golem15/msd-core
3
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

feat(#2632): executor actuals and the closed estimate-calibration loop (#2672)

Browse Source 
* feat(#2632): record executor actuals and close the estimate calibration loop

* fix(#2632): calibrate against the raw projection so the loop converges

* test(#2632): add closed-loop convergence guard and codify the feedback-loop rule

* fix(#2632): pair calibration samples per plan; atomic write; amend adr

* chore(#2632): backfill changeset pr to 2672

* fix(#2632): retry renameSync on transient windows errnos and clean up the temp
This commit is contained in:
Tom Boucher
2026-07-26 16:28:56 -04:00
committed by GitHub
parent 920a5f3f06
commit bd570618d4
37 changed files with 943 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/adr/2629-phase-effort-estimation-calibration.md
View File

@@ -66,7 +66,7 @@ The effective ceiling is model-, task-, and distractor-dependent. **100k is a co
### 4. Calibration: median ratio, clamped, with a minimum sample count
```
ratio_i = actuals_i.tokens / estimate_i.tokens (phases with BOTH fields)
ratio_i = actuals_i.tokens / estimate_i.raw_tokens (per PLAN, with BOTH fields)
factor = clamp(median(ratio_i), 0.5, 3.0) when n >= 3
factor = 1.0 when n < 3
```
@@ -74,6 +74,8 @@ factor = 1.0 when n < 3
- **Median, not mean** — one pathological phase (an aborted run, a mass rename) must not swing the projection for every later phase.
- **Clamped to [0.5, 3.0]** — bounds the blast radius of a degenerate history; a factor outside that range indicates the estimator is wrong in kind, not in degree, and should be fixed rather than amplified.
- **`n >= 3` before any correction applies** — below that, the sample says more about variance than about bias.
- **The denominator is the RAW projection, not the emitted (already-corrected) figure** — amended #2632. Measuring `actual / calibrated` is self-defeating: once the correction works the observed ratio approaches 1, which drags the median back toward 1 and un-corrects the next estimate. Simulated over 10 phases against a true 2x miss it oscillates and settles near 1.41 instead of converging on 2.0. Plans therefore record `estimate.raw_tokens` alongside the calibrated `estimate.tokens`, and `calibrationBasis()` prefers it (falling back to `tokens` for plans written before #2632, where no factor had yet been applied).
- **Samples are per PLAN, not per phase** — amended #2632. A phase holds several `<NN>-<PP>-PLAN.md` files; pairing at phase granularity cross-pairs one plan's projection with another's cost and discards the rest.
Persisted to `.planning/estimation-calibration.json` with a `schema_version` field, written by `extract-learnings`, read at plan time. Versioned from the first write so the schema can migrate without a silent misread.