fix(#4129): derive completed_phases from the ROADMAP authority; honor the progress-ratchet on every state write (#4359)

* test(#4129): failing-first regressions — completed_phases clobber on resyncing writes and phase-complete failure to increment

* fix(#4129): completed_phases derives from the ROADMAP authority and the write path honors the progress-ratchet

Three coordinated prongs (diagnosis in .gsd/bug/fix-4129-completed-phases-recompute/):

P1 — buildStateFrontmatter's disk scan floors the completed-phases numerator at
the milestone-scoped ROADMAP Complete-row count (deriveProgressFromRoadmap, the
one owner), gated inside the same safeToUseRoadmapCount / not-withheld branch
that owns the denominator. A completed phase whose verification routes stale
(#2348 clean-commit-time drift) or is missing no longer under-counts forever.

P2 — applyPreserveAlways's resync arm merges instead of wholesale-replacing on
a measured scan: totals derived both directions (#2440), completed counters
up-only (#2969 — the schema-declared progress-ratchet, now enforced on the
write path like the read path always has), percent recomputed from the merged
counters. The #3756 unmeasured guard and the #3242 explicit-progress contract
are unchanged.

P3 — phase complete's atomic 3-file commit passes the post-completion
ROADMAP-derived counters through the #2736 authoritativeFm seam (new object
direction for the progress key; completedOnlyRaise at the post-preservation
re-assert), because the transaction's disk scan reads the pre-completion
ROADMAP and failed to increment on the completing phase's own write.

* fix(#4129): adversarial-review hardening — intent is a floor at BOTH authoritativeFm sites

The pre-preservation merge could lower a correctly-higher disk-derived
counter (a verification-passed phase whose ROADMAP table row drifted behind
the disk signal). completedOnlyRaise now governs both application sites: the
intent and the derivation agree on direction (up), never on subtraction.

* fix(#4129): the ratchet merge keeps derived values verbatim when numerically equal

The re-parsed derived block carries string scalars ("2") while the curated
snapshot carries numbers (2); substituting the curated spelling over an
equal derived one was a no-op in substance but a shape churn the ADR-3473
§8.7 reporting loop surfaced as a phantom preserved-over-disagreeing-derived
warning on phase complete (ADR-3408 §8.5 Matrix B). Only a strictly-greater
curated counter replaces the derived value now; percent gets the same
verbatim rule.

* changeset(#4129): backfill PR 4359

---------

Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
2026-09-05 23:02:07 -04:00
committed by GitHub
parent bd75d42f52
commit e6d047decc
7 changed files with 827 additions and 34 deletions

View File

@@ -59,6 +59,12 @@ const { findPhaseInternal, getArchivedPhaseDirs, listMilestonePhaseDirs } = phas
// eslint-disable-next-line @typescript-eslint/no-require-imports -- roadmap-parser.cjs is an export= CommonJS module
import roadmapParserMod = require('./roadmap-parser.cjs');
const { stripShippedMilestones, extractCurrentMilestone, currentMilestoneRawRanges, withPhaseSection, findMilestoneScopeHeadingLines } = roadmapParserMod;
// #4129: the single owner of "count the ROADMAP's milestone Complete rows"
// (pure computation, no I/O — no cycle on this path) for the intent-first
// progress counters the phase-complete transaction passes downstream.
// eslint-disable-next-line @typescript-eslint/no-require-imports -- phase-lifecycle.cjs is an export= CommonJS module
import phaseLifecycleMod = require('./phase-lifecycle.cjs');
const { deriveProgressFromRoadmap: deriveProgressFromRoadmapForIntent, clampPercent: clampPercentForIntent } = phaseLifecycleMod;
// eslint-disable-next-line @typescript-eslint/no-require-imports -- planning-workspace.cjs is an export= CommonJS module
import planningWorkspace = require('./planning-workspace.cjs');
// eslint-disable-next-line @typescript-eslint/no-require-imports -- frontmatter.cjs is an export= CommonJS module
@@ -4395,14 +4401,57 @@ function cmdPhaseComplete(cwd: string, phaseNum: string, raw: boolean): void {
const bodyHasPhaseField =
stateExtractField(fmBody, 'Current Phase') != null ||
stateExtractField(fmBody, 'Phase') != null;
const authoritativeFm: Record<string, string> | undefined = nextPhaseDisplayName
? bodyHasPhaseField || !nextPhaseNum
? { current_phase_name: nextPhaseDisplayName }
: {
current_phase: String(nextPhaseNum),
current_phase_name: nextPhaseDisplayName,
}
: undefined;
// #4129: the POST-completion progress counters, derived from the very
// ROADMAP this transaction just mutated (still in memory — it hits disk
// only at writePlanningFileSet, AFTER this content was assembled).
// buildStateFrontmatter's disk scan inside syncAndPreserveStateMd
// reads the PRE-completion ROADMAP (and any stale-dated sibling
// verification), so without this intent the persisted counter failed
// to increment on the completing phase's own transaction. Routed
// through the #2736 authoritativeFm seam's object direction: the
// pre-preservation merge makes it the derived truth the ratchet
// compares, and the post-preservation re-assert (completedOnlyRaise)
// is a floor no preservation branch can drop below. clampPercent is
// completePhaseCore's own percent formula (state-transition.cts),
// reused so the frontmatter and the body `Progress:` line agree.
const postCompletionRoadmapScope = roadmapContent !== null
? extractCurrentMilestone(roadmapContent, cwd)
: null;
const postCompletionRoadmapProgress = postCompletionRoadmapScope !== null
? deriveProgressFromRoadmapForIntent(postCompletionRoadmapScope)
: null;
const authoritativeProgress: Record<string, number> | undefined =
postCompletionRoadmapProgress && postCompletionRoadmapProgress.completedPhases !== null
? postCompletionRoadmapProgress.totalPhases !== null && postCompletionRoadmapProgress.totalPhases > 0
? {
completed_phases: postCompletionRoadmapProgress.completedPhases,
percent: clampPercentForIntent(
postCompletionRoadmapProgress.completedPhases,
postCompletionRoadmapProgress.totalPhases,
),
}
: { completed_phases: postCompletionRoadmapProgress.completedPhases }
: undefined;
const authoritativeFm: Record<string, unknown> | undefined = authoritativeProgress
? {
...(nextPhaseDisplayName
? bodyHasPhaseField || !nextPhaseNum
? { current_phase_name: nextPhaseDisplayName }
: {
current_phase: String(nextPhaseNum),
current_phase_name: nextPhaseDisplayName,
}
: {}),
progress: authoritativeProgress,
}
: nextPhaseDisplayName
? bodyHasPhaseField || !nextPhaseNum
? { current_phase_name: nextPhaseDisplayName }
: {
current_phase: String(nextPhaseNum),
current_phase_name: nextPhaseDisplayName,
}
: undefined;
// ADR-3408 §8.3 / #3469: this deliberately bypasses
// readModifyWriteStateMd (STATE.md is committed atomically with
// ROADMAP/REQUIREMENTS), so it calls the single write-seam

View File

@@ -17,11 +17,16 @@
// eslint-disable-next-line @typescript-eslint/no-require-imports
import frontmatter = require('./frontmatter.cjs');
import { stateReplaceField, stateExtractField, stateReplaceFieldIfTemplate, stateReplaceFieldWithFallback, stateReplaceFieldInSession, stateCurrentPositionSlice } from './state-document.cjs';
import { KNOWN_TEMPLATE_DEFAULTS, toFiniteNumber } from './state-document.cjs';
import { KNOWN_TEMPLATE_DEFAULTS, toFiniteNumber, computeProgressPercent } from './state-document.cjs';
import { tokenizeHeadings } from './markdown-sectionizer.cjs';
import type { HeadingToken } from './markdown-sectionizer.cjs';
import { deriveProgressFromRoadmap, clampPercent, clampPercentFromFraction } from './phase-lifecycle.cjs';
import { escapeRegex } from './pattern.cjs';
// #4129: the completion-ratio kernel for the resync-arm ratchet's percent
// (planning-scope's SCOPE — state-document's own dependency, no cycle here:
// state-document never imports this module).
// eslint-disable-next-line @typescript-eslint/no-require-imports
import planningScopeMod = require('./planning-scope.cjs');
// eslint-disable-next-line @typescript-eslint/no-require-imports
import stateMdSchemaMod = require('./state-md-schema.cjs');
const { STATE_FIELD_SCHEMA } = stateMdSchemaMod;
@@ -684,6 +689,78 @@ function preservedValuesEqual(a: unknown, b: unknown): boolean {
return a === b;
}
/**
* #4129: the resync-arm progress merge. A resyncing write whose scan MEASURED
* something no longer wholesale-replaces the curated block — the declared
* `progress-ratchet` mergeStrategy ("completed_plans/completed_phases only ever
* ratchet UP toward the derived value (#2969)", state-md-schema.cts) now holds
* on the write path too, matching what the read path (`shouldPreserveExistingProgress`)
* has always enforced. Rules, mirroring the `deriveProgressKeys` branch above:
*
* - total_plans / total_phases always take the derived value (#2440 — totals
* correct in BOTH directions).
* - completed_plans / completed_phases take the derived value only when it is
* strictly GREATER (#2969's `>` not `>=`); else the curated value survives
* (a hand-corrected or previously-correct counter can never be re-derived
* downward — the #4129 clobber).
* - any other key keeps the curated value (the existing branch's convention).
* - percent is RECOMPUTED from the merged counters through the single kernel
* (`computeProgressPercent`), because either side's stored percent was
* computed against that side's counters and the merged block may mix them
* (curated completed, derived totals). Recomputation runs ONLY when the
* derived block itself carried a percent — an upstream withhold
* (#1761 milestone-unbounded, #3217 scope) nulled percent deliberately and
* this merge must not resurrect it.
*
* Frontmatter scalars arrive as STRINGS ("2", not 2), so every comparison
* coerces through `toFiniteNumber` — never a `typeof === 'number'` test
* (scanMeasuredSomething's own convention).
*/
function mergeResyncProgressRatchet(
curatedRecord: Record<string, unknown>,
derivedRecord: Record<string, unknown>,
): Record<string, unknown> {
const merged: Record<string, unknown> = { ...derivedRecord };
for (const [key, value] of Object.entries(curatedRecord)) {
if (key === 'total_plans' || key === 'total_phases' || key === 'percent') continue;
if (key === 'completed_plans' || key === 'completed_phases') {
const derivedNum = toFiniteNumber(derivedRecord[key]) ?? -Infinity;
const curatedNum = toFiniteNumber(value) ?? -Infinity;
// Ratchet up only (strictly greater, #2969); else keep curated.
if (derivedNum > curatedNum) continue;
// Numerically EQUAL keeps the derived value VERBATIM. The two sides
// arrive in different scalar shapes (the re-parsed derived block
// carries string totals "2" while the curated snapshot carries numbers
// 2), and substituting the curated spelling over an equal derived one
// is a no-op in substance but a shape churn the §8.7 reporting loop
// would surface as a phantom `preserved-over-disagreeing-derived`
// warning (it diffs structurally). Only a curated counter that is
// STRICTLY greater replaces the derived value.
if (derivedNum === curatedNum) continue;
merged[key] = value;
} else {
merged[key] = value;
}
}
if (toFiniteNumber(derivedRecord.percent) !== null) {
const recomputed = computeProgressPercent(
toFiniteNumber(merged.completed_plans),
toFiniteNumber(merged.total_plans),
toFiniteNumber(merged.completed_phases),
toFiniteNumber(merged.total_phases),
planningScopeMod.SCOPE.COMPLETE,
);
// Same verbatim rule for percent: assign only when the recomputed value
// numerically differs, so a string-spelled derived percent ("67") is not
// churned into a number-spelled 67 (phantom-divergence noise, not a
// change).
if (recomputed !== null && recomputed !== toFiniteNumber(merged.percent)) {
merged.percent = recomputed;
}
}
return merged;
}
/**
* Executor for `preservation: 'preserve-always'` (ADR-3408 §8.1). Only
* `progress` carries this policy today. Preserves #3242/#1446/#2440/#2969
@@ -698,25 +775,41 @@ function applyPreserveAlways(field: string, cls: FieldClassification, ctx: Prese
const derived = ctx.postFm[field];
const derivedMeasured = scanMeasuredSomething(cls, derived);
const curatedMeasured = scanMeasuredSomething(cls, curated);
// On a resyncing write the fresh derivation is authoritative — UNLESS it
// measured nothing while the curated block did (#3756), AND the caller did
// not explicitly name a progress-affecting field this write. The
// unmeasured-scan guard exists to stop an INCIDENTAL resync (e.g. `state
// add-decision`, whose `resync` defaults true for reasons that have
// nothing to do with `progress`) from dropping a real curated block when a
// milestone-scoped disk scan measures nothing (#3756's archived-milestone
// case). It must not also block a write the user pointed AT `progress` on
// purpose: `preserve-always`'s own contract is "never overwrite unless the
// caller explicitly names this field" (FIELD_CLASSIFICATION doc comment),
// and `state update Progress` / `state patch Progress=...` are exactly
// that naming — the resync they trigger must win even when the disk scan
// it also drives (e.g. because there are no phase dirs at all) reads as
// "unmeasured" (tests/frontmatter.test.cjs: "state.update \"Progress\"
// resyncs progress frontmatter from the updated body", pre-existing, #3242).
if (ctx.resync && (derivedMeasured || !curatedMeasured || ctx.explicitProgressField)) return;
// On a resyncing write the fresh derivation is authoritative in two cases
// (#3756 / ADR-3473 §8.6, unchanged): when the caller EXPLICITLY named a
// progress-affecting field (`preserve-always`'s contract is "never
// overwrite unless the caller explicitly names this field" — `state update
// Progress` is exactly that naming, pre-existing #3242 behavior), and when
// the derivation measured something the curated block did not (an
// unmeasured CURATED block is not worth protecting). The unmeasured-DERIVED
// guard also stands: an incidental resync (e.g. `state add-decision`, whose
// `resync` defaults true for reasons that have nothing to do with
// `progress`) that measured nothing must not drop a real curated block
// (#3756's archived-milestone case) — that falls through to the wholesale
// restore below.
//
// #4129 narrows the remaining arm. A resyncing write whose scan MEASURED
// something while the curated block is also real previously wholesale-
// replaced the curated block with the derived one — no monotonic guard, so
// any under-counting derivation (a stale-dated verification, #2348) silently
// reverted every hand-correction and every correct value an earlier write
// had persisted, while the read path (`shouldPreserveExistingProgress`)
// kept reporting the higher stored counters. That arm now falls through to
// `mergeResyncProgressRatchet` — the declared `progress-ratchet`
// mergeStrategy, finally enforced on the write path: totals derived both
// directions (#2440), completed counters up-only (#2969), percent
// recomputed from the merged counters.
if (ctx.resync && (ctx.explicitProgressField || (derivedMeasured && !curatedMeasured))) return;
let next: unknown;
if (cls.mergeStrategy === 'progress-ratchet' && ctx.deriveProgressKeys && derived && derivedMeasured) {
if (cls.mergeStrategy === 'progress-ratchet' && ctx.resync && derived && derivedMeasured && curatedMeasured) {
// #4129 resync arm — see mergeResyncProgressRatchet's doc. Reached only
// after the early-out above, so curatedMeasured is guaranteed true here.
next = mergeResyncProgressRatchet(
curated as Record<string, unknown>,
(derived ?? {}) as Record<string, unknown>,
);
} else if (cls.mergeStrategy === 'progress-ratchet' && ctx.deriveProgressKeys && derived && derivedMeasured) {
// #2440: total_plans and total_phases always take the derived (post-sync)
// value even under !resync. This is used by cmdStatePlannedPhase where
// total_plans must correct upward after plans are added. For body-only

View File

@@ -69,6 +69,13 @@ import planDependencyGraphMod = require('./plan-dependency-graph.cjs');
// eslint-disable-next-line @typescript-eslint/no-require-imports
import verificationMod = require('./verification.cjs');
const { isPhaseComplete } = verificationMod;
// #4129: the single owner of "count the ROADMAP's milestone Complete rows"
// (phase-lifecycle.cts) — reused for the completed-phases numerator floor so
// this scan cannot grow a second ROADMAP parser. Pure computation module (no
// I/O), so it introduces no cycle on this path.
// eslint-disable-next-line @typescript-eslint/no-require-imports
import phaseLifecycleMod = require('./phase-lifecycle.cjs');
const { deriveProgressFromRoadmap } = phaseLifecycleMod;
// eslint-disable-next-line @typescript-eslint/no-require-imports
import planningScopeMod = require('./planning-scope.cjs');
const { SCOPE } = planningScopeMod;
@@ -3014,6 +3021,34 @@ function buildStateFrontmatter(
// write silently clobbered the three stored siblings with the
// under-scoped disk numbers.
const diskCountsWithheld = milestonedButUnbounded || roadmapAbsentWithAssertedMilestone;
// #4129: floor the completed-phases numerator at the ROADMAP's own
// milestone Complete-row count. The disk numerator counts ONLY
// phase dirs whose *-VERIFICATION.md routes `passed` (isPhaseComplete,
// #2957 disk-strict — the gate stays untouched), so a completed
// phase whose verification reads `stale` (a SUMMARY committed or
// edited after it, #2348 clean-commit-time clock) or `missing`
// (pre-verification era, hand-flipped ROADMAP row) drops out of the
// count forever — while every other surface (the ROADMAP row
// `phase complete` just flipped, the body `Completed Phases` field
// completePhaseCore derives from deriveProgressFromRoadmap) still
// asserts the phase complete. max(disk, ROADMAP) keeps the disk
// signal for gap detection (a verification-passed phase whose ROADMAP
// row is not yet flipped still counts) while never UNDER-counting
// what the ROADMAP asserts. Scoped exactly like the denominator:
// the same milestone window (roadmapScope), the same
// safeToUseRoadmapCount gate, and never under the #3354/#3573
// withhold — a whole-document Complete-row count must not leak
// through an untrustworthy scope. Reuses deriveProgressFromRoadmap
// (phase-lifecycle.cts, the one owner of "read the Progress table")
// — no second ROADMAP parser here. A ROADMAP without a canonical
// `## Progress` table resolves no table → floor inert (disk count
// stands), the owner's own answer to "what is countable".
const roadmapCompletedPhases = roadmapScope !== null && safeToUseRoadmapCount && !diskCountsWithheld
? deriveProgressFromRoadmap(roadmapScope).completedPhases
: null;
const flooredCompletedPhases = roadmapCompletedPhases !== null
? Math.max(diskCompletedPhases, roadmapCompletedPhases)
: diskCompletedPhases;
return {
// The two WITHHOLD shapes (#3354 milestoned-but-unbounded, #3573
// roadmap-absent-with-asserted-milestone) must be evaluated BEFORE
@@ -3024,7 +3059,7 @@ function buildStateFrontmatter(
? null
: (safeToUseRoadmapCount ? Math.max(phaseDirs.length, roadmapPhaseCount) : phaseDirs.length),
milestoneBounded,
completedPhases: diskCountsWithheld ? null : diskCompletedPhases,
completedPhases: diskCountsWithheld ? null : flooredCompletedPhases,
totalPlans: diskCountsWithheld ? null : diskTotalPlans,
completedPlans: diskCountsWithheld ? null : diskTotalSummaries,
phaseDirScope,
@@ -3395,6 +3430,76 @@ function readStoredCompletedPlans(existingFm: Record<string, unknown> | null | u
return readStoredProgressCounter(existingFm, 'completed_plans');
}
/**
* #4129: is this authoritativeFm value a PARTIAL progress intent? The #2736
* seam was string-only (names); #4129 extends it with one object direction —
* the `progress` key carrying the sub-keys a transition resolved
* authoritatively (completePhase's ROADMAP-derived completed_phases/percent).
* Anything else keeps the seam's existing contract untouched.
*/
function isPartialProgressIntent(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value);
}
/**
* #4129: merge a PARTIAL progress intent (see isPartialProgressIntent) into a
* frontmatter object's `progress` block. Sub-keys are accepted only when they
* are a declared `progress.*` row in FIELD_CLASSIFICATION — the single policy
* source (ADR-3408 §8.5) decides which leaves exist; an intent may not invent
* one. Returns whether anything changed.
*
* `completedOnlyRaise` (both application sites use it): completed counters
* apply only when strictly greater than what is already in the block, so no
* intent can LOWER a count another trustworthy signal already established —
* at the pre-preservation site the disk derivation's own count (a
* verification-passed phase whose ROADMAP row drifted behind), at the
* post-preservation re-assert the #2969 monotonic property preservation just
* enforced. `percent` follows its sibling: it is applied when a completed
* counter moved this call (the intent percent was computed from the intent
* counters and is coherent with them) or when the block has no percent to
* lose (a repair, never a regression of an upstream withhold — the withhold
* nulled percent upstream precisely so no write would re-assert one over
* untrustworthy counts; here the intent's own counts ARE the trustworthy
* source, the post-completion ROADMAP).
*/
function applyAuthoritativeProgressSubkeys(
fm: Record<string, unknown>,
intent: Record<string, unknown>,
opts: { completedOnlyRaise: boolean },
): boolean {
const current = fm['progress'];
const base: Record<string, unknown> = isPartialProgressIntent(current)
? { ...current }
: {};
let changed = false;
let completedMoved = false;
for (const [subkey, value] of Object.entries(intent)) {
if (typeof value !== 'number' || !Number.isFinite(value)) continue;
if (!getFieldClassification(`progress.${subkey}`)) continue;
const isCompletedCounter = subkey === 'completed_phases' || subkey === 'completed_plans';
if (isCompletedCounter && opts.completedOnlyRaise) {
const currentNum = toFiniteNumber(base[subkey]);
if (currentNum !== null && currentNum >= value) continue;
}
if (isCompletedCounter && !Object.is(base[subkey], value)) completedMoved = true;
if (!Object.is(base[subkey], value)) {
base[subkey] = value;
changed = true;
}
}
// percent: applied only when a completed counter moved (coherent with the
// counters that just landed) or when no percent exists to contradict.
const intentPercent = intent['percent'];
if (typeof intentPercent === 'number' && Number.isFinite(intentPercent) && (completedMoved || toFiniteNumber(base['percent']) === null)) {
if (!Object.is(base['percent'], intentPercent)) {
base['percent'] = intentPercent;
changed = true;
}
}
if (changed) fm['progress'] = base;
return changed;
}
function syncStateFrontmatter(
content: string,
cwd: string | undefined,
@@ -3620,10 +3725,20 @@ function syncStateFrontmatter(
// parenthetical (`Closer-ruling measurement (D1a)` → `D1a`) — never runs
// the final word on a field the transition just resolved. The prose parser
// remains the fallback for genuinely unknown prose only.
// #4129: the `progress` key carries a PARTIAL block (the object direction of
// this seam — see applyAuthoritativeProgressSubkeys) for the same reason:
// completePhase holds the POST-completion ROADMAP, and the disk scan this
// function drives reads the PRE-completion one. The intent is applied as a
// FLOOR here too (completedOnlyRaise): a derivation that already counted
// MORE completed phases than the ROADMAP table asserts (verification-passed
// phases whose table rows drifted behind) must not be lowered by the intent
// — the two signals agree on direction (up), never on subtraction.
if (authoritativeFm) {
for (const [key, value] of Object.entries(authoritativeFm)) {
if (typeof value === 'string' && value.trim().length > 0) {
derivedFm[key] = value;
} else if (key === 'progress' && isPartialProgressIntent(value)) {
applyAuthoritativeProgressSubkeys(derivedFm, value, { completedOnlyRaise: true });
}
}
}
@@ -4194,12 +4309,20 @@ function applyPostSyncPreservation(
// (equal), so the #1695 restore fires and would put the stale pre-transition
// name back over the authoritative one. Intent beats both the prose
// re-derivation and the curated restore — the transition just resolved it.
// #4129: for the `progress` key the re-assert is a FLOOR, not an override —
// the #2969 monotonic property preservation just enforced (completed
// counters never move down) must not be undone by the intent, so completed
// sub-keys apply only-raise here (see applyAuthoritativeProgressSubkeys).
let authoritativeReasserted = false;
if (authoritativeFm) {
for (const [key, value] of Object.entries(authoritativeFm)) {
if (typeof value === 'string' && value.trim().length > 0 && preservation.postFm[key] !== value) {
preservation.postFm[key] = value;
authoritativeReasserted = true;
} else if (key === 'progress' && isPartialProgressIntent(value)) {
if (applyAuthoritativeProgressSubkeys(preservation.postFm, value, { completedOnlyRaise: true })) {
authoritativeReasserted = true;
}
}
}
}