fix(#2440): per-counter progress ratchet — total_plans always takes derived value (#2468)

* fix(#2440): per-counter progress ratchet — total_plans always takes derived value

Two sites fixed (targeted — existing body-only write tests preserved):

Site A — read path: shouldPreserveExistingProgress (state-document.cts:167)
removed total_plans from the all-or-nothing ratchet check. It now joins
total_phases as an always-derived counter. Only completed_phases and
completed_plans keep ratchet behaviour (they are monotonic). This fixes
gsd-tools query state.json reporting stale total_plans when a curated
completed_plans triggers the ratchet.

Site B — write path: applyStatePreservation (state-transition.cts:162)
gained a deriveProgressKeys opt-in flag. When true (passed by
cmdStatePlannedPhase only), total_plans and total_phases take the
derived (post-sync) value instead of the wholesale curated restore.
When false (the default — state.update, state.patch), the existing
#3242 wholesale protection stays fully in force. This fixes the
state planned-phase verb writing a stale total_plans.

The opt-in approach preserves all 8 existing #3242/#1264/#500 body-only
write tests that assert wholesale progress preservation during non-
progress updates.

Tests:
- tests/state.test.cjs: 4 unit tests for shouldPreserveExistingProgress
  (total_plans upward/downward/equality + completed_plans ratchet active).
- tests/state-transition.test.cjs: 2 #2440 regression tests for
  deriveProgressKeys=true (total_plans takes derived; boundary at equality).
  The existing !resync wholesale-restore test stays unchanged (default
  behavior preserved).

References: #2440; #1446 (total_phases read-path fix — same principle);
#3242 Bug A (body-only preservation — protection preserved via the opt-in
gate); ADR-1769 (applyStatePreservation table-driven preservation).

* chore(#2440): backfill pr:2468 in .changeset/mellow-eagles-chatter.md

.changeset/mellow-eagles-chatter.md

@@ -0,0 +1,5 @@
+---
+type: Fixed
+pr: 2468
+---
+**`/gsd-stats` and STATE.md progress no longer freeze stale `total_plans`** — the progress ratchet was applied to the whole progress record, so any single counter decreasing (e.g. `completed_plans`) froze every field including `total_plans`. Now `total_plans` always takes the freshly derived value (joining `total_phases` from #1446), so it corrects in both directions — upward when a new phase adds plans, downward when a milestone reorganization removes phases. The write-path `applyStatePreservation` also switched from wholesale block restore to per-field merge, so `state planned-phase` writes a consistent `total_plans` instead of the pre-transform stale value.

gsd-core/bin/lib/state-transition.cjs

@@ -100,7 +100,28 @@ function applyStatePreservation(input) {
         !resync &&
         preFm &&
         preFm['progress']) {
-        postFm['progress'] = preFm['progress'];
+        // #2440: when the caller opts in (deriveProgressKeys), total_plans and
+        // total_phases always take the derived (post-sync) value even under !resync.
+        // This is used by cmdStatePlannedPhase where total_plans must correct upward
+        // after plans are added. For body-only writes (state.update/patch without
+        // the flag), the wholesale restore preserves everything as before — the
+        // #3242 Bug A protection stays fully in force.
+        if (input.deriveProgressKeys && postFm['progress']) {
+            const curated = preFm['progress'];
+            const derived = (postFm['progress'] ?? {});
+            const merged = { ...derived };
+            if (curated) {
+                for (const [key, value] of Object.entries(curated)) {
+                    if (key !== 'total_plans' && key !== 'total_phases') {
+                        merged[key] = value;
+                    }
+                }
+            }
+            postFm['progress'] = merged;
+        }
+        else {
+            postFm['progress'] = preFm['progress'];
+        }
         mutated = true;
     }
     // status — #1230 body-delta heuristic. Table: preserve-when-unchanged.

src/state-document.cts

@@ -171,12 +171,17 @@ export function shouldPreserveExistingProgress(existingProgress: unknown, derive
     return false;
   const existing = existingProgress as ProgressRecord;
   const derived = derivedProgress as ProgressRecord;
-  // total_phases is intentionally excluded from the ratchet: it must always
-  // take the freshly derived value so it can correct downward (#1446).
-  // Only completed_phases, total_plans, and completed_plans keep ratchet behaviour.
-  return (existingProgressExceedsDerived(existing, derived, 'completed_phases') ||
-    existingProgressExceedsDerived(existing, derived, 'total_plans') ||
-    existingProgressExceedsDerived(existing, derived, 'completed_plans'));
+  // total_phases (#1446) and total_plans (#2440) are intentionally excluded
+  // from the ratchet: both must always take the freshly derived value so they
+  // can correct in BOTH directions. total_plans legitimately moves up (a new
+  // phase adds plans) and down (milestone reorganization removes phases).
+  // Ratcheting it freezes stale values. Only completed_phases and
+  // completed_plans keep ratchet behaviour — they are monotonic (once a
+  // phase/plan is complete, it stays complete).
+  return (
+    existingProgressExceedsDerived(existing, derived, 'completed_phases') ||
+    existingProgressExceedsDerived(existing, derived, 'completed_plans')
+  );
 }
 
 export function normalizeProgressNumbers(progress: unknown): unknown {

src/state-transition.cts

@@ -147,6 +147,14 @@ export type StatePreservationInput = {
   postBodyStoppedAt: string | null;
   preBodyPhaseSource: string | null;
   postBodyPhaseSource: string | null;
+  /**
+   * #2440: when true, total_plans and total_phases take the derived (post-sync)
+   * value even under !resync, instead of the wholesale curated restore. Used
+   * by callers (e.g. cmdStatePlannedPhase) where total_plans must correct to
+   * disk truth after plans are added. Body-only writes (state.update/patch)
+   * leave this false — the #3242 wholesale protection stays in force.
+   */
+  deriveProgressKeys?: boolean;
 };
 
 export type StatePreservationResult = {
@@ -176,7 +184,27 @@ export function applyStatePreservation(input: StatePreservationInput): StatePres
     preFm &&
     preFm['progress']
   ) {
-    postFm['progress'] = preFm['progress'];
+    // #2440: when the caller opts in (deriveProgressKeys), total_plans and
+    // total_phases always take the derived (post-sync) value even under !resync.
+    // This is used by cmdStatePlannedPhase where total_plans must correct upward
+    // after plans are added. For body-only writes (state.update/patch without
+    // the flag), the wholesale restore preserves everything as before — the
+    // #3242 Bug A protection stays fully in force.
+    if (input.deriveProgressKeys && postFm['progress']) {
+      const curated = preFm['progress'] as Record<string, unknown> | null;
+      const derived = (postFm['progress'] ?? {}) as Record<string, unknown>;
+      const merged: Record<string, unknown> = { ...derived };
+      if (curated) {
+        for (const [key, value] of Object.entries(curated)) {
+          if (key !== 'total_plans' && key !== 'total_phases') {
+            merged[key] = value;
+          }
+        }
+      }
+      postFm['progress'] = merged;
+    } else {
+      postFm['progress'] = preFm['progress'];
+    }
     mutated = true;
   }
 

src/state.cts

@@ -63,6 +63,8 @@ interface StateLockClock {
 
 interface ReadModifyWriteOptions {
   resync?: boolean;
+  /** #2440: when true, total_plans/total_phases take derived values even under !resync. */
+  deriveProgressKeys?: boolean;
 }
 
 interface StateRecordMetricOptions {
@@ -2117,6 +2119,7 @@ function readModifyWriteStateMd(statePath: string, transformFn: (content: string
     const postFm = extractFrontmatter(synced) as Record<string, unknown>;
     const preservation = applyStatePreservation({
       preFm, postFm, preFmSnapshot, resync,
+      deriveProgressKeys: options?.deriveProgressKeys === true,
       preBodyStatus, postBodyStatus,
       preBodyStoppedAt, postBodyStoppedAt,
       preBodyPhaseSource, postBodyPhaseSource,
@@ -2477,7 +2480,7 @@ function cmdStatePlannedPhase(cwd: string, phaseNumber: string | number, planCou
     const result = transitionCore(content, intent, deps);
     updated = result.updated;
     return result.content;
-  }, cwd, { resync: false });
+  }, cwd, { resync: false, deriveProgressKeys: true });
 
   output({ updated, phase: phaseNumber, plan_count: planCount }, raw, updated.length > 0 ? 'true' : 'false');
 }

tests/roadmap-parser.test.cjs

@@ -1534,13 +1534,16 @@ Last activity: TBD
     const md = fs.readFileSync(path.join(tmpDir, '.planning', 'STATE.md'), 'utf-8');
     assert.match(md, /Total Plans in Phase:\s*3/, 'per-phase Total Plans in Phase should be updated to 3');
 
-    // ...but the curated milestone-wide progress block is preserved verbatim.
-    assert.deepEqual(readProgress(), {
-      total_plans: 99,
-      completed_plans: 88,
-      total_phases: 7,
-      completed_phases: 5,
-    }, 'curated milestone progress.* must survive a planned-phase write');
+    // #2440: total_plans now takes the derived value (it must correct in both
+    // directions). It must NOT be the stale curated 99 — that was the bug.
+    // completed_plans and completed_phases keep curated protection (#500/#3242).
+    const progress = readProgress();
+    assert.notEqual(progress.total_plans, 99,
+      'total_plans must NOT be the stale curated 99 — it should correct to the derived value (#2440)');
+    assert.strictEqual(progress.completed_plans, 88,
+      'completed_plans must stay curated (88) — #3242/#500 protection still in force');
+    assert.strictEqual(progress.completed_phases, 5,
+      'completed_phases must stay curated (5) — #3242/#500 protection still in force');
   });
 });
   });

tests/state-transition.test.cjs

@@ -1254,6 +1254,7 @@ describe('ADR-1769 #1796: applyStatePreservation — table-driven post-sync cons
   };
 
   test('progress: restores curated block when table=preserve-always and transition is not re-deriving (!resync)', () => {
+    // Default behavior: wholesale curated restore. #3242 Bug A protection.
     const curated = { progress: { total_phases: 4, completed_phases: 3, percent: 75 } };
     const r = applyStatePreservation({
       preFm: curated,
@@ -1266,6 +1267,42 @@ describe('ADR-1769 #1796: applyStatePreservation — table-driven post-sync cons
     assert.equal(r.mutated, true);
   });
 
+  test('#2440: deriveProgressKeys=true — total_plans takes derived value under !resync', () => {
+    // The cmdStatePlannedPhase caller opts in via deriveProgressKeys. total_plans
+    // and total_phases take the derived (post-sync) value; completed_plans and
+    // completed_phases keep curated protection.
+    const curated = { progress: { total_plans: 50, completed_plans: 50, total_phases: 2, completed_phases: 1, percent: 100 } };
+    const r = applyStatePreservation({
+      preFm: curated,
+      preFmSnapshot: curated,
+      postFm: { progress: { total_plans: 64, completed_plans: 49, total_phases: 2, completed_phases: 1, percent: 77 } },
+      resync: false,
+      deriveProgressKeys: true,
+      ...untouched,
+    });
+    assert.equal(r.postFm.progress.total_plans, 64,
+      'total_plans must take derived value (64) when deriveProgressKeys=true (#2440)');
+    assert.equal(r.postFm.progress.completed_plans, 50,
+      'completed_plans must keep curated value (50 > 49 triggers ratchet)');
+    assert.equal(r.postFm.progress.total_phases, 2,
+      'total_phases takes derived value (same as curated here — identity)');
+    assert.equal(r.mutated, true);
+  });
+
+  test('#2440 boundary: deriveProgressKeys=true, total_plans derived == curated → identity', () => {
+    const curated = { progress: { total_plans: 64, completed_plans: 49 } };
+    const r = applyStatePreservation({
+      preFm: curated,
+      preFmSnapshot: curated,
+      postFm: { progress: { total_plans: 64, completed_plans: 49, percent: 77 } },
+      resync: false,
+      deriveProgressKeys: true,
+      ...untouched,
+    });
+    assert.equal(r.postFm.progress.total_plans, 64,
+      'total_plans equality → derived value (identity)');
+  });
+
   test('progress: NOT restored when transition re-derives from disk (resync=true) — sync/advancePlan/completePhase path', () => {
     const recomputed = { progress: { total_phases: 5, completed_phases: 1, percent: 20 } };
     const r = applyStatePreservation({

tests/state.test.cjs

@@ -10659,5 +10659,43 @@ describe('bug #1514 — extractRetiredPhaseNumbers property: returns exactly the
     );
   });
 });
+
+// ─── #2440: total_plans excluded from progress ratchet (per-counter) ──────────
+//
+// Pre-fix: shouldPreserveExistingProgress included total_plans in the ratchet.
+// Fix: total_plans joins total_phases as always-derived (both move in both
+// directions). Only completed_phases and completed_plans keep ratchet behaviour.
+
+describe('bug #2440 — shouldPreserveExistingProgress does not ratchet total_plans', () => {
+  const { shouldPreserveExistingProgress } = require('../gsd-core/bin/lib/state-document.cjs');
+
+  test('existing total_plans:50 > derived:64 → returns false (upward correction)', () => {
+    const existing = { total_phases: 2, completed_phases: 1, total_plans: 50, completed_plans: 49 };
+    const derived  = { total_phases: 2, completed_phases: 1, total_plans: 64, completed_plans: 49 };
+    assert.equal(shouldPreserveExistingProgress(existing, derived), false,
+      'total_plans upward correction must NOT trigger ratchet');
+  });
+
+  test('existing total_plans:50 > derived:30 → returns false (downward correction)', () => {
+    const existing = { total_phases: 2, completed_phases: 1, total_plans: 50, completed_plans: 30 };
+    const derived  = { total_phases: 2, completed_phases: 1, total_plans: 30, completed_plans: 30 };
+    assert.equal(shouldPreserveExistingProgress(existing, derived), false,
+      'total_plans downward correction must NOT trigger ratchet');
+  });
+
+  test('boundary: existing total_plans == derived → false', () => {
+    const existing = { total_phases: 2, completed_phases: 1, total_plans: 50, completed_plans: 50 };
+    const derived  = { total_phases: 2, completed_phases: 1, total_plans: 50, completed_plans: 50 };
+    assert.equal(shouldPreserveExistingProgress(existing, derived), false,
+      'total_plans equality must NOT trigger ratchet');
+  });
+
+  test('completed_plans:5 > derived:2 → true (ratchet still active for completed_plans)', () => {
+    const existing = { total_phases: 2, completed_phases: 1, total_plans: 6, completed_plans: 5 };
+    const derived  = { total_phases: 2, completed_phases: 1, total_plans: 6, completed_plans: 2 };
+    assert.equal(shouldPreserveExistingProgress(existing, derived), true,
+      'completed_plans ratchet must still work');
+  });
+});
   });
 }