From dfff25f1bdf404bea68b944f50aaf476135dbe09 Mon Sep 17 00:00:00 2001 From: Tom Boucher Date: Mon, 29 Jun 2026 15:27:22 -0400 Subject: [PATCH] docs(#1817): add adr-1817 state.md rebuild derivability contract (#1828) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(context): scrub nul byte from consent-store predicate CONTEXT.md line 206 (Capability Consent Store predicate) contained a literal NUL byte between ${realpath(projectRoot)} and documenting the disk-key join format. The byte was intentional but made the file binary-detected, breaking grep/rg searches (hit while preparing ADR-1817). Replace with the 4-char \x00 escape. Preserves the byte-level disk-key documentation; surrounding prose 'prototype-pollution-safe NUL-joined keys' carries the semantic context. file(1) now reports 'Unicode text'. * docs(#1817): add adr-1817 state.md rebuild derivability contract Phase 0 of approved feature #1817 (epic). Lands the design contract for the new `rebuild` transition in the STATE.md Transition Module (ADR-1769): - ADR-1817 (new): `rebuild` is the capstone 11th transition. Six design decisions: (1) core substrate, non-toggleable, same tier as the other 10; (2) section taxonomy — re-derivable (`## Current Position` prose from frontmatter, `## By-Phase Progress` table from disk) vs preserved (`## Session`, `## Decisions`, unknown sections) vs de-duplicated (`## Session Continuity Archive`); (3) orphaned data is logged + dropped with a structured audit entry in `## Rebuild Log` (ADR-1411 provenance principle); (4) idempotency is a hard guarantee — a no-mutation rebuild appends no log entry; (5) non-overlapping scope with `sync` (3 frontmatter fields, auto-triggered); (6) orthogonal to `auto_prune_state` (rebuild reconciles with current canonical sources, prune removes by retention policy). - CONTEXT.md (STATE.md Transition Module section): list `rebuild` as the 11th intent; add contract predicates mirroring the ADR. - docs/adr/README.md: add ADR-1817 to the index (Accepted). Targets the #1776/#1761/#1591 body-drift cluster that survived ADR-1769's per-field transitions. Phased per ADR-1817: this PR (Phase 0) closes #1817; Phase 1 (#1827) lands `rebuildCore` + intent dispatch + drift-class unit tests; Phase 2 (#1826) lands `cmdStateRebuild` CLI + dry-run + integration tests + docs + changeset. * docs(#1817): reword state-doctor alternative to satisfy docs-parity lint The docs-parity-live-registry test scans every docs/*.md (including docs/adr/) for slash-command tokens and asserts each one resolves to a live command in the registry. The rejected-alternative #4 in ADR-1817 mentioned a hypothetical `/gsd:state-doctor` workflow, which tripped the lint (`unknown command token(s): [/gsd:state-doctor]`). Reword to 'standalone state-doctor workflow' (no slash prefix). The extractor is aggressive — backticks and space-preceding tokens are both extracted per the test's own polarity-invariant cases — so the only sound fix is to not form a slash token at all for hypothetical names. Verified locally: `node --test tests/docs-parity-live-registry.test.cjs` now passes 31/31 (was 30/1). --- CONTEXT.md | 4 +- ...-state-md-rebuild-derivability-contract.md | 279 ++++++++++++++++++ docs/adr/README.md | 1 + 3 files changed, 282 insertions(+), 2 deletions(-) create mode 100644 docs/adr/1817-state-md-rebuild-derivability-contract.md diff --git a/CONTEXT.md b/CONTEXT.md index d5d11c09d..7439f2959 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -53,7 +53,7 @@ Module owning projection from dispatch results/errors to CLI `{ exitCode, stdout Module owning STATE.md parse, field extraction, field replacement, status normalization, and frontmatter reconstruction. It does not scan `.planning/phases` and does not own persistence or locking; phase/plan/summary counts arrive from inventory/progress Modules as inputs, and read-modify-write paths remain Adapters. Source of truth: `gsd-core/bin/lib/state-document.cjs`. ### STATE.md Transition Module -Module owning STATE.md lifecycle/maintenance transitions as intent-based methods (`beginPhase`, `advancePlan`, `completePhase`, `plannedPhase`, `milestoneSwitch`, `milestoneComplete`, `patch`, `sync`, `prune`, `update`). Pure core `(content, intent, deps) → newContent` with injected I/O (file read/write, lock, disk scan); consults a field-classification table that names each STATE.md field's class (`derived-from-body` | `derived-from-disk` | `derived-from-external` | `curated` | `free`) and its preservation policy. Supersedes the 14 scattered RMW callbacks in `state.cts` and the direct `writeStateMd` callers in `milestone.cts:352` and `phase.cts:1770`; verify's `regenerateState` factory-reset primitive stays as a direct `writeStateMd` call. Absorbs `syncStateFrontmatter` + `readModifyWriteStateMd`'s post-sync preservation block; Encoding 3 (`cmdStateBuildFrontmatter`) stays separate — read path concern. Sibling/super-module of the STATE.md Document Module; consumes its `stateReplaceField`/`stateExtractField` primitives. Body section structure (`## Current Position`, `## Session`, etc.) lives as a constants block inside the Module. Append-only transitions (`addDecision`, `addBlocker`, etc.) stay on today's RMW seam for now. Targets the #1760/#1761/#1743/#1695/#1264/#1255/#1257/#3242 bug cluster. Migration per ADR-1372 §T6 sequenced as substrate + `beginPhase` first (PR1), then transition-by-transition with characterization tests first per transition. Source of truth: `gsd-core/bin/lib/state-transition.cjs` (generated from `src/state-transition.cts`). +Module owning STATE.md lifecycle/maintenance transitions as intent-based methods (`beginPhase`, `advancePlan`, `completePhase`, `plannedPhase`, `milestoneSwitch`, `milestoneComplete`, `patch`, `sync`, `prune`, `update`, `rebuild`). Pure core `(content, intent, deps) → newContent` with injected I/O (file read/write, lock, disk scan); consults a field-classification table that names each STATE.md field's class (`derived-from-body` | `derived-from-disk` | `derived-from-external` | `curated` | `free`) and its preservation policy. Supersedes the 14 scattered RMW callbacks in `state.cts` and the direct `writeStateMd` callers in `milestone.cts:352` and `phase.cts:1770`; verify's `regenerateState` factory-reset primitive stays as a direct `writeStateMd` call. Absorbs `syncStateFrontmatter` + `readModifyWriteStateMd`'s post-sync preservation block; Encoding 3 (`cmdStateBuildFrontmatter`) stays separate — read path concern. Sibling/super-module of the STATE.md Document Module; consumes its `stateReplaceField`/`stateExtractField` primitives. Body section structure (`## Current Position`, `## Session`, etc.) lives as a constants block inside the Module. Append-only transitions (`addDecision`, `addBlocker`, etc.) stay on today's RMW seam for now. Targets the #1760/#1761/#1743/#1695/#1264/#1255/#1257/#3242 bug cluster. Migration per ADR-1372 §T6 sequenced as substrate + `beginPhase` first (PR1), then transition-by-transition with characterization tests first per transition. **ADR-1817 adds `rebuild` as the capstone 11th transition — the body-structure derivability contract.** Re-derives `## Current Position` prose from frontmatter and `## By-Phase Progress` table from phase dirs on disk; preserves `## Session` / `## Decisions` / unknown sections verbatim; de-duplicates `## Session Continuity Archive` (keep most-recent N, default 3); appends a structured audit entry to `## Rebuild Log` (`timestamp`, `kind`, `section`, `before`, `after`, `reason`) for every mutation. Hard idempotency guarantee: a no-mutation rebuild appends no log entry, so two successive invocations on a clean file are byte-identical. Non-overlapping with `sync` (3 lightweight frontmatter fields, auto-triggered) and orthogonal to `auto_prune_state` (age-based removal) — `rebuild` reconciles with current canonical sources, `prune` removes by retention policy, the two compose (rebuild first, then prune). Section ordering is invariant: rebuild rewrites content in place, never reorders. Targets the #1776/#1761/#1591 body-drift cluster that survived ADR-1769's per-field transitions. Phased per ADR-1817: Phase 0 = this ADR + predicates (closes #1817), Phase 1 = `rebuildCore` body + `rebuild` dispatch case + drift-class unit tests (#1827), Phase 2 = `cmdStateRebuild` CLI + `--dry-run`/`--verbose` + integration tests + docs + changeset (#1826). Source of truth: `gsd-core/bin/lib/state-transition.cjs` (generated from `src/state-transition.cts`). ### Query Execution Policy Module Module owning query transport routing policy projection (`preferNative`, fallback policy, workstream subprocess forcing) at execution seam. @@ -203,7 +203,7 @@ ADR-1244 D3 fetch-and-stage seam (`gsd-core/bin/lib/capability-source.cjs`). Pri ADR-1244 D4 per-runtime install manifest (`gsd-core/bin/lib/capability-ledger.cjs`). Leaf module (only `node:fs`/`node:path` plus `shell-command-projection`'s `platformWriteSync`). Records `{ id, version, source, integrity, files[], sharedEdits[{file,marker}] }` per installed capability in `.gsd-capabilities.json` at the runtime config dir root. Exports: `readLedger` (structural-validated, never throws), `writeLedger` (atomic via `platformWriteSync`), `recordInstall` (idempotent, prototype-pollution-guarded), `removeEntry`, and `reconcile` (reports orphans whose `files[]` are missing on disk; hardened against non-string/`..` members; never mutates). Serves as the atomic commit point for Phase-4 upgrade/remove and the reconciliation basis for detecting stale entries after out-of-band deletions. ### Capability Consent Store -Issue #1459 user-owned consent seam (`gsd-core/bin/lib/capability-consent.cjs`, generated from `src/capability-consent.cts`). Leaf module (`node:fs`/`node:path`/`node:os`/`node:crypto` + the ledger's shared bounded `readSmallRegularFile`/`readSmallRegularFileBuffer` + the shared `capability-lock` primitive). Stores `{ version:"1", records: { "": { projectRoot, id, scope:'project', integrity, disclosureSignature, contentHash, consentedAt } } }` at `${GSD_HOME||homedir()}/.gsd/consent.json` — a USER-OWNED file OUTSIDE any repository. Exports: `consentStorePath(gsdHome?)`, `readConsentStore(gsdHome?)` (bounded via `readSmallRegularFile` + 8 MiB cap, NON-THROWING — missing/corrupt/oversized/FIFO/wrong-shape → empty `{records:{}}`; caps records at `MAX_RECORDS=4096`), `bundleContentHash(capDir)` (THE security binding — a `sha512-` over a DETERMINISTIC, INJECTIVE, LOSSLESS serialization of EVERY regular file AND directory under the bundle: length-FRAMED entry COUNT + per-entry TYPE tag + uint32 path-byte-len + RAW path bytes from a `{encoding:'buffer'}` dir walk [finding 4] + for files uint64 content-byte-len + RAW content bytes via `readSmallRegularFileBuffer` [finding 1b], plus typed DIR markers binding empty directories [finding 2]; symlinks/non-regular rejected; size+count bounded), `hasProjectConsent({gsdHome,projectRoot,id,contentHash})` (true iff a record for `${realpath(projectRoot)}` exists AND its stored `contentHash` equals the supplied recomputed hash — the binding is `contentHash`, NOT `integrity` and NOT `disclosureSignature` (those remain on the record purely for the human disclosure + re-consent-on-executable-change UX); unsafe ids → false; prototype-pollution-safe NUL-joined keys + `Object.prototype.hasOwnProperty`), `recordProjectConsent({gsdHome,projectRoot,id,integrity,disclosureSignature,contentHash})` (LOCKED, atomic+durable write — tmp `wx`/fsync/rename/dir-fsync mirroring `writeLedger`; enforces the record cap at write time) and `revokeProjectConsent({gsdHome,projectRoot,id})` (LOCKED atomic delete, no-op if absent) — BOTH **THROW** rather than perform an UNLOCKED read-modify-write when the consent-store lock cannot be acquired (finding 3; the lifecycle treats a consent-write failure as non-fatal, and the `trust revoke` CLI catches the throw and emits a clean error). This is the authoritative consent signal the loader recomputes (`bundleContentHash(capDir)`) and checks at load before activating a PROJECT-scope third-party overlay (declarative surfaces AND command dispatch): a forged/cloned in-repo project ledger, OR any post-consent tamper (swapped declarative manifest, edited hook script, empty-integrity local install — all change the recomputed hash), leaves the cap DISCOVERED-BUT-INACTIVE until the user consents on THIS machine to the EXACT bundle (the lifecycle records the consent on a consented project install/upgrade and revokes it on remove; install/lookup/revoke share one canonical `consentProjectRoot` root key). GLOBAL-scope overlays (under the user's own home) need no record; and when `GSD_HOME` resolves (via realpath, defeating symlink aliasing — finding 1) to a genuine project root the in-repo bundle still requires a record. The consent lock is the SHARED hardened primitive (below), so it never stale-steals a slow-but-live writer (finding 4). See `docs/explanation/capability-trust-model.md` "project-scope trust boundary". +Issue #1459 user-owned consent seam (`gsd-core/bin/lib/capability-consent.cjs`, generated from `src/capability-consent.cts`). Leaf module (`node:fs`/`node:path`/`node:os`/`node:crypto` + the ledger's shared bounded `readSmallRegularFile`/`readSmallRegularFileBuffer` + the shared `capability-lock` primitive). Stores `{ version:"1", records: { "": { projectRoot, id, scope:'project', integrity, disclosureSignature, contentHash, consentedAt } } }` at `${GSD_HOME||homedir()}/.gsd/consent.json` — a USER-OWNED file OUTSIDE any repository. Exports: `consentStorePath(gsdHome?)`, `readConsentStore(gsdHome?)` (bounded via `readSmallRegularFile` + 8 MiB cap, NON-THROWING — missing/corrupt/oversized/FIFO/wrong-shape → empty `{records:{}}`; caps records at `MAX_RECORDS=4096`), `bundleContentHash(capDir)` (THE security binding — a `sha512-` over a DETERMINISTIC, INJECTIVE, LOSSLESS serialization of EVERY regular file AND directory under the bundle: length-FRAMED entry COUNT + per-entry TYPE tag + uint32 path-byte-len + RAW path bytes from a `{encoding:'buffer'}` dir walk [finding 4] + for files uint64 content-byte-len + RAW content bytes via `readSmallRegularFileBuffer` [finding 1b], plus typed DIR markers binding empty directories [finding 2]; symlinks/non-regular rejected; size+count bounded), `hasProjectConsent({gsdHome,projectRoot,id,contentHash})` (true iff a record for `${realpath(projectRoot)}\x00` exists AND its stored `contentHash` equals the supplied recomputed hash — the binding is `contentHash`, NOT `integrity` and NOT `disclosureSignature` (those remain on the record purely for the human disclosure + re-consent-on-executable-change UX); unsafe ids → false; prototype-pollution-safe NUL-joined keys + `Object.prototype.hasOwnProperty`), `recordProjectConsent({gsdHome,projectRoot,id,integrity,disclosureSignature,contentHash})` (LOCKED, atomic+durable write — tmp `wx`/fsync/rename/dir-fsync mirroring `writeLedger`; enforces the record cap at write time) and `revokeProjectConsent({gsdHome,projectRoot,id})` (LOCKED atomic delete, no-op if absent) — BOTH **THROW** rather than perform an UNLOCKED read-modify-write when the consent-store lock cannot be acquired (finding 3; the lifecycle treats a consent-write failure as non-fatal, and the `trust revoke` CLI catches the throw and emits a clean error). This is the authoritative consent signal the loader recomputes (`bundleContentHash(capDir)`) and checks at load before activating a PROJECT-scope third-party overlay (declarative surfaces AND command dispatch): a forged/cloned in-repo project ledger, OR any post-consent tamper (swapped declarative manifest, edited hook script, empty-integrity local install — all change the recomputed hash), leaves the cap DISCOVERED-BUT-INACTIVE until the user consents on THIS machine to the EXACT bundle (the lifecycle records the consent on a consented project install/upgrade and revokes it on remove; install/lookup/revoke share one canonical `consentProjectRoot` root key). GLOBAL-scope overlays (under the user's own home) need no record; and when `GSD_HOME` resolves (via realpath, defeating symlink aliasing — finding 1) to a genuine project root the in-repo bundle still requires a record. The consent lock is the SHARED hardened primitive (below), so it never stale-steals a slow-but-live writer (finding 4). See `docs/explanation/capability-trust-model.md` "project-scope trust boundary". ### Capability Lock Issue #1459 finding 4 shared cross-process lock primitive (`gsd-core/bin/lib/capability-lock.cjs`, generated from `src/capability-lock.cts`). Leaf module (`node:fs`/`node:path`/`node:os`/`node:crypto` + the ledger's bounded `readSmallRegularFile` + `shell-command-projection`'s `execTool` for the rare start-time shell-out). THE single hardened lockfile protocol shared by BOTH `capability-lifecycle` (the `.gsd/capabilities/.lock` mutation lock) and `capability-consent` (the consent-store `.consent.lock`) — extracted so the two locks cannot diverge (mirrors the shared-validator / shared bounded-reader lessons). Exports: `acquireLock(lockPath, opts?)` (O_EXCL create with a JSON `{token,pid,hostname,startTime,ts}` body; steal protocol binds age to the body's own `ts`, never stale-steals a VERIFIED-LIVE same-host holder — pid alive AND recorded start-time matches the pid's current start-time, defeating pid-reuse without ever stealing a live holder — and reclaims only a dead/unverifiable holder via the dead-pid fast path or the hard `LOCK_DEADMAN_MS` deadman; `opts.maxAttempts` raises the bounded retry budget and `opts.waitForFresh` makes a contended fresh/live holder be WAITED FOR rather than failed-fast so genuinely-racing consent writers serialize), `releaseLock(handle)` (token + inode owner-safe — never deletes a successor's lock), `getProcessStartTime`, and the `_setLockProbes`/`_resetLockProbes` test seams. Carries the #1462 lifecycle-lock invariants (process-start-time liveness, TOCTOU-safe pre-rename identity recheck, bounded iterative loop). diff --git a/docs/adr/1817-state-md-rebuild-derivability-contract.md b/docs/adr/1817-state-md-rebuild-derivability-contract.md new file mode 100644 index 000000000..08cd472fb --- /dev/null +++ b/docs/adr/1817-state-md-rebuild-derivability-contract.md @@ -0,0 +1,279 @@ +# ADR-1817: STATE.md rebuild — derivability contract (capstone transition) + +- **Status:** Accepted (Phase 0 — ADR + CONTEXT.md update; lands ahead of Phases 1–2) +- **Date:** 2026-06-29 +- **Issue:** [#1817](https://github.com/open-gsd/gsd-core/issues/1817) — epic +- **Builds on:** [ADR-1769](1769-state-md-transition-module.md) (STATE.md Transition Module). Adds the 11th transition (`rebuild`) on top of ADR-1769's 10 lifecycle/maintenance intents. +- **Supersedes:** nothing. Extends ADR-1769's transition set; does not revisit its design. + +## Context + +ADR-1769 landed the STATE.md Transition Module with 10 intent-based transitions +(`beginPhase`, `advancePlan`, `completePhase`, `plannedPhase`, `milestoneSwitch`, +`milestoneComplete`, `patch`, `sync`, `prune`, `update`) and a field-classification +table that killed the per-call-site preservation-policy bug cluster (#1760, #1761, +#1743, #1695, #1264, #1255, #1257, #3242). Each transition touches **individual +fields**. + +A second bug class survived ADR-1769: **body-structure drift** that no current +command can reconcile. `syncCore` (the lightest-weight transition) only patches +three frontmatter fields — `Total Plans in Phase`, `Progress`, `Last Activity` — +and intentionally does not re-derive body structure. `buildStateFrontmatter` +re-derives all frontmatter from body + disk scan but does not touch the body +itself. The result: **the body can diverge from ground truth indefinitely while +`gsd-tools state sync` reports `synced: true`.** + +Observed drift signatures (full list in epic #1817): + +- `## Current Position` prose fields (Phase, Status, Current Plan) contradict + frontmatter after a milestone switch or prune. +- `## By-Phase Progress` table has orphaned rows for phases from a prior + milestone or rows with zero-padded phase IDs that were renamed. +- Template-placeholder field values (`[phase name]`, `[date]`) left in place + when an AI agent wrote partial state. +- Duplicate `## Session Continuity Archive` blocks from repeated + `state record-session` calls on a corrupt file. +- `stopped_at` in frontmatter sourced from the wrong section (an archive block + rather than the current `## Session` block) — bug #2444's guard only applies + in `buildStateFrontmatter`, not to the body itself. +- `progress.total_phases` in frontmatter correct, but `Phase: [N] of [M]` prose + still shows the old milestone's count. + +Three open issues sit in this defect class: **#1776** (`cmdStatePrune` phase +fallback matches any `| Phase | N |` table cell, not `## Current Position` +prose → requires scoped body extraction), **#1761** (`state sync` writes wrong +progress when ROADMAP lacks versioned milestone headings → a rebuild would +re-derive from disk + ROADMAP together), **#1591** (`phase.complete` mis-parses +`
`-wrapped roadmaps and garbles counters → a rebuild can reconcile +from canonical disk sources rather than parsing ROADMAP mid-transition). + +The deeper shape: **every body-level drift bug today requires a per-bug regex +fix.** Ten-plus closed issues (#1658, #1659, #1668, #1446, #1230, #948, #549, +#500, #363, #316) each added a narrow guard that didn't prevent the next +variant. A `rebuild` transition that re-derives the **canonical body sections** +from ROADMAP + phase dirs + session history would resolve the entire class +without per-bug patches. + +## Decision + +Add `rebuild` as the **11th intent** in `transitionCore` (ADR-1769 §6 "Core +scope: writes only"). Six design decisions, resolved via `/grilling`: + +### 1. The `rebuild` intent is a maintenance transition, same tier as the other 10 + +`rebuild` is a STATE.md Transition Module method, dispatched from +`transitionCore`'s switch alongside `sync`, `prune`, `update`, etc. Pure core +`(content, intent, deps) → newContent`, same shape as ADR-1769 §3. + +**Capability tier (ADR-857 analog):** `rebuild` is **core substrate**, not a +Feature Capability. It is non-toggleable, lives inside the transition module, +and is not subject to ADR-857 capability consent/overlay. This mirrors ADR-550 +§83's "verifier↔predicate contract is core/non-toggleable" rule for the +verification seam: the derivability contract documented here is the state-seam +equivalent. + +*Rejected:* (B) Implement `rebuild` as a Feature Capability that users opt into +— rejected because the bug class is core STATE.md correctness, not optional +behavior. (C) Keep `rebuild` outside the Transition Module (a sibling +utility) — rejected: it must own the same lock→read→apply→preserve→write +transaction ADR-1769 §1 specified for the other 10 transitions; a sibling +utility would re-encapsulate that machinery and drift. + +### 2. Section taxonomy: derived vs preserved + +Each STATE.md body section is classified as **re-derivable** or **preserved**. +`rebuild` consults the taxonomy; it never re-derives a preserved section and +never preserves a re-derivable one verbatim when the canonical source +disagrees. + +| Section | Class | Source of truth | Rebuild behavior | +|---|---|---|---| +| `## Current Position` prose | re-derivable | Frontmatter (which `buildStateFrontmatter` already derives correctly from disk + ROADMAP) | Re-derive each prose field from the corresponding frontmatter field; replace verbatim. | +| `## By-Phase Progress` table | re-derivable | Phase dirs on disk (same source as `buildStateFrontmatter`'s disk scan) | Re-derive the entire table from disk; drop orphaned rows. | +| `## Session` block | preserved | Human-curated current-session data | Preserve verbatim. (Only the *current* `## Session` block; archived sessions are de-duplicated per §4.) | +| `## Decisions` | preserved | Human-curated decision log | Preserve verbatim. Staleness is `pruneCore`'s concern, not rebuild's. | +| `## Session Continuity Archive` | preserved, de-duplicated | Prior session snapshots | Keep the most-recent N (configurable; default 3); drop duplicates; preserve kept entries verbatim. | +| `## Rebuild Log` | appended (new section) | The rebuild transition itself | Append one entry per rebuild that mutated the file; never re-derive or edit prior entries. | +| Any other `## …` section | preserved (unknown) | Human-curated | Preserve verbatim. Rebuild does not recognize or rewrite sections outside the taxonomy. | + +**Section ordering is invariant.** Rebuild rewrites the *content* of +re-derivable sections in place; it does not reorder sections, insert new +sections (other than `## Rebuild Log` if absent), or remove sections. + +*Principle:* derive what is derivable, preserve what is curated, log what is +dropped. (Postel's Law applied to a state file: be liberal in what you accept +— any drifted input — and conservative in what you send — canonical form for +derived, verbatim for preserved.) + +*Rejected:* (α) Treat all sections as re-derivable — rejected: destroys +human-curated content (session notes, decisions). (β) Treat all sections as +preserved — rejected: this is the status quo; the drift class survives. + +### 3. Orphaned data: log + drop (audit trail mandatory) + +When `rebuild` encounters canonical-source-disagreement that requires dropping +data, it MUST append a structured entry to `## Rebuild Log` recording: + +- `timestamp` (ISO-8601, from the injected `clock`) +- `kind` — one of `orphaned-row`, `placeholder-removed`, `archive-deduplicated`, + `wrong-section-source`, `milestone-count-stale`, or `section-rewritten` +- `section` — which body section was mutated +- `before` / `after` — the dropped/changed content (truncated to 512 chars per + entry to bound log growth) +- `reason` — short structured string explaining the canonical source that won + +`## Rebuild Log` is itself **preserved** (per §2). Rebuild never rewrites or +truncates prior log entries; it only appends. A separate prune step (out of +scope here) governs log retention. + +**Why log everything:** dropping user-adjacent data without a trace is hostile +even when the drop is correct. The log gives the user an undo path (manual +re-add) and gives the maintainer a debugging signal when rebuild drops +something it shouldn't have. (Hyrum's Law mitigation: the drop is observable, +the audit trail is the contract.) + +*Rejected:* silent drop — rejected: violates the ADR-1411 resolution-provenance +principle that mutation decisions report what they did, not fall open silently. + +### 4. Idempotency is a hard guarantee + +`rebuild` is **idempotent**: invoking it twice in succession on the same file +produces no change on the second invocation. This is testable and tested. + +The idempotency contract has two parts: + +1. **Body content idempotency:** re-running rebuild on a file rebuild just + canonicalized produces byte-identical `## Current Position` and + `## By-Phase Progress` sections. +2. **Rebuild Log idempotency:** a rebuild that mutates nothing appends no log + entry. (This is what makes the second-invocation case truly byte-identical + — without it, the second run would always append a no-op log entry and + violate idempotency.) + +The log-appends-only-on-mutation rule is the load-bearing constraint. If +rebuild wrote a log entry unconditionally on every invocation, idempotency +would break. + +### 5. Interaction with `sync` — non-overlapping scopes + +`sync` and `rebuild` compose; they do not compete. + +| Transition | Scope | Trigger | Latency | +|---|---|---|---| +| `sync` | 3 frontmatter fields (`Total Plans in Phase`, `Progress`, `Last Activity`) | Auto-triggered on every state transition | Lightweight, runs on every transition | +| `rebuild` | Body structure (`## Current Position`, `## By-Phase Progress`, archive dedup) | Manual (`gsd-tools state rebuild`) | Heavier; reads disk + ROADMAP; explicit user invocation | + +Running `sync` after `rebuild` is safe: `sync`'s 3 fields are a strict subset +of what `rebuild` reconciled (in canonical form), so sync's derivation will +produce the same values rebuild just wrote. Running `rebuild` after `sync` is +also safe: rebuild re-derives body from canonical sources; sync's just-written +frontmatter is one of those sources. + +`sync` stays as the auto-triggered lightweight path; `rebuild` is the +user-invoked heavy reconciliation. Neither subsumes the other. + +### 6. Interaction with `auto_prune_state` — orthogonal concerns + +`rebuild` does NOT prune. Pruning (removing data that is no longer relevant, +e.g. decisions older than N sessions, prior-milestone state) is a separate +concern governed by `auto_prune_state` and `pruneCore`. + +The distinction: + +- **Rebuild reconciles** body with current canonical sources. A `## By-Phase + Progress` row for a phase that no longer exists on disk is dropped because + it is *canonical-mismatched*, not because it is *old*. +- **Prune removes** data based on age/staleness policy. A `## Decisions` + entry from 6 months ago stays under rebuild (preserved) but may be removed + by prune based on retention policy. + +The two compose: rebuild first (reconcile with canonical sources), then prune +(remove per policy). Rebuild never makes pruning decisions; prune never +re-derives structure. + +## Consequences + +**Positive:** + +- The body-structure drift bug class is killed structurally. #1776, #1761, + and #1591 each become either directly fixable by `rebuild` or indirectly + addressable (the rebuild provides the scoped body extraction those bugs + need). +- The derivability contract is a new correctness invariant: STATE.md body + is **derivable from canonical sources at any time**, not just incrementally + updatable. This is the capstone property ADR-1769's per-field transitions + couldn't deliver alone. +- The audit log gives the maintainer a debugging signal when STATE.md editing + (manual or AI-driven) produces drift that rebuild later reconciles. +- Future drift classes (anything not in the §2 taxonomy today) can be added + by extending the taxonomy + a new `kind` in the log enum, without + re-touching the transition core's dispatch shape (Gall's Law: extend, don't + rewrite). + +**Negative:** + +- A new body section (`## Rebuild Log`) is added to STATE.md. Older GSD + versions reading the file ignore the section (preserved verbatim by + `readModifyWriteStateMd`'s post-sync block — the section name is not in + the field-classification table, so it falls through as "unknown, preserved"). +- The derivability contract is a new shared artifact: any future STATE.md + body section must declare its taxonomy class. Adding a new re-derivable + section is a non-trivial change (rebuild must learn the derivation rule); + adding a new preserved section is mechanical. +- The first invocation of `rebuild` on a long-lived project will produce a + substantial audit log entry (the project's accumulated drift is reconciled + in one pass). This is honest — the drift existed; rebuild surfaces it — + but users may be surprised by the log size on first run. Mitigation: + `--dry-run` flag (Phase 2) previews the diff before writing. + +**Neutral:** + +- `rebuildCore` is a pure function over `(content, intent, deps)`, callable + inside any orchestration shape (single-file write, multi-file transaction, + dry-run preview). Same property that let ADR-1769 §3 run `completePhase` + inside `writePlanningFileSet`. +- The existing 10 transitions are unchanged. `transitionCore`'s switch grows + from 10 cases to 11; the missing-case-compile-time-error guarantee + (ADR-1769 §1) extends to the new case. + +## Alternatives considered + +1. **Fix each body-level bug individually (status quo).** Rejected: already + done for 10+ closed issues. Each fix is a narrow regex guard that doesn't + prevent the next variant. Does not scale; the open issues (#1776, #1761, + #1591) are evidence. +2. **`state sync` expansion — extend `syncCore` to cover body structure.** + Rejected: `sync` is intentionally lightweight and auto-triggers on every + transition. Making it re-derive body structure would make every state + transition pay the disk-scan + ROADMAP-read cost, and would couple + auto-triggered behavior to a heavier and riskier code path. The + manual/auto split (§5) is the right factoring. +3. **Regenerate STATE.md from scratch (nuke-and-rebuild).** Rejected: loses + curated human content (session notes, decisions, archives). The + derivability contract is *selective* — derived sections re-derive, + preserved sections survive — which is exactly what a nuke-and-rebuild + cannot do. +4. **A standalone `state-doctor` workflow outside the transition module.** Rejected: + same drift-from-canonical-shape risk that motivated ADR-1769's + consolidation. A workflow that bypasses the transition module re-imports + the lock/scan/preservation machinery and re-creates the bug class. +5. **Defer until ADR-1769's amendments (#1796) finish independently.** + Rejected: ADR-1769 is closed (Phase 7 closeout + #1796 amendment landed). + There is no consumer-driven sequencing constraint; the rebuild transition + composes cleanly with the existing 10. + +## Phases + +This epic (#1817) is implemented in three phases, each its own PR. Phase 0 +closes this issue (the epic); Phases 1 and 2 close their own sub-issues. + +| Phase | Scope | Closes issue | Bug coverage | +|---|---|---|---| +| 0 | ADR + CONTEXT.md update (derivability contract, preserved-vs-derived taxonomy, idempotency, sync/prune interaction) | #1817 | — | +| 1 | `rebuildCore` body + `rebuild` intent dispatch case + drift-class unit tests | #1827 | surfaces the class; #1776, #1761, #1591 become directly addressable | +| 2 | `cmdStateRebuild` CLI + `--dry-run` / `--verbose` + integration tests + `docs/commands/state.md` + changeset | #1826 | end-to-end reconciliation available to users | + +Per-transition discipline (inherited from ADR-1769 §7): characterization tests +first (capture the drift signatures we want to reconcile), then implement +`rebuildCore`, then verify existing `pruneCore` / `syncCore` tests still pass, +then add idempotency tests. diff --git a/docs/adr/README.md b/docs/adr/README.md index a1a9ee823..afe84e846 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -62,6 +62,7 @@ See **[CONTRIBUTING.md — "Proposing an ADR or PRD"](../../CONTRIBUTING.md#prop | [1508-runtime-artifact-conversion-module.md](1508-runtime-artifact-conversion-module.md) | Runtime Artifact Conversion Module owns per-runtime content rewriting | Accepted | | [1593-skill-mapping-converter-methodology.md](1593-skill-mapping-converter-methodology.md) | Skill mapping & converter methodology across runtimes | Accepted | | [1769-state-md-transition-module.md](1769-state-md-transition-module.md) | STATE.md Transition Module — intent-based transitions over scattered RMW callbacks | Proposed | +| [1817-state-md-rebuild-derivability-contract.md](1817-state-md-rebuild-derivability-contract.md) | STATE.md rebuild — derivability contract (capstone 11th transition) | Accepted | ## Seam map