Files
msd-core/src/frontmatter.cts
Tom Boucher c1885df9e5 chore(#2143): prohibition-with-teeth + migrate remaining ad-hoc table sites — Phase 4 (final) (#2253)
* chore(#2143): prohibition-with-teeth + migrate remaining table sites — Phase 4

Phase 4 of epic #2143 (ADR-2143 §7). Completes the markdown table/mutation
consolidation by (a) giving the ad-hoc-parsing prohibition teeth and (b)
migrating the last ad-hoc table sites onto the shared seam.

- src/markdown-table.cts: new formatting-preserving `updateTableCell` primitive
  (self-contained, ragged-row-tolerant header/delimiter/cell-range scan; splices
  only the target cell's raw span, preserving all other bytes incl. padding/CRLF;
  no-op-preserves-padding when a transformer returns the current value). Exports
  splitTableRow/isDelimiterRow/findTableStartOffset for tolerant reuse.
- eslint-rules/no-adhoc-markdown-parsing.cjs: TABLE-REGEX detector extended to
  `new RegExp(<literal|static-template>)`; new `.replace()`-mutation detector for
  roadmap/state/content receivers with a table/section-shaped pattern.
- scripts/lint-table-schema-drift.cjs (wired into lint:ci): fails if a TABLE_SCHEMA
  header drifts from its authored table; tests import its logic (single source).
- Migrated onto the seam (behaviour-preserving vs pre-Phase-4 HEAD, verified
  byte-diff old-vs-new): roadmap.cts cmdRoadmapUpdatePlanProgress, phase.cts
  cmdPhaseComplete + traceability, milestone.cts cmdRequirementsMarkComplete,
  uat.cts read path, state.cts metrics/decisions/By-Phase.
- Incidental correctness gains from the migration: a decoy table can no longer
  swallow a phase-progress update (## Progress scoping); a ragged neighbouring
  row no longer silently aborts an edit; completing integer phase N no longer
  touches a decimal sub-phase N.x row; record-metric no longer drops trailing
  section content or duplicates the ## Performance Metrics section.
- Kept justified allow-adhoc-markdown markers only where genuinely not a table
  (security.cts <|role|> token) or a loose non-GFM section (uat human-verify).

Two orthogonal isolated reviews (correctness/adversarial + security) passed;
correctness found 4 behaviour regressions in the first migration pass, all fixed
and re-verified byte-identical-or-better vs OLD.

Surfaced for maintainer (pre-existing, ambiguous domain logic, NOT changed here):
templates/state.md places a By-Phase table under ## Performance Metrics while
cmdStateRecordMetric assumes a Plan|Duration|Tasks|Files table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): match traceability row by first-cell value, not Requirement header

Phase 4's migration matched the REQUIREMENTS.md traceability row by a column
literally named `Requirement` (`row['Requirement']`), but real tables head that
column `REQ-ID`. The by-name lookup found nothing, so `phase complete` and
`requirements mark-complete` left the Status cell `Pending` (regressed #2769 /
#2203, caught by gsd-test — 8 failures, both node 22/24).

- src/phase.cts, src/milestone.cts: match the row by its FIRST cell's value
  (the requirement-ID column) regardless of that column's HEADER name, via
  `Object.values(row)[0]` (updateTableCell builds the record in header order).
  This mirrors OLD's first-cell `\|\s*<id>\s*\|` anchor, restoring header-name
  independence while keeping the seam.
- src/milestone.cts hasTable: broadened from `Requirement`-only to also
  recognize `Requirement ID` / `REQ-ID` / `REQ ID` headers, kept in sync with
  the now-positional rowMatch/hasRow so a REQ-ID-headed table participates in
  the ADR-2143 §6 write-set and the #2140 table_unmatched drift check (it was
  silently omitted before — a checkbox-only partial reconcile against a REQ-ID
  table could report as fully reconciled). The `Requirement`-headed path is
  byte-identical to OLD.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(#2143): replace stale structural milestone guards with behavioural suite

The `milestone.cjs regex global state fix` block was a source-structure guard
(allow-test-rule: structural-regression-guard) — it readFileSync'd the compiled
milestone.cjs and asserted removed regex idioms (`tablePattern.test`,
`afterTable !== reqContent`, `doneTable = new RegExp(...)`). Phase 4's migration
deleted those regexes (table update is now updateTableCell), making the
assertions obsolete. Per the Test Cleanup rule, replace them in-PR with a
behavioural suite driving the compiled CLI:

- multi-ID mark-complete flips all IDs (guards the lastIndex/global-state class),
- Pending->Complete flip under both `REQ-ID` and `Requirement` headers (#2769),
- idempotent already_complete detection with no corruption,
- REQ-ID-headed table participates in write_set (traceability entry, applied),
- REQ-ID-headed table trips #2140 table_unmatched drift on a missing row.

Pruned the now-nonexistent structural-regression-guard entry from the
lint-allow-test-rule-refs allowlist (the source-text-is-the-product entry for
the same file remains valid).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(changeset): backfill PR number 2253

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): record-metric targets its own metrics table, not By-Phase velocity

`state record-metric` appended its per-plan row (`| Phase 1 P1 | 5min | 3 tasks |
4 files |`) into the FIRST table under `## Performance Metrics` — which on a real
template-derived STATE.md is the By-Phase velocity table `| Phase | Plans | Total
| Avg/Plan |`, polluting it on EVERY plan completion (execute-plan.md:414 is a
per-plan call). The command's own metrics table is `| Plan | Duration | Tasks |
Files |`, which the template does not ship, so the row never reached it; the
scaffold branch also emitted a wrong `| Phase | Plan | Duration | Notes |` header
matching neither the row nor the canonical table.

Pre-existing (predates Phase 4); surfaced while migrating this site and fixed here
per no-defer, on the user's explicit go-ahead.

- src/state.cts cmdStateRecordMetric: locate the metrics table by its own header
  shape (`Plan|Duration|Tasks|Files`, via splitTableRow/isDelimiterRow) rather
  than "first table in the section". When the section exists but has no metrics
  table (only the By-Phase table), self-heal by appending a fresh **Per-Plan
  Metrics:** table to the END of the section body — By-Phase table, Recent Trend
  and footer preserved verbatim, no duplicate `## Performance Metrics` heading,
  created stays false. Absent-section scaffold header corrected to the canonical
  `| Plan | Duration | Tasks | Files |`. Ragged-tolerance + None-yet preserved.
- Not touching templates/state.md (golden-install-parity hashed) — record-metric
  self-creates the table on first use instead.

Failing-first regression test (tests/state.test.cjs) demonstrates the By-Phase
pollution on the pre-fix build, then green after. Verified: no pollution, self-
heal idempotency, both-tables isolation, content/heading preservation, flags,
None-yet, corrected scaffold header (23-check adversarial harness + all existing
record-metric scenarios).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(#2143): deleteSection seam primitive (level-bounded whole-section removal)

ADR-2143 §4 shipped withSection/collectSection (replace a section BODY) but no
way to DELETE a section (heading + body). Phase 4 suppressed the phase-remove
section delete instead of building it. deleteSection(content, predicate, opts)
locates the section via the collectSection machinery and splices out from the
heading's start offset to the next same-or-higher-level heading — so a level-3
`### Phase N` delete stops at a following level-2 `## Progress`, never past it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): phase remove no longer deletes ## Progress on last-phase removal

updateRoadmapAfterPhaseRemoval deleted a `### Phase N` detail section with a
greedy raw regex whose lazy scan, on the LAST phase, ran to EOF and destroyed
the following `## Progress` heading and its entire tracking table — silent data
loss, uncovered by tests (removal tests only exercised a middle phase). Migrated
onto the new deleteSection seam (level-bounded, stops at `## Progress`); dropped
the allow-adhoc-markdown SECTION-DELETION suppression. Failing-first regression
(tests/phase.test.cjs) removes the LAST phase and asserts the ## Progress heading
+ table survive; middle-phase removal is byte-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(#2143): deleteTableRow seam primitive (row removal, ragged-tolerant)

Sibling of updateTableCell: locates the first GFM table, matches a DATA row by
predicate (ragged-tolerant record build, header order), and splices out that
row's whole line preserving every other byte. Returns {ok:false,reason} on no
table / no match. Enables migrating the phase-remove Progress-table row delete
off its ad-hoc regex (ADR-2143 §7 — the "future row-delete seam" Phase 4 punted).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): phase remove deletes the Progress row via deleteTableRow

The Progress-table row delete used a whole-document regex with two defects:
(a) `\.?\s` required whitespace after the phase number, so a COMPACT row
`|2|Beta|` was never deleted (stale row left behind); (b) unscoped — it could
strike a row in a different table (e.g. an earlier `| Phase | Requirements |`
table). Migrated onto deleteTableRow, scoped to the `## Progress` section
(mirrors deriveProgressFromRoadmap), matching the row by first-cell phase number
(integer zero-pad-insensitive; decimal exact; removing `2` never touches `2.5`).
Both allow-adhoc-markdown suppressions removed. New behavioural tests: compact
unpadded row deleted; padded byte-parity on the surviving rows (their ordinal
correctly renumbers via the pre-existing renumber block).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): deleteTableRow leaves no dangling newline on last EOL-less row

Deleting the final row of a table with no trailing EOL sliced from the row's
start to end-of-string, stranding the newline that terminated the previous line.
Back rowStart over the preceding \r?\n in that branch so the table ends cleanly.
(Caught by the primitive's own unit test on gsd-test; local scenario checks
missed the no-trailing-EOL edge.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): migrate read-only section-collects onto collectSection

Six hand-rolled `## Section` read-extract regexes replaced by the collectSection
seam (behaviour-preserving; extracted bodies feed the same downstream parsers):
state.cts matchSessionSection (## Session / ## Session Continuity) + ## Blockers,
smart-entry.cts ## Blockers, audit.cts ## Current Focus + ## Open Questions.
Removes 6 allow-adhoc-markdown "pending #1372" suppressions. Incidental fix: the
old Session regex `## Session[ \t]*\n` silently failed on a CRLF `## Session\r\n`
heading (Windows STATE.md), nulling all session fields; collectSection is
CRLF-safe, so session state now resolves on Windows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): fence-safe state-transition section writes + dedup stripFrontmatter

- milestoneCompleteCore's `## Current Position` and `## Operator Next Steps`
  section resets used fence-blind raw regexes that a fenced `##` inside the body
  could truncate/mis-target (#2130/#2067/#2080 class). Migrated onto a
  fence-aware tokenizeHeadings-based helper (resetSectionVerbatim) that is
  byte-identical to the old output on the canonical path (9/9 fixtures) and
  correctly ignores a fenced fake heading (proven robustness gain).
- mutateCurrentPositionFirstTime: hand-rolled locate+splice → collectSection +
  replaceSection (byte-parity).
- stripFrontmatter was inlined byte-identically in state.cts AND
  state-transition.cts; hoisted the single canonical copy into frontmatter.cts
  (both call sites now import it) + unit tests — eliminates the divergence risk
  per CLAUDE.md "Generative Fix Divergence". Removes 3 allow-adhoc-markdown /
  #1372 markers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): name-address By-Phase sum + uat parse, eslint recall hole, catches

- state.cts By-Phase "Total plans completed" sum: positional 2nd-cell regex →
  name-addressed splitTableRow read (correct on a reordered header, where the
  old code silently summed the wrong column). Marker removed.
- uat.cts parseVerificationItems: loose pipe regex → splitTableRow within the
  existing table/numbered/bullet union scan (item list byte-identical; does NOT
  reintroduce the reverted strict-parseMarkdownTable item-drop). Marker removed.
- eslint no-adhoc-markdown-parsing: close the `new RegExp(identifier)` recall
  hole — resolve a const-declared table-shaped regex identifier (mirrors the
  .replace() detector) + RuleTester cases; param/call args stay out (boundary).
- commands.cts: delete a lying comment that claimed the scaffold date "stays on
  raw UTC / deferred" — #2136 already moved it to realClock.localToday().
- Empty catches (classified, not blind-swept): removed 4 dead try/catch;
  fixed 3 error-hiding (phase-insert decimal-dir I/O collision now fails loud;
  phase-remove rename partial-failure surfaced; milestone-archive true count via
  finally); left best-effort swallows with justification comments.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): extractFencedBlock seam + migrate api-coverage named fence

parseCoverageMatrix extracted its ```coverage fenced block with an ad-hoc regex
(the last real allow-adhoc-markdown suppression). Added extractFencedBlock to the
markdown-sectionizer seam (reuses stripFencedCode's CommonMark fence engine —
info-string match, ~~~/backtick, nesting, indent) and migrated onto it; byte-
parity on the parsed CoverageMatrix across 8 fixtures. Only security.cts:367
(a genuine `<|role|>` protocol-token false-positive, not a GFM table) remains
marked in src/ — the "prohibition with teeth" goal (nothing grandfathered but a
true FP) is met.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): By-Phase row insert is name-addressed (insertTableRow seam)

updatePerformanceMetricsSection's INSERT-new-row branch located the By-Phase
table with a canonical-column-order-only regex + a hardcoded positional row
literal, so on a reordered header it silently inserted nothing — inconsistent
with the now name-addressed UPDATE and SUM halves of the same function. Added
insertTableRow (markdown-table seam sibling of updateTableCell/deleteTableRow:
name-addressed, header-order-agnostic, EOL-preserving) and migrated the branch
onto it, mapping By-Phase values by column NAME. Canonical-order output is
byte-identical; a reordered header now inserts a correctly-mapped row; a
pre-existing CRLF mixed-EOL splice glitch is incidentally fixed. Retired the
now-dead byPhaseTablePattern const.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): phase-list checkbox flip via updateBullet seam

Added updateBullet (markdown-sectionizer): a fence-aware, offset-tracked
single-bullet write primitive (GFM 1–4-space marker tolerance) — the write
counterpart to read-only iterateBullets. Migrated mutateMilestonePhase's
phase-list checkbox flip (`- [ ] Phase N …` → `- [x] … (completed <date>)`)
off its whole-slice regex onto it, same milestone-slice scope + clock seam.
Byte-identical across simple / idempotent / metachar-title / double-space /
CRLF scenarios.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): scope the Progress-ordinal renumber to ## Progress via seam

phase remove's integer-renumber decremented Progress-table phase ordinals with a
whole-document `content.replace(/(\|\s*)(\d+)(\.\s)/g, …)` — unscoped, so it also
rewrote any `| N. …` cell in an unrelated/decoy table (same class as the batch-2
row-delete scoping bug). Migrated onto updateTableCell, scoped to the ## Progress
section, decrementing each affected row's leading phase ordinal by column name.
Byte-identical on canonical Progress tables + multi-row + decimal-sibling cases;
a decoy `| 3. … |` row before ## Progress is now correctly left untouched. The
sibling heading / checkbox-bullet / PLAN.md-filename / Depends-on-prose renumbers
are not GFM-table mutations (outside ADR-2143's table/section mandate) — left as-is.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2143): review fixes — scope traceability write, restore Current Position H3-stop

Adversarial review of the remediation (BLOCK verdict) — all 9 findings fixed:
- F1 (BLOCKER): requirements mark-complete / phase complete flipped the checkbox
  but NOT the traceability row on the shipped template, because updateTableCell
  bound to the FIRST table (## Out of Scope, no Status column) instead of the
  ## Traceability table — the #2140 silent-divergence class, re-introduced by the
  seam migration and missed by tests (fixtures had Traceability first). Scoped
  the write + hasRow probe to the ## Traceability section slice (updateTraceability
  Cell helper) in milestone.cts + phase.cts. Failing-first tests on the
  Out-of-Scope-before-Traceability layout; the #2769 first-cell match preserved.
- F2 (MAJOR): mutateCurrentPositionFirstTime restored to locateCurrentPosition
  (STOP_H2_PLUS) — collectSection's default H2-stop swallowed a level-3 subsection
  and the field regexes clobbered it (#2130 class).
- F3/F8: Progress-ordinal renumber re-escapes via escapeCell + keys padding
  recovery by row index (was de-escaping `\|` and losing padding on dup values).
- F4: insertTableRow escapes cell values internally.
- F5: updateBullet accepts a tab after the marker (`[ \t]{1,4}`).
- F7: resetSectionVerbatim consumes CRLF blank lines (byte-parity on CRLF).
- F6/F9: corrected two misleading comments.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(changeset): data-loss + CRLF-session user-facing fixes (#2253)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(#2143): de-flake the G10 windsurf ReDoS-guard wall-clock assertion

The G10 test asserted `elapsedMs < 1000` for a 200k-char payload — a wall-clock
assertion (CLAUDE.md: never assert on wall-clock time) that flaked on a loaded
node24 bench at ~1.1s. It was redundant: runHook's spawnSync `timeout: 10000`
already SIGKILLs a catastrophic-backtracking hook, so the exit-0 assertion is the
real ReDoS guard. Removed the timing assertion; kept exit-0 + documented the
subprocess-timeout mechanism. Surfaced (not caused) by this branch's gsd-test
runs loading the bench; unrelated to the markdown-parsing changes but fixed in
place per the no-flaky-tests rule.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 14:25:44 -04:00

646 lines
30 KiB
TypeScript

/**
* Frontmatter — YAML frontmatter parsing, serialization, and CRUD commands
*
* ADR-457 build-at-publish: the hand-written bin/lib/frontmatter.cjs collapsed
* to a TypeScript source of truth. Behaviour is preserved byte-for-behaviour
* from the prior hand-written .cjs; only strict types are added.
*/
import fs from 'node:fs';
import path from 'node:path';
// eslint-disable-next-line @typescript-eslint/no-require-imports
import ioMod = require('./io.cjs');
const { output, error } = ioMod;
import { platformReadSync as safeReadFile, platformWriteSync } from './shell-command-projection.cjs';
// ─── Types ────────────────────────────────────────────────────────────────────
type FrontmatterValue = string | string[] | Record<string, unknown>;
type Frontmatter = Record<string, FrontmatterValue>;
// ─── Parsing engine ───────────────────────────────────────────────────────────
/**
* Split a YAML inline array body on commas, respecting quoted strings.
* e.g. '"a, b", c' → ['a, b', 'c']
*/
function splitInlineArray(body: string): string[] {
const items: string[] = [];
let current = '';
let inQuote: string | null = null;
for (let i = 0; i < body.length; i++) {
const ch = body[i];
if (inQuote) {
if (ch === inQuote) {
inQuote = null;
} else {
current += ch;
}
} else if (ch === '"' || ch === "'") {
inQuote = ch;
} else if (ch === ',') {
const trimmed = current.trim();
if (trimmed) items.push(trimmed);
current = '';
} else {
current += ch;
}
}
const trimmed = current.trim();
if (trimmed) items.push(trimmed);
return items;
}
function extractFrontmatter(content: string): Frontmatter {
const frontmatter: Frontmatter = {};
// Match frontmatter only at byte 0 — a `---` block later in the document
// body (YAML examples, horizontal rules) must never be treated as frontmatter.
const headerEnd = content.startsWith('---\r\n') ? 5 : content.startsWith('---\n') ? 4 : -1;
if (headerEnd === -1) return frontmatter;
const closingLineStart = content.indexOf('\n---', headerEnd);
if (closingLineStart === -1) return frontmatter;
const yamlEnd = content[closingLineStart - 1] === '\r' ? closingLineStart - 1 : closingLineStart;
const yaml = content.slice(headerEnd, yamlEnd);
const lines = yaml.split(/\r?\n/);
// Stack to track nested objects: [{obj, key, indent}]
type StackEntry = { obj: Record<string, unknown> | unknown[]; key: string | null; indent: number };
const stack: StackEntry[] = [{ obj: frontmatter, key: null, indent: -1 }];
for (const line of lines) {
// Skip empty lines
if (line.trim() === '') continue;
// Calculate indentation (number of leading spaces)
const indentMatch = line.match(/^(\s*)/);
const indent = indentMatch ? indentMatch[1].length : 0;
// Pop stack back to appropriate level
while (stack.length > 1 && indent <= stack[stack.length - 1].indent) {
stack.pop();
}
const current = stack[stack.length - 1];
// Check for key: value pattern
const keyMatch = line.match(/^(\s*)([a-zA-Z0-9_-]+):\s*(.*)/);
if (keyMatch) {
const key = keyMatch[2];
const value = keyMatch[3].trim();
if (value === '' || value === '[') {
// Key with no value or opening bracket — could be nested object or array
const newObj: Record<string, unknown> | unknown[] = value === '[' ? [] : {};
(current.obj as Record<string, unknown>)[key] = newObj;
current.key = null;
// Push new context for potential nested content
stack.push({ obj: newObj, key: null, indent });
} else if (value.startsWith('[') && value.endsWith(']')) {
// Inline array: key: [a, b, c] — quote-aware split (REG-04 fix)
(current.obj as Record<string, unknown>)[key] = splitInlineArray(value.slice(1, -1));
current.key = null;
} else {
// Simple key: value
(current.obj as Record<string, unknown>)[key] = value.replace(/^["']|["']$/g, '');
current.key = null;
}
} else if (line.trim().startsWith('- ')) {
// Array item
const itemValue = line.trim().slice(2).replace(/^["']|["']$/g, '');
// If current context is an empty object, convert to array
if (typeof current.obj === 'object' && !Array.isArray(current.obj) && Object.keys(current.obj).length === 0) {
// Find the key in parent that points to this object and convert it
const parent = stack.length > 1 ? stack[stack.length - 2] : null;
if (parent) {
for (const k of Object.keys(parent.obj)) {
if ((parent.obj as Record<string, unknown>)[k] === current.obj) {
(parent.obj as Record<string, unknown>)[k] = [itemValue];
current.obj = (parent.obj as Record<string, unknown>)[k] as unknown[];
break;
}
}
}
} else if (Array.isArray(current.obj)) {
current.obj.push(itemValue);
}
}
}
return frontmatter;
}
/**
* Escape a string for emission inside a YAML double-quoted scalar (#1779).
* Backslash must be escaped first so the backslashes added for embedded quotes
* (and control chars) are not themselves doubled. Without this, a value
* carrying an indicator (`:`/`#`) that also contains a literal `"` serializes
* to invalid YAML, e.g. `upstream: "https://x (Tom; "Git. Ship. Done")"`. A
* literal newline/tab/control char inside the quotes likewise breaks (or
* silently alters) the scalar, so those are escaped to their YAML forms too.
*/
function escapeDoubleQuoted(s: string): string {
return s
.replace(/\\/g, '\\\\')
.replace(/"/g, '\\"')
.replace(/\n/g, '\\n')
.replace(/\t/g, '\\t')
.replace(/\r/g, '\\r')
// Remaining C0 controls + DEL → \xHH (a valid YAML double-quoted escape).
.replace(/[\u0000-\u001f\u007f]/g, (c) => `\\x${c.charCodeAt(0).toString(16).padStart(2, '0')}`);
}
/**
* A plain (unquoted) scalar that would mis-parse or round-trip lossily when
* emitted bare must instead go through the double-quoted + escaped form
* (#1779): the empty string (bare `k:` reloads as null), an embedded `"`/`\`
* or control char, a leading YAML indicator (quote, `&`/`*`/`!` anchor/alias/
* tag, `|`/`>` block scalar, flow `[]{},`, `#`, reserved `%`/`@`/backtick, or
* `-`/`?`/`:` before a space), or leading/trailing whitespace. This helper is
* the correctness complement of `escapeDoubleQuoted`: it broadens the *trigger*
* for quoting without broadening the lossy object-list handling deferred to
* #1572/#1660.
*/
function scalarNeedsDoubleQuoting(s: string): boolean {
if (s === '') return true;
if (/["\\\u0000-\u001f\u007f]/.test(s)) return true;
// Always-unsafe leading indicators, or leading/trailing whitespace.
if (/^[,[\]{}#&*!|>'"%@`]/.test(s) || /^\s|\s$/.test(s)) return true;
// `-` `?` `:` only start a plain scalar safely when NOT followed by a space.
if (/^[-?:](\s|$)/.test(s)) return true;
return false;
}
function reconstructFrontmatter(obj: Frontmatter): string {
const lines: string[] = [];
for (const [key, value] of Object.entries(obj)) {
if (value === null || value === undefined) continue;
if (Array.isArray(value)) {
if (value.length === 0) {
lines.push(`${key}: []`);
} else if (value.every(v => typeof v === 'string') && value.length <= 3 && (value).join(', ').length < 60) {
lines.push(`${key}: [${(value).join(', ')}]`);
} else {
lines.push(`${key}:`);
for (const item of value) {
lines.push(` - ${typeof item === 'string' && (item.includes(':') || item.includes('#') || scalarNeedsDoubleQuoting(item)) ? `"${escapeDoubleQuoted(item)}"` : item}`);
}
}
} else if (typeof value === 'object') {
lines.push(`${key}:`);
for (const [subkey, subval] of Object.entries(value)) {
if (subval === null || subval === undefined) continue;
if (Array.isArray(subval)) {
if (subval.length === 0) {
lines.push(` ${subkey}: []`);
} else if (subval.every((v: unknown) => typeof v === 'string') && subval.length <= 3 && (subval).join(', ').length < 60) {
lines.push(` ${subkey}: [${(subval).join(', ')}]`);
} else {
lines.push(` ${subkey}:`);
for (const item of subval) {
lines.push(` - ${typeof item === 'string' && (item.includes(':') || item.includes('#') || scalarNeedsDoubleQuoting(item)) ? `"${escapeDoubleQuoted(item)}"` : item}`);
}
}
} else if (typeof subval === 'object') {
lines.push(` ${subkey}:`);
for (const [subsubkey, subsubval] of Object.entries(subval as Record<string, unknown>)) {
if (subsubval === null || subsubval === undefined) continue;
if (Array.isArray(subsubval)) {
if (subsubval.length === 0) {
lines.push(` ${subsubkey}: []`);
} else {
lines.push(` ${subsubkey}:`);
for (const item of subsubval) {
lines.push(` - ${item}`);
}
}
} else {
// eslint-disable-next-line @typescript-eslint/no-base-to-string, @typescript-eslint/restrict-template-expressions
lines.push(` ${subsubkey}: ${subsubval}`);
}
}
} else {
// eslint-disable-next-line @typescript-eslint/no-base-to-string
const sv = String(subval);
lines.push(` ${subkey}: ${sv.includes(':') || sv.includes('#') || scalarNeedsDoubleQuoting(sv) ? `"${escapeDoubleQuoted(sv)}"` : sv}`);
}
}
} else {
const sv = String(value);
if (sv.includes(':') || sv.includes('#') || sv.startsWith('[') || sv.startsWith('{') || scalarNeedsDoubleQuoting(sv)) {
lines.push(`${key}: "${escapeDoubleQuoted(sv)}"`);
} else {
lines.push(`${key}: ${sv}`);
}
}
}
return lines.join('\n');
}
/**
* Slice a frontmatter YAML body into per-top-level-key raw text segments. Each segment
* runs from a column-0 `key:` line through the line before the next column-0 key (or the
* end), capturing all nested indented content. Used by `spliceFrontmatter` for per-key
* identity preservation (#1572): a structurally-unchanged key keeps its original raw
* text, so the lossy `reconstructFrontmatter` never touches object-lists the caller did
* not modify (e.g. must_haves.artifacts / .prohibitions).
*/
function sliceTopLevelFrontmatterSegments(yaml: string): Array<{ key: string; raw: string }> {
const lines = yaml.split(/\r?\n/);
const segments: Array<{ key: string; raw: string }> = [];
let current: { key: string; raw: string[] } | null = null;
for (const line of lines) {
// A column-0 `key:` (no leading whitespace) starts a new top-level segment.
if (/^[A-Za-z0-9_-]+:/.test(line)) {
if (current) segments.push({ key: current.key, raw: current.raw.join('\n') });
const keyName = (line.match(/^([A-Za-z0-9_-]+):/) as RegExpMatchArray)[1];
current = { key: keyName, raw: [line] };
} else if (current) {
current.raw.push(line);
}
// Stray lines before the first top-level key (rare in frontmatter) are dropped.
}
if (current) segments.push({ key: current.key, raw: current.raw.join('\n') });
return segments;
}
/**
* Regenerate one frontmatter key's serialization, fail-closed if the lossy
* `reconstructFrontmatter` cannot represent the value (#1572 codex review). Object-list
* items (e.g. must_haves.artifacts `{path, provides}` maps) serialize as the literal
* string "[object Object]"; rather than silently emit that and destroy the data, refuse
* so the caller (cmdFrontmatterSet/Merge) errors out WITHOUT writing — directing the
* user to edit the file directly. The reported #1572 case (mutating an UNRELATED field)
* is unaffected: unchanged keys preserve their original raw text and never reach here.
*/
function regenerateFrontmatterKey(key: string, value: FrontmatterValue): string {
const rendered = reconstructFrontmatter({ [key]: value });
if (/\[object Object\]/.test(rendered)) {
throw new Error(
`frontmatter: cannot faithfully serialize key "${key}" — it contains a nested object-list ` +
`(e.g. must_haves.artifacts) the frontmatter writer cannot represent, and serializing it would ` +
`emit "[object Object]". Edit the file directly instead of using frontmatter set/merge.`,
);
}
return rendered;
}
function spliceFrontmatter(content: string, newObj: Frontmatter): string {
const match = content.match(/^---\r?\n[\s\S]+?\r?\n---/);
if (match) {
const fmBlock = match[0];
// Whole-document no-op guard: a true no-op returns content verbatim (byte-exact,
// including any formatting the lossy serializer would normalize).
try {
if (frontmatterDeepEqual(extractFrontmatter(content), newObj)) {
return content;
}
} catch {
/* fall through to regeneration on any comparison hiccup */
}
// Per-key identity preservation (#1572). `reconstructFrontmatter` is a deliberately
// lossy serializer — it cannot faithfully re-emit nested object-list items (e.g.
// must_haves.artifacts / .prohibitions, whose items are `{ path, provides }` /
// `{ statement, status }` maps; `extractFrontmatter` flattens those to scalar
// strings, so a round-trip drops `provides:` and collapses the list to a malformed
// inline array). For any top-level key whose value is STRUCTURALLY UNCHANGED between
// the original parse and `newObj`, preserve that key's ORIGINAL raw text verbatim;
// regenerate only keys that actually changed. This generalizes the whole-document
// no-op guard above to per-key fidelity, so mutating `wave` no longer destroys an
// unrelated `must_haves` block. Keys absent from the original (genuinely new) are
// regenerated and appended; keys absent from `newObj` are preserved (never silently
// deleted by a set/merge).
const fmLines = fmBlock.split(/\r?\n/);
const inner = fmLines.slice(1, -1).join('\n'); // drop the opening `---` and closing `---`
let originalParsed: Frontmatter;
try { originalParsed = extractFrontmatter(fmBlock); } catch { originalParsed = {}; }
const segments = sliceTopLevelFrontmatterSegments(inner);
const emitted: string[] = [];
const seen: Set<string> = new Set();
for (const seg of segments) {
seen.add(seg.key);
if (Object.prototype.hasOwnProperty.call(newObj, seg.key)) {
// Key is in newObj: preserve original raw text if structurally unchanged,
// otherwise regenerate. The key SET is defined by newObj — keys that were in
// the original but are absent from newObj are intentionally dropped (the real
// cmdSet/cmdMerge flow always passes the full merged object, so this only
// matters for direct unit callers and matches spliceFrontmatter's contract:
// the result frontmatter IS newObj).
if (frontmatterDeepEqual(newObj[seg.key], originalParsed[seg.key])) {
emitted.push(seg.raw); // unchanged → preserve original raw text verbatim
} else {
emitted.push(regenerateFrontmatterKey(seg.key, newObj[seg.key])); // changed → regenerate (fail-closed on object-lists)
}
}
// else: key absent from newObj → drop (not emitted).
}
// Append genuinely-new keys not present in the original frontmatter.
for (const k of Object.keys(newObj)) {
if (!seen.has(k)) {
emitted.push(regenerateFrontmatterKey(k, newObj[k]));
}
}
const yamlStr = emitted.join('\n');
return `---\n${yamlStr}\n---` + content.slice(fmBlock.length);
}
// No existing frontmatter — generate from scratch, fail-closed on unrepresentable values.
const yamlStr = reconstructFrontmatter(newObj);
if (/\[object Object\]/.test(yamlStr)) {
throw new Error(
'frontmatter: cannot faithfully serialize the requested frontmatter — it contains a nested ' +
'object-list (e.g. must_haves.artifacts) the writer cannot represent. Edit the file directly.',
);
}
return `---\n${yamlStr}\n---\n\n` + content;
}
/**
* Structural deep-equality for two parsed frontmatter objects. Order-sensitive for arrays
* (YAML lists are ordered), key-order-insensitive for objects. Used only by `spliceFrontmatter`
* to recognize a no-op write-back; intentionally narrow (handles the string / string[] /
* nested-object shapes `extractFrontmatter` produces).
*/
function frontmatterDeepEqual(a: unknown, b: unknown): boolean {
if (a === b) return true;
if (a == null || b == null) return a === b;
if (Array.isArray(a) || Array.isArray(b)) {
if (!Array.isArray(a) || !Array.isArray(b) || a.length !== b.length) return false;
return a.every((v, i) => frontmatterDeepEqual(v, b[i]));
}
if (typeof a === 'object' && typeof b === 'object') {
const ao: Record<string, unknown> = a as Record<string, unknown>;
const bo: Record<string, unknown> = b as Record<string, unknown>;
const ak = Object.keys(ao);
const bk = Object.keys(bo);
if (ak.length !== bk.length) return false;
return ak.every((k) =>
Object.prototype.hasOwnProperty.call(bo, k) && frontmatterDeepEqual(ao[k], bo[k]),
);
}
return false;
}
function parseMustHavesBlock(content: string, blockName: string): unknown[] {
// Extract a specific block from must_haves in raw frontmatter YAML
// Handles 3-level nesting: must_haves > artifacts/key_links > [{path, provides, ...}]
const fmMatch = content.match(/^---\r?\n([\s\S]+?)\r?\n---/);
if (!fmMatch) return [];
const yaml = fmMatch[1];
// Find must_haves: first to detect its indentation level
const mustHavesMatch = yaml.match(/^(\s*)must_haves:\s*$/m);
if (!mustHavesMatch) return [];
const mustHavesIndent = mustHavesMatch[1].length;
// Find the block (e.g., "truths:", "artifacts:", "key_links:") under must_haves
// It must be indented more than must_haves but we detect the actual indent dynamically
const blockPattern = new RegExp(`^(\\s+)${blockName}:\\s*$`, 'm');
const blockMatch = yaml.match(blockPattern);
if (!blockMatch) return [];
const blockIndent = blockMatch[1].length;
// The block must be nested under must_haves (more indented)
if (blockIndent <= mustHavesIndent) return [];
// Find where the block starts in the yaml string
const blockStart = yaml.indexOf(blockMatch[0]);
if (blockStart === -1) return [];
const afterBlock = yaml.slice(blockStart);
const blockLines = afterBlock.split(/\r?\n/).slice(1); // skip the header line
// List items are indented one level deeper than blockIndent
// Continuation KVs are indented one level deeper than list items
const items: unknown[] = [];
let current: string | Record<string, unknown> | null = null;
let listItemIndent = -1; // detected from first "- " line
for (const line of blockLines) {
// Skip empty lines
if (line.trim() === '') continue;
const indentMatch = line.match(/^(\s*)/);
const indent = indentMatch ? indentMatch[1].length : 0;
// Stop at same or lower indent level than the block header
if (indent <= blockIndent && line.trim() !== '') break;
const trimmed = line.trim();
if (trimmed.startsWith('- ')) {
// Detect list item indent from the first occurrence
if (listItemIndent === -1) listItemIndent = indent;
// Only treat as a top-level list item if at the expected indent
if (indent === listItemIndent) {
if (current) items.push(current);
const afterDash = trimmed.slice(2);
const trimmedAfterDash = afterDash.trim();
// Check if it's a fully-quoted string (may contain ':' inside the quotes)
if ((trimmedAfterDash.startsWith('"') && trimmedAfterDash.endsWith('"')) ||
(trimmedAfterDash.startsWith("'") && trimmedAfterDash.endsWith("'"))) {
current = trimmedAfterDash.slice(1, -1);
// Check if it's a simple string item (no colon means not a key-value)
} else if (!afterDash.includes(':')) {
current = afterDash.replace(/^["']|["']$/g, '');
} else {
// Key-value on same line as dash: "- path: value"
// YAML KV always has at least one space after the colon: "key: value"
// Requiring \s+ rejects "Class::Method" and "db:seed" (no space after colon)
const kvMatch = afterDash.match(/^(\w+):\s+"?([^"]*)"?\s*$/);
if (kvMatch) {
current = {};
(current)[kvMatch[1]] = kvMatch[2];
} else {
// Looks like KV but doesn't match — treat as plain string (#2757)
current = afterDash.replace(/^["']|["']$/g, '');
}
}
continue;
}
}
if (current && typeof current === 'object' && indent > listItemIndent) {
// Continuation key-value or nested array item
if (trimmed.startsWith('- ')) {
// Array item under a key
const arrVal = trimmed.slice(2).replace(/^["']|["']$/g, '');
const keys = Object.keys(current);
const lastKey = keys[keys.length - 1];
if (lastKey && !Array.isArray((current)[lastKey])) {
const existing = (current)[lastKey];
(current)[lastKey] = existing ? [existing] : [];
}
if (lastKey) ((current)[lastKey] as unknown[]).push(arrVal);
} else {
const kvMatch = trimmed.match(/^(\w+):\s*"?([^"]*)"?\s*$/);
if (kvMatch) {
// Trim: a quoted value like `"backstop "` captures the inner trailing space in group 2.
// Left untrimmed, a hand-authored `must_haves` marker degrades (a `backstop` truth silently
// grades green instead of abstaining — #1905, the #1154 false-pass; also the sibling
// check_target/violationFixture path). Whitespace is never semantic in a scalar KV value.
const val = kvMatch[2].trim();
// Try to parse as number
(current)[kvMatch[1]] = /^\d+$/.test(val) ? parseInt(val, 10) : val;
}
}
}
}
if (current) items.push(current);
// Warn when must_haves block exists but parsed as empty -- likely YAML formatting issue.
// This is a critical diagnostic: empty must_haves causes verification to silently degrade
// to Option C (LLM-derived truths) instead of checking documented contracts.
if (items.length === 0 && blockLines.length > 0) {
const nonEmptyLines = blockLines.filter(l => l.trim() !== '').length;
if (nonEmptyLines > 0) {
process.stderr.write(
`[gsd-tools] WARNING: must_haves.${blockName} block has ${nonEmptyLines} content lines but parsed 0 items. ` +
`Possible YAML formatting issue — verification will fall back to LLM-derived truths.\n`
);
}
}
return items;
}
// ─── Frontmatter CRUD commands ────────────────────────────────────────────────
const FRONTMATTER_SCHEMAS: Record<string, { required: string[] }> = {
plan: { required: ['phase', 'plan', 'type', 'wave', 'depends_on', 'files_modified', 'autonomous', 'must_haves'] },
summary: { required: ['phase', 'plan', 'subsystem', 'tags', 'duration', 'completed'] },
verification: { required: ['phase', 'verified', 'status', 'score'] },
};
/**
* Strip ALL frontmatter blocks from the start of `content`.
*
* Handles CRLF line endings and multiple stacked blocks (corruption
* recovery): greedily strips consecutive `---...---` blocks separated by
* optional whitespace, so a doubled/tripled frontmatter header (e.g. from a
* botched merge) is fully removed, not just the first block.
*
* Canonical home for this primitive (#2143 audit dedup): previously
* duplicated byte-identically in both `state.cts` and `state-transition.cts`.
*/
function stripFrontmatter(content: string): string {
let result = content;
while (true) {
const stripped = result.replace(/^\s*---\r?\n[\s\S]*?\r?\n---\s*/, '');
if (stripped === result) break;
result = stripped;
}
return result;
}
function cmdFrontmatterGet(cwd: string, filePath: string, field: string | undefined, raw: boolean): void {
if (!filePath) { error('file path required'); }
// Path traversal guard: reject null bytes
if (filePath.includes('\0')) { error('file path contains null bytes'); }
const fullPath = path.isAbsolute(filePath) ? filePath : path.join(cwd, filePath);
const content = safeReadFile(fullPath);
if (!content) { output({ error: 'File not found', path: filePath }, raw, undefined); return; }
const fm = extractFrontmatter(content);
if (field) {
const value = fm[field];
if (value === undefined) { output({ error: 'Field not found', field }, raw, undefined); return; }
output({ [field]: value }, raw, JSON.stringify(value));
} else {
output(fm, raw, undefined);
}
}
function cmdFrontmatterSet(cwd: string, filePath: string, field: string | undefined, value: string | undefined, raw: boolean): void {
if (!filePath || !field || value === undefined) { error('file, field, and value required'); }
// Path traversal guard: reject null bytes
if (filePath.includes('\0')) { error('file path contains null bytes'); }
const fullPath = path.isAbsolute(filePath) ? filePath : path.join(cwd, filePath);
if (!fs.existsSync(fullPath)) { output({ error: 'File not found', path: filePath }, raw, undefined); return; }
const content = fs.readFileSync(fullPath, 'utf-8');
const fm = extractFrontmatter(content);
let parsedValue: unknown;
try { parsedValue = JSON.parse(value as string); } catch { parsedValue = value; }
fm[field as string] = parsedValue as FrontmatterValue;
const newContent = spliceFrontmatter(content, fm);
// #1660: a no-op set (newContent unchanged) with a dict-valued field means the lossy
// frontmatter parser made the new value's projection equal the original's — the change
// did not apply (bites object-list fields like must_haves). Detection lives in the pure
// exported helper noOpObjectListSetError so the mutation gate (property/unit set) covers
// it — the cmd path itself is not in that set.
const noOpErr = noOpObjectListSetError(content, newContent, parsedValue);
if (noOpErr) {
output({ error: noOpErr, field }, raw, undefined);
return;
}
platformWriteSync(fullPath, newContent);
output({ updated: true, field, value: parsedValue }, raw, 'true');
}
/**
* #1660: detect a frontmatter `set` that would be a silent no-op on a dict-valued field.
* Returns an error message when the splice produced no content change but the new value
* is a dict (object-list fields like must_haves, whose `{path, provides}` items flatten to
* scalar strings under extractFrontmatter so a replacement can deep-equal the original's
* projection), else null. Scalars and scalar arrays round-trip faithfully, so idempotent
* sets of those are intentionally NOT flagged. Pure and unit-tested directly (the cmd path
* is not in Stryker's property/unit set, so the detection must be testable in isolation).
*/
function noOpObjectListSetError(originalContent: string, newContent: string, parsedValue: unknown): string | null {
if (newContent !== originalContent) return null;
if (parsedValue === null || typeof parsedValue !== 'object' || Array.isArray(parsedValue)) return null;
return 'frontmatter set had no effect — the supplied value is equivalent to the existing field under the frontmatter parser, which cannot faithfully round-trip object-list fields like must_haves. Edit the file directly.';
}
function cmdFrontmatterMerge(cwd: string, filePath: string, data: string | undefined, raw: boolean): void {
if (!filePath || !data) { error('file and data required'); }
const fullPath = path.isAbsolute(filePath) ? filePath : path.join(cwd, filePath);
if (!fs.existsSync(fullPath)) { output({ error: 'File not found', path: filePath }, raw, undefined); return; }
const content = fs.readFileSync(fullPath, 'utf-8');
const fm = extractFrontmatter(content);
let mergeData: Record<string, FrontmatterValue>;
try { mergeData = JSON.parse(data as string) as Record<string, FrontmatterValue>; } catch { error('Invalid JSON for --data'); return; }
Object.assign(fm, mergeData);
const newContent = spliceFrontmatter(content, fm);
platformWriteSync(fullPath, newContent);
output({ merged: true, fields: Object.keys(mergeData) }, raw, 'true');
}
function cmdFrontmatterValidate(cwd: string, filePath: string, schemaName: string | undefined, raw: boolean): void {
if (!filePath || !schemaName) { error('file and schema required'); }
const schema = FRONTMATTER_SCHEMAS[schemaName as string];
if (!schema) { error(`Unknown schema: ${schemaName}. Available: ${Object.keys(FRONTMATTER_SCHEMAS).join(', ')}`); }
const fullPath = path.isAbsolute(filePath) ? filePath : path.join(cwd, filePath);
const content = safeReadFile(fullPath);
if (!content) { output({ error: 'File not found', path: filePath }, raw, undefined); return; }
const fm = extractFrontmatter(content);
const missing = schema.required.filter(f => fm[f] === undefined);
const present = schema.required.filter(f => fm[f] !== undefined);
output({ valid: missing.length === 0, missing, present, schema: schemaName }, raw, missing.length === 0 ? 'valid' : 'invalid');
}
export = {
extractFrontmatter,
// Additive alias (#644 prohibition-probe schema contract): the probe round-trip seam reads a
// frontmatter object via `parseFrontmatter` (the name the contract test pins). It is the SAME
// function as `extractFrontmatter` — a bare-object parse with no behavior change — exposed under
// the alias so the prohibition schema round-trip and any future caller can use the canonical name.
parseFrontmatter: extractFrontmatter,
reconstructFrontmatter,
spliceFrontmatter,
stripFrontmatter,
noOpObjectListSetError,
parseMustHavesBlock,
FRONTMATTER_SCHEMAS,
cmdFrontmatterGet,
cmdFrontmatterSet,
cmdFrontmatterMerge,
cmdFrontmatterValidate,
};