Files
msd-core/tests/bug-2796-arg-parsing-regression.test.cjs
Tom Boucher ef43f5161f fix(#2969): deterministic Step 5 verification gate for /gsd-reapply-patches (#2972)
* fix(#2969): deterministic Step 5 verification gate for /gsd-reapply-patches

The prior Step 5 "Hunk Verification Gate" was prescribed correctly in the
workflow text — but executed laxly by the LLM, which filled in `verified: yes`
without actually checking content presence. The reporter observed three
distinct files (skills/gsd-discuss-phase/SKILL.md, skills/gsd-autonomous/
SKILL.md, get-shit-done/workflows/new-project.md) where archives contained
substantive user-added blocks that did not survive into the merged result, yet
the gate reported clean.

Move verification from LLM-driven prose into a deterministic Node script the
workflow calls. The script can't be shortcut.

Changes:

- scripts/verify-reapply-patches.cjs (new): pure Node, no external deps.
  For each file in the patches dir, computes user-added significant lines as
  the line-set diff between backup and pristine baseline (when available;
  falls back to "every significant backup line" when no pristine — over-broad
  but the safe direction for this bug class). Asserts each line appears
  literally in the merged installed file via String.prototype.includes.
  Filters trivial lines (length < 12 chars, pure punctuation, decorative
  comments) so harmless drift doesn't trigger false failures. Exits 0 on
  pass, 1 on any miss with per-file diagnostic, 2 on usage error.
  Supports --json for workflow consumption.

- get-shit-done/workflows/reapply-patches.md: rewrite Step 5 to call the
  script and parse its JSON output. The Step 4 Hunk Verification Table
  remains as advisory Claude-readable summary, but the gate is now the
  script's exit code.

- tests/bug-2969-verify-reapply-patches.test.cjs (new): 6 tests covering
  (a) pass when every line survives, (b) fail when a line is missing,
  (c) fail when the merged file is deleted entirely, (d) --json structured
  report shape, (e) backup-meta.json is correctly skipped as metadata,
  (f) no-pristine-dir fallback exercises the safe over-broad path. All pass.

Out of scope: the manifest-baseline tightening described in #2969 Failure 1
(saveLocalPatches comparing against the wrong baseline so prior silent wipes
poison subsequent updates). That's a separate, bigger architectural change
involving pristine-content infrastructure; this PR addresses the gate fidelity
half so users at least see the diagnostic when content goes missing.

Closes #2969 (partial — Failure 2 only)

* fix(#2969): preserve #1999 Hunk Verification Table assertions alongside new script gate

CI failure on PR #2972 surfaced that tests/reapply-patches.test.cjs (the
#1999 contract) asserts Step 5 references:
  - "Hunk Verification Table"
  - `verified: no` failure condition
  - explicit STOP/halt/abort directive
  - "table absent / missing" halt path

My initial Step 5 rewrite for #2969 substituted the deterministic script
for the table-based gate entirely, stripping those references. The script
is the strictly stronger gate, but the existing #1999 test enforces the
table-based safety net as a defense-in-depth contract.

Restore both gates as a layered Step 5:

  - 5a (binding): deterministic verifier script — script gate, exits
    non-zero on any miss, cannot be shortcut by the LLM
  - 5b (advisory): Hunk Verification Table review — preserved as
    redundant safety net for the case where the script has a bug or the
    pristine baseline is unavailable

Both gates must pass. Verified: tests/reapply-patches.test.cjs (5 tests
in the #1999 suite) and tests/bug-2969-verify-reapply-patches.test.cjs
(6 tests in the #2969 suite) all pass — 21/21 total in this fixture.

* fix(#2969): address CodeRabbit findings on workflow + script

Five CR findings on PR #2972, all valid; addressed in this commit:

1. (Major) Stderr was merged into VERIFY_OUTPUT via `2>&1`, so any Node
   warning, deprecation notice, or stack trace would corrupt the JSON
   parse downstream. Capture stdout only; stderr remains on the
   controlling terminal for operator visibility.

2. (Major) verifyFile() crashed with EISDIR/EACCES instead of producing
   a structured diagnostic when the installed path was a directory or
   unreadable. Wrap statSync/readFileSync in try/catch and emit a
   per-file fail row; the whole-run gate continues with structured
   output. Added test case asserting the directory-at-installed-path
   case fails with `not a regular file` diagnostic instead of crashing.

3. (Minor) PRISTINE_FLAG built as a single string + unquoted expansion
   would split paths with spaces. Switched to a bash array (VERIFY_ARGS)
   that preserves whitespace through expansion.

4. (Minor) Fenced code block missing language tag (markdownlint MD040).
   Added `text` tag to the error message block.

5. (Minor) Usage comment said pristine fallback was "backup-meta lookup"
   but the actual code path falls back to significant-line checks from
   backup content. Corrected the comment to match implementation.

Verified all 21 tests in tests/reapply-patches.test.cjs (#1999 contract)
+ tests/bug-2969-verify-reapply-patches.test.cjs (now 7 tests with the
new directory case) pass.

* test(#2969): structured JSON assertions, no substring matching on script output

Replace every assert.match(r.stdout, /pattern/) call with structured
assertions on the parsed JSON report from the script's own --json mode.
The script's --json contract IS the structured shape we test against —
the test author should never depend on the human-readable formatter
output, just as no test should depend on substring presence in source.

Changes:

  - All 7 tests now run the verifier with --json (via a runVerifier()
    helper) and parse the resulting JSON document into { status, report,
    stderr }. Diagnostic stderr is preserved as a separate channel for
    debug output but is not used for assertions.
  - Each previously substring-matched diagnostic ("Failures: 1",
    "not a regular file", "installed file missing after merge",
    file path, dropped line) is now a deepEqual / equal / Array.includes
    against typed report fields: report.failures, report.results[i].status,
    report.results[i].reason, report.results[i].file,
    report.results[i].missing[].
  - Added an explicit "documented shape" test asserting the JSON output
    has exactly the keys { file, missing, reason, status } per result —
    locks the public contract of the --json mode.
  - DRY'd up fixture reset into a resetFixture() helper since every test
    starts with a fresh patches/installed/pristine triple.

Linter: scripts/lint-no-source-grep.cjs reports 0 violations across 348
test files. Combined run of bug-2969-...test.cjs (7 tests) +
reapply-patches.test.cjs (5 tests in the #1999 suite) all pass —
22/22 in the relevant fixture.

* fix(#2969): typed REASON enum + raw-text-matching rule shipped repo-wide

This commit closes the loop on the no-source-grep discipline:

1. scripts/verify-reapply-patches.cjs:
   - Frozen REASON enum exposes the diagnostic surface as stable codes:
     OK_NO_USER_LINES_VS_PRISTINE, OK_NO_SIGNIFICANT_BACKUP_LINES,
     FAIL_INSTALLED_MISSING, FAIL_INSTALLED_NOT_REGULAR_FILE,
     FAIL_READ_ERROR, FAIL_USER_LINES_MISSING.
   - Each result.reason is now a code from this enum, not free text.
     Tests assert via REASON.X equality, not regex on prose.
   - REASON exported from module.exports.

2. tests/bug-2969-verify-reapply-patches.test.cjs:
   - Full rewrite. Every assertion on typed structured fields:
     report.results[0].status === 'fail',
     report.results[0].reason === REASON.FAIL_INSTALLED_NOT_REGULAR_FILE,
     report.results[0].missing.includes(droppedLine) (Array set membership,
     not String substring).
   - Locks the REASON enum surface via Object.keys(REASON).sort() deepEqual.
   - Locks the JSON report shape via Object.keys(report).sort() deepEqual.
   - Zero regex, zero String#includes, zero startsWith/endsWith on text.

3. CONTRIBUTING.md:
   - New section "Prohibited: Raw Text Matching on Test Outputs" with
     concrete BAD/GOOD examples (substring on file content; assert.match
     on stdout; "structured parser" hiding string ops; regex on free-form
     reason fields).
   - The rule statement: "Tests assert on typed structured values. If
     the code under test produces text, the code under test must also
     expose a structured intermediate representation, and the test must
     assert on that IR — never on the rendered text."
   - Required structured-surface table: file IR, --json mode, frozen
     enum, fs facts.
   - "Hiding grep behind a function is still grep" callout — the
     parser-wrapper anti-pattern.
   - New `pre-existing-text-matching` exemption category for the 8
     grandfathered files. Marked Transitional; new tests cannot use it.

4. scripts/lint-no-source-grep.cjs:
   - Three new patterns enforced (in addition to the existing .cjs-source
     readFileSync rule):
     - assert.match/doesNotMatch on .stdout/.stderr
     - .stdout/.stderr.<includes|startsWith|endsWith>(
     - readFileSync(...).<includes|startsWith|endsWith>(
   - Aggregated violations per file (multiple findings now report together).
   - Updated diagnostic message references both CONTRIBUTING.md sections.

5. 8 pre-existing tests annotated with `// allow-test-rule:
   pre-existing-text-matching` so the lint passes on this commit; each
   carries the prose "Tracked for migration to typed-IR assertions; do
   not copy this pattern." Files: bug-2649, bug-2687, bug-2796, bug-2838,
   bug-2943, graphify, hooks-opt-in, security-scan.

Verification: lint 0 violations across 348 test files; full suite passes.

* fix(#2969): rename exemption category to pending-migration-to-typed-ir + cite tracking issue

Per maintainer feedback:
1. "Grandfathered" / "legacy" framing is wrong — both terms imply
   permanent or condoned exemption. The 8 files are tracked for
   correction, not exempted.
2. Each annotated file must cite the tracking issue so the migration
   work is auditable.

Changes:
- CONTRIBUTING.md: rename exemption category from
  `pre-existing-text-matching` to `pending-migration-to-typed-ir`. Update
  prose to "Tracked for correction, not exempted" and require each
  annotation to cite the open migration issue (e.g.
  `// allow-test-rule: pending-migration-to-typed-ir [#NNNN]`).
- 8 test files: update annotation to cite #2974 (the tracking issue
  opened for migrating these files to typed-IR assertions).
2026-05-01 16:14:39 -04:00

158 lines
5.3 KiB
JavaScript

/**
* Regression test for bug #2796
*
* roadmap.update-plan-progress used positional-only arg destructuring:
* `const phaseNum = args[0]`. When called with the flag form documented in
* execute-phase.md:228 (`--phase "TEST" --plan "01" --status "complete"`),
* args[0] was the literal string "--phase", which was passed to findPhase().
* findPhase found no phase named "--phase" and returned `updated: false` with
* `reason: "no matching checkbox found"`, silently no-oping. ROADMAP.md plan
* checkboxes never advanced.
*
* The stateBeginPhase handler already uses parseNamedArgs and is NOT affected.
*
* Fix: roadmap-update-plan-progress.ts now checks for --phase <value> before
* falling back to positional arg[0] (filtering out flag tokens).
*/
'use strict';
// allow-test-rule: pending-migration-to-typed-ir [#2974]
// Tracked in #2974 for migration to typed-IR assertions per CONTRIBUTING.md
// "Prohibited: Raw Text Matching on Test Outputs". Do not copy this pattern.
const { describe, test, beforeEach, afterEach } = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const { execFileSync } = require('node:child_process');
const { createTempGitProject, cleanup } = require('./helpers.cjs');
const REPO_ROOT = path.join(__dirname, '..');
const SDK_CLI = path.join(REPO_ROOT, 'sdk', 'dist', 'cli.js');
function runSdkQuery(subcommand, args, projectDir) {
const argv = ['query', subcommand, ...args, '--project-dir', projectDir];
let stdout = '';
let stderr = '';
let exitCode = 0;
try {
stdout = execFileSync(process.execPath, [SDK_CLI, ...argv], {
encoding: 'utf-8',
stdio: ['pipe', 'pipe', 'pipe'],
env: { ...process.env, GSD_SESSION_KEY: '' },
});
} catch (err) {
exitCode = err.status ?? 1;
stdout = err.stdout?.toString() ?? '';
stderr = err.stderr?.toString() ?? '';
}
let json = null;
try { json = JSON.parse(stdout.trim()); } catch { /* ok */ }
return { exitCode, stdout: stdout.trim(), stderr: stderr.trim(), json };
}
/** Create a minimal ROADMAP.md with a phase checkbox */
function createRoadmap(projectDir, phaseNum, planLabel) {
const planningDir = path.join(projectDir, '.planning');
fs.mkdirSync(path.join(planningDir, 'phases'), { recursive: true });
const roadmap = [
'# My Project Roadmap',
'',
'## v1.0 — MVP',
'',
`### Phase ${phaseNum}: Test Phase`,
'',
`**Plans:** 1/1 plans complete`,
'',
`| # | Phase | Plans | Status | Date |`,
`|---|-------|-------|--------|------|`,
`| ${phaseNum} | Test Phase | 0/1 | Planned | |`,
'',
`- [ ] ${planLabel}: Do the thing`,
'',
].join('\n');
fs.writeFileSync(path.join(planningDir, 'ROADMAP.md'), roadmap);
// Create the phase directory so findPhase finds it
const phaseDir = path.join(
planningDir, 'phases',
`${String(phaseNum).padStart(2, '0')}-test-phase`
);
fs.mkdirSync(phaseDir, { recursive: true });
// Create a plan file and a summary so progress = 1/1
fs.writeFileSync(path.join(phaseDir, `${planLabel}-PLAN.md`), '# Plan\n');
fs.writeFileSync(path.join(phaseDir, `${planLabel}-SUMMARY.md`), '# Summary\n');
return phaseDir;
}
describe('bug-2796: roadmap update-plan-progress accepts --phase flag', () => {
let tmpDir;
beforeEach(() => {
tmpDir = createTempGitProject('gsd-test-2796-');
});
afterEach(() => {
cleanup(tmpDir);
});
test('flag form --phase <N> resolves the correct phase (not literal "--phase")', () => {
createRoadmap(tmpDir, '9', '01');
// Flag form: this is the form execute-phase.md:228 uses
const result = runSdkQuery(
'roadmap.update-plan-progress',
['--phase', '9'],
tmpDir
);
// Before fix: exitCode=1 with "phase --phase not found" or updated:false
// After fix: should succeed with phase="9" and updated:true
assert.strictEqual(result.exitCode, 0, `should exit 0; stderr: ${result.stderr}`);
assert.ok(result.json !== null, 'should emit JSON');
assert.ok(result.json.updated === true, 'updated should be true');
assert.strictEqual(String(result.json.phase), '9', 'phase should be "9", not "--phase"');
});
test('positional form still works (backward compat)', () => {
createRoadmap(tmpDir, '9', '01');
const result = runSdkQuery(
'roadmap.update-plan-progress',
['9'],
tmpDir
);
assert.strictEqual(result.exitCode, 0, `should exit 0; stderr: ${result.stderr}`);
assert.ok(result.json?.updated === true, 'updated should be true');
assert.strictEqual(String(result.json.phase), '9', 'phase should be "9"');
});
test('flag form does not pass "--phase" as the phase value to findPhase', () => {
// Before fix: findPhase("--phase") returned found:false, causing updated:false
// This test confirms the phase value is not the flag name itself.
createRoadmap(tmpDir, '5', '01');
const result = runSdkQuery(
'roadmap.update-plan-progress',
['--phase', '5'],
tmpDir
);
// If the phase was "--phase", we'd get an error like "Phase --phase not found"
assert.ok(
!result.stderr.includes('--phase not found'),
'stderr must not say "--phase not found"'
);
assert.ok(
!result.stderr.includes('"--phase"'),
'stderr must not treat "--phase" as the phase value'
);
});
});