* chore(#2371): representative gate-fixture corpus + document-shaped property test Adds tests/fixtures/representative/ — a permanent corpus of verbatim, incident-sourced fixtures (never author-invented) from #2286, #2347, #2365, #2366, each labeled with its expected gate verdict in a MANIFEST.json and driven through the real CLI gate entrypoint via tests/representative-corpus.test.cjs. Adds a document-shaped fast-check property test alongside the existing writer-seeded bijection test in tests/api-coverage.test.cjs: the existing generator produces rows and renders them through the writer, so the document shape is a constant and it cannot fail against a decoy table; the new one generates the document space instead. Two gates (#2365, #2347) are still open, so their corpus/property assertions are marked with node:test's official `todo` option — the test executes and reports its failure without affecting the process exit code (https://nodejs.org/api/test.html#test-options). The audit-uat corpus (#2286, fixed by #2317) is a normal passing assertion, proving the methodology works end to end and not just cataloguing gaps. Records the fixture-provenance rule in CONTRIBUTING.md: a gate's fixtures may not be derived from the gate's own writer, grammar, or docstring examples; a negative fixture must come from a source that doesn't know the gate exists. No production src/*.cts changes — validation only, per #2371's scope. Closes #2371 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#2371): address orthogonal-review findings — dedup generators, fix field naming, wire dead fields Standards-axis review findings, all fixed: - Deduplicated the row-shape generators (capabilityGen/rowGen/validRowGen) that were copy-pasted between the parse/render bijection test and the new document-shaped property test in tests/api-coverage.test.cjs — a future edit to one could have silently desynced the two properties. Hoisted to a single module-scope declaration both tests reference. - Renamed decision-coverage-guard/MANIFEST.json's expectedOutcome -> expectedReason. It asserted against the gate's `reason` field, but this codebase already has a real, different `outcome` field at parser altitude (extractDecisions' DecisionOutcome) — naming the manifest field after the wrong altitude's term was exactly the ambiguity the "Fixture provenance" rule this PR adds exists to eliminate. - Removed the unused `role` field from three MANIFEST.json files (never read by any test) and wired the previously-dead per-fixture `expectedMinItems` in audit-uat/MANIFEST.json into a real per-file assertion in tests/representative-corpus.test.cjs, using cmdAuditUat's `results` array — catches a regression that moves items between the two fixture files while preserving the aggregate total, which the existing total_items check alone would miss. No changes to test intent or coverage — same assertions, correctly named and fully wired. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: remove dead capabilityState/capabilityWriter requires from gsd-tools.cjs Surfaced by the mandatory pre-PR lint gate (no-unused-vars) while preparing this PR — unrelated to #2371's own changes, but a defect found while working is fixed in place rather than deferred. Leftover from #2368/#2370 (merged just before this branch rebased onto it): the case 'capability' arm that needed these two requires was relocated to bin/lib/capability-command-router.cjs, which already requires both modules directly (lines 24-25) and is their only real consumer (cmdCapabilityState, resolveCapabilityRuntimeState, cmdCapabilitySet). The two requires left behind in gsd-tools.cjs had zero other references in the file and were never re-exported — confirmed via grep across the file and its module.exports. Behavior-preserving: Node's require cache means the underlying modules still load exactly once via capability-command-router.cjs's own requires; gsd-tools.cjs never used its now-removed local bindings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#2371): replace todo-marked assertions with characterization tests gsd-test's own JSONL result parser (gsd-test-runner's internal/pipeline/parse.go, verified directly against that repo's source) has no concept of node:test's `todo` option — it only recognizes kind:"pass"|"fail" and hard-errors on anything else. A { todo: true } test whose body throws is counted as a real failure in gsd-test's own verdict, exactly as if it weren't marked todo — proven by an actual gsd-test run against this branch, which reported outcome:"failed" with all six todo-marked assertions (the property test plus five representative-corpus fixtures) in the failure list, each carrying the correct raw node:test `todo` field the tool's parser simply doesn't read. Replaces todo with characterization: MANIFEST.json now carries both the correct target verdict (expected*) and the exact current observed verdict (currentBuggyOutput, directly verified against live CLI output for all five fixtures). Tests assert currentBuggyOutput — an honest, non-vacuous pin of today's known-broken reality that passes today and will fail loudly the moment the referenced fix changes the observed output, at which point the assertion should be flipped to expected* and currentBuggyOutput deleted. The document-shaped property test switches from throwing fc.assert to non-throwing fc.check (returns RunDetails per fast-check's own docs) and asserts report.failed === true directly, for the same reason. Updates all prose (CONTRIBUTING.md, the fixture READMEs) that previously claimed todo would be respected — that claim was factually wrong for this repo's actual tooling and must not ship. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: regenerate golden-install-parity fixtures after rebase onto next Rebasing onto the current next (which now includes #2381's todo-severity changes to gsd-core/bin/gsd-tools.cjs) produced real conflicts in all 18 golden-install-parity fixtures — expected, since both branches changed the same gsd-tools.cjs hash entry. Resolved by taking one side to unblock the rebase, then regenerating fresh from source via npm run gen:golden and verifying the result; every file's diff is exactly the one hash line for gsd-core/bin/gsd-tools.cjs, correcting a stale intermediate hash from the arbitrary conflict pick. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -501,6 +501,14 @@ Required cases where relevant:
|
||||
|
||||
Property-style parser tests are encouraged for high-risk parsers. They must be deterministic: pin the seed, bound the iteration count, and print replay data on failure.
|
||||
|
||||
##### Fixture provenance (#2371)
|
||||
|
||||
**A gate's fixtures may not be derived from the gate's own writer, grammar, or docstring examples. A negative fixture must come from a source that does not know the gate exists.**
|
||||
|
||||
This is stricter than the adversarial-input rule above and exists because of it: `tests/fixtures/adversarial/` covers hostile input, but a fixture written by the parser's own author — even a deliberately "realistic" one — is still drawn from the author's mental model of the format. It can only ever confirm what the author already believed, never surface what they didn't anticipate. A property-test generator has the same failure mode one level up: seeding the generator from the writer/render function that produces the same format makes the document shape a constant, so the property can never explore a document the writer wouldn't produce (see the document-shaped vs. writer-seeded property tests in `tests/api-coverage.test.cjs` for a worked example — the writer-seeded one cannot fail against a decoy table; the document-shaped one can).
|
||||
|
||||
For a gate whose fixtures come from real user reports, put them under `tests/fixtures/representative/<gate>/` with a `MANIFEST.json` labeling each fixture's source issue and expected gate verdict, and drive them through the gate's real CLI entrypoint (gate-verdict altitude), not the parser function in isolation — see `tests/fixtures/representative/README.md` and `tests/representative-corpus.test.cjs`. If the gate is not yet fixed, do not mark the assertion `{ todo: true }` and do not skip it: this repo's test-runner (`gsd-test` / `gsd-test-runner`) has no concept of node:test's `todo` option — its JSONL result parser only recognizes `kind: "pass" | "fail"`, so a thrown todo-marked test is still counted as a real failure and blocks the push gate. Instead record BOTH the correct target verdict (`expected*`) and the exact current observed verdict (`currentBuggyOutput`) in the manifest, and assert against `currentBuggyOutput` — an honest, non-vacuous characterization of today's known-broken behavior that passes today and breaks loudly the moment the real fix changes the observed output, forcing the assertion to be flipped to `expected*`.
|
||||
|
||||
#### Filesystem writes and installers
|
||||
|
||||
Changes to install/uninstall flows, generated artifact writers, state/config writers, worktree safety, or any code that writes under `.planning`, runtime config dirs, `.claude`, `.codex`, `hooks`, or generated files must include fault-injection coverage where the seam allows it.
|
||||
|
||||
@@ -285,8 +285,6 @@ const { routeInitCommand } = require('./lib/init-command-router.cjs');
|
||||
// here, invoked from case 'init' below.
|
||||
const { warnIfStaleBake } = require('./lib/stale-bake-guard.cjs');
|
||||
const loopResolver = require('./lib/loop-resolver.cjs');
|
||||
const capabilityState = require('./lib/capability-state.cjs');
|
||||
const capabilityWriter = require('./lib/capability-writer.cjs');
|
||||
const { routePhaseCommand } = require('./lib/phase-command-router.cjs');
|
||||
const { routePhasesCommand } = require('./lib/phases-command-router.cjs');
|
||||
const { routeValidateCommand } = require('./lib/validate-command-router.cjs');
|
||||
|
||||
@@ -20,6 +20,25 @@ const fc = require('fast-check');
|
||||
|
||||
const MODULE_PATH = path.join(__dirname, '..', 'gsd-core', 'bin', 'lib', 'api-coverage.cjs');
|
||||
|
||||
// Shared row-shape generators for the coverage-matrix property tests below
|
||||
// (the parse/render bijection and the #2371 document-shaped property both
|
||||
// build matrices from the same canonical row shape — a single declaration
|
||||
// here means the two properties can't silently desync).
|
||||
const capabilityGen = fc.stringMatching(/^[a-z][a-z0-9-]{0,14}$/);
|
||||
const rowGen = fc.record({
|
||||
capability: capabilityGen,
|
||||
decision: fc.constantFrom('INTEGRATE', 'OPT-OUT'),
|
||||
// Reasons are short prose (e.g. "not needed yet"). The matrix is a
|
||||
// markdown table, so cell text is format-safe: no pipes / newlines.
|
||||
reason: fc.stringMatching(/^[a-z0-9 ,.\-!?]{0,20}$/),
|
||||
});
|
||||
// OPT-OUT rows must carry a non-empty reason for the round-trip to validate.
|
||||
const validRowGen = rowGen.map((r) =>
|
||||
r.decision === 'OPT-OUT' && r.reason.trim() === ''
|
||||
? { ...r, reason: 'because' }
|
||||
: { ...r, reason: r.reason.trim() }
|
||||
);
|
||||
|
||||
describe('detectApiIntegration — pure detector (#1562)', () => {
|
||||
let mod;
|
||||
try {
|
||||
@@ -346,20 +365,6 @@ describe('coverage matrix — parse/render bijection (fast-check)', () => {
|
||||
const { renderCoverageMatrix, validateCoverageMatrix } = mod;
|
||||
|
||||
test('any valid row set renders and re-validates to the same counts', () => {
|
||||
const capabilityGen = fc.stringMatching(/^[a-z][a-z0-9-]{0,14}$/);
|
||||
const rowGen = fc.record({
|
||||
capability: capabilityGen,
|
||||
decision: fc.constantFrom('INTEGRATE', 'OPT-OUT'),
|
||||
// Reasons are short prose (e.g. "not needed yet"). The matrix is a
|
||||
// markdown table, so cell text is format-safe: no pipes / newlines.
|
||||
reason: fc.stringMatching(/^[a-z0-9 ,.\-!?]{0,20}$/),
|
||||
});
|
||||
// OPT-OUT rows must carry a non-empty reason for the round-trip to validate.
|
||||
const validRowGen = rowGen.map((r) =>
|
||||
r.decision === 'OPT-OUT' && r.reason.trim() === ''
|
||||
? { ...r, reason: 'because' }
|
||||
: { ...r, reason: r.reason.trim() }
|
||||
);
|
||||
const matrixGen = fc.uniqueArray(validRowGen, {
|
||||
minLength: 1,
|
||||
maxLength: 8,
|
||||
@@ -379,6 +384,121 @@ describe('coverage matrix — parse/render bijection (fast-check)', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ──────────────────────────────────────────────────────────────────────────────
|
||||
// Document-shaped property (#2371): the bijection test above generates ROWS and
|
||||
// renders them through the writer, so the document shape is a constant — it
|
||||
// cannot generate a second table, a decoy table, or surrounding prose, and so
|
||||
// cannot fail against #2366's bugs. This property generates the DOCUMENT
|
||||
// space instead: a canonical matrix interleaved with content a real
|
||||
// COVERAGE.md may legitimately contain that is NOT the matrix. See
|
||||
// tests/fixtures/representative/README.md and CONTRIBUTING.md's "Fixture
|
||||
// provenance" section for the full rationale.
|
||||
// ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
describe('coverage matrix — document-shaped fast-check (extract-exactly-canonical, #2371)', () => {
|
||||
let mod;
|
||||
try {
|
||||
mod = require(MODULE_PATH);
|
||||
} catch (err) {
|
||||
throw new Error(`Could not require ${MODULE_PATH}. Run "npm run build:lib" first. Underlying: ${err.message}`);
|
||||
}
|
||||
const { parseCoverageMatrix, renderCoverageMatrix } = mod;
|
||||
|
||||
// Uses the shared capabilityGen/rowGen/validRowGen declared at module scope
|
||||
// above (same generators the bijection test uses), so the two properties
|
||||
// exercise the same canonical-row space and can't silently desync.
|
||||
const canonicalMatrixGen = fc.uniqueArray(validRowGen, {
|
||||
minLength: 1,
|
||||
maxLength: 5,
|
||||
selector: (r) => r.capability.toLowerCase(),
|
||||
});
|
||||
|
||||
// Decoy blocks: content a document may legitimately contain that is NOT the
|
||||
// canonical matrix. Kept to three explicit, independently-readable shapes
|
||||
// rather than a generic "random markdown" generator — a combinatorial but
|
||||
// opaque generator is exactly the kind of cleverness that's unrunnable to
|
||||
// debug when it fails (Kernighan's Law).
|
||||
const proseDecoyGen = fc.constantFrom(
|
||||
'## Notes\n\nSee the ADR for background.',
|
||||
'This phase also touches the auth helper.',
|
||||
'## Risks\n\n- Rollout risk is low.',
|
||||
);
|
||||
|
||||
const summaryTableDecoyGen = fc
|
||||
.record({
|
||||
label: fc.stringMatching(/^[a-z][a-z0-9 ]{0,10}$/),
|
||||
integrateCount: fc.nat({ max: 50 }),
|
||||
optoutCount: fc.nat({ max: 50 }),
|
||||
})
|
||||
.map(
|
||||
({ label, integrateCount, optoutCount }) =>
|
||||
`## Coverage summary\n\n| tier | INTEGRATE | OPT-OUT |\n|---|---|---|\n` +
|
||||
`| ${label} | ${integrateCount} | ${optoutCount} |`
|
||||
);
|
||||
|
||||
const secondSectionMatrixGen = fc
|
||||
.uniqueArray(validRowGen, { minLength: 1, maxLength: 3, selector: (r) => r.capability.toLowerCase() })
|
||||
.map((rows) => `## Transferred to a later phase\n\n${renderCoverageMatrix(rows)}`);
|
||||
|
||||
const decoyGen = fc.oneof(proseDecoyGen, summaryTableDecoyGen, secondSectionMatrixGen);
|
||||
|
||||
const documentGen = fc.record({
|
||||
canonicalRows: canonicalMatrixGen,
|
||||
decoysBefore: fc.array(decoyGen, { maxLength: 2 }),
|
||||
decoysAfter: fc.array(decoyGen, { maxLength: 2 }),
|
||||
});
|
||||
|
||||
// #2371's own test-runner (gsd-test / gsd-test-runner v1.6.2) has no concept
|
||||
// of node:test's `todo` option: its JSONL result parser
|
||||
// (internal/pipeline/parse.go's parseJSONL, gsd-test-runner repo) only
|
||||
// recognizes `kind: "pass" | "fail"` and hard-errors on anything else, so a
|
||||
// `{ todo: true }` test whose body throws is still counted as a failure in
|
||||
// the tool's own verdict — verified directly against that source, not
|
||||
// assumed. So this property uses fc's non-throwing `fc.check` (returns
|
||||
// `RunDetails` instead of throwing — see fast-check's runners docs) and
|
||||
// asserts on `.failed` directly: today the invariant genuinely does NOT
|
||||
// hold (that is #2366), so `report.failed === true` is an honest,
|
||||
// non-vacuous, currently-PASSING characterization of today's known-broken
|
||||
// reality — not a fake pass. The moment #2366 makes the invariant hold for
|
||||
// real, `report.failed` becomes `false` and THIS assertion fails loudly,
|
||||
// forcing whoever's fix landed to notice and flip it. The fix itself stays
|
||||
// owned by #2366.
|
||||
test(
|
||||
'given a document containing exactly one canonical matrix plus arbitrary other content, ' +
|
||||
'the parser extracts exactly that matrix\'s rows and ignores everything else ' +
|
||||
'(currently violated — #2366)',
|
||||
() => {
|
||||
const report = fc.check(
|
||||
fc.property(documentGen, ({ canonicalRows, decoysBefore, decoysAfter }) => {
|
||||
const canonicalBlock = renderCoverageMatrix(canonicalRows);
|
||||
const doc = [...decoysBefore, canonicalBlock, ...decoysAfter].join('\n\n');
|
||||
|
||||
const result = parseCoverageMatrix(doc);
|
||||
|
||||
const expectedByCap = new Map(canonicalRows.map((r) => [r.capability.toLowerCase(), r]));
|
||||
const actualByCap = new Map(result.rows.map((r) => [r.capability.toLowerCase(), r]));
|
||||
|
||||
if (actualByCap.size !== expectedByCap.size) return false;
|
||||
for (const [cap, expected] of expectedByCap) {
|
||||
const actual = actualByCap.get(cap);
|
||||
if (!actual || actual.decision !== expected.decision) return false;
|
||||
}
|
||||
return result.errors.length === 0;
|
||||
}),
|
||||
{ numRuns: 100 }
|
||||
);
|
||||
assert.strictEqual(
|
||||
report.failed,
|
||||
true,
|
||||
'This property is expected to be VIOLATED today (#2366 — a decoy summary table or a ' +
|
||||
'second canonical-schema section corrupts the result or spuriously errors). If this ' +
|
||||
'assertion fails, the property now HOLDS — #2366 appears fixed; replace this ' +
|
||||
'characterization with a real fc.assert of the invariant.'
|
||||
);
|
||||
}
|
||||
);
|
||||
});
|
||||
|
||||
// ──────────────────────────────────────────────────────────────────────────────
|
||||
// CLI entry point (STDIN → exit codes mirror grep, like assumption-delta)
|
||||
// ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "ea841e2865248e74",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48a2355e15c585c2",
|
||||
"gsd-core/bin/gsd-tools.cjs": "0cfb0889b6fc5928",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -110,7 +110,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "51bc64467ab30f62",
|
||||
"gsd-core/bin/gsd-tools.cjs": "2041725176ce6b66",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48b736dd16863746",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -109,7 +109,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "51bc64467ab30f62",
|
||||
"gsd-core/bin/gsd-tools.cjs": "2041725176ce6b66",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48b736dd16863746",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -38,7 +38,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "51bc64467ab30f62",
|
||||
"gsd-core/bin/gsd-tools.cjs": "2041725176ce6b66",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48b736dd16863746",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -42,7 +42,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "476aa24e8c4f03cf",
|
||||
"gsd-core/bin/gsd-tools.cjs": "bc864a9bf3a21f8b",
|
||||
"gsd-core/bin/gsd-tools.cjs": "ecb7831b68ecb4a9",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -110,7 +110,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "51bc64467ab30f62",
|
||||
"gsd-core/bin/gsd-tools.cjs": "2041725176ce6b66",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48b736dd16863746",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -145,7 +145,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "51bc64467ab30f62",
|
||||
"gsd-core/bin/gsd-tools.cjs": "2041725176ce6b66",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48b736dd16863746",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -40,7 +40,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "ea841e2865248e74",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48a2355e15c585c2",
|
||||
"gsd-core/bin/gsd-tools.cjs": "0cfb0889b6fc5928",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -110,7 +110,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "2525f1ae8b086828",
|
||||
"gsd-core/bin/gsd-tools.cjs": "39106b83bdc47046",
|
||||
"gsd-core/bin/gsd-tools.cjs": "6e31bf5c3515b677",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "3a3409215044af9f",
|
||||
"gsd-core/bin/gsd-tools.cjs": "b663dd5ecf091d51",
|
||||
"gsd-core/bin/gsd-tools.cjs": "b0385bba7000b281",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -110,7 +110,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "51bc64467ab30f62",
|
||||
"gsd-core/bin/gsd-tools.cjs": "2041725176ce6b66",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48b736dd16863746",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -103,7 +103,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "51bc64467ab30f62",
|
||||
"gsd-core/bin/gsd-tools.cjs": "2041725176ce6b66",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48b736dd16863746",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -110,7 +110,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "51bc64467ab30f62",
|
||||
"gsd-core/bin/gsd-tools.cjs": "2041725176ce6b66",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48b736dd16863746",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
2
tests/fixtures/golden-install-parity/pi.json
vendored
2
tests/fixtures/golden-install-parity/pi.json
vendored
@@ -6,7 +6,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "51bc64467ab30f62",
|
||||
"gsd-core/bin/gsd-tools.cjs": "2041725176ce6b66",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48b736dd16863746",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "6e98d76e955e35a2",
|
||||
"gsd-core/bin/gsd-tools.cjs": "b0e0c49e82e33a9c",
|
||||
"gsd-core/bin/gsd-tools.cjs": "8bdd0b02837b7a6e",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "de4627dff103d527",
|
||||
"gsd-core/bin/gsd-tools.cjs": "4e19039b6a346562",
|
||||
"gsd-core/bin/gsd-tools.cjs": "d7b5484a84a5bd15",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "5636ca0b726871b2",
|
||||
"gsd-core/bin/gsd-tools.cjs": "196f1439ef0da939",
|
||||
"gsd-core/bin/gsd-tools.cjs": "09a4dc673d2fd74e",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
@@ -110,7 +110,7 @@
|
||||
"gsd-core/VERSION": "ef0deccd81a6723c",
|
||||
"gsd-core/bin/check-latest-version.cjs": "e4a224058c8f4d74",
|
||||
"gsd-core/bin/ensure-runtime-build.cjs": "51bc64467ab30f62",
|
||||
"gsd-core/bin/gsd-tools.cjs": "2041725176ce6b66",
|
||||
"gsd-core/bin/gsd-tools.cjs": "48b736dd16863746",
|
||||
"gsd-core/bin/gsd_run": "62d9b647ede212e6",
|
||||
"gsd-core/bin/shared/config-defaults.manifest.json": "517e6a7c1e9f4f16",
|
||||
"gsd-core/bin/shared/config-schema.manifest.json": "0109a5a9866a24e0",
|
||||
|
||||
82
tests/fixtures/representative/README.md
vendored
Normal file
82
tests/fixtures/representative/README.md
vendored
Normal file
@@ -0,0 +1,82 @@
|
||||
# Representative Gate Fixtures (#2371)
|
||||
|
||||
Every fixture in this tree is **verbatim** (or, where noted, a minimal
|
||||
faithful subset) of an artifact that a real user actually produced and
|
||||
reported against a real GSD gate. None of it was written by a gate's own
|
||||
author to exercise that gate.
|
||||
|
||||
## Why this directory exists, and why `tests/fixtures/adversarial/` isn't enough
|
||||
|
||||
`tests/fixtures/adversarial/` covers hostile input — unicode, CRLF, nested
|
||||
fences, heredoc breakout. Nobody attacked the gates these fixtures target.
|
||||
A developer wrote an ordinary, well-formed artifact — a plan, a coverage
|
||||
matrix, a CONTEXT.md — that a gate misjudged anyway. That input is neither
|
||||
synthetic-happy nor adversarial; it's simply *real*, and until #2371 no gate
|
||||
had coverage for it.
|
||||
|
||||
The pattern this corpus exists to break: a gate's test fixtures were
|
||||
authored by the same person (or model) who wrote the gate, from the same
|
||||
mental model, so they can only confirm what the author already believed —
|
||||
never surface what the author didn't anticipate. See #2371 for the full
|
||||
diagnosis (four incidents across three gates in ten days, including a
|
||||
fast-check property test whose generator was seeded from the parser's own
|
||||
writer function and therefore could not fail).
|
||||
|
||||
## Rule
|
||||
|
||||
**A gate's fixtures may not be derived from the gate's own writer, grammar,
|
||||
or docstring examples. A negative fixture must come from a source that
|
||||
does not know the gate exists.** Recorded in `CONTRIBUTING.md` under
|
||||
"Fixture provenance."
|
||||
|
||||
## Layout
|
||||
|
||||
Each subdirectory is one gate:
|
||||
|
||||
- `api-coverage-detector/` — `detectApiIntegration` (#2365)
|
||||
- `api-coverage-matrix/` — `parseCoverageMatrix` (#2366)
|
||||
- `audit-uat/` — `parseUatItems` / `parseVerificationItems` (#2286, fixed by #2317)
|
||||
- `decision-coverage-guard/` — `extractDecisions`'s could-not-parse guard (#1365 gap, #2347)
|
||||
|
||||
Each carries its own `README.md` (what each fixture is and where it came
|
||||
from) and `MANIFEST.json` (machine-readable: file → source issue → gate →
|
||||
expected verdict). `tests/representative-corpus.test.cjs` loads every
|
||||
manifest and drives each fixture through the gate's real CLI entrypoint —
|
||||
never the parser function directly — so the assertion is at gate-verdict
|
||||
altitude (the boolean/JSON a user actually sees), not parse-tree altitude.
|
||||
|
||||
## Why the still-broken fixtures assert `currentBuggyOutput`, not a red `todo`
|
||||
|
||||
Three of the four gates here are still open bugs (#2365, #2366, #2347) at
|
||||
the time this corpus was added. Their `MANIFEST.json` entries carry BOTH
|
||||
the correct target verdict (`expected*` — what the eventual fix must
|
||||
produce) and the exact CURRENT observed verdict (`currentBuggyOutput` —
|
||||
what today's code actually returns). The test asserts against
|
||||
`currentBuggyOutput`: an honest, non-vacuous characterization of today's
|
||||
known-broken reality, not a fake pass.
|
||||
|
||||
This is deliberately NOT node:test's `todo` option. `todo` looked like the
|
||||
right tool — a todo test executes and reports its failure without
|
||||
affecting Node's own process exit code
|
||||
(https://nodejs.org/api/test.html#test-options) — but this repo's actual
|
||||
test-runner (`gsd-test` / `gsd-test-runner` v1.6.2) has no concept of it:
|
||||
its JSONL result parser (`internal/pipeline/parse.go`'s `parseJSONL`, in
|
||||
the separate `gsd-test-runner` repo) only recognizes `kind: "pass" | "fail"`
|
||||
and hard-errors on anything else — verified directly against that source,
|
||||
not assumed. A `{ todo: true }` test whose body throws is still counted as
|
||||
a real failure in `gsd-test`'s own verdict, which would block the push
|
||||
gate exactly as if it weren't marked todo at all.
|
||||
|
||||
Asserting `currentBuggyOutput` sidesteps this because the test genuinely
|
||||
passes today — no runner-level "expected failure" feature required. The
|
||||
fixes belong to #2365 / #2366 / #2347, not to this corpus. When one of
|
||||
those lands, the corresponding assertion will fail (the gate now returns
|
||||
something other than the pinned buggy value) — at that point, flip the
|
||||
test to assert `expected*` instead and delete the stale
|
||||
`currentBuggyOutput`.
|
||||
|
||||
The `audit-uat/` corpus has no `todo`: #2286 was fixed by #2317 before this
|
||||
corpus was written, so its assertions are ordinary, currently-passing
|
||||
tests — the proof that a representative fixture, driven through the real
|
||||
gate, is not automatically doomed to fail. It demonstrates the methodology
|
||||
working, not just the gaps it finds.
|
||||
29
tests/fixtures/representative/api-coverage-detector/MANIFEST.json
vendored
Normal file
29
tests/fixtures/representative/api-coverage-detector/MANIFEST.json
vendored
Normal file
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"gate": "api-coverage.verify-pre",
|
||||
"sourceIssue": "#2365",
|
||||
"fixtures": [
|
||||
{
|
||||
"file": "nextjs-route-path.txt",
|
||||
"expectedDetected": false,
|
||||
"currentBuggyOutput": { "detected": true, "signal": { "verb": "integration", "noun": "api" } },
|
||||
"note": "First-party Next.js route path. The noun-boundary class [^a-zA-Z0-9] treats '/' as a word boundary, so 'api' inside the path matches as if it were prose. Highest blast radius: any Next.js project names a route file."
|
||||
},
|
||||
{
|
||||
"file": "unrelated-verb-noun.txt",
|
||||
"expectedDetected": false,
|
||||
"currentBuggyOutput": { "detected": true, "signal": { "verb": "wiring", "noun": "endpoint" } },
|
||||
"note": "'wiring' and 'endpoint' co-occur on one line in unrelated clauses, reverse semantic order, no integration described. The verb/noun regexes are independently exec'd over the whole line with no proximity or grammatical relation."
|
||||
},
|
||||
{
|
||||
"file": "threat-model-prose.txt",
|
||||
"expectedDetected": false,
|
||||
"currentBuggyOutput": { "detected": true, "signal": { "verb": "(surface)", "noun": "api" } },
|
||||
"note": "Threat-model table cell describing a LOCAL interface. SERVICE_SURFACE_API_RE matches any capitalized word before API/SDK/REST/GraphQL; the stopword denylist cannot enumerate every ordinary English word that precedes 'API' in a sentence."
|
||||
},
|
||||
{
|
||||
"file": "non-integration-assertion.txt",
|
||||
"expectedDetected": false,
|
||||
"note": "This line explicitly ASSERTS non-integration ('no new command/dependency') and is still read as an integration signal in the real $gsd-verify-work occurrence this was drawn from — but in isolation it already returns detected:false today (the multi-signal real occurrence needed the OTHER lines' signals to trip the gate). No currentBuggyOutput: this fixture already passes for the right reason."
|
||||
}
|
||||
]
|
||||
}
|
||||
23
tests/fixtures/representative/api-coverage-detector/README.md
vendored
Normal file
23
tests/fixtures/representative/api-coverage-detector/README.md
vendored
Normal file
@@ -0,0 +1,23 @@
|
||||
# API-coverage detector fixtures (#2365)
|
||||
|
||||
Verbatim reproduction lines from #2365, each used as a `PLAN.md` body and
|
||||
driven through `check api-coverage.verify-pre <phaseDir>` — the real
|
||||
blocking gate, not `detectApiIntegration()` called in isolation.
|
||||
|
||||
- `nextjs-route-path.txt` — a first-party Next.js route path read as an
|
||||
external API signal because `/` is a word boundary.
|
||||
- `unrelated-verb-noun.txt` — a verb and a noun on the same line, unrelated
|
||||
clauses, no compound relation.
|
||||
- `threat-model-prose.txt` — a threat-model table cell describing a LOCAL
|
||||
interface, misread as a third-party service name.
|
||||
- `non-integration-assertion.txt` — a line that explicitly states no new
|
||||
integration was added, misread as evidence of one.
|
||||
|
||||
The first three currently `detected: true`; the gate expects `false` for
|
||||
all four (see `MANIFEST.json`'s `expectedDetected`/`currentBuggyOutput`
|
||||
fields). `non-integration-assertion.txt` already returns `detected: false`
|
||||
in isolation, so it's asserted directly. The other three are asserted
|
||||
against their `currentBuggyOutput` in `tests/representative-corpus.test.cjs`
|
||||
— a characterization of today's known-broken behavior, not a `todo` (see
|
||||
`tests/fixtures/representative/README.md` for why `todo` doesn't work with
|
||||
this repo's test-runner) — until #2365 lands.
|
||||
1
tests/fixtures/representative/api-coverage-detector/nextjs-route-path.txt
vendored
Normal file
1
tests/fixtures/representative/api-coverage-detector/nextjs-route-path.txt
vendored
Normal file
@@ -0,0 +1 @@
|
||||
Run integration tests for src/app/api/profile/route.test.ts
|
||||
1
tests/fixtures/representative/api-coverage-detector/non-integration-assertion.txt
vendored
Normal file
1
tests/fixtures/representative/api-coverage-detector/non-integration-assertion.txt
vendored
Normal file
@@ -0,0 +1 @@
|
||||
Reuse existing validator APIs; no new command/dependency.
|
||||
1
tests/fixtures/representative/api-coverage-detector/threat-model-prose.txt
vendored
Normal file
1
tests/fixtures/representative/api-coverage-detector/threat-model-prose.txt
vendored
Normal file
@@ -0,0 +1 @@
|
||||
| Tampering | Resolver-only API rejects arbitrary caller URLs. |
|
||||
1
tests/fixtures/representative/api-coverage-detector/unrelated-verb-noun.txt
vendored
Normal file
1
tests/fixtures/representative/api-coverage-detector/unrelated-verb-noun.txt
vendored
Normal file
@@ -0,0 +1 @@
|
||||
Render the page and prove label endpoint, filename, and CSV/XLSX wiring.
|
||||
23
tests/fixtures/representative/api-coverage-matrix/MANIFEST.json
vendored
Normal file
23
tests/fixtures/representative/api-coverage-matrix/MANIFEST.json
vendored
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"gate": "api-coverage.verify-pre",
|
||||
"sourceIssue": "#2366",
|
||||
"fixtures": [
|
||||
{
|
||||
"file": "multi-table-with-summary.md",
|
||||
"pairedPlan": "Integrate the Stripe API for payment processing.",
|
||||
"expectedCounts": { "surface": 3, "integrate": 1, "optout": 2 },
|
||||
"expectedErrorCount": 0,
|
||||
"expectedBlock": false,
|
||||
"currentBuggyOutput": {
|
||||
"block": true,
|
||||
"error_count": 3,
|
||||
"errors": [
|
||||
"row: decision \"**OPT-OUT**\" not in {INTEGRATE, OPT-OUT}",
|
||||
"row: decision \"DECISION\" not in {INTEGRATE, OPT-OUT}",
|
||||
"row: decision \"12\" not in {INTEGRATE, OPT-OUT}"
|
||||
]
|
||||
},
|
||||
"note": "Self-contained 15-line COVERAGE.md from #2366's own reproduction: one canonical matrix (search/skip), a second section-split canonical table (widget) for a transferred capability, and a decoy 3-column summary table whose 2nd cell reads 'INTEGRATE'. Today's parser invents a 'tier' capability from the summary table's header row (silent corruption, zero errors) and drops 'skip' (bolded decision rejected), while also spuriously erroring on the second header — the errors array above shows the rejected-DECISION-header and the rejected-bolded-OPT-OUT and rejected-numeric-'12'-cell paths, none of which mention the silently-invented 'tier' row (that corruption produces no error at all, which is the headline finding). Expected once fixed: exactly 3 rows (search, skip, widget), 0 errors — see expectedCounts/expectedErrorCount/expectedBlock."
|
||||
}
|
||||
]
|
||||
}
|
||||
23
tests/fixtures/representative/api-coverage-matrix/README.md
vendored
Normal file
23
tests/fixtures/representative/api-coverage-matrix/README.md
vendored
Normal file
@@ -0,0 +1,23 @@
|
||||
# API-coverage matrix fixture (#2366)
|
||||
|
||||
`multi-table-with-summary.md` is the verbatim 15-line `repro-coverage.md`
|
||||
from #2366's own reproduction, used as a `COVERAGE.md` body and driven
|
||||
through `check api-coverage.verify-pre <phaseDir>` (paired with a PLAN.md
|
||||
that integrates an API, so a matrix is required) — the real blocking gate,
|
||||
not `parseCoverageMatrix()` called in isolation.
|
||||
|
||||
Contains, in one file: the canonical `| capability | decision | reason |`
|
||||
matrix, a second canonical-schema table under a "Transferred to a later
|
||||
phase" heading (the section-split use case #2366 names as legitimate), and
|
||||
a decoy 3-column "Coverage summary" table whose header row happens to read
|
||||
`| tier | INTEGRATE | OPT-OUT |`.
|
||||
|
||||
Expected once fixed: 3 rows (`search`, `skip`, `widget`), 0 errors (see
|
||||
`MANIFEST.json`'s `expectedCounts`/`expectedErrorCount`/`expectedBlock`).
|
||||
Today's parser instead invents a `tier` capability from the summary table
|
||||
(silent corruption — zero errors reported for that path) while rejecting
|
||||
the bolded `skip` decision and two other cells, producing 3 errors and
|
||||
`block: true` — pinned in `MANIFEST.json`'s `currentBuggyOutput` and
|
||||
asserted directly in `tests/representative-corpus.test.cjs` (a
|
||||
characterization of today's known-broken behavior, not a `todo` — see
|
||||
`tests/fixtures/representative/README.md` for why) until #2366 lands.
|
||||
18
tests/fixtures/representative/api-coverage-matrix/multi-table-with-summary.md
vendored
Normal file
18
tests/fixtures/representative/api-coverage-matrix/multi-table-with-summary.md
vendored
Normal file
@@ -0,0 +1,18 @@
|
||||
# API Coverage — demo
|
||||
|
||||
| capability | decision | reason |
|
||||
|---|---|---|
|
||||
| search | INTEGRATE | |
|
||||
| skip | **OPT-OUT** | not needed yet |
|
||||
|
||||
## Transferred to a later phase
|
||||
|
||||
| capability | decision | reason |
|
||||
|---|---|---|
|
||||
| widget | OPT-OUT | deferred to 9 |
|
||||
|
||||
## Coverage summary
|
||||
|
||||
| tier | INTEGRATE | OPT-OUT |
|
||||
|---|---|---|
|
||||
| phase 8 | 12 | 6 |
|
||||
20
tests/fixtures/representative/audit-uat/MANIFEST.json
vendored
Normal file
20
tests/fixtures/representative/audit-uat/MANIFEST.json
vendored
Normal file
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"gate": "audit-uat",
|
||||
"sourceIssue": "#2286",
|
||||
"fixedBy": "#2317",
|
||||
"fixtures": [
|
||||
{
|
||||
"file": "gaps-section-uat.md",
|
||||
"filenameSuffix": "-UAT.md",
|
||||
"expectedMinItems": 1,
|
||||
"note": "A '## Gaps' bullet entry with status: open. Before #2317, parseUatItems never scanned this section at all, so this file's real outstanding item was silently invisible."
|
||||
},
|
||||
{
|
||||
"file": "human-verification-frontmatter.md",
|
||||
"filenameSuffix": "-VERIFICATION.md",
|
||||
"expectedMinItems": 1,
|
||||
"note": "Frontmatter declares status: human_needed with a populated human_verification: array. Before #2317, parseVerificationItems never read the frontmatter's structured array and only recognized specific body shapes, so this file read as zero items."
|
||||
}
|
||||
],
|
||||
"expectedTotalItems": 2
|
||||
}
|
||||
17
tests/fixtures/representative/audit-uat/README.md
vendored
Normal file
17
tests/fixtures/representative/audit-uat/README.md
vendored
Normal file
@@ -0,0 +1,17 @@
|
||||
# Audit-UAT fixtures (#2286, fixed by #2317)
|
||||
|
||||
Verbatim reproduction files from #2286, driven through `gsd-tools
|
||||
audit-uat --raw` (the real CLI gate) rather than `parseUatItems` /
|
||||
`parseVerificationItems` called in isolation.
|
||||
|
||||
- `gaps-section-uat.md` — a UAT file whose only outstanding finding lives
|
||||
in a `## Gaps` bullet entry.
|
||||
- `human-verification-frontmatter.md` — a VERIFICATION file whose
|
||||
frontmatter declares a structured `human_verification:` array.
|
||||
|
||||
This is the one corpus in `tests/fixtures/representative/` with **no**
|
||||
`todo` marker. #2286 was fixed by #2317 (merged) before this corpus was
|
||||
written, so `total_items >= 2` is a normal, currently-passing assertion —
|
||||
proof that a representative fixture, driven through the real gate, is not
|
||||
automatically doomed to fail. It demonstrates the methodology working end
|
||||
to end, not just the gaps it finds in the other three gates.
|
||||
4
tests/fixtures/representative/audit-uat/gaps-section-uat.md
vendored
Normal file
4
tests/fixtures/representative/audit-uat/gaps-section-uat.md
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
## Gaps
|
||||
|
||||
- truth: "SC1: some success criterion"
|
||||
status: open
|
||||
11
tests/fixtures/representative/audit-uat/human-verification-frontmatter.md
vendored
Normal file
11
tests/fixtures/representative/audit-uat/human-verification-frontmatter.md
vendored
Normal file
@@ -0,0 +1,11 @@
|
||||
---
|
||||
status: human_needed
|
||||
human_verification:
|
||||
- test: "Confirm the widget renders correctly"
|
||||
---
|
||||
|
||||
## Human Verification Required
|
||||
|
||||
### 1. Widget render check
|
||||
|
||||
**Confirm the widget appears as expected on the dashboard.**
|
||||
18
tests/fixtures/representative/decision-coverage-guard/MANIFEST.json
vendored
Normal file
18
tests/fixtures/representative/decision-coverage-guard/MANIFEST.json
vendored
Normal file
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"gate": "check.decision-coverage-plan",
|
||||
"sourceIssue": "#2347",
|
||||
"fixtures": [
|
||||
{
|
||||
"file": "d5-prefix-context.md",
|
||||
"expectedReason": "could-not-parse",
|
||||
"expectedPassed": false,
|
||||
"currentBuggyOutput": {
|
||||
"passed": true,
|
||||
"skipped": true,
|
||||
"reason": "no trackable decisions",
|
||||
"total": 0
|
||||
},
|
||||
"note": "A <decisions> block using the D5-01 ID-prefix shape from #2347's own reproduction ('- **D5-01:** some decision'), repeated twice so 'populated but 0 extracted' is unambiguous. The original report used 23 real decisions under a project-specific D5- prefix convention; this fixture preserves the exact grammar mismatch, not the count. The #1365 guard's evidence test (/\\bD-[A-Za-z0-9]/) shares the parser's own D- grammar, so it is blind to exactly the input class it exists to catch: both see nothing, and the gate reports passed:true, skipped:true, reason:'no trackable decisions' instead of failing loud."
|
||||
}
|
||||
]
|
||||
}
|
||||
24
tests/fixtures/representative/decision-coverage-guard/README.md
vendored
Normal file
24
tests/fixtures/representative/decision-coverage-guard/README.md
vendored
Normal file
@@ -0,0 +1,24 @@
|
||||
# Decision-coverage guard fixture (#2347)
|
||||
|
||||
`d5-prefix-context.md` is the verbatim reproduction shape from #2347 — the
|
||||
`- **D5-01:** some decision` bullet given in the issue's own "Steps to
|
||||
reproduce" — used as a CONTEXT.md `<decisions>` block and driven through
|
||||
`query check.decision-coverage-plan <phaseDir> <contextPath>` (the real
|
||||
CLI gate; see `tests/decisions.test.cjs` for the established pattern this
|
||||
follows), not `extractDecisions()` called in isolation.
|
||||
|
||||
The #1365 fail-loud guard's "is this decision-shaped?" evidence test
|
||||
(`/\bD-[A-Za-z0-9]/`) reuses the same `D-` grammar as the parser it guards.
|
||||
For any ID prefix the parser cannot read — `D5-01` here — the guard sees
|
||||
no evidence either, so the two failure modes the guard exists to
|
||||
distinguish (`none-present` vs `could-not-parse`) collapse into
|
||||
`none-present`, and a populated, genuinely decision-shaped CONTEXT.md
|
||||
passes silently.
|
||||
|
||||
Expected once fixed: `reason: 'could-not-parse'`, `passed: false` (see
|
||||
`MANIFEST.json`'s `expectedReason`/`expectedPassed`). Today's gate instead
|
||||
reports `passed: true, skipped: true, reason: 'no trackable decisions'` —
|
||||
pinned in `MANIFEST.json`'s `currentBuggyOutput` and asserted directly in
|
||||
`tests/representative-corpus.test.cjs` (a characterization of today's
|
||||
known-broken behavior, not a `todo` — see
|
||||
`tests/fixtures/representative/README.md` for why) until #2347 lands.
|
||||
4
tests/fixtures/representative/decision-coverage-guard/d5-prefix-context.md
vendored
Normal file
4
tests/fixtures/representative/decision-coverage-guard/d5-prefix-context.md
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
<decisions>
|
||||
- **D5-01:** some decision
|
||||
- **D5-02:** some other decision
|
||||
</decisions>
|
||||
261
tests/representative-corpus.test.cjs
Normal file
261
tests/representative-corpus.test.cjs
Normal file
@@ -0,0 +1,261 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Representative-corpus gate tests (#2371).
|
||||
*
|
||||
* Every fixture under tests/fixtures/representative/ is verbatim (or a
|
||||
* minimal faithful subset) of a real reported artifact — never invented to
|
||||
* match a gate's own grammar. See tests/fixtures/representative/README.md
|
||||
* for the full rationale and CONTRIBUTING.md's "Fixture provenance" rule.
|
||||
*
|
||||
* Each gate is driven through its real CLI entrypoint (gate-verdict
|
||||
* altitude), matching the established pattern in
|
||||
* tests/api-coverage-gate-e2e.test.cjs and tests/decisions.test.cjs — never
|
||||
* the parser function called in isolation.
|
||||
*
|
||||
* Three fixtures (across api-coverage-detector, api-coverage-matrix,
|
||||
* decision-coverage-guard) encode gates that are still open bugs (#2365,
|
||||
* #2366, #2347). For those, MANIFEST.json carries BOTH the correct target
|
||||
* verdict (`expected*` — what the fix must produce) and the exact CURRENT
|
||||
* observed verdict (`currentBuggyOutput` — what today's code actually
|
||||
* returns). The test asserts against `currentBuggyOutput`: an honest,
|
||||
* non-vacuous characterization of today's known-broken reality, not a fake
|
||||
* pass. This assertion WILL fail, loudly, the moment the underlying bug is
|
||||
* fixed and the gate starts returning something other than the pinned
|
||||
* buggy value — at which point whoever's fix landed must update the
|
||||
* assertion to check `expected*` instead (and can delete `currentBuggyOutput`).
|
||||
*
|
||||
* Why not node:test's `todo` option: this repo's own test-runner
|
||||
* (gsd-test / gsd-test-runner v1.6.2) has no concept of it. Its JSONL
|
||||
* result parser (internal/pipeline/parse.go's parseJSONL, gsd-test-runner
|
||||
* repo) only recognizes `kind: "pass" | "fail"` — verified directly against
|
||||
* that source — so a `{ todo: true }` test whose body throws is still
|
||||
* counted as a real failure in the tool's own verdict. Characterization
|
||||
* (assert the known-current value) sidesteps this because the test
|
||||
* genuinely passes today; it needs no runner-level "expected failure"
|
||||
* feature at all.
|
||||
*
|
||||
* The audit-uat corpus (#2286, fixed by #2317) has no currentBuggyOutput:
|
||||
* it already asserts the correct behavior directly, because the bug is
|
||||
* already fixed — proof the methodology works end to end, not just a
|
||||
* record of gaps.
|
||||
*/
|
||||
|
||||
const { describe, test, afterEach } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { execFileSync } = require('node:child_process');
|
||||
|
||||
const { cleanup } = require('./helpers.cjs');
|
||||
|
||||
const TOOLS_PATH = path.join(__dirname, '..', 'gsd-core', 'bin', 'gsd-tools.cjs');
|
||||
const FIXTURES_ROOT = path.join(__dirname, 'fixtures', 'representative');
|
||||
|
||||
const TEST_ENV_BASE = {
|
||||
GSD_SESSION_KEY: '',
|
||||
CODEX_THREAD_ID: '',
|
||||
CLAUDE_SESSION_ID: '',
|
||||
CLAUDE_CODE_SSE_PORT: '',
|
||||
OPENCODE_SESSION_ID: '',
|
||||
GEMINI_SESSION_ID: '',
|
||||
CURSOR_SESSION_ID: '',
|
||||
WINDSURF_SESSION_ID: '',
|
||||
TERM_SESSION: '',
|
||||
WT_SESSION: '',
|
||||
TMUX_PANE: '',
|
||||
ZELLIJ_SESSION_NAME: '',
|
||||
TTY: '',
|
||||
SSH_TTY: '',
|
||||
};
|
||||
|
||||
function runTools(args, cwd) {
|
||||
try {
|
||||
const stdout = execFileSync(process.execPath, [TOOLS_PATH, ...args], {
|
||||
cwd,
|
||||
encoding: 'utf-8',
|
||||
env: { ...process.env, ...TEST_ENV_BASE },
|
||||
timeout: 60000,
|
||||
});
|
||||
return { success: true, output: stdout.trim(), error: '' };
|
||||
} catch (err) {
|
||||
return {
|
||||
success: false,
|
||||
output: err.stdout?.toString().trim() || '',
|
||||
error: err.stderr?.toString().trim() || err.message,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function readManifest(gateDir) {
|
||||
const raw = fs.readFileSync(path.join(FIXTURES_ROOT, gateDir, 'MANIFEST.json'), 'utf8');
|
||||
return JSON.parse(raw);
|
||||
}
|
||||
|
||||
function readFixture(gateDir, file) {
|
||||
return fs.readFileSync(path.join(FIXTURES_ROOT, gateDir, file), 'utf8');
|
||||
}
|
||||
|
||||
function makeProject() {
|
||||
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-repcorpus-'));
|
||||
fs.mkdirSync(path.join(tmpDir, '.planning', 'phases'), { recursive: true });
|
||||
fs.writeFileSync(path.join(tmpDir, '.planning', 'config.json'), '{}', 'utf8');
|
||||
return tmpDir;
|
||||
}
|
||||
|
||||
function makePhaseDir(projectDir, slug) {
|
||||
const dir = path.join(projectDir, '.planning', 'phases', slug);
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
return dir;
|
||||
}
|
||||
|
||||
// ─── api-coverage-detector (#2365) ────────────────────────────────────────────
|
||||
|
||||
describe('representative corpus — api-coverage detector (#2365)', () => {
|
||||
let tmpDir;
|
||||
afterEach(() => { if (tmpDir) { cleanup(tmpDir); tmpDir = null; } });
|
||||
|
||||
const manifest = readManifest('api-coverage-detector');
|
||||
|
||||
for (const fx of manifest.fixtures) {
|
||||
const label = fx.currentBuggyOutput ? `${fx.file} → currently detected:true (#2365)` : `${fx.file} → detected:false`;
|
||||
test(label, () => {
|
||||
tmpDir = makeProject();
|
||||
const phaseDir = makePhaseDir(tmpDir, '01-repcorpus');
|
||||
const body = readFixture('api-coverage-detector', fx.file);
|
||||
fs.writeFileSync(path.join(phaseDir, '01-PLAN.md'), `# Plan\n${body}\n`, 'utf8');
|
||||
|
||||
const r = runTools(['check', 'api-coverage.verify-pre', phaseDir, '--raw'], tmpDir);
|
||||
assert.ok(r.success, `gate should succeed (JSON). stderr: ${r.error}`);
|
||||
const j = JSON.parse(r.output);
|
||||
|
||||
if (fx.currentBuggyOutput) {
|
||||
assert.strictEqual(j.detected, fx.currentBuggyOutput.detected,
|
||||
`${fx.file}: expected today's known-buggy detected:${fx.currentBuggyOutput.detected}, got ${JSON.stringify(j)}. ` +
|
||||
`If this now differs, #2365 may be fixed — check against expectedDetected:${fx.expectedDetected} instead.`);
|
||||
assert.strictEqual(j.signals?.[0]?.verb, fx.currentBuggyOutput.signal.verb, `${fx.file}: signal.verb`);
|
||||
assert.strictEqual(j.signals?.[0]?.noun, fx.currentBuggyOutput.signal.noun, `${fx.file}: signal.noun`);
|
||||
} else {
|
||||
assert.strictEqual(j.detected, fx.expectedDetected,
|
||||
`${fx.file}: expected detected:${fx.expectedDetected}, got ${JSON.stringify(j)}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// ─── api-coverage-matrix (#2366) ──────────────────────────────────────────────
|
||||
|
||||
describe('representative corpus — api-coverage matrix (#2366)', () => {
|
||||
let tmpDir;
|
||||
afterEach(() => { if (tmpDir) { cleanup(tmpDir); tmpDir = null; } });
|
||||
|
||||
const manifest = readManifest('api-coverage-matrix');
|
||||
|
||||
for (const fx of manifest.fixtures) {
|
||||
const label = fx.currentBuggyOutput
|
||||
? `${fx.file} → currently silently-corrupted + spurious errors (#2366)`
|
||||
: `${fx.file} → exactly the canonical rows, 0 errors`;
|
||||
test(label, () => {
|
||||
tmpDir = makeProject();
|
||||
const phaseDir = makePhaseDir(tmpDir, '01-repcorpus');
|
||||
fs.writeFileSync(path.join(phaseDir, '01-PLAN.md'), `# Plan\n${fx.pairedPlan}\n`, 'utf8');
|
||||
fs.writeFileSync(path.join(phaseDir, 'COVERAGE.md'), readFixture('api-coverage-matrix', fx.file), 'utf8');
|
||||
|
||||
const r = runTools(['check', 'api-coverage.verify-pre', phaseDir, '--raw'], tmpDir);
|
||||
assert.ok(r.success, `gate should succeed (JSON). stderr: ${r.error}`);
|
||||
const j = JSON.parse(r.output);
|
||||
|
||||
if (fx.currentBuggyOutput) {
|
||||
assert.strictEqual(j.block, fx.currentBuggyOutput.block,
|
||||
`${fx.file}: expected today's known-buggy block:${fx.currentBuggyOutput.block}, got ${JSON.stringify(j)}. ` +
|
||||
`If this now differs, #2366 may be fixed — check against expectedBlock:${fx.expectedBlock} instead.`);
|
||||
assert.strictEqual(j.error_count, fx.currentBuggyOutput.error_count, `${fx.file}: error_count`);
|
||||
assert.deepStrictEqual(j.errors, fx.currentBuggyOutput.errors, `${fx.file}: errors`);
|
||||
} else {
|
||||
assert.strictEqual(j.block, fx.expectedBlock, `${fx.file}: block. Got ${JSON.stringify(j)}`);
|
||||
assert.deepStrictEqual(j.counts, fx.expectedCounts, `${fx.file}: counts. Got ${JSON.stringify(j)}`);
|
||||
assert.strictEqual((j.errors || []).length, fx.expectedErrorCount,
|
||||
`${fx.file}: errors. Got ${JSON.stringify(j.errors)}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// ─── audit-uat (#2286, fixed by #2317 — asserts correct behavior directly) ────
|
||||
|
||||
describe('representative corpus — audit-uat (#2286, fixed by #2317)', () => {
|
||||
let tmpDir;
|
||||
afterEach(() => { if (tmpDir) { cleanup(tmpDir); tmpDir = null; } });
|
||||
|
||||
const manifest = readManifest('audit-uat');
|
||||
|
||||
test('Gaps-section + human-verification-frontmatter fixtures both surface as real items', () => {
|
||||
tmpDir = makeProject();
|
||||
const phaseDir = makePhaseDir(tmpDir, '01-repcorpus');
|
||||
for (const fx of manifest.fixtures) {
|
||||
fs.writeFileSync(
|
||||
path.join(phaseDir, `01${fx.filenameSuffix}`),
|
||||
readFixture('audit-uat', fx.file),
|
||||
'utf8',
|
||||
);
|
||||
}
|
||||
|
||||
const r = runTools(['audit-uat', '--raw'], tmpDir);
|
||||
assert.ok(r.success, `audit-uat should succeed. stderr: ${r.error}`);
|
||||
const j = JSON.parse(r.output);
|
||||
assert.ok(
|
||||
j.summary.total_items >= manifest.expectedTotalItems,
|
||||
`expected total_items >= ${manifest.expectedTotalItems}, got ${JSON.stringify(j.summary)}`,
|
||||
);
|
||||
// Per-fixture check (not just the aggregate): a regression that moves
|
||||
// items between files while preserving the total would slip past the
|
||||
// total_items check above but not this one.
|
||||
for (const fx of manifest.fixtures) {
|
||||
const fileName = `01${fx.filenameSuffix}`;
|
||||
const fileResult = j.results.find((r2) => r2.file === fileName);
|
||||
assert.ok(fileResult, `expected a result entry for ${fileName}, got ${JSON.stringify(j.results)}`);
|
||||
assert.ok(
|
||||
fileResult.items.length >= fx.expectedMinItems,
|
||||
`${fileName}: expected items.length >= ${fx.expectedMinItems}, got ${fileResult.items.length}`,
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ─── decision-coverage-guard (#2347) ──────────────────────────────────────────
|
||||
|
||||
describe('representative corpus — decision-coverage guard (#2347)', () => {
|
||||
let tmpDir;
|
||||
afterEach(() => { if (tmpDir) { cleanup(tmpDir); tmpDir = null; } });
|
||||
|
||||
const manifest = readManifest('decision-coverage-guard');
|
||||
|
||||
for (const fx of manifest.fixtures) {
|
||||
const label = fx.currentBuggyOutput
|
||||
? `${fx.file} → currently passed:true, skipped:true (#2347)`
|
||||
: `${fx.file} → outcome could-not-parse, passed:false`;
|
||||
test(label, () => {
|
||||
tmpDir = makeProject();
|
||||
const phaseDir = makePhaseDir(tmpDir, '01-repcorpus');
|
||||
const contextPath = path.join(phaseDir, 'CONTEXT.md');
|
||||
fs.writeFileSync(contextPath, readFixture('decision-coverage-guard', fx.file), 'utf8');
|
||||
|
||||
const r = runTools(['query', 'check.decision-coverage-plan', phaseDir, contextPath], tmpDir);
|
||||
assert.ok(r.success, `gate should succeed (JSON). stderr: ${r.error}`);
|
||||
const j = JSON.parse(r.output);
|
||||
|
||||
if (fx.currentBuggyOutput) {
|
||||
assert.strictEqual(j.passed, fx.currentBuggyOutput.passed,
|
||||
`${fx.file}: expected today's known-buggy passed:${fx.currentBuggyOutput.passed}, got ${JSON.stringify(j)}. ` +
|
||||
`If this now differs, #2347 may be fixed — check against expectedPassed:${fx.expectedPassed} instead.`);
|
||||
assert.strictEqual(j.skipped, fx.currentBuggyOutput.skipped, `${fx.file}: skipped`);
|
||||
assert.strictEqual(j.reason, fx.currentBuggyOutput.reason, `${fx.file}: reason`);
|
||||
assert.strictEqual(j.total, fx.currentBuggyOutput.total, `${fx.file}: total`);
|
||||
} else {
|
||||
assert.strictEqual(j.passed, fx.expectedPassed, `${fx.file}: passed. Got ${JSON.stringify(j)}`);
|
||||
assert.strictEqual(j.reason, fx.expectedReason, `${fx.file}: reason. Got ${JSON.stringify(j)}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
});
|
||||
Reference in New Issue
Block a user