enhance(#3304): gate re-verification blockers on deterministic evidence (#4085)

* test(#3304): add failing-first suite for the convergence evidence gate

Content-assertion suite for the Step 7 re-verification evidence gate
(agents/gsd-verifier.md / gsd-core/references/verifier-evidence-gate.md).
Committed before the implementation to prove RED via gsd-test.

* enhance(#3304): gate re-verification blockers on deterministic evidence

Step 7's anti-pattern scan re-runs at full, unbounded scope on every
re-verification pass, independent of the must-haves established in Step 2.
A blocker it finds — other than the self-evidencing debt-marker check —
previously reverted a completed gap-closure round and started another
--gaps cycle on nothing more than the verifier's own new judgment call,
with no bound on how many times that could repeat.

A Step 7 blocker now blocks unconditionally in re-verification mode only
if it is a carried-forward gap (present in the prior VERIFICATION.md's
gaps: list) or the flagged file was git-modified since the prior pass
(a regression; fails closed toward blocking when history is unresolvable).
Otherwise it predates the gap-closure round unflagged and needs
deterministic evidence — a named test run red, or another concrete
reproducible artifact — to stay blocking. Unevidenced, it downgrades to
a new advisory: frontmatter list and report section instead of setting
status: gaps_found, and never reverts a completed must-have.

Maintainer approval was narrowed to this evidence condition only,
explicitly rejecting the broader "advisory whenever untraceable to a
requirement/decision/prior-gap" proposal — implemented and pinned by
tests/verifier-evidence-gate.test.cjs and documented as rejected in
gsd-core/references/verifier-evidence-gate.md so it can't silently
re-expand.

Closes #3304

* fix(#3304): correct window-truncation and indentation bugs in evidence-gate tests

gsd-test's GREEN checkpoint caught 3 real bugs in the test file itself
(not the production prose): a {0,600} match window was shorter than the
724-char paragraph it was scanning (the "exclude from Step 9 Rule 1"
phrase starts at offset 662), and two regexes assumed no indentation
after a markdown list-continuation line break. All three phrases are
confirmed unique across agents/gsd-verifier.md, so the windowed
submatches are replaced with direct whole-string assertions instead of
just widening the window.

Also acknowledges the deliberate byte growth in agents/gsd-verifier.md
that the differential-attribution check (ADR-2719) correctly flagged.

Emitted-Drift-Ack-Growth: gsd-verifier.md — adds the #3304 re-verification evidence gate (Step 7 rule, Advisory bucket, advisory: frontmatter, report section); 1488 bytes, still within the LARGE-tier 48 KiB cap (48751/49152).

* docs(#3304): backfill changeset PR number

---------

Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
2026-08-30 16:51:52 -04:00
committed by GitHub
parent 62b0d939b6
commit 6beaa66b25
27 changed files with 360 additions and 2 deletions

View File

@@ -169,6 +169,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -240,6 +240,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -240,6 +240,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -169,6 +169,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -171,6 +171,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -240,6 +240,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -205,6 +205,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -170,6 +170,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -169,6 +169,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -169,6 +169,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -240,6 +240,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -170,6 +170,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -206,6 +206,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -240,6 +240,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -136,6 +136,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -169,6 +169,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -169,6 +169,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -169,6 +169,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -240,6 +240,7 @@
"gsd-core/references/user-story-template.md",
"gsd-core/references/verification-overrides.md",
"gsd-core/references/verification-patterns.md",
"gsd-core/references/verifier-evidence-gate.md",
"gsd-core/references/verifier-phase-gates.md",
"gsd-core/references/verifier-wiring-patterns.md",
"gsd-core/references/verify-command-path-resolvability.md",

View File

@@ -0,0 +1,154 @@
'use strict';
// Issue #3304 — re-verification convergence: an out-of-contract, unevidenced
// blocker must not revert a completed gap-closure round or trigger another
// --gaps cycle. Content-assertion contract for the gsd-verifier agent (same
// basis as tests/verifier-behavior-unverified.test.cjs and
// tests/verifier-deferred-items.test.cjs — the agent .md file IS the deployed
// product; testing its text tests the shipped contract).
//
// Maintainer approval was narrowed to condition C only (deterministic
// evidence gates a new-scope blocker), rejecting the reporter's broader A/B
// (advisory whenever untraceable to a requirement/decision/prior-gap,
// regardless of evidence) — several tests below pin that narrowing so a
// future edit can't silently widen it back out.
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const ROOT = path.join(__dirname, '..');
const verifierPath = path.join(ROOT, 'agents', 'gsd-verifier.md');
const verifier = fs.readFileSync(verifierPath, 'utf-8');
const gatePath = path.join(ROOT, 'gsd-core', 'references', 'verifier-evidence-gate.md');
const gate = fs.readFileSync(gatePath, 'utf-8');
test('gsd-verifier.md required_reading loads the evidence-gate reference', () => {
assert.match(
verifier,
/<required_reading>[\s\S]*?@~\/\.claude\/gsd-core\/references\/verifier-evidence-gate\.md[\s\S]*?<\/required_reading>/,
);
});
test('verifier-evidence-gate.md reference file exists and is non-trivial', () => {
assert.ok(fs.existsSync(gatePath), 'gsd-core/references/verifier-evidence-gate.md should exist');
assert.ok(gate.length > 500, 'reference file should contain the full algorithm, not a stub');
});
test('Step 7 gates on re-verification mode only', () => {
assert.match(verifier, /Re-verification evidence gate[\s\S]{0,80}#3304/);
assert.match(verifier, /in re-verification mode[\s\S]{0,400}?blocks unconditionally only if/);
});
test('debt marker gate stays self-evidencing and unconditional (not routed through the new gate)', () => {
assert.match(verifier, /Re-verification evidence gate/);
assert.match(verifier, /other than an unresolved debt marker \(always self-evidencing\)/);
});
test('carried-forward gaps and regressions still block unconditionally, evidence or not', () => {
assert.match(verifier, /carried-forward gap \(Step 0's `gaps:`\)/);
assert.match(verifier, /git-modified since the prior `verified:` timestamp/);
});
test('unresolvable git history fails closed toward blocking', () => {
assert.match(verifier, /fail closed: unresolvable history counts as modified/);
assert.match(gate, /\*\*Fail closed\*\*/);
});
test('unevidenced new-scope findings route to advisory, not gaps_found', () => {
assert.match(verifier, /Unevidenced → 📋 Advisory/);
assert.match(verifier, /exclude from Step 9 Rule 1/);
assert.match(verifier, /never revert a completed must-have/);
});
test('Categorize line adds the Advisory bucket alongside Blocker/Warning/Info', () => {
const line = verifier.match(/^Categorize:.*$/m);
assert.ok(line, 'Categorize line must exist');
assert.match(line[0], /🛑 Blocker/);
assert.match(line[0], /⚠️ Warning/);
assert.match(line[0], /ℹ️ Info/);
assert.match(line[0], /📋 Advisory/);
});
test('VERIFICATION.md frontmatter template carries the advisory: list', () => {
assert.match(
verifier,
/advisory: # Only if unevidenced new-scope findings exist \(Step 7, re-verification only\)/,
);
assert.match(verifier, /advisory:[\s\S]{0,200}?category: architectural \| security \| other/);
assert.match(verifier, /advisory:[\s\S]{0,300}?evidence_status:/);
});
test('report body template includes an Advisory section, positioned after Deferred Items', () => {
const deferredIdx = verifier.indexOf('### Deferred Items');
const advisoryIdx = verifier.indexOf('### Advisory (New Scope, Unevidenced)');
assert.notEqual(deferredIdx, -1, 'Deferred Items section must exist');
assert.notEqual(advisoryIdx, -1, 'Advisory section must exist');
assert.ok(advisoryIdx > deferredIdx, 'Advisory section should follow Deferred Items');
});
test('Advisory section instructs inclusion even when empty', () => {
const sectionStart = verifier.indexOf('### Advisory (New Scope, Unevidenced)');
const nextSection = verifier.indexOf('### Required Artifacts', sectionStart);
assert.notEqual(sectionStart, -1);
assert.notEqual(nextSection, -1);
const section = verifier.slice(sectionStart, nextSection);
assert.match(section, /even "None"/);
});
// ─── Reference file: definitions and rejected-scope pins ──────────────────
test('reference file scopes the gate to Step 7 only — truths/artifacts/key-links excluded', () => {
assert.match(
gate,
/Truths, artifacts, and key links[\s\S]{0,200}?can \*\*never\*\* produce this\s*\nfailure mode/,
);
});
test('reference file explicitly rejects conditions A/B from the issue (contract-traceability alone)', () => {
const m = gate.match(/## What this deliberately does NOT implement[\s\S]{0,700}/);
assert.ok(m, 'the "does NOT implement" section must exist');
assert.match(m[0], /conditions A and B/);
assert.match(m[0], /maintainer approved \*\*condition C only\*\*/);
assert.match(m[0], /Do not implement A\/B/);
});
test('reference file defines deterministic evidence as a run-red test or a reproducible artifact', () => {
assert.match(gate, /named test that FAILS when actually run/);
assert.match(gate, /Another concrete, reproducible artifact/);
assert.match(gate, /is not evidence, however well-reasoned/);
});
test('reference file uses file-level git check, not line-level (reliability tradeoff is disclosed)', () => {
assert.match(gate, /git log --since="\$PREV_VERIFIED_TS" --oneline -- "\$file"/);
assert.match(gate, /Check file-level, not\s*\n\s*line-level/);
});
test('reference file carries a worked example grounded in the reported incident', () => {
assert.match(gate, /## Worked example \(from the issue's reported incident\)/);
assert.match(gate, /→ \*\*advisory\*\*, does not\s*\n\s*block/);
});
// ─── Parity: advisory is a per-finding annotation, never an overall status ─
test('PARITY: advisory never leaks into the overall-status vocabulary', () => {
const unionLines = verifier.match(/^status:\s+[a-z_]+(?:\s*\|\s*[a-z_]+)+\s*$/gm) || [];
assert.ok(unionLines.length > 0, 'expected at least one status union line');
for (const line of unionLines) {
assert.doesNotMatch(line, /advisory/i);
}
});
test('PARITY: overall-status enum in verification.cts does not gain an "advisory" status', () => {
const verificationLib = require(path.join(ROOT, 'gsd-core', 'bin', 'lib', 'verification.cjs'));
const { VERIFIER_STATUSES } = verificationLib;
assert.ok(Array.isArray(VERIFIER_STATUSES), 'VERIFIER_STATUSES array must be present');
assert.ok(
!VERIFIER_STATUSES.includes('advisory'),
'VERIFIER_STATUSES must not gain a per-finding "advisory" state — it stays a per-finding annotation',
);
for (const s of ['passed', 'gaps_found', 'human_needed']) {
assert.ok(VERIFIER_STATUSES.includes(s), `VERIFIER_STATUSES must still contain ${s}`);
}
});