* test(#4663): add failing-first contract coverage for the blocked-uat canonicalization gate verify-work.md's complete_session step flips VERIFICATION.md to passed on 'zero issues' alone, so a session whose every UAT row is blocked (a session that observed nothing) canonicalizes the report. Pins the deployed contract the fix must satisfy: the flip runs the unflagged phase uat-passed predicate inside the human_needed branch, frontmatter.set sits inside a passed==true guard, a refusal message carries the blocker count and keeps human_needed, and an indeterminate pre-check fails closed. All four new assertions are RED until the workflow grows the guard. * fix(#4663): gate the canonicalization on the uat-passed predicate complete_session flipped VERIFICATION.md to passed whenever the session recorded zero issues and the status was human_needed — but blocked rows are not issues by this workflow's own rule, so a 0-passed / 0-issues / N-blocked session (one that observed nothing) rewrote the canonical report to passed. Every later reader (transition.md's preliminary check, resume paths, validate-phase, verification.status) then inherited the unearned pass while the phase-close predicate correctly refused it. The flip now runs the phase-close predicate in a new --uat-only form before canonicalizing: UAT rows evaluated (at least one pass, no pending/blocked/failed/unexplained-skip row), VERIFICATION-status blockers skipped — they must be, because the report still reads human_needed at pre-check time and that status is itself a blocking verification entry, so the full predicate could never pass there and the flip would deadlock (found by isolated review, probed). The --require-verification call stays the transition gate; the refusal branch reports the blocker count and keeps human_needed; an indeterminate pre-check fails closed. Emitted-Drift-Ack-Growth: verify-work.md — canonicalize block gains the uat-only pre-check and refusal branch (#4663) * test(#4663): align the canonicalize pre-check needles with the shipped line The workflow line carries a 2>/dev/null redirect the needles did not include, so both pre-check assertions fail against the committed fix (fixed-string grep verified). Reviewer-found; needle and message aligned. * fix(#4663): reword the canonicalize prose and refresh its size baseline The rationale paragraph mentioned the flagged transition-gate call by its flag, putting a --require-verification literal before the first phase uat-passed occurrence and breaking the existing ordering pin; the prose now describes it without the literal. verify-work.md's growth also drifted the committed compact-content baseline; regenerated via benchmark-compact-content.cjs --write (derived artifact, report-not-gate contract). Emitted-Drift-Ack-Growth: verify-work.md — canonicalize block gains the uat-only pre-check and refusal branch (#4663) * docs(#4663): backfill changeset PR number --------- Co-authored-by: sim <sim@local>
This commit is contained in:
@@ -33,14 +33,14 @@
|
||||
"reductionPct": 11.9
|
||||
},
|
||||
"verify-work": {
|
||||
"offTokens": 12664,
|
||||
"onTokens": 10691,
|
||||
"reductionPct": 15.58
|
||||
"offTokens": 12992,
|
||||
"onTokens": 11019,
|
||||
"reductionPct": 15.19
|
||||
}
|
||||
},
|
||||
"aggregate": {
|
||||
"offTokens": 108302,
|
||||
"onTokens": 91615,
|
||||
"reductionPct": 15.41
|
||||
"offTokens": 108630,
|
||||
"onTokens": 91943,
|
||||
"reductionPct": 15.36
|
||||
}
|
||||
}
|
||||
|
||||
@@ -599,6 +599,62 @@ describe('evaluateUatPassed — policy.requireVerification', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─── evaluateUatPassed — policy.uatOnly (#4663) ───────────────────────────────
|
||||
//
|
||||
// The verify-work canonicalize pre-check runs WHILE the report still reads
|
||||
// `human_needed` — and `human_needed` is itself a blocking verification
|
||||
// status, so the full predicate can never pass at pre-check time and the
|
||||
// flip would deadlock. uatOnly evaluates the UAT rows ONLY: verification-file
|
||||
// blockers (and only those) are skipped, while every UAT-row blocker
|
||||
// (pending/blocked/failed/skip-without-reason) and the vacuous-pass guard
|
||||
// still apply.
|
||||
|
||||
describe('evaluateUatPassed — policy.uatOnly (#4663)', () => {
|
||||
let tmpDir;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = makeTmpDir();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmDir(tmpDir);
|
||||
});
|
||||
|
||||
test('uatOnly ignores the human_needed verification blocker the flip is about to remove', () => {
|
||||
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
||||
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_needed\n---\n\nWaiting on hardware.');
|
||||
const full = evaluateUatPassed(tmpDir, {});
|
||||
assert.strictEqual(full.passed, false,
|
||||
'the full predicate must refuse while verification is human_needed (the blocker the flip removes)');
|
||||
assert.ok(full.blockers.some(b => /human_needed/.test(b)),
|
||||
`expected a human_needed blocker, got: ${JSON.stringify(full.blockers)}`);
|
||||
const report = evaluateUatPassed(tmpDir, { policy: { uatOnly: true } });
|
||||
assert.strictEqual(report.passed, true, 'uat-only: all UAT rows pass → the pre-check may flip');
|
||||
assert.strictEqual(report.blockers.length, 0);
|
||||
assert.strictEqual(report.policy.uat_only, true);
|
||||
assert.strictEqual(report.policy.require_verification, false);
|
||||
});
|
||||
|
||||
test('uatOnly still refuses blocked UAT rows — the vacuous pass stays impossible', () => {
|
||||
writeFile(tmpDir, 'phase-UAT.md', [
|
||||
'---', 'status: human_needed', '---', '', '# UAT', '',
|
||||
'### 1. Test 1', 'expected: It works', 'result: blocked', '',
|
||||
].join('\n'));
|
||||
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_needed\n---\n\nWaiting on hardware.');
|
||||
const report = evaluateUatPassed(tmpDir, { policy: { uatOnly: true } });
|
||||
assert.strictEqual(report.passed, false, 'a blocked row is a non-pass result even in uat-only mode');
|
||||
assert.ok(report.blockers.length > 0);
|
||||
});
|
||||
|
||||
test('uatOnly and requireVerification are mutually exclusive — uatOnly wins', () => {
|
||||
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
||||
const report = evaluateUatPassed(tmpDir, { policy: { uatOnly: true, requireVerification: true } });
|
||||
assert.strictEqual(report.passed, true, 'uatOnly takes precedence; the verification policy is not evaluated');
|
||||
assert.strictEqual(report.policy.require_verification, false);
|
||||
assert.strictEqual(report.policy.uat_only, true);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── evaluateUatPassed — #3057 B3: staleness-check indeterminate is surfaced ──
|
||||
//
|
||||
// readVerificationStatus's internal staleness check can fail (fs /
|
||||
|
||||
@@ -129,6 +129,58 @@ describe('verify-work.md — auto-transition after UAT passes with 0 issues', ()
|
||||
});
|
||||
});
|
||||
|
||||
// ── #4663 — the canonicalize flip requires the UAT predicate, not a vacuous zero ──
|
||||
// "zero issues" is not pass evidence: blocked rows are not issues by this same
|
||||
// workflow's rule, so a 0-passed / 0-issues / N-blocked session must NOT flip
|
||||
// VERIFICATION.md to `passed`. The flip now consumes the SAME predicate the
|
||||
// phase-close uses in its --uat-only form (the verification-status blocker
|
||||
// is exactly what the flip removes, so the full predicate could never pass
|
||||
// at pre-check time); the flagged call stays the later transition gate.
|
||||
describe('verify-work.md — canonicalize flip is gated by the UAT predicate (#4663)', () => {
|
||||
test('canonicalize flips to passed only when the uat-passed predicate reports passed (#4663)', () => {
|
||||
const content = fs.readFileSync(VERIFY_WORK, 'utf-8');
|
||||
const humanNeededIdx = content.indexOf('if [ "$VERIFICATION_STATUS_VALUE" = "human_needed" ]; then');
|
||||
const precheckIdx = content.indexOf('UAT_PRECHECK=$(gsd_run phase uat-passed "{phase}" --uat-only 2>/dev/null)');
|
||||
const flipGuardIdx = content.indexOf('if [ "$UAT_PRECHECK_PASSED" = "true" ]; then');
|
||||
const setPassedIdx = content.indexOf('gsd_run query frontmatter.set "$VERIFICATION_FILE" --field status --value passed');
|
||||
|
||||
assert.ok(precheckIdx !== -1, 'the canonicalize block must run the uat-passed predicate before flipping');
|
||||
assert.ok(humanNeededIdx !== -1 && precheckIdx > humanNeededIdx, 'the pre-check must sit inside the human_needed branch');
|
||||
assert.ok(content.includes(".passed // false"), 'the verdict must be extracted from the typed report with a false default');
|
||||
assert.ok(flipGuardIdx !== -1 && flipGuardIdx > precheckIdx, 'the flip must be guarded on the extracted passed verdict');
|
||||
assert.ok(setPassedIdx > flipGuardIdx, 'frontmatter.set must sit INSIDE the passed==true guard');
|
||||
});
|
||||
|
||||
test('the canonicalize pre-check runs uat-passed without --require-verification (#4663)', () => {
|
||||
const content = fs.readFileSync(VERIFY_WORK, 'utf-8');
|
||||
const precheckIdx = content.indexOf('UAT_PRECHECK=$(gsd_run phase uat-passed "{phase}" --uat-only 2>/dev/null)');
|
||||
const flaggedIdx = content.indexOf('PHASE_COMPLETE=$(gsd_run phase uat-passed "{phase}" --require-verification)');
|
||||
|
||||
assert.ok(precheckIdx !== -1, 'the --uat-only pre-check must exist');
|
||||
assert.ok(
|
||||
!content.slice(precheckIdx, precheckIdx + 120).includes('--require-verification'),
|
||||
'the pre-check is the unflagged predicate - requiring verification there would evaluate the very report being written'
|
||||
);
|
||||
assert.ok(flaggedIdx !== -1 && flaggedIdx > precheckIdx, 'the flagged predicate remains the later transition gate');
|
||||
});
|
||||
|
||||
test('refused canonicalization says the verification stays human_needed (#4663)', () => {
|
||||
const content = fs.readFileSync(VERIFY_WORK, 'utf-8');
|
||||
assert.match(content, /stays human_needed/, 'the refusal message must say verification stays human_needed');
|
||||
assert.match(content, /blockers \| length/, 'the refusal must carry the blocking-row count');
|
||||
});
|
||||
|
||||
test('an indeterminate pre-check must not flip the report (fail closed) (#4663)', () => {
|
||||
const content = fs.readFileSync(VERIFY_WORK, 'utf-8');
|
||||
// jq -r '.passed // false' with an `|| echo "false"` fallback: empty or
|
||||
// failed gsd_run output must yield no-flip, never a flip.
|
||||
assert.ok(
|
||||
content.includes(`jq -r '.passed // false' 2>/dev/null || echo "false"`),
|
||||
'the extraction must default to false on empty/failed output'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Folded from tests/bug-3381-verify-work-workstream.test.cjs — consolidation epic #1969 (B4 #1973)
|
||||
|
||||
Reference in New Issue
Block a user