* test(#4546): failing-first tests for deferred uat follow-ups * chore(#4546): regenerate derived lists for the deferred-promotion suite The new verify-work-deferred-promotion suite changes the tests/ tree the macOS conformance-tier classifier tracks and is a novel file under the verify prefix in the test-file-count ratchet; both derived lists are regenerated/registered per their own guards' instructions. * fix(#4546): deferred uat follow-ups no longer block, and get promoted Two halves of one disconnect (#1921's deferral design vs the completion predicate): - uat-predicate: the item parser now captures the block's reason: line alongside result:. A skipped item whose reason carries the verify-work writer's 'Deferred follow-up:' template is a deliberate deferral -- non-blocking, flagged deferred in the report. Quote- tolerant (the writer wraps the value) and case-insensitive. A reasonless skip, a non-deferral reason, pending/blocked/issue/ failed/missing all still block, exactly as before. - verify-work complete_session: when the Deferred Follow-Ups section is non-empty, offer to promote the items to a ROADMAP.md 999.x backlog entry reusing next.md's prior_phase_completeness entry shape, with a --files-scoped commit. Offer, not auto-mutation -- matches the workflow's interactive convention and next.md's own prompt style. * chore(#4546): refresh compact-content benchmark baseline verify-work.md grew (the #4546 deferred-follow-up promotion offer in complete_session); the registered split's token counts moved with it. Baseline recomputed with the script's own --write. Emitted-Drift-Ack-Growth: verify-work.md — complete_session gained the deferred-follow-up promotion offer (detection, [P]/[K] choice, the next.md-shaped 999.x entry template, and the --files-scoped ROADMAP.md commit); the growth is the new contract text, not duplication * fix(#4546): gate/audit agreement and review fixes for deferred follow-ups - src/uat.cts categorizeItem: a skipped item carrying the deferred follow-up template reason now categorizes as 'deferred' (the category already existed for deferred-items.md entries) instead of being misfiled into the blocked families by keyword match -- the gate/audit agreement #3078-CR expects, restored in the permissive direction the #1921 design intends. Checked BEFORE the keyword families so '... on the release build next version' is not build_needed. - verify-work.md promotion step: numbering scans for the smallest free 999.n (count races + non-contiguous history), one backlog entry per deferred follow-up, ROADMAP.md-absent behavior specified, idea text newline-flattened, Deferred at placeholder harmonized with next.md. - DEFERRED_REASON_RE: trust assumption documented (authoring contract, not a security boundary; non-matching spellings block fail-closed). - tests: the property now drives evaluateUatPassed and derives expectations from the input spec (never restates the matcher), includes the no-result-line branch, and pins its seed; the parity test drops try/finally for the approved pattern, uses createTempDir, sites its allow-test-rule marker at the suppression site, and asserts the literal [P]/[K] choices. * fix(#4546): close promotion-test docstring, drop fc replay-path misuse, refresh baseline The final matrix run caught three defects in my own review-fix commit: the parity test file's JSDoc was left unterminated (the whole file parsed as one comment -- zero tests registered, hence the file-level 'test failed' the runner reported); fast-check's replay-path parameter was misused as a label (invalid path at replay); and the workflow-text ambiguity fixes re-drifted the compact-content benchmark baseline. * docs(#4546): add Fixed changeset for deferred follow-up coverage * docs(#4546): backfill changeset PR number * fix(#4546): use the pattern seam escapeRegex for shape-marker matching The hand-rolled metacharacter escape in the shape-marker assertion tripped local/no-adhoc-regex-escape, whose named remedy this adopts. --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
@@ -1654,3 +1654,147 @@ describe('#3078-CR: evaluateUatPassed agrees with the audit surface (parseUatIte
|
||||
assert.equal(auditReport.items.length, 0, 'audit: a clean file must have no outstanding rows');
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
// ─── #4546 — deferred follow-up skips are non-blocking ────────────────────────
|
||||
|
||||
describe('#4546 — deferred follow-up skips', () => {
|
||||
let tmpDir;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = makeTmpDir();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmDir(tmpDir);
|
||||
});
|
||||
|
||||
function makeDeferredUatItem(n, name, reason) {
|
||||
const lines = [`### ${n}. ${name}`, `expected: ${name} works`];
|
||||
if (reason === null) {
|
||||
lines.push('result: skipped');
|
||||
} else {
|
||||
lines.push('result: skipped', `reason: ${reason}`);
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
test('deferred follow-up skip is non-blocking (#4546)', () => {
|
||||
const content = [
|
||||
'---', 'status: complete', '---', '',
|
||||
'### 1. Test A', 'expected: A', 'result: passed', '',
|
||||
...makeDeferredUatItem(2, 'Test B', '"Deferred follow-up: nice to have, next version"'), '',
|
||||
].join('\n');
|
||||
writeFile(tmpDir, 'phase-UAT.md', content);
|
||||
const report = evaluateUatPassed(tmpDir);
|
||||
assert.strictEqual(report.passed, true,
|
||||
`a deliberately deferred follow-up must not block: ${JSON.stringify(report.blockers)}`);
|
||||
assert.strictEqual(report.blockers.length, 0);
|
||||
const deferred = report.checks.find(c => c.test === 2);
|
||||
assert.ok(deferred, 'deferred check present');
|
||||
assert.strictEqual(deferred.passing, true, 'deferred skip counts as passing');
|
||||
assert.strictEqual(deferred.deferred, true, 'deferred skip is flagged deferred in the report');
|
||||
const passing = report.checks.find(c => c.test === 1);
|
||||
assert.strictEqual(passing.deferred, false, 'a real pass is not flagged deferred');
|
||||
});
|
||||
|
||||
test('plain skipped without a reason still blocks (#4546 negative space)', () => {
|
||||
const content = [
|
||||
'---', 'status: complete', '---', '',
|
||||
'### 1. Test A', 'expected: A', 'result: passed', '',
|
||||
...makeDeferredUatItem(2, 'Test B', null), '',
|
||||
].join('\n');
|
||||
writeFile(tmpDir, 'phase-UAT.md', content);
|
||||
const report = evaluateUatPassed(tmpDir);
|
||||
assert.strictEqual(report.passed, false,
|
||||
'a skipped test with no reason is unresolved and must block');
|
||||
assert.ok(report.blockers.some(b => /test 2/.test(b)),
|
||||
`expected a blocker for test 2, got: ${JSON.stringify(report.blockers)}`);
|
||||
});
|
||||
|
||||
test('skipped with a non-deferral reason still blocks (#4546 negative space)', () => {
|
||||
const content = [
|
||||
'---', 'status: complete', '---', '',
|
||||
'### 1. Test A', 'expected: A', 'result: passed', '',
|
||||
...makeDeferredUatItem(2, 'Test B', '"waiting on credentials"'), '',
|
||||
].join('\n');
|
||||
writeFile(tmpDir, 'phase-UAT.md', content);
|
||||
const report = evaluateUatPassed(tmpDir);
|
||||
assert.strictEqual(report.passed, false,
|
||||
'a skipped test whose reason is not a deferral must block');
|
||||
});
|
||||
|
||||
test('deferred reason match is case-insensitive (#4546 boundary)', () => {
|
||||
const content = [
|
||||
'---', 'status: complete', '---', '',
|
||||
...makeDeferredUatItem(1, 'Test A', '"deferred follow-up: later"'), '',
|
||||
].join('\n');
|
||||
writeFile(tmpDir, 'phase-UAT.md', content);
|
||||
const report = evaluateUatPassed(tmpDir);
|
||||
assert.strictEqual(report.passed, true,
|
||||
'the matcher anchors on the shipped template text case-insensitively');
|
||||
});
|
||||
|
||||
test('a deferred skip does not mask other blockers (#4546 independence)', () => {
|
||||
const content = [
|
||||
'---', 'status: complete', '---', '',
|
||||
...makeDeferredUatItem(1, 'Test A', '"Deferred follow-up: later"'), '',
|
||||
'### 2. Test B', 'expected: B', 'result: pending', '',
|
||||
'### 3. Test C', 'expected: C', 'result: blocked', 'blocked_by: server', '',
|
||||
'### 4. Test D', 'expected: D', 'result: issue', 'reported: "crashes"', '',
|
||||
].join('\n');
|
||||
writeFile(tmpDir, 'phase-UAT.md', content);
|
||||
const report = evaluateUatPassed(tmpDir);
|
||||
assert.strictEqual(report.passed, false,
|
||||
'deferred skip must not mask pending/blocked/issue blockers');
|
||||
assert.ok(report.blockers.some(b => /test 2/.test(b)), 'pending still blocks');
|
||||
assert.ok(report.blockers.some(b => /test 3/.test(b)), 'blocked still blocks');
|
||||
assert.ok(report.blockers.some(b => /test 4/.test(b)), 'issue still blocks');
|
||||
assert.ok(!report.blockers.some(b => /test 1/.test(b)),
|
||||
`the deferred item must not appear among blockers: ${JSON.stringify(report.blockers)}`);
|
||||
});
|
||||
|
||||
test('property: deferred-skip acceptance drives the real gate over arbitrary result/reason pairs (#4546)', () => {
|
||||
// Gate-driven: expectations are derived from the INPUT (the spec sentence
|
||||
// in #4546), then asserted against evaluateUatPassed's report — the
|
||||
// property never restates the implementation's regex. The generated
|
||||
// classes include the no-result-line shape (the parser's 'missing'
|
||||
// branch) and reasonless skips.
|
||||
const deferredRe = /^["']?deferred follow-up\b/i;
|
||||
fc.assert(
|
||||
fc.property(
|
||||
fc.constantFrom('passed', 'pass', 'skipped', 'pending', 'blocked', 'issue', 'missing'),
|
||||
fc.option(fc.stringMatching(/^["']?[a-z ]{0,30}$/), { nil: undefined }),
|
||||
(result, reason) => {
|
||||
const itemLines = [`### 1. Test X`, 'expected: X works'];
|
||||
if (result !== 'missing') {
|
||||
itemLines.push(`result: ${result}`);
|
||||
if (reason !== undefined) itemLines.push(`reason: ${reason}`);
|
||||
}
|
||||
const content = [
|
||||
'---', 'status: complete', '---', '',
|
||||
...itemLines, '',
|
||||
].join('\n');
|
||||
fs.writeFileSync(path.join(tmpDir, 'phase-UAT.md'), content, 'utf-8');
|
||||
const report = evaluateUatPassed(tmpDir);
|
||||
assert.strictEqual(report.checks.length, 1);
|
||||
const check = report.checks[0];
|
||||
|
||||
const isDeferral = result === 'skipped' &&
|
||||
typeof reason === 'string' && deferredRe.test(reason);
|
||||
const specPassing = result === 'passed' || result === 'pass' || isDeferral;
|
||||
|
||||
assert.strictEqual(check.result, result === 'missing' ? 'missing' : result);
|
||||
assert.strictEqual(check.passing, specPassing,
|
||||
`result=${result} reason=${JSON.stringify(reason)}: gate must ${specPassing ? 'pass' : 'block'}`);
|
||||
assert.strictEqual(check.deferred, isDeferral,
|
||||
`result=${result} reason=${JSON.stringify(reason)}: deferred flag`);
|
||||
assert.strictEqual(report.passed, specPassing,
|
||||
'a single-item file passes exactly when the item passes');
|
||||
assert.deepStrictEqual(report.blockers, specPassing ? [] : [report.blockers[0]]);
|
||||
}
|
||||
),
|
||||
{ numRuns: 120, seed: 4546 }
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user