4e8927b0b9
* test(#3707): failing-first coverage for reverting the fence-shortfall fold shield Pins the post-revert contract: a phase whose only gap is a fence shortfall must degrade the fold and withhold the milestone percentages, like every other gap class. Five of the eight rows are CONTROLS that pass before the change, and they carry more weight than the failing row. The failure mode of this revert is degrading TOO MUCH: a revert that sets foldScope outside the headingsSeen > 0 branch would withhold every percentage in the project, and only the no-gap control catches that. Another control catches a revert that collapses the two scopes into one and loses the distinction between what a phase reports and what the fold folds -- uat.scope must stay TRUNCATED for every gap either way, which it already is. The row that pinned the shielded behavior is rewritten rather than deleted. Deleting a test because the behavior it asserts is being reversed leaves the reversal unguarded. * fix(#3707): degrade the fold for every UAT gap class, reverting the fence-shortfall shield Maintainer decision. The two orthogonal engines split on this during #3707 and neither filed it as blocking, so it shipped in the shape the engine that raised the objection endorsed after verifying seven fixtures. The call has now gone the other way, restoring the fail-safe direction chosen twice already on this issue. The shield exempted one gap class from the fold's teeth. It could not do that safely: shortfallBlocks is a single tally incremented at exactly one site and spans BOTH a harmless fenced documentation sample AND a genuinely fence-straddled result: blocked row. Exempting it therefore could not exempt only the harmless case -- it also published a milestone percentage over a real, unread outstanding row. SCOPE.TRUNCATED means the scan could not SEE part of the evidence, which is exactly that case. scope and foldScope now agree: every gap class degrades both. The accepted over-report documented in uat.cts is unchanged and still documented there; what changed is only that it no longer buys an exemption from the fold. The comment block above it argued FOR the shield and is rewritten, because a comment defending behavior the code no longer has is worse than no comment. shortfallBlocks leaves this function's destructure but is untouched upstream, where audit-uat still consumes it. * fix(#3707): correct the caller comment, add the changeset, and name what the order tests guard Review found a SECOND comment still documenting the removed shield -- the caller's, beside the worstScope fold, stating that foldScope differs from scope for exactly one case which must not raise phase_scope_degraded or withhold the milestone's percentages. That is now the opposite of what the code does. I rewrote the buildUatRows comment in the previous commit and asserted in its message that a comment defending behavior the code no longer has is worse than no comment, then left exactly that one standing a few hundred lines away. The change had no changeset. It is user-visible: a milestone's percentage goes from published to withheld whenever any phase has a fence-shortfall-only gap. PR gates hard-fail a user-facing code diff without one. The two scopes are now identical at every return site. They are NOT collapsed -- that would change the return shape and the caller on what is meant to be a one-condition revert, and the seam is worth keeping if the distinction is ever wanted again -- but the declaration now says plainly that they agree by decision rather than by accident, so a reader does not have to re-derive it. The two order-independence tests were renamed. foldScope is monotonic with no reset path, so file order is structurally irrelevant and those rows could never have failed for the ordering reason their names promised. They do guard something real -- a multi-file phase degrading when any one file has a shortfall-only gap -- so they now say that instead. * test(#3707): failing-first coverage for the lone-CR UAT false-clean The parser splits on newline only, and the heading tokenizer agrees with it, so a lone carriage return is not a line boundary anywhere in it. CommonMark treats a lone CR as a line ending, so such a row renders to a human reader while being invisible to BOTH sides of the parser's symmetry invariant: no item, no shortfall, no headingsSeen. A phase hiding a result: blocked row this way reports 100 percent with zero diagnostics. Found by the security review of the fold-shield revert. It is the one false-clean class that revert does not reach, and it is the same bug class this issue exists to fix -- an unreadable row reported as clean. Nine rows. The LF control is what proves this is a separator defect rather than a content defect: identical bodies, one separator apart, and only one of them hides the row. CRLF and CR-inside-a-fence controls guard the coming normalization against double-counting or tearing content that legitimately contains a carriage return. Two further manifestations turned up while writing them: a leading CR breaks column-0 anchoring of the first heading, and an all-CR document flags a shortfall it cannot attribute to any row. * fix(#3707): treat a lone carriage return as a line ending in the UAT parser A lone CR was not a line boundary anywhere in the parser -- it split on newline only, and the heading tokenizer agreed with it. CommonMark treats a lone CR as a line ending, so such a row rendered to a human reader while being invisible to BOTH sides of the parser's symmetry invariant: no item, no shortfall, no headingsSeen. A phase hiding a result: blocked row that way reported 100 percent with zero diagnostics. Line endings are now normalized once at document ingress -- CRLF and lone CR both to newline -- at the two independent entry points, rather than teaching each split site about CR. Every downstream scan, offset and span therefore reads one convention. That single-frame property is deliberate: this issue already cost a HIGH when two scans read the same document through different frames. MY OWN END-TO-END TEST WAS WRONG and is replaced rather than weakened. It asserted that a lone-CR document must withhold its percentage, which reasons from the pre-fix symptom: after the fix the row is not hidden, it is surfaced, and this module deliberately keeps visible outstanding UAT work separate from completion percentages -- only unreadable evidence degrades scope. The success of the fix is what made the assertion false. The implementing agent refused to satisfy both it and the architecture and asked instead of bending either; it was right. What replaces it is a stronger contract: a lone-CR document and its LF twin, built from one source, must produce identical audit output -- scope, percent, every unresolved row by identity, and the diagnostic set. That is what 'a line-ending convention must not change what the audit reports' actually means, and it carries a non-vacuity check so it cannot pass with both sides empty. shortfallBlocks keeps being returned, now documented as currently unconsumed. An earlier reviewer told me audit-uat still consumed it and I passed that on as an instruction; it was wrong, and it was caught by checking rather than by me. * fix(#3707): normalize at the document read boundary, not at two call sites The lone-CR fix was half-applied and both review engines caught it independently. cmdAuditUat has four document ingresses, not the two I normalized: VERIFICATION.md and deferred-items.md still handed raw text to newline-only splitters, and the frontmatter extract in the UAT loop read raw content while its parser read normalized -- one audit entry mixing the two frames the fix exists to unify. Measured: a phase written twice from one source gave total_files 2 / total_items 4 under LF and results [] / total_items 0 under lone CR, with zero diagnostics. Normalizing two call sites and declaring it done is exactly why two were missed, so this moves it to the read boundary: every document now enters through a helper that normalizes, in audit-uat, in planning-inspect's readDocument, and in the shared verification-status read. Future parsers downstream get normalized text by construction rather than because someone remembered. That last seam also fixes an under-reporting case of the same root: a lone-CR VERIFICATION.md saying status: passed was read as missing, telling the user a verify step that had completed never ran. The parity test's load-bearing assertion is now marked as such. Four of its five equality checks still pass with the bug present -- only the unresolved-row identity differs -- so trimming that one as redundant would make the row vacuous. Second changeset added: the CR fix is user-visible independently of the fold revert, and one fragment covering both would have described neither. * test(#3707): failing-first coverage for the U+2028 and duplicate-result false-cleans Two more of the same class, both found by the security review of this branch and both reproduced before writing a line. normalizeLineEndings folds only carriage returns, but a JS /m anchor also treats U+2028 and U+2029 as line terminators while split on newline does not. That is the identical asymmetry the carriage-return bug exploited, one separator over, and worse in one respect: these are not CommonMark line endings, so a reader still sees the column-0 result: blocked that the tool discards. Measured: a scalar-internal result: pass placed after U+2028 wins over the real blocked line and the row disappears with no gap raised. Separately, and independent of any separator, a block with two column-0 result: lines resolves to the first with no ambiguity signalled. Prepending result: pass to a block therefore deletes an outstanding row silently; reversing the order surfaces it. Order deciding meaning is the defect, so the pair of rows pins the contract as ambiguity-is-a-gap rather than last-one-wins, leaving the fix room to implement the gap sensibly. Four controls: an ordinary marker in the same position (proving separator not content), legitimate U+2028 inside prose that must not be torn, a single result line, and a result line inside a fence that must not count as a second occurrence. * fix(#3707): scan result lines by split, not by a multiline anchor Two more false-cleans from the security review, both closed by the same change. A JS /m anchor treats U+2028 and U+2029 as line terminators while split on newline does not. A scalar-internal result: pass placed after one of those separators therefore matched as a line start and beat the real column-0 result: blocked, and the row vanished at 100 percent with no gap. Worse than the carriage-return case in one respect: these are not CommonMark line endings, so a reader still saw the blocked row the tool discarded. Separately, the non-global match returned the leftmost hit, so a block with two column-0 result: lines silently resolved to the first. Prepending result: pass deleted an outstanding row; reversing the order surfaced it. Order deciding meaning was the defect. Both close by scanning lines produced by split rather than by anchoring a regex inside the whole document: each line is tested on its own, and a count other than exactly one is reported as a parse gap instead of resolved to either candidate. I asked for U+2028 to be folded in normalizeLineEndings and that was wrong. Folding is length-preserving, so it would have made the U+2028 fixture byte-identical to the genuine two-result-line fixture -- while one requires a confident item and the other requires an ambiguity gap. No implementation can satisfy both once the distinguishing character is erased. The agent proved that and deviated rather than forcing it, which is why normalizeLineEndings still folds only carriage returns, now with a comment saying why. * fix(#3707): bound the ambiguity scan at the next heading-shaped line The split-based result scan regressed four pre-existing #3078/#3707 guards, each off by exactly one gap. My diagnosis was wrong. I read the off-by-one as double counting -- zero-result blocks taking both the new path and the pre-existing one -- and said to change the ambiguity condition from not-equal-one to greater-than-one. The agent checked and refused: the zero path was never duplicated. The real cause is double ATTRIBUTION. A block is sliced to the next TOKENIZED heading, so when the next row is untokenized -- hidden by a straddling fence, or indented and already counted by the shortfall scan -- that row's own result: line is absorbed into the previous block. The scan then saw two result lines across what are really two rows and raised a second, redundant gap on top of the one already counted elsewhere. Had the greater-than-one change gone in, the counts would have matched while the double attribution stayed. That is the compensating-adjustment failure I had asked it to refuse, and it did. The scan is now bounded at the first following heading-shaped line, either indent class, so a genuine same-block ambiguity is untouched while spillover from a row counted elsewhere is excluded. * fix(#3707): keep the U+2028 immunity, revert the ambiguity detection The ambiguity half of this change regressed the suite twice and is coming out. Attempt one double-attributed: a block is sliced to the next TOKENIZED heading, so when the real next row is untokenized its result: line was absorbed into the previous block and raised a second gap on a row already counted elsewhere. Four guards broke. Attempt two bounded the scan at the next heading-shaped line and broke thirty. An indented ### N. inside a block scalar is legitimate scalar CONTENT, not a heading, and truncating there defeats every #3078 guard that exists to stop scalar bodies being read as rows. Telling a genuinely hidden indented row apart from indented scalar text is a classification countUnattributedIndentedRows already owns; a raw regex does not have that information. What survives is the half that is sound and was never implicated in either regression: the result scan tests each line produced by split rather than anchoring a regex with the multiline flag over the whole block. split never treats U+2028 or U+2029 as a delimiter, so those separators can no longer manufacture a line start and steal a row. Everything else returns to first-match-wins, byte-identical to origin/next. The two tests pinning ambiguity-as-a-gap are removed with it, since the contract is no longer implemented here. The defect they described is real, pre-existing and independent of any separator -- result: pass before result: blocked silently deletes an outstanding row -- and it needs its own change with a scalar-aware counter rather than being wedged into a branch already carrying three fixes. * fix(#3707): correct the shared-seam rationale and restore U+2028 trailing text The revert left a stale rationale in core-utils, justifying the decision not to fold U+2028 by claiming uat.cts must tell a fake line start apart from a real second column-0 result: declaration that gets flagged as ambiguous. Nothing flags ambiguity any more; that behavior was reverted and the same file says so a few lines away. The decision is still right, the stated reason was false. This is the third stale comment this branch has shipped and had to fix, and the worst placed of them: core-utils is a shared leaf that every future document consumer will read for guidance. Rewritten to the true reason -- the scan tests each split line individually rather than anchoring over the block, so an exotic separator cannot manufacture a line start and folding is unnecessary. Also a real behavior delta I had not noticed. Dropping the multiline flag left the pattern's trailing .*$ in place, and dot never matches U+2028, so a genuine column-0 result: blocked whose TRAILING text contained one stopped parsing entirely -- a visible parse gap rather than a false clean, so fail-safe, but a regression against origin/next that nothing pinned. The trailing portion now matches any character and a test pins it by identity against its plain-LF twin. Plus the JSDoc orphaned when normalizeLineEndings moved to core-utils, and the changeset, which described neither the separator fix nor planning-inspect surfacing lone-CR rows. * fix(#3707): harden the acceptance gate, which had both halves of the same bug uat-predicate is a SECOND, independent UAT parser, and it is the one that decides phase uat-passed. It read raw bytes and anchored a multiline regex over unsplit text -- exactly the two defects this branch closed one module away in uat.cts. The consequence is worse than the audit surface it mirrors. Measured on identical bytes: a U+2028 scalar injection made the gate return passed true while planning inspect reported the same row as blocked and outstanding. The hardened surface and the gate disagreed, and the gate was the permissive one -- so a phase could be accepted over a row the audit could see and the gate could not. Both raw reads now go through the shared normalize seam and both scans test lines produced by split rather than anchoring over the document. First-match-wins, matching uat.cts; no ambiguity counting is reintroduced. Tests assert the AGREEMENT between the two surfaces rather than each separately, because divergence is the defect. Also finishes the same root cause one module over: phase complete's advisory pre-scan read raw bytes, so a lone-CR VERIFICATION.md lost its human_needed or gaps_found warning -- the fix verification.cts already got on this branch. And narrows the core-utils rationale I reworded last commit, which claimed consumers already avoid multiline anchors. uat.cts still has five over unsplit text. That is the fourth comment on this branch to assert something the code does not do, so it now states only what is true of core-utils itself. * fix(#3707): give structure and attribution different line frames, normalize the close audit Two more from review, and the first was a regression I introduced one commit earlier. Converting the gate's heading scan to split-then-match removed a detection origin/next had: a ### N. heading delimited by U+2028 was found by the old multiline scan and was not found after. So hardening the result scan quietly weakened the heading scan, and the gate stopped blocking on rows origin/next blocked -- the permissive direction, on the surface that decides acceptance. The insight I had missed is that the two scans need DIFFERENT frames. Heading detection is structure: there is no distinction to preserve, so it splits on newline or either exotic separator and finds a heading however it is delimited. The result scan is attribution: the newline-only frame is exactly what stops a scalar-internal result: from being read as a column-0 line, so it stays. One frame applied uniformly was the error. Second, a THIRD unnormalized parser family: the milestone-close audit read every artifact raw. A lone-CR VERIFICATION.md degraded to status unknown and was skipped, and deferred entries vanished outright -- measured as three items requiring decisions under LF and one under CR, on identical bytes. All nine scanner reads now normalize; six of them had the identical defect beyond the three review named. The acknowledge path stays deliberately raw, since it splices by byte offset, and now says so. Also pins the cross-newline result: divergence, and replaces three raw U+2028 literals in test source with escapes. A raw separator in a fixture is one formatter away from becoming an ordinary-character control that still passes -- vacuous in the only test pinning the separator fix. * fix(#3707): share one frame between the acknowledge writer and the audit reader Normalizing the audit scanners left the writer and the reader on different frames. cmdAuditAcknowledge derives its stored snapshot values from raw content -- correct for the SPLICE, which rewrites by byte offset -- but scanUatGaps and scanContextQuestions now recompute those same values from normalized content. For a lone-CR artifact the two can never match, so an acknowledgement never suppresses its item and it resurfaces on every audit: acknowledge became a silent no-op. Fail-safe in direction, since the item stays visible rather than being wrongly suppressed, but it is the writer and reader disagreeing about what a line is -- the exact class this branch exists to eliminate, and the fourth instance of it here. The derive functions now read a normalized copy while the splice keeps raw bytes and raw offsets, so both sides share one frame and the byte-offset rewrite is untouched. Round trip pinned for lone-CR and LF, with an existing LF marker asserted still recognised so the change cannot silently invalidate acknowledgements already in users' files. Also tightens an assertion that pinned this branch's own heading fix with a proxy: notStrictEqual against 'passed' also passes on 'pass', which IS a passing token, so it could not have caught a regression attributing a passing result to the recovered heading. It now pins the exact token. * chore(#3707): backfill changeset pr numbers Both fragments still carried the pr: 0 placeholder, which failed changeset-lint and docs-lint on PR 3903. The review had flagged the backfill as pending and I opened the PR without doing it. --------- Co-authored-by: sim <sim@local>
1657 lines
65 KiB
JavaScript
1657 lines
65 KiB
JavaScript
'use strict';
|
|
|
|
/**
|
|
* Unit tests for uat-predicate.cjs
|
|
* Tests the pure-computation module: stripFalsePositiveContexts,
|
|
* parseUatResultItems, evaluateUatPassed.
|
|
*
|
|
* Issue #247 — phase uat-passed predicate
|
|
*/
|
|
|
|
const { test, describe, beforeEach, afterEach } = require('node:test');
|
|
const assert = require('node:assert/strict');
|
|
const fs = require('node:fs');
|
|
const path = require('node:path');
|
|
const os = require('node:os');
|
|
const fc = require('fast-check');
|
|
|
|
const {
|
|
stripFalsePositiveContexts,
|
|
parseUatResultItems,
|
|
analyzeMarkdown,
|
|
evaluateUatPassed,
|
|
} = require('../gsd-core/bin/lib/uat-predicate.cjs');
|
|
const { parseUatItemsWithStats } = require('../gsd-core/bin/lib/uat.cjs');
|
|
const { cleanup } = require('./helpers.cjs');
|
|
|
|
// ─── Helpers ──────────────────────────────────────────────────────────────────
|
|
|
|
function makeTmpDir() {
|
|
return fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-uat-pred-test-'));
|
|
}
|
|
|
|
function rmDir(dir) {
|
|
cleanup(dir);
|
|
}
|
|
|
|
function writeFile(dir, name, content) {
|
|
fs.writeFileSync(path.join(dir, name), content, 'utf-8');
|
|
}
|
|
|
|
function makePassingUat(n = 1) {
|
|
const tests = Array.from({ length: n }, (_, i) => [
|
|
`### ${i + 1}. Test ${i + 1}`,
|
|
`expected: It works`,
|
|
`result: passed`,
|
|
'',
|
|
].join('\n')).join('\n');
|
|
return `---\nstatus: passed\n---\n\n# UAT\n\n${tests}`;
|
|
}
|
|
|
|
// ─── stripFalsePositiveContexts ────────────────────────────────────────────────
|
|
|
|
describe('stripFalsePositiveContexts — frontmatter', () => {
|
|
test('removes leading frontmatter block', () => {
|
|
const input = '---\nstatus: pending\nresult: pending\n---\n\nReal content here.';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'frontmatter result: pending should be stripped');
|
|
assert.ok(out.includes('Real content here.'), 'body content must be preserved');
|
|
});
|
|
|
|
test('does not strip non-frontmatter --- dividers later in document', () => {
|
|
const input = '---\nstatus: ok\n---\n\n# Section\n\n---\n\nMore content.';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(out.includes('More content.'), 'content after a non-frontmatter divider must survive');
|
|
});
|
|
|
|
test('handles CRLF frontmatter', () => {
|
|
const input = '---\r\nstatus: partial\r\nresult: pending\r\n---\r\n\r\nBody text.';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'CRLF frontmatter must be stripped');
|
|
assert.ok(out.includes('Body text.'), 'body after CRLF frontmatter must survive');
|
|
});
|
|
});
|
|
|
|
describe('stripFalsePositiveContexts — HTML comments', () => {
|
|
test('removes single-line HTML comment', () => {
|
|
const input = 'Before\n<!-- result: pending -->\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'HTML comment content must be stripped');
|
|
assert.ok(out.includes('Before'), 'content before comment must survive');
|
|
assert.ok(out.includes('After'), 'content after comment must survive');
|
|
});
|
|
|
|
test('removes multi-line HTML comment', () => {
|
|
const input = 'A\n<!--\n### 1. Test\nexpected: Foo\nresult: pending\n-->\nB';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'multi-line HTML comment content must be stripped');
|
|
assert.ok(out.includes('A'), 'content before comment must survive');
|
|
assert.ok(out.includes('B'), 'content after comment must survive');
|
|
});
|
|
|
|
test('unterminated HTML comment swallows to EOF (fail-closed)', () => {
|
|
const input = 'Before\n<!--\nresult: passed\nstill in comment';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: passed'), 'unterminated comment must swallow to EOF');
|
|
assert.ok(out.includes('Before'), 'content before comment must survive');
|
|
});
|
|
});
|
|
|
|
describe('stripFalsePositiveContexts — fenced code blocks', () => {
|
|
test('removes backtick fence with result inside', () => {
|
|
const input = 'Before\n```\n### 1. Test\nexpected: X\nresult: pending\n```\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'content inside ``` fence must be stripped');
|
|
assert.ok(out.includes('Before'), 'content before fence must survive');
|
|
assert.ok(out.includes('After'), 'content after fence must survive');
|
|
});
|
|
|
|
test('removes tilde fence', () => {
|
|
const input = 'Before\n~~~\n### 1. Test\nexpected: X\nresult: blocked\n~~~\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: blocked'), 'content inside ~~~ fence must be stripped');
|
|
assert.ok(out.includes('After'), 'content after tilde fence must survive');
|
|
});
|
|
|
|
test('handles indented fence', () => {
|
|
const input = 'Before\n ```\n### 1. Test\nresult: pending\n ```\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'content inside indented fence must be stripped');
|
|
});
|
|
|
|
test('handles CRLF in fenced block', () => {
|
|
const input = 'Before\r\n```\r\nresult: pending\r\n```\r\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'CRLF fenced content must be stripped');
|
|
});
|
|
|
|
test('preserves content after multiple fenced blocks', () => {
|
|
const input = [
|
|
'Real intro',
|
|
'```',
|
|
'result: pending',
|
|
'```',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Works',
|
|
'result: passed',
|
|
'',
|
|
'```',
|
|
'result: pending',
|
|
'```',
|
|
'Trailing',
|
|
].join('\n');
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(out.includes('result: passed'), 'real result outside fence must survive');
|
|
assert.ok(!out.includes('result: pending'), 'fenced result: pending must be stripped');
|
|
});
|
|
});
|
|
|
|
describe('stripFalsePositiveContexts — blockquotes', () => {
|
|
test('removes blockquote lines', () => {
|
|
const input = 'Before\n> ### 1. Test\n> result: pending\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'blockquote content must be stripped');
|
|
assert.ok(out.includes('Before'), 'content before blockquote must survive');
|
|
assert.ok(out.includes('After'), 'content after blockquote must survive');
|
|
});
|
|
|
|
test('removes indented blockquote lines', () => {
|
|
const input = 'Before\n > result: pending\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'indented blockquote content must be stripped');
|
|
});
|
|
});
|
|
|
|
// ─── parseUatResultItems ───────────────────────────────────────────────────────
|
|
|
|
describe('parseUatResultItems', () => {
|
|
test('parses a single passing test', () => {
|
|
const content = '### 1. Login flow\nexpected: User logs in\nresult: passed\n';
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].test, 1);
|
|
assert.strictEqual(items[0].name, 'Login flow');
|
|
assert.strictEqual(items[0].result, 'passed');
|
|
});
|
|
|
|
test('parses bracketed result [passed] (#2273)', () => {
|
|
const content = '### 1. Login flow\nexpected: User logs in\nresult: [passed]\n';
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].result, 'passed');
|
|
});
|
|
|
|
test('parses bracketed result [pass]', () => {
|
|
const content = '### 1. Login flow\nexpected: User logs in\nresult: [pass]\n';
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].result, 'pass');
|
|
});
|
|
|
|
test('parses multiple tests with mixed results', () => {
|
|
const content = [
|
|
'### 1. Test A',
|
|
'expected: A',
|
|
'result: passed',
|
|
'',
|
|
'### 2. Test B',
|
|
'expected: B',
|
|
'result: pending',
|
|
'',
|
|
'### 3. Test C',
|
|
'expected: C',
|
|
'result: failed',
|
|
'',
|
|
].join('\n');
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 3);
|
|
assert.strictEqual(items[0].result, 'passed');
|
|
assert.strictEqual(items[1].result, 'pending');
|
|
assert.strictEqual(items[2].result, 'failed');
|
|
});
|
|
|
|
test('returns empty array for content with no test blocks', () => {
|
|
const items = parseUatResultItems('Just some markdown with no test blocks.');
|
|
assert.strictEqual(items.length, 0);
|
|
});
|
|
|
|
test('lowercases result value', () => {
|
|
const content = '### 1. Test\nexpected: Foo\nresult: PASSED\n';
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items[0].result, 'passed');
|
|
});
|
|
|
|
test('heading with NO result line → result:missing (blocker state, not dropped)', () => {
|
|
const content = [
|
|
'### 1. Test without result',
|
|
'expected: Something',
|
|
'some notes here',
|
|
'',
|
|
].join('\n');
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].result, 'missing');
|
|
});
|
|
|
|
test('result: line with leading whitespace (indented scalar) is NOT parsed as column-0', () => {
|
|
// A real failing test uses block-scalar expected: with blank line before result
|
|
// The result: with leading whitespace should not be treated as a column-0 result
|
|
const content = [
|
|
'### 1. Test A',
|
|
'expected: |',
|
|
' multi',
|
|
' line',
|
|
'',
|
|
' result: pending',
|
|
'',
|
|
].join('\n');
|
|
// The indented result: should not match; heading has no column-0 result → 'missing'
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].result, 'missing',
|
|
'Indented result: line must not be parsed as column-0 result');
|
|
});
|
|
|
|
test('block-scalar expected: with blank line then column-0 result: is parsed correctly', () => {
|
|
// Real failing test: block-scalar expected: | followed by blank + real result: pending
|
|
const content = [
|
|
'### 1. Test A',
|
|
'expected: |',
|
|
' multi',
|
|
' line',
|
|
'',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].result, 'pending',
|
|
'Column-0 result: after blank line must be parsed correctly');
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — boundary coverage ────────────────────────────────────
|
|
|
|
describe('evaluateUatPassed — boundary coverage', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('0 UAT files → passed:false, no_uat_artifacts:true (fail-closed, no vacuous pass)', () => {
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'No UAT files must NOT vacuously pass — fail-closed');
|
|
assert.strictEqual(report.no_uat_artifacts, true);
|
|
assert.deepStrictEqual(report.uat_files, []);
|
|
assert.deepStrictEqual(report.blockers, []);
|
|
});
|
|
|
|
test('1 passing test → passed:true, no_uat_artifacts:false', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.no_uat_artifacts, false);
|
|
assert.strictEqual(report.blockers.length, 0);
|
|
assert.strictEqual(report.checks.length, 1);
|
|
assert.strictEqual(report.checks[0].passing, true);
|
|
});
|
|
|
|
test('N all-passing tests → passed:true', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(5));
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.checks.every(c => c.passing), true);
|
|
});
|
|
|
|
test('N-1 passing + 1 pending → passed:false', () => {
|
|
// 3 tests total: tests 1,2 passed; test 3 pending
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: passed', '',
|
|
'### 2. Test B', 'expected: B', 'result: passed', '',
|
|
'### 3. Test C', 'expected: C', 'result: pending', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /test 3/i.test(b) || /pending/i.test(b)),
|
|
`Expected blocker for pending test, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('1 blocked test → passed:false', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: blocked', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
});
|
|
|
|
test('1 skipped test → passed:false', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: skipped', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
});
|
|
|
|
test('1 failed test → passed:false', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: failed', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
});
|
|
|
|
test('1 human_needed test → passed:false', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: human_needed', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
});
|
|
|
|
test('heading with no result line → result:missing → blocker → passed:false', () => {
|
|
const content = [
|
|
'### 1. Test without result',
|
|
'expected: It should work',
|
|
'some notes but no result',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.checks.some(c => c.result === 'missing' && !c.passing),
|
|
`Expected missing check item, got checks: ${JSON.stringify(report.checks)}`);
|
|
assert.ok(report.blockers.some(b => /missing/i.test(b)),
|
|
`Expected missing blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — frontmatter status checks ────────────────────────────
|
|
|
|
describe('evaluateUatPassed — frontmatter status', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
const failingUatStatuses = ['partial', 'diagnosed', 'pending', 'blocked', 'in_progress', 'failed'];
|
|
const failingUatResults = ['pending', 'blocked', 'failed'];
|
|
|
|
for (const status of failingUatStatuses) {
|
|
test(`UAT frontmatter status=${status} → passed:false`, () => {
|
|
const content = [
|
|
'---', `status: ${status}`, '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: passed', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
`status=${status} should cause failure`);
|
|
assert.ok(report.blockers.some(b => b.includes(status)),
|
|
`Blocker should mention status=${status}, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
}
|
|
|
|
for (const result of failingUatResults) {
|
|
test(`UAT frontmatter result=${result} → passed:false`, () => {
|
|
const content = [
|
|
'---', `result: ${result}`, '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: passed', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
`fm result=${result} should cause failure`);
|
|
});
|
|
}
|
|
});
|
|
|
|
// ─── evaluateUatPassed — VERIFICATION file checks ─────────────────────────────
|
|
|
|
describe('evaluateUatPassed — VERIFICATION files', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('VERIFICATION with status human_needed → passed:false', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_needed\n---\n\nNeeds human check.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /human_needed/i.test(b)),
|
|
`Expected human_needed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with status gaps_found → passed:false', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: gaps_found\n---\n\nHas gaps.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /gaps_found/i.test(b)),
|
|
`Expected gaps_found blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with status pending → passed:false', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: pending\n---\n');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
});
|
|
|
|
test('VERIFICATION with status failed → passed:false (blocking)', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: failed\n---\n');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /failed/i.test(b)),
|
|
`Expected failed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with status in_progress → passed:false (blocking)', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: in_progress\n---\n');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /in_progress/i.test(b)),
|
|
`Expected in_progress blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with missing status does NOT satisfy --require-verification', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '# Verification\nNo frontmatter status.\n');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false,
|
|
'Missing verification status must not satisfy requireVerification');
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with unknown status does NOT satisfy --require-verification', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: some_unknown_status\n---\n');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false,
|
|
'Unknown verification status must not satisfy requireVerification');
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with status passed → does not block', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.verification_files.length, 1);
|
|
});
|
|
|
|
test('VERIFICATION with status complete → does not block', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: complete\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
});
|
|
|
|
test('VERIFICATION with status verified → does not block', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: verified\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
});
|
|
|
|
test('VERIFICATION with status human_passed → does not block', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_passed\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
});
|
|
|
|
test('VERIFICATION status complete does NOT satisfy --require-verification', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: complete\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false);
|
|
assert.strictEqual(report.policy.require_verification, true);
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION status verified does NOT satisfy --require-verification', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: verified\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION status human_passed does NOT satisfy --require-verification', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_passed\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — policy.requireVerification ───────────────────────────
|
|
|
|
describe('evaluateUatPassed — policy.requireVerification', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('requireVerification=true with no verification file → passed:false', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
assert.strictEqual(report.policy.require_verification, true);
|
|
});
|
|
|
|
test('requireVerification=true with passing verification → passed:true', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n\nOK.');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.policy.require_verification, true);
|
|
});
|
|
|
|
test('requireVerification=false (default) with no verification file → passed:true', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: false } });
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.policy.require_verification, false);
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — #3057 B3: staleness-check indeterminate is surfaced ──
|
|
//
|
|
// readVerificationStatus's internal staleness check can fail (fs /
|
|
// scanPhasePlans / clock error). Pre-#3057 B3 wiring, the requireVerification
|
|
// policy check used only `.status`, dropping `.staleCheckIndeterminate` on
|
|
// the floor — so cmdPhaseUatPassed's JSON output (which spreads this whole
|
|
// report) could never distinguish "checked; nothing is stale" from "could
|
|
// not check". `verification_stale_check_indeterminate` must never itself gate
|
|
// `passed`/`blockers` — only readVerificationStatus's `.status` may.
|
|
|
|
describe('#3057 B3: evaluateUatPassed — verification staleness-check indeterminate is surfaced', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
function seedVerifiedPhase() {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n\nOK.');
|
|
writeFile(tmpDir, 'phase-SUMMARY.md', '# Summary');
|
|
const verificationPath = path.join(tmpDir, 'phase-VERIFICATION.md');
|
|
const summaryPath = path.join(tmpDir, 'phase-SUMMARY.md');
|
|
// Deterministic mtime ordering — never rely on write-order clock ties.
|
|
const older = new Date('2026-01-01T00:00:00.000Z');
|
|
const newer = new Date('2026-01-01T00:01:00.000Z');
|
|
fs.utimesSync(summaryPath, older, older);
|
|
fs.utimesSync(verificationPath, newer, newer);
|
|
return { summaryPath, verificationPath };
|
|
}
|
|
|
|
test('an fs failure inside the staleness check sets verification_stale_check_indeterminate:true; passed/blockers unchanged', (t) => {
|
|
const { summaryPath, verificationPath } = seedVerifiedPhase();
|
|
const origStatSync = fs.statSync;
|
|
|
|
t.mock.method(fs, 'statSync', function injectedStaleCheckFault(target, ...args) {
|
|
const targetPath = String(target);
|
|
if (targetPath === verificationPath || targetPath === summaryPath) {
|
|
throw new Error('injected stat failure (#3057 B3)');
|
|
}
|
|
return origStatSync.call(fs, target, ...args);
|
|
});
|
|
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
|
|
// Pre-existing no-throw fail-open routing is UNCHANGED — `passed`/
|
|
// `blockers` are exactly what they would be without the injected fault.
|
|
assert.strictEqual(report.passed, true);
|
|
assert.deepStrictEqual(report.blockers, []);
|
|
assert.strictEqual(report.verification_stale_check_indeterminate, true);
|
|
});
|
|
|
|
test('a completed staleness check that finds nothing stale reports verification_stale_check_indeterminate:false', () => {
|
|
seedVerifiedPhase();
|
|
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.verification_stale_check_indeterminate, false);
|
|
});
|
|
|
|
test('requireVerification not set → verification_stale_check_indeterminate is always false (readVerificationStatus never reached)', () => {
|
|
seedVerifiedPhase();
|
|
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: false } });
|
|
|
|
assert.strictEqual(report.verification_stale_check_indeterminate, false);
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — malformed markdown guard ─────────────────────────────
|
|
|
|
describe('evaluateUatPassed — malformed markdown blocker', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('unterminated code fence → malformed blocker → passed:false (even with real result:passed)', () => {
|
|
const content = [
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: passed',
|
|
'',
|
|
'```',
|
|
'unterminated fence — no closing delimiter',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'Unterminated fence must always block, even if a result:passed exists');
|
|
assert.ok(report.blockers.some(b => /malformed/i.test(b)),
|
|
`Expected malformed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('unterminated HTML comment → malformed blocker → passed:false', () => {
|
|
const content = [
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: passed',
|
|
'',
|
|
'<!-- unterminated comment',
|
|
'no closing arrow',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'Unterminated comment must always block');
|
|
assert.ok(report.blockers.some(b => /malformed/i.test(b)),
|
|
`Expected malformed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('well-formed fences → no malformed blocker', () => {
|
|
const content = [
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: passed',
|
|
'',
|
|
'```',
|
|
'code here',
|
|
'```',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.ok(!report.blockers.some(b => /malformed/i.test(b)),
|
|
'Well-formed fences must not produce malformed blocker');
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — hardening regressions (the heart of #247) ────────────
|
|
|
|
describe('evaluateUatPassed — false-positive hardening regressions', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('#247: result:passed inside fenced block: parseUatResultItems returns [] for fake, evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
|
|
// ONLY result: passed is inside a fenced block — no real test blocks
|
|
const rawContent = [
|
|
'```',
|
|
'### 1. Fake Test',
|
|
'expected: Example output',
|
|
'result: passed',
|
|
'```',
|
|
].join('\n');
|
|
// After stripping, the clean content has no headings at all
|
|
const clean = stripFalsePositiveContexts(rawContent);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 0, 'Fake result inside fence must produce no items');
|
|
|
|
writeFile(tmpDir, 'phase-UAT.md', rawContent);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'result:passed inside a fenced block must not flip passed to true');
|
|
assert.strictEqual(report.no_uat_artifacts, true,
|
|
'no real UAT items → no_uat_artifacts:true');
|
|
});
|
|
|
|
test('#247: result:passed inside blockquote: parseUatResultItems returns [] + evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
|
|
const rawContent = [
|
|
'> ### 1. Test',
|
|
'> expected: Example',
|
|
'> result: passed',
|
|
].join('\n');
|
|
const clean = stripFalsePositiveContexts(rawContent);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 0, 'Fake result inside blockquote must produce no items');
|
|
|
|
writeFile(tmpDir, 'phase-UAT.md', rawContent);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.strictEqual(report.no_uat_artifacts, true);
|
|
});
|
|
|
|
test('#247: result:passed inside HTML comment: parseUatResultItems returns [] + evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
|
|
const rawContent = [
|
|
'<!-- ### 1. Test',
|
|
'expected: Example',
|
|
'result: passed -->',
|
|
].join('\n');
|
|
const clean = stripFalsePositiveContexts(rawContent);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 0, 'Fake result inside HTML comment must produce no items');
|
|
|
|
writeFile(tmpDir, 'phase-UAT.md', rawContent);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.strictEqual(report.no_uat_artifacts, true);
|
|
});
|
|
|
|
test('#247: result:passed inside frontmatter: parseUatResultItems returns [] + evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
|
|
const rawContent = [
|
|
'---',
|
|
'example_result: passed',
|
|
'---',
|
|
'',
|
|
'No real test blocks here.',
|
|
].join('\n');
|
|
const clean = stripFalsePositiveContexts(rawContent);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 0, 'Fake result inside frontmatter must produce no items');
|
|
|
|
writeFile(tmpDir, 'phase-UAT.md', rawContent);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.strictEqual(report.no_uat_artifacts, true);
|
|
});
|
|
|
|
test('#247: result:passed inside fenced block is NOT treated as passing test (with real failing test)', () => {
|
|
const content = [
|
|
'---',
|
|
'status: partial',
|
|
'---',
|
|
'',
|
|
'# Example',
|
|
'',
|
|
'```',
|
|
'### 1. Test',
|
|
'expected: Example output',
|
|
'result: passed',
|
|
'```',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'result:passed inside a fenced block must not flip passed to true');
|
|
assert.ok(report.checks.some(c => c.result === 'pending' && !c.passing),
|
|
'Real pending test must be captured');
|
|
});
|
|
|
|
test('#247: result:passed inside blockquote is NOT treated as passing test', () => {
|
|
const content = [
|
|
'---',
|
|
'status: partial',
|
|
'---',
|
|
'',
|
|
'> ### 1. Test',
|
|
'> expected: Example',
|
|
'> result: passed',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'result:passed inside a blockquote must not flip passed to true');
|
|
});
|
|
|
|
test('#247: result:passed inside HTML comment is NOT treated as passing test', () => {
|
|
const content = [
|
|
'---',
|
|
'status: partial',
|
|
'---',
|
|
'',
|
|
'<!-- ### 1. Test',
|
|
'expected: Example',
|
|
'result: passed -->',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'result:passed inside an HTML comment must not flip passed to true');
|
|
});
|
|
|
|
test('#247: result:passed inside frontmatter example is NOT treated as passing test', () => {
|
|
const content = [
|
|
'---',
|
|
'status: partial',
|
|
'example_result: passed',
|
|
'---',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'result:passed in frontmatter must not flip passed to true');
|
|
});
|
|
|
|
test('#247: real passing test block outside all contexts → passed:true', () => {
|
|
const content = [
|
|
'---',
|
|
'status: passed',
|
|
'---',
|
|
'',
|
|
'# UAT Results',
|
|
'',
|
|
'### 1. Login works',
|
|
'expected: User logs in',
|
|
'result: passed',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true,
|
|
'A real passing test outside false-positive contexts must pass');
|
|
});
|
|
|
|
test('block-scalar expected: followed by blank line + result: pending → parsed as blocker (not dropped)', () => {
|
|
const content = [
|
|
'### 1. Test A',
|
|
'expected: |',
|
|
' multi',
|
|
' line expected output',
|
|
'',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'Block-scalar expected: with result: pending must be captured as a blocker');
|
|
assert.ok(report.checks.some(c => c.result === 'pending' && !c.passing),
|
|
`Expected pending check, got: ${JSON.stringify(report.checks)}`);
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — output shape contract ───────────────────────────────
|
|
|
|
describe('evaluateUatPassed — output shape (Hyrum\'s Law contract)', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('returns all required fields in the locked shape including no_uat_artifacts', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir);
|
|
|
|
// Locked field names
|
|
assert.ok('passed' in report, 'report.passed must exist');
|
|
assert.ok('uat_files' in report, 'report.uat_files must exist');
|
|
assert.ok('verification_files' in report, 'report.verification_files must exist');
|
|
assert.ok('checks' in report, 'report.checks must exist');
|
|
assert.ok('blockers' in report, 'report.blockers must exist');
|
|
assert.ok('no_uat_artifacts' in report, 'report.no_uat_artifacts must exist');
|
|
assert.ok('policy' in report, 'report.policy must exist');
|
|
assert.ok('require_verification' in report.policy, 'report.policy.require_verification must exist');
|
|
|
|
// checks item shape
|
|
if (report.checks.length > 0) {
|
|
const c = report.checks[0];
|
|
assert.ok('file' in c, 'check.file must exist');
|
|
assert.ok('test' in c, 'check.test must exist');
|
|
assert.ok('name' in c, 'check.name must exist');
|
|
assert.ok('result' in c, 'check.result must exist');
|
|
assert.ok('passing' in c, 'check.passing must exist');
|
|
}
|
|
});
|
|
|
|
test('no_uat_artifacts is false when real checks exist', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.no_uat_artifacts, false);
|
|
});
|
|
|
|
test('no_uat_artifacts is true when no checks exist', () => {
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.no_uat_artifacts, true);
|
|
});
|
|
|
|
test('uat_files contains the filename', () => {
|
|
writeFile(tmpDir, 'my-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.ok(report.uat_files.includes('my-UAT.md'),
|
|
`uat_files should include 'my-UAT.md', got: ${JSON.stringify(report.uat_files)}`);
|
|
});
|
|
|
|
test('verification_files contains the filename', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.ok(report.verification_files.includes('phase-VERIFICATION.md'),
|
|
`verification_files should include 'phase-VERIFICATION.md', got: ${JSON.stringify(report.verification_files)}`);
|
|
});
|
|
});
|
|
|
|
// ─── #3511: phase-scoped UAT/VERIFICATION scanning — the transition gate ─────
|
|
//
|
|
// evaluateUatPassed is the CRITICAL anchor for #3511: its `passed`/`blockers`
|
|
// fields directly gate a phase transition. A cross-phase stray file sitting
|
|
// in a phase directory must never contribute a blocker to a phase it does not
|
|
// belong to, and the phase's own artifacts must keep behaving exactly as
|
|
// before.
|
|
|
|
describe('#3511: evaluateUatPassed — cross-phase stray files do not contribute blockers', () => {
|
|
let baseDir;
|
|
let phaseDir;
|
|
|
|
beforeEach(() => {
|
|
baseDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-3511-uat-pred-'));
|
|
// Phase-shaped basename ("03-…") so isPhaseArtifact actually scopes —
|
|
// extractPhaseToken('03-uat-predicate') derives token "03".
|
|
phaseDir = path.join(baseDir, '03-uat-predicate');
|
|
fs.mkdirSync(phaseDir);
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(baseDir);
|
|
});
|
|
|
|
test('passed:true with a passing own UAT + own VERIFICATION, despite a blocking cross-phase stray VERIFICATION', () => {
|
|
writeFile(phaseDir, '03-UAT.md', makePassingUat(1));
|
|
writeFile(phaseDir, '03-VERIFICATION.md', '---\nstatus: passed\n---\n\nOK.');
|
|
// Cross-phase stray: a "04" VERIFICATION file sitting in phase 03's
|
|
// directory, with a BLOCKING status. Pre-#3511, this unscoped scan would
|
|
// have picked it up and blocked phase 03's transition.
|
|
writeFile(phaseDir, '04-VERIFICATION.md', '---\nstatus: human_needed\n---\n\nNeeds human check.');
|
|
|
|
const report = evaluateUatPassed(phaseDir);
|
|
assert.strictEqual(report.passed, true,
|
|
'a cross-phase stray VERIFICATION file must not block this phase\'s transition');
|
|
assert.ok(
|
|
!report.blockers.some(b => /04-VERIFICATION\.md/.test(b) || /human_needed/i.test(b)),
|
|
`blockers must not name the stray file; got: ${JSON.stringify(report.blockers)}`,
|
|
);
|
|
assert.strictEqual(report.verification_files.includes('04-VERIFICATION.md'), false,
|
|
'the stray must not even be counted as a verification_files entry for this phase');
|
|
});
|
|
|
|
test('passed:false with own report still failing, unaffected by an unrelated passing cross-phase stray (non-stray case unchanged)', () => {
|
|
writeFile(phaseDir, '03-UAT.md', makePassingUat(1));
|
|
// This phase's own VERIFICATION is blocking.
|
|
writeFile(phaseDir, '03-VERIFICATION.md', '---\nstatus: gaps_found\n---\n\nHas gaps.');
|
|
// A cross-phase stray that is itself passing must not paper over the
|
|
// phase's own real failure either — over-exclusion is as dangerous as
|
|
// under-exclusion here.
|
|
writeFile(phaseDir, '99-VERIFICATION.md', '---\nstatus: passed\n---\n\nOK.');
|
|
|
|
const report = evaluateUatPassed(phaseDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'the phase\'s own gaps_found VERIFICATION must still block, exactly as before #3511');
|
|
assert.ok(report.blockers.some(b => /gaps_found/i.test(b)),
|
|
`blockers must still name this phase's own gaps_found status; got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('#3511 follow-up: passed:true from a NON-canonical dir shape "1-unpadded" (over-exclusion / no_uat_artifacts check)', () => {
|
|
// "1-unpadded" tokenizes to literal "1"; scaffold writes the PADDED
|
|
// "01-…" form (normalizePhaseName). A literal token compare excluded the
|
|
// phase's own artifacts here, flipping `no_uat_artifacts: true` and
|
|
// false-blocking the transition gate this predicate feeds.
|
|
const unpaddedDir = path.join(baseDir, '1-unpadded');
|
|
fs.mkdirSync(unpaddedDir);
|
|
writeFile(unpaddedDir, '01-UAT.md', makePassingUat(1));
|
|
writeFile(unpaddedDir, '01-VERIFICATION.md', '---\nstatus: passed\n---\n\nOK.');
|
|
|
|
const report = evaluateUatPassed(unpaddedDir);
|
|
assert.strictEqual(report.no_uat_artifacts, false,
|
|
`own UAT/VERIFICATION files in an unpadded-dir phase must be found; got: ${JSON.stringify(report)}`);
|
|
assert.strictEqual(report.passed, true,
|
|
`the phase's own passing files in a non-canonical dir must pass the gate; got: ${JSON.stringify(report)}`);
|
|
});
|
|
});
|
|
|
|
// ─── FIX A regression: nested-fence (~~~ inside ```) ─────────────────────────
|
|
|
|
describe('FIX A — nested fence: ~~~ inside ``` does not prematurely close outer fence', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => { tmpDir = makeTmpDir(); });
|
|
afterEach(() => { rmDir(tmpDir); });
|
|
|
|
test('parseUatResultItems sees [] for fake inside ``` that encloses ~~~', () => {
|
|
// A backtick fence that contains an inner ~~~ fence with a fake test block.
|
|
// The ~~~ must NOT close the ``` fence — the whole interior is content and is dropped.
|
|
const raw = [
|
|
'```',
|
|
'~~~',
|
|
'### 1. Fake',
|
|
'expected: X',
|
|
'result: passed',
|
|
'~~~',
|
|
'```',
|
|
].join('\n');
|
|
const clean = stripFalsePositiveContexts(raw);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 0, 'fake inside nested fence must not leak through');
|
|
});
|
|
|
|
test('evaluateUatPassed → passed:false + no_uat_artifacts:true for nested-fence-only file', () => {
|
|
const raw = [
|
|
'```',
|
|
'~~~',
|
|
'### 1. Fake',
|
|
'expected: X',
|
|
'result: passed',
|
|
'~~~',
|
|
'```',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', raw);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false, 'nested-fence fake must not flip passed');
|
|
assert.strictEqual(report.no_uat_artifacts, true, 'no real items → no_uat_artifacts:true');
|
|
assert.ok(!report.checks.some(c => c.name === 'Fake'), 'fake must not appear in checks');
|
|
});
|
|
|
|
test('balanced nested fence (``` inside ~~~) is NOT flagged as malformed', () => {
|
|
// ~~~ outer, ``` inner — properly closed — should not trigger unterminatedFence
|
|
const raw = [
|
|
'~~~',
|
|
'```',
|
|
'code',
|
|
'```',
|
|
'~~~',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Works',
|
|
'result: passed',
|
|
].join('\n');
|
|
const { unterminatedFence } = analyzeMarkdown(raw);
|
|
assert.strictEqual(unterminatedFence, false, 'balanced nested fence must not be flagged');
|
|
const clean = stripFalsePositiveContexts(raw);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 1, 'real test outside fence must still be found');
|
|
assert.strictEqual(items[0].result, 'passed');
|
|
});
|
|
|
|
test('real result:passed outside ``` that encloses ~~~ → passed:true (no false-block)', () => {
|
|
const raw = [
|
|
'---',
|
|
'status: passed',
|
|
'---',
|
|
'',
|
|
'```',
|
|
'~~~',
|
|
'### 1. Fake',
|
|
'result: passed',
|
|
'~~~',
|
|
'```',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Works',
|
|
'result: passed',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', raw);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true, 'real result outside nested fence must still pass');
|
|
assert.ok(!report.checks.some(c => c.name === 'Fake'), 'fake must not appear in checks');
|
|
});
|
|
});
|
|
|
|
// ─── FIX B regression: cross-line result value ────────────────────────────────
|
|
|
|
describe('FIX B — cross-line result: value must be on the same line', () => {
|
|
test('result: with value on next line → result:missing (not passed)', () => {
|
|
const content = [
|
|
'### 1. Cross-line Test',
|
|
'expected: Y',
|
|
'result:',
|
|
'',
|
|
'passed',
|
|
].join('\n');
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.notStrictEqual(items[0].result, 'passed',
|
|
'result value on a subsequent line must not be captured as passed');
|
|
assert.strictEqual(items[0].result, 'missing',
|
|
'cross-line result must yield missing (blocker)');
|
|
});
|
|
|
|
test('result: whose value sits on a following INDENTED line → missing (pinned #3078-CR divergence)', () => {
|
|
// #3078-CR (security review follow-up): on origin/next, the old
|
|
// `/^result:\s*\[?(\w+)\]?.*$/im` regex's `\s*` is greedy and matches
|
|
// ACROSS a newline, so `result:\n blocked` parsed as `blocked` — a real
|
|
// row this shape reported 1/blocked. The split-then-match rewrite tests
|
|
// `result:` against a SINGLE already-split line, per the documented
|
|
// "value must sit on the SAME line as result:" contract (see the comment
|
|
// above RESULT_LINE_RE), so this now yields 'missing' (a parse gap, with
|
|
// the percentage withheld) instead of silently crossing the newline.
|
|
// This is a DELIBERATE, FAIL-SAFE divergence from the old cross-newline
|
|
// `\s*` behavior — pinned here so it is never "fixed" back by accident.
|
|
const content = [
|
|
'### 1. Indented-continuation Test',
|
|
'expected: Y',
|
|
'result:',
|
|
' blocked',
|
|
'',
|
|
].join('\n');
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.notStrictEqual(items[0].result, 'blocked',
|
|
'a value on a following indented line must not be captured across the newline');
|
|
assert.strictEqual(items[0].result, 'missing',
|
|
'result: with its value on the next (even indented) line must yield missing, not the old cross-newline capture');
|
|
});
|
|
|
|
test('evaluateUatPassed → passed:false for cross-line result:passed', () => {
|
|
const tmpDir = makeTmpDir();
|
|
try {
|
|
const content = [
|
|
'### 1. Cross-line Test',
|
|
'expected: Y',
|
|
'result:',
|
|
'',
|
|
'passed',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(!report.checks.some(c => c.result === 'passed'),
|
|
'cross-line result must not produce a passing check');
|
|
} finally {
|
|
rmDir(tmpDir);
|
|
}
|
|
});
|
|
});
|
|
|
|
// ─── FIX C regression: masked-comment (earlier closed comment + later unterminated) ──
|
|
|
|
describe('FIX C — dangling comment survives earlier balanced comment', () => {
|
|
test('analyzeMarkdown detects unterminated comment after a properly closed one', () => {
|
|
const raw = [
|
|
'<!-- ok -->',
|
|
'Some text',
|
|
'<!--',
|
|
'### 1. Fake',
|
|
'result: passed',
|
|
].join('\n');
|
|
const { unterminatedComment } = analyzeMarkdown(raw);
|
|
assert.strictEqual(unterminatedComment, true,
|
|
'later unterminated comment must be detected even after a balanced one');
|
|
});
|
|
|
|
test('evaluateUatPassed → passed:false (malformed) when later comment is unterminated', () => {
|
|
const tmpDir = makeTmpDir();
|
|
try {
|
|
const raw = [
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: passed',
|
|
'',
|
|
'<!-- properly closed -->',
|
|
'',
|
|
'<!--',
|
|
'### 2. Fake',
|
|
'result: passed',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', raw);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'unterminated later comment must trigger malformed blocker');
|
|
assert.ok(report.blockers.some(b => /malformed/i.test(b)),
|
|
`Expected malformed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
} finally {
|
|
rmDir(tmpDir);
|
|
}
|
|
});
|
|
|
|
test('analyzeMarkdown does NOT flag a file with only properly balanced comments', () => {
|
|
const raw = [
|
|
'<!-- first comment -->',
|
|
'Some text',
|
|
'<!-- second comment -->',
|
|
'',
|
|
'### 1. Real Test',
|
|
'result: passed',
|
|
].join('\n');
|
|
const { unterminatedComment } = analyzeMarkdown(raw);
|
|
assert.strictEqual(unterminatedComment, false,
|
|
'only balanced comments must not be flagged as unterminated');
|
|
});
|
|
});
|
|
|
|
// ─── FIX D regression: balanced mixed fences are NOT flagged ─────────────────
|
|
|
|
describe('FIX D — odd-fence-count heuristic replaced: balanced mixed fences not flagged', () => {
|
|
test('analyzeMarkdown: ``` followed by ~~~ (both balanced) → unterminatedFence:false', () => {
|
|
const raw = [
|
|
'```',
|
|
'code',
|
|
'```',
|
|
'',
|
|
'~~~',
|
|
'more code',
|
|
'~~~',
|
|
].join('\n');
|
|
const { unterminatedFence } = analyzeMarkdown(raw);
|
|
assert.strictEqual(unterminatedFence, false,
|
|
'two separate balanced fences must not trigger unterminatedFence');
|
|
});
|
|
|
|
test('evaluateUatPassed: multiple balanced fences + real passing test → passed:true, no malformed blocker', () => {
|
|
const tmpDir = makeTmpDir();
|
|
try {
|
|
const raw = [
|
|
'---',
|
|
'status: passed',
|
|
'---',
|
|
'',
|
|
'```',
|
|
'result: fake',
|
|
'```',
|
|
'',
|
|
'~~~',
|
|
'result: also fake',
|
|
'~~~',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Works',
|
|
'result: passed',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', raw);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.ok(!report.blockers.some(b => /malformed/i.test(b)),
|
|
`Balanced mixed fences must not produce malformed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
assert.strictEqual(report.passed, true,
|
|
'real test outside balanced fences must still pass');
|
|
} finally {
|
|
rmDir(tmpDir);
|
|
}
|
|
});
|
|
});
|
|
|
|
// ─── FIX E extra: fake-not-in-checks assertions for existing mixed tests ──────
|
|
|
|
describe('FIX E — fake items must NOT appear in checks (absence assertions)', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => { tmpDir = makeTmpDir(); });
|
|
afterEach(() => { rmDir(tmpDir); });
|
|
|
|
test('fake inside fence + real pending: fake NOT in checks', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'```',
|
|
'### 10. Fake',
|
|
'expected: Fake',
|
|
'result: passed',
|
|
'```',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(!report.checks.some(c => c.name === 'Fake'),
|
|
'fake test inside fence must not appear in checks');
|
|
});
|
|
|
|
test('fake inside blockquote + real pending: fake NOT in checks', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'> ### 10. Fake',
|
|
'> expected: Fake',
|
|
'> result: passed',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(!report.checks.some(c => c.name === 'Fake'),
|
|
'fake test inside blockquote must not appear in checks');
|
|
});
|
|
|
|
test('fake inside HTML comment + real pending: fake NOT in checks', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'<!-- ### 10. Fake',
|
|
'expected: Fake',
|
|
'result: passed -->',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(!report.checks.some(c => c.name === 'Fake'),
|
|
'fake test inside HTML comment must not appear in checks');
|
|
});
|
|
});
|
|
|
|
// ─── Property-based test (fast-check) ─────────────────────────────────────────
|
|
|
|
describe('evaluateUatPassed — property: wrapping in false-positive context never flips to passed', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('fc: inserting result:passed inside wrapper context never flips a failing UAT to passed', () => {
|
|
// A baseline UAT file that has a pending item — it must always evaluate to passed:false
|
|
// regardless of how many "result: passed" lines we inject inside fenced/blockquote/comment wrappers.
|
|
const baseFailingBody = [
|
|
'### 1. Real Test',
|
|
'expected: It works',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
|
|
const wrappers = fc.constantFrom(
|
|
// backtick fence
|
|
(inner) => '```\n' + inner + '\n```',
|
|
// tilde fence
|
|
(inner) => '~~~\n' + inner + '\n~~~',
|
|
// HTML comment
|
|
(inner) => '<!--\n' + inner + '\n-->',
|
|
// blockquote — prefix each line
|
|
(inner) => inner.split('\n').map(l => '> ' + l).join('\n'),
|
|
);
|
|
|
|
fc.assert(
|
|
fc.property(wrappers, fc.nat(3), (wrap, extraCount) => {
|
|
// Build a "fake passing block" that would fool a naive regex
|
|
const fakePassingLines = Array.from({ length: extraCount + 1 }, (_, i) =>
|
|
`### ${i + 10}. Fake Test ${i + 10}\nexpected: Fake\nresult: passed`
|
|
).join('\n');
|
|
|
|
const fullContent = [
|
|
'---',
|
|
'status: partial',
|
|
'---',
|
|
'',
|
|
wrap(fakePassingLines),
|
|
'',
|
|
baseFailingBody,
|
|
].join('\n');
|
|
|
|
// Write to a unique tmp file to avoid cross-test state
|
|
const fcDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-fc-uat-'));
|
|
try {
|
|
fs.writeFileSync(path.join(fcDir, 'feature-UAT.md'), fullContent, 'utf-8');
|
|
const report = evaluateUatPassed(fcDir);
|
|
// The pending item must always keep passed:false
|
|
// AND fake items injected via wrappers must never appear in checks
|
|
const hasFakeInChecks = report.checks.some(c => c.name.startsWith('Fake Test'));
|
|
return report.passed === false && !hasFakeInChecks;
|
|
} finally {
|
|
cleanup(fcDir);
|
|
}
|
|
}),
|
|
{ numRuns: 50 }
|
|
);
|
|
});
|
|
});
|
|
|
|
// ─── #3078-CR MEDIUM: acceptance gate (uat-predicate.cjs) must AGREE with the ──
|
|
// ─── audit surface (uat.cjs's parseUatItemsWithStats) on the same bytes ───────
|
|
|
|
describe('#3078-CR: evaluateUatPassed agrees with the audit surface (parseUatItemsWithStats)', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('U+2028 scalar-injection: gate blocks, audit surface reports the outstanding row — they AGREE', () => {
|
|
// A `result:` line reachable only via a U+2028 LINE SEPARATOR sitting inside
|
|
// an `expected: |` block-scalar body must not be read as a genuine
|
|
// column-0 match by EITHER surface. The real, later `result: blocked` line
|
|
// is the one that must win.
|
|
const LS = String.fromCharCode(0x2028); // never a raw separator in source: a formatter that normalizes line separators would silently turn this fixture into an ordinary-character control that still passes
|
|
const body = [
|
|
'---',
|
|
'status: passed',
|
|
'---',
|
|
'',
|
|
'# UAT',
|
|
'',
|
|
'### 1. Alpha',
|
|
'expected: |',
|
|
' x' + LS + 'result: pass',
|
|
'result: blocked',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, '01-alpha-UAT.md', body);
|
|
|
|
const gateReport = evaluateUatPassed(tmpDir);
|
|
const auditReport = parseUatItemsWithStats(body);
|
|
|
|
// AGREEMENT, asserted explicitly (not each surface independently): both
|
|
// surfaces must consider this phase NOT clean, on the same test row.
|
|
assert.equal(gateReport.passed, false, 'gate: must not accept a blocked test as passed');
|
|
assert.equal(auditReport.items.length, 1, 'audit: the blocked row must surface as outstanding');
|
|
assert.equal(auditReport.items[0].result, 'blocked', 'audit: must read the real result, not the injected one');
|
|
const gateCheck = gateReport.checks.find((c) => c.test === 1);
|
|
assert.ok(gateCheck, 'gate: must record the test-1 check');
|
|
assert.equal(gateCheck.result, 'blocked', 'gate: must read the real result, not the injected one');
|
|
assert.equal(gateCheck.passing, false);
|
|
// Cross-surface identity: same test number, same result token.
|
|
assert.equal(gateCheck.result, auditReport.items[0].result, 'gate and audit surface must agree on the result token');
|
|
});
|
|
|
|
test('H-U28 restored: a heading delimited by U+2028 (not \n) is still found and blocks', () => {
|
|
// #3078-CR MEDIUM 1 (security review follow-up): origin/next found this
|
|
// heading via an /m-anchored scan whose LineTerminator set includes
|
|
// U+2028/U+2029; a naive split('\n')-only port of that scan silently
|
|
// stopped finding it, making the gate MORE PERMISSIVE than origin/next
|
|
// (measured: HEAD passed:true/0 blockers, origin/next passed:false/1
|
|
// blocker, for this exact shape). The heading scan now splits on
|
|
// \n/U+2028/U+2029 (a STRUCTURE frame) while the result: scan below it
|
|
// stays \n-only (an ATTRIBUTION frame, unchanged) -- so the heading is
|
|
// found, but its result: line -- separated from the heading by the same
|
|
// exotic separator -- is correctly NOT read across that boundary (that is
|
|
// the attribution guard I-U28 below exists to prove), yielding
|
|
// 'missing' rather than 'blocked'. Either token is a non-passing,
|
|
// blocking state, so the gate still BLOCKS -- the outcome origin/next
|
|
// produced, restored.
|
|
const LS = String.fromCharCode(0x2028); // never a raw separator in source: a formatter that normalizes line separators would silently turn this fixture into an ordinary-character control that still passes
|
|
const content = 'Notes.' + LS + '### 2. B' + LS + 'result: blocked';
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1, 'a U+2028-delimited heading must still be found');
|
|
assert.strictEqual(items[0].test, 2);
|
|
assert.strictEqual(items[0].name, 'B');
|
|
// IDENTITY, not a proxy: the exact token. `notStrictEqual(..., 'passed')` also passes on
|
|
// 'pass', which IS in UAT_PASS_RESULTS -- so it could not catch a regression that
|
|
// attributed a PASSING result to the recovered heading, which is the whole risk here.
|
|
assert.strictEqual(items[0].result, 'missing',
|
|
'the result: line sits across the same exotic separator, so it is correctly NOT attributed -- '
|
|
+ 'missing is a non-passing, blocking state');
|
|
});
|
|
|
|
test('H-U28 restored: evaluateUatPassed BLOCKS on the U+2028-delimited heading shape', () => {
|
|
const tmpDir = makeTmpDir();
|
|
try {
|
|
const LS = String.fromCharCode(0x2028); // never a raw separator in source: a formatter that normalizes line separators would silently turn this fixture into an ordinary-character control that still passes
|
|
const content = 'Notes.' + LS + '### 2. B' + LS + 'result: blocked';
|
|
const body = ['---', 'status: passed', '---', '', '# UAT', '', content, ''].join('\n');
|
|
writeFile(tmpDir, '01-h28-UAT.md', body);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false, 'gate must block on the U+2028-delimited heading -- origin/next parity');
|
|
assert.ok(report.blockers.length > 0, 'a blocker must be recorded');
|
|
} finally {
|
|
rmDir(tmpDir);
|
|
}
|
|
});
|
|
|
|
test('CR-fenced case: gate and audit surface (parseUatItemsWithStats) agree -- neither silently clean', () => {
|
|
// #3078-CR MEDIUM 1 follow-up evidence: a lone-CR document
|
|
// (see:CRfenceCR### 2. BCRresult: blockedCRfence) has its CRs normalized
|
|
// to \n before parsing, which turns a literal fence-marker sequence into
|
|
// a REAL fence delimiter it was not before normalization -- the row ends
|
|
// up fenced and stripped on the gate side. Both surfaces must agree this
|
|
// phase is NOT clean (the gate must not pass while the audit surface
|
|
// reports a shortfall/gap for the same document).
|
|
const crBody = ['see:', '```', '### 2. B', 'result: blocked', '```'].join('\r');
|
|
const tmpDir = makeTmpDir();
|
|
try {
|
|
writeFile(tmpDir, '01-crfence-UAT.md', crBody);
|
|
const gateReport = evaluateUatPassed(tmpDir);
|
|
const auditReport = parseUatItemsWithStats(crBody);
|
|
assert.strictEqual(gateReport.passed, false, 'gate must not report a clean pass for this document');
|
|
assert.ok(auditReport.headingsSeen > 0, 'audit surface must see the heading exists');
|
|
assert.ok(auditReport.items.length === 0 && auditReport.shortfallBlocks > 0,
|
|
'audit surface must record the fenced row as an unresolved shortfall, not silently drop it');
|
|
} finally {
|
|
rmDir(tmpDir);
|
|
}
|
|
});
|
|
|
|
test('lone-CR frontmatter: gate no longer silently drops a blocking status hidden by an unnormalized read', () => {
|
|
// A lone-CR-terminated frontmatter fence (`---\rstatus: partial\r---`) must
|
|
// still be recognised as frontmatter — the raw, unnormalized read this
|
|
// fix replaces treated the whole fence as one unbroken line, so
|
|
// `extractFrontmatter` never matched it and the blocking `status: partial`
|
|
// was silently dropped (fail-OPEN, the false-clean this fix closes).
|
|
const body = [
|
|
'---\rstatus: partial\r---',
|
|
'',
|
|
'# UAT',
|
|
'',
|
|
'### 1. Alpha\rresult: passed',
|
|
'### 2. Beta\rresult: pass',
|
|
'',
|
|
].join('\r');
|
|
writeFile(tmpDir, '01-beta-UAT.md', body);
|
|
|
|
const gateReport = evaluateUatPassed(tmpDir);
|
|
assert.equal(gateReport.passed, false, 'gate: the hidden status: partial must now block');
|
|
assert.ok(
|
|
gateReport.blockers.some((b) => b.includes('status=partial')),
|
|
'gate: the frontmatter status blocker must be surfaced, not silently dropped',
|
|
);
|
|
});
|
|
|
|
test('clean control: a normally-passing file agrees as passed on both surfaces', () => {
|
|
const body = [
|
|
'---',
|
|
'status: passed',
|
|
'---',
|
|
'',
|
|
'# UAT',
|
|
'',
|
|
'### 1. Alpha',
|
|
'result: passed',
|
|
'',
|
|
'### 2. Beta',
|
|
'result: pass',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, '01-gamma-UAT.md', body);
|
|
|
|
const gateReport = evaluateUatPassed(tmpDir);
|
|
const auditReport = parseUatItemsWithStats(body);
|
|
|
|
// AGREEMENT: the gate accepts, and the audit surface reports NO outstanding
|
|
// (non-passing) rows for the same bytes.
|
|
assert.equal(gateReport.passed, true, 'gate: a clean file must still pass');
|
|
assert.equal(auditReport.items.length, 0, 'audit: a clean file must have no outstanding rows');
|
|
});
|
|
});
|