Files
msd-core/tests/uat-predicate.test.cjs
Tom Boucher 53ea8e0664 fix(#3057): make a guard's failure distinguishable from its benign result — Wave 1 (#3088)
* fix(#3057): refuse the write when the duplicate scan cannot complete

writeManifest documents itself as a fail-closed duplicate guard: if any
existing manifest shares plan_id with a different, non-terminal job_id it must
refuse, because dispatching again would duplicate the external job.

It could not honour that. The scan reads every sibling manifest looking for the
duplicate, and an unreadable or unparseable sibling was `continue`d past. If
the corrupt file was the one holding the live duplicate, the scan found nothing
and a duplicate external job dispatched.

The asymmetry is what gives it away: a malformed TARGET refused with
malformed_existing because clobbering is unacceptable, while a malformed
SIBLING was skipped — yet siblings are the only thing the duplicate check
reads.

Adds a scan_incomplete verdict that refuses and names the offending file, so an
operator can quarantine or repair it. Fail-closed alone would let one stale
corrupt manifest wedge every dispatch for that planning dir permanently; naming
the file is what makes refusing survivable. malformed_existing is untouched, so
the target/sibling distinction stays visible. The docstring is updated — it
previously stated a rule the function did not keep.

memFs() gains an optional failReads map so these branches are reachable at all;
they had zero coverage because the fake could not express a per-file read
fault. The signature is additive and every existing caller is unchanged.

The regression is proved by a pair, not a single test. A control writes a
readable sibling holding a genuine non-terminal duplicate and asserts
duplicate_plan_id, establishing the scenario is real; the regression then makes
that same path unreadable and asserts scan_incomplete. A first draft of this
test used a corrupt-JSON fixture containing no plan_id at all while its comment
claimed otherwise — it duplicated the unparseable-sibling case and proved
nothing, which is the defect class this phase exists to remove.

Refs #3051

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#3057): make a guard's failure distinguishable from its benign result

Wave 1 of the negative-space backfill: the branches where a guard that could
not verify something reported the same value it reports when everything is
fine. That indistinguishability is the defect; every fix here makes the two
states tellable apart, and every test proves it with a pair — one for the
failure, one for the benign case. A single test cannot establish that two
states are distinguishable, which is the whole property being fixed.

state.cts phaseInventoryProvider returned null for both a real disk-scan
failure and a genuinely empty phases dir, so `state rebuild` could report
success while phase-table reconciliation never ran. It now returns a
discriminated result and the CLI surfaces phase_inventory_scan_failed plus a
reason. The reason field turned out never to have been wired into the emitted
JSON at all — it existed only as an internal variable — so a test could only
assert on the operator-facing note. It is a real field now.

state.cts treated an unreadable lock body the same as an empty one, applying
the 1-second stealable floor. A lock we cannot read is not a lock we know is
stale; an unreadable body is now held to the deadman ceiling like a live
holder.

verification.cts findStaleVerificationSummary returned null on any fs, scan or
clock failure — meaning "not stale". It now returns a discriminated
StaleCheckResult and the caller records that the check was indeterminate.

git-base-branch resolveBaseBranch returned 'main' both when no candidate branch
existed and when every git tier timed out. A diagnostics variant now reports
whether the answer was verified, and the CLI writes an unverified-fallback note
to stderr. The stdout contract five workflows parse is untouched.

worktree-safety snapshotWorktreeInventory left exists:true when statSync threw,
so a guard that could not check reported the worktree present; exists is now
tri-state and a stat failure surfaces as an 'unverified' finding.
planWorktreePrune reported 'no_worktrees' for a parse failure, which is not the
same as an empty list — and it drives a prune. It now reports 'parse_failed'.

Fixing the inventory change exposed a second fail-open in verify.cts: the
validate-health consumer silently dropped findings whose kind it did not
recognise, so the new kind would have vanished. That is closed too — worth
noting that the survey enumerated producers of degraded verdicts, not consumers
that discard them.

worktree-base-ref and state-transition gain the distinguishing signal without
changing what they do: headAbsenceVerified, and a phase-inventory scan meta.
Whether those guards should ACT differently is a product question this change
does not answer, and both are flagged rather than quietly settled.

rescueSummaryArtifacts is left alone: rescuing on an uncertain cat-file is
deliberate per #2556. It now has tests proving it, and a recorded negative
finding — git cat-file -e returns 128 for both "absent from HEAD" and a fatal
error, so "uncertain" and "certain-and-fine" are not separable at the git
level.

Refs #3051

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#3057): assert typed values, not rendered text

Ten assertions in the rebuild CLI suite matched substrings of produced output —
STATE.md body fields, a markdown table row, an audit-log heading, and JSON keys
read as text. CONTRIBUTING prohibits that: if the code under test produces
text, the test asserts on its structured surface instead.

No production surface had to be built. Every one already existed and was
already compiled into bin/lib: stateExtractField for body fields,
parseMarkdownTable for the phase table, collectSection for the audit-log
section, and result.data.log — already a typed RebuildLogEntry[]. The tests
were matching rendered text sitting next to the structured data.

One of those assertions was passing for the wrong reason. `stdout.includes
('rebuilt')` matched the JSON KEY name, not a value: the dry-run path emits
`mutated` and the real path emits `rebuilt`, so it would have passed whether
the value was true or false. It now asserts the value.

external-job's refusal already had to name the offending file — that naming is
why the fail-closed variant is survivable rather than a permanent wedge — but
the tests proved it by substring of a prose message. The failure result now
carries offendingPath as its own field and the tests assert it by value. The
human message is unchanged; operators read it.

Array membership is left alone. `phaseIds.includes('99')` and
`result.updated.includes('Completed Phases')` are membership checks on real
arrays, not text matching, and converting them would weaken nothing and clarify
nothing.

Refs #3051

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#3057): execute acquireStateLock instead of grepping its source

The non-EEXIST lock test asserted on the TEXT of the built .cjs and never
called acquireStateLock. It carried an allow-test-rule: architectural-invariant
exemption to permit that. A source grep proves a literal is present in a file,
not that the behaviour works — it is weaker than a liveness test, which at
least runs the code, and it was the only coverage the fatal-errno path had.

Replaced with tests that inject the errno through fs and assert what actually
happens: a fatal EACCES propagates out of acquireStateLock with zero backoff
sleeps, while EAGAIN/EINTR/EINVAL/EIO/ENOENT/ESTALE/EPERM/EBUSY retry once and
succeed. The exemption is removed and its allowlist entry with it.

One old assertion is deliberately not carried over: it checked the retryable
errnos were expressed as a Set rather than an inline literal. That is a shape
check with no runtime signature; the behavioural tests fail if the code reverts
to the old inline check, which is the regression it was really guarding.

The #3057 lock-body tests move into that same file rather than a new one, which
is what lint-test-file-count asks for and puts every acquireStateLock test in
one place.

Refs #3051

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#3057): surface an indeterminate staleness check to its callers

An isolated review caught an inconsistency inside this wave. Two of the three
"add the distinguishing signal" fixes wire through to something a user sees:
git base-branch writes an unverified-fallback diagnostic to stderr, and an
unverifiable worktree surfaces as a W020 finding. The third set
staleCheckIndeterminate on readVerificationStatus's result and nothing read it.

A signal nobody consumes leaves the fail-open exactly as silent as before: the
staleness check could fail and the operator saw precisely what they would see
if the answer were genuinely "not stale". That is the defect this issue exists
to remove, so it is not defensible as scaffolding when its two siblings in the
same change already wire through.

All five callers now surface it, each through the channel it already had rather
than a mechanism imposed uniformly: phase complete adds it to its existing
warnings array and, on the blocked path, as an additive note on the error text;
init and roadmap carry it as a field on output they already emit; the UAT
report carries it without ever gating passed/blockers; workstream inventory
takes an injectable writeDiagnostic mirroring the git base-branch idiom,
because its return shape had nowhere to hang a per-phase field without
rippling the builder's types.

The routing decision is unchanged everywhere. What changes is only that a
caller and an operator can now tell a failed check from a completed one.

That diagnostic carries structured meta rather than being asserted by regex —
the default still writes only the human message to stderr, but tests assert
phaseDir and reason by value. Two earlier assertions in this branch were
converted the same way; this was the last raw-text assertion left.

Also records a scope correction: the completePhaseCore guards now compare
stateReplaceField's result to the body instead of testing truthiness, so a
field whose substitution produced identical text no longer reports as updated.
That is a real behaviour fix, not the signal-only change this file was
described as carrying, and its tests cover both the changed and unchanged
cases.

Refs #3051

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#3057): bound two heavy subprocesses for a loaded bench, not an idle one

The remote matrix surfaced three failures unrelated to this branch's changes.
All were bad tests, and a re-run would have hidden every one of them.

The reviewer-flags parse block bounded bash -> node -> a full gsd-tools cold
start at 5 seconds. On a bench running thirty thousand tests in parallel that
is not a hang, it is a busy machine. Raised to 30s, matching the convention
sibling suites already use for script invocations, with a comment saying what
the budget covers so nobody tightens it back. Two further copies of the same
5-second spawn in the same file had the identical defect and are raised too —
they were not in the failure report, but they will be next time.

The fragment-propagation test bounded npm run regen:derived — a full build plus
eight generators, the heaviest subprocess in the suite — at five minutes, and
node22 was killed near the end. The captured output proves it: every generator
had written its files and gen:install-tree had emitted all fifteen runtimes
before the kill. Raised to fifteen minutes.

That failure read as `null !== 0`, which says nothing. status null means killed,
not a non-zero exit, and the two want different responses: one is a timeout to
size correctly, the other is a real build break. The assertion now distinguishes
them and names the signal.

Neither test's assertions were weakened and no retry was added. A retry here
would suppress exactly the signal the timeout exists to produce.

Refs #3051

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#3057): capture fd 1 through the mock tracker, not a raw reassignment

The phase suite reported zero test results on both lanes while running for five
and a half minutes and exiting 1. No assertion text, no stderr, four events for
the whole file: enqueue, start, dequeue, complete. That shape is not a failing
assertion — it is the runner being unable to read the child at all, because it
parses its event stream from the child's stdout.

The cause was the capture helper reassigning fs.writeSync directly. Proven
rather than assumed: a standalone probe patched fs.writeSync and called
process.stdout.write, and the interception fired only when fd 1 resolved to a
FILE, not when it was a pipe. The remote runner captures the event stream to a
file, so a helper that was invisible against a pipe swallowed the reporter's own
output on the bench. That is also why the two sibling suites wired the same way
in this change pass cleanly — they use the mock tracker, the seam io.test.cjs
established for this exact function.

The helper now uses t.mock.method with an explicit restore after each call, so
teardown belongs to node:test rather than a second hand-rolled implementation,
and the interception cannot outlive the one synchronous call it wraps even if
that call throws. Ten call sites thread the test context through; three test
callbacks gained the parameter they lacked.

The three B3 tests are untouched — same assertions, same fault injection. Only
how the context reaches the helper changed.

Refs #3051

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(#3057): capture phase-complete output from a subprocess, not fd 1

Two attempts to make in-process fd-1 interception safe both failed on the
bench. The suite reported zero test results on either lane while exiting 1 —
four events for the whole file — because the runner parses its event stream
from the child's stdout, and process.stdout.write routes through fs.writeSync
whenever fd 1 resolves to a file, which is how the runner captures. Patching
that seam anywhere in a file can therefore destroy the file's own reporting,
and tightening the window only moved the runtime from 326s to 125s without
recovering a single event.

So the interception is gone rather than tuned. The helper now spawns gsd-tools
as a real subprocess and reads stdout the way the OS already gives it to us,
which is what the rest of the suite does. It asserts the command succeeded
before parsing, so a genuine failure can no longer present as a JSON parse
error.

The two fault-injecting tests could not survive that move as written: a
subprocess cannot see a mock installed in the parent. Instead of reinstating
the interception they now produce the fault on disk — the summary artifact is
created as a dangling symlink, so the staleness check's real statSync throws
inside the child. That is a more honest fixture than a mock in any case, since
it is a condition a user's tree can actually be in. Skipped on Windows, matching
the existing symlink precedent in the write-guard suite.

Three further call sites turned out to depend on parent-process writeFileSync
mocks the subprocess could not see. Those call the CJS function directly, which
is what they always wanted — they never needed stdout at all.

Refs #3051

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#3057): one name for one signal, one encoding for one distinction

Standards review found four things this branch introduced, all of them
inconsistencies with itself rather than with the repo.

One upstream bit reached its consumers under three names —
verification_stale_check_indeterminate in two modules, the same value with
"stale" dropped in a third, and stderr only in the fourth. Standardised on the
long name wherever it is a field. The workstream inventory keeps its stderr
channel, since its return shape has nowhere to hang a per-phase field without
rippling the builder's types, but it now says the same word for the same thing.

worktree-safety encoded one three-way distinction two ways in a single file: a
named union for a finding's kind, and boolean|null for an inventory entry's
existence. The second is now a named union too.

Two assertions matched human prose because the blocked and non-blocked
completion paths carried no typed field for the signal. Both now assert typed
values. The first round of this fix added the field but left the regex beside
it, which is the banned pattern sitting next to its own replacement; the second
removed it and added an assertion on the reason enum so nothing was lost.

The remaining two were reasoned away before being fixed, and both reasons were
bad. "No typed surface exists" is the condition CONTRIBUTING says to fix by
adding one — it took three lines. "The file already does this dozens of times"
is not licence to add instance number thirty-one; a convention that violates a
documented rule is debt, not precedent.

Vocabulary differing across DIFFERENT modules is left alone: CONTEXT.md rejects
a single shared result envelope, so per-module shapes are precedented, and a
baseline smell does not outrank a documented standard.

A census of every line this branch adds to a test file now finds no regex or
substring assertion on produced prose: 87 strictEqual, 25 ok (all non-empty or
shape guards), 12 equal, 3 throws (all typed err.code predicates), 3
deepStrictEqual, 2 notStrictEqual.

Refs #3051

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(#3057): backfill changeset pr number to 3088

---------

Co-authored-by: sim <sim@local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 16:00:52 -04:00

1383 lines
51 KiB
JavaScript

'use strict';
/**
* Unit tests for uat-predicate.cjs
* Tests the pure-computation module: stripFalsePositiveContexts,
* parseUatResultItems, evaluateUatPassed.
*
* Issue #247 — phase uat-passed predicate
*/
const { test, describe, beforeEach, afterEach } = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const os = require('node:os');
const fc = require('fast-check');
const {
stripFalsePositiveContexts,
parseUatResultItems,
analyzeMarkdown,
evaluateUatPassed,
} = require('../gsd-core/bin/lib/uat-predicate.cjs');
const { cleanup } = require('./helpers.cjs');
// ─── Helpers ──────────────────────────────────────────────────────────────────
function makeTmpDir() {
return fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-uat-pred-test-'));
}
function rmDir(dir) {
cleanup(dir);
}
function writeFile(dir, name, content) {
fs.writeFileSync(path.join(dir, name), content, 'utf-8');
}
function makePassingUat(n = 1) {
const tests = Array.from({ length: n }, (_, i) => [
`### ${i + 1}. Test ${i + 1}`,
`expected: It works`,
`result: passed`,
'',
].join('\n')).join('\n');
return `---\nstatus: passed\n---\n\n# UAT\n\n${tests}`;
}
// ─── stripFalsePositiveContexts ────────────────────────────────────────────────
describe('stripFalsePositiveContexts — frontmatter', () => {
test('removes leading frontmatter block', () => {
const input = '---\nstatus: pending\nresult: pending\n---\n\nReal content here.';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: pending'), 'frontmatter result: pending should be stripped');
assert.ok(out.includes('Real content here.'), 'body content must be preserved');
});
test('does not strip non-frontmatter --- dividers later in document', () => {
const input = '---\nstatus: ok\n---\n\n# Section\n\n---\n\nMore content.';
const out = stripFalsePositiveContexts(input);
assert.ok(out.includes('More content.'), 'content after a non-frontmatter divider must survive');
});
test('handles CRLF frontmatter', () => {
const input = '---\r\nstatus: partial\r\nresult: pending\r\n---\r\n\r\nBody text.';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: pending'), 'CRLF frontmatter must be stripped');
assert.ok(out.includes('Body text.'), 'body after CRLF frontmatter must survive');
});
});
describe('stripFalsePositiveContexts — HTML comments', () => {
test('removes single-line HTML comment', () => {
const input = 'Before\n<!-- result: pending -->\nAfter';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: pending'), 'HTML comment content must be stripped');
assert.ok(out.includes('Before'), 'content before comment must survive');
assert.ok(out.includes('After'), 'content after comment must survive');
});
test('removes multi-line HTML comment', () => {
const input = 'A\n<!--\n### 1. Test\nexpected: Foo\nresult: pending\n-->\nB';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: pending'), 'multi-line HTML comment content must be stripped');
assert.ok(out.includes('A'), 'content before comment must survive');
assert.ok(out.includes('B'), 'content after comment must survive');
});
test('unterminated HTML comment swallows to EOF (fail-closed)', () => {
const input = 'Before\n<!--\nresult: passed\nstill in comment';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: passed'), 'unterminated comment must swallow to EOF');
assert.ok(out.includes('Before'), 'content before comment must survive');
});
});
describe('stripFalsePositiveContexts — fenced code blocks', () => {
test('removes backtick fence with result inside', () => {
const input = 'Before\n```\n### 1. Test\nexpected: X\nresult: pending\n```\nAfter';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: pending'), 'content inside ``` fence must be stripped');
assert.ok(out.includes('Before'), 'content before fence must survive');
assert.ok(out.includes('After'), 'content after fence must survive');
});
test('removes tilde fence', () => {
const input = 'Before\n~~~\n### 1. Test\nexpected: X\nresult: blocked\n~~~\nAfter';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: blocked'), 'content inside ~~~ fence must be stripped');
assert.ok(out.includes('After'), 'content after tilde fence must survive');
});
test('handles indented fence', () => {
const input = 'Before\n ```\n### 1. Test\nresult: pending\n ```\nAfter';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: pending'), 'content inside indented fence must be stripped');
});
test('handles CRLF in fenced block', () => {
const input = 'Before\r\n```\r\nresult: pending\r\n```\r\nAfter';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: pending'), 'CRLF fenced content must be stripped');
});
test('preserves content after multiple fenced blocks', () => {
const input = [
'Real intro',
'```',
'result: pending',
'```',
'',
'### 1. Real Test',
'expected: Works',
'result: passed',
'',
'```',
'result: pending',
'```',
'Trailing',
].join('\n');
const out = stripFalsePositiveContexts(input);
assert.ok(out.includes('result: passed'), 'real result outside fence must survive');
assert.ok(!out.includes('result: pending'), 'fenced result: pending must be stripped');
});
});
describe('stripFalsePositiveContexts — blockquotes', () => {
test('removes blockquote lines', () => {
const input = 'Before\n> ### 1. Test\n> result: pending\nAfter';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: pending'), 'blockquote content must be stripped');
assert.ok(out.includes('Before'), 'content before blockquote must survive');
assert.ok(out.includes('After'), 'content after blockquote must survive');
});
test('removes indented blockquote lines', () => {
const input = 'Before\n > result: pending\nAfter';
const out = stripFalsePositiveContexts(input);
assert.ok(!out.includes('result: pending'), 'indented blockquote content must be stripped');
});
});
// ─── parseUatResultItems ───────────────────────────────────────────────────────
describe('parseUatResultItems', () => {
test('parses a single passing test', () => {
const content = '### 1. Login flow\nexpected: User logs in\nresult: passed\n';
const items = parseUatResultItems(content);
assert.strictEqual(items.length, 1);
assert.strictEqual(items[0].test, 1);
assert.strictEqual(items[0].name, 'Login flow');
assert.strictEqual(items[0].result, 'passed');
});
test('parses bracketed result [passed] (#2273)', () => {
const content = '### 1. Login flow\nexpected: User logs in\nresult: [passed]\n';
const items = parseUatResultItems(content);
assert.strictEqual(items.length, 1);
assert.strictEqual(items[0].result, 'passed');
});
test('parses bracketed result [pass]', () => {
const content = '### 1. Login flow\nexpected: User logs in\nresult: [pass]\n';
const items = parseUatResultItems(content);
assert.strictEqual(items.length, 1);
assert.strictEqual(items[0].result, 'pass');
});
test('parses multiple tests with mixed results', () => {
const content = [
'### 1. Test A',
'expected: A',
'result: passed',
'',
'### 2. Test B',
'expected: B',
'result: pending',
'',
'### 3. Test C',
'expected: C',
'result: failed',
'',
].join('\n');
const items = parseUatResultItems(content);
assert.strictEqual(items.length, 3);
assert.strictEqual(items[0].result, 'passed');
assert.strictEqual(items[1].result, 'pending');
assert.strictEqual(items[2].result, 'failed');
});
test('returns empty array for content with no test blocks', () => {
const items = parseUatResultItems('Just some markdown with no test blocks.');
assert.strictEqual(items.length, 0);
});
test('lowercases result value', () => {
const content = '### 1. Test\nexpected: Foo\nresult: PASSED\n';
const items = parseUatResultItems(content);
assert.strictEqual(items[0].result, 'passed');
});
test('heading with NO result line → result:missing (blocker state, not dropped)', () => {
const content = [
'### 1. Test without result',
'expected: Something',
'some notes here',
'',
].join('\n');
const items = parseUatResultItems(content);
assert.strictEqual(items.length, 1);
assert.strictEqual(items[0].result, 'missing');
});
test('result: line with leading whitespace (indented scalar) is NOT parsed as column-0', () => {
// A real failing test uses block-scalar expected: with blank line before result
// The result: with leading whitespace should not be treated as a column-0 result
const content = [
'### 1. Test A',
'expected: |',
' multi',
' line',
'',
' result: pending',
'',
].join('\n');
// The indented result: should not match; heading has no column-0 result → 'missing'
const items = parseUatResultItems(content);
assert.strictEqual(items.length, 1);
assert.strictEqual(items[0].result, 'missing',
'Indented result: line must not be parsed as column-0 result');
});
test('block-scalar expected: with blank line then column-0 result: is parsed correctly', () => {
// Real failing test: block-scalar expected: | followed by blank + real result: pending
const content = [
'### 1. Test A',
'expected: |',
' multi',
' line',
'',
'result: pending',
'',
].join('\n');
const items = parseUatResultItems(content);
assert.strictEqual(items.length, 1);
assert.strictEqual(items[0].result, 'pending',
'Column-0 result: after blank line must be parsed correctly');
});
});
// ─── evaluateUatPassed — boundary coverage ────────────────────────────────────
describe('evaluateUatPassed — boundary coverage', () => {
let tmpDir;
beforeEach(() => {
tmpDir = makeTmpDir();
});
afterEach(() => {
rmDir(tmpDir);
});
test('0 UAT files → passed:false, no_uat_artifacts:true (fail-closed, no vacuous pass)', () => {
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
'No UAT files must NOT vacuously pass — fail-closed');
assert.strictEqual(report.no_uat_artifacts, true);
assert.deepStrictEqual(report.uat_files, []);
assert.deepStrictEqual(report.blockers, []);
});
test('1 passing test → passed:true, no_uat_artifacts:false', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, true);
assert.strictEqual(report.no_uat_artifacts, false);
assert.strictEqual(report.blockers.length, 0);
assert.strictEqual(report.checks.length, 1);
assert.strictEqual(report.checks[0].passing, true);
});
test('N all-passing tests → passed:true', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(5));
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, true);
assert.strictEqual(report.checks.every(c => c.passing), true);
});
test('N-1 passing + 1 pending → passed:false', () => {
// 3 tests total: tests 1,2 passed; test 3 pending
const content = [
'---', 'status: partial', '---', '',
'### 1. Test A', 'expected: A', 'result: passed', '',
'### 2. Test B', 'expected: B', 'result: passed', '',
'### 3. Test C', 'expected: C', 'result: pending', '',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.ok(report.blockers.some(b => /test 3/i.test(b) || /pending/i.test(b)),
`Expected blocker for pending test, got: ${JSON.stringify(report.blockers)}`);
});
test('1 blocked test → passed:false', () => {
const content = [
'---', 'status: partial', '---', '',
'### 1. Test A', 'expected: A', 'result: blocked', '',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
});
test('1 skipped test → passed:false', () => {
const content = [
'---', 'status: partial', '---', '',
'### 1. Test A', 'expected: A', 'result: skipped', '',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
});
test('1 failed test → passed:false', () => {
const content = [
'---', 'status: partial', '---', '',
'### 1. Test A', 'expected: A', 'result: failed', '',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
});
test('1 human_needed test → passed:false', () => {
const content = [
'---', 'status: partial', '---', '',
'### 1. Test A', 'expected: A', 'result: human_needed', '',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
});
test('heading with no result line → result:missing → blocker → passed:false', () => {
const content = [
'### 1. Test without result',
'expected: It should work',
'some notes but no result',
'',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.ok(report.checks.some(c => c.result === 'missing' && !c.passing),
`Expected missing check item, got checks: ${JSON.stringify(report.checks)}`);
assert.ok(report.blockers.some(b => /missing/i.test(b)),
`Expected missing blocker, got: ${JSON.stringify(report.blockers)}`);
});
});
// ─── evaluateUatPassed — frontmatter status checks ────────────────────────────
describe('evaluateUatPassed — frontmatter status', () => {
let tmpDir;
beforeEach(() => {
tmpDir = makeTmpDir();
});
afterEach(() => {
rmDir(tmpDir);
});
const failingUatStatuses = ['partial', 'diagnosed', 'pending', 'blocked', 'in_progress', 'failed'];
const failingUatResults = ['pending', 'blocked', 'failed'];
for (const status of failingUatStatuses) {
test(`UAT frontmatter status=${status} → passed:false`, () => {
const content = [
'---', `status: ${status}`, '---', '',
'### 1. Test A', 'expected: A', 'result: passed', '',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
`status=${status} should cause failure`);
assert.ok(report.blockers.some(b => b.includes(status)),
`Blocker should mention status=${status}, got: ${JSON.stringify(report.blockers)}`);
});
}
for (const result of failingUatResults) {
test(`UAT frontmatter result=${result} → passed:false`, () => {
const content = [
'---', `result: ${result}`, '---', '',
'### 1. Test A', 'expected: A', 'result: passed', '',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
`fm result=${result} should cause failure`);
});
}
});
// ─── evaluateUatPassed — VERIFICATION file checks ─────────────────────────────
describe('evaluateUatPassed — VERIFICATION files', () => {
let tmpDir;
beforeEach(() => {
tmpDir = makeTmpDir();
});
afterEach(() => {
rmDir(tmpDir);
});
test('VERIFICATION with status human_needed → passed:false', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_needed\n---\n\nNeeds human check.');
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.ok(report.blockers.some(b => /human_needed/i.test(b)),
`Expected human_needed blocker, got: ${JSON.stringify(report.blockers)}`);
});
test('VERIFICATION with status gaps_found → passed:false', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: gaps_found\n---\n\nHas gaps.');
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.ok(report.blockers.some(b => /gaps_found/i.test(b)),
`Expected gaps_found blocker, got: ${JSON.stringify(report.blockers)}`);
});
test('VERIFICATION with status pending → passed:false', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: pending\n---\n');
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
});
test('VERIFICATION with status failed → passed:false (blocking)', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: failed\n---\n');
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.ok(report.blockers.some(b => /failed/i.test(b)),
`Expected failed blocker, got: ${JSON.stringify(report.blockers)}`);
});
test('VERIFICATION with status in_progress → passed:false (blocking)', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: in_progress\n---\n');
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.ok(report.blockers.some(b => /in_progress/i.test(b)),
`Expected in_progress blocker, got: ${JSON.stringify(report.blockers)}`);
});
test('VERIFICATION with missing status does NOT satisfy --require-verification', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '# Verification\nNo frontmatter status.\n');
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
assert.strictEqual(report.passed, false,
'Missing verification status must not satisfy requireVerification');
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
});
test('VERIFICATION with unknown status does NOT satisfy --require-verification', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: some_unknown_status\n---\n');
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
assert.strictEqual(report.passed, false,
'Unknown verification status must not satisfy requireVerification');
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
});
test('VERIFICATION with status passed → does not block', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n\nAll good.');
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, true);
assert.strictEqual(report.verification_files.length, 1);
});
test('VERIFICATION with status complete → does not block', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: complete\n---\n\nAll good.');
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, true);
});
test('VERIFICATION with status verified → does not block', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: verified\n---\n\nAll good.');
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, true);
});
test('VERIFICATION with status human_passed → does not block', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_passed\n---\n\nAll good.');
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, true);
});
test('VERIFICATION status complete does NOT satisfy --require-verification', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: complete\n---\n\nAll good.');
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
assert.strictEqual(report.passed, false);
assert.strictEqual(report.policy.require_verification, true);
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
});
test('VERIFICATION status verified does NOT satisfy --require-verification', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: verified\n---\n\nAll good.');
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
assert.strictEqual(report.passed, false);
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
});
test('VERIFICATION status human_passed does NOT satisfy --require-verification', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_passed\n---\n\nAll good.');
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
assert.strictEqual(report.passed, false);
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
});
});
// ─── evaluateUatPassed — policy.requireVerification ───────────────────────────
describe('evaluateUatPassed — policy.requireVerification', () => {
let tmpDir;
beforeEach(() => {
tmpDir = makeTmpDir();
});
afterEach(() => {
rmDir(tmpDir);
});
test('requireVerification=true with no verification file → passed:false', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
assert.strictEqual(report.passed, false);
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
assert.strictEqual(report.policy.require_verification, true);
});
test('requireVerification=true with passing verification → passed:true', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n\nOK.');
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
assert.strictEqual(report.passed, true);
assert.strictEqual(report.policy.require_verification, true);
});
test('requireVerification=false (default) with no verification file → passed:true', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: false } });
assert.strictEqual(report.passed, true);
assert.strictEqual(report.policy.require_verification, false);
});
});
// ─── evaluateUatPassed — #3057 B3: staleness-check indeterminate is surfaced ──
//
// readVerificationStatus's internal staleness check can fail (fs /
// scanPhasePlans / clock error). Pre-#3057 B3 wiring, the requireVerification
// policy check used only `.status`, dropping `.staleCheckIndeterminate` on
// the floor — so cmdPhaseUatPassed's JSON output (which spreads this whole
// report) could never distinguish "checked; nothing is stale" from "could
// not check". `verification_stale_check_indeterminate` must never itself gate
// `passed`/`blockers` — only readVerificationStatus's `.status` may.
describe('#3057 B3: evaluateUatPassed — verification staleness-check indeterminate is surfaced', () => {
let tmpDir;
beforeEach(() => {
tmpDir = makeTmpDir();
});
afterEach(() => {
rmDir(tmpDir);
});
function seedVerifiedPhase() {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n\nOK.');
writeFile(tmpDir, 'phase-SUMMARY.md', '# Summary');
const verificationPath = path.join(tmpDir, 'phase-VERIFICATION.md');
const summaryPath = path.join(tmpDir, 'phase-SUMMARY.md');
// Deterministic mtime ordering — never rely on write-order clock ties.
const older = new Date('2026-01-01T00:00:00.000Z');
const newer = new Date('2026-01-01T00:01:00.000Z');
fs.utimesSync(summaryPath, older, older);
fs.utimesSync(verificationPath, newer, newer);
return { summaryPath, verificationPath };
}
test('an fs failure inside the staleness check sets verification_stale_check_indeterminate:true; passed/blockers unchanged', (t) => {
const { summaryPath, verificationPath } = seedVerifiedPhase();
const origStatSync = fs.statSync;
t.mock.method(fs, 'statSync', function injectedStaleCheckFault(target, ...args) {
const targetPath = String(target);
if (targetPath === verificationPath || targetPath === summaryPath) {
throw new Error('injected stat failure (#3057 B3)');
}
return origStatSync.call(fs, target, ...args);
});
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
// Pre-existing no-throw fail-open routing is UNCHANGED — `passed`/
// `blockers` are exactly what they would be without the injected fault.
assert.strictEqual(report.passed, true);
assert.deepStrictEqual(report.blockers, []);
assert.strictEqual(report.verification_stale_check_indeterminate, true);
});
test('a completed staleness check that finds nothing stale reports verification_stale_check_indeterminate:false', () => {
seedVerifiedPhase();
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
assert.strictEqual(report.passed, true);
assert.strictEqual(report.verification_stale_check_indeterminate, false);
});
test('requireVerification not set → verification_stale_check_indeterminate is always false (readVerificationStatus never reached)', () => {
seedVerifiedPhase();
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: false } });
assert.strictEqual(report.verification_stale_check_indeterminate, false);
});
});
// ─── evaluateUatPassed — malformed markdown guard ─────────────────────────────
describe('evaluateUatPassed — malformed markdown blocker', () => {
let tmpDir;
beforeEach(() => {
tmpDir = makeTmpDir();
});
afterEach(() => {
rmDir(tmpDir);
});
test('unterminated code fence → malformed blocker → passed:false (even with real result:passed)', () => {
const content = [
'### 1. Real Test',
'expected: Something',
'result: passed',
'',
'```',
'unterminated fence — no closing delimiter',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
'Unterminated fence must always block, even if a result:passed exists');
assert.ok(report.blockers.some(b => /malformed/i.test(b)),
`Expected malformed blocker, got: ${JSON.stringify(report.blockers)}`);
});
test('unterminated HTML comment → malformed blocker → passed:false', () => {
const content = [
'### 1. Real Test',
'expected: Something',
'result: passed',
'',
'<!-- unterminated comment',
'no closing arrow',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
'Unterminated comment must always block');
assert.ok(report.blockers.some(b => /malformed/i.test(b)),
`Expected malformed blocker, got: ${JSON.stringify(report.blockers)}`);
});
test('well-formed fences → no malformed blocker', () => {
const content = [
'### 1. Real Test',
'expected: Something',
'result: passed',
'',
'```',
'code here',
'```',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.ok(!report.blockers.some(b => /malformed/i.test(b)),
'Well-formed fences must not produce malformed blocker');
});
});
// ─── evaluateUatPassed — hardening regressions (the heart of #247) ────────────
describe('evaluateUatPassed — false-positive hardening regressions', () => {
let tmpDir;
beforeEach(() => {
tmpDir = makeTmpDir();
});
afterEach(() => {
rmDir(tmpDir);
});
test('#247: result:passed inside fenced block: parseUatResultItems returns [] for fake, evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
// ONLY result: passed is inside a fenced block — no real test blocks
const rawContent = [
'```',
'### 1. Fake Test',
'expected: Example output',
'result: passed',
'```',
].join('\n');
// After stripping, the clean content has no headings at all
const clean = stripFalsePositiveContexts(rawContent);
const items = parseUatResultItems(clean);
assert.strictEqual(items.length, 0, 'Fake result inside fence must produce no items');
writeFile(tmpDir, 'phase-UAT.md', rawContent);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
'result:passed inside a fenced block must not flip passed to true');
assert.strictEqual(report.no_uat_artifacts, true,
'no real UAT items → no_uat_artifacts:true');
});
test('#247: result:passed inside blockquote: parseUatResultItems returns [] + evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
const rawContent = [
'> ### 1. Test',
'> expected: Example',
'> result: passed',
].join('\n');
const clean = stripFalsePositiveContexts(rawContent);
const items = parseUatResultItems(clean);
assert.strictEqual(items.length, 0, 'Fake result inside blockquote must produce no items');
writeFile(tmpDir, 'phase-UAT.md', rawContent);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.strictEqual(report.no_uat_artifacts, true);
});
test('#247: result:passed inside HTML comment: parseUatResultItems returns [] + evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
const rawContent = [
'<!-- ### 1. Test',
'expected: Example',
'result: passed -->',
].join('\n');
const clean = stripFalsePositiveContexts(rawContent);
const items = parseUatResultItems(clean);
assert.strictEqual(items.length, 0, 'Fake result inside HTML comment must produce no items');
writeFile(tmpDir, 'phase-UAT.md', rawContent);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.strictEqual(report.no_uat_artifacts, true);
});
test('#247: result:passed inside frontmatter: parseUatResultItems returns [] + evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
const rawContent = [
'---',
'example_result: passed',
'---',
'',
'No real test blocks here.',
].join('\n');
const clean = stripFalsePositiveContexts(rawContent);
const items = parseUatResultItems(clean);
assert.strictEqual(items.length, 0, 'Fake result inside frontmatter must produce no items');
writeFile(tmpDir, 'phase-UAT.md', rawContent);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.strictEqual(report.no_uat_artifacts, true);
});
test('#247: result:passed inside fenced block is NOT treated as passing test (with real failing test)', () => {
const content = [
'---',
'status: partial',
'---',
'',
'# Example',
'',
'```',
'### 1. Test',
'expected: Example output',
'result: passed',
'```',
'',
'### 1. Real Test',
'expected: Something',
'result: pending',
'',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
'result:passed inside a fenced block must not flip passed to true');
assert.ok(report.checks.some(c => c.result === 'pending' && !c.passing),
'Real pending test must be captured');
});
test('#247: result:passed inside blockquote is NOT treated as passing test', () => {
const content = [
'---',
'status: partial',
'---',
'',
'> ### 1. Test',
'> expected: Example',
'> result: passed',
'',
'### 1. Real Test',
'expected: Something',
'result: pending',
'',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
'result:passed inside a blockquote must not flip passed to true');
});
test('#247: result:passed inside HTML comment is NOT treated as passing test', () => {
const content = [
'---',
'status: partial',
'---',
'',
'<!-- ### 1. Test',
'expected: Example',
'result: passed -->',
'',
'### 1. Real Test',
'expected: Something',
'result: pending',
'',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
'result:passed inside an HTML comment must not flip passed to true');
});
test('#247: result:passed inside frontmatter example is NOT treated as passing test', () => {
const content = [
'---',
'status: partial',
'example_result: passed',
'---',
'',
'### 1. Real Test',
'expected: Something',
'result: pending',
'',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
'result:passed in frontmatter must not flip passed to true');
});
test('#247: real passing test block outside all contexts → passed:true', () => {
const content = [
'---',
'status: passed',
'---',
'',
'# UAT Results',
'',
'### 1. Login works',
'expected: User logs in',
'result: passed',
'',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, true,
'A real passing test outside false-positive contexts must pass');
});
test('block-scalar expected: followed by blank line + result: pending → parsed as blocker (not dropped)', () => {
const content = [
'### 1. Test A',
'expected: |',
' multi',
' line expected output',
'',
'result: pending',
'',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
'Block-scalar expected: with result: pending must be captured as a blocker');
assert.ok(report.checks.some(c => c.result === 'pending' && !c.passing),
`Expected pending check, got: ${JSON.stringify(report.checks)}`);
});
});
// ─── evaluateUatPassed — output shape contract ───────────────────────────────
describe('evaluateUatPassed — output shape (Hyrum\'s Law contract)', () => {
let tmpDir;
beforeEach(() => {
tmpDir = makeTmpDir();
});
afterEach(() => {
rmDir(tmpDir);
});
test('returns all required fields in the locked shape including no_uat_artifacts', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
const report = evaluateUatPassed(tmpDir);
// Locked field names
assert.ok('passed' in report, 'report.passed must exist');
assert.ok('uat_files' in report, 'report.uat_files must exist');
assert.ok('verification_files' in report, 'report.verification_files must exist');
assert.ok('checks' in report, 'report.checks must exist');
assert.ok('blockers' in report, 'report.blockers must exist');
assert.ok('no_uat_artifacts' in report, 'report.no_uat_artifacts must exist');
assert.ok('policy' in report, 'report.policy must exist');
assert.ok('require_verification' in report.policy, 'report.policy.require_verification must exist');
// checks item shape
if (report.checks.length > 0) {
const c = report.checks[0];
assert.ok('file' in c, 'check.file must exist');
assert.ok('test' in c, 'check.test must exist');
assert.ok('name' in c, 'check.name must exist');
assert.ok('result' in c, 'check.result must exist');
assert.ok('passing' in c, 'check.passing must exist');
}
});
test('no_uat_artifacts is false when real checks exist', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.no_uat_artifacts, false);
});
test('no_uat_artifacts is true when no checks exist', () => {
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.no_uat_artifacts, true);
});
test('uat_files contains the filename', () => {
writeFile(tmpDir, 'my-UAT.md', makePassingUat(1));
const report = evaluateUatPassed(tmpDir);
assert.ok(report.uat_files.includes('my-UAT.md'),
`uat_files should include 'my-UAT.md', got: ${JSON.stringify(report.uat_files)}`);
});
test('verification_files contains the filename', () => {
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n');
const report = evaluateUatPassed(tmpDir);
assert.ok(report.verification_files.includes('phase-VERIFICATION.md'),
`verification_files should include 'phase-VERIFICATION.md', got: ${JSON.stringify(report.verification_files)}`);
});
});
// ─── FIX A regression: nested-fence (~~~ inside ```) ─────────────────────────
describe('FIX A — nested fence: ~~~ inside ``` does not prematurely close outer fence', () => {
let tmpDir;
beforeEach(() => { tmpDir = makeTmpDir(); });
afterEach(() => { rmDir(tmpDir); });
test('parseUatResultItems sees [] for fake inside ``` that encloses ~~~', () => {
// A backtick fence that contains an inner ~~~ fence with a fake test block.
// The ~~~ must NOT close the ``` fence — the whole interior is content and is dropped.
const raw = [
'```',
'~~~',
'### 1. Fake',
'expected: X',
'result: passed',
'~~~',
'```',
].join('\n');
const clean = stripFalsePositiveContexts(raw);
const items = parseUatResultItems(clean);
assert.strictEqual(items.length, 0, 'fake inside nested fence must not leak through');
});
test('evaluateUatPassed → passed:false + no_uat_artifacts:true for nested-fence-only file', () => {
const raw = [
'```',
'~~~',
'### 1. Fake',
'expected: X',
'result: passed',
'~~~',
'```',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', raw);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false, 'nested-fence fake must not flip passed');
assert.strictEqual(report.no_uat_artifacts, true, 'no real items → no_uat_artifacts:true');
assert.ok(!report.checks.some(c => c.name === 'Fake'), 'fake must not appear in checks');
});
test('balanced nested fence (``` inside ~~~) is NOT flagged as malformed', () => {
// ~~~ outer, ``` inner — properly closed — should not trigger unterminatedFence
const raw = [
'~~~',
'```',
'code',
'```',
'~~~',
'',
'### 1. Real Test',
'expected: Works',
'result: passed',
].join('\n');
const { unterminatedFence } = analyzeMarkdown(raw);
assert.strictEqual(unterminatedFence, false, 'balanced nested fence must not be flagged');
const clean = stripFalsePositiveContexts(raw);
const items = parseUatResultItems(clean);
assert.strictEqual(items.length, 1, 'real test outside fence must still be found');
assert.strictEqual(items[0].result, 'passed');
});
test('real result:passed outside ``` that encloses ~~~ → passed:true (no false-block)', () => {
const raw = [
'---',
'status: passed',
'---',
'',
'```',
'~~~',
'### 1. Fake',
'result: passed',
'~~~',
'```',
'',
'### 1. Real Test',
'expected: Works',
'result: passed',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', raw);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, true, 'real result outside nested fence must still pass');
assert.ok(!report.checks.some(c => c.name === 'Fake'), 'fake must not appear in checks');
});
});
// ─── FIX B regression: cross-line result value ────────────────────────────────
describe('FIX B — cross-line result: value must be on the same line', () => {
test('result: with value on next line → result:missing (not passed)', () => {
const content = [
'### 1. Cross-line Test',
'expected: Y',
'result:',
'',
'passed',
].join('\n');
const items = parseUatResultItems(content);
assert.strictEqual(items.length, 1);
assert.notStrictEqual(items[0].result, 'passed',
'result value on a subsequent line must not be captured as passed');
assert.strictEqual(items[0].result, 'missing',
'cross-line result must yield missing (blocker)');
});
test('evaluateUatPassed → passed:false for cross-line result:passed', () => {
const tmpDir = makeTmpDir();
try {
const content = [
'### 1. Cross-line Test',
'expected: Y',
'result:',
'',
'passed',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.ok(!report.checks.some(c => c.result === 'passed'),
'cross-line result must not produce a passing check');
} finally {
rmDir(tmpDir);
}
});
});
// ─── FIX C regression: masked-comment (earlier closed comment + later unterminated) ──
describe('FIX C — dangling comment survives earlier balanced comment', () => {
test('analyzeMarkdown detects unterminated comment after a properly closed one', () => {
const raw = [
'<!-- ok -->',
'Some text',
'<!--',
'### 1. Fake',
'result: passed',
].join('\n');
const { unterminatedComment } = analyzeMarkdown(raw);
assert.strictEqual(unterminatedComment, true,
'later unterminated comment must be detected even after a balanced one');
});
test('evaluateUatPassed → passed:false (malformed) when later comment is unterminated', () => {
const tmpDir = makeTmpDir();
try {
const raw = [
'### 1. Real Test',
'expected: Something',
'result: passed',
'',
'<!-- properly closed -->',
'',
'<!--',
'### 2. Fake',
'result: passed',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', raw);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false,
'unterminated later comment must trigger malformed blocker');
assert.ok(report.blockers.some(b => /malformed/i.test(b)),
`Expected malformed blocker, got: ${JSON.stringify(report.blockers)}`);
} finally {
rmDir(tmpDir);
}
});
test('analyzeMarkdown does NOT flag a file with only properly balanced comments', () => {
const raw = [
'<!-- first comment -->',
'Some text',
'<!-- second comment -->',
'',
'### 1. Real Test',
'result: passed',
].join('\n');
const { unterminatedComment } = analyzeMarkdown(raw);
assert.strictEqual(unterminatedComment, false,
'only balanced comments must not be flagged as unterminated');
});
});
// ─── FIX D regression: balanced mixed fences are NOT flagged ─────────────────
describe('FIX D — odd-fence-count heuristic replaced: balanced mixed fences not flagged', () => {
test('analyzeMarkdown: ``` followed by ~~~ (both balanced) → unterminatedFence:false', () => {
const raw = [
'```',
'code',
'```',
'',
'~~~',
'more code',
'~~~',
].join('\n');
const { unterminatedFence } = analyzeMarkdown(raw);
assert.strictEqual(unterminatedFence, false,
'two separate balanced fences must not trigger unterminatedFence');
});
test('evaluateUatPassed: multiple balanced fences + real passing test → passed:true, no malformed blocker', () => {
const tmpDir = makeTmpDir();
try {
const raw = [
'---',
'status: passed',
'---',
'',
'```',
'result: fake',
'```',
'',
'~~~',
'result: also fake',
'~~~',
'',
'### 1. Real Test',
'expected: Works',
'result: passed',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', raw);
const report = evaluateUatPassed(tmpDir);
assert.ok(!report.blockers.some(b => /malformed/i.test(b)),
`Balanced mixed fences must not produce malformed blocker, got: ${JSON.stringify(report.blockers)}`);
assert.strictEqual(report.passed, true,
'real test outside balanced fences must still pass');
} finally {
rmDir(tmpDir);
}
});
});
// ─── FIX E extra: fake-not-in-checks assertions for existing mixed tests ──────
describe('FIX E — fake items must NOT appear in checks (absence assertions)', () => {
let tmpDir;
beforeEach(() => { tmpDir = makeTmpDir(); });
afterEach(() => { rmDir(tmpDir); });
test('fake inside fence + real pending: fake NOT in checks', () => {
const content = [
'---', 'status: partial', '---', '',
'```',
'### 10. Fake',
'expected: Fake',
'result: passed',
'```',
'',
'### 1. Real Test',
'expected: Something',
'result: pending',
'',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.ok(!report.checks.some(c => c.name === 'Fake'),
'fake test inside fence must not appear in checks');
});
test('fake inside blockquote + real pending: fake NOT in checks', () => {
const content = [
'---', 'status: partial', '---', '',
'> ### 10. Fake',
'> expected: Fake',
'> result: passed',
'',
'### 1. Real Test',
'expected: Something',
'result: pending',
'',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.ok(!report.checks.some(c => c.name === 'Fake'),
'fake test inside blockquote must not appear in checks');
});
test('fake inside HTML comment + real pending: fake NOT in checks', () => {
const content = [
'---', 'status: partial', '---', '',
'<!-- ### 10. Fake',
'expected: Fake',
'result: passed -->',
'',
'### 1. Real Test',
'expected: Something',
'result: pending',
'',
].join('\n');
writeFile(tmpDir, 'phase-UAT.md', content);
const report = evaluateUatPassed(tmpDir);
assert.strictEqual(report.passed, false);
assert.ok(!report.checks.some(c => c.name === 'Fake'),
'fake test inside HTML comment must not appear in checks');
});
});
// ─── Property-based test (fast-check) ─────────────────────────────────────────
describe('evaluateUatPassed — property: wrapping in false-positive context never flips to passed', () => {
let tmpDir;
beforeEach(() => {
tmpDir = makeTmpDir();
});
afterEach(() => {
rmDir(tmpDir);
});
test('fc: inserting result:passed inside wrapper context never flips a failing UAT to passed', () => {
// A baseline UAT file that has a pending item — it must always evaluate to passed:false
// regardless of how many "result: passed" lines we inject inside fenced/blockquote/comment wrappers.
const baseFailingBody = [
'### 1. Real Test',
'expected: It works',
'result: pending',
'',
].join('\n');
const wrappers = fc.constantFrom(
// backtick fence
(inner) => '```\n' + inner + '\n```',
// tilde fence
(inner) => '~~~\n' + inner + '\n~~~',
// HTML comment
(inner) => '<!--\n' + inner + '\n-->',
// blockquote — prefix each line
(inner) => inner.split('\n').map(l => '> ' + l).join('\n'),
);
fc.assert(
fc.property(wrappers, fc.nat(3), (wrap, extraCount) => {
// Build a "fake passing block" that would fool a naive regex
const fakePassingLines = Array.from({ length: extraCount + 1 }, (_, i) =>
`### ${i + 10}. Fake Test ${i + 10}\nexpected: Fake\nresult: passed`
).join('\n');
const fullContent = [
'---',
'status: partial',
'---',
'',
wrap(fakePassingLines),
'',
baseFailingBody,
].join('\n');
// Write to a unique tmp file to avoid cross-test state
const fcDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-fc-uat-'));
try {
fs.writeFileSync(path.join(fcDir, 'feature-UAT.md'), fullContent, 'utf-8');
const report = evaluateUatPassed(fcDir);
// The pending item must always keep passed:false
// AND fake items injected via wrappers must never appear in checks
const hasFakeInChecks = report.checks.some(c => c.name.startsWith('Fake Test'));
return report.passed === false && !hasFakeInChecks;
} finally {
cleanup(fcDir);
}
}),
{ numRuns: 50 }
);
});
});