* fix(#3057): refuse the write when the duplicate scan cannot complete writeManifest documents itself as a fail-closed duplicate guard: if any existing manifest shares plan_id with a different, non-terminal job_id it must refuse, because dispatching again would duplicate the external job. It could not honour that. The scan reads every sibling manifest looking for the duplicate, and an unreadable or unparseable sibling was `continue`d past. If the corrupt file was the one holding the live duplicate, the scan found nothing and a duplicate external job dispatched. The asymmetry is what gives it away: a malformed TARGET refused with malformed_existing because clobbering is unacceptable, while a malformed SIBLING was skipped — yet siblings are the only thing the duplicate check reads. Adds a scan_incomplete verdict that refuses and names the offending file, so an operator can quarantine or repair it. Fail-closed alone would let one stale corrupt manifest wedge every dispatch for that planning dir permanently; naming the file is what makes refusing survivable. malformed_existing is untouched, so the target/sibling distinction stays visible. The docstring is updated — it previously stated a rule the function did not keep. memFs() gains an optional failReads map so these branches are reachable at all; they had zero coverage because the fake could not express a per-file read fault. The signature is additive and every existing caller is unchanged. The regression is proved by a pair, not a single test. A control writes a readable sibling holding a genuine non-terminal duplicate and asserts duplicate_plan_id, establishing the scenario is real; the regression then makes that same path unreadable and asserts scan_incomplete. A first draft of this test used a corrupt-JSON fixture containing no plan_id at all while its comment claimed otherwise — it duplicated the unparseable-sibling case and proved nothing, which is the defect class this phase exists to remove. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#3057): make a guard's failure distinguishable from its benign result Wave 1 of the negative-space backfill: the branches where a guard that could not verify something reported the same value it reports when everything is fine. That indistinguishability is the defect; every fix here makes the two states tellable apart, and every test proves it with a pair — one for the failure, one for the benign case. A single test cannot establish that two states are distinguishable, which is the whole property being fixed. state.cts phaseInventoryProvider returned null for both a real disk-scan failure and a genuinely empty phases dir, so `state rebuild` could report success while phase-table reconciliation never ran. It now returns a discriminated result and the CLI surfaces phase_inventory_scan_failed plus a reason. The reason field turned out never to have been wired into the emitted JSON at all — it existed only as an internal variable — so a test could only assert on the operator-facing note. It is a real field now. state.cts treated an unreadable lock body the same as an empty one, applying the 1-second stealable floor. A lock we cannot read is not a lock we know is stale; an unreadable body is now held to the deadman ceiling like a live holder. verification.cts findStaleVerificationSummary returned null on any fs, scan or clock failure — meaning "not stale". It now returns a discriminated StaleCheckResult and the caller records that the check was indeterminate. git-base-branch resolveBaseBranch returned 'main' both when no candidate branch existed and when every git tier timed out. A diagnostics variant now reports whether the answer was verified, and the CLI writes an unverified-fallback note to stderr. The stdout contract five workflows parse is untouched. worktree-safety snapshotWorktreeInventory left exists:true when statSync threw, so a guard that could not check reported the worktree present; exists is now tri-state and a stat failure surfaces as an 'unverified' finding. planWorktreePrune reported 'no_worktrees' for a parse failure, which is not the same as an empty list — and it drives a prune. It now reports 'parse_failed'. Fixing the inventory change exposed a second fail-open in verify.cts: the validate-health consumer silently dropped findings whose kind it did not recognise, so the new kind would have vanished. That is closed too — worth noting that the survey enumerated producers of degraded verdicts, not consumers that discard them. worktree-base-ref and state-transition gain the distinguishing signal without changing what they do: headAbsenceVerified, and a phase-inventory scan meta. Whether those guards should ACT differently is a product question this change does not answer, and both are flagged rather than quietly settled. rescueSummaryArtifacts is left alone: rescuing on an uncertain cat-file is deliberate per #2556. It now has tests proving it, and a recorded negative finding — git cat-file -e returns 128 for both "absent from HEAD" and a fatal error, so "uncertain" and "certain-and-fine" are not separable at the git level. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#3057): assert typed values, not rendered text Ten assertions in the rebuild CLI suite matched substrings of produced output — STATE.md body fields, a markdown table row, an audit-log heading, and JSON keys read as text. CONTRIBUTING prohibits that: if the code under test produces text, the test asserts on its structured surface instead. No production surface had to be built. Every one already existed and was already compiled into bin/lib: stateExtractField for body fields, parseMarkdownTable for the phase table, collectSection for the audit-log section, and result.data.log — already a typed RebuildLogEntry[]. The tests were matching rendered text sitting next to the structured data. One of those assertions was passing for the wrong reason. `stdout.includes ('rebuilt')` matched the JSON KEY name, not a value: the dry-run path emits `mutated` and the real path emits `rebuilt`, so it would have passed whether the value was true or false. It now asserts the value. external-job's refusal already had to name the offending file — that naming is why the fail-closed variant is survivable rather than a permanent wedge — but the tests proved it by substring of a prose message. The failure result now carries offendingPath as its own field and the tests assert it by value. The human message is unchanged; operators read it. Array membership is left alone. `phaseIds.includes('99')` and `result.updated.includes('Completed Phases')` are membership checks on real arrays, not text matching, and converting them would weaken nothing and clarify nothing. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#3057): execute acquireStateLock instead of grepping its source The non-EEXIST lock test asserted on the TEXT of the built .cjs and never called acquireStateLock. It carried an allow-test-rule: architectural-invariant exemption to permit that. A source grep proves a literal is present in a file, not that the behaviour works — it is weaker than a liveness test, which at least runs the code, and it was the only coverage the fatal-errno path had. Replaced with tests that inject the errno through fs and assert what actually happens: a fatal EACCES propagates out of acquireStateLock with zero backoff sleeps, while EAGAIN/EINTR/EINVAL/EIO/ENOENT/ESTALE/EPERM/EBUSY retry once and succeed. The exemption is removed and its allowlist entry with it. One old assertion is deliberately not carried over: it checked the retryable errnos were expressed as a Set rather than an inline literal. That is a shape check with no runtime signature; the behavioural tests fail if the code reverts to the old inline check, which is the regression it was really guarding. The #3057 lock-body tests move into that same file rather than a new one, which is what lint-test-file-count asks for and puts every acquireStateLock test in one place. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#3057): surface an indeterminate staleness check to its callers An isolated review caught an inconsistency inside this wave. Two of the three "add the distinguishing signal" fixes wire through to something a user sees: git base-branch writes an unverified-fallback diagnostic to stderr, and an unverifiable worktree surfaces as a W020 finding. The third set staleCheckIndeterminate on readVerificationStatus's result and nothing read it. A signal nobody consumes leaves the fail-open exactly as silent as before: the staleness check could fail and the operator saw precisely what they would see if the answer were genuinely "not stale". That is the defect this issue exists to remove, so it is not defensible as scaffolding when its two siblings in the same change already wire through. All five callers now surface it, each through the channel it already had rather than a mechanism imposed uniformly: phase complete adds it to its existing warnings array and, on the blocked path, as an additive note on the error text; init and roadmap carry it as a field on output they already emit; the UAT report carries it without ever gating passed/blockers; workstream inventory takes an injectable writeDiagnostic mirroring the git base-branch idiom, because its return shape had nowhere to hang a per-phase field without rippling the builder's types. The routing decision is unchanged everywhere. What changes is only that a caller and an operator can now tell a failed check from a completed one. That diagnostic carries structured meta rather than being asserted by regex — the default still writes only the human message to stderr, but tests assert phaseDir and reason by value. Two earlier assertions in this branch were converted the same way; this was the last raw-text assertion left. Also records a scope correction: the completePhaseCore guards now compare stateReplaceField's result to the body instead of testing truthiness, so a field whose substitution produced identical text no longer reports as updated. That is a real behaviour fix, not the signal-only change this file was described as carrying, and its tests cover both the changed and unchanged cases. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#3057): bound two heavy subprocesses for a loaded bench, not an idle one The remote matrix surfaced three failures unrelated to this branch's changes. All were bad tests, and a re-run would have hidden every one of them. The reviewer-flags parse block bounded bash -> node -> a full gsd-tools cold start at 5 seconds. On a bench running thirty thousand tests in parallel that is not a hang, it is a busy machine. Raised to 30s, matching the convention sibling suites already use for script invocations, with a comment saying what the budget covers so nobody tightens it back. Two further copies of the same 5-second spawn in the same file had the identical defect and are raised too — they were not in the failure report, but they will be next time. The fragment-propagation test bounded npm run regen:derived — a full build plus eight generators, the heaviest subprocess in the suite — at five minutes, and node22 was killed near the end. The captured output proves it: every generator had written its files and gen:install-tree had emitted all fifteen runtimes before the kill. Raised to fifteen minutes. That failure read as `null !== 0`, which says nothing. status null means killed, not a non-zero exit, and the two want different responses: one is a timeout to size correctly, the other is a real build break. The assertion now distinguishes them and names the signal. Neither test's assertions were weakened and no retry was added. A retry here would suppress exactly the signal the timeout exists to produce. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#3057): capture fd 1 through the mock tracker, not a raw reassignment The phase suite reported zero test results on both lanes while running for five and a half minutes and exiting 1. No assertion text, no stderr, four events for the whole file: enqueue, start, dequeue, complete. That shape is not a failing assertion — it is the runner being unable to read the child at all, because it parses its event stream from the child's stdout. The cause was the capture helper reassigning fs.writeSync directly. Proven rather than assumed: a standalone probe patched fs.writeSync and called process.stdout.write, and the interception fired only when fd 1 resolved to a FILE, not when it was a pipe. The remote runner captures the event stream to a file, so a helper that was invisible against a pipe swallowed the reporter's own output on the bench. That is also why the two sibling suites wired the same way in this change pass cleanly — they use the mock tracker, the seam io.test.cjs established for this exact function. The helper now uses t.mock.method with an explicit restore after each call, so teardown belongs to node:test rather than a second hand-rolled implementation, and the interception cannot outlive the one synchronous call it wraps even if that call throws. Ten call sites thread the test context through; three test callbacks gained the parameter they lacked. The three B3 tests are untouched — same assertions, same fault injection. Only how the context reaches the helper changed. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#3057): capture phase-complete output from a subprocess, not fd 1 Two attempts to make in-process fd-1 interception safe both failed on the bench. The suite reported zero test results on either lane while exiting 1 — four events for the whole file — because the runner parses its event stream from the child's stdout, and process.stdout.write routes through fs.writeSync whenever fd 1 resolves to a file, which is how the runner captures. Patching that seam anywhere in a file can therefore destroy the file's own reporting, and tightening the window only moved the runtime from 326s to 125s without recovering a single event. So the interception is gone rather than tuned. The helper now spawns gsd-tools as a real subprocess and reads stdout the way the OS already gives it to us, which is what the rest of the suite does. It asserts the command succeeded before parsing, so a genuine failure can no longer present as a JSON parse error. The two fault-injecting tests could not survive that move as written: a subprocess cannot see a mock installed in the parent. Instead of reinstating the interception they now produce the fault on disk — the summary artifact is created as a dangling symlink, so the staleness check's real statSync throws inside the child. That is a more honest fixture than a mock in any case, since it is a condition a user's tree can actually be in. Skipped on Windows, matching the existing symlink precedent in the write-guard suite. Three further call sites turned out to depend on parent-process writeFileSync mocks the subprocess could not see. Those call the CJS function directly, which is what they always wanted — they never needed stdout at all. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#3057): one name for one signal, one encoding for one distinction Standards review found four things this branch introduced, all of them inconsistencies with itself rather than with the repo. One upstream bit reached its consumers under three names — verification_stale_check_indeterminate in two modules, the same value with "stale" dropped in a third, and stderr only in the fourth. Standardised on the long name wherever it is a field. The workstream inventory keeps its stderr channel, since its return shape has nowhere to hang a per-phase field without rippling the builder's types, but it now says the same word for the same thing. worktree-safety encoded one three-way distinction two ways in a single file: a named union for a finding's kind, and boolean|null for an inventory entry's existence. The second is now a named union too. Two assertions matched human prose because the blocked and non-blocked completion paths carried no typed field for the signal. Both now assert typed values. The first round of this fix added the field but left the regex beside it, which is the banned pattern sitting next to its own replacement; the second removed it and added an assertion on the reason enum so nothing was lost. The remaining two were reasoned away before being fixed, and both reasons were bad. "No typed surface exists" is the condition CONTRIBUTING says to fix by adding one — it took three lines. "The file already does this dozens of times" is not licence to add instance number thirty-one; a convention that violates a documented rule is debt, not precedent. Vocabulary differing across DIFFERENT modules is left alone: CONTEXT.md rejects a single shared result envelope, so per-module shapes are precedented, and a baseline smell does not outrank a documented standard. A census of every line this branch adds to a test file now finds no regex or substring assertion on produced prose: 87 strictEqual, 25 ok (all non-empty or shape guards), 12 equal, 3 throws (all typed err.code predicates), 3 deepStrictEqual, 2 notStrictEqual. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(#3057): backfill changeset pr number to 3088 --------- Co-authored-by: sim <sim@local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1383 lines
51 KiB
JavaScript
1383 lines
51 KiB
JavaScript
'use strict';
|
|
|
|
/**
|
|
* Unit tests for uat-predicate.cjs
|
|
* Tests the pure-computation module: stripFalsePositiveContexts,
|
|
* parseUatResultItems, evaluateUatPassed.
|
|
*
|
|
* Issue #247 — phase uat-passed predicate
|
|
*/
|
|
|
|
const { test, describe, beforeEach, afterEach } = require('node:test');
|
|
const assert = require('node:assert/strict');
|
|
const fs = require('node:fs');
|
|
const path = require('node:path');
|
|
const os = require('node:os');
|
|
const fc = require('fast-check');
|
|
|
|
const {
|
|
stripFalsePositiveContexts,
|
|
parseUatResultItems,
|
|
analyzeMarkdown,
|
|
evaluateUatPassed,
|
|
} = require('../gsd-core/bin/lib/uat-predicate.cjs');
|
|
const { cleanup } = require('./helpers.cjs');
|
|
|
|
// ─── Helpers ──────────────────────────────────────────────────────────────────
|
|
|
|
function makeTmpDir() {
|
|
return fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-uat-pred-test-'));
|
|
}
|
|
|
|
function rmDir(dir) {
|
|
cleanup(dir);
|
|
}
|
|
|
|
function writeFile(dir, name, content) {
|
|
fs.writeFileSync(path.join(dir, name), content, 'utf-8');
|
|
}
|
|
|
|
function makePassingUat(n = 1) {
|
|
const tests = Array.from({ length: n }, (_, i) => [
|
|
`### ${i + 1}. Test ${i + 1}`,
|
|
`expected: It works`,
|
|
`result: passed`,
|
|
'',
|
|
].join('\n')).join('\n');
|
|
return `---\nstatus: passed\n---\n\n# UAT\n\n${tests}`;
|
|
}
|
|
|
|
// ─── stripFalsePositiveContexts ────────────────────────────────────────────────
|
|
|
|
describe('stripFalsePositiveContexts — frontmatter', () => {
|
|
test('removes leading frontmatter block', () => {
|
|
const input = '---\nstatus: pending\nresult: pending\n---\n\nReal content here.';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'frontmatter result: pending should be stripped');
|
|
assert.ok(out.includes('Real content here.'), 'body content must be preserved');
|
|
});
|
|
|
|
test('does not strip non-frontmatter --- dividers later in document', () => {
|
|
const input = '---\nstatus: ok\n---\n\n# Section\n\n---\n\nMore content.';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(out.includes('More content.'), 'content after a non-frontmatter divider must survive');
|
|
});
|
|
|
|
test('handles CRLF frontmatter', () => {
|
|
const input = '---\r\nstatus: partial\r\nresult: pending\r\n---\r\n\r\nBody text.';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'CRLF frontmatter must be stripped');
|
|
assert.ok(out.includes('Body text.'), 'body after CRLF frontmatter must survive');
|
|
});
|
|
});
|
|
|
|
describe('stripFalsePositiveContexts — HTML comments', () => {
|
|
test('removes single-line HTML comment', () => {
|
|
const input = 'Before\n<!-- result: pending -->\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'HTML comment content must be stripped');
|
|
assert.ok(out.includes('Before'), 'content before comment must survive');
|
|
assert.ok(out.includes('After'), 'content after comment must survive');
|
|
});
|
|
|
|
test('removes multi-line HTML comment', () => {
|
|
const input = 'A\n<!--\n### 1. Test\nexpected: Foo\nresult: pending\n-->\nB';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'multi-line HTML comment content must be stripped');
|
|
assert.ok(out.includes('A'), 'content before comment must survive');
|
|
assert.ok(out.includes('B'), 'content after comment must survive');
|
|
});
|
|
|
|
test('unterminated HTML comment swallows to EOF (fail-closed)', () => {
|
|
const input = 'Before\n<!--\nresult: passed\nstill in comment';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: passed'), 'unterminated comment must swallow to EOF');
|
|
assert.ok(out.includes('Before'), 'content before comment must survive');
|
|
});
|
|
});
|
|
|
|
describe('stripFalsePositiveContexts — fenced code blocks', () => {
|
|
test('removes backtick fence with result inside', () => {
|
|
const input = 'Before\n```\n### 1. Test\nexpected: X\nresult: pending\n```\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'content inside ``` fence must be stripped');
|
|
assert.ok(out.includes('Before'), 'content before fence must survive');
|
|
assert.ok(out.includes('After'), 'content after fence must survive');
|
|
});
|
|
|
|
test('removes tilde fence', () => {
|
|
const input = 'Before\n~~~\n### 1. Test\nexpected: X\nresult: blocked\n~~~\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: blocked'), 'content inside ~~~ fence must be stripped');
|
|
assert.ok(out.includes('After'), 'content after tilde fence must survive');
|
|
});
|
|
|
|
test('handles indented fence', () => {
|
|
const input = 'Before\n ```\n### 1. Test\nresult: pending\n ```\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'content inside indented fence must be stripped');
|
|
});
|
|
|
|
test('handles CRLF in fenced block', () => {
|
|
const input = 'Before\r\n```\r\nresult: pending\r\n```\r\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'CRLF fenced content must be stripped');
|
|
});
|
|
|
|
test('preserves content after multiple fenced blocks', () => {
|
|
const input = [
|
|
'Real intro',
|
|
'```',
|
|
'result: pending',
|
|
'```',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Works',
|
|
'result: passed',
|
|
'',
|
|
'```',
|
|
'result: pending',
|
|
'```',
|
|
'Trailing',
|
|
].join('\n');
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(out.includes('result: passed'), 'real result outside fence must survive');
|
|
assert.ok(!out.includes('result: pending'), 'fenced result: pending must be stripped');
|
|
});
|
|
});
|
|
|
|
describe('stripFalsePositiveContexts — blockquotes', () => {
|
|
test('removes blockquote lines', () => {
|
|
const input = 'Before\n> ### 1. Test\n> result: pending\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'blockquote content must be stripped');
|
|
assert.ok(out.includes('Before'), 'content before blockquote must survive');
|
|
assert.ok(out.includes('After'), 'content after blockquote must survive');
|
|
});
|
|
|
|
test('removes indented blockquote lines', () => {
|
|
const input = 'Before\n > result: pending\nAfter';
|
|
const out = stripFalsePositiveContexts(input);
|
|
assert.ok(!out.includes('result: pending'), 'indented blockquote content must be stripped');
|
|
});
|
|
});
|
|
|
|
// ─── parseUatResultItems ───────────────────────────────────────────────────────
|
|
|
|
describe('parseUatResultItems', () => {
|
|
test('parses a single passing test', () => {
|
|
const content = '### 1. Login flow\nexpected: User logs in\nresult: passed\n';
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].test, 1);
|
|
assert.strictEqual(items[0].name, 'Login flow');
|
|
assert.strictEqual(items[0].result, 'passed');
|
|
});
|
|
|
|
test('parses bracketed result [passed] (#2273)', () => {
|
|
const content = '### 1. Login flow\nexpected: User logs in\nresult: [passed]\n';
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].result, 'passed');
|
|
});
|
|
|
|
test('parses bracketed result [pass]', () => {
|
|
const content = '### 1. Login flow\nexpected: User logs in\nresult: [pass]\n';
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].result, 'pass');
|
|
});
|
|
|
|
test('parses multiple tests with mixed results', () => {
|
|
const content = [
|
|
'### 1. Test A',
|
|
'expected: A',
|
|
'result: passed',
|
|
'',
|
|
'### 2. Test B',
|
|
'expected: B',
|
|
'result: pending',
|
|
'',
|
|
'### 3. Test C',
|
|
'expected: C',
|
|
'result: failed',
|
|
'',
|
|
].join('\n');
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 3);
|
|
assert.strictEqual(items[0].result, 'passed');
|
|
assert.strictEqual(items[1].result, 'pending');
|
|
assert.strictEqual(items[2].result, 'failed');
|
|
});
|
|
|
|
test('returns empty array for content with no test blocks', () => {
|
|
const items = parseUatResultItems('Just some markdown with no test blocks.');
|
|
assert.strictEqual(items.length, 0);
|
|
});
|
|
|
|
test('lowercases result value', () => {
|
|
const content = '### 1. Test\nexpected: Foo\nresult: PASSED\n';
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items[0].result, 'passed');
|
|
});
|
|
|
|
test('heading with NO result line → result:missing (blocker state, not dropped)', () => {
|
|
const content = [
|
|
'### 1. Test without result',
|
|
'expected: Something',
|
|
'some notes here',
|
|
'',
|
|
].join('\n');
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].result, 'missing');
|
|
});
|
|
|
|
test('result: line with leading whitespace (indented scalar) is NOT parsed as column-0', () => {
|
|
// A real failing test uses block-scalar expected: with blank line before result
|
|
// The result: with leading whitespace should not be treated as a column-0 result
|
|
const content = [
|
|
'### 1. Test A',
|
|
'expected: |',
|
|
' multi',
|
|
' line',
|
|
'',
|
|
' result: pending',
|
|
'',
|
|
].join('\n');
|
|
// The indented result: should not match; heading has no column-0 result → 'missing'
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].result, 'missing',
|
|
'Indented result: line must not be parsed as column-0 result');
|
|
});
|
|
|
|
test('block-scalar expected: with blank line then column-0 result: is parsed correctly', () => {
|
|
// Real failing test: block-scalar expected: | followed by blank + real result: pending
|
|
const content = [
|
|
'### 1. Test A',
|
|
'expected: |',
|
|
' multi',
|
|
' line',
|
|
'',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.strictEqual(items[0].result, 'pending',
|
|
'Column-0 result: after blank line must be parsed correctly');
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — boundary coverage ────────────────────────────────────
|
|
|
|
describe('evaluateUatPassed — boundary coverage', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('0 UAT files → passed:false, no_uat_artifacts:true (fail-closed, no vacuous pass)', () => {
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'No UAT files must NOT vacuously pass — fail-closed');
|
|
assert.strictEqual(report.no_uat_artifacts, true);
|
|
assert.deepStrictEqual(report.uat_files, []);
|
|
assert.deepStrictEqual(report.blockers, []);
|
|
});
|
|
|
|
test('1 passing test → passed:true, no_uat_artifacts:false', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.no_uat_artifacts, false);
|
|
assert.strictEqual(report.blockers.length, 0);
|
|
assert.strictEqual(report.checks.length, 1);
|
|
assert.strictEqual(report.checks[0].passing, true);
|
|
});
|
|
|
|
test('N all-passing tests → passed:true', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(5));
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.checks.every(c => c.passing), true);
|
|
});
|
|
|
|
test('N-1 passing + 1 pending → passed:false', () => {
|
|
// 3 tests total: tests 1,2 passed; test 3 pending
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: passed', '',
|
|
'### 2. Test B', 'expected: B', 'result: passed', '',
|
|
'### 3. Test C', 'expected: C', 'result: pending', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /test 3/i.test(b) || /pending/i.test(b)),
|
|
`Expected blocker for pending test, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('1 blocked test → passed:false', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: blocked', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
});
|
|
|
|
test('1 skipped test → passed:false', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: skipped', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
});
|
|
|
|
test('1 failed test → passed:false', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: failed', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
});
|
|
|
|
test('1 human_needed test → passed:false', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: human_needed', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
});
|
|
|
|
test('heading with no result line → result:missing → blocker → passed:false', () => {
|
|
const content = [
|
|
'### 1. Test without result',
|
|
'expected: It should work',
|
|
'some notes but no result',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.checks.some(c => c.result === 'missing' && !c.passing),
|
|
`Expected missing check item, got checks: ${JSON.stringify(report.checks)}`);
|
|
assert.ok(report.blockers.some(b => /missing/i.test(b)),
|
|
`Expected missing blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — frontmatter status checks ────────────────────────────
|
|
|
|
describe('evaluateUatPassed — frontmatter status', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
const failingUatStatuses = ['partial', 'diagnosed', 'pending', 'blocked', 'in_progress', 'failed'];
|
|
const failingUatResults = ['pending', 'blocked', 'failed'];
|
|
|
|
for (const status of failingUatStatuses) {
|
|
test(`UAT frontmatter status=${status} → passed:false`, () => {
|
|
const content = [
|
|
'---', `status: ${status}`, '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: passed', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
`status=${status} should cause failure`);
|
|
assert.ok(report.blockers.some(b => b.includes(status)),
|
|
`Blocker should mention status=${status}, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
}
|
|
|
|
for (const result of failingUatResults) {
|
|
test(`UAT frontmatter result=${result} → passed:false`, () => {
|
|
const content = [
|
|
'---', `result: ${result}`, '---', '',
|
|
'### 1. Test A', 'expected: A', 'result: passed', '',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
`fm result=${result} should cause failure`);
|
|
});
|
|
}
|
|
});
|
|
|
|
// ─── evaluateUatPassed — VERIFICATION file checks ─────────────────────────────
|
|
|
|
describe('evaluateUatPassed — VERIFICATION files', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('VERIFICATION with status human_needed → passed:false', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_needed\n---\n\nNeeds human check.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /human_needed/i.test(b)),
|
|
`Expected human_needed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with status gaps_found → passed:false', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: gaps_found\n---\n\nHas gaps.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /gaps_found/i.test(b)),
|
|
`Expected gaps_found blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with status pending → passed:false', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: pending\n---\n');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
});
|
|
|
|
test('VERIFICATION with status failed → passed:false (blocking)', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: failed\n---\n');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /failed/i.test(b)),
|
|
`Expected failed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with status in_progress → passed:false (blocking)', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: in_progress\n---\n');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /in_progress/i.test(b)),
|
|
`Expected in_progress blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with missing status does NOT satisfy --require-verification', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '# Verification\nNo frontmatter status.\n');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false,
|
|
'Missing verification status must not satisfy requireVerification');
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with unknown status does NOT satisfy --require-verification', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: some_unknown_status\n---\n');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false,
|
|
'Unknown verification status must not satisfy requireVerification');
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION with status passed → does not block', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.verification_files.length, 1);
|
|
});
|
|
|
|
test('VERIFICATION with status complete → does not block', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: complete\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
});
|
|
|
|
test('VERIFICATION with status verified → does not block', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: verified\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
});
|
|
|
|
test('VERIFICATION with status human_passed → does not block', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_passed\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true);
|
|
});
|
|
|
|
test('VERIFICATION status complete does NOT satisfy --require-verification', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: complete\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false);
|
|
assert.strictEqual(report.policy.require_verification, true);
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION status verified does NOT satisfy --require-verification', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: verified\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('VERIFICATION status human_passed does NOT satisfy --require-verification', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: human_passed\n---\n\nAll good.');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — policy.requireVerification ───────────────────────────
|
|
|
|
describe('evaluateUatPassed — policy.requireVerification', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('requireVerification=true with no verification file → passed:false', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(report.blockers.some(b => /verification required/i.test(b)),
|
|
`Expected verification-required blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
assert.strictEqual(report.policy.require_verification, true);
|
|
});
|
|
|
|
test('requireVerification=true with passing verification → passed:true', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n\nOK.');
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.policy.require_verification, true);
|
|
});
|
|
|
|
test('requireVerification=false (default) with no verification file → passed:true', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: false } });
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.policy.require_verification, false);
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — #3057 B3: staleness-check indeterminate is surfaced ──
|
|
//
|
|
// readVerificationStatus's internal staleness check can fail (fs /
|
|
// scanPhasePlans / clock error). Pre-#3057 B3 wiring, the requireVerification
|
|
// policy check used only `.status`, dropping `.staleCheckIndeterminate` on
|
|
// the floor — so cmdPhaseUatPassed's JSON output (which spreads this whole
|
|
// report) could never distinguish "checked; nothing is stale" from "could
|
|
// not check". `verification_stale_check_indeterminate` must never itself gate
|
|
// `passed`/`blockers` — only readVerificationStatus's `.status` may.
|
|
|
|
describe('#3057 B3: evaluateUatPassed — verification staleness-check indeterminate is surfaced', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
function seedVerifiedPhase() {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n\nOK.');
|
|
writeFile(tmpDir, 'phase-SUMMARY.md', '# Summary');
|
|
const verificationPath = path.join(tmpDir, 'phase-VERIFICATION.md');
|
|
const summaryPath = path.join(tmpDir, 'phase-SUMMARY.md');
|
|
// Deterministic mtime ordering — never rely on write-order clock ties.
|
|
const older = new Date('2026-01-01T00:00:00.000Z');
|
|
const newer = new Date('2026-01-01T00:01:00.000Z');
|
|
fs.utimesSync(summaryPath, older, older);
|
|
fs.utimesSync(verificationPath, newer, newer);
|
|
return { summaryPath, verificationPath };
|
|
}
|
|
|
|
test('an fs failure inside the staleness check sets verification_stale_check_indeterminate:true; passed/blockers unchanged', (t) => {
|
|
const { summaryPath, verificationPath } = seedVerifiedPhase();
|
|
const origStatSync = fs.statSync;
|
|
|
|
t.mock.method(fs, 'statSync', function injectedStaleCheckFault(target, ...args) {
|
|
const targetPath = String(target);
|
|
if (targetPath === verificationPath || targetPath === summaryPath) {
|
|
throw new Error('injected stat failure (#3057 B3)');
|
|
}
|
|
return origStatSync.call(fs, target, ...args);
|
|
});
|
|
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
|
|
// Pre-existing no-throw fail-open routing is UNCHANGED — `passed`/
|
|
// `blockers` are exactly what they would be without the injected fault.
|
|
assert.strictEqual(report.passed, true);
|
|
assert.deepStrictEqual(report.blockers, []);
|
|
assert.strictEqual(report.verification_stale_check_indeterminate, true);
|
|
});
|
|
|
|
test('a completed staleness check that finds nothing stale reports verification_stale_check_indeterminate:false', () => {
|
|
seedVerifiedPhase();
|
|
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: true } });
|
|
|
|
assert.strictEqual(report.passed, true);
|
|
assert.strictEqual(report.verification_stale_check_indeterminate, false);
|
|
});
|
|
|
|
test('requireVerification not set → verification_stale_check_indeterminate is always false (readVerificationStatus never reached)', () => {
|
|
seedVerifiedPhase();
|
|
|
|
const report = evaluateUatPassed(tmpDir, { policy: { requireVerification: false } });
|
|
|
|
assert.strictEqual(report.verification_stale_check_indeterminate, false);
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — malformed markdown guard ─────────────────────────────
|
|
|
|
describe('evaluateUatPassed — malformed markdown blocker', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('unterminated code fence → malformed blocker → passed:false (even with real result:passed)', () => {
|
|
const content = [
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: passed',
|
|
'',
|
|
'```',
|
|
'unterminated fence — no closing delimiter',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'Unterminated fence must always block, even if a result:passed exists');
|
|
assert.ok(report.blockers.some(b => /malformed/i.test(b)),
|
|
`Expected malformed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('unterminated HTML comment → malformed blocker → passed:false', () => {
|
|
const content = [
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: passed',
|
|
'',
|
|
'<!-- unterminated comment',
|
|
'no closing arrow',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'Unterminated comment must always block');
|
|
assert.ok(report.blockers.some(b => /malformed/i.test(b)),
|
|
`Expected malformed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
});
|
|
|
|
test('well-formed fences → no malformed blocker', () => {
|
|
const content = [
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: passed',
|
|
'',
|
|
'```',
|
|
'code here',
|
|
'```',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.ok(!report.blockers.some(b => /malformed/i.test(b)),
|
|
'Well-formed fences must not produce malformed blocker');
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — hardening regressions (the heart of #247) ────────────
|
|
|
|
describe('evaluateUatPassed — false-positive hardening regressions', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('#247: result:passed inside fenced block: parseUatResultItems returns [] for fake, evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
|
|
// ONLY result: passed is inside a fenced block — no real test blocks
|
|
const rawContent = [
|
|
'```',
|
|
'### 1. Fake Test',
|
|
'expected: Example output',
|
|
'result: passed',
|
|
'```',
|
|
].join('\n');
|
|
// After stripping, the clean content has no headings at all
|
|
const clean = stripFalsePositiveContexts(rawContent);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 0, 'Fake result inside fence must produce no items');
|
|
|
|
writeFile(tmpDir, 'phase-UAT.md', rawContent);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'result:passed inside a fenced block must not flip passed to true');
|
|
assert.strictEqual(report.no_uat_artifacts, true,
|
|
'no real UAT items → no_uat_artifacts:true');
|
|
});
|
|
|
|
test('#247: result:passed inside blockquote: parseUatResultItems returns [] + evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
|
|
const rawContent = [
|
|
'> ### 1. Test',
|
|
'> expected: Example',
|
|
'> result: passed',
|
|
].join('\n');
|
|
const clean = stripFalsePositiveContexts(rawContent);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 0, 'Fake result inside blockquote must produce no items');
|
|
|
|
writeFile(tmpDir, 'phase-UAT.md', rawContent);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.strictEqual(report.no_uat_artifacts, true);
|
|
});
|
|
|
|
test('#247: result:passed inside HTML comment: parseUatResultItems returns [] + evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
|
|
const rawContent = [
|
|
'<!-- ### 1. Test',
|
|
'expected: Example',
|
|
'result: passed -->',
|
|
].join('\n');
|
|
const clean = stripFalsePositiveContexts(rawContent);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 0, 'Fake result inside HTML comment must produce no items');
|
|
|
|
writeFile(tmpDir, 'phase-UAT.md', rawContent);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.strictEqual(report.no_uat_artifacts, true);
|
|
});
|
|
|
|
test('#247: result:passed inside frontmatter: parseUatResultItems returns [] + evaluateUatPassed → passed:false + no_uat_artifacts:true', () => {
|
|
const rawContent = [
|
|
'---',
|
|
'example_result: passed',
|
|
'---',
|
|
'',
|
|
'No real test blocks here.',
|
|
].join('\n');
|
|
const clean = stripFalsePositiveContexts(rawContent);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 0, 'Fake result inside frontmatter must produce no items');
|
|
|
|
writeFile(tmpDir, 'phase-UAT.md', rawContent);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.strictEqual(report.no_uat_artifacts, true);
|
|
});
|
|
|
|
test('#247: result:passed inside fenced block is NOT treated as passing test (with real failing test)', () => {
|
|
const content = [
|
|
'---',
|
|
'status: partial',
|
|
'---',
|
|
'',
|
|
'# Example',
|
|
'',
|
|
'```',
|
|
'### 1. Test',
|
|
'expected: Example output',
|
|
'result: passed',
|
|
'```',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'result:passed inside a fenced block must not flip passed to true');
|
|
assert.ok(report.checks.some(c => c.result === 'pending' && !c.passing),
|
|
'Real pending test must be captured');
|
|
});
|
|
|
|
test('#247: result:passed inside blockquote is NOT treated as passing test', () => {
|
|
const content = [
|
|
'---',
|
|
'status: partial',
|
|
'---',
|
|
'',
|
|
'> ### 1. Test',
|
|
'> expected: Example',
|
|
'> result: passed',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'result:passed inside a blockquote must not flip passed to true');
|
|
});
|
|
|
|
test('#247: result:passed inside HTML comment is NOT treated as passing test', () => {
|
|
const content = [
|
|
'---',
|
|
'status: partial',
|
|
'---',
|
|
'',
|
|
'<!-- ### 1. Test',
|
|
'expected: Example',
|
|
'result: passed -->',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'result:passed inside an HTML comment must not flip passed to true');
|
|
});
|
|
|
|
test('#247: result:passed inside frontmatter example is NOT treated as passing test', () => {
|
|
const content = [
|
|
'---',
|
|
'status: partial',
|
|
'example_result: passed',
|
|
'---',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'result:passed in frontmatter must not flip passed to true');
|
|
});
|
|
|
|
test('#247: real passing test block outside all contexts → passed:true', () => {
|
|
const content = [
|
|
'---',
|
|
'status: passed',
|
|
'---',
|
|
'',
|
|
'# UAT Results',
|
|
'',
|
|
'### 1. Login works',
|
|
'expected: User logs in',
|
|
'result: passed',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true,
|
|
'A real passing test outside false-positive contexts must pass');
|
|
});
|
|
|
|
test('block-scalar expected: followed by blank line + result: pending → parsed as blocker (not dropped)', () => {
|
|
const content = [
|
|
'### 1. Test A',
|
|
'expected: |',
|
|
' multi',
|
|
' line expected output',
|
|
'',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'Block-scalar expected: with result: pending must be captured as a blocker');
|
|
assert.ok(report.checks.some(c => c.result === 'pending' && !c.passing),
|
|
`Expected pending check, got: ${JSON.stringify(report.checks)}`);
|
|
});
|
|
});
|
|
|
|
// ─── evaluateUatPassed — output shape contract ───────────────────────────────
|
|
|
|
describe('evaluateUatPassed — output shape (Hyrum\'s Law contract)', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('returns all required fields in the locked shape including no_uat_artifacts', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir);
|
|
|
|
// Locked field names
|
|
assert.ok('passed' in report, 'report.passed must exist');
|
|
assert.ok('uat_files' in report, 'report.uat_files must exist');
|
|
assert.ok('verification_files' in report, 'report.verification_files must exist');
|
|
assert.ok('checks' in report, 'report.checks must exist');
|
|
assert.ok('blockers' in report, 'report.blockers must exist');
|
|
assert.ok('no_uat_artifacts' in report, 'report.no_uat_artifacts must exist');
|
|
assert.ok('policy' in report, 'report.policy must exist');
|
|
assert.ok('require_verification' in report.policy, 'report.policy.require_verification must exist');
|
|
|
|
// checks item shape
|
|
if (report.checks.length > 0) {
|
|
const c = report.checks[0];
|
|
assert.ok('file' in c, 'check.file must exist');
|
|
assert.ok('test' in c, 'check.test must exist');
|
|
assert.ok('name' in c, 'check.name must exist');
|
|
assert.ok('result' in c, 'check.result must exist');
|
|
assert.ok('passing' in c, 'check.passing must exist');
|
|
}
|
|
});
|
|
|
|
test('no_uat_artifacts is false when real checks exist', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.no_uat_artifacts, false);
|
|
});
|
|
|
|
test('no_uat_artifacts is true when no checks exist', () => {
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.no_uat_artifacts, true);
|
|
});
|
|
|
|
test('uat_files contains the filename', () => {
|
|
writeFile(tmpDir, 'my-UAT.md', makePassingUat(1));
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.ok(report.uat_files.includes('my-UAT.md'),
|
|
`uat_files should include 'my-UAT.md', got: ${JSON.stringify(report.uat_files)}`);
|
|
});
|
|
|
|
test('verification_files contains the filename', () => {
|
|
writeFile(tmpDir, 'phase-UAT.md', makePassingUat(1));
|
|
writeFile(tmpDir, 'phase-VERIFICATION.md', '---\nstatus: passed\n---\n');
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.ok(report.verification_files.includes('phase-VERIFICATION.md'),
|
|
`verification_files should include 'phase-VERIFICATION.md', got: ${JSON.stringify(report.verification_files)}`);
|
|
});
|
|
});
|
|
|
|
// ─── FIX A regression: nested-fence (~~~ inside ```) ─────────────────────────
|
|
|
|
describe('FIX A — nested fence: ~~~ inside ``` does not prematurely close outer fence', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => { tmpDir = makeTmpDir(); });
|
|
afterEach(() => { rmDir(tmpDir); });
|
|
|
|
test('parseUatResultItems sees [] for fake inside ``` that encloses ~~~', () => {
|
|
// A backtick fence that contains an inner ~~~ fence with a fake test block.
|
|
// The ~~~ must NOT close the ``` fence — the whole interior is content and is dropped.
|
|
const raw = [
|
|
'```',
|
|
'~~~',
|
|
'### 1. Fake',
|
|
'expected: X',
|
|
'result: passed',
|
|
'~~~',
|
|
'```',
|
|
].join('\n');
|
|
const clean = stripFalsePositiveContexts(raw);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 0, 'fake inside nested fence must not leak through');
|
|
});
|
|
|
|
test('evaluateUatPassed → passed:false + no_uat_artifacts:true for nested-fence-only file', () => {
|
|
const raw = [
|
|
'```',
|
|
'~~~',
|
|
'### 1. Fake',
|
|
'expected: X',
|
|
'result: passed',
|
|
'~~~',
|
|
'```',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', raw);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false, 'nested-fence fake must not flip passed');
|
|
assert.strictEqual(report.no_uat_artifacts, true, 'no real items → no_uat_artifacts:true');
|
|
assert.ok(!report.checks.some(c => c.name === 'Fake'), 'fake must not appear in checks');
|
|
});
|
|
|
|
test('balanced nested fence (``` inside ~~~) is NOT flagged as malformed', () => {
|
|
// ~~~ outer, ``` inner — properly closed — should not trigger unterminatedFence
|
|
const raw = [
|
|
'~~~',
|
|
'```',
|
|
'code',
|
|
'```',
|
|
'~~~',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Works',
|
|
'result: passed',
|
|
].join('\n');
|
|
const { unterminatedFence } = analyzeMarkdown(raw);
|
|
assert.strictEqual(unterminatedFence, false, 'balanced nested fence must not be flagged');
|
|
const clean = stripFalsePositiveContexts(raw);
|
|
const items = parseUatResultItems(clean);
|
|
assert.strictEqual(items.length, 1, 'real test outside fence must still be found');
|
|
assert.strictEqual(items[0].result, 'passed');
|
|
});
|
|
|
|
test('real result:passed outside ``` that encloses ~~~ → passed:true (no false-block)', () => {
|
|
const raw = [
|
|
'---',
|
|
'status: passed',
|
|
'---',
|
|
'',
|
|
'```',
|
|
'~~~',
|
|
'### 1. Fake',
|
|
'result: passed',
|
|
'~~~',
|
|
'```',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Works',
|
|
'result: passed',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', raw);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, true, 'real result outside nested fence must still pass');
|
|
assert.ok(!report.checks.some(c => c.name === 'Fake'), 'fake must not appear in checks');
|
|
});
|
|
});
|
|
|
|
// ─── FIX B regression: cross-line result value ────────────────────────────────
|
|
|
|
describe('FIX B — cross-line result: value must be on the same line', () => {
|
|
test('result: with value on next line → result:missing (not passed)', () => {
|
|
const content = [
|
|
'### 1. Cross-line Test',
|
|
'expected: Y',
|
|
'result:',
|
|
'',
|
|
'passed',
|
|
].join('\n');
|
|
const items = parseUatResultItems(content);
|
|
assert.strictEqual(items.length, 1);
|
|
assert.notStrictEqual(items[0].result, 'passed',
|
|
'result value on a subsequent line must not be captured as passed');
|
|
assert.strictEqual(items[0].result, 'missing',
|
|
'cross-line result must yield missing (blocker)');
|
|
});
|
|
|
|
test('evaluateUatPassed → passed:false for cross-line result:passed', () => {
|
|
const tmpDir = makeTmpDir();
|
|
try {
|
|
const content = [
|
|
'### 1. Cross-line Test',
|
|
'expected: Y',
|
|
'result:',
|
|
'',
|
|
'passed',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(!report.checks.some(c => c.result === 'passed'),
|
|
'cross-line result must not produce a passing check');
|
|
} finally {
|
|
rmDir(tmpDir);
|
|
}
|
|
});
|
|
});
|
|
|
|
// ─── FIX C regression: masked-comment (earlier closed comment + later unterminated) ──
|
|
|
|
describe('FIX C — dangling comment survives earlier balanced comment', () => {
|
|
test('analyzeMarkdown detects unterminated comment after a properly closed one', () => {
|
|
const raw = [
|
|
'<!-- ok -->',
|
|
'Some text',
|
|
'<!--',
|
|
'### 1. Fake',
|
|
'result: passed',
|
|
].join('\n');
|
|
const { unterminatedComment } = analyzeMarkdown(raw);
|
|
assert.strictEqual(unterminatedComment, true,
|
|
'later unterminated comment must be detected even after a balanced one');
|
|
});
|
|
|
|
test('evaluateUatPassed → passed:false (malformed) when later comment is unterminated', () => {
|
|
const tmpDir = makeTmpDir();
|
|
try {
|
|
const raw = [
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: passed',
|
|
'',
|
|
'<!-- properly closed -->',
|
|
'',
|
|
'<!--',
|
|
'### 2. Fake',
|
|
'result: passed',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', raw);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false,
|
|
'unterminated later comment must trigger malformed blocker');
|
|
assert.ok(report.blockers.some(b => /malformed/i.test(b)),
|
|
`Expected malformed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
} finally {
|
|
rmDir(tmpDir);
|
|
}
|
|
});
|
|
|
|
test('analyzeMarkdown does NOT flag a file with only properly balanced comments', () => {
|
|
const raw = [
|
|
'<!-- first comment -->',
|
|
'Some text',
|
|
'<!-- second comment -->',
|
|
'',
|
|
'### 1. Real Test',
|
|
'result: passed',
|
|
].join('\n');
|
|
const { unterminatedComment } = analyzeMarkdown(raw);
|
|
assert.strictEqual(unterminatedComment, false,
|
|
'only balanced comments must not be flagged as unterminated');
|
|
});
|
|
});
|
|
|
|
// ─── FIX D regression: balanced mixed fences are NOT flagged ─────────────────
|
|
|
|
describe('FIX D — odd-fence-count heuristic replaced: balanced mixed fences not flagged', () => {
|
|
test('analyzeMarkdown: ``` followed by ~~~ (both balanced) → unterminatedFence:false', () => {
|
|
const raw = [
|
|
'```',
|
|
'code',
|
|
'```',
|
|
'',
|
|
'~~~',
|
|
'more code',
|
|
'~~~',
|
|
].join('\n');
|
|
const { unterminatedFence } = analyzeMarkdown(raw);
|
|
assert.strictEqual(unterminatedFence, false,
|
|
'two separate balanced fences must not trigger unterminatedFence');
|
|
});
|
|
|
|
test('evaluateUatPassed: multiple balanced fences + real passing test → passed:true, no malformed blocker', () => {
|
|
const tmpDir = makeTmpDir();
|
|
try {
|
|
const raw = [
|
|
'---',
|
|
'status: passed',
|
|
'---',
|
|
'',
|
|
'```',
|
|
'result: fake',
|
|
'```',
|
|
'',
|
|
'~~~',
|
|
'result: also fake',
|
|
'~~~',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Works',
|
|
'result: passed',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', raw);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.ok(!report.blockers.some(b => /malformed/i.test(b)),
|
|
`Balanced mixed fences must not produce malformed blocker, got: ${JSON.stringify(report.blockers)}`);
|
|
assert.strictEqual(report.passed, true,
|
|
'real test outside balanced fences must still pass');
|
|
} finally {
|
|
rmDir(tmpDir);
|
|
}
|
|
});
|
|
});
|
|
|
|
// ─── FIX E extra: fake-not-in-checks assertions for existing mixed tests ──────
|
|
|
|
describe('FIX E — fake items must NOT appear in checks (absence assertions)', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => { tmpDir = makeTmpDir(); });
|
|
afterEach(() => { rmDir(tmpDir); });
|
|
|
|
test('fake inside fence + real pending: fake NOT in checks', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'```',
|
|
'### 10. Fake',
|
|
'expected: Fake',
|
|
'result: passed',
|
|
'```',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(!report.checks.some(c => c.name === 'Fake'),
|
|
'fake test inside fence must not appear in checks');
|
|
});
|
|
|
|
test('fake inside blockquote + real pending: fake NOT in checks', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'> ### 10. Fake',
|
|
'> expected: Fake',
|
|
'> result: passed',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(!report.checks.some(c => c.name === 'Fake'),
|
|
'fake test inside blockquote must not appear in checks');
|
|
});
|
|
|
|
test('fake inside HTML comment + real pending: fake NOT in checks', () => {
|
|
const content = [
|
|
'---', 'status: partial', '---', '',
|
|
'<!-- ### 10. Fake',
|
|
'expected: Fake',
|
|
'result: passed -->',
|
|
'',
|
|
'### 1. Real Test',
|
|
'expected: Something',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
writeFile(tmpDir, 'phase-UAT.md', content);
|
|
const report = evaluateUatPassed(tmpDir);
|
|
assert.strictEqual(report.passed, false);
|
|
assert.ok(!report.checks.some(c => c.name === 'Fake'),
|
|
'fake test inside HTML comment must not appear in checks');
|
|
});
|
|
});
|
|
|
|
// ─── Property-based test (fast-check) ─────────────────────────────────────────
|
|
|
|
describe('evaluateUatPassed — property: wrapping in false-positive context never flips to passed', () => {
|
|
let tmpDir;
|
|
|
|
beforeEach(() => {
|
|
tmpDir = makeTmpDir();
|
|
});
|
|
|
|
afterEach(() => {
|
|
rmDir(tmpDir);
|
|
});
|
|
|
|
test('fc: inserting result:passed inside wrapper context never flips a failing UAT to passed', () => {
|
|
// A baseline UAT file that has a pending item — it must always evaluate to passed:false
|
|
// regardless of how many "result: passed" lines we inject inside fenced/blockquote/comment wrappers.
|
|
const baseFailingBody = [
|
|
'### 1. Real Test',
|
|
'expected: It works',
|
|
'result: pending',
|
|
'',
|
|
].join('\n');
|
|
|
|
const wrappers = fc.constantFrom(
|
|
// backtick fence
|
|
(inner) => '```\n' + inner + '\n```',
|
|
// tilde fence
|
|
(inner) => '~~~\n' + inner + '\n~~~',
|
|
// HTML comment
|
|
(inner) => '<!--\n' + inner + '\n-->',
|
|
// blockquote — prefix each line
|
|
(inner) => inner.split('\n').map(l => '> ' + l).join('\n'),
|
|
);
|
|
|
|
fc.assert(
|
|
fc.property(wrappers, fc.nat(3), (wrap, extraCount) => {
|
|
// Build a "fake passing block" that would fool a naive regex
|
|
const fakePassingLines = Array.from({ length: extraCount + 1 }, (_, i) =>
|
|
`### ${i + 10}. Fake Test ${i + 10}\nexpected: Fake\nresult: passed`
|
|
).join('\n');
|
|
|
|
const fullContent = [
|
|
'---',
|
|
'status: partial',
|
|
'---',
|
|
'',
|
|
wrap(fakePassingLines),
|
|
'',
|
|
baseFailingBody,
|
|
].join('\n');
|
|
|
|
// Write to a unique tmp file to avoid cross-test state
|
|
const fcDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-fc-uat-'));
|
|
try {
|
|
fs.writeFileSync(path.join(fcDir, 'feature-UAT.md'), fullContent, 'utf-8');
|
|
const report = evaluateUatPassed(fcDir);
|
|
// The pending item must always keep passed:false
|
|
// AND fake items injected via wrappers must never appear in checks
|
|
const hasFakeInChecks = report.checks.some(c => c.name.startsWith('Fake Test'));
|
|
return report.passed === false && !hasFakeInChecks;
|
|
} finally {
|
|
cleanup(fcDir);
|
|
}
|
|
}),
|
|
{ numRuns: 50 }
|
|
);
|
|
});
|
|
});
|