* feat(spec-phase): spec-completeness edge-probe (#550) — relocated to gsd-core/
Rebased onto current next and relocated the whole feature from get-shit-done/ to
gsd-core/ per #615 (trek-e re-review #4, option 1). The artifact now builds to
gsd-core/bin/lib/edge-probe.cjs; all hard-coded path strings (tests, workflow
@-refs, run-tests.cjs sentinel, eslint ADR-457 ignore, .gitignore) updated.
Content conflicts in .gitignore / eslint.config.mjs / run-tests.cjs resolved.
Feature: Step 5.5 edge-completeness probe walks each SPEC requirement against a
closed 8-category edge taxonomy, proposes applicable candidate edges, and resolves
each (covered/dismissed/backstop/unresolved). covered/backstop criteria are lifted
by plan-phase into must_haves.truths, extending the goal-backward verifier's reach
to boundary edges no requirement was written for. Engine authored as strict TS
(src/edge-probe.cts, ADR-457), compiled to the gitignored gsd-core/bin/lib/edge-probe.cjs.
Folds in every prior review round on PR #584:
- RR-01..03: plan-phase resolves the phase *-SPEC.md and injects {SPEC_PATH} into the
planner; must_haves<->Edge-Coverage quality_gate; held-out planner-contract test.
- RR-04/11: Step 5.5 invokes the compiled engine at runtime (npm --prefix-pinned,
source-checkout-gated build fallback) instead of LLM re-derivation; the engine
capture is exit-checked and the report JSON-validated before use (fail closed).
- RR-05..10: all six fixtures embedded + count-equality; backstop/covered require a
resolution; Array.isArray(shapes); duplicate-resolution rejection; CLI JSON exit(2);
per-artifact build sentinel.
- Authored-shape validation: invalid (non-empty) shapes fail closed (VALID_SHAPES).
- Adversarial-review hardening: orphan/typo resolution rejection, requirement input
validation (id/text/shapes, duplicate id, non-array), zero-applicable guard.
Full suite 0 failures; npm run lint 0 errors; edge-probe suite 72/72.
* test(#550): RED — status×verification re-cut + probe-core engine specs
Re-cut the edge-probe resolution model onto two orthogonal axes per
ADR-550 Decision 7 (trek-e #644 comments 2026-06-03 14:36 + 14:44):
status: resolved | dismissed | unresolved (lifecycle, shared)
verification: explicit | backstop | null (only when resolved)
- tests/probe-core.test.cjs (new): behavioral specs for the generic engine
to be extracted — validateResolution(r, validators), validateRequirement,
analyzeCoverage(items, resolutions?, validators), byVerification rollup,
runProbeCli I/O scaffold (injected-io unit tests).
- tests/edge-probe.test.cjs: covered→{resolved,explicit}, backstop→
{resolved,backstop}; coverage gains byVerification.{explicit,backstop};
proposeEdges items gain verification:null.
- 6 fixtures re-genned + re-embedded in edge-probe.md; coverage.resolved
COUNT preserved on every fixture (closed set = resolved+dismissed; doc
line: 'adjacency=covered + ordering=dismissed' -> 2). edge-probe.md prose
rewritten to the two-axis model.
Fails as expected: probe-core.cjs has no source yet; edge-probe still
emits the old covered/backstop enum (27/61 edge specs red).
* feat(#550): extract probe-core seam + refactor edge-probe onto it (ADR-550 D7)
Extract the generic resolution model into src/probe-core.cts (the shared
seam the prohibition probe #644 is born on) and refactor edge-probe.cts
into its first adapter.
probe-core owns (probe-agnostic):
- the status×verification re-cut: status: resolved|dismissed|unresolved ×
verification: <probe-defined>|null
- validateResolution(r, validators) / validateRequirement (generic id+text)
- analyzeCoverage(items, resolutions?, validators) over ALREADY-PROPOSED
items[] (core never assumes propose is deterministic — edge resolves via
LLM, #644 proposes via LLM), with merge / dup-reject / orphan-reject
- byVerification rollup; coverage.resolved = closed set (resolved+dismissed),
count-preserved from the pre-re-cut engine
- runProbeCli I/O scaffold (injected io; one bin per probe)
- hybrid typing: generic params + injected runtime validators
{categories, verification, requiredFieldsByVerification} (ADR-550 #5)
edge-probe keeps ONLY the edge cluster: Shape/SHAPE_CUES/VALID_SHAPES/
classifyShape/TAXONOMY/applicableCategories/proposeEdges + EDGE_VALIDATORS
{explicit,backstop}; delegates merge/rollup/CLI to probe-core. Every shipped
#584 guarantee preserved (fail-closed shapes, orphan/dup rejection, input
validation, CLI exit 2). 104/104 edge+probe-core+docs+contract specs green.
* chore(#550): register probe-core.cjs artifact in ledgers + inventory
New gitignored build artifact gsd-core/bin/lib/probe-core.cjs (compiled
from src/probe-core.cts) needs registering in every artifact ledger:
- .gitignore + eslint.config.mjs ADR-457 ignore: lint the .cts source,
never the emitted .cjs.
- scripts/run-tests.cjs per-artifact build sentinel: build if probe-core.cjs
is missing on a clean checkout.
- docs/INVENTORY.md: CLI Modules 83 -> 84, new probe-core.cjs row, and the
edge-probe.cjs row updated to reflect it is now the first probe-core adapter.
- docs/INVENTORY-MANIFEST.json: regenerated (gen-inventory-manifest.cjs --write).
probe-core.test.cjs is a single test file (under the 2-file cap), so no
lint-test-file-count allowlist entry is needed.
* docs(adr-550): spec-phase probe pattern + prohibition contract [Accepted]
trek-e's final ADR-550 body, verbatim (open-gsd/gsd-core#644 comment
2026-06-03T15:23Z), Accepted by both maintainer and #550 author. Lands on
PR #584 alongside the probe-core extraction (Decision 7) it governs, so the
contract and its first implementation arrive together.
Decisions: probe packaging (3 layers); recall->precision protocol;
prohibition home = SPEC acceptance criteria + optional must_haves.prohibitions:
(truths untouched, no polarity); tiered verification test|judgment
(judgment = mode-dependent soft-gate-with-flags, never silent pass / never
hard-halt); CI tests the contract not the classifier; secure-phase ownership
seam; and Decision 7 — probe-core seam + status×verification re-cut (7a-7e),
which this PR implements.
* feat(#550): fail-closed probe-core across full status×verification + runProbeCli structural guard
Re-review #5 (trek-e) seam-hardening on the generic probe-core contract #644 inherits:
- validateResolution now enforces the 'verification is null unless resolved'
invariant for EVERY status (not just resolved): a dismissed/unresolved
resolution carrying a verification tier is rejected instead of merging verbatim.
- An unresolved resolution carrying a resolution/reason payload is rejected
(was silently dropped into the unresolved count).
- runProbeCli structurally validates the report an adapter returns before writing
it (was: any malformed object stringified as green output) — fails closed → exit 2.
- coverage.resolved kept count-preserved (closed set) per the blessed migration
contract; a new test locks that an all-dismissed run is NOT affirmatively covered
(byVerification is the honest gate).
Tests: probe-core 37/37; full edge-probe suite 113/113; full suite 1816/1816; lint 0.
* docs(adr-550): annotate Decision 5 #584/#644 scope + correct 7a coverage.resolved semantics
Re-review #5 (trek-e) clarity edits:
- Decision 5: annotate that only contract item (a) ships on #584 (the edge
adapter's parse+validate test); (b)–(d) are #644 scope, matching Consequences.
- Decision 7a: correct the 'coverage.resolved is preserved (status === resolved)'
parenthetical — the blessed/implemented semantics are count-preserved = the
CLOSED set (resolved + dismissed = applicable − unresolved), with byVerification
carrying the per-tier resolved-status breakdown. The old parenthetical
contradicted the shipped count.
* test(#550): cover runProbeCli structural-guard numeric-count branch
Second-pass coverage audit found the 'coverage object present but counts
non-numeric' branch of isValidReport (built probe-core.cjs:60-61) unexercised —
the {nope:true} malformed test fails earlier at the items[] check. Add a report
with well-formed items[] + a coverage object carrying non-numeric counts so the
numeric branch is hit. No source change; closes the line gap.
* fix(#550): reject edge requirement with missing/empty text when no shapes override (M2)
The edge adapter's `text` is the classification signal and a required field, but
core `validateRequirement` left it optional, so a `{ id }` requirement classified to
zero shapes -> zero edges -> was silently DROPPED from coverage with no signal -- the
exact fail-open this feature exists to eliminate. Reject missing/empty text unless an
authored `shapes` override (incl. `[]`) opts out of prose classification.
* fix(#550): validate verbatim items in analyzeCoverage shared seam (m1)
A proposed item with no matching author resolution is rolled up VERBATIM, but its own
status/fields were never validated -- an item carrying an out-of-enum status (the dropped
"covered") or `dismissed` without a reason would be counted closed. The edge adapter only
proposes `unresolved` items, but the prohibition adapter (#644) proposes LLM-generated
items that arrive populated. An Item is structurally a superset of a Resolution, so reuse
validateResolution to fail closed. ADR-550 Decision 5 hardens this shared seam.
* fix(#550): move edge-coverage lift instruction to runtime planner surface (M1)
templates/planner-subagent-prompt.md is loaded by nothing at runtime (no @-import in
agents/gsd-planner.md; plan-phase.md spawns the planner from its own inline
<planning_context>), so the precise covered/backstop -> must_haves.truths lift instruction
this PR added there never reached the planner -- and the RR-02 contract test asserted it in
that dead file, giving false green. Move the instruction into plan-phase.md's runtime
<downstream_consumer> block (where the rest of the wire already lives), revert the dead-template
edit, and retarget RR-02 to the loaded surface with a guard against re-orphaning.
* test(#550): lock machine<->SPEC vocabulary mapping against drift (m2)
The machine contract uses orthogonal status x verification; the SPEC table renders a flat
covered/dismissed/backstop/unresolved. The migration map (ADR-550 Decision 7a) was prose-only
with no test, so the layers could silently drift. The SPEC table is LLM-rendered (no JS
renderer to round-trip), so pin the canonical bijection as code AND ground it in every doc
surface that renders the vocabulary (ADR migration clause, spec.md legend, reference mapping
table) -- a rename or remap now fails the suite.
* docs(#550): add how-to for resolving edge-coverage findings (B1)
Feature shipped reference coverage (FEATURES.md, COMMANDS.md, references/edge-probe.md) but
no how-to -- reference-only does not satisfy the Diataxis docs standard for a user-facing
capability. Add a single-mode how-to (imperative, goal-directed) walking each resolution
state (specify/dismiss/backstop/defer), the soft gate, and --auto, with taxonomy/concepts
linked out to the reference. Register it in the docs/how-to index.
* docs(#550): add Probe Core + Edge Probe glossary entries to CONTEXT.md (N1)
trek-e re-review #7 N1 (Major): adding probe-core/edge-probe as src/*.cts-derived
seam modules (ADR-550 Decision 7) requires CONTEXT.md Domain-terms glossary entries
per the maintainer-enforced new-seam gate. Adds '### Probe Core Module' and
'### Edge Probe Module' with exports, generated source paths, and the ADR-550 seam
contract, placed beside the Research Module feature-seam entries.
* test(#550): add fast-check property suite for probe-core analyzeCoverage (N2)
trek-e re-review #7 N2 (RULESET.TESTS.property-based-testing): analyzeCoverage is a
transformation/rollup module, the class the property-testing predicate covers, and
fast-check is already a dependency with an established *.property.test.cjs pattern.
Adds 5 properties over the algebraic invariants: closed-set identity
(applicable === resolved + unresolved), byVerification sums ≤ resolved, per-tier
recount + resolved-status-only counting, rollup determinism, and stable orphan
rejection. 200 runs/seed 42 via helpers/fast-check-setup.cjs.
* test(#550): align allow-test-rule tokens to canonical runtime-contract-is-the-product (N3)
trek-e re-review #7 N3 (Nit): the // allow-test-rule: tokens (source-text-is-the-product,
docs-parity) differed from CONTEXT.md's canonical exemption category
'runtime-contract-is-the-product' (RULESET.TESTS.no-source-grep.exemption, CONTEXT.md:240).
All three tests assert deployed runtime-contract surfaces (spec-phase.md Step 5.5, the
plan-phase.md planner prompt, the rendered reference/SPEC/ADR vocabulary), so the canonical
category fits; each now carries a one-line justification per the ruleset format. Free-text
reason — lint behavior unchanged.
* test(#550): re-baseline plan-phase + spec-phase byte sizes for edge-probe
Rebased onto next (e4f0910d), which replaced the line-based tier-max
size ratchet (#597) with the byte-based per-file baseline guard (#1074).
The edge-probe feature legitimately grows two workflows:
- spec-phase.md 15131 -> 23094 (+7963): Step 5.5 Edge-Completeness Probe
- plan-phase.md 93135 -> 94253 (+1118): covered/backstop edge lift into
must_haves.truths (the live <downstream_consumer> block)
Both remain under their tier hard caps (plan-phase XL 98304, ~4KB
headroom; spec-phase DEFAULT 40960). Growth is real inline workflow
content the feature requires at that step — not eager @-import proxy
gaming. Drops the obsolete line-based XL_BUDGET 93000->94000 bump
(superseded by the byte baseline) via rebase.
* chore(#550): reconcile INVENTORY headline counts after rebase onto next
Rebase onto current next dropped the prior reconcile commit (stale counts
refs 68 / CLI 107). Current next + the edge-probe additions yield:
- References (68 -> 69 shipped): + gsd-core/references/edge-probe.md
- CLI Modules (107 -> 109 shipped): + edge-probe.cjs + probe-core.cjs
Rows for all three already present; only the headline counts were stale.
Caught by tests/inventory-counts.test.cjs (CI ubuntu-24 leg).
This commit is contained in:
Rezolv
115
tests/edge-probe-docs-fixtures.test.cjs
Normal file
115
tests/edge-probe-docs-fixtures.test.cjs
Normal file
@@ -0,0 +1,115 @@
|
||||
// allow-test-rule: runtime-contract-is-the-product — the rendered reference/SPEC/ADR vocab surfaces are the runtime contract; this pins their bijection to the code (docs-parity)
|
||||
// Asserts the portable reference doc (gsd-core/references/edge-probe.md) keeps its
|
||||
// worked-example JSON blocks in sync with the source-of-truth fixture files under
|
||||
// gsd-core/references/edge-probe-fixtures/. The fixtures are the canonical data; the
|
||||
// doc embeds copies. Per the CONTRIBUTING.md exception matrix this is `docs-parity`: a
|
||||
// reference doc must mirror source-defined data and there is no runtime enumeration API.
|
||||
// The comparison is PARSED JSON (deepEqual of JSON.parse on both sides), never a raw-text
|
||||
// substring match — so a reformat that preserves the data does not fail, and any semantic
|
||||
// drift between doc and fixture does.
|
||||
'use strict';
|
||||
process.env.GSD_TEST_MODE = '1';
|
||||
|
||||
const { test, describe } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const docPath = path.join(__dirname, '..', 'gsd-core', 'references', 'edge-probe.md');
|
||||
const fixturesRoot = path.join(__dirname, '..', 'gsd-core', 'references', 'edge-probe-fixtures');
|
||||
const adrPath = path.join(__dirname, '..', 'docs', 'adr', '550-spec-phase-probe-contract.md');
|
||||
const specTemplatePath = path.join(__dirname, '..', 'gsd-core', 'templates', 'spec.md');
|
||||
|
||||
// Extract fenced blocks tagged ```json edge-probe:<dir>/<file> from the doc, keyed by ref.
|
||||
// The \n? before the closing fence allows blocks whose closing fence has no preceding newline
|
||||
// (fixes the silent-skip bug where a trailing-fence-with-no-newline was not matched).
|
||||
function taggedJsonBlocks(md) {
|
||||
const re = /```json edge-probe:([^\n]+)\n([\s\S]*?)\n?```/g;
|
||||
const out = {};
|
||||
let m;
|
||||
while ((m = re.exec(md))) out[m[1].trim()] = m[2];
|
||||
return out;
|
||||
}
|
||||
|
||||
describe('edge-probe doc/fixture sync', () => {
|
||||
test('reference doc exists', () => {
|
||||
assert.ok(fs.existsSync(docPath), `${docPath} must exist`);
|
||||
});
|
||||
|
||||
test('doc embeds tagged fixture blocks for every expected-coverage.json fixture (count-equality)', () => {
|
||||
const md = fs.readFileSync(docPath, 'utf8');
|
||||
const blocks = taggedJsonBlocks(md);
|
||||
// Count the expected-coverage.json files under the fixtures root (one per fixture dir).
|
||||
const expectedCount = fs.readdirSync(fixturesRoot, { withFileTypes: true })
|
||||
.filter(e => e.isDirectory())
|
||||
.filter(dir => fs.existsSync(path.join(fixturesRoot, dir.name, 'expected-coverage.json')))
|
||||
.length;
|
||||
assert.strictEqual(
|
||||
Object.keys(blocks).length,
|
||||
expectedCount,
|
||||
`edge-probe.md must embed exactly ${expectedCount} tagged blocks (one per fixture expected-coverage.json)`
|
||||
);
|
||||
});
|
||||
|
||||
test('every tagged doc block parses and deepEquals its fixture file', () => {
|
||||
const md = fs.readFileSync(docPath, 'utf8');
|
||||
const blocks = taggedJsonBlocks(md);
|
||||
for (const [ref, body] of Object.entries(blocks)) {
|
||||
const fixtureFile = path.join(fixturesRoot, ref);
|
||||
const onDisk = fs.readFileSync(fixtureFile, 'utf8');
|
||||
assert.deepEqual(JSON.parse(body), JSON.parse(onDisk),
|
||||
`doc block edge-probe:${ref} must deepEqual ${fixtureFile}`);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// m2: lock the machine↔SPEC vocabulary mapping so the two layers cannot silently drift. The
|
||||
// machine contract uses orthogonal `status` × `verification`; the SPEC table and planner prose
|
||||
// render a flat `covered/dismissed/backstop/unresolved`. The migration is documented in ADR-550
|
||||
// Decision 7a and the reference's "Generic mapping" table, but the SPEC table is rendered by the
|
||||
// LLM workflow (no JS renderer to round-trip against). So this pins the canonical map as code AND
|
||||
// grounds it in every doc surface that renders the vocabulary — a renderer/parser drift fails here.
|
||||
describe('edge-probe machine↔SPEC vocabulary mapping (m2 drift lock)', () => {
|
||||
// The canonical migration map (ADR-550 Decision 7a): machine state → SPEC display label.
|
||||
// `verification` is null for the lifecycle-only states (dismissed/unresolved).
|
||||
const MACHINE_TO_DISPLAY = [
|
||||
{ status: 'resolved', verification: 'explicit', display: 'covered' },
|
||||
{ status: 'resolved', verification: 'backstop', display: 'backstop' },
|
||||
{ status: 'dismissed', verification: null, display: 'dismissed' },
|
||||
{ status: 'unresolved', verification: null, display: 'unresolved' },
|
||||
];
|
||||
const machineKey = (s, v) => `${s}|${v ?? '∅'}`;
|
||||
|
||||
test('the mapping is a bijection — machine→display→machine is identity, no shared labels', () => {
|
||||
const toDisplay = new Map(MACHINE_TO_DISPLAY.map((m) => [machineKey(m.status, m.verification), m.display]));
|
||||
const fromDisplay = new Map(MACHINE_TO_DISPLAY.map((m) => [m.display, machineKey(m.status, m.verification)]));
|
||||
// No two machine states collapse onto the same display label (the silent-drift failure mode).
|
||||
assert.equal(toDisplay.size, MACHINE_TO_DISPLAY.length, 'each machine state must have a distinct key');
|
||||
assert.equal(fromDisplay.size, MACHINE_TO_DISPLAY.length, 'each display label must map back to exactly one machine state');
|
||||
// Round-trip identity.
|
||||
for (const m of MACHINE_TO_DISPLAY) {
|
||||
const key = machineKey(m.status, m.verification);
|
||||
assert.equal(fromDisplay.get(toDisplay.get(key)), key, `${key} must round-trip through its display label`);
|
||||
}
|
||||
});
|
||||
|
||||
test('ADR-550 Decision 7a documents the resolved/explicit↔covered and resolved/backstop↔backstop migration', () => {
|
||||
const adr = fs.readFileSync(adrPath, 'utf8');
|
||||
assert.match(adr, /covered\b[^.]*resolved[^.]*explicit/i, 'ADR must document covered → {resolved, explicit}');
|
||||
assert.match(adr, /backstop\b[^.]*resolved[^.]*backstop/i, 'ADR must document backstop → {resolved, backstop}');
|
||||
assert.match(adr, /count-for-count|count-preserved/i, 'ADR must state coverage.resolved is count-preserved across the migration');
|
||||
});
|
||||
|
||||
test('the SPEC template legend renders exactly the four canonical display labels', () => {
|
||||
const spec = fs.readFileSync(specTemplatePath, 'utf8');
|
||||
for (const { display } of MACHINE_TO_DISPLAY) {
|
||||
assert.match(spec, new RegExp(`\\b${display}\\b`, 'i'), `spec.md Edge Coverage legend must render the "${display}" label`);
|
||||
}
|
||||
});
|
||||
|
||||
test('the reference Generic-mapping table distinguishes the resolved/explicit and resolved/backstop tiers', () => {
|
||||
const md = fs.readFileSync(docPath, 'utf8');
|
||||
assert.match(md, /resolved`?\/`?explicit/i, 'reference must name the resolved/explicit tier in the mapping table');
|
||||
assert.match(md, /resolved`?\/`?backstop/i, 'reference must name the resolved/backstop tier in the mapping table');
|
||||
});
|
||||
});
|
||||
167
tests/edge-probe-planner-contract.test.cjs
Normal file
167
tests/edge-probe-planner-contract.test.cjs
Normal file
@@ -0,0 +1,167 @@
|
||||
// allow-test-rule: runtime-contract-is-the-product — plan-phase.md's planner prompt is the deployed runtime contract under assertion
|
||||
// plan-phase.md is the deployed planning workflow contract; these checks lock
|
||||
// the SPEC path wiring and quality-gate that the edge-probe review (RR-01/02/03)
|
||||
// requires — assertions scope to extracted sub-blocks to avoid false positives.
|
||||
|
||||
'use strict';
|
||||
|
||||
process.env.GSD_TEST_MODE = '1';
|
||||
|
||||
const { test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const PLAN_PHASE_PATH = path.join(__dirname, '..', 'gsd-core', 'workflows', 'plan-phase.md');
|
||||
const PROMPT_PATH = path.join(__dirname, '..', 'gsd-core', 'templates', 'planner-subagent-prompt.md');
|
||||
|
||||
function readPlanPhase() {
|
||||
return fs.readFileSync(PLAN_PHASE_PATH, 'utf8');
|
||||
}
|
||||
|
||||
// Extract the planner <downstream_consumer> block from plan-phase.md. The runtime planner is
|
||||
// spawned from plan-phase.md's own inline <planning_context>/<downstream_consumer>, so the
|
||||
// load-bearing lift instruction must live here — NOT in templates/planner-subagent-prompt.md,
|
||||
// which nothing loads at runtime (no @-import in agents/gsd-planner.md, no read in plan-phase.md).
|
||||
function extractDownstreamConsumerBlock(content) {
|
||||
const start = content.indexOf('<downstream_consumer>');
|
||||
if (start === -1) return '';
|
||||
const end = content.indexOf('</downstream_consumer>', start);
|
||||
if (end === -1) return '';
|
||||
return content.slice(start, end + '</downstream_consumer>'.length);
|
||||
}
|
||||
|
||||
// Extract the planner <files_to_read> block that contains {UI_SPEC_PATH}
|
||||
// There are multiple <files_to_read> blocks in plan-phase.md; we need the one
|
||||
// at ~line 890-912 inside the planning_context markdown block.
|
||||
function extractPlannerFilesBlock(content) {
|
||||
let pos = 0;
|
||||
while (true) {
|
||||
const start = content.indexOf('<files_to_read>', pos);
|
||||
if (start === -1) return '';
|
||||
const end = content.indexOf('</files_to_read>', start);
|
||||
if (end === -1) return '';
|
||||
const block = content.slice(start, end + '</files_to_read>'.length);
|
||||
if (block.includes('{UI_SPEC_PATH}')) {
|
||||
return block;
|
||||
}
|
||||
pos = end + 1;
|
||||
}
|
||||
}
|
||||
|
||||
// Extract the planner <quality_gate> block (the last one in plan-phase.md,
|
||||
// inside the planner prompt template)
|
||||
function extractQualityGateBlock(content) {
|
||||
const start = content.lastIndexOf('<quality_gate>');
|
||||
if (start === -1) return '';
|
||||
const end = content.indexOf('</quality_gate>', start);
|
||||
if (end === -1) return '';
|
||||
return content.slice(start, end + '</quality_gate>'.length);
|
||||
}
|
||||
|
||||
// Test A (RR-01): plan-phase.md resolves a phase *-SPEC.md into SPEC_FILE/SPEC_PATH
|
||||
// Uses new RegExp to correctly match literal $ and ( characters in bash snippets.
|
||||
// This MUST FAIL before the RR-01 fix (no SPEC_PATH resolution exists today)
|
||||
test('RR-01: plan-phase.md resolves phase *-SPEC.md (excluding AI/UI variants) into SPEC_FILE/SPEC_PATH', () => {
|
||||
const content = readPlanPhase();
|
||||
|
||||
// Assert the canonical SPEC_FILE resolution form is present.
|
||||
// new RegExp used so that \$ and \( are treated as literal dollar-sign and open-paren
|
||||
// (JS regex literals interpret \$ as end-anchor and \( as group open).
|
||||
assert.match(
|
||||
content,
|
||||
new RegExp('SPEC_FILE=\\$\\(ls "\\$\\{[A-Z_]*PHASE_DIR[A-Z_]*\\}"[/][*]-SPEC\\.md'),
|
||||
'plan-phase.md must resolve SPEC_FILE using ls "${...PHASE_DIR...}"/*-SPEC.md pattern'
|
||||
);
|
||||
|
||||
// Assert {SPEC_PATH} token appears in the planner files_to_read block
|
||||
const filesBlock = extractPlannerFilesBlock(content);
|
||||
assert.match(
|
||||
filesBlock,
|
||||
/[{]SPEC_PATH[}]/,
|
||||
'The planner <files_to_read> block (containing {UI_SPEC_PATH}) must also contain {SPEC_PATH}'
|
||||
);
|
||||
});
|
||||
|
||||
// Test B (RR-01): The {SPEC_PATH} entry is labelled as carrying the ## Edge Coverage section
|
||||
// This MUST FAIL before the RR-01 fix (no {SPEC_PATH} entry exists today)
|
||||
test('RR-01: {SPEC_PATH} entry in files_to_read is labelled with Edge Coverage', () => {
|
||||
const content = readPlanPhase();
|
||||
const filesBlock = extractPlannerFilesBlock(content);
|
||||
|
||||
assert.match(
|
||||
filesBlock,
|
||||
/[{]SPEC_PATH[}][^\n]*Edge Coverage/,
|
||||
'{SPEC_PATH} entry in planner files_to_read must be labelled as carrying the ## Edge Coverage section'
|
||||
);
|
||||
});
|
||||
|
||||
// Extract a "## " section from its heading until the next "## " heading (or EOF).
|
||||
function extractSection(content, heading) {
|
||||
const start = content.indexOf(heading);
|
||||
if (start === -1) return '';
|
||||
const next = content.indexOf('\n## ', start + heading.length);
|
||||
return content.slice(start, next === -1 ? content.length : next);
|
||||
}
|
||||
|
||||
// Test E (RR-01 REACHABILITY — the assertion that catches the original no-op):
|
||||
// Token presence is not enough. The SPEC resolution must live on an UN-GATED path. §4.5
|
||||
// "Check AI-SPEC" is skipped on every non-AI phase (ai_integration_phase_enabled false /
|
||||
// --skip-ai-spec), so a resolution placed there leaves SPEC_PATH unbound and the planner
|
||||
// never receives the SPEC — exactly the #550 silent no-op. Assert it is NOT in §4.5.
|
||||
test('RR-01 reachability: SPEC_FILE resolution is NOT gated inside the skippable "## 4.5 Check AI-SPEC" section', () => {
|
||||
const content = readPlanPhase();
|
||||
const aiSpecSection = extractSection(content, '## 4.5. Check AI-SPEC');
|
||||
const specResolution = new RegExp('SPEC_FILE=\\$\\(ls "\\$\\{[A-Z_]*PHASE_DIR[A-Z_]*\\}"[/][*]-SPEC\\.md');
|
||||
assert.ok(aiSpecSection.length > 0, 'sanity: the §4.5 Check AI-SPEC section must exist to scope this test');
|
||||
assert.doesNotMatch(
|
||||
aiSpecSection,
|
||||
specResolution,
|
||||
'SPEC_FILE resolution must NOT live inside the skippable §4.5 "Check AI-SPEC" block — gating it there silently starves the planner of the SPEC on non-AI phases (the original #550 no-op)'
|
||||
);
|
||||
assert.match(content, specResolution, 'SPEC_FILE resolution must still exist on an un-gated path elsewhere in plan-phase.md');
|
||||
});
|
||||
|
||||
// Test C (RR-02 consumer end): the lift instruction lives in the RUNTIME planner surface.
|
||||
// plan-phase.md spawns the planner from its own inline <planning_context>; the
|
||||
// templates/planner-subagent-prompt.md file is orphaned (loaded by nothing), so asserting the
|
||||
// contract there is false assurance — the test stays green even if the runtime never consumes it.
|
||||
// Pin the contract to the <downstream_consumer> block plan-phase.md actually sends the planner.
|
||||
test('RR-02 consumer: plan-phase.md downstream_consumer instructs lifting covered/backstop edges into must_haves.truths', () => {
|
||||
const block = extractDownstreamConsumerBlock(readPlanPhase());
|
||||
|
||||
assert.ok(block.length > 0, 'sanity: plan-phase.md must contain a <downstream_consumer> block to scope this test');
|
||||
assert.match(
|
||||
block,
|
||||
/##\s*Edge Coverage/,
|
||||
'plan-phase.md <downstream_consumer> must reference ## Edge Coverage'
|
||||
);
|
||||
assert.match(
|
||||
block,
|
||||
/must_haves\.truths/,
|
||||
'plan-phase.md <downstream_consumer> must reference must_haves.truths as the lift target'
|
||||
);
|
||||
|
||||
// Regression guard against re-orphaning: the lift instruction must NOT be relocated back into
|
||||
// templates/planner-subagent-prompt.md (which nothing loads) and presented as the consumer
|
||||
// contract — that is exactly the false-assurance this retarget fixes.
|
||||
const prompt = fs.readFileSync(PROMPT_PATH, 'utf8');
|
||||
assert.doesNotMatch(
|
||||
prompt,
|
||||
/Edge Coverage/,
|
||||
'planner-subagent-prompt.md is not loaded at runtime — the Edge Coverage lift instruction must not live there'
|
||||
);
|
||||
});
|
||||
|
||||
// Test D (RR-03): plan-phase.md <quality_gate> contains a covered/backstop ↔ must_haves item
|
||||
// This MUST FAIL before the RR-03 fix (no such quality_gate item exists today)
|
||||
test('RR-03: planner quality_gate requires covered/backstop edges represented in must_haves', () => {
|
||||
const content = readPlanPhase();
|
||||
const qgBlock = extractQualityGateBlock(content);
|
||||
|
||||
assert.match(
|
||||
qgBlock,
|
||||
/covered.*backstop.*must_haves|backstop.*covered.*must_haves/i,
|
||||
'planner quality_gate must contain a checklist item tying covered/backstop edges to must_haves'
|
||||
);
|
||||
});
|
||||
193
tests/edge-probe-spec-phase-contract.test.cjs
Normal file
193
tests/edge-probe-spec-phase-contract.test.cjs
Normal file
@@ -0,0 +1,193 @@
|
||||
// allow-test-rule: runtime-contract-is-the-product — spec-phase.md Step 5.5 is the deployed workflow runtime contract under assertion
|
||||
// spec-phase.md is the deployed spec workflow contract; these checks lock
|
||||
// the Step 5.5 wiring so the edge-probe.cjs runtime invocation cannot
|
||||
// silently rot the way the original plan-phase no-op did (reviewer finding RR-11).
|
||||
// Assertions scope to the extracted Step 5.5 block to avoid false positives
|
||||
// from incidental mentions elsewhere in the file.
|
||||
|
||||
'use strict';
|
||||
|
||||
process.env.GSD_TEST_MODE = '1';
|
||||
|
||||
const { test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const SPEC_PHASE_PATH = path.join(__dirname, '..', 'gsd-core', 'workflows', 'spec-phase.md');
|
||||
|
||||
function readSpecPhase() {
|
||||
return fs.readFileSync(SPEC_PHASE_PATH, 'utf8');
|
||||
}
|
||||
|
||||
// Slice the Step 5.5 block: from the "Step 5.5" heading to the next "## " or "Step " heading.
|
||||
// This scopes assertions to Step 5.5 only, preventing false positives from mentions elsewhere.
|
||||
function extractStep55Block(content) {
|
||||
const startIdx = content.indexOf('## Step 5.5');
|
||||
if (startIdx === -1) {
|
||||
// Also try without the ## prefix
|
||||
const altIdx = content.indexOf('Step 5.5');
|
||||
if (altIdx === -1) return '';
|
||||
// Find end: next heading starting with ## or Step N (not Step 5.5)
|
||||
const rest = content.slice(altIdx + 'Step 5.5'.length);
|
||||
const nextHeading = rest.search(/\n## |\nStep \d/);
|
||||
if (nextHeading === -1) return content.slice(altIdx);
|
||||
return content.slice(altIdx, altIdx + 'Step 5.5'.length + nextHeading);
|
||||
}
|
||||
const rest = content.slice(startIdx + '## Step 5.5'.length);
|
||||
const nextHeading = rest.search(/\n## /);
|
||||
if (nextHeading === -1) return content.slice(startIdx);
|
||||
return content.slice(startIdx, startIdx + '## Step 5.5'.length + nextHeading);
|
||||
}
|
||||
|
||||
// Test A (RR-11): Step 5.5 resolves and invokes edge-probe.cjs via node.
|
||||
// MUST FAIL before the RR-04 wire (Step 5.5 is prose-only today — no CLI invocation).
|
||||
test('RR-11: spec-phase Step 5.5 resolves edge-probe.cjs via path-fallback loop', () => {
|
||||
const content = readSpecPhase();
|
||||
const block = extractStep55Block(content);
|
||||
|
||||
assert.ok(block.length > 0, 'Step 5.5 block must be extractable from spec-phase.md');
|
||||
|
||||
// Assert the path-fallback resolution loop for edge-probe.cjs is present in Step 5.5.
|
||||
// The token "edge-probe.cjs" must appear inside the block (the artifact being resolved).
|
||||
assert.match(
|
||||
block,
|
||||
/edge-probe\.cjs/,
|
||||
'Step 5.5 must reference edge-probe.cjs as the artifact being resolved'
|
||||
);
|
||||
|
||||
// Assert node invocation of edge-probe.cjs in Step 5.5.
|
||||
// Matches: node "$EDGE_PROBE_JS" or node ... edge-probe.cjs
|
||||
assert.match(
|
||||
block,
|
||||
/node\s+["$].*[Ee][Dd][Gg][Ee][-_][Pp][Rr][Oo][Bb][Ee]/,
|
||||
'Step 5.5 must invoke edge-probe.cjs via node (e.g. node "$EDGE_PROBE_JS" ...)'
|
||||
);
|
||||
});
|
||||
|
||||
// Test C (RR-11 FUNCTION — the assertion that catches "decorative bash"):
|
||||
// Token presence is not enough. The invocation is a no-op unless $REQS_JSON is actually
|
||||
// POPULATED before the engine runs (the original block only mktemp'd it + left a comment,
|
||||
// so the CLI parsed an empty file). Assert the block (a) writes $REQS_JSON via a redirect,
|
||||
// (b) does so BEFORE the node invocation, and (c) guards against an empty/invalid file.
|
||||
test('RR-11 function: Step 5.5 writes $REQS_JSON before invoking, and guards against empty input', () => {
|
||||
const content = readSpecPhase();
|
||||
const block = extractStep55Block(content);
|
||||
assert.ok(block.length > 0, 'Step 5.5 block must be extractable from spec-phase.md');
|
||||
|
||||
// (a) A redirect that writes the requirements into $REQS_JSON (e.g. `cat > "$REQS_JSON"`).
|
||||
const writeIdx = block.search(/>\s*"\$REQS_JSON"/);
|
||||
assert.ok(
|
||||
writeIdx !== -1,
|
||||
'Step 5.5 must WRITE requirements into $REQS_JSON (a redirect like `cat > "$REQS_JSON"`), not just mktemp it — an empty file makes the probe a silent no-op'
|
||||
);
|
||||
|
||||
// (b) The write must precede the node invocation of the engine.
|
||||
const invokeIdx = block.search(/node\s+["$].*[Ee][Dd][Gg][Ee][-_][Pp][Rr][Oo][Bb][Ee]/);
|
||||
assert.ok(invokeIdx !== -1, 'Step 5.5 must invoke the engine via node');
|
||||
assert.ok(
|
||||
writeIdx < invokeIdx,
|
||||
'Step 5.5 must populate $REQS_JSON BEFORE invoking edge-probe.cjs (write precedes the node call)'
|
||||
);
|
||||
|
||||
// (c) A guard that refuses to run on an empty/invalid requirements array.
|
||||
assert.match(
|
||||
block,
|
||||
/Array\.isArray|empty\/invalid|empty or invalid|REQS_JSON[^\n]*empty/i,
|
||||
'Step 5.5 must guard against an empty/invalid $REQS_JSON before invoking (fail loud, not silent no-op)'
|
||||
);
|
||||
});
|
||||
|
||||
// Test B (RR-11): Step 5.5 has an explicit not-found branch — build:lib or error token.
|
||||
// MUST FAIL before the RR-04 wire (no not-found handling today).
|
||||
test('RR-11: spec-phase Step 5.5 has an explicit not-found branch (build:lib or blocking error)', () => {
|
||||
const content = readSpecPhase();
|
||||
const block = extractStep55Block(content);
|
||||
|
||||
assert.ok(block.length > 0, 'Step 5.5 block must be extractable from spec-phase.md');
|
||||
|
||||
// Assert either a build:lib invocation or an explicit "not found" / error message exists.
|
||||
// This prevents the wire from being added as a silent-skip with no fallback.
|
||||
assert.match(
|
||||
block,
|
||||
/build:lib|not found|ERROR.*edge-probe|edge-probe.*not found/i,
|
||||
'Step 5.5 must have an explicit not-found branch (build:lib attempt or clear blocking error)'
|
||||
);
|
||||
});
|
||||
|
||||
// Test D (review High): the build fallback must NEVER run the CONSUMING project's package
|
||||
// scripts. Every executable `build:lib` invocation must be pinned to the GSD dir with
|
||||
// `npm --prefix`, and the build must be gated behind a verified GSD source checkout. A bare
|
||||
// `npm run build:lib` (no --prefix) uses cwd — which, under the git-toplevel fallback, is the
|
||||
// consumer repo — and would execute its codegen/migrations during a spec workflow.
|
||||
test('review High: Step 5.5 build:lib is --prefix-pinned to the GSD dir and gated on a source checkout', () => {
|
||||
const content = readSpecPhase();
|
||||
const block = extractStep55Block(content);
|
||||
assert.ok(block.length > 0, 'Step 5.5 block must be extractable from spec-phase.md');
|
||||
|
||||
// Collect lines that actually INVOKE npm (trimmed start === "npm"), excluding echo/comment
|
||||
// mentions (e.g. the error message that quotes `npm run build:lib` for the user).
|
||||
const buildInvocations = block
|
||||
.split('\n')
|
||||
.filter((l) => l.trim().startsWith('npm') && l.includes('build:lib'));
|
||||
|
||||
assert.ok(
|
||||
buildInvocations.length > 0,
|
||||
'Step 5.5 must contain at least one npm build:lib invocation (the dev-checkout fallback)'
|
||||
);
|
||||
for (const line of buildInvocations) {
|
||||
assert.match(
|
||||
line,
|
||||
/npm\s+--prefix\s+"?\$?\{?_?GSD_RT/,
|
||||
`build:lib must be pinned with \`npm --prefix "$_GSD_RT"\` so it never runs the consuming project's scripts — offending line: ${line.trim()}`
|
||||
);
|
||||
}
|
||||
|
||||
// The build must be gated behind a verified GSD source checkout (tsconfig.build.json present),
|
||||
// so it cannot fire inside a plain consumer repo where the artifact merely happens to be absent.
|
||||
assert.match(
|
||||
block,
|
||||
/tsconfig\.build\.json/,
|
||||
'Step 5.5 must gate the build behind a GSD source checkout (e.g. test -f "$_GSD_RT/tsconfig.build.json")'
|
||||
);
|
||||
});
|
||||
|
||||
// Test E (review #4 High): the engine's fail-closed exit(2) must NOT be swallowed by command
|
||||
// substitution. A bare `COVERAGE=$(node "$EDGE_PROBE_JS" ...)` discards the exit status, leaves
|
||||
// $COVERAGE empty on an invalid-shapes failure, and lets the workflow proceed into prose
|
||||
// re-derivation — fail-OPEN at the very boundary the engine validation exists to protect.
|
||||
test('review #4 High: Step 5.5 exit-checks the engine capture and validates the report (no fail-open)', () => {
|
||||
const content = readSpecPhase();
|
||||
const block = extractStep55Block(content);
|
||||
assert.ok(block.length > 0, 'Step 5.5 block must be extractable from spec-phase.md');
|
||||
|
||||
// The engine capture must be FATAL — guarded by `if ! COVERAGE=$(node "$EDGE_PROBE_JS" …)`
|
||||
// (or an explicit exit-status check) that exits non-zero on failure.
|
||||
assert.match(
|
||||
block,
|
||||
/if\s+!\s+COVERAGE=\$\(node\s+"\$EDGE_PROBE_JS"/,
|
||||
'Step 5.5 must exit-check the engine invocation (e.g. `if ! COVERAGE=$(node "$EDGE_PROBE_JS" …)`) — a bare command substitution swallows the engine exit code and fails open'
|
||||
);
|
||||
|
||||
// And the captured report must be validated as JSON before the resolution loop consumes it
|
||||
// (guards against an exit-0-but-garbage capture).
|
||||
assert.match(
|
||||
block,
|
||||
/(COVERAGE[\s\S]{0,500}JSON\.parse)|(JSON\.parse[\s\S]{0,500}\$COVERAGE)/,
|
||||
'Step 5.5 must validate $COVERAGE parses as JSON before use (guard against exit-0-but-malformed output)'
|
||||
);
|
||||
});
|
||||
|
||||
// Test F (adversarial review): a report with ZERO applicable edges across all requirements is
|
||||
// the likely-classification-miss fail-open (same shape as an invalid shape yielding applicable:0).
|
||||
// Step 5.5 must surface it, not silently emit a green empty ## Edge Coverage section.
|
||||
test('adversarial review: Step 5.5 guards a zero-applicable coverage report', () => {
|
||||
const content = readSpecPhase();
|
||||
const block = extractStep55Block(content);
|
||||
assert.ok(block.length > 0, 'Step 5.5 block must be extractable from spec-phase.md');
|
||||
assert.match(
|
||||
block,
|
||||
/coverage\.applicable/,
|
||||
'Step 5.5 must read coverage.applicable and guard the zero-applicable case (warn/confirm, not silently proceed)'
|
||||
);
|
||||
});
|
||||
441
tests/edge-probe.test.cjs
Normal file
441
tests/edge-probe.test.cjs
Normal file
@@ -0,0 +1,441 @@
|
||||
/**
|
||||
* Edge-probe reference core unit tests.
|
||||
*
|
||||
* Asserts the LOCKED export surface of the spec-completeness edge-probe against
|
||||
* the BUILT artifact (`gsd-core/bin/lib/edge-probe.cjs`), which
|
||||
* `npm run build:lib` (run by pretest) emits from `src/edge-probe.cts`.
|
||||
*
|
||||
* Post ADR-550 Decision 7: the generic resolution model lives in `probe-core`;
|
||||
* edge-probe is its first adapter (shapes/TAXONOMY/proposeEdges + the
|
||||
* `{explicit, backstop}` verification validators). The resolution model is the
|
||||
* status×verification re-cut: `status: resolved | dismissed | unresolved` ×
|
||||
* `verification: explicit | backstop`. `covered`/`backstop` are no longer status
|
||||
* values — `covered → {resolved, explicit}`, `backstop → {resolved, backstop}`.
|
||||
*/
|
||||
'use strict';
|
||||
process.env.GSD_TEST_MODE = '1';
|
||||
|
||||
const { test, describe } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const path = require('node:path');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const { execFileSync, spawnSync } = require('node:child_process');
|
||||
const { cleanup } = require('./helpers.cjs');
|
||||
|
||||
const BUILT_SCRIPT = path.join(__dirname, '..', 'gsd-core', 'bin', 'lib', 'edge-probe.cjs');
|
||||
const ep = require(BUILT_SCRIPT);
|
||||
|
||||
describe('edge-probe: classifyShape', () => {
|
||||
test('detects numeric-range from rounding/threshold cues', () => {
|
||||
assert.deepEqual(ep.classifyShape('Round a number to N decimal places').sort(),
|
||||
['numeric-range']);
|
||||
});
|
||||
test('detects collection from interval/merge cues', () => {
|
||||
const shapes = ep.classifyShape('Merge a list of overlapping intervals');
|
||||
assert.ok(shapes.includes('collection'));
|
||||
});
|
||||
test('detects text from truncate/string cues', () => {
|
||||
const shapes = ep.classifyShape('Truncate a string to a maximum length');
|
||||
assert.ok(shapes.includes('text'));
|
||||
});
|
||||
test('returns [] when no cue matches', () => {
|
||||
assert.deepEqual(ep.classifyShape('Display the company logo'), []);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: TAXONOMY + applicableCategories', () => {
|
||||
test('TAXONOMY has the 8 documented categories in order', () => {
|
||||
assert.deepEqual(ep.TAXONOMY.map((c) => c.id),
|
||||
['boundary', 'adjacency', 'empty', 'encoding', 'ordering', 'precision', 'idempotency', 'concurrency']);
|
||||
});
|
||||
test('every category has name, shapes[], probe', () => {
|
||||
for (const c of ep.TAXONOMY) {
|
||||
assert.equal(typeof c.name, 'string');
|
||||
assert.ok(Array.isArray(c.shapes) && c.shapes.length >= 1);
|
||||
assert.equal(typeof c.probe, 'string');
|
||||
}
|
||||
});
|
||||
test('numeric-range raises boundary + precision only', () => {
|
||||
assert.deepEqual(ep.applicableCategories(['numeric-range']).sort(),
|
||||
['boundary', 'precision']);
|
||||
});
|
||||
test('collection raises adjacency, empty, ordering', () => {
|
||||
assert.deepEqual(ep.applicableCategories(['collection']).sort(),
|
||||
['adjacency', 'empty', 'ordering']);
|
||||
});
|
||||
test('text raises empty + encoding', () => {
|
||||
assert.deepEqual(ep.applicableCategories(['text']).sort(),
|
||||
['empty', 'encoding']);
|
||||
});
|
||||
test('no shapes raises nothing', () => {
|
||||
assert.deepEqual(ep.applicableCategories([]), []);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: proposeEdges', () => {
|
||||
test('rounding requirement proposes boundary + precision, all unresolved (verification null)', () => {
|
||||
const edges = ep.proposeEdges({ id: 'R1', text: 'Round a number to N decimal places' });
|
||||
assert.deepEqual(edges.map((e) => e.category).sort(), ['boundary', 'precision']);
|
||||
for (const e of edges) {
|
||||
assert.equal(e.requirement_id, 'R1');
|
||||
assert.equal(e.status, 'unresolved');
|
||||
assert.equal(e.verification, null);
|
||||
assert.equal(e.resolution, null);
|
||||
assert.equal(e.reason, null);
|
||||
assert.equal(typeof e.probe, 'string');
|
||||
}
|
||||
});
|
||||
test('authored shapes override prose classification', () => {
|
||||
const edges = ep.proposeEdges({ id: 'R9', text: 'opaque label', shapes: ['collection'] });
|
||||
assert.deepEqual(edges.map((e) => e.category).sort(), ['adjacency', 'empty', 'ordering']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: validateResolution', () => {
|
||||
test('rejects an unknown status', () => {
|
||||
assert.throws(() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'maybe' }),
|
||||
/invalid status/i);
|
||||
});
|
||||
test('rejects a former covered status (re-cut: covered is no longer a status)', () => {
|
||||
assert.throws(() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'covered', resolution: 'x' }),
|
||||
/invalid status/i);
|
||||
});
|
||||
test('rejects dismissed without a reason', () => {
|
||||
assert.throws(() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'dismissed', reason: '' }),
|
||||
/dismissed requires a reason/i);
|
||||
});
|
||||
test('accepts dismissed with a reason', () => {
|
||||
assert.equal(ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'dismissed', reason: 'bounded enum' }), true);
|
||||
});
|
||||
test('rejects resolved with a missing verification tier', () => {
|
||||
assert.throws(() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'resolved', resolution: 'AC' }),
|
||||
/verification/i);
|
||||
});
|
||||
test('rejects resolved with a verification tier outside {explicit, backstop}', () => {
|
||||
assert.throws(() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'resolved', verification: 'judgment', resolution: 'AC' }),
|
||||
/invalid verification/i);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: analyzeCoverage', () => {
|
||||
const reqs = [{ id: 'R1', text: 'Merge a list of overlapping intervals' }];
|
||||
test('with no resolutions, every applicable edge is unresolved (byVerification zeroed)', () => {
|
||||
const rep = ep.analyzeCoverage(reqs, []);
|
||||
assert.deepEqual(rep.coverage, { applicable: 3, resolved: 0, unresolved: 3, byVerification: { explicit: 0, backstop: 0 } });
|
||||
});
|
||||
test('merges a resolved/explicit resolution and counts it resolved', () => {
|
||||
const rep = ep.analyzeCoverage(reqs, [
|
||||
{ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: 'AC#6: touching intervals merge' },
|
||||
]);
|
||||
const adj = rep.items.find((i) => i.category === 'adjacency');
|
||||
assert.equal(adj.status, 'resolved');
|
||||
assert.equal(adj.verification, 'explicit');
|
||||
assert.equal(adj.resolution, 'AC#6: touching intervals merge');
|
||||
assert.equal(rep.coverage.resolved, 1);
|
||||
assert.equal(rep.coverage.unresolved, 2);
|
||||
assert.deepEqual(rep.coverage.byVerification, { explicit: 1, backstop: 0 });
|
||||
});
|
||||
test('throws if a resolution is invalid (dismissed w/o reason)', () => {
|
||||
assert.throws(() => ep.analyzeCoverage(reqs, [
|
||||
{ requirement_id: 'R1', category: 'empty', status: 'dismissed' },
|
||||
]), /dismissed requires a reason/i);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: CLI (built artifact)', () => {
|
||||
test('reads a requirements file and prints a coverage report as JSON', () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'edge-probe-'));
|
||||
const reqPath = path.join(dir, 'requirements.json');
|
||||
fs.writeFileSync(reqPath, JSON.stringify([{ id: 'R1', text: 'Round a number to N decimal places' }]));
|
||||
const out = execFileSync('node', [BUILT_SCRIPT, reqPath], { encoding: 'utf8' });
|
||||
const rep = JSON.parse(out);
|
||||
assert.deepEqual(rep.coverage, { applicable: 2, resolved: 0, unresolved: 2, byVerification: { explicit: 0, backstop: 0 } });
|
||||
});
|
||||
test('with no args exits with status 2 (assert on exit code, not stderr prose)', () => {
|
||||
let status;
|
||||
try {
|
||||
execFileSync('node', [BUILT_SCRIPT], { stdio: 'pipe' });
|
||||
status = 0;
|
||||
} catch (error) {
|
||||
status = error.status;
|
||||
}
|
||||
assert.equal(status, 2);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: CLI JSON.parse error handling (RR-10)', () => {
|
||||
test('invalid requirements JSON exits with status 2 (handled error, not uncaught throw)', () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'edge-probe-rr10-'));
|
||||
const badJson = path.join(dir, 'bad-req.json');
|
||||
fs.writeFileSync(badJson, 'not valid json {{{');
|
||||
try {
|
||||
const r = spawnSync(process.execPath, [BUILT_SCRIPT, badJson], { stdio: 'pipe' });
|
||||
assert.equal(r.status, 2);
|
||||
} finally {
|
||||
cleanup(dir);
|
||||
}
|
||||
});
|
||||
test('invalid resolutions JSON exits with status 2 (handled error, not uncaught throw)', () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'edge-probe-rr10-'));
|
||||
const goodReq = path.join(dir, 'req.json');
|
||||
const badRes = path.join(dir, 'bad-res.json');
|
||||
fs.writeFileSync(goodReq, JSON.stringify([{ id: 'R1', text: 'Round a number to N decimal places' }]));
|
||||
fs.writeFileSync(badRes, 'not valid json {{{');
|
||||
try {
|
||||
const r = spawnSync(process.execPath, [BUILT_SCRIPT, goodReq, badRes], { stdio: 'pipe' });
|
||||
assert.equal(r.status, 2);
|
||||
} finally {
|
||||
cleanup(dir);
|
||||
}
|
||||
});
|
||||
test('valid requirements file exits 0 and stdout is parseable JSON', () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'edge-probe-rr10-'));
|
||||
const reqPath = path.join(dir, 'req.json');
|
||||
fs.writeFileSync(reqPath, JSON.stringify([{ id: 'R1', text: 'Round a number to N decimal places' }]));
|
||||
try {
|
||||
const r = spawnSync(process.execPath, [BUILT_SCRIPT, reqPath], { stdio: 'pipe', encoding: 'utf8' });
|
||||
assert.equal(r.status, 0);
|
||||
const rep = JSON.parse(r.stdout);
|
||||
assert.deepEqual(rep.coverage, { applicable: 2, resolved: 0, unresolved: 2, byVerification: { explicit: 0, backstop: 0 } });
|
||||
} finally {
|
||||
cleanup(dir);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: proposeEdges — empty-shapes override (RR-06)', () => {
|
||||
test('shapes: [] returns zero edges (explicit empty-shapes override)', () => {
|
||||
const edges = ep.proposeEdges({ id: 'R1', text: 'merge intervals', shapes: [] });
|
||||
assert.deepEqual(edges, []);
|
||||
});
|
||||
test('absent shapes key classifies from prose (no override)', () => {
|
||||
const edges = ep.proposeEdges({ id: 'R1', text: 'merge intervals' });
|
||||
assert.ok(edges.length > 0, 'should classify collection edges from prose');
|
||||
});
|
||||
test('shapes: [collection] overrides prose and proposes collection categories', () => {
|
||||
const edges = ep.proposeEdges({ id: 'R9', text: 'opaque text with no cues', shapes: ['collection'] });
|
||||
assert.deepEqual(edges.map((e) => e.category).sort(), ['adjacency', 'empty', 'ordering']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: proposeEdges — invalid authored shapes fail closed (re-review #3 High)', () => {
|
||||
// A non-empty but INVALID shapes array must NOT silently suppress every probe.
|
||||
// shapes:['numeric'] (typo for the locked 'numeric-range') previously passed
|
||||
// Array.isArray, matched no category, and returned applicable:0 — failing OPEN.
|
||||
test('rejects an unknown shape value (typo for a locked shape)', () => {
|
||||
assert.throws(
|
||||
() => ep.proposeEdges({ id: 'R1', text: 'Round a number', shapes: ['numeric'] }),
|
||||
/invalid shape/i,
|
||||
);
|
||||
});
|
||||
test('rejects a mixed array where one entry is invalid', () => {
|
||||
assert.throws(
|
||||
() => ep.proposeEdges({ id: 'R1', text: 'Round a number', shapes: ['numeric-range', 'bogus'] }),
|
||||
/invalid shape/i,
|
||||
);
|
||||
});
|
||||
test('rejects a non-string shape entry', () => {
|
||||
assert.throws(
|
||||
() => ep.proposeEdges({ id: 'R1', text: 'Round a number', shapes: [42] }),
|
||||
/invalid shape/i,
|
||||
);
|
||||
});
|
||||
test('analyzeCoverage propagates the invalid-shape throw', () => {
|
||||
assert.throws(
|
||||
() => ep.analyzeCoverage([{ id: 'R1', text: 'Round a number', shapes: ['numeric'] }]),
|
||||
/invalid shape/i,
|
||||
);
|
||||
});
|
||||
test('CLI exits 2 (handled) on an invalid authored shape, not an uncaught trace', () => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'edge-probe-shape-'));
|
||||
const reqPath = path.join(dir, 'req.json');
|
||||
fs.writeFileSync(reqPath, JSON.stringify([{ id: 'R1', text: 'Round a number', shapes: ['numeric'] }]));
|
||||
try {
|
||||
const r = spawnSync(process.execPath, [BUILT_SCRIPT, reqPath], { stdio: 'pipe' });
|
||||
assert.equal(r.status, 2);
|
||||
} finally {
|
||||
cleanup(dir);
|
||||
}
|
||||
});
|
||||
test('a valid locked shape still proposes its categories (no false rejection)', () => {
|
||||
const edges = ep.proposeEdges({ id: 'R1', text: 'opaque', shapes: ['numeric-range'] });
|
||||
assert.deepEqual(edges.map((e) => e.category).sort(), ['boundary', 'precision']);
|
||||
});
|
||||
test('shapes: [] remains a valid zero-edge override (RR-06 intact)', () => {
|
||||
assert.deepEqual(ep.proposeEdges({ id: 'R1', text: 'merge intervals', shapes: [] }), []);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: input validation & orphan-resolution rejection (adversarial review)', () => {
|
||||
// HIGH: a resolution whose (requirement_id, category) matches no proposed edge — a typo'd
|
||||
// category or a non-applicable one — was silently DROPPED, so an author who typos `precison`
|
||||
// sees the precision edge as still-unresolved with no error (a confirmed money-rounding exploit).
|
||||
test('rejects an orphan resolution (typo category — no matching proposed edge)', () => {
|
||||
assert.throws(
|
||||
() => ep.analyzeCoverage(
|
||||
[{ id: 'R1', text: 'Round a number to N decimal places' }],
|
||||
[{ requirement_id: 'R1', category: 'precison', status: 'resolved', verification: 'explicit', resolution: 'AC: precision handled' }],
|
||||
),
|
||||
/unknown resolution|no matching proposed edge/i,
|
||||
);
|
||||
});
|
||||
test('rejects a resolution for a valid-but-non-applicable category', () => {
|
||||
// 'encoding' is a real taxonomy id but applies to text, not the numeric-range requirement.
|
||||
assert.throws(
|
||||
() => ep.analyzeCoverage(
|
||||
[{ id: 'R1', text: 'Round a number to N decimal places' }],
|
||||
[{ requirement_id: 'R1', category: 'encoding', status: 'resolved', verification: 'explicit', resolution: 'AC' }],
|
||||
),
|
||||
/unknown resolution|no matching proposed edge/i,
|
||||
);
|
||||
});
|
||||
test('a matching resolution still resolves (no false orphan rejection)', () => {
|
||||
const rep = ep.analyzeCoverage(
|
||||
[{ id: 'R1', text: 'Round a number to N decimal places' }],
|
||||
[{ requirement_id: 'R1', category: 'precision', status: 'resolved', verification: 'explicit', resolution: 'AC: precision tested' }],
|
||||
);
|
||||
assert.equal(rep.coverage.resolved, 1);
|
||||
});
|
||||
test('rejects requirements that is not an array', () => {
|
||||
assert.throws(() => ep.analyzeCoverage('nope'), /requirements must be an array/i);
|
||||
});
|
||||
test('rejects a duplicate requirement id', () => {
|
||||
assert.throws(
|
||||
() => ep.analyzeCoverage([{ id: 'R1', text: 'a' }, { id: 'R1', text: 'b' }]),
|
||||
/duplicate requirement/i,
|
||||
);
|
||||
});
|
||||
test('rejects a truthy non-array shapes (string instead of array)', () => {
|
||||
// A bare string `shapes: "numeric-range"` previously fell through to prose classification,
|
||||
// silently ignoring the authored override instead of honoring or rejecting it.
|
||||
assert.throws(
|
||||
() => ep.proposeEdges({ id: 'R1', text: 'x', shapes: 'numeric-range' }),
|
||||
/shapes must be an array/i,
|
||||
);
|
||||
});
|
||||
test('rejects a missing requirement id', () => {
|
||||
assert.throws(() => ep.proposeEdges({ text: 'x' }), /requirement id must be a non-empty string/i);
|
||||
});
|
||||
test('rejects an empty requirement id', () => {
|
||||
assert.throws(() => ep.proposeEdges({ id: ' ', text: 'x' }), /requirement id must be a non-empty string/i);
|
||||
});
|
||||
test('rejects a non-string requirement text', () => {
|
||||
assert.throws(() => ep.proposeEdges({ id: 'R1', text: 42 }), /text must be a string/i);
|
||||
});
|
||||
test('rejects a missing requirement text when no shapes override (M2 fail-open)', () => {
|
||||
// Without text or an authored shape, prose classification yields zero shapes → zero edges →
|
||||
// the requirement is silently DROPPED from coverage with no signal — the exact fail-open this
|
||||
// feature exists to eliminate. The edge adapter's `text` is required, so reject it.
|
||||
assert.throws(
|
||||
() => ep.proposeEdges({ id: 'R1' }),
|
||||
/text must be a non-empty string when no shapes override/i,
|
||||
);
|
||||
assert.throws(
|
||||
() => ep.analyzeCoverage([{ id: 'R1' }]),
|
||||
/text must be a non-empty string when no shapes override/i,
|
||||
);
|
||||
});
|
||||
test('rejects an empty/whitespace requirement text when no shapes override (M2)', () => {
|
||||
assert.throws(() => ep.proposeEdges({ id: 'R1', text: '' }), /text must be a non-empty string when no shapes override/i);
|
||||
assert.throws(() => ep.proposeEdges({ id: 'R1', text: ' ' }), /text must be a non-empty string when no shapes override/i);
|
||||
});
|
||||
test('allows missing/empty text WHEN an explicit shapes override is provided (M2 legitimate path)', () => {
|
||||
// An authored `shapes` array (including `[]` for "no applicable categories") opts out of prose
|
||||
// classification, so `text` is not required — this must remain valid.
|
||||
assert.deepEqual(ep.proposeEdges({ id: 'R1', shapes: [] }), []);
|
||||
const edges = ep.proposeEdges({ id: 'R1', shapes: ['numeric-range'] });
|
||||
assert.ok(edges.length > 0, 'an explicit shape override must still propose edges without text');
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: validateResolution — explicit-needs-resolution (RR-07, re-cut)', () => {
|
||||
test('rejects resolved/explicit with empty resolution string', () => {
|
||||
assert.throws(
|
||||
() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'resolved', verification: 'explicit', resolution: '' }),
|
||||
/explicit requires a resolution/i,
|
||||
);
|
||||
});
|
||||
test('rejects resolved/explicit with whitespace-only resolution', () => {
|
||||
assert.throws(
|
||||
() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'resolved', verification: 'explicit', resolution: ' ' }),
|
||||
/explicit requires a resolution/i,
|
||||
);
|
||||
});
|
||||
test('rejects resolved/explicit with missing resolution', () => {
|
||||
assert.throws(
|
||||
() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'resolved', verification: 'explicit' }),
|
||||
/explicit requires a resolution/i,
|
||||
);
|
||||
});
|
||||
test('accepts resolved/explicit with a non-empty resolution', () => {
|
||||
assert.equal(
|
||||
ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'resolved', verification: 'explicit', resolution: 'AC#3: boundary tested in suite' }),
|
||||
true,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: validateResolution — backstop-needs-resolution (RR-07 follow-up, re-cut)', () => {
|
||||
test('rejects resolved/backstop with empty resolution string', () => {
|
||||
assert.throws(
|
||||
() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'resolved', verification: 'backstop', resolution: '' }),
|
||||
/backstop requires a resolution/i,
|
||||
);
|
||||
});
|
||||
test('rejects resolved/backstop with whitespace-only resolution', () => {
|
||||
assert.throws(
|
||||
() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'resolved', verification: 'backstop', resolution: ' ' }),
|
||||
/backstop requires a resolution/i,
|
||||
);
|
||||
});
|
||||
test('rejects resolved/backstop with missing resolution', () => {
|
||||
assert.throws(
|
||||
() => ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'resolved', verification: 'backstop' }),
|
||||
/backstop requires a resolution/i,
|
||||
);
|
||||
});
|
||||
test('accepts resolved/backstop with a non-empty resolution note', () => {
|
||||
assert.equal(
|
||||
ep.validateResolution({ requirement_id: 'R1', category: 'boundary', status: 'resolved', verification: 'backstop', resolution: 'held-out: covered by integration fuzz suite' }),
|
||||
true,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: analyzeCoverage — duplicate rejection (RR-09)', () => {
|
||||
const reqs = [{ id: 'R1', text: 'Merge a list of overlapping intervals' }];
|
||||
test('rejects duplicate (requirement_id, category) resolution', () => {
|
||||
assert.throws(
|
||||
() => ep.analyzeCoverage(reqs, [
|
||||
{ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: 'AC#1' },
|
||||
{ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: 'AC#2' },
|
||||
]),
|
||||
/duplicate resolution/i,
|
||||
);
|
||||
});
|
||||
test('distinct pairs still analyze without throwing', () => {
|
||||
// mirrors fixture 06-resolved-mixed
|
||||
assert.doesNotThrow(() => ep.analyzeCoverage(reqs, [
|
||||
{ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: 'AC#6: touching intervals merge' },
|
||||
{ requirement_id: 'R1', category: 'ordering', status: 'dismissed', resolution: null, reason: 'output is canonically sorted; no tie possible' },
|
||||
]));
|
||||
});
|
||||
});
|
||||
|
||||
describe('edge-probe: golden fixtures', () => {
|
||||
const root = path.join(__dirname, '..', 'gsd-core', 'references', 'edge-probe-fixtures');
|
||||
const fixtures = fs.readdirSync(root).filter((d) =>
|
||||
fs.statSync(path.join(root, d)).isDirectory());
|
||||
assert.ok(fixtures.length >= 6, 'expected at least 6 fixtures');
|
||||
for (const name of fixtures) {
|
||||
test(`fixture ${name} matches its golden coverage`, () => {
|
||||
const dir = path.join(root, name);
|
||||
const reqs = JSON.parse(fs.readFileSync(path.join(dir, 'requirements.json'), 'utf8'));
|
||||
const resPath = path.join(dir, 'resolutions.json');
|
||||
const res = fs.existsSync(resPath) ? JSON.parse(fs.readFileSync(resPath, 'utf8')) : [];
|
||||
const expected = JSON.parse(fs.readFileSync(path.join(dir, 'expected-coverage.json'), 'utf8'));
|
||||
assert.deepEqual(ep.analyzeCoverage(reqs, res), expected);
|
||||
});
|
||||
}
|
||||
});
|
||||
167
tests/probe-core.property.test.cjs
Normal file
167
tests/probe-core.property.test.cjs
Normal file
@@ -0,0 +1,167 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Property-based tests for probe-core.cjs (ADR-550 Decision 7).
|
||||
*
|
||||
* Module: gsd-core/bin/lib/probe-core.cjs (generated from src/probe-core.cts)
|
||||
* Exercised: analyzeCoverage(items, resolutions?, validators) — the generic
|
||||
* merge/rollup/orphan-reject engine shared by the edge probe and the #644
|
||||
* prohibition probe.
|
||||
*
|
||||
* trek-e re-review #7 N2 (RULESET.TESTS.property-based-testing): analyzeCoverage is a
|
||||
* transformation/rollup module (items × resolutions → CoverageReport) — exactly the
|
||||
* class the predicate covers. The example-based suite (tests/probe-core.test.cjs)
|
||||
* pins specific scenarios; these properties pin the algebraic invariants that must
|
||||
* hold for EVERY valid scenario.
|
||||
*
|
||||
* Properties tested:
|
||||
* (a) closed-set identity: applicable === resolved + unresolved (and === items.length)
|
||||
* (b) byVerification sums ≤ resolved (dismissed is closed but unverified)
|
||||
* (c) per-tier byVerification ≤ resolved, and only `resolved`-status items are counted
|
||||
* (d) determinism: same input → identical CoverageReport (stable rollup)
|
||||
* (e) orphan rejection is stable: a resolution matching no proposed item always throws
|
||||
*/
|
||||
|
||||
const { describe, test } = require('node:test');
|
||||
const path = require('node:path');
|
||||
const fc = require('./helpers/fast-check-setup.cjs');
|
||||
|
||||
const BUILT_SCRIPT = path.join(__dirname, '..', 'gsd-core', 'bin', 'lib', 'probe-core.cjs');
|
||||
const pc = require(BUILT_SCRIPT);
|
||||
|
||||
// The same representative validators bundle the edge adapter injects (see
|
||||
// tests/probe-core.test.cjs) — exercises the generic engine independent of any one probe.
|
||||
const VALIDATORS = {
|
||||
categories: ['adjacency', 'empty', 'ordering'],
|
||||
verification: ['explicit', 'backstop'],
|
||||
requiredFieldsByVerification: { explicit: ['resolution'], backstop: ['resolution'] },
|
||||
};
|
||||
|
||||
function bareItem(requirement_id, category) {
|
||||
return {
|
||||
requirement_id,
|
||||
category,
|
||||
status: 'unresolved',
|
||||
verification: null,
|
||||
resolution: null,
|
||||
reason: null,
|
||||
probe: `probe-for-${category}`,
|
||||
};
|
||||
}
|
||||
|
||||
const catArb = fc.constantFrom(...VALIDATORS.categories);
|
||||
const idArb = fc.constantFrom('R1', 'R2', 'R3', 'R4', 'R5');
|
||||
const keyArb = fc.record({ requirement_id: idArb, category: catArb });
|
||||
// Unique (requirement_id, category) keys — the merge keys analyzeCoverage maps on.
|
||||
const keyOf = (k) => `${k.requirement_id}::${k.category}`;
|
||||
const uniqueKeysArb = fc.uniqueArray(keyArb, { selector: keyOf, minLength: 0, maxLength: 12 });
|
||||
|
||||
// Each unique item key gets one resolution disposition. Resolution text/reason use fixed
|
||||
// non-empty literals — the counting invariants are independent of their content, and this
|
||||
// keeps the generator off the validateResolution rejection paths (which the example suite
|
||||
// already covers exhaustively).
|
||||
const DISPOSITIONS = ['none', 'resolved-explicit', 'resolved-backstop', 'dismissed', 'unresolved'];
|
||||
|
||||
function resolutionFor(k, disposition) {
|
||||
const base = { requirement_id: k.requirement_id, category: k.category };
|
||||
switch (disposition) {
|
||||
case 'resolved-explicit':
|
||||
return { ...base, status: 'resolved', verification: 'explicit', resolution: 'AC#1' };
|
||||
case 'resolved-backstop':
|
||||
return { ...base, status: 'resolved', verification: 'backstop', resolution: 'held-out PBT suite' };
|
||||
case 'dismissed':
|
||||
return { ...base, status: 'dismissed', reason: 'bounded enum — not applicable' };
|
||||
case 'unresolved':
|
||||
return { ...base, status: 'unresolved' };
|
||||
default: // 'none' — author left no resolution; item rolls up verbatim (bare unresolved)
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// A fully valid scenario: unique items (all bare-unresolved) + a per-item resolution choice.
|
||||
const scenarioArb = uniqueKeysArb.chain((keys) =>
|
||||
fc.tuple(...keys.map(() => fc.constantFrom(...DISPOSITIONS))).map((choices) => {
|
||||
const items = keys.map((k) => bareItem(k.requirement_id, k.category));
|
||||
const resolutions = [];
|
||||
keys.forEach((k, i) => {
|
||||
const r = resolutionFor(k, choices[i]);
|
||||
if (r) resolutions.push(r);
|
||||
});
|
||||
return { items, resolutions };
|
||||
}),
|
||||
);
|
||||
|
||||
describe('probe-core property: analyzeCoverage algebraic invariants', () => {
|
||||
test('(a) closed-set identity: applicable === resolved + unresolved === items.length', () => {
|
||||
fc.assert(
|
||||
fc.property(scenarioArb, ({ items, resolutions }) => {
|
||||
const { coverage } = pc.analyzeCoverage(items, resolutions, VALIDATORS);
|
||||
return (
|
||||
coverage.applicable === coverage.resolved + coverage.unresolved &&
|
||||
coverage.applicable === items.length
|
||||
);
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
test('(b) sum(byVerification) ≤ resolved — dismissed counts closed but unverified', () => {
|
||||
fc.assert(
|
||||
fc.property(scenarioArb, ({ items, resolutions }) => {
|
||||
const { coverage } = pc.analyzeCoverage(items, resolutions, VALIDATORS);
|
||||
const verifiedTotal = Object.values(coverage.byVerification).reduce((a, b) => a + b, 0);
|
||||
return verifiedTotal <= coverage.resolved && verifiedTotal >= 0;
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
test('(c) byVerification only counts resolved-status items, and matches a direct recount', () => {
|
||||
fc.assert(
|
||||
fc.property(scenarioArb, ({ items, resolutions }) => {
|
||||
const rep = pc.analyzeCoverage(items, resolutions, VALIDATORS);
|
||||
for (const tier of VALIDATORS.verification) {
|
||||
const recount = rep.items.filter((i) => i.status === 'resolved' && i.verification === tier).length;
|
||||
if (rep.coverage.byVerification[tier] !== recount) return false;
|
||||
if (rep.coverage.byVerification[tier] > rep.coverage.resolved) return false;
|
||||
}
|
||||
return true;
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
test('(d) determinism: identical inputs produce an identical CoverageReport', () => {
|
||||
fc.assert(
|
||||
fc.property(scenarioArb, ({ items, resolutions }) => {
|
||||
const a = pc.analyzeCoverage(items, resolutions, VALIDATORS);
|
||||
const b = pc.analyzeCoverage(items, resolutions, VALIDATORS);
|
||||
return JSON.stringify(a) === JSON.stringify(b);
|
||||
}),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('probe-core property: orphan rejection is stable', () => {
|
||||
// An orphan resolution carries an id ('Z9') that no generated item ever uses, so its
|
||||
// (requirement_id, category) key never matches a proposed item. The resolution is itself
|
||||
// structurally VALID (a bare unresolved), so it clears validateResolution and reaches the
|
||||
// orphan-reject guard — isolating that guard from input-validation throws.
|
||||
const orphanScenarioArb = fc.record({
|
||||
keys: uniqueKeysArb,
|
||||
orphanCategory: catArb,
|
||||
});
|
||||
|
||||
test('(e) a resolution matching no proposed item always throws', () => {
|
||||
fc.assert(
|
||||
fc.property(orphanScenarioArb, ({ keys, orphanCategory }) => {
|
||||
const items = keys.map((k) => bareItem(k.requirement_id, k.category));
|
||||
const orphan = { requirement_id: 'Z9', category: orphanCategory, status: 'unresolved' };
|
||||
let threwForOrphan = false;
|
||||
try {
|
||||
pc.analyzeCoverage(items, [orphan], VALIDATORS);
|
||||
} catch (e) {
|
||||
threwForOrphan = /unknown resolution|no matching proposed item/i.test(e.message);
|
||||
}
|
||||
return threwForOrphan;
|
||||
}),
|
||||
);
|
||||
});
|
||||
});
|
||||
335
tests/probe-core.test.cjs
Normal file
335
tests/probe-core.test.cjs
Normal file
@@ -0,0 +1,335 @@
|
||||
/**
|
||||
* probe-core reference-model unit tests (ADR-550 Decision 7).
|
||||
*
|
||||
* probe-core is the GENERIC spec-phase probe resolution model extracted from the
|
||||
* edge-probe (the first adapter): the resolution lifecycle, the two-axis
|
||||
* status×verification re-cut, `validateResolution`/`validateRequirement`, the
|
||||
* `analyzeCoverage(items, resolutions?, validators)` merge/rollup/orphan-reject
|
||||
* engine, the `byVerification` rollup, and the `runProbeCli` I/O scaffold.
|
||||
*
|
||||
* Asserts the LOCKED export surface against the BUILT artifact
|
||||
* (`gsd-core/bin/lib/probe-core.cjs`), which `npm run build:lib` (run by pretest /
|
||||
* the run-tests sentinel) emits from `src/probe-core.cts`.
|
||||
*
|
||||
* The injected runtime validators are the enforcement contract (ADR-550 #5): the
|
||||
* CLI runs over JSON where TS types are erased, so `analyzeCoverage` is told its
|
||||
* probe's closed vocabularies — `{ categories, verification, requiredFieldsByVerification }`
|
||||
* — rather than relying on the type system. These tests pin the validators' behavior.
|
||||
*/
|
||||
'use strict';
|
||||
process.env.GSD_TEST_MODE = '1';
|
||||
|
||||
const { test, describe } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const path = require('node:path');
|
||||
|
||||
const BUILT_SCRIPT = path.join(__dirname, '..', 'gsd-core', 'bin', 'lib', 'probe-core.cjs');
|
||||
const pc = require(BUILT_SCRIPT);
|
||||
|
||||
// A representative validators bundle — the shape the edge adapter injects, used here
|
||||
// to exercise the generic engine independent of any one probe.
|
||||
const VALIDATORS = {
|
||||
categories: ['adjacency', 'empty', 'ordering'],
|
||||
verification: ['explicit', 'backstop'],
|
||||
requiredFieldsByVerification: { explicit: ['resolution'], backstop: ['resolution'] },
|
||||
};
|
||||
|
||||
function item(category, overrides = {}) {
|
||||
return {
|
||||
requirement_id: 'R1',
|
||||
category,
|
||||
status: 'unresolved',
|
||||
verification: null,
|
||||
resolution: null,
|
||||
reason: null,
|
||||
probe: `probe-for-${category}`,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
const UNRESOLVED_ITEMS = [item('adjacency'), item('empty'), item('ordering')];
|
||||
|
||||
describe('probe-core: VALID_STATUS is the re-cut lifecycle enum', () => {
|
||||
test('exposes exactly resolved | dismissed | unresolved (no covered/backstop)', () => {
|
||||
assert.deepEqual([...ep_sorted(pc.VALID_STATUS)], ['dismissed', 'resolved', 'unresolved']);
|
||||
assert.ok(!pc.VALID_STATUS.includes('covered'), 'covered must not survive the re-cut as a status');
|
||||
assert.ok(!pc.VALID_STATUS.includes('backstop'), 'backstop must not survive the re-cut as a status');
|
||||
});
|
||||
});
|
||||
|
||||
function ep_sorted(arr) {
|
||||
return [...arr].sort();
|
||||
}
|
||||
|
||||
describe('probe-core: validateResolution (status×verification)', () => {
|
||||
const v = (r) => pc.validateResolution(r, VALIDATORS);
|
||||
test('rejects an unknown status', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'maybe' }), /invalid status/i);
|
||||
});
|
||||
test('rejects a former covered status (re-cut: no longer a valid status)', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'covered', resolution: 'x' }), /invalid status/i);
|
||||
});
|
||||
test('rejects dismissed without a reason', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'dismissed', reason: '' }), /dismissed requires a reason/i);
|
||||
});
|
||||
test('accepts dismissed with a reason', () => {
|
||||
assert.equal(v({ requirement_id: 'R1', category: 'adjacency', status: 'dismissed', reason: 'bounded enum' }), true);
|
||||
});
|
||||
test('rejects resolved with a missing verification tier', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'resolved', resolution: 'AC' }), /verification/i);
|
||||
});
|
||||
test('rejects resolved with an unknown verification tier', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'judgment', resolution: 'AC' }), /invalid verification/i);
|
||||
});
|
||||
test('rejects resolved/explicit with empty resolution text', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: ' ' }), /explicit requires a resolution/i);
|
||||
});
|
||||
test('rejects resolved/backstop with missing resolution note', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'backstop' }), /backstop requires a resolution/i);
|
||||
});
|
||||
test('accepts resolved/explicit with a resolution', () => {
|
||||
assert.equal(v({ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: 'AC#6' }), true);
|
||||
});
|
||||
test('accepts resolved/backstop with a resolution note', () => {
|
||||
assert.equal(v({ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'backstop', resolution: 'held-out PBT suite' }), true);
|
||||
});
|
||||
// Re-review #5 Medium — fail closed across the FULL status×verification model, not just
|
||||
// `resolved`. The invariant (probe-core header): verification is null unless status is
|
||||
// resolved. A dismissed/unresolved resolution carrying a tier otherwise merges verbatim
|
||||
// (analyzeCoverage line ~189), breaking the model for the second adapter (#644) that
|
||||
// inherits this seam.
|
||||
test('rejects a dismissed resolution carrying a verification tier (null unless resolved)', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'dismissed', reason: 'n/a', verification: 'explicit' }), /verification must be null/i);
|
||||
});
|
||||
test('rejects an unresolved resolution carrying a verification tier', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'unresolved', verification: 'backstop' }), /verification must be null/i);
|
||||
});
|
||||
// An unresolved resolution is an UNACTED item — a populated resolution/reason payload is an
|
||||
// authoring mistake (the author meant resolved/dismissed) that today is silently dropped into
|
||||
// the unresolved count with no error pointing at it. Reject the payload.
|
||||
test('rejects an unresolved resolution carrying a resolution payload', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'unresolved', resolution: 'AC#1' }), /unresolved must not carry/i);
|
||||
});
|
||||
test('rejects an unresolved resolution carrying a reason payload', () => {
|
||||
assert.throws(() => v({ requirement_id: 'R1', category: 'adjacency', status: 'unresolved', reason: 'because' }), /unresolved must not carry/i);
|
||||
});
|
||||
test('accepts a bare unresolved resolution (no payload — a harmless no-op merge)', () => {
|
||||
assert.equal(v({ requirement_id: 'R1', category: 'adjacency', status: 'unresolved' }), true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('probe-core: validateRequirement (generic id/text only)', () => {
|
||||
test('rejects a missing id', () => {
|
||||
assert.throws(() => pc.validateRequirement({ text: 'x' }), /requirement id must be a non-empty string/i);
|
||||
});
|
||||
test('rejects an empty id', () => {
|
||||
assert.throws(() => pc.validateRequirement({ id: ' ', text: 'x' }), /requirement id must be a non-empty string/i);
|
||||
});
|
||||
test('rejects a non-string text', () => {
|
||||
assert.throws(() => pc.validateRequirement({ id: 'R1', text: 42 }), /text must be a string/i);
|
||||
});
|
||||
test('accepts a valid requirement', () => {
|
||||
assert.doesNotThrow(() => pc.validateRequirement({ id: 'R1', text: 'a testable statement' }));
|
||||
});
|
||||
});
|
||||
|
||||
describe('probe-core: analyzeCoverage (merge · rollup · byVerification)', () => {
|
||||
test('no resolutions → every item unresolved; resolved 0; byVerification zeroed per tier', () => {
|
||||
const rep = pc.analyzeCoverage(UNRESOLVED_ITEMS, [], VALIDATORS);
|
||||
assert.deepEqual(rep.coverage, {
|
||||
applicable: 3, resolved: 0, unresolved: 3, byVerification: { explicit: 0, backstop: 0 },
|
||||
});
|
||||
});
|
||||
test('merges a resolved/explicit resolution and counts byVerification.explicit', () => {
|
||||
const rep = pc.analyzeCoverage(UNRESOLVED_ITEMS, [
|
||||
{ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: 'AC#6: touching intervals merge' },
|
||||
], VALIDATORS);
|
||||
const adj = rep.items.find((i) => i.category === 'adjacency');
|
||||
assert.equal(adj.status, 'resolved');
|
||||
assert.equal(adj.verification, 'explicit');
|
||||
assert.equal(adj.resolution, 'AC#6: touching intervals merge');
|
||||
assert.equal(rep.coverage.resolved, 1);
|
||||
assert.equal(rep.coverage.unresolved, 2);
|
||||
assert.deepEqual(rep.coverage.byVerification, { explicit: 1, backstop: 0 });
|
||||
});
|
||||
test('a dismissed item counts toward coverage.resolved (closed set) but NOT byVerification', () => {
|
||||
// coverage.resolved preserves the pre-re-cut "closed" semantic = applicable - unresolved
|
||||
// (covered + dismissed + backstop), per edge-probe.md and the migration contract.
|
||||
const rep = pc.analyzeCoverage(UNRESOLVED_ITEMS, [
|
||||
{ requirement_id: 'R1', category: 'ordering', status: 'dismissed', reason: 'canonically sorted; no tie' },
|
||||
], VALIDATORS);
|
||||
assert.equal(rep.coverage.resolved, 1);
|
||||
assert.equal(rep.coverage.unresolved, 2);
|
||||
assert.deepEqual(rep.coverage.byVerification, { explicit: 0, backstop: 0 });
|
||||
const ord = rep.items.find((i) => i.category === 'ordering');
|
||||
assert.equal(ord.status, 'dismissed');
|
||||
assert.equal(ord.verification, null);
|
||||
assert.equal(ord.reason, 'canonically sorted; no tie');
|
||||
});
|
||||
test('mixed resolved/explicit + backstop + dismissed: resolved = closed = applicable - unresolved', () => {
|
||||
const rep = pc.analyzeCoverage(UNRESOLVED_ITEMS, [
|
||||
{ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: 'AC#6' },
|
||||
{ requirement_id: 'R1', category: 'empty', status: 'resolved', verification: 'backstop', resolution: 'held-out empty-input PBT' },
|
||||
{ requirement_id: 'R1', category: 'ordering', status: 'dismissed', reason: 'canonically sorted' },
|
||||
], VALIDATORS);
|
||||
assert.equal(rep.coverage.applicable, 3);
|
||||
assert.equal(rep.coverage.unresolved, 0);
|
||||
assert.equal(rep.coverage.resolved, 3); // 2 resolved-status + 1 dismissed
|
||||
assert.deepEqual(rep.coverage.byVerification, { explicit: 1, backstop: 1 });
|
||||
});
|
||||
test('an all-dismissed run is CLOSED but NOT affirmatively covered (byVerification is the honest gate)', () => {
|
||||
// Re-review #5 Medium — coverage.resolved is the closed set (resolved + dismissed), kept
|
||||
// count-preserved per the blessed migration contract. So an all-dismissed spec has
|
||||
// resolved === applicable while NOTHING was affirmatively resolved/backstopped. A CI gate
|
||||
// keying on `resolved === applicable` would read green here; the honest signal is
|
||||
// byVerification (all tiers zero). This test locks that distinction.
|
||||
const rep = pc.analyzeCoverage(UNRESOLVED_ITEMS, [
|
||||
{ requirement_id: 'R1', category: 'adjacency', status: 'dismissed', reason: 'bounded enum' },
|
||||
{ requirement_id: 'R1', category: 'empty', status: 'dismissed', reason: 'guaranteed non-empty' },
|
||||
{ requirement_id: 'R1', category: 'ordering', status: 'dismissed', reason: 'canonically sorted' },
|
||||
], VALIDATORS);
|
||||
assert.equal(rep.coverage.unresolved, 0);
|
||||
assert.equal(rep.coverage.resolved, 3); // closed set counts the dismissals
|
||||
assert.deepEqual(rep.coverage.byVerification, { explicit: 0, backstop: 0 }); // nothing affirmatively verified
|
||||
});
|
||||
test('rejects a duplicate (requirement_id, category) resolution', () => {
|
||||
assert.throws(() => pc.analyzeCoverage(UNRESOLVED_ITEMS, [
|
||||
{ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: 'AC#1' },
|
||||
{ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: 'AC#2' },
|
||||
], VALIDATORS), /duplicate resolution/i);
|
||||
});
|
||||
test('rejects an orphan resolution (no matching proposed item)', () => {
|
||||
assert.throws(() => pc.analyzeCoverage(UNRESOLVED_ITEMS, [
|
||||
{ requirement_id: 'R1', category: 'boundary', status: 'resolved', verification: 'explicit', resolution: 'AC' },
|
||||
], VALIDATORS), /unknown resolution|no matching proposed/i);
|
||||
});
|
||||
test('propagates an invalid-resolution throw from validateResolution', () => {
|
||||
assert.throws(() => pc.analyzeCoverage(UNRESOLVED_ITEMS, [
|
||||
{ requirement_id: 'R1', category: 'empty', status: 'dismissed' },
|
||||
], VALIDATORS), /dismissed requires a reason/i);
|
||||
});
|
||||
test('rejects items that is not an array', () => {
|
||||
assert.throws(() => pc.analyzeCoverage('nope', [], VALIDATORS), /items must be an array/i);
|
||||
});
|
||||
test('rejects a proposed item whose category is not in validators.categories', () => {
|
||||
// Adapter self-consistency: an item carrying a category outside the probe's closed
|
||||
// vocabulary is an adapter bug, caught here rather than silently rolled up.
|
||||
assert.throws(() => pc.analyzeCoverage([item('bogus-category')], [], VALIDATORS), /unknown category/i);
|
||||
});
|
||||
|
||||
// m1: a verbatim item (no matching author resolution) is rolled up as-is, so its OWN
|
||||
// status/fields must also be validated. The edge adapter only proposes `unresolved` items, but
|
||||
// the prohibition adapter (#644) proposes LLM-generated items that arrive already populated —
|
||||
// an out-of-enum status or a `dismissed` with no reason must fail closed, not count as closed.
|
||||
test('m1: rejects a verbatim item carrying an out-of-enum status (e.g. the dropped "covered")', () => {
|
||||
assert.throws(
|
||||
() => pc.analyzeCoverage([item('adjacency', { status: 'covered' })], [], VALIDATORS),
|
||||
/invalid status/i,
|
||||
);
|
||||
});
|
||||
test('m1: rejects a verbatim item dismissed without a reason', () => {
|
||||
assert.throws(
|
||||
() => pc.analyzeCoverage([item('adjacency', { status: 'dismissed' })], [], VALIDATORS),
|
||||
/dismissed requires a reason/i,
|
||||
);
|
||||
});
|
||||
test('m1: rejects a verbatim resolved item missing its verification tier', () => {
|
||||
assert.throws(
|
||||
() => pc.analyzeCoverage([item('adjacency', { status: 'resolved', resolution: 'AC' })], [], VALIDATORS),
|
||||
/requires a verification tier/i,
|
||||
);
|
||||
});
|
||||
test('m1: a matching resolution still governs — item validation targets VERBATIM items only', () => {
|
||||
// When a resolution matches, the item is rebuilt from the (already-validated) resolution, so a
|
||||
// pre-populated item status is irrelevant and must NOT cause a throw. Guards against the m1
|
||||
// fix over-reaching into the merge path.
|
||||
const rep = pc.analyzeCoverage([item('adjacency', { status: 'covered' })], [
|
||||
{ requirement_id: 'R1', category: 'adjacency', status: 'resolved', verification: 'explicit', resolution: 'AC#6' },
|
||||
], VALIDATORS);
|
||||
const adj = rep.items.find((i) => i.category === 'adjacency');
|
||||
assert.equal(adj.status, 'resolved');
|
||||
assert.equal(adj.verification, 'explicit');
|
||||
assert.equal(rep.coverage.resolved, 1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('probe-core: runProbeCli (generic I/O scaffold, injected io)', () => {
|
||||
const report = { items: [], coverage: { applicable: 0, resolved: 0, unresolved: 0, byVerification: {} } };
|
||||
test('no requirements path → writes usage to stderr and exits 2', () => {
|
||||
let code; let err = '';
|
||||
pc.runProbeCli(() => report, {
|
||||
usage: 'demo-probe.cjs <requirements.json> [resolutions.json]',
|
||||
argv: ['node', 'demo'], writeErr: (s) => { err += s; }, exit: (c) => { code = c; },
|
||||
});
|
||||
assert.equal(code, 2);
|
||||
assert.match(err, /usage: demo-probe\.cjs/);
|
||||
});
|
||||
test('valid requirements path → calls analyze and prints the report as JSON', () => {
|
||||
let out = '';
|
||||
pc.runProbeCli((reqs, res) => {
|
||||
assert.deepEqual(reqs, [{ id: 'R1', text: 'x' }]);
|
||||
assert.deepEqual(res, []);
|
||||
return report;
|
||||
}, {
|
||||
usage: 'demo', argv: ['node', 'demo', '/fake/req.json'],
|
||||
readFile: () => '[{"id":"R1","text":"x"}]', write: (s) => { out += s; }, exit: () => {},
|
||||
});
|
||||
assert.deepEqual(JSON.parse(out), report);
|
||||
});
|
||||
test('reads the optional resolutions file when a second path is given', () => {
|
||||
let seenRes;
|
||||
pc.runProbeCli((reqs, res) => { seenRes = res; return report; }, {
|
||||
usage: 'demo', argv: ['node', 'demo', '/req.json', '/res.json'],
|
||||
readFile: (p) => (p === '/res.json' ? '[{"requirement_id":"R1"}]' : '[{"id":"R1"}]'),
|
||||
write: () => {}, exit: () => {},
|
||||
});
|
||||
assert.deepEqual(seenRes, [{ requirement_id: 'R1' }]);
|
||||
});
|
||||
test('invalid requirements JSON → exits 2 (handled, not an uncaught throw)', () => {
|
||||
let code;
|
||||
pc.runProbeCli(() => report, {
|
||||
usage: 'demo', argv: ['node', 'demo', '/req.json'],
|
||||
readFile: () => 'not json {{{', writeErr: () => {}, exit: (c) => { code = c; },
|
||||
});
|
||||
assert.equal(code, 2);
|
||||
});
|
||||
test('an analyze throw → exits 2 (the engine fail-closed surfaces, never silently passes)', () => {
|
||||
let code;
|
||||
pc.runProbeCli(() => { throw new Error('boom'); }, {
|
||||
usage: 'demo', argv: ['node', 'demo', '/req.json'],
|
||||
readFile: () => '[]', writeErr: () => {}, exit: (c) => { code = c; },
|
||||
});
|
||||
assert.equal(code, 2);
|
||||
});
|
||||
// Re-review #5 Low — the scaffold trusts each adapter's `as` cast for the returned report.
|
||||
// A future adapter (#644) that forgets to validate inside its closure would otherwise have a
|
||||
// structurally-broken report written as green output. Guard the report shape structurally.
|
||||
test('a structurally-invalid report from analyze → exits 2 and writes nothing (no silent malformed output)', () => {
|
||||
let code; let out = '';
|
||||
pc.runProbeCli(() => ({ nope: true }), {
|
||||
usage: 'demo', argv: ['node', 'demo', '/req.json'],
|
||||
readFile: () => '[]', write: (s) => { out += s; }, writeErr: () => {}, exit: (c) => { code = c; },
|
||||
});
|
||||
assert.equal(code, 2);
|
||||
assert.equal(out, '');
|
||||
});
|
||||
test('a report whose coverage object carries non-numeric counts → exits 2 (partial-guard case)', () => {
|
||||
// items[] is well-formed and `coverage` IS an object, so the cheaper checks pass — this
|
||||
// exercises the numeric-count branch of the structural guard specifically.
|
||||
let code; let out = '';
|
||||
pc.runProbeCli(() => ({ items: [], coverage: { applicable: 'x', resolved: null, unresolved: undefined, byVerification: {} } }), {
|
||||
usage: 'demo', argv: ['node', 'demo', '/req.json'],
|
||||
readFile: () => '[]', write: (s) => { out += s; }, writeErr: () => {}, exit: (c) => { code = c; },
|
||||
});
|
||||
assert.equal(code, 2);
|
||||
assert.equal(out, '');
|
||||
});
|
||||
test('a well-formed report still writes and does not trip the structural guard', () => {
|
||||
let out = '';
|
||||
pc.runProbeCli(() => report, {
|
||||
usage: 'demo', argv: ['node', 'demo', '/req.json'],
|
||||
readFile: () => '[]', write: (s) => { out += s; }, exit: () => {},
|
||||
});
|
||||
assert.deepEqual(JSON.parse(out), report);
|
||||
});
|
||||
});
|
||||
@@ -51,7 +51,7 @@
|
||||
"note.md": 6563,
|
||||
"pause-work.md": 13654,
|
||||
"plan-milestone-gaps.md": 11765,
|
||||
"plan-phase.md": 93135,
|
||||
"plan-phase.md": 94253,
|
||||
"plan-review-convergence.md": 22949,
|
||||
"plant-seed.md": 11741,
|
||||
"pr-branch.md": 4994,
|
||||
@@ -72,7 +72,7 @@
|
||||
"ship.md": 20896,
|
||||
"sketch-wrap-up.md": 14223,
|
||||
"sketch.md": 19960,
|
||||
"spec-phase.md": 15131,
|
||||
"spec-phase.md": 23094,
|
||||
"spike-wrap-up.md": 15092,
|
||||
"spike.md": 24517,
|
||||
"stats.md": 6718,
|
||||
|
||||
Reference in New Issue
Block a user