* chore(#2794): single-source the reviewer invocation contract Phase 1 of epic #2782 (ADR-2782). Introduces one core descriptor table as the declared contract for all 11 cross-AI reviewer lanes, and the DEFECT.GENERATIVE-FIX parity assertion the roster has never had. The lane contract lived in three unrelated surfaces — the roster, ~640 lines of hand-authored per-CLI bash in invoke_reviewers, and the write_reviews section headings — so cross-cutting fixes landed per-leg (#2494 and #2605 were the same empty-output defect filed twice). - src/review-lane-descriptor.cts: frozen table declaring per lane the slug, flags, probe, invoke shape, timeout floor, empty-output policy, REVIEWS.md section, evidence class, required binaries, prompt-budget key and handler. Field names track ADR-2782 D1/D2/D6/D7 verbatim so Phase 2 harvests the shape with no translation layer. It declares; it does not execute — invoke_reviewers iterates in Phase 5b. - checkReviewerLaneParity: bidirectional parity across descriptor, roster, invoke_reviewers legs and write_reviews sections. Forward-only would miss the failure it exists to catch (#2718 added a leg, #2781 was the drift). ADR-1517 instance headings are exempt per D8. - Legs carry an explicit <!-- reviewer-lane: slug --> marker; five non-lane bold labels share the bold-then-fence shape a heuristic matcher would key on. - ADR-2782 D4: an explicitly-flagged reviewer that cannot run is now an error in both the core module and the workflow prose that mirrors it. A code-only change would be unobservable — the module has no production caller; the workflow narrates the policy. Discovery paths (--all, review.default_reviewers) stay lenient. - Fixes the qwen leg, the last one discarding stderr to /dev/null. Two ADR-2782 D2 vocabulary widenings were forced by surveying the shipped legs: promptChannel 'none' (CodeRabbit is fed no prompt) and outputChannel 'file-arg' (Codex writes via -o and discards stdout, #1698). Both are additive and closed; Phase 2 owns the validator. Closes #2690 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#2794): make the parity checker total and pin the lane slug grammar Findings from the orthogonal review passes. Spec axis — the module claimed its vocabulary tracked ADR-2782 D1/D2 "verbatim" while diverging in three undisclosed ways, which is the translation layer Phase 2 was supposed to be spared: - `transport` moves from `invoke.transport` to the LANE level, a sibling of `probe`/`invoke`, exactly as D1's manifest example places it. The nested form read better as a TS discriminated union; the union is now discriminated at the lane level instead, which costs nothing. - The header and the CONTEXT.md glossary now enumerate all FOUR widenings (adding `outputArg` and `flags[]`), not two. Standards axis — CLAUDE.md requires a fast-check property test for a parser, and `checkReviewerLaneParity` parses markdown for markers and headings. Adding one found two real defects that the hand-written matrix missed: - NOT TOTAL: a malformed descriptor entry threw on `lane.flags` iteration, contradicting the module's own "never throws" claim. Every field is now narrowed from `unknown` at the trust boundary and reported as MALFORMED_LANE / INVALID_SLUG. This matters because Phase 2 feeds this function third-party overlay data, and a parity gate that crashes is indistinguishable from one never run. - SILENT GRAMMAR MISMATCH: LEG_MARKER_RE captures only [a-z0-9_-], so a slug outside that class was unmatchable — its marker could be present and correct and the scan would still report LEG_MARKER_MISSING forever. LANE_SLUG_RE now pins the grammar and a violating slug is reported INVALID_SLUG. A loud named violation beats a silent miss. Generators are document-shaped, not writer-seeded (CONTRIBUTING #2371): seeding from the module's own matchers could only produce documents those matchers already recognize. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#2794): register the new bin/lib module in the ESLint ignore list The remote runner caught this; lint:ci did not, because the invariant lives in the test suite rather than the lint chain: tests/repo-invariants.test.cjs "each bin/lib/*.cjs is linted xor ignored according to migration state" -> tsc-generated bin/lib modules not yet added to ESLint ignore list: review-lane-descriptor.cjs Adding a src/*.cts module ripples to six surfaces (.gitignore, the ESLint ignore list, docs/INVENTORY-MANIFEST.json, the CONTEXT.md glossary, the capability/inventory manifests, and any size baseline). The other five were covered; this was the miss. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#2794): amend ADR-2782 D1/D2/D8 with the vocabulary Phase 1 surfaced Building the Phase 1 descriptor table against all eleven shipped legs is the first time every lane's contract was written in one place, and it surfaced four cases the ADR's original survey did not cover. Amending the design lock rather than diverging from it, so Phase 2 (#2795) implements the manifest validator against the amended vocabulary instead of rediscovering the gaps. All four are additive widenings of closed enums; no decision reverses: - D2 promptChannel gains `none` — coderabbit is fed no prompt at all, it reviews the working-tree diff. - D2 outputChannel gains `file-arg` — the ADR called a file-writing lane a shape a real CLI *could* take; codex already is one, writing via -o/--output-last-message and discarding stdout (#1698). - D2 gains `outputArg`, required iff file-arg — knowing the review lands in a file is useless without the argument naming it. - D1 `flag` becomes `flags[]` and D8's uniqueness flattens across lanes — antigravity is selected by both --antigravity and --agy, which a single-valued field cannot express. This is the same evidence path that produced the openai-http transport: the vocabulary widens on a lane that exists, under review, never on speculation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(#2794): backfill changeset pr number to 2820 --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
8
tests/emitted-drift-ack.json
Normal file
8
tests/emitted-drift-ack.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"version": 1,
|
||||
"paths": {
|
||||
"review.md": {
|
||||
"reason": "#2794 / ADR-2782 Phase 1. Three deliberate additions to gsd-core/workflows/review.md: (1) an explicit `<!-- reviewer-lane: <slug> -->` marker above each of the 11 invoke_reviewers legs, which is what makes a leg machine-identifiable for the DEFECT.GENERATIVE-FIX parity assertion — the legs are prose-labelled and five NON-lane bold labels in the same step share the bold-then-fence shape a heuristic matcher would key on, so inference was not an option; (2) the ADR-2782 D4 explicit-selection contract in the detect_clis prose, without which the selector correction is unobservable (review-reviewer-selection.cts has no production caller — the workflow narrates the policy and the reviewing agent executes it, so code and prose must move together); (3) the qwen leg's stderr sidecar, the last lane still discarding stderr to /dev/null (#2494/#2605 class). Growth is prose and comments only; no lane's observable command shape changed apart from the qwen redirect."
|
||||
}
|
||||
}
|
||||
}
|
||||
166
tests/fix-2794-review-qwen-empty-guard.test.cjs
Normal file
166
tests/fix-2794-review-qwen-empty-guard.test.cjs
Normal file
@@ -0,0 +1,166 @@
|
||||
// allow-test-rule: source-text-is-the-product (see #2794)
|
||||
// The qwen reviewer dispatch block in gsd-core/workflows/review.md IS the runtime
|
||||
// contract — the workflow's text is what the reviewing agent executes. This suite
|
||||
// extracts that shell block verbatim and runs it under a real bash against a stubbed
|
||||
// `qwen`, so the shipped guard is what gets exercised rather than a reimplementation
|
||||
// of it. Assertions are on the guard's documented output contract (the stub line the
|
||||
// consensus step must be able to tell apart from a clean empty review, and the
|
||||
// presence of the captured stderr), not incidental string matching.
|
||||
|
||||
/**
|
||||
* Regression tests for #2794 — the qwen reviewer leg was the last one still
|
||||
* sending stderr to /dev/null.
|
||||
*
|
||||
* Every other lane captures stderr to a `.err` sidecar and appends it to the
|
||||
* empty-output stub (#2494 for claude/gemini, #2605 for the local servers and
|
||||
* CodeRabbit). qwen alone ran `2>/dev/null` and wrote a bare
|
||||
* "Qwen review failed or returned empty output." with no diagnostic, so a missing
|
||||
* binary, an auth prompt, a rate-limit, and a genuinely empty review were all
|
||||
* indistinguishable — the same cross-cutting defect landing per-leg that ADR-2782
|
||||
* exists to end.
|
||||
*
|
||||
* These tests fail against pre-fix review.md: the stub carries no stderr, so the
|
||||
* diagnostic assertion in E2 trips.
|
||||
*/
|
||||
|
||||
'use strict';
|
||||
|
||||
const { describe, test, before, after } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { spawnSync } = require('node:child_process');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { createTempDir, cleanup } = require('./helpers.cjs');
|
||||
|
||||
const ROOT = path.join(__dirname, '..');
|
||||
const REVIEW_PATH = path.join(ROOT, 'gsd-core', 'workflows', 'review.md');
|
||||
|
||||
// Normalize CRLF: on a Windows autocrlf checkout every line carries a trailing
|
||||
// \r, which would mangle the extracted block's redirect tokens and defeat the
|
||||
// fence regex below.
|
||||
const WORKFLOW = fs.readFileSync(REVIEW_PATH, 'utf-8').replace(/\r\n/g, '\n');
|
||||
|
||||
/**
|
||||
* Extract the qwen dispatch block verbatim. If review.md changes the block's
|
||||
* shape this throws and the test fails loudly — intended coupling, the same
|
||||
* contract the #2494 and #2605 suites pin for their legs.
|
||||
*/
|
||||
function extractQwenBlock() {
|
||||
// `\r?\n` throughout: WORKFLOW is CRLF-normalized above, but the anchors stay
|
||||
// CRLF-tolerant so the regex cannot silently miss on a Windows checkout.
|
||||
const m = WORKFLOW.match(
|
||||
/<!--\s*reviewer-lane:\s*qwen\s*-->\r?\n[\s\S]*?```bash\r?\n([\s\S]*?)\r?\n```/,
|
||||
);
|
||||
assert.ok(m, 'review.md must define the qwen reviewer dispatch as a bash block (#2794)');
|
||||
return m[1];
|
||||
}
|
||||
|
||||
const QWEN_BLOCK = extractQwenBlock();
|
||||
|
||||
// The extracted block is POSIX shell. Git Bash on Windows ignores Node's chmod
|
||||
// exec bit for PATH-executed extension-less scripts, so the stub would never run
|
||||
// there; the guard logic is platform-independent and is asserted in full on every
|
||||
// macOS/Linux leg.
|
||||
const WIN32_SKIP =
|
||||
'extracted block is POSIX shell; guard logic is platform-independent and asserted on macOS/Linux';
|
||||
|
||||
let sandbox;
|
||||
|
||||
before(() => { sandbox = createTempDir('gsd-2794-'); });
|
||||
after(() => { cleanup(sandbox); });
|
||||
|
||||
/**
|
||||
* Run the extracted block with `{run_dir}` pointed at a fresh run directory and
|
||||
* `qwenBody` installed on PATH as `qwen`. Pass `qwenBody: null` to run with no
|
||||
* qwen on PATH at all.
|
||||
*/
|
||||
function runQwenLeg(qwenBody) {
|
||||
// Single exec site so the Windows guard lives in one place: Git Bash (msys2)
|
||||
// ignores Node's chmod exec bit for PATH-executed extension-less scripts
|
||||
// (DEFECT.WINDOWS-TEST-PORTABILITY). Every test below is skipped on win32 —
|
||||
// this early return keeps the exec unreachable there rather than relying on
|
||||
// the skip alone.
|
||||
if (process.platform === 'win32') return null;
|
||||
|
||||
const caseDir = fs.mkdtempSync(path.join(sandbox, 'run-'));
|
||||
const runDir = path.join(caseDir, 'run');
|
||||
const binDir = path.join(caseDir, 'bin');
|
||||
fs.mkdirSync(runDir);
|
||||
fs.mkdirSync(binDir);
|
||||
|
||||
if (qwenBody !== null) {
|
||||
const stub = path.join(binDir, 'qwen');
|
||||
fs.writeFileSync(stub, qwenBody);
|
||||
fs.chmodSync(stub, 0o755);
|
||||
}
|
||||
|
||||
fs.writeFileSync(path.join(runDir, 'gsd-review-prompt.md'), '# review prompt\n');
|
||||
|
||||
const script = QWEN_BLOCK.split('{run_dir}').join(runDir);
|
||||
const result = spawnSync('bash', ['-c', script], {
|
||||
encoding: 'utf8',
|
||||
timeout: 30000,
|
||||
killSignal: 'SIGKILL',
|
||||
// An empty PATH would also break `cat`; prepend the stub dir instead so the
|
||||
// no-qwen case still resolves the shell builtins the block needs.
|
||||
env: { ...process.env, PATH: `${binDir}${path.delimiter}${process.env.PATH}` },
|
||||
});
|
||||
|
||||
const reviewPath = path.join(runDir, 'gsd-review-qwen.md');
|
||||
return {
|
||||
result,
|
||||
reviewPath,
|
||||
errPath: path.join(runDir, 'gsd-review-qwen.err'),
|
||||
review: fs.existsSync(reviewPath) ? fs.readFileSync(reviewPath, 'utf-8') : null,
|
||||
};
|
||||
}
|
||||
|
||||
const STUB_STDERR = 'gsd-2794-stub: qwen: authentication required';
|
||||
|
||||
describe('qwen reviewer leg — empty-output guard (#2794)', () => {
|
||||
test('writes the review on success', { skip: process.platform === 'win32' ? WIN32_SKIP : false }, () => {
|
||||
const out = runQwenLeg('#!/bin/sh\necho "## Qwen findings"\nexit 0\n');
|
||||
assert.ok(out.review !== null, 'review file must exist');
|
||||
assert.match(out.review, /Qwen findings/);
|
||||
// A successful lane must NOT be decorated with the failure stub.
|
||||
assert.doesNotMatch(out.review, /failed or returned empty output/i);
|
||||
});
|
||||
|
||||
test('a failed lane surfaces its stderr in the review stub', { skip: process.platform === 'win32' ? WIN32_SKIP : false }, () => {
|
||||
// THE regression row. Pre-fix the block ran `2>/dev/null`, so this stderr
|
||||
// was discarded and the stub carried no diagnostic at all.
|
||||
const out = runQwenLeg(`#!/bin/sh\necho "${STUB_STDERR}" >&2\nexit 1\n`);
|
||||
assert.ok(out.review !== null, 'review file must exist after a failed lane');
|
||||
assert.notStrictEqual(out.review.trim(), '', 'review file must not be empty');
|
||||
assert.match(
|
||||
out.review,
|
||||
/Qwen review failed or returned empty output/i,
|
||||
'stub must name the lane as failed-or-empty so consensus can tell it apart',
|
||||
);
|
||||
assert.ok(
|
||||
out.review.includes(STUB_STDERR),
|
||||
`stub must carry the captured stderr; got: ${JSON.stringify(out.review)}`,
|
||||
);
|
||||
assert.ok(fs.existsSync(out.errPath), 'stderr sidecar must be written');
|
||||
});
|
||||
|
||||
test('a silently empty lane still produces a diagnosable stub', { skip: process.platform === 'win32' ? WIN32_SKIP : false }, () => {
|
||||
// Boundary: nothing on either stream. The stub line is the only signal, and
|
||||
// it must still be present so write_reviews does not render the lane as a
|
||||
// reviewer that ran cleanly with nothing to report.
|
||||
const out = runQwenLeg('#!/bin/sh\nexit 0\n');
|
||||
assert.ok(out.review !== null, 'review file must exist');
|
||||
assert.notStrictEqual(out.review.trim(), '', 'review file must not be zero-byte');
|
||||
assert.match(out.review, /Qwen review failed or returned empty output/i);
|
||||
});
|
||||
|
||||
test('a missing qwen binary produces a diagnosable stub', { skip: process.platform === 'win32' ? WIN32_SKIP : false }, () => {
|
||||
const out = runQwenLeg(null);
|
||||
assert.ok(out.review !== null, 'review file must exist when the binary is absent');
|
||||
assert.notStrictEqual(out.review.trim(), '', 'review file must not be zero-byte');
|
||||
assert.match(out.review, /Qwen review failed or returned empty output/i);
|
||||
// The shell's own "command not found" lands in the sidecar and is appended,
|
||||
// which is the whole point of capturing instead of discarding.
|
||||
assert.ok(fs.existsSync(out.errPath), 'stderr sidecar must be written');
|
||||
});
|
||||
});
|
||||
734
tests/review-lane-descriptor.test.cjs
Normal file
734
tests/review-lane-descriptor.test.cjs
Normal file
@@ -0,0 +1,734 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Reviewer Lane Descriptor + DEFECT.GENERATIVE-FIX parity (#2794, ADR-2782 Phase 1).
|
||||
*
|
||||
* `CONTEXT.md:797` requires that a constant shared between two parallel surfaces
|
||||
* carry a parity assertion failing when they diverge. The reviewer roster has
|
||||
* never had one: it is declared across four surfaces — the descriptor, the
|
||||
* roster in `review-reviewer-selection.cts`, the `invoke_reviewers` legs, and the
|
||||
* `write_reviews` section headings — and only the Cursor lane has ever been
|
||||
* parity-checked at all.
|
||||
*
|
||||
* The assertion is exercised in BOTH directions, because a forward-only check
|
||||
* ("does every declared lane resolve?") misses the failure this exists to catch:
|
||||
* #2718 added a lane leg and #2781 was the documentation drift that followed. So
|
||||
* every negative row below feeds a SYNTHETIC divergence to the pure checker and
|
||||
* asserts the specific violation — a parity test that has never been seen to
|
||||
* fail is a green light on drift, not a guarantee.
|
||||
*
|
||||
* Assertions are on the frozen `PARITY_VIOLATION` reason enum, never on rendered
|
||||
* prose (CONTRIBUTING.md — "tests assert on typed structured values").
|
||||
*/
|
||||
|
||||
const { describe, test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const fc = require('fast-check');
|
||||
|
||||
const {
|
||||
REVIEWER_LANES,
|
||||
PARITY_VIOLATION,
|
||||
checkReviewerLaneParity,
|
||||
} = require('../gsd-core/bin/lib/review-lane-descriptor.cjs');
|
||||
const {
|
||||
KNOWN_REVIEWER_SLUGS,
|
||||
} = require('../gsd-core/bin/lib/review-reviewer-selection.cjs');
|
||||
|
||||
const ROOT = path.join(__dirname, '..');
|
||||
// Normalized to LF on read so the CRLF cases below can construct a Windows
|
||||
// checkout deterministically from a known-LF baseline.
|
||||
const WORKFLOW_TEXT = fs
|
||||
.readFileSync(path.join(ROOT, 'gsd-core', 'workflows', 'review.md'), 'utf-8')
|
||||
.replace(/\r\n/g, '\n');
|
||||
|
||||
/** Render the LF baseline as a Windows autocrlf checkout would store it. */
|
||||
function asCrlf(text) {
|
||||
return text.split('\n').join('\r\n');
|
||||
}
|
||||
|
||||
/** Run the checker against the shipped inputs, with targeted overrides. */
|
||||
function check(overrides = {}) {
|
||||
return checkReviewerLaneParity({
|
||||
descriptor: REVIEWER_LANES,
|
||||
roster: KNOWN_REVIEWER_SLUGS,
|
||||
workflowText: WORKFLOW_TEXT,
|
||||
...overrides,
|
||||
});
|
||||
}
|
||||
|
||||
/** The violation reasons produced, as a plain sorted array of `reason:subject`. */
|
||||
function reasons(result) {
|
||||
return result.violations.map((v) => `${v.reason}:${v.subject}`).sort();
|
||||
}
|
||||
|
||||
/** A lane object that is structurally valid but names nothing real. */
|
||||
function fakeLane(slug) {
|
||||
return { ...REVIEWER_LANES[0], slug, flags: [`--${slug}`], reviewsSection: slug };
|
||||
}
|
||||
|
||||
describe('reviewer lane parity — the shipped repo', () => {
|
||||
test('descriptor, roster, workflow legs and output sections all agree', () => {
|
||||
const r = check();
|
||||
assert.deepStrictEqual(
|
||||
r.violations,
|
||||
[],
|
||||
`shipped repo must satisfy lane parity; got: ${JSON.stringify(r.violations)}`,
|
||||
);
|
||||
assert.strictEqual(r.ok, true);
|
||||
});
|
||||
|
||||
test('the descriptor covers every roster slug and vice versa', () => {
|
||||
assert.deepStrictEqual(
|
||||
REVIEWER_LANES.map((l) => l.slug).sort(),
|
||||
[...KNOWN_REVIEWER_SLUGS].sort(),
|
||||
);
|
||||
});
|
||||
|
||||
test('parity is evaluated over a non-empty lane set', () => {
|
||||
// Guards the vacuous-truth failure mode: an empty descriptor trivially
|
||||
// satisfies every forward check.
|
||||
assert.ok(REVIEWER_LANES.length >= 11, 'expected at least the 11 shipped lanes');
|
||||
});
|
||||
});
|
||||
|
||||
describe('reviewer lane parity — descriptor vs roster', () => {
|
||||
test('a roster slug with no descriptor entry is a violation', () => {
|
||||
const r = check({ roster: [...KNOWN_REVIEWER_SLUGS, 'kimi_code'] });
|
||||
assert.deepStrictEqual(reasons(r), [
|
||||
`${PARITY_VIOLATION.ROSTER_SLUG_UNDECLARED}:kimi_code`,
|
||||
]);
|
||||
});
|
||||
|
||||
test('a descriptor lane absent from the roster is a violation', () => {
|
||||
const r = check({ descriptor: [...REVIEWER_LANES, fakeLane('acme')] });
|
||||
assert.ok(
|
||||
reasons(r).includes(`${PARITY_VIOLATION.DESCRIPTOR_LANE_NOT_IN_ROSTER}:acme`),
|
||||
`expected a not-in-roster violation, got: ${JSON.stringify(reasons(r))}`,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('reviewer lane parity — descriptor vs invoke_reviewers legs', () => {
|
||||
test('a leg added without a descriptor entry is a violation', () => {
|
||||
// The #2718 shape: a new lane's bash block lands in the workflow and nothing
|
||||
// else moves. This is the row a forward-only assertion cannot catch.
|
||||
const r = check({
|
||||
workflowText: WORKFLOW_TEXT.replace(
|
||||
'<!-- reviewer-lane: qwen -->',
|
||||
'<!-- reviewer-lane: qwen -->\n<!-- reviewer-lane: kimi_code -->',
|
||||
),
|
||||
});
|
||||
assert.deepStrictEqual(reasons(r), [
|
||||
`${PARITY_VIOLATION.LEG_MARKER_UNDECLARED}:kimi_code`,
|
||||
]);
|
||||
});
|
||||
|
||||
test('a declared lane whose workflow leg was removed is a violation', () => {
|
||||
const r = check({
|
||||
workflowText: WORKFLOW_TEXT.replace('<!-- reviewer-lane: qwen -->', ''),
|
||||
});
|
||||
assert.deepStrictEqual(reasons(r), [
|
||||
`${PARITY_VIOLATION.LEG_MARKER_MISSING}:qwen`,
|
||||
]);
|
||||
});
|
||||
|
||||
test('a duplicated leg marker is a violation', () => {
|
||||
const r = check({
|
||||
workflowText: WORKFLOW_TEXT.replace(
|
||||
'<!-- reviewer-lane: qwen -->',
|
||||
'<!-- reviewer-lane: qwen -->\n<!-- reviewer-lane: qwen -->',
|
||||
),
|
||||
});
|
||||
assert.deepStrictEqual(reasons(r), [
|
||||
`${PARITY_VIOLATION.LEG_MARKER_DUPLICATED}:qwen`,
|
||||
]);
|
||||
});
|
||||
|
||||
test('marker matching tolerates whitespace variation', () => {
|
||||
const r = check({
|
||||
workflowText: WORKFLOW_TEXT.replace(
|
||||
'<!-- reviewer-lane: qwen -->',
|
||||
'<!--reviewer-lane:qwen-->',
|
||||
),
|
||||
});
|
||||
assert.deepStrictEqual(r.violations, []);
|
||||
});
|
||||
|
||||
test('a marker outside the invoke_reviewers step does not satisfy the leg', () => {
|
||||
// Scoped, not file-wide: a marker parked in write_reviews must not be
|
||||
// mistaken for a dispatch leg.
|
||||
const moved = WORKFLOW_TEXT
|
||||
.replace('<!-- reviewer-lane: qwen -->', '')
|
||||
.replace('## Qwen Review', '<!-- reviewer-lane: qwen -->\n## Qwen Review');
|
||||
const r = check({ workflowText: moved });
|
||||
assert.deepStrictEqual(reasons(r), [
|
||||
`${PARITY_VIOLATION.LEG_MARKER_MISSING}:qwen`,
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('reviewer lane parity — descriptor vs write_reviews sections', () => {
|
||||
test('an output section with no declared lane is a violation', () => {
|
||||
const r = check({
|
||||
workflowText: WORKFLOW_TEXT.replace(
|
||||
'## Qwen Review',
|
||||
'## Qwen Review\n\n{qwen}\n\n---\n\n## Kimi Review',
|
||||
),
|
||||
});
|
||||
assert.deepStrictEqual(reasons(r), [
|
||||
`${PARITY_VIOLATION.SECTION_UNDECLARED}:Kimi`,
|
||||
]);
|
||||
});
|
||||
|
||||
test('a declared lane with no output section is a violation', () => {
|
||||
const r = check({
|
||||
workflowText: WORKFLOW_TEXT.replace('## Qwen Review', '## Renamed Heading'),
|
||||
});
|
||||
assert.ok(
|
||||
reasons(r).includes(`${PARITY_VIOLATION.SECTION_MISSING}:Qwen`),
|
||||
`expected a section-missing violation, got: ${JSON.stringify(reasons(r))}`,
|
||||
);
|
||||
});
|
||||
|
||||
test('a duplicated output section is a violation', () => {
|
||||
// Two lanes under one heading would silently MERGE in REVIEWS.md, producing
|
||||
// a review that appears to have consensus it does not have (ADR-2782 D8).
|
||||
const r = check({
|
||||
workflowText: WORKFLOW_TEXT.replace(
|
||||
'## Qwen Review',
|
||||
'## Qwen Review\n\n{a}\n\n---\n\n## Qwen Review',
|
||||
),
|
||||
});
|
||||
assert.deepStrictEqual(reasons(r), [
|
||||
`${PARITY_VIOLATION.SECTION_DUPLICATED}:Qwen`,
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('reviewer lane parity — not-corruption (must NOT fire)', () => {
|
||||
test('ADR-1517 instance sections are exempt from lane parity', () => {
|
||||
// `## OpenCode Review (opencode-deepseek)` and `(opencode-mimo)` are already
|
||||
// in the shipped file. ADR-2782 D8: reviewer instances are not lanes. A
|
||||
// naive `## … Review` matcher fails against these on day one.
|
||||
const r = check();
|
||||
assert.deepStrictEqual(r.violations, []);
|
||||
|
||||
const withNewInstance = WORKFLOW_TEXT.replace(
|
||||
'## Qwen Review',
|
||||
'## Qwen Review (qwen-turbo)\n\n{x}\n\n---\n\n## Qwen Review',
|
||||
);
|
||||
assert.deepStrictEqual(
|
||||
checkReviewerLaneParity({
|
||||
descriptor: REVIEWER_LANES,
|
||||
roster: KNOWN_REVIEWER_SLUGS,
|
||||
workflowText: withNewInstance,
|
||||
}).violations,
|
||||
[],
|
||||
'adding a reviewer instance section must not trip lane parity',
|
||||
);
|
||||
});
|
||||
|
||||
test('the h1 title and non-lane headings are not read as lane sections', () => {
|
||||
// `# Cross-AI Plan Review — Phase {N}` contains "Review" but is h1;
|
||||
// `## Consensus Summary` is h2 but has no ` Review` suffix.
|
||||
const r = check();
|
||||
assert.deepStrictEqual(r.violations, []);
|
||||
|
||||
const withExtras = WORKFLOW_TEXT.replace(
|
||||
'## Consensus Summary',
|
||||
'## Another Summary\n\n---\n\n## Consensus Summary',
|
||||
);
|
||||
assert.deepStrictEqual(
|
||||
checkReviewerLaneParity({
|
||||
descriptor: REVIEWER_LANES,
|
||||
roster: KNOWN_REVIEWER_SLUGS,
|
||||
workflowText: withExtras,
|
||||
}).violations,
|
||||
[],
|
||||
);
|
||||
});
|
||||
|
||||
test('bold prose in invoke_reviewers is not read as a leg', () => {
|
||||
// Five non-lane bold labels share the bold-then-fence shape a heuristic
|
||||
// matcher would key on. Adding another must not register a lane.
|
||||
const withProse = WORKFLOW_TEXT.replace(
|
||||
'<!-- reviewer-lane: qwen -->',
|
||||
'**Some new maintainer note (#9999):**\n\n```bash\necho hi\n```\n\n<!-- reviewer-lane: qwen -->',
|
||||
);
|
||||
assert.deepStrictEqual(
|
||||
checkReviewerLaneParity({
|
||||
descriptor: REVIEWER_LANES,
|
||||
roster: KNOWN_REVIEWER_SLUGS,
|
||||
workflowText: withProse,
|
||||
}).violations,
|
||||
[],
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('reviewer lane parity — cross-platform and hostile input', () => {
|
||||
test('parity is CRLF-insensitive', () => {
|
||||
// A Windows autocrlf checkout puts \r on every line; without normalization
|
||||
// every marker and heading would miss and the whole roster would report
|
||||
// missing.
|
||||
const r = check({ workflowText: asCrlf(WORKFLOW_TEXT) });
|
||||
assert.deepStrictEqual(r.violations, []);
|
||||
});
|
||||
|
||||
test('a divergence is still detected under CRLF', () => {
|
||||
const crlf = asCrlf(
|
||||
WORKFLOW_TEXT.replace('<!-- reviewer-lane: qwen -->', ''),
|
||||
);
|
||||
assert.deepStrictEqual(reasons(check({ workflowText: crlf })), [
|
||||
`${PARITY_VIOLATION.LEG_MARKER_MISSING}:qwen`,
|
||||
]);
|
||||
});
|
||||
|
||||
test('empty workflow text degrades to violations rather than throwing', () => {
|
||||
// A read failure must never be mistaken for a clean bill of health.
|
||||
const r = check({ workflowText: '' });
|
||||
assert.strictEqual(r.ok, false);
|
||||
assert.strictEqual(
|
||||
r.violations.filter((v) => v.reason === PARITY_VIOLATION.LEG_MARKER_MISSING)
|
||||
.length,
|
||||
REVIEWER_LANES.length,
|
||||
);
|
||||
});
|
||||
|
||||
test('non-string workflow text is coerced, never thrown on', () => {
|
||||
for (const bad of [undefined, null]) {
|
||||
const r = check({ workflowText: bad });
|
||||
assert.strictEqual(r.ok, false, `expected violations for ${String(bad)}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('repeated evaluation is stable (no leaked regex lastIndex)', () => {
|
||||
// A module-level /g regex carries state between calls and would silently
|
||||
// skip matches on the second invocation.
|
||||
const first = check();
|
||||
const second = check();
|
||||
assert.deepStrictEqual(second.violations, first.violations);
|
||||
assert.deepStrictEqual(second.violations, []);
|
||||
});
|
||||
});
|
||||
|
||||
describe('reviewer lane parity — descriptor-internal uniqueness (ADR-2782 D8)', () => {
|
||||
test('duplicate lane slugs are a violation', () => {
|
||||
const r = check({ descriptor: [...REVIEWER_LANES, REVIEWER_LANES[0]] });
|
||||
assert.ok(reasons(r).some((x) => x.startsWith(PARITY_VIOLATION.DUPLICATE_SLUG)));
|
||||
});
|
||||
|
||||
test('duplicate lane flags are a violation', () => {
|
||||
const clash = { ...fakeLane('acme'), flags: ['--gemini'] };
|
||||
const r = check({ descriptor: [...REVIEWER_LANES, clash] });
|
||||
assert.ok(
|
||||
reasons(r).includes(`${PARITY_VIOLATION.DUPLICATE_FLAG}:--gemini`),
|
||||
`expected a duplicate-flag violation, got: ${JSON.stringify(reasons(r))}`,
|
||||
);
|
||||
});
|
||||
|
||||
test('duplicate reviewsSection is a violation', () => {
|
||||
const clash = { ...fakeLane('acme'), reviewsSection: 'Gemini' };
|
||||
const r = check({ descriptor: [...REVIEWER_LANES, clash] });
|
||||
assert.ok(
|
||||
reasons(r).includes(`${PARITY_VIOLATION.DUPLICATE_SECTION}:Gemini`),
|
||||
`expected a duplicate-section violation, got: ${JSON.stringify(reasons(r))}`,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* `checkReviewerLaneParity` parses markdown for lane markers and section
|
||||
* headings, so CLAUDE.md's TEST RULES ("Parsers... must include at least one
|
||||
* fast-check property test") applies.
|
||||
*
|
||||
* FIXTURE PROVENANCE (CONTRIBUTING.md #2371): the generators below are
|
||||
* DOCUMENT-shaped, not writer-seeded. They emit arbitrary markdown — arbitrary
|
||||
* noise lines, arbitrary heading levels, arbitrary bold labels, markers placed
|
||||
* at arbitrary positions — rather than being built by calling the same regexes
|
||||
* the checker uses. A generator seeded from the module's own matchers could only
|
||||
* ever produce documents the matchers already recognize, which makes the
|
||||
* document shape a constant and the property unfalsifiable.
|
||||
*
|
||||
* Deterministic per repo rules: seed pinned, run count bounded.
|
||||
*/
|
||||
describe('reviewer lane parity — properties', () => {
|
||||
const FC = { seed: 20260729, numRuns: 200 };
|
||||
|
||||
/** Slugs inside the declared grammar — the only ones a marker can carry. */
|
||||
const slugArb = fc.stringMatching(/^[a-z][a-z0-9_-]{0,12}$/);
|
||||
|
||||
/**
|
||||
* Slugs OUTSIDE the grammar. These are unmatchable by LEG_MARKER_RE, so the
|
||||
* contract is that they are reported as INVALID_SLUG rather than silently
|
||||
* reported missing. Includes regex metacharacters and prototype-pollution
|
||||
* shaped keys.
|
||||
*/
|
||||
const badSlugArb = fc.constantFrom(
|
||||
'a.b', 'a*b', 'a+b', '(a)', '[a]', 'a|b', 'A', '-lead', '_lead', '__proto__', '',
|
||||
);
|
||||
|
||||
/**
|
||||
* Slugs that ARE inside the grammar but collide with Object.prototype keys.
|
||||
* `constructor` is all-lowercase, so it is a legitimate slug — the risk is
|
||||
* prototype pollution in the counting maps, not validation.
|
||||
*/
|
||||
const prototypeKeyArb = fc.constantFrom('constructor', 'tostring', 'valueof', 'hasownproperty');
|
||||
|
||||
/** Arbitrary markdown noise that must never be read as a marker or a lane heading. */
|
||||
const noiseArb = fc.array(
|
||||
fc.oneof(
|
||||
fc.constant(''),
|
||||
fc.constant('**Timeout guidance (#2194):**'),
|
||||
fc.constant('```bash'),
|
||||
fc.constant('```'),
|
||||
fc.constant('# Cross-AI Plan Review — Phase {N}'),
|
||||
fc.constant('## Consensus Summary'),
|
||||
fc.constant('### Agreed Concerns'),
|
||||
fc.constant('<!-- not-a-lane-marker: xyz -->'),
|
||||
fc.stringMatching(/^[A-Za-z0-9 ,.()#*_-]{0,40}$/),
|
||||
),
|
||||
{ maxLength: 12 },
|
||||
);
|
||||
|
||||
/** Build a review.md-shaped document declaring exactly `slugs`. */
|
||||
function docFor(slugs, sections, noise, eol) {
|
||||
const legs = slugs.map((s) => `<!-- reviewer-lane: ${s} -->\n**${s}:**`).join('\n');
|
||||
const heads = sections.map((s) => `## ${s} Review`).join('\n\n');
|
||||
const body = [
|
||||
'<step name="invoke_reviewers">',
|
||||
...noise,
|
||||
legs,
|
||||
'</step>',
|
||||
'<step name="write_reviews">',
|
||||
...noise,
|
||||
heads,
|
||||
'## Consensus Summary',
|
||||
'</step>',
|
||||
].join('\n');
|
||||
return body.split('\n').join(eol);
|
||||
}
|
||||
|
||||
const laneSetArb = fc
|
||||
.uniqueArray(slugArb, { minLength: 1, maxLength: 6 })
|
||||
.map((slugs) =>
|
||||
slugs.map((slug, i) => ({
|
||||
slug,
|
||||
flags: [`--${slug}`],
|
||||
transport: 'spawn',
|
||||
probe: { kind: 'command-exists', binary: slug },
|
||||
invoke: {
|
||||
binary: slug,
|
||||
args: [],
|
||||
promptChannel: 'stdin',
|
||||
outputChannel: 'stdout',
|
||||
modelArg: null,
|
||||
effortChannel: 'none',
|
||||
},
|
||||
timeoutFloorMs: 1000,
|
||||
emptyOutput: 'stub-with-stderr',
|
||||
// Section names are index-tagged so they stay unique even when two slugs
|
||||
// differ only by a character the heading grammar would not distinguish.
|
||||
reviewsSection: `Sec${i}`,
|
||||
evidenceClass: 'source-grounded',
|
||||
requiresBinaries: [],
|
||||
promptBudgetKey: null,
|
||||
handler: null,
|
||||
})),
|
||||
);
|
||||
|
||||
test('never throws on arbitrary input, and ok always agrees with the violations', () => {
|
||||
// Totality is a real requirement, not a nicety: Phase 2 (#2795) feeds this
|
||||
// same function manifest-derived data from third-party overlays. A checker
|
||||
// that throws on bad input cannot report on it, and a parity gate that
|
||||
// crashes is indistinguishable from one that was never run.
|
||||
fc.assert(
|
||||
fc.property(
|
||||
fc.anything(),
|
||||
fc.anything(),
|
||||
fc.anything(),
|
||||
(descriptor, roster, workflowText) => {
|
||||
let r;
|
||||
try {
|
||||
r = checkReviewerLaneParity({ descriptor, roster, workflowText });
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
return (
|
||||
Array.isArray(r.violations) && r.ok === (r.violations.length === 0)
|
||||
);
|
||||
},
|
||||
),
|
||||
FC,
|
||||
);
|
||||
});
|
||||
|
||||
test('a slug outside the marker grammar is reported, never silently missing', () => {
|
||||
// The silent-miss this prevents: LEG_MARKER_RE captures only [a-z0-9_-], so
|
||||
// a slug like `acme.reviewer` can have a present, correct marker that the
|
||||
// scan can never see — reporting LEG_MARKER_MISSING forever with no clue why.
|
||||
fc.assert(
|
||||
fc.property(badSlugArb, (bad) => {
|
||||
const lane = { ...fakeLane('placeholder'), slug: bad };
|
||||
const r = checkReviewerLaneParity({
|
||||
descriptor: [lane],
|
||||
roster: [bad],
|
||||
workflowText: `<step name="invoke_reviewers">\n<!-- reviewer-lane: ${bad} -->\n</step>`,
|
||||
});
|
||||
const reasonsOut = r.violations.map((v) => v.reason);
|
||||
return (
|
||||
reasonsOut.includes(PARITY_VIOLATION.INVALID_SLUG) &&
|
||||
!reasonsOut.includes(PARITY_VIOLATION.LEG_MARKER_MISSING)
|
||||
);
|
||||
}),
|
||||
FC,
|
||||
);
|
||||
});
|
||||
|
||||
test('a prototype-key slug behaves like any other valid slug', () => {
|
||||
// The counting layer uses Map/Set, not bare objects, so a slug named
|
||||
// `constructor` cannot reach Object.prototype. Locking it: a bare-object
|
||||
// counter would make this lane appear present when it is absent.
|
||||
fc.assert(
|
||||
fc.property(prototypeKeyArb, (name) => {
|
||||
const lane = { ...fakeLane('placeholder'), slug: name, reviewsSection: 'Sec' };
|
||||
const declared = checkReviewerLaneParity({
|
||||
descriptor: [lane],
|
||||
roster: [name],
|
||||
workflowText:
|
||||
`<step name="invoke_reviewers">\n<!-- reviewer-lane: ${name} -->\n</step>\n` +
|
||||
'<step name="write_reviews">\n## Sec Review\n</step>',
|
||||
});
|
||||
const absent = checkReviewerLaneParity({
|
||||
descriptor: [lane],
|
||||
roster: [name],
|
||||
workflowText: '<step name="invoke_reviewers">\n</step>',
|
||||
});
|
||||
return (
|
||||
declared.ok &&
|
||||
absent.violations.some(
|
||||
(v) => v.reason === PARITY_VIOLATION.LEG_MARKER_MISSING && v.subject === name,
|
||||
)
|
||||
);
|
||||
}),
|
||||
FC,
|
||||
);
|
||||
});
|
||||
|
||||
test('a non-object lane entry is reported as malformed, not thrown on', () => {
|
||||
fc.assert(
|
||||
fc.property(
|
||||
fc.constantFrom(0, 1, '', 'x', null, true, [], NaN),
|
||||
(junk) => {
|
||||
const r = checkReviewerLaneParity({
|
||||
descriptor: [junk],
|
||||
roster: [],
|
||||
workflowText: '',
|
||||
});
|
||||
const reasonsOut = r.violations.map((v) => v.reason);
|
||||
return (
|
||||
reasonsOut.includes(PARITY_VIOLATION.MALFORMED_LANE) ||
|
||||
reasonsOut.includes(PARITY_VIOLATION.INVALID_SLUG)
|
||||
);
|
||||
},
|
||||
),
|
||||
FC,
|
||||
);
|
||||
});
|
||||
|
||||
test('a document declaring exactly the descriptor satisfies parity', () => {
|
||||
fc.assert(
|
||||
fc.property(laneSetArb, noiseArb, fc.constantFrom('\n', '\r\n'), (lanes, noise, eol) => {
|
||||
const doc = docFor(
|
||||
lanes.map((l) => l.slug),
|
||||
lanes.map((l) => l.reviewsSection),
|
||||
noise,
|
||||
eol,
|
||||
);
|
||||
const r = checkReviewerLaneParity({
|
||||
descriptor: lanes,
|
||||
roster: lanes.map((l) => l.slug),
|
||||
workflowText: doc,
|
||||
});
|
||||
return r.ok;
|
||||
}),
|
||||
FC,
|
||||
);
|
||||
});
|
||||
|
||||
test('removing one lane marker always yields exactly that lane missing', () => {
|
||||
fc.assert(
|
||||
fc.property(laneSetArb, noiseArb, fc.nat(), (lanes, noise, pick) => {
|
||||
const victim = lanes[pick % lanes.length];
|
||||
const kept = lanes.filter((l) => l.slug !== victim.slug);
|
||||
const doc = docFor(
|
||||
kept.map((l) => l.slug),
|
||||
lanes.map((l) => l.reviewsSection),
|
||||
noise,
|
||||
'\n',
|
||||
);
|
||||
const r = checkReviewerLaneParity({
|
||||
descriptor: lanes,
|
||||
roster: lanes.map((l) => l.slug),
|
||||
workflowText: doc,
|
||||
});
|
||||
const missing = r.violations.filter(
|
||||
(v) => v.reason === PARITY_VIOLATION.LEG_MARKER_MISSING,
|
||||
);
|
||||
return missing.length === 1 && missing[0].subject === victim.slug;
|
||||
}),
|
||||
FC,
|
||||
);
|
||||
});
|
||||
|
||||
test('evaluation is deterministic across repeated calls', () => {
|
||||
// Guards regex lastIndex leaking between invocations of a module-level /g.
|
||||
fc.assert(
|
||||
fc.property(laneSetArb, noiseArb, (lanes, noise) => {
|
||||
const input = {
|
||||
descriptor: lanes,
|
||||
roster: lanes.map((l) => l.slug),
|
||||
workflowText: docFor(
|
||||
lanes.map((l) => l.slug),
|
||||
lanes.map((l) => l.reviewsSection),
|
||||
noise,
|
||||
'\n',
|
||||
),
|
||||
};
|
||||
const a = checkReviewerLaneParity(input);
|
||||
const b = checkReviewerLaneParity(input);
|
||||
return JSON.stringify(a) === JSON.stringify(b);
|
||||
}),
|
||||
FC,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('reviewer lane descriptor — declared shape (ADR-2782 D1/D2/D6/D7)', () => {
|
||||
test('every lane declares a closed transport at the lane level', () => {
|
||||
// ADR-2782 D1 places `transport` as a sibling of `probe` and `invoke`, not
|
||||
// nested inside `invoke`. Locking the placement keeps Phase 2's manifest
|
||||
// harvest free of a translation step.
|
||||
for (const lane of REVIEWER_LANES) {
|
||||
assert.ok(
|
||||
['spawn', 'openai-http'].includes(lane.transport),
|
||||
`${lane.slug}: unexpected transport ${lane.transport}`,
|
||||
);
|
||||
assert.strictEqual(
|
||||
lane.invoke.transport,
|
||||
undefined,
|
||||
`${lane.slug}: transport must not be duplicated inside invoke`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('the transport sub-shape is respected per lane', () => {
|
||||
// A descriptor carrying fields from both sub-shapes — or neither — has
|
||||
// undefined meaning, which is what a closed vocabulary exists to prevent.
|
||||
for (const lane of REVIEWER_LANES) {
|
||||
const i = lane.invoke;
|
||||
if (lane.transport === 'spawn') {
|
||||
assert.ok(i.binary, `${lane.slug}: spawn lane must declare a binary`);
|
||||
assert.ok(Array.isArray(i.args), `${lane.slug}: spawn lane must declare args`);
|
||||
assert.strictEqual(i.hostConfigKey, undefined, `${lane.slug}: spawn lane must not declare hostConfigKey`);
|
||||
} else {
|
||||
assert.ok(i.hostConfigKey, `${lane.slug}: http lane must declare hostConfigKey`);
|
||||
assert.ok(i.path, `${lane.slug}: http lane must declare a path`);
|
||||
assert.strictEqual(i.binary, undefined, `${lane.slug}: http lane must not declare a binary`);
|
||||
assert.strictEqual(i.effortChannel, 'none', `${lane.slug}: http lanes carry no effort channel`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('every probe kind is in the closed enum', () => {
|
||||
for (const lane of REVIEWER_LANES) {
|
||||
assert.ok(
|
||||
['command-exists', 'command-capability', 'http-reachable'].includes(lane.probe.kind),
|
||||
`${lane.slug}: unexpected probe kind ${lane.probe.kind}`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('every probe that opens a connection declares a bound', () => {
|
||||
// DEFECT.UNBOUNDED-SUBPROCESS: an unbounded probe hangs every future review,
|
||||
// including reviews that never asked for that lane.
|
||||
for (const lane of REVIEWER_LANES) {
|
||||
if (lane.probe.kind === 'command-exists') continue;
|
||||
assert.ok(
|
||||
Number.isInteger(lane.probe.timeoutMs) && lane.probe.timeoutMs > 0,
|
||||
`${lane.slug}: ${lane.probe.kind} probe must declare a positive timeoutMs`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('handler is a closed first-party enum', () => {
|
||||
const allowed = [null, 'antigravity', 'openai-compatible'];
|
||||
for (const lane of REVIEWER_LANES) {
|
||||
assert.ok(allowed.includes(lane.handler), `${lane.slug}: unexpected handler ${lane.handler}`);
|
||||
}
|
||||
assert.deepStrictEqual(
|
||||
REVIEWER_LANES.filter((l) => l.handler !== null).map((l) => l.slug).sort(),
|
||||
['antigravity', 'llama_cpp', 'lm_studio', 'ollama'],
|
||||
);
|
||||
});
|
||||
|
||||
test('every lane declares a positive timeout floor', () => {
|
||||
for (const lane of REVIEWER_LANES) {
|
||||
assert.ok(
|
||||
Number.isInteger(lane.timeoutFloorMs) && lane.timeoutFloorMs > 0,
|
||||
`${lane.slug}: timeoutFloorMs must be a positive integer`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('empty-output policy is normalized across lanes', () => {
|
||||
// Only Antigravity opts out, and it does so by owning its own diagnostics
|
||||
// through a handler (ADR-2782 D6) — not by discarding stderr.
|
||||
assert.deepStrictEqual(
|
||||
REVIEWER_LANES.filter((l) => l.emptyOutput !== 'stub-with-stderr').map((l) => l.slug),
|
||||
['antigravity'],
|
||||
);
|
||||
});
|
||||
|
||||
test('the descriptor table is frozen', () => {
|
||||
assert.ok(Object.isFrozen(REVIEWER_LANES));
|
||||
for (const lane of REVIEWER_LANES) {
|
||||
assert.ok(Object.isFrozen(lane), `${lane.slug}: lane must be frozen`);
|
||||
}
|
||||
});
|
||||
|
||||
test('flag uniqueness holds across the flattened multi-flag set', () => {
|
||||
// ADR-2782 D8 states uniqueness over a singular `reviewer.flag`; this module
|
||||
// widens that field to `flags[]` (Antigravity is --antigravity AND --agy), so
|
||||
// the invariant is enforced over every lane's flattened flag set.
|
||||
const all = REVIEWER_LANES.flatMap((l) => l.flags);
|
||||
assert.deepStrictEqual([...new Set(all)].sort(), [...all].sort());
|
||||
assert.ok(
|
||||
REVIEWER_LANES.some((l) => l.flags.length > 1),
|
||||
'expected at least one multi-flag lane, else the widening is untested',
|
||||
);
|
||||
});
|
||||
|
||||
test('the violation reason enum is locked', () => {
|
||||
// Adding a reason is three coordinated changes: enum, emitting site, and
|
||||
// this assertion.
|
||||
assert.deepStrictEqual(Object.keys(PARITY_VIOLATION).sort(), [
|
||||
'DESCRIPTOR_LANE_NOT_IN_ROSTER',
|
||||
'DUPLICATE_FLAG',
|
||||
'DUPLICATE_SECTION',
|
||||
'DUPLICATE_SLUG',
|
||||
'INVALID_SLUG',
|
||||
'LEG_MARKER_DUPLICATED',
|
||||
'LEG_MARKER_MISSING',
|
||||
'LEG_MARKER_UNDECLARED',
|
||||
'MALFORMED_LANE',
|
||||
'ROSTER_SLUG_UNDECLARED',
|
||||
'SECTION_DUPLICATED',
|
||||
'SECTION_MISSING',
|
||||
'SECTION_UNDECLARED',
|
||||
]);
|
||||
assert.ok(Object.isFrozen(PARITY_VIOLATION));
|
||||
});
|
||||
});
|
||||
@@ -144,3 +144,166 @@ describe('resolveReviewerSelection', () => {
|
||||
assert.deepStrictEqual(r.errors, []);
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* ADR-2782 D4 (#2794) — absent-safe governs DISCOVERY, never explicit selection.
|
||||
*
|
||||
* "Not finding a lane nobody asked for is normal; failing to run a lane somebody
|
||||
* asked for is an error."
|
||||
*
|
||||
* Before this change every explicit miss was an `info`. A TOTAL miss still
|
||||
* errored, but only as a side effect of the selected set coming out empty — so
|
||||
* the PARTIAL miss (`--gemini --qwen` on a host without qwen) had no signal at
|
||||
* all: the review ran with a thinner reviewer set and present_results reported
|
||||
* success. The workflow's own guidance names why that is wrong — "a cross-AI
|
||||
* review that silently drops a lane is blind in one eye".
|
||||
*/
|
||||
describe('resolveReviewerSelection — explicit flags are an assertion (ADR-2782 D4)', () => {
|
||||
const errorsMentioning = (r, slug) => r.errors.filter((e) => e.includes(slug));
|
||||
|
||||
test('an explicit flag for a detected reviewer selects it with no message', () => {
|
||||
const r = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
explicitFlags: ['gemini'],
|
||||
});
|
||||
assert.deepStrictEqual(r.selected, ['gemini']);
|
||||
assert.deepStrictEqual(r.errors, []);
|
||||
assert.deepStrictEqual(r.infos, []);
|
||||
});
|
||||
|
||||
test('a partial explicit miss errors instead of degrading silently', () => {
|
||||
// THE regression row. Pre-fix this produced errors: [] and an info note,
|
||||
// and the run proceeded one-eyed.
|
||||
const r = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
explicitFlags: ['gemini', 'qwen'],
|
||||
});
|
||||
assert.deepStrictEqual(r.selected, ['gemini'], 'the detected lane is still selected');
|
||||
assert.strictEqual(
|
||||
errorsMentioning(r, 'qwen').length,
|
||||
1,
|
||||
`expected exactly one error naming qwen, got: ${JSON.stringify(r.errors)}`,
|
||||
);
|
||||
assert.deepStrictEqual(r.infos, [], 'the miss must not be downgraded to an info');
|
||||
});
|
||||
|
||||
test('a sole explicit flag that is undetected errors per-slug and in aggregate', () => {
|
||||
const r = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
explicitFlags: ['qwen'],
|
||||
});
|
||||
assert.deepStrictEqual(r.selected, []);
|
||||
assert.strictEqual(errorsMentioning(r, 'qwen').length, 1);
|
||||
// The pre-existing aggregate message is preserved, not replaced — the
|
||||
// per-slug errors must not suppress it.
|
||||
assert.ok(
|
||||
r.errors.some((e) => e.includes('no selected reviewers are available')),
|
||||
`expected the aggregate error to survive, got: ${JSON.stringify(r.errors)}`,
|
||||
);
|
||||
});
|
||||
|
||||
test('every missing explicit flag produces its own error, in a stable order', () => {
|
||||
const forward = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
explicitFlags: ['gemini', 'qwen', 'codex'],
|
||||
});
|
||||
const reversed = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
explicitFlags: ['gemini', 'codex', 'qwen'],
|
||||
});
|
||||
assert.strictEqual(errorsMentioning(forward, 'qwen').length, 1);
|
||||
assert.strictEqual(errorsMentioning(forward, 'codex').length, 1);
|
||||
// Order must not depend on the order flags appeared on the command line.
|
||||
assert.deepStrictEqual(forward.errors, reversed.errors);
|
||||
});
|
||||
|
||||
test('a duplicate explicit flag produces exactly one error', () => {
|
||||
const r = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
explicitFlags: ['qwen', 'qwen'],
|
||||
});
|
||||
assert.strictEqual(errorsMentioning(r, 'qwen').length, 1);
|
||||
});
|
||||
|
||||
test('explicit flag matching is case-insensitive', () => {
|
||||
const r = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
explicitFlags: ['QWEN'],
|
||||
});
|
||||
assert.strictEqual(errorsMentioning(r, 'qwen').length, 1);
|
||||
});
|
||||
|
||||
test('an explicit flag with nothing detected at all errors', () => {
|
||||
const r = resolveReviewerSelection({
|
||||
detected: [],
|
||||
explicitFlags: ['gemini'],
|
||||
});
|
||||
assert.deepStrictEqual(r.selected, []);
|
||||
assert.strictEqual(errorsMentioning(r, 'gemini').length, 1);
|
||||
});
|
||||
|
||||
test('pre-existing config errors do not suppress per-slug explicit errors', () => {
|
||||
const r = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
explicitFlags: ['qwen'],
|
||||
configuredDefaultReviewers: 'not-an-array',
|
||||
});
|
||||
assert.strictEqual(errorsMentioning(r, 'qwen').length, 1);
|
||||
assert.ok(r.errors.some((e) => e.includes('must be a JSON array')));
|
||||
// Guarded on the PRE-branch error count, so the aggregate fires exactly when
|
||||
// it did before this change — i.e. not here, because a config error already
|
||||
// existed.
|
||||
assert.ok(!r.errors.some((e) => e.includes('no selected reviewers are available')));
|
||||
});
|
||||
|
||||
test('non-string explicit flags are coerced, never thrown on', () => {
|
||||
const r = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
explicitFlags: [null, 0, { a: 1 }],
|
||||
});
|
||||
assert.strictEqual(r.source, 'explicit_flags');
|
||||
assert.strictEqual(r.errors.length, 3 + 1, 'three unknown flags plus the aggregate');
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* The other half of D4, and the reason the carve-out is scoped to explicit
|
||||
* flags only: discovery paths stay lenient. `--all` is a quantifier over what
|
||||
* exists; `review.default_reviewers` is a preference evaluated across many
|
||||
* hosts. Neither is an assertion about a specific lane, so neither errors.
|
||||
*/
|
||||
describe('resolveReviewerSelection — discovery paths stay lenient (ADR-2782 D4)', () => {
|
||||
test('--all does not error on undetected lanes', () => {
|
||||
const r = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
allFlag: true,
|
||||
});
|
||||
assert.equal(r.source, 'all_flag');
|
||||
assert.deepStrictEqual(r.selected, ['gemini']);
|
||||
assert.deepStrictEqual(r.errors, []);
|
||||
assert.deepStrictEqual(r.infos, []);
|
||||
});
|
||||
|
||||
test('a configured default that is undetected stays an info, not an error', () => {
|
||||
const r = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
configuredDefaultReviewers: ['gemini', 'codex'],
|
||||
});
|
||||
assert.equal(r.source, 'config_default');
|
||||
assert.deepStrictEqual(r.selected, ['gemini']);
|
||||
assert.deepStrictEqual(r.errors, [], 'a preference miss must not become an error');
|
||||
assert.ok(
|
||||
r.infos.some((i) => i.includes('codex')),
|
||||
`expected an info naming codex, got: ${JSON.stringify(r.infos)}`,
|
||||
);
|
||||
});
|
||||
|
||||
test('an unknown configured slug stays a warning', () => {
|
||||
const r = resolveReviewerSelection({
|
||||
detected: ['gemini'],
|
||||
configuredDefaultReviewers: ['gemini', '__nope__'],
|
||||
});
|
||||
assert.deepStrictEqual(r.errors, []);
|
||||
assert.ok(r.warnings.some((w) => w.includes('__nope__')));
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user