Files
msd-core/tests/review-reviewer-selection.test.cjs
Tom Boucher 8b44a0da43 chore(#2794): single-source the reviewer invocation contract + parity assertion (#2820)
* chore(#2794): single-source the reviewer invocation contract

Phase 1 of epic #2782 (ADR-2782). Introduces one core descriptor table as
the declared contract for all 11 cross-AI reviewer lanes, and the
DEFECT.GENERATIVE-FIX parity assertion the roster has never had.

The lane contract lived in three unrelated surfaces — the roster, ~640
lines of hand-authored per-CLI bash in invoke_reviewers, and the
write_reviews section headings — so cross-cutting fixes landed per-leg
(#2494 and #2605 were the same empty-output defect filed twice).

- src/review-lane-descriptor.cts: frozen table declaring per lane the
  slug, flags, probe, invoke shape, timeout floor, empty-output policy,
  REVIEWS.md section, evidence class, required binaries, prompt-budget
  key and handler. Field names track ADR-2782 D1/D2/D6/D7 verbatim so
  Phase 2 harvests the shape with no translation layer. It declares;
  it does not execute — invoke_reviewers iterates in Phase 5b.
- checkReviewerLaneParity: bidirectional parity across descriptor,
  roster, invoke_reviewers legs and write_reviews sections. Forward-only
  would miss the failure it exists to catch (#2718 added a leg, #2781
  was the drift). ADR-1517 instance headings are exempt per D8.
- Legs carry an explicit <!-- reviewer-lane: slug --> marker; five
  non-lane bold labels share the bold-then-fence shape a heuristic
  matcher would key on.
- ADR-2782 D4: an explicitly-flagged reviewer that cannot run is now an
  error in both the core module and the workflow prose that mirrors it.
  A code-only change would be unobservable — the module has no
  production caller; the workflow narrates the policy. Discovery paths
  (--all, review.default_reviewers) stay lenient.
- Fixes the qwen leg, the last one discarding stderr to /dev/null.

Two ADR-2782 D2 vocabulary widenings were forced by surveying the
shipped legs: promptChannel 'none' (CodeRabbit is fed no prompt) and
outputChannel 'file-arg' (Codex writes via -o and discards stdout,
#1698). Both are additive and closed; Phase 2 owns the validator.

Closes #2690

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#2794): make the parity checker total and pin the lane slug grammar

Findings from the orthogonal review passes.

Spec axis — the module claimed its vocabulary tracked ADR-2782 D1/D2
"verbatim" while diverging in three undisclosed ways, which is the
translation layer Phase 2 was supposed to be spared:
- `transport` moves from `invoke.transport` to the LANE level, a sibling
  of `probe`/`invoke`, exactly as D1's manifest example places it. The
  nested form read better as a TS discriminated union; the union is now
  discriminated at the lane level instead, which costs nothing.
- The header and the CONTEXT.md glossary now enumerate all FOUR
  widenings (adding `outputArg` and `flags[]`), not two.

Standards axis — CLAUDE.md requires a fast-check property test for a
parser, and `checkReviewerLaneParity` parses markdown for markers and
headings. Adding one found two real defects that the hand-written
matrix missed:
- NOT TOTAL: a malformed descriptor entry threw on `lane.flags`
  iteration, contradicting the module's own "never throws" claim. Every
  field is now narrowed from `unknown` at the trust boundary and
  reported as MALFORMED_LANE / INVALID_SLUG. This matters because
  Phase 2 feeds this function third-party overlay data, and a parity
  gate that crashes is indistinguishable from one never run.
- SILENT GRAMMAR MISMATCH: LEG_MARKER_RE captures only [a-z0-9_-], so a
  slug outside that class was unmatchable — its marker could be present
  and correct and the scan would still report LEG_MARKER_MISSING
  forever. LANE_SLUG_RE now pins the grammar and a violating slug is
  reported INVALID_SLUG. A loud named violation beats a silent miss.

Generators are document-shaped, not writer-seeded (CONTRIBUTING #2371):
seeding from the module's own matchers could only produce documents
those matchers already recognize.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#2794): register the new bin/lib module in the ESLint ignore list

The remote runner caught this; lint:ci did not, because the invariant
lives in the test suite rather than the lint chain:

  tests/repo-invariants.test.cjs
  "each bin/lib/*.cjs is linted xor ignored according to migration state"
  -> tsc-generated bin/lib modules not yet added to ESLint ignore list:
     review-lane-descriptor.cjs

Adding a src/*.cts module ripples to six surfaces (.gitignore, the
ESLint ignore list, docs/INVENTORY-MANIFEST.json, the CONTEXT.md
glossary, the capability/inventory manifests, and any size baseline).
The other five were covered; this was the miss.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(#2794): amend ADR-2782 D1/D2/D8 with the vocabulary Phase 1 surfaced

Building the Phase 1 descriptor table against all eleven shipped legs is
the first time every lane's contract was written in one place, and it
surfaced four cases the ADR's original survey did not cover. Amending
the design lock rather than diverging from it, so Phase 2 (#2795)
implements the manifest validator against the amended vocabulary instead
of rediscovering the gaps.

All four are additive widenings of closed enums; no decision reverses:

- D2 promptChannel gains `none` — coderabbit is fed no prompt at all, it
  reviews the working-tree diff.
- D2 outputChannel gains `file-arg` — the ADR called a file-writing lane
  a shape a real CLI *could* take; codex already is one, writing via
  -o/--output-last-message and discarding stdout (#1698).
- D2 gains `outputArg`, required iff file-arg — knowing the review lands
  in a file is useless without the argument naming it.
- D1 `flag` becomes `flags[]` and D8's uniqueness flattens across lanes —
  antigravity is selected by both --antigravity and --agy, which a
  single-valued field cannot express.

This is the same evidence path that produced the openai-http transport:
the vocabulary widens on a lane that exists, under review, never on
speculation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(#2794): backfill changeset pr number to 2820

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 07:31:32 -04:00

310 lines
11 KiB
JavaScript

'use strict';
/**
* Characterization tests for the reviewer selection module.
* Locks the normalizeConfiguredDefaultReviewers and resolveReviewerSelection
* export shapes and key policy decisions.
*/
const { describe, test } = require('node:test');
const assert = require('node:assert/strict');
const {
KNOWN_REVIEWER_SLUGS,
normalizeConfiguredDefaultReviewers,
resolveReviewerSelection,
} = require('../gsd-core/bin/lib/review-reviewer-selection.cjs');
describe('KNOWN_REVIEWER_SLUGS', () => {
test('known slug appears in selected with no warning; unknown slug produces a warning and is dropped', () => {
const knownSlug = KNOWN_REVIEWER_SLUGS[0];
const unknownSlug = '__not_a_real_reviewer__';
const knownResult = resolveReviewerSelection({
detected: [knownSlug],
explicitFlags: [],
allFlag: false,
configuredDefaultReviewers: [knownSlug],
});
assert.ok(
knownResult.selected.includes(knownSlug),
`expected known slug "${knownSlug}" to appear in selected`,
);
assert.ok(
knownResult.warnings.length === 0,
`expected no warnings for known slug "${knownSlug}", got: ${JSON.stringify(knownResult.warnings)}`,
);
const unknownResult = resolveReviewerSelection({
detected: [unknownSlug],
explicitFlags: [],
allFlag: false,
configuredDefaultReviewers: [unknownSlug],
});
assert.ok(
!unknownResult.selected.includes(unknownSlug),
`expected unknown slug "${unknownSlug}" to be dropped from selected`,
);
assert.ok(
unknownResult.warnings.some((w) => w.includes(unknownSlug)),
`expected a warning mentioning "${unknownSlug}", got: ${JSON.stringify(unknownResult.warnings)}`,
);
});
});
describe('normalizeConfiguredDefaultReviewers', () => {
test('returns absent=true for undefined', () => {
const r = normalizeConfiguredDefaultReviewers(undefined);
assert.ok(r.absent);
assert.deepStrictEqual(r.values, []);
assert.deepStrictEqual(r.errors, []);
});
test('returns absent=true for null', () => {
const r = normalizeConfiguredDefaultReviewers(null);
assert.ok(r.absent);
});
test('returns error for non-array', () => {
const r = normalizeConfiguredDefaultReviewers('gemini');
assert.ok(!r.absent);
assert.ok(r.errors.length > 0);
});
test('returns error for empty array', () => {
const r = normalizeConfiguredDefaultReviewers([]);
assert.ok(!r.absent);
assert.ok(r.errors.length > 0);
});
test('normalizes slugs to lowercase', () => {
const r = normalizeConfiguredDefaultReviewers(['Gemini', 'CLAUDE']);
assert.ok(!r.absent);
assert.ok(r.values.includes('gemini'));
assert.ok(r.values.includes('claude'));
});
test('deduplicates slugs case-insensitively', () => {
const r = normalizeConfiguredDefaultReviewers(['gemini', 'GEMINI']);
assert.ok(!r.absent);
assert.strictEqual(r.values.filter((s) => s === 'gemini').length, 1);
});
test('records error for invalid slug format', () => {
const r = normalizeConfiguredDefaultReviewers(['gem@ini']);
assert.ok(r.errors.some((e) => e.includes('invalid reviewer slug')));
});
});
describe('resolveReviewerSelection', () => {
test('explicit_flags source — returns intersection of flags and detected', () => {
const r = resolveReviewerSelection({
detected: ['gemini', 'claude'],
explicitFlags: ['gemini'],
allFlag: false,
});
assert.equal(r.source, 'explicit_flags');
assert.deepStrictEqual(r.selected, ['gemini']);
});
test('all_flag source — returns all detected', () => {
const r = resolveReviewerSelection({
detected: ['gemini', 'claude'],
explicitFlags: [],
allFlag: true,
});
assert.equal(r.source, 'all_flag');
assert.ok(r.selected.includes('gemini'));
assert.ok(r.selected.includes('claude'));
});
test('no_config_all_detected source — returns all detected when no config', () => {
const r = resolveReviewerSelection({
detected: ['gemini'],
explicitFlags: [],
allFlag: false,
});
assert.equal(r.source, 'no_config_all_detected');
assert.deepStrictEqual(r.selected, ['gemini']);
});
test('selected is sorted alphabetically', () => {
const r = resolveReviewerSelection({
detected: ['claude', 'gemini'],
explicitFlags: [],
allFlag: true,
});
assert.deepStrictEqual(r.selected, [...r.selected].sort());
});
test('empty detected with no flags/config falls back to no_config_all_detected with empty selected, warnings, and errors', () => {
const r = resolveReviewerSelection({ detected: [] });
assert.equal(r.source, 'no_config_all_detected');
assert.deepStrictEqual(r.selected, []);
assert.deepStrictEqual(r.warnings, []);
assert.deepStrictEqual(r.errors, []);
});
});
/**
* ADR-2782 D4 (#2794) — absent-safe governs DISCOVERY, never explicit selection.
*
* "Not finding a lane nobody asked for is normal; failing to run a lane somebody
* asked for is an error."
*
* Before this change every explicit miss was an `info`. A TOTAL miss still
* errored, but only as a side effect of the selected set coming out empty — so
* the PARTIAL miss (`--gemini --qwen` on a host without qwen) had no signal at
* all: the review ran with a thinner reviewer set and present_results reported
* success. The workflow's own guidance names why that is wrong — "a cross-AI
* review that silently drops a lane is blind in one eye".
*/
describe('resolveReviewerSelection — explicit flags are an assertion (ADR-2782 D4)', () => {
const errorsMentioning = (r, slug) => r.errors.filter((e) => e.includes(slug));
test('an explicit flag for a detected reviewer selects it with no message', () => {
const r = resolveReviewerSelection({
detected: ['gemini'],
explicitFlags: ['gemini'],
});
assert.deepStrictEqual(r.selected, ['gemini']);
assert.deepStrictEqual(r.errors, []);
assert.deepStrictEqual(r.infos, []);
});
test('a partial explicit miss errors instead of degrading silently', () => {
// THE regression row. Pre-fix this produced errors: [] and an info note,
// and the run proceeded one-eyed.
const r = resolveReviewerSelection({
detected: ['gemini'],
explicitFlags: ['gemini', 'qwen'],
});
assert.deepStrictEqual(r.selected, ['gemini'], 'the detected lane is still selected');
assert.strictEqual(
errorsMentioning(r, 'qwen').length,
1,
`expected exactly one error naming qwen, got: ${JSON.stringify(r.errors)}`,
);
assert.deepStrictEqual(r.infos, [], 'the miss must not be downgraded to an info');
});
test('a sole explicit flag that is undetected errors per-slug and in aggregate', () => {
const r = resolveReviewerSelection({
detected: ['gemini'],
explicitFlags: ['qwen'],
});
assert.deepStrictEqual(r.selected, []);
assert.strictEqual(errorsMentioning(r, 'qwen').length, 1);
// The pre-existing aggregate message is preserved, not replaced — the
// per-slug errors must not suppress it.
assert.ok(
r.errors.some((e) => e.includes('no selected reviewers are available')),
`expected the aggregate error to survive, got: ${JSON.stringify(r.errors)}`,
);
});
test('every missing explicit flag produces its own error, in a stable order', () => {
const forward = resolveReviewerSelection({
detected: ['gemini'],
explicitFlags: ['gemini', 'qwen', 'codex'],
});
const reversed = resolveReviewerSelection({
detected: ['gemini'],
explicitFlags: ['gemini', 'codex', 'qwen'],
});
assert.strictEqual(errorsMentioning(forward, 'qwen').length, 1);
assert.strictEqual(errorsMentioning(forward, 'codex').length, 1);
// Order must not depend on the order flags appeared on the command line.
assert.deepStrictEqual(forward.errors, reversed.errors);
});
test('a duplicate explicit flag produces exactly one error', () => {
const r = resolveReviewerSelection({
detected: ['gemini'],
explicitFlags: ['qwen', 'qwen'],
});
assert.strictEqual(errorsMentioning(r, 'qwen').length, 1);
});
test('explicit flag matching is case-insensitive', () => {
const r = resolveReviewerSelection({
detected: ['gemini'],
explicitFlags: ['QWEN'],
});
assert.strictEqual(errorsMentioning(r, 'qwen').length, 1);
});
test('an explicit flag with nothing detected at all errors', () => {
const r = resolveReviewerSelection({
detected: [],
explicitFlags: ['gemini'],
});
assert.deepStrictEqual(r.selected, []);
assert.strictEqual(errorsMentioning(r, 'gemini').length, 1);
});
test('pre-existing config errors do not suppress per-slug explicit errors', () => {
const r = resolveReviewerSelection({
detected: ['gemini'],
explicitFlags: ['qwen'],
configuredDefaultReviewers: 'not-an-array',
});
assert.strictEqual(errorsMentioning(r, 'qwen').length, 1);
assert.ok(r.errors.some((e) => e.includes('must be a JSON array')));
// Guarded on the PRE-branch error count, so the aggregate fires exactly when
// it did before this change — i.e. not here, because a config error already
// existed.
assert.ok(!r.errors.some((e) => e.includes('no selected reviewers are available')));
});
test('non-string explicit flags are coerced, never thrown on', () => {
const r = resolveReviewerSelection({
detected: ['gemini'],
explicitFlags: [null, 0, { a: 1 }],
});
assert.strictEqual(r.source, 'explicit_flags');
assert.strictEqual(r.errors.length, 3 + 1, 'three unknown flags plus the aggregate');
});
});
/**
* The other half of D4, and the reason the carve-out is scoped to explicit
* flags only: discovery paths stay lenient. `--all` is a quantifier over what
* exists; `review.default_reviewers` is a preference evaluated across many
* hosts. Neither is an assertion about a specific lane, so neither errors.
*/
describe('resolveReviewerSelection — discovery paths stay lenient (ADR-2782 D4)', () => {
test('--all does not error on undetected lanes', () => {
const r = resolveReviewerSelection({
detected: ['gemini'],
allFlag: true,
});
assert.equal(r.source, 'all_flag');
assert.deepStrictEqual(r.selected, ['gemini']);
assert.deepStrictEqual(r.errors, []);
assert.deepStrictEqual(r.infos, []);
});
test('a configured default that is undetected stays an info, not an error', () => {
const r = resolveReviewerSelection({
detected: ['gemini'],
configuredDefaultReviewers: ['gemini', 'codex'],
});
assert.equal(r.source, 'config_default');
assert.deepStrictEqual(r.selected, ['gemini']);
assert.deepStrictEqual(r.errors, [], 'a preference miss must not become an error');
assert.ok(
r.infos.some((i) => i.includes('codex')),
`expected an info naming codex, got: ${JSON.stringify(r.infos)}`,
);
});
test('an unknown configured slug stays a warning', () => {
const r = resolveReviewerSelection({
detected: ['gemini'],
configuredDefaultReviewers: ['gemini', '__nope__'],
});
assert.deepStrictEqual(r.errors, []);
assert.ok(r.warnings.some((w) => w.includes('__nope__')));
});
});