* test(#3301): add failing-first plan coverage manifest tests Failing-first regression tests for the plan-id manifest, the updated Review Instructions, and the mechanical per-reviewer coverage check, ahead of the review.md implementation. RED baseline before the fix lands. * test(#3301): raise allow-test-rule-refs unverified ceiling for new marker Adding tests/review-plan-coverage-manifest.test.cjs's source-text-is-the-product marker grows the unverified-exemption pool by one (282 -> 283), the same documented growth path scripts/lint-allow-test-rule-refs.cjs's own failure output names. Confirmed clean via 'npm run lint:allow-test-rule-refs' locally. * feat(#3301): tell reviewers the plan ids and total count, grade coverage build_prompt now derives a plan-id manifest from each *-PLAN.md filename (stripping the -PLAN.md suffix) and appends it, with the total plan count, to both gsd-review-instructions.md and gsd-review-prompt.md. The Review Instructions prose requires one heading-verbatim section per id before any cross-plan or overall-risk content. write_reviews grades each dispatched lane's real (non-stub, non-empty) review against that same manifest and records an optional plan_coverage: frontmatter block, present only when a lane is incomplete. The match escapes regex metacharacters in the id and excludes a preceding/trailing hyphen or word character as a boundary, closing the two traps named in the issue (a decimal phase like 12.6 satisfied by 12X6-01; a threat id like T-04-07 registering as coverage of plan 04-07). CodeRabbit is exempt, since it never receives the source-grounding prompt carrying the manifest. This closes the gap where a review that silently covers only some plans in a multi-plan phase is indistinguishable from one that covers all of them. * docs(#3301): add changeset fragment * test(#3301): use t.after() instead of try/finally for cleanup CONTRIBUTING.md bans try/finally inside test bodies. Code review caught this in the new coverage-manifest test file; switch every fixture-cleanup site to the approved t.after() pattern. * test: use t.after() instead of try/finally in #3300's build_prompt tests Pre-existing try/finally-for-cleanup pattern in this file (landed for #3300) violates CONTRIBUTING.md's explicit ban on try/finally inside test bodies. Surfaced incidentally while reviewing #3301's diff, which cites this file as its extraction-pattern precedent; fixed inline per the no-defer rule rather than deferred to a separate PR. * test(#3301): anchor coverage-check extraction on the fence line, not prose `.plans-manifest.md` also appears in write_reviews' own prose ahead of the ```bash fence, so indexOf found that occurrence first and the backward-walk-to-fence-open landed on the earlier, unrelated gate-check block instead. gsd-test caught this: coverage-check tests expecting a real verdict got null, because the wrong block ran and never writes .plan-coverage-<slug>.json. Anchor on the fence-only bash assignment line instead. Emitted-Drift-Ack-Growth: review.md — #3301 adds the plan-coverage manifest and mechanical coverage check to build_prompt/write_reviews. * fix(#3301): route id escaping through the canonical pattern seam ADR-3212 (epic #3212) consolidated ~44 hand-rolled regex-escape copies into one owner, src/pattern.cts's escapeRegex, specifically to stop this exact class of duplication. My coverage-check node -e script hand-rolled the identical metachar-escape regex — invisible to eslint-rules/no-adhoc-regex-escape.cjs only because it lives inside a workflow markdown file, not a .cts/.cjs source file the shape-matching guard scans. Require the compiled seam (gsd-core/bin/lib/pattern.cjs) instead, matching the established node -e-requires-a-compiled-lib idiom already used elsewhere in this workflow (code-review.md's code-review-flags.cjs/code-review-depth.cjs calls). Verified both named traps from the issue still resolve correctly under escapeRegex's RegExp.escape-backed implementation, which differs in escaped-text shape (hex-escapes hyphens/leading chars) but not match result. * test(#3301): run coverage-check block with cwd at the repo root The block's node -e now requires ./gsd-core/bin/lib/pattern.cjs, a path relative to the repo root (correct for production, which always runs from there). The test harness ran it with cwd at the fixture's own temp dir instead, so the require failed. Add an optional cwd param to runScript (default: root, unchanged for the plan-copy-block tests) and pass the real repo root for every coverage-check call site. Manually verified end-to-end before spending another remote run: the extracted block now produces the expected {complete:true} verdict. * docs(#3301): backfill changeset pr number (pr:0 -> pr:4084) --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
372
tests/review-plan-coverage-manifest.test.cjs
Normal file
372
tests/review-plan-coverage-manifest.test.cjs
Normal file
@@ -0,0 +1,372 @@
|
||||
// allow-test-rule: source-text-is-the-product (#3301)
|
||||
// The workflow markdown IS the installed orchestration contract; these rows
|
||||
// extract the real shipped bash and execute it, never a hand-copied duplicate.
|
||||
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* #3301 — reviewers in the cross-AI plan-review workflow are never told the
|
||||
* plan ids or the total plan count, so a review that silently covers 6 of 7
|
||||
* plans is indistinguishable from one that covers all 7.
|
||||
*
|
||||
* Two behavioral seams, both extracted from the REAL shipped bash in
|
||||
* gsd-core/workflows/review.md (the same pattern as
|
||||
* tests/review-build-prompt-optional-sections.test.cjs):
|
||||
*
|
||||
* 1. build_prompt's plan-copy block — must derive a `.plans-manifest.md`
|
||||
* (plan ids + total count) from the `*-PLAN.md` filenames and append it
|
||||
* to both gsd-review-instructions.md and gsd-review-prompt.md.
|
||||
* 2. write_reviews' plan-coverage-check block — must grade each dispatched
|
||||
* lane's review file against that manifest, honoring two named regex
|
||||
* traps (escaped ids, hyphen-boundary exclusion) and skipping stub/empty/
|
||||
* CodeRabbit lanes.
|
||||
*
|
||||
* Script transport is temp-FILE based, never `bash -c <script>` (#2650 argv
|
||||
* mangling on Windows), matching the established convention.
|
||||
*/
|
||||
|
||||
const { describe, test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { spawnSync } = require('node:child_process');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const {
|
||||
cleanup,
|
||||
createTempDir,
|
||||
readWorkflowCombined,
|
||||
} = require('./helpers.cjs');
|
||||
const { PROBE_TIMEOUT_MS } = require('./helpers/timeouts.cjs');
|
||||
const { scanFencedBlocks } = require('../gsd-core/bin/lib/markdown-sectionizer.cjs');
|
||||
|
||||
const REVIEW_WORKFLOW = path.join(__dirname, '..', 'gsd-core', 'workflows', 'review.md');
|
||||
const REPO_ROOT = path.join(__dirname, '..');
|
||||
|
||||
function detectShells() {
|
||||
const shells = [{ name: 'bash', cmd: 'bash' }];
|
||||
const probe = spawnSync('zsh', ['-c', 'exit 0'], { timeout: PROBE_TIMEOUT_MS, windowsHide: true });
|
||||
if (!probe.error && probe.status === 0) shells.push({ name: 'zsh', cmd: 'zsh' });
|
||||
return shells;
|
||||
}
|
||||
const SHELLS = detectShells();
|
||||
|
||||
/**
|
||||
* Extract the fenced ```bash block of build_prompt that copies *-PLAN.md
|
||||
* files — the one that also writes gsd-review-context.md and
|
||||
* gsd-review-instructions.md. Same anchor/walk-backward pattern as
|
||||
* tests/review-build-prompt-optional-sections.test.cjs.
|
||||
*/
|
||||
function extractPlanCopyBlock() {
|
||||
const content = readWorkflowCombined(REVIEW_WORKFLOW);
|
||||
const anchorIdx = content.indexOf('gsd-review-context.md');
|
||||
assert.notEqual(anchorIdx, -1, 'gsd-review-context.md no longer appears in review.md (+steps)');
|
||||
const before = content.slice(0, anchorIdx);
|
||||
const fenceOpenRe = /```bash\r?\n/g;
|
||||
let lastOpen = -1;
|
||||
let m;
|
||||
while ((m = fenceOpenRe.exec(before)) !== null) lastOpen = m.index + m[0].length;
|
||||
assert.notEqual(lastOpen, -1, 'gsd-review-context.md is not inside a ```bash fence of review.md (+steps)');
|
||||
const after = content.slice(lastOpen);
|
||||
const closeIdx = after.indexOf('\n```');
|
||||
assert.notEqual(closeIdx, -1, 'unterminated ```bash fence around gsd-review-context.md');
|
||||
const body = after.slice(0, closeIdx);
|
||||
assert.ok(
|
||||
body.includes('gsd-review-instructions.md'),
|
||||
'extracted block writes gsd-review-context.md but not gsd-review-instructions.md — wrong block',
|
||||
);
|
||||
return body;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the fenced ```bash block of write_reviews that grades plan
|
||||
* coverage — anchored on the manifest variable this PR introduces. Walks
|
||||
* forward from the write_reviews step marker so it never grabs the earlier
|
||||
* (unrelated) gate-check bash block in the same step.
|
||||
*/
|
||||
function extractCoverageCheckBlock() {
|
||||
const content = readWorkflowCombined(REVIEW_WORKFLOW);
|
||||
const stepIdx = content.indexOf('<step name="write_reviews">');
|
||||
assert.notEqual(stepIdx, -1, 'write_reviews step must exist');
|
||||
const fromStep = content.slice(stepIdx);
|
||||
const anchorIdx = fromStep.indexOf('MANIFEST="$RUN_DIR/.plans-manifest.md"');
|
||||
assert.notEqual(
|
||||
anchorIdx,
|
||||
-1,
|
||||
'write_reviews must reference .plans-manifest.md — the plan-coverage-check block is missing',
|
||||
);
|
||||
const before = fromStep.slice(0, anchorIdx);
|
||||
const fenceOpenRe = /```bash\r?\n/g;
|
||||
let lastOpen = -1;
|
||||
let m;
|
||||
while ((m = fenceOpenRe.exec(before)) !== null) lastOpen = m.index + m[0].length;
|
||||
assert.notEqual(lastOpen, -1, '.plans-manifest.md reference is not inside a ```bash fence of write_reviews');
|
||||
const after = fromStep.slice(lastOpen);
|
||||
const closeIdx = after.indexOf('\n```');
|
||||
assert.notEqual(closeIdx, -1, 'unterminated ```bash fence around the plan-coverage-check block');
|
||||
return after.slice(0, closeIdx);
|
||||
}
|
||||
|
||||
/** Every fenced ```bash block of review.md (+steps) — for the CodeRabbit-slug structural row. */
|
||||
function extractAllBashBlocks() {
|
||||
const content = readWorkflowCombined(REVIEW_WORKFLOW);
|
||||
const lines = content.split(/\r?\n/);
|
||||
return scanFencedBlocks(lines)
|
||||
.filter((b) => b.closeLineIdx !== -1 && (b.infoString || '').trim() === 'bash')
|
||||
.map((b) => lines.slice(b.openLineIdx + 1, b.closeLineIdx).join('\n'));
|
||||
}
|
||||
|
||||
/** Fill the workflow's `{run_dir}` placeholder and stage the script in a file. */
|
||||
function stageScript(shell, body, root, runDir) {
|
||||
const scriptPath = path.join(root, `block-${shell.name}-${Math.random().toString(36).slice(2)}.sh`);
|
||||
fs.writeFileSync(scriptPath, body.split('{run_dir}').join(runDir));
|
||||
return scriptPath;
|
||||
}
|
||||
|
||||
function runScript(shell, body, root, runDir, env, cwd = root) {
|
||||
const scriptPath = stageScript(shell, body, root, runDir);
|
||||
return spawnSync(shell.cmd, [scriptPath], {
|
||||
cwd,
|
||||
env: { ...process.env, ...env },
|
||||
encoding: 'utf8',
|
||||
stdin: 'ignore',
|
||||
timeout: PROBE_TIMEOUT_MS,
|
||||
windowsHide: true,
|
||||
});
|
||||
}
|
||||
|
||||
const readIfPresent = (file) => (fs.existsSync(file) ? fs.readFileSync(file, 'utf8') : null);
|
||||
|
||||
/**
|
||||
* Fixture for the plan-copy block: a PHASE_DIR with the given *-PLAN.md
|
||||
* filenames, a fresh RUN_DIR, and the two always-copied section sources the
|
||||
* block expects from prompt assembly.
|
||||
*/
|
||||
function buildPlanCopyFixture(planFileNames) {
|
||||
const root = createTempDir('gsd-3301-copy-');
|
||||
const phaseDir = path.join(root, 'phase');
|
||||
const runDir = path.join(root, 'run');
|
||||
fs.mkdirSync(phaseDir);
|
||||
fs.mkdirSync(runDir);
|
||||
for (const name of planFileNames) {
|
||||
fs.writeFileSync(path.join(phaseDir, name), 'plan body\n');
|
||||
}
|
||||
fs.writeFileSync(path.join(root, 'instr.md'), 'instructions\n');
|
||||
fs.writeFileSync(path.join(root, 'roadmap.md'), 'roadmap\n');
|
||||
return {
|
||||
root,
|
||||
phaseDir,
|
||||
runDir,
|
||||
env: {
|
||||
PHASE_DIR: phaseDir,
|
||||
INSTRUCTIONS_BLOCK_FILE: path.join(root, 'instr.md'),
|
||||
ROADMAP_SECTION_FILE: path.join(root, 'roadmap.md'),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Fixture for the coverage-check block: a RUN_DIR pre-populated with a manifest and lane files. */
|
||||
function buildCoverageFixture(planIds, lanes) {
|
||||
const root = createTempDir('gsd-3301-coverage-');
|
||||
const runDir = path.join(root, 'run');
|
||||
fs.mkdirSync(runDir);
|
||||
const manifestLines = ['', '## Plan Coverage Manifest', '', `Total plans in this review: ${planIds.length}`, ''];
|
||||
for (const id of planIds) manifestLines.push(`- ${id}`);
|
||||
fs.writeFileSync(path.join(runDir, '.plans-manifest.md'), manifestLines.join('\n') + '\n');
|
||||
for (const [slug, content] of Object.entries(lanes)) {
|
||||
fs.writeFileSync(path.join(runDir, `gsd-review-${slug}.md`), content);
|
||||
}
|
||||
return {
|
||||
root,
|
||||
runDir,
|
||||
env: { SELECTED_REVIEWERS: Object.keys(lanes).join(',') },
|
||||
};
|
||||
}
|
||||
|
||||
function readCoverageJson(runDir, slug) {
|
||||
const p = path.join(runDir, `.plan-coverage-${slug}.json`);
|
||||
return fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, 'utf8')) : null;
|
||||
}
|
||||
|
||||
describe('#3301 build_prompt derives and appends a plan coverage manifest', () => {
|
||||
for (const shell of SHELLS) {
|
||||
test(`[${shell.name}] zero plans: manifest reports Total plans: 0 and no ids`, (t) => {
|
||||
const fx = buildPlanCopyFixture([]);
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractPlanCopyBlock(), fx.root, fx.runDir, fx.env);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const manifest = readIfPresent(path.join(fx.runDir, '.plans-manifest.md'));
|
||||
assert.notEqual(manifest, null, '.plans-manifest.md must be written even with zero plans');
|
||||
assert.match(manifest, /Total plans in this review: 0/);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] one plan: manifest lists the single id and count 1`, (t) => {
|
||||
const fx = buildPlanCopyFixture(['01-PLAN.md']);
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractPlanCopyBlock(), fx.root, fx.runDir, fx.env);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const manifest = readIfPresent(path.join(fx.runDir, '.plans-manifest.md'));
|
||||
assert.match(manifest, /Total plans in this review: 1/);
|
||||
assert.match(manifest, /^- 01$/m);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] decimal-phase ids are preserved verbatim in the manifest`, (t) => {
|
||||
const fx = buildPlanCopyFixture(['12.6-01-PLAN.md', '12.6-02-PLAN.md']);
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractPlanCopyBlock(), fx.root, fx.runDir, fx.env);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const manifest = readIfPresent(path.join(fx.runDir, '.plans-manifest.md'));
|
||||
assert.match(manifest, /Total plans in this review: 2/);
|
||||
assert.match(manifest, /^- 12\.6-01$/m);
|
||||
assert.match(manifest, /^- 12\.6-02$/m);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] manifest is appended to both gsd-review-instructions.md and gsd-review-prompt.md`, (t) => {
|
||||
const fx = buildPlanCopyFixture(['01-PLAN.md']);
|
||||
t.after(() => cleanup(fx.root));
|
||||
// gsd-review-prompt.md is written earlier in build_prompt (the fenced
|
||||
// markdown template); simulate that so the append target pre-exists.
|
||||
fs.writeFileSync(path.join(fx.runDir, 'gsd-review-prompt.md'), '# prompt\n');
|
||||
const res = runScript(shell, extractPlanCopyBlock(), fx.root, fx.runDir, fx.env);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const instructions = readIfPresent(path.join(fx.runDir, 'gsd-review-instructions.md'));
|
||||
const prompt = readIfPresent(path.join(fx.runDir, 'gsd-review-prompt.md'));
|
||||
assert.match(instructions, /Plan Coverage Manifest/);
|
||||
assert.match(prompt, /Plan Coverage Manifest/);
|
||||
});
|
||||
}
|
||||
|
||||
test('manifest filename does not collide with either existing RUN_DIR glob', () => {
|
||||
const name = '.plans-manifest.md';
|
||||
assert.ok(!/^gsd-review-.*\.md$/.test(name), 'must not match the gsd-review-*.md reviewer-report glob');
|
||||
assert.ok(!/^gsd-review-plan-.*\.md$/.test(name), 'must not match the gsd-review-plan-*.md plan-copy glob');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#3301 Review Instructions require one section per manifest id', () => {
|
||||
const workflow = readWorkflowCombined(REVIEW_WORKFLOW);
|
||||
|
||||
test('documents mandatory per-id section requirement before cross-plan content', () => {
|
||||
assert.ok(
|
||||
workflow.includes('Plan Coverage Manifest'),
|
||||
'build_prompt prompt template must reference the Plan Coverage Manifest section',
|
||||
);
|
||||
assert.ok(
|
||||
/plan coverage is mandatory/i.test(workflow),
|
||||
'Review Instructions must state that plan coverage is mandatory',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#3301 write_reviews grades each lane against the plan coverage manifest', () => {
|
||||
for (const shell of SHELLS) {
|
||||
test(`[${shell.name}] coverage check: complete when every id appears`, (t) => {
|
||||
const fx = buildCoverageFixture(['01', '02'], {
|
||||
gemini: '## 01\n\nlooks good\n\n## 02\n\nlooks good too\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const cov = readCoverageJson(fx.runDir, 'gemini');
|
||||
assert.deepEqual(cov, { complete: true, missing_ids: [], total: 2 });
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: reports the specific missing id (the field-observed "6 of 7" case)`, (t) => {
|
||||
const ids = ['01', '02', '03', '04', '05', '06', '07'];
|
||||
const body = ids
|
||||
.filter((id) => id !== '07')
|
||||
.map((id) => `## ${id}\n\ncovered\n`)
|
||||
.join('\n');
|
||||
const fx = buildCoverageFixture(ids, { qwen: body });
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const cov = readCoverageJson(fx.runDir, 'qwen');
|
||||
assert.strictEqual(cov.complete, false);
|
||||
assert.deepEqual(cov.missing_ids, ['07']);
|
||||
assert.strictEqual(cov.total, 7);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: unescaped dot cannot be satisfied by 12X6-01 (issue trap 1)`, (t) => {
|
||||
const fx = buildCoverageFixture(['12.6-01'], {
|
||||
codex: 'discussion of 12X6-01 but never the real id\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const cov = readCoverageJson(fx.runDir, 'codex');
|
||||
assert.strictEqual(cov.complete, false, 'unescaped "." would let 12X6-01 wrongly satisfy 12.6-01');
|
||||
assert.deepEqual(cov.missing_ids, ['12.6-01']);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: a hyphen-prefixed token (T-04-07) does not satisfy id 04-07 (issue trap 2)`, (t) => {
|
||||
const fx = buildCoverageFixture(['04-07'], {
|
||||
claude: 'six sections away, threat id T-04-07 is discussed at length\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const cov = readCoverageJson(fx.runDir, 'claude');
|
||||
assert.strictEqual(cov.complete, false, 'a preceding hyphen must not count as a boundary for id 04-07');
|
||||
assert.deepEqual(cov.missing_ids, ['04-07']);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: a plain-prose mention counts as covered, no heading required`, (t) => {
|
||||
const fx = buildCoverageFixture(['12.6-01'], {
|
||||
gemini: 'This was already covered by plan 12.6-01 above, no separate section needed.\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const cov = readCoverageJson(fx.runDir, 'gemini');
|
||||
assert.strictEqual(cov.complete, true);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: a budget-skipped stub is not graded`, (t) => {
|
||||
const fx = buildCoverageFixture(['01'], {
|
||||
ollama: 'ollama review skipped: prompt budget (500 tokens) too small for the minimum review set.\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(readCoverageJson(fx.runDir, 'ollama'), null, 'a budget-skip stub must not be graded');
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: an empty review file is not graded`, (t) => {
|
||||
const fx = buildCoverageFixture(['01'], { codex: '' });
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(readCoverageJson(fx.runDir, 'codex'), null, 'an empty review file must not be graded');
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: coderabbit lane is exempt from plan-coverage grading`, (t) => {
|
||||
const fx = buildCoverageFixture(['01', '02'], {
|
||||
coderabbit: 'diff-only review, mentions nothing about plans\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(
|
||||
readCoverageJson(fx.runDir, 'coderabbit'),
|
||||
null,
|
||||
'coderabbit never receives the source-grounding prompt and must not be graded',
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
test('structural: no lane slug other than coderabbit is hardcoded-excluded (the exemption is named, not general)', () => {
|
||||
const offenders = extractAllBashBlocks().filter((b) => /\[\s*"\$SLUG"\s*=\s*"(?!coderabbit)/.test(b));
|
||||
assert.deepEqual(offenders, [], 'only the coderabbit exemption may hardcode a slug comparison');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#3301 REVIEWS.md documents the plan_coverage frontmatter key', () => {
|
||||
const workflow = readWorkflowCombined(REVIEW_WORKFLOW);
|
||||
|
||||
test('documents plan_coverage: frontmatter is present only when a lane is incomplete', () => {
|
||||
assert.ok(workflow.includes('plan_coverage'), 'write_reviews must document a plan_coverage frontmatter key');
|
||||
assert.ok(
|
||||
/plan_coverage.*only present if at least one/is.test(workflow),
|
||||
'plan_coverage must be documented as present only when at least one graded lane is incomplete',
|
||||
);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user