* test(#3301): add failing-first plan coverage manifest tests Failing-first regression tests for the plan-id manifest, the updated Review Instructions, and the mechanical per-reviewer coverage check, ahead of the review.md implementation. RED baseline before the fix lands. * test(#3301): raise allow-test-rule-refs unverified ceiling for new marker Adding tests/review-plan-coverage-manifest.test.cjs's source-text-is-the-product marker grows the unverified-exemption pool by one (282 -> 283), the same documented growth path scripts/lint-allow-test-rule-refs.cjs's own failure output names. Confirmed clean via 'npm run lint:allow-test-rule-refs' locally. * feat(#3301): tell reviewers the plan ids and total count, grade coverage build_prompt now derives a plan-id manifest from each *-PLAN.md filename (stripping the -PLAN.md suffix) and appends it, with the total plan count, to both gsd-review-instructions.md and gsd-review-prompt.md. The Review Instructions prose requires one heading-verbatim section per id before any cross-plan or overall-risk content. write_reviews grades each dispatched lane's real (non-stub, non-empty) review against that same manifest and records an optional plan_coverage: frontmatter block, present only when a lane is incomplete. The match escapes regex metacharacters in the id and excludes a preceding/trailing hyphen or word character as a boundary, closing the two traps named in the issue (a decimal phase like 12.6 satisfied by 12X6-01; a threat id like T-04-07 registering as coverage of plan 04-07). CodeRabbit is exempt, since it never receives the source-grounding prompt carrying the manifest. This closes the gap where a review that silently covers only some plans in a multi-plan phase is indistinguishable from one that covers all of them. * docs(#3301): add changeset fragment * test(#3301): use t.after() instead of try/finally for cleanup CONTRIBUTING.md bans try/finally inside test bodies. Code review caught this in the new coverage-manifest test file; switch every fixture-cleanup site to the approved t.after() pattern. * test: use t.after() instead of try/finally in #3300's build_prompt tests Pre-existing try/finally-for-cleanup pattern in this file (landed for #3300) violates CONTRIBUTING.md's explicit ban on try/finally inside test bodies. Surfaced incidentally while reviewing #3301's diff, which cites this file as its extraction-pattern precedent; fixed inline per the no-defer rule rather than deferred to a separate PR. * test(#3301): anchor coverage-check extraction on the fence line, not prose `.plans-manifest.md` also appears in write_reviews' own prose ahead of the ```bash fence, so indexOf found that occurrence first and the backward-walk-to-fence-open landed on the earlier, unrelated gate-check block instead. gsd-test caught this: coverage-check tests expecting a real verdict got null, because the wrong block ran and never writes .plan-coverage-<slug>.json. Anchor on the fence-only bash assignment line instead. Emitted-Drift-Ack-Growth: review.md — #3301 adds the plan-coverage manifest and mechanical coverage check to build_prompt/write_reviews. * fix(#3301): route id escaping through the canonical pattern seam ADR-3212 (epic #3212) consolidated ~44 hand-rolled regex-escape copies into one owner, src/pattern.cts's escapeRegex, specifically to stop this exact class of duplication. My coverage-check node -e script hand-rolled the identical metachar-escape regex — invisible to eslint-rules/no-adhoc-regex-escape.cjs only because it lives inside a workflow markdown file, not a .cts/.cjs source file the shape-matching guard scans. Require the compiled seam (gsd-core/bin/lib/pattern.cjs) instead, matching the established node -e-requires-a-compiled-lib idiom already used elsewhere in this workflow (code-review.md's code-review-flags.cjs/code-review-depth.cjs calls). Verified both named traps from the issue still resolve correctly under escapeRegex's RegExp.escape-backed implementation, which differs in escaped-text shape (hex-escapes hyphens/leading chars) but not match result. * test(#3301): run coverage-check block with cwd at the repo root The block's node -e now requires ./gsd-core/bin/lib/pattern.cjs, a path relative to the repo root (correct for production, which always runs from there). The test harness ran it with cwd at the fixture's own temp dir instead, so the require failed. Add an optional cwd param to runScript (default: root, unchanged for the plan-copy-block tests) and pass the real repo root for every coverage-check call site. Manually verified end-to-end before spending another remote run: the extracted block now produces the expected {complete:true} verdict. * docs(#3301): backfill changeset pr number (pr:0 -> pr:4084) --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
5
.changeset/bold-sloths-dance.md
Normal file
5
.changeset/bold-sloths-dance.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
type: Changed
|
||||
pr: 4084
|
||||
---
|
||||
**`/gsd-review` now tells every reviewer the exact plan ids and total count, and grades coverage against them** — a review that silently covers only some of a multi-plan phase is no longer indistinguishable from one that covered every plan. (#3301)
|
||||
@@ -1811,6 +1811,8 @@ Reviewers reached through `--all` or `review.default_reviewers` behave different
|
||||
|
||||
Its frontmatter records the model each reviewer resolved to, as `models:` (the model id, or `unknown`, with a `(reasoning=<level>)` suffix when GSD applied a reasoning effort to that lane) and `model_sources:` (how each value was determined — `pinned`, `served`, `requested`, `banner`, `transcript`, or `unknown`). See [Resolved model recording](CONFIGURATION.md#resolved-model-recording-2295).
|
||||
|
||||
**Plan coverage manifest (#3301):** the assembled prompt tells every prompt-fed reviewer exactly which plan ids exist in this review and the total count, and asks for one heading-verbatim section per id before any cross-plan or overall-risk content — so a review that silently stops partway through a multi-plan phase is no longer indistinguishable from one that covered every plan. A mechanical check grades each reviewer's output against that same id list and records an optional `plan_coverage:` frontmatter block, present only when a reviewer's output does not mention every id (diagnostic only — it never blocks the run). CodeRabbit is not graded — it is a diff-only reviewer that never receives the prompt carrying the manifest.
|
||||
|
||||
```bash
|
||||
# set project default reviewers for no-flag /gsd-review runs
|
||||
gsd config-set review.default_reviewers '["gemini","codex"]'
|
||||
|
||||
@@ -220,6 +220,13 @@ Provide structured feedback on plan quality, completeness, and risks.
|
||||
|
||||
Findings citing `file:line` evidence are weighted far more heavily than impressionistic ones; a review that only restates the plan's own claims has low value.
|
||||
|
||||
**Plan coverage is mandatory (#3301).** The exact list of plan ids and the total plan count for
|
||||
this review are given in the "## Plan Coverage Manifest" section below. Give **every** listed id
|
||||
its own `##`-level section headed with that id **verbatim** (e.g. `## 12.6-01`) before writing any
|
||||
cross-plan comparison, an overall risk assessment, or a consensus-style summary. A review that
|
||||
stops before every id has its own section is an incomplete review, not a summary — if you must
|
||||
stop early, say so explicitly and name which ids you did not reach.
|
||||
|
||||
Analyze each plan and provide:
|
||||
|
||||
1. **Summary** — One-paragraph assessment
|
||||
@@ -262,6 +269,42 @@ for PLAN_FILE in "${PHASE_DIR}"/*-PLAN.md; do
|
||||
PLAN_INDEX=$((PLAN_INDEX + 1))
|
||||
done
|
||||
|
||||
# #3301: plan coverage manifest — tell reviewers exactly which plan ids exist and
|
||||
# how many there are, so a review that silently covers 6 of 7 plans is no longer
|
||||
# indistinguishable from one that covers all 7. The id is the plan file's own
|
||||
# basename with the `-PLAN.md` suffix stripped (e.g. `12.6-01-PLAN.md` ->
|
||||
# `12.6-01`) — NOT the `plan:` frontmatter key, which holds only the bare
|
||||
# in-phase sequence number ("01") and can never reconstruct the phase-qualified
|
||||
# id reviewers need to cite. The filename is guaranteed present for every copied
|
||||
# plan, so no plan is ever dropped from the manifest for lacking a key.
|
||||
# A plain string accumulator, not an array: zsh and bash disagree on array
|
||||
# indexing and this block runs under both (see the nullglob/NULL_GLOB pairing
|
||||
# above).
|
||||
PLAN_IDS=""
|
||||
for PLAN_FILE in "${PHASE_DIR}"/*-PLAN.md; do
|
||||
PLAN_BASENAME=$(basename "$PLAN_FILE")
|
||||
PLAN_IDS="$PLAN_IDS ${PLAN_BASENAME%-PLAN.md}"
|
||||
done
|
||||
PLAN_IDS="${PLAN_IDS# }"
|
||||
PLAN_COUNT=0
|
||||
for _ in $PLAN_IDS; do PLAN_COUNT=$((PLAN_COUNT + 1)); done
|
||||
|
||||
# Named to avoid BOTH existing RUN_DIR globs: `gsd-review-*.md` (reviewer
|
||||
# reports, invoke_reviewers) and `gsd-review-plan-*.md` (the plan copies just
|
||||
# above) — a manifest matching either would be picked up as a report or as a
|
||||
# plan to review.
|
||||
{
|
||||
echo ""
|
||||
echo "## Plan Coverage Manifest"
|
||||
echo ""
|
||||
echo "Total plans in this review: ${PLAN_COUNT}"
|
||||
echo ""
|
||||
echo "Plan ids (give each one its own \`##\`-level section, headed verbatim):"
|
||||
for PLAN_ID in $PLAN_IDS; do
|
||||
echo "- ${PLAN_ID}"
|
||||
done
|
||||
} > "${RUN_DIR}/.plans-manifest.md"
|
||||
|
||||
# Optional section files (only if content was included in the combined prompt)
|
||||
if [ -f ".planning/PROJECT.md" ]; then
|
||||
cp .planning/PROJECT.md "${RUN_DIR}/gsd-review-project.md"
|
||||
@@ -277,6 +320,15 @@ fi
|
||||
if [ -f ".planning/REQUIREMENTS.md" ]; then
|
||||
cp .planning/REQUIREMENTS.md "${RUN_DIR}/gsd-review-requirements.md"
|
||||
fi
|
||||
|
||||
# #3301: append the manifest to BOTH files reviewers actually read — the
|
||||
# per-lane budget-trimmed instructions file (descriptor lanes get
|
||||
# `--instructions-file`) and the full combined prompt (combined-prompt lanes
|
||||
# read the whole file). The `instructions` fragment is in prompt-budget's
|
||||
# `minimumFor` floor set and is never trimmed, so this survives per-lane
|
||||
# budget trimming intact.
|
||||
cat "${RUN_DIR}/.plans-manifest.md" >> "${RUN_DIR}/gsd-review-instructions.md"
|
||||
cat "${RUN_DIR}/.plans-manifest.md" >> "${RUN_DIR}/gsd-review-prompt.md"
|
||||
```
|
||||
|
||||
Note: `INSTRUCTIONS_BLOCK_FILE`, `ROADMAP_SECTION_FILE`, and `PHASE_DIR` come from prompt assembly; `RUN_DIR` is the run-scoped dir from `gather_context` (#2358) re-assigned from `{run_dir}` above. Copy the temp files written during prompt assembly to these section paths (or write each section here if the prompt was built inline).
|
||||
@@ -536,6 +588,79 @@ fi
|
||||
their `.err`/stub files preserved under `.review-diagnostics/` by `present_results`) and stop.
|
||||
- **Otherwise** (at least one lane produced a result — R1, unchanged): proceed exactly as below.
|
||||
|
||||
**#3301: plan coverage check.** For each dispatched lane that produced a *real* review (not a
|
||||
stub, not budget-skipped, not empty), check whether its output mentions every plan id from
|
||||
`.plans-manifest.md` — the same manifest `build_prompt` gave the reviewer, so the expected-id list
|
||||
here can never diverge from what the reviewer was actually told. This is diagnostic only: it never
|
||||
blocks the workflow, never fails a lane, and never changes the `TOTAL_LANE_FAILURE`/
|
||||
`ALL_LANES_SKIPPED` gate above.
|
||||
|
||||
CodeRabbit is excluded — it is a diff-only lane that never receives the source-grounding prompt
|
||||
(and therefore never receives the manifest or the per-id section instruction either), the same fact
|
||||
that already excludes it from grounded-review weighting in the Consensus Summary below.
|
||||
|
||||
The match is intentionally lenient about *where* an id appears (a `##`-headed section is asked for,
|
||||
but plain prose mentioning the id still counts as coverage — grading only the letter of the
|
||||
formatting instruction would produce false INCOMPLETE verdicts against a reviewer that cited real
|
||||
evidence correctly). It is strict about *what* counts as a match: the id is regex-escaped (a
|
||||
decimal phase like `12.6` must not let `12X6-01` satisfy `12.6-01` through an unescaped `.`), and a
|
||||
`-`/word character immediately before or after the candidate match does not count as a boundary (so
|
||||
a threat id like `T-04-07` elsewhere in the review must not register as covering plan `04-07`).
|
||||
|
||||
```bash
|
||||
RUN_DIR="{run_dir}"
|
||||
MANIFEST="$RUN_DIR/.plans-manifest.md"
|
||||
|
||||
# Recompute — a shell variable does not survive across separate fenced blocks
|
||||
# (each is its own process), so DISPATCH_SLUGS from the gate-check block above
|
||||
# cannot be assumed to still be set here. Same recomputation as that block and
|
||||
# as invoke_reviewers.
|
||||
DISPATCH_SLUGS=""
|
||||
for SLUG in $(echo "$SELECTED_REVIEWERS" | tr ',' ' '); do
|
||||
case " $DISPATCH_SLUGS " in
|
||||
*" $SLUG "*) continue ;;
|
||||
esac
|
||||
DISPATCH_SLUGS="$DISPATCH_SLUGS $SLUG"
|
||||
done
|
||||
|
||||
for SLUG in $DISPATCH_SLUGS; do
|
||||
[ "$SLUG" = "coderabbit" ] && continue
|
||||
REVIEW_FILE="$RUN_DIR/gsd-review-$SLUG.md"
|
||||
[ -f "$REVIEW_FILE" ] || continue
|
||||
[ -s "$REVIEW_FILE" ] || continue
|
||||
grep -q "review skipped: prompt budget" "$REVIEW_FILE" 2>/dev/null && continue
|
||||
grep -q "failed or returned empty output" "$REVIEW_FILE" 2>/dev/null && continue
|
||||
|
||||
node -e '
|
||||
const fs = require("fs");
|
||||
const { escapeRegex } = require("./gsd-core/bin/lib/pattern.cjs");
|
||||
const manifest = fs.readFileSync(process.argv[1], "utf8");
|
||||
const review = fs.readFileSync(process.argv[2], "utf8");
|
||||
const ids = manifest.split("\n")
|
||||
.filter((l) => l.startsWith("- "))
|
||||
.map((l) => l.slice(2).trim())
|
||||
.filter(Boolean);
|
||||
const missing = ids.filter((id) => {
|
||||
const re = new RegExp("(?<![\\w-])" + escapeRegex(id) + "(?![\\w-])");
|
||||
return !re.test(review);
|
||||
});
|
||||
process.stdout.write(JSON.stringify({ complete: missing.length === 0, missing_ids: missing, total: ids.length }));
|
||||
' "$MANIFEST" "$REVIEW_FILE" > "$RUN_DIR/.plan-coverage-$SLUG.json"
|
||||
done
|
||||
```
|
||||
|
||||
Each `${RUN_DIR}/.plan-coverage-<slug>.json` carries `{complete, missing_ids, total}` for one
|
||||
graded lane. Collect these into a `plan_coverage` frontmatter block — **only** when at least one
|
||||
graded lane has `complete: false` (mirrors the existing `trimmed_reviewers` precedent: present
|
||||
only when there is something to report):
|
||||
|
||||
```yaml
|
||||
plan_coverage: # only present if at least one graded lane is incomplete
|
||||
<slug>:
|
||||
total: 7
|
||||
missing: ["12.6-07"]
|
||||
```
|
||||
|
||||
Combine all review responses into `{phase_dir}/{padded_phase}-REVIEWS.md`:
|
||||
|
||||
After all reviewers complete, collect trim metadata files written during the run. For each reviewer that was trimmed (i.e. a `.metadata.json` file exists and `hardFailed` or `omitted` is non-empty, or `projectMdShrunk` is true, or `planTruncationPct > 0`), include a `trimmed_reviewers` block in the frontmatter. Omit the key entirely if no reviewer was trimmed.
|
||||
@@ -580,6 +705,10 @@ trimmed_reviewers: # only present if at least one reviewer was trimmed
|
||||
plan_truncation_pct: 22
|
||||
hard_failed: false
|
||||
note_injected: true
|
||||
plan_coverage: # only present if at least one graded lane is incomplete (#3301)
|
||||
ollama:
|
||||
total: 7
|
||||
missing: ["12.6-07"]
|
||||
---
|
||||
|
||||
# Cross-AI Plan Review — Phase {N}
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
{
|
||||
"maxFiles": 282
|
||||
"maxFiles": 283
|
||||
}
|
||||
|
||||
@@ -198,83 +198,68 @@ describe('#3300 build_prompt optional-section guards under nullglob', () => {
|
||||
});
|
||||
|
||||
for (const shell of SHELLS) {
|
||||
test(`[${shell.name}] absent optional sources: no context/research output file is created at all`, () => {
|
||||
test(`[${shell.name}] absent optional sources: no context/research output file is created at all`, (t) => {
|
||||
const fx = buildFixture({ '01-PLAN.md': 'plan\n' }); // the common case: PLAN only
|
||||
try {
|
||||
const res = runBlock(shell, extractBuildPromptBlock(), fx);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
// Row 1/2 of the matrix — the failing-first core. Pre-fix these exist
|
||||
// as 0-byte files (the `>` redirect creates them before cat blocks or
|
||||
// EOFs); post-fix they must not exist at all.
|
||||
assert.strictEqual(
|
||||
readIfPresent(path.join(fx.runDir, 'gsd-review-context.md')),
|
||||
null,
|
||||
'gsd-review-context.md must NOT be created when no *-CONTEXT.md exists',
|
||||
);
|
||||
assert.strictEqual(
|
||||
readIfPresent(path.join(fx.runDir, 'gsd-review-research.md')),
|
||||
null,
|
||||
'gsd-review-research.md must NOT be created when no *-RESEARCH.md exists',
|
||||
);
|
||||
// The always-on parts of the block still did their job.
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-plan-00.md')), 'plan\n');
|
||||
} finally {
|
||||
cleanup(fx.root);
|
||||
}
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runBlock(shell, extractBuildPromptBlock(), fx);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
// Row 1/2 of the matrix — the failing-first core. Pre-fix these exist
|
||||
// as 0-byte files (the `>` redirect creates them before cat blocks or
|
||||
// EOFs); post-fix they must not exist at all.
|
||||
assert.strictEqual(
|
||||
readIfPresent(path.join(fx.runDir, 'gsd-review-context.md')),
|
||||
null,
|
||||
'gsd-review-context.md must NOT be created when no *-CONTEXT.md exists',
|
||||
);
|
||||
assert.strictEqual(
|
||||
readIfPresent(path.join(fx.runDir, 'gsd-review-research.md')),
|
||||
null,
|
||||
'gsd-review-research.md must NOT be created when no *-RESEARCH.md exists',
|
||||
);
|
||||
// The always-on parts of the block still did their job.
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-plan-00.md')), 'plan\n');
|
||||
});
|
||||
|
||||
test(`[${shell.name}] present optional sources: output matches the source exactly`, () => {
|
||||
test(`[${shell.name}] present optional sources: output matches the source exactly`, (t) => {
|
||||
const fx = buildFixture({
|
||||
'01-PLAN.md': 'plan\n',
|
||||
'01-CONTEXT.md': 'ctx\n',
|
||||
'01-RESEARCH.md': 'research\n',
|
||||
});
|
||||
try {
|
||||
const res = runBlock(shell, extractBuildPromptBlock(), fx);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-context.md')), 'ctx\n');
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-research.md')), 'research\n');
|
||||
} finally {
|
||||
cleanup(fx.root);
|
||||
}
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runBlock(shell, extractBuildPromptBlock(), fx);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-context.md')), 'ctx\n');
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-research.md')), 'research\n');
|
||||
});
|
||||
|
||||
test(`[${shell.name}] research-only phase: context output stays absent`, () => {
|
||||
test(`[${shell.name}] research-only phase: context output stays absent`, (t) => {
|
||||
const fx = buildFixture({ '01-PLAN.md': 'plan\n', '01-RESEARCH.md': 'research\n' });
|
||||
try {
|
||||
const res = runBlock(shell, extractBuildPromptBlock(), fx);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-context.md')), null);
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-research.md')), 'research\n');
|
||||
} finally {
|
||||
cleanup(fx.root);
|
||||
}
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runBlock(shell, extractBuildPromptBlock(), fx);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-context.md')), null);
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-research.md')), 'research\n');
|
||||
});
|
||||
|
||||
test(`[${shell.name}] multiple matches concatenate in glob order (original cat <glob> semantics)`, () => {
|
||||
test(`[${shell.name}] multiple matches concatenate in glob order (original cat <glob> semantics)`, (t) => {
|
||||
const fx = buildFixture({
|
||||
'01-PLAN.md': 'plan\n',
|
||||
'01-CONTEXT.md': 'A\n',
|
||||
'02-CONTEXT.md': 'B\n',
|
||||
});
|
||||
try {
|
||||
const res = runBlock(shell, extractBuildPromptBlock(), fx);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-context.md')), 'A\nB\n');
|
||||
} finally {
|
||||
cleanup(fx.root);
|
||||
}
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runBlock(shell, extractBuildPromptBlock(), fx);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(readIfPresent(path.join(fx.runDir, 'gsd-review-context.md')), 'A\nB\n');
|
||||
});
|
||||
|
||||
test(`[${shell.name}] open, unconsumed stdin pipe: block completes within ${STDIN_BOUND_MS}ms`, async () => {
|
||||
test(`[${shell.name}] open, unconsumed stdin pipe: block completes within ${STDIN_BOUND_MS}ms`, async (t) => {
|
||||
const fx = buildFixture({ '01-PLAN.md': 'plan\n' });
|
||||
try {
|
||||
const res = await runBlockWithOpenStdin(shell, extractBuildPromptBlock(), fx);
|
||||
assert.ok(!res.timedOut, `block blocked on stdin (killed at ${STDIN_BOUND_MS}ms) — cat ran operand-less`);
|
||||
assert.strictEqual(res.code, 0, `block exited ${res.code}${res.signal ? ` (signal ${res.signal})` : ''}`);
|
||||
} finally {
|
||||
cleanup(fx.root);
|
||||
}
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = await runBlockWithOpenStdin(shell, extractBuildPromptBlock(), fx);
|
||||
assert.ok(!res.timedOut, `block blocked on stdin (killed at ${STDIN_BOUND_MS}ms) — cat ran operand-less`);
|
||||
assert.strictEqual(res.code, 0, `block exited ${res.code}${res.signal ? ` (signal ${res.signal})` : ''}`);
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
372
tests/review-plan-coverage-manifest.test.cjs
Normal file
372
tests/review-plan-coverage-manifest.test.cjs
Normal file
@@ -0,0 +1,372 @@
|
||||
// allow-test-rule: source-text-is-the-product (#3301)
|
||||
// The workflow markdown IS the installed orchestration contract; these rows
|
||||
// extract the real shipped bash and execute it, never a hand-copied duplicate.
|
||||
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* #3301 — reviewers in the cross-AI plan-review workflow are never told the
|
||||
* plan ids or the total plan count, so a review that silently covers 6 of 7
|
||||
* plans is indistinguishable from one that covers all 7.
|
||||
*
|
||||
* Two behavioral seams, both extracted from the REAL shipped bash in
|
||||
* gsd-core/workflows/review.md (the same pattern as
|
||||
* tests/review-build-prompt-optional-sections.test.cjs):
|
||||
*
|
||||
* 1. build_prompt's plan-copy block — must derive a `.plans-manifest.md`
|
||||
* (plan ids + total count) from the `*-PLAN.md` filenames and append it
|
||||
* to both gsd-review-instructions.md and gsd-review-prompt.md.
|
||||
* 2. write_reviews' plan-coverage-check block — must grade each dispatched
|
||||
* lane's review file against that manifest, honoring two named regex
|
||||
* traps (escaped ids, hyphen-boundary exclusion) and skipping stub/empty/
|
||||
* CodeRabbit lanes.
|
||||
*
|
||||
* Script transport is temp-FILE based, never `bash -c <script>` (#2650 argv
|
||||
* mangling on Windows), matching the established convention.
|
||||
*/
|
||||
|
||||
const { describe, test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { spawnSync } = require('node:child_process');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const {
|
||||
cleanup,
|
||||
createTempDir,
|
||||
readWorkflowCombined,
|
||||
} = require('./helpers.cjs');
|
||||
const { PROBE_TIMEOUT_MS } = require('./helpers/timeouts.cjs');
|
||||
const { scanFencedBlocks } = require('../gsd-core/bin/lib/markdown-sectionizer.cjs');
|
||||
|
||||
const REVIEW_WORKFLOW = path.join(__dirname, '..', 'gsd-core', 'workflows', 'review.md');
|
||||
const REPO_ROOT = path.join(__dirname, '..');
|
||||
|
||||
function detectShells() {
|
||||
const shells = [{ name: 'bash', cmd: 'bash' }];
|
||||
const probe = spawnSync('zsh', ['-c', 'exit 0'], { timeout: PROBE_TIMEOUT_MS, windowsHide: true });
|
||||
if (!probe.error && probe.status === 0) shells.push({ name: 'zsh', cmd: 'zsh' });
|
||||
return shells;
|
||||
}
|
||||
const SHELLS = detectShells();
|
||||
|
||||
/**
|
||||
* Extract the fenced ```bash block of build_prompt that copies *-PLAN.md
|
||||
* files — the one that also writes gsd-review-context.md and
|
||||
* gsd-review-instructions.md. Same anchor/walk-backward pattern as
|
||||
* tests/review-build-prompt-optional-sections.test.cjs.
|
||||
*/
|
||||
function extractPlanCopyBlock() {
|
||||
const content = readWorkflowCombined(REVIEW_WORKFLOW);
|
||||
const anchorIdx = content.indexOf('gsd-review-context.md');
|
||||
assert.notEqual(anchorIdx, -1, 'gsd-review-context.md no longer appears in review.md (+steps)');
|
||||
const before = content.slice(0, anchorIdx);
|
||||
const fenceOpenRe = /```bash\r?\n/g;
|
||||
let lastOpen = -1;
|
||||
let m;
|
||||
while ((m = fenceOpenRe.exec(before)) !== null) lastOpen = m.index + m[0].length;
|
||||
assert.notEqual(lastOpen, -1, 'gsd-review-context.md is not inside a ```bash fence of review.md (+steps)');
|
||||
const after = content.slice(lastOpen);
|
||||
const closeIdx = after.indexOf('\n```');
|
||||
assert.notEqual(closeIdx, -1, 'unterminated ```bash fence around gsd-review-context.md');
|
||||
const body = after.slice(0, closeIdx);
|
||||
assert.ok(
|
||||
body.includes('gsd-review-instructions.md'),
|
||||
'extracted block writes gsd-review-context.md but not gsd-review-instructions.md — wrong block',
|
||||
);
|
||||
return body;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the fenced ```bash block of write_reviews that grades plan
|
||||
* coverage — anchored on the manifest variable this PR introduces. Walks
|
||||
* forward from the write_reviews step marker so it never grabs the earlier
|
||||
* (unrelated) gate-check bash block in the same step.
|
||||
*/
|
||||
function extractCoverageCheckBlock() {
|
||||
const content = readWorkflowCombined(REVIEW_WORKFLOW);
|
||||
const stepIdx = content.indexOf('<step name="write_reviews">');
|
||||
assert.notEqual(stepIdx, -1, 'write_reviews step must exist');
|
||||
const fromStep = content.slice(stepIdx);
|
||||
const anchorIdx = fromStep.indexOf('MANIFEST="$RUN_DIR/.plans-manifest.md"');
|
||||
assert.notEqual(
|
||||
anchorIdx,
|
||||
-1,
|
||||
'write_reviews must reference .plans-manifest.md — the plan-coverage-check block is missing',
|
||||
);
|
||||
const before = fromStep.slice(0, anchorIdx);
|
||||
const fenceOpenRe = /```bash\r?\n/g;
|
||||
let lastOpen = -1;
|
||||
let m;
|
||||
while ((m = fenceOpenRe.exec(before)) !== null) lastOpen = m.index + m[0].length;
|
||||
assert.notEqual(lastOpen, -1, '.plans-manifest.md reference is not inside a ```bash fence of write_reviews');
|
||||
const after = fromStep.slice(lastOpen);
|
||||
const closeIdx = after.indexOf('\n```');
|
||||
assert.notEqual(closeIdx, -1, 'unterminated ```bash fence around the plan-coverage-check block');
|
||||
return after.slice(0, closeIdx);
|
||||
}
|
||||
|
||||
/** Every fenced ```bash block of review.md (+steps) — for the CodeRabbit-slug structural row. */
|
||||
function extractAllBashBlocks() {
|
||||
const content = readWorkflowCombined(REVIEW_WORKFLOW);
|
||||
const lines = content.split(/\r?\n/);
|
||||
return scanFencedBlocks(lines)
|
||||
.filter((b) => b.closeLineIdx !== -1 && (b.infoString || '').trim() === 'bash')
|
||||
.map((b) => lines.slice(b.openLineIdx + 1, b.closeLineIdx).join('\n'));
|
||||
}
|
||||
|
||||
/** Fill the workflow's `{run_dir}` placeholder and stage the script in a file. */
|
||||
function stageScript(shell, body, root, runDir) {
|
||||
const scriptPath = path.join(root, `block-${shell.name}-${Math.random().toString(36).slice(2)}.sh`);
|
||||
fs.writeFileSync(scriptPath, body.split('{run_dir}').join(runDir));
|
||||
return scriptPath;
|
||||
}
|
||||
|
||||
function runScript(shell, body, root, runDir, env, cwd = root) {
|
||||
const scriptPath = stageScript(shell, body, root, runDir);
|
||||
return spawnSync(shell.cmd, [scriptPath], {
|
||||
cwd,
|
||||
env: { ...process.env, ...env },
|
||||
encoding: 'utf8',
|
||||
stdin: 'ignore',
|
||||
timeout: PROBE_TIMEOUT_MS,
|
||||
windowsHide: true,
|
||||
});
|
||||
}
|
||||
|
||||
const readIfPresent = (file) => (fs.existsSync(file) ? fs.readFileSync(file, 'utf8') : null);
|
||||
|
||||
/**
|
||||
* Fixture for the plan-copy block: a PHASE_DIR with the given *-PLAN.md
|
||||
* filenames, a fresh RUN_DIR, and the two always-copied section sources the
|
||||
* block expects from prompt assembly.
|
||||
*/
|
||||
function buildPlanCopyFixture(planFileNames) {
|
||||
const root = createTempDir('gsd-3301-copy-');
|
||||
const phaseDir = path.join(root, 'phase');
|
||||
const runDir = path.join(root, 'run');
|
||||
fs.mkdirSync(phaseDir);
|
||||
fs.mkdirSync(runDir);
|
||||
for (const name of planFileNames) {
|
||||
fs.writeFileSync(path.join(phaseDir, name), 'plan body\n');
|
||||
}
|
||||
fs.writeFileSync(path.join(root, 'instr.md'), 'instructions\n');
|
||||
fs.writeFileSync(path.join(root, 'roadmap.md'), 'roadmap\n');
|
||||
return {
|
||||
root,
|
||||
phaseDir,
|
||||
runDir,
|
||||
env: {
|
||||
PHASE_DIR: phaseDir,
|
||||
INSTRUCTIONS_BLOCK_FILE: path.join(root, 'instr.md'),
|
||||
ROADMAP_SECTION_FILE: path.join(root, 'roadmap.md'),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** Fixture for the coverage-check block: a RUN_DIR pre-populated with a manifest and lane files. */
|
||||
function buildCoverageFixture(planIds, lanes) {
|
||||
const root = createTempDir('gsd-3301-coverage-');
|
||||
const runDir = path.join(root, 'run');
|
||||
fs.mkdirSync(runDir);
|
||||
const manifestLines = ['', '## Plan Coverage Manifest', '', `Total plans in this review: ${planIds.length}`, ''];
|
||||
for (const id of planIds) manifestLines.push(`- ${id}`);
|
||||
fs.writeFileSync(path.join(runDir, '.plans-manifest.md'), manifestLines.join('\n') + '\n');
|
||||
for (const [slug, content] of Object.entries(lanes)) {
|
||||
fs.writeFileSync(path.join(runDir, `gsd-review-${slug}.md`), content);
|
||||
}
|
||||
return {
|
||||
root,
|
||||
runDir,
|
||||
env: { SELECTED_REVIEWERS: Object.keys(lanes).join(',') },
|
||||
};
|
||||
}
|
||||
|
||||
function readCoverageJson(runDir, slug) {
|
||||
const p = path.join(runDir, `.plan-coverage-${slug}.json`);
|
||||
return fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, 'utf8')) : null;
|
||||
}
|
||||
|
||||
describe('#3301 build_prompt derives and appends a plan coverage manifest', () => {
|
||||
for (const shell of SHELLS) {
|
||||
test(`[${shell.name}] zero plans: manifest reports Total plans: 0 and no ids`, (t) => {
|
||||
const fx = buildPlanCopyFixture([]);
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractPlanCopyBlock(), fx.root, fx.runDir, fx.env);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const manifest = readIfPresent(path.join(fx.runDir, '.plans-manifest.md'));
|
||||
assert.notEqual(manifest, null, '.plans-manifest.md must be written even with zero plans');
|
||||
assert.match(manifest, /Total plans in this review: 0/);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] one plan: manifest lists the single id and count 1`, (t) => {
|
||||
const fx = buildPlanCopyFixture(['01-PLAN.md']);
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractPlanCopyBlock(), fx.root, fx.runDir, fx.env);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const manifest = readIfPresent(path.join(fx.runDir, '.plans-manifest.md'));
|
||||
assert.match(manifest, /Total plans in this review: 1/);
|
||||
assert.match(manifest, /^- 01$/m);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] decimal-phase ids are preserved verbatim in the manifest`, (t) => {
|
||||
const fx = buildPlanCopyFixture(['12.6-01-PLAN.md', '12.6-02-PLAN.md']);
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractPlanCopyBlock(), fx.root, fx.runDir, fx.env);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const manifest = readIfPresent(path.join(fx.runDir, '.plans-manifest.md'));
|
||||
assert.match(manifest, /Total plans in this review: 2/);
|
||||
assert.match(manifest, /^- 12\.6-01$/m);
|
||||
assert.match(manifest, /^- 12\.6-02$/m);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] manifest is appended to both gsd-review-instructions.md and gsd-review-prompt.md`, (t) => {
|
||||
const fx = buildPlanCopyFixture(['01-PLAN.md']);
|
||||
t.after(() => cleanup(fx.root));
|
||||
// gsd-review-prompt.md is written earlier in build_prompt (the fenced
|
||||
// markdown template); simulate that so the append target pre-exists.
|
||||
fs.writeFileSync(path.join(fx.runDir, 'gsd-review-prompt.md'), '# prompt\n');
|
||||
const res = runScript(shell, extractPlanCopyBlock(), fx.root, fx.runDir, fx.env);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const instructions = readIfPresent(path.join(fx.runDir, 'gsd-review-instructions.md'));
|
||||
const prompt = readIfPresent(path.join(fx.runDir, 'gsd-review-prompt.md'));
|
||||
assert.match(instructions, /Plan Coverage Manifest/);
|
||||
assert.match(prompt, /Plan Coverage Manifest/);
|
||||
});
|
||||
}
|
||||
|
||||
test('manifest filename does not collide with either existing RUN_DIR glob', () => {
|
||||
const name = '.plans-manifest.md';
|
||||
assert.ok(!/^gsd-review-.*\.md$/.test(name), 'must not match the gsd-review-*.md reviewer-report glob');
|
||||
assert.ok(!/^gsd-review-plan-.*\.md$/.test(name), 'must not match the gsd-review-plan-*.md plan-copy glob');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#3301 Review Instructions require one section per manifest id', () => {
|
||||
const workflow = readWorkflowCombined(REVIEW_WORKFLOW);
|
||||
|
||||
test('documents mandatory per-id section requirement before cross-plan content', () => {
|
||||
assert.ok(
|
||||
workflow.includes('Plan Coverage Manifest'),
|
||||
'build_prompt prompt template must reference the Plan Coverage Manifest section',
|
||||
);
|
||||
assert.ok(
|
||||
/plan coverage is mandatory/i.test(workflow),
|
||||
'Review Instructions must state that plan coverage is mandatory',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('#3301 write_reviews grades each lane against the plan coverage manifest', () => {
|
||||
for (const shell of SHELLS) {
|
||||
test(`[${shell.name}] coverage check: complete when every id appears`, (t) => {
|
||||
const fx = buildCoverageFixture(['01', '02'], {
|
||||
gemini: '## 01\n\nlooks good\n\n## 02\n\nlooks good too\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const cov = readCoverageJson(fx.runDir, 'gemini');
|
||||
assert.deepEqual(cov, { complete: true, missing_ids: [], total: 2 });
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: reports the specific missing id (the field-observed "6 of 7" case)`, (t) => {
|
||||
const ids = ['01', '02', '03', '04', '05', '06', '07'];
|
||||
const body = ids
|
||||
.filter((id) => id !== '07')
|
||||
.map((id) => `## ${id}\n\ncovered\n`)
|
||||
.join('\n');
|
||||
const fx = buildCoverageFixture(ids, { qwen: body });
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const cov = readCoverageJson(fx.runDir, 'qwen');
|
||||
assert.strictEqual(cov.complete, false);
|
||||
assert.deepEqual(cov.missing_ids, ['07']);
|
||||
assert.strictEqual(cov.total, 7);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: unescaped dot cannot be satisfied by 12X6-01 (issue trap 1)`, (t) => {
|
||||
const fx = buildCoverageFixture(['12.6-01'], {
|
||||
codex: 'discussion of 12X6-01 but never the real id\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const cov = readCoverageJson(fx.runDir, 'codex');
|
||||
assert.strictEqual(cov.complete, false, 'unescaped "." would let 12X6-01 wrongly satisfy 12.6-01');
|
||||
assert.deepEqual(cov.missing_ids, ['12.6-01']);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: a hyphen-prefixed token (T-04-07) does not satisfy id 04-07 (issue trap 2)`, (t) => {
|
||||
const fx = buildCoverageFixture(['04-07'], {
|
||||
claude: 'six sections away, threat id T-04-07 is discussed at length\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const cov = readCoverageJson(fx.runDir, 'claude');
|
||||
assert.strictEqual(cov.complete, false, 'a preceding hyphen must not count as a boundary for id 04-07');
|
||||
assert.deepEqual(cov.missing_ids, ['04-07']);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: a plain-prose mention counts as covered, no heading required`, (t) => {
|
||||
const fx = buildCoverageFixture(['12.6-01'], {
|
||||
gemini: 'This was already covered by plan 12.6-01 above, no separate section needed.\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
const cov = readCoverageJson(fx.runDir, 'gemini');
|
||||
assert.strictEqual(cov.complete, true);
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: a budget-skipped stub is not graded`, (t) => {
|
||||
const fx = buildCoverageFixture(['01'], {
|
||||
ollama: 'ollama review skipped: prompt budget (500 tokens) too small for the minimum review set.\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(readCoverageJson(fx.runDir, 'ollama'), null, 'a budget-skip stub must not be graded');
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: an empty review file is not graded`, (t) => {
|
||||
const fx = buildCoverageFixture(['01'], { codex: '' });
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(readCoverageJson(fx.runDir, 'codex'), null, 'an empty review file must not be graded');
|
||||
});
|
||||
|
||||
test(`[${shell.name}] coverage check: coderabbit lane is exempt from plan-coverage grading`, (t) => {
|
||||
const fx = buildCoverageFixture(['01', '02'], {
|
||||
coderabbit: 'diff-only review, mentions nothing about plans\n',
|
||||
});
|
||||
t.after(() => cleanup(fx.root));
|
||||
const res = runScript(shell, extractCoverageCheckBlock(), fx.root, fx.runDir, fx.env, REPO_ROOT);
|
||||
assert.strictEqual(res.status, 0, `block exited ${res.status}: ${res.stderr}`);
|
||||
assert.strictEqual(
|
||||
readCoverageJson(fx.runDir, 'coderabbit'),
|
||||
null,
|
||||
'coderabbit never receives the source-grounding prompt and must not be graded',
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
test('structural: no lane slug other than coderabbit is hardcoded-excluded (the exemption is named, not general)', () => {
|
||||
const offenders = extractAllBashBlocks().filter((b) => /\[\s*"\$SLUG"\s*=\s*"(?!coderabbit)/.test(b));
|
||||
assert.deepEqual(offenders, [], 'only the coderabbit exemption may hardcode a slug comparison');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#3301 REVIEWS.md documents the plan_coverage frontmatter key', () => {
|
||||
const workflow = readWorkflowCombined(REVIEW_WORKFLOW);
|
||||
|
||||
test('documents plan_coverage: frontmatter is present only when a lane is incomplete', () => {
|
||||
assert.ok(workflow.includes('plan_coverage'), 'write_reviews must document a plan_coverage frontmatter key');
|
||||
assert.ok(
|
||||
/plan_coverage.*only present if at least one/is.test(workflow),
|
||||
'plan_coverage must be documented as present only when at least one graded lane is incomplete',
|
||||
);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user