feat(#2398): consensus gate for CYCLE_SUMMARY on multi-reviewer runs (#3755)

* test(#2398): failing-first suite for the CYCLE_SUMMARY consensus gate

Binds the gate before it exists, so the suite is RED against next.

The load-bearing rows are the two the closed PR #2417 did not have. The B2
regression row asserts a judgment-class lone HIGH counts WITHOUT corroboration
when its raiser is unmarked — if anyone re-couples that class to corroboration,
more reviewers again produce a weaker gate than one, which is what closed #2417.
The parity row asserts every marker literal the gate names is one
review-lane-runner actually emits, so the gate cannot key on a signal nothing
produces; a mutation row and a seeded fast-check property prove that guard runs
its failure branch rather than only reading a correct tree.

Also pinned: gate position before Counting rules, the untouched CYCLE_SUMMARY
line shape the orchestrator greps, fence balance, the single-reviewer no-op,
classification by what a claim asserts rather than by citation presence, the
all-marked fail-open, current_actionable staying out of scope, and the
leading-marker requirement that stops a review which merely quotes a marker
from suppressing its own findings.

* feat(#2398): consensus gate for CYCLE_SUMMARY on multi-reviewer runs

With review.reviewer_instances running several reviewer identities off one
adapter, any single instance's fabricated HIGH could force a full replan cycle
on its own. Across ~9 real cycles on two projects each of four instances
fabricated at least once, and each was also the most accurate reviewer in some
other cycle, so dropping to fewer reviewers trades away real signal.

The gate engages only when 2+ reviewers actually ran, and weighs a lone HIGH by
what the claim asserts rather than by whether anyone agreed with it. An
existence claim -- a symbol, file, flag, commit or ID exists, is absent, or says
something specific -- counts only if source-grounding confirms it or another
reviewer raised the same concern. A judgment claim -- a design or correctness
property -- counts unless that reviewer's own section opens with an
evidence-quality discount marker the review lane already stamps
([reviewed-without-source-citations] #3194, [reviewed-without-repo-access]
#2176, or a diff-only lane).

That split is what resolves B2, the finding that closed PR #2417. B2 showed the
approved wording made more reviewers produce a WEAKER gate than one: condition
(a) pointed at the source-grounding pass, which verifies every symbol THE PLAN
cites and never takes reviewer claims as input, so a genuine architectural HIGH
that one reviewer caught and another missed was neither groundable nor
corroborated and stopped gating. Judgment-class findings are therefore exempt
from corroboration entirely -- reviewers catch materially different classes of
issue, and demanding two of them independently raise the same architectural
concern suppresses exactly what a multi-reviewer setup exists to surface.

Guards on the gate itself: an all-marked cycle disengages it, so a cycle in
which nothing was verified can never be counted as converged; the marker must
OPEN a reviewer's section, so a review that merely quotes a marker does not
suppress its own findings; a suppressed HIGH stays listed and tagged rather
than dropped; current_actionable is untouched; and a single-reviewer run is
unchanged.

No new command, config key, or dependency -- the gate reads signals that
already exist. The CYCLE_SUMMARY line shape the orchestrator greps is
unchanged; only the integer it computes moves, and only for 2+ reviewers.

Known limit, inherited rather than introduced: SOURCE_CITATION_RE checks
citation presence, not resolution, which src/review-lane-runner.cts records as
a deliberate #3194 scope boundary. A fabricated but plausible file:line still
gates.

Scope revised and re-approved on the issue before any code was written.

* test(#2398): make marker parity behavioral, and stop overclaiming the gate

Review found the parity tests were vacuous: they asserted a marker STRING
appeared in review-lane-runner.cjs's source text, never requiring the module or
calling the stampers, so they would pass even if stampUngroundedReview were
broken or never invoked. They now invoke the real exported functions and assert
what those functions PRODUCE — that an uncited review gains a leading marker
blockquote, that a review carrying a file:line does not, that a self-reported
blind review is stamped, and that stamping is idempotent. Removing the source
read also removes an incidental no-source-grep evasion via a parameterized path.

Review also found the changeset headline false for the class it matters most
in. The discount markers detect 'cited nothing' and 'had no repo access'; they
cannot detect 'drew a wrong conclusion from a real citation', so a judgment-class
finding invented by an evidence-bearing reviewer still counts alone. That is the
deliberate side of the tradeoff jags-faith named when closing #2417 — the
alternative is requiring corroboration for design findings, which is B2 — but
the changeset claimed lone hallucinations no longer force a cycle, full stop.
Corrected there, and stated plainly in docs/COMMANDS.md and the design record.

Also dropped the reviewer-instances.md entry from the emitted-drift ack: the
growth ratchet's currentSizes() scans only gsd-core/workflows/ and agents/
(tests/helpers/emitted-runtime.cjs:916-929), so references/ is outside it and
that entry acknowledged a delta the gate cannot see.

* chore(#2398): backfill changeset pr number to 3755

---------

Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
2026-08-22 10:53:14 -04:00
committed by GitHub
parent 444069d601
commit 738f42f4fd
6 changed files with 373 additions and 1 deletions

View File

@@ -200,9 +200,50 @@ Your final response MUST include a machine-readable line of exactly this form:
Where <N> is the integer count of HIGH-severity concerns that REMAIN UNRESOLVED in this cycle's findings.
Where <M> is the integer count of actionable MEDIUM/LOW concerns that REMAIN UNRESOLVED because the latest PLAN.md files do not yet incorporate them or explicitly defer/reject them.
Consensus gate (applies to NEWLY RAISED HIGHs only; evaluate before the counting rules below):
This gate engages ONLY when 2 or more reviewers actually ran and produced a review section this
cycle. With exactly one reviewer, skip this entire gate — a single reviewer's HIGH always counts,
exactly as before.
Classify each newly raised HIGH by what the claim ASSERTS, not by whether it happens to contain a
file:line citation:
- EXISTENCE-CLASS — asserts that a named symbol, file, path, flag, commit, or ID exists,
is absent, or says something specific ("X does not exist", "the plan cites Y which is missing",
"file Z contains Q").
- JUDGMENT-CLASS — asserts a design or correctness property ("no idempotency on retried writes",
"race between A and B", "missing rate limit"). A judgment-class HIGH stays judgment-class even
when it cites a file for context.
A HIGH raised by 2+ reviewers is corroborated and always counts.
For a HIGH raised by exactly ONE reviewer:
- EXISTENCE-CLASS — counts only if the source-grounding pass independently confirms it against
real project source, or another reviewer raised the same or a materially overlapping concern
(i.e. it lands in REVIEWS.md's Consensus Summary "Agreed Concerns").
- JUDGMENT-CLASS — counts UNLESS that reviewer's own section OPENS with an evidence-quality
discount marker blockquote: `[reviewed-without-source-citations]` or
`[reviewed-without-repo-access]`, or the reviewer is a diff-only lane (CodeRabbit). The marker
must be the LEADING blockquote of that reviewer's section — a review that merely quotes a
marker while discussing it is NOT marked. Corroboration by another reviewer overrides the
marker and the HIGH counts.
Judgment-class findings are deliberately NOT subject to corroboration. Different reviewers catch
materially different classes of issue, so requiring two of them to independently raise the same
architectural concern would suppress exactly the findings a multi-reviewer setup exists to surface.
FAIL OPEN: if EVERY reviewer that ran this cycle carries a discount marker, this gate does not
apply at all — count as if it were absent. A gate must never manufacture convergence out of a
cycle in which nothing was verified.
A HIGH suppressed by this gate is still listed under "## Current HIGH Concerns", tagged
`(single-reviewer, unconfirmed)`. It is excluded from current_high only — never silently dropped,
and never removed from the report.
This gate governs current_high only. current_actionable is unaffected.
Counting rules:
INCLUDE in the count:
- Newly raised HIGHs in this cycle
- Newly raised HIGHs in this cycle (subject to the consensus gate above)
- PARTIALLY RESOLVED HIGHs: concern acknowledged and a mitigation is in progress, but not yet verified/completed
- Previously raised HIGHs that are still unresolved