Files

Tom Boucher 738f42f4fd feat(#2398 ): consensus gate for CYCLE_SUMMARY on multi-reviewer runs (#3755 )

* test(#2398): failing-first suite for the CYCLE_SUMMARY consensus gate

Binds the gate before it exists, so the suite is RED against next.

The load-bearing rows are the two the closed PR #2417 did not have. The B2
regression row asserts a judgment-class lone HIGH counts WITHOUT corroboration
when its raiser is unmarked — if anyone re-couples that class to corroboration,
more reviewers again produce a weaker gate than one, which is what closed #2417.
The parity row asserts every marker literal the gate names is one
review-lane-runner actually emits, so the gate cannot key on a signal nothing
produces; a mutation row and a seeded fast-check property prove that guard runs
its failure branch rather than only reading a correct tree.

Also pinned: gate position before Counting rules, the untouched CYCLE_SUMMARY
line shape the orchestrator greps, fence balance, the single-reviewer no-op,
classification by what a claim asserts rather than by citation presence, the
all-marked fail-open, current_actionable staying out of scope, and the
leading-marker requirement that stops a review which merely quotes a marker
from suppressing its own findings.

* feat(#2398): consensus gate for CYCLE_SUMMARY on multi-reviewer runs

With review.reviewer_instances running several reviewer identities off one
adapter, any single instance's fabricated HIGH could force a full replan cycle
on its own. Across ~9 real cycles on two projects each of four instances
fabricated at least once, and each was also the most accurate reviewer in some
other cycle, so dropping to fewer reviewers trades away real signal.

The gate engages only when 2+ reviewers actually ran, and weighs a lone HIGH by
what the claim asserts rather than by whether anyone agreed with it. An
existence claim -- a symbol, file, flag, commit or ID exists, is absent, or says
something specific -- counts only if source-grounding confirms it or another
reviewer raised the same concern. A judgment claim -- a design or correctness
property -- counts unless that reviewer's own section opens with an
evidence-quality discount marker the review lane already stamps
([reviewed-without-source-citations] #3194, [reviewed-without-repo-access]
#2176, or a diff-only lane).

That split is what resolves B2, the finding that closed PR #2417. B2 showed the
approved wording made more reviewers produce a WEAKER gate than one: condition
(a) pointed at the source-grounding pass, which verifies every symbol THE PLAN
cites and never takes reviewer claims as input, so a genuine architectural HIGH
that one reviewer caught and another missed was neither groundable nor
corroborated and stopped gating. Judgment-class findings are therefore exempt
from corroboration entirely -- reviewers catch materially different classes of
issue, and demanding two of them independently raise the same architectural
concern suppresses exactly what a multi-reviewer setup exists to surface.

Guards on the gate itself: an all-marked cycle disengages it, so a cycle in
which nothing was verified can never be counted as converged; the marker must
OPEN a reviewer's section, so a review that merely quotes a marker does not
suppress its own findings; a suppressed HIGH stays listed and tagged rather
than dropped; current_actionable is untouched; and a single-reviewer run is
unchanged.

No new command, config key, or dependency -- the gate reads signals that
already exist. The CYCLE_SUMMARY line shape the orchestrator greps is
unchanged; only the integer it computes moves, and only for 2+ reviewers.

Known limit, inherited rather than introduced: SOURCE_CITATION_RE checks
citation presence, not resolution, which src/review-lane-runner.cts records as
a deliberate #3194 scope boundary. A fabricated but plausible file:line still
gates.

Scope revised and re-approved on the issue before any code was written.

* test(#2398): make marker parity behavioral, and stop overclaiming the gate

Review found the parity tests were vacuous: they asserted a marker STRING
appeared in review-lane-runner.cjs's source text, never requiring the module or
calling the stampers, so they would pass even if stampUngroundedReview were
broken or never invoked. They now invoke the real exported functions and assert
what those functions PRODUCE — that an uncited review gains a leading marker
blockquote, that a review carrying a file:line does not, that a self-reported
blind review is stamped, and that stamping is idempotent. Removing the source
read also removes an incidental no-source-grep evasion via a parameterized path.

Review also found the changeset headline false for the class it matters most
in. The discount markers detect 'cited nothing' and 'had no repo access'; they
cannot detect 'drew a wrong conclusion from a real citation', so a judgment-class
finding invented by an evidence-bearing reviewer still counts alone. That is the
deliberate side of the tradeoff jags-faith named when closing #2417 — the
alternative is requiring corroboration for design findings, which is B2 — but
the changeset claimed lone hallucinations no longer force a cycle, full stop.
Corrected there, and stated plainly in docs/COMMANDS.md and the design record.

Also dropped the reviewer-instances.md entry from the emitted-drift ack: the
growth ratchet's currentSizes() scans only gsd-core/workflows/ and agents/
(tests/helpers/emitted-runtime.cjs:916-929), so references/ is outside it and
that entry acknowledged a delta the gate cannot see.

* chore(#2398): backfill changeset pr number to 3755

---------

Co-authored-by: sim <sim@local>

2026-08-22 10:53:14 -04:00

7.0 KiB

Raw Blame History

Reviewer Instances (#1517)

Custom reviewer instances for /gsd:review: run one model-capable adapter (e.g. OpenCode) as several independent reviewer identities in a single review pass. Loaded lazily by gsd-core/workflows/review.md when review.reviewer_instances is configured. See ADR-1517 for the contract.

Config shape

A review.reviewer_instances object under the review namespace. Each entry maps an instance name to { cli, model?, agent? }:

{
  "review": {
    "reviewer_instances": {
      "opencode-deepseek": { "cli": "opencode", "model": "deepseek/deepseek-v4-pro", "agent": "review" },
      "opencode-mimo":     { "cli": "opencode", "model": "xiaomi/mimo-v2.5-pro" }
    },
    "default_reviewers": ["opencode-deepseek", "opencode-mimo", "codex"]
  }
}

Instance name: ^[a-z0-9][a-z0-9-]*$, must not equal a built-in slug. Validated at config-set time.
cli: MUST be a known adapter (KNOWN_REVIEWER_SLUGS) — never an arbitrary shell command.
model: opaque provider/model string, passed through verbatim. GSD does not parse it.
agent: opaque string; honoured only by adapters with a native agent concept (OpenCode --agent in v1). Ignored by other adapters.

Resolution rules (single source)

The canonical logic lives in resolveReviewerSelection / normalizeReviewerInstances in review-reviewer-selection.cjs. Apply the SAME rules in the workflow so the two surfaces cannot diverge (DEFECT.GENERATIVE-FIX; parity-locked in tests/review-reviewer-instances.test.cjs).

Instances participate ONLY via review.default_reviewers. They never appear under --all or explicit --<cli> flags, and there are no per-instance CLI flags.
Expand instance references BEFORE the built-in-slug check: an entry that is a key in review.reviewer_instances is an instance; an entry that is a built-in slug is a builtin.
An instance is available iff its base cli is detected (e.g. opencode-deepseek is available iff opencode is available).
An entry that is NEITHER a defined instance NOR a built-in slug is a hard error (likely a typo'd instance name) — stop and report it. Do NOT silently drop it. (When review.reviewer_instances is absent entirely, fall back to the legacy unknown-slug warn-and-drop behaviour for backward compatibility.)
model/agent/instance-name are opaque: pass them as separate argv elements. They are NEVER interpolated into shell strings.

Invocation

An instance resolves through a lane; it is not a lane itself (ADR-2782 D8). It takes no part in the roster, the flag set, or lane uniqueness — which is why an instance heading (## OpenCode Review (opencode-deepseek)) must never be read as a lane section.

Since Phase 5b (#2799) invoke_reviewers iterates declared lanes rather than hand-authored per-CLI blocks, so an instance is invoked through the same single seam as its base lane, with two substitutions:

# $INSTANCE_NAME is the reviewer identity (e.g. opencode-deepseek); $INSTANCE_MODEL / $INSTANCE_AGENT
# come from the instance spec. --run-dir is the run-scoped mktemp directory created once in
# gather_context (#2358) — the same directory every lane uses.
#
# The instance's OWN model replaces the lane's configured model, and the output lands under the
# INSTANCE name so two instances of one adapter never overwrite each other.
gsd_run query review-lane invoke \
  --slug "$INSTANCE_CLI" \
  --run-dir "$RUN_DIR" --repo-root "$REPO_ROOT" \
  --model "$INSTANCE_MODEL" ${INSTANCE_AGENT:+--agent "$INSTANCE_AGENT"} \
  --as "$INSTANCE_NAME"

--as is what makes the run write {run_dir}/gsd-review-${INSTANCE_NAME}.md instead of the lane's own {run_dir}/gsd-review-<slug>.md.

Everything the lane declares — probe, prompt channel, output channel, timeout floor, empty-output policy, handler — applies unchanged to an instance. That is the point of routing instances through the lane rather than duplicating its invocation: a cross-cutting fix reaches instances for free, where the previous per-adapter block had to be copied and kept in sync by hand.

Only opencode honours an agent field in v1; it is ignored by other adapters. model and agent are opaque pass-through strings and are NEVER interpolated into a shell string — the runner spawns with an argv array and shell: false.

REVIEWS.md contract

Frontmatter reviewers: records the actual identities invoked. For a built-in slug use the slug (opencode); for an instance use the instance name (opencode-deepseek), so frontmatter distinguishes the independent voices. Example: reviewers: [opencode-deepseek, opencode-mimo, codex].
Section headers: each instance gets its OWN top-level section, headed with the base adapter's display name plus the instance name in parentheses: ## OpenCode Review (opencode-deepseek). Same-cli instances are never collapsed.
Shared-adapter caveat: when ≥2 invoked instances share the same base cli, print a one-line caveat immediately after the frontmatter (before the first section), e.g.: > Note: opencode-deepseek and opencode-mimo share the opencode adapter; their consensus is cross-model, not cross-tool.

Interaction with the convergence loop (#2398)

Running 2+ instances changes how /gsd:plan-review-convergence counts HIGHs. Its consensus gate (plan-review-convergence.md, step 5a, immediately before the counting rules) engages only when two or more reviewers actually ran in a cycle — which is precisely the configuration this file enables.

Under that gate, a HIGH raised by exactly one instance is treated by what the claim asserts:

an existence-class claim (a symbol, file, flag, commit or ID exists / is absent / says X) counts toward current_high only if source-grounding confirms it or another reviewer raised the same concern;
a judgment-class claim (a design or correctness property) counts unless that instance's own section opens with an evidence-quality discount marker — [reviewed-without-source-citations] (#3194) or [reviewed-without-repo-access] (#2176).

Judgment-class findings are deliberately exempt from the corroboration requirement: instances catch materially different classes of issue, so demanding two of them independently raise the same architectural concern would suppress the findings this feature exists to surface.

A suppressed HIGH is still reported, tagged (single-reviewer, unconfirmed). If every instance that ran carries a discount marker the gate disengages entirely, so a cycle in which nothing was verified can never be counted as converged.

Practical consequence for this file's use case: instances of uneven reliability are safe to configure. A weak instance that returns no file:line evidence gets stamped, and its lone judgment-class HIGHs stop forcing replan cycles — while any instance that does produce grounded evidence keeps full blocking weight, alone, on exactly the architectural findings it was added to catch.

7.0 KiB Raw Blame History