`review-lane plan` resolved every cross-AI reviewer lane's reasoning effort by spawning `query resolve-execution gsd-plan-checker --host <slug>`. The agent id was a hardcoded literal, so `--host` chose only the argv RENDERING while the LEVEL always came from the installed plan-checker's frontmatter — `low` under every shipped model profile. Every prompt-fed lane therefore ran at a fast structural verifier's effort, and because the rendered argument is a CLI config override it silently beat the effort the operator had configured for that CLI. At `low` a large source-grounded prompt makes a model end its turn with no final message, so the lane came back empty and its stub read as a crash. Effort is a property of the review, so the lane declares it. Two new fields on ReviewerLane — `effortConfigKey` (`review.effort.<slug>`) and `defaultEffort` — carried through each capability manifest and the generated registry, set on the three lanes with an argv effort channel and null on the other nine. A new pure `resolveLaneEffort()` resolves config key -> lane default -> nothing, where "nothing" emits no effort argument at all and the reviewer CLI's own configuration decides; `inherit` selects that path explicitly and an unrecognized level falls back to the lane default rather than being forwarded to a CLI that would reject it. The host's negotiated effortSurface still gates the rendering, so ADR-1239/#2481's trust boundary holds on this path too. Resolving in-process also removes up to twelve subprocess spawns per review. The empty-output stub now names the effort the lane ran at and distinguishes a clean exit from a timeout kill, a non-zero exit, and a process that never ran — `status` is null for both a timeout and a signal, so those were indistinguishable before. The hint is hedged: a clean empty exit is most often a model stopping short, but it is also consistent with a CLI writing its output elsewhere. Also: the capability validator now knows both fields, rejects a malformed key or an out-of-vocabulary default, and rejects a default declared without a config key (a level the operator could never override). An existing end-to-end row in tests/effort-surface-axis.test.cjs asserted the old coupling; it now configures the lane's own key and pins the decoupling in the same real spawn, with the agent execution tier set to a level that must not appear. Emitted-Drift-Ack-Growth: review.md — the effort/model resolution-order table this fix adds. The workflow is where an operator looks to find out which knob set a lane's model and effort; leaving the new key undocumented there is the same invisibility that made the plan-checker coupling survive this long. Emitted-Drift-Ack-Growth: review.md — the effort/model resolution-order table this fix adds. The workflow is where an operator looks to find out which knob set a lane's model and effort, so leaving the new key undocumented there is the same invisibility that let the plan-checker coupling survive. Claude-Session: https://claude.ai/code/session_01CRMEuzNMWn3gs5uUW2ghcF Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Tom Boucher <trekkie@nomorestars.com>
This commit is contained in:
committed by
GitHub
parent
925a363879
commit
5ad9a36f35
@@ -14,7 +14,7 @@ const path = require('node:path');
|
||||
const fc = require('fast-check');
|
||||
|
||||
const { REVIEWER_LANES } = require('../gsd-core/bin/lib/review-lane-descriptor.cjs');
|
||||
const { resolveLanePlan, LANE_UNAVAILABLE } = require('../gsd-core/bin/lib/review-lane-invocation.cjs');
|
||||
const { resolveLanePlan, resolveLaneEffort, LANE_UNAVAILABLE } = require('../gsd-core/bin/lib/review-lane-invocation.cjs');
|
||||
const {
|
||||
checkEgressHost,
|
||||
probeLane,
|
||||
@@ -233,6 +233,108 @@ describe('runner — empty-output policy (#2494 / #2605 / #2794)', () => {
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* A plan built through the SAME two steps the CLI runs (#4255): resolve the lane's review
|
||||
* effort, then expand it into argv. `plan()` above passes no effort at all, which is the right
|
||||
* default for rows that do not care — but a stub row asserting the effort must not invent it.
|
||||
*/
|
||||
function planWithEffort(slug) {
|
||||
const lane = REVIEWER_LANES.find((l) => l.slug === slug);
|
||||
const eff = resolveLaneEffort(lane, () => undefined, (host, level) => ({
|
||||
argv: ['-c', `model_reasoning_effort=${level}`], value: level,
|
||||
}));
|
||||
const r = resolveLanePlan({
|
||||
lane, configGet: () => undefined, runDir: RUN, repoRoot: ROOT,
|
||||
effortArgs: eff.argv, effortValue: eff.value,
|
||||
});
|
||||
assert.equal(r.ok, true, `${slug} failed to resolve`);
|
||||
return r.plan;
|
||||
}
|
||||
|
||||
test('#4255 — the stub names the effort and says the exit was clean', () => {
|
||||
// A crash, a timeout kill and a model that ended its turn without writing a final message all
|
||||
// reach the stub as the same zero bytes. The third is what a too-low reasoning effort produces
|
||||
// on a large source-grounded prompt, and it was indistinguishable from the first two — the
|
||||
// operator read "Codex failed" and went looking for a broken CLI. The stub now names the level
|
||||
// it ran at (the value they would change) and how the process actually ended.
|
||||
const p = planWithEffort('codex');
|
||||
const d = deps();
|
||||
writeReviewOrStub(p, '', d, undefined, { status: 0 });
|
||||
const out = d.files[p.reviewPath];
|
||||
assert.ok(out.includes('failed or returned empty output'),
|
||||
'the header every downstream reader greps for must be untouched');
|
||||
assert.ok(/ran at effort=high/.test(out), 'the stub must name the effort the lane ran at');
|
||||
assert.ok(/exited cleanly inside the timeout/.test(out));
|
||||
assert.ok(/ending its turn without writing a final message/.test(out),
|
||||
'a clean exit with no output must be named as the likeliest cause, not left reading as a crash');
|
||||
assert.ok(/most often/.test(out),
|
||||
'the cause is hedged on purpose: a clean empty exit is consistent with a stopped-short model '
|
||||
+ 'AND with a CLI writing its output somewhere this lane did not read');
|
||||
});
|
||||
|
||||
test('#4255 — a timeout kill and a crash are NOT reported as stopping short', () => {
|
||||
// The non-vacuity half: the stopped-short hint must be earned by a clean exit, or it is
|
||||
// advice that sends the operator to raise effort on a lane that was killed or crashed.
|
||||
const timedOut = deps();
|
||||
writeReviewOrStub(planWithEffort('codex'), '', timedOut, undefined, { status: null, errorCode: 'ETIMEDOUT' });
|
||||
const t = timedOut.files[planWithEffort('codex').reviewPath];
|
||||
assert.ok(/killed by the outer timeout/.test(t));
|
||||
assert.ok(!/most often/.test(t), 'a timeout is not a model stopping short');
|
||||
|
||||
const crashed = deps();
|
||||
writeReviewOrStub(planWithEffort('codex'), '', crashed, undefined, { status: 127 });
|
||||
const c = crashed.files[planWithEffort('codex').reviewPath];
|
||||
assert.ok(/exited with status 127/.test(c));
|
||||
assert.ok(!/most often/.test(c), 'a non-zero exit is not a model stopping short');
|
||||
|
||||
// `status` is null for a process that never started or died on a signal, exactly as it is for
|
||||
// a timeout kill. Reporting "status null" named nothing the operator could act on, and the
|
||||
// two must not collapse into one another (Codex review of #4255).
|
||||
for (const [label, out, expect] of [
|
||||
['binary missing', { status: null, errorCode: 'ENOENT' }, /did not exit normally \(ENOENT\)/],
|
||||
['killed by a signal', { status: null }, /did not exit normally \(killed by a signal\)/],
|
||||
]) {
|
||||
const d = deps();
|
||||
writeReviewOrStub(planWithEffort('codex'), '', d, undefined, out);
|
||||
const text = d.files[planWithEffort('codex').reviewPath];
|
||||
assert.match(text, expect, `${label} must be named, not reported as "status null"`);
|
||||
assert.ok(!/status null/.test(text), `${label}: "status null" tells the operator nothing`);
|
||||
assert.ok(!/most often/.test(text), `${label} is not a model stopping short`);
|
||||
}
|
||||
});
|
||||
|
||||
test('#4255 — an HTTP lane is not described as having a reviewer CLI', () => {
|
||||
// ollama is reached directly over HTTP: there is no CLI, so "the CLI's own configuration
|
||||
// applied" would be a lie about what ran (Codex review of #4255).
|
||||
const p = plan('ollama');
|
||||
const d = deps();
|
||||
writeReviewOrStub(p, '', d, undefined, { status: 0 });
|
||||
const out = d.files[p.reviewPath];
|
||||
assert.ok(/HTTP lane/.test(out));
|
||||
assert.ok(!/reviewer CLI/.test(out), 'an HTTP lane has no reviewer CLI to attribute anything to');
|
||||
assert.ok(!/effort=/.test(out), 'no level may be claimed for a transport that carries none');
|
||||
});
|
||||
|
||||
test('#4255 — a lane that sent no effort argument says so, rather than naming a level', () => {
|
||||
// gemini declares no effort channel, so `plan.effort` is null. Printing a level there would
|
||||
// be a lie about what reached the CLI; the stub says the CLI's own configuration applied.
|
||||
const p = plan('gemini');
|
||||
const d = deps();
|
||||
writeReviewOrStub(p, '', d, undefined, { status: 0 });
|
||||
const out = d.files[p.reviewPath];
|
||||
assert.ok(/no effort argument, so the reviewer CLI's own configuration applied/.test(out));
|
||||
assert.ok(!/effort=/.test(out), 'no level may be claimed for a lane that sent none');
|
||||
});
|
||||
|
||||
test('#4255 — the diagnosis is added even when the outcome is unknown', () => {
|
||||
// The HTTP path has no spawn outcome to pass. The effort half still applies, so the line is
|
||||
// still written — just without the exit clause it cannot honestly make.
|
||||
const p = planWithEffort('codex');
|
||||
const d = deps();
|
||||
writeReviewOrStub(p, '', d);
|
||||
assert.ok(/ran at effort=high/.test(d.files[p.reviewPath]));
|
||||
});
|
||||
|
||||
test('the stub is distinguishable from a real review', () => {
|
||||
// The ambiguity between "failed" and "ran cleanly with nothing to report" IS the defect.
|
||||
const p = plan('gemini');
|
||||
|
||||
Reference in New Issue
Block a user