Phase 2 of #3464, following #3465. Rewrites every assertion that read a shipped .cjs/.js file and text-searched it, so the file no longer needs an allow-test-rule exemption. 6 of the 8 files are now marker-free; the ceiling drops 285 -> 278 against a measured 277. A source-grep passes when a STRING is present, not when the code WORKS. It survives a refactor that keeps the string but breaks the behavior, and breaks on a refactor that keeps the behavior but renames the string. Both failure modes are silent about the thing the test claims to protect. That is the anti-pattern ADR-456 and local/no-source-grep exist to prevent. Rewrites, each against the real exported seam: - discuss-mode: calls cmdInitPlanPhase() against a fixture whose config sets workflow.text_mode, asserts the value propagates to its emitted JSON. - effort-surface-axis: runs review-lane invoke against a real project with a fake `claude` PATH shim, asserts the resolved --effort actually lands in the shim's captured argv. - install-minimal-hooks: calls applySettingsJsonHooks() with a hook source missing, asserts it is neither registered nor silently registers wholesale, with sibling present hooks as the positive control. - install: calls the exported inferPreferredRuntime({fs, env, preferredConfigDir}) via its injected fs seam, asserting 'kilo' from both the config-marker and env-var paths. - opencode-permissions: spawns the real installer with a custom config dir, asserts the written opencode.json permission paths are anchored on it. - repo-layout: spawns the installer for copilot local vs global, asserting AGENTS.md is written only in the local case. - runtime-config-adapter-registry: stubs resolveInstallPlan and force-reloads install.js, asserting the runtime's artifact stops being written -- proving install.js genuinely routes through the registry. - runtime-homes-descriptor-drive: calls buildAgentSkillsBlock() for cursor and claude against real fixture SKILL.md files, asserting each runtime's refs land under its own config dir and never the other's. Every rewrite was mutation-checked before its marker was dropped. Because node --test is not runnable locally in this repo, each assertion was replicated in a standalone probe that requires the same module: run green against the real file, then red against a deliberately broken one (text_mode propagation deleted, existsSync guard removed, config dir hardcoded, !isGlobal guard dropped, kilo branch removed, effort resolution bypassed, skills base hardcoded back to .claude), then the production file restored and confirmed byte-identical. An assertion that could not be made to fail would not have shipped -- a behavioral test that passes regardless of correctness is strictly worse than the source-grep it replaces, because it looks rigorous while asserting nothing. Three claims were checked rather than trusted. All three were wrong: - repo-layout's own marker cited #1188 asserting the `!isGlobal` lexical scope was "unprovable at runtime". It is provable: the guard decides whether AGENTS.md is written, which is directly observable. Both directions verified. - The triage for runtime-config-adapter-registry claimed its source-grep was redundant with the file's EXPECTED_TABLE tests, so deletion would be safe. Those tests only exercise resolveInstallPlan() directly and never load bin/install.js, so they do not cover it. A real behavioral assertion was written instead of deleting coverage. - An earlier revision of this change dropped runtime-config-adapter-registry's two markers on the grounds that ESLint stayed silent without them. An adversarial review caught that this was wrong, and it is restored here. The file still genuinely source-greps bin/install.js at two sites; ESLint is silent only because no-source-grep's TEXT_METHODS omits matchAll. Dropping a marker because the linter cannot see the violation is exploiting the blind spot, not resolving it -- and it would go red the moment the rule is widened. Those two assertions are also irreducible: they assert that EVERY inline `runtime === '...'` branch in install.js names a registry-known runtime, and a branch naming an unregistered runtime would simply never execute, so no runtime observation can prove its absence. The markers now say so explicitly. Two coverage gaps in no-source-grep surfaced while doing this, recorded on #3464 rather than fixed here, since widening the rule is its own change with its own blast radius: - TEXT_METHODS omits matchAll, so a matchAll source-grep never trips the rule (the case above). - The path test requires a literal quoted bin/lib/gsd-core/src segment and tracks the binding one hop, so a read through a dynamic path or an intermediate variable evades it. install-minimal-hooks' genuinely load-bearing read at line 975 is itself unmarked for a related reason. install-minimal-hooks therefore keeps its markers too: its remaining real source-grep of bin/install.js (the Codex legacy gsd-update-check migration check, line 975) is outside this issue's 8 sites. It is the last blocker for that file and is a clean follow-up. Closes #3466 Co-authored-by: sim <sim@local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
@@ -1,12 +1,8 @@
|
||||
// allow-test-rule: source-text-is-the-product (see #2481, #2615)
|
||||
// The final describe block asserts on gsd-core/workflows/review.md's text. A
|
||||
// workflow .md IS what the runtime loads — its literal command lines are the
|
||||
// deployed contract, and there is no runtime seam that executes review.md here.
|
||||
// The #2615 matrix-parity block below is the same kind of contract assertion:
|
||||
// docs/reference/host-integration-capability-matrix.md IS the cited source of
|
||||
// truth for every descriptor axis (ADR-1239), so asserting a shipped axis
|
||||
// value appears there and matches is a contract assertion, not a source grep.
|
||||
// Every other block in this file is behavioral (CLI + module surface).
|
||||
// #2615 the matrix-parity block below (the file's final describe block) is a
|
||||
// contract assertion, not a source grep: docs/reference/host-integration-capability-matrix.md
|
||||
// IS the cited source of truth for every descriptor axis (ADR-1239), so asserting a shipped
|
||||
// axis value appears there and matches is a contract assertion. Every other block in this file
|
||||
// is behavioral (CLI + module surface).
|
||||
|
||||
/**
|
||||
* #2481 — ADR-1239 `effortSurface` axis + ADR-443 path (a).
|
||||
@@ -23,6 +19,8 @@ const { describe, test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const os = require('node:os');
|
||||
const cp = require('node:child_process');
|
||||
const fc = require('fast-check');
|
||||
|
||||
const { runGsdTools, createTempProject, cleanup } = require('./helpers.cjs');
|
||||
@@ -418,18 +416,95 @@ describe('#2481 — ADR-443 mechanism callers, as they actually exist', () => {
|
||||
});
|
||||
|
||||
describe('#2481 review workflow resolves effort per reviewer', () => {
|
||||
test('shipped orchestration invokes resolve-execution — the grep ADR-443 said returned zero hits', () => {
|
||||
test('shipped orchestration: the live claude lane genuinely receives --effort <level> in its spawned argv', () => {
|
||||
// Phase 5b (#2799) moved the call out of review.md's per-lane bash and into the review-lane
|
||||
// route, which resolves effort once per selected lane through the SAME surface. ADR-443's
|
||||
// invariant is about shipped orchestration calling resolve-execution at all, not about which
|
||||
// file it lives in — so the assertion follows the call rather than pinning the old location.
|
||||
const toolsSrc = fs.readFileSync(
|
||||
path.join(__dirname, '..', 'gsd-core', 'bin', 'gsd-tools.cjs'), 'utf-8',
|
||||
);
|
||||
assert.ok(
|
||||
toolsSrc.includes('resolve-execution'),
|
||||
'ADR-443 blocks on no shipped orchestration calling resolve-execution',
|
||||
);
|
||||
// route's `effortFor()`, which SPAWNS `query resolve-execution … --pick effort_argv_string`
|
||||
// once per selected lane and folds the result into that lane's argv template. A text grep for
|
||||
// the string "resolve-execution" in gsd-tools.cjs would pass even if effortFor's result were
|
||||
// silently dropped before reaching the spawned reviewer, or if the call were dead code. This
|
||||
// drives the REAL `review-lane invoke` route end-to-end — real cp.spawnSync, a real project
|
||||
// config, a real claude-shaped shim on PATH — and inspects the argv the shim actually received,
|
||||
// which is the only way to prove the resolved effort reaches the invocation rather than merely
|
||||
// that some file mentions the command name.
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-2481-orchestration-e2e-'));
|
||||
const projectDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-2481-orchestration-project-'));
|
||||
try {
|
||||
const bin = path.join(dir, 'bin');
|
||||
const runDir = path.join(dir, 'run');
|
||||
fs.mkdirSync(bin);
|
||||
fs.mkdirSync(runDir);
|
||||
fs.writeFileSync(path.join(runDir, 'gsd-review-prompt.md'), 'prompt');
|
||||
|
||||
const seenArgv = path.join(dir, 'argv.txt');
|
||||
fs.writeFileSync(
|
||||
path.join(bin, 'claude'),
|
||||
'#!/usr/bin/env bash\n'
|
||||
+ 'cat >/dev/null\n'
|
||||
+ `printf '%s\\n' "$@" > "${seenArgv}"\n`
|
||||
+ 'echo "a review body long enough to clear the empty-output guard."\n',
|
||||
{ mode: 0o755 },
|
||||
);
|
||||
|
||||
// An extensionless file with a POSIX shebang is not executable on Windows:
|
||||
// CreateProcess resolves a bare `claude` command against PATHEXT
|
||||
// (.COM;.EXE;.BAT;.CMD;...), and a shebang-only file matches none of them,
|
||||
// so the shim above is invisible there. Ship a second shim recognized by
|
||||
// PATHEXT that writes the SAME newline-per-argv capture format the
|
||||
// assertion below parses. Delegating the actual argv capture to a small
|
||||
// Node script (invoked via `%*`) rather than parsing `%*` in batch avoids
|
||||
// cmd.exe's fragile re-splitting of quoted/spaced arguments — Node parses
|
||||
// the raw Windows command line itself, the same way the real `claude`
|
||||
// binary's argv would be parsed.
|
||||
if (process.platform === 'win32') {
|
||||
const captureScript = path.join(bin, '_claude-capture.cjs');
|
||||
fs.writeFileSync(
|
||||
captureScript,
|
||||
'const fs = require("fs");\n'
|
||||
+ 'process.stdin.resume();\n'
|
||||
+ 'process.stdin.on("end", () => {\n'
|
||||
+ ` fs.writeFileSync(${JSON.stringify(seenArgv)}, process.argv.slice(2).join("\\n") + "\\n");\n`
|
||||
+ ' console.log("a review body long enough to clear the empty-output guard.");\n'
|
||||
+ '});\n',
|
||||
);
|
||||
fs.writeFileSync(
|
||||
path.join(bin, 'claude.cmd'),
|
||||
`@echo off\r\n"${process.execPath}" "${captureScript}" %*\r\n`,
|
||||
);
|
||||
}
|
||||
|
||||
fs.mkdirSync(path.join(projectDir, '.planning'), { recursive: true });
|
||||
fs.writeFileSync(
|
||||
path.join(projectDir, '.planning', 'config.json'),
|
||||
JSON.stringify({ effort: { default: 'xhigh' } }, null, 2),
|
||||
);
|
||||
|
||||
const r = cp.spawnSync(
|
||||
process.execPath,
|
||||
[
|
||||
path.join(REPO_ROOT, 'gsd-core', 'bin', 'gsd-tools.cjs'),
|
||||
'review-lane', 'invoke', '--slug', 'claude',
|
||||
'--run-dir', runDir, '--repo-root', REPO_ROOT, '--json',
|
||||
],
|
||||
{
|
||||
cwd: projectDir,
|
||||
encoding: 'utf8',
|
||||
timeout: 60000,
|
||||
killSignal: 'SIGKILL',
|
||||
env: { ...process.env, PATH: `${bin}${path.delimiter}${process.env.PATH}` },
|
||||
},
|
||||
);
|
||||
assert.equal(r.status, 0, `review-lane invoke failed: ${r.stderr}`);
|
||||
assert.ok(fs.existsSync(seenArgv), `the claude shim never ran; stdout was: ${r.stdout}`);
|
||||
|
||||
const argv = fs.readFileSync(seenArgv, 'utf8').trim().split(/\r?\n/);
|
||||
assert.ok(
|
||||
argv.includes('--effort') && argv.includes('xhigh'),
|
||||
`resolved effort ("xhigh") did not reach the spawned claude reviewer's argv: ${JSON.stringify(argv)}`,
|
||||
);
|
||||
} finally {
|
||||
cleanup(dir);
|
||||
cleanup(projectDir);
|
||||
}
|
||||
});
|
||||
|
||||
test('each argv-effort reviewer places effort in its resolved command line', () => {
|
||||
|
||||
Reference in New Issue
Block a user