test(#3466): replace 8 source-grep assertions with behavioral tests (#3500)

Phase 2 of #3464, following #3465. Rewrites every assertion that read a
shipped .cjs/.js file and text-searched it, so the file no longer needs an
allow-test-rule exemption. 6 of the 8 files are now marker-free; the ceiling
drops 285 -> 278 against a measured 277.

A source-grep passes when a STRING is present, not when the code WORKS. It
survives a refactor that keeps the string but breaks the behavior, and breaks
on a refactor that keeps the behavior but renames the string. Both failure
modes are silent about the thing the test claims to protect. That is the
anti-pattern ADR-456 and local/no-source-grep exist to prevent.

Rewrites, each against the real exported seam:

- discuss-mode: calls cmdInitPlanPhase() against a fixture whose config sets
  workflow.text_mode, asserts the value propagates to its emitted JSON.
- effort-surface-axis: runs review-lane invoke against a real project with a
  fake `claude` PATH shim, asserts the resolved --effort actually lands in the
  shim's captured argv.
- install-minimal-hooks: calls applySettingsJsonHooks() with a hook source
  missing, asserts it is neither registered nor silently registers wholesale,
  with sibling present hooks as the positive control.
- install: calls the exported inferPreferredRuntime({fs, env,
  preferredConfigDir}) via its injected fs seam, asserting 'kilo' from both
  the config-marker and env-var paths.
- opencode-permissions: spawns the real installer with a custom config dir,
  asserts the written opencode.json permission paths are anchored on it.
- repo-layout: spawns the installer for copilot local vs global, asserting
  AGENTS.md is written only in the local case.
- runtime-config-adapter-registry: stubs resolveInstallPlan and force-reloads
  install.js, asserting the runtime's artifact stops being written -- proving
  install.js genuinely routes through the registry.
- runtime-homes-descriptor-drive: calls buildAgentSkillsBlock() for cursor and
  claude against real fixture SKILL.md files, asserting each runtime's refs
  land under its own config dir and never the other's.

Every rewrite was mutation-checked before its marker was dropped. Because
node --test is not runnable locally in this repo, each assertion was
replicated in a standalone probe that requires the same module: run green
against the real file, then red against a deliberately broken one (text_mode
propagation deleted, existsSync guard removed, config dir hardcoded, !isGlobal
guard dropped, kilo branch removed, effort resolution bypassed, skills base
hardcoded back to .claude), then the production file restored and confirmed
byte-identical. An assertion that could not be made to fail would not have
shipped -- a behavioral test that passes regardless of correctness is strictly
worse than the source-grep it replaces, because it looks rigorous while
asserting nothing.

Three claims were checked rather than trusted. All three were wrong:

- repo-layout's own marker cited #1188 asserting the `!isGlobal` lexical scope
  was "unprovable at runtime". It is provable: the guard decides whether
  AGENTS.md is written, which is directly observable. Both directions verified.
- The triage for runtime-config-adapter-registry claimed its source-grep was
  redundant with the file's EXPECTED_TABLE tests, so deletion would be safe.
  Those tests only exercise resolveInstallPlan() directly and never load
  bin/install.js, so they do not cover it. A real behavioral assertion was
  written instead of deleting coverage.
- An earlier revision of this change dropped runtime-config-adapter-registry's
  two markers on the grounds that ESLint stayed silent without them. An
  adversarial review caught that this was wrong, and it is restored here. The
  file still genuinely source-greps bin/install.js at two sites; ESLint is
  silent only because no-source-grep's TEXT_METHODS omits matchAll. Dropping a
  marker because the linter cannot see the violation is exploiting the blind
  spot, not resolving it -- and it would go red the moment the rule is
  widened. Those two assertions are also irreducible: they assert that EVERY
  inline `runtime === '...'` branch in install.js names a registry-known
  runtime, and a branch naming an unregistered runtime would simply never
  execute, so no runtime observation can prove its absence. The markers now
  say so explicitly.

Two coverage gaps in no-source-grep surfaced while doing this, recorded on
#3464 rather than fixed here, since widening the rule is its own change with
its own blast radius:

- TEXT_METHODS omits matchAll, so a matchAll source-grep never trips the rule
  (the case above).
- The path test requires a literal quoted bin/lib/gsd-core/src segment and
  tracks the binding one hop, so a read through a dynamic path or an
  intermediate variable evades it. install-minimal-hooks' genuinely
  load-bearing read at line 975 is itself unmarked for a related reason.

install-minimal-hooks therefore keeps its markers too: its remaining real
source-grep of bin/install.js (the Codex legacy gsd-update-check migration
check, line 975) is outside this issue's 8 sites. It is the last blocker for
that file and is a clean follow-up.

Closes #3466

Co-authored-by: sim <sim@local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-08-14 15:57:25 -04:00
committed by GitHub
parent 895d9df96d
commit 8a6d87538f
10 changed files with 588 additions and 311 deletions

View File

@@ -1,6 +1,6 @@
// allow-test-rule: source-text-is-the-product
// Reads .md/.json/.yml product files whose deployed text IS what the
// runtime loads — testing text content tests the deployed contract.
// runtime loads — testing text content tests the deployed contract. (No
// .cjs/.js/.ts source-grep remains in this file — see #3466.)
/**
* Installer Module — Sections 1–5.
@@ -736,15 +736,16 @@ describe('configureKiloPermissions', () => {
});
describe('Kilo integration — install/uninstall behaviour', () => {
// Product-text reads for test 6 only — update.md and update-context.cjs
// are deployed artifacts whose text IS the runtime contract (allow-test-rule).
// update.md IS the deployed workflow contract — its literal command lines are
// what the runtime loads, and there is no runtime seam that executes update.md
// here, so this .md read stays a text assertion (does not trigger no-source-grep).
const updateWorkflowSrc = fs.readFileSync(
path.join(__dirname, '..', 'gsd-core', 'workflows', 'update.md'), 'utf8');
// #498: update.md's runtime/scope/config-dir resolution moved into the tested
// projection gsd-core/bin/lib/update-context.cjs. Custom-config-dir
// detection (kilo.jsonc, KILO_CONFIG) is now asserted there.
const updateContextSrc = fs.readFileSync(
path.join(__dirname, '..', 'gsd-core', 'bin', 'lib', 'update-context.cjs'), 'utf8');
// projection gsd-core/bin/lib/update-context.cjs. Custom-config-dir detection
// (kilo.jsonc, KILO_CONFIG) is asserted behaviorally below via
// inferPreferredRuntime() itself, not via a source grep on update-context.cjs.
const { inferPreferredRuntime } = require('../gsd-core/bin/lib/update-context.cjs');
let tmpDir;
let previousCwd;
@@ -882,10 +883,35 @@ describe('Kilo integration — install/uninstall behaviour', () => {
test('update workflow checks preferred custom config dirs', () => {
// update.md still derives the preferred config dir from execution_context…
assert.ok(updateWorkflowSrc.includes('PREFERRED_CONFIG_DIR'));
// …and the custom-dir detection (kilo.jsonc config marker, KILO_CONFIG env)
// now lives in the tested update-context projection (#498).
assert.ok(updateContextSrc.includes('kilo.jsonc'));
assert.ok(updateContextSrc.includes('KILO_CONFIG'));
});
test('inferPreferredRuntime infers "kilo" from a kilo.jsonc marker in preferredConfigDir', () => {
// Behavioural replacement for the update-context.cjs source grep (#3466):
// the custom-dir detection (kilo.jsonc config marker) lives in this exact
// projection (#498) — calling it directly, with an injected fs seam, proves
// the kilo branch actually resolves rather than merely that the string
// "kilo.jsonc" appears in the file.
const fakeFs = {
exists: (p) => String(p).endsWith('kilo.jsonc'),
};
const runtime = inferPreferredRuntime({
fs: fakeFs,
env: {},
preferredConfigDir: '/fake/kilo-config-dir',
});
assert.strictEqual(runtime, 'kilo');
});
test('inferPreferredRuntime infers "kilo" from KILO_CONFIG_DIR / KILO_CONFIG env when no config-dir marker is present', () => {
const fakeFs = { exists: () => false };
assert.strictEqual(
inferPreferredRuntime({ fs: fakeFs, env: { KILO_CONFIG_DIR: '/custom/kilo' }, preferredConfigDir: '' }),
'kilo',
);
assert.strictEqual(
inferPreferredRuntime({ fs: fakeFs, env: { KILO_CONFIG: '/custom/kilo/kilo.jsonc' }, preferredConfigDir: '' }),
'kilo',
);
});
});
@@ -1156,7 +1182,6 @@ describe('readCmdNames() — tolerates missing commands/gsd directory (#1223)',
});
// ─── Section N: Antigravity .agents canonical workspace dir (#791) ─────────────
// allow-test-rule: source-text-is-the-product
// Reads deployed agent .md files whose text IS the product surface the
// Antigravity runtime loads at startup (path references, command names).
@@ -1292,7 +1317,6 @@ describe('install — --devin-desktop CLI flag routes to windsurf runtime (#792)
});
});
// ─── Section N: Windsurf workflow slash-command install (#1615) ─────────────
// allow-test-rule: source-text-is-the-product
// Reads deployed workflow .md files whose text IS the product surface the
// Windsurf runtime loads at startup (path references, command names).
@@ -5985,7 +6009,6 @@ test('install.js tier-defaults object has exactly the same keys as manifest effo
{
const { describe: __foldDescribe } = require('node:test');
__foldDescribe("folded:bug-1367-claude-local-flat-command-layout (consolidation epic #1969 B6 #1975)", () => {
// allow-test-rule: source-text-is-the-product #1367
// Installed command `.md` files — their on-disk path determines the slash-command
// namespace registered by Claude Code. Asserting the layout (flat vs. subdirectory)
// IS a behavioral test of the deploy contract, not source-grep theater.
@@ -6189,8 +6212,7 @@ describe('bug #1367 — Claude local install uses flat gsd-<cmd>.md command layo
__gtmAfter(() => { if (__savedGsdTestMode === undefined) delete process.env.GSD_TEST_MODE; else process.env.GSD_TEST_MODE = __savedGsdTestMode; });
'use strict';
// allow-test-rule: source-text-is-the-product (see #2380)
// Reads .md/.json/.yml product files whose deployed text IS what the
// (see #2380) Reads .md/.json/.yml product files whose deployed text IS what the
// runtime loads — testing text content tests the deployed contract.
/**
@@ -7453,8 +7475,7 @@ describe('#3184: scripts/lib/ and scripts/changeset/ install/uninstall parity',
{
const { describe: __foldDescribe } = require('node:test');
__foldDescribe("folded:issue-607-installer-dry-run (test-hygiene sweep #3336 H3 Wave 4)", () => {
// allow-test-rule: integration-test-input (#607)
// Test-created temp dirs are the only filesystem reads here — not repo source files.
// (#607) Test-created temp dirs are the only filesystem reads here — not repo source files.
// This is an integration test that seeds fixture files in OS temp dirs and
// asserts that the installer correctly handles --dry-run and the
// cleanupLegacyGsdCc exported helper.