Commit Graph

4 Commits

Author SHA1 Message Date
Tom Boucher
e2bfc06558 fix(#4709): a retired runtime id must not resolve to Claude Code (#4756)
* fix(#4709): a retired runtime id must not resolve to Claude Code

AC#1 of epic #4709 — the last unmet acceptance criterion. Every other phase
(#4711, #4716, #4732, #4743, #4753) is merged; the epic does not close until
this lands.

THE DEFECT, MEASURED

Five runtime-resolution accessors resolved a RETIRED id to a plausible-looking
value, indistinguishable from the same call with a canonical id. Measured on
5d4c98cde7 by executing the built modules:

  getRuntimeLabel('gemini')              -> 'Claude Code'
  getProjectInstructionFile('gemini')    -> 'AGENTS.md'
  getGlobalConfigHomeFragment('gemini')  -> "'.claude'"
  getGlobalConfigDir('gemini')           -> ~/.claude   (byte-identical to 'claude')
  getDirName('gemini')                   -> '.claude'

So asking for a runtime Google sunset on 2026-06-18 wrote into Claude Code's
global config home and labelled the install "Claude Code". Nothing errored and
nothing warned.

AC#1 names four accessors. getDirName is the fifth, found by a reviewer: same
module, same silent-wrong-answer class, and it feeds capability-state's
runtimeConfigDir. Fixing only the four the criterion happened to list would
have left the defect reachable, so it is guarded too.

WHY THE CHECK CANNOT LIVE IN CANONICALIZATION

canonicalizeRuntimeName returns null for 'gemini', 'gemini-cli', 'Gemini',
'GEMINI' AND for ''. After canonicalization a retired id, an unknown id and an
empty string are the same value, so anything keyed off the canonical form
cannot tell them apart — it would have to treat all three alike, which is the
behaviour being fixed. The check therefore runs on the RAW input.

WHAT THIS DELIBERATELY DOES NOT DO

The criterion reads "reject a non-canonical runtime id". Taken literally that
overturns three recorded decisions, so the narrower reading was put to the
maintainer as a blocking question and this implements the answer: RETIRED ids
throw, unknown and future ids keep falling back.

Preserved:

  - The #1529 contract, written into getProjectInstructionFile's own docblock
    as a mapping table ending "unknown / future runtimes -> AGENTS.md (safe
    cross-agent default)". That default exists so a runtime GSD has never heard
    of still gets a working instruction file.
  - ADR-1239 Phase B / #1679, which preserved GLOBAL_CONFIG_HOME_FRAGMENTS
    BYTE-FOR-BYTE when it collapsed a 14-branch chain, with golden install
    parity asserting generated hook output is unchanged across every runtime.
  - The explicit `if (!runtime) return <default>` branch. Empty string is a
    supported input, not a non-canonical id.

The distinction the code encodes: ABSENCE OF KNOWLEDGE IS NOT THE SAME AS
RECORDED RETIREMENT. Unknown means "no information, degrade safely". Retired
means "we know it is gone and we know what replaced it" — and silently
substituting a different product for it is the defect.

ONE INACCURACY IN THE CRITERION, RECORDED RATHER THAN REPEATED

AC#1 says the accessors return "a Claude Code value". True for getRuntimeLabel,
getGlobalConfigHomeFragment, getGlobalConfigDir and getDirName — but
getProjectInstructionFile returns 'AGENTS.md', which is not a Claude value at
all. The defect it points at is real for all of them, so the fix covers all of
them, but the wording is wrong for one.

MATCHING

RETIRED_RUNTIME_DETAILS is a Map keyed by canonical retired id, and
RETIRED_RUNTIME_SPELLINGS maps every spelling to that id. Both are Maps, not
object literals: a literal indexed by a computed key resolves INHERITED
properties, so '__proto__' and 'constructor' were truthy and threw with every
field `undefined`, while isRetiredRuntimeId — which already went through a Set
— correctly answered false for the same input. Two guards disagreeing about one
id is worse than either answer. A Map has no prototype keys, so that hazard is
structural rather than patched. The predicate and the assertion now share one
normaliser and one table and cannot diverge.

Candidates are normalised NFKC + lowercase + strip non-alphanumerics. Folding
the separators makes 'gemini-cli', 'gemini_cli', 'gemini.cli' and 'geminicli'
one key instead of four near-misses found one at a time, and NFKC folds the
full-width 'gemini' a CJK keyboard produces. It stays MEMBERSHIP matching,
never prefix or substring: 'gemini-2.5-pro' folds to 'gemini25pro' and
'gemini-3.1-pro-preview' to 'gemini31propreview', neither a member, so Google's
live model ids — part of Antigravity's real on-disk contract — are untouched.

Homoglyph folding is deliberately not attempted, and a Cyrillic 'і' would slip
through. These values arrive from argv and env, trusted inputs here, and a
mapping broad enough to catch deliberate homoglyphs would start catching
legitimate ids. Stated rather than left for the next reader to discover.

This over-broad-match trap is the recurring shape of the whole epic: an
exclusion or match written wider than its subject. Four occurrences, each
cited: #4716's `gemini-[0-9]` sweep exclusion hid a stale review.models.gemini
row whose value was "gemini-2.5-pro" on the same line; #4753's first
model-display escape was a blanket /^ \d/ that laundered "Gemini 2.5 CLI as a
supported runtime."; its dialect rule then used a +/-24-character window that
let one legitimate reference license a live claim 21 characters away; and its
model rule treated the ABSENCE of a runtime word as a grant, passing five
unqualified live-runtime claims. Earlier drafts of this message and its
artifacts said "five" in one place and "three" in another with nothing cited;
it is four, listed here, and the artifacts now agree.

THE THROW

RetiredRuntimeError carries `code: 'GSD_RETIRED_RUNTIME'` so a caller can
handle this case without string-matching a message that may be reworded, and
the message names the id, the successor and the retiring issue.
assertNotRetiredRuntime runs as the FIRST statement of each accessor, including
before getGlobalConfigDir's explicitDir branch, so an explicit directory cannot
mask a runtime that is gone.

`gsd-tools query project-instruction-file --runtime gemini` answered the new
throw with a raw stack trace — a user-facing regression this change introduced.
Its sibling routeSkillsRoot already emitted a clean single-line error for an
unknown runtime, so that route now maps GSD_RETIRED_RUNTIME through the same
`error()` helper, and a test asserts the contract directly: non-zero exit,
stderr naming Antigravity and #1928, and no stack frame. It was the only
unwrapped call site in that CLI; I checked the rest rather than assuming.

getRuntimeNewProjectCommand is deliberately NOT guarded: its value does not
vary by runtime in a way that makes a retired id a wrong answer, so throwing
would cost callers a crash without correcting anything. Verified by observing
it return the same value across claude, codex, opencode, kimi, antigravity,
copilot and an unknown id.

RECONCILING THE TESTS THAT PINNED THE DEFECT

The full remote matrix went red with 14 failures, and every one was a
pre-existing test asserting the fallback this criterion calls a defect. One had
already been caught locally by review; the matrix found the other thirteen
across four files. They were reconciled by intent, not blanket-inverted:

  - Tests whose SUBJECT is the retired runtime — "gemini falls back on label /
    config-fragment / new-project surfaces", "gemini no longer maps to
    GEMINI.md (defaults to AGENTS.md)", "gemini is no longer a known runtime —
    falls back to AGENTS.md" — had pinned the defect, titles and all. Their
    assertions are INVERTED rather than deleted, so the history of what the
    behaviour used to be stays attached to the test that pinned it.
  - Tests whose SUBJECT is "an unregistered id falls back generically", with
    gemini merely the SAMPLE, still assert a TRUE property that this change
    deliberately preserved. Those keep their assertion and switch the sample to
    a genuinely unknown id, with a retired-id refusal pinned alongside so both
    halves of the distinction sit together.
  - The project-instruction-file parity loop dropped gemini from its
    parametrised runtimes — both sides now refuse, so there is no value to
    agree on — and gained a dedicated refusal-parity test.

A FIFTEENTH was then found by executing the touched suites locally, in process,
one file at a time — `tests/runtime-name-policy.test.cjs:135` asserted
`getProjectInstructionFile('gemini-cli') === 'AGENTS.md'`, and its own comment
read "gemini-cli was an alias for gemini", which is exactly why that spelling
is now a retired one rather than a merely-unrecognised one. Inverted like the
rest.

Two remote runs on this change were avoidable: the first by reconciling the
tests that pinned the old behaviour before shipping, the second by executing
the touched suites locally first. The matrix is the authority; it is not the
discovery mechanism. Local per-file execution is bounded and cheap and is not
the banned `node --test` fan-out.

All five touched suites now pass in process: runtime-name-policy 47/47,
gemini-runtime-removed 32/32, project-instruction-file-parity 12/12,
runtime-homes-legacy-ids-drift-guard 2/2, install 452/452.

COVERAGE

Failing-first, one per accessor as the criterion demands, each proven RED
against 5d4c98cde7 before the fix existed — the table at the top of this
message IS that baseline, and the exports the tests import did not exist yet
either.

Asserting only the throw would pass if every id threw, which would break every
install, so each property is paired with its opposite: every canonical id still
resolves on all five accessors with byte-identical values; '' keeps its
documented branch; a genuinely unknown id keeps 'Claude Code' / 'AGENTS.md' /
'.claude' / ~/.claude. That last one is the load-bearing negative — it is the
decision the maintainer chose to preserve, so a later patch that "tightens" the
guard to reject all non-canonical ids turns it red with the reason attached.

Boundary coverage maps limit-1/limit/limit+1 onto set membership: 'gemin',
'geminix', 'gemini-2.5-pro' and 'gemini-3.1-pro-preview' must NOT throw, the
retired id and its folded spellings must. '__proto__', 'constructor' and
'  CONSTRUCTOR  ' are pinned as must-not-throw, and predicate/assertion
agreement is asserted directly. Several assert.throws calls initially passed a
string as the second argument, which node treats as the MESSAGE rather than a
matcher, so they asserted nothing about the error; they now use a real
predicate checking the code.

The tests live in the owning modules' suites rather than a new issue-named
file: lint-regression-test-names rejects new bug-NNNN/fix-NNNN/issue-NNNN test
files outright and directs the regression to the owning module's suite.
scripts/lib/macos-conformance-tier.generated.cjs regenerated through its own
--write path, since the tracked test-file count moved.

Fixes #4709

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#4709): backfill changeset PR number (#4756)

---------

Co-authored-by: sim <sim@local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 22:22:16 -04:00
Tom Boucher
9faacc0c15 test(#3148): bound the long tail and delete the unbounded-spawn allowlist (#3192)
* test(#3148): bound the long tail and delete the allowlist

Migrates the final 170 unbounded sync spawn sites across 49 files, then
removes the allowlist entirely. local/no-unbounded-spawn now runs with no
exemption surface across tests/**: there is no file to add a name to.

drift-detection's throw-native git() helper routes to gitOrThrow -- bare
runGit would have taken 16 call sites quiet on failure. commands.test.cjs
has two independently-scoped runGsdTools/runCli helpers, one already bounded
and one not; they are kept distinct rather than unified, the same trap as the
two same-named git() helpers in Wave 1.

runNpm's bound was erasable. Its options spread callerOptions after the
defaults, so an explicit timeout:undefined silently dropped the 180000ms
bound -- the rule flagged it and was right; it was not a false positive. Fixed
by destructuring with a default, with a test that fails when the default is
removed.

Two sites stay on a raw spawn with an explicit timeout because the seam
cannot express them: one needs shell:true for npm.cmd on Windows, one
redirects stdout to a real fd. Both are the rule's own documented second
option, not an escape from it.

Closure verified rather than asserted: the derivation scan reports 0 unbounded
spawn helpers and 0 unbounded direct git call sites, and a temporary file
carrying an unbounded spawn still errors with the allowlist gone.

Closes #3064.

* test(#3148): close a hole in the guard's own eslint-disable ban

The ban listed only the top level of tests/, so it was blind to 37 .cjs
files under tests/helpers, qa, observability, fixtures and dispatch. With the
allowlist deleted this test is the sole remaining way to detect someone
silencing the rule inline, so the gap was load-bearing: a nested file could
carry an unbounded spawn plus an eslint-disable and pass everything.

Proven before and after. A probe planted under tests/helpers with both was
invisible to the guard and clean under eslint; after making the listing
recursive the guard fails on it. The scanned set goes from 771 files to 808.

Pre-existing since the guard shipped, but this wave is what promoted it to
sole defense, so it is fixed here rather than filed.

Also converts the last hand-rolled throw check to throwIfFailed and the last
re-derived legacy shape to compose toLegacyResult, which makes the epic's
none-remain claim true rather than nearly true. toLegacyResult itself is not
widened -- eight callers depend on its shape and one consumer does not
justify changing a shared contract.

* fix(#3148): correct seam incoherence at the bound and a slow review-lane error path

Two real failures from the remote runner, both fixed at the cause.

The seam could return outcome TIMED_OUT together with exitCode 0. At the
exact bound spawnSync reports ETIMEDOUT while the child has already exited
with a real status, and toSeamResult classified on the error code while
passing status straight through -- an incoherent pair its own boundary test
was written to catch, and did. A status that is not null is direct evidence
the child exited on its own, so it now decides the outcome before the
error-code branches run. process-seam.cjs was deliberately untouched by every
earlier wave; this is a defect in the module itself, kept surgical, with a
unit test that fails against the old logic.

review-lane with an unknown subcommand fell through to its usage error only
after loading the capability registry and building a per-lane plan, which
spawns one child process per lane -- up to twelve. The error path took
~1288ms instead of ~119ms, and under bench load it outran a caller's spawn
timeout and was killed before writing anything, which is the empty stdout and
stderr CI saw. It now fails fast before any of that work begins.

This is the epic's first production change. It is user-facing, so it carries
a changeset rather than a no-changelog label.

* test(#3148): replace a real-race timeout test with a deterministic one

E9 raced git rev-parse against a 1ms bound and assumed git always lost. On a
warm container git finishes first, spawnSync returns status 0 with no error
at all, the seam correctly classifies EXITED, and gitOrThrow correctly does
not throw -- so the test failed on both lanes. A probe confirms a genuine
timeout always carries status null, so this was never the seam misbehaving.

Raising the bound would only lengthen the odds, which is the same defect with
better luck. The test now drives gitOrThrow against a stubbed runGit that
returns a synthetic TIMED_OUT result, so it asserts exactly what it always
meant to -- that a timeout propagates as a throw -- with no timing
dependence. Five consecutive runs are identical where the old one varied.

I wrote this test in Wave 0; it is a real-race test by construction and
CLAUDE.md says to replace those rather than re-run them.

* chore(#3148): backfill changeset PR number 3192

---------

Co-authored-by: sim <sim@local>
2026-08-07 21:03:50 -04:00
sim
7dd9e59f6b test(#3090): stop exempting violations under categories that do not fit
An allow-test-rule annotation citing a category that does not apply is worse
than no annotation, because it reads as reviewed. Eight were confirmed by
reading the assertions each one covered, and auditing the rest found five more
plus one refutation — a converter test whose wording described the wrong
mechanism while the covered assertion genuinely was deployed-text.

The instructive one used the CANONICAL string for the same mistake: STATE.md
command output labelled as a deployed artifact. A canonical string is not
evidence the category fits, which is why normalising strings alone would have
laundered the problem rather than fixed it. Every mapping the audit had inferred
rather than code-verified was spot-checked before rewriting, and the ones that
turned out not to fit were re-annotated rather than relabelled.

Fourteen STATE.md assertions had a typed extractor available all along and now
use it; their annotations came out because nothing needs exempting. Eight
assertions genuinely need a production change first — CLI stdout and stderr with
no structured mode — and are tagged pending-migration-to-typed-ir citing #3090,
which is what that category is for. It had zero real uses before this, while one
file carried a real citation to migration issue #2974 under a non-canonical tag.

Six annotations covered assertions that do no text matching at all. An exemption
for a violation that does not exist is noise that makes the real ones harder to
audit; those are removed.

atomic-write-coverage gains the annotation it always warranted — its own
docstring describes a structural-regression-guard while the file carried none.

Fifty-nine non-canonical strings across roughly thirty files are normalised, and
the allow-test-rule allowlist is regenerated to match. 472 annotations became
463: every one now uses a canonical category, and the two remaining
non-canonical strings are ESLint RuleTester fixtures, not annotations.

Refs #3057

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 17:20:56 -04:00
Tom Boucher
bf9bd1f4e0 fix(#1529): emit runtime-native instruction file from new-project 2026-06-22 09:32:29 -04:00