A PR whose diff is entirely under docs/ runs zero tests, so a guard whose INPUT
is shipped prose cannot protect the PR lane of the diffs it exists to check. Its
only firing opportunity is after merge, on the shared branch -- which is how next
went red on dacae9273 while the PR that caused it (#3746) was green on every
check.
The docs-lint job in .github/workflows/docs-required.yml -- an ALREADY-REQUIRED
context -- now selects and runs the docs guards that read the specific docs files
the PR changed.
scripts/docs-guard-registry.cjs test file -> the docs paths it reads (63)
scripts/select-docs-guards.cjs pure (changedPaths, registry) -> test files
scripts/lint-docs-guard-registration.cjs drift guard, wired into lint:ci
scripts/ci-test-scope.cjs is NOT touched -- `git diff origin/next --` on it is
empty -- so #764's saving stands and its 21 pinning tests are untouched.
Selection: exact path; trailing-slash directory prefix (boundary-checked --
docs/adrenaline.md does NOT match docs/adr/, which a naive startsWith gets
wrong); and '*' for the 6 entries that walk docs/ generally or read a computed
path. Unknown maps to '*' -- guessing narrow is how a guard silently stops
running. Measured: a typo fix selects 6 of 63; docs/AGENTS.md selects 12;
docs/COMMANDS.md selects 18.
Four things this got wrong first, each found by an independent reviewer or by
probe, and each having been asserted safe in a comment:
1. The registry started as a RULE in ci-test-scope.cjs's RULES, on the theory
that classify()'s !codeChanged normalization made it inert. True for
docs-ONLY diffs; false for MIXED docs+code diffs, where codeChanged is true
and the normalization never runs:
node scripts/ci-test-scope.cjs --files "docs/a.md src/semver.cts"
with the RULE: 25 targeted_tests
origin/next: 3 targeted_tests
Category error: RULES is the scoped lane's input; a docs-guard registry is a
lane manifest for a consumer that never calls classify(). Extracted; pinned
by value.
2. The second attempt was a dedicated workflow with paths: [docs/**]. Such a
workflow never reports on a non-docs PR, so it can never be a required
context without hanging every non-docs PR -- and a non-required check does not
block a merge, so the guard would have been advisory and #3753 unfixed.
docs-required.yml already has no paths: filter, already supplies the required
docs-lint context, already computes docs_changed, and already ran one docs
guard gated on it. Generalizing that step needs no ruleset edit at all.
3. The registry and the drift lint were built from ONE path-segment heuristic, so
both were blind identically -- and blind at the guard that motivated the issue.
The reader-call regex required a character BEFORE its keyword, so a callee
named exactly read( / load( / parse( / doc( / file( / content( could never
match; and only an INLINE path.join(ROOT,'docs','X.md') argument was caught,
missing the two-step-via-variable form -- the MAJORITY spelling -- plus
template literals and concatenation. Detector 1 fired on 14 of ~450 files, so
35 genuine guards sat unregistered while the lint reported 0 violations,
including cursor-reviewer (reads docs/COMMANDS.md, asserts
.includes('--cursor')) and inventory-headings-countfree. The "accepted blind
spot" this shipped with was the common case, not a fringe.
4. With detection fixed the true population is 115 files: 63 genuine guards, 52
incidental. Running all 63 in a REQUIRED check on a one-line typo fix is the
cost #764 exists to avoid -- install.test.cjs is 7840 lines and reads exactly
one docs file, docs/AGENTS.md, for its frontmatter. Dropping it reproduces the
bug; running it for a typo elsewhere is waste. Hence the map.
Then a second review round found six more, all fixed here:
- fragment-single-edit-propagation.install.test.cjs was EXEMPTED as
"overlay fixture only". False: it reads the real docs/registries/eos.json and
asserts on a registry entry name, and reads the real ADR-0001 and asserts its
H1. A docs-only PR touching either would have gone green and red next -- #3753
shipping again, from inside the fix for it. Now registered against both paths,
and all 52 remaining exemptions were re-audited one by one.
- The SUITES-collision guard compared RAW registry keys, but run-tests.cjs strips
a leading `tests/` BEFORE its suite check. So it caught 'all' and missed
'tests/all' -- the only spelling that can actually occur, since every key
carries the prefix. One typo would have run all 824 test files inside the
required job. Now normalized the same way run-tests.cjs normalizes.
- The lint failed OPEN on an unreadable tests dir or candidate file: 0 violations,
ok:true. A guard that cannot read its input must never report success.
- The exemption ratchet gated identity only, so a baselined file that later
STARTED asserting on shipped docs stayed exempt silently -- 52 permanently blind
files. The baseline now fingerprints the docs paths each exempted file
references and fails when that set changes, naming what changed.
- The exemption marker was still honored inside a multi-line template literal in
the header window. The scanner now tracks template-literal and block-comment
state.
- `git diff --name-only | grep '^docs/'` silently dropped C-quoted non-ASCII docs
paths, making docs_changed=false a green zero-guard check. Both call sites now
pass -c core.quotepath=false.
- The run step was gated on hashFiles(), which a force-committed
.docs-guard-tests.txt would satisfy. The step now rm -f's both scratch files
first and gates on an output it sets itself.
Three empty states, deliberately distinct, because conflating them rebuilds
#3753: an empty or malformed registry HARD-FAILS; docs changed with no guard
covering them logs and skips; no docs change is already gated. The middle state
must never be expressed as an empty --files-from, which prints `no tests in suite
"all"` and exits 0 -- a green check that guarded nothing. With the current
registry that state is unreachable, because the six '*' entries always match;
the branch is kept as defensive handling for a future registry and says so.
timeout-minutes: 15 bounds the required job against a hanging fork-supplied test;
it had none. npm ci was added because the job never installed dependencies -- the
previous single-file step got away without it, the registry does not.
docs/contributing/docs-guard-registration.md documents the rule, following its
sibling cross-platform-portability-rules.md, and CONTRIBUTING.md's CI Test
Quality Checks table links to it. It is also load-bearing: without a docs/ file
in the diff this PR would not have triggered its own lane, shipping an
unexercised change to a required check.
One unrelated fix, included because this PR surfaced it and CLAUDE.md forbids
deferring a defect found while working. On this branch's first CI run,
`full test (windows-latest, 24, shard 3/3)` was CANCELLED at exactly 30 minutes;
tests were still passing 0.8s before the cancel, so it is a wall-clock timeout,
not a hang, and a cancelled job reddens `Required tests`.
The cause is not this PR's test file, which costs ~60ms. Shard composition is
unstable: adding ONE file to the unit suite reshuffled 115 of 268 files between
shards, and shard 3 drew a heavier mix. Underneath that is a real pre-existing
defect. tests/ci-test-job-timeout-budget.test.cjs requires every lane's budget to
be >= 1.5x its MEASURED cost -- "a lane that got slower must be re-budgeted, not
excused" -- and its test-full entry recorded 19m from a windows-22 shard. That is
stale. Measured on `next` with none of this PR's changes present: 26m18s (run
32614439702, windows-latest/24 shard 3/3), 23m36s and 23m17s on shard 2/3. So the
lane costs ~26m and the 30-minute cap carried 1.14x headroom, not 1.5x. The gate
had been out of compliance with its own rule; this PR was merely the file
addition that reshuffled shard 3 past the cliff.
Fixed as that file prescribes: measuredMinutes 19 -> 27 with fresh evidence, and
test-full timeout-minutes 30 -> 45. The rule's minimum for 27m is 41; 45 is
deliberately above it because the reshuffle means per-shard worst case moves run
to run, and a budget pinned to the exact minimum would be re-breached by the next
test file anyone adds. Only that one job's timeout changed; test.yml's scope,
matrix and steps are untouched, so #764's saving is unaffected.
Raising that cap let the Windows shard finish (28m45s, inside 45) and uncovered
a real failure the 30-minute cancel had been masking:
`new quick-task branch branches off origin/main (#2916)` died with
`outcome=timed_out exitCode=null`, SIGTERM, at the 15000ms bound.
tests/quick-branching.test.cjs:149 `runStep` runs a `#!/usr/bin/env bash` script
executing MULTIPLE git commands, but was bound to GIT_TIMEOUT_MS (15000) -- the
norm for a SINGLE git plumbing call. tests/helpers/timeouts.cjs already documents
this exact failure and exists to fix it: HOOK_FANOUT_TIMEOUT_MS was created after
PR #3285 recorded "outcome=timed_out exitCode=null at exactly the 15000ms probe
bound while every other lane passed the same commit", and calls that "a bound
sized for the wrong class, not a slow machine". Our failure is that case
verbatim, so both sites move to the class norm rather than to a bigger number.
The same class also failed on `next` itself 21 hours earlier -- run 32608945654,
windows-latest/24 shard 1/3, `plan touching only src/ in a submodule project
keeps worktree isolation ENABLED` -- where tests/worktree-safety.test.cjs:5845
`runGate` fans out to `git config --file .gitmodules` under a hardcoded 30000.
Fixed too, since it is a defect in the tree regardless of which branch surfaced
it.
A survey of the whole tests/ tree found the same class-mismatch at further
bash fan-out sites bound under 60000ms, and the maintainer approved sweeping
them rather than leaving them latent to surface the same way one at a time. 16
fan-out sites across 16 files now use the class norm.
The sweep is class-correctness, not raising numbers until things pass. Sites
were moved ONLY where the bash body demonstrably spawns something (git, node,
npm, a CLI); self-contained shell snippets were left where they are, and are
listed as deliberately unchanged: pure if/printf bodies (copilot-install), pure
array/case builtins (code-review-pipeline-regression:638), a documented
pure-shell gsd_run stub (host-integration), single-process hook calls
(workflow-guard:222/271/302), and a deliberately tight 5000ms fast-check hook
(gsd-write-guard.property). Nothing was lowered. process-seam.test.cjs:513
(literal 300) is untouched on purpose -- it tests timeout BEHAVIOR, so raising
it would destroy what it asserts.
Shared file-level constants were the trap here, and were handled per file rather
than by redefinition: GIT_TIMEOUT_MS has ~15 users in git-base-branch and only 1
is a fan-out; WORKTREE_TIMEOUT_MS has 16 users in worktree.test.cjs and 3 are;
PROBE_TIMEOUT_MS has several in three more files. In each the CALL SITE was
changed and the constant left alone, so no single-plumbing-call site silently
inherited a 60s bound. The one exception is hooks-opt-in.test.cjs, where
HOOK_TIMEOUT_MS has exactly one consumer -- spawnHook, the fan-out itself -- so
redefining it is identical in effect and reads better.
Only two of these sites have actually been observed failing. The rest cite that
shared class and those two run ids rather than inventing evidence of their own.
Co-authored-by: sim <sim@local>
Phase 5 of #3464, following #3465, #3466, #3502 and #3508. Those cut the
ceiling 305 -> 278, made the rule accurate, and ended file-wide amnesty. This
one fixes the number itself.
scripts/lint-allow-test-rule-refs.cjs counted FILES CONTAINING MARKER TEXT.
Only 5 of those files carry a marker that actually suppresses a violation the
rule detects, across 10 sites. The ratcheted number was ~98% noise, which is
exactly why bumping it was frictionless: the metric was never coupled to the
thing it claimed to govern. That is the whole complaint this epic opened with,
stated precisely.
Verified directly rather than assumed: only eslint-rules/no-source-grep.cjs
functionally honors the marker. Four other rule files mention allow-test-rule
in prose only, and no-raw-rmsync-in-tests.cjs:24 explicitly states it does not
apply. So the large count was not legitimately large because several rules
share the annotation.
Now two numbers, only the first ratcheted:
EFFECTIVE EXEMPTIONS -- markers that actually suppress a detected violation.
10 sites across 5 files. Tightly ratcheted in both directions, as before:
over the ceiling fails, and slack beyond grace fails.
UNVERIFIED MARKERS -- marker-bearing files with no detectable violation. 273
files. Reported and given a loose ceiling so the pool cannot silently
balloon, but deliberately NOT tightly ratcheted, because shrinking it is a
rule-coverage problem and not a delete-the-markers problem.
A file with at least one effective site counts as effective and is not also
counted as unverified; the two numbers never double-count.
Reporting ONLY the effective count was considered and rejected. It would say
five files and look excellent while being falsely reassuring, because "no
detectable violation" is not "no violation". This phase's own measurement found
two genuine source-greps that are unsuppressed AND undetected --
tests/security-prompt-injection.security.test.cjs:852 and
tests/check-update-config-dir.test.cjs:91 -- each reading a real shipped file
and text-searching it, invisible only because the path is bound to a separate
const the rule never resolves back to its literal. Markers guarding that class
count as zero-effective and would look vestigial. Trading a number that is too
big and meaningless for one that is too small and falsely reassuring is not
progress, so the script prints the known-limit caveat alongside the numbers and
the two undetected violations are filed separately rather than lost.
Single source of truth is structural, not a matter of discipline. The script
does not re-implement detection or the site-scoped adjacency predicate -- that
is the generative-fix-divergence class this repo has shipped before. The rule
now exports MAX_MARKER_LOOKAHEAD_LINES, MARKER_COMMENT_RE,
collectMarkerAndCommentLines and isSuppressedAt (extracted verbatim, no logic
change), plus a default-off neutralizeSuppression option so the counter can
enumerate every site through the real rule via ESLint's Linter API and then
classify each with the rule's own predicate, replicating reportUnlessSuppressed's
search-line-OR-read-line check exactly. Default rule behavior is byte-identical:
tests/eslint-rules.test.cjs passes 168/168 unchanged. A parity test asserts the
script's suppressed/not verdict equals the rule's own report/no-report outcome
for every site in a fixture corpus.
A real silent-failure bug surfaced and was fixed while building this: ESLint's
flat-config Linter reports "No matching configuration" and returns ZERO messages
for any filename resolving outside its cwd. That would have quietly
misclassified every sandboxed test fixture as having no violations -- a test
suite that passes while asserting nothing. Fixed by anchoring the Linter to the
tests dir, with a defensive throw if it ever recurs.
Two earlier claims of mine are corrected by this phase's measurement. Widening
the source-dir allowlist to include hooks/ -- the "fifth blind spot" recorded in
#3508 -- rescues ZERO sites; it is real in principle and has no practical
effect, because the hooks reads that exist are missed for other reasons
(.sh extension, identifier-indirection, dynamic filenames). And the #3508
correction that attributed those reads to the hooks/ gap rather than to variable
indirection was itself incomplete: both are independently sufficient, so fixing
either alone changes nothing. I accepted the reviewer's causal claim as
uncritically as I had made my own.
The unverified count is 273, not the ~289 in the phase design doc. That is
legitimate drift -- the baseline was measured at fba7c9032 and other merged work
has since removed markers. Left as measured rather than adjusted to match the
document.
Adversarial review found a BLOCKER in the first revision and it is fixed here.
The counter walked only tests/**/*.test.cjs, but the rule is registered on
tests/**/*.cjs -- every .cjs, not just test files -- plus scripts/**,
eslint-rules/**, bin/lib/**, pi/**, examples/**, gsd-core/bin/** and three
plugin globs. 35 non-.test.cjs files under tests/ were never walked, and
tests/helpers/live-command-registry.cjs:1 carries a real marker that appeared in
NEITHER reported number. A counter that undercounts is worse than the
meaningless one it replaces, because it will be trusted.
The scan set is now derived programmatically: the script dynamically imports
eslint.config.mjs and extracts the `files` globs from every config block that
enables local/no-source-grep. There is no hardcoded list to drift, which is the
same divergence class the parity test already guards.
Two further defects surfaced while fixing it. The silent-clean catch around
linter.verify() was worse than reported -- ESLint signals a parse error by
returning a message with fatal:true rather than throwing, so the original catch
would not even have fired for the common case. Both paths now throw with the
file path. That silent swallow was actively hiding a broken fixture in this
suite's own tests: 'no marker here\n' is not valid JS and the case only
"passed" because the parse failure was discarded. Fixed.
And once the scan widened, the raw-text marker scanner started matching this
tooling's own doc comments and RuleTester fixture strings, so marker extraction
moved to AST comment nodes. That corrected three long-standing FALSE POSITIVES:
allowlist entries for tests/eslint-rules.test.cjs that were never real markers,
only fixture payload. Allowlist 137 -> 135: three false positives pruned, one
real entry added for live-command-registry.
The headline numbers are coincidentally unchanged (10/10 effective across 5
files, 273/280 unverified) but the composition is corrected: one real file
gained, one phantom dropped. Verified by direct diff rather than inferred from
the totals matching.
The first remote run of this branch came back RED with 23 failures, all one
cause, and it is fixed here. Classification drove ESLint's flat-config Linter,
which resolves configuration relative to a cwd and refuses to lint any file
outside it. The test harness writes fixtures into an OS temp dir, so every
sandbox row hit "No matching configuration found" and tripped the defensive
throw. It surfaced only on the container because the repo lives at /work there
and the fixtures at /tmp, making the mismatch unmissable; a local run had
reported the suite green, which it was not.
Fixed by not depending on config *resolution* at all: the script now builds an
eslintrc-format Linter and registers the rule directly with defineRule, since it
already knows exactly which rule and options it wants. That removes the
cwd-ancestor constraint entirely and makes repo files and out-of-tree fixtures
classify identically. Verified out-of-tree explicitly, not just in-repo, because
the local temp path shape is what hid the bug the first time. Parse failures
still throw loudly -- that behavior is required and tested. Classifications are
unchanged for real repo files (10/10 effective across 5, 273/280 unverified,
0 live), which is the check that the config swap did not quietly alter results.
The second remote run cut the failures from 23 to 2, and the survivors were a
genuinely different and more interesting defect:
tests/packaging-shipped-scripts-require-only-shipped.test.cjs caught that
scripts/ SHIPS in the published package while eslint-rules/ does not, so
importing the rule for single-source-of-truth would MODULE_NOT_FOUND in a real
consumer's install. That test statically extracts require() calls, so hiding the
import inside a function would have dodged the check without fixing the problem.
Resolved along the grain of existing convention rather than by weakening
anything: package.json already excludes several repo-internal lint gates from
the shipped set via `!scripts/...`, including
`!scripts/lint-no-adhoc-regex-escape.cjs`, which is the same situation. This
gate is CI-only and has no meaning in a consumer install, so it joins them.
Verified with `npm pack --dry-run` that the .cjs is genuinely absent from the
tarball (its inert JSON config files remain, and carry no requires).
The third remote run failed on shard 3/3 across all three OSes with exit code
NULL and empty output -- the spawned gate was killed by a timeout, not failing
an assertion. Cause: replacing a raw text scan with a full ESLint Linter pass
over every file in every registered glob took the gate from 0.57s to ~7-12s,
and GitHub's runners are slower than the bench that had just passed it green.
Fixed by narrowing the work rather than raising the timeout to hide it. Both
numbers the gate computes are properties of files that CONTAIN a marker, so
only those (~294) need linting; the repo-wide byte walk that finds them stays,
since that was the undercount fix. Live violations in files carrying NO marker
are already enforced by npm run lint over exactly these globs, so re-detecting
them here was redundant. 2.87s now, from ~7s measured locally.
That narrowing changes what one reported number means, so the wording changed
with it: "live violations in marker-bearing files: 0 (unmarked files are
enforced separately by npm run lint)". A number that quietly covers less than
it reads is the failure this whole epic is about, so it is stated rather than
left implicit, and the test row that asserted the old broader behavior was
split -- an unmarked live violation now passes this gate (and is caught by
eslint), while a live violation in a marker-bearing file still fails it.
The repo-baseline test's timeout was also raised to 30s with a comment, since a
gate that legitimately takes ~3s must not sit at a timeout close to its own
runtime. Every other row keeps the shorter sandbox-scoped timeout.
Closes#3520
Co-authored-by: sim <sim@local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Extends lint-allow-test-rule-refs.cjs with a second, independent check
alongside the existing uncited-citation identity ratchet: the total
number of distinct test files carrying any allow-test-rule marker
(cited or not) is now checked against a tight ceiling via the
previously-unwired assertTightCeiling primitive (allowlist-ratchet.cjs,
0 prior callers). A cited exemption is legitimate under ADR-456 but
nothing stopped the raw total from growing forever - this closes that
gap without duplicating the file walk (both checks consume one shared
walkTestFiles pass).
Ceiling introduced at the exact measured high-water mark (314 files,
grace 3) rather than a padded estimate, per "budgets may only
decrease."
Also lands the two F17 pieces (absorbed from the now-closed #1885)
that had no precondition:
- --max-warnings 0 added to lint/lint:ci
- local/no-source-grep promoted warn->error in the scripts/bin/
eslint-rules glob block (already error in the tests/ glob)
Both promotions were pre-verified against a zero-warning tree (fresh
non-cached eslint run) before flipping, per the maintainer's clean-
tree-first decision.
Not included: local/no-elapsed-assertion promotion, which stays warn
pending #3314 (H2) - 10 of 19 clock-touching src modules have no
sanctioned time-control mechanism until ADR-456 is amended there.
H1 of epic #3053, absorbing #1885 F17.
Co-authored-by: sim <sim@local>
Tom Boucher