A PR whose diff is entirely under docs/ runs zero tests, so a guard whose INPUT
is shipped prose cannot protect the PR lane of the diffs it exists to check. Its
only firing opportunity is after merge, on the shared branch -- which is how next
went red on dacae9273 while the PR that caused it (#3746) was green on every
check.
The docs-lint job in .github/workflows/docs-required.yml -- an ALREADY-REQUIRED
context -- now selects and runs the docs guards that read the specific docs files
the PR changed.
scripts/docs-guard-registry.cjs test file -> the docs paths it reads (63)
scripts/select-docs-guards.cjs pure (changedPaths, registry) -> test files
scripts/lint-docs-guard-registration.cjs drift guard, wired into lint:ci
scripts/ci-test-scope.cjs is NOT touched -- `git diff origin/next --` on it is
empty -- so #764's saving stands and its 21 pinning tests are untouched.
Selection: exact path; trailing-slash directory prefix (boundary-checked --
docs/adrenaline.md does NOT match docs/adr/, which a naive startsWith gets
wrong); and '*' for the 6 entries that walk docs/ generally or read a computed
path. Unknown maps to '*' -- guessing narrow is how a guard silently stops
running. Measured: a typo fix selects 6 of 63; docs/AGENTS.md selects 12;
docs/COMMANDS.md selects 18.
Four things this got wrong first, each found by an independent reviewer or by
probe, and each having been asserted safe in a comment:
1. The registry started as a RULE in ci-test-scope.cjs's RULES, on the theory
that classify()'s !codeChanged normalization made it inert. True for
docs-ONLY diffs; false for MIXED docs+code diffs, where codeChanged is true
and the normalization never runs:
node scripts/ci-test-scope.cjs --files "docs/a.md src/semver.cts"
with the RULE: 25 targeted_tests
origin/next: 3 targeted_tests
Category error: RULES is the scoped lane's input; a docs-guard registry is a
lane manifest for a consumer that never calls classify(). Extracted; pinned
by value.
2. The second attempt was a dedicated workflow with paths: [docs/**]. Such a
workflow never reports on a non-docs PR, so it can never be a required
context without hanging every non-docs PR -- and a non-required check does not
block a merge, so the guard would have been advisory and #3753 unfixed.
docs-required.yml already has no paths: filter, already supplies the required
docs-lint context, already computes docs_changed, and already ran one docs
guard gated on it. Generalizing that step needs no ruleset edit at all.
3. The registry and the drift lint were built from ONE path-segment heuristic, so
both were blind identically -- and blind at the guard that motivated the issue.
The reader-call regex required a character BEFORE its keyword, so a callee
named exactly read( / load( / parse( / doc( / file( / content( could never
match; and only an INLINE path.join(ROOT,'docs','X.md') argument was caught,
missing the two-step-via-variable form -- the MAJORITY spelling -- plus
template literals and concatenation. Detector 1 fired on 14 of ~450 files, so
35 genuine guards sat unregistered while the lint reported 0 violations,
including cursor-reviewer (reads docs/COMMANDS.md, asserts
.includes('--cursor')) and inventory-headings-countfree. The "accepted blind
spot" this shipped with was the common case, not a fringe.
4. With detection fixed the true population is 115 files: 63 genuine guards, 52
incidental. Running all 63 in a REQUIRED check on a one-line typo fix is the
cost #764 exists to avoid -- install.test.cjs is 7840 lines and reads exactly
one docs file, docs/AGENTS.md, for its frontmatter. Dropping it reproduces the
bug; running it for a typo elsewhere is waste. Hence the map.
Then a second review round found six more, all fixed here:
- fragment-single-edit-propagation.install.test.cjs was EXEMPTED as
"overlay fixture only". False: it reads the real docs/registries/eos.json and
asserts on a registry entry name, and reads the real ADR-0001 and asserts its
H1. A docs-only PR touching either would have gone green and red next -- #3753
shipping again, from inside the fix for it. Now registered against both paths,
and all 52 remaining exemptions were re-audited one by one.
- The SUITES-collision guard compared RAW registry keys, but run-tests.cjs strips
a leading `tests/` BEFORE its suite check. So it caught 'all' and missed
'tests/all' -- the only spelling that can actually occur, since every key
carries the prefix. One typo would have run all 824 test files inside the
required job. Now normalized the same way run-tests.cjs normalizes.
- The lint failed OPEN on an unreadable tests dir or candidate file: 0 violations,
ok:true. A guard that cannot read its input must never report success.
- The exemption ratchet gated identity only, so a baselined file that later
STARTED asserting on shipped docs stayed exempt silently -- 52 permanently blind
files. The baseline now fingerprints the docs paths each exempted file
references and fails when that set changes, naming what changed.
- The exemption marker was still honored inside a multi-line template literal in
the header window. The scanner now tracks template-literal and block-comment
state.
- `git diff --name-only | grep '^docs/'` silently dropped C-quoted non-ASCII docs
paths, making docs_changed=false a green zero-guard check. Both call sites now
pass -c core.quotepath=false.
- The run step was gated on hashFiles(), which a force-committed
.docs-guard-tests.txt would satisfy. The step now rm -f's both scratch files
first and gates on an output it sets itself.
Three empty states, deliberately distinct, because conflating them rebuilds
#3753: an empty or malformed registry HARD-FAILS; docs changed with no guard
covering them logs and skips; no docs change is already gated. The middle state
must never be expressed as an empty --files-from, which prints `no tests in suite
"all"` and exits 0 -- a green check that guarded nothing. With the current
registry that state is unreachable, because the six '*' entries always match;
the branch is kept as defensive handling for a future registry and says so.
timeout-minutes: 15 bounds the required job against a hanging fork-supplied test;
it had none. npm ci was added because the job never installed dependencies -- the
previous single-file step got away without it, the registry does not.
docs/contributing/docs-guard-registration.md documents the rule, following its
sibling cross-platform-portability-rules.md, and CONTRIBUTING.md's CI Test
Quality Checks table links to it. It is also load-bearing: without a docs/ file
in the diff this PR would not have triggered its own lane, shipping an
unexercised change to a required check.
One unrelated fix, included because this PR surfaced it and CLAUDE.md forbids
deferring a defect found while working. On this branch's first CI run,
`full test (windows-latest, 24, shard 3/3)` was CANCELLED at exactly 30 minutes;
tests were still passing 0.8s before the cancel, so it is a wall-clock timeout,
not a hang, and a cancelled job reddens `Required tests`.
The cause is not this PR's test file, which costs ~60ms. Shard composition is
unstable: adding ONE file to the unit suite reshuffled 115 of 268 files between
shards, and shard 3 drew a heavier mix. Underneath that is a real pre-existing
defect. tests/ci-test-job-timeout-budget.test.cjs requires every lane's budget to
be >= 1.5x its MEASURED cost -- "a lane that got slower must be re-budgeted, not
excused" -- and its test-full entry recorded 19m from a windows-22 shard. That is
stale. Measured on `next` with none of this PR's changes present: 26m18s (run
32614439702, windows-latest/24 shard 3/3), 23m36s and 23m17s on shard 2/3. So the
lane costs ~26m and the 30-minute cap carried 1.14x headroom, not 1.5x. The gate
had been out of compliance with its own rule; this PR was merely the file
addition that reshuffled shard 3 past the cliff.
Fixed as that file prescribes: measuredMinutes 19 -> 27 with fresh evidence, and
test-full timeout-minutes 30 -> 45. The rule's minimum for 27m is 41; 45 is
deliberately above it because the reshuffle means per-shard worst case moves run
to run, and a budget pinned to the exact minimum would be re-breached by the next
test file anyone adds. Only that one job's timeout changed; test.yml's scope,
matrix and steps are untouched, so #764's saving is unaffected.
Raising that cap let the Windows shard finish (28m45s, inside 45) and uncovered
a real failure the 30-minute cancel had been masking:
`new quick-task branch branches off origin/main (#2916)` died with
`outcome=timed_out exitCode=null`, SIGTERM, at the 15000ms bound.
tests/quick-branching.test.cjs:149 `runStep` runs a `#!/usr/bin/env bash` script
executing MULTIPLE git commands, but was bound to GIT_TIMEOUT_MS (15000) -- the
norm for a SINGLE git plumbing call. tests/helpers/timeouts.cjs already documents
this exact failure and exists to fix it: HOOK_FANOUT_TIMEOUT_MS was created after
PR #3285 recorded "outcome=timed_out exitCode=null at exactly the 15000ms probe
bound while every other lane passed the same commit", and calls that "a bound
sized for the wrong class, not a slow machine". Our failure is that case
verbatim, so both sites move to the class norm rather than to a bigger number.
The same class also failed on `next` itself 21 hours earlier -- run 32608945654,
windows-latest/24 shard 1/3, `plan touching only src/ in a submodule project
keeps worktree isolation ENABLED` -- where tests/worktree-safety.test.cjs:5845
`runGate` fans out to `git config --file .gitmodules` under a hardcoded 30000.
Fixed too, since it is a defect in the tree regardless of which branch surfaced
it.
A survey of the whole tests/ tree found the same class-mismatch at further
bash fan-out sites bound under 60000ms, and the maintainer approved sweeping
them rather than leaving them latent to surface the same way one at a time. 16
fan-out sites across 16 files now use the class norm.
The sweep is class-correctness, not raising numbers until things pass. Sites
were moved ONLY where the bash body demonstrably spawns something (git, node,
npm, a CLI); self-contained shell snippets were left where they are, and are
listed as deliberately unchanged: pure if/printf bodies (copilot-install), pure
array/case builtins (code-review-pipeline-regression:638), a documented
pure-shell gsd_run stub (host-integration), single-process hook calls
(workflow-guard:222/271/302), and a deliberately tight 5000ms fast-check hook
(gsd-write-guard.property). Nothing was lowered. process-seam.test.cjs:513
(literal 300) is untouched on purpose -- it tests timeout BEHAVIOR, so raising
it would destroy what it asserts.
Shared file-level constants were the trap here, and were handled per file rather
than by redefinition: GIT_TIMEOUT_MS has ~15 users in git-base-branch and only 1
is a fan-out; WORKTREE_TIMEOUT_MS has 16 users in worktree.test.cjs and 3 are;
PROBE_TIMEOUT_MS has several in three more files. In each the CALL SITE was
changed and the constant left alone, so no single-plumbing-call site silently
inherited a 60s bound. The one exception is hooks-opt-in.test.cjs, where
HOOK_TIMEOUT_MS has exactly one consumer -- spawnHook, the fan-out itself -- so
redefining it is identical in effect and reads better.
Only two of these sites have actually been observed failing. The rest cite that
shared class and those two run ids rather than inventing evidence of their own.
Co-authored-by: sim <sim@local>
The scoped Windows lane reached exactly 15m05s and was cancelled on four
consecutive shas of PR #3094. A job that exceeds timeout-minutes reports as
CANCELLED rather than FAILURE, which is why it first read as infrastructure
noise; the giveaway is that the duration equals the cap. The Required tests
rollup fans that job in, so it red-blocked merge while every other lane —
including all three sharded full-Windows shards — was green.
The trigger was a change to the shared test helper, which scopes the
install-heavy suites into the selected list. The lane normally runs about eight
minutes; with that list it does not fit. It was the only unsharded lane left in
this job, so it was the only one without headroom to absorb a large scoped
list.
Issue #869 hit this exact cliff on the sibling lane and named the durable answer
in its own follow-up: a timeout bump moves the cliff, sharding removes it.
#2952 then sharded the full lane. This finishes that work.
The runner already supports it — the shard partition is applied after scope
selection, so it composes with a selected file list rather than only with a
suite, and the partition is cost-weighted from the measured timings table. The
job name template already renders a shard suffix when one is present, so the
three entries name themselves. No individual matrix job is a required status
check; the rollup is, and it is name-independent, so renaming these jobs does
not touch branch protection.
timeout-minutes stays at 15. Each shard now does roughly a third of the work,
so the cap goes from binding to backstop without being raised.
The lane-shape tests were generalized rather than relaxed: the complete-shard-set
invariant now runs per sharded scope instead of only over the full lane, and
"only the full lane is sharded" became "targeted is the only unsharded lane". A
new assertion pins the shard through to the runner — without it the three shards
would each run the entire selected list, triple the cost and no speedup, and
every check would stay green.
No LANE_COSTS entry is added for the new shards. The only recorded cost for that
lane is the pre-sharding run that hit the cap, and inventing a post-sharding
number would be exactly the kind of unmeasured claim the rest of that table
avoids. The estimate and the reason are written down instead, to be replaced by
a real measurement.
Refs #3057
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* ci(#2952): budget CI job timeouts by headroom over measured cost
`origin/next` was red. The only failing check was `Required tests`, and its
sole cause was `test (ubuntu-latest, 24)` reported `cancelled` — GitHub's
conclusion for a job that exceeds its `timeout-minutes`, not a button press.
That job's `ubuntu-latest / 24` entry is the only `scope: full` matrix entry:
it runs the whole unit suite under c8 coverage, then the scripts/ coverage
floor, integration, security, install and slow, serially on one runner. On
next@5a0a9f097 it ran 15m16s against `timeout-minutes: 15` and was axed 23s
into `npm run test:slow`. Projected to a completed slow step (27s on the last
green run) the lane costs ~15m20s.
Confirmed hypothesis: the budget, not the suite. The lane had been riding the
ceiling all day — 12m03s, 11m34s, 11m50s, 14m25s, 14m51s — and crossed on
three of the last four full-lane runs (d2d2f7c08, 07603df8f, 5a0a9f097). There
is no pathological test: the unit run is cost-first bin-packed into 12 chunks,
chunk 1 is gated by run-tests-harness.test.cjs at 169s (expensive by design —
it spawns real harness subprocesses, one of which exercises the per-chunk
timeout), and the remaining chunks are 32-94s. 769s is the honest cost of 689
files under c8. Re-running could not have helped; the work exceeded the budget.
Review of the first cut surfaced the same defect one runner away: `full test
(windows-latest, 22, shard 3/3)` reached 18m59s against its own 20-minute cap
on 05b170e44 (94%) and 18m14s on 81eeb8a53 (91%). That lane has already blown
its cap twice (#1051, #1212). Fixed here rather than deferred.
The first cut also asserted `test >= test-full`, which is unsound — those two
budgets are dominated by different platforms, so their ordering carries no
meaning. Replaced with the invariant that actually generalises: every lane is
held to a headroom FACTOR over its own measured cost. `test` 15 -> 25 (1.5x of
16m), `test-full` 20 -> 30 (1.5x of 19m), `test-inert` unchanged at 15.
tests/ci-test-job-timeout-budget.test.cjs locks that rule. No unit test can
prove a lane still FITS its budget — only a real run measures that — but a
budget can no longer be lowered back beneath what its lane is known to need,
and a lane that gets slower must be re-measured rather than excused.
Separately: the earlier `failure` at 05b170e44 was an unrelated, already-fixed
CONTEXT-INDEX.json drift (07603df8f re-synced it; lint-tests is green at HEAD).
07603df8f's own run hit this same timeout, which is why it never reported green.
Refs #869, #1051, #1212
* ci(#2952): shard the full test lane instead of widening its cap
The `scope: full` lane was the only unsharded lane in this file. It ran the
entire unit suite under c8 on one runner, grew past a 15-minute cap, and
reddened `next`. Raising the cap bought room; it did not change the shape, and
the same lane would have walked back into the ceiling. Shard it, the way #1212
answered this for the Windows lane.
Balance comes from measurement, not file counts. scripts/run-tests.cjs already
partitions by measured per-file duration using LPT (#2472); the table it reads
was 10 days stale — 638 of 695 files timed, 64 missing, including the whole
context-predicates group. Regenerated from a verified matrix run: 700 files, 0
missing. On that table the 685-file unit suite splits 19.37m / 19.37m / 19.37m
— 0.0% spread — and the split is a total, disjoint cover with 0 files dropped.
Completeness, disjointness, balance and determinism of the partition itself are
already pinned against selectShard in run-tests-harness.test.cjs, including a
fast-check property, so this change does not restate them.
Sharding a COVERAGE run is the part that needs care. A per-shard percentage is
meaningless — shard 2 never executes shard 1's files, so those read 0% — and
leaving the gate on the shards would have quietly measured a third of the tree.
Each shard now renders no report and only leaves raw V8 dumps; a new
`coverage-gate` job merges all three into one coverage/tmp and runs the gate
there. c8's default temp directory is where the download lands, so the ≥70%
lines / ≥60% branches gate and the ≥55% scripts floor run unmodified against
merged data.
Both surfaces call the same npm scripts rather than inlining c8 into YAML, so
the include/exclude globs and both thresholds stay defined once in package.json.
The workflow holding its own copy is the divergence this repo has a rule
against, and the new test cross-checks package.json so an inline reintroduction
fails rather than drifts.
tests/ci-full-lane-sharding.test.cjs covers the two ways this stays GREEN while
being wrong: an incomplete shard set (declare 1/3 and 2/3, never 3/3, and a
third of the suite silently stops running) and a coverage gate that stops being
required. required-tests now depends on coverage-gate and fails on it, while
still tolerating `skipped` so docs-only PRs are not blocked.
`timeout-minutes: 25` on the lane is deliberately left alone. The budget test
requires a real measurement before a lane's declared cost changes, and the
sharded cost is not measured until this PR's own CI run.
Refs #1212, #2472
* ci(#2952): tighten the sharded lane's budget to its measured cost
The sharding commit deliberately left `timeout-minutes: 25` alone, because
tests/ci-test-job-timeout-budget.test.cjs requires a real measurement before a
lane's declared cost changes and the sharded cost did not exist yet.
It exists now. Run 30677442953: shard 1/3 7m12s, 2/3 4m32s, 3/3 3m59s, and
coverage-gate 1m20s. Shard 1 is the long pole because the unsharded aux suites
ride along on it, which is deliberate — they total ~1m35s and sharding them
would cost more than it saves.
So the lane's budget is 15 against a slowest measured shard of 8 minutes
(~1.9x), and coverage-gate joins LANE_COSTS at 2 minutes. 15 is the same number
the lane blew before sharding; the work behind it is now a third the size.
Merged coverage was checked against the pre-shard single-runner baseline rather
than assumed from a green check: 94.36 stmts / 96.3 funcs / 94.36 lines
identical, branches 84.22 vs 84.21 — one branch across two different trees,
noise rather than a regression.
---------
Co-authored-by: sim <sim@local>
Tom Boucher