* chore(#4603): retire the test-full CI job Phase 2 (#4591) added test-conformance but left test-full (the pre-existing full-suite Windows/macOS replay) running unchanged, gated on the same full_matrix flag, downgraded only from a hard gate to a non-blocking ::warning:: -- framed as "a non-gating safety net for one release cycle." No phase or issue ever retired it. Result: every full_matrix=true PR ran 10 OS-specific jobs (test-full's 6 + test-conformance's 4, purely additive) instead of the original 6 -- the epic's own goal (reduce runner-minutes) was measurably regressing, not improving, for the majority of PRs. This phase was missing from the original 4-phase epic decomposition; the epic (#4589) has been amended to add it as Phase 5 (see its comment thread), and this issue was filed as the tracked sub-issue. Deletes the test-full job from .github/workflows/test.yml entirely, along with every reference to it: required-tests' needs/FULL_TEST_RESULT warning branch, ci-timeout-report.cjs's JOB_RULES entry, ci-test-job-timeout-budget.test.cjs's LANE_COSTS/staticLanes/testFullRule entries, ci-test-scope.test.cjs's test-full-specific tests (preserving three unrelated tests that were nested in the same describe block, moved under a renamed describe rather than deleted), and docs mentions. test-conformance is now the sole gating signal for real-OS coverage. Two separate defects found and fixed while auditing every test-full reference: - tests/ci-pr-mergeability.test.cjs's GATED['test.yml'] safety-critical array (jobs that must needs: the mergeability preflight) had test-full but was missing test-conformance entirely -- Phase 2 never added it. Verified the real workflow wiring was already correct (test-conformance does have needs: [changes, preflight]); this was a test-coverage gap, not a live defect. Fixed by swapping the array entry. - docs/TESTING-SUITES.md's "## CI matrix" section was substantially stale independent of this phase (predating even #2952's coverage-gate split). Rewritten against the real, current job topology, verified directly against test.yml rather than trusted from memory. An isolated code-review pass found and fixed two minor inaccuracies in the rewritten docs table (two jobs' "Gated on" column didn't match their real if: condition exactly). An isolated security-review pass found no qualifying findings -- every compute-provisioning job already carries needs: preflight directly, unaffected by this deletion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(ci): isolate 7 more heavy test files from chunk-weight packing `next`'s own push-triggered Tests run failed: `conformance test (windows-latest, 24, shard 2/3)` chunk 3/6 was killed after 600019ms. Root cause: state.test.cjs (weight 21.35, measured) was packed alongside companions by run-tests.cjs's LPT chunk packer, the same failure mode that previously hit codex-config.test.cjs (weight 17.87) twice and got a dedicated fix (ISOLATED_HEAVY_FILES, #4497) -- but state.test.cjs was never added to that set. This is a direct, unintended consequence of epic #4589 Phase 2: the new platform-conformance-tier job packs only ~546 files per shard (vs. the ~950-file full suite the packer used to balance against), so the same absolute-weight outlier now represents a larger share of a smaller, more homogeneous pool -- the LPT packer has fewer light files to pad around it with. This was a real, foreseeable side effect of shrinking the packing pool that nobody checked for when Phase 2 shipped. A first attempt at this fix hand-picked 4 candidates by eyeballing a truncated weight list and missed 3 heavier ones -- caught by an isolated code-review pass (blocker: emitted-attribution.test.cjs at 66.2% of the Windows chunk budget, install-minimal-hooks.test.cjs at 61.1%, install.test.cjs at 47.1%, all above codex-config.test.cjs's own 44.7% -- the ratio that already proved dangerous twice). Corrected by systematically computing weight/budget for every unit-suite file and isolating everything at or above that same ratio: 7 files total, plus the pre-existing codex-config.test.cjs (8 total). Added a durable regression test (tests/run-tests-harness.test.cjs) that re-derives this exact computation from the live tests/test-timings.json on every run, so a future heavy file crossing this threshold fails the test instead of silently reintroducing this failure -- not just a one-time manual sweep. Verified end-to-end: simulated the real 3-way windows shard split of the actual conformance-tier file list with the real packing functions. Max packable-chunk weight across all 3 shards is now 27.04 / 24.10 / 23.91 (shard 2 is the exact shard that failed on next), comfortably under the 40 budget -- versus 40+ and a 600s kill before this fix. A second isolated code-review + security-review pass on the corrected diff found nothing further. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: sim <sim@local> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
@@ -403,7 +403,7 @@ Two properties are load-bearing and easy to break:
|
||||
running* — gating on it self-deadlocks the pipeline. It is read for the
|
||||
failure message only.
|
||||
|
||||
Gated: `test.yml` (`lint-tests`, `test`, `test-inert`, `test-full`,
|
||||
Gated: `test.yml` (`lint-tests`, `test`, `test-inert`,
|
||||
`test-conformance`, `coverage-gate`, `qa-loop-walk`, `required-tests`), `install-smoke.yml`,
|
||||
`mutation.yml`, `security-scan.yml`, `docs-required.yml`,
|
||||
`changeset-required.yml`, `default-flip-documentation.yml`, `branch-naming.yml`.
|
||||
@@ -457,43 +457,33 @@ sha, which invalidates any prior remote-runner verification. Rebase *last*.
|
||||
|
||||
## CI matrix
|
||||
|
||||
The `Tests` workflow runs every PR through a scoped gate generated by
|
||||
`scripts/ci-test-scope.cjs`.
|
||||
The `Tests` workflow (`.github/workflows/test.yml`) runs every PR through a scope
|
||||
computed by `scripts/ci-test-scope.cjs`'s `classify()`, which sets two flags —
|
||||
`product_changed` and `full_matrix` — from the changed-file list.
|
||||
|
||||
All lanes run on **Node 24** — the `engines.node` floor (`>=24.0.0`) and the
|
||||
only supported runtime.
|
||||
|
||||
| Lane | Scope |
|
||||
|---|---|
|
||||
| `ubuntu-latest` (scoped) | scoped tests — fast PR signal |
|
||||
| `ubuntu-latest` (full, sharded) | unit + integration + security |
|
||||
| `windows-latest` | scoped Windows/path/shell tests |
|
||||
| `macos-latest` | full parity when required |
|
||||
| Job | Lanes | Gated on | Purpose |
|
||||
|---|---|---|---|
|
||||
| `test` | `ubuntu-latest` (1 targeted + 3-shard full) + `windows-latest` (3-shard, Windows/path/shell-scoped) | `product_changed == 'true'` | The default, always-scoped PR signal — the full `unit`/`integration`/`security` suites run once, sharded, on Linux; Windows runs the Windows-sensitive subset plus every changed test file |
|
||||
| `test-inert` | `ubuntu-latest` | `code_changed == 'true' && product_changed != 'true'` | A lightweight lane for PRs that touch only administrative/policy workflow files (code changed, but nothing that needs the real matrix) |
|
||||
| `test-conformance` | `windows-latest` (3-shard) + `macos-latest` (unsharded) | `code_changed == 'true' && full_matrix == 'true'` | Runs only the **platform-conformance-tier** file list (`scripts/lib/platform-conformance-tier.generated.cjs`, epic #4589 Phase 2/#4591) on real Windows/macOS — the sole gating signal for real-OS coverage. Retired the parallel legacy full-suite matrix in #4603. |
|
||||
| `coverage-gate` | `ubuntu-latest` | `product_changed == 'true' && test.result == 'success'` | Merges every `test` shard's coverage dumps and evaluates the threshold once (sharding moved this out of the `test` job itself — #2952) |
|
||||
| `qa-loop-walk` | `ubuntu-latest` | `product_changed == 'true'` | The QA smell-ratchet scenario walk (see "The QA smell ratchet" below) |
|
||||
| `required-tests` | `ubuntu-latest` | `always()` | Aggregates every job above into the one branch-protection-required check |
|
||||
|
||||
- **Scoped tests** are selected from the changed paths, plus a small CLI/package
|
||||
smoke set. They are for confidence on the affected surface, not for counting
|
||||
tests.
|
||||
`full_matrix` fires on any changed `tests/**/*.test.cjs` file unconditionally (restored
|
||||
by #4421 after #962's narrowing let a real regression through undetected), plus a
|
||||
handful of curated `RULES` (workflow/installer/hooks/env-gate changes) — see
|
||||
`scripts/ci-test-scope.cjs`'s own `classify()` for the exact, current rule set; this
|
||||
doc intentionally does not restate it in full, to avoid drifting out of sync with it.
|
||||
|
||||
The default PR gate runs the broad `unit` (under the c8 coverage gate),
|
||||
`integration`, and `security` suites once on Ubuntu / Node 24, scoped tests on
|
||||
a second Ubuntu / Node 24 lane, and scoped tests on Windows / Node 24.
|
||||
"Scoped" means the diff-selected list from the rule table — not the full suite
|
||||
and not a fixed smoke set (the fixed smoke list is only the empty-selection
|
||||
fallback). The Windows lane's list is the Windows-sensitive subset of the
|
||||
selection, plus **every changed test file, unconditionally** (the #494
|
||||
invariant, narrowed): a modified test is exercised on the divergent OS before
|
||||
merge at per-file cost, without paying for the three full parity lanes.
|
||||
|
||||
PRs touching workflow, package, test-runner, install, release, or
|
||||
Windows-sensitive surfaces also run the full parity matrix on macOS and the
|
||||
older Windows runtime, plus `install` and `slow` on the primary Ubuntu lane.
|
||||
Everything (including the full parity matrix) runs on every push to `next`,
|
||||
which covers the residual macOS / Windows cross-product for scoped PRs.
|
||||
|
||||
Coverage runs inside the Ubuntu / Node 24 full lane (not a separate job — that
|
||||
duplicated the entire unit run) and stays single-lane because multiplying
|
||||
coverage across OS/runtime lanes adds cost without improving the threshold
|
||||
signal. Note the gate's deliberate blind spot: it measures
|
||||
Coverage is evaluated by the dedicated `coverage-gate` job (moved out of the `test`
|
||||
job by #2952, once sharding meant no single `test` runner saw the whole picture) and
|
||||
stays single-lane (Ubuntu / Node 24 only) because multiplying coverage across
|
||||
OS/runtime lanes adds cost without improving the threshold signal. Note the gate's
|
||||
deliberate blind spot: it measures
|
||||
`gsd-core/bin/lib/*.cjs` only — `scripts/`, `hooks/`, and `bin/` are
|
||||
unenforced, and `stryker.config.mjs` additionally excludes ~48% of lib lines
|
||||
from mutation testing (see the UNMUTATED list there). Widening either gate is
|
||||
@@ -540,18 +530,18 @@ would, so a badly stale table cannot collapse the suite into a few fat chunks.
|
||||
|
||||
### CI job timeout budgets: report + near-cap warning (#4036)
|
||||
|
||||
Every matrixed job — `test` and `test-full` in `test.yml`, `mutate` in
|
||||
Every matrixed job — `test` in `test.yml`, `mutate` in
|
||||
`mutation.yml`, `smoke` in `install-smoke.yml` — declares a `timeout-minutes`
|
||||
cap. `tests/ci-test-job-timeout-budget.test.cjs` enforces that each checked-in
|
||||
cap stays at least the **headroom factor** (1.5x) above a documented,
|
||||
hand-measured cost for that job — now all four of the jobs above, not just
|
||||
`test`/`test-full`/`coverage-gate`/`test-inert` as before.
|
||||
hand-measured cost for that job — now all three of the jobs above, not just
|
||||
`test`/`coverage-gate`/`test-inert` as before.
|
||||
|
||||
`test-conformance` in `test.yml` (#4591, epic #4589 Phase 2) runs the
|
||||
`scripts/lib/platform-conformance-tier.generated.cjs` file list on
|
||||
`windows-latest` (sharded three ways) and `macos-latest` (unsharded) — the
|
||||
gating signal for real-OS coverage, replacing `test-full`'s old gating role
|
||||
for one release cycle while `test-full` runs as a non-gating safety net. It
|
||||
sole gating signal for real-OS coverage (#4603 retired the parallel
|
||||
legacy full-matrix safety-net job). It
|
||||
also declares a `timeout-minutes` cap and runs the same in-job near-cap check
|
||||
described below, but it has no `LANE_COSTS` entry in
|
||||
`tests/ci-test-job-timeout-budget.test.cjs` yet — no real, completed
|
||||
@@ -561,7 +551,7 @@ does not cover it until one lands.
|
||||
Two runtime mechanisms sit on top of that static gate, both new in #4036:
|
||||
|
||||
- **In-job near-cap check** (`scripts/ci-check-job-near-cap.cjs`) — the last
|
||||
step of each of the four jobs computes elapsed-vs-cap from a start-time
|
||||
step of each of the three jobs computes elapsed-vs-cap from a start-time
|
||||
marker recorded as that job's first step. At >=90% of budget it emits a
|
||||
`::warning::` annotation (visible in the PR Checks UI) and a
|
||||
`$GITHUB_STEP_SUMMARY` block. Advisory only — it never fails the job. Known
|
||||
@@ -572,7 +562,7 @@ Two runtime mechanisms sit on top of that static gate, both new in #4036:
|
||||
`scripts/ci-timeout-report.cjs`) — runs daily and on `workflow_dispatch`. It
|
||||
polls GitHub's Actions REST API for recently completed jobs across
|
||||
`test.yml`, `mutation.yml`, and `install-smoke.yml`, resolves each job's
|
||||
declared cap (a literal `timeout-minutes` for `test`/`test-full`/`test-conformance`/`smoke`, or
|
||||
declared cap (a literal `timeout-minutes` for `test`/`test-conformance`/`smoke`, or
|
||||
`scripts/mutation-matrix.cjs`'s `COVERED[<module>].timeoutMinutes` for
|
||||
`mutate`'s per-module shards), and appends any new `(runId, jobName)`
|
||||
records to `tests/ci-timeout-budget-history.jsonl`. Unlike the in-job check,
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
# How to read CI timeout budget signals
|
||||
|
||||
Every matrixed CI job (`test`, `test-full`, `test-conformance` in `.github/workflows/test.yml`;
|
||||
Every matrixed CI job (`test`, `test-conformance` in `.github/workflows/test.yml`;
|
||||
`mutate` in `mutation.yml`; `smoke` in `install-smoke.yml`) now reports how close it ran to its
|
||||
`timeout-minutes` cap. This page is for a maintainer trying to answer: *is a lane drifting
|
||||
toward its cap, and where do I look?* (`test-conformance` runs the platform-conformance-tier file
|
||||
list — `scripts/lib/platform-conformance-tier.generated.cjs` — on `windows-latest`, sharded three
|
||||
ways, and `macos-latest`, unsharded; it is the gating signal for real-OS coverage, replacing
|
||||
`test-full`'s old gating role for one release cycle.)
|
||||
ways, and `macos-latest`, unsharded; it is the sole gating signal for real-OS coverage.)
|
||||
|
||||
## 1. A single run crossed 90% of its budget
|
||||
|
||||
|
||||
Reference in New Issue
Block a user