* chore(#4603): retire the test-full CI job Phase 2 (#4591) added test-conformance but left test-full (the pre-existing full-suite Windows/macOS replay) running unchanged, gated on the same full_matrix flag, downgraded only from a hard gate to a non-blocking ::warning:: -- framed as "a non-gating safety net for one release cycle." No phase or issue ever retired it. Result: every full_matrix=true PR ran 10 OS-specific jobs (test-full's 6 + test-conformance's 4, purely additive) instead of the original 6 -- the epic's own goal (reduce runner-minutes) was measurably regressing, not improving, for the majority of PRs. This phase was missing from the original 4-phase epic decomposition; the epic (#4589) has been amended to add it as Phase 5 (see its comment thread), and this issue was filed as the tracked sub-issue. Deletes the test-full job from .github/workflows/test.yml entirely, along with every reference to it: required-tests' needs/FULL_TEST_RESULT warning branch, ci-timeout-report.cjs's JOB_RULES entry, ci-test-job-timeout-budget.test.cjs's LANE_COSTS/staticLanes/testFullRule entries, ci-test-scope.test.cjs's test-full-specific tests (preserving three unrelated tests that were nested in the same describe block, moved under a renamed describe rather than deleted), and docs mentions. test-conformance is now the sole gating signal for real-OS coverage. Two separate defects found and fixed while auditing every test-full reference: - tests/ci-pr-mergeability.test.cjs's GATED['test.yml'] safety-critical array (jobs that must needs: the mergeability preflight) had test-full but was missing test-conformance entirely -- Phase 2 never added it. Verified the real workflow wiring was already correct (test-conformance does have needs: [changes, preflight]); this was a test-coverage gap, not a live defect. Fixed by swapping the array entry. - docs/TESTING-SUITES.md's "## CI matrix" section was substantially stale independent of this phase (predating even #2952's coverage-gate split). Rewritten against the real, current job topology, verified directly against test.yml rather than trusted from memory. An isolated code-review pass found and fixed two minor inaccuracies in the rewritten docs table (two jobs' "Gated on" column didn't match their real if: condition exactly). An isolated security-review pass found no qualifying findings -- every compute-provisioning job already carries needs: preflight directly, unaffected by this deletion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(ci): isolate 7 more heavy test files from chunk-weight packing `next`'s own push-triggered Tests run failed: `conformance test (windows-latest, 24, shard 2/3)` chunk 3/6 was killed after 600019ms. Root cause: state.test.cjs (weight 21.35, measured) was packed alongside companions by run-tests.cjs's LPT chunk packer, the same failure mode that previously hit codex-config.test.cjs (weight 17.87) twice and got a dedicated fix (ISOLATED_HEAVY_FILES, #4497) -- but state.test.cjs was never added to that set. This is a direct, unintended consequence of epic #4589 Phase 2: the new platform-conformance-tier job packs only ~546 files per shard (vs. the ~950-file full suite the packer used to balance against), so the same absolute-weight outlier now represents a larger share of a smaller, more homogeneous pool -- the LPT packer has fewer light files to pad around it with. This was a real, foreseeable side effect of shrinking the packing pool that nobody checked for when Phase 2 shipped. A first attempt at this fix hand-picked 4 candidates by eyeballing a truncated weight list and missed 3 heavier ones -- caught by an isolated code-review pass (blocker: emitted-attribution.test.cjs at 66.2% of the Windows chunk budget, install-minimal-hooks.test.cjs at 61.1%, install.test.cjs at 47.1%, all above codex-config.test.cjs's own 44.7% -- the ratio that already proved dangerous twice). Corrected by systematically computing weight/budget for every unit-suite file and isolating everything at or above that same ratio: 7 files total, plus the pre-existing codex-config.test.cjs (8 total). Added a durable regression test (tests/run-tests-harness.test.cjs) that re-derives this exact computation from the live tests/test-timings.json on every run, so a future heavy file crossing this threshold fails the test instead of silently reintroducing this failure -- not just a one-time manual sweep. Verified end-to-end: simulated the real 3-way windows shard split of the actual conformance-tier file list with the real packing functions. Max packable-chunk weight across all 3 shards is now 27.04 / 24.10 / 23.91 (shard 2 is the exact shard that failed on next), comfortably under the 40 budget -- versus 40+ and a 600s kill before this fix. A second isolated code-review + security-review pass on the corrected diff found nothing further. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: sim <sim@local> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
@@ -33,7 +33,6 @@ const WORKFLOWS_DIR = path.join(__dirname, '..', '.github', 'workflows');
|
||||
const JOB_RULES = [
|
||||
{ workflowFile: 'test.yml', jobKey: 'test-inert', test: (name) => name === 'test (inert CI)' },
|
||||
{ workflowFile: 'test.yml', jobKey: 'test', test: (name) => name.startsWith('test (') && name !== 'test (inert CI)' },
|
||||
{ workflowFile: 'test.yml', jobKey: 'test-full', test: (name) => name.startsWith('full test (') },
|
||||
{ workflowFile: 'test.yml', jobKey: 'coverage-gate', test: (name) => name === 'Coverage gate (merged shards)' },
|
||||
{ workflowFile: 'test.yml', jobKey: 'test-conformance', test: (name) => name.startsWith('conformance test (') },
|
||||
{ workflowFile: 'install-smoke.yml', jobKey: 'smoke', test: (name) => name.startsWith('smoke (') },
|
||||
|
||||
@@ -16,8 +16,8 @@
|
||||
* (`scripts/affected-tests-lib.cjs`'s `PR_EXCLUDED_SUITES`; "PRs must never
|
||||
* select or run these"), and this generator's output feeds a `pull_request`-
|
||||
* triggered job; (2) `integration`/`security` already run via their own
|
||||
* separate, dedicated, unsharded, shard-1-only steps in the `test`/
|
||||
* `test-full` jobs (.github/workflows/test.yml) — folding any of them into
|
||||
* separate, dedicated, unsharded, shard-1-only steps in the `test` job
|
||||
* (.github/workflows/test.yml) — folding any of them into
|
||||
* this job's generic `--files-from` + `--shard` invocation is unproven and,
|
||||
* per the incident below, unsafe. (3) `qa` (loop-walk-suite files) already
|
||||
* runs via its own separate, dedicated `qa-loop-walk` job
|
||||
@@ -46,10 +46,10 @@
|
||||
* What stands in for it: (1) the most recent push-triggered run on `next`
|
||||
* (unconditionally full-matrix) is green on every OS for every file in this
|
||||
* classification, confirmed before this classifier was built; (2) the
|
||||
* existing `test-full` job keeps running the WHOLE suite on real Windows/
|
||||
* macOS as a non-gating safety net for one release cycle (.github/workflows/
|
||||
* test.yml) — a classifier miss surfaces as a visible warning there, not a
|
||||
* silent gap, before the safety net is retired. This is the same
|
||||
* legacy full-matrix job ran the WHOLE suite on real Windows/macOS as a
|
||||
* non-gating safety net for one release cycle (.github/workflows/test.yml)
|
||||
* before it was retired (#4603) — a classifier miss during that cycle would
|
||||
* have surfaced as a visible warning there, not a silent gap. This is the same
|
||||
* static-analysis-substitutes-for-real-OS-execution stance ADR-1703's whole
|
||||
* rule catalog already takes; it is a real, disclosed limit, not a silent
|
||||
* substitution.
|
||||
|
||||
@@ -671,16 +671,56 @@ function packChunks(files, { weightOf, maxWeight, maxChars, fixedOverhead }) {
|
||||
// which files end up as that gamble's companions is not something a future
|
||||
// PR can predict or control.
|
||||
//
|
||||
// Isolating it into its own chunk, unconditionally, on every platform,
|
||||
// removes the gamble at its source rather than tuning the shared budget a
|
||||
// third time around a moving target: no other file's packing changes (this
|
||||
// file simply never enters the shared pool `packChunks` balances), and no
|
||||
// future single-file addition can silently reintroduce this exact failure by
|
||||
// landing in its chunk. If a future profiling pass genuinely speeds up
|
||||
// codex-config.test.cjs itself, this isolation can be revisited — this is a
|
||||
// packing-side mitigation for a KNOWN file's cost, not a statement that the
|
||||
// cost is irreducible.
|
||||
const ISOLATED_HEAVY_FILES = new Set(['codex-config.test.cjs']);
|
||||
// 2026-09-10 (epic #4589 Phase 2/#4591, #4603): the SAME failure hit
|
||||
// state.test.cjs (weight 21.35, heavier than codex-config.test.cjs) on
|
||||
// `next`'s own push-triggered Tests run, `conformance test (windows-latest,
|
||||
// 24, shard 2/3)` chunk 3/6 — 600019ms, killed. Root cause is not a new
|
||||
// outlier: state.test.cjs was already this heavy before Phase 2 existed.
|
||||
// What changed is the POOL it gets packed against. Phase 2's
|
||||
// platform-conformance-tier job packs only the ~546 conformance-tier files
|
||||
// per shard (vs. the ~950-file full suite `packChunks` used to balance
|
||||
// against), so the same absolute-weight outlier now represents a much
|
||||
// larger share of a much smaller, more homogeneous pool — the LPT packer has
|
||||
// fewer light files available to pad around it. This is a structural risk of
|
||||
// the smaller conformance-tier pool, not a one-off.
|
||||
//
|
||||
// A first attempt at this fix hand-picked a handful of candidates by eye and
|
||||
// missed three heavier files — caught by an isolated code-review pass, which
|
||||
// is the reason this comment says "systematically", not "we looked at the
|
||||
// obvious ones". The corrected method: codex-config.test.cjs's own weight
|
||||
// (17.87) is 44.7% of the Windows MAX_FILES_PER_CHUNK budget (40) — that
|
||||
// ratio, not a round "~45%", is the actual established threshold, since it's
|
||||
// the exact file two prior documented incidents already proved dangerous.
|
||||
// Computing weight/budget for EVERY unit-suite file in the timings table
|
||||
// (`suiteOf(f) === null` — the same eligibility test that decides
|
||||
// conformance-tier membership) and keeping everything at or above that ratio
|
||||
// found SEVEN files, not four: run-tests-harness.test.cjs (31.23, 78.1%),
|
||||
// emitted-attribution.test.cjs (26.47, 66.2%), install-minimal-hooks.test.cjs
|
||||
// (24.45, 61.1%), phase.test.cjs (23.31, 58.3%), state.test.cjs (21.35,
|
||||
// 53.4%), config.test.cjs (19.76, 49.4%), install.test.cjs (18.84, 47.1%) —
|
||||
// each at or above codex-config.test.cjs's own proven-dangerous ratio.
|
||||
//
|
||||
// Isolating all eight (these seven plus codex-config.test.cjs) into their own
|
||||
// chunk, unconditionally, on every platform, removes the gamble at its
|
||||
// source rather than tuning the shared budget again around a moving target:
|
||||
// no other file's packing changes (these files simply never enter the shared
|
||||
// pool `packChunks` balances), and no future single-file addition can
|
||||
// silently reintroduce this exact failure by landing in one of their chunks.
|
||||
// If a future profiling pass genuinely speeds any of them up, this isolation
|
||||
// can be revisited — this is a packing-side mitigation for KNOWN files' cost,
|
||||
// not a statement that the cost is irreducible. If a FUTURE file's measured
|
||||
// weight ever crosses this same ratio, it needs the same treatment; nothing
|
||||
// currently re-runs this sweep automatically when the timings table changes.
|
||||
const ISOLATED_HEAVY_FILES = new Set([
|
||||
'codex-config.test.cjs',
|
||||
'run-tests-harness.test.cjs',
|
||||
'emitted-attribution.test.cjs',
|
||||
'install-minimal-hooks.test.cjs',
|
||||
'phase.test.cjs',
|
||||
'state.test.cjs',
|
||||
'config.test.cjs',
|
||||
'install.test.cjs',
|
||||
]);
|
||||
|
||||
/**
|
||||
* Split `files` (absolute or repo-relative paths) into `{isolated, packable}`
|
||||
|
||||
Reference in New Issue
Block a user