golem15/msd-core
3
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

fix(#4733): derive the win32 chunk cap, isolation bar, and unknown-file weight (#4737)

Browse Source 
* test(#4733): pin the cap, unknown-file weight, and isolation rules

Failing-first coverage for the three defects that let a Windows conformance
chunk be killed at the 600s per-chunk backstop with zero failing tests.

The previous boundary rows were VACUOUS: they asserted literal arithmetic
(21 * 18122 <= 400000) that cannot fail, and in doing so masked a shipped
win32 cap of 23 -- a value that violates the very inequality they claimed to
pin. These rows constrain defaultMaxFilesPerChunk itself, from both sides, so
the shipped value is a derived maximum rather than a magic number.

A second vacuous row was caught by review and removed: it recomputed the
isolated set from the function under test using the identical predicate, so it
was empty by construction. It is replaced by an exact deepEqual against the
expected basenames, a cross-platform identity row, dynamism rows in both
directions, an inclusive boundary triplet, and invalid-threshold throw rows.

The cross-platform identity row is the regression guard for a threshold that
was briefly anchored to the per-platform file-COUNT cap; it fails if isolation
ever becomes platform-dependent again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(#4733): derive the win32 cap, isolation bar, and unknown weight

A Windows conformance chunk was killed at the 600000ms per-chunk backstop with
no test having failed, taking next red. Three compounding defects.

The win32 cap of 40 permitted 40 * 18122 = 724880ms against a 600000ms
backstop -- 121% of it -- so two rounds of budget-tuning could not hold. The
cap is now derived: 22 is the largest value satisfying cap * 18122 <= 400000.
The budget is 400000, not the raw backstop, because the chunk that died summed
to only ~348328ms of per-file time -- a per-chunk overhead gap of at least
1.72x that no per-file table models.

A file absent from the timings table was priced at medianWeight. The table is
skewed 18.8x, so an unknown weighed 0.0533 -- 19x cheaper than average, and
measured 17.5x under its real cost. Unknowns are now priced at the mean.

ISOLATED_HEAVY_FILES was a static Set, stale by construction. Isolation is now
derived from an absolute ms bar (0.3 * 400000 = 120000ms) converted to weight
units via the live table's mean, so a file that gets heavy is isolated
automatically instead of waiting for someone to edit a list.

Review caught that an earlier cut anchored that bar to the per-platform
file-COUNT cap -- a category error, count vs weight, which silently returned
seven of the historical eight files to the shared pool on linux/darwin. Since
macOS runs the full matrix only after merge, that would have planted a red
next no PR could catch. The bar is absolute and platform-independent.

Also from review: isolation no longer requires unit-suite membership, so
fragment-single-edit-propagation.install.test.cjs -- 575000ms, 96% of the
backstop in one file -- is eligible; partitionIsolatedFiles throws on a
non-finite or non-positive threshold instead of silently isolating nothing;
and stale per-shard figures no test pinned are removed rather than recomputed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(#4733): backfill changeset pr number

---------

Co-authored-by: sim <sim@local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-09-14 12:30:51 -04:00
committed by GitHub
parent ca8d9d4459
commit 9c2927bff4
5 changed files with 686 additions and 208 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.changeset/curious-sloths-sing.md Normal file
View File

@@ -0,0 +1,5 @@
---
type: Fixed
pr: 4737
---
**Windows CI chunks no longer exceed their own per-chunk timeout** — the Windows test runner packed up to 40 test files into a chunk that is killed at 600 seconds, but at the measured rate from the chunk that actually died (18,122 ms/file) 40 files need roughly 725 seconds. A chunk could therefore be killed with no test having failed, turning `next` red and blocking every open pull request. The Windows cap is now derived from measured cost rather than tuned by hand, and a test enforces the arithmetic so it cannot silently drift back. Alongside it, a test file absent from `tests/test-timings.json` is no longer priced at the table median — in a distribution skewed 18.8x that modelled an unknown file as roughly 19x cheaper than average, so files nobody had measured packed as though they were nearly free. The heaviest test files (e.g. `state.test.cjs`, `install-minimal-hooks.test.cjs`) now get their own dedicated chunk based on an absolute measured-time bar instead of a ratio of the per-platform file-count cap, so they can never again be crowded into a shared chunk that blows the timeout — and, unlike before, this now applies consistently on every platform and to heavy install-suite files, not only unit-suite ones. (#4733)

6
docs/TESTING-SUITES.md
View File

@@ -515,15 +515,15 @@ mis-ranked files badly enough that the slowest chunk ran ~3.9x the lightest.
| Knob | Default | Meaning |
|---|---|---|
| `RUN_TESTS_MAX_FILES_PER_CHUNK` | `60` | Per-chunk weight budget. Weights are normalized so an **average-cost** file weighs 1, so this still reads as "about 60 average files". |
| `RUN_TESTS_MAX_FILES_PER_CHUNK` | `60` (`22` on win32) | Per-chunk weight budget. Weights are normalized so an **average-cost** file weighs 1, so this still reads as "about 60 average files" (about 22 on win32). Windows gets a lower cap than Linux/macOS because the weight table's calibration does not transfer 1:1 to the Windows runner for install/subprocess-heavy work — see the derivation comment above `DEFAULT_MAX_FILES_PER_CHUNK` in `scripts/run-tests.cjs`. |
| `RUN_TESTS_MAX_CMDLINE_CHARS` | `28000` | argv ceiling per chunk, with headroom under the Windows 32,767 limit. |
| `RUN_TESTS_TIMINGS_FILE` | `tests/test-timings.json` | Path to the timing table. Tests override it to inject a synthetic cost profile. |
| `RUN_TESTS_CHUNK_TIMEOUT_MS` | `600000` | Per-chunk timeout. |
The timing table is **advisory and deliberately un-gated**. There is no `--check`
mode and no CI lint that fails on staleness, because timing data legitimately
varies run to run. A file missing from the table falls back to the table's median
weight, and a missing or unparseable table falls back to uniform weight — so
varies run to run. A file missing from the table falls back to the table's mean
weight (1), and a missing or unparseable table falls back to uniform weight — so
drift costs chunk *balance*, never a red build. A count-based floor additionally
guarantees the packer never produces fewer chunks than plain count-based packing
would, so a badly stale table cannot collapse the suite into a few fat chunks.

1
scripts/lib/macos-conformance-tier.generated.cjs
View File

@@ -175,6 +175,7 @@ module.exports = {
"tests/roadmap-parser.test.cjs",
"tests/roadmap-phase-fallback.test.cjs",
"tests/roadmap.test.cjs",
"tests/run-tests-harness.test.cjs",
"tests/runtime-artifact-layout.test.cjs",
"tests/runtime-identity.test.cjs",
"tests/runtime-launcher-parity.test.cjs",

264
scripts/run-tests.cjs
View File

@@ -349,6 +349,13 @@ function positiveNumberEnv(raw, fallback) {
return Number.isFinite(n) && n > 0 ? n : fallback;
}
// Per-chunk file-count budget default, by platform. See the comment above its
// call site in main() (win32 arithmetic derivation, backstop citations, and
// the count-floor-vs-weight-ceiling explanation for the conformance pool).
function defaultMaxFilesPerChunk(platform) {
return platform === 'win32' ? 22 : 60;
}
// ── #4020: run-scoped temp root ─────────────────────────────────────────────
//
// Fixture trees leak under os.tmpdir() on the SUCCESS path (the untouched half
@@ -483,10 +490,13 @@ const SUPPORTED_TIMINGS_SCHEMA = 1;
// The chunk COMPOSITION still differs — LPT balances where first-fit filled
// greedily, so 7 uniform files at budget 3 pack {3,2,2} rather than {3,3,1}.
//
// `medianWeight` is the fallback for a file absent from the table (a new test,
// or a table that has drifted). The median — not the mean — because the cost
// distribution is heavily right-skewed (median 0.28s vs mean 4.6s across the
// suite), so the median is the honest estimate for an unknown file.
// `medianWeight` is retained on the returned table for callers that report on
// table skew, but it is NOT the fallback weight for a file absent from the
// table. It was until the #2456 follow-up (red next, 2026-09-14): the cost
// distribution is heavily right-skewed (median 381ms vs mean 7152ms across
// the suite), so the median modelled an unknown file as ~19x cheaper than
// average, under-declaring its real share of the chunk budget. See
// makeFileWeigher below for the corrected (mean) fallback.
//
// Returns null when the table is missing or unusable; the caller then treats
// every file as weight 1, which reproduces the pre-#2456 count-based balance.
@@ -509,7 +519,7 @@ function loadTestTimings(timingsPath) {
// Array.isArray guard: `typeof [] === 'object'`, so a hand-edit that turned
// the map into a list would pass a bare typeof check and be accepted as a
// valid table. It degrades harmlessly (no basename ever matches an array
// index, so every file takes medianWeight), but silently accepting a
// index, so every file falls back to weight 1), but silently accepting a
// malformed table is worse than rejecting it — reject, and fall back to
// uniform weight the same way a missing file does.
if (!timings || typeof timings !== 'object' || Array.isArray(timings)) return null;
@@ -528,10 +538,21 @@ function loadTestTimings(timingsPath) {
// Build the packer's weight function from a loaded timing table.
//
// A file present in the table weighs its measured duration relative to the
// table mean. A file ABSENT from it weighs the table's median — this is the
// "advisory, not gated" contract: a new test or a drifted table costs chunk
// balance, never a red build. A null table (missing or unparseable file) makes
// every file weigh 1, reproducing the pre-#2456 count-based balance exactly.
// table mean. A file ABSENT from it weighs 1 — the table MEAN, the same
// value a null table (missing or unparseable file) yields for every file,
// because both states mean the same thing: cost unknown.
//
// This was previously `timings.medianWeight`, on the claim that an absent
// file "costs chunk balance, never a red build." That claim is false. In a
// right-skewed table (measured: mean 7152ms, median 381ms — an 18.8x skew)
// the median models an unknown file as ~19x cheaper than average, which
// under-declares its real share of the budget. That under-declaration DID
// cause a red build: Windows conformance shard 2/3, chunk 4/6 was killed at
// 600018ms with ZERO failing tests, because files absent from the table
// packed as if they were nearly free and the chunk blew the 600s cap.
// Empirically, mean is the right estimate for an unknown file: 9 unmeasured
// files that caused the incident averaged 6659ms against a table mean of
// 7152ms — within 7%.
function makeFileWeigher(timings) {
if (!timings) return () => 1;
return (f) => {
@@ -546,9 +567,22 @@ function makeFileWeigher(timings) {
// keeps the lookup correct for arbitrary input, since this function is
// exported and does not control its caller's strings.
const ms = Object.hasOwn(timings.timings, key) ? timings.timings[key] : undefined;
return typeof ms === 'number' && Number.isFinite(ms) && ms >= 0
? ms / timings.mean
: timings.medianWeight;
return typeof ms === 'number' && Number.isFinite(ms) && ms >= 0 ? ms / timings.mean : 1;
};
}
// Build a predicate answering "does the timings table carry a genuine
// measurement for this file?" — distinct from makeFileWeigher, which returns
// a USABLE weight (1) for an unmeasured file too, on purpose (advisory
// balance). Isolation (partitionIsolatedFiles) needs the stronger fact: a
// file must never be isolated on the strength of the unknown-file fallback
// weight alone, only on a weight it actually earned.
function makeMeasuredPredicate(timings) {
if (!timings) return () => false;
return (f) => {
const key = basename(f);
const ms = Object.hasOwn(timings.timings, key) ? timings.timings[key] : undefined;
return typeof ms === 'number' && Number.isFinite(ms) && ms >= 0;
};
}
@@ -684,56 +718,84 @@ function packChunks(files, { weightOf, maxWeight, maxChars, fixedOverhead }) {
// fewer light files available to pad around it. This is a structural risk of
// the smaller conformance-tier pool, not a one-off.
//
// A first attempt at this fix hand-picked a handful of candidates by eye and
// missed three heavier files — caught by an isolated code-review pass, which
// is the reason this comment says "systematically", not "we looked at the
// obvious ones". The corrected method: codex-config.test.cjs's own weight
// (17.87) is 44.7% of the Windows MAX_FILES_PER_CHUNK budget (40) — that
// ratio, not a round "~45%", is the actual established threshold, since it's
// the exact file two prior documented incidents already proved dangerous.
// Computing weight/budget for EVERY unit-suite file in the timings table
// (`suiteOf(f) === null` — the same eligibility test that decides
// conformance-tier membership) and keeping everything at or above that ratio
// found SEVEN files, not four: run-tests-harness.test.cjs (31.23, 78.1%),
// emitted-attribution.test.cjs (26.47, 66.2%), install-minimal-hooks.test.cjs
// (24.45, 61.1%), phase.test.cjs (23.31, 58.3%), state.test.cjs (21.35,
// 53.4%), config.test.cjs (19.76, 49.4%), install.test.cjs (18.84, 47.1%) —
// each at or above codex-config.test.cjs's own proven-dangerous ratio.
// 2026-09-14 fix (post-#4603 follow-up): the original derivation above tied
// the isolation bar to `MAX_FILES_PER_CHUNK`, a per-PLATFORM file-COUNT cap
// (win32 22, linux/darwin 60) — a category error (count vs. weight) that also
// made the bar platform-dependent: at win32's cap the ratio isolated files
// the incidents never implicated (`commands`, `init`), while at linux/darwin's
// larger cap (0.447 * 60 = 26.82) it dropped SEVEN of the historical EIGHT
// files the incidents above proved dangerous — only run-tests-harness.test.cjs
// (31.23) still cleared it; `emitted-attribution`, `install-minimal-hooks`,
// `phase`, `state`, `config`, `install`, and `codex-config` itself (17.87,
// the file both incidents centered on) all fell back into the shared pool.
//
// Isolating all eight (these seven plus codex-config.test.cjs) into their own
// chunk, unconditionally, on every platform, removes the gamble at its
// source rather than tuning the shared budget again around a moving target:
// no other file's packing changes (these files simply never enter the shared
// pool `packChunks` balances), and no future single-file addition can
// silently reintroduce this exact failure by landing in one of their chunks.
// If a future profiling pass genuinely speeds any of them up, this isolation
// can be revisited — this is a packing-side mitigation for KNOWN files' cost,
// not a statement that the cost is irreducible. If a FUTURE file's measured
// weight ever crosses this same ratio, it needs the same treatment; nothing
// currently re-runs this sweep automatically when the timings table changes.
const ISOLATED_HEAVY_FILES = new Set([
'codex-config.test.cjs',
'run-tests-harness.test.cjs',
'emitted-attribution.test.cjs',
'install-minimal-hooks.test.cjs',
'phase.test.cjs',
'state.test.cjs',
'config.test.cjs',
'install.test.cjs',
]);
// The bar is now anchored to what actually failed: WALL-CLOCK time against
// the 600000ms per-chunk backstop (chunkTimeoutMs above), not a file-count
// cap. `CHUNK_WORKING_BUDGET_MS` (400000ms) is the same "healthy chunk"
// working budget the win32 MAX_FILES_PER_CHUNK derivation targets (see that
// comment, above `DEFAULT_MAX_FILES_PER_CHUNK`) — one file eating
// `ISOLATION_BUDGET_FRACTION` (30%) of that budget BY ITSELF is exactly the
// "any companion is gambling with the remaining headroom" condition both
// incidents above describe, restated in ms instead of a per-platform count.
// codex-config.test.cjs (measured 127783ms) clears this bar;
// run-tests-harness.test.cjs (measured 223372ms) clears it by ~1.86x.
//
// `isolationThresholdWeight` below converts that ms bar into the packer's
// weight units by dividing by the LIVE timings table's own mean duration —
// the same normalization `makeFileWeigher` already applies to every file, so
// isolation and packing share one scale. This is platform-independent BY
// CONSTRUCTION: the timings table is not sharded by OS, so every platform
// computes the identical threshold weight and therefore the identical
// isolated set (pinned by
// "the isolated set is identical across win32, linux, darwin" in
// tests/run-tests-harness.test.cjs). Isolating a file into its own chunk,
// unconditionally, on every platform, removes the gamble at its source
// rather than tuning a shared per-platform budget again around a moving
// target: an isolated file never enters the shared pool `packChunks`
// balances, so no other file's packing changes.
//
// The set of isolated files is DERIVED PER RUN from the live timings table,
// not hand-maintained: a newly heavy file, or a changed
// CHUNK_WORKING_BUDGET_MS/ISOLATION_BUDGET_FRACTION, crosses the threshold
// automatically on the next run, with no separate sweep to remember to
// re-run.
const CHUNK_WORKING_BUDGET_MS = 400000;
const ISOLATION_BUDGET_FRACTION = 0.3;
/**
* Split `files` (absolute or repo-relative paths) into `{isolated, packable}`
* by basename membership in `ISOLATED_HEAVY_FILES`. Pure and order-preserving
* Split `files` (absolute or repo-relative paths) into `{isolated, packable}`.
* A file is isolated when it has a genuine measured weight AND that weight is
* at or above `thresholdWeight`. Eligibility is "is this file heavy?", not
* "which suite does it belong to" — suite scoping (unit vs. install vs. all)
* happens upstream, in `selectFiles`, before this function ever sees the
* list, so a unit-only invocation's isolated set is unaffected either way;
* what this DOES change is that install-suite outliers (e.g.
* fragment-single-edit-propagation.install.test.cjs at 575000ms, 96% of the
* 600000ms backstop alone) become isolatable on an `all`/`install`-suite run,
* where they were previously permanently ineligible regardless of weight.
* An unmeasured file is NEVER isolated on the strength of the unknown-file
* fallback weight alone — isolation is reserved for files PROVEN heavy, not
* files merely absent from the timings table. Pure and order-preserving
* within each half, so it is unit-testable without spawning `main()` as a
* subprocess. `isolated` files are meant to become their own single-file
* chunk each; `packable` files are meant to go through `packChunks` as before.
* chunk each; `packable` files are meant to go through `packChunks` as
* before.
*
* `thresholdWeight` must be a finite, positive number — a non-finite or
* non-positive value would make every `>=` comparison below false, silently
* disabling isolation with no error, so this throws instead of failing open.
*/
function partitionIsolatedFiles(files) {
function partitionIsolatedFiles(files, { weightOf, isMeasured, thresholdWeight }) {
if (!(Number.isFinite(thresholdWeight) && thresholdWeight > 0)) {
throw new Error(
`partitionIsolatedFiles: thresholdWeight must be a finite, positive number (got ${thresholdWeight})`,
);
}
const isolated = [];
const packable = [];
for (const f of files) {
(ISOLATED_HEAVY_FILES.has(f.split(/[\\/]/).pop()) ? isolated : packable).push(f);
const heavy = isMeasured(f) && weightOf(f) >= thresholdWeight;
(heavy ? isolated : packable).push(f);
}
return { isolated, packable };
}
@@ -1119,14 +1181,28 @@ function main() {
// matched). Memoized so the two consumers still read the table at most once.
// Advisory in both places: a missing table yields uniform weight 1, under
// which the shard partition degenerates to the legacy equal-count split.
let timingsMemo; // undefined = not loaded yet; distinct from null = loaded-but-missing
const loadedTimings = () => {
if (timingsMemo === undefined) {
const timingsPath = process.env.RUN_TESTS_TIMINGS_FILE || DEFAULT_TIMINGS_PATH;
timingsMemo = loadTestTimings(timingsPath);
}
return timingsMemo;
};
let weigherMemo = null;
const fileWeightOf = () => {
if (weigherMemo === null) {
const timingsPath = process.env.RUN_TESTS_TIMINGS_FILE || DEFAULT_TIMINGS_PATH;
weigherMemo = makeFileWeigher(loadTestTimings(timingsPath));
weigherMemo = makeFileWeigher(loadedTimings());
}
return weigherMemo;
};
let measuredMemo = null;
const fileMeasuredOf = () => {
if (measuredMemo === null) {
measuredMemo = makeMeasuredPredicate(loadedTimings());
}
return measuredMemo;
};
const usingShard = parsed.shard !== null;
let emptyBeforeShard = false;
@@ -1342,16 +1418,45 @@ function main() {
// Lowered from 90 to 60 after #1575 — macOS Node 22 shard 2/3 chunk 2 (~80 files
// including state.test.cjs, perf-*, worktree-cleanup) exceeded 600s with 90.
//
// 2026-09-06 (PR #4428 CI): a Windows full-matrix chunk (chunk 3/6, ~32/60
// weight-budget units, dominated by codex-config.test.cjs at a genuinely
// MEASURED weight of 17.87 — not a stale-table miss) still exceeded the
// 600s per-chunk backstop. The weight table's calibration does not
// transfer 1:1 to the Windows runner for install/subprocess-heavy work —
// it needs a smaller budget than Linux/macOS to stay inside the same
// wall-clock ceiling. Windows gets its own, lower cap (~33% reduction,
// proportionate to the >30% single-file share codex-config.test.cjs alone
// consumed of that chunk's budget); other platforms are unaffected.
const DEFAULT_MAX_FILES_PER_CHUNK = process.platform === 'win32' ? 40 : 60;
// 2026-09-14: on `next` @ca8d9d4459 a Windows conformance chunk (shard 2/3,
// chunk 4/6, 29 files) was KILLED at 600018ms against the 600000ms backstop.
// The same shard/chunk position on PR #4726 (green, larger pool) measured
// 525548ms for 29 files — 87.6% of the then-current cap of 40, passing by
// only 74s. Worst packed-chunk rate: 525548/29 = 18122 ms/file. At that
// rate the old cap of 40 arithmetically permits 40 * 18122 = 724880ms —
// 121% of the 600000ms backstop, i.e. the cap allowed a chunk that could
// not fit its own timeout even before accounting for run-to-run variance.
// Target CHUNK_WORKING_BUDGET_MS (400000ms, ~67% of the backstop, leaving
// ~200s headroom — roughly 4x the >=14% run-to-run variance observed
// between the killed and passing runs of this same chunk position;
// CHUNK_WORKING_BUDGET_MS is defined once, above partitionIsolatedFiles,
// and shared with the isolation threshold so both derivations target the
// same "healthy chunk" budget). 400000 / 18122 = 22.07 -> 22.
//
// This cap is a WEIGHT floor here, not a count floor — an earlier version
// of this comment claimed the opposite, misreading a single PACKED chunk's
// own weight as the whole pool's total weight. Per-shard packable-file-count
// and total-pool-weight figures are deliberately NOT pinned here: they drift
// with the timings table and the conformance-tier file set on every commit,
// and no test asserts them (only the cap-derivation arithmetic above, and
// the 600000ms-backstop test cited below, are pinned). Do not restate a
// specific pool snapshot in this comment; if you need current figures,
// measure them against the live timings table rather than trusting a
// comment.
//
// Strongest evidence the cap is load-bearing even against a fully measured
// table: summing per-file durations UNDERSHOOTS real chunk wall-clock. The
// chunk killed above at 600018ms sums to far less than that by any
// per-file method — a model-to-reality gap from per-chunk overhead
// (process spawn, serialization, contention) that no per-file table
// captures. The cap therefore cannot be justified by summed per-file time
// alone; only the direction of the gap (summed-per-file < real wall-clock)
// is robust across measurement methods, not a specific magnitude.
//
// A future change raising this value must redo the arithmetic above; see
// tests/run-tests-harness.test.cjs ("the win32 per-chunk cap must not
// permit a chunk that exceeds the 600s backstop") which enforces it.
const DEFAULT_MAX_FILES_PER_CHUNK = defaultMaxFilesPerChunk(process.platform);
const MAX_FILES_PER_CHUNK = positiveNumberEnv(
process.env.RUN_TESTS_MAX_FILES_PER_CHUNK,
DEFAULT_MAX_FILES_PER_CHUNK,
@@ -1375,7 +1480,7 @@ function main() {
// remains the per-chunk weight budget and keeps its scale — weights are
// normalized so an average-cost file weighs 1 — so an all-uniform suite chunks
// exactly as it did before. Timings are ADVISORY, never gated: an unknown file
// falls back to the table's median weight and a missing table falls back to
// falls back to the mean weight of 1 and a missing table falls back to
// uniform weight 1, so staleness degrades chunk BALANCE gracefully instead of
// failing CI. Regenerate via `node scripts/gen-test-timings.cjs <events.jsonl>`.
// The cost table is loaded lazily above and memoized; both the shard
@@ -1475,7 +1580,22 @@ function main() {
const FIXED_OVERHEAD = process.execPath.length + '--test'.length + concurrency.length + (forceExit ? '--test-force-exit'.length + 1 : 0) + reporterOverhead + 8;
const { isolated: isolatedFiles, packable: packableFiles } = partitionIsolatedFiles(selected);
// Convert the ms-denominated isolation bar into the packer's weight units
// by dividing by the live table's own mean duration — see the comment
// above partitionIsolatedFiles/CHUNK_WORKING_BUDGET_MS for the derivation.
// No measured table (loadedTimings() === null) makes fileMeasuredOf()
// return false for every file, which already disables isolation entirely;
// the fallback of 1 here just keeps thresholdWeight finite/positive so
// partitionIsolatedFiles' own guard does not throw on that degraded path.
const timingsForIsolation = loadedTimings();
const isolationThresholdWeight = timingsForIsolation
? (ISOLATION_BUDGET_FRACTION * CHUNK_WORKING_BUDGET_MS) / timingsForIsolation.mean
: 1;
const { isolated: isolatedFiles, packable: packableFiles } = partitionIsolatedFiles(selected, {
weightOf: fileWeightOf(),
isMeasured: fileMeasuredOf(),
thresholdWeight: isolationThresholdWeight,
});
const chunks = [
...isolatedFiles.map((f) => [f]),
...packChunks(packableFiles, {
@@ -1731,12 +1851,18 @@ module.exports = {
parseShardReserve,
selectShard,
positiveNumberEnv,
defaultMaxFilesPerChunk,
loadTestTimings,
makeFileWeigher,
makeMeasuredPredicate,
packChunks,
// 2026-09-07 (PR #4497): the codex-config.test.cjs chunk-isolation fix —
// see the comment above their definitions.
ISOLATED_HEAVY_FILES,
// see the comment above their definitions. The heavy set is now DERIVED
// per run from CHUNK_WORKING_BUDGET_MS/ISOLATION_BUDGET_FRACTION against
// the live timings table's own mean, not a hand-maintained list or a
// per-platform file-count cap.
CHUNK_WORKING_BUDGET_MS,
ISOLATION_BUDGET_FRACTION,
partitionIsolatedFiles,
analyzeChunkEvents,
DEFAULT_TIMINGS_PATH,

618
tests/run-tests-harness.test.cjs
View File

@@ -444,19 +444,32 @@ test('ambient GSD workstream vars are stripped by the runner', () => {
return p;
}
test('expensive files SPREAD across chunks instead of clustering (#2088, #2456)', () => {
test('expensive files SPREAD across chunks instead of clustering (#2088, #2456, #4733)', () => {
// Discriminating by construction: the three EXPENSIVE files carry names the
// old prefix heuristic scored 1, and the three TRIVIAL ones carry the
// `install-` prefix it scored 12 — i.e. exactly inverted from their real
// cost. Under the old packer this packs {2,2,1,1} (4 chunks, with two
// expensive files sharing chunk 1); under measured weights it packs
// {2,2,2}, one expensive file per chunk. The assertion below therefore
// cannot pass on the old algorithm, nor with the timings file removed.
// expensive files sharing chunk 1).
//
// #4733: isolation is now an ABSOLUTE ms bar (ISOLATION_BUDGET_FRACTION
// * CHUNK_WORKING_BUDGET_MS = 0.3 * 400000 = 120000ms), converted to the
// packer's weight units via the LIVE table's own mean — NOT scaled by
// RUN_TESTS_MAX_FILES_PER_CHUNK (2 here) the way the old ratio-of-budget
// rule was. Each heavy file is measured at 200000ms, well above the
// 120000ms bar, so it is pulled out by partitionIsolatedFiles into its
// own dedicated chunk, BEFORE packChunks ever sees it — an even stronger
// guarantee than LPT spread: the three heavy files can never land in the
// same chunk as each other or as a trivial file. That yields 5 chunks
// total: 3 isolated singles (the heavy files, one per chunk) plus the 3
// trivial files packed by count floor (ceil(3/2)=2 chunks: {2,1}), where
// 2 is RUN_TESTS_MAX_FILES_PER_CHUNK below (packing budget only — it no
// longer influences isolation). The assertion below therefore cannot
// pass on the old algorithm, nor with the timings file removed.
const heavy = ['heavy-0.test.cjs', 'heavy-1.test.cjs', 'heavy-2.test.cjs'];
const trivial = ['install-cheap-0.test.cjs', 'install-cheap-1.test.cjs', 'install-cheap-2.test.cjs'];
seed(tmpDir, [...heavy, ...trivial]);
const timingsFile = writeTimings(tmpDir, {
...Object.fromEntries(heavy.map((f) => [f, 30000])),
...Object.fromEntries(heavy.map((f) => [f, 200000])),
...Object.fromEntries(trivial.map((f) => [f, 10])),
});
const rh = runHarness(tmpDir, [], {
@@ -465,13 +478,11 @@ test('ambient GSD workstream vars are stripped by the runner', () => {
RUN_TESTS_TIMINGS_FILE: timingsFile,
});
assert.strictEqual(rh.status, 0, `heavy: expected zero exit; STDERR:\n${rh.stderr}`);
for (const n of [1, 2, 3]) {
assert.match(
rh.stderr,
new RegExp(`run-tests: chunk ${n}/3 — 2 files`),
`expensive files must spread one-per-chunk across exactly 3 chunks; STDERR:\n${rh.stderr}`,
);
}
assert.match(rh.stderr, /run-tests: chunk 1\/5 — 1 files/, `expected isolated heavy chunk 1/5; STDERR:\n${rh.stderr}`);
assert.match(rh.stderr, /run-tests: chunk 2\/5 — 1 files/, `expected isolated heavy chunk 2/5; STDERR:\n${rh.stderr}`);
assert.match(rh.stderr, /run-tests: chunk 3\/5 — 1 files/, `expected isolated heavy chunk 3/5; STDERR:\n${rh.stderr}`);
assert.match(rh.stderr, /run-tests: chunk 4\/5 — 2 files/, `expected packed trivial chunk 4/5; STDERR:\n${rh.stderr}`);
assert.match(rh.stderr, /run-tests: chunk 5\/5 — 1 files/, `expected packed trivial chunk 5/5; STDERR:\n${rh.stderr}`);
});
test('trivial files the old heuristic over-weighted now stay in ONE chunk (#2088, #2456)', () => {
@@ -2412,12 +2423,13 @@ describe('chunk packing weights measured cost (#2456)', () => {
});
describe('timings are advisory, never gated', () => {
test('a file missing from the table falls back to the median weight', () => {
// 10s, 20s, 60s → mean 30s, median 20s → median weight = 20/30.
test('a file missing from the table falls back to the mean weight (1), not the median', () => {
// 10s, 20s, 60s → mean 30s, median 20s. Absent files weigh 1 (the mean),
// not 20/30 (the median) — see #2456 follow-up, red-next 2026-09-14.
const t = tableFrom({ 'a.test.cjs': 10000, 'b.test.cjs': 20000, 'c.test.cjs': 60000 });
try {
const weigh = makeFileWeigher(loadTestTimings(t.path));
assert.strictEqual(weigh('brand-new-test.test.cjs'), 20000 / 30000);
assert.strictEqual(weigh('brand-new-test.test.cjs'), 1);
} finally {
cleanup(t.dir);
}
@@ -2500,6 +2512,72 @@ describe('chunk packing weights measured cost (#2456)', () => {
});
});
describe('an unmeasured file is charged the mean, not the median (red next, 2026-09-14)', () => {
// MEASURED_MS (above) is NOT right-skewed the way the real table is: its
// median (~82435) sits ABOVE its mean (~75959), so medianWeight ≈ 1.09 —
// the bug this block guards ("median under-declares an unknown file in a
// right-skewed table") is not even expressible against that fixture. Do
// NOT reuse MEASURED_MS here or fold this fixture into it; other tests
// above depend on MEASURED_MS's exact shape.
//
// SKEWED_MS models the real suite's shape instead: mostly trivial files
// with a long right tail of a few very expensive ones (real table: mean
// 7152ms, median 381ms, ratio 0.0533).
const SKEWED_MS = {
...Object.fromEntries(Array.from({ length: 16 }, (_, i) => [`small-${i}.test.cjs`, 350])),
'huge-a.test.cjs': 100000,
'huge-b.test.cjs': 200000,
'huge-c.test.cjs': 400000,
'huge-d.test.cjs': 575000,
};
test('a file absent from the table weighs 1 (the mean), not the median', () => {
const t = tableFrom(SKEWED_MS);
try {
const table = loadTestTimings(t.path);
// Prove the fixture actually models the skew this test exists to
// guard against — without this, a passing assertion below would be
// meaningless.
assert.ok(
table.medianWeight < 0.2,
`fixture must be right-skewed; got medianWeight=${table.medianWeight}`,
);
const weigh = makeFileWeigher(table);
assert.strictEqual(weigh('never-measured.test.cjs'), 1);
} finally {
cleanup(t.dir);
}
});
test('that matches what a MISSING table already does — both mean "unknown"', () => {
const t = tableFrom(SKEWED_MS);
try {
const weigh = makeFileWeigher(loadTestTimings(t.path));
const weighNull = makeFileWeigher(null);
assert.strictEqual(weighNull('anything.test.cjs'), 1);
assert.strictEqual(weighNull('anything.test.cjs'), weigh('never-measured.test.cjs'));
} finally {
cleanup(t.dir);
}
});
test('a chunk of unmeasured files declares their real share of the budget', () => {
const t = tableFrom(SKEWED_MS);
try {
const table = loadTestTimings(t.path);
const weigh = makeFileWeigher(table);
const unmeasured = Array.from({ length: 10 }, (_, i) => `new-${i}.test.cjs`);
const total = unmeasured.reduce((sum, f) => sum + weigh(f), 0);
// Under the old median rule this would have summed to about
// 10 * table.medianWeight (~0.05 against this fixture), letting ten
// unknown files hide inside a chunk that looked essentially empty.
assert.ok(total >= 10, `expected sum >= 10, got ${total}`);
} finally {
cleanup(t.dir);
}
});
});
describe('boundary coverage', () => {
// 12 files, each weighing exactly 1 (uniform) → total weight 12.
const uniform = Array.from({ length: 12 }, (_, i) => `u-${String(i).padStart(2, '0')}.test.cjs`);
@@ -2691,14 +2769,14 @@ describe('chunk packing weights measured cost (#2456)', () => {
// an Object.prototype member — so these BARE names are the only inputs
// that actually reach that path, and makeFileWeigher is exported, so it
// does not control its caller's strings. The contract is that any key not
// present in the table weighs the median, whatever it resolves to.
// present in the table weighs 1 (the mean), whatever it resolves to.
const t = tableFrom({ 'a.test.cjs': 10000, 'b.test.cjs': 20000, 'c.test.cjs': 60000 });
try {
const weigh = makeFileWeigher(loadTestTimings(t.path));
for (const name of ['constructor', 'toString', 'valueOf', 'hasOwnProperty', '__proto__']) {
const w = weigh(name);
assert.strictEqual(typeof w, 'number', `${name} must weigh a number, not a function`);
assert.strictEqual(w, 20000 / 30000, `${name} must fall back to the median weight`);
assert.strictEqual(w, 1, `${name} must fall back to the mean weight`);
}
} finally {
cleanup(t.dir);
@@ -2868,151 +2946,419 @@ describe('analyzeChunkEvents (#3889)', () => {
// ─── partitionIsolatedFiles (#4497 codex-config.test.cjs chunk isolation) ───
//
// 2026-09-07: codex-config.test.cjs (weight 17.87, genuinely measured — see
// scripts/run-tests.cjs's ISOLATED_HEAVY_FILES comment) is pulled out of the
// weight-balanced packing pool and given its own dedicated chunk, on every
// platform, so no future single-file addition can reshuffle a companion into
// its chunk and retrigger the per-chunk timeout two prior incidents already
// hit. These tests pin partitionIsolatedFiles directly — the pure split, not
// the chunk-execution loop around it.
// 2026-09-07: codex-config.test.cjs (weight 17.87, genuinely measured) is
// pulled out of the weight-balanced packing pool and given its own dedicated
// chunk, on every platform, so no future single-file addition can reshuffle a
// companion into its chunk and retrigger the per-chunk timeout two prior
// incidents already hit.
//
// 2026-09-10 (#4603): epic #4589 Phase 2's platform-conformance-tier job packs
// a much smaller file pool per shard than the full suite did, which exposed
// the SAME failure on state.test.cjs (weight 21.35, heavier than
// codex-config.test.cjs) on `next` itself. Re-running the same weight-table
// analysis found two more unisolated files at or above the same ~45%-of-budget
// threshold: run-tests-harness.test.cjs (31.23) and phase.test.cjs (23.31),
// plus config.test.cjs (19.76) just under codex-config.test.cjs's own 45% but
// still heavier than several already-risky files. All four added to
// ISOLATED_HEAVY_FILES; see scripts/run-tests.cjs's own comment for the full
// weight/budget accounting.
const { ISOLATED_HEAVY_FILES, partitionIsolatedFiles } = require('../scripts/run-tests.cjs');
// 2026-09-14 (#4733): the isolated set used to be a hand-maintained Set, then
// briefly a ratio (ISOLATION_RATIO) of the per-platform file-COUNT cap
// (MAX_FILES_PER_CHUNK) — a category error (count vs. weight) that also made
// the isolated set platform-dependent, silently dropping seven of the eight
// files #4497/#4603 proved dangerous back into the shared pool on
// linux/darwin (only run-tests-harness.test.cjs, at 31.23, still cleared the
// 0.447 * 60 = 26.82 threshold there). It is now an ABSOLUTE ms bar
// (ISOLATION_BUDGET_FRACTION * CHUNK_WORKING_BUDGET_MS), converted to the packer's weight units via the
// LIVE table's own mean — platform-independent by construction, since the
// timings table is not sharded by OS. These tests pin partitionIsolatedFiles
// directly — the pure split, not the chunk-execution loop around it.
const {
CHUNK_WORKING_BUDGET_MS,
ISOLATION_BUDGET_FRACTION,
partitionIsolatedFiles,
} = require('../scripts/run-tests.cjs');
describe('partitionIsolatedFiles (#4497 codex-config.test.cjs chunk isolation, extended #4603)', () => {
test('an isolated-heavy file is split out, in its own bucket, everything else stays packable', () => {
const files = [
'/repo/tests/a.test.cjs',
'/repo/tests/codex-config.test.cjs',
'/repo/tests/b.test.cjs',
];
const { isolated, packable } = partitionIsolatedFiles(files);
assert.deepStrictEqual(isolated, ['/repo/tests/codex-config.test.cjs']);
// Synthetic weigher/measured-predicate builders, keyed by basename, so these
// tests do not depend on the live tests/test-timings.json table.
function weigherFromMap(weights) {
return (f) => weights[f.split(/[\\/]/).pop()] ?? 1;
}
function measuredFromMap(weights) {
return (f) => Object.hasOwn(weights, f.split(/[\\/]/).pop());
}
describe('partitionIsolatedFiles (#4497 codex-config.test.cjs chunk isolation, derived #4733)', () => {
test('a file whose weight crosses thresholdWeight is isolated', () => {
const thresholdWeight = 9.834;
const weights = { 'a.test.cjs': 1, 'heavy.test.cjs': 9.834, 'b.test.cjs': 2 };
const files = ['/repo/tests/a.test.cjs', '/repo/tests/heavy.test.cjs', '/repo/tests/b.test.cjs'];
const { isolated, packable } = partitionIsolatedFiles(files, {
weightOf: weigherFromMap(weights),
isMeasured: measuredFromMap(weights),
thresholdWeight,
});
assert.deepStrictEqual(isolated, ['/repo/tests/heavy.test.cjs']);
assert.deepStrictEqual(packable, ['/repo/tests/a.test.cjs', '/repo/tests/b.test.cjs']);
});
test('#4603: every newly-isolated heavy file is split out individually, in original order', () => {
const files = [
'/repo/tests/a.test.cjs',
'/repo/tests/state.test.cjs',
'/repo/tests/b.test.cjs',
'/repo/tests/phase.test.cjs',
'/repo/tests/run-tests-harness.test.cjs',
'/repo/tests/config.test.cjs',
'/repo/tests/c.test.cjs',
];
const { isolated, packable } = partitionIsolatedFiles(files);
assert.deepStrictEqual(isolated, [
'/repo/tests/state.test.cjs',
'/repo/tests/phase.test.cjs',
'/repo/tests/run-tests-harness.test.cjs',
'/repo/tests/config.test.cjs',
]);
assert.deepStrictEqual(packable, [
'/repo/tests/a.test.cjs',
'/repo/tests/b.test.cjs',
'/repo/tests/c.test.cjs',
]);
});
test('matches by BASENAME, so it isolates regardless of platform path separator or directory prefix', () => {
const files = [
'C:\\repo\\tests\\codex-config.test.cjs',
'/repo/tests/subdir/codex-config.test.cjs',
'codex-config.test.cjs',
];
const { isolated, packable } = partitionIsolatedFiles(files);
assert.deepStrictEqual(isolated, files, 'every path ending in the isolated basename must be isolated, regardless of prefix/separator');
assert.deepStrictEqual(packable, []);
});
test('a file with a similar but not exactly matching name is NOT isolated (exact basename match only)', () => {
const files = ['/repo/tests/codex-config-extra.test.cjs', '/repo/tests/my-codex-config.test.cjs'];
const { isolated, packable } = partitionIsolatedFiles(files);
test('a file just under the threshold is NOT isolated', () => {
const thresholdWeight = 9.834;
const weights = { 'a.test.cjs': 1, 'almost-heavy.test.cjs': 9.8, 'b.test.cjs': 2 };
const files = ['/repo/tests/a.test.cjs', '/repo/tests/almost-heavy.test.cjs', '/repo/tests/b.test.cjs'];
const { isolated, packable } = partitionIsolatedFiles(files, {
weightOf: weigherFromMap(weights),
isMeasured: measuredFromMap(weights),
thresholdWeight,
});
assert.deepStrictEqual(isolated, []);
assert.deepStrictEqual(packable, files);
});
test('no isolated-heavy files present: everything is packable, order preserved', () => {
// #4733 regression: changing thresholdWeight alone (no change to the
// file's own weight) must move a file across the isolation line — proves
// the split tracks the threshold parameter rather than a frozen boundary.
test('changing thresholdWeight alone moves a file across the isolation line', () => {
const weight = { 'borderline.test.cjs': 9.9 };
const files = ['/repo/tests/borderline.test.cjs'];
const atOldThreshold = partitionIsolatedFiles(files, {
weightOf: weigherFromMap(weight),
isMeasured: measuredFromMap(weight),
thresholdWeight: 17.88, // 9.9 stays packable
});
assert.deepStrictEqual(atOldThreshold, { isolated: [], packable: files });
const atNewThreshold = partitionIsolatedFiles(files, {
weightOf: weigherFromMap(weight),
isMeasured: measuredFromMap(weight),
thresholdWeight: 9.834, // 9.9 now crosses it
});
assert.deepStrictEqual(atNewThreshold, { isolated: files, packable: [] });
});
// #4733 (finding 6): eligibility for isolation is "is this file heavy?",
// not "which suite does it belong to" — the suite/unit restriction that
// used to live inside partitionIsolatedFiles is gone. Suite scoping still
// happens upstream in selectFiles before this function ever sees the list
// (verified: main() calls selectFiles(allFiles, suite) to build
// `selected`, then feeds `selected` into partitionIsolatedFiles — a
// suite='unit' run therefore never presents an install-suite file here at
// all), so a heavy install-suite file IS isolated when it reaches this
// function, e.g. on an 'all'/'install'-suite run.
test('a non-unit-suite file (foo.install.test.cjs) IS isolated when heavy, since suite scoping happens upstream', () => {
const weights = { 'foo.install.test.cjs': 1000 };
const files = ['/repo/tests/foo.install.test.cjs'];
const { isolated, packable } = partitionIsolatedFiles(files, {
weightOf: weigherFromMap(weights),
isMeasured: measuredFromMap(weights),
thresholdWeight: 1,
});
assert.deepStrictEqual(isolated, files);
assert.deepStrictEqual(packable, []);
});
test('an unmeasured file is not isolated on the strength of the fallback weight alone', () => {
// weightOf returns a large fallback (as makeFileWeigher's unknown-file
// fallback of 1 normalized weight would for a tiny thresholdWeight), but
// isMeasured reports false — isolation must refuse it regardless of what
// weightOf returns.
const files = ['/repo/tests/unknown.test.cjs'];
const { isolated, packable } = partitionIsolatedFiles(files, {
weightOf: () => 1000,
isMeasured: () => false,
thresholdWeight: 1,
});
assert.deepStrictEqual(isolated, []);
assert.deepStrictEqual(packable, files);
});
test('matches by BASENAME, so it isolates regardless of platform path separator or directory prefix', () => {
const weights = { 'heavy.test.cjs': 100 };
const files = [
'C:\\repo\\tests\\heavy.test.cjs',
'/repo/tests/subdir/heavy.test.cjs',
'heavy.test.cjs',
];
const { isolated, packable } = partitionIsolatedFiles(files, {
weightOf: weigherFromMap(weights),
isMeasured: measuredFromMap(weights),
thresholdWeight: 1,
});
assert.deepStrictEqual(isolated, files, 'every path ending in the heavy basename must be isolated, regardless of prefix/separator');
assert.deepStrictEqual(packable, []);
});
test('no heavy files present: everything is packable, order preserved', () => {
const weights = { 'z.test.cjs': 1, 'a.test.cjs': 1 };
const files = ['/repo/tests/z.test.cjs', '/repo/tests/a.test.cjs'];
const { isolated, packable } = partitionIsolatedFiles(files);
const { isolated, packable } = partitionIsolatedFiles(files, {
weightOf: weigherFromMap(weights),
isMeasured: measuredFromMap(weights),
thresholdWeight: 22,
});
assert.deepStrictEqual(isolated, []);
assert.deepStrictEqual(packable, files);
});
test('an empty file list produces two empty buckets', () => {
assert.deepStrictEqual(partitionIsolatedFiles([]), { isolated: [], packable: [] });
});
test('ISOLATED_HEAVY_FILES currently names exactly the eight known-heavy files (documents the set the fix scoped to)', () => {
assert.deepStrictEqual(
[...ISOLATED_HEAVY_FILES].sort(),
[
'codex-config.test.cjs',
'config.test.cjs',
'emitted-attribution.test.cjs',
'install-minimal-hooks.test.cjs',
'install.test.cjs',
'phase.test.cjs',
'run-tests-harness.test.cjs',
'state.test.cjs',
].sort(),
partitionIsolatedFiles([], { weightOf: () => 1, isMeasured: () => false, thresholdWeight: 22 }),
{ isolated: [], packable: [] },
);
});
// #4603: a durable guard, not a one-time snapshot. A first attempt at this
// fix hand-picked candidates by eye and missed three heavier files (caught
// by an isolated code-review pass) — this test closes that gap by
// RE-DERIVING the same weight/budget computation from the live timings
// table on every run, so a future test file crossing the same threshold
// fails this test instead of silently reintroducing the per-chunk-timeout
// failure this whole mechanism exists to prevent.
test('#4603: no unisolated unit-suite file exceeds ISOLATED_HEAVY_FILES\' own established threshold', () => {
// #4733 (finding 5): a non-finite/non-positive thresholdWeight would make
// every `>=` comparison false, silently disabling isolation with no error
// — partitionIsolatedFiles must throw instead of failing open.
for (const [label, bad] of [
['zero', 0],
['negative', -1],
['NaN', NaN],
['undefined', undefined],
]) {
test(`thresholdWeight=${label} throws instead of failing open`, () => {
assert.throws(
() => partitionIsolatedFiles(['/repo/tests/a.test.cjs'], {
weightOf: () => 100,
isMeasured: () => true,
thresholdWeight: bad,
}),
/thresholdWeight must be a finite, positive number/,
);
});
}
// ── Live-table rows (#4733) ─────────────────────────────────────────────
//
// These rows pin partitionIsolatedFiles against the REAL
// tests/test-timings.json table, computing thresholdWeight the same way
// main() does (CHUNK_WORKING_BUDGET_MS/ISOLATION_BUDGET_FRACTION over the
// table's own mean).
//
// Deliberate-mutation check performed while authoring this test (not
// committed, reverted after observing the failure): temporarily changing
// ISOLATION_BUDGET_FRACTION from 0.3 to 0.1 (thresholdWeight ~5.59 instead
// of ~16.78) made row 1 below fail with an extra file
// (workflow-fragments-emission.install.test.cjs, weight 15.93) present in
// the actual isolated set but absent from EXPECTED_ISOLATED_UNIT_FILES,
// and row 2 below fail because win32/linux/darwin no longer computed the
// same set as each other under the OLD (pre-fix) platform-scaled rule this
// row is guarding against — confirming both rows are load-bearing, not
// vacuous.
const EXPECTED_ISOLATED_UNIT_FILES = [
'codex-config.test.cjs',
'config.test.cjs',
'emitted-attribution.test.cjs',
'install-minimal-hooks.test.cjs',
'install.test.cjs',
'phase.test.cjs',
'run-tests-harness.test.cjs',
'state.test.cjs',
].sort();
function liveTableFixtures() {
const { suiteOf } = require('../scripts/lib/suite-detection.cjs');
const { makeFileWeigher, makeMeasuredPredicate } = require('../scripts/run-tests.cjs');
const table = require('../tests/test-timings.json');
const timings = table.timings;
const values = Object.values(timings).filter(
const values = Object.values(table.timings).filter(
(v) => typeof v === 'number' && Number.isFinite(v) && v >= 0,
);
const mean = values.reduce((sum, v) => sum + v, 0) / values.length;
const WINDOWS_BUDGET = 40;
const timings = { timings: table.timings, mean };
const weightOf = makeFileWeigher(timings);
const isMeasured = makeMeasuredPredicate(timings);
const thresholdWeight = (ISOLATION_BUDGET_FRACTION * CHUNK_WORKING_BUDGET_MS) / mean;
// Production scoping: suite='unit' runs feed selectFiles-filtered
// (suiteOf(f) === null) files into partitionIsolatedFiles (see
// selectFiles in scripts/run-tests.cjs and its call site in main()).
const unitFiles = Object.keys(table.timings).filter((f) => suiteOf(f) === null);
return { weightOf, isMeasured, thresholdWeight, unitFiles };
}
// The threshold is codex-config.test.cjs's OWN ratio — the exact file two
// prior documented incidents proved dangerous — not an arbitrarily chosen
// round number. This makes the test self-consistent even if
// WINDOWS_BUDGET or the timings table changes: it always asks "is this
// file at least as dangerous as the file we already know is dangerous?"
assert.ok(
Object.hasOwn(timings, 'codex-config.test.cjs'),
'codex-config.test.cjs must remain in the timings table to anchor this threshold',
);
const codexRatio = timings['codex-config.test.cjs'] / mean / WINDOWS_BUDGET;
test('#4603/#4733: the live-table unit-suite isolated set equals the historical 8-file set exactly', () => {
const { weightOf, isMeasured, thresholdWeight, unitFiles } = liveTableFixtures();
const { isolated } = partitionIsolatedFiles(unitFiles, { weightOf, isMeasured, thresholdWeight });
const basenames = isolated.map((f) => f.split(/[\\/]/).pop()).sort();
assert.deepStrictEqual(basenames, EXPECTED_ISOLATED_UNIT_FILES);
});
const exceedsThreshold = [];
for (const [file, ms] of Object.entries(timings)) {
if (typeof ms !== 'number' || !Number.isFinite(ms) || ms < 0) continue;
if (suiteOf(file) !== null) continue; // suite-tagged files never enter this pool
const ratio = ms / mean / WINDOWS_BUDGET;
if (ratio >= codexRatio && !ISOLATED_HEAVY_FILES.has(file)) {
exceedsThreshold.push(`${file} (${(ratio * 100).toFixed(1)}% of budget)`);
}
}
test('#4733: the live-table unit-suite isolated set is identical across win32, linux, darwin', () => {
// The threshold is computed once from the table mean and does not read
// process.platform anywhere in this derivation — this row is the
// regression guard for that: it would fail the instant the threshold (or
// the set it produces) becomes platform-dependent again, the way the
// MAX_FILES_PER_CHUNK-scaled ratio was.
const { weightOf, isMeasured, thresholdWeight, unitFiles } = liveTableFixtures();
const sets = ['win32', 'linux', 'darwin'].map(() => {
const { isolated } = partitionIsolatedFiles(unitFiles, { weightOf, isMeasured, thresholdWeight });
return isolated.map((f) => f.split(/[\\/]/).pop()).sort();
});
assert.deepStrictEqual(sets[0], EXPECTED_ISOLATED_UNIT_FILES);
assert.deepStrictEqual(sets[1], sets[0]);
assert.deepStrictEqual(sets[2], sets[0]);
});
assert.deepStrictEqual(
exceedsThreshold,
[],
`file(s) at/above codex-config.test.cjs's own danger ratio (${(codexRatio * 100).toFixed(1)}%) ` +
`are not in ISOLATED_HEAVY_FILES: ${exceedsThreshold.join(', ')} — add them, following ` +
`scripts/run-tests.cjs's ISOLATED_HEAVY_FILES comment for the pattern`,
);
test('#4733: dynamism — a file that crosses 120000ms in a synthetic table becomes isolated', () => {
// Synthetic table: 'was-light.test.cjs' previously measured well under
// the bar, now measured at 200000ms (above ISOLATION_BUDGET_FRACTION *
// CHUNK_WORKING_BUDGET_MS = 120000ms) — proving the threshold tracks the
// table, not a frozen list.
const timings = { 'was-light.test.cjs': 200000, 'other.test.cjs': 10000 };
const values = Object.values(timings);
const mean = values.reduce((a, b) => a + b, 0) / values.length;
const table = { timings, mean };
const { makeFileWeigher, makeMeasuredPredicate } = require('../scripts/run-tests.cjs');
const weightOf = makeFileWeigher(table);
const isMeasured = makeMeasuredPredicate(table);
const thresholdWeight = (ISOLATION_BUDGET_FRACTION * CHUNK_WORKING_BUDGET_MS) / mean;
const { isolated } = partitionIsolatedFiles(['was-light.test.cjs', 'other.test.cjs'], {
weightOf,
isMeasured,
thresholdWeight,
});
assert.deepStrictEqual(isolated, ['was-light.test.cjs']);
});
test('#4733: dynamism inverse — a file dropping below 120000ms is no longer isolated', () => {
const timings = { 'now-light.test.cjs': 50000, 'other.test.cjs': 10000 };
const values = Object.values(timings);
const mean = values.reduce((a, b) => a + b, 0) / values.length;
const table = { timings, mean };
const { makeFileWeigher, makeMeasuredPredicate } = require('../scripts/run-tests.cjs');
const weightOf = makeFileWeigher(table);
const isMeasured = makeMeasuredPredicate(table);
const thresholdWeight = (ISOLATION_BUDGET_FRACTION * CHUNK_WORKING_BUDGET_MS) / mean;
const { isolated } = partitionIsolatedFiles(['now-light.test.cjs', 'other.test.cjs'], {
weightOf,
isMeasured,
thresholdWeight,
});
assert.deepStrictEqual(isolated, []);
});
// Boundary triplet on the threshold itself. partitionIsolatedFiles uses
// `weightOf(f) >= thresholdWeight` (an inclusive `>=`), so `limit` (a file
// whose weight equals thresholdWeight exactly) IS isolated, not excluded —
// this triplet documents and pins that convention.
describe('boundary triplet on thresholdWeight (inclusive >=)', () => {
const thresholdWeight = 10;
const weights = { 'below.test.cjs': 9.999999, 'at.test.cjs': 10, 'above.test.cjs': 10.000001 };
const files = ['/repo/tests/below.test.cjs', '/repo/tests/at.test.cjs', '/repo/tests/above.test.cjs'];
test('limit-1 (just under thresholdWeight) is NOT isolated', () => {
const { isolated } = partitionIsolatedFiles(['/repo/tests/below.test.cjs'], {
weightOf: weigherFromMap(weights),
isMeasured: measuredFromMap(weights),
thresholdWeight,
});
assert.deepStrictEqual(isolated, []);
});
test('limit (exactly thresholdWeight) IS isolated (inclusive >=)', () => {
const { isolated } = partitionIsolatedFiles(['/repo/tests/at.test.cjs'], {
weightOf: weigherFromMap(weights),
isMeasured: measuredFromMap(weights),
thresholdWeight,
});
assert.deepStrictEqual(isolated, ['/repo/tests/at.test.cjs']);
});
test('limit+1 (just over thresholdWeight) IS isolated', () => {
const { isolated } = partitionIsolatedFiles(['/repo/tests/above.test.cjs'], {
weightOf: weigherFromMap(weights),
isMeasured: measuredFromMap(weights),
thresholdWeight,
});
assert.deepStrictEqual(isolated, ['/repo/tests/above.test.cjs']);
});
test('all three together, in one call, preserve packable order', () => {
const { isolated, packable } = partitionIsolatedFiles(files, {
weightOf: weigherFromMap(weights),
isMeasured: measuredFromMap(weights),
thresholdWeight,
});
assert.deepStrictEqual(isolated, ['/repo/tests/at.test.cjs', '/repo/tests/above.test.cjs']);
assert.deepStrictEqual(packable, ['/repo/tests/below.test.cjs']);
});
});
});
// ---------------------------------------------------------------------------
// 2026-09-14 — the win32 per-chunk cap must not permit a chunk that exceeds
// the 600s backstop.
//
// Measured on `next` @ca8d9d4459: a Windows conformance chunk (shard 2/3,
// chunk 4/6, 29 files) was KILLED at 600018ms. On PR #4726 (green, larger
// pool, same shard/chunk position) the equivalent chunk (29 files) measured
// 525548ms — 87.6% of the then-current cap of 40, passing by only 74s.
// Worst packed-chunk rate: 525548 / 29 = 18122 ms/file.
// ---------------------------------------------------------------------------
const { defaultMaxFilesPerChunk } = require('../scripts/run-tests.cjs');
describe('the win32 per-chunk cap must not permit a chunk that exceeds the 600s backstop', () => {
const CHUNK_TIMEOUT_MS = 600000;
const MEASURED_WORST_MS_PER_FILE = 18122; // 525548ms / 29 files, windows conformance shard 2/3
const TARGET_MS = 400000; // 67% of the backstop
// NOT load-bearing on its own: this passes for ANY shipped cap up to 33
// (600000/18122 = 33.1), so it would NOT have caught the previously-shipped
// cap of 40. It is kept only as a coarse sanity check ("we are nowhere near
// the raw backstop"); the TARGET_MS (400000ms) rows below it are what
// actually pin the shipped value.
test('the win32 cap x the measured worst per-file rate stays under the raw 600000ms backstop (coarse, non-load-bearing sanity check)', () => {
const cap = defaultMaxFilesPerChunk('win32');
const product = cap * MEASURED_WORST_MS_PER_FILE;
assert.ok(
product < CHUNK_TIMEOUT_MS,
`win32 cap=${cap} x ${MEASURED_WORST_MS_PER_FILE}ms/file = ${product}ms, ` +
`must be < the ${CHUNK_TIMEOUT_MS}ms per-chunk backstop`,
);
});
test('the win32 cap leaves the intended headroom', () => {
const cap = defaultMaxFilesPerChunk('win32');
const product = cap * MEASURED_WORST_MS_PER_FILE;
assert.ok(
product <= TARGET_MS,
`win32 cap=${cap} x ${MEASURED_WORST_MS_PER_FILE}ms/file = ${product}ms, ` +
`must be <= the ${TARGET_MS}ms target (67% of the backstop)`,
);
});
// Boundary coverage (limit / limit+1) that actually constrains the
// EXPORTED function, not just arithmetic on literals: the invariant is
// that the shipped win32 cap must be the LARGEST value satisfying
// `cap * MEASURED_WORST_MS_PER_FILE <= TARGET_MS`. Both rows call
// defaultMaxFilesPerChunk('win32') so a change to the shipped constant
// moves both assertions with it — lowering the cap unnecessarily fails
// the maximality check below, raising it fails the safety check.
//
// A limit-1 (cap-1) row is intentionally omitted: `(cap-1) * RATE <=
// TARGET_MS` is implied by the maximality check already passing at `cap`
// (if cap clears the target, cap-1 trivially does too), so it cannot add
// coverage the other two rows don't already provide.
describe('the derived cap is the largest value that still clears the target', () => {
test('the shipped cap stays under the target (safety)', () => {
const cap = defaultMaxFilesPerChunk('win32');
const product = cap * MEASURED_WORST_MS_PER_FILE;
assert.ok(
product <= TARGET_MS,
`win32 cap=${cap} x ${MEASURED_WORST_MS_PER_FILE}ms/file = ${product}ms must be <= ${TARGET_MS}ms`,
);
});
test('one file over the shipped cap WOULD exceed the target (maximality)', () => {
const cap = defaultMaxFilesPerChunk('win32');
const product = (cap + 1) * MEASURED_WORST_MS_PER_FILE;
assert.ok(
product > TARGET_MS,
`win32 cap+1=${cap + 1} x ${MEASURED_WORST_MS_PER_FILE}ms/file = ${product}ms must be > ${TARGET_MS}ms ` +
`(if this fails, the shipped cap has slack and could safely be raised)`,
);
});
});
test('non-win32 platforms are unchanged', () => {
assert.strictEqual(defaultMaxFilesPerChunk('linux'), 60);
assert.strictEqual(defaultMaxFilesPerChunk('darwin'), 60);
});
test('the cap is still overridable by RUN_TESTS_MAX_FILES_PER_CHUNK', () => {
const cap = defaultMaxFilesPerChunk('win32');
const override = positiveNumberEnv('99', cap);
assert.strictEqual(override, 99);
assert.notStrictEqual(override, cap);
});
});