From 953b8043eabea64f4ee2c5f34650697cf0f35c36 Mon Sep 17 00:00:00 2001 From: Tom Boucher Date: Mon, 20 Jul 2026 16:51:35 -0400 Subject: [PATCH] fix(#2456): weight test chunks by measured cost and pack with LPT (#2463) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(#2456): weight test chunks by measured cost and pack with LPT scripts/run-tests.cjs guessed each test file's cost from its filename (basename matching /^(?:install|codex-)/ scored 12, everything else 1). Measured durations show that guess is wrong in both directions: installer-migration-authoring.test.cjs scored 12 while running ~0.1s, and the two most expensive files in the suite both scored 1 — run-tests-harness.test.cjs never matched the prefix, and release-tarball-smoke.install.test.cjs was missed because the regex is anchored to the START of the basename. Chunks were therefore balanced by file COUNT, not cost. On the real shard 2/3 the two heaviest files packed into the SAME chunk, leaving the slowest chunk 2.8x the lightest and sitting near the 600s per-chunk timeout while other chunks idled. Weight each file by its measured duration from a checked-in, regenerable timings table and pack with LPT (heaviest first, into the lightest chunk). On the same shard this drops the slowest chunk from 383s to 238s and the imbalance from 2.79x to 1.00x, and separates the two heavy files. Timings are advisory, never gated: an unknown file falls back to the table's median weight, a missing or corrupt table falls back to uniform weight, and a count-based floor guarantees the packer never produces fewer chunks than plain count-based packing would. Closes #2456 Co-Authored-By: Claude Opus 4.8 * fix(#2456): harden chunk packing against degenerate knobs and table keys Follow-up hardening found while reviewing the packer, fixed inline. The chunk knobs are read from the environment with Number(), so a typo (RUN_TESTS_MAX_FILES_PER_CHUNK=abc) yields NaN and an explicit 0 yields 0. Both flow into the new chunk-count arithmetic: NaN made Math.ceil return NaN, Array.from({length: NaN}) produce zero bins, and packChunks' retry loop spin forever — a hung CI job with no output. Zero made the count Infinity and threw RangeError: Invalid array length. The previous count-based packer degraded to a single chunk instead, so this was a regression introduced by the LPT rewrite. Normalize the knobs at the environment boundary (positiveNumberEnv: anything not a positive finite number falls back to the default) and guard packChunks itself, since it is exported and cannot assume its caller normalized. Non-finite weights from an arbitrary weightOf are clamped too. RUN_TESTS_CHUNK_TIMEOUT_MS gets the same treatment. Also resolve timing-table lookups with Object.hasOwn: the table is JSON-parsed, so a bare index would walk the prototype chain and return a function for a file named constructor.test.cjs or toString.test.cjs. The typeof guard already rejected that, but the lookup now resolves correctly rather than relying on the downstream check. Refs #2456 Co-Authored-By: Claude Opus 4.8 * fix(#2456): correct prototype-lookup rationale and guard generator keys Two findings from independent security review, fixed inline. The makeFileWeigher comment claimed a bare table lookup "would return a FUNCTION for a file named constructor.test.cjs". That premise is false: basename('constructor.test.cjs') is 'constructor.test.cjs', which is not an Object.prototype key, and walkTestFiles only ever collects *.test.cjs. The prototype chain was never reachable from a real selection, and the existing typeof guard already rejected the function it would return, so Object.hasOwn is defense-in-depth rather than a behavior change. The comment now says that instead of asserting something untrue. The accompanying test inherited the same false premise: it fed constructor.test.cjs and asserted a median fallback that would have held with or without the guard, so it passed for a reason unrelated to what it claimed to prove. It now uses BARE keys (constructor, toString, valueOf, hasOwnProperty, __proto__) — the only inputs that actually resolve on Object.prototype — and asserts the real exported contract: any key absent from the table weighs the median, never a function. gen-test-timings.cjs built its output object by computed-key assignment from basenames taken out of a reporter stream it does not control — the js/prototype-polluting-assignment shape, and this repo has a CodeQL barrier for exactly that pattern. It was not exploitable (the value is always a rounded number, so the __proto__ setter is a silent no-op), but it silently DROPPED such an entry rather than reporting it. Validate every key against a test-basename pattern and fail loudly instead, and build the table with a null prototype. Refs #2456 Co-Authored-By: Claude Opus 4.8 * fix(#2456): replace tautological chunking tests and clamp chunk count Six findings from independent correctness review, all reproduced and fixed inline. The two subprocess tests written to carry the #2088 guarantee forward were tautological: every seeded file weighed exactly 1, so both passed under the OLD prefix-heuristic packer and with the timings file deleted entirely. Neither could fail for the reason it existed. Both are rebuilt so the old algorithm produces a different packing and the assertion goes red: the spread test now uses three expensive files named so the old heuristic scored them 1 alongside three trivial `install-`-prefixed files it scored 12 — inverted from real cost, giving {2,2,1,1} under the old packer versus {2,2,2} under measured weights. The companion test covers the other direction: four trivial `install-` files the old heuristic split into four single-file chunks now stay in one. packChunks clamped the chunk count from below but not above, so a legitimate but tiny budget (RUN_TESTS_MAX_FILES_PER_CHUNK=1e-9, which positiveNumberEnv accepts) asked for 637,000,000,000 bins and threw RangeError. More chunks than files is never useful; the count now clamps at one file per chunk. The generator's basename-collision guard compared full dirnames, so two OS lanes reporting the same file under different container roots (/work/tests vs C:/work/tests) flagged every shared basename as a collision — on the script's own documented multi-lane usage. Detection is now scoped per stream, where the root is constant; a genuine same-lane collision is still caught. Also: the LPT tie-break compared raw paths, so a path separator (0x2F vs 0x5C) could order a subdir file differently per platform, contradicting the documented byte-identical guarantee — it now normalizes separators. loadTestTimings now honors schema_version instead of writing it and never reading it, falling back to uniform weight on an unknown version. A comment claiming an all-uniform suite "chunks exactly as it did before" was false and contradicted by this PR's own test: the chunk count is preserved, the composition is not. And the missing-table test created a temp dir it never cleaned up, for a path that only needed to not exist. Refs #2456 Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: Claude Opus 4.8 --- docs/TESTING-SUITES.md | 49 +++ scripts/gen-test-timings.cjs | 201 ++++++++++ scripts/run-tests.cjs | 297 +++++++++++--- tests/run-tests-harness.test.cjs | 542 ++++++++++++++++++++++++-- tests/test-timings.json | 649 +++++++++++++++++++++++++++++++ 5 files changed, 1668 insertions(+), 70 deletions(-) create mode 100644 scripts/gen-test-timings.cjs create mode 100644 tests/test-timings.json diff --git a/docs/TESTING-SUITES.md b/docs/TESTING-SUITES.md index b58c8d22a..32911f440 100644 --- a/docs/TESTING-SUITES.md +++ b/docs/TESTING-SUITES.md @@ -209,6 +209,55 @@ npm run ci:test-scope -- --files "commands/gsd/plan-phase.md" node scripts/ci-test-scope.cjs --base origin/next --head HEAD ``` +## Chunk packing and the test timing table + +`scripts/run-tests.cjs` does not hand the whole selected file list to one +`node --test` process. It packs the files into **chunks**, each spawned +separately, because Windows caps a command line at 32,767 characters and because +each chunk gets its own 600s timeout (`RUN_TESTS_CHUNK_TIMEOUT_MS`) and a fresh +process, which bounds memory pressure. + +How files are distributed across those chunks decides whether the slowest chunk +sits near that timeout while the others idle. The packer weights each file by its +**measured duration**, read from `tests/test-timings.json`, and places files with +LPT (longest-processing-time-first: heaviest file first, each into the currently +lightest chunk). Before #2456 the weight was guessed from the filename, which +mis-ranked files badly enough that the slowest chunk ran ~3.9x the lightest. + +### Reference + +| Knob | Default | Meaning | +|---|---|---| +| `RUN_TESTS_MAX_FILES_PER_CHUNK` | `60` | Per-chunk weight budget. Weights are normalized so an **average-cost** file weighs 1, so this still reads as "about 60 average files". | +| `RUN_TESTS_MAX_CMDLINE_CHARS` | `28000` | argv ceiling per chunk, with headroom under the Windows 32,767 limit. | +| `RUN_TESTS_TIMINGS_FILE` | `tests/test-timings.json` | Path to the timing table. Tests override it to inject a synthetic cost profile. | +| `RUN_TESTS_CHUNK_TIMEOUT_MS` | `600000` | Per-chunk timeout. | + +The timing table is **advisory and deliberately un-gated**. There is no `--check` +mode and no CI lint that fails on staleness, because timing data legitimately +varies run to run. A file missing from the table falls back to the table's median +weight, and a missing or unparseable table falls back to uniform weight — so +drift costs chunk *balance*, never a red build. A count-based floor additionally +guarantees the packer never produces fewer chunks than plain count-based packing +would, so a badly stale table cannot collapse the suite into a few fat chunks. + +### How-to: regenerate the timing table + +Regenerate when the suite's cost profile has visibly drifted — after adding or +removing expensive tests, not on a schedule. The input is a `node:test` reporter +event stream from a `gsd-test` run: + +```bash +node scripts/gen-test-timings.cjs \ + ~/.local/state/gsd-test/runs//test-events-linux-node22.jsonl \ + ~/.local/state/gsd-test/runs//test-events-linux-node24.jsonl +``` + +Pass every lane you have. A file's recorded time is the **max** across the +supplied streams, not the mean: the packer exists to keep the *slowest* lane's +slowest chunk away from the timeout, so the conservative bound is the right one. +Keys are sorted so a regeneration diff shows only the files whose cost moved. + ## Best practices for forward-compat (Node 24/26) - Use `process.execPath` when spawning Node in tests so each matrix lane exercises the lane's Node version. diff --git a/scripts/gen-test-timings.cjs b/scripts/gen-test-timings.cjs new file mode 100644 index 000000000..6448556d9 --- /dev/null +++ b/scripts/gen-test-timings.cjs @@ -0,0 +1,201 @@ +#!/usr/bin/env node +// Regenerate the per-file test timing table used to weight chunk packing in +// scripts/run-tests.cjs (issue #2456). +// +// The chunk packer needs to know what each test file actually COSTS. Before +// #2456 it guessed from the filename (`^(?:install|codex-)` scored 12x, +// everything else 1) and was wrong in both directions — installer-migration- +// authoring.test.cjs scored 12x but runs ~0.1s, while the two heaviest files in +// the suite (run-tests-harness.test.cjs and release-tarball-smoke.install +// .test.cjs) both scored 1. This script replaces the guess with measurement. +// +// Input is one or more node:test reporter event streams as emitted by +// `gsd-test` (`~/.local/state/gsd-test/runs//test-events--node +// .jsonl`). Each stream carries one `test:summary` event per test FILE, whose +// `data.duration_ms` is that file's total wall-clock and whose `data.file` is +// its absolute in-container path. +// +// Usage: +// node scripts/gen-test-timings.cjs [ ...] +// node scripts/gen-test-timings.cjs ~/.local/state/gsd-test/runs/*/test-events-*.jsonl +// node scripts/gen-test-timings.cjs events.jsonl --out tests/test-timings.json +// +// When several streams are supplied (multiple lanes, e.g. node22 + node24), a +// file's recorded time is the MAX across them, not the mean: the packer exists +// to keep the SLOWEST lane's slowest chunk away from the per-chunk timeout, so +// the conservative bound is the right one to balance against. +// +// The table is ADVISORY and deliberately un-gated — there is no `--check` mode +// and no CI lint that fails on staleness, because timing data legitimately +// varies run to run. A file missing from the table falls back to the table's +// median weight, so a stale table degrades chunk BALANCE gracefully instead of +// failing the build. Regenerate it when the suite's cost profile has visibly +// drifted, not on a schedule. +'use strict'; + +const fs = require('fs'); +const { basename, dirname, join } = require('path'); +const { ExitError, runMain } = require('./lib/cli-exit.cjs'); + +const DEFAULT_OUT = join(__dirname, '..', 'tests', 'test-timings.json'); +const SCHEMA_VERSION = 1; + +function parseArgs(argv) { + const inputs = []; + let out = DEFAULT_OUT; + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]; + if (arg === '--out') { + const value = argv[++i]; + if (!value) return { error: '--out requires a path' }; + out = value; + } else if (arg.startsWith('--out=')) { + const value = arg.slice('--out='.length); + if (!value) return { error: '--out requires a path' }; + out = value; + } else if (arg.startsWith('-')) { + return { error: `unknown flag "${arg}"` }; + } else { + inputs.push(arg); + } + } + if (inputs.length === 0) { + return { error: 'usage: gen-test-timings.cjs [...] [--out ]' }; + } + return { inputs, out }; +} + +// Fold one reporter event stream into `acc`, keeping the MAX duration seen for +// each test file. Returns per-stream counters plus any basename collisions +// found WITHIN this stream. +// +// Keying is by BASENAME, matching how run-tests.cjs weights a selected file: +// the reporter reports absolute in-container paths (/work/tests/foo.test.cjs) +// while the harness carries paths relative to its test dir, so the basename is +// the only stable join key between the two. A basename seen in two different +// directories would make the table ambiguous, so it must fail loudly. +// +// Collision detection is scoped to a SINGLE stream deliberately. Every lane +// writes its own stream under its own container root (`/work/tests` on Linux, +// `C:/work/tests` on Windows), so comparing directories ACROSS streams reports +// every shared basename as a collision — which is the script's own documented +// usage (globbing `test-events-*.jsonl` across lanes). Within one stream the +// root is constant, so a differing directory is a real collision. +function foldStream(text, acc) { + const dirsByBase = new Map(); + let files = 0; + let malformed = 0; + for (const line of text.split('\n')) { + if (line.trim() === '') continue; + let event; + try { + event = JSON.parse(line); + } catch { + malformed++; + continue; + } + if (!event || event.type !== 'test:summary') continue; + const data = event.data; + if (!data || typeof data.file !== 'string') continue; + const ms = data.duration_ms; + if (typeof ms !== 'number' || !Number.isFinite(ms) || ms < 0) continue; + const path = data.file.replace(/\\/g, '/'); + const base = basename(path); + if (!dirsByBase.has(base)) dirsByBase.set(base, new Set()); + dirsByBase.get(base).add(dirname(path)); + const prev = acc.get(base); + if (prev === undefined || ms > prev) acc.set(base, ms); + files++; + } + const collisions = [...dirsByBase.entries()] + .filter(([, dirs]) => dirs.size > 1) + .map(([base, dirs]) => `${base} (${[...dirs].sort().join(', ')})`); + return { files, malformed, collisions }; +} + +function main() { + const parsed = parseArgs(process.argv.slice(2)); + if (parsed.error) throw new ExitError(2, `gen-test-timings: ${parsed.error}`); + + const acc = new Map(); + const allCollisions = new Set(); + const sources = []; + for (const input of parsed.inputs) { + let text; + try { + text = fs.readFileSync(input, 'utf8'); + } catch (err) { + throw new ExitError(2, `gen-test-timings: cannot read "${input}": ${err.message}`); + } + const { files, malformed, collisions } = foldStream(text, acc); + for (const c of collisions) allCollisions.add(c); + sources.push(basename(input)); + console.error( + `gen-test-timings: ${basename(input)} — ${files} file summaries` + + (malformed > 0 ? `, ${malformed} unparseable lines skipped` : ''), + ); + } + + if (acc.size === 0) { + throw new ExitError( + 2, + 'gen-test-timings: no `test:summary` events with a file and duration_ms were found. ' + + 'Check that the input is a node:test reporter event stream (test-events--node.jsonl).', + ); + } + + // A basename that resolves to two different directories within one lane makes + // the table ambiguous: run-tests.cjs joins on basename alone, so one file's + // measured cost would silently be applied to the other. Fail rather than emit + // a table that lies. + if (allCollisions.size > 0) { + throw new ExitError( + 2, + `gen-test-timings: basename collision — the table cannot key on basename alone:\n ${[...allCollisions].sort().join('\n ')}`, + ); + } + + // Every key must be a plain test-file basename. This is a data-integrity + // check on a stream we do not control (the reporter emits whatever path the + // runner saw), and it structurally excludes a computed key like `__proto__` + // or `constructor` from being written into the table object below — the + // `js/prototype-polluting-assignment` shape, even though the value here is + // always a number and could not actually pollute. + const SAFE_BASENAME_RE = /^[A-Za-z0-9._-]+\.test\.cjs$/; + const rejected = [...acc.keys()].filter((base) => !SAFE_BASENAME_RE.test(base)); + if (rejected.length > 0) { + throw new ExitError( + 2, + `gen-test-timings: refusing to emit non-test-file keys: ${rejected.sort().join(', ')}`, + ); + } + + // Sorted keys keep the checked-in diff reviewable: a regeneration shows only + // the files whose cost actually moved, not a reshuffled object. + const timings = Object.create(null); + for (const base of [...acc.keys()].sort()) { + timings[base] = Math.round(acc.get(base)); + } + + const payload = { + schema_version: SCHEMA_VERSION, + generated_by: 'scripts/gen-test-timings.cjs', + unit: 'ms', + sources: sources.sort(), + file_count: acc.size, + timings, + }; + + fs.writeFileSync(parsed.out, `${JSON.stringify(payload, null, 2)}\n`, 'utf8'); + const totalMs = [...acc.values()].reduce((sum, ms) => sum + ms, 0); + console.error( + `gen-test-timings: wrote ${parsed.out} — ${acc.size} files, ${(totalMs / 1000).toFixed(1)}s total`, + ); + return 0; +} + +if (require.main === module) { + runMain(main); +} + +module.exports = { parseArgs, foldStream }; diff --git a/scripts/run-tests.cjs b/scripts/run-tests.cjs index e03ab8a98..0c470d3f7 100644 --- a/scripts/run-tests.cjs +++ b/scripts/run-tests.cjs @@ -29,7 +29,7 @@ // See docs/TESTING-SUITES.md for full grouping policy. 'use strict'; -const { readdirSync } = require('fs'); +const { readdirSync, readFileSync } = require('fs'); const { join, basename } = require('path'); const { execFileSync } = require('child_process'); const { ExitError, runMain } = require('./lib/cli-exit.cjs'); @@ -220,6 +220,206 @@ function selectShard(sortedFiles, { index, total }) { return sortedFiles.filter((_, k) => k % total === index - 1); } +// Read an operator-supplied numeric env knob, falling back to the default for +// anything that is not a positive finite number. +// +// This is a strict-input boundary (Postel's Law: a typo must fail SAFE, not +// silently poison arithmetic downstream). `Number('abc')` is NaN and +// `Number('')` is 0, and both are load-bearing here: a NaN chunk budget makes +// the chunk-count computation NaN, which spins packChunks' retry loop forever +// (a hung CI job with no output); a zero budget makes it Infinity, which throws +// `RangeError: Invalid array length`. Neither is an acceptable response to a +// mistyped environment variable. +function positiveNumberEnv(raw, fallback) { + if (raw === undefined || raw === null || String(raw).trim() === '') return fallback; + const n = Number(raw); + return Number.isFinite(n) && n > 0 ? n : fallback; +} + +// Per-file measured durations, regenerated by scripts/gen-test-timings.cjs from +// gsd-test reporter event streams. Overridable so tests can inject a synthetic +// table instead of depending on the real suite's cost profile. +const DEFAULT_TIMINGS_PATH = join(__dirname, '..', 'tests', 'test-timings.json'); +// Must track SCHEMA_VERSION in scripts/gen-test-timings.cjs. +const SUPPORTED_TIMINGS_SCHEMA = 1; + +// Load the timing table and reduce it to what the packer needs. +// +// Weights are normalized by the table's MEAN duration, so an average-cost file +// weighs exactly 1 and `MAX_FILES_PER_CHUNK` keeps its original meaning ("about +// N average files per chunk"). When every file costs the same, total weight +// equals file count, so the chunk COUNT matches count-based packing exactly. +// The chunk COMPOSITION still differs — LPT balances where first-fit filled +// greedily, so 7 uniform files at budget 3 pack {3,2,2} rather than {3,3,1}. +// +// `medianWeight` is the fallback for a file absent from the table (a new test, +// or a table that has drifted). The median — not the mean — because the cost +// distribution is heavily right-skewed (median 0.28s vs mean 4.6s across the +// suite), so the median is the honest estimate for an unknown file. +// +// Returns null when the table is missing or unusable; the caller then treats +// every file as weight 1, which reproduces the pre-#2456 count-based balance. +function loadTestTimings(timingsPath) { + let parsed; + try { + parsed = JSON.parse(readFileSync(timingsPath, 'utf8')); + } catch { + return null; + } + if (!parsed || typeof parsed !== 'object') return null; + // Refuse a table written by a future generator: a v2 schema could change the + // unit or the key format, and consuming it under v1 semantics would silently + // mis-weight every file. Returning null falls back to uniform weight, which + // is the same graceful degradation as a missing table. + if (parsed.schema_version !== undefined && parsed.schema_version !== SUPPORTED_TIMINGS_SCHEMA) { + return null; + } + const timings = parsed.timings; + if (!timings || typeof timings !== 'object') return null; + const values = Object.values(timings).filter( + (v) => typeof v === 'number' && Number.isFinite(v) && v >= 0, + ); + if (values.length === 0) return null; + const mean = values.reduce((sum, v) => sum + v, 0) / values.length; + if (!(mean > 0)) return null; + const sorted = [...values].sort((a, b) => a - b); + const mid = sorted.length >> 1; + const median = sorted.length % 2 === 1 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2; + return { timings, mean, medianWeight: median / mean }; +} + +// Build the packer's weight function from a loaded timing table. +// +// A file present in the table weighs its measured duration relative to the +// table mean. A file ABSENT from it weighs the table's median — this is the +// "advisory, not gated" contract: a new test or a drifted table costs chunk +// balance, never a red build. A null table (missing or unparseable file) makes +// every file weigh 1, reproducing the pre-#2456 count-based balance exactly. +function makeFileWeigher(timings) { + if (!timings) return () => 1; + return (f) => { + const key = basename(f); + // Own-property check before the lookup. This is defense-in-depth, NOT a + // behavior change: the table is JSON-parsed, so a bare `timings[key]` would + // walk the prototype chain, but the only keys that resolve there are + // Object.prototype members (`constructor`, `toString`, …) and every real + // selection is a `*.test.cjs` basename, which can never equal one. Even if + // it could, the `typeof ms === 'number'` guard below already rejects the + // function it would return. `Object.hasOwn` makes the intent explicit and + // keeps the lookup correct for arbitrary input, since this function is + // exported and does not control its caller's strings. + const ms = Object.hasOwn(timings.timings, key) ? timings.timings[key] : undefined; + return typeof ms === 'number' && Number.isFinite(ms) && ms >= 0 + ? ms / timings.mean + : timings.medianWeight; + }; +} + +// Pack `files` into chunks using LPT (longest-processing-time-first): sort by +// weight descending, then place each file into the currently-LIGHTEST chunk. +// +// #2456: the previous packer was a sequential first-fit that appended files in +// selection order and closed a chunk once its weight budget was hit. Because +// sorted-adjacent files land together, the two heaviest files in a shard packed +// into the SAME chunk, leaving the slowest chunk ~3.9x the lightest and sitting +// near the 600s per-chunk timeout while other chunks idled. LPT is the standard +// greedy approximation for exactly this makespan problem and balanced the same +// real shard to ~1.0x. +// +// Chunk COUNT is fixed before placement so LPT has bins to balance across: +// ceil(totalWeight / maxWeight) — the weighted budget, and +// ceil(fileCount / maxWeight) — a floor that pins the count at what the +// old count-based packing would produce. +// The floor is what makes a stale or missing timings table safe: unknown files +// fall back to a small median weight, which on its own would collapse many files +// into few fat chunks. With the floor, a degraded table can only ever reproduce +// today's chunking, never something coarser. +// +// `maxChars` still bounds each chunk's argv (Windows CreateProcess caps +// lpCommandLine at 32,767). A chunk that cannot fit the next file is skipped for +// that file; when no chunk has room, the chunk count grows and packing restarts. +// A single file longer than the budget lands alone rather than looping forever. +// +// Ordering is fully deterministic — ties break on the separator-normalized file +// path, and each chunk's files are emitted in their original selection order — +// so the packing is byte-identical across Windows/macOS/Linux. +function packChunks(files, { weightOf, maxWeight, maxChars, fixedOverhead }) { + if (files.length === 0) return []; + // packChunks is exported, so it cannot assume its caller normalized these. + // A non-finite or non-positive budget makes the chunk-count arithmetic + // non-finite, which spins the retry loop below forever or throws from + // Array.from; a non-finite weight propagates into the same computation. + // Degrade to a safe bound instead. + const weightBudget = Number.isFinite(maxWeight) && maxWeight > 0 ? maxWeight : files.length; + const charBudget = Number.isFinite(maxChars) && maxChars > 0 ? maxChars : Number.MAX_SAFE_INTEGER; + const overhead = Number.isFinite(fixedOverhead) && fixedOverhead >= 0 ? fixedOverhead : 0; + const safeWeight = (file) => { + const w = weightOf(file); + return Number.isFinite(w) && w >= 0 ? w : 0; + }; + const entries = files.map((file, index) => ({ + file, + index, + weight: safeWeight(file), + chars: file.length + 1, // +1 for the inter-arg separator + })); + const totalWeight = entries.reduce((sum, e) => sum + e.weight, 0); + // Ties break on a SEPARATOR-NORMALIZED path so a subdir file orders the same + // on Windows as on POSIX: '/' is 0x2F and '\' is 0x5C, which straddle the + // uppercase range, so comparing raw paths can order `sub/x.test.cjs` against + // `subZ.test.cjs` differently per platform and silently produce a different + // (still valid, but non-reproducible) packing. + const sortKey = (f) => f.replace(/\\/g, '/'); + const heaviestFirst = [...entries].sort((a, b) => { + if (b.weight !== a.weight) return b.weight - a.weight; + const ka = sortKey(a.file); + const kb = sortKey(b.file); + return ka < kb ? -1 : ka > kb ? 1 : 0; + }); + + // Termination: the empty-bin rule below guarantees every file is placeable + // once chunkCount reaches files.length, so the retry loop cannot run forever. + // The upper clamp matters as much as the lower bound: a legitimate but tiny + // budget (RUN_TESTS_MAX_FILES_PER_CHUNK=1e-9) would otherwise ask for + // 637,000,000,000 bins and throw `RangeError: Invalid array length`. More + // chunks than files is never useful — one file per chunk is the finest + // possible packing. + let chunkCount = Math.min( + files.length, + Math.max(1, Math.ceil(totalWeight / weightBudget), Math.ceil(files.length / weightBudget)), + ); + for (;;) { + const bins = Array.from({ length: chunkCount }, () => ({ + entries: [], + weight: 0, + chars: overhead, + })); + let overflowed = false; + for (const entry of heaviestFirst) { + let target = null; + for (const bin of bins) { + // An empty bin always accepts, so an over-long single file lands alone + // instead of growing the chunk count forever. + if (bin.entries.length > 0 && bin.chars + entry.chars > charBudget) continue; + if (target === null || bin.weight < target.weight) target = bin; + } + if (target === null) { + overflowed = true; + break; + } + target.entries.push(entry); + target.weight += entry.weight; + target.chars += entry.chars; + } + if (!overflowed) { + return bins + .filter((bin) => bin.entries.length > 0) + .map((bin) => bin.entries.sort((a, b) => a.index - b.index).map((e) => e.file)); + } + chunkCount++; + } +} + function parseArgs(argv) { let suite = null; let seen = false; @@ -559,9 +759,10 @@ function main() { // fine there. Split into chunks sized for the tightest target so behavior // is identical across platforms. (#3597) // Operator override (also used by tests to force chunking with short paths). - const MAX_CMDLINE_CHARS = process.env.RUN_TESTS_MAX_CMDLINE_CHARS - ? Number(process.env.RUN_TESTS_MAX_CMDLINE_CHARS) - : 28000; // headroom below the 32,767 Windows ceiling + const MAX_CMDLINE_CHARS = positiveNumberEnv( + process.env.RUN_TESTS_MAX_CMDLINE_CHARS, + 28000, // headroom below the 32,767 Windows ceiling + ); // A full-lane shard (~171 files) fit in ONE chunk at the old cap of 180, so the // entire shard's wall-clock ran against a single per-chunk timeout. On the slow // Windows runner the install-heavy files in a shard (e.g. install-minimal-hooks @@ -574,26 +775,31 @@ function main() { // node process (also relieving per-process memory pressure from 170+ files at once). // Lowered from 90 to 60 after #1575 — macOS Node 22 shard 2/3 chunk 2 (~80 files // including state.test.cjs, perf-*, worktree-cleanup) exceeded 600s with 90. - const MAX_FILES_PER_CHUNK = process.env.RUN_TESTS_MAX_FILES_PER_CHUNK - ? Number(process.env.RUN_TESTS_MAX_FILES_PER_CHUNK) - : 60; - // #2088: file COUNT is a poor proxy for a chunk's wall-clock — install-heavy - // files (real installs; install-minimal-hooks.test.cjs alone runs ~250 cases - // doing dozens of installs) are ~10× a unit file. When several land in the SAME - // chunk — e.g. a PR touching the whole install surface, whose *targeted* lane is - // unsharded (13 install-heavy files → one chunk) — that chunk blows the 600s - // backstop while unit-only chunks finish in seconds. WEIGHT install-heavy files - // so they fill a chunk's budget faster and therefore SPREAD across chunks - // instead of clustering. Light files keep weight 1, so pure-unit chunking (and - // its harness tests) is byte-for-byte unchanged. `MAX_FILES_PER_CHUNK` is now a - // per-chunk WEIGHT budget (backwards-compatible: it equals the file count when - // every file is light). Tune via RUN_TESTS_HEAVY_FILE_WEIGHT; classify via the - // basename prefix (install*/installer*/codex-* are the real install-heavy suites). - const HEAVY_TEST_RE = /^(?:install|codex-)/; - const HEAVY_FILE_WEIGHT = process.env.RUN_TESTS_HEAVY_FILE_WEIGHT - ? Number(process.env.RUN_TESTS_HEAVY_FILE_WEIGHT) - : 12; - const fileWeight = (f) => (HEAVY_TEST_RE.test(basename(f)) ? HEAVY_FILE_WEIGHT : 1); + const MAX_FILES_PER_CHUNK = positiveNumberEnv(process.env.RUN_TESTS_MAX_FILES_PER_CHUNK, 60); + // #2088 established that file COUNT is a poor proxy for a chunk's wall-clock: + // install-heavy files (real installs) cost ~10x a unit file, and when several + // land in the SAME chunk it blows the 600s backstop while unit-only chunks + // finish in seconds. #2088 approximated cost from the filename — basename + // matching /^(?:install|codex-)/ scored 12, everything else 1. + // + // #2456: that approximation is miscalibrated in BOTH directions, so chunks were + // still balanced by file count rather than by cost. Measured durations show + // installer-migration-authoring.test.cjs scoring 12 while running ~0.1s, and the + // two heaviest files in the whole suite scoring 1 — run-tests-harness.test.cjs + // (never matched the prefix) and release-tarball-smoke.install.test.cjs (the + // regex is anchored to the START of the basename, so a mid-name "install" never + // matches). Both landed in the same chunk, leaving the slowest chunk ~3.9x the + // lightest and sitting near the timeout. + // + // Weight each file by its MEASURED duration instead. `MAX_FILES_PER_CHUNK` + // remains the per-chunk weight budget and keeps its scale — weights are + // normalized so an average-cost file weighs 1 — so an all-uniform suite chunks + // exactly as it did before. Timings are ADVISORY, never gated: an unknown file + // falls back to the table's median weight and a missing table falls back to + // uniform weight 1, so staleness degrades chunk BALANCE gracefully instead of + // failing CI. Regenerate via `node scripts/gen-test-timings.cjs `. + const timingsPath = process.env.RUN_TESTS_TIMINGS_FILE || DEFAULT_TIMINGS_PATH; + const fileWeight = makeFileWeigher(loadTestTimings(timingsPath)); // node:test does not exit until the event loop drains. A unit test that leaks // an open handle (un-terminated Worker, un-killed child_process, ref'd timer) @@ -609,35 +815,19 @@ function main() { const forceExit = nodeMajor >= 22 && !process.env.RUN_TESTS_NO_FORCE_EXIT; const FIXED_OVERHEAD = process.execPath.length + '--test'.length + concurrency.length + (forceExit ? '--test-force-exit'.length + 1 : 0) + 8; - const chunks = []; - let current = []; - let currentLen = FIXED_OVERHEAD; - let currentWeight = 0; - for (const file of selected) { - const add = file.length + 1; // +1 for the inter-arg separator - if ( - current.length > 0 && - (currentLen + add > MAX_CMDLINE_CHARS || currentWeight >= MAX_FILES_PER_CHUNK) - ) { - chunks.push(current); - current = []; - currentLen = FIXED_OVERHEAD; - currentWeight = 0; - } - current.push(file); - currentLen += add; - currentWeight += fileWeight(file); // heavy install files count for more - } - if (current.length > 0) chunks.push(current); + const chunks = packChunks(selected, { + weightOf: fileWeight, + maxWeight: MAX_FILES_PER_CHUNK, + maxChars: MAX_CMDLINE_CHARS, + fixedOverhead: FIXED_OVERHEAD, + }); // A chunk that still hangs (a leak the backstop somehow misses, or a wedged // subprocess) must fail loudly rather than silently burn the job's wall-clock // budget until the CI runner cancels the whole job. Default 10 min per chunk: // well above a healthy chunk (~4-5 min on the windows lane) but below the 20m // job cap. Operator/test override via RUN_TESTS_CHUNK_TIMEOUT_MS. - const chunkTimeoutMs = process.env.RUN_TESTS_CHUNK_TIMEOUT_MS - ? Number(process.env.RUN_TESTS_CHUNK_TIMEOUT_MS) - : 600000; + const chunkTimeoutMs = positiveNumberEnv(process.env.RUN_TESTS_CHUNK_TIMEOUT_MS, 600000); let firstFailureExit = 0; for (let i = 0; i < chunks.length; i++) { @@ -685,4 +875,15 @@ if (require.main === module) { runMain(main); } -module.exports = { suiteOf, ensureBuiltArtifacts, ensureBuiltHooks, parseShardArg, selectShard }; +module.exports = { + suiteOf, + ensureBuiltArtifacts, + ensureBuiltHooks, + parseShardArg, + selectShard, + positiveNumberEnv, + loadTestTimings, + makeFileWeigher, + packChunks, + DEFAULT_TIMINGS_PATH, +}; diff --git a/tests/run-tests-harness.test.cjs b/tests/run-tests-harness.test.cjs index 7310ff42a..30ffc46fa 100644 --- a/tests/run-tests-harness.test.cjs +++ b/tests/run-tests-harness.test.cjs @@ -383,46 +383,78 @@ test('ambient GSD workstream vars are stripped by the runner', () => { /run-tests: chunk 1\/3 — 3 files/, `expected file-count chunking marker in stderr; STDERR:\n${r.stderr}`, ); + // #2456 changed the packer from sequential first-fit to LPT, so 7 equal-cost + // files across 3 chunks now balance {3,2,2} instead of filling greedily to + // {3,3,1}. The chunk COUNT is unchanged; only the tail is no longer starved. assert.match( r.stderr, - /run-tests: chunk 3\/3 — 1 files/, + /run-tests: chunk 3\/3 — 2 files/, `expected final file-count chunking marker in stderr; STDERR:\n${r.stderr}`, ); }); - // #2088: install-heavy files (real installs) are weighted so they never all - // land in one chunk — otherwise the unsharded targeted lane packs the whole - // install surface into a single chunk that blows the 600s per-chunk backstop - // on the slow Windows runner. - test('install-heavy files carry more weight so they SPREAD across chunks (#2088)', () => { - // 4 install-* files at weight 3 = 12 against a 6-weight cap → 2 chunks - // (2 heavy each). The whole install load is never in a single chunk. - const heavy = Array.from({ length: 4 }, (_, i) => `install-weighttest-${i}.test.cjs`); - seed(tmpDir, heavy); + // #2088: expensive files must never all land in one chunk — otherwise the + // unsharded targeted lane packs the whole install surface into a single chunk + // that blows the 600s per-chunk backstop on the slow Windows runner. #2088 + // approximated "expensive" from the filename prefix; #2456 replaced that with + // measured durations. This test carries the #2088 GUARANTEE forward onto the + // new mechanism: the cost signal is now the timings table, injected via + // RUN_TESTS_TIMINGS_FILE so the assertion does not depend on the real suite's + // (regenerable, drifting) cost profile. + function writeTimings(dir, timings) { + const p = path.join(dir, 'timings.json'); + fs.writeFileSync(p, JSON.stringify({ schema_version: 1, unit: 'ms', timings }), 'utf8'); + return p; + } + + test('expensive files SPREAD across chunks instead of clustering (#2088, #2456)', () => { + // Discriminating by construction: the three EXPENSIVE files carry names the + // old prefix heuristic scored 1, and the three TRIVIAL ones carry the + // `install-` prefix it scored 12 — i.e. exactly inverted from their real + // cost. Under the old packer this packs {2,2,1,1} (4 chunks, with two + // expensive files sharing chunk 1); under measured weights it packs + // {2,2,2}, one expensive file per chunk. The assertion below therefore + // cannot pass on the old algorithm, nor with the timings file removed. + const heavy = ['heavy-0.test.cjs', 'heavy-1.test.cjs', 'heavy-2.test.cjs']; + const trivial = ['install-cheap-0.test.cjs', 'install-cheap-1.test.cjs', 'install-cheap-2.test.cjs']; + seed(tmpDir, [...heavy, ...trivial]); + const timingsFile = writeTimings(tmpDir, { + ...Object.fromEntries(heavy.map((f) => [f, 30000])), + ...Object.fromEntries(trivial.map((f) => [f, 10])), + }); const rh = runHarness(tmpDir, [], { RUN_TESTS_MAX_CMDLINE_CHARS: '100000', - RUN_TESTS_MAX_FILES_PER_CHUNK: '6', - RUN_TESTS_HEAVY_FILE_WEIGHT: '3', + RUN_TESTS_MAX_FILES_PER_CHUNK: '2', + RUN_TESTS_TIMINGS_FILE: timingsFile, }); assert.strictEqual(rh.status, 0, `heavy: expected zero exit; STDERR:\n${rh.stderr}`); - assert.match(rh.stderr, /run-tests: chunk 1\/2 — 2 files/, `install-heavy files must split into 2 chunks; STDERR:\n${rh.stderr}`); - assert.match(rh.stderr, /run-tests: chunk 2\/2 — 2 files/, `STDERR:\n${rh.stderr}`); + for (const n of [1, 2, 3]) { + assert.match( + rh.stderr, + new RegExp(`run-tests: chunk ${n}/3 — 2 files`), + `expensive files must spread one-per-chunk across exactly 3 chunks; STDERR:\n${rh.stderr}`, + ); + } }); - test('light files of the same count stay in ONE chunk — split is weight-driven, not count-driven (#2088)', () => { - // Same file COUNT (4) but LIGHT (weight 1 each): 4 < 6 cap → a single - // chunk. Proves the split above is driven by install-heavy WEIGHT. - const light = Array.from({ length: 4 }, (_, i) => `lightweighttest-${i}.test.cjs`); - seed(tmpDir, light); + test('trivial files the old heuristic over-weighted now stay in ONE chunk (#2088, #2456)', () => { + // The other direction of the miscalibration. These four files carry the + // `install-` prefix the old heuristic scored 12, so it split them into + // FOUR single-file chunks against a 6-weight budget. Measured, they cost + // 10ms each and belong together. The absence of a split marker cannot be + // produced by the old algorithm. + const trivial = Array.from({ length: 4 }, (_, i) => `install-triv-${i}.test.cjs`); + seed(tmpDir, trivial); + const timingsFile = writeTimings(tmpDir, Object.fromEntries(trivial.map((f) => [f, 10]))); const rl = runHarness(tmpDir, [], { RUN_TESTS_MAX_CMDLINE_CHARS: '100000', RUN_TESTS_MAX_FILES_PER_CHUNK: '6', - RUN_TESTS_HEAVY_FILE_WEIGHT: '3', + RUN_TESTS_TIMINGS_FILE: timingsFile, }); assert.strictEqual(rl.status, 0, `light: expected zero exit; STDERR:\n${rl.stderr}`); // A single chunk emits NO `chunk N/M` split marker (it only prints when - // chunks.length > 1), so its absence proves the 4 light files stayed together. - assert.doesNotMatch(rl.stderr, /run-tests: chunk \d+\/\d+ — /, `4 light files must stay in one chunk (no split marker); STDERR:\n${rl.stderr}`); + // chunks.length > 1), so its absence proves the four stayed together. + assert.doesNotMatch(rl.stderr, /run-tests: chunk \d+\/\d+ — /, `4 trivial install-prefixed files must stay in one chunk; STDERR:\n${rl.stderr}`); }); }); @@ -1302,3 +1334,469 @@ describe('bug #969 C — ensureBuiltHooks populates hooks/dist before concurrent }); }); } + +// --------------------------------------------------------------------------- +// #2456 — chunk weights must reflect MEASURED cost, not a filename guess. +// +// These tests drive the packer's pure IR (`packChunks` / `makeFileWeigher` / +// `loadTestTimings`) rather than the `run-tests: chunk N/M` stderr line, so they +// assert on typed values (chunk composition, weights) instead of rendered text. +// --------------------------------------------------------------------------- + +const { + packChunks, + makeFileWeigher, + loadTestTimings, + positiveNumberEnv, + DEFAULT_TIMINGS_PATH, +} = require('../scripts/run-tests.cjs'); + +describe('chunk packing weights measured cost (#2456)', () => { + // A cost profile modelled on the real measurements in the issue. It is the + // MISCALIBRATION that matters: the pre-#2456 heuristic scored basenames + // matching /^(?:install|codex-)/ at 12 and everything else at 1, which is + // wrong in BOTH directions here — + // * run-tests-harness / release-tarball-smoke.install are the two most + // expensive files yet scored 1 (the regex is anchored to the START of the + // basename, so a mid-name "install" never matches), and + // * installer-migration-authoring / codex-declarative-reference scored 12 + // while costing almost nothing. + const MEASURED_MS = { + 'run-tests-harness.test.cjs': 150180, + 'release-tarball-smoke.install.test.cjs': 144420, + 'phase.test.cjs': 136770, + 'install-minimal-hooks.test.cjs': 136330, + 'config.test.cjs': 114420, + 'state.test.cjs': 94290, + 'commands.test.cjs': 70580, + 'init.test.cjs': 64100, + 'installer-migration-authoring.test.cjs': 90, + 'installer-migration-report.test.cjs': 120, + 'install-update-marker.test.cjs': 95, + 'codex-declarative-reference.test.cjs': 110, + }; + const FILES = Object.keys(MEASURED_MS); + const FIXED_OVERHEAD = 120; + const ROOMY_CHARS = 100000; + + function tableFrom(timings) { + const tmp = createTempDir('gsd-2456-timings-'); + const p = path.join(tmp, 'timings.json'); + fs.writeFileSync(p, JSON.stringify({ schema_version: 1, unit: 'ms', timings }), 'utf8'); + return { path: p, dir: tmp }; + } + + function packMeasured(files, maxWeight, extra = {}) { + const t = tableFrom(MEASURED_MS); + try { + return packChunks(files, { + weightOf: makeFileWeigher(loadTestTimings(t.path)), + maxWeight, + maxChars: ROOMY_CHARS, + fixedOverhead: FIXED_OVERHEAD, + ...extra, + }); + } finally { + cleanup(t.dir); + } + } + + const costOf = (chunk) => chunk.reduce((sum, f) => sum + MEASURED_MS[f], 0); + + /** + * THE REGRESSION (#2456). Under the pre-fix packer the two most expensive + * files both scored weight 1 and, being adjacent in selection order, packed + * into the SAME chunk — that chunk ran ~3.9x the lightest and sat near the + * 600s per-chunk timeout. Weighting by measured cost and packing with LPT must + * separate them. + */ + test('the two most expensive files never land in the same chunk', () => { + const chunks = packMeasured(FILES, 4); + const chunkOf = (f) => chunks.findIndex((c) => c.includes(f)); + const a = chunkOf('run-tests-harness.test.cjs'); + const b = chunkOf('release-tarball-smoke.install.test.cjs'); + assert.ok(a !== -1 && b !== -1, 'both heavy files must be packed'); + assert.notStrictEqual( + a, + b, + `the two most expensive files must be split across chunks; got both in chunk ${a + 1}. ` + + `Chunk costs (s): ${chunks.map((c) => (costOf(c) / 1000).toFixed(1)).join(', ')}`, + ); + }); + + test('chunk costs are balanced — slowest chunk stays well under 2x the fastest', () => { + const chunks = packMeasured(FILES, 4); + const costs = chunks.map(costOf); + const ratio = Math.max(...costs) / Math.min(...costs); + assert.ok( + ratio < 2, + `LPT must balance measured cost; imbalance was ${ratio.toFixed(2)}x ` + + `(costs in s: ${costs.map((c) => (c / 1000).toFixed(1)).join(', ')})`, + ); + }); + + test('weights follow measured duration, correcting the old prefix heuristic both ways', () => { + const t = tableFrom(MEASURED_MS); + try { + const weigh = makeFileWeigher(loadTestTimings(t.path)); + // Old heuristic: 1. Actual: the most expensive file in the suite. + assert.ok( + weigh('run-tests-harness.test.cjs') > 1, + 'the heaviest file must weigh above the table average', + ); + // Old heuristic: 12 (install prefix). Actual: ~0.1s. + assert.ok( + weigh('installer-migration-authoring.test.cjs') < 1, + 'a trivially cheap install-prefixed file must weigh below the table average', + ); + // The old regex is anchored to the START of the basename, so a mid-name + // "install" scored 1. Measured, it is the second most expensive file. + assert.ok( + weigh('release-tarball-smoke.install.test.cjs') + > weigh('installer-migration-authoring.test.cjs') * 100, + 'mid-name "install" must be ranked by cost, not by prefix position', + ); + } finally { + cleanup(t.dir); + } + }); + + test('an average-cost file weighs exactly 1, preserving the MAX_FILES_PER_CHUNK scale', () => { + // Three files at 10s, 20s, 30s → mean 20s. The 20s file is the average. + const t = tableFrom({ 'a.test.cjs': 10000, 'b.test.cjs': 20000, 'c.test.cjs': 30000 }); + try { + const weigh = makeFileWeigher(loadTestTimings(t.path)); + assert.strictEqual(weigh('b.test.cjs'), 1, 'the mean-cost file must weigh 1'); + assert.strictEqual(weigh('a.test.cjs'), 0.5); + assert.strictEqual(weigh('c.test.cjs'), 1.5); + } finally { + cleanup(t.dir); + } + }); + + describe('timings are advisory, never gated', () => { + test('a file missing from the table falls back to the median weight', () => { + // 10s, 20s, 60s → mean 30s, median 20s → median weight = 20/30. + const t = tableFrom({ 'a.test.cjs': 10000, 'b.test.cjs': 20000, 'c.test.cjs': 60000 }); + try { + const weigh = makeFileWeigher(loadTestTimings(t.path)); + assert.strictEqual(weigh('brand-new-test.test.cjs'), 20000 / 30000); + } finally { + cleanup(t.dir); + } + }); + + test('an unknown file packs without error rather than failing the run', () => { + const chunks = packMeasured([...FILES, 'never-measured.test.cjs'], 4); + assert.ok( + chunks.flat().includes('never-measured.test.cjs'), + 'a file absent from the timings table must still be packed', + ); + }); + + test('a missing timings file degrades to uniform weight, not an error', () => { + // No temp dir needed — the point is a path that does NOT exist. Creating + // one here would leak it (createTempDir has no registry). + const missing = path.join(__dirname, 'no-such-dir-2456', 'does-not-exist.json'); + assert.strictEqual(loadTestTimings(missing), null, 'a missing table must load as null'); + const weigh = makeFileWeigher(null); + assert.strictEqual(weigh('anything.test.cjs'), 1, 'a null table must weigh every file 1'); + }); + + test('a corrupt timings file degrades to uniform weight, not an error', () => { + const tmp = createTempDir('gsd-2456-corrupt-'); + try { + const p = path.join(tmp, 'timings.json'); + fs.writeFileSync(p, '{ this is not json', 'utf8'); + assert.strictEqual(loadTestTimings(p), null, 'unparseable JSON must load as null'); + } finally { + cleanup(tmp); + } + }); + + test('a structurally valid but empty table degrades to uniform weight', () => { + const t = tableFrom({}); + try { + assert.strictEqual(loadTestTimings(t.path), null, 'an empty table must load as null'); + } finally { + cleanup(t.dir); + } + }); + + test('chunk count never drops below what count-based packing would produce', () => { + const files = Array.from({ length: 30 }, (_, i) => `unmeasured-${i}.test.cjs`); + assert.ok( + packMeasured(files, 6).length >= Math.ceil(files.length / 6), + 'the count floor must hold for a fully-unknown file set', + ); + }); + + test('a right-skewed table cannot collapse unknown files into fat chunks', () => { + // The safety floor in its worst case. In a realistically right-skewed suite + // (many trivial files, a few very expensive ones) the median weight is far + // below 1, so weighting ALONE would pack 30 unknown files into a single + // chunk — exactly the failure mode this fix exists to prevent. The count + // floor pins the result at what the pre-#2456 packer produced. + const skewed = { + ...Object.fromEntries(Array.from({ length: 10 }, (_, i) => [`t-${i}.test.cjs`, 100])), + 'one-huge.test.cjs': 100000, + }; + const t = tableFrom(skewed); + try { + const table = loadTestTimings(t.path); + assert.ok(table.medianWeight < 0.05, 'fixture must actually be right-skewed'); + const files = Array.from({ length: 30 }, (_, i) => `unmeasured-${i}.test.cjs`); + const chunks = packChunks(files, { + weightOf: makeFileWeigher(table), + maxWeight: 6, + maxChars: ROOMY_CHARS, + fixedOverhead: FIXED_OVERHEAD, + }); + assert.strictEqual( + chunks.length, + Math.ceil(files.length / 6), + 'a right-skewed table must still chunk exactly as count-based packing did', + ); + } finally { + cleanup(t.dir); + } + }); + }); + + describe('boundary coverage', () => { + // 12 files, each weighing exactly 1 (uniform) → total weight 12. + const uniform = Array.from({ length: 12 }, (_, i) => `u-${String(i).padStart(2, '0')}.test.cjs`); + const packUniform = (maxWeight) => + packChunks(uniform, { + weightOf: () => 1, + maxWeight, + maxChars: ROOMY_CHARS, + fixedOverhead: FIXED_OVERHEAD, + }); + + test('weight budget at limit-1 forces a second chunk', () => { + assert.strictEqual(packUniform(11).length, 2, 'total weight 12 over budget 11 → 2 chunks'); + }); + + test('weight budget exactly at the limit stays in one chunk', () => { + assert.strictEqual(packUniform(12).length, 1, 'total weight 12 at budget 12 → 1 chunk'); + }); + + test('weight budget at limit+1 stays in one chunk', () => { + assert.strictEqual(packUniform(13).length, 1, 'total weight 12 under budget 13 → 1 chunk'); + }); + + // argv ceiling: two files of identical length in one chunk occupies + // fixedOverhead + 2*(len+1) chars. + const two = ['argv-boundary-aaa.test.cjs', 'argv-boundary-bbb.test.cjs']; + const exactChars = FIXED_OVERHEAD + two.reduce((sum, f) => sum + f.length + 1, 0); + const packChars = (maxChars) => + packChunks(two, { + weightOf: () => 1, + maxWeight: 1000, // never the binding constraint here + maxChars, + fixedOverhead: FIXED_OVERHEAD, + }); + + test('argv ceiling at limit-1 splits the chunk', () => { + assert.strictEqual(packChars(exactChars - 1).length, 2, 'one char short → must split'); + }); + + test('argv ceiling exactly at the limit keeps one chunk', () => { + assert.strictEqual(packChars(exactChars).length, 1, 'exactly at the ceiling → fits'); + }); + + test('argv ceiling at limit+1 keeps one chunk', () => { + assert.strictEqual(packChars(exactChars + 1).length, 1, 'one char spare → fits'); + }); + + test('a single file wider than the argv ceiling is packed alone, not dropped or looped', () => { + const chunks = packChunks(['x'.repeat(500) + '.test.cjs', 'small.test.cjs'], { + weightOf: () => 1, + maxWeight: 1000, + maxChars: FIXED_OVERHEAD + 50, // narrower than the long file alone + fixedOverhead: FIXED_OVERHEAD, + }); + assert.strictEqual(chunks.flat().length, 2, 'both files must survive packing'); + assert.strictEqual(chunks.length, 2, 'the over-long file must occupy its own chunk'); + }); + + test('an empty selection packs to no chunks', () => { + assert.deepStrictEqual( + packChunks([], { + weightOf: () => 1, + maxWeight: 60, + maxChars: ROOMY_CHARS, + fixedOverhead: FIXED_OVERHEAD, + }), + [], + ); + }); + }); + + describe('packing invariants', () => { + test('every selected file is packed exactly once', () => { + const chunks = packMeasured(FILES, 3); + const flat = chunks.flat(); + assert.strictEqual(flat.length, FILES.length, 'no file may be dropped or duplicated'); + assert.deepStrictEqual([...flat].sort(), [...FILES].sort(), 'the packed set must equal the selection'); + }); + + test('no chunk is empty', () => { + const chunks = packMeasured(FILES, 3); + for (const [i, c] of chunks.entries()) { + assert.ok(c.length > 0, `chunk ${i + 1} must not be empty`); + } + }); + + test('packing is deterministic — identical input yields byte-identical chunks', () => { + assert.deepStrictEqual(packMeasured(FILES, 4), packMeasured(FILES, 4)); + }); + + test('files keep their original selection order within a chunk', () => { + const chunks = packMeasured(FILES, 3); + for (const chunk of chunks) { + const positions = chunk.map((f) => FILES.indexOf(f)); + assert.deepStrictEqual( + positions, + [...positions].sort((a, b) => a - b), + 'within a chunk, files must stay in selection order', + ); + } + }); + }); + + describe('degenerate knobs degrade safely rather than hanging or crashing', () => { + // These knobs come from the environment via Number(), so a typo yields NaN + // and an explicit 0 yields 0. Both reach the chunk-count arithmetic. Before + // hardening, NaN spun packChunks' retry loop forever (a hung CI job with no + // output) and 0 threw `RangeError: Invalid array length` from Array.from. + // Every case below must return a valid packing instead. + const files = ['a.test.cjs', 'b.test.cjs', 'c.test.cjs']; + const packWith = (opts) => + packChunks(files, { + weightOf: () => 1, + maxWeight: 60, + maxChars: ROOMY_CHARS, + fixedOverhead: FIXED_OVERHEAD, + ...opts, + }); + const conserves = (chunks) => + JSON.stringify(chunks.flat().sort()) === JSON.stringify([...files].sort()); + + for (const [label, opts] of [ + ['a NaN weight budget', { maxWeight: NaN }], + ['a zero weight budget', { maxWeight: 0 }], + ['a negative weight budget', { maxWeight: -5 }], + ['a NaN argv ceiling', { maxChars: NaN }], + ['a zero argv ceiling', { maxChars: 0 }], + ['a NaN fixed overhead', { fixedOverhead: NaN }], + ['a weight function returning NaN', { weightOf: () => NaN }], + ['a weight function returning Infinity', { weightOf: () => Infinity }], + ['a weight function returning a negative', { weightOf: () => -1 }], + ]) { + test(`${label} still packs every file exactly once`, () => { + const chunks = packWith(opts); + assert.ok(conserves(chunks), `${label} must still pack all files; got ${JSON.stringify(chunks)}`); + }); + } + + test('a legitimate but tiny weight budget does not explode the chunk count', () => { + // positiveNumberEnv accepts any positive finite number, so 1e-9 is a valid + // budget. Without an upper clamp the packer asks for files.length / 1e-9 + // bins and throws `RangeError: Invalid array length`. More chunks than + // files is never useful, so the count clamps at one file per chunk. + const many = Array.from({ length: 200 }, (_, i) => `m-${i}.test.cjs`); + const chunks = packChunks(many, { + weightOf: () => 1, + maxWeight: 1e-9, + maxChars: ROOMY_CHARS, + fixedOverhead: FIXED_OVERHEAD, + }); + assert.strictEqual(chunks.length, many.length, 'must clamp to one file per chunk'); + assert.strictEqual(chunks.flat().length, many.length, 'no file may be dropped'); + }); + + test('a timings table from a future schema is ignored rather than mis-read', () => { + // A v2 table could change the unit or key format; consuming it under v1 + // semantics would silently mis-weight every file. Falling back to null + // (uniform weight) is the same graceful degradation as a missing table. + const tmp = createTempDir('gsd-2456-schema-'); + try { + const p2 = path.join(tmp, 'timings.json'); + fs.writeFileSync(p2, JSON.stringify({ schema_version: 2, timings: { 'a.test.cjs': 100 } }), 'utf8'); + assert.strictEqual(loadTestTimings(p2), null, 'an unknown schema_version must load as null'); + fs.writeFileSync(p2, JSON.stringify({ schema_version: 1, timings: { 'a.test.cjs': 100 } }), 'utf8'); + assert.ok(loadTestTimings(p2) !== null, 'the supported schema_version must load'); + } finally { + cleanup(tmp); + } + }); + + test('positiveNumberEnv falls back for every non-positive-finite input', () => { + for (const bad of [undefined, null, '', ' ', 'abc', '0', '-1', 'NaN', 'Infinity', '1e999']) { + assert.strictEqual( + positiveNumberEnv(bad, 60), + 60, + `${JSON.stringify(bad)} must fall back to the default`, + ); + } + }); + + test('positiveNumberEnv accepts a legitimate override', () => { + assert.strictEqual(positiveNumberEnv('3', 60), 3); + assert.strictEqual(positiveNumberEnv('0.5', 60), 0.5); + }); + + test('a key that resolves on Object.prototype still weighs a number, not a function', () => { + // The table is JSON-parsed, so a bare index would walk the prototype + // chain. Real selections are `*.test.cjs` basenames, which can never equal + // an Object.prototype member — so these BARE names are the only inputs + // that actually reach that path, and makeFileWeigher is exported, so it + // does not control its caller's strings. The contract is that any key not + // present in the table weighs the median, whatever it resolves to. + const t = tableFrom({ 'a.test.cjs': 10000, 'b.test.cjs': 20000, 'c.test.cjs': 60000 }); + try { + const weigh = makeFileWeigher(loadTestTimings(t.path)); + for (const name of ['constructor', 'toString', 'valueOf', 'hasOwnProperty', '__proto__']) { + const w = weigh(name); + assert.strictEqual(typeof w, 'number', `${name} must weigh a number, not a function`); + assert.strictEqual(w, 20000 / 30000, `${name} must fall back to the median weight`); + } + } finally { + cleanup(t.dir); + } + }); + + test('a __proto__ key in the table does not pollute Object.prototype', () => { + const t = tableFrom(JSON.parse('{"__proto__":{"polluted":true},"a.test.cjs":1000}')); + try { + makeFileWeigher(loadTestTimings(t.path))('a.test.cjs'); + assert.strictEqual({}.polluted, undefined, 'Object.prototype must not be polluted'); + } finally { + cleanup(t.dir); + } + }); + }); + + describe('the checked-in timings table', () => { + test('loads, is non-empty, and yields usable weights', () => { + const table = loadTestTimings(DEFAULT_TIMINGS_PATH); + assert.ok(table !== null, `${DEFAULT_TIMINGS_PATH} must parse into a usable timing table`); + assert.ok(Object.keys(table.timings).length > 0, 'the table must not be empty'); + assert.ok(table.mean > 0, 'the table mean must be positive'); + assert.ok(table.medianWeight > 0, 'the median fallback weight must be positive'); + }); + + // Schema guard on a static checked-in data file — NOT a timing assertion. + // Reported as a list so a corrupt regeneration names every bad entry at once + // rather than failing on the first. + test('every recorded cost is a finite non-negative number', () => { + const table = loadTestTimings(DEFAULT_TIMINGS_PATH); + const invalid = Object.entries(table.timings) + .filter(([, value]) => typeof value !== 'number' || !Number.isFinite(value) || value < 0) + .map(([file, value]) => `${file}=${value}`); + assert.deepStrictEqual(invalid, [], 'every timing-table entry must be a finite non-negative number'); + }); + }); +}); diff --git a/tests/test-timings.json b/tests/test-timings.json new file mode 100644 index 000000000..314ba8754 --- /dev/null +++ b/tests/test-timings.json @@ -0,0 +1,649 @@ +{ + "schema_version": 1, + "generated_by": "scripts/gen-test-timings.cjs", + "unit": "ms", + "sources": [ + "test-events-linux-node22.jsonl", + "test-events-linux-node24.jsonl" + ], + "file_count": 637, + "timings": { + "active-workstream-store.test.cjs": 102, + "active-workstream-store.unit.test.cjs": 332, + "adapter-declarative-equivalence.test.cjs": 31, + "adapter-imperative.test.cjs": 63, + "adr-15-progress-converge.test.cjs": 50, + "adr-218-release-version-validation.test.cjs": 62, + "adr-22-plan-drift-guard.test.cjs": 2737, + "adr-230-pr-target-policy.test.cjs": 117, + "adr-612-collision-characterization.test.cjs": 88, + "adr-index-gate.test.cjs": 4976, + "adr-parser.property.test.cjs": 1305, + "adr-parser.test.cjs": 34, + "adr-parser.unit.test.cjs": 380, + "adr857-contribution-merge.test.cjs": 90, + "adr857-core-without-capabilities.test.cjs": 4109, + "adr857-predicate-boundary.test.cjs": 100, + "affected-tests-lib.test.cjs": 175, + "agent-classification-parity.test.cjs": 92, + "agent-frontmatter.test.cjs": 526, + "agent-install-check.test.cjs": 91, + "agent-install-validation.test.cjs": 7395, + "agent-required-reading-consistency.test.cjs": 142, + "agent-size-budget.test.cjs": 111, + "agent-skills-awareness.test.cjs": 55, + "agent-skills-bootstrap.test.cjs": 315, + "agent-skills.test.cjs": 20045, + "ai-evals.test.cjs": 4637, + "allowlist-ratchet.test.cjs": 70, + "analyze-dependencies.test.cjs": 30, + "anti-pattern-enforcement.test.cjs": 59, + "antigravity-repo-grounding.test.cjs": 220, + "antigravity-reviewer.test.cjs": 58, + "antigravity-upgrades.test.cjs": 4908, + "api-coverage-gate-e2e.test.cjs": 4990, + "api-coverage.test.cjs": 597, + "artifacts.test.cjs": 246, + "ask-user-questions-fallback.test.cjs": 108, + "assumption-delta-checkpoint-e2e.test.cjs": 3823, + "assumption-delta.test.cjs": 1017, + "atomic-write-coverage.test.cjs": 72, + "audit-command-cutover.test.cjs": 4706, + "audit-fix-command.test.cjs": 159, + "augment-conversion.test.cjs": 83, + "augment-upgrades.test.cjs": 6978, + "autonomous-allowed-tools.test.cjs": 19, + "autonomous-converge.test.cjs": 41, + "autonomous-decomposition.test.cjs": 42, + "autonomous-interactive.test.cjs": 47, + "autonomous-to-flag.test.cjs": 44, + "autonomous-ui-steps.test.cjs": 74, + "backwards-compat-phase-id.test.cjs": 51, + "broken-windows.test.cjs": 9220, + "bugs-1656-1657.test.cjs": 192, + "capability-cli.test.cjs": 37059, + "capability-command-dispatch.test.cjs": 177, + "capability-consent.test.cjs": 6516, + "capability-consumption.test.cjs": 203, + "capability-ledger.test.cjs": 1050, + "capability-lifecycle.test.cjs": 5010, + "capability-loader.test.cjs": 667, + "capability-manifest-version.test.cjs": 266, + "capability-matrix-sync.test.cjs": 31, + "capability-precedence-parity.test.cjs": 94, + "capability-registry.test.cjs": 31553, + "capability-source.test.cjs": 944, + "capability-state.test.cjs": 6635, + "capability-trust.test.cjs": 232, + "capability-writer.test.cjs": 3085, + "chain-flag-plan-phase.test.cjs": 91, + "changeset-cli.test.cjs": 6838, + "changeset-github-release-notes.test.cjs": 814, + "changeset-lint.test.cjs": 1006, + "changeset-new.test.cjs": 170, + "changeset-parse.test.cjs": 122, + "changeset-render.test.cjs": 37, + "changeset-serialize.test.cjs": 87, + "check-env.test.cjs": 9927, + "check-gap-analysis-plan-post-e2e.test.cjs": 14298, + "check-glossary-refs.test.cjs": 1108, + "check-predicate.test.cjs": 306, + "check-tdd-review-checkpoint-e2e.test.cjs": 5647, + "check-ui-plan-gate.test.cjs": 127, + "check-ui-safety-gate.test.cjs": 1839, + "check-update-config-dir.test.cjs": 277, + "ci-rebase-check.test.cjs": 1643, + "ci-test-scope.test.cjs": 40081, + "cjs-command-router-adapter.test.cjs": 6540, + "claude-imperative-reference.test.cjs": 72, + "claude-md-path.test.cjs": 3670, + "claude-md.test.cjs": 7773, + "claude-orchestration-command-router.test.cjs": 2626, + "claude-orchestration.test.cjs": 464, + "claude-skills-migration.test.cjs": 214, + "cleanup-branch-pruning.test.cjs": 49, + "cli-exit.test.cjs": 719, + "cline-beforetool-upgrade.test.cjs": 66, + "cline-dispatch-degradation.test.cjs": 135, + "cline-imperative-reference.test.cjs": 101, + "cline-install.test.cjs": 3279, + "cline-model-override-upgrade.test.cjs": 51, + "cline-support.test.cjs": 40, + "clock-seam.test.cjs": 8384, + "close-phase-todos-stage-deletion.test.cjs": 24, + "clusters.test.cjs": 59, + "code-review-agent-skills.test.cjs": 32, + "code-review-command.test.cjs": 413, + "code-review-flags.test.cjs": 119, + "code-review-pipeline-regression.test.cjs": 150, + "code-review-summary-parser.test.cjs": 87, + "code-review.test.cjs": 2574, + "codebuddy-install.test.cjs": 2807, + "codebuddy-upgrades.test.cjs": 2532, + "codex-config.test.cjs": 108295, + "codex-declarative-reference.test.cjs": 110, + "command-arg-projection.test.cjs": 50, + "command-contract.test.cjs": 1023, + "command-routing-hub.test.cjs": 29430, + "commands.test.cjs": 70578, + "commit-docs-bypass.test.cjs": 2188, + "commit-files-deletion.test.cjs": 2583, + "commit-files-pathspec.test.cjs": 4154, + "concurrency-safety.test.cjs": 12250, + "config-field-docs.test.cjs": 1203, + "config-get-default.test.cjs": 18007, + "config-loader.test.cjs": 6320, + "config-schema.property.test.cjs": 5229, + "config.test.cjs": 114417, + "configuration-migrate-config.test.cjs": 1751, + "context-enrichment.test.cjs": 124, + "context-utilization.property.test.cjs": 203, + "context-utilization.test.cjs": 109, + "context7-plugin-grant-parity.test.cjs": 111, + "continuation-grammar-parity.test.cjs": 62, + "contributor-standards.test.cjs": 63, + "conventional-title.property.test.cjs": 265, + "conventional-title.test.cjs": 175, + "copilot-install.test.cjs": 44480, + "copilot-upgrades.test.cjs": 4816, + "core-utils.test.cjs": 247, + "coverage-metadata-parser.test.cjs": 10769, + "coverage-uat-routing.test.cjs": 1959, + "cross-ai-execution.test.cjs": 97, + "cursor-conversion.test.cjs": 79, + "cursor-dispatch-upgrade.test.cjs": 68, + "cursor-hook-bus-upgrade.test.cjs": 69, + "cursor-hooks.test.cjs": 407, + "cursor-imperative-reference.test.cjs": 164, + "cursor-reviewer.test.cjs": 165, + "debug-session-management.test.cjs": 105, + "debugger-bug-taxonomy.test.cjs": 103, + "debugger-fix-acceptance.test.cjs": 83, + "debugger-prevention.test.cjs": 109, + "debugger-rca-branching.test.cjs": 112, + "debugger-repro-hardening.test.cjs": 90, + "debugger-sbfl.test.cjs": 207, + "debugger-semantic-recall.test.cjs": 73, + "decisions.test.cjs": 10172, + "declarative-reference-antigravity.test.cjs": 1177, + "declarative-reference-augment.test.cjs": 1271, + "declarative-reference-codebuddy.test.cjs": 1354, + "declarative-reference-copilot.test.cjs": 1410, + "declarative-reference-windsurf.test.cjs": 1779, + "declarative-reference-zcode.test.cjs": 1756, + "defaults-json-fallback.test.cjs": 118, + "derive-progress.property.test.cjs": 207, + "discord-release-summary.test.cjs": 35, + "discuss-all-flag.test.cjs": 100, + "discuss-checkpoint.test.cjs": 69, + "discuss-mode.test.cjs": 84, + "discuss-phase-power.test.cjs": 214, + "dispatcher.test.cjs": 15150, + "docs-parity-live-registry.test.cjs": 585, + "docs-update.test.cjs": 7877, + "drift-detection.test.cjs": 8590, + "edge-probe-docs-fixtures.test.cjs": 95, + "edge-probe-planner-contract.test.cjs": 33, + "edge-probe-spec-phase-contract.test.cjs": 39, + "edge-probe.test.cjs": 1251, + "edit-phase.test.cjs": 159, + "effort-sync-installed-runtime.test.cjs": 2194, + "ensure-runtime-build.test.cjs": 6601, + "eslint-rules.test.cjs": 793, + "eval.property.test.cjs": 207, + "eval.test.cjs": 92, + "event.test.cjs": 162, + "execute-mvp-tdd-gate.test.cjs": 1420, + "execute-phase-active-flags.test.cjs": 199, + "execute-phase-step-5-5-deviation-doc.test.cjs": 321, + "execute-phase-wave.test.cjs": 1293, + "execute-phase-worktree-artifacts.test.cjs": 73, + "execute-wave-post-gate-pipeline-e2e.test.cjs": 9138, + "executor-mvp-tdd-section.test.cjs": 109, + "explore-command.test.cjs": 44, + "external-descriptor-confinement.test.cjs": 56, + "external-descriptor-loader-wiring.test.cjs": 261, + "external-job-waiting.test.cjs": 82, + "external-job.test.cjs": 179, + "extract-learnings.test.cjs": 102, + "extraction-discipline-consistency.test.cjs": 67, + "federated-config-key-removal.test.cjs": 137, + "federated-config-loadconfig.test.cjs": 915, + "federated-config.test.cjs": 206, + "few-shot-calibration.test.cjs": 68, + "fix-1941-quick-worktree-stale-base.test.cjs": 31, + "fix-2068-resolve-execution-dynamic-routing.test.cjs": 4537, + "fix-2136-clock-local-today.test.cjs": 746, + "fix-2138-ship-note-lost-on-merge.test.cjs": 27, + "fix-2194-review-timeout-guidance.test.cjs": 57, + "fix-2196-debug-agent-handoff.test.cjs": 51, + "fix-2257-debug-nonterminal-resume.test.cjs": 104, + "fix-2284-hermes-agent-delegate-task-projection.test.cjs": 7880, + "fix-2285-claude-orchestration-wiring.test.cjs": 3226, + "fix-2287-deferred-items-reader.test.cjs": 2854, + "fix-2289-context-monitor-event-allowlist.test.cjs": 4471, + "fix-2297-resolve-model-ids-runtime-scoping.test.cjs": 333, + "fix-2337-add-todo-severity.test.cjs": 29, + "fix-2358-review-temp-path-scoping.test.cjs": 117, + "fixture-builder.test.cjs": 198, + "forensics.test.cjs": 92, + "format-github-release-notes.test.cjs": 57, + "frontmatter-cli.test.cjs": 14617, + "frontmatter.property.test.cjs": 2347, + "frontmatter.test.cjs": 7786, + "frontmatter.unit.test.cjs": 438, + "gap-checker.property.test.cjs": 10443, + "gate-predicate-evaluator.test.cjs": 113, + "gates-taxonomy.test.cjs": 92, + "gemini-runtime-removed.test.cjs": 5019, + "gen-registry.test.cjs": 1299, + "getdirname-registry-derivation.test.cjs": 41, + "git-base-branch.test.cjs": 6663, + "global-config-home-fragment.test.cjs": 56, + "golden-install-parity.test.cjs": 22621, + "golden-install-tree.test.cjs": 24864, + "golden-parity-single-source.test.cjs": 56, + "graphify-auto-update.slow.test.cjs": 31613, + "graphify-command-cutover.test.cjs": 10275, + "graphify-graph-path.test.cjs": 681, + "graphify-query.test.cjs": 1101, + "graphify-visualization.test.cjs": 3720, + "graphify.test.cjs": 1175, + "gsd-check-update-worker-platform-gate.test.cjs": 1382, + "gsd-mcp-server-bin.test.cjs": 451, + "gsd-mcp-server.test.cjs": 1033, + "gsd-researcher-app-aware.test.cjs": 79, + "gsd-researcher-flow-diagram.test.cjs": 101, + "gsd-settings-advanced.test.cjs": 2641, + "gsd-statusline-git.property.test.cjs": 1600, + "gsd-statusline-state.property.test.cjs": 223, + "gsd-statusline.test.cjs": 6310, + "gsd-tools-path-refs.test.cjs": 298, + "gsd-workflow.structure.test.cjs": 128, + "gsd2-import.test.cjs": 3459, + "handshake-serialized.test.cjs": 93, + "hardcoded-paths.test.cjs": 497, + "health-validation.test.cjs": 13246, + "helpers-cleanup.test.cjs": 89, + "helpers-process-isolation.test.cjs": 38, + "hermes-dispatch-upgrade.test.cjs": 75, + "hermes-imperative-reference.test.cjs": 169, + "hermes-skills-migration.test.cjs": 139, + "hook-bus.test.cjs": 65, + "hook-validation.test.cjs": 87, + "hooks-opt-in.test.cjs": 6422, + "host-integration-descriptors.test.cjs": 310, + "host-integration-validator-parity.test.cjs": 150, + "host-integration.test.cjs": 343, + "hub-logger-integration.test.cjs": 293, + "import-command.test.cjs": 207, + "ingest-docs.test.cjs": 3456, + "init-manager.test.cjs": 20615, + "init.test.cjs": 64096, + "injection-blocking-config.test.cjs": 661, + "inline-plan-threshold.test.cjs": 815, + "install-minimal-hooks.test.cjs": 136334, + "install-nested-layout.test.cjs": 1852, + "install-path-detection.test.cjs": 5261, + "install-regressions.test.cjs": 30044, + "install-runtime-artifacts.test.cjs": 30370, + "install-update-marker.test.cjs": 137, + "install-write-confinement.test.cjs": 5239, + "install.test.cjs": 114366, + "installer-migration-authoring.test.cjs": 90, + "installer-migration-install.integration.test.cjs": 55927, + "installer-migration-prune-stale-pristine.test.cjs": 118, + "installer-migration-rename-gsd-core.test.cjs": 145, + "installer-migration-report.test.cjs": 273, + "installer-migrations.test.cjs": 4798, + "intel-command-cutover.test.cjs": 6280, + "intel.test.cjs": 6372, + "inventory-headings-countfree.test.cjs": 51, + "inventory-manifest-sync.test.cjs": 42, + "io.test.cjs": 2003, + "ios-scaffold-safety.test.cjs": 97, + "issue-1575-agent-descriptor-parity.test.cjs": 2781, + "issue-1855-marketplace-manifest.test.cjs": 146, + "issue-2045-third-party-skills-surface.test.cjs": 2049, + "issue-2517-runtime-aware-profiles.test.cjs": 1598, + "issue-2639-codex-toml-neutralization.test.cjs": 47, + "issue-429-comment-text-gate.test.cjs": 1232, + "issue-498-identity-drift-lint.test.cjs": 211, + "issue-498-package-identity.test.cjs": 99, + "issue-498-update-backup-runtime-dir.test.cjs": 50, + "issue-498-update-context.test.cjs": 791, + "issue-57-runtime-install-no-drift.test.cjs": 148, + "issue-607-cache-lineage.test.cjs": 109, + "issue-607-installer-dry-run.install.test.cjs": 866, + "issue-607-legacy-cleanup.test.cjs": 181, + "issue-69-surface-keeps-nested.test.cjs": 463, + "issue-766-plugin-manifest.test.cjs": 442, + "issue-787-cline-hooks-agents.test.cjs": 7503, + "issue-815-update-next-channel.test.cjs": 125, + "issue-844-manifest-version-sync.test.cjs": 290, + "issue-dedupe.test.cjs": 303, + "issue-version-gate.test.cjs": 238, + "kilo-imperative-reference.test.cjs": 168, + "kilo-upgrades.test.cjs": 11403, + "kimi-agent-converter.test.cjs": 7497, + "kimi-imperative-reference.test.cjs": 95, + "kimi-skill-converter.test.cjs": 44, + "kimi-tool-mapping.test.cjs": 77, + "kimi-upgrades.test.cjs": 7445, + "learnings.test.cjs": 1921, + "lint-docs-required.test.cjs": 127, + "lint-legacy-dir-name.test.cjs": 957, + "lint-pr-check-project-dir.test.cjs": 520, + "lint-regression-test-names.test.cjs": 2172, + "lint-resolution-provenance.test.cjs": 91, + "lint-skill-deps.test.cjs": 1579, + "lint-test-file-count.test.cjs": 1486, + "list-seeds.property.test.cjs": 184, + "list-seeds.test.cjs": 8729, + "locking-bugs-1909-1916-1925-1927.test.cjs": 3269, + "logger.test.cjs": 146, + "loop-hook-firing-spike.test.cjs": 70, + "loop-hooks-empty-points-e2e.test.cjs": 11918, + "loop-hooks-ship-pre-e2e.test.cjs": 3698, + "loop-hooks-verify-post-e2e.test.cjs": 5985, + "loop-host-contract.test.cjs": 246, + "loop-render-hooks.test.cjs": 6447, + "m8-writestatemd-scan-after-lock.test.cjs": 58, + "m9-statelock-write-error-orphan.test.cjs": 40, + "managed-hooks.test.cjs": 10224, + "markdown-sectionizer.test.cjs": 2010, + "markdown-table.test.cjs": 313, + "mcp-tool-inheritance.test.cjs": 217, + "mempalace-capture-headless-invocation.test.cjs": 50, + "methodology-artifact.test.cjs": 85, + "milestone-archive.test.cjs": 8439, + "milestone-helper.test.cjs": 70, + "milestone-prefixed-convention.test.cjs": 3208, + "milestone-summary.test.cjs": 1646, + "milestone.test.cjs": 30666, + "model-adapter.test.cjs": 31, + "model-alias-map.test.cjs": 25, + "model-catalog-runtime-defaults.test.cjs": 74, + "model-catalog-valid-tiers.test.cjs": 58, + "model-profiles.test.cjs": 2200, + "model-resolver.test.cjs": 35231, + "model-routing-spawn-threading.test.cjs": 235, + "multi-runtime-select.test.cjs": 97, + "mutation-matrix-ratchet.test.cjs": 373, + "mutation-matrix-stdin-eagain.test.cjs": 35, + "mvp-phase-command.test.cjs": 61, + "mvp-phase-integration.test.cjs": 2508, + "mvp-phase-spidr.test.cjs": 32, + "new-milestone-clear-phases.test.cjs": 10113, + "new-project-mvp-prompt.test.cjs": 103, + "next-decimal-roadmap-scan.test.cjs": 4986, + "next-safety-gates.test.cjs": 97, + "next-up-clear-order.test.cjs": 223, + "no-bare-npm-exec.rule.test.cjs": 262, + "no-crlf-fragile-split.rule.test.cjs": 361, + "no-dead-sdk-refs.test.cjs": 155, + "no-hardcoded-home-gsd-tools.test.cjs": 285, + "no-hardcoded-tmp.rule.test.cjs": 369, + "no-path-literal-in-assert.rule.test.cjs": 588, + "no-phantom-issue-refs.test.cjs": 890, + "no-posix-mode-bit-assert.rule.test.cjs": 384, + "no-unguarded-nonportable-exec.rule.test.cjs": 670, + "non-claude-runtimes-registry-derivation.test.cjs": 57, + "normalize-path-in-content.rule.test.cjs": 1379, + "normalize-test-command.test.cjs": 257, + "npm-integrity-gate.test.cjs": 4285, + "onboard-command.test.cjs": 10665, + "opencode-command-dir-plural.test.cjs": 14645, + "opencode-imperative-reference.test.cjs": 71, + "opencode-permissions.test.cjs": 42, + "opencode-plugin-adapter.test.cjs": 2776, + "opencode-review-reconstruction.property.test.cjs": 35, + "orphan-worktree-detection.test.cjs": 1618, + "orphaned-hooks.test.cjs": 70, + "package-legitimacy-gate.test.cjs": 378, + "package-legitimacy.property.test.cjs": 167, + "package-legitimacy.test.cjs": 294, + "package-manifest.test.cjs": 59, + "package-name-single-source.test.cjs": 143, + "parallel-dependent-plans.test.cjs": 100, + "path-replacement.test.cjs": 347, + "pattern-mapper.test.cjs": 3386, + "pause-work-improvements.test.cjs": 337, + "perf-315-loadconfig-subrepo-scan.test.cjs": 46, + "perf-316-state-lock-buffer-alloc.test.cjs": 516, + "perf-317-context-monitor-fs.test.cjs": 5249, + "perf-407-planning-lock-buffer-alloc.test.cjs": 80, + "phase-command-router.test.cjs": 2814, + "phase-dependency-levels.test.cjs": 86, + "phase-id-drift-guard.test.cjs": 505, + "phase-id.test.cjs": 500, + "phase-lifecycle.test.cjs": 105, + "phase-locator.test.cjs": 239, + "phase.test.cjs": 136772, + "phase6-capability-docs.test.cjs": 71, + "phase6-capstone-conformance.test.cjs": 3569, + "phase6-planning-capabilities.test.cjs": 60, + "phase6-review-capabilities.test.cjs": 135, + "phases-command-router.test.cjs": 1234, + "pi-extension-reachability.test.cjs": 1653, + "pi-imperative-reference.test.cjs": 48, + "pi-upgrades.test.cjs": 114, + "pick-flag.test.cjs": 4302, + "plan-bounce.test.cjs": 1329, + "plan-phase-drift-guard.test.cjs": 631, + "plan-phase-mvp-flag.test.cjs": 1426, + "plan-phase-ui-redirect.test.cjs": 232, + "plan-pre-hook-e2e.test.cjs": 7778, + "plan-review-convergence.test.cjs": 1818, + "planner-decomposition.test.cjs": 189, + "planner-language-regression.test.cjs": 1075, + "planner-mvp-mode.test.cjs": 194, + "planning-workspace.test.cjs": 6988, + "platform-guard.unit.test.cjs": 693, + "playwright-ui-verify.test.cjs": 68, + "policy-138-nyquist-config-default.test.cjs": 33, + "policy-160-route0-resume.test.cjs": 102, + "policy-lint-shallow-checkout.test.cjs": 67, + "policy-release-no-npm-self-upgrade.test.cjs": 62, + "policy-shell-pinning.test.cjs": 396, + "portability-rule-disable-ban.test.cjs": 24574, + "portability-vocab-drift.test.cjs": 1546, + "post-planning-gaps-2493.test.cjs": 21648, + "pr-template-policy.test.cjs": 175, + "precommit-alias-drift-hook.test.cjs": 352, + "precondition-element.test.cjs": 1141, + "prepush-enterprise-email-hook.test.cjs": 178, + "probe-core.property.test.cjs": 325, + "probe-core.test.cjs": 252, + "product-name-purity.test.cjs": 69, + "profile-output.test.cjs": 10693, + "profile-pipeline.test.cjs": 4712, + "progress-forensic.test.cjs": 432, + "progress-mvp-display.test.cjs": 50, + "prohibition-enforcement.property.test.cjs": 291, + "prohibition-enforcement.test.cjs": 54884, + "prohibition-probe.docs-fixtures.test.cjs": 82, + "prohibition-probe.planner-contract.test.cjs": 25, + "prohibition-probe.schema.test.cjs": 140, + "prohibition-probe.spec-phase-contract.test.cjs": 56, + "prohibition-probe.validators.test.cjs": 37, + "prohibition-probe.verify-tier.test.cjs": 620, + "project-instruction-file-parity.test.cjs": 5145, + "project-root.test.cjs": 114, + "prompt-budget-cli.test.cjs": 2758, + "prompt-budget.property.test.cjs": 1749, + "prompt-budget.test.cjs": 68, + "prompt-budget.unit.test.cjs": 408, + "prompt-injection-scan.security.test.cjs": 1318, + "prompt-thinning.test.cjs": 356, + "prune-orphaned-worktrees.test.cjs": 1375, + "quick-branching.test.cjs": 1315, + "quick-commit-boundary.test.cjs": 67, + "quick-research.test.cjs": 3543, + "quick-session-management.test.cjs": 49, + "qwen-imperative-reference.test.cjs": 101, + "qwen-skills-migration.test.cjs": 534, + "qwen-upgrades.test.cjs": 4761, + "reachability-check.test.cjs": 55, + "read-guard.test.cjs": 2465, + "read-injection-scanner.property.test.cjs": 20842, + "read-injection-scanner.security.test.cjs": 4662, + "reapply-patches.test.cjs": 178, + "reapply-verify-hunks.test.cjs": 4848, + "redaction.test.cjs": 61, + "refactor-1390-t3-characterization.test.cjs": 138, + "registry-schema.test.cjs": 378, + "release-coverage-scope.test.cjs": 31, + "release-finalize-syncs-next-version.test.cjs": 55, + "release-tarball-smoke-bin-invocation.test.cjs": 35, + "release-tarball-smoke-workflow.test.cjs": 59, + "release-tarball-smoke.install.test.cjs": 144419, + "repo-invariants.test.cjs": 21850, + "repo-layout.test.cjs": 101, + "representative-corpus.test.cjs": 4322, + "require-fs-op-fallback.rule.test.cjs": 430, + "require-userprofile-with-home.rule.test.cjs": 205, + "research-agent-profiles.test.cjs": 316, + "research-cli.test.cjs": 11909, + "research-provider.property.test.cjs": 262, + "research-provider.test.cjs": 144, + "research-store.property.test.cjs": 326, + "research-store.test.cjs": 405, + "resolution.test.cjs": 37, + "review-default-reviewers-config.test.cjs": 6391, + "review-default-reviewers-resolution.test.cjs": 81, + "review-default-reviewers-workflow.test.cjs": 128, + "review-model-config.test.cjs": 9411, + "review-reviewer-instances-config.test.cjs": 1998, + "review-reviewer-instances.test.cjs": 84, + "review-reviewer-selection.test.cjs": 61, + "roadmap-command-router.test.cjs": 77, + "roadmap-mode-field.test.cjs": 2558, + "roadmap-parser.test.cjs": 33080, + "roadmap-phase-fallback.test.cjs": 3736, + "roadmap-upgrade.test.cjs": 957, + "roadmap.test.cjs": 55342, + "roadmapper-granularity.test.cjs": 5203, + "run-tests-harness.test.cjs": 161263, + "run-with-timeout.test.cjs": 18008, + "runtime-artifact-install-plan.test.cjs": 142, + "runtime-artifact-layout-descriptor-drive.test.cjs": 322, + "runtime-artifact-layout-install-profiles.test.cjs": 2604, + "runtime-artifact-layout-surface.test.cjs": 1754, + "runtime-artifact-layout.test.cjs": 3980, + "runtime-config-adapter-registry.test.cjs": 196, + "runtime-converters.test.cjs": 893, + "runtime-flags.test.cjs": 62, + "runtime-homes-descriptor-drive.test.cjs": 473, + "runtime-homes.property.test.cjs": 501, + "runtime-label-policy.test.cjs": 51, + "runtime-launcher-parity.test.cjs": 3251, + "runtime-name-policy.test.cjs": 255, + "scan-command.test.cjs": 91, + "schema-drift.test.cjs": 6347, + "sdk-removal-query-family-dispatch.test.cjs": 3029, + "sdk-smoke.test.cjs": 42, + "secret-scan-lint.security.test.cjs": 4752, + "secure-phase-single-writer.test.cjs": 52, + "secure-phase.test.cjs": 261, + "security-dead-exports.regression.test.cjs": 112, + "security-prompt-injection.security.test.cjs": 9359, + "security-scan.security.test.cjs": 11654, + "security.test.cjs": 1956, + "seed-scan-new-milestone.test.cjs": 54, + "semver-compare.test.cjs": 168, + "settings-integrations.test.cjs": 13052, + "settings-jsonc.test.cjs": 149, + "sh-hook-paths.test.cjs": 195, + "shell-command-projection-dispatch.test.cjs": 3711, + "shell-command-projection-path-sep.test.cjs": 173, + "skill-frontmatter-contract.test.cjs": 872, + "skill-manifest.test.cjs": 4312, + "slash-command-namespace.test.cjs": 4520, + "slurm-adapter.test.cjs": 112, + "smart-entry.unit.test.cjs": 3312, + "spawn-liveness-banner.test.cjs": 160, + "spec-section.test.cjs": 606, + "stale-bake-guard.test.cjs": 376, + "state-acquirestatelock-non-eexist.test.cjs": 126, + "state-command-cutover.test.cjs": 1467, + "state-io.test.cjs": 72, + "state-prune.test.cjs": 6401, + "state-rebuild-cli.test.cjs": 3801, + "state-rebuild.test.cjs": 85, + "state-transition.test.cjs": 455, + "state.test.cjs": 116217, + "stats-mvp-display.test.cjs": 1082, + "subagent-timeout.test.cjs": 6840, + "surface-empty-manifest-agents.test.cjs": 119, + "surface-md-paths.regression.test.cjs": 38, + "sync-next-version.test.cjs": 148, + "table-schema-drift-lint.test.cjs": 789, + "tdd-mode.test.cjs": 8685, + "teams-status.test.cjs": 3434, + "temp-subdir.test.cjs": 93, + "template.test.cjs": 5110, + "test-failure-reasons.test.cjs": 47, + "test-gate-watch-mode.test.cjs": 198, + "thinking-model-guidance.test.cjs": 296, + "thinking-partner.test.cjs": 716, + "thread-session-management.test.cjs": 114, + "trace-correlation.test.cjs": 74, + "tracer-bullet.test.cjs": 2778, + "trae-imperative-reference.test.cjs": 66, + "trae-upgrades.test.cjs": 4593, + "transition-verification-gate.test.cjs": 38, + "tsconfig-noemit.test.cjs": 6383, + "uat-predicate.test.cjs": 668, + "uat.test.cjs": 15879, + "ui-checker-adversarial-stance.test.cjs": 26, + "ui-consideration-probe-docs-fixtures.test.cjs": 88, + "ui-consideration-probe.test.cjs": 347, + "ui-review-next-guidance.test.cjs": 65, + "ui-safety-gate.test.cjs": 78, + "ultraplan-phase.test.cjs": 169, + "untrusted-input-isolation.test.cjs": 706, + "update-custom-backup.test.cjs": 8333, + "update-size-baseline.test.cjs": 115, + "validate-context.test.cjs": 4346, + "validate-registry.test.cjs": 1266, + "verification-overrides.test.cjs": 136, + "verification-status.test.cjs": 614, + "verifier-behavior-unverified.test.cjs": 116, + "verifier-deferred-items.test.cjs": 92, + "verifier-mvp-section.test.cjs": 72, + "verifier-spotcheck-test-discipline.test.cjs": 139, + "verify-health.test.cjs": 34751, + "verify-mvp-uat.test.cjs": 63, + "verify-npm-publish.test.cjs": 100, + "verify-test-quality.test.cjs": 532, + "verify-work-auto-transition.test.cjs": 78, + "verify.test.cjs": 26790, + "vscode-browser-no-node-api.test.cjs": 36, + "vscode-extension-reachability.test.cjs": 1838, + "vscode-ide-reference.test.cjs": 155, + "vscode-lm-tools.test.cjs": 1616, + "vscode-subagent-dispatch.test.cjs": 1866, + "windows-robustness.test.cjs": 234, + "windsurf-conversion.test.cjs": 133, + "windsurf-hooks-bridge.test.cjs": 5096, + "windsurf-install.test.cjs": 47, + "workflow-compat.test.cjs": 386, + "workflow-guard-registration.test.cjs": 375, + "workflow-maintainer-skip.test.cjs": 142, + "workflow-shell-pinning.test.cjs": 101, + "workflow-size-budget.test.cjs": 473, + "workflow-size.test.cjs": 196, + "workflow-step-tag-balance.test.cjs": 336, + "workspace.test.cjs": 7399, + "workstream-inventory.test.cjs": 73, + "workstream-name-policy.test.cjs": 61, + "workstream.test.cjs": 40951, + "worktree-base-ref.test.cjs": 202, + "worktree-baseref-install.test.cjs": 11950, + "worktree-cleanup.test.cjs": 1901, + "worktree-safety.test.cjs": 14332, + "worktree.test.cjs": 1409, + "write-set.test.cjs": 172 + } +}