golem15/msd-core
3
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

fix(#1051): force-exit + per-chunk timeout for the windows full-test lane; close leaked test handles (#1054)

Browse Source 
The `full test (windows-latest, 22)` job intermittently got CANCELLED at its
20m wall-clock cap with no failed test step — a false-negative gate (recurrence
of #869). Root cause: a unit test leaves an open event-loop handle, so the
chunk's `node --test` child hangs ~150s on Windows after its last test prints;
two such stalls push the already-~13m job past 20m.

Fix (defense in depth):
- run-tests.cjs: pass --test-force-exit (Node >=22; engines requires >=22.0.0)
  so the runner exits once all tests finish regardless of lingering handles —
  the durable backstop. Account for the flag in the argv-length ceiling.
- run-tests.cjs: add a per-chunk execFileSync timeout (default 600000ms, env
  RUN_TESTS_CHUNK_TIMEOUT_MS) that fails loudly with a diagnostic naming the
  chunk's files, so a hung chunk can never silently eat the job budget.
- perf-316 test: terminate both Worker threads on all paths (afterEach +
  finally) so they cannot outlive the test.
- locking-bugs test: kill spawned children in a finally that wraps the whole
  spawn -> waitFor -> barrier-release -> Promise.all sequence, so a barrier
  timeout no longer leaks live child processes.
- Refresh the stale synckit comment (synckit/SDK bridge was removed).

Regression tests in run-tests-harness: a hung chunk hits the per-chunk timeout
and fails with a clear message; force-exit lets a chunk with a leaked handle
exit cleanly.

Closes #1051

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-06-11 14:11:50 -04:00
committed by GitHub
parent 58ed55683e
commit 4698b3e349
4 changed files with 181 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files

67
scripts/run-tests.cjs
View File

@@ -319,14 +319,13 @@ function main() {
// Default concurrency: 4 on Linux/macOS, 2 on Windows.
//
// Windows has significantly higher per-subprocess overhead than Linux/macOS:
// - Windows Defender scans each spawned process
// - NTFS has higher file-system latency under concurrent access
// - synckit worker_threads (used by the SDK bridge in gsd-tools.cjs) spawn
// native threads that contend on SharedArrayBuffer + Atomics.wait; under
// Node 24 on Windows, 4-way concurrent gsd-tools invocations (each spawning
// a synckit worker) caused intermittent process crashes with empty stderr —
// a signature of OS-level resource exhaustion killing worker threads before
// they could flush. Reducing to 2 halves the peak concurrent worker count.
// - Windows Defender scans each spawned process on first execution, adding
// latency proportional to the number of concurrent spawns.
// - NTFS has higher file-system latency under concurrent access compared to
// ext4/APFS, which amplifies contention when multiple test chunks run in
// parallel and all read/write the same fixture directories.
// Reducing to 2 halves the peak concurrent subprocess count on Windows and
// keeps per-chunk wall-clock time well within the 20m CI job cap.
//
// Operator override via TEST_CONCURRENCY env var for local debugging.
const defaultConcurrency = process.platform === 'win32' ? 2 : 4;
@@ -343,7 +342,21 @@ function main() {
const MAX_CMDLINE_CHARS = process.env.RUN_TESTS_MAX_CMDLINE_CHARS
? Number(process.env.RUN_TESTS_MAX_CMDLINE_CHARS)
: 28000; // headroom below the 32,767 Windows ceiling
const FIXED_OVERHEAD = process.execPath.length + '--test'.length + concurrency.length + 8;
// node:test does not exit until the event loop drains. A unit test that leaks
// an open handle (un-terminated Worker, un-killed child_process, ref'd timer)
// makes a chunk's `node --test` child hang ~150s on Windows AFTER its last test
// prints; two such stalls push the windows full lane past its 20m cap and the
// job is CANCELLED with no failed step — a false-negative gate (#1051, recurrence
// of #869). --test-force-exit (Node >=22; engines requires >=22.0.0) exits the
// runner once all tests finish regardless of lingering handles. The leaking
// tests are also fixed at the source; this is the defensive backstop.
// RUN_TESTS_NO_FORCE_EXIT=1 disables it (used by the harness regression test to
// observe the pre-fix hang).
const nodeMajor = Number(process.versions.node.split('.')[0]);
const forceExit = nodeMajor >= 22 && !process.env.RUN_TESTS_NO_FORCE_EXIT;
const FIXED_OVERHEAD = process.execPath.length + '--test'.length + concurrency.length + (forceExit ? '--test-force-exit'.length + 1 : 0) + 8;
const chunks = [];
let current = [];
let currentLen = FIXED_OVERHEAD;
@@ -359,17 +372,45 @@ function main() {
}
if (current.length > 0) chunks.push(current);
// A chunk that still hangs (a leak the backstop somehow misses, or a wedged
// subprocess) must fail loudly rather than silently burn the job's wall-clock
// budget until the CI runner cancels the whole job. Default 10 min per chunk:
// well above a healthy chunk (~4-5 min on the windows lane) but below the 20m
// job cap. Operator/test override via RUN_TESTS_CHUNK_TIMEOUT_MS.
const chunkTimeoutMs = process.env.RUN_TESTS_CHUNK_TIMEOUT_MS
? Number(process.env.RUN_TESTS_CHUNK_TIMEOUT_MS)
: 600000;
let firstFailureExit = 0;
for (let i = 0; i < chunks.length; i++) {
if (chunks.length > 1) {
console.error(`run-tests: chunk ${i + 1}/${chunks.length} — ${chunks[i].length} files`);
}
try {
execFileSync(process.execPath, ['--test', concurrency, ...chunks[i]], {
stdio: 'inherit',
env: { ...process.env },
});
execFileSync(
process.execPath,
['--test', ...(forceExit ? ['--test-force-exit'] : []), concurrency, ...chunks[i]],
{
stdio: 'inherit',
env: { ...process.env },
timeout: chunkTimeoutMs,
},
);
} catch (err) {
// When the per-chunk timeout fires, execFileSync kills the child and
// surfaces it as err.code === 'ETIMEDOUT' (POSIX) and/or err.killed === true
// (platform-dependent). Check both so detection holds on Windows and POSIX.
const timedOut = err.killed === true || err.code === 'ETIMEDOUT';
if (timedOut) {
console.error(
`run-tests: chunk ${i + 1}/${chunks.length} exceeded the per-chunk timeout ` +
`of ${chunkTimeoutMs}ms and was killed — a test in this chunk is likely leaking ` +
`an open handle (un-terminated Worker, un-killed child process, or ref'd timer) ` +
`so node --test never exits. Files: ${chunks[i]
.map(f => f.split(/[\\/]/).pop())
.join(' ')}`,
);
}
const code = err.status || 1;
// Run every chunk so the operator sees all failures in one pass; report
// the first non-zero exit at the end.