fix(#4012): a killed chunk now names the file that was hanging

The per-chunk timeout fired correctly but reported almost nothing, so every
diagnosis cost a CI round-trip. run-tests.cjs logged chunk START only — no
timestamp, no duration, no end line — then on a kill printed all ~55 basenames
and asked the operator to work out whether output kept flowing (slow) or
stopped early (hang). It could not name the in-flight file because the child is
spawned with stdio inherit, deliberately, per #3597/#1051.

Three additions.

Per-chunk elapsed timing on every path, not just failures, so drift toward the
cap is visible before it becomes a kill. Every timing number in the
investigation behind this had to be reconstructed by hand from GitHub log
timestamps.

A second, machine-readable reporter running ALONGSIDE the human one, writing
NDJSON to its own file. On a kill that file is read back and the files with a
test:start and no matching completion are named, with the staleness of the last
event, so "stopped 480s ago at X" reads differently from "still emitting at
kill". stdio stays inherit and nothing is piped or tee'd — the maxBuffer and
live-output risks that shaped the original design are untouched.

Ranking of the killed chunk's files by known weight, flagging any absent from
tests/test-timings.json, since an unweighted file is an unknown quantity.

Two details that are correct rather than lucky. Passing --test-reporter at all
replaces node's implicit default, so the human reporter is now named explicitly
and reproduces node's own selection (spec on a TTY, tap otherwise) — visible
output is unchanged. And the destination path's chunk index is zero-padded to a
fixed width because FIXED_OVERHEAD is computed ONCE before chunking; a
variable-length path would have silently mis-accounted the Windows 32,767-char
argv ceiling and reintroduced #3597. The reporter flags are added to
FIXED_OVERHEAD exactly as --test-force-exit is.

The multi-reporter pairing and the stream.compose reporter contract were
confirmed against Node's v24 documentation, not recalled — the first draft
carried them as an unverified assumption and said so.

Also corrects a stale comment claiming the 600s cap sits "below the 20m job
cap". The lane is sharded 3x at timeout-minutes: 45; the windows shards were at
19m when chunk 1/5 was killed on b351c83e0 and c3e667df3. The per-chunk cap is
now the binding constraint, and the old silent-cancel model leads to the wrong
conclusion.

Verification runs on the remote runner.

Closes #4012
This commit is contained in:
sim
2026-08-28 18:50:55 -04:00
parent dd4f179672
commit 8497833a15
4 changed files with 317 additions and 10 deletions

View File

@@ -814,6 +814,95 @@ test('noop', () => {});
);
});
});
describe('chunk-timeout instrumentation (#3889)', () => {
// T2: per-chunk elapsed timing appears on the normal SUCCESS path, not
// only when something goes wrong — this is what makes "which chunk is
// drifting toward the cap" readable across ordinary green runs.
test('a successful chunk prints its elapsed time', () => {
seed(tmpDir, ['a.test.cjs']);
const r = runHarness(tmpDir, []);
assert.strictEqual(r.status, 0, `expected a clean pass; STDERR:\n${r.stderr}`);
assert.match(
r.stderr,
/run-tests: chunk 1\/1 completed in \d+ms/,
`expected a per-chunk completion timing line; STDERR:\n${r.stderr}`,
);
});
// T3: the ndjson companion reporter's destination file is a temp
// artifact of the instrumentation, not a product output — it must not
// survive a successful run. Assert against the OS temp root's own
// "gsd-run-tests-events-*" prefix (scripts/run-tests.cjs's mkdtemp
// prefix) rather than any run-tests-owned directory, since that IS the
// leak surface being guarded.
test('the ndjson reporter temp dir is cleaned up after a successful run', () => {
seed(tmpDir, ['a.test.cjs']);
const before = fs.readdirSync(require('os').tmpdir())
.filter((n) => n.startsWith('gsd-run-tests-events-'));
const r = runHarness(tmpDir, []);
assert.strictEqual(r.status, 0, `expected a clean pass; STDERR:\n${r.stderr}`);
const after = fs.readdirSync(require('os').tmpdir())
.filter((n) => n.startsWith('gsd-run-tests-events-'));
assert.deepStrictEqual(
after,
before,
`expected no leaked gsd-run-tests-events-* temp dir after a successful run; ` +
`before=${JSON.stringify(before)} after=${JSON.stringify(after)}`,
);
});
// T1: on a chunk timeout, the diagnostic must NAME the file that was
// still executing — not merely list every file the chunk contained (the
// pre-instrumentation behavior). A test that hangs INSIDE its own body
// (never resolving) keeps its test:start event unmatched by any
// test:pass/test:fail in the ndjson companion reporter's output, which
// is exactly the signal the diagnostic reads back on timeout.
const HANGS_FOREVER_BODY = `'use strict';
const { test } = require('node:test');
test('hangs forever', () => new Promise(() => {}));
`;
test('a chunk timeout names the file that was in flight when killed', () => {
fs.writeFileSync(path.join(tmpDir, 'hangs.test.cjs'), HANGS_FOREVER_BODY, 'utf8');
const r = runHarness(tmpDir, [], {
RUN_TESTS_NO_FORCE_EXIT: '1',
RUN_TESTS_CHUNK_TIMEOUT_MS: '2000',
});
assert.notStrictEqual(
r.status,
0,
`expected non-zero exit from a timed-out chunk; got status=${r.status}\nSTDERR:\n${r.stderr}`,
);
assert.match(
r.stderr,
/In flight when killed.*hangs\.test\.cjs/s,
`expected the diagnostic to NAME the in-flight file, not just list the chunk; STDERR:\n${r.stderr}`,
);
});
// T4: the pre-existing timeout / abort / force-exit behavior (#1051)
// still holds with the reporter instrumentation wired in — the new
// --test-reporter flags must not change detection, the abort-on-timeout
// control flow, or the exit code.
test('existing timeout diagnostic and abort behavior are unchanged', () => {
fs.writeFileSync(path.join(tmpDir, 'hangs.test.cjs'), HANGS_FOREVER_BODY, 'utf8');
const r = runHarness(tmpDir, [], {
RUN_TESTS_NO_FORCE_EXIT: '1',
RUN_TESTS_CHUNK_TIMEOUT_MS: '2000',
});
assert.notStrictEqual(r.status, 0, `expected non-zero exit; STDERR:\n${r.stderr}`);
assert.match(
r.stderr,
/exceeded the per-chunk timeout/,
`expected the original timeout diagnostic wording to survive; STDERR:\n${r.stderr}`,
);
assert.match(
r.stderr,
/run-tests: chunk 1\/1 was killed after \d+ms/,
`expected the new killed/elapsed line; STDERR:\n${r.stderr}`,
);
});
});
});
// Pure partition contract for the shard selector (#1212). Imported directly