The remote run proved the instrumentation did not work. Timing landed —
"chunk 1/1 was killed after 2006ms" — but the in-flight-file naming produced
nothing and fell through to the pre-existing generic message. The feature I
wrote to diagnose a kill was itself destroyed by the kill.
Root cause, confirmed rather than assumed. The reporter yielded strings, which
node pipes into the --test-reporter-destination WriteStream. That stream
BUFFERS. execFileSync's timeout sends SIGKILL, which is uncatchable and gives
nothing a chance to flush, so the events sat in a buffer that died with the
child. The parent's own timer reported correctly because it lives in the
parent — which is exactly why half the feature looked fine.
The reporter now writes each event with fs.appendFileSync, unbuffered and
durable at the moment it happens, to a path passed through
GSD_RUN_TESTS_EVENTS_FILE. Env vars do not count toward the Windows
32,767-char argv ceiling, so moving the path out of argv also REDUCES
FIXED_OVERHEAD; the accounting moved with it rather than being left stale. The
destination is now a fixed devNull sink that stays empty by design.
Silence was the reason this was invisible for a whole run. Failing to read the
events file now says so explicitly, and distinguishes a file that could not be
read at all from one that exists but is empty — the generic fallback firing
quietly is what let a broken feature look like a working one. A write is
unbuffered but not atomic, so a kill can still interleave a partial line; the
reader tolerates exactly one unparsable trailing line and reports the complete
ones before it.
The failing T1 was left red and untouched rather than weakened to pass. Three
new unit tests cover the reader directly, with no subprocess, so the parsing
half is verifiable without a full runner pass: missing file, existing-but-empty
file, and a truncated final line.
Also adds ndjson-reporter.cjs to GSD_SCRIPTS_LIB_FILES in bin/install.js —
scripts/lib/ ships, and omitting it meant the file would install everywhere and
orphan on uninstall. That single omission caused 4 of the 7 remote failures.
Verification runs on the remote runner.
Refs #4012
The per-chunk timeout fired correctly but reported almost nothing, so every
diagnosis cost a CI round-trip. run-tests.cjs logged chunk START only — no
timestamp, no duration, no end line — then on a kill printed all ~55 basenames
and asked the operator to work out whether output kept flowing (slow) or
stopped early (hang). It could not name the in-flight file because the child is
spawned with stdio inherit, deliberately, per #3597/#1051.
Three additions.
Per-chunk elapsed timing on every path, not just failures, so drift toward the
cap is visible before it becomes a kill. Every timing number in the
investigation behind this had to be reconstructed by hand from GitHub log
timestamps.
A second, machine-readable reporter running ALONGSIDE the human one, writing
NDJSON to its own file. On a kill that file is read back and the files with a
test:start and no matching completion are named, with the staleness of the last
event, so "stopped 480s ago at X" reads differently from "still emitting at
kill". stdio stays inherit and nothing is piped or tee'd — the maxBuffer and
live-output risks that shaped the original design are untouched.
Ranking of the killed chunk's files by known weight, flagging any absent from
tests/test-timings.json, since an unweighted file is an unknown quantity.
Two details that are correct rather than lucky. Passing --test-reporter at all
replaces node's implicit default, so the human reporter is now named explicitly
and reproduces node's own selection (spec on a TTY, tap otherwise) — visible
output is unchanged. And the destination path's chunk index is zero-padded to a
fixed width because FIXED_OVERHEAD is computed ONCE before chunking; a
variable-length path would have silently mis-accounted the Windows 32,767-char
argv ceiling and reintroduced #3597. The reporter flags are added to
FIXED_OVERHEAD exactly as --test-force-exit is.
The multi-reporter pairing and the stream.compose reporter contract were
confirmed against Node's v24 documentation, not recalled — the first draft
carried them as an unverified assumption and said so.
Also corrects a stale comment claiming the 600s cap sits "below the 20m job
cap". The lane is sharded 3x at timeout-minutes: 45; the windows shards were at
19m when chunk 1/5 was killed on b351c83e0 and c3e667df3. The per-chunk cap is
now the binding constraint, and the old silent-cancel model leads to the wrong
conclusion.
Verification runs on the remote runner.
Closes#4012
