fix(#2455): abort remaining chunks when one hits the per-chunk timeout (#2465)

The per-chunk timeout exists so a bad chunk fails loudly 'rather than
silently burn the job's wall-clock budget until the CI runner cancels the
whole job' (run-tests.cjs:633-637). The control flow defeated that: after a
timeout kill the loop fell through to the next chunk. Since the timeout
(600000ms) is half the 20m job cap and a healthy Windows full pass is
~11m42s, continuing after a timeout can essentially never finish.

Observed on run 29749380190 (windows shard 2/3): chunk 1/5 was killed at
exactly 600s, the loop pressed on through chunks 2-4, and the job was
cancelled mid-chunk-5 at the 20m wall. The failure surfaced as
'##[error]The operation was canceled.' — the timeout diagnostic ended up
~38,000 log lines from the end and 'gh run view --log-failed' returned
nothing, making the real cause very hard to find.

Abort the remaining chunks on a timeout so the diagnostic survives as the
visible failure. Ordinary test failures still run every chunk, so the
operator keeps seeing all failures in one pass.

Fixes #2455

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-07-20 16:58:13 -04:00
committed by GitHub
parent 903182fed3
commit 448e148058
2 changed files with 90 additions and 2 deletions

View File

@@ -863,9 +863,35 @@ function main() {
);
}
const code = err.status || 1;
// Run every chunk so the operator sees all failures in one pass; report
// the first non-zero exit at the end.
if (firstFailureExit === 0) firstFailureExit = code;
if (timedOut) {
// A timeout has already burned a large share of the job's budget
// (chunkTimeoutMs defaults to 600000ms, i.e. half the 20m CI job
// cap), so — unlike an ordinary test failure — letting the loop
// fall through to the remaining chunks risks the CI runner
// cancelling the whole job before they finish. That cancellation
// replaces the loud, specific diagnostic printed above with an
// opaque "The operation was canceled." buried at the very end of
// the log, thousands of lines past the real cause (observed live on
// CI run 29749380190: chunk 1/5 timed out, the loop pressed on
// through chunks 2-4, and the job was cancelled mid-chunk-5 — the
// timeout message was ~38,000 log lines from the end and
// `gh run view --log-failed` returned nothing). Abort the remaining
// chunks instead so the operator actually sees this message.
const skipped = chunks.length - (i + 1);
if (skipped > 0) {
console.error(
`run-tests: aborting — skipping the remaining ${skipped} chunk${skipped === 1 ? '' : 's'} ` +
`after the chunk ${i + 1}/${chunks.length} timeout rather than risk the CI runner ` +
`cancelling the job (and burying this diagnostic) before they finish.`,
);
}
break;
}
// A non-timeout failure is cheap in wall-clock terms (the child exits
// promptly on its own), so — unlike the timeout case above — run every
// remaining chunk anyway: the operator sees all failures in one pass,
// and the first non-zero exit is reported at the end.
}
}
if (firstFailureExit !== 0) return firstFailureExit;