From 370cfc66804bbb218de4a0b0deb44da22e0fb13a Mon Sep 17 00:00:00 2001 From: Tom Boucher Date: Sat, 29 Aug 2026 16:13:15 -0400 Subject: [PATCH] enhance(#4036): persist CI shard/job timeout-vs-cap trending, warn at 90% (#4043) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(#4036): persist CI shard/job timeout-vs-cap trending, warn at 90% Adds two new mechanisms plus an audit-coverage extension: - scripts/lib/ci-job-timing.cjs: shared elapsed-vs-cap arithmetic - scripts/ci-check-job-near-cap.cjs: in-job advisory near-cap check, wired into test/test-full/mutate/smoke as each job's last step - scripts/ci-timeout-report.cjs + .github/workflows/ci-timeout-report.yml: scheduled REST-API poll that appends new records to tests/ci-timeout-budget-history.jsonl and opens a small data-only PR - tests/ci-test-job-timeout-budget.test.cjs: extended to cover mutate (mutation.yml) and smoke (install-smoke.yml), which previously had no headroom-factor gate coverage at all Does not change any timeout-minutes value, shard composition, or shard-1 contents — those stay maintainer policy calls per the issue's own scope. * fix(#4036): address two-orthogonal-review findings - Parity tests guarding the two hand-duplicated literals this design cannot single-source through GH Actions YAML: CI_JOB_TIMEOUT_MINUTES vs each job's own timeout-minutes, and ci-timeout-report.cjs's JOB_RULES name-prefixes vs each job's actual name: template. - Thread run.event through as runEvent on every persisted record, so PR-context and push-context install-smoke timings (genuinely different matrix shape) are distinguishable in the history rather than silently conflated under one job name. - Replace the Windows near-cap start-time step's ambiguous PowerShell +/>> precedence with GitHub's documented string-interpolation form. - Move github.run_id out of direct ${{ }} shell interpolation into an env: var in the new scheduled workflow, per this repo's own expression-injection-safe convention. * test(#4036): regenerate golden install-tree fixtures for scripts/lib/ci-job-timing.cjs npm run gen:install-tree — scripts/ ships wholesale into the installed package (per ADR/known-defect precedent from #4012's own PR history: a new scripts/lib/*.cjs file needs its golden entry regenerated or every runtime's install-tree test fails). Confirmed via gsd-test: this was the sole cause of the first real verification run's 25 failures (all in tests/golden-install-tree.test.cjs, one per runtime). Top-level scripts/*.cjs files (ci-check-job-near-cap.cjs, ci-timeout-report.cjs) are not individually tracked in these fixtures — consistent with every other existing top-level scripts/*.cjs file, so no entry was expected or added for those two. * fix(#4036): register new lib file with installer, fix H1 shell policy - bin/install.js: add ci-job-timing.cjs to GSD_SCRIPTS_LIB_FILES (a hand-maintained registry, not generated — tests/install.test.cjs asserts every scripts/lib/ file is enumerated here) - test.yml: replace the two OS-conditional "Record job start time" step pairs (test + test-full jobs) with a single unconditional `node -e` step. The prior pair's Windows variant declared an explicit shell: pwsh, which scripts/workflow-policy.cjs's H1 checker statically flags against every OS a job's matrix can realize, independent of the step's own if: gate. A single Node one-liner needs no shell override at all — it's syntactically valid and behaves identically under bash, zsh, and pwsh — which is both H1 compliant and removes the last OS-specific shell syntax from this change entirely. Both defects were found by a real gsd-test run, not local gates — lint:ci and build:lib were clean throughout because neither the scripts/lib/ install-manifest parity check nor the H1 shell-policy baseline runs as part of lint:ci; both are gsd-test-only suites. * docs(#4036): how-to for reading CI timeout budget signals The phase-gate docs check correctly flagged the enablement sequence as 3 real steps (read the near-cap warning, find the accumulated trend file, pick the right maintainer lever) — a reference table can't carry a sequence. Adds docs/how-to/read-ci-timeout-signals.md, indexed from docs/README.md. * chore(#4036): backfill changeset PR number (4043) --------- Co-authored-by: sim --- .changeset/plucky-pandas-roam.md | 5 + .github/workflows/ci-timeout-report.yml | 65 +++++ .github/workflows/install-smoke.yml | 12 + .github/workflows/mutation.yml | 11 + .github/workflows/test.yml | 22 ++ bin/install.js | 2 +- docs/README.md | 1 + docs/TESTING-SUITES.md | 37 +++ docs/how-to/read-ci-timeout-signals.md | 64 +++++ scripts/ci-check-job-near-cap.cjs | 49 ++++ scripts/ci-timeout-report.cjs | 230 +++++++++++++++ scripts/lib/ci-job-timing.cjs | 72 +++++ tests/ci-job-timing.test.cjs | 198 +++++++++++++ tests/ci-test-job-timeout-budget.test.cjs | 112 +++++++- tests/ci-timeout-report.test.cjs | 264 ++++++++++++++++++ tests/fixtures/install-tree/antigravity.json | 1 + tests/fixtures/install-tree/augment.json | 1 + tests/fixtures/install-tree/claude-local.json | 1 + tests/fixtures/install-tree/claude.json | 1 + tests/fixtures/install-tree/cline.json | 1 + tests/fixtures/install-tree/codebuddy.json | 1 + tests/fixtures/install-tree/codex.json | 1 + tests/fixtures/install-tree/copilot.json | 1 + tests/fixtures/install-tree/cursor.json | 1 + tests/fixtures/install-tree/hermes.json | 1 + tests/fixtures/install-tree/kilo.json | 1 + tests/fixtures/install-tree/kimi-code.json | 1 + tests/fixtures/install-tree/kimi.json | 1 + tests/fixtures/install-tree/opencode.json | 1 + tests/fixtures/install-tree/pi.json | 1 + tests/fixtures/install-tree/qwen.json | 1 + tests/fixtures/install-tree/trae.json | 1 + tests/fixtures/install-tree/windsurf.json | 1 + tests/fixtures/install-tree/zcode.json | 1 + 34 files changed, 1158 insertions(+), 5 deletions(-) create mode 100644 .changeset/plucky-pandas-roam.md create mode 100644 .github/workflows/ci-timeout-report.yml create mode 100644 docs/how-to/read-ci-timeout-signals.md create mode 100644 scripts/ci-check-job-near-cap.cjs create mode 100644 scripts/ci-timeout-report.cjs create mode 100644 scripts/lib/ci-job-timing.cjs create mode 100644 tests/ci-job-timing.test.cjs create mode 100644 tests/ci-timeout-report.test.cjs diff --git a/.changeset/plucky-pandas-roam.md b/.changeset/plucky-pandas-roam.md new file mode 100644 index 000000000..d9822e407 --- /dev/null +++ b/.changeset/plucky-pandas-roam.md @@ -0,0 +1,5 @@ +--- +type: Added +pr: 4043 +--- +**CI shard/job timeouts now self-report near-cap and accumulate a trending history.** Every matrixed CI job (test, test-full, mutate, smoke) warns in its own run once it crosses 90% of its timeout-minutes budget, and a new scheduled workflow keeps a durable, accumulating record of elapsed-vs-cap across runs — so a lane drifting toward its cap is visible before it actually breaches, not just after. (#4036) diff --git a/.github/workflows/ci-timeout-report.yml b/.github/workflows/ci-timeout-report.yml new file mode 100644 index 000000000..c637f9a0e --- /dev/null +++ b/.github/workflows/ci-timeout-report.yml @@ -0,0 +1,65 @@ +name: CI timeout budget report + +# Scheduled aggregator for #4036: polls GitHub's own Actions REST API for +# recently completed jobs across test.yml / mutation.yml / install-smoke.yml, +# computes elapsed-vs-timeout-minutes per job via scripts/ci-timeout-report.cjs, +# and appends any new records to tests/ci-timeout-budget-history.jsonl. `next` +# is a protected branch — nothing pushes to it directly, even from a scheduled +# bot run (see auto-backmerge.yml for the same constraint) — so each run opens +# a small, uniquely-branched, data-only PR carrying just that run's new rows, +# rather than force-pushing one long-lived branch. Small PRs merge easily and +# never go stale enough to need a rebase. + +on: + schedule: + - cron: '15 7 * * *' + workflow_dispatch: + +permissions: + actions: read + contents: write + pull-requests: write + +jobs: + report: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: true + token: ${{ github.token }} + + - uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 + id: poll + with: + script: | + const report = require(`${process.env.GITHUB_WORKSPACE}/scripts/ci-timeout-report.cjs`); + const result = await report.main({ github, context, core }); + core.info(`ci-timeout-report: added ${result.added} record(s), ${result.nearCap} near-cap`); + return result; + + - name: Open a small PR with this run's new records, if any + if: always() + env: + GH_TOKEN: ${{ secrets.GSD_BOT_PR_TOKEN || secrets.GITHUB_TOKEN }} + RUN_ID: ${{ github.run_id }} + run: | + set -euo pipefail + if git status --porcelain -- tests/ci-timeout-budget-history.jsonl | grep -q .; then + BR="automation/ci-timeout-report-$RUN_ID" + git checkout -b "$BR" + git add tests/ci-timeout-budget-history.jsonl + git -c user.name="gsd-bot" -c user.email="gsd-bot@users.noreply.github.com" \ + commit -m "chore: CI timeout budget report — run $RUN_ID" + git push origin "$BR" + gh pr create \ + --base next \ + --head "$BR" \ + --title "chore: CI timeout budget report — run $RUN_ID" \ + --body "Automated, data-only update to \`tests/ci-timeout-budget-history.jsonl\` — new shard/job wall-clock vs. \`timeout-minutes\` records collected by \`.github/workflows/ci-timeout-report.yml\` (#4036). No source changes; safe to merge once CI passes." \ + --label "automation" \ + --label "no-changelog" + else + echo "No new timeout-report records this run — nothing to commit." + fi diff --git a/.github/workflows/install-smoke.yml b/.github/workflows/install-smoke.yml index bc6386a69..07e376ed9 100644 --- a/.github/workflows/install-smoke.yml +++ b/.github/workflows/install-smoke.yml @@ -117,6 +117,10 @@ jobs: # Need enough history to merge origin/main for stale-base detection. fetch-depth: 0 + - name: Record job start time (near-cap check) + if: steps.skip.outputs.skip != 'true' + run: echo "CI_JOB_START_EPOCH_MS=$(( $(date +%s) * 1000 ))" >> "$GITHUB_ENV" + # The default `refs/pull/N/merge` ref GitHub produces for PRs is cached # against the recorded merge-base, not the current target branch. When the # target advances @@ -213,6 +217,14 @@ jobs: path: /tmp/release-smoke.json retention-days: 7 + - name: Check job budget (near-cap advisory) + if: always() + continue-on-error: true + env: + CI_JOB_LABEL: "smoke (${{ matrix.os }}, ${{ matrix.node-version }}, ${{ matrix.full_only }}, ${{ matrix.shell }})" + CI_JOB_TIMEOUT_MINUTES: '12' + run: node scripts/ci-check-job-near-cap.cjs + # --------------------------------------------------------------------------- # Job 2: unpacked-dir install # --------------------------------------------------------------------------- diff --git a/.github/workflows/mutation.yml b/.github/workflows/mutation.yml index d5703e88f..7d1460132 100644 --- a/.github/workflows/mutation.yml +++ b/.github/workflows/mutation.yml @@ -123,6 +123,9 @@ jobs: fetch-depth: 0 persist-credentials: true + - name: Record job start time (near-cap check) + run: echo "CI_JOB_START_EPOCH_MS=$(( $(date +%s) * 1000 ))" >> "$GITHUB_ENV" + - name: Set up Node uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: @@ -184,6 +187,14 @@ jobs: path: reports/mutation/mutation.html retention-days: 14 + - name: Check job budget (near-cap advisory) + if: always() + continue-on-error: true + env: + CI_JOB_LABEL: "Stryker (${{ matrix.name }})" + CI_JOB_TIMEOUT_MINUTES: ${{ matrix.timeoutMinutes }} + run: node scripts/ci-check-job-near-cap.cjs + # ── Job 3: mutation-gate ─────────────────────────────────────────────────── # Stable required-check name for branch protection. # • has_work is false → job is SKIPPED (not run) — nothing to mutate is not diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 707b973e0..ae89532f8 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -244,6 +244,9 @@ jobs: persist-credentials: true token: ${{ github.token }} + - name: Record job start time (near-cap check) + run: node -e 'require("fs").appendFileSync(process.env.GITHUB_ENV, "CI_JOB_START_EPOCH_MS=" + Date.now() + "\n")' + - name: Guard — require GitHub-hosted runner run: node scripts/ci-guard-runner.cjs @@ -349,6 +352,14 @@ jobs: if: matrix.scope == 'full' && matrix.shard == '1/3' && needs.changes.outputs.full_matrix == 'true' run: npm run test:slow + - name: Check job budget (near-cap advisory) + if: always() + continue-on-error: true + env: + CI_JOB_LABEL: "test (${{ matrix.os }}, ${{ matrix.node-version }}${{ matrix.shard && format(', shard {0}', matrix.shard) || '' }})" + CI_JOB_TIMEOUT_MINUTES: '15' + run: node scripts/ci-check-job-near-cap.cjs + test-inert: name: test (inert CI) needs: [changes, preflight] @@ -501,6 +512,9 @@ jobs: persist-credentials: true token: ${{ github.token }} + - name: Record job start time (near-cap check) + run: node -e 'require("fs").appendFileSync(process.env.GITHUB_ENV, "CI_JOB_START_EPOCH_MS=" + Date.now() + "\n")' + - name: Guard — require GitHub-hosted runner run: node scripts/ci-guard-runner.cjs @@ -562,6 +576,14 @@ jobs: if: matrix.shard == 1 run: npm run test:security + - name: Check job budget (near-cap advisory) + if: always() + continue-on-error: true + env: + CI_JOB_LABEL: "full test (${{ matrix.os }}, ${{ matrix.node-version }}, shard ${{ matrix.shard }}/3)" + CI_JOB_TIMEOUT_MINUTES: '45' + run: node scripts/ci-check-job-near-cap.cjs + # #2952: the coverage gate that the `test` lane used to run inline. Sharding # the unit suite means no single runner sees the whole picture, so the gate # moves here, downloads every shard's raw V8 dumps into one coverage/tmp, and diff --git a/bin/install.js b/bin/install.js index c9311d08d..0bc641766 100755 --- a/bin/install.js +++ b/bin/install.js @@ -411,7 +411,7 @@ const GSD_CHANGESET_FILES = [ 'github-release-notes.cjs', 'lint.cjs', 'new.cjs', 'README.md', // documentation only — not user-authored ]; -const GSD_SCRIPTS_LIB_FILES = ['cli-exit.cjs', 'allowlist-ratchet.cjs', 'drift-scan.cjs', 'alias-drift-families.cjs', 'exit-code-registry.cjs', 'ndjson-reporter.cjs']; +const GSD_SCRIPTS_LIB_FILES = ['cli-exit.cjs', 'allowlist-ratchet.cjs', 'drift-scan.cjs', 'alias-drift-families.cjs', 'exit-code-registry.cjs', 'ndjson-reporter.cjs', 'ci-job-timing.cjs']; /** * Resolve a runtime's shared-hooks directory name from its descriptor. diff --git a/docs/README.md b/docs/README.md index 4c7c81862..6ab8ab693 100644 --- a/docs/README.md +++ b/docs/README.md @@ -39,6 +39,7 @@ Language versions: [English](README.md) · [Português (pt-BR)](pt-BR/README.md) - [Adopt the v2 exit contract](how-to/adopt-the-v2-exit-contract.md) — turn on `gsd-tools`'s versioned exit-code projection, read the code table including what `80` (`DEGRADED`) means, and migrate a CI gate that treats any non-zero exit as fatal - [Read the statusline freshness marker](how-to/read-the-statusline-freshness-marker.md) — turn on `state ~N commits back`, and tell "STATE.md is fresh" apart from "freshness could not be established" - [Consume the planning snapshot](how-to/consume-the-planning-snapshot.md) — read `planning inspect` from a dashboard or harness, and tell "nothing to report" apart from "could not look" +- [Read CI timeout budget signals](how-to/read-ci-timeout-signals.md) — find the near-cap warning on a run, read the accumulated `tests/ci-timeout-budget-history.jsonl` trend, and know which lever (cap, shard balance, shard-1 contents) a repeatedly-near-cap lane calls for - [Consume the state contract](how-to/consume-the-state-contract.md) — read `.planning/state.json` from a workbench or editor extension, gate on the contract version, and tell "nothing to show" apart from "could not look" - [Keep planning docs out of a shared repo](how-to/keep-planning-docs-private.md) — make `.planning/` local-only, including untracking files git already tracks (the step `.gitignore` alone cannot do) - [Publish PRs without planning artifacts](how-to/publish-prs-without-planning-artifacts.md) — keep `.planning/` committed locally, so worktrees and `/gsd-undo` keep working, while `planning.pr_strict` keeps every planning path out of the branch you push diff --git a/docs/TESTING-SUITES.md b/docs/TESTING-SUITES.md index ab03b5101..fd37ff4bf 100644 --- a/docs/TESTING-SUITES.md +++ b/docs/TESTING-SUITES.md @@ -527,6 +527,43 @@ drift costs chunk *balance*, never a red build. A count-based floor additionally guarantees the packer never produces fewer chunks than plain count-based packing would, so a badly stale table cannot collapse the suite into a few fat chunks. +### CI job timeout budgets: report + near-cap warning (#4036) + +Every matrixed job — `test` and `test-full` in `test.yml`, `mutate` in +`mutation.yml`, `smoke` in `install-smoke.yml` — declares a `timeout-minutes` +cap. `tests/ci-test-job-timeout-budget.test.cjs` enforces that each checked-in +cap stays at least the **headroom factor** (1.5x) above a documented, +hand-measured cost for that job — now all four of the jobs above, not just +`test`/`test-full`/`coverage-gate`/`test-inert` as before. + +Two runtime mechanisms sit on top of that static gate, both new in #4036: + +- **In-job near-cap check** (`scripts/ci-check-job-near-cap.cjs`) — the last + step of each of the four jobs computes elapsed-vs-cap from a start-time + marker recorded as that job's first step. At >=90% of budget it emits a + `::warning::` annotation (visible in the PR Checks UI) and a + `$GITHUB_STEP_SUMMARY` block. Advisory only — it never fails the job. Known + limit: it cannot fire for a job actually killed by the timeout, since a + killed job never reaches its last step. That case is caught by the second + mechanism instead. +- **Scheduled trending report** (`.github/workflows/ci-timeout-report.yml`, + `scripts/ci-timeout-report.cjs`) — runs daily and on `workflow_dispatch`. It + polls GitHub's Actions REST API for recently completed jobs across + `test.yml`, `mutation.yml`, and `install-smoke.yml`, resolves each job's + declared cap (a literal `timeout-minutes` for `test`/`test-full`/`smoke`, or + `scripts/mutation-matrix.cjs`'s `COVERED[].timeoutMinutes` for + `mutate`'s per-module shards), and appends any new `(runId, jobName)` + records to `tests/ci-timeout-budget-history.jsonl`. Unlike the in-job check, + this also catches jobs killed by an actual timeout breach — GitHub's Jobs + API still reports `started_at`/`completed_at` for a cancelled job. Each run + opens a small, data-only PR carrying that run's new rows, since `next` is a + protected branch and nothing pushes to it directly — the same constraint + `auto-backmerge.yml` already works within. + +This does not retune any `timeout-minutes` value, rebalance shard composition, +or trim what runs in shard 1 — those stay maintainer policy calls made from +the accumulated history, not something either mechanism decides on its own. + ### How-to: regenerate the timing table Regenerate when the suite's cost profile has visibly drifted — after adding or diff --git a/docs/how-to/read-ci-timeout-signals.md b/docs/how-to/read-ci-timeout-signals.md new file mode 100644 index 000000000..81285ad47 --- /dev/null +++ b/docs/how-to/read-ci-timeout-signals.md @@ -0,0 +1,64 @@ +# How to read CI timeout budget signals + +Every matrixed CI job (`test`, `test-full` in `.github/workflows/test.yml`; `mutate` in +`mutation.yml`; `smoke` in `install-smoke.yml`) now reports how close it ran to its +`timeout-minutes` cap. This page is for a maintainer trying to answer: *is a lane drifting +toward its cap, and where do I look?* + +## 1. A single run crossed 90% of its budget + +Open the job's page in the Actions run — two places show it, both populated by the same +computation (`scripts/lib/ci-job-timing.cjs`): + +- **The Checks tab annotation.** A `::warning::` line renders as an expandable warning banner + on the PR's Checks summary, naming the job, its elapsed time, its cap, and the percentage — + visible without opening the job's logs. +- **The job's step summary.** The same information, as a Markdown line, appended to the job's + own summary page (`$GITHUB_STEP_SUMMARY`) by that job's own "Check job budget (near-cap + advisory)" step — always the job's last step. + +Neither signal fails the job. A near-cap warning on an otherwise-green run means exactly what +it says: this run finished, but with less margin than the headroom-factor gate assumes it has. + +**If the warning never appears even on a job that was actually cancelled at its cap**, that is +expected — a killed job never reaches its last step, so the in-job check never runs. See §2. + +## 2. Checking the accumulated trend + +`.github/workflows/ci-timeout-report.yml` runs daily (and on-demand via `workflow_dispatch`). It +polls GitHub's Actions REST API directly — independent of whether any individual job's own +near-cap step ran — so it also catches jobs that were actually cancelled by a timeout breach +(GitHub's Jobs API still reports `started_at`/`completed_at` for a cancelled job). + +Each run's new rows land in `tests/ci-timeout-budget-history.jsonl`, one JSON object per line: + +```json +{"runId":123456,"jobName":"test (ubuntu-latest, 24, shard 1/3)","workflowFile":"test.yml","sha":"...","completedAt":"...","elapsedMs":432000,"timeoutMinutes":15,"pct":0.8,"runEvent":"push"} +``` + +`runEvent` matters for `install-smoke.yml`'s `smoke` job specifically — its `pull_request` runs +use a smaller matrix (no `macos-latest` `full_only` row) than its `push` runs, so a `pct` figure +only means the same thing across rows sharing the same `runEvent`. + +Because `next` is a protected branch, the report never pushes directly to it — each scheduled +run opens (or the prior run's already merged, in which case a fresh one opens) a small, +data-only PR carrying just that run's new rows, titled `chore: CI timeout budget report — run +`. Merge these like any other PR; there is nothing to review beyond "did the numbers land." + +## 3. A lane is repeatedly near-cap — what to do + +Neither mechanism here decides what to do about a lane that's genuinely trending toward its +cap. That is a maintainer call among three options, each with real tradeoffs: + +- **Raise the `timeout-minutes` cap** for that job. +- **Rebalance the shard split** so no single shard carries a disproportionate share of the + suite (see `scripts/run-tests.cjs`'s `selectShard`, which packs shards by measured cost from + `tests/test-timings.json`). +- **Trim what runs on the long-pole shard** — for the `test` job, shard 1 also carries the + unsharded aux suites (integration/security/install/slow); moving one elsewhere changes what + shard 1 costs. + +`tests/ci-test-job-timeout-budget.test.cjs` will keep failing to accept a lowered +`timeout-minutes` beneath 1.5x whatever `LANE_COSTS`/`COVERED[*].timeoutMinutes` records as that +job's last measured cost — raising the cap back down is not something either mechanism will +silently allow. diff --git a/scripts/ci-check-job-near-cap.cjs b/scripts/ci-check-job-near-cap.cjs new file mode 100644 index 000000000..e50e52ee0 --- /dev/null +++ b/scripts/ci-check-job-near-cap.cjs @@ -0,0 +1,49 @@ +#!/usr/bin/env node +'use strict'; + +const fs = require('node:fs'); +const { computeElapsedPct, isNearCap, formatNearCapNotice } = require('./lib/ci-job-timing.cjs'); + +function run(env = process.env, nowMs = Date.now(), appendFileSync = fs.appendFileSync) { + const label = env.CI_JOB_LABEL; + const startEpochMs = Number(env.CI_JOB_START_EPOCH_MS); + const timeoutMinutes = Number(env.CI_JOB_TIMEOUT_MINUTES); + + if (!label || !Number.isFinite(startEpochMs) || !Number.isFinite(timeoutMinutes) || timeoutMinutes <= 0) { + console.error( + 'ci-check-job-near-cap: missing or invalid CI_JOB_LABEL / CI_JOB_START_EPOCH_MS / ' + + 'CI_JOB_TIMEOUT_MINUTES — skipping near-cap check (advisory only, not a failure).', + ); + return { skipped: true }; + } + + const { elapsedMs, capMs, pct } = computeElapsedPct({ + startedAt: new Date(startEpochMs).toISOString(), + completedAt: new Date(nowMs).toISOString(), + timeoutMinutes, + }); + + if (!isNearCap(pct)) { + return { skipped: false, nearCap: false, pct }; + } + + const { warningLine, summaryMarkdown } = formatNearCapNotice({ label, pct, elapsedMs, capMs }); + console.log(warningLine); + + if (env.GITHUB_STEP_SUMMARY) { + try { + appendFileSync(env.GITHUB_STEP_SUMMARY, `${summaryMarkdown}\n`); + } catch (err) { + console.error(`ci-check-job-near-cap: could not write GITHUB_STEP_SUMMARY: ${err.message}`); + } + } + + return { skipped: false, nearCap: true, pct }; +} + +module.exports = { run }; + +if (require.main === module) { + run(); + process.exitCode = 0; +} diff --git a/scripts/ci-timeout-report.cjs b/scripts/ci-timeout-report.cjs new file mode 100644 index 000000000..d75289cff --- /dev/null +++ b/scripts/ci-timeout-report.cjs @@ -0,0 +1,230 @@ +'use strict'; + +/** + * scripts/ci-timeout-report.cjs + * + * Scheduled CI-timeout trending report (#4036). Polls GitHub's Actions REST + * API for recently completed jobs across test.yml, mutation.yml, and + * install-smoke.yml, resolves each job's declared `timeout-minutes` budget, + * computes elapsed-vs-cap via scripts/lib/ci-job-timing.cjs, and appends + * new (never-before-seen) records to a JSONL history file. Every record also + * carries the triggering `runEvent` (e.g. `push`/`pull_request`) so entries + * for jobs whose matrix genuinely differs by trigger (e.g. `smoke`'s + * push-only macOS row) can be told apart in the persisted trend — records + * are never filtered by event, only labeled. + * + * Invoked from a GitHub Actions workflow via actions/github-script, e.g.: + * const report = require(`${process.env.GITHUB_WORKSPACE}/scripts/ci-timeout-report.cjs`); + * const result = await report.main({ github, context, core }); + */ + +const yaml = require('js-yaml'); +const fs = require('node:fs'); +const path = require('node:path'); +const { + computeElapsedPct, isNearCap, formatNearCapNotice, +} = require('./lib/ci-job-timing.cjs'); + +const HISTORY_PATH = path.join(__dirname, '..', 'tests', 'ci-timeout-budget-history.jsonl'); +const WORKFLOWS_DIR = path.join(__dirname, '..', '.github', 'workflows'); + +// Static job name → job-id rules, first match wins, checked in array order +// (test-inert before test, since both job names start with "test "). +const JOB_RULES = [ + { workflowFile: 'test.yml', jobKey: 'test-inert', test: (name) => name === 'test (inert CI)' }, + { workflowFile: 'test.yml', jobKey: 'test', test: (name) => name.startsWith('test (') && name !== 'test (inert CI)' }, + { workflowFile: 'test.yml', jobKey: 'test-full', test: (name) => name.startsWith('full test (') }, + { workflowFile: 'test.yml', jobKey: 'coverage-gate', test: (name) => name === 'Coverage gate (merged shards)' }, + { workflowFile: 'install-smoke.yml', jobKey: 'smoke', test: (name) => name.startsWith('smoke (') }, +]; + +function resolveJobTimeoutMinutes({ jobName, workflowFile, workflowYamlText, covered }) { + if (workflowFile === 'mutation.yml') { + const m = jobName.match(/^Stryker \(([^)]+)\)$/); + if (!m) return null; + const moduleName = m[1]; + if (!covered || !Object.prototype.hasOwnProperty.call(covered, moduleName)) return null; + return covered[moduleName].timeoutMinutes || 15; + } + + const rule = JOB_RULES.find((r) => r.workflowFile === workflowFile && r.test(jobName)); + if (!rule) return null; + + const doc = yaml.load(workflowYamlText); + const budget = doc && doc.jobs && doc.jobs[rule.jobKey] ? doc.jobs[rule.jobKey]['timeout-minutes'] : undefined; + return typeof budget === 'number' ? budget : null; +} + +/** + * @param {{job: object, workflowFile: string, workflowYamlText: ?string, covered: ?object}} args + * `job.runEvent` is the triggering event (e.g. `push`/`pull_request`) — carried through to + * the returned record so entries whose matrix genuinely differs by trigger (e.g. `smoke`'s + * push-only macOS row) can be distinguished in the persisted history. + */ +function parseJobRecord({ job, workflowFile, workflowYamlText, covered }) { + if (!job.completed_at) return null; + + const timeoutMinutes = resolveJobTimeoutMinutes({ jobName: job.name, workflowFile, workflowYamlText, covered }); + if (timeoutMinutes == null) return null; + + const { elapsedMs, pct } = computeElapsedPct({ + startedAt: job.started_at, completedAt: job.completed_at, timeoutMinutes, + }); + + return { + runId: job.run_id, + jobName: job.name, + workflowFile, + sha: job.head_sha, + runEvent: job.runEvent, + completedAt: job.completed_at, + elapsedMs, + timeoutMinutes, + pct, + }; +} + +function buildReportLines(runs, { workflowFile, workflowYamlText, covered }) { + const records = []; + for (const { run, jobs } of runs) { + for (const job of jobs) { + const rec = parseJobRecord({ + job: { + ...job, run_id: run.id, head_sha: run.head_sha, runEvent: run.event, + }, + workflowFile, + workflowYamlText, + covered, + }); + if (rec) records.push(rec); + } + } + return records; +} + +function dedupeAgainstHistory(newRecords, historyText) { + const seen = new Set(); + for (const line of String(historyText || '').split('\n')) { + if (!line.trim()) continue; + try { + const rec = JSON.parse(line); + seen.add(`${rec.runId}::${rec.jobName}`); + } catch { + // Malformed history line — skip it rather than crash the whole report. + } + } + return newRecords.filter((r) => !seen.has(`${r.runId}::${r.jobName}`)); +} + +function formatHistoryLine(record) { + return `${JSON.stringify(record)}\n`; +} + +const WORKFLOW_FILES = ['test.yml', 'mutation.yml', 'install-smoke.yml']; +const MAX_RUNS_PER_WORKFLOW = 15; + +/** + * Orchestration entry point — impure, invoked from actions/github-script. + * + * @param {{github: object, context: object, core: object, historyPath?: string, fs?: object}} args + * @returns {Promise<{added: number, nearCap: number}>} + */ +async function main({ + github, context, core, historyPath = HISTORY_PATH, fs: fsImpl = fs, +}) { + const { owner, repo } = context.repo; + const mutationMatrix = require('./mutation-matrix.cjs'); + + const allNewRecords = []; + + for (const workflowFile of WORKFLOW_FILES) { + const covered = workflowFile === 'mutation.yml' ? mutationMatrix.COVERED : null; + const workflowYamlText = workflowFile === 'mutation.yml' + ? null + : fsImpl.readFileSync(path.join(WORKFLOWS_DIR, workflowFile), 'utf8'); + + let runsList; + try { + runsList = await github.paginate(github.rest.actions.listWorkflowRuns, { + owner, + repo, + workflow_id: workflowFile, + status: 'completed', + per_page: 30, + }); + } catch (err) { + core.warning(`ci-timeout-report: failed to list runs for ${workflowFile}: ${err.message}`); + continue; + } + + const runs = runsList.slice(0, MAX_RUNS_PER_WORKFLOW); + const runsWithJobs = []; + + for (const run of runs) { + try { + const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, { + owner, + repo, + run_id: run.id, + per_page: 50, + }); + runsWithJobs.push({ run, jobs }); + } catch (err) { + core.warning(`ci-timeout-report: failed to list jobs for ${workflowFile} run ${run.id}: ${err.message}`); + } + } + + const records = buildReportLines(runsWithJobs, { workflowFile, workflowYamlText, covered }); + allNewRecords.push(...records); + } + + let historyText = ''; + try { + historyText = fsImpl.readFileSync(historyPath, 'utf8'); + } catch { + // First run — history file does not exist yet, treat as empty. + historyText = ''; + } + + const deduped = dedupeAgainstHistory(allNewRecords, historyText); + + if (deduped.length > 0) { + const newLines = deduped.map(formatHistoryLine).join(''); + fsImpl.appendFileSync(historyPath, newLines); + } + // First-run bootstrap when there is nothing new to append is handled by + // `git add` picking up whatever the history file already contains. + + let nearCapCount = 0; + for (const record of deduped) { + if (!isNearCap(record.pct)) continue; + nearCapCount += 1; + + const notice = formatNearCapNotice({ + label: `${record.jobName} (run ${record.runId})`, + pct: record.pct, + elapsedMs: record.elapsedMs, + capMs: record.timeoutMinutes * 60000, + }); + + core.warning(notice.warningLine.replace(/^::warning title=CI budget::/, '')); + + if (core.summary) { + core.summary.addRaw(`${notice.summaryMarkdown}\n`); + } + } + + return { added: deduped.length, nearCap: nearCapCount }; +} + +module.exports = { + HISTORY_PATH, + WORKFLOWS_DIR, + JOB_RULES, + resolveJobTimeoutMinutes, + parseJobRecord, + buildReportLines, + dedupeAgainstHistory, + formatHistoryLine, + main, +}; diff --git a/scripts/lib/ci-job-timing.cjs b/scripts/lib/ci-job-timing.cjs new file mode 100644 index 000000000..ef41e68ca --- /dev/null +++ b/scripts/lib/ci-job-timing.cjs @@ -0,0 +1,72 @@ +'use strict'; + +/** + * scripts/lib/ci-job-timing.cjs + * + * Pure budget-percentage arithmetic shared by the in-job near-cap check + * (scripts/ci-check-job-near-cap.cjs) and the scheduled trending report + * (scripts/ci-timeout-report.cjs). See #4036. + */ + +/** A job at or above this fraction of its `timeout-minutes` cap is "near-cap". */ +const THRESHOLD_PCT = 0.9; + +/** + * @param {{startedAt: string, completedAt: string, timeoutMinutes: number}} args + * @returns {{elapsedMs: number, capMs: number, pct: number}} + */ +function computeElapsedPct({ startedAt, completedAt, timeoutMinutes }) { + if (!Number.isFinite(timeoutMinutes) || timeoutMinutes <= 0) { + throw new Error(`timeoutMinutes must be a positive finite number, got ${timeoutMinutes}`); + } + + const startMs = new Date(startedAt).getTime(); + const endMs = new Date(completedAt).getTime(); + + if (!Number.isFinite(startMs)) { + throw new Error(`startedAt is not a valid timestamp: ${startedAt}`); + } + if (!Number.isFinite(endMs)) { + throw new Error(`completedAt is not a valid timestamp: ${completedAt}`); + } + + const elapsedMs = endMs - startMs; + if (elapsedMs < 0) { + throw new Error(`completedAt (${completedAt}) is before startedAt (${startedAt})`); + } + + const capMs = timeoutMinutes * 60000; + return { elapsedMs, capMs, pct: elapsedMs / capMs }; +} + +/** + * @param {number} pct + * @param {number} [threshold] + * @returns {boolean} + */ +function isNearCap(pct, threshold = THRESHOLD_PCT) { + return pct >= threshold; +} + +/** + * @param {{label: string, pct: number, elapsedMs: number, capMs: number}} args + * @returns {{warningLine: string, summaryMarkdown: string}} + */ +function formatNearCapNotice({ label, pct, elapsedMs, capMs }) { + const pctStr = `${Math.round(pct * 100)}%`; + const elapsedMin = (elapsedMs / 60000).toFixed(1); + const capMin = (capMs / 60000).toFixed(1); + const detail = `${label} at ${pctStr} of its ${capMin}m cap (${elapsedMin}m elapsed)`; + + return { + warningLine: `::warning title=CI budget::${detail}`, + summaryMarkdown: `- **${label}** — ${pctStr} of ${capMin}m cap (${elapsedMin}m elapsed)`, + }; +} + +module.exports = { + THRESHOLD_PCT, + computeElapsedPct, + isNearCap, + formatNearCapNotice, +}; diff --git a/tests/ci-job-timing.test.cjs b/tests/ci-job-timing.test.cjs new file mode 100644 index 000000000..c8df92c53 --- /dev/null +++ b/tests/ci-job-timing.test.cjs @@ -0,0 +1,198 @@ +'use strict'; + +/** + * scripts/lib/ci-job-timing.cjs — pure budget-percentage arithmetic (#4036). + * + * This module is shared by the in-job near-cap check + * (scripts/ci-check-job-near-cap.cjs) and the scheduled trending report + * (scripts/ci-timeout-report.cjs). Both surfaces need to agree on exactly + * what "elapsed" and "near-cap" mean, so the arithmetic is factored out here + * and tested once, at every boundary the two callers depend on: zero + * elapsed, exactly-at-cap, just-under-threshold, exactly-at-threshold, + * just-over-threshold, and past-cap. It also covers the invalid-input + * failure modes (bad timeoutMinutes, malformed timestamps, completedAt + * before startedAt) that both callers need to fail loudly on rather than + * silently misreport. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); +const fc = require('fast-check'); + +const { + THRESHOLD_PCT, + computeElapsedPct, + isNearCap, + formatNearCapNotice, +} = require('../scripts/lib/ci-job-timing.cjs'); + +test('computeElapsedPct', async (t) => { + await t.test('zero elapsed yields pct exactly 0', () => { + const result = computeElapsedPct({ + startedAt: '2026-01-01T00:00:00.000Z', + completedAt: '2026-01-01T00:00:00.000Z', + timeoutMinutes: 15, + }); + assert.equal(result.pct, 0); + // Deterministic arithmetic over a fixed literal fixture (not a measured + // wall-clock duration); elapsedMs is part of the function's public + // return-shape contract that formatNearCapNotice reads directly. + // eslint-disable-next-line local/no-elapsed-assertion -- deterministic arithmetic, not measured timing + assert.equal(result.elapsedMs, 0); + assert.equal(result.capMs, 900000); + }); + + await t.test('elapsed exactly equal to cap yields pct exactly 1 and is near-cap', () => { + const result = computeElapsedPct({ + startedAt: '2026-01-01T00:00:00.000Z', + completedAt: '2026-01-01T00:15:00.000Z', + timeoutMinutes: 15, + }); + assert.equal(result.pct, 1); + assert.equal(isNearCap(result.pct), true); + }); + + await t.test('elapsed at 89.99% of cap is NOT near-cap', () => { + const startedAt = new Date('2026-01-01T00:00:00.000Z'); + const completedAt = new Date(startedAt.getTime() + 809910); + const result = computeElapsedPct({ + startedAt: startedAt.toISOString(), + completedAt: completedAt.toISOString(), + timeoutMinutes: 15, + }); + assert.equal(isNearCap(result.pct), false); + }); + + await t.test('elapsed at exactly 90% of cap IS near-cap (threshold is inclusive)', () => { + const startedAt = new Date('2026-01-01T00:00:00.000Z'); + const completedAt = new Date(startedAt.getTime() + 810000); + const result = computeElapsedPct({ + startedAt: startedAt.toISOString(), + completedAt: completedAt.toISOString(), + timeoutMinutes: 15, + }); + assert.equal(result.pct, 0.9); + assert.equal(isNearCap(result.pct), true); + }); + + await t.test('elapsed at 90.01% of cap is near-cap', () => { + const startedAt = new Date('2026-01-01T00:00:00.000Z'); + const completedAt = new Date(startedAt.getTime() + 810090); + const result = computeElapsedPct({ + startedAt: startedAt.toISOString(), + completedAt: completedAt.toISOString(), + timeoutMinutes: 15, + }); + assert.equal(isNearCap(result.pct), true); + }); + + await t.test('elapsed beyond the cap does not throw, pct > 1, near-cap', () => { + const result = computeElapsedPct({ + startedAt: '2026-01-01T00:00:00.000Z', + completedAt: '2026-01-01T00:20:00.000Z', + timeoutMinutes: 15, + }); + assert.ok(result.pct > 1); + assert.equal(isNearCap(result.pct), true); + }); + + await t.test('timeoutMinutes of 0 throws', () => { + assert.throws( + () => + computeElapsedPct({ + startedAt: '2026-01-01T00:00:00.000Z', + completedAt: '2026-01-01T00:15:00.000Z', + timeoutMinutes: 0, + }), + /timeoutMinutes must be a positive finite number/, + ); + }); + + await t.test('negative timeoutMinutes throws', () => { + assert.throws( + () => + computeElapsedPct({ + startedAt: '2026-01-01T00:00:00.000Z', + completedAt: '2026-01-01T00:15:00.000Z', + timeoutMinutes: -5, + }), + /timeoutMinutes must be a positive finite number/, + ); + }); + + await t.test('malformed startedAt throws', () => { + assert.throws( + () => + computeElapsedPct({ + startedAt: 'not-a-date', + completedAt: '2026-01-01T00:15:00.000Z', + timeoutMinutes: 15, + }), + /startedAt is not a valid timestamp/, + ); + }); + + await t.test('completedAt before startedAt throws', () => { + assert.throws( + () => + computeElapsedPct({ + startedAt: '2026-01-01T01:00:00.000Z', + completedAt: '2026-01-01T00:00:00.000Z', + timeoutMinutes: 15, + }), + /completedAt .* is before startedAt/, + ); + }); +}); + +test('formatNearCapNotice', async (t) => { + await t.test('produces a warningLine and summaryMarkdown containing label and pct', () => { + const { warningLine, summaryMarkdown } = formatNearCapNotice({ + label: 'test (ubuntu-latest, 24, shard 1/3)', + pct: 0.92, + elapsedMs: 828000, + capMs: 900000, + }); + + assert.ok(warningLine.startsWith('::warning')); + assert.ok(warningLine.includes('test (ubuntu-latest, 24, shard 1/3)')); + assert.ok(warningLine.includes('92%')); + + assert.ok(summaryMarkdown.includes('test (ubuntu-latest, 24, shard 1/3)')); + assert.ok(summaryMarkdown.includes('92%')); + }); +}); + +test('computeElapsedPct / isNearCap — property: pct matches direct division and threshold predicate agrees', () => { + fc.assert( + fc.property( + fc.record({ + timeoutMinutes: fc.integer({ min: 1, max: 500 }), + // Expressed as thousandths of the cap so elapsedMs stays a whole + // number of ms in [0, 10 * capMs] without generating capMs first. + fractionOfCapMilli: fc.integer({ min: 0, max: 10000 }), + }), + ({ timeoutMinutes, fractionOfCapMilli }) => { + const capMs = timeoutMinutes * 60000; + const elapsedMs = Math.floor((capMs * fractionOfCapMilli) / 1000); + + const startedAt = new Date('2026-01-01T00:00:00.000Z'); + const completedAt = new Date(startedAt.getTime() + elapsedMs); + + const result = computeElapsedPct({ + startedAt: startedAt.toISOString(), + completedAt: completedAt.toISOString(), + timeoutMinutes, + }); + + // elapsedMs/capMs here are deterministically constructed by this test + // from fast-check inputs (see above), not a measured wall-clock + // duration; this is the core invariant under test. + // eslint-disable-next-line local/no-elapsed-assertion -- deterministic arithmetic, not measured timing + assert.equal(result.pct, elapsedMs / capMs); + assert.equal(isNearCap(result.pct), result.pct >= THRESHOLD_PCT); + }, + ), + { numRuns: 200 }, + ); +}); diff --git a/tests/ci-test-job-timeout-budget.test.cjs b/tests/ci-test-job-timeout-budget.test.cjs index 76f75ba01..e5180c701 100644 --- a/tests/ci-test-job-timeout-budget.test.cjs +++ b/tests/ci-test-job-timeout-budget.test.cjs @@ -31,6 +31,7 @@ const assert = require('node:assert/strict'); const fs = require('node:fs'); const path = require('node:path'); const yaml = require('js-yaml'); +const { COVERED } = require('../scripts/mutation-matrix.cjs'); const WORKFLOWS_DIR = path.join(__dirname, '..', '.github', 'workflows'); @@ -98,6 +99,19 @@ const LANE_COSTS = [ // around a minute. Listed so its budget cannot be dropped to nothing. evidence: 'targeted-only lane, ~1m observed', }, + { + job: 'smoke', + workflowFile: 'install-smoke.yml', + measuredMinutes: 2, + // Worst observed wall-clock across the two most recent PUSH-triggered + // (full-matrix, macos-latest included) runs: 65s on macos-latest, run + // 32260569855 (2026-08-19). A second push run (31240989202, 2026-08-08) + // measured 43-56s across its three jobs, all under this figure. PR-context + // runs are faster (~50-62s, ubuntu only, macOS full_only row skipped) and + // are not the binding case. Rounded up to whole minutes per this file's + // convention. + evidence: 'run 32260569855 — 65s, macos-latest push (full matrix)', + }, ]; function requiredBudgetMinutes(measuredMinutes, headroomFactor = HEADROOM_FACTOR) { @@ -110,13 +124,14 @@ function hasSufficientBudget(budgetMinutes, measuredMinutes, headroomFactor = HE } test('CI job timeout budgets carry headroom over measured cost (#2952)', async (t) => { - const workflow = loadWorkflow('test.yml'); - for (const lane of LANE_COSTS) { await t.test(`${lane.job} is budgeted above its measured cost`, () => { + const workflowFile = lane.workflowFile || 'test.yml'; + const workflow = loadWorkflow(workflowFile); + assert.ok( workflow.jobs && Object.prototype.hasOwnProperty.call(workflow.jobs, lane.job), - `.github/workflows/test.yml declares no job \`${lane.job}\`. If it was ` + `.github/workflows/${workflowFile} declares no job \`${lane.job}\`. If it was ` + 'renamed or removed, update LANE_COSTS in this file to match — do not ' + 'delete the entry to make this pass.', ); @@ -126,7 +141,7 @@ test('CI job timeout budgets carry headroom over measured cost (#2952)', async ( assert.equal( typeof budget, 'number', - `.github/workflows/test.yml jobs.${lane.job} must declare timeout-minutes`, + `.github/workflows/${workflowFile} jobs.${lane.job} must declare timeout-minutes`, ); assert.ok( hasSufficientBudget(budget, lane.measuredMinutes), @@ -170,3 +185,92 @@ test('CI job timeout budgets carry headroom over measured cost (#2952)', async ( assert.equal(requiredBudgetMinutes(10, 1.5), 15); }); }); + +test('mutation.yml mutate job timeout budgets (#4036)', async (t) => { + await t.test('mutate job timeout-minutes is matrix-driven, not a fixed literal', () => { + const workflow = loadWorkflow('mutation.yml'); + assert.equal( + workflow.jobs.mutate['timeout-minutes'], + '${{ matrix.timeoutMinutes }}', + 'mutation.yml jobs.mutate.timeout-minutes must stay matrix-driven so each covered ' + + 'module can declare its own per-shard budget via scripts/mutation-matrix.cjs — a ' + + 'fixed literal here would either under-budget a slow module or over-budget every ' + + 'fast one.', + ); + }); + + await t.test('every covered module declares a sane per-shard timeout', () => { + for (const [name, mod] of Object.entries(COVERED)) { + const timeoutMinutes = mod.timeoutMinutes || 15; + assert.ok( + Number.isInteger(timeoutMinutes) && timeoutMinutes >= 15, + `COVERED.${name}.timeoutMinutes resolves to ${timeoutMinutes}, but must be an ` + + 'integer >= 15 (the shared default) — a module\'s override must never budget ' + + 'BELOW the shared floor every other module gets for free.', + ); + } + }); + + await t.test('frontmatter mutate shard is budgeted above its measured cost', () => { + // Measured: CI run 33026833181 — 713s (11m53s) under the tap runner + // (coverageAnalysis: 'perTest'). See scripts/mutation-matrix.cjs's own + // comment on the `frontmatter` COVERED entry for the full citation. + const measuredMinutes = 12; // 713s rounded up + const declared = COVERED.frontmatter.timeoutMinutes || 15; + const required = requiredBudgetMinutes(measuredMinutes); + assert.ok( + hasSufficientBudget(declared, measuredMinutes), + `COVERED.frontmatter.timeoutMinutes is ${declared}, but the shard measured ` + + `${measuredMinutes}m (run 33026833181 — 713s) and needs at least ${required} — ` + + `${HEADROOM_FACTOR}x — so this module cannot quietly regress toward its cap.`, + ); + }); +}); + +test('near-cap check CI_JOB_TIMEOUT_MINUTES literals match each job\'s own timeout-minutes (#4036)', async (t) => { + const staticLanes = [ + { workflowFile: 'test.yml', jobKey: 'test', envLiteral: '15' }, + { workflowFile: 'test.yml', jobKey: 'test-full', envLiteral: '45' }, + { workflowFile: 'install-smoke.yml', jobKey: 'smoke', envLiteral: '12' }, + ]; + + for (const lane of staticLanes) { + await t.test(`${lane.workflowFile} jobs.${lane.jobKey}: CI_JOB_TIMEOUT_MINUTES matches timeout-minutes`, () => { + const workflow = loadWorkflow(lane.workflowFile); + const declared = workflow.jobs[lane.jobKey]['timeout-minutes']; + assert.equal( + String(declared), lane.envLiteral, + `.github/workflows/${lane.workflowFile} jobs.${lane.jobKey}.timeout-minutes is ${declared}, ` + + `but the near-cap check step's CI_JOB_TIMEOUT_MINUTES literal is hardcoded to '${lane.envLiteral}' ` + + '— these two must be updated together (GH Actions has no expression to read a sibling job-level ' + + 'key from within a step\'s env, so this parity test is the drift guard instead). Update BOTH the ' + + 'literal in this test AND the CI_JOB_TIMEOUT_MINUTES env value in the workflow step when the cap changes.', + ); + }); + } + + await t.test('ci-timeout-report.cjs JOB_RULES prefixes still match each job\'s declared name: template', () => { + const { JOB_RULES } = require('../scripts/ci-timeout-report.cjs'); + const testWorkflow = loadWorkflow('test.yml'); + + const testRule = JOB_RULES.find((r) => r.workflowFile === 'test.yml' && r.jobKey === 'test'); + assert.ok(testWorkflow.jobs.test.name.startsWith('test ('), + 'test.yml jobs.test.name no longer starts with "test (" — update JOB_RULES in scripts/ci-timeout-report.cjs to match'); + assert.equal(testRule.test('test (ubuntu-latest, 24, shard 1/3)'), true); + + const testFullRule = JOB_RULES.find((r) => r.workflowFile === 'test.yml' && r.jobKey === 'test-full'); + assert.ok(testWorkflow.jobs['test-full'].name.startsWith('full test ('), + 'test.yml jobs.test-full.name no longer starts with "full test (" — update JOB_RULES to match'); + assert.equal(testFullRule.test('full test (windows-latest, 24, shard 1/3)'), true); + + const testInertRule = JOB_RULES.find((r) => r.workflowFile === 'test.yml' && r.jobKey === 'test-inert'); + assert.equal(testWorkflow.jobs['test-inert'].name, 'test (inert CI)', + 'test.yml jobs.test-inert.name changed — update JOB_RULES to match'); + assert.equal(testInertRule.test('test (inert CI)'), true); + + const coverageGateRule = JOB_RULES.find((r) => r.workflowFile === 'test.yml' && r.jobKey === 'coverage-gate'); + assert.equal(testWorkflow.jobs['coverage-gate'].name, 'Coverage gate (merged shards)', + 'test.yml jobs.coverage-gate.name changed — update JOB_RULES to match'); + assert.equal(coverageGateRule.test('Coverage gate (merged shards)'), true); + }); +}); diff --git a/tests/ci-timeout-report.test.cjs b/tests/ci-timeout-report.test.cjs new file mode 100644 index 000000000..8b22b60bc --- /dev/null +++ b/tests/ci-timeout-report.test.cjs @@ -0,0 +1,264 @@ +'use strict'; + +/** + * tests/ci-timeout-report.test.cjs + * + * Unit tests for scripts/ci-timeout-report.cjs's pure exports (#4036). + * main() is impure orchestration requiring a live Octokit/GitHub Actions + * context and is intentionally NOT covered here. + */ + +const test = require('node:test'); +const assert = require('node:assert/strict'); + +const { + resolveJobTimeoutMinutes, + parseJobRecord, + buildReportLines, + dedupeAgainstHistory, +} = require('../scripts/ci-timeout-report.cjs'); + +test('resolveJobTimeoutMinutes', async (t) => { + await t.test('static job: resolves timeout-minutes from workflow YAML', () => { + const yamlText = [ + 'jobs:', + ' test:', + ' timeout-minutes: 15', + '', + ].join('\n'); + + const result = resolveJobTimeoutMinutes({ + jobName: 'test (ubuntu-latest, 24, shard 1/3)', + workflowFile: 'test.yml', + workflowYamlText: yamlText, + covered: null, + }); + + assert.equal(result, 15); + }); + + await t.test('mutation job: resolves override timeoutMinutes from COVERED', () => { + const result = resolveJobTimeoutMinutes({ + jobName: 'Stryker (frontmatter)', + workflowFile: 'mutation.yml', + workflowYamlText: null, + covered: { frontmatter: { timeoutMinutes: 20 }, 'adr-parser': {} }, + }); + + assert.equal(result, 20); + }); + + await t.test('mutation job: falls back to default 15 when no override', () => { + const result = resolveJobTimeoutMinutes({ + jobName: 'Stryker (adr-parser)', + workflowFile: 'mutation.yml', + workflowYamlText: null, + covered: { frontmatter: { timeoutMinutes: 20 }, 'adr-parser': {} }, + }); + + assert.equal(result, 15); + }); + + await t.test('mutation job: unknown module returns null', () => { + const result = resolveJobTimeoutMinutes({ + jobName: 'Stryker (totally-unknown-module)', + workflowFile: 'mutation.yml', + workflowYamlText: null, + covered: { frontmatter: { timeoutMinutes: 20 }, 'adr-parser': {} }, + }); + + assert.equal(result, null); + }); + + await t.test('test-inert resolves against the test-inert job key, not test', () => { + const yamlText = [ + 'jobs:', + ' test:', + ' timeout-minutes: 15', + ' test-inert:', + ' timeout-minutes: 2', + '', + ].join('\n'); + + const result = resolveJobTimeoutMinutes({ + jobName: 'test (inert CI)', + workflowFile: 'test.yml', + workflowYamlText: yamlText, + covered: null, + }); + + assert.equal(result, 2); + }); +}); + +test('parseJobRecord', async (t) => { + await t.test('still-running job (completed_at null) returns null', () => { + const result = parseJobRecord({ + job: { + name: 'test (ubuntu-latest, 24, shard 1/3)', + completed_at: null, + started_at: '2026-08-29T00:00:00Z', + run_id: 1, + head_sha: 'abc123', + runEvent: 'pull_request', + }, + workflowFile: 'test.yml', + workflowYamlText: 'jobs:\n test:\n timeout-minutes: 15\n', + covered: null, + }); + + assert.equal(result, null); + }); + + await t.test('untracked job name returns null', () => { + for (const jobName of ['preflight', 'changes', 'lint-tests']) { + const result = parseJobRecord({ + job: { + name: jobName, + completed_at: '2026-08-29T00:10:00Z', + started_at: '2026-08-29T00:00:00Z', + run_id: 1, + head_sha: 'abc123', + runEvent: 'pull_request', + }, + workflowFile: 'test.yml', + workflowYamlText: 'jobs:\n test:\n timeout-minutes: 15\n', + covered: null, + }); + + assert.equal(result, null, `expected null for job name ${jobName}`); + } + }); + + await t.test('valid smoke job returns a full record', () => { + const yamlText = [ + 'jobs:', + ' smoke:', + ' timeout-minutes: 12', + '', + ].join('\n'); + + const result = parseJobRecord({ + job: { + name: 'smoke (ubuntu-latest)', + started_at: '2026-08-29T00:00:00Z', + completed_at: '2026-08-29T00:06:00Z', + run_id: 42, + head_sha: 'deadbeef', + runEvent: 'push', + }, + workflowFile: 'install-smoke.yml', + workflowYamlText: yamlText, + covered: null, + }); + + assert.ok(result); + assert.equal(result.jobName, 'smoke (ubuntu-latest)'); + assert.equal(result.workflowFile, 'install-smoke.yml'); + assert.equal(result.runId, 42); + assert.equal(result.sha, 'deadbeef'); + assert.equal(result.runEvent, 'push'); + assert.equal(result.timeoutMinutes, 12); + assert.equal(typeof result.pct, 'number'); + }); +}); + +test('dedupeAgainstHistory', async (t) => { + await t.test('excludes only the record already present in history', () => { + const records = [ + { runId: 1, jobName: 'test (ubuntu-latest, 24, shard 1/3)', pct: 0.5 }, + { runId: 2, jobName: 'test (ubuntu-latest, 24, shard 2/3)', pct: 0.6 }, + ]; + const historyText = `${JSON.stringify({ runId: 1, jobName: 'test (ubuntu-latest, 24, shard 1/3)' })}\n`; + + const result = dedupeAgainstHistory(records, historyText); + + assert.equal(result.length, 1); + assert.equal(result[0].runId, 2); + }); + + await t.test('same runId, different jobName: both kept when history is empty', () => { + const records = [ + { runId: 1, jobName: 'test (ubuntu-latest, 24, shard 1/3)', pct: 0.5 }, + { runId: 1, jobName: 'test (ubuntu-latest, 24, shard 2/3)', pct: 0.6 }, + ]; + + const result = dedupeAgainstHistory(records, ''); + + assert.equal(result.length, 2); + }); + + await t.test('malformed history lines are skipped, not thrown', () => { + const records = [ + { runId: 1, jobName: 'test (ubuntu-latest, 24, shard 1/3)', pct: 0.5 }, + { runId: 2, jobName: 'test (ubuntu-latest, 24, shard 2/3)', pct: 0.6 }, + ]; + const historyText = [ + JSON.stringify({ runId: 1, jobName: 'test (ubuntu-latest, 24, shard 1/3)' }), + '', + 'not json{', + '', + ].join('\n'); + + const result = dedupeAgainstHistory(records, historyText); + + assert.equal(result.length, 1); + assert.equal(result[0].runId, 2); + }); +}); + +test('buildReportLines', async (t) => { + await t.test('end-to-end: only tracked+completed jobs produce records', () => { + const workflowYamlText = [ + 'jobs:', + ' test:', + ' timeout-minutes: 15', + '', + ].join('\n'); + + const runs = [ + { + run: { id: 1, head_sha: 'sha1', event: 'pull_request' }, + jobs: [ + { + name: 'test (ubuntu-latest, 24, shard 1/3)', + started_at: '2026-08-29T00:00:00Z', + completed_at: '2026-08-29T00:05:00Z', + }, + { + name: 'test (ubuntu-latest, 24, shard 2/3)', + started_at: '2026-08-29T00:00:00Z', + completed_at: null, + }, + { + name: 'lint-tests', + started_at: '2026-08-29T00:00:00Z', + completed_at: '2026-08-29T00:01:00Z', + }, + ], + }, + { + run: { id: 2, head_sha: 'sha2', event: 'push' }, + jobs: [ + { + name: 'test (ubuntu-latest, 24, shard 3/3)', + started_at: '2026-08-29T00:00:00Z', + completed_at: '2026-08-29T00:07:00Z', + }, + ], + }, + ]; + + const result = buildReportLines(runs, { workflowFile: 'test.yml', workflowYamlText, covered: null }); + + assert.equal(result.length, 2); + const names = result.map((r) => r.jobName).sort(); + assert.deepEqual(names, [ + 'test (ubuntu-latest, 24, shard 1/3)', + 'test (ubuntu-latest, 24, shard 3/3)', + ]); + for (const name of names) { + assert.equal(names.filter((n) => n === name).length, 1); + } + }); +}); diff --git a/tests/fixtures/install-tree/antigravity.json b/tests/fixtures/install-tree/antigravity.json index 0b6ec95e5..2aa171cf8 100644 --- a/tests/fixtures/install-tree/antigravity.json +++ b/tests/fixtures/install-tree/antigravity.json @@ -430,6 +430,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/augment.json b/tests/fixtures/install-tree/augment.json index 33ef834fe..1e2846c75 100644 --- a/tests/fixtures/install-tree/augment.json +++ b/tests/fixtures/install-tree/augment.json @@ -500,6 +500,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/claude-local.json b/tests/fixtures/install-tree/claude-local.json index d180006de..3676036a3 100644 --- a/tests/fixtures/install-tree/claude-local.json +++ b/tests/fixtures/install-tree/claude-local.json @@ -500,6 +500,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/claude.json b/tests/fixtures/install-tree/claude.json index e4d4d20c8..9234e6329 100644 --- a/tests/fixtures/install-tree/claude.json +++ b/tests/fixtures/install-tree/claude.json @@ -429,6 +429,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/cline.json b/tests/fixtures/install-tree/cline.json index 9061cad27..c2bd74306 100644 --- a/tests/fixtures/install-tree/cline.json +++ b/tests/fixtures/install-tree/cline.json @@ -392,6 +392,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/codebuddy.json b/tests/fixtures/install-tree/codebuddy.json index 80033205e..ff233ec9b 100644 --- a/tests/fixtures/install-tree/codebuddy.json +++ b/tests/fixtures/install-tree/codebuddy.json @@ -500,6 +500,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/codex.json b/tests/fixtures/install-tree/codex.json index 33d634a39..5bb869a38 100644 --- a/tests/fixtures/install-tree/codex.json +++ b/tests/fixtures/install-tree/codex.json @@ -431,6 +431,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/copilot.json b/tests/fixtures/install-tree/copilot.json index a2dfc0a36..f98192bab 100644 --- a/tests/fixtures/install-tree/copilot.json +++ b/tests/fixtures/install-tree/copilot.json @@ -392,6 +392,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/cursor.json b/tests/fixtures/install-tree/cursor.json index 5f4870a96..8cb07da4c 100644 --- a/tests/fixtures/install-tree/cursor.json +++ b/tests/fixtures/install-tree/cursor.json @@ -403,6 +403,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/hermes.json b/tests/fixtures/install-tree/hermes.json index 9243a9537..26dc283de 100644 --- a/tests/fixtures/install-tree/hermes.json +++ b/tests/fixtures/install-tree/hermes.json @@ -429,6 +429,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/kilo.json b/tests/fixtures/install-tree/kilo.json index d447453d7..154e553de 100644 --- a/tests/fixtures/install-tree/kilo.json +++ b/tests/fixtures/install-tree/kilo.json @@ -503,6 +503,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/kimi-code.json b/tests/fixtures/install-tree/kimi-code.json index 9cd3a0c33..391b54599 100644 --- a/tests/fixtures/install-tree/kimi-code.json +++ b/tests/fixtures/install-tree/kimi-code.json @@ -430,6 +430,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/kimi.json b/tests/fixtures/install-tree/kimi.json index 8de8044da..35c06483b 100644 --- a/tests/fixtures/install-tree/kimi.json +++ b/tests/fixtures/install-tree/kimi.json @@ -427,6 +427,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/opencode.json b/tests/fixtures/install-tree/opencode.json index b49094e86..ca6fc5b74 100644 --- a/tests/fixtures/install-tree/opencode.json +++ b/tests/fixtures/install-tree/opencode.json @@ -503,6 +503,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/pi.json b/tests/fixtures/install-tree/pi.json index a178f40e4..a68c00c88 100644 --- a/tests/fixtures/install-tree/pi.json +++ b/tests/fixtures/install-tree/pi.json @@ -396,6 +396,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/qwen.json b/tests/fixtures/install-tree/qwen.json index 323ed882c..228ff0937 100644 --- a/tests/fixtures/install-tree/qwen.json +++ b/tests/fixtures/install-tree/qwen.json @@ -429,6 +429,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/trae.json b/tests/fixtures/install-tree/trae.json index 027c10f81..9c6bb1909 100644 --- a/tests/fixtures/install-tree/trae.json +++ b/tests/fixtures/install-tree/trae.json @@ -390,6 +390,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/windsurf.json b/tests/fixtures/install-tree/windsurf.json index a6716c4e1..0d029ce51 100644 --- a/tests/fixtures/install-tree/windsurf.json +++ b/tests/fixtures/install-tree/windsurf.json @@ -393,6 +393,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs", diff --git a/tests/fixtures/install-tree/zcode.json b/tests/fixtures/install-tree/zcode.json index 943441131..d6a63ba7c 100644 --- a/tests/fixtures/install-tree/zcode.json +++ b/tests/fixtures/install-tree/zcode.json @@ -461,6 +461,7 @@ "scripts/gen-loop-host-contract.cjs", "scripts/lib/alias-drift-families.cjs", "scripts/lib/allowlist-ratchet.cjs", + "scripts/lib/ci-job-timing.cjs", "scripts/lib/cli-exit.cjs", "scripts/lib/drift-scan.cjs", "scripts/lib/exit-code-registry.cjs",