enhance(#4036): persist CI shard/job timeout-vs-cap trending, warn at 90% (#4043)

* feat(#4036): persist CI shard/job timeout-vs-cap trending, warn at 90%

Adds two new mechanisms plus an audit-coverage extension:

- scripts/lib/ci-job-timing.cjs: shared elapsed-vs-cap arithmetic
- scripts/ci-check-job-near-cap.cjs: in-job advisory near-cap check,
  wired into test/test-full/mutate/smoke as each job's last step
- scripts/ci-timeout-report.cjs + .github/workflows/ci-timeout-report.yml:
  scheduled REST-API poll that appends new records to
  tests/ci-timeout-budget-history.jsonl and opens a small data-only PR
- tests/ci-test-job-timeout-budget.test.cjs: extended to cover mutate
  (mutation.yml) and smoke (install-smoke.yml), which previously had no
  headroom-factor gate coverage at all

Does not change any timeout-minutes value, shard composition, or shard-1
contents — those stay maintainer policy calls per the issue's own scope.

* fix(#4036): address two-orthogonal-review findings

- Parity tests guarding the two hand-duplicated literals this design
  cannot single-source through GH Actions YAML: CI_JOB_TIMEOUT_MINUTES
  vs each job's own timeout-minutes, and ci-timeout-report.cjs's
  JOB_RULES name-prefixes vs each job's actual name: template.
- Thread run.event through as runEvent on every persisted record, so
  PR-context and push-context install-smoke timings (genuinely
  different matrix shape) are distinguishable in the history rather
  than silently conflated under one job name.
- Replace the Windows near-cap start-time step's ambiguous PowerShell
  +/>> precedence with GitHub's documented string-interpolation form.
- Move github.run_id out of direct ${{ }} shell interpolation into an
  env: var in the new scheduled workflow, per this repo's own
  expression-injection-safe convention.

* test(#4036): regenerate golden install-tree fixtures for scripts/lib/ci-job-timing.cjs

npm run gen:install-tree — scripts/ ships wholesale into the installed
package (per ADR/known-defect precedent from #4012's own PR history: a
new scripts/lib/*.cjs file needs its golden entry regenerated or every
runtime's install-tree test fails). Confirmed via gsd-test: this was the
sole cause of the first real verification run's 25 failures (all in
tests/golden-install-tree.test.cjs, one per runtime). Top-level
scripts/*.cjs files (ci-check-job-near-cap.cjs, ci-timeout-report.cjs)
are not individually tracked in these fixtures — consistent with every
other existing top-level scripts/*.cjs file, so no entry was expected
or added for those two.

* fix(#4036): register new lib file with installer, fix H1 shell policy

- bin/install.js: add ci-job-timing.cjs to GSD_SCRIPTS_LIB_FILES (a
  hand-maintained registry, not generated — tests/install.test.cjs
  asserts every scripts/lib/ file is enumerated here)
- test.yml: replace the two OS-conditional "Record job start time"
  step pairs (test + test-full jobs) with a single unconditional
  `node -e` step. The prior pair's Windows variant declared an
  explicit shell: pwsh, which scripts/workflow-policy.cjs's H1 checker
  statically flags against every OS a job's matrix can realize,
  independent of the step's own if: gate. A single Node one-liner
  needs no shell override at all — it's syntactically valid and
  behaves identically under bash, zsh, and pwsh — which is both H1
  compliant and removes the last OS-specific shell syntax from this
  change entirely.

Both defects were found by a real gsd-test run, not local gates —
lint:ci and build:lib were clean throughout because neither the
scripts/lib/ install-manifest parity check nor the H1 shell-policy
baseline runs as part of lint:ci; both are gsd-test-only suites.

* docs(#4036): how-to for reading CI timeout budget signals

The phase-gate docs check correctly flagged the enablement sequence as
3 real steps (read the near-cap warning, find the accumulated trend
file, pick the right maintainer lever) — a reference table can't carry
a sequence. Adds docs/how-to/read-ci-timeout-signals.md, indexed from
docs/README.md.

* chore(#4036): backfill changeset PR number (4043)

---------

Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
2026-08-29 16:13:15 -04:00
committed by GitHub
parent b431ae9f0d
commit 370cfc6680
34 changed files with 1158 additions and 5 deletions

View File

@@ -527,6 +527,43 @@ drift costs chunk *balance*, never a red build. A count-based floor additionally
guarantees the packer never produces fewer chunks than plain count-based packing
would, so a badly stale table cannot collapse the suite into a few fat chunks.
### CI job timeout budgets: report + near-cap warning (#4036)
Every matrixed job — `test` and `test-full` in `test.yml`, `mutate` in
`mutation.yml`, `smoke` in `install-smoke.yml` — declares a `timeout-minutes`
cap. `tests/ci-test-job-timeout-budget.test.cjs` enforces that each checked-in
cap stays at least the **headroom factor** (1.5x) above a documented,
hand-measured cost for that job — now all four of the jobs above, not just
`test`/`test-full`/`coverage-gate`/`test-inert` as before.
Two runtime mechanisms sit on top of that static gate, both new in #4036:
- **In-job near-cap check** (`scripts/ci-check-job-near-cap.cjs`) — the last
step of each of the four jobs computes elapsed-vs-cap from a start-time
marker recorded as that job's first step. At >=90% of budget it emits a
`::warning::` annotation (visible in the PR Checks UI) and a
`$GITHUB_STEP_SUMMARY` block. Advisory only — it never fails the job. Known
limit: it cannot fire for a job actually killed by the timeout, since a
killed job never reaches its last step. That case is caught by the second
mechanism instead.
- **Scheduled trending report** (`.github/workflows/ci-timeout-report.yml`,
`scripts/ci-timeout-report.cjs`) — runs daily and on `workflow_dispatch`. It
polls GitHub's Actions REST API for recently completed jobs across
`test.yml`, `mutation.yml`, and `install-smoke.yml`, resolves each job's
declared cap (a literal `timeout-minutes` for `test`/`test-full`/`smoke`, or
`scripts/mutation-matrix.cjs`'s `COVERED[<module>].timeoutMinutes` for
`mutate`'s per-module shards), and appends any new `(runId, jobName)`
records to `tests/ci-timeout-budget-history.jsonl`. Unlike the in-job check,
this also catches jobs killed by an actual timeout breach — GitHub's Jobs
API still reports `started_at`/`completed_at` for a cancelled job. Each run
opens a small, data-only PR carrying that run's new rows, since `next` is a
protected branch and nothing pushes to it directly — the same constraint
`auto-backmerge.yml` already works within.
This does not retune any `timeout-minutes` value, rebalance shard composition,
or trim what runs in shard 1 — those stay maintainer policy calls made from
the accumulated history, not something either mechanism decides on its own.
### How-to: regenerate the timing table
Regenerate when the suite's cost profile has visibly drifted — after adding or