golem15/msd-core
3
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

enhance(#3915): stryker on the official tap runner, 29m to 12m on the critical path (#3919)

Browse Source 
* enhance(#3915): stryker on the official tap runner, per-test coverage

The 'command' runner is the one runner Stryker excludes from coverage
analysis, which forced coverageAnalysis:'off' and made mutation cost
strictly linear in (mutants x whole-shard test time). The frontmatter
shard measured 1751s against 212s for the next slowest.

Swap to @stryker-mutator/tap-runner (official plugin, peer-pinned to the
@stryker-mutator/core@9.6.1 already installed) and turn coverage analysis
on, so Stryker re-runs only the test files that cover each mutated line.

Per-shard injection moves from a MUTATION_TEST_CMD command string to
MUTATION_TEST_FILES, read by a new fail-closed resolveMutationTestFiles()
that mirrors resolveMutationBreak. It stays derived from
scripts/mutation-matrix.cjs and now has a single owner: the union moves
behind allCoveredTests() and stryker.config.mjs no longer imports COVERED.
The resolver existence-checks every entry, because the tap runner resolves
tap.testFiles with glob() and a non-matching pattern yields an empty list
silently - a fast, confident, meaningless run.

tap.forceBail is off by measurement, not preference: a structural AST
audit found 3 of 26 shard test files spawn subprocesses, and bail fires on
every killed mutant, so leaving it on would kill processes mid-spawnSync
and orphan their children.

The matrix 'isolation' field is removed - per-file process isolation is
inherent to the tap runner, so the field had no consumer left.

Score arithmetic is unchanged and now pinned: mutationScore counts
NoCoverage in the same denominator as Survived, which both Stryker's
thresholds.break and check-mutation-score-ratchet.cjs read. A new
non-vacuity test proves it diverges from mutationScoreBasedOnCoveredCode,
so the gate cannot be quietly swapped to the field that would make every
floor trivially satisfiable.

Refs #3915

* fix(#3915): enforce the resolver's documented containment contract

Review findings, all fixed in place.

resolveMutationTestFiles claimed to verify each entry exists 'relative to
the repo root' but used a bare fs.existsSync(path.join(...)), which
accepts an existing DIRECTORY and lets ../ segments escape the root
(path.join('/repo/root','../../etc/passwd') resolves to /etc/passwd).
Not reachable from PR content - the value only ever comes from the static
COVERED registry via CI env - but a fail-closed contract that overstates
its own guarantee is a defect in the contract. Each entry is now resolved,
rejected if path.relative puts it outside the root, and required to be a
regular file. Three hostile-input tests added; the original missing-file
wording is preserved so the existing assertion still binds.

Also: removed a stale buildResult comment still naming the isolation
field this branch deleted; hoisted one top-level path require in place of
three inline ones; de-duplicated the derived-test-list expression in the
tests to a single const, deliberately still re-derived from COVERED
rather than calling allCoveredTests() so the assertion cannot become a
tautology; tightened the workflow-parity assertion to exact equality;
changed the tap-runner range from an exact 9.6.1 to ^9.6.1 so it tracks
the caret-ranged core its peerDependency pins exactly.

Regenerated examples/dynamic-context-management/CONTEXT-INDEX.json, which
the earlier CONTEXT.md edit left stale - lint:ci was red on
lint-example-parser-parity until it was refreshed.

Refs #3915

* test(#3915): kill model-catalog survivors, set frontmatter budget from measurement

From mutation run 33026833181 (all 13 shards, dispatched on this branch before any PR).

WALL TIME: frontmatter measured 713s (11m53s) vs 1751s (29m11s) on the command runner, a 59% cut. timeoutMinutes 60 -> 20 (1.68x measured). Not deleted outright: the shared 15-minute default would leave only 21% headroom, and this module's mutant count grew 1.8x in one change.

MODEL-CATALOG: came back 57.91 against its floor of 58. Diagnosed from the report JSON, not assumed - all 24 of its new RuntimeError mutants are in the load-time catalog bootstrap, so mutating them makes the module throw at require. Under node --test that is a failed test file (Killed); under the tap runner the process dies before emitting TAP, which Stryker classifies RuntimeError and excludes from the denominator. Add them back as killed and the shard is 248/416 = 59.62, the pre-change number exactly. Detection did not regress, classification changed.

The floor is NOT lowered to absorb that. 11 new behavioural tests target ~42 genuinely surviving mutants: exact Set equality on the EFFORT_RENDERING/EFFORT_ARGV supported sets, an exact-string table render that makes the column-width arithmetic observable (padEnd never truncates, so the old substring assertion could not see a too-small width), clampEffortForHost's full null matrix plus a spoofed-toString host, and a prototype-pollution guard driven through Object.fromEntries.

Mutants judged equivalent were skipped rather than papered over; reasoning is in the phase artifacts. All 15 new assertions verified against unmutated code. lint:ci exit 0.

Refs #3915

* test(#3915): ratchet model-catalog's floor to 74 on measured 75.26%

CI run 33029755081 measured model-catalog at 75.26% (295 killed / 49 survived / 48 no-coverage / 24 runtime-error, totalValid 392), so check-mutation-score-ratchet.cjs correctly failed the shard for unclaimed headroom: 17.26 points above the declared floor of 58, well past the 5-point slack.

Floor raised 58 -> 74 (floor(75.26)-1, this file's documented convention), with RATCHET_BASELINE updated in the same diff as the equality assertion requires.

Worth recording why the number moved this far. The shard first came back at 57.91 under the tap runner and the temptation was to lower the floor to match. The drop was not a regression: all 24 of its RuntimeError mutants sit in the load-time catalog bootstrap, and adding them back as killed reproduces 248/416 = 59.62, the pre-swap figure exactly. Rather than absorb a reporting artifact by weakening the gate, 11 behavioural tests went after the genuinely surviving mutants and killed 68 of them - carrying the module from 59.62 past its old ceiling to 75.26, within reach of the ADR-456 target of 80.

The #3007 measurement is kept as clearly-labelled prior context rather than deleted, so the entry does not read as carrying two current numbers.

Refs #3915

---------

Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
2026-08-26 22:43:14 -04:00
committed by GitHub
parent 8641d0a468
commit 7e9d33c378
12 changed files with 1190 additions and 344 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
.github/workflows/mutation.yml vendored
View File

@@ -110,7 +110,7 @@ jobs:
runs-on: ubuntu-latest
# Per-shard; lower than the old 30-min serial budget. Matrix-driven so only shards that
# document a measured need (mutation-matrix.cjs COVERED[<module>].timeoutMinutes, e.g.
# frontmatter's 180) get more; every other shard keeps the 15-minute default emitted by
# frontmatter's 60) get more; every other shard keeps the 15-minute default emitted by
# buildResult() in scripts/mutation-matrix.cjs.
timeout-minutes: ${{ matrix.timeoutMinutes }}
strategy:
@@ -133,7 +133,10 @@ jobs:
run: npm ci
- name: Run Stryker — ${{ matrix.name }}
# MUTATION_TEST_CMD scopes the command runner to only this module's tests.
# MUTATION_TEST_FILES supplies the per-shard tap.testFiles list to
# stryker.config.mjs (resolveMutationTestFiles). With coverageAnalysis:
# perTest, Stryker re-runs only the files that actually cover each mutant
# rather than the whole list per mutant (#3915: tap-runner swap).
# --mutate scopes mutation to only the changed module's built artifact.
# --incremental reuses cached results for unchanged mutants.
# MUTATION_BREAK passes the per-module minScore ratchet floor to
@@ -147,15 +150,14 @@ jobs:
# repo's no-defer rule.
env:
NODE_OPTIONS: '--max-old-space-size=4096'
MUTATION_TEST_CMD: node --test --test-isolation=${{ matrix.isolation }} ${{ matrix.tests }}
MUTATION_TEST_FILES: ${{ matrix.tests }}
MUTATION_BREAK: ${{ matrix.minScore }}
MODULE_NAME: ${{ matrix.name }}
MUTATE_GLOB: ${{ matrix.mutate }}
MODULE_TESTS: ${{ matrix.tests }}
run: |
echo "Module: ${MODULE_NAME}"
echo "Mutate: ${MUTATE_GLOB}"
echo "Tests: ${MODULE_TESTS}"
echo "Tests: ${MUTATION_TEST_FILES}"
echo "MinScore: ${MUTATION_BREAK}"
npx stryker run --incremental --mutate "${MUTATE_GLOB}"

2
CONTEXT.md
View File

@@ -609,6 +609,8 @@ The prompt-level data/instruction isolation seam for untrusted web/document ingr
`RULESET.TESTS.clock-seam=concurrency logic must accept an optional {clock=Date} parameter; tests control time via t.mock.timers.enable(['Date']) + t.mock.timers.setTime(0) + t.mock.timers.tick(N); real OS scheduler races are not a permitted test pattern after ADR 456 (2026-05-28); real-race tests are deleted once deterministic seam tests cover the same logical path; clock.cjs realClock adds nowIso() (→ new Date(this.now()).toISOString()) and today() (→ nowIso().split('T')[0]) so all date-stamping in state.cjs routes through the seam; subprocess time-pin adapter: set GSD_TEST_MODE=1 + GSD_NOW_MS=<epoch-ms> in runGsdTools env to pin the date written by the SUT without touching real wall-clock (issue #474)`
`RULESET.TESTS.property-based-testing=modules implementing parsing / transformation / budget-limit / bijective contracts must include at least one fast-check (fc) property test asserting a domain invariant; invariant categories: round-trip, monotonicity, boundary-containment, idempotency; property tests live in *.test.cjs alongside unit tests; CI signal: Stryker mutation score below 80% blocks merge`
`RULESET.TESTS.mutation-score=Stryker runs incremental (--since origin/next) on ubuntu-latest/Node24 CI leg; default threshold 80% killed/total; surviving mutants in scope block merge unless path is listed in stryker.config.mjs with documented reason; treat surviving mutant as a failing test specification`
`RULESET.TESTS.mutation-runner=Stryker executes every shard through the OFFICIAL @stryker-mutator/tap-runner (testRunner:'tap'), never the built-in 'command' runner (#3915); 'command' is the one runner Stryker excludes from coverage analysis, which forced coverageAnalysis:'off' and made cost strictly linear in (mutants x whole-shard test time) — the frontmatter shard measured 1751s on run 33021042847 vs 212s for the next slowest. tap.testFiles is injected per shard via MUTATION_TEST_FILES (mutation.yml env <- matrix.tests <- scripts/mutation-matrix.cjs buildResult); resolveMutationTestFiles is the SINGLE fail-closed reader and existence-checks every entry, because the tap runner's findTestyLookingFiles resolves the list with glob() and a non-matching pattern yields an EMPTY list SILENTLY (a fast, confident, meaningless run). tap.forceBail is FALSE by measurement, not preference: 3 of 26 shard test files spawn subprocesses (config-schema.property, core-utils, feat-3881-yaml-parser-consequences) and bail fires on every KILLED mutant, so leaving it on kills processes mid-spawnSync and orphans their children; Stryker's separate disableBail still skips remaining FILES, which is most of the win. tap.nodeArgs and top-level buildCommand stay UNSET so no rebuild lands between mutation and test (ADR-457). Coverage granularity is per FILE, not per test ("a test is always a test file"), so the #2790 excludeTests bans on spawn-heavy integration files remain necessary and unchanged`
`RULESET.TESTS.mutation-score-denominator=the gated number is mutation-testing-metrics' mutationScore = totalDetected/totalValid, which counts NoCoverage in the denominator EXACTLY as Survived; both Stryker's own thresholds.break (core dist/src/reporters/mutation-test-report-helper.js) and scripts/check-mutation-score-ratchet.cjs read THAT field, which is what makes the #3915 coverageAnalysis 'off'->'perTest' switch score-neutral. NEVER gate on mutationScoreBasedOnCoveredCode — it EXCLUDES NoCoverage and inflates sharply under perTest (measured on a synthetic report: 8 killed/2 survived = 80 and 80; 8 killed/2 noCoverage = 80 and 100), so swapping to the better-sounding field would make every minScore floor trivially satisfiable and the gate decorative. Under the pre-#3915 coverageAnalysis:'off' the two fields were ALWAYS identical (noCoverage was structurally 0), which is why nothing had ever pinned the choice; tests/mutation-score-ratchet.test.cjs now pins it with a non-vacuity assertion that the two numbers genuinely diverge`
`RULESET.TESTS.delete-bad-tests=pass-always / vacuous-truth / source-grep / elapsed-time / real-race / permanent-allow-test-rule tests are DELETED and replaced with compliant tests in the same PR; not skipped, not commented out, not permanently exempted; replacement must cover the same logical path via typed-surface assertion or clock-seam pattern`
`RULESET.TESTS.eslint-harness=ADR 452 (2026-05-28): ESLint flat config + typescript-eslint + eslint-plugin-n + eslint-plugin-no-only-tests + local plugin at eslint-rules/ (repo root, NOT scripts/eslint-rules/); replaces scripts/lint-*.cjs regex scanners (fully removed in #632); all three test-rigor rules now ship at error in tests/**/*.test.cjs scope: local/no-source-grep and local/no-magic-sleep-in-tests promoted by #3313, local/no-elapsed-assertion promoted by #3331 once #3314 delivered its ADR-456 §(a) precondition (epic #1885 was subsumed into epic #3053 and closed stale before this promotion landed)`

14
docs/CONTEXT-INDEX.json
View File

@@ -1,6 +1,6 @@
{
"schemaVersion": 1,
"count": 261,
"count": 263,
"classes": {
"ARCH": 1,
"CI": 2,
@@ -17,7 +17,7 @@
"PROC": 14,
"PROHIB": 10,
"RELEASE-NOTES": 31,
"RULESET": 58,
"RULESET": 60,
"SESSION": 9,
"WAVE": 5,
"WORKSTREAM": 5,
@@ -1094,11 +1094,21 @@
"klass": "RULESET",
"value": "module-level const src = readFileSync(...) throws before any test() registers — wrap in try/catch in test() or use lazy load"
},
{
"id": "RULESET.TESTS.mutation-runner",
"klass": "RULESET",
"value": "Stryker executes every shard through the OFFICIAL @stryker-mutator/tap-runner (testRunner:'tap'), never the built-in 'command' runner (#3915); 'command' is the one runner Stryker excludes from coverage analysis, which forced coverageAnalysis:'off' and made cost strictly linear in (mutants x whole-shard test time) — the frontmatter shard measured 1751s on run 33021042847 vs 212s for the next slowest. tap.testFiles is injected per shard via MUTATION_TEST_FILES (mutation.yml env <- matrix.tests <- scripts/mutation-matrix.cjs buildResult); resolveMutationTestFiles is the SINGLE fail-closed reader and existence-checks every entry, because the tap runner's findTestyLookingFiles resolves the list with glob() and a non-matching pattern yields an EMPTY list SILENTLY (a fast, confident, meaningless run). tap.forceBail is FALSE by measurement, not preference: 3 of 26 shard test files spawn subprocesses (config-schema.property, core-utils, feat-3881-yaml-parser-consequences) and bail fires on every KILLED mutant, so leaving it on kills processes mid-spawnSync and orphans their children; Stryker's separate disableBail still skips remaining FILES, which is most of the win. tap.nodeArgs and top-level buildCommand stay UNSET so no rebuild lands between mutation and test (ADR-457). Coverage granularity is per FILE, not per test (\"a test is always a test file\"), so the #2790 excludeTests bans on spawn-heavy integration files remain necessary and unchanged"
},
{
"id": "RULESET.TESTS.mutation-score",
"klass": "RULESET",
"value": "Stryker runs incremental (--since origin/next) on ubuntu-latest/Node24 CI leg; default threshold 80% killed/total; surviving mutants in scope block merge unless path is listed in stryker.config.mjs with documented reason; treat surviving mutant as a failing test specification"
},
{
"id": "RULESET.TESTS.mutation-score-denominator",
"klass": "RULESET",
"value": "the gated number is mutation-testing-metrics' mutationScore = totalDetected/totalValid, which counts NoCoverage in the denominator EXACTLY as Survived; both Stryker's own thresholds.break (core dist/src/reporters/mutation-test-report-helper.js) and scripts/check-mutation-score-ratchet.cjs read THAT field, which is what makes the #3915 coverageAnalysis 'off'->'perTest' switch score-neutral. NEVER gate on mutationScoreBasedOnCoveredCode — it EXCLUDES NoCoverage and inflates sharply under perTest (measured on a synthetic report: 8 killed/2 survived = 80 and 80; 8 killed/2 noCoverage = 80 and 100), so swapping to the better-sounding field would make every minScore floor trivially satisfiable and the gate decorative. Under the pre-#3915 coverageAnalysis:'off' the two fields were ALWAYS identical (noCoverage was structurally 0), which is why nothing had ever pinned the choice; tests/mutation-score-ratchet.test.cjs now pins it with a non-vacuity assertion that the two numbers genuinely diverge"
},
{
"id": "RULESET.TESTS.no-dead-regex-in-includes",
"klass": "RULESET",

538
examples/dynamic-context-management/CONTEXT-INDEX.json
View File

File diff suppressed because one or more lines are too long

92
package-lock.json generated
View File

@@ -21,6 +21,7 @@
"devDependencies": {
"@eslint/js": "^9.39.4",
"@stryker-mutator/core": "^9.6.1",
"@stryker-mutator/tap-runner": "^9.6.1",
"@types/node": "^22.19.19",
"c8": "^11.0.0",
"eslint": "^9.39.4",
@@ -1747,6 +1748,26 @@
"node": ">=20.0.0"
}
},
"node_modules/@stryker-mutator/tap-runner": {
"version": "9.6.1",
"resolved": "https://registry.npmjs.org/@stryker-mutator/tap-runner/-/tap-runner-9.6.1.tgz",
"integrity": "sha512-b5ryfiRQHH5VoWP++VEA9KYiU6lhVbE9znooFaWRr7umaAtqKmlWrFinKA6fghwiFpdxerRpctLDT8uVKzeQEw==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"@stryker-mutator/api": "9.6.1",
"@stryker-mutator/util": "9.6.1",
"glob": "~13.0.0",
"tap-parser": "~17.0.0",
"tslib": "~2.8.0"
},
"engines": {
"node": ">=14.18.0"
},
"peerDependencies": {
"@stryker-mutator/core": "9.6.1"
}
},
"node_modules/@stryker-mutator/util": {
"version": "9.6.1",
"resolved": "https://registry.npmjs.org/@stryker-mutator/util/-/util-9.6.1.tgz",
@@ -3010,6 +3031,16 @@
"node": ">= 0.6"
}
},
"node_modules/events-to-array": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/events-to-array/-/events-to-array-2.0.3.tgz",
"integrity": "sha512-f/qE2gImHRa4Cp2y1stEOSgw8wTFyUdVJX7G//bMwbaV9JqISFxg99NbmVQeP7YLnDUZ2un851jlaDrlpmGehQ==",
"dev": true,
"license": "ISC",
"engines": {
"node": ">=12"
}
},
"node_modules/eventsource": {
"version": "3.0.7",
"resolved": "https://registry.npmjs.org/eventsource/-/eventsource-3.0.7.tgz",
@@ -4870,6 +4901,37 @@
"node": ">=8"
}
},
"node_modules/tap-parser": {
"version": "17.0.0",
"resolved": "https://registry.npmjs.org/tap-parser/-/tap-parser-17.0.0.tgz",
"integrity": "sha512-Na7kB4ML7T77abJtYIlXh/aJcz54Azv0iAtOaDnLqsL4uWjU40uNFIFnZ5IvnGTuCIk5M6vjx7ZsceNGc1mcag==",
"dev": true,
"license": "BlueOak-1.0.0",
"dependencies": {
"events-to-array": "^2.0.3",
"tap-yaml": "3.0.0"
},
"bin": {
"tap-parser": "bin/cmd.cjs"
},
"engines": {
"node": ">= 18.6.0"
}
},
"node_modules/tap-yaml": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/tap-yaml/-/tap-yaml-3.0.0.tgz",
"integrity": "sha512-qtbgXJqE9xdWqlE520y+vG4c1lgqWrDHN7Y2YcrV1XudLuc2Y5aMXhAyPBGl57h8MNoprvL/mAJiISUIadvS9w==",
"dev": true,
"license": "BlueOak-1.0.0",
"dependencies": {
"yaml": "^2.4.1",
"yaml-types": "^0.3.0"
},
"engines": {
"node": ">= 18.6.0"
}
},
"node_modules/tapable": {
"version": "2.3.3",
"resolved": "https://registry.npmjs.org/tapable/-/tapable-2.3.3.tgz",
@@ -5298,6 +5360,36 @@
"dev": true,
"license": "ISC"
},
"node_modules/yaml": {
"version": "2.9.0",
"resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.0.tgz",
"integrity": "sha512-2AvhNX3mb8zd6Zy7INTtSpl1F15HW6Wnqj0srWlkKLcpYl/gMIMJiyuGq2KeI2YFxUPjdlB+3Lc10seMLtL4cA==",
"dev": true,
"license": "ISC",
"bin": {
"yaml": "bin.mjs"
},
"engines": {
"node": ">= 14.6"
},
"funding": {
"url": "https://github.com/sponsors/eemeli"
}
},
"node_modules/yaml-types": {
"version": "0.3.0",
"resolved": "https://registry.npmjs.org/yaml-types/-/yaml-types-0.3.0.tgz",
"integrity": "sha512-i9RxAO/LZBiE0NJUy9pbN5jFz5EasYDImzRkj8Y81kkInTi1laia3P3K/wlMKzOxFQutZip8TejvQP/DwgbU7A==",
"dev": true,
"license": "ISC",
"engines": {
"node": ">= 16",
"npm": ">= 7"
},
"peerDependencies": {
"yaml": "^2.3.0"
}
},
"node_modules/yargs": {
"version": "17.7.2",
"resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.2.tgz",

1
package.json
View File

@@ -66,6 +66,7 @@
"devDependencies": {
"@eslint/js": "^9.39.4",
"@stryker-mutator/core": "^9.6.1",
"@stryker-mutator/tap-runner": "^9.6.1",
"@types/node": "^22.19.19",
"c8": "^11.0.0",
"eslint": "^9.39.4",

209
scripts/mutation-matrix.cjs
View File

@@ -31,6 +31,7 @@
const { execFileSync } = require('child_process');
const fs = require('fs');
const path = require('node:path');
const { ExitError, runMain } = require('./lib/cli-exit.cjs');
@@ -92,11 +93,14 @@ function readStdinSync() {
// confirmed equivalent mutant is acceptable.
//
// HOW TO UPDATE:
// 1. The per-module Stryker shard CANNOT be run locally: Stryker's command
// runner invokes `node --test` once per mutant (see stryker.config.mjs),
// and this repo hard-blocks local `node --test` via
// .claude/hooks/block-local-node-test.sh. Push the branch instead and
// let CI run the shard for the changed module.
// 1. The per-module Stryker shard CANNOT be run locally: Stryker's tap
// runner (see stryker.config.mjs) spawns
// `node --test-reporter=tap -r <hook> <testFile>` once per covering test
// file per mutant, and .claude/hooks/block-local-node-test.sh's matcher
// still denies that form — its pattern `node(\s+-\S+)*\s+--test([\s=-]|$)`
// matches `--test-reporter` because `-` is in the trailing character
// class. Push the branch instead and let CI run the shard for the
// changed module.
// 2. Read the measured score from the CI shard's output.
// 3. Set minScore = floor(measured) - 1 (never lower than current value)
// and update the matching RATCHET_BASELINE entry in the same diff.
@@ -168,7 +172,7 @@ const TARGET_MUTATION_SCORE = 80;
// named in extraTests, or named in excludeTests — so a file that newly starts
// matching the naming rule (like #3888's four files would have, had they been
// named `frontmatter*`) cannot silently fall through the cracks again.
const TESTS_DIR = require('node:path').join(__dirname, '..', 'tests');
const TESTS_DIR = path.join(__dirname, '..', 'tests');
let _testRequireCache = null;
/**
@@ -191,7 +195,7 @@ function scanTestRequires() {
entries = [];
}
for (const file of entries) {
const text = fs.readFileSync(require('node:path').join(TESTS_DIR, file), 'utf8');
const text = fs.readFileSync(path.join(TESTS_DIR, file), 'utf8');
let m;
REQUIRE_RE.lastIndex = 0;
while ((m = REQUIRE_RE.exec(text))) {
@@ -403,27 +407,26 @@ const COVERED = {
'frontmatter.test.cjs',
],
minScore: 65,
// Wall-time projection, re-derived after piece 1 (dropping frontmatter.test.cjs) using
// this file's own documented method. Mutant-count factor is unchanged: source grew 1.8x
// for #3881 (1030 -> ~1850 mutants; see the #3888-era note this superseded for that
// derivation). Per-run test-command cost is re-measured on the CURRENT 6-file derived
// set (frontmatter.property/.unit/-golden-parity/-roundtrip.property + unusable-input +
// feat-3881-yaml-parser-consequences — the shard minus frontmatter.test.cjs and minus
// frontmatter-cli.test.cjs, neither of which was ever in a measured baseline): 1520ms,
// vs the documented OLD 3-file baseline of 593ms — a 2.56x per-run cost increase (down
// from the pre-piece-1 8x, since the file responsible for 3132ms of the old 4669ms
// 7-file run is gone). Applying both factors the same way the prior note did: 586s
// (documented 3-file/1030-mutant CI baseline) * 1.8 (mutants) * 2.56 (test cost) ~=
// 2700s (~45 minutes). Set to 60 minutes for margin above that projection (the same
// ~1.3x margin ratio the prior 180-minute budget used over its own 140-minute
// projection), well under GitHub Actions' 360-minute job ceiling and a 3x cut from the
// previous 180. Scoped to this shard only via timeoutMinutes below — every other shard
// keeps the 15-minute default.
timeoutMinutes: 60,
// isolation: intentionally NOT set (defaults to 'process' below) — unchanged from the
// prior audit: 'none' showed no reliable win once the test set grew past 3 files
// (overlapping distributions), and dropping frontmatter.test.cjs only shrinks the set
// further, so there is no new basis to revisit that call.
// MEASUREMENT, not a projection. Under the tap runner with coverageAnalysis: 'perTest'
// (#3915), Stryker now re-runs only the test files that cover each mutated line instead
// of all six files for every one of ~1900 mutants. Measured result: the frontmatter shard
// completed in 713s (11m53s) — GitHub Actions run 33026833181, job wall time including
// checkout and `npm ci` — against 1751s (29m11s) on the command runner in run
// 33021042847. A 59% reduction.
// 20 minutes is 1.68x the measured 713s. The override is not simply deleted because the
// shared default is 15 minutes, which 11m53s would fit inside — but only at 79% of
// budget — and this module's mutant count grew 1.8x in a single change (#3881), so a
// shard sitting at 79% of the shared default is one growth spurt from a red lane. 20
// keeps a real margin while still cutting the previous 60-minute budget by 3x.
// Scoped to this shard only via timeoutMinutes below — every other shard keeps the
// 15-minute default emitted by buildResult(), well under GitHub Actions' 360-minute job
// ceiling.
timeoutMinutes: 20,
// isolation: no knob left to tune (#3915). Per-file process isolation is now INHERENT
// to @stryker-mutator/tap-runner — it drives Node's own `--test-reporter=tap` once per
// covering test FILE, so every file already runs in its own process by construction.
// The prior audit's 'none' vs 'process' comparison (recorded here before this change)
// is moot: there is nothing left to opt in or out of.
},
// adr-parser / config-schema / active-workstream-store / core-utils: derivation reproduces
// their prior hand lists exactly (every constraining file's own name already matched the
@@ -534,13 +537,9 @@ const COVERED = {
// per mutant). tests/model-catalog.unit.test.cjs is spawn-free, in-process,
// and runs in well under a second.
//
// Measured CI score (GitHub Actions run 32605073352, job 97108869486):
// Prior context (#3007, GitHub Actions run 32605073352, job 97108869486):
// model-catalog 59.62% → floor 58 (248 killed, 168 survived, 0 timeouts,
// 0 errors; below TARGET_MUTATION_SCORE (80) — ratchet candidate like
// planning-inspect (56): comfortably clears its own floor but has real
// room to grow. Raise as its tests improve, never lower it.)
// Floor follows this file's documented rule, minScore = floor(measured) - 1,
// matching the sibling precedent exactly (57.03 → 56, 76.58 → 75, 95.65 → 94).
// 0 errors). SUPERSEDED by the 2026-08-27 measurement below.
//
// The shard completed in 57 seconds — concrete evidence the spawn-free
// unit-file design above worked: the #2790 precedent's 15-minute shard-cap
@@ -551,9 +550,31 @@ const COVERED = {
// directly require model-catalog.cjs and match the "model-catalog*" naming rule. Measured
// cost: 50ms (1-file) -> 196ms (3-file), in-process, 0 subprocess spawns; still far under
// the 57s the shard already measured for the single-file set.
//
// CI run 33029755081 (2026-08-27, #3915): measured 75.26% (295 killed / 49 survived /
// 48 no-coverage / 24 runtime-error, totalValid 392). Floor = floor(75.26) - 1 = 74,
// following this file's documented convention.
//
// Why it moved so far: under #3915's tap-runner swap this shard first came back at
// 57.91% against the old floor of 58. Diagnosis from the mutation report's own JSON:
// all 24 of its RuntimeError mutants are in the module's load-time catalog bootstrap,
// so mutating them makes model-catalog.cjs throw at `require`. Under `node --test`
// that is a failed test file and the mutant counts as Killed; under the tap runner
// the process dies before emitting TAP, which Stryker classifies RuntimeError and
// EXCLUDES from the denominator. Adding those 24 back as killed reproduces
// 248/416 = 59.62 exactly — the pre-swap #3007 number — so detection never
// regressed, only its classification changed.
//
// The floor was NOT lowered to absorb that. 11 new behavioural tests in
// tests/model-catalog.unit.test.cjs killed 68 previously-surviving mutants (227 ->
// 295 killed), taking the module from 57.91 to 75.26 — now within striking distance
// of TARGET_MUTATION_SCORE (80) instead of the 59.62 it sat at before this change.
//
// The floor MUST come from a CI shard, never a local run — same rule as every other
// entry in this file.
'model-catalog': {
cjs: 'gsd-core/bin/lib/model-catalog.cjs',
minScore: 58,
minScore: 74,
},
// state-contract: net-new module from #3227. Without this entry the
// Stryker gate reports has_work: "false" and SKIPS it entirely — the
@@ -696,14 +717,10 @@ function buildResult(moduleNames) {
mutate: COVERED[name].cjs,
tests: COVERED[name].tests.join(' '),
minScore: COVERED[name].minScore,
// node:test's default per-file process isolation; only modules that document a
// measured, audited need for 'none' opt out.
isolation: COVERED[name].isolation || 'process',
// Per-shard CI job timeout in minutes. Defaults to 15 (the shared per-shard budget);
// only a module that documents a measured need for more (see the frontmatter entry
// above) sets a higher value. Threaded through mutation.yml's job-level
// `timeout-minutes: ${{ matrix.timeoutMinutes }}` the same way `isolation` is threaded
// through the test-runner env.
// `timeout-minutes: ${{ matrix.timeoutMinutes }}`.
timeoutMinutes: COVERED[name].timeoutMinutes || 15,
}));
@@ -724,7 +741,6 @@ function printHuman(result, changedFiles) {
console.log(` mutate: ${shard.mutate}`);
console.log(` tests: ${shard.tests}`);
console.log(` minScore: ${shard.minScore}`);
console.log(` isolation:${shard.isolation}`);
console.log(` timeoutMinutes:${shard.timeoutMinutes}`);
}
}
@@ -786,12 +802,123 @@ function resolveMutationBreak(raw) {
return n;
}
/**
* Sorted, de-duplicated union of every COVERED module's `tests` array. This
* is the tap-runner's local/full-run default (see resolveMutationTestFiles
* below) — the same union stryker.config.mjs's since-removed DEFAULT_TEST_CMD
* string used to build for the command runner.
*
* @returns {string[]}
*/
function allCoveredTests() {
return [...new Set(Object.values(COVERED).flatMap((mod) => mod.tests))].sort();
}
// ── MUTATION_TEST_FILES resolver ──────────────────────────────────────────────
/**
* Resolves the per-shard tap-runner test-file list from the MUTATION_TEST_FILES
* env var. Fail-closed twin of resolveMutationBreak above, for #3915's swap from
* Stryker's `command` runner to `@stryker-mutator/tap-runner`: the tap runner
* takes an explicit `tap.testFiles` array rather than a shell command, so there
* is no single string to inject a per-shard test list into — this function is
* that injection point instead.
*
* Fail-closed contract:
* - undefined → allCoveredTests() (local run: no env set, documented backstop)
* - non-string → throws (the realistic caller mistake: COVERED[*].tests is an
* array, but the env var this reads is the SPACE-JOINED STRING form of it —
* passing the array itself, or any other non-string, is a wiring bug)
* - set but empty/whitespace-only → throws (CI shard wiring is broken:
* matrix.tests missing)
* - otherwise → trim, split on whitespace, de-duplicate, sort, and, per
* entry: (1) resolve it against the repo root and reject any entry whose
* resolved path escapes the repo root (e.g. via `../` segments); (2)
* reject any entry that does not exist on disk, or that exists but is
* not a regular file (e.g. names a directory) — each failure throws
* naming the offending entry(ies)
*
* This function is the single call site for reading MUTATION_TEST_FILES.
* stryker.config.mjs imports and calls it, so a bad value must fail
* immediately rather than silently degrade: the tap runner's own
* `findTestyLookingFiles` resolves `tap.testFiles` via `glob()`, and a
* non-matching glob pattern yields an EMPTY list SILENTLY — which would
* produce a fast, confident, meaningless mutation run (every mutant reported
* killed or survived against zero tests) instead of a loud error.
*
* The `undefined` branch also runs the same on-disk existence check as every
* other branch, so a stale `extraTests`/`excludeTests` entry in COVERED fails
* loudly here rather than silently producing a shard pointed at a phantom file.
*
* @param {string|undefined} raw - value of process.env.MUTATION_TEST_FILES
* @returns {string[]} sorted, de-duplicated, existence-checked test file paths
*/
function resolveMutationTestFiles(raw) {
let entries;
if (raw === undefined) {
// Local run with no MUTATION_TEST_FILES set — use the derived full-run default.
entries = allCoveredTests();
} else if (typeof raw !== 'string') {
throw new Error(
`MUTATION_TEST_FILES must be a string (space-joined test file paths), got ${typeof raw} — ` +
"COVERED[*].tests is an array internally, but the env var this reads is always the " +
'SPACE-JOINED STRING form of it; passing the array (or any other non-string) directly is a wiring bug'
);
} else if (raw.trim() === '') {
throw new Error(
'MUTATION_TEST_FILES is set but empty — CI shard wiring is broken (matrix.tests missing?)'
);
} else {
entries = [...new Set(raw.trim().split(/\s+/))].sort();
}
const repoRoot = path.join(__dirname, '..');
const escaped = [];
const missing = [];
const notFile = [];
for (const entry of entries) {
const resolved = path.resolve(repoRoot, entry);
const rel = path.relative(repoRoot, resolved);
if (rel === '' || rel.startsWith('..') || path.isAbsolute(rel)) {
escaped.push(entry);
continue;
}
if (!fs.existsSync(resolved)) {
missing.push(entry);
continue;
}
if (!fs.statSync(resolved).isFile()) {
notFile.push(entry);
}
}
if (escaped.length > 0) {
throw new Error(
`MUTATION_TEST_FILES names ${escaped.length} entry(ies) that escape the repo root: ${escaped.join(', ')}`
);
}
if (missing.length > 0) {
throw new Error(
`MUTATION_TEST_FILES names ${missing.length} file(s) that do not exist on disk: ${missing.join(', ')}`
);
}
if (notFile.length > 0) {
throw new Error(
`MUTATION_TEST_FILES names ${notFile.length} entry(ies) that are not a regular file: ${notFile.join(', ')}`
);
}
return entries;
}
// Export internals for programmatic use (tests/mutation-matrix-ratchet.test.cjs).
// The require.main guard prevents main() from running when this file is require()d.
module.exports = {
COVERED,
TARGET_MUTATION_SCORE,
resolveMutationBreak,
allCoveredTests,
resolveMutationTestFiles,
readStdinSync,
// Derivation-engine internals — exported for tests/mutation-test-derivation-drift.test.cjs
// and scripts/lint-mutation-test-derivation-drift.cjs.

74
stryker.config.mjs
View File

@@ -3,13 +3,15 @@
*
* Mutation testing configuration for gsd-core.
*
* Test runner: 'command' (built into @stryker-mutator/core)
* Runs: node --test over the lib test files via the repo's run-tests invocation.
* Test runner: 'tap' (@stryker-mutator/tap-runner)
* Runs: node --test-reporter=tap over each per-shard test file (tap.testFiles,
* resolved by scripts/mutation-matrix.cjs's resolveMutationTestFiles), one
* process per covering test file per mutant.
*
* Mutate scope: bin/lib/**\/*.cjs, excluding generated files and test files.
*
* coverageAnalysis: 'off' — command runner does not support per-mutant coverage
* thresholds: high=80, low=60, break=50
* coverageAnalysis: 'perTest' — the tap runner supports per-mutant coverage
* thresholds: high=80, low=60, break=per-shard MUTATION_BREAK (local fallback 60)
* incremental: true — caches results; PR-scoped runs pass --mutate <changed-files>
*
* Reports:
@@ -25,12 +27,11 @@ import { createRequire } from 'node:module';
const _require = createRequire(import.meta.url);
// resolveMutationBreak: fail-closed resolver for MUTATION_BREAK env var.
// undefined → 60 (local backstop); set-but-empty or non-numeric → throws.
// COVERED: the same derived-tests registry CI's per-shard MUTATION_TEST_CMD is built from
// (mutation.yml's `matrix.tests`) — DEFAULT_TEST_CMD below derives from it too, rather than
// hand-duplicating the union in a second literal (#3881 follow-up, mutation-matrix piece 2:
// this exact split — six modules' tests missing from the old hand-written DEFAULT_TEST_CMD,
// plus two entries belonging to no module — is what this derivation removes).
const { resolveMutationBreak, COVERED } = _require('./scripts/mutation-matrix.cjs');
// resolveMutationTestFiles: fail-closed resolver for MUTATION_TEST_FILES env var (#3915).
// undefined → the derived union of every COVERED module's tests (local backstop);
// set-but-empty, non-string, or naming a nonexistent file → throws. See that
// function's doc-comment in scripts/mutation-matrix.cjs for the full contract.
const { resolveMutationBreak, resolveMutationTestFiles } = _require('./scripts/mutation-matrix.cjs');
// ADR-457: bin/lib/*.cjs are gitignored build artifacts (compiled from
// src/*.cts by `npm run build:lib`, which the mutation CI job runs via `npm ci`
@@ -64,29 +65,28 @@ const UNMUTATED = [
'!gsd-core/bin/lib/gsd2-import.cjs',
];
// Full test command used by local runs and as the fallback when CI does not inject a
// per-shard command via MUTATION_TEST_CMD. DERIVED — never hand-edit this list; it is the
// sorted union of every COVERED module's `tests` array (itself derived by
// scripts/mutation-matrix.cjs's computeModuleTests from direct `require()`s of each
// module's built artifact — see that file's derivation-engine header). A previous
// hand-maintained literal here drifted independently of COVERED's own hand-written `tests`
// arrays: six modules' tests were missing from it, plus two entries (broken-windows.test.cjs,
// complexity-trigger.test.cjs) belonging to no COVERED module at all. Deriving both from the
// same COVERED object makes that class of drift structurally impossible.
const DEFAULT_TEST_CMD = `node --test ${
[...new Set(Object.values(COVERED).flatMap((mod) => mod.tests))].sort().join(' ')
}`;
/** @type {import('@stryker-mutator/core').PartialStrykerOptions} */
export default {
// ── Test runner ──────────────────────────────────────────────────────────────
testRunner: 'command',
commandRunner: {
// Run property + unit tests over lib only (avoids the slow integration
// suite). NO build step here: Stryker mutates the already-built .cjs and the
// tests load it directly — adding a build would rebuild over the mutation.
// In CI each matrix shard injects MUTATION_TEST_CMD with only its own tests.
command: process.env.MUTATION_TEST_CMD || DEFAULT_TEST_CMD,
testRunner: 'tap',
tap: {
testFiles: resolveMutationTestFiles(process.env.MUTATION_TEST_FILES),
// forceBail is OFF (MEASURED, #3915): a structural AST audit of all 26 shard test
// files found 3 that spawn subprocesses — tests/config-schema.property.test.cjs
// (6 runGsdTools calls), tests/core-utils.test.cjs (1), and
// tests/feat-3881-yaml-parser-consequences.test.cjs (2 runGsdTools + 2 runNode).
// @stryker-mutator/tap-runner's docs warn that with forceBail on, a runner that
// spawns child processes can be terminated prematurely — bail fires on every KILLED
// mutant (i.e. most of them), so leaving it on would kill hundreds of processes
// mid-spawnSync per shard and orphan their children. The cost of leaving it off is
// only INTRA-file early exit: Stryker's separate `disableBail` (unset, default false)
// still skips the remaining FILES in tap.testFiles after a failure, so this is no
// worse than the command runner's behaviour was.
forceBail: false,
// No nodeArgs, and no top-level buildCommand (ADR-457): the plugin's default argv,
// ["--test-reporter=tap", "-r", "{{hookFile}}", "{{testFile}}"], contains no build
// step, and ADR-457 requires Stryker to test the ALREADY-BUILT gsd-core/bin/lib/*.cjs
// artifacts with no rebuild between mutation and test.
},
// ── Files to mutate ──────────────────────────────────────────────────────────
@@ -99,8 +99,18 @@ export default {
],
// ── Coverage ─────────────────────────────────────────────────────────────────
// 'off' is required for the command test runner — it cannot instrument per-mutant.
coverageAnalysis: 'off',
// 'perTest' (#3915): 'off' was required only because the command runner could not
// instrument per-mutant coverage. @stryker-mutator/tap-runner is an official plugin
// that does support it, so Stryker now re-runs only the test files that cover each
// mutated line rather than the full per-shard test list for every mutant.
//
// Arithmetic note: 'perTest' makes Stryker able to report a mutant as NoCoverage, and
// NoCoverage counts in the SAME denominator as Survived for `mutationScore` — so this
// reclassification is score-neutral for both `thresholds.break` and
// scripts/check-mutation-score-ratchet.cjs. Never gate on
// `mutationScoreBasedOnCoveredCode`, which EXCLUDES NoCoverage and inflates sharply
// under this setting.
coverageAnalysis: 'perTest',
// ── Thresholds ───────────────────────────────────────────────────────────────
// ADR-456 / issue #1187: CI passes the per-module minScore (from

130
tests/model-catalog.unit.test.cjs
View File

@@ -41,11 +41,17 @@ const {
CODEX_MODEL_EFFORT,
MODEL_ALIAS_MAP,
PROVIDER_PRESETS,
AGENT_TO_PHASE_TYPE,
AGENT_DEFAULT_TIERS,
RUNTIME_PROFILE_MAP,
EFFORT_RENDERING,
EFFORT_ARGV,
isAnthropicFlavoredModel,
getAgentToModelMapForProfile,
formatAgentToModelMapAsTable,
renderEffortArgv,
renderEffortForRuntime,
clampEffortForHost,
nextTier,
mergeEffortTierDefaults,
} = catalog;
@@ -378,4 +384,128 @@ describe('model-catalog: mergeEffortTierDefaults (#3531)', () => {
mergeEffortTierDefaults(manifest, { sonnet: 'opus' }, isValid);
assert.deepEqual(manifest, original);
});
// #3915 — a plain object literal's `__proto__: value` key is a prototype
// setter, not an own-enumerable property, so `Object.entries()` never
// yields it and the house-pollution guard's `tier === '__proto__'` branch
// is unreachable via a normal test fixture. `Object.fromEntries` uses
// CreateDataPropertyOrThrow internally and DOES create a genuine own
// property literally named "__proto__", which `Object.entries()` then
// yields — the only way to actually drive `tier` to that value and
// exercise the guard's first disjunct. A permissive `isValid` (accepts
// any value) is required here specifically so a disabled guard would let
// the object-valued override reach `merged['__proto__'] = value`, which
// (unlike a string value) really does repoint the prototype.
test('__proto__ pollution guard fires even for a genuine own-enumerable "__proto__" key', () => {
const permissive = () => true;
const overrideProto = Object.fromEntries([['__proto__', { polluted: true }]]);
const merged = mergeEffortTierDefaults({}, overrideProto, permissive);
assert.strictEqual(Object.getPrototypeOf(merged), Object.prototype);
assert.strictEqual(merged.polluted, undefined);
});
});
// #3915 — mutation-score restoration (Stryker survivors in model-catalog.cjs).
// Each test below targets specific surviving mutants identified from the
// mutation report; see the PR/issue for the full mutant-to-test mapping.
describe('model-catalog: module load shape (#3915)', () => {
test('the compiled module is flagged __esModule (defineProperty descriptor, not a loose object literal)', () => {
assert.strictEqual(catalog.__esModule, true);
});
});
describe('model-catalog: AGENT_TO_PHASE_TYPE / AGENT_DEFAULT_TIERS (#3915)', () => {
test('AGENT_TO_PHASE_TYPE maps a known agent to its exact catalog phaseType', () => {
assert.equal(AGENT_TO_PHASE_TYPE['gsd-planner'], 'planning');
assert.equal(AGENT_TO_PHASE_TYPE['gsd-verifier'], 'verification');
});
test('AGENT_DEFAULT_TIERS maps a known agent to its exact catalog routingTier', () => {
assert.equal(AGENT_DEFAULT_TIERS['gsd-planner'], 'heavy');
assert.equal(AGENT_DEFAULT_TIERS['gsd-codebase-mapper'], 'light');
});
});
describe('model-catalog: RUNTIME_PROFILE_MAP filtering (#3915)', () => {
// The catalog's runtimeTierDefaults has runtimes whose opus/sonnet/haiku
// entries are ALL null (e.g. 'cline') as a deliberate "no defaults yet"
// sentinel, and runtimes fully populated (e.g. 'claude'). This pair is
// the exact boundary the filter's `Object.keys(filtered).length > 0`
// check exists for.
test('a runtime whose tier entries are all null is dropped entirely', () => {
assert.equal('cline' in RUNTIME_PROFILE_MAP, false);
assert.equal('kimi' in RUNTIME_PROFILE_MAP, false);
});
test('a fully-populated runtime keeps every tier entry, unmodified', () => {
assert.deepStrictEqual(RUNTIME_PROFILE_MAP.claude, {
opus: { model: 'claude-opus-4-8' },
sonnet: { model: 'claude-sonnet-5' },
haiku: { model: 'claude-haiku-4-5' },
});
});
});
describe('model-catalog: EFFORT_RENDERING / EFFORT_ARGV supported sets (#3915)', () => {
test('EFFORT_RENDERING.claude.supported is exactly low/medium/high/xhigh/max', () => {
assert.deepStrictEqual(EFFORT_RENDERING.claude.supported, new Set(['low', 'medium', 'high', 'xhigh', 'max']));
});
test('EFFORT_RENDERING.codex.supported is exactly low/medium/high/xhigh/max', () => {
assert.deepStrictEqual(EFFORT_RENDERING.codex.supported, new Set(['low', 'medium', 'high', 'xhigh', 'max']));
});
test("EFFORT_RENDERING.codex.clamp maps 'minimal' to 'low' and leaves other levels unchanged", () => {
assert.equal(EFFORT_RENDERING.codex.clamp('minimal'), 'low');
assert.equal(EFFORT_RENDERING.codex.clamp('high'), 'high');
});
test('EFFORT_ARGV.claude.supported is exactly low/medium/high/xhigh/max', () => {
assert.deepStrictEqual(EFFORT_ARGV.claude.supported, new Set(['low', 'medium', 'high', 'xhigh', 'max']));
});
test('EFFORT_ARGV.opencode.supported additionally includes minimal', () => {
assert.deepStrictEqual(EFFORT_ARGV.opencode.supported, new Set(['minimal', 'low', 'medium', 'high', 'xhigh', 'max']));
});
test('EFFORT_ARGV.codex.supported is exactly low/medium/high/xhigh/max', () => {
assert.deepStrictEqual(EFFORT_ARGV.codex.supported, new Set(['low', 'medium', 'high', 'xhigh', 'max']));
});
});
describe('model-catalog: formatAgentToModelMapAsTable — exact column widths (#3915)', () => {
// Existing coverage only used `.includes()` on longer-than-header
// agent/model names, which never exercises `Math.max('Agent'.length, ...)`
// vs `Math.min` (padEnd never truncates, so a too-small width is
// invisible to a substring check). Short entries plus a full-string
// comparison make the width computation and the +2/-2 separator padding
// observable.
test('short agent/model names still pad to the header width, and the separator is exactly width+2 wide', () => {
const out = formatAgentToModelMapAsTable({ ab: 'xy' });
const expected = ` Agent │ Model\n${'─'.repeat(7)}┼${'─'.repeat(7)}\n ab │ xy \n`;
assert.equal(out, expected);
});
});
describe('model-catalog: clampEffortForHost (#3915)', () => {
test('non-string host, unknown host, non-string effort, and empty effort all return null; valid input clamps', () => {
assert.equal(clampEffortForHost(123, 'high'), null);
assert.equal(clampEffortForHost('totally-bogus-host', 'high'), null);
assert.equal(clampEffortForHost('claude', 123), null);
assert.equal(clampEffortForHost('claude', ''), null);
assert.equal(clampEffortForHost('claude', 'minimal'), 'low');
assert.equal(clampEffortForHost('claude', 'high'), 'high');
});
// The own-property guard must reject a host BEFORE any property lookup
// that could be coerced into a real key. `typeof host !== 'string'` short-
// circuits the `||` for a non-string host, so `EFFORT_ARGV[host]` is never
// reached even when the host's `toString()` would resolve to a real key —
// proving the type check (not just the hasOwnProperty check) is load-
// bearing, and that neither the `||` nor either disjunct can be disabled.
test('a non-string host is rejected even when it stringifies to a known key', () => {
const spoofedHost = { toString: () => 'claude' };
assert.equal(clampEffortForHost(spoofedHost, 'high'), null);
assert.equal(clampEffortForHost('claude', 'high'), 'high');
});
});

217
tests/mutation-matrix-ratchet.test.cjs
View File

@@ -23,13 +23,24 @@
const { test, describe } = require('node:test');
const assert = require('node:assert/strict');
const path = require('node:path');
const fs = require('node:fs');
const fc = require('fast-check');
const { runNode } = require('./helpers/process-seam.cjs');
const { throwIfFailed } = require('./helpers/git-fixture.cjs');
const { PROBE_TIMEOUT_MS } = require('./helpers/timeouts.cjs');
const REPO_ROOT = path.resolve(__dirname, '..');
const MATRIX_SCRIPT = path.resolve(__dirname, '../scripts/mutation-matrix.cjs');
const matrix = require(MATRIX_SCRIPT);
// Deliberately RE-DERIVED here from matrix.COVERED, rather than calling the
// exported matrix.allCoveredTests() — resolveMutationTestFiles(undefined)'s
// production implementation itself calls allCoveredTests(), so comparing its
// output against that same function would make the assertion a tautology
// that passes no matter what either function does. This independent
// derivation is what makes the comparison test the production code.
const derived = [...new Set(Object.values(matrix.COVERED).flatMap((mod) => mod.tests))].sort();
// ── (a) TARGET_MUTATION_SCORE ─────────────────────────────────────────────────
describe('mutation-matrix ratchet: TARGET_MUTATION_SCORE export', () => {
test('exports TARGET_MUTATION_SCORE', () => {
@@ -202,7 +213,10 @@ const RATCHET_BASELINE = {
'planning-inspect': 56, // CI run 32392791843: 57.03% (unit shard); ratchet candidate vs TARGET 80
'plan-document': 75, // CI run 32392791843: 76.58% (unit shard)
'planning-command-router': 94, // CI run 32392791843: 95.65% (unit shard); already exceeds TARGET 80
'model-catalog': 58, // #3007: measured 59.62% in CI (248 killed / 168 survived); floor(59.62)-1
'model-catalog': 74, // CI run 33029755081 (2026-08-27, #3915): measured 75.26%;
// floor(75.26)-1. Supersedes the #3007 59.62% measurement
// (248 killed / 168 survived) — see scripts/mutation-matrix.cjs
// for the RuntimeError classification-change diagnosis.
'state-contract': 65, // #3227: CI run 32769289750, job 97565813640,
// `Stryker (state-contract)`: measured 66.25%; floor(66.25)-1.
// Below TARGET_MUTATION_SCORE (80) — ratchet candidate like
@@ -305,3 +319,204 @@ describe('resolveMutationBreak: fail-closed env-var resolver', () => {
assert.strictEqual(resolveMutationBreak('62'), 62);
});
});
// ── (g) resolveMutationTestFiles behaviour (#3915: tap-runner test-file resolver) ────
// stryker's tap-runner has no single `command` to inject a per-shard test list into —
// it takes an explicit `tap.testFiles` array — so the derived-union default that used to
// live only in stryker.config.mjs's DEFAULT_TEST_CMD string needs a resolver twin to
// resolveMutationBreak: same fail-closed shape, same single call site, this time
// producing a file-list array instead of a numeric threshold.
describe('resolveMutationTestFiles: fail-closed test-file-list resolver', () => {
const { resolveMutationTestFiles } = matrix;
test('exports resolveMutationTestFiles as a function', () => {
assert.strictEqual(typeof resolveMutationTestFiles, 'function');
});
test('undefined → sorted, de-duplicated union of every COVERED[*].tests', () => {
assert.deepStrictEqual(resolveMutationTestFiles(undefined), derived);
});
test('every entry of the undefined-default result exists on disk', () => {
for (const entry of resolveMutationTestFiles(undefined)) {
assert.ok(
fs.existsSync(path.resolve(REPO_ROOT, entry)),
`derived default test file does not exist on disk: ${entry}`
);
}
});
test('single test file (boundary 1) → one-element array', () => {
assert.deepStrictEqual(
resolveMutationTestFiles('tests/frontmatter.unit.test.cjs'),
['tests/frontmatter.unit.test.cjs']
);
});
test('two space-separated test files → two-element array', () => {
assert.deepStrictEqual(
resolveMutationTestFiles('tests/frontmatter.unit.test.cjs tests/unusable-input.test.cjs'),
['tests/frontmatter.unit.test.cjs', 'tests/unusable-input.test.cjs']
);
});
test("'' (empty string) → throws (CI shard wiring error)", () => {
assert.throws(() => resolveMutationTestFiles(''), /set but empty/);
});
test("' ' (whitespace-only) → throws (CI shard wiring error)", () => {
assert.throws(() => resolveMutationTestFiles(' '), /set but empty/);
});
test("'\\t\\n ' (mixed whitespace-only) → throws (CI shard wiring error)", () => {
assert.throws(() => resolveMutationTestFiles('\t\n '), /set but empty/);
});
test('null → throws', () => {
assert.throws(() => resolveMutationTestFiles(null));
});
test('123 (non-string) → throws', () => {
assert.throws(() => resolveMutationTestFiles(123));
});
test('an array (not a string) → throws', () => {
assert.throws(() => resolveMutationTestFiles(['tests/frontmatter.unit.test.cjs']));
});
test('leading/trailing whitespace is trimmed to exactly one entry with no surrounding whitespace', () => {
const result = resolveMutationTestFiles(' tests/frontmatter.unit.test.cjs ');
assert.strictEqual(result.length, 1);
assert.strictEqual(result[0], result[0].trim());
assert.strictEqual(result[0], 'tests/frontmatter.unit.test.cjs');
});
test('mixed tab/newline/space separators → exactly 3 non-empty entries', () => {
const result = resolveMutationTestFiles(
'tests/frontmatter.unit.test.cjs\t\ttests/unusable-input.test.cjs\n tests/frontmatter.property.test.cjs'
);
assert.strictEqual(result.length, 3);
for (const entry of result) assert.notStrictEqual(entry, '');
});
test('CRLF-separated entries → exactly 2 entries, none carrying a \\r', () => {
const result = resolveMutationTestFiles(
'tests/frontmatter.unit.test.cjs\r\ntests/unusable-input.test.cjs'
);
assert.strictEqual(result.length, 2);
for (const entry of result) assert.ok(!entry.includes('\r'), `entry retained a \\r: ${JSON.stringify(entry)}`);
});
test('a repeated entry is de-duplicated to exactly 1', () => {
const result = resolveMutationTestFiles(
'tests/frontmatter.unit.test.cjs tests/frontmatter.unit.test.cjs'
);
assert.deepStrictEqual(result, ['tests/frontmatter.unit.test.cjs']);
});
test('a nonexistent test file throws, naming the bad path', () => {
assert.throws(
() => resolveMutationTestFiles('tests/does-not-exist-3915.test.cjs'),
/does-not-exist-3915/
);
});
test('one bad entry among otherwise-valid entries still throws', () => {
assert.throws(
() => resolveMutationTestFiles('tests/frontmatter.unit.test.cjs tests/does-not-exist-3915.test.cjs'),
/does-not-exist-3915/
);
});
test('an entry naming an existing directory throws, indicating it is not a regular file', () => {
assert.throws(
() => resolveMutationTestFiles('tests'),
/not a regular file/
);
});
test('an entry escaping the repo root via ../../../etc/passwd throws, indicating the escape', () => {
assert.throws(
() => resolveMutationTestFiles('../../../etc/passwd'),
/escape the repo root/
);
});
test('an entry escaping via a valid-looking prefix (tests/../../outside-3915.cjs) throws for escaping the repo root', () => {
assert.throws(
() => resolveMutationTestFiles('tests/../../outside-3915.cjs'),
/escape the repo root/
);
});
test('the full derived default, space-joined, round-trips through the resolver', () => {
assert.deepStrictEqual(resolveMutationTestFiles(derived.join(' ')), derived);
});
});
describe('resolveMutationTestFiles: round-trip property', () => {
const { resolveMutationTestFiles } = matrix;
test('any non-empty subset of the derived default, space-joined, resolves back to that de-duplicated subset', () => {
fc.assert(
fc.property(
fc.subarray(derived, { minLength: 1 }),
(subset) => {
const expected = [...new Set(subset)].sort();
const actual = resolveMutationTestFiles(subset.join(' '));
assert.deepStrictEqual(actual, expected);
}
),
{ seed: 3915, numRuns: 100 }
);
});
});
// ── (h) mutation matrix: isolation field removed (#3915) ─────────────────────
// stryker's tap-runner has no equivalent of `node --test --test-isolation=<mode>`
// (it drives Node's own TAP test-reporter file-by-file), so the per-shard
// `isolation` field this matrix used to emit for mutation.yml's
// MUTATION_TEST_CMD env line has nothing left to wire into and must be removed
// from buildResult()'s output entirely, not merely left unused.
describe('mutation matrix: isolation field removed (#3915)', () => {
const { resolveMutationTestFiles } = matrix;
test('every emitted matrix entry has no isolation field but keeps the rest of the contract', () => {
const covered = matrix.COVERED || {};
const moduleNames = Object.keys(covered);
const stdinLines = moduleNames.map((name) => `src/${name}.cts`).join('\n');
const spawnResult = runNode(
[MATRIX_SCRIPT],
{
input: stdinLines + '\n',
cwd: REPO_ROOT,
timeoutMs: PROBE_TIMEOUT_MS,
}
);
throwIfFailed(spawnResult, `node ${MATRIX_SCRIPT}`);
const result = JSON.parse(spawnResult.stdout);
assert.ok(result.matrix.include.length > 0, 'matrix.include must not be empty');
for (const entry of result.matrix.include) {
assert.ok(
!Object.prototype.hasOwnProperty.call(entry, 'isolation'),
`matrix entry for '${entry.name}' must NOT include an isolation field (#3915: tap-runner has no test-isolation flag)`
);
for (const key of ['name', 'mutate', 'tests', 'minScore', 'timeoutMinutes']) {
assert.ok(
Object.prototype.hasOwnProperty.call(entry, key),
`matrix entry for '${entry.name}' must still include '${key}'`
);
}
// Round-trip between the two halves of the wiring: the tests string this
// matrix emits must resolve back to exactly the COVERED entry's own tests.
assert.deepStrictEqual(
resolveMutationTestFiles(entry.tests),
matrix.COVERED[entry.name].tests
);
}
});
});

56
tests/mutation-score-ratchet.test.cjs
View File

@@ -81,6 +81,62 @@ describe('extractAchievedScore: Stryker json-reporter document', () => {
test('throws when the report has no scoreable mutants (empty files map — a wiring bug, never a 0% score)', () => {
assert.throws(() => extractAchievedScore({ schemaVersion: '1.0', thresholds: {}, files: {} }), /no scoreable mutants/);
});
// #3915: coverageAnalysis flips 'off' -> 'perTest' for the tap-runner swap, which is
// what makes NoCoverage a status Stryker can now actually emit for these shards (the
// command runner's 'off' analysis never produced it). extractAchievedScore must already
// treat NoCoverage exactly like Survived in the denominator, or every shard's floor
// silently becomes easier to clear the moment mutants start reporting NoCoverage instead
// of Survived.
function buildCountReport({ Killed = 0, Survived = 0, NoCoverage = 0 } = {}) {
const mutants = [];
let id = 0;
const push = (status, n) => {
for (let i = 0; i < n; i += 1) {
mutants.push({
id: String(id++),
mutatorName: 'a',
status,
location: { start: { line: 1, column: 1 }, end: { line: 1, column: 2 } },
});
}
};
push('Killed', Killed);
push('Survived', Survived);
push('NoCoverage', NoCoverage);
return {
schemaVersion: '1.0',
thresholds: { high: 80, low: 60 },
files: { 'foo.js': { language: 'javascript', source: 'x', mutants } },
};
}
test('8 Killed + 2 Survived → 80', () => {
assert.strictEqual(extractAchievedScore(buildCountReport({ Killed: 8, Survived: 2 })), 80);
});
// NoCoverage counts in the mutation-score denominator exactly as Survived does, which is
// what makes the #3915 coverageAnalysis 'off'->'perTest' switch score-neutral; Stryker's
// own break threshold reads this same field (mutation-test-report-helper.js).
test('8 Killed + 2 NoCoverage → 80, NOT 100', () => {
assert.strictEqual(extractAchievedScore(buildCountReport({ Killed: 8, NoCoverage: 2 })), 80);
});
// Non-vacuity guard (Goodhart guard): proves the previous test is not a tautology by
// showing mutationScore and mutationScoreBasedOnCoveredCode genuinely diverge on the same
// report. This fails the moment someone "improves" extractAchievedScore to read the
// better-sounding covered-code field, which would make every floor trivially satisfiable.
test('non-vacuity: mutationScoreBasedOnCoveredCode (100) diverges from extractAchievedScore (80) on the same report', () => {
const { calculateMutationTestMetrics } = require('mutation-testing-metrics');
const report = buildCountReport({ Killed: 8, NoCoverage: 2 });
const metrics = calculateMutationTestMetrics(report);
assert.strictEqual(metrics.systemUnderTestMetrics.metrics.mutationScoreBasedOnCoveredCode, 100);
assert.strictEqual(extractAchievedScore(report), 80);
});
test('0 Killed + 10 NoCoverage → 0 (a wholly-uncovered shard scores zero, not 100)', () => {
assert.strictEqual(extractAchievedScore(buildCountReport({ NoCoverage: 10 })), 0);
});
});
// ── CLI end-to-end: fail on a planted over-floor score, pass when the floor is raised ───

189
tests/mutation-tap-runner-wiring.test.cjs Normal file
View File

@@ -0,0 +1,189 @@
'use strict';
/**
* tests/mutation-tap-runner-wiring.test.cjs
*
* Pins the #3915 tap-runner wiring — the config contract AND the
* workflow<->config parity, because the injected env token lives on two
* surfaces (`.github/workflows/mutation.yml`'s per-shard `env:` block and
* `stryker.config.mjs`'s reader of it) and silent drift between them would
* make every shard silently fall back to running the FULL default test list
* instead of its own module's tests — slow, wrong, and (because Stryker
* would still find SOME test constraining each mutant) green.
*
* FAILING-FIRST (#3915): stryker.config.mjs still declares `testRunner:
* 'command'` with a `commandRunner.command` built from `MUTATION_TEST_CMD`,
* and mutation.yml still injects `MUTATION_TEST_CMD` / `matrix.isolation`.
* Every test below targets the tap-runner shape those files do not have yet.
*/
const { test, describe } = require('node:test');
const assert = require('node:assert/strict');
const path = require('node:path');
const fs = require('node:fs');
const { pathToFileURL } = require('node:url');
const yaml = require('js-yaml');
const REPO_ROOT = path.resolve(__dirname, '..');
const CONFIG_PATH = path.join(REPO_ROOT, 'stryker.config.mjs');
const WORKFLOW_PATH = path.join(REPO_ROOT, '.github', 'workflows', 'mutation.yml');
// Env keys this config is known to read. Every key is saved/restored so a
// test's env override can never leak into a sibling test.
const RELEVANT_ENV_KEYS = ['MUTATION_TEST_FILES', 'MUTATION_TEST_CMD', 'MUTATION_BREAK'];
// Cache-busting counter: importing the SAME file:// URL twice returns the
// SAME cached ES module record, so an env-dependent config test that reused
// one URL across calls would silently observe the FIRST call's env forever —
// every subsequent env-dependent assertion would pass or fail for the wrong
// reason. Incrementing this per call forces a fresh module evaluation.
let _importCounter = 0;
/**
* Load stryker.config.mjs with `env` applied on top of process.env for the
* duration of the import, then restore process.env exactly. `env` values of
* `undefined` delete the corresponding key rather than setting it.
*
* @param {Record<string, string|undefined>} env
* @returns {Promise<object>} the config module's default export
*/
async function loadConfig(env) {
// Save/restore the UNION of RELEVANT_ENV_KEYS and Object.keys(env), not just the fixed
// list: the PARITY test below discovers its env key NAME from the parsed workflow file
// rather than hardcoding it, so a key outside RELEVANT_ENV_KEYS can reach here — saving
// only the fixed list would let that discovered key leak into sibling tests.
const keysToRestore = new Set([...RELEVANT_ENV_KEYS, ...Object.keys(env)]);
const saved = {};
for (const key of keysToRestore) saved[key] = process.env[key];
for (const [key, value] of Object.entries(env)) {
if (value === undefined) delete process.env[key];
else process.env[key] = value;
}
try {
const mod = await import(`${pathToFileURL(CONFIG_PATH).href}?v=${_importCounter++}`);
return mod.default;
} finally {
for (const key of keysToRestore) {
if (saved[key] === undefined) delete process.env[key];
else process.env[key] = saved[key];
}
}
}
describe('stryker.config.mjs: tap runner contract (#3915)', () => {
test("testRunner === 'tap'", async () => {
const config = await loadConfig({});
assert.strictEqual(config.testRunner, 'tap');
});
test("coverageAnalysis === 'perTest'", async () => {
const config = await loadConfig({});
assert.strictEqual(config.coverageAnalysis, 'perTest');
});
test("coverageAnalysis !== 'off' (the literal condition #3915 requires to stop being true)", async () => {
const config = await loadConfig({});
assert.notStrictEqual(config.coverageAnalysis, 'off');
});
test('no own-property commandRunner (the command runner is gone)', async () => {
const config = await loadConfig({});
assert.ok(!Object.prototype.hasOwnProperty.call(config, 'commandRunner'));
});
test('tap.forceBail === false', async () => {
const config = await loadConfig({});
assert.strictEqual(config.tap.forceBail, false);
});
test('no own-property buildCommand, and tap has no own-property nodeArgs (no rebuild step reintroduced, ADR-457)', async () => {
const config = await loadConfig({});
assert.ok(!Object.prototype.hasOwnProperty.call(config, 'buildCommand'));
assert.ok(!Object.prototype.hasOwnProperty.call(config.tap, 'nodeArgs'));
});
test('MUTATION_TEST_FILES set → tap.testFiles deep-equals the parsed entries', async () => {
const config = await loadConfig({
MUTATION_TEST_FILES: 'tests/frontmatter.unit.test.cjs tests/unusable-input.test.cjs',
});
assert.deepStrictEqual(config.tap.testFiles, [
'tests/frontmatter.unit.test.cjs',
'tests/unusable-input.test.cjs',
]);
});
test('MUTATION_BREAK set → thresholds.break reflects it unchanged (re-proved after the runner swap)', async () => {
const config = await loadConfig({ MUTATION_BREAK: '72' });
assert.strictEqual(config.thresholds.break, 72);
});
test('MUTATION_TEST_FILES set but empty → fail-closed survives all the way to config load', async () => {
await assert.rejects(() => loadConfig({ MUTATION_TEST_FILES: '' }));
});
test('mutate scope unchanged by the runner swap', async () => {
const config = await loadConfig({});
assert.ok(Array.isArray(config.mutate));
assert.ok(config.mutate.includes('gsd-core/bin/lib/**/*.cjs'));
assert.ok(
config.mutate.some((entry) => entry.startsWith('!gsd-core/bin/lib/')),
'mutate array must still carry at least one !gsd-core/bin/lib/... exclusion'
);
});
});
describe('mutation.yml <-> stryker.config.mjs: injected env parity (#3915)', () => {
const workflowDoc = yaml.load(fs.readFileSync(WORKFLOW_PATH, 'utf8'));
const mutateJob = workflowDoc.jobs.mutate;
const runStrykerStep = mutateJob.steps.find(
(step) => typeof step.name === 'string' && step.name.startsWith('Run Stryker')
);
test("the 'Run Stryker' step exists", () => {
assert.ok(runStrykerStep, "no step in the mutate job's steps has a name starting with 'Run Stryker'");
});
test('step.env has key MUTATION_TEST_FILES', () => {
assert.ok(Object.prototype.hasOwnProperty.call(runStrykerStep.env, 'MUTATION_TEST_FILES'));
});
test('step.env does NOT have key MUTATION_TEST_CMD', () => {
assert.ok(!Object.prototype.hasOwnProperty.call(runStrykerStep.env, 'MUTATION_TEST_CMD'));
});
test('MUTATION_TEST_FILES value is exactly the ${{ matrix.tests }} expression (derived from mutation-matrix.cjs)', () => {
assert.strictEqual(String(runStrykerStep.env.MUTATION_TEST_FILES).trim(), '${{ matrix.tests }}');
});
test('MUTATION_BREAK still references matrix.minScore', () => {
assert.ok(Object.prototype.hasOwnProperty.call(runStrykerStep.env, 'MUTATION_BREAK'));
assert.ok(String(runStrykerStep.env.MUTATION_BREAK).includes('matrix.minScore'));
});
test('no value anywhere in the step env mentions --test-isolation', () => {
for (const value of Object.values(runStrykerStep.env)) {
assert.ok(!String(value).includes('--test-isolation'), `unexpected --test-isolation reference: ${value}`);
}
});
test('no value anywhere in the whole mutate job mentions matrix.isolation', () => {
const serialized = JSON.stringify(mutateJob);
assert.ok(!serialized.includes('matrix.isolation'), 'mutate job still references matrix.isolation');
});
test("the step's run string does not contain '${{' (no direct interpolation inside run:, CONTRIBUTING.md)", () => {
assert.ok(!runStrykerStep.run.includes('${{'), 'run: block contains a direct ${{ }} interpolation');
});
test('PARITY: the env key name discovered from the workflow drives the config test directly, so the two surfaces cannot drift', async () => {
// Find the key in step.env that starts with MUTATION_TEST_ — this is read
// from the PARSED workflow document, not hardcoded, so a rename on either
// surface (the workflow's env key, or stryker.config.mjs's reader of it)
// breaks this test instead of the two silently drifting apart.
const discoveredKey = Object.keys(runStrykerStep.env).find((k) => k.startsWith('MUTATION_TEST_'));
assert.ok(discoveredKey, 'no MUTATION_TEST_* key found in the Run Stryker step env');
const config = await loadConfig({ [discoveredKey]: 'tests/frontmatter.unit.test.cjs' });
assert.deepStrictEqual(config.tap.testFiles, ['tests/frontmatter.unit.test.cjs']);
});
});