* feat(#1187): per-module mutation-score ratchet + graduate core-utils ADR-456's 80% mutation floor was unenforceable as a single global break=50: 4 of 6 covered modules sit at 63-79% and forcing them to 80 would require brittle exact-string assertions on equivalent string-literal mutants (a Goodhart's-Law trap). Instead, each covered module declares a minScore floor (locked at its measured score, TARGET 80) enforced per CI shard via stryker --break, ratcheting up over time without brittle tests. - mutation-matrix.cjs: minScore per module + TARGET_MUTATION_SCORE=80, emitted in the matrix; require.main guard + exports for testability. - mutation.yml: per-shard --break <minScore>. - stryker.config.mjs: global break 50->60 as a local backstop (CI uses minScore). - Graduated core-utils (measured 77.5%, floor 75). - context-utilization 79.5->92.3% via behavioral killers (state classification outputs + error-value contract, not exact-string matches) -> minScore 80 (TARGET). - ratchet-integrity guard test (28 cases). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#1187): pass mutation break via MUTATION_BREAK env (no stryker --break flag) Adversarial review caught that Stryker 9.x has no --break CLI flag, so the per-shard 'stryker run --break <minScore>' errored out every mutation shard. Read the per-module floor from process.env.MUTATION_BREAK in stryker.config.mjs and set it per shard via env in mutation.yml. Red-green verified: MUTATION_BREAK=99 exits 1, =80 exits 0. Also make the ratchet guard monotonic (RATCHET_BASELINE floors; lowering a floor now fails the guard unless the baseline is edited). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#1187): fail closed on bad MUTATION_BREAK + monotonic ratchet baseline Code review: Number(env)||60 failed OPEN — an empty/invalid MUTATION_BREAK (e.g. a future module missing minScore -> matrix expands to '') silently degraded the shard to break 60, letting a high-floor module regress undetected. resolveMutationBreak() now returns 60 only when the env is truly unset (local backstop) and THROWS on present-but-empty/non-numeric/out-of-range (fail closed); stryker.config.mjs imports it via createRequire. Also make RATCHET_BASELINE an equality mirror (=== not >=) so any floor change is explicit in review and no floor can be silently lowered. Tests: 46 (incl resolveMutationBreak cases). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#1187): recalibrate config-schema/prompt-budget floors to CI scores First CI mutation run failed two shards: the floors were set from local Stryker runs whose TIMEOUTS were counted as kills (env-variable), inflating scores. CI runs with timeout~0, so the real deterministic scores are lower: - config-schema: local 69.7% -> CI 54.55% (5 local timeouts vanished) -> floor 52 - prompt-budget: local 99.6% -> CI 68.33% (239 local timeouts vanished) -> floor 66 Calibrate floors from CI (the documented source of truth) and record the lesson in the comment so future floors aren't set from timeout-inflated local runs. Baseline updated to match. The other 5 shards passed (deterministic CI scores above their floors). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
129 lines
6.9 KiB
JavaScript
129 lines
6.9 KiB
JavaScript
/**
|
|
* stryker.config.mjs
|
|
*
|
|
* Mutation testing configuration for gsd-core.
|
|
*
|
|
* Test runner: 'command' (built into @stryker-mutator/core)
|
|
* Runs: node --test over the lib test files via the repo's run-tests invocation.
|
|
*
|
|
* Mutate scope: bin/lib/**\/*.cjs, excluding generated files and test files.
|
|
*
|
|
* coverageAnalysis: 'off' — command runner does not support per-mutant coverage
|
|
* thresholds: high=80, low=60, break=50
|
|
* incremental: true — caches results; PR-scoped runs pass --mutate <changed-files>
|
|
*
|
|
* Reports:
|
|
* - html: reports/mutation/mutation.html
|
|
* - clear-text (console)
|
|
* - progress (spinner)
|
|
*
|
|
* NOTE: This is incremental / changed-files-only in CI (--mutate <changed-files>)
|
|
* to stay bounded. Full runs are for local exploration only.
|
|
*/
|
|
|
|
import { createRequire } from 'node:module';
|
|
const _require = createRequire(import.meta.url);
|
|
// resolveMutationBreak: fail-closed resolver for MUTATION_BREAK env var.
|
|
// undefined → 60 (local backstop); set-but-empty or non-numeric → throws.
|
|
const { resolveMutationBreak } = _require('./scripts/mutation-matrix.cjs');
|
|
|
|
// ADR-457: bin/lib/*.cjs are gitignored build artifacts (compiled from
|
|
// src/*.cts by `npm run build:lib`, which the mutation CI job runs via `npm ci`
|
|
// → prepare before Stryker). Stryker mutates the *built* .cjs directly — the
|
|
// command runner runs the tests with NO rebuild, so each mutation to the
|
|
// shipped artifact is seen by the tests. (Mutating src/*.cts instead would
|
|
// force a full tsc rebuild per mutant — far too slow for the 30-min CI budget.)
|
|
// Large/low-coverage modules are excluded (the command's test set does not
|
|
// exercise them, so they would only ever produce survived mutants).
|
|
//
|
|
// KNOWN BLIND SPOT (2026-06 CI audit): this list excludes ~14.2k of ~29.8k
|
|
// lib lines (~48%), including the most central modules (state, core,
|
|
// commands, phase, verify). Mutation results therefore speak only for the
|
|
// well-tested half of the lib. Shrinking the list is deliberate tracked work:
|
|
// bring one module into scope per release by first giving it per-module
|
|
// *.unit.test.cjs / *.property.test.cjs coverage, then deleting its entry —
|
|
// never delete an entry without that coverage (it will only produce survived
|
|
// mutants and trip the break threshold).
|
|
const UNMUTATED = [
|
|
'!gsd-core/bin/lib/command-aliases.cjs',
|
|
'!gsd-core/bin/lib/commands.cjs',
|
|
'!gsd-core/bin/lib/core.cjs',
|
|
'!gsd-core/bin/lib/install-profiles.cjs',
|
|
'!gsd-core/bin/lib/installer-migrations.cjs',
|
|
'!gsd-core/bin/lib/phase.cjs',
|
|
'!gsd-core/bin/lib/profile-output.cjs',
|
|
'!gsd-core/bin/lib/state.cjs',
|
|
'!gsd-core/bin/lib/verify.cjs',
|
|
'!gsd-core/bin/lib/init.cjs',
|
|
'!gsd-core/bin/lib/audit.cjs',
|
|
'!gsd-core/bin/lib/gsd2-import.cjs',
|
|
];
|
|
|
|
// Full test command used by local runs and as the fallback when CI does not
|
|
// inject a per-shard command via MUTATION_TEST_CMD.
|
|
// Keep this list in sync with the tests arrays in scripts/mutation-matrix.cjs COVERED.
|
|
const DEFAULT_TEST_CMD = 'node --test tests/context-utilization.property.test.cjs tests/prompt-budget.property.test.cjs tests/frontmatter.property.test.cjs tests/adr-parser.property.test.cjs tests/config-schema.property.test.cjs tests/adr-parser.test.cjs tests/active-workstream-store.test.cjs tests/active-workstream-store.unit.test.cjs tests/prompt-budget.unit.test.cjs tests/adr-parser.unit.test.cjs tests/frontmatter.unit.test.cjs tests/core-utils.test.cjs';
|
|
|
|
/** @type {import('@stryker-mutator/core').PartialStrykerOptions} */
|
|
export default {
|
|
// ── Test runner ──────────────────────────────────────────────────────────────
|
|
testRunner: 'command',
|
|
commandRunner: {
|
|
// Run property + unit tests over lib only (avoids the slow integration
|
|
// suite). NO build step here: Stryker mutates the already-built .cjs and the
|
|
// tests load it directly — adding a build would rebuild over the mutation.
|
|
// In CI each matrix shard injects MUTATION_TEST_CMD with only its own tests.
|
|
command: process.env.MUTATION_TEST_CMD || DEFAULT_TEST_CMD,
|
|
},
|
|
|
|
// ── Files to mutate ──────────────────────────────────────────────────────────
|
|
// The built bin/lib/*.cjs artifacts (ADR-457). CI overrides this with
|
|
// --mutate <changed, covered modules> computed in mutation.yml.
|
|
mutate: [
|
|
'gsd-core/bin/lib/**/*.cjs',
|
|
'!gsd-core/bin/lib/**/*.test.cjs',
|
|
...UNMUTATED,
|
|
],
|
|
|
|
// ── Coverage ─────────────────────────────────────────────────────────────────
|
|
// 'off' is required for the command test runner — it cannot instrument per-mutant.
|
|
coverageAnalysis: 'off',
|
|
|
|
// ── Thresholds ───────────────────────────────────────────────────────────────
|
|
// ADR-456 / issue #1187: CI passes the per-module minScore (from
|
|
// scripts/mutation-matrix.cjs) via the MUTATION_BREAK environment variable.
|
|
// Each CI shard sets MUTATION_BREAK to its module's floor so Stryker enforces
|
|
// the ratchet. Local runs without MUTATION_BREAK fall back to 60 (backstop).
|
|
// Do NOT raise the fallback here; raise individual minScore values in
|
|
// mutation-matrix.cjs instead.
|
|
thresholds: {
|
|
high: 80,
|
|
low: 60,
|
|
break: resolveMutationBreak(process.env.MUTATION_BREAK),
|
|
},
|
|
|
|
// ── Incremental mode ─────────────────────────────────────────────────────────
|
|
// Cache mutation results; re-run only changed mutants on subsequent calls.
|
|
// In CI the workflow computes changed files and passes: stryker run --incremental --mutate <list>
|
|
incremental: true,
|
|
incrementalFile: '.stryker-incremental.json',
|
|
|
|
// ── Reporters ────────────────────────────────────────────────────────────────
|
|
reporters: ['html', 'clear-text', 'progress'],
|
|
htmlReporter: {
|
|
fileName: 'reports/mutation/mutation.html',
|
|
},
|
|
|
|
// ── Temp directory ───────────────────────────────────────────────────────────
|
|
tempDirName: '.stryker-tmp',
|
|
|
|
// ── Ignore patterns ──────────────────────────────────────────────────────────
|
|
ignorePatterns: [
|
|
'node_modules',
|
|
'reports',
|
|
'.stryker-tmp',
|
|
'coverage',
|
|
'hooks/dist',
|
|
],
|
|
};
|