feat(#1187): per-module Stryker mutation-score ratchet (ADR-456 80% floor) (#1200)

* feat(#1187): per-module mutation-score ratchet + graduate core-utils ADR-456's 80% mutation floor was unenforceable as a single global break=50: 4 of 6 covered modules sit at 63-79% and forcing them to 80 would require brittle exact-string assertions on equivalent string-literal mutants (a Goodhart's-Law trap). Instead, each covered module declares a minScore floor (locked at its measured score, TARGET 80) enforced per CI shard via stryker --break, ratcheting up over time without brittle tests. - mutation-matrix.cjs: minScore per module + TARGET_MUTATION_SCORE=80, emitted in the matrix; require.main guard + exports for testability. - mutation.yml: per-shard --break <minScore>. - stryker.config.mjs: global break 50->60 as a local backstop (CI uses minScore). - Graduated core-utils (measured 77.5%, floor 75). - context-utilization 79.5->92.3% via behavioral killers (state classification outputs + error-value contract, not exact-string matches) -> minScore 80 (TARGET). - ratchet-integrity guard test (28 cases). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#1187): pass mutation break via MUTATION_BREAK env (no stryker --break flag) Adversarial review caught that Stryker 9.x has no --break CLI flag, so the per-shard 'stryker run --break <minScore>' errored out every mutation shard. Read the per-module floor from process.env.MUTATION_BREAK in stryker.config.mjs and set it per shard via env in mutation.yml. Red-green verified: MUTATION_BREAK=99 exits 1, =80 exits 0. Also make the ratchet guard monotonic (RATCHET_BASELINE floors; lowering a floor now fails the guard unless the baseline is edited). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#1187): fail closed on bad MUTATION_BREAK + monotonic ratchet baseline Code review: Number(env)||60 failed OPEN — an empty/invalid MUTATION_BREAK (e.g. a future module missing minScore -> matrix expands to '') silently degraded the shard to break 60, letting a high-floor module regress undetected. resolveMutationBreak() now returns 60 only when the env is truly unset (local backstop) and THROWS on present-but-empty/non-numeric/out-of-range (fail closed); stryker.config.mjs imports it via createRequire. Also make RATCHET_BASELINE an equality mirror (=== not >=) so any floor change is explicit in review and no floor can be silently lowered. Tests: 46 (incl resolveMutationBreak cases). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#1187): recalibrate config-schema/prompt-budget floors to CI scores First CI mutation run failed two shards: the floors were set from local Stryker runs whose TIMEOUTS were counted as kills (env-variable), inflating scores. CI runs with timeout~0, so the real deterministic scores are lower: - config-schema: local 69.7% -> CI 54.55% (5 local timeouts vanished) -> floor 52 - prompt-budget: local 99.6% -> CI 68.33% (239 local timeouts vanished) -> floor 66 Calibrate floors from CI (the documented source of truth) and record the lesson in the comment so future floors aren't set from timeout-inflated local runs. Baseline updated to match. The other 5 shards passed (deterministic CI scores above their floors). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-14 08:25:41 -04:00
parent f660a9e55a
commit 98866a0c69
5 changed files with 511 additions and 13 deletions
--- a/scripts/mutation-matrix.cjs
+++ b/scripts/mutation-matrix.cjs
@@ -34,17 +34,59 @@ const { readFileSync } = require('fs');

 const { ExitError, runMain } = require('./lib/cli-exit.cjs');

+// ── Per-module mutation score ratchet ─────────────────────────────────────────
+// ADR-456 / issue #1187: every covered module declares a minScore floor.
+//
+// HOW THE RATCHET WORKS:
+//   • minScore locks in the current measured mutation score (minus a 1–2 pt
+//     margin for run-to-run timeout variance).
+//   • CI fails a shard if the module's live score drops below its minScore.
+//   • Raise minScore (never lower) as a module's tests improve.
+//   • The goal is every module reaching TARGET_MUTATION_SCORE (80).
+//
+// GOODHART SAFETY: scores are improved by writing genuine behavioural
+// assertions that kill real mutants — never by adding brittle exact-string
+// matches on incidental output. A justified `// Stryker disable` on a
+// confirmed equivalent mutant is acceptable.
+//
+// HOW TO UPDATE:
+//   1. Run the per-module Stryker shard locally.
+//   2. Note the reported score.
+//   3. Set minScore = floor(score) - 1 (never lower than current value).
+//   4. Open a PR — the CI gate will enforce the new floor on every future run.
+
+/** Long-run target for all modules (ADR-456). */
+const TARGET_MUTATION_SCORE = 80;
+
 // ── Single source of truth: covered modules ───────────────────────────────────
-// Each entry: { cjs: '<built artifact>', tests: ['tests/...', ...] }
-// A module is "covered" iff its tests are wired into the Stryker command runner
-// (stryker.config.mjs commandRunner.command). Mutating an uncovered module can
-// only ever produce survived mutants — so we scope strictly to these 6.
+// Each entry: { cjs: '<built artifact>', tests: ['tests/...', ...], minScore: N }
+//
+// minScore is the CI break threshold for this module's shard.
+// Floors are measured scores minus 1–2 pts for run-to-run variance.
+// Measured CI scores 2026-06-14 (issue #1187, timeout-free — source of truth):
+//   context-utilization     92.31% → floor 80  (target already met)
+//   prompt-budget           68.33% → floor 66  (local was 99.6% — TIMEOUT INFLATION; CI is the truth)
+//   frontmatter             63.35% → floor 62
+//   adr-parser              69.30% → floor 68
+//   config-schema           54.55% → floor 52  (local was 69.7% — TIMEOUT INFLATION; CI is the truth)
+//   active-workstream-store 81.91% → floor 80
+//   core-utils              77.52% → floor 75
+//
+// LESSON: floors MUST be calibrated from CI mutation runs (CI runs with
+// timeout≈0, deterministic). Local runs count timeouts as kills and
+// inflate scores significantly (prompt-budget: 99.6% local vs 68.3% CI;
+// config-schema: 69.7% local vs 54.55% CI). Never set a floor from a
+// local run without CI cross-check.
 const COVERED = {
  'context-utilization': {
    cjs: 'gsd-core/bin/lib/context-utilization.cjs',
    tests: [
      'tests/context-utilization.property.test.cjs',
    ],
+    // After mutation-killer assertions added in #1187: measured 92.31% (2026-06-14).
+    // 3 survivors are __esModule boilerplate (genuinely equivalent CJS interop mutants).
+    // minScore raised to TARGET (80) — module now meets ADR-456 goal.
+    minScore: 80,
  },
  'prompt-budget': {
    cjs: 'gsd-core/bin/lib/prompt-budget.cjs',
@@ -52,6 +94,9 @@ const COVERED = {
      'tests/prompt-budget.property.test.cjs',
      'tests/prompt-budget.unit.test.cjs',
    ],
+    // CI 68.33% timeout-free (164 killed / 1 timeout / 240 total) 2026-06-14;
+    // local was 99.6% — timeout inflation. Floor = 68 - 2 margin.
+    minScore: 66,
  },
  frontmatter: {
    cjs: 'gsd-core/bin/lib/frontmatter.cjs',
@@ -59,6 +104,7 @@ const COVERED = {
      'tests/frontmatter.property.test.cjs',
      'tests/frontmatter.unit.test.cjs',
    ],
+    minScore: 62,
  },
  'adr-parser': {
    cjs: 'gsd-core/bin/lib/adr-parser.cjs',
@@ -67,12 +113,16 @@ const COVERED = {
      'tests/adr-parser.test.cjs',
      'tests/adr-parser.unit.test.cjs',
    ],
+    minScore: 68,
  },
  'config-schema': {
    cjs: 'gsd-core/bin/lib/config-schema.cjs',
    tests: [
      'tests/config-schema.property.test.cjs',
    ],
+    // CI 54.55% timeout-free (18 killed / 0 timeout / 33 total) 2026-06-14;
+    // local was 69.7% — timeout inflation. Floor = 54 - 2 margin.
+    minScore: 52,
  },
  'active-workstream-store': {
    cjs: 'gsd-core/bin/lib/active-workstream-store.cjs',
@@ -80,6 +130,14 @@ const COVERED = {
      'tests/active-workstream-store.test.cjs',
      'tests/active-workstream-store.unit.test.cjs',
    ],
+    minScore: 80,
+  },
+  'core-utils': {
+    cjs: 'gsd-core/bin/lib/core-utils.cjs',
+    tests: [
+      'tests/core-utils.test.cjs',
+    ],
+    minScore: 75,  // measured 77.52% (2026-06-14, issue #1187); floor = 77 - 2
  },
 };

@@ -178,6 +236,7 @@ function buildResult(moduleNames) {
    name,
    mutate: COVERED[name].cjs,
    tests: COVERED[name].tests.join(' '),
+    minScore: COVERED[name].minScore,
  }));

  return {
@@ -194,8 +253,9 @@ function printHuman(result, changedFiles) {
  console.log(`Shards (${result.matrix.include.length}):`);
  for (const shard of result.matrix.include) {
    console.log(`  [${shard.name}]`);
-    console.log(`    mutate: ${shard.mutate}`);
-    console.log(`    tests:  ${shard.tests}`);
+    console.log(`    mutate:   ${shard.mutate}`);
+    console.log(`    tests:    ${shard.tests}`);
+    console.log(`    minScore: ${shard.minScore}`);
  }
 }

@@ -219,4 +279,45 @@ function main() {
  }
 }

-runMain(main);
+// ── MUTATION_BREAK resolver ───────────────────────────────────────────────────
+/**
+ * Resolves the per-shard mutation break threshold from the MUTATION_BREAK env var.
+ *
+ * Fail-closed contract:
+ *   - undefined  → 60  (local run: no env set, documented backstop)
+ *   - set but empty (e.g. CI matrix.minScore missing) → throws (wiring error)
+ *   - non-numeric or out-of-range [1, 100] → throws (invalid config)
+ *   - valid integer string → returns that number
+ *
+ * This function is the single call site for reading MUTATION_BREAK.
+ * stryker.config.mjs imports and calls it so CI shards with a bad
+ * MUTATION_BREAK fail immediately rather than silently falling back to 60
+ * and bypassing a per-module floor above 60 (e.g. prompt-budget: 90).
+ *
+ * @param {string|undefined} raw - value of process.env.MUTATION_BREAK
+ * @returns {number}
+ */
+function resolveMutationBreak(raw) {
+  if (raw === undefined) {
+    // Local run with no MUTATION_BREAK set — use documented backstop.
+    return 60;
+  }
+  if (typeof raw !== 'string' || raw.trim() === '') {
+    throw new Error(
+      'MUTATION_BREAK is set but empty — CI shard wiring is broken (matrix.minScore missing?)'
+    );
+  }
+  const n = Number(raw);
+  if (!Number.isFinite(n) || n < 1 || n > 100) {
+    throw new Error(
+      `MUTATION_BREAK invalid: "${raw}" (expected a per-module minScore 1-100)`
+    );
+  }
+  return n;
+}
+
+// Export internals for programmatic use (tests/mutation-matrix-ratchet.test.cjs).
+// The require.main guard prevents main() from running when this file is require()d.
+module.exports = { COVERED, TARGET_MUTATION_SCORE, resolveMutationBreak };
+
+if (require.main === module) runMain(main);