* enhance(#3915): stryker on the official tap runner, per-test coverage The 'command' runner is the one runner Stryker excludes from coverage analysis, which forced coverageAnalysis:'off' and made mutation cost strictly linear in (mutants x whole-shard test time). The frontmatter shard measured 1751s against 212s for the next slowest. Swap to @stryker-mutator/tap-runner (official plugin, peer-pinned to the @stryker-mutator/core@9.6.1 already installed) and turn coverage analysis on, so Stryker re-runs only the test files that cover each mutated line. Per-shard injection moves from a MUTATION_TEST_CMD command string to MUTATION_TEST_FILES, read by a new fail-closed resolveMutationTestFiles() that mirrors resolveMutationBreak. It stays derived from scripts/mutation-matrix.cjs and now has a single owner: the union moves behind allCoveredTests() and stryker.config.mjs no longer imports COVERED. The resolver existence-checks every entry, because the tap runner resolves tap.testFiles with glob() and a non-matching pattern yields an empty list silently - a fast, confident, meaningless run. tap.forceBail is off by measurement, not preference: a structural AST audit found 3 of 26 shard test files spawn subprocesses, and bail fires on every killed mutant, so leaving it on would kill processes mid-spawnSync and orphan their children. The matrix 'isolation' field is removed - per-file process isolation is inherent to the tap runner, so the field had no consumer left. Score arithmetic is unchanged and now pinned: mutationScore counts NoCoverage in the same denominator as Survived, which both Stryker's thresholds.break and check-mutation-score-ratchet.cjs read. A new non-vacuity test proves it diverges from mutationScoreBasedOnCoveredCode, so the gate cannot be quietly swapped to the field that would make every floor trivially satisfiable. Refs #3915 * fix(#3915): enforce the resolver's documented containment contract Review findings, all fixed in place. resolveMutationTestFiles claimed to verify each entry exists 'relative to the repo root' but used a bare fs.existsSync(path.join(...)), which accepts an existing DIRECTORY and lets ../ segments escape the root (path.join('/repo/root','../../etc/passwd') resolves to /etc/passwd). Not reachable from PR content - the value only ever comes from the static COVERED registry via CI env - but a fail-closed contract that overstates its own guarantee is a defect in the contract. Each entry is now resolved, rejected if path.relative puts it outside the root, and required to be a regular file. Three hostile-input tests added; the original missing-file wording is preserved so the existing assertion still binds. Also: removed a stale buildResult comment still naming the isolation field this branch deleted; hoisted one top-level path require in place of three inline ones; de-duplicated the derived-test-list expression in the tests to a single const, deliberately still re-derived from COVERED rather than calling allCoveredTests() so the assertion cannot become a tautology; tightened the workflow-parity assertion to exact equality; changed the tap-runner range from an exact 9.6.1 to ^9.6.1 so it tracks the caret-ranged core its peerDependency pins exactly. Regenerated examples/dynamic-context-management/CONTEXT-INDEX.json, which the earlier CONTEXT.md edit left stale - lint:ci was red on lint-example-parser-parity until it was refreshed. Refs #3915 * test(#3915): kill model-catalog survivors, set frontmatter budget from measurement From mutation run 33026833181 (all 13 shards, dispatched on this branch before any PR). WALL TIME: frontmatter measured 713s (11m53s) vs 1751s (29m11s) on the command runner, a 59% cut. timeoutMinutes 60 -> 20 (1.68x measured). Not deleted outright: the shared 15-minute default would leave only 21% headroom, and this module's mutant count grew 1.8x in one change. MODEL-CATALOG: came back 57.91 against its floor of 58. Diagnosed from the report JSON, not assumed - all 24 of its new RuntimeError mutants are in the load-time catalog bootstrap, so mutating them makes the module throw at require. Under node --test that is a failed test file (Killed); under the tap runner the process dies before emitting TAP, which Stryker classifies RuntimeError and excludes from the denominator. Add them back as killed and the shard is 248/416 = 59.62, the pre-change number exactly. Detection did not regress, classification changed. The floor is NOT lowered to absorb that. 11 new behavioural tests target ~42 genuinely surviving mutants: exact Set equality on the EFFORT_RENDERING/EFFORT_ARGV supported sets, an exact-string table render that makes the column-width arithmetic observable (padEnd never truncates, so the old substring assertion could not see a too-small width), clampEffortForHost's full null matrix plus a spoofed-toString host, and a prototype-pollution guard driven through Object.fromEntries. Mutants judged equivalent were skipped rather than papered over; reasoning is in the phase artifacts. All 15 new assertions verified against unmutated code. lint:ci exit 0. Refs #3915 * test(#3915): ratchet model-catalog's floor to 74 on measured 75.26% CI run 33029755081 measured model-catalog at 75.26% (295 killed / 49 survived / 48 no-coverage / 24 runtime-error, totalValid 392), so check-mutation-score-ratchet.cjs correctly failed the shard for unclaimed headroom: 17.26 points above the declared floor of 58, well past the 5-point slack. Floor raised 58 -> 74 (floor(75.26)-1, this file's documented convention), with RATCHET_BASELINE updated in the same diff as the equality assertion requires. Worth recording why the number moved this far. The shard first came back at 57.91 under the tap runner and the temptation was to lower the floor to match. The drop was not a regression: all 24 of its RuntimeError mutants sit in the load-time catalog bootstrap, and adding them back as killed reproduces 248/416 = 59.62, the pre-swap figure exactly. Rather than absorb a reporting artifact by weakening the gate, 11 behavioural tests went after the genuinely surviving mutants and killed 68 of them - carrying the module from 59.62 past its old ceiling to 75.26, within reach of the ADR-456 target of 80. The #3007 measurement is kept as clearly-labelled prior context rather than deleted, so the entry does not read as carrying two current numbers. Refs #3915 --------- Co-authored-by: sim <sim@local>
This commit is contained in:
@@ -41,11 +41,17 @@ const {
|
||||
CODEX_MODEL_EFFORT,
|
||||
MODEL_ALIAS_MAP,
|
||||
PROVIDER_PRESETS,
|
||||
AGENT_TO_PHASE_TYPE,
|
||||
AGENT_DEFAULT_TIERS,
|
||||
RUNTIME_PROFILE_MAP,
|
||||
EFFORT_RENDERING,
|
||||
EFFORT_ARGV,
|
||||
isAnthropicFlavoredModel,
|
||||
getAgentToModelMapForProfile,
|
||||
formatAgentToModelMapAsTable,
|
||||
renderEffortArgv,
|
||||
renderEffortForRuntime,
|
||||
clampEffortForHost,
|
||||
nextTier,
|
||||
mergeEffortTierDefaults,
|
||||
} = catalog;
|
||||
@@ -378,4 +384,128 @@ describe('model-catalog: mergeEffortTierDefaults (#3531)', () => {
|
||||
mergeEffortTierDefaults(manifest, { sonnet: 'opus' }, isValid);
|
||||
assert.deepEqual(manifest, original);
|
||||
});
|
||||
|
||||
// #3915 — a plain object literal's `__proto__: value` key is a prototype
|
||||
// setter, not an own-enumerable property, so `Object.entries()` never
|
||||
// yields it and the house-pollution guard's `tier === '__proto__'` branch
|
||||
// is unreachable via a normal test fixture. `Object.fromEntries` uses
|
||||
// CreateDataPropertyOrThrow internally and DOES create a genuine own
|
||||
// property literally named "__proto__", which `Object.entries()` then
|
||||
// yields — the only way to actually drive `tier` to that value and
|
||||
// exercise the guard's first disjunct. A permissive `isValid` (accepts
|
||||
// any value) is required here specifically so a disabled guard would let
|
||||
// the object-valued override reach `merged['__proto__'] = value`, which
|
||||
// (unlike a string value) really does repoint the prototype.
|
||||
test('__proto__ pollution guard fires even for a genuine own-enumerable "__proto__" key', () => {
|
||||
const permissive = () => true;
|
||||
const overrideProto = Object.fromEntries([['__proto__', { polluted: true }]]);
|
||||
const merged = mergeEffortTierDefaults({}, overrideProto, permissive);
|
||||
assert.strictEqual(Object.getPrototypeOf(merged), Object.prototype);
|
||||
assert.strictEqual(merged.polluted, undefined);
|
||||
});
|
||||
});
|
||||
|
||||
// #3915 — mutation-score restoration (Stryker survivors in model-catalog.cjs).
|
||||
// Each test below targets specific surviving mutants identified from the
|
||||
// mutation report; see the PR/issue for the full mutant-to-test mapping.
|
||||
describe('model-catalog: module load shape (#3915)', () => {
|
||||
test('the compiled module is flagged __esModule (defineProperty descriptor, not a loose object literal)', () => {
|
||||
assert.strictEqual(catalog.__esModule, true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('model-catalog: AGENT_TO_PHASE_TYPE / AGENT_DEFAULT_TIERS (#3915)', () => {
|
||||
test('AGENT_TO_PHASE_TYPE maps a known agent to its exact catalog phaseType', () => {
|
||||
assert.equal(AGENT_TO_PHASE_TYPE['gsd-planner'], 'planning');
|
||||
assert.equal(AGENT_TO_PHASE_TYPE['gsd-verifier'], 'verification');
|
||||
});
|
||||
|
||||
test('AGENT_DEFAULT_TIERS maps a known agent to its exact catalog routingTier', () => {
|
||||
assert.equal(AGENT_DEFAULT_TIERS['gsd-planner'], 'heavy');
|
||||
assert.equal(AGENT_DEFAULT_TIERS['gsd-codebase-mapper'], 'light');
|
||||
});
|
||||
});
|
||||
|
||||
describe('model-catalog: RUNTIME_PROFILE_MAP filtering (#3915)', () => {
|
||||
// The catalog's runtimeTierDefaults has runtimes whose opus/sonnet/haiku
|
||||
// entries are ALL null (e.g. 'cline') as a deliberate "no defaults yet"
|
||||
// sentinel, and runtimes fully populated (e.g. 'claude'). This pair is
|
||||
// the exact boundary the filter's `Object.keys(filtered).length > 0`
|
||||
// check exists for.
|
||||
test('a runtime whose tier entries are all null is dropped entirely', () => {
|
||||
assert.equal('cline' in RUNTIME_PROFILE_MAP, false);
|
||||
assert.equal('kimi' in RUNTIME_PROFILE_MAP, false);
|
||||
});
|
||||
|
||||
test('a fully-populated runtime keeps every tier entry, unmodified', () => {
|
||||
assert.deepStrictEqual(RUNTIME_PROFILE_MAP.claude, {
|
||||
opus: { model: 'claude-opus-4-8' },
|
||||
sonnet: { model: 'claude-sonnet-5' },
|
||||
haiku: { model: 'claude-haiku-4-5' },
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('model-catalog: EFFORT_RENDERING / EFFORT_ARGV supported sets (#3915)', () => {
|
||||
test('EFFORT_RENDERING.claude.supported is exactly low/medium/high/xhigh/max', () => {
|
||||
assert.deepStrictEqual(EFFORT_RENDERING.claude.supported, new Set(['low', 'medium', 'high', 'xhigh', 'max']));
|
||||
});
|
||||
|
||||
test('EFFORT_RENDERING.codex.supported is exactly low/medium/high/xhigh/max', () => {
|
||||
assert.deepStrictEqual(EFFORT_RENDERING.codex.supported, new Set(['low', 'medium', 'high', 'xhigh', 'max']));
|
||||
});
|
||||
|
||||
test("EFFORT_RENDERING.codex.clamp maps 'minimal' to 'low' and leaves other levels unchanged", () => {
|
||||
assert.equal(EFFORT_RENDERING.codex.clamp('minimal'), 'low');
|
||||
assert.equal(EFFORT_RENDERING.codex.clamp('high'), 'high');
|
||||
});
|
||||
|
||||
test('EFFORT_ARGV.claude.supported is exactly low/medium/high/xhigh/max', () => {
|
||||
assert.deepStrictEqual(EFFORT_ARGV.claude.supported, new Set(['low', 'medium', 'high', 'xhigh', 'max']));
|
||||
});
|
||||
|
||||
test('EFFORT_ARGV.opencode.supported additionally includes minimal', () => {
|
||||
assert.deepStrictEqual(EFFORT_ARGV.opencode.supported, new Set(['minimal', 'low', 'medium', 'high', 'xhigh', 'max']));
|
||||
});
|
||||
|
||||
test('EFFORT_ARGV.codex.supported is exactly low/medium/high/xhigh/max', () => {
|
||||
assert.deepStrictEqual(EFFORT_ARGV.codex.supported, new Set(['low', 'medium', 'high', 'xhigh', 'max']));
|
||||
});
|
||||
});
|
||||
|
||||
describe('model-catalog: formatAgentToModelMapAsTable — exact column widths (#3915)', () => {
|
||||
// Existing coverage only used `.includes()` on longer-than-header
|
||||
// agent/model names, which never exercises `Math.max('Agent'.length, ...)`
|
||||
// vs `Math.min` (padEnd never truncates, so a too-small width is
|
||||
// invisible to a substring check). Short entries plus a full-string
|
||||
// comparison make the width computation and the +2/-2 separator padding
|
||||
// observable.
|
||||
test('short agent/model names still pad to the header width, and the separator is exactly width+2 wide', () => {
|
||||
const out = formatAgentToModelMapAsTable({ ab: 'xy' });
|
||||
const expected = ` Agent │ Model\n${'─'.repeat(7)}┼${'─'.repeat(7)}\n ab │ xy \n`;
|
||||
assert.equal(out, expected);
|
||||
});
|
||||
});
|
||||
|
||||
describe('model-catalog: clampEffortForHost (#3915)', () => {
|
||||
test('non-string host, unknown host, non-string effort, and empty effort all return null; valid input clamps', () => {
|
||||
assert.equal(clampEffortForHost(123, 'high'), null);
|
||||
assert.equal(clampEffortForHost('totally-bogus-host', 'high'), null);
|
||||
assert.equal(clampEffortForHost('claude', 123), null);
|
||||
assert.equal(clampEffortForHost('claude', ''), null);
|
||||
assert.equal(clampEffortForHost('claude', 'minimal'), 'low');
|
||||
assert.equal(clampEffortForHost('claude', 'high'), 'high');
|
||||
});
|
||||
|
||||
// The own-property guard must reject a host BEFORE any property lookup
|
||||
// that could be coerced into a real key. `typeof host !== 'string'` short-
|
||||
// circuits the `||` for a non-string host, so `EFFORT_ARGV[host]` is never
|
||||
// reached even when the host's `toString()` would resolve to a real key —
|
||||
// proving the type check (not just the hasOwnProperty check) is load-
|
||||
// bearing, and that neither the `||` nor either disjunct can be disabled.
|
||||
test('a non-string host is rejected even when it stringifies to a known key', () => {
|
||||
const spoofedHost = { toString: () => 'claude' };
|
||||
assert.equal(clampEffortForHost(spoofedHost, 'high'), null);
|
||||
assert.equal(clampEffortForHost('claude', 'high'), 'high');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -23,13 +23,24 @@
|
||||
const { test, describe } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const path = require('node:path');
|
||||
const fs = require('node:fs');
|
||||
const fc = require('fast-check');
|
||||
const { runNode } = require('./helpers/process-seam.cjs');
|
||||
const { throwIfFailed } = require('./helpers/git-fixture.cjs');
|
||||
const { PROBE_TIMEOUT_MS } = require('./helpers/timeouts.cjs');
|
||||
const REPO_ROOT = path.resolve(__dirname, '..');
|
||||
|
||||
const MATRIX_SCRIPT = path.resolve(__dirname, '../scripts/mutation-matrix.cjs');
|
||||
const matrix = require(MATRIX_SCRIPT);
|
||||
|
||||
// Deliberately RE-DERIVED here from matrix.COVERED, rather than calling the
|
||||
// exported matrix.allCoveredTests() — resolveMutationTestFiles(undefined)'s
|
||||
// production implementation itself calls allCoveredTests(), so comparing its
|
||||
// output against that same function would make the assertion a tautology
|
||||
// that passes no matter what either function does. This independent
|
||||
// derivation is what makes the comparison test the production code.
|
||||
const derived = [...new Set(Object.values(matrix.COVERED).flatMap((mod) => mod.tests))].sort();
|
||||
|
||||
// ── (a) TARGET_MUTATION_SCORE ─────────────────────────────────────────────────
|
||||
describe('mutation-matrix ratchet: TARGET_MUTATION_SCORE export', () => {
|
||||
test('exports TARGET_MUTATION_SCORE', () => {
|
||||
@@ -202,7 +213,10 @@ const RATCHET_BASELINE = {
|
||||
'planning-inspect': 56, // CI run 32392791843: 57.03% (unit shard); ratchet candidate vs TARGET 80
|
||||
'plan-document': 75, // CI run 32392791843: 76.58% (unit shard)
|
||||
'planning-command-router': 94, // CI run 32392791843: 95.65% (unit shard); already exceeds TARGET 80
|
||||
'model-catalog': 58, // #3007: measured 59.62% in CI (248 killed / 168 survived); floor(59.62)-1
|
||||
'model-catalog': 74, // CI run 33029755081 (2026-08-27, #3915): measured 75.26%;
|
||||
// floor(75.26)-1. Supersedes the #3007 59.62% measurement
|
||||
// (248 killed / 168 survived) — see scripts/mutation-matrix.cjs
|
||||
// for the RuntimeError classification-change diagnosis.
|
||||
'state-contract': 65, // #3227: CI run 32769289750, job 97565813640,
|
||||
// `Stryker (state-contract)`: measured 66.25%; floor(66.25)-1.
|
||||
// Below TARGET_MUTATION_SCORE (80) — ratchet candidate like
|
||||
@@ -305,3 +319,204 @@ describe('resolveMutationBreak: fail-closed env-var resolver', () => {
|
||||
assert.strictEqual(resolveMutationBreak('62'), 62);
|
||||
});
|
||||
});
|
||||
|
||||
// ── (g) resolveMutationTestFiles behaviour (#3915: tap-runner test-file resolver) ────
|
||||
// stryker's tap-runner has no single `command` to inject a per-shard test list into —
|
||||
// it takes an explicit `tap.testFiles` array — so the derived-union default that used to
|
||||
// live only in stryker.config.mjs's DEFAULT_TEST_CMD string needs a resolver twin to
|
||||
// resolveMutationBreak: same fail-closed shape, same single call site, this time
|
||||
// producing a file-list array instead of a numeric threshold.
|
||||
describe('resolveMutationTestFiles: fail-closed test-file-list resolver', () => {
|
||||
const { resolveMutationTestFiles } = matrix;
|
||||
|
||||
test('exports resolveMutationTestFiles as a function', () => {
|
||||
assert.strictEqual(typeof resolveMutationTestFiles, 'function');
|
||||
});
|
||||
|
||||
test('undefined → sorted, de-duplicated union of every COVERED[*].tests', () => {
|
||||
assert.deepStrictEqual(resolveMutationTestFiles(undefined), derived);
|
||||
});
|
||||
|
||||
test('every entry of the undefined-default result exists on disk', () => {
|
||||
for (const entry of resolveMutationTestFiles(undefined)) {
|
||||
assert.ok(
|
||||
fs.existsSync(path.resolve(REPO_ROOT, entry)),
|
||||
`derived default test file does not exist on disk: ${entry}`
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('single test file (boundary 1) → one-element array', () => {
|
||||
assert.deepStrictEqual(
|
||||
resolveMutationTestFiles('tests/frontmatter.unit.test.cjs'),
|
||||
['tests/frontmatter.unit.test.cjs']
|
||||
);
|
||||
});
|
||||
|
||||
test('two space-separated test files → two-element array', () => {
|
||||
assert.deepStrictEqual(
|
||||
resolveMutationTestFiles('tests/frontmatter.unit.test.cjs tests/unusable-input.test.cjs'),
|
||||
['tests/frontmatter.unit.test.cjs', 'tests/unusable-input.test.cjs']
|
||||
);
|
||||
});
|
||||
|
||||
test("'' (empty string) → throws (CI shard wiring error)", () => {
|
||||
assert.throws(() => resolveMutationTestFiles(''), /set but empty/);
|
||||
});
|
||||
|
||||
test("' ' (whitespace-only) → throws (CI shard wiring error)", () => {
|
||||
assert.throws(() => resolveMutationTestFiles(' '), /set but empty/);
|
||||
});
|
||||
|
||||
test("'\\t\\n ' (mixed whitespace-only) → throws (CI shard wiring error)", () => {
|
||||
assert.throws(() => resolveMutationTestFiles('\t\n '), /set but empty/);
|
||||
});
|
||||
|
||||
test('null → throws', () => {
|
||||
assert.throws(() => resolveMutationTestFiles(null));
|
||||
});
|
||||
|
||||
test('123 (non-string) → throws', () => {
|
||||
assert.throws(() => resolveMutationTestFiles(123));
|
||||
});
|
||||
|
||||
test('an array (not a string) → throws', () => {
|
||||
assert.throws(() => resolveMutationTestFiles(['tests/frontmatter.unit.test.cjs']));
|
||||
});
|
||||
|
||||
test('leading/trailing whitespace is trimmed to exactly one entry with no surrounding whitespace', () => {
|
||||
const result = resolveMutationTestFiles(' tests/frontmatter.unit.test.cjs ');
|
||||
assert.strictEqual(result.length, 1);
|
||||
assert.strictEqual(result[0], result[0].trim());
|
||||
assert.strictEqual(result[0], 'tests/frontmatter.unit.test.cjs');
|
||||
});
|
||||
|
||||
test('mixed tab/newline/space separators → exactly 3 non-empty entries', () => {
|
||||
const result = resolveMutationTestFiles(
|
||||
'tests/frontmatter.unit.test.cjs\t\ttests/unusable-input.test.cjs\n tests/frontmatter.property.test.cjs'
|
||||
);
|
||||
assert.strictEqual(result.length, 3);
|
||||
for (const entry of result) assert.notStrictEqual(entry, '');
|
||||
});
|
||||
|
||||
test('CRLF-separated entries → exactly 2 entries, none carrying a \\r', () => {
|
||||
const result = resolveMutationTestFiles(
|
||||
'tests/frontmatter.unit.test.cjs\r\ntests/unusable-input.test.cjs'
|
||||
);
|
||||
assert.strictEqual(result.length, 2);
|
||||
for (const entry of result) assert.ok(!entry.includes('\r'), `entry retained a \\r: ${JSON.stringify(entry)}`);
|
||||
});
|
||||
|
||||
test('a repeated entry is de-duplicated to exactly 1', () => {
|
||||
const result = resolveMutationTestFiles(
|
||||
'tests/frontmatter.unit.test.cjs tests/frontmatter.unit.test.cjs'
|
||||
);
|
||||
assert.deepStrictEqual(result, ['tests/frontmatter.unit.test.cjs']);
|
||||
});
|
||||
|
||||
test('a nonexistent test file throws, naming the bad path', () => {
|
||||
assert.throws(
|
||||
() => resolveMutationTestFiles('tests/does-not-exist-3915.test.cjs'),
|
||||
/does-not-exist-3915/
|
||||
);
|
||||
});
|
||||
|
||||
test('one bad entry among otherwise-valid entries still throws', () => {
|
||||
assert.throws(
|
||||
() => resolveMutationTestFiles('tests/frontmatter.unit.test.cjs tests/does-not-exist-3915.test.cjs'),
|
||||
/does-not-exist-3915/
|
||||
);
|
||||
});
|
||||
|
||||
test('an entry naming an existing directory throws, indicating it is not a regular file', () => {
|
||||
assert.throws(
|
||||
() => resolveMutationTestFiles('tests'),
|
||||
/not a regular file/
|
||||
);
|
||||
});
|
||||
|
||||
test('an entry escaping the repo root via ../../../etc/passwd throws, indicating the escape', () => {
|
||||
assert.throws(
|
||||
() => resolveMutationTestFiles('../../../etc/passwd'),
|
||||
/escape the repo root/
|
||||
);
|
||||
});
|
||||
|
||||
test('an entry escaping via a valid-looking prefix (tests/../../outside-3915.cjs) throws for escaping the repo root', () => {
|
||||
assert.throws(
|
||||
() => resolveMutationTestFiles('tests/../../outside-3915.cjs'),
|
||||
/escape the repo root/
|
||||
);
|
||||
});
|
||||
|
||||
test('the full derived default, space-joined, round-trips through the resolver', () => {
|
||||
assert.deepStrictEqual(resolveMutationTestFiles(derived.join(' ')), derived);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveMutationTestFiles: round-trip property', () => {
|
||||
const { resolveMutationTestFiles } = matrix;
|
||||
|
||||
test('any non-empty subset of the derived default, space-joined, resolves back to that de-duplicated subset', () => {
|
||||
fc.assert(
|
||||
fc.property(
|
||||
fc.subarray(derived, { minLength: 1 }),
|
||||
(subset) => {
|
||||
const expected = [...new Set(subset)].sort();
|
||||
const actual = resolveMutationTestFiles(subset.join(' '));
|
||||
assert.deepStrictEqual(actual, expected);
|
||||
}
|
||||
),
|
||||
{ seed: 3915, numRuns: 100 }
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// ── (h) mutation matrix: isolation field removed (#3915) ─────────────────────
|
||||
// stryker's tap-runner has no equivalent of `node --test --test-isolation=<mode>`
|
||||
// (it drives Node's own TAP test-reporter file-by-file), so the per-shard
|
||||
// `isolation` field this matrix used to emit for mutation.yml's
|
||||
// MUTATION_TEST_CMD env line has nothing left to wire into and must be removed
|
||||
// from buildResult()'s output entirely, not merely left unused.
|
||||
describe('mutation matrix: isolation field removed (#3915)', () => {
|
||||
const { resolveMutationTestFiles } = matrix;
|
||||
|
||||
test('every emitted matrix entry has no isolation field but keeps the rest of the contract', () => {
|
||||
const covered = matrix.COVERED || {};
|
||||
const moduleNames = Object.keys(covered);
|
||||
const stdinLines = moduleNames.map((name) => `src/${name}.cts`).join('\n');
|
||||
|
||||
const spawnResult = runNode(
|
||||
[MATRIX_SCRIPT],
|
||||
{
|
||||
input: stdinLines + '\n',
|
||||
cwd: REPO_ROOT,
|
||||
timeoutMs: PROBE_TIMEOUT_MS,
|
||||
}
|
||||
);
|
||||
throwIfFailed(spawnResult, `node ${MATRIX_SCRIPT}`);
|
||||
const result = JSON.parse(spawnResult.stdout);
|
||||
|
||||
assert.ok(result.matrix.include.length > 0, 'matrix.include must not be empty');
|
||||
|
||||
for (const entry of result.matrix.include) {
|
||||
assert.ok(
|
||||
!Object.prototype.hasOwnProperty.call(entry, 'isolation'),
|
||||
`matrix entry for '${entry.name}' must NOT include an isolation field (#3915: tap-runner has no test-isolation flag)`
|
||||
);
|
||||
for (const key of ['name', 'mutate', 'tests', 'minScore', 'timeoutMinutes']) {
|
||||
assert.ok(
|
||||
Object.prototype.hasOwnProperty.call(entry, key),
|
||||
`matrix entry for '${entry.name}' must still include '${key}'`
|
||||
);
|
||||
}
|
||||
|
||||
// Round-trip between the two halves of the wiring: the tests string this
|
||||
// matrix emits must resolve back to exactly the COVERED entry's own tests.
|
||||
assert.deepStrictEqual(
|
||||
resolveMutationTestFiles(entry.tests),
|
||||
matrix.COVERED[entry.name].tests
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -81,6 +81,62 @@ describe('extractAchievedScore: Stryker json-reporter document', () => {
|
||||
test('throws when the report has no scoreable mutants (empty files map — a wiring bug, never a 0% score)', () => {
|
||||
assert.throws(() => extractAchievedScore({ schemaVersion: '1.0', thresholds: {}, files: {} }), /no scoreable mutants/);
|
||||
});
|
||||
|
||||
// #3915: coverageAnalysis flips 'off' -> 'perTest' for the tap-runner swap, which is
|
||||
// what makes NoCoverage a status Stryker can now actually emit for these shards (the
|
||||
// command runner's 'off' analysis never produced it). extractAchievedScore must already
|
||||
// treat NoCoverage exactly like Survived in the denominator, or every shard's floor
|
||||
// silently becomes easier to clear the moment mutants start reporting NoCoverage instead
|
||||
// of Survived.
|
||||
function buildCountReport({ Killed = 0, Survived = 0, NoCoverage = 0 } = {}) {
|
||||
const mutants = [];
|
||||
let id = 0;
|
||||
const push = (status, n) => {
|
||||
for (let i = 0; i < n; i += 1) {
|
||||
mutants.push({
|
||||
id: String(id++),
|
||||
mutatorName: 'a',
|
||||
status,
|
||||
location: { start: { line: 1, column: 1 }, end: { line: 1, column: 2 } },
|
||||
});
|
||||
}
|
||||
};
|
||||
push('Killed', Killed);
|
||||
push('Survived', Survived);
|
||||
push('NoCoverage', NoCoverage);
|
||||
return {
|
||||
schemaVersion: '1.0',
|
||||
thresholds: { high: 80, low: 60 },
|
||||
files: { 'foo.js': { language: 'javascript', source: 'x', mutants } },
|
||||
};
|
||||
}
|
||||
|
||||
test('8 Killed + 2 Survived → 80', () => {
|
||||
assert.strictEqual(extractAchievedScore(buildCountReport({ Killed: 8, Survived: 2 })), 80);
|
||||
});
|
||||
|
||||
// NoCoverage counts in the mutation-score denominator exactly as Survived does, which is
|
||||
// what makes the #3915 coverageAnalysis 'off'->'perTest' switch score-neutral; Stryker's
|
||||
// own break threshold reads this same field (mutation-test-report-helper.js).
|
||||
test('8 Killed + 2 NoCoverage → 80, NOT 100', () => {
|
||||
assert.strictEqual(extractAchievedScore(buildCountReport({ Killed: 8, NoCoverage: 2 })), 80);
|
||||
});
|
||||
|
||||
// Non-vacuity guard (Goodhart guard): proves the previous test is not a tautology by
|
||||
// showing mutationScore and mutationScoreBasedOnCoveredCode genuinely diverge on the same
|
||||
// report. This fails the moment someone "improves" extractAchievedScore to read the
|
||||
// better-sounding covered-code field, which would make every floor trivially satisfiable.
|
||||
test('non-vacuity: mutationScoreBasedOnCoveredCode (100) diverges from extractAchievedScore (80) on the same report', () => {
|
||||
const { calculateMutationTestMetrics } = require('mutation-testing-metrics');
|
||||
const report = buildCountReport({ Killed: 8, NoCoverage: 2 });
|
||||
const metrics = calculateMutationTestMetrics(report);
|
||||
assert.strictEqual(metrics.systemUnderTestMetrics.metrics.mutationScoreBasedOnCoveredCode, 100);
|
||||
assert.strictEqual(extractAchievedScore(report), 80);
|
||||
});
|
||||
|
||||
test('0 Killed + 10 NoCoverage → 0 (a wholly-uncovered shard scores zero, not 100)', () => {
|
||||
assert.strictEqual(extractAchievedScore(buildCountReport({ NoCoverage: 10 })), 0);
|
||||
});
|
||||
});
|
||||
|
||||
// ── CLI end-to-end: fail on a planted over-floor score, pass when the floor is raised ───
|
||||
|
||||
189
tests/mutation-tap-runner-wiring.test.cjs
Normal file
189
tests/mutation-tap-runner-wiring.test.cjs
Normal file
@@ -0,0 +1,189 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* tests/mutation-tap-runner-wiring.test.cjs
|
||||
*
|
||||
* Pins the #3915 tap-runner wiring — the config contract AND the
|
||||
* workflow<->config parity, because the injected env token lives on two
|
||||
* surfaces (`.github/workflows/mutation.yml`'s per-shard `env:` block and
|
||||
* `stryker.config.mjs`'s reader of it) and silent drift between them would
|
||||
* make every shard silently fall back to running the FULL default test list
|
||||
* instead of its own module's tests — slow, wrong, and (because Stryker
|
||||
* would still find SOME test constraining each mutant) green.
|
||||
*
|
||||
* FAILING-FIRST (#3915): stryker.config.mjs still declares `testRunner:
|
||||
* 'command'` with a `commandRunner.command` built from `MUTATION_TEST_CMD`,
|
||||
* and mutation.yml still injects `MUTATION_TEST_CMD` / `matrix.isolation`.
|
||||
* Every test below targets the tap-runner shape those files do not have yet.
|
||||
*/
|
||||
|
||||
const { test, describe } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const path = require('node:path');
|
||||
const fs = require('node:fs');
|
||||
const { pathToFileURL } = require('node:url');
|
||||
const yaml = require('js-yaml');
|
||||
|
||||
const REPO_ROOT = path.resolve(__dirname, '..');
|
||||
const CONFIG_PATH = path.join(REPO_ROOT, 'stryker.config.mjs');
|
||||
const WORKFLOW_PATH = path.join(REPO_ROOT, '.github', 'workflows', 'mutation.yml');
|
||||
|
||||
// Env keys this config is known to read. Every key is saved/restored so a
|
||||
// test's env override can never leak into a sibling test.
|
||||
const RELEVANT_ENV_KEYS = ['MUTATION_TEST_FILES', 'MUTATION_TEST_CMD', 'MUTATION_BREAK'];
|
||||
|
||||
// Cache-busting counter: importing the SAME file:// URL twice returns the
|
||||
// SAME cached ES module record, so an env-dependent config test that reused
|
||||
// one URL across calls would silently observe the FIRST call's env forever —
|
||||
// every subsequent env-dependent assertion would pass or fail for the wrong
|
||||
// reason. Incrementing this per call forces a fresh module evaluation.
|
||||
let _importCounter = 0;
|
||||
|
||||
/**
|
||||
* Load stryker.config.mjs with `env` applied on top of process.env for the
|
||||
* duration of the import, then restore process.env exactly. `env` values of
|
||||
* `undefined` delete the corresponding key rather than setting it.
|
||||
*
|
||||
* @param {Record<string, string|undefined>} env
|
||||
* @returns {Promise<object>} the config module's default export
|
||||
*/
|
||||
async function loadConfig(env) {
|
||||
// Save/restore the UNION of RELEVANT_ENV_KEYS and Object.keys(env), not just the fixed
|
||||
// list: the PARITY test below discovers its env key NAME from the parsed workflow file
|
||||
// rather than hardcoding it, so a key outside RELEVANT_ENV_KEYS can reach here — saving
|
||||
// only the fixed list would let that discovered key leak into sibling tests.
|
||||
const keysToRestore = new Set([...RELEVANT_ENV_KEYS, ...Object.keys(env)]);
|
||||
const saved = {};
|
||||
for (const key of keysToRestore) saved[key] = process.env[key];
|
||||
for (const [key, value] of Object.entries(env)) {
|
||||
if (value === undefined) delete process.env[key];
|
||||
else process.env[key] = value;
|
||||
}
|
||||
try {
|
||||
const mod = await import(`${pathToFileURL(CONFIG_PATH).href}?v=${_importCounter++}`);
|
||||
return mod.default;
|
||||
} finally {
|
||||
for (const key of keysToRestore) {
|
||||
if (saved[key] === undefined) delete process.env[key];
|
||||
else process.env[key] = saved[key];
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
describe('stryker.config.mjs: tap runner contract (#3915)', () => {
|
||||
test("testRunner === 'tap'", async () => {
|
||||
const config = await loadConfig({});
|
||||
assert.strictEqual(config.testRunner, 'tap');
|
||||
});
|
||||
|
||||
test("coverageAnalysis === 'perTest'", async () => {
|
||||
const config = await loadConfig({});
|
||||
assert.strictEqual(config.coverageAnalysis, 'perTest');
|
||||
});
|
||||
|
||||
test("coverageAnalysis !== 'off' (the literal condition #3915 requires to stop being true)", async () => {
|
||||
const config = await loadConfig({});
|
||||
assert.notStrictEqual(config.coverageAnalysis, 'off');
|
||||
});
|
||||
|
||||
test('no own-property commandRunner (the command runner is gone)', async () => {
|
||||
const config = await loadConfig({});
|
||||
assert.ok(!Object.prototype.hasOwnProperty.call(config, 'commandRunner'));
|
||||
});
|
||||
|
||||
test('tap.forceBail === false', async () => {
|
||||
const config = await loadConfig({});
|
||||
assert.strictEqual(config.tap.forceBail, false);
|
||||
});
|
||||
|
||||
test('no own-property buildCommand, and tap has no own-property nodeArgs (no rebuild step reintroduced, ADR-457)', async () => {
|
||||
const config = await loadConfig({});
|
||||
assert.ok(!Object.prototype.hasOwnProperty.call(config, 'buildCommand'));
|
||||
assert.ok(!Object.prototype.hasOwnProperty.call(config.tap, 'nodeArgs'));
|
||||
});
|
||||
|
||||
test('MUTATION_TEST_FILES set → tap.testFiles deep-equals the parsed entries', async () => {
|
||||
const config = await loadConfig({
|
||||
MUTATION_TEST_FILES: 'tests/frontmatter.unit.test.cjs tests/unusable-input.test.cjs',
|
||||
});
|
||||
assert.deepStrictEqual(config.tap.testFiles, [
|
||||
'tests/frontmatter.unit.test.cjs',
|
||||
'tests/unusable-input.test.cjs',
|
||||
]);
|
||||
});
|
||||
|
||||
test('MUTATION_BREAK set → thresholds.break reflects it unchanged (re-proved after the runner swap)', async () => {
|
||||
const config = await loadConfig({ MUTATION_BREAK: '72' });
|
||||
assert.strictEqual(config.thresholds.break, 72);
|
||||
});
|
||||
|
||||
test('MUTATION_TEST_FILES set but empty → fail-closed survives all the way to config load', async () => {
|
||||
await assert.rejects(() => loadConfig({ MUTATION_TEST_FILES: '' }));
|
||||
});
|
||||
|
||||
test('mutate scope unchanged by the runner swap', async () => {
|
||||
const config = await loadConfig({});
|
||||
assert.ok(Array.isArray(config.mutate));
|
||||
assert.ok(config.mutate.includes('gsd-core/bin/lib/**/*.cjs'));
|
||||
assert.ok(
|
||||
config.mutate.some((entry) => entry.startsWith('!gsd-core/bin/lib/')),
|
||||
'mutate array must still carry at least one !gsd-core/bin/lib/... exclusion'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('mutation.yml <-> stryker.config.mjs: injected env parity (#3915)', () => {
|
||||
const workflowDoc = yaml.load(fs.readFileSync(WORKFLOW_PATH, 'utf8'));
|
||||
const mutateJob = workflowDoc.jobs.mutate;
|
||||
const runStrykerStep = mutateJob.steps.find(
|
||||
(step) => typeof step.name === 'string' && step.name.startsWith('Run Stryker')
|
||||
);
|
||||
|
||||
test("the 'Run Stryker' step exists", () => {
|
||||
assert.ok(runStrykerStep, "no step in the mutate job's steps has a name starting with 'Run Stryker'");
|
||||
});
|
||||
|
||||
test('step.env has key MUTATION_TEST_FILES', () => {
|
||||
assert.ok(Object.prototype.hasOwnProperty.call(runStrykerStep.env, 'MUTATION_TEST_FILES'));
|
||||
});
|
||||
|
||||
test('step.env does NOT have key MUTATION_TEST_CMD', () => {
|
||||
assert.ok(!Object.prototype.hasOwnProperty.call(runStrykerStep.env, 'MUTATION_TEST_CMD'));
|
||||
});
|
||||
|
||||
test('MUTATION_TEST_FILES value is exactly the ${{ matrix.tests }} expression (derived from mutation-matrix.cjs)', () => {
|
||||
assert.strictEqual(String(runStrykerStep.env.MUTATION_TEST_FILES).trim(), '${{ matrix.tests }}');
|
||||
});
|
||||
|
||||
test('MUTATION_BREAK still references matrix.minScore', () => {
|
||||
assert.ok(Object.prototype.hasOwnProperty.call(runStrykerStep.env, 'MUTATION_BREAK'));
|
||||
assert.ok(String(runStrykerStep.env.MUTATION_BREAK).includes('matrix.minScore'));
|
||||
});
|
||||
|
||||
test('no value anywhere in the step env mentions --test-isolation', () => {
|
||||
for (const value of Object.values(runStrykerStep.env)) {
|
||||
assert.ok(!String(value).includes('--test-isolation'), `unexpected --test-isolation reference: ${value}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('no value anywhere in the whole mutate job mentions matrix.isolation', () => {
|
||||
const serialized = JSON.stringify(mutateJob);
|
||||
assert.ok(!serialized.includes('matrix.isolation'), 'mutate job still references matrix.isolation');
|
||||
});
|
||||
|
||||
test("the step's run string does not contain '${{' (no direct interpolation inside run:, CONTRIBUTING.md)", () => {
|
||||
assert.ok(!runStrykerStep.run.includes('${{'), 'run: block contains a direct ${{ }} interpolation');
|
||||
});
|
||||
|
||||
test('PARITY: the env key name discovered from the workflow drives the config test directly, so the two surfaces cannot drift', async () => {
|
||||
// Find the key in step.env that starts with MUTATION_TEST_ — this is read
|
||||
// from the PARSED workflow document, not hardcoded, so a rename on either
|
||||
// surface (the workflow's env key, or stryker.config.mjs's reader of it)
|
||||
// breaks this test instead of the two silently drifting apart.
|
||||
const discoveredKey = Object.keys(runStrykerStep.env).find((k) => k.startsWith('MUTATION_TEST_'));
|
||||
assert.ok(discoveredKey, 'no MUTATION_TEST_* key found in the Run Stryker step env');
|
||||
|
||||
const config = await loadConfig({ [discoveredKey]: 'tests/frontmatter.unit.test.cjs' });
|
||||
assert.deepStrictEqual(config.tap.testFiles, ['tests/frontmatter.unit.test.cjs']);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user