msd-core/docs/adr/4641-windows-selector-consolidation.md

ADR-4641: One Windows test selector, and a proportional ceiling on the conformance tier

• Status: Accepted
• Date: 2026-09-11
• Issue: #4641
• Twin of: ADR-4593 (macOS conformance tier) — that ADR applied evidence-first sizing to the macOS signal set; this one applies the same discipline to the Windows signal set and to the number of selectors, which ADR-4593 did not cover.
• Closes a gap in: epic #4589, whose goal — "the OS-agnostic majority of the suite runs once, on Linux, while a small and explicitly-scoped platform-conformance tier covers genuinely OS-specific behavior" — was not achieved by its five merged phases.

Context

Epic #4589 closed 2026-09-10 with every acceptance box checked. Measured on PR #4640 (run 34618834118), a routine tests/**-touching PR, two things were true that the goal text forbids.

Two independent Windows selectors

selector	gate	what it runs
test job, three scope: windows rows	product_changed	windows_tests = every changed tests/*.test.cjs, unconditionally
test-conformance, three windows shards	code_changed && full_matrix	the generated conformance tier

The epic replaced test-full with test-conformance and never touched the first selector, which predates it (#494, sharded in #3057). On PR #4640, 5 of 7 changed test files ran on a real Windows runner twice. Non-Linux job count was 7, not the 4 the epic's closeout reported — that figure compared test-full (6) against test-conformance (4) and omitted the three always-on scope: windows shards from both sides. Counting every non-Linux job, the epic moved 9 → 7, not 6 → 4.

The tier was 59% of the suite

node scripts/gen-platform-conformance-tier.cjs reported 546 of 930 eligible unit-suite files (58.7%) when this was diagnosed. Measured on the tree this PR actually ships against (932 eligible, after #4253 and #4644 landed on next mid-flight) the same comparison is 548 → 257 by detector removal alone. Absolute counts drift every time next gains a test file; the percentages did not move at all across three rebases (58.8% → 28.5%), which is the whole reason the ceilings are ratios. Measured per-category contribution, where UNIQUE is the count of files for which that category is the sole signal — i.e. the marginal cost of keeping it:

category	total	UNIQUE
process-seam-subprocess	335	118
hardcoded-path-vs-path-call	328	108
raw-child-process	96	19
symlink-keyword	86	6
win32-darwin-literal	77	1
process-platform	73	1
chmod-mode-bit	73	4
windows-env-var	69	6
windows-shell-token	23	4
os-platform	2	0

Two categories carried 226 of the tier's sole-signal membership; the other eight carried 41 combined.

• process-seam-subprocess matches runNode( / runGit( / runHook( / runMsdTools( / gitOrThrow( — the repo's own tests/helpers.cjs entry points, used by nearly every CLI test. For the overwhelming majority of them, going through the seam is the opposite of a platform signal: src/shell-command-projection.cts takes platform as an injected parameter, and tests/shell-command-projection-dispatch.test.cjs already exercises PowerShell/cmd.exe/PATHEXT in-process on Linux by passing platform: 'win32' as data. That is epic #4589's own argument for why the cutover was safe, applied against itself. But see the Consequences section: this detector was 99% noise wrapping a real signal — the ~9 tests that spawn a real shell via runHook's interpreter option — and that signal is preserved by a narrow replacement category rather than lost with the blanket one.
• hardcoded-path-vs-path-call requires a path.join|resolve|…( call anywhere in the file AND a quoted '/…' literal anywhere in the file, with no proximity. In a Node test suite both are universal. The defect class it gestures at is already enforced by Linux-runnable ESLint rules under ADR-1703 (no-hardcoded-tmp, no-path-literal-in-assert).

The repo had already reached this conclusion for one consumer and not the other. gen-platform-conformance-tier.cjs carried:

// Two CATEGORIES entries precise enough for TEST-file classification (this
// module's own purpose) but far too broad for SOURCE-file reachability
const NOISY_FOR_SOURCE_REACHABILITY = new Set(['hardcoded-path-vs-path-call', 'symlink-keyword']);

Measured there: classifyContent over src/ flagged 100 of 235 files; excluding these two narrowed it to 28, "all verified to carry a genuine platform-conditional branch." The same over-breadth verdict was reached, recorded, and then not applied to the tier itself.

Why no gate caught it

Phase 2's acceptance criterion was "conformance-tier file list exists as a single source of truth." It checked that the list exists. No phase asserted it was small, and no test would have failed if the classifier had put all 930 files in. The #4591 per-file parity-baseline diff that would have caught it was explicitly never performed — disclosed in the generator's own header as a KNOWN LIMIT — and Phase 5 then retired test-full, the safety net that disclosure named as its compensating control.

Decision

1. Delete the scope: windows lane; port its residue to reachability

The alternative — gating windows.add(file) on reachesConformanceTierOrSeam — was evaluated and rejected as provably redundant. For a test file that predicate is literally:

if (file.startsWith('tests/') && file.endsWith('.test.cjs')) {
  const { CONFORMANCE_TIER_FILES } = loadConformanceTier();
  return CONFORMANCE_TIER_FILES.includes(file);
}

and the same predicate is what sets full_matrix, which is what turns test-conformance on. So every file a gated lane would run is (a) already in the tier and (b) has already caused the conformance lane to run the whole tier in the same workflow run. Gating does not reduce the duplication; it makes it total.

The lane's one non-redundant contribution is the isWindowsHint arm — tests pulled in by a path RULE whose filename contains windows/win32/shell/path. That is a filename substring heuristic, precisely the kind of unproven heuristic #4592 replaced with reachability. It is therefore ported into reachesConformanceTierOrSeam: such a RULE-pulled test now sets full_matrix = true, and the conformance lane covers it. The signal is preserved; the parallel lane is not.

The escalation is tier-backed, and that condition is load-bearing. Three variants were measured over the 16 entries of RULES:

variant	predicate	rules firing	verdict
A	isWindowsHint(t)	6/16	Can fire on a test that is not in the tier — full_matrix goes true, the conformance lane runs, and the hinted test still never runs on Windows. Cost without coverage.
B (shipped)	isWindowsHint(t) && reachesConformanceTierOrSeam(t)	6/16	Identical firing set to A today, so no behavior change — but correct by construction: it can only escalate when the conformance lane will actually run the file.
C	reachesConformanceTierOrSeam(t) alone	14/16	Rejected as over-broad. Would newly escalate most ordinary product-code PRs (src/, agents/, commands/, hooks/, skills/, config paths) — tier membership alone is too weak a trigger.

A and B coincide only because every windows-hint test currently pulled in by a rule happens to be in the tier except one (tests/normalize-path-in-content.rule.test.cjs, which has zero signals). B is shipped because that coincidence is not an invariant.

Live effect is deliberately small: of the six rules that fire, four already set fullMatrix: true (no-op), inert CI's escalation is overridden downstream by the inert-CI reset, and exactly one — portability lint rules (ADR-1703) — genuinely changes behavior.

test-conformance becomes the sole Windows selector, matching how it already is the sole macOS selector.

2. Remove the two house-idiom detectors, and add one narrow replacement

process-seam-subprocess and hardcoded-path-vs-path-call are deleted from CATEGORIES. hardcoded-path-vs-path-call leaves NOISY_FOR_SOURCE_REACHABILITY with it (the set now holds symlink-keyword alone). Tier, all measured on one tree (932 eligible): 548 → 257 (58.8% → 27.6%) by removing the two detectors, then 257 → 266 (28.5%) once the narrow shell-interpreter-spawn replacement added 9 genuinely shell-spawning tests back. Net: 282 files removed, 9 restored. ALWAYS_REAL_OS currently adds 0 — see Consequences for why it is still there.

Measured, the change is surgical: src/ reachability is 28 → 28, zero files change status, because hardcoded-path-vs-path-call was already excluded there and no src/ file matches the test-helper regexes. Phase 3's classifier behavior for src/ diffs is provably unchanged.

3. A proportional ceiling, asserted failing-first

The missing Phase 2 gate is added as a test, and it is expressed as a ratio against a live denominator, not a count:

tier	measured	ceiling
Windows (CONFORMANCE_TIER_FILES)	28.5%	33%
macOS (MACOS_CONFORMANCE_TIER_FILES)	21.2%	25%

An absolute count goes stale as the suite grows and silently stops binding; the property that matters — "a tier, not the suite" — is inherently proportional. The ceiling is deliberately not today's emitted value, which #4641 rules out explicitly as a non-bound.

Consequences

• Non-Linux jobs on a full_matrix PR: 7 → 4. Against the true pre-epic baseline of 9, epic #4589 plus this ADR deliver 9 → 4 (-56%), versus the -33% its closeout claimed against a denominator that excluded this lane.

  Measured, not computed — read off real job lists rather than derived from the workflow file, which is the verification epic #4589's own closeout skipped:

  	PR #4640 (the trigger)	PR #4643 (this change)
  jobs in the completed test.yml run	21	17
  non-Linux jobs	7	4
  test job	4 ubuntu + 3 windows	4 ubuntu, 0 windows
  conformance tier size	548 files	266 files

  Both job totals are counted the same way — every job in the completed run, which includes the post-test Coverage gate and baseline-publisher jobs. An earlier draft of this table compared #4640's completed total against this run's count at matrix-expansion time, before those trailing jobs exist; that is an apples-to-oranges comparison and the kind of error this ADR is otherwise about, so it is called out rather than quietly corrected.

  One caveat stated rather than glossed: a PR's total check count is not a clean before/after, because many gates are path-scoped and this change touches a broader path set than #4640. The like-for-like figure is the test.yml job count and its non-Linux portion, which is what the epic's goal was about.

  Wall-clock, measured on both runs — and the honest read is that this is a correctness win more than a speed one:

  conformance job	#4640 (548-file tier)	#4643 (266-file tier)
  windows shard 1/3	29m47s	21m12s	-29%
  windows shard 2/3	29m00s	26m21s	-9%
  windows shard 3/3	40m24s	31m27s	-22%
  macOS	17m48s	21m02s	+18%

  File count fell 52% but wall-clock only 9-29%, because the files removed were the cheap static ones — the tier that remains is concentrated in genuinely expensive spawn-heavy work, which is exactly what it should contain. Do not expect a future narrowing to buy time proportional to file count. The macOS figure moved the wrong way while its tier was unchanged by this PR (198 files; it tracks next's test count, not this change), which fixes it as runner variance rather than an effect of this change, and is a caution against reading any single duration as signal.

  The load-bearing number is shard 3/3: it ran at 40m24s against a 45-minute cap, 90% of the cliff that #869 and #3057 were both filed about. Pulling it to 31m27s restores real headroom.

• 282 test files leave real-OS Windows execution — 291 dropped when the two detectors were removed, 9 restored by the narrow shell-interpreter-spawn replacement. This is a real coverage change, not a refactor. It is defensible because every file that stays out does so by losing a signal that was never a platform signal — each remains covered by the Linux run, and the files that genuinely spawn a real binary are untouched or restored (raw-child-process, 96 files; shell-interpreter-spawn, 33).

  The drop-out set was audited rather than assumed. Of those initially dropped, 14 had a filename suggesting platform relevance (/windows|win32|shell|path|platform|posix|crlf|symlink|exec|spawn|subprocess/i), and each was inspected. Six carry an explicit allow-test-rule: source-text-is-the-product or structural-regression-guard marker; the rest were read individually.

  That audit initially reached the wrong conclusion, and the correction is the most important thing in this ADR. Its first pass concluded all 14 were static analyses or seam-mediated CLI tests. An adversarial review found a counterexample by reading call semantics rather than filenames: tests/execute-phase-worktree-guard.test.cjs calls

  runHook('-c', [guardScript()], { interpreter: 'bash', cwd: dir, … })
  

  and tests/helpers/process-seam.cjs's runHook spawns options.interpreter through a real spawnSync. With interpreter: 'bash' that is a real bash binary executing a shell script extracted from workflow markdown, doing real git plumbing — bash availability, quoting, and git output parsing all differ on Windows. No injected-platform unit test stands in for that.

  The seam argument therefore needs a boundary it did not originally state. "Going through the seam is not a platform signal" is true of src/shell-command-projection.cts, which takes platform as an injected parameter. It is not true of tests/helpers/process-seam.cjs, whose runHook/runGit spawn real binaries. Conflating the two is what made the original process-seam-subprocess detector look purely noisy: it was 99% noise wrapping a real signal.

  On ALWAYS_REAL_OS adding zero today — disclosed, not hidden. The allowlist holds one entry, tests/external-descriptor-confinement.test.cjs, and it currently contributes 0 files, because this PR's own win32 test cases introduced the literal win32 into that file and it now classifies in on content via win32-darwin-literal. A future reader measuring the allowlist's marginal contribution will get zero and may conclude the mechanism is dead. It is not, and the entry stays: the file's real-OS need is a property of the CODE UNDER TEST — isPathConfined reads the ambient path module — not of the test's text, and the text that currently saves it is incidental. Rewrite those cases to use a helper without the literal and the file drops out silently. The pin exists precisely for that, and the tests assert every entry names a file that exists so a stale entry fails loudly rather than rotting.

  The fix is a narrow replacement category rather than restoring the blanket one:

  { name: 'shell-interpreter-spawn',
    test: (c) => /interpreter:\s*['"`](bash|sh|zsh|dash|pwsh|powershell|cmd)['"`]/.test(c) }
  

  Measured 2026-09-11: 33 eligible files match, 9 of them were outside the tier and are added back, taking it from 257 to 266 of 932 (27.6% → 28.5%), which is the committed total. Still under the 33% ceiling. Every one of the 9 was confirmed by reading the matching source line — all are live interpreter: options on real runHook/runHookSeam calls, zero comment or fixture matches. Two narrower alternatives (runGit( alone; non-node spawnSeam() were measured and rejected: each adds 9 files but misses the counterexample entirely, because it spawns through runHook's interpreter option rather than through runGit.

  The lesson is recorded deliberately: an audit that selects candidates by filename inherits exactly the defect this ADR is fixing in the classifier. The 14-file filename sweep was the right first cut and the wrong last word.

  The worked example is tests/windows-robustness.test.cjs, which was on this ADR's own first-draft "must remain in the tier" list because of its filename. It does not spawn anything: it reads other files' source text and asserts on it (assert.match(region, /windowsHide:\s*true/)), and its apparent spawnSync( / execFileSync( occurrences are string literals used as search anchors into those other files. It is fully Linux-runnable and correctly drops out. Selecting it by name would have been the same error the classifier makes — and a test now pins that it drops out, with the reason, so nobody "fixes" it back in.

  The clearest statement of this ADR's thesis is one the repo already wrote. tests/hardcoded-paths.test.cjs, itself a drop-out, opens: "Statically scans source files to catch hardcoded platform-specific paths… Catches issues that previously required a real Windows runner to detect."

• classify() no longer returns a windows_tests key; ci-prepare-test-scope.cjs no longer accepts a windows scope. Both are removed rather than left inert, so a future reader cannot mistake a dead output for a live one.

• ADR-4593's decision is unaffected: MACOS_CATEGORIES is a separate array, chmod-mode-bit and symlink-keyword keep their recorded rationale and their definitions, and the macOS tier is unchanged — the regenerated macos-conformance-tier.generated.cjs is byte-identical to the one on next (git diff reports zero changed lines). Its five prose citations of the 546 figure are left as written: they were accurate on 2026-09-10 and an ADR is a dated record, not a live reference page. ADR-4593 instead carries a short amendment note pointing here, so a reader who arrives at the 546 figure learns it has since moved without the original reasoning being rewritten underneath them.

  Neither tier's size is asserted as a literal count anywhere in the test suite: the ceilings are ratios against a live denominator, and the macOS list is pinned by comparing the committed file to a fresh classification of the live tree. A count hardcoded in a test is a failure scheduled for whenever the suite next grows — which is exactly how the first draft of this work broke.

Risk accepted, and why it is not a rerun of #962

#962 narrowed Windows coverage and was rescinded (#4421) after a macOS-only failure merged green. That failure was root-caused to a rendered-text-length assertion sensitive to tmpdir path length — a violation of ADR-456's typed-surface mandate that nothing enforced. Epic #4589 Phase 1 shipped that enforcement (local/no-rendered-text-length-assert, error from the moment it landed). The specific defect class that made the last narrowing unsafe is now statically prevented, which is the condition #4589 itself named as the precondition for narrowing. That is the difference, and it is why this narrowing rests on an enforced invariant rather than on optimism.

Rejected alternatives

• Gate the lane instead of deleting it. Rejected: provably redundant, shown above. Gating would have produced a lane whose main arm duplicates the conformance lane exactly and whose residual arm is a filename heuristic.
• Keep the lane ungated as deliberate belt-and-braces. Rejected: it does not function as a safety net for the files it duplicates, and for the files it does not duplicate it selects them by filename substring. Paying three Windows runners per PR for that is not a trade-off, it is an accident preserved.
• Narrow hardcoded-path-vs-path-call to same-line proximity rather than deleting it. Rejected: ADR-1703's Linux-runnable rules already enforce the class, so real-OS execution buys nothing.
• Also drop symlink-keyword (measured at the time as 228 rather than 254, before the shell-interpreter-spawn replacement took the tier to its final 266). Rejected: worth 6 unique files, and ADR-4593 reuses it in MACOS_CATEGORIES with recorded rationale.
• Narrow chmod-mode-bit's bare-octal arm. #4641's text named this as a co-driver. Measurement says otherwise: 51 files match only via the bare-octal arm, but for 4 is chmod-mode-bit the sole signal. Changing it would invalidate ADR-4593's measured macOS table for a 4-file benefit. Rejected on evidence; the issue's claim is corrected here.
• An absolute file-count ceiling. Rejected: goes stale under suite growth and stops binding without anyone noticing — the same failure shape as Phase 2's "the list exists" criterion.
• A companion "sole-signal concentration" ceiling — no single category may be the sole signal for more than N% of the tier. Proposed because the ratio ceiling has a real Goodhart weakness: a ratio can be satisfied by inflating the denominator, so adding OS-agnostic tests loosens it without narrowing the tier. Concentration looked like the harder-to-fake companion, since the original defect was precisely one detector carrying half the tier. Measured, and rejected on the numbers. Post-fix the peak sole-signal share is raw-child-process at ~53/266 = ~20%, against the two historic offenders at 21.6% (process-seam-subprocess) and 19.8% (hardcoded-path-vs-path-call). Any threshold above 20% would have missed the original defect; any threshold below it fails today on a category that is entirely legitimate — a test that spawns a real subprocess genuinely needs a real OS. Concentration cannot separate "a big honest category" from "a big dishonest one"; the discriminator is whether the signal is platform-meaningful, which is a judgement no threshold encodes. The ratio ceiling stands alone, with its denominator-inflation weakness disclosed rather than papered over by a second gate that does not actually bind.
• Relaxing raw-child-process to drop its content.includes('child_process') precondition. Investigated and rejected on measurement. The narrowing appeared to unmask a false negative: tests/windows-robustness.test.cjs contains spawnSync( and execFileSync( yet does not match raw-child-process, which looked like the precondition being over-tight (its stated rationale — stopping a local identifier such as spawnResult from matching — is already served by the trailing paren in \bspawnSync\(). Relaxing it was measured to add 13 files, and reading the matching line in each showed all 13 are false positives: comment text, jsdoc prose describing a return shape, template-literal code fixtures fed to an ESLint rule under test, and the search-anchor string literals described above. There is no false negative. The precondition stays exactly as written, and this paragraph exists so the same apparent bug is not "fixed" next time.