* test(#4641): failing-first tests for the tier ceiling and a single Windows selector Tests only, committed ahead of the implementation so the RED run is real. - tests/platform-conformance-tier.test.cjs: tier-size ceiling asserted as a ratio against a live denominator (Windows 33%, macOS 25%); per-helper negative cases proving seam calls and path-call-plus-slash-literal are not platform signals; positive pins that genuine platform content, seam-bypassing spawns, chmod and symlink still classify in; macOS signal set and generated list unchanged. - tests/ci-full-lane-sharding.test.cjs: the test job has zero windows-latest rows and test-conformance still has 3 windows + 1 macOS. - tests/ci-test-scope.test.cjs: windows_tests is absent rather than empty, a non-tier test file no longer forces full_matrix, a RULE-pulled windows-hint test does, and resolveSelection rejects the retired windows scope. Refs #4589, #4591, #4592, #4593, #4603 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#4641): delete the second Windows selector and narrow the conformance tier Epic #4589's goal — the OS-agnostic bulk on Linux, a small explicitly-scoped conformance tier on real Windows/macOS — was not met. Measured on PR #4640 (run 34618834118): 7 non-Linux jobs, a 546/930 (58.7%) "tier", and 5 of 7 changed test files running on a real Windows runner twice. Two selectors, only one in the epic's scope. The test job's three scope:windows shards predate the epic (#494, sharded #3057) and gate on product_changed, not full_matrix, so they fire on every product PR whatever Phase 3's classifier decides. They are deleted; test-conformance becomes the sole Windows selector, as it already was for macOS. Non-Linux jobs 7 -> 4. Gating the lane instead was rejected as provably redundant: for a test file reachesConformanceTierOrSeam is literally CONFORMANCE_TIER_FILES.includes(file), and that same predicate sets full_matrix, which turns test-conformance on. Every file a gated lane would run is already covered in the same run. The lane's one non-redundant residue -- RULE-pulled tests matched by the isWindowsHint filename heuristic -- is ported into reachesConformanceTierOrSeam so it sets full_matrix instead of feeding a parallel lane. Two detectors matched the repo's own test idiom rather than any platform signal and carried 226 of the tier's sole-signal membership against 41 for the other eight: process-seam-subprocess (335 files, 118 unique) matches the tests/helpers.cjs entry points nearly every CLI test uses, and going through the seam is the opposite of a platform signal since shell-command-projection takes platform as an injected parameter; hardcoded-path-vs-path-call (328, 108) needs only a path call anywhere plus a slash literal anywhere, and that class is already enforced by ADR-1703's Linux-runnable ESLint rules. Both are removed. Tier 546 -> 254 (27.3%). src/ reachability is unchanged at 28 files, measured. Adds the size gate Phase 2 never had, as a ratio against a live denominator so it cannot stop binding as the suite grows. 292 files leave real-OS Windows execution. The drop-out set was audited: 14 have a platform-suggestive filename and all 14 are static source-text analyses or seam-mediated CLI tests. raw-child-process was investigated as a suspected false negative and left unchanged -- relaxing it adds 13 files, all false positives. macOS is untouched: MACOS_CATEGORIES is a separate array and the regenerated macos-conformance-tier.generated.cjs is byte-identical at 196 files. Fixes #4641 Refs #4589, #4591, #4592, #4593, #4603 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#4641): register the new ADR path in the docs-guard exempt baseline tests/ci-test-scope.test.cjs references docs/adr/4641-windows-selector-consolidation.md in a comment justifying the retired windows scope; lint-docs-guard-registration tracks that reference set, so the baseline needs the new path. Verified the exemption still holds: the path is prose, not a filesystem read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#4641): make the escalation tier-backed and drop every hardcoded count Three follow-ups from measuring the first pass rather than trusting it. The windows-hint escalation now requires tier membership as well as the filename hint. Setting full_matrix runs test-conformance, which runs only the tier; escalating on a test that is NOT in the tier costs four jobs and still never runs that test on Windows. Measured over the 16 RULES entries the narrowed predicate fires on exactly the same rules today, so this is correct-by-construction rather than a behavior change. The broader variant -- escalate on any tier member a rule pulls in, ignoring the hint -- was measured at 14/16 rules and rejected as over-broad. Removes the hardcoded counts. A hardcoded macOS tier length of 196 broke as soon as the rebase pulled in one new test file from #4253, which is the whole argument against them: the ceilings are ratios against a live denominator, the committed lists are pinned by comparison against a fresh classification of the live tree, and the three named probe files now assert on their SIGNAL rather than on membership in a literal list -- asserting by filename is the exact error this PR fixes in the classifier. Regenerates both lists against the rebased tree. Same-tree figures are now 547 -> 255 of 931 eligible (58.8% -> 27.4%), 292 entries removed and none added; macOS is unchanged at 197 with a zero-line diff. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#4641): restore real-shell-spawn coverage and repair assertions the narrowing broke An isolated adversarial review found a real false negative. Removing the blanket process-seam-subprocess detector also removed the only coverage for tests that spawn a REAL shell: tests/helpers/process-seam.cjs's runHook spawns options.interpreter via real spawnSync, so runHook('-c', [script], { interpreter: 'bash' }) runs a real bash binary executing a shell script extracted from workflow markdown. The seam argument holds for src/shell-command-projection.cts, which takes platform as an injected parameter; it does NOT hold for the test helpers, which spawn real binaries. Conflating the two is what made the blanket detector look purely noisy -- it was 99% noise wrapping a real signal. Adds a narrow shell-interpreter-spawn category keyed on a real interpreter option. Measured 2026-09-11: 33 files match, 9 were outside the tier and are added back, taking it 255 -> 264 of 931 (27.4% -> 28.4%), still under the 33% ceiling. All 9 confirmed by reading the matching source line, zero comment or fixture matches. runGit-alone and non-node-spawnSeam alternatives were measured and rejected -- each adds 9 files but misses the counterexample entirely. Fixes a real bug the suite caught: jobs.test is ubuntu-only now that its scope:windows rows are gone, so it must wire GSD_STRICT_LIVE_CONFIG_GUARD strictly rather than carrying the Windows report-only carve-out. The carve-out now lives solely on test-conformance, whose matrix does include windows. Repairs seven pre-existing assertions the category removal invalidated, preserving each case's purpose rather than deleting coverage, and converts the last hardcoded tier bounds to live-derived ratios -- including the macOS sanity range that was still a magic [100, 350]. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#4641): keep the confinement test on a real OS via a documented allowlist A security review found tests/external-descriptor-confinement.test.cjs had dropped out of the Windows tier. It must stay in, and no content signal can express why: it exercises isPathConfined (src/external-descriptor-trust.cts), which uses the AMBIENT path module -- path.resolve(root, target) and path.sep -- with no injection. Its win32 semantics (drive letters, UNC, separator) are only reachable by actually running on Windows, and it is a security-relevant write-confinement gate. A content classifier cannot see 'this module reads the ambient path module', so no regex belongs here. Adds ALWAYS_REAL_OS, a Map of path -> recorded reason, unioned into the Windows tier only. A Map rather than a list so an entry without a reason is impossible by construction, and tests assert every entry names a file that exists on disk so a stale entry fails loudly instead of rotting. This is the centrally- enumerated single source of truth epic #4589 Phase 2 asked for and ADR-1703's portability-vocab.cjs already models -- deliberately not a heuristic. Windows tier 264 -> 265 of 931 (28.5%), still under the 33% ceiling. macOS is untouched and byte-identical: the win32 concern does not apply to a POSIX runner, and a test asserts the allowlist does not leak into that tier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#4641): inject the path impl into isPathConfined and correct the ADR count Two review findings, both fixed rather than dispositioned. A security review found tests/external-descriptor-confinement.test.cjs had left real-OS execution. The allowlist pinned it back, but that only restored INCIDENTAL coverage: isPathConfined used the ambient path module, and its test carried POSIX-only literals, so a win32 confinement escape was unverified on every platform including Windows. isPathConfined now takes an optional third parameter carrying the path implementation, defaulting to the ambient module. Blast radius is CRITICAL -- 53 affected symbols across 19 files -- so the change is purely additive and every existing two-argument caller is byte-identical. Tests now inject path.win32 and path.posix, covering a different drive letter, a cross-drive absolute, backslash and forward-slash traversal, UNC, and the startsWith prefix-boundary bug (.gsdEVIL against root .gsd) on both separators. Proved load-bearing: dropping the + p.sep from the prefix check fails exactly the two boundary cases and nothing else. Callers' suites 149/149. The spec review caught an off-by-one: the ADR narrated a 264-file tier while the committed list holds 265. The ADR now records the full chain 547 -> 255 -> 264 -> 265 (28.5%). Also corrects a stale comment in scripts/docs-guard-registry.cjs that narrated classify() as zeroing windows_tests, a key this change removes -- kept as historical narration but labelled as such. Refs #4641 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#131): make the unwritable-HOME test actually test something Found by sweeping for the root-bypass class after fixing commit-files-deletion. This one is the silent variant, and it was broken twice over. First, the condition: the test made a fake HOME unwritable with chmod 0o500. The gsd-test Docker bench runs as root, root bypasses mode bits, so HOME stayed writable and the hostile condition never existed. Replaced with a HOME whose PARENT is a regular file, so every write under it fails ENOTDIR at the VFS layer for every uid -- no permission check is involved at all. Second, and more fundamental: the probe was npm --version, which on npm 11.19.0 performs zero filesystem I/O against HOME. Proven rather than assumed -- neutralizing runNpm()'s isolation turned the sibling test red while this one stayed green, so its assertion could never detect the regression it guards, on any uid, with or without the condition fix. npm config get cache was tried next and proved vacuous the same way (it only string-resolves the path). The probe is now npm cache verify, which really does mkdir _cacache under HOME. Re-proved load-bearing after the change: with isolation neutralized the test now fails with ENOTDIR on <blocker>/home/.npm/_cacache. tests/helpers.cjs was restored and verified diff-clean; suite 13/13. Refs #4641 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#4641): correct the net drop-out figure in ADR-4641 The Consequences section still said 292 files leave real-OS Windows execution. That was the count before the narrow shell-interpreter-spawn replacement restored 9 and ALWAYS_REAL_OS pinned 1. Net is 282. Also names both real-binary categories rather than only raw-child-process, and clarifies that the 14-file filename audit was against the 292 initially dropped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#4641): record the rejected concentration ceiling and its measurement Applying Goodhart's own question to the new ceiling -- how would you make this metric look good without improving what it represents -- surfaces a real weakness: a ratio can be satisfied by inflating the denominator, so adding OS-agnostic tests loosens it without narrowing the tier. The obvious companion gate was a sole-signal concentration ceiling, since the original defect was one detector carrying half the tier. Measured and rejected: peak concentration post-fix is raw-child-process at 53/265 = 20.0%, against the historic offenders at 21.6% and 19.8%. Any threshold above 20% misses the original defect; any threshold below it fails on a legitimate category. The discriminator is whether a signal is platform-meaningful, which no threshold encodes. Weakness disclosed rather than covered by a gate that does not bind. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(#4641): add the changeset fragment for the confinement-check change changeset-lint failed on PR #4643: the PR touches user-facing paths and carried no fragment. The earlier no-changeset call matched #4604's CI-only precedent and was correct then; it was not revisited once the PR grew a src/ change, which is my miss. The fragment describes the real user-visible improvement: the external-descriptor write-confinement check's Windows semantics are now verified deterministically rather than only when the suite happened to run on Windows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#4641): correct the tier count in TESTING-SUITES.md Said the tier narrowed from 546 to 254. The final committed list is 265 of 931 eligible (58.8% -> 28.5%) after the shell-interpreter-spawn replacement restored 9 files and ALWAYS_REAL_OS pinned 1. Same error class the spec review caught in the ADR, in a live reference page rather than a dated record, so it states the current truth rather than carrying an amendment note. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#4641): record the measured aggregate from real CI job lists Epic #4589's closeout asserted its reduction from a static count; #4641's acceptance criterion asks for a figure read off a real run. Recorded here: test.yml job count 21 -> 15 and non-Linux 7 -> 4, comparing PR #4640's run against this PR's own. Against the true pre-epic baseline of 9, that is 9 -> 4. Also states the caveat that a PR's total CHECK count is not a clean before/after comparison, since many gates are path-scoped and this change touches a broader path set -- the like-for-like figure is the test.yml job count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#4641): compare job totals the same way on both sides The measured-aggregate table put #4640's COMPLETED run total (21) against this run's count at matrix-expansion time (15). Those are not the same measurement: the completed total includes the post-test Coverage gate and baseline-publisher jobs. Counted identically, it is 21 -> 17. The load-bearing figure, non-Linux jobs 7 -> 4, was correct and is unchanged. Called out in the table rather than silently corrected -- comparing two differently-derived numbers is exactly the error class this ADR is about. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#4641): record measured conformance wall-clock and date the stale counterfactual Adds the per-job durations from both runs. The honest read is that this is a correctness win more than a speed one: file count fell 52% but wall-clock only 9-29%, because what was removed were the cheap static tests and what remains is concentrated in expensive spawn-heavy work. Stated explicitly so nobody expects a future narrowing to buy time proportional to file count. The load-bearing figure is windows shard 3/3: 40m24s against a 45-minute cap on the 547-file tier -- 90% of the cliff #869 and #3057 were both filed about -- pulled back to 31m27s. macOS moved the wrong way (17m48s -> 21m02s) while its tier was UNCHANGED at 197 files, which fixes that as runner variance and is noted as a caution against reading a single duration as signal. Also dates the symlink-keyword counterfactual, which cited a 254-file tier from before the replacement category and allowlist took it to its final 265. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#4641): re-measure against the rebased tree and disclose the allowlist's zero next gained #4644 mid-flight, so every absolute count shifted. Re-measured on the tree this actually ships against (932 eligible): 548 -> 257 by detector removal, 257 -> 266 once shell-interpreter-spawn restores 9. Net 282 removed, 9 restored. macOS 198, unchanged by this PR. The percentages did not move across three rebases (58.8% -> 28.5%), which is the whole argument for expressing the ceilings as ratios rather than counts -- noted in the ADR since it is now evidence rather than assertion. Also discloses that ALWAYS_REAL_OS now contributes ZERO files: this PR's own win32 test cases introduced the literal win32 into the pinned file, so it classifies in on content via win32-darwin-literal. The entry stays and the reason is written down, because the file's real-OS need is a property of the code under test (isPathConfined reads the ambient path module), not of the test's text -- the text that currently saves it is incidental and could be refactored away silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: sim <sim@local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
@@ -73,14 +73,65 @@ function isCompleteShardSet(specs) {
|
||||
return seen.size === total && [...seen].every((i) => i >= 1 && i <= total);
|
||||
}
|
||||
|
||||
test('the test job has no windows lane; test-conformance is the sole Windows selector (#4641)', () => {
|
||||
const workflow = loadWorkflow('test.yml');
|
||||
const testInclude = workflow.jobs.test.strategy.matrix.include;
|
||||
const testWindowsEntries = testInclude.filter((e) => e.os === 'windows-latest');
|
||||
assert.equal(
|
||||
testWindowsEntries.length, 0,
|
||||
`the \`test\` job's matrix.include still has ${testWindowsEntries.length} windows-latest `
|
||||
+ `entr${testWindowsEntries.length === 1 ? 'y' : 'ies'} (${JSON.stringify(testWindowsEntries)}). `
|
||||
+ '#4641 deletes the `test` job\'s windows lane; test-conformance is the sole Windows selector.',
|
||||
);
|
||||
|
||||
const conformanceInclude = workflow.jobs['test-conformance'].strategy.matrix.include;
|
||||
const conformanceWindowsEntries = conformanceInclude.filter((e) => e.os === 'windows-latest');
|
||||
const conformanceMacosEntries = conformanceInclude.filter((e) => e.os === 'macos-latest');
|
||||
|
||||
// Derived, not pinned: windows-latest must be a non-empty, COMPLETE shard
|
||||
// set (same denominator, numerators 1..N — see isCompleteShardSet above),
|
||||
// whatever N the workflow currently declares. A bare `.length === 3` here
|
||||
// would break the moment the workflow is rebalanced to a different shard
|
||||
// count without a wiring regression, exactly the shape of bug this sweep
|
||||
// exists to remove (the sibling macOS-tier-count magic number already did
|
||||
// this once).
|
||||
assert.ok(
|
||||
conformanceWindowsEntries.length > 0,
|
||||
`test-conformance declares no windows-latest entries: ${JSON.stringify(conformanceInclude)}`,
|
||||
);
|
||||
assert.ok(
|
||||
isCompleteShardSet(conformanceWindowsEntries.map((e) => e.shard)),
|
||||
'the windows-latest entries in test-conformance are not a complete shard set: '
|
||||
+ JSON.stringify(conformanceWindowsEntries),
|
||||
);
|
||||
|
||||
// macos-latest is, by design, a single UNSHARDED entry (see the workflow's
|
||||
// "macos-latest stays unsharded; it has real headroom" comment above the
|
||||
// `test-conformance` job) — unlike the windows/full shard counts, this "1"
|
||||
// is not a fact about the live tree that grows with the suite, it is the
|
||||
// structural claim the test exists to pin: more than one entry here would
|
||||
// silently duplicate full macOS runs, and a `shard` key would mean the
|
||||
// workflow started partitioning a lane the run-tests.cjs invocation below
|
||||
// does not expect to be partitioned.
|
||||
assert.equal(
|
||||
conformanceMacosEntries.length, 1,
|
||||
`expected a single unsharded macos-latest entry in test-conformance, got ${conformanceMacosEntries.length}: `
|
||||
+ JSON.stringify(conformanceMacosEntries),
|
||||
);
|
||||
assert.equal(
|
||||
conformanceMacosEntries[0] && conformanceMacosEntries[0].shard, undefined,
|
||||
'the macos-latest entry in test-conformance declares a shard, contradicting the '
|
||||
+ `workflow's "macos-latest stays unsharded" design: ${JSON.stringify(conformanceMacosEntries)}`,
|
||||
);
|
||||
});
|
||||
|
||||
test('the full test lane is sharded and complete (#2952)', async (t) => {
|
||||
const workflow = loadWorkflow('test.yml');
|
||||
const include = workflow.jobs.test.strategy.matrix.include;
|
||||
const fullLanes = include.filter((e) => e.scope === 'full');
|
||||
const windowsLanes = include.filter((e) => e.scope === 'windows');
|
||||
// The only lane in this job with no shard is `scope: targeted` — the fast,
|
||||
// single-runner default lane. Both `full` and `windows` are sharded.
|
||||
const shardedScopes = { full: fullLanes, windows: windowsLanes };
|
||||
// #4641: the `test` job's `scope: windows` lane is deleted — there is no
|
||||
// longer a second sharded scope in this job to pin alongside `full`.
|
||||
const shardedScopes = { full: fullLanes };
|
||||
|
||||
for (const [scope, lanes] of Object.entries(shardedScopes)) {
|
||||
await t.test(`the \`scope: ${scope}\` lane is actually sharded, not a single runner`, () => {
|
||||
@@ -112,12 +163,13 @@ test('the full test lane is sharded and complete (#2952)', async (t) => {
|
||||
}
|
||||
|
||||
await t.test('no other lane is sharded', () => {
|
||||
for (const lane of include.filter((e) => e.scope !== 'full' && e.scope !== 'windows')) {
|
||||
// #4641: the `test` job's `scope: windows` lane is deleted, so `full` is
|
||||
// the only sharded scope left in this job's matrix.
|
||||
for (const lane of include.filter((e) => e.scope !== 'full')) {
|
||||
assert.equal(
|
||||
lane.shard, undefined,
|
||||
`lane ${JSON.stringify(lane)} declares a shard but is neither \`scope: full\` `
|
||||
+ 'nor `scope: windows` — the targeted lane runs a selected file list, not '
|
||||
+ 'a partition.',
|
||||
`lane ${JSON.stringify(lane)} declares a shard but is not \`scope: full\` — `
|
||||
+ 'the targeted lane runs a selected file list, not a partition.',
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -56,7 +56,11 @@ describe('ci-test-scope.cjs', () => {
|
||||
assert.strictEqual(result.full_matrix, true);
|
||||
assert.ok(result.targeted_tests.includes('tests/workflow-shell-pinning.test.cjs'));
|
||||
assert.ok(result.targeted_tests.includes('tests/release-tarball-smoke-workflow.test.cjs'));
|
||||
assert.ok(result.windows_tests.includes('tests/workflow-shell-pinning.test.cjs'));
|
||||
// #4641: the `test` job's windows lane is deleted; full_matrix (already
|
||||
// asserted true above) is the only Windows signal left, and classify()
|
||||
// emits no windows_tests key at all.
|
||||
assert.strictEqual(Object.hasOwn(result, 'windows_tests'), false,
|
||||
`expected no windows_tests property, got keys: ${JSON.stringify(Object.keys(result))}`);
|
||||
});
|
||||
|
||||
test('pipeline workflow (install-smoke.yml) — product_changed true, full_matrix true', () => {
|
||||
@@ -307,18 +311,21 @@ describe('ci-test-scope superset invariant (#494, rescinded by #4421)', () => {
|
||||
`expected full_matrix=true for a tests/**-only change (rescinded #494 carve-out, see #4421), got: ${JSON.stringify(result)}`);
|
||||
assert.ok(result.targeted_tests.includes('tests/perf-317-context-monitor-fs.test.cjs'),
|
||||
`expected the changed test in targeted_tests, got: ${JSON.stringify(result.targeted_tests)}`);
|
||||
assert.ok(result.windows_tests.includes('tests/perf-317-context-monitor-fs.test.cjs'),
|
||||
`expected the changed test in windows_tests, got: ${JSON.stringify(result.windows_tests)}`);
|
||||
// #4641: the `test` job's windows lane is deleted; full_matrix (already
|
||||
// asserted true above) is the only Windows signal left.
|
||||
assert.strictEqual(Object.hasOwn(result, 'windows_tests'), false,
|
||||
`expected no windows_tests property, got keys: ${JSON.stringify(Object.keys(result))}`);
|
||||
});
|
||||
|
||||
test('A2: a changed test file with no windows hint still joins the windows lane and triggers full_matrix', () => {
|
||||
// commands.test.cjs matches none of the WINDOWS_HINTS substrings — the
|
||||
// unconditional changed-test → windows lane rule must include it anyway.
|
||||
test('A2: a changed test file with no windows hint still triggers full_matrix, with no side lane (#4641)', () => {
|
||||
// commands.test.cjs matches none of the WINDOWS_HINTS substrings — under
|
||||
// the old #494/#4421 behavior it still joined the windows lane. Post-#4641
|
||||
// there is no windows lane to join; full_matrix is the sole signal.
|
||||
const result = scopeFor(['tests/commands.test.cjs']);
|
||||
assert.strictEqual(result.full_matrix, true,
|
||||
`expected full_matrix=true (rescinded #494 carve-out, see #4421), got: ${JSON.stringify(result)}`);
|
||||
assert.ok(result.windows_tests.includes('tests/commands.test.cjs'),
|
||||
`expected hint-less changed test in windows_tests, got: ${JSON.stringify(result.windows_tests)}`);
|
||||
assert.strictEqual(Object.hasOwn(result, 'windows_tests'), false,
|
||||
`expected no windows_tests property, got keys: ${JSON.stringify(Object.keys(result))}`);
|
||||
});
|
||||
|
||||
test('A3: a deleted/nonexistent test path falls back to the unit token; full_matrix now depends on conformance-tier reachability (#4592)', () => {
|
||||
@@ -1029,6 +1036,66 @@ describe('#4592 reachability-based full_matrix classifier', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('#4641 windows lane removal: test-conformance becomes the sole Windows selector', () => {
|
||||
const { classify } = require('../scripts/ci-test-scope.cjs');
|
||||
|
||||
// Case C: a changed test file that is not tagged in Phase 2's
|
||||
// CONFORMANCE_TIER_FILES no longer needs to force a Windows run of its own —
|
||||
// once the `test` job's `scope: windows` lane is deleted, the only Windows
|
||||
// signal left is full_matrix (routed to test-conformance).
|
||||
test('a non-tier test file no longer forces a Windows run (#4641)', () => {
|
||||
const { CONFORMANCE_TIER_FILES } = require('../scripts/lib/platform-conformance-tier.generated.cjs');
|
||||
const file = 'tests/some-brand-new-non-tier-4641.test.cjs';
|
||||
assert.ok(!CONFORMANCE_TIER_FILES.includes(file), 'precondition: fixture path must not be tier-tagged');
|
||||
const result = classify([file]);
|
||||
assert.strictEqual(result.full_matrix, false,
|
||||
`expected full_matrix=false for a non-conformance-tier test file, got: ${JSON.stringify(result)}`);
|
||||
});
|
||||
|
||||
// Case D: post-#4641, classify() must not emit a `windows_tests` key at all —
|
||||
// "absent" and "empty array" are different claims, and only the former
|
||||
// matches a workflow with no windows_tests output to consume.
|
||||
test('windows_tests is gone, not merely empty (#4641)', () => {
|
||||
const result = classify(['tests/some-brand-new-non-tier-4641.test.cjs']);
|
||||
assert.strictEqual(
|
||||
Object.hasOwn(result, 'windows_tests'), false,
|
||||
`expected classify() to return no windows_tests property at all, got keys: ${JSON.stringify(Object.keys(result))}`,
|
||||
);
|
||||
});
|
||||
|
||||
// Case E: the one non-redundant residue of the deleted windows lane — a
|
||||
// RULE whose tests[] includes a filename matching isWindowsHint — must be
|
||||
// preserved by escalating to full_matrix instead of a side lane.
|
||||
// 'portability lint rules (ADR-1703)' pulls in
|
||||
// tests/no-path-literal-in-assert.rule.test.cjs and
|
||||
// tests/normalize-path-in-content.rule.test.cjs, both matching the 'path'
|
||||
// hint in WINDOWS_HINTS, and today carries no fullMatrix of its own.
|
||||
test('RULE-pulled windows-hint tests force full_matrix, not a side lane (#4641)', () => {
|
||||
const file = 'eslint-rules/no-path-literal-in-assert.cjs';
|
||||
const result = classify([file]);
|
||||
assert.strictEqual(result.full_matrix, true,
|
||||
`expected full_matrix=true because the matched rule pulls in a windows-hint test, got: ${JSON.stringify(result)}`);
|
||||
assert.ok(
|
||||
result.reasons.some(r => r.startsWith(`${file}: portability lint rules (ADR-1703)`)),
|
||||
`expected reasons to name the windows-hint rule, got: ${JSON.stringify(result.reasons)}`,
|
||||
);
|
||||
});
|
||||
|
||||
// Case F: MUST-PASS pin, already covered by the "full_matrix true for a
|
||||
// conformance-tier test file" test in the '#4592 reachability-based
|
||||
// full_matrix classifier' describe block above — a changed test file that
|
||||
// IS in CONFORMANCE_TIER_FILES still yields full_matrix=true. Restated here
|
||||
// so the #4641 removal's regression surface is pinned in one place too.
|
||||
test('a conformance-tier test file still yields full_matrix=true (#4641 pin)', () => {
|
||||
const { CONFORMANCE_TIER_FILES } = require('../scripts/lib/platform-conformance-tier.generated.cjs');
|
||||
assert.ok(CONFORMANCE_TIER_FILES.length > 0, 'precondition: committed tier list must be non-empty');
|
||||
const file = CONFORMANCE_TIER_FILES[0];
|
||||
const result = classify([file]);
|
||||
assert.strictEqual(result.full_matrix, true,
|
||||
`expected full_matrix=true for conformance-tier file ${file}, got: ${JSON.stringify(result)}`);
|
||||
});
|
||||
});
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Folded from tests/bug-641-files-from-suite-token.test.cjs — consolidation epic #1969 (B6 #1975)
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
@@ -1250,7 +1317,7 @@ describe('bug #1329 — ci-prepare-test-scope fallback never emits a deleted fil
|
||||
fs.writeFileSync(path.join(tmpDir, f), PASS_BODY, 'utf8');
|
||||
}
|
||||
|
||||
const lines = resolveSelection({ scope: 'targeted', targeted: '', windows: '', root: tmpDir });
|
||||
const lines = resolveSelection({ scope: 'targeted', targeted: '', root: tmpDir });
|
||||
|
||||
assert.ok(!lines.includes(absent), `absent file "${absent}" must be filtered out, got: ${lines.join(', ')}`);
|
||||
for (const f of present) {
|
||||
@@ -1260,17 +1327,30 @@ describe('bug #1329 — ci-prepare-test-scope fallback never emits a deleted fil
|
||||
|
||||
test('empty detection with no surviving fallback files falls back to the unit sentinel', () => {
|
||||
// tmpDir/tests exists but contains none of the FALLBACK files.
|
||||
const lines = resolveSelection({ scope: 'windows', targeted: '', windows: '', root: tmpDir });
|
||||
// #4641: this used to run under scope: 'windows'; that scope was retired
|
||||
// with the windows CI lane. Re-pointed at 'targeted' (still empty-detected)
|
||||
// to keep the fallback-sentinel contract covered.
|
||||
const lines = resolveSelection({ scope: 'targeted', targeted: '', root: tmpDir });
|
||||
assert.deepStrictEqual(lines, [FALLBACK_SENTINEL]);
|
||||
});
|
||||
|
||||
test('resolveSelection rejects the retired windows scope (#4641)', () => {
|
||||
// #4641: the `test` job's `scope: windows` lane was deleted (see
|
||||
// docs/adr/4641-windows-selector-consolidation.md). Do not restore this
|
||||
// scope — resolveSelection must now fail loudly for it rather than
|
||||
// silently falling back.
|
||||
assert.throws(
|
||||
() => resolveSelection({ scope: 'windows', targeted: '', root: tmpDir }),
|
||||
/Unknown test scope: windows/,
|
||||
);
|
||||
});
|
||||
|
||||
test('detected list passes through verbatim — files and suite sentinels preserved, not existence-filtered', () => {
|
||||
// The detected list is already filtered by affected-tests-lib and may carry
|
||||
// a suite sentinel; ci-prepare-test-scope must not touch it.
|
||||
const lines = resolveSelection({
|
||||
scope: 'targeted',
|
||||
targeted: 'tests/does-not-exist.test.cjs unit',
|
||||
windows: '',
|
||||
root: tmpDir,
|
||||
});
|
||||
assert.deepStrictEqual(lines, ['tests/does-not-exist.test.cjs', 'unit']);
|
||||
@@ -1288,7 +1368,7 @@ describe('bug #1329 — ci-prepare-test-scope fallback never emits a deleted fil
|
||||
[path.join(REPO_ROOT, 'scripts', 'ci-prepare-test-scope.cjs')],
|
||||
{
|
||||
cwd: tmpDir,
|
||||
env: { ...process.env, TEST_SCOPE: 'targeted', TARGETED_TESTS: '', WINDOWS_TESTS: '' },
|
||||
env: { ...process.env, TEST_SCOPE: 'targeted', TARGETED_TESTS: '' },
|
||||
timeoutMs: PROBE_TIMEOUT_MS,
|
||||
},
|
||||
);
|
||||
|
||||
@@ -25,6 +25,53 @@ test('isPathConfined: confined paths are true, escapes are false', () => {
|
||||
assert.ok(!isPathConfined('skills', ''), 'empty root is NOT confined');
|
||||
});
|
||||
|
||||
test('isPathConfined: win32 injection — confined and escape cases', () => {
|
||||
const win32 = path.win32;
|
||||
const root = 'C:\\Users\\me\\.gsd';
|
||||
assert.ok(
|
||||
isPathConfined('sub\\file.md', root, { pathImpl: win32 }),
|
||||
'win32 target under root is confined',
|
||||
);
|
||||
assert.ok(
|
||||
!isPathConfined('D:\\evil', root, { pathImpl: win32 }),
|
||||
'a different drive letter is NOT confined',
|
||||
);
|
||||
assert.ok(
|
||||
!isPathConfined('C:\\Windows\\system32', root, { pathImpl: win32 }),
|
||||
'an absolute path on the same drive outside root is NOT confined',
|
||||
);
|
||||
assert.ok(
|
||||
!isPathConfined('..\\..\\evil', root, { pathImpl: win32 }),
|
||||
'a backslash traversal is NOT confined',
|
||||
);
|
||||
assert.ok(
|
||||
!isPathConfined('\\\\server\\share\\x', root, { pathImpl: win32 }),
|
||||
'a UNC path is NOT confined',
|
||||
);
|
||||
assert.ok(
|
||||
!isPathConfined('../../x', root, { pathImpl: win32 }),
|
||||
'a forward-slash traversal is NOT confined (win32 accepts / too)',
|
||||
);
|
||||
});
|
||||
|
||||
test('isPathConfined: win32 prefix-boundary — sibling with root as a string prefix is refused', () => {
|
||||
const win32 = path.win32;
|
||||
const root = 'C:\\Users\\me\\.gsd';
|
||||
assert.ok(
|
||||
!isPathConfined('..\\.gsdEVIL', root, { pathImpl: win32 }),
|
||||
'C:\\Users\\me\\.gsdEVIL must be refused despite sharing the "C:\\Users\\me\\.gsd" string prefix',
|
||||
);
|
||||
});
|
||||
|
||||
test('isPathConfined: posix prefix-boundary — sibling with root as a string prefix is refused', () => {
|
||||
const posix = path.posix;
|
||||
const root = '/home/me/.gsd';
|
||||
assert.ok(
|
||||
!isPathConfined('../.gsdEVIL', root, { pathImpl: posix }),
|
||||
'/home/me/.gsdEVIL must be refused despite sharing the "/home/me/.gsd" string prefix',
|
||||
);
|
||||
});
|
||||
|
||||
test('assertDescriptorConfined: a benign descriptor (all destSubpaths under configHome) passes', () => {
|
||||
const desc = {
|
||||
id: 'community-host',
|
||||
|
||||
@@ -28,6 +28,10 @@ const {
|
||||
NOISY_FOR_SOURCE_REACHABILITY,
|
||||
classifyMacosContent,
|
||||
classifyMacosTree,
|
||||
CATEGORIES,
|
||||
MACOS_CATEGORIES,
|
||||
walkTestFiles,
|
||||
ALWAYS_REAL_OS,
|
||||
} = require('../scripts/gen-platform-conformance-tier.cjs');
|
||||
|
||||
const ROOT = path.resolve(__dirname, '..');
|
||||
@@ -35,6 +39,21 @@ const SCRIPT = path.join(ROOT, 'scripts', 'gen-platform-conformance-tier.cjs');
|
||||
const GENERATED_PATH = path.join(ROOT, 'scripts', 'lib', 'platform-conformance-tier.generated.cjs');
|
||||
const MACOS_GENERATED_PATH = path.join(ROOT, 'scripts', 'lib', 'macos-conformance-tier.generated.cjs');
|
||||
|
||||
// Policy ceilings, not derived facts — a bound like this has to be a number
|
||||
// somewhere, so it is hoisted here once (module scope, shared by every case
|
||||
// below that needs it) rather than left as a bare literal inside an
|
||||
// assertion. Measured at authoring time (2026-09-11): the Windows tier sat at
|
||||
// 264/931 eligible unit-suite files (~28.4%), the macOS tier at 197/931
|
||||
// (~21.2%). Each ceiling below leaves headroom over that measurement — enough
|
||||
// to absorb ordinary suite growth (new test files that happen to touch a real
|
||||
// platform signal) without going so loose that a regression toward
|
||||
// re-matching a removed house idiom (process-seam calls, path-call-plus-
|
||||
// slash-literal) would slip back under the ceiling undetected. If the
|
||||
// measured ratio moves, update the ratio in this comment and re-justify the
|
||||
// ceiling — do not just raise the number to make a red test green.
|
||||
const WINDOWS_TIER_RATIO_CEILING = 0.33;
|
||||
const MACOS_TIER_RATIO_CEILING = 0.25;
|
||||
|
||||
// ─── Rows 1-15: classifyContent, pure fixtures ────────────────────────────────
|
||||
|
||||
describe('classifyContent — happy-path signals', () => {
|
||||
@@ -88,16 +107,25 @@ describe('classifyContent — happy-path signals', () => {
|
||||
}
|
||||
});
|
||||
|
||||
test('flags process-seam subprocess helpers', () => {
|
||||
// Replaces the former 'flags process-seam subprocess helpers' case: that
|
||||
// category ('process-seam-subprocess') was removed outright from CATEGORIES
|
||||
// (#4641 — it matched the house test idiom of calling the process seam at
|
||||
// all, not a genuine platform signal). Its narrower, evidence-backed
|
||||
// replacement is 'shell-interpreter-spawn', which keys on a REAL shell
|
||||
// binary name passed as the process-seam helpers' `interpreter` option
|
||||
// (tests/helpers/process-seam.cjs's `runHook`/`runHookSeam`) — genuinely
|
||||
// platform-dependent (bash/zsh/cmd availability, quoting, output parsing
|
||||
// all differ across OSes), unlike the removed category's over-broad "any
|
||||
// process-seam call" signal.
|
||||
test('flags shell-interpreter-spawn (real interpreter option on a process-seam helper)', () => {
|
||||
for (const fixture of [
|
||||
"runNode(['--check'])",
|
||||
"runGit(['status'])",
|
||||
"runHook(HOOK_PATH, [])",
|
||||
"runGsdTools(['state', 'show'])",
|
||||
"runHook(HOOK_PATH, [], { interpreter: 'bash' })",
|
||||
"runHookSeam(HOOK_PATH, [], { interpreter: 'zsh' })",
|
||||
"runHook(HOOK_PATH, [], { interpreter: 'cmd' })",
|
||||
]) {
|
||||
const { needsRealOs, signals } = classifyContent(fixture);
|
||||
assert.equal(needsRealOs, true, fixture);
|
||||
assert.ok(signals.includes('process-seam-subprocess'), fixture);
|
||||
assert.ok(signals.includes('shell-interpreter-spawn'), fixture);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -114,12 +142,18 @@ describe('classifyContent — happy-path signals', () => {
|
||||
assert.ok(signals.includes('symlink-keyword'));
|
||||
});
|
||||
|
||||
test('flags hardcoded path literal vs path.* call', () => {
|
||||
const fixture = "const p = path.join(root, 'x');\nassert.equal(rendered, '/etc/passwd');";
|
||||
const { needsRealOs, signals } = classifyContent(fixture);
|
||||
assert.equal(needsRealOs, true);
|
||||
assert.ok(signals.includes('hardcoded-path-vs-path-call'));
|
||||
});
|
||||
// The former 'flags hardcoded path literal vs path.* call' case asserted
|
||||
// the 'hardcoded-path-vs-path-call' category, which #4641 removed outright
|
||||
// from CATEGORIES (measured to be, alongside process-seam-subprocess, the
|
||||
// largest driver of Windows-tier over-inclusion — a universal Node
|
||||
// test-suite idiom, not a platform signal). Unlike process-seam-subprocess
|
||||
// above, this category has no narrower evidence-backed replacement: no
|
||||
// still-existing CATEGORIES entry keys on "a hardcoded path literal
|
||||
// alongside a path.* call". Every other CATEGORIES entry already has
|
||||
// dedicated happy-path coverage elsewhere in this describe block, so there
|
||||
// is genuinely nothing left for a rewritten fixture here to assert; the
|
||||
// case is retired rather than kept as dead weight around a deleted
|
||||
// detector.
|
||||
});
|
||||
|
||||
describe('classifyContent — negative / hostile inputs', () => {
|
||||
@@ -154,13 +188,23 @@ describe('classifyContent — negative / hostile inputs', () => {
|
||||
// ─── Row 15 (#4592): NOISY_FOR_SOURCE_REACHABILITY drift guard ────────────────
|
||||
|
||||
describe('NOISY_FOR_SOURCE_REACHABILITY (#4592)', () => {
|
||||
test('NOISY_FOR_SOURCE_REACHABILITY exports exactly the two noisy categories', () => {
|
||||
// #4641 removed 'hardcoded-path-vs-path-call' from CATEGORIES outright (it
|
||||
// was the single largest driver of Windows-tier over-inclusion, a house
|
||||
// test idiom rather than a genuine platform signal) — not merely from this
|
||||
// exemption set. NOISY_FOR_SOURCE_REACHABILITY therefore now holds exactly
|
||||
// one member, 'symlink-keyword': still precise enough for test-file
|
||||
// classification but too noisy for source reachability.
|
||||
test('NOISY_FOR_SOURCE_REACHABILITY exports exactly the one remaining noisy category', () => {
|
||||
assert.ok(NOISY_FOR_SOURCE_REACHABILITY instanceof Set,
|
||||
`expected a Set, got: ${typeof NOISY_FOR_SOURCE_REACHABILITY}`);
|
||||
assert.deepEqual(
|
||||
[...NOISY_FOR_SOURCE_REACHABILITY].sort(),
|
||||
['hardcoded-path-vs-path-call', 'symlink-keyword'].sort(),
|
||||
`expected exactly the two named noisy categories, got: ${JSON.stringify([...NOISY_FOR_SOURCE_REACHABILITY])}`,
|
||||
['symlink-keyword'],
|
||||
`expected exactly the one remaining noisy category, got: ${JSON.stringify([...NOISY_FOR_SOURCE_REACHABILITY])}`,
|
||||
);
|
||||
assert.ok(
|
||||
!CATEGORIES.map((c) => c.name).includes('hardcoded-path-vs-path-call'),
|
||||
'hardcoded-path-vs-path-call was removed from CATEGORIES outright (#4641), not merely exempted here',
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -283,13 +327,19 @@ describe('gen-platform-conformance-tier.cjs — real repo tree (regression)', ()
|
||||
fresh = classifyTree(realTestsDir);
|
||||
}, 'a full sweep of the real tests/ tree must complete without throwing');
|
||||
|
||||
// Measured 546/952 at authoring time (#4591, post suite-exclusion fix) —
|
||||
// the range below is a sanity ballpark with headroom for organic
|
||||
// test-suite growth in either direction, not a brittle exact-match on
|
||||
// that literal.
|
||||
// Post-#4641 the tier is a ratio-bounded MINORITY of eligible unit-suite
|
||||
// files (WINDOWS_TIER_RATIO_CEILING, same policy ceiling the #4641 block
|
||||
// below enforces), not a brittle absolute-count range against a literal
|
||||
// that goes stale every time the category set or the suite's file count
|
||||
// changes (a hardcoded exact-count sanity range already broke once in
|
||||
// this PR). Derived from the live tree, not a hardcoded number.
|
||||
const { suiteOf } = require('../scripts/lib/suite-detection.cjs');
|
||||
const eligibleCount = walkTestFiles(realTestsDir).filter((absPath) => suiteOf(absPath) === null).length;
|
||||
const ratio = fresh.files.length / eligibleCount;
|
||||
assert.ok(
|
||||
fresh.files.length >= 450 && fresh.files.length <= 700,
|
||||
`expected a real, current, sanity-checked count in [450, 700], got ${fresh.files.length}`,
|
||||
ratio > 0 && ratio <= WINDOWS_TIER_RATIO_CEILING,
|
||||
`expected a nonzero conformance tier within the #4641 ratio ceiling (<=${WINDOWS_TIER_RATIO_CEILING * 100}%), ` +
|
||||
`got ${fresh.files.length}/${eligibleCount} (${(ratio * 100).toFixed(1)}%)`,
|
||||
);
|
||||
|
||||
delete require.cache[require.resolve(GENERATED_PATH)];
|
||||
@@ -474,11 +524,19 @@ describe('gen-platform-conformance-tier.cjs — real repo tree, macOS target (re
|
||||
fresh = classifyMacosTree(realTestsDir);
|
||||
}, 'a full sweep of the real tests/ tree must complete without throwing');
|
||||
|
||||
// Measured 196/930 at authoring time (#4593 design doc). Sanity ballpark
|
||||
// with headroom for organic test-suite growth, not a brittle exact match.
|
||||
// Post-#4641 the tier is a ratio-bounded MINORITY of eligible unit-suite
|
||||
// files (MACOS_TIER_RATIO_CEILING, same policy ceiling the #4641 block
|
||||
// below enforces), not a brittle absolute-count range against a literal
|
||||
// that goes stale every time the category set or the suite's file count
|
||||
// changes (a hardcoded exact-count sanity range already broke once in
|
||||
// this PR). Derived from the live tree, not a hardcoded number.
|
||||
const { suiteOf } = require('../scripts/lib/suite-detection.cjs');
|
||||
const eligibleCount = walkTestFiles(realTestsDir).filter((absPath) => suiteOf(absPath) === null).length;
|
||||
const ratio = fresh.files.length / eligibleCount;
|
||||
assert.ok(
|
||||
fresh.files.length >= 100 && fresh.files.length <= 350,
|
||||
`expected a real, current, sanity-checked count in [100, 350], got ${fresh.files.length}`,
|
||||
ratio > 0 && ratio <= MACOS_TIER_RATIO_CEILING,
|
||||
`expected a nonzero macOS conformance tier within the #4641 ratio ceiling (<=${MACOS_TIER_RATIO_CEILING * 100}%), ` +
|
||||
`got ${fresh.files.length}/${eligibleCount} (${(ratio * 100).toFixed(1)}%)`,
|
||||
);
|
||||
|
||||
delete require.cache[require.resolve(MACOS_GENERATED_PATH)];
|
||||
@@ -511,3 +569,280 @@ describe('gen-platform-conformance-tier.cjs — real repo tree, macOS target (re
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── #4641: narrow the Windows conformance tier away from house-idiom noise ───
|
||||
|
||||
describe('conformance tier narrowing (#4641)', () => {
|
||||
// WINDOWS_TIER_RATIO_CEILING / MACOS_TIER_RATIO_CEILING are hoisted to
|
||||
// module scope above (shared with the real-repo-tree regression case
|
||||
// further down, which needs the same ceiling rather than a second
|
||||
// independently-drifting copy of it).
|
||||
|
||||
test('conformance tier stays a tier, not the suite (#4641)', () => {
|
||||
const realTestsDir = path.join(ROOT, 'tests');
|
||||
const { suiteOf } = require('../scripts/lib/suite-detection.cjs');
|
||||
|
||||
const absoluteFiles = walkTestFiles(realTestsDir);
|
||||
const eligibleFiles = absoluteFiles.filter((absPath) => suiteOf(absPath) === null);
|
||||
const eligibleCount = eligibleFiles.length;
|
||||
|
||||
let tierCount = 0;
|
||||
for (const absPath of eligibleFiles) {
|
||||
const content = fs.readFileSync(absPath, 'utf8');
|
||||
const { needsRealOs } = classifyContent(content);
|
||||
if (needsRealOs) tierCount++;
|
||||
}
|
||||
|
||||
const ratio = tierCount / eligibleCount;
|
||||
assert.ok(
|
||||
ratio <= WINDOWS_TIER_RATIO_CEILING,
|
||||
`expected the conformance tier to be at most ${WINDOWS_TIER_RATIO_CEILING * 100}% of eligible unit-suite files, ` +
|
||||
`got ${tierCount}/${eligibleCount} (${(ratio * 100).toFixed(1)}%)`,
|
||||
);
|
||||
});
|
||||
|
||||
test('macos conformance tier stays within its evidence-backed ceiling (#4641)', () => {
|
||||
const realTestsDir = path.join(ROOT, 'tests');
|
||||
const { suiteOf } = require('../scripts/lib/suite-detection.cjs');
|
||||
|
||||
const absoluteFiles = walkTestFiles(realTestsDir);
|
||||
const eligibleFiles = absoluteFiles.filter((absPath) => suiteOf(absPath) === null);
|
||||
const eligibleCount = eligibleFiles.length;
|
||||
|
||||
let tierCount = 0;
|
||||
for (const absPath of eligibleFiles) {
|
||||
const content = fs.readFileSync(absPath, 'utf8');
|
||||
const { needsRealOs } = classifyMacosContent(content);
|
||||
if (needsRealOs) tierCount++;
|
||||
}
|
||||
|
||||
const ratio = tierCount / eligibleCount;
|
||||
assert.ok(
|
||||
ratio <= MACOS_TIER_RATIO_CEILING,
|
||||
`expected the macOS conformance tier to be at most ${MACOS_TIER_RATIO_CEILING * 100}% of eligible unit-suite files, ` +
|
||||
`got ${tierCount}/${eligibleCount} (${(ratio * 100).toFixed(1)}%)`,
|
||||
);
|
||||
});
|
||||
|
||||
test('seam-helper calls are not a platform signal (#4641)', () => {
|
||||
for (const call of ['runNode(', 'runGit(', 'runHook(', 'runGsdTools(', 'gitOrThrow(']) {
|
||||
const fixture = `${call}args);\nassert.equal(result.exitCode, 0);\n`;
|
||||
const { needsRealOs, signals } = classifyContent(fixture);
|
||||
assert.equal(needsRealOs, false, call);
|
||||
assert.deepEqual(signals, [], call);
|
||||
}
|
||||
});
|
||||
|
||||
test('a path call plus a slash literal is not a platform signal (#4641)', () => {
|
||||
const fixture = "const p = path.join(dir, 'sub');\nassert.equal(p, '/tmp/fixture');\n";
|
||||
const { needsRealOs, signals } = classifyContent(fixture);
|
||||
assert.equal(needsRealOs, false);
|
||||
assert.deepEqual(signals, []);
|
||||
});
|
||||
|
||||
test('the two house-idiom detectors are gone (#4641)', () => {
|
||||
const names = CATEGORIES.map((c) => c.name);
|
||||
assert.ok(!names.includes('process-seam-subprocess'), `CATEGORIES still contains process-seam-subprocess: ${JSON.stringify(names)}`);
|
||||
assert.ok(!names.includes('hardcoded-path-vs-path-call'), `CATEGORIES still contains hardcoded-path-vs-path-call: ${JSON.stringify(names)}`);
|
||||
});
|
||||
|
||||
test('genuine platform-conditional content still classifies IN (#4641)', () => {
|
||||
const { needsRealOs, signals } = classifyContent("if (process.platform === 'win32') { doThing(); }");
|
||||
assert.equal(needsRealOs, true);
|
||||
assert.ok(signals.includes('process-platform'));
|
||||
});
|
||||
|
||||
test('seam-BYPASSING spawn still classifies IN (#4641)', () => {
|
||||
const fixture = "const { spawnSync } = require('node:child_process');\nspawnSync('ls', []);";
|
||||
const { needsRealOs, signals } = classifyContent(fixture);
|
||||
assert.equal(needsRealOs, true);
|
||||
assert.ok(signals.includes('raw-child-process'));
|
||||
});
|
||||
|
||||
test('chmodSync content still classifies IN via chmod-mode-bit (#4641)', () => {
|
||||
const { needsRealOs, signals } = classifyContent('fs.chmodSync(target, mode);');
|
||||
assert.equal(needsRealOs, true);
|
||||
assert.ok(signals.includes('chmod-mode-bit'));
|
||||
});
|
||||
|
||||
test('symlinkSync content still classifies IN via symlink-keyword (#4641)', () => {
|
||||
const { needsRealOs, signals } = classifyContent('fs.symlinkSync(target, link);');
|
||||
assert.equal(needsRealOs, true);
|
||||
assert.ok(signals.includes('symlink-keyword'));
|
||||
});
|
||||
|
||||
test('macOS signal set is untouched by the Windows narrowing (#4641)', () => {
|
||||
assert.deepEqual(
|
||||
MACOS_CATEGORIES.map((c) => c.name),
|
||||
['darwin-literal', 'zsh-dispatch', 'case-sensitivity', 'chmod-mode-bit', 'symlink-keyword'],
|
||||
);
|
||||
});
|
||||
|
||||
test('macOS generated tier matches a fresh classification of the live tests/ tree (#4641)', () => {
|
||||
const realTestsDir = path.join(ROOT, 'tests');
|
||||
const fresh = classifyMacosTree(realTestsDir);
|
||||
|
||||
delete require.cache[require.resolve(MACOS_GENERATED_PATH)];
|
||||
const { MACOS_CONFORMANCE_TIER_FILES } = require(MACOS_GENERATED_PATH);
|
||||
|
||||
assert.deepEqual(
|
||||
MACOS_CONFORMANCE_TIER_FILES.slice().sort(),
|
||||
fresh.files.slice().sort(),
|
||||
'the committed macOS generated file must be fresh — run ' +
|
||||
'`node scripts/gen-platform-conformance-tier.cjs --target macos --write`',
|
||||
);
|
||||
});
|
||||
|
||||
test('named probe files still classify IN on a genuine platform signal, not by filename (#4641)', () => {
|
||||
// These three files are examples, not the assertion. Each was picked
|
||||
// because it probes a DIFFERENT genuine platform-signal family, so the
|
||||
// three together exercise the narrowed classifier's breadth, not just its
|
||||
// presence: shell-command-projection-dispatch pulls in raw
|
||||
// spawn/shell-dispatch signals, review-lane-windows-spawn-resolution pulls
|
||||
// in Windows path/spawn-resolution signals, and prohibition-enforcement
|
||||
// pulls in process.platform/os.platform conditionals. Asserting by
|
||||
// FILENAME membership is exactly the classifier bug #4641 fixes — a file
|
||||
// being in this literal list proves nothing about why it is in the tier.
|
||||
// So the assertion here is on the SIGNAL: each file must still classify
|
||||
// needsRealOs === true, and its surviving `signals` must be non-empty and
|
||||
// drawn from the real (non-house-idiom) category names still present in
|
||||
// CATEGORIES after the #4641 narrowing.
|
||||
const files = [
|
||||
'tests/shell-command-projection-dispatch.test.cjs',
|
||||
'tests/review-lane-windows-spawn-resolution.test.cjs',
|
||||
'tests/prohibition-enforcement.test.cjs',
|
||||
];
|
||||
const validSignalNames = new Set(CATEGORIES.map((c) => c.name));
|
||||
for (const rel of files) {
|
||||
const absPath = path.join(ROOT, rel);
|
||||
const content = fs.readFileSync(absPath, 'utf8');
|
||||
const { needsRealOs, signals } = classifyContent(content);
|
||||
assert.equal(needsRealOs, true, rel);
|
||||
assert.ok(signals.length > 0, `${rel}: expected a non-empty signal set, got none`);
|
||||
for (const signal of signals) {
|
||||
assert.ok(
|
||||
validSignalNames.has(signal),
|
||||
`${rel}: signal "${signal}" is not a genuine platform category name (${JSON.stringify([...validSignalNames])})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('CATEGORIES contains the shell-interpreter-spawn detector (#4641)', () => {
|
||||
const names = CATEGORIES.map((c) => c.name);
|
||||
assert.ok(
|
||||
names.includes('shell-interpreter-spawn'),
|
||||
`CATEGORIES must contain shell-interpreter-spawn: ${JSON.stringify(names)}`,
|
||||
);
|
||||
});
|
||||
|
||||
test('a real interpreter: bash spawn (the adversarial-review finding) still classifies IN (#4641)', () => {
|
||||
// tests/execute-phase-worktree-guard.test.cjs calls tests/helpers/
|
||||
// process-seam.cjs's runHook(..., { interpreter: 'bash', ... }), which
|
||||
// spawns a REAL bash binary via spawnSync. That is a genuine
|
||||
// platform-dependent signal (bash availability, quoting, git output
|
||||
// parsing all differ across OSes) that the removed
|
||||
// 'process-seam-subprocess' category used to catch incidentally, and
|
||||
// which silently dropped out of the tier when that category was removed
|
||||
// — an adversarial review caught this as a real false negative (#4641).
|
||||
// Assert directly against the real file's content so nobody can
|
||||
// "fix" a regression here by re-editing a hand-written fixture string.
|
||||
const absPath = path.join(ROOT, 'tests/execute-phase-worktree-guard.test.cjs');
|
||||
const content = fs.readFileSync(absPath, 'utf8');
|
||||
const { needsRealOs, signals } = classifyContent(content);
|
||||
assert.equal(needsRealOs, true, 'tests/execute-phase-worktree-guard.test.cjs');
|
||||
assert.ok(
|
||||
signals.includes('shell-interpreter-spawn'),
|
||||
`expected shell-interpreter-spawn among signals, got: ${JSON.stringify(signals)}`,
|
||||
);
|
||||
});
|
||||
|
||||
test('runHook without an interpreter option does not match shell-interpreter-spawn (#4641)', () => {
|
||||
// The detector must key on a real shell name being passed as the
|
||||
// `interpreter` option, not on the mere presence of `runHook(...)` —
|
||||
// the default (no `interpreter:` option) spawns node, not a real shell,
|
||||
// and is not a platform signal.
|
||||
const fixture = "runHook(HOOK_PATH, [], { cwd: dir });\nassert.equal(result.exitCode, 0);\n";
|
||||
const { needsRealOs, signals } = classifyContent(fixture);
|
||||
assert.equal(needsRealOs, false);
|
||||
assert.ok(!signals.includes('shell-interpreter-spawn'), JSON.stringify(signals));
|
||||
});
|
||||
|
||||
test('a source-text-analysis test drops out of the tier even when its filename says windows (#4641)', () => {
|
||||
// tests/windows-robustness.test.cjs carries `// allow-test-rule:
|
||||
// source-text-is-the-product` and only ever reads OTHER files' source
|
||||
// text and asserts on it (e.g. `assert.match(region, /windowsHide:\s*true/)`).
|
||||
// Its apparent `spawnSync(` / `execFileSync(` hits are string-literal
|
||||
// search anchors into other files' source, not real subprocess calls —
|
||||
// it spawns nothing itself and is fully Linux-runnable. Despite the
|
||||
// filename, it must classify OUT of the real-OS tier. Do not "fix" this
|
||||
// by re-adding the file to the four-named-files case above.
|
||||
const absPath = path.join(ROOT, 'tests/windows-robustness.test.cjs');
|
||||
const content = fs.readFileSync(absPath, 'utf8');
|
||||
const { needsRealOs } = classifyContent(content);
|
||||
assert.equal(needsRealOs, false, 'tests/windows-robustness.test.cjs');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── #4641: ALWAYS_REAL_OS escape hatch for code-under-test-only signals ───
|
||||
|
||||
describe('ALWAYS_REAL_OS escape hatch (#4641)', () => {
|
||||
test('is a Map, so every entry is forced to carry a reason', () => {
|
||||
assert.ok(ALWAYS_REAL_OS instanceof Map, 'ALWAYS_REAL_OS must be a Map');
|
||||
});
|
||||
|
||||
test('every key is present in the committed Windows CONFORMANCE_TIER_FILES', () => {
|
||||
delete require.cache[require.resolve(GENERATED_PATH)];
|
||||
const { CONFORMANCE_TIER_FILES } = require(GENERATED_PATH);
|
||||
const committedSet = new Set(CONFORMANCE_TIER_FILES);
|
||||
for (const relPath of ALWAYS_REAL_OS.keys()) {
|
||||
assert.ok(
|
||||
committedSet.has(relPath),
|
||||
`${relPath} is in ALWAYS_REAL_OS but missing from the committed Windows tier — ` +
|
||||
'run `node scripts/gen-platform-conformance-tier.cjs --write`',
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('every entry has a non-empty recorded reason', () => {
|
||||
for (const [relPath, reason] of ALWAYS_REAL_OS.entries()) {
|
||||
assert.equal(typeof reason, 'string', `${relPath}: reason must be a string`);
|
||||
assert.ok(reason.trim().length > 0, `${relPath}: reason must be non-empty`);
|
||||
}
|
||||
});
|
||||
|
||||
test('every key names a file that actually exists on disk', () => {
|
||||
for (const relPath of ALWAYS_REAL_OS.keys()) {
|
||||
const absPath = path.join(ROOT, relPath);
|
||||
assert.ok(
|
||||
fs.existsSync(absPath),
|
||||
`${relPath} is enumerated in ALWAYS_REAL_OS but does not exist on disk — stale allowlist entry ` +
|
||||
'(silent rot: the file was likely deleted or renamed)',
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('tests/external-descriptor-confinement.test.cjs is enumerated (#4641)', () => {
|
||||
// It exercises isPathConfined (src/external-descriptor-trust.cts:41-49),
|
||||
// which uses the AMBIENT path module (path.resolve/path.sep) with no
|
||||
// platform/path injection — its win32 branch (drive letters, UNC paths,
|
||||
// \ separator) is only reachable by actually running on Windows. A
|
||||
// security-relevant write-confinement gate; do not remove this entry to
|
||||
// "clean up" the allowlist.
|
||||
assert.ok(
|
||||
ALWAYS_REAL_OS.has('tests/external-descriptor-confinement.test.cjs'),
|
||||
'tests/external-descriptor-confinement.test.cjs must stay in ALWAYS_REAL_OS',
|
||||
);
|
||||
});
|
||||
|
||||
test('does not leak into the macOS tier — macOS is POSIX, the win32 concern does not apply', () => {
|
||||
delete require.cache[require.resolve(MACOS_GENERATED_PATH)];
|
||||
const { MACOS_CONFORMANCE_TIER_FILES } = require(MACOS_GENERATED_PATH);
|
||||
const macosSet = new Set(MACOS_CONFORMANCE_TIER_FILES);
|
||||
assert.ok(
|
||||
!macosSet.has('tests/external-descriptor-confinement.test.cjs'),
|
||||
'tests/external-descriptor-confinement.test.cjs must be absent from MACOS_CONFORMANCE_TIER_FILES ' +
|
||||
'(the ALWAYS_REAL_OS entry is Windows-only)',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -511,25 +511,42 @@ function safeRealpath(p) {
|
||||
|
||||
describe('bug-131: runNpm isolates HOME from the caller environment', () => {
|
||||
// ── Test 1 — runNpm works with an unwritable HOME ────────────────────────
|
||||
// Spawn a child Node process that sets HOME to a chmod-0500 directory, then
|
||||
// invokes runNpm(['--version']). Without the fix, npm tries to read/write
|
||||
// HOME/.npmrc and HOME/.npm, fails with EACCES, and runNpm throws.
|
||||
// With the fix, runNpm injects its own isolated HOME and npm succeeds.
|
||||
// Spawn a child Node process that sets HOME to an unwritable directory, then
|
||||
// invokes runNpm(['cache', 'verify']). `npm --version` performs zero
|
||||
// filesystem I/O against HOME/.npm or HOME/.npmrc on modern npm, and even
|
||||
// `npm config get cache` only *resolves* the cache path as a string without
|
||||
// touching disk — both stay green even without HOME isolation, making the
|
||||
// assertion vacuous. `npm cache verify` genuinely creates/reads/writes the
|
||||
// cache directory under HOME (mkdir _cacache, write logs), so without the
|
||||
// fix it fails with ENOTDIR against the unwritable HOME, and with the fix
|
||||
// runNpm's injected isolated HOME lets it succeed. (Proven empirically: with
|
||||
// this exact probe, neutralising runNpm()'s isolation flips this test from
|
||||
// green to red, whereas `npm config get cache` stayed green either way.)
|
||||
test('runNpm succeeds even when process HOME is unwritable', () => {
|
||||
// Create an unwritable dir to serve as a poisoned HOME.
|
||||
const poisonedHome = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-bug131-poison-'));
|
||||
// Simulate an unwritable HOME with a mechanism that holds for every uid,
|
||||
// including root (as gsd-test Docker benches run). chmod 0o500 is
|
||||
// insufficient because root bypasses mode bits entirely, silently making
|
||||
// this assertion vacuous under root — see CLAUDE.md section 4. Instead,
|
||||
// make the PARENT of "HOME" a regular file rather than a directory: any
|
||||
// attempt to create or write an entry under a non-directory parent fails
|
||||
// with ENOTDIR at the filesystem/VFS level, a property that has nothing
|
||||
// to do with permission bits and therefore cannot be bypassed by root.
|
||||
const poisonedHomeBlocker = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-bug131-poison-'));
|
||||
const blockerFile = path.join(poisonedHomeBlocker, 'blocker');
|
||||
fs.writeFileSync(blockerFile, ''); // regular file, not a directory
|
||||
const poisonedHome = path.join(blockerFile, 'home'); // parent is a file → ENOTDIR
|
||||
try {
|
||||
fs.chmodSync(poisonedHome, 0o500); // r-x only — not writable
|
||||
|
||||
// We exercise the real runNpm() path by running a tiny inline Node script
|
||||
// that requires helpers.cjs and calls runNpm(['--version']) with HOME set
|
||||
// to the unwritable dir. The script exits 0 on success, non-zero on throw.
|
||||
// that requires helpers.cjs and calls runNpm(['cache', 'verify']) with
|
||||
// HOME set to the unwritable dir. The script exits 0 on success, non-zero
|
||||
// on throw.
|
||||
const script = `
|
||||
process.env.HOME = ${JSON.stringify(poisonedHome)};
|
||||
process.env.USERPROFILE = ${JSON.stringify(poisonedHome)};
|
||||
const { runNpm } = require(${JSON.stringify(path.join(__dirname, 'helpers.cjs'))});
|
||||
try {
|
||||
const out = runNpm(['--version']);
|
||||
const out = runNpm(['cache', 'verify']);
|
||||
if (!out || out.trim() === '') process.exit(2); // vacuous success guard
|
||||
process.stdout.write(out);
|
||||
process.exit(0);
|
||||
@@ -558,16 +575,16 @@ describe('bug-131: runNpm isolates HOME from the caller environment', () => {
|
||||
0,
|
||||
`runNpm should succeed with an unwritable HOME but exited ${exitCode}. stderr: ${stderr}`,
|
||||
);
|
||||
// npm --version returns something like "10.x.y"
|
||||
// npm cache verify reports what it found/fixed in the cache directory.
|
||||
assert.match(
|
||||
stdout.trim(),
|
||||
/^\d+\.\d+/,
|
||||
`expected semver output from npm --version, got: ${stdout}`,
|
||||
stdout,
|
||||
/cache verified|content verified/i,
|
||||
`expected npm cache verify output, got: ${stdout}`,
|
||||
);
|
||||
} finally {
|
||||
// Restore write permission before cleanup so the directory can be deleted.
|
||||
try { fs.chmodSync(poisonedHome, 0o700); } catch (_) { /* best-effort */ }
|
||||
cleanup(poisonedHome);
|
||||
// poisonedHome itself was never created (its parent is a file), so only
|
||||
// the directory holding the blocker file needs cleanup.
|
||||
cleanup(poisonedHomeBlocker);
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user