feat(#4593): add a macOS-specific conformance tier, final phase of epic #4589 (#4607)

test-conformance's macos-latest leg (Phase 2, #4591) has been running the same 546-file, Windows-oriented conformance-tier list as windows-latest -- built from signals like windows-shell-token/windows-env-var that have nothing to do with macOS. Issue #4593 asked for macOS coverage sized to its own evidence-backed surface (zsh dispatch, case-sensitivity, darwin- specific behavior) instead. Issue #4593 was filed before Phase 5 (#4603) existed and referenced updating test-full's macOS legs -- that job is gone. Corrected the issue's body before any code was touched: the "shrink from full replay" half of the original ask was already done by Phase 5; what remained was narrowing the still-Windows-oriented tier macOS was inheriting. Two design assumptions were measured and rejected before accepting a design (documented in docs/adr/4593-macos-conformance-tier-architecture.md): - Reusing the general tier's signals minus its 3 Windows-specific categories barely narrows anything (546 -> 424, 78% retained) -- most files match multiple signals and only need one to survive exclusion. - A standalone CRLF/autocrlf signal, despite the issue naming "CRLF-checkout behavior": even narrowed to /\bCRLF\b|autocrlf/i it hit 143/930 files. Root cause: CRLF is primarily a Windows checkout concern in this codebase (ADR-1703 files it under DEFECT.WINDOWS-TEST- PORTABILITY), so the signal was really re-selecting Windows-relevant files already covered by the general tier, not narrowing macOS specifically. Built 5 new, genuinely macOS-specific signals instead: darwin-literal (darwin alone, not the general tier's win32-OR-darwin), zsh-dispatch, case-sensitivity, plus chmod-mode-bit and symlink-keyword reused verbatim from the general tier (genuinely Unix-relevant, not Windows-motivated). Measured against the real tree: 196 of 930 eligible unit-suite files (21%), versus the general tier's 546 (59%) -- a real, evidence-backed narrowing. scripts/gen-platform-conformance-tier.cjs gains classifyMacosContent/ classifyMacosTree/renderMacosGeneratedFile and a --target windows (default, unchanged)/--target macos CLI flag, so the same generator produces two independent, gated outputs rather than needing a second script. New committed output: scripts/lib/macos-conformance-tier. generated.cjs. .github/workflows/test.yml's test-conformance job: only the macos-latest leg's file-list source changes; windows-latest is byte-for-byte untouched. New shipped-file ripples handled proactively (19 install-tree fixtures regenerated, bin/install.js registered). An isolated code-review pass found one real defect: the ADR's per- category count table had drifted by 1 (zsh-dispatch, case-sensitivity) because the new test file's own fixture strings joined the tree it classifies after the table was authored -- fixed, with the union total (196, what CI actually gates on) confirmed unaffected. An isolated security-review pass found no qualifying findings. The ADR also records an explicit requirement for any future widening proposal: check whether the motivating regression is already covered by Phase 1's no-rendered-text-length-assert lint rule (#4590) before re-proposing full macOS/Linux parity, since that is exactly what #4421's root cause was (a rendered-text-length assertion, not a real behavioral divergence). Co-authored-by: sim <sim@local> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-10 14:37:15 -04:00
parent 557a0b2876
commit 1e47560e34
27 changed files with 719 additions and 23 deletions
--- a/tests/fixtures/install-tree/antigravity.json
+++ b/tests/fixtures/install-tree/antigravity.json
@@ -622,6 +622,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/augment.json
+++ b/tests/fixtures/install-tree/augment.json
@@ -693,6 +693,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/claude-local.json
+++ b/tests/fixtures/install-tree/claude-local.json
@@ -557,6 +557,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/claude.json
+++ b/tests/fixtures/install-tree/claude.json
@@ -621,6 +621,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/cline.json
+++ b/tests/fixtures/install-tree/cline.json
@@ -583,6 +583,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/codebuddy.json
+++ b/tests/fixtures/install-tree/codebuddy.json
@@ -693,6 +693,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/codex.json
+++ b/tests/fixtures/install-tree/codex.json
@@ -621,6 +621,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/copilot.json
+++ b/tests/fixtures/install-tree/copilot.json
@@ -583,6 +583,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/cursor.json
+++ b/tests/fixtures/install-tree/cursor.json
@@ -594,6 +594,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/hermes.json
+++ b/tests/fixtures/install-tree/hermes.json
@@ -621,6 +621,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/kilo.json
+++ b/tests/fixtures/install-tree/kilo.json
@@ -696,6 +696,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/kimi-code.json
+++ b/tests/fixtures/install-tree/kimi-code.json
@@ -622,6 +622,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/kimi.json
+++ b/tests/fixtures/install-tree/kimi.json
@@ -589,6 +589,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/opencode.json
+++ b/tests/fixtures/install-tree/opencode.json
@@ -696,6 +696,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/pi.json
+++ b/tests/fixtures/install-tree/pi.json
@@ -423,6 +423,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/qwen.json
+++ b/tests/fixtures/install-tree/qwen.json
@@ -621,6 +621,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/trae.json
+++ b/tests/fixtures/install-tree/trae.json
@@ -581,6 +581,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/windsurf.json
+++ b/tests/fixtures/install-tree/windsurf.json
@@ -516,6 +516,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/fixtures/install-tree/zcode.json
+++ b/tests/fixtures/install-tree/zcode.json
@@ -653,6 +653,7 @@
  "scripts/lib/cli-exit.cjs",
  "scripts/lib/drift-scan.cjs",
  "scripts/lib/exit-code-registry.cjs",
+  "scripts/lib/macos-conformance-tier.generated.cjs",
  "scripts/lib/ndjson-reporter.cjs",
  "scripts/lib/npm-version-check-diagnosis.cjs",
  "scripts/lib/platform-conformance-tier.generated.cjs",
--- a/tests/platform-conformance-tier.test.cjs
+++ b/tests/platform-conformance-tier.test.cjs
@@ -22,11 +22,18 @@ const path = require('node:path');

 const { createTempDir, cleanup } = require('./helpers.cjs');
 const { runNode } = require('./helpers/process-seam.cjs');
-const { classifyContent, classifyTree, NOISY_FOR_SOURCE_REACHABILITY } = require('../scripts/gen-platform-conformance-tier.cjs');
+const {
+  classifyContent,
+  classifyTree,
+  NOISY_FOR_SOURCE_REACHABILITY,
+  classifyMacosContent,
+  classifyMacosTree,
+} = require('../scripts/gen-platform-conformance-tier.cjs');

 const ROOT = path.resolve(__dirname, '..');
 const SCRIPT = path.join(ROOT, 'scripts', 'gen-platform-conformance-tier.cjs');
 const GENERATED_PATH = path.join(ROOT, 'scripts', 'lib', 'platform-conformance-tier.generated.cjs');
+const MACOS_GENERATED_PATH = path.join(ROOT, 'scripts', 'lib', 'macos-conformance-tier.generated.cjs');

 // ─── Rows 1-15: classifyContent, pure fixtures ────────────────────────────────

@@ -319,3 +326,188 @@ describe('gen-platform-conformance-tier.cjs — real repo tree (regression)', ()
    );
  });
 });
+
+// ─── #4593: macOS-specific classifier (classifyMacosContent / MACOS_CATEGORIES)
+
+describe('classifyMacosContent — happy-path signals', () => {
+  test('flags darwin literal', () => {
+    const { needsRealOs, signals } = classifyMacosContent("const platforms = ['darwin', 'linux'];");
+    assert.equal(needsRealOs, true);
+    assert.ok(signals.includes('darwin-literal'));
+  });
+
+  test('flags zsh dispatch', () => {
+    const { needsRealOs, signals } = classifyMacosContent("shell: 'zsh {0}'");
+    assert.equal(needsRealOs, true);
+    assert.ok(signals.includes('zsh-dispatch'));
+  });
+
+  test('flags case-sensitivity phrasing', () => {
+    for (const fixture of [
+      '// verify the case-insensitive lookup',
+      '// verify the case-sensitive lookup',
+      '// verify case insensitivity',
+    ]) {
+      const { needsRealOs, signals } = classifyMacosContent(fixture);
+      assert.equal(needsRealOs, true, fixture);
+      assert.ok(signals.includes('case-sensitivity'), fixture);
+    }
+  });
+
+  test('flags chmod mode-bit octal', () => {
+    const { needsRealOs, signals } = classifyMacosContent('fs.chmodSync(target, 0o755);');
+    assert.equal(needsRealOs, true);
+    assert.ok(signals.includes('chmod-mode-bit'));
+  });
+
+  test('flags symlink keyword', () => {
+    const { needsRealOs, signals } = classifyMacosContent('fs.symlinkSync(target, link);');
+    assert.equal(needsRealOs, true);
+    assert.ok(signals.includes('symlink-keyword'));
+  });
+});
+
+describe('classifyMacosContent — negative case', () => {
+  test('does not flag a clean file with none of the 5 macOS signals', () => {
+    const fixture =
+      "const assert = require('node:assert/strict');\n" +
+      "if (process.platform === 'win32') { doThing(); }\n" +
+      "test('adds numbers', () => { assert.equal(1 + 1, 2); });";
+    const { needsRealOs, signals } = classifyMacosContent(fixture);
+    assert.equal(needsRealOs, false);
+    assert.deepEqual(signals, []);
+  });
+
+  test('does not crash on empty content', () => {
+    assert.doesNotThrow(() => classifyMacosContent(''));
+    const { needsRealOs, signals } = classifyMacosContent('');
+    assert.equal(needsRealOs, false);
+    assert.deepEqual(signals, []);
+  });
+});
+
+// ─── #4593: CLI --target macos ─────────────────────────────────────────────
+
+describe('gen-platform-conformance-tier.cjs CLI --target macos (temp fixture tree)', () => {
+  test('--target macos --check passes when the generated file is fresh', () => {
+    const tmpDir = createTempDir('gen-platform-conformance-tier-macos-');
+    try {
+      const testsDir = path.join(tmpDir, 'tests');
+      fs.mkdirSync(testsDir, { recursive: true });
+      fs.writeFileSync(path.join(testsDir, 'flagged.test.cjs'), 'fs.symlinkSync(target, link);\n');
+      fs.writeFileSync(path.join(testsDir, 'clean.test.cjs'), "assert.equal(1 + 1, 2);\n");
+      const outPath = path.join(tmpDir, 'macos-conformance-tier.generated.cjs');
+
+      const write = runGen(['--target', 'macos', '--write', '--tests-dir', testsDir, '--out', outPath]);
+      assert.equal(write.exitCode, 0, write.stderr);
+      assert.ok(fs.existsSync(outPath));
+
+      delete require.cache[require.resolve(outPath)];
+      const generated = require(outPath);
+      assert.deepEqual(generated.MACOS_CONFORMANCE_TIER_FILES, ['tests/flagged.test.cjs']);
+
+      const check = runGen(['--target', 'macos', '--check', '--tests-dir', testsDir, '--out', outPath]);
+      assert.equal(check.exitCode, 0, check.stderr);
+      assert.match(check.stdout, /ok gen-platform-conformance-tier --target macos/);
+    } finally {
+      cleanup(tmpDir);
+    }
+  });
+
+  test('--target macos --check fails and names the drift when the list is stale', () => {
+    const tmpDir = createTempDir('gen-platform-conformance-tier-macos-');
+    try {
+      const testsDir = path.join(tmpDir, 'tests');
+      fs.mkdirSync(testsDir, { recursive: true });
+      fs.writeFileSync(path.join(testsDir, 'flagged.test.cjs'), 'fs.symlinkSync(target, link);\n');
+      const outPath = path.join(tmpDir, 'macos-conformance-tier.generated.cjs');
+
+      const write = runGen(['--target', 'macos', '--write', '--tests-dir', testsDir, '--out', outPath]);
+      assert.equal(write.exitCode, 0, write.stderr);
+
+      fs.writeFileSync(path.join(testsDir, 'new-signal.test.cjs'), "const platforms = ['darwin'];\n");
+
+      const check = runGen(['--target', 'macos', '--check', '--tests-dir', testsDir, '--out', outPath]);
+      assert.equal(check.exitCode, 1);
+      const combined = check.stdout + check.stderr;
+      assert.match(combined, /tests\/new-signal\.test\.cjs/, 'the drift report must name the new file');
+    } finally {
+      cleanup(tmpDir);
+    }
+  });
+
+  test('a suite-suffixed file is excluded even with a qualifying macOS content signal', () => {
+    const tmpDir = createTempDir('gen-platform-conformance-tier-macos-suite-');
+    try {
+      const testsDir = path.join(tmpDir, 'tests');
+      fs.mkdirSync(testsDir, { recursive: true });
+      const signal = 'fs.symlinkSync(target, link);\n';
+      fs.writeFileSync(path.join(testsDir, 'foo.test.cjs'), signal);
+      fs.writeFileSync(path.join(testsDir, 'foo.install.test.cjs'), signal);
+      const outPath = path.join(tmpDir, 'macos-conformance-tier.generated.cjs');
+
+      const write = runGen(['--target', 'macos', '--write', '--tests-dir', testsDir, '--out', outPath]);
+      assert.equal(write.exitCode, 0, write.stderr);
+
+      delete require.cache[require.resolve(outPath)];
+      const generated = require(outPath);
+
+      assert.deepEqual(
+        generated.MACOS_CONFORMANCE_TIER_FILES,
+        ['tests/foo.test.cjs'],
+        'the install-suite file must be excluded even though its content would otherwise qualify',
+      );
+    } finally {
+      cleanup(tmpDir);
+    }
+  });
+});
+
+// ─── #4593: real tests/ tree, regression against the committed macOS artifact
+
+describe('gen-platform-conformance-tier.cjs — real repo tree, macOS target (regression)', () => {
+  test('real tests/ tree macOS classification matches the committed list', () => {
+    const realTestsDir = path.join(ROOT, 'tests');
+
+    let fresh;
+    assert.doesNotThrow(() => {
+      fresh = classifyMacosTree(realTestsDir);
+    }, 'a full sweep of the real tests/ tree must complete without throwing');
+
+    // Measured 196/930 at authoring time (#4593 design doc). Sanity ballpark
+    // with headroom for organic test-suite growth, not a brittle exact match.
+    assert.ok(
+      fresh.files.length >= 100 && fresh.files.length <= 350,
+      `expected a real, current, sanity-checked count in [100, 350], got ${fresh.files.length}`,
+    );
+
+    delete require.cache[require.resolve(MACOS_GENERATED_PATH)];
+    const committed = require(MACOS_GENERATED_PATH);
+
+    assert.equal(
+      committed.MACOS_CONFORMANCE_TIER_FILES.length,
+      fresh.files.length,
+      'the committed macOS generated file must be fresh — run ' +
+        '`node scripts/gen-platform-conformance-tier.cjs --target macos --write`',
+    );
+    assert.deepEqual(
+      committed.MACOS_CONFORMANCE_TIER_FILES.slice().sort(),
+      fresh.files.slice().sort(),
+      'the committed macOS list must match a fresh sweep exactly, not just in length',
+    );
+  });
+
+  test('the committed macOS conformance-tier list contains zero suite-tagged files', () => {
+    delete require.cache[require.resolve(MACOS_GENERATED_PATH)];
+    const { MACOS_CONFORMANCE_TIER_FILES } = require(MACOS_GENERATED_PATH);
+
+    const suiteTaggedPattern = /\.(install|security|slow|integration|qa)\.test\.cjs$/;
+    const offenders = MACOS_CONFORMANCE_TIER_FILES.filter((f) => suiteTaggedPattern.test(f));
+
+    assert.deepEqual(
+      offenders,
+      [],
+      'suite-tagged files must never appear in the macOS conformance-tier list either',
+    );
+  });
+});