Files
msd-core/docs/issueevidence/1192-adr-test-audit-2026-06-13.md
Tom Boucher d30c99bc92 chore(#3421): delete orphan verify-phase workflow, migrate live gates to verifier (#3422)
* chore(#1892): delete orphan verify-phase workflow, migrate live gates to verifier reference

* test(#1892): retarget structural suites from verify-phase.md to verifier-phase-gates.md

* chore(#1892): reword retired-workflow mentions for removed-but-needed lint

* test(#1892): correct stale surface labels in retargeted suites

* docs(#1892): add verifier-phase-gates row to locale inventories

* chore(#3421): backfill changeset pr number

---------

Co-authored-by: sim <sim@local>
2026-08-13 21:22:03 -04:00

35 KiB
Raw Blame History

ADR Test-Coverage Audit — 2026-06-13

Lead QA architect final report. Scope: 38 ADRs in docs/adr/, evaluated with the qa-test-architect methodology, plus 4 platform-capability lenses over the test infrastructure. Every RETIRE verdict was independently re-checked by an adversarial verification pass.

Scope note. ADR-857 (capability system core) was intentionally excluded from this audit — it is being worked by another instance. The capability cluster (ADR-894, ADR-959, ADR-1016, ADR-1143) is coupled to ADR-857, so all findings for those four ADRs are provisional and may shift once ADR-857 lands.


Executive summary

The test portfolio is, on the whole, healthier than its volume suggests: the deterministic seams (dispatch hub, model catalog, planning-path resolution, installer migration, shell-command projection) are genuinely behavioral and well-covered, and the project has built real test-rigor infrastructure (an injectable clock seam, fast-check with a pinned seed, Stryker mutation testing wired into CI, and four custom ESLint rules). The dominant problem is not absent tests — it is enforcement gaps that let the rigor infrastructure run as theater, and a long tail of weak tests concentrated in three families: source-text-grep assertions, tautological assert.ok(true) placeholders, and architecture-retirement guards that can never go red.

Five headline problems demand attention first: (1) Stryker mutation testing covers only ~4–6% of the lib surface and excludes every high-risk module (state, phase, verify, init, commands), with the break threshold set to 50% instead of the ADR-456-mandated 80% — the mutation gate trivially passes for PRs touching those modules. (2) 332 of 335 allow-test-rule exemptions carry no tracking issue, directly violating ADR-456's delete-bad-tests policy. (3) A subdirectory blind spot in scripts/run-tests.cjs silently excludes all of tests/observability/ and tests/dispatch/ from npm test — meaning ADR-227's entire UUID-validation seam is unguarded by any continuously-executed test. (4) Multiple ADRs marked Accepted or Proposed have zero attributed tests or test files that cannot run because their source modules are gitignored build artifacts absent from a clean worktree (ADR-218, ADR-0011 family). (5) A scatter of vacuous and self-referential tests (verify-test-quality.test.cjs, assert.ok(true) placeholders, environment-dependent path-guard branches) that provide false confidence on security-relevant code.

Do first: raise the Stryker break threshold and bring core.cts/config-loader.cts into the mutation-covered set (P0); fix the run-tests.cjs non-recursive file scan so observability/dispatch tests actually execute (P0); retire the 5 verified-worthless tests and redesign the 6 downgraded retire-candidates (P0/P1).


Testing-platform robustness

Synthesis of the four platform lenses: adr-456-test-rigor, coverage-mutation, anti-pattern-sweep, determinism-flake.

(a) What the platform CAN do today

  • Deterministic clock seam (tests/helpers/clock.cjs, makeFakeClock) — well-designed and correctly used by lock/state tests (clock-seam.test.cjs, perf-407, bug-474). Eliminates wall-clock races in the lock subsystem.
  • Property-based testing via fast-check with a pinned seed (tests/helpers/fast-check-setup.cjs, seed=42) — 9 *.property.test.cjs files exist and run reproducibly.
  • Mutation testing — Stryker is configured (stryker.config.mjs) and wired into a PR-gating workflow (.github/workflows/mutation.yml) that shards per changed module.
  • Custom ESLint rules — no-source-grep, no-magic-sleep-in-tests, no-elapsed-assertion, no-raw-rmsync-in-tests, plus no-only-tests — exist and are unit-tested via RuleTester (tests/eslint-rules.test.cjs).
  • Env hermeticity — scripts/run-tests.cjs deletes GSD_PROJECT/GSD_WORKSTREAM before spawning child test processes (mitigates the ambient-env hazard recorded in MEMORY.md).
  • Line-coverage gate — c8 --lines 70 (lib) enforced in CI as part of the unit lane.

(b) ENFORCED vs aspirational

Mechanism Status Note
Clock seam in lock/state tests Enforced Used correctly; but not mandated by lint — bug-3707 and phase.test.cjs:4362 still use real Date.now()
fast-check seed pinned Enforced seed=42; but no lint rule requires *.property.test.cjs to import the setup, and context-utilization.property.test.cjs uses Math.random() inside fc.property, breaking reproducibility
Stryker mutation gate Aspirational Runs in CI, but break=50% (not 80%) and covers 6/109 modules; trivially passes for excluded modules
Line coverage 70% Enforced But satisfiable via CLI integration tests that execute lines without asserting sub-behaviors; no branch-coverage floor
no-magic-sleep-in-tests Enforced (error) Promoted to error
no-raw-rmsync-in-tests Enforced (error)
no-only-tests Aspirational Wired at error but has zero RuleTester coverage and no integration test
no-source-grep Aspirational At warn only, and NOT applied to tests/** at all — the file-level allow-test-rule bypass blanket-exempts source-grep in test files
no-elapsed-assertion Aspirational At warn; only matches .elapsed/.duration/.took/.ms properties, missing Date.now()-start and computed-variable patterns (e.g. elapsedMs)
allow-test-rule tracking-issue requirement Not enforced 332/335 exemptions lack an issue number; no CI gate prevents new untracked exemptions
Subdirectory test discovery Broken run-tests.cjs uses flat readdirSync(testDir); tests/observability/ and tests/dispatch/ are silently never run by npm test
Fail-on-zero-executed Not enforced run-tests.cjs exits 0 when a suite resolves to zero files

(c) Prioritized capability upgrades

Capability Current state Recommendation Priority
Recursive test discovery Flat readdirSync skips tests/observability/, tests/dispatch/ Replace with the existing findTestFilesRecursive (already in the same file) P0
Stryker break threshold break: 50 in stryker.config.mjs Raise to 80 per ADR-456; move sub-80% modules to UNMUTATED with documented reasons P0
Mutation coverage breadth 6/109 modules covered; 12 critical modules excluded Graduate one module/release cycle starting with core.cts, config-loader.cts; require any module with a *.unit/property.test.cjs to be in COVERED or UNMUTATED P0
allow-test-rule audit 332/335 untracked exemptions Lint gate requiring either an approved category tag or #NNN; one-time tracking-issue sweep P1
no-source-grep in test files Not applied to tests/**; warn-only Apply at error to tests/**; split rule to allow product-file (.md/.json) reads but block source-file (.cjs/.js/.ts) grep assertions P1
Branch-coverage floor Only line coverage gated Add --branches 60; per-file branch floor for the UNMUTATED critical modules P1
no-elapsed-assertion reach Property-name match only Extend to Date.now()-start binary expressions and identifiers ending Ms/Elapsed/Duration in assert args P1
no-only-tests coverage Wired error, untested Add RuleTester valid/invalid cases + an integration assertion it fires P1
no-tautological-assert rule None New ~30-line local rule flagging assert.ok(true)/assert.ok(1) in tests/** P1
Fail-on-zero-executed Exits 0 on empty suite Exit 1 by default; gate the empty-suite allowance behind RUN_TESTS_ALLOW_EMPTY_SUITE=1 P2
Shared-GSD_TEMP_DIR isolation reap tests write to global os.tmpdir()/gsd Redirect via the existing dir option to per-test tmpdirs; lint against writing to the shared path without cleanup P2
Build-artifact preflight in artifact-dependent tests Top-level require() of gitignored .cjs hard-crashes on clean worktree before() existsSync guard with a clear "run npm run build:lib" message P2

ADR-456 compliance — per mandate

ADR-456 defines four test-rigor mandates. Compliance is partial on all four:

  • (a) Concurrency via injectable clock seam, not real OS races — Substantially met. The clock seam exists and the lock subsystem uses it. Gaps: bug-3707-locked-worktree-cleanup.test.cjs computes stale-lock mtimes from real Date.now() (7 sites); phase.test.cjs:4362 asserts a 60-second wall-clock window; core.test.cjs timeAgo boundary tests use real clock at the 5s bucket edge. No lint rule mandates the seam.
  • (b) fast-check + Stryker antagonistic tier, 80% kill threshold — Largely aspirational. fast-check is wired with a pinned seed, but Stryker covers ~4–6% of source lines, excludes all high-risk modules, and the break threshold is 50%, not the mandated 80%. The antagonistic tier exists in name but does not guard the dangerous surface.
  • (c) Assert on typed --json fields and exported registries, not rendered text or source literals — Partially met, weakly enforced. New tests largely follow it, but no-source-grep is warn-only and not applied to test files, so source-grep assertions persist (e.g. install.test.cjs .includes() on bin/install.js; sh-hook-paths.test.cjs variable-name scans). Several CLI tests still assert on stdout.includes() substrings rather than parsed JSON.
  • (d) Delete pass-always / vacuous / source-grep / elapsed / real-race / permanent-allow-rule tests — Not met. Concrete pass-always tests survive (research-cli.test.cjs:423, worktree-baseref-install.test.cjs:444, 24 assert.ok(true) in eslint-rules.test.cjs); 332/335 allow-test-rule exemptions are effectively permanent and untracked, directly contradicting the policy clause requiring // allow-test-rule: see #NNN on post-ADR exemptions.

Per-ADR coverage

ADR Status #Tests Happy Boundary Negative Independence Verdict
0001 Dispatch Policy Module Accepted 12 covered partial covered covered Core protected; SDK bridge / exit-code mapping unverified
0002 Command Contract Validation Accepted 6 covered partial missing covered Integration-strong; helper unit tests + lint negatives missing
0003 Model Catalog Module Accepted 13 covered partial covered covered CJS solid; SDK-parity invariant untested
0004 Worktree/Workstream Seam Accepted 13 covered partial covered covered Substantially protected; cross-path naming + .planning/.lock 2-proc gaps
0005 SDK Architecture Seam Map Superseded (0174) 13 partial partial partial covered Retirement guards OK; thin-adapter enforcement absent
0006 Planning-Path Projection Accepted 6 partial partial covered covered Critical gap: init-handler planningPaths() consumption untested
0007 SDK Package Seam Superseded (0174) 10 covered partial partial covered Retirement guarded; classifyPromptUserAction boundary untested
0008 Installer Migration Module Accepted 16 covered partial covered covered Rich suite; rewrite-json apply + local-scope gaps
0009 Shell Command Projection Accepted 21 covered partial covered covered Strong; 3 ADR-listed test files absent; source-grep in sh-hook-paths
0010a Skill Surface Budget Proposed (impl w/ gaps) 8 covered partial partial covered ~75% protected; stageCommandsForRuntimeFlat + migration one-shot untested
0010b File Operation Engine Superseded (0009) 10 covered partial covered covered Canonical seam solid; divergent atomicWriteFileSync + path-containment gaps
0011a Skill Surface Budget Accepted 12 covered partial partial covered Broad; source modules absent from worktree → MODULE_NOT_FOUND; cyclic-requires untested
0011b Review Default Reviewers (PRD) Draft 5 partial partial partial covered infos-path + --all+flags edge untested; build-artifact dependency
0011c Review Default Reviewers Proposed 4 covered partial partial covered Mid-tier risks (infos, empty-selection) missing; build-artifact dependency
0012 Command Routing Hub Superseded (0174) 5 covered partial covered covered Hub well-covered; uat-passed adapter arg-shape gap; 3 redundant tests
15 Autonomous Cross-AI Convergence Proposed 8 partial partial partial partial Primary surface (/gsd-progress --next --auto --converge) has zero coverage
22 Plan Drift Guard Proposed 6 partial partial covered covered Severity mapping, authority auto-upgrade, rung≥3 hard-block untested
58 Runtime Install Policy Accepted 7 covered partial partial covered Strong projection coverage; purity guarantee + sandboxTier-invalid untested
0174 Retire gsd-sdk Package Accepted 12 covered covered covered covered Strong & behavioral; init-composer trace + tsc-output invariant gaps
218 Release Version Validation Accepted 0 missing missing missing na ZERO TESTS — entirely unprotected (regex lives only in release.yml bash)
227 Input Validation Shape Accepted 5 covered covered partial partial All 5 test files silently excluded from npm test (subdir blind spot)
230 Next Integration Branch Proposed 5 partial partial partial covered Most critical seams untested; 2 files misattributed (unrelated features)
415 Stale-Base Token Reintro Accepted 12 covered partial partial covered Token reintro guarded; propagator + ruleset-strict-mode untested
443 Unified Effort / Fast-Mode Proposed 12 covered covered covered partial Resolver exceptionally covered; end-to-end effort propagation untested
452 ESLint Lint Harness Accepted 2 partial partial partial covered Local rules covered; no-only-tests + plugin-n + script-retirement untested
456 Test Rigor Architecture Accepted 19 covered partial partial covered Clock seam strong; mutation breadth + warn→error + allow-rule tracking gaps
457 Generated CJS Single Source Accepted 8 covered partial partial covered Value-baking solid; .gitignore↔eslint sync + emit-path gaps
550 Spec-Phase Probe Contract Accepted 6 covered covered covered covered Core seam strong; Decisions 4(b–d)/6 deferred to #644 (untested by design)
0656 Research Module Seam Accepted 9 covered covered covered covered Strong; slopcheck "cannot-lower" + scrape-dispatch gaps
660 Release From Next Head Proposed 13 partial partial covered covered Scripts covered; orchestration (release.yml/auto-backmerge) largely untested
766 Claude Plugin Manifest Accepted 3 covered partial partial partial agents-absence + post-#770 event drift untested; CI-skipped schema validate
894 Capability Decl Format Proposed (coupled 857) 5 covered partial covered partial Provisional. Strong; when-ownership, tier-monotone std→full gaps
959 Capability Command Contribution Proposed (coupled 857) 6 covered partial covered covered Provisional. Strong; proto-pollution family-name + case-removal gaps
1016 Runtime Capability Descriptor Proposed (coupled 857) 9 covered partial covered covered Provisional. Dense; sandboxTier fail-loud + 5f required-axes gaps
1143 Claude Orchestration Proposed (coupled 857) 9 missing missing missing na Provisional. No impl; primary surface entirely untested
3524 CJS-SDK Hard Seam Superseded (0174) 12 partial partial covered covered Drift-bugs guarded; all subjects gitignored artifacts (crash on clean worktree)
3660 Runtime Artifact Layout Accepted 6 covered partial partial covered Core protected; findInstallSourceRoot fail-path + agent tools:-strip gaps

ADRs with zero tests or notable gaps

ADR-218 (Release Version Validation) — ZERO attributed tests. This is the most acute hole. Both ADR decisions — the leading-zero-rejecting format regex (^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.0$) and the npm duplicate-version precheck — live exclusively in a bash run: block in .github/workflows/release.yml. No test exercises either. The 1.01.0 incident that motivated the ADR caused an irrecoverable production divergence, and nothing prevents a reversion to the old broken regex. issue-version-gate.test.cjs is a superficially-related but wholly separate feature. This is a critical gap in a high-blast-radius zone.

ADR-227 (Input Validation Shape) — 5 tests that never run. All five attributed files live in tests/observability/ and tests/dispatch/, which scripts/run-tests.cjs:269 never enters (flat readdirSync). The tests are individually high quality (8 boundary/negative UUID-v4 cases, real file I/O, per-test isolation) but are silently excluded from npm test, CI, and gsd-test. ADR-227's two-layer validation guarantee — the correlation-poisoning and log-blowup classes it was written to prevent — is effectively unguarded until the runner is fixed.

ADR-0006 (Planning-Path Projection) — critical handler-consumption gap. The ADR's single most important invariant — that cmdInitExecutePhase/PlanPhase/PhaseOp/MilestoneOp route through planningPaths().planning rather than flat path.join(cwd, '.planning') — has no regression test. The banned .planning/projects/<project> form also has no guard.

ADR-15 (Autonomous Cross-AI Convergence) — primary surface untested. The ADR's designated primary operator surface, /gsd-progress --next --auto --converge, is both unimplemented and untested; the existing cross-ai-execution.test.cjs tests a different feature (execute-phase delegation) and is misattributed.

ADR-0011a / 0011b / 0011c (Skill Surface / Review Default Reviewers). Source modules (install-profiles.cjs, surface.cjs, clusters.cjs, review-reviewer-selection.cjs, config-schema.cjs) are gitignored build artifacts absent from a clean worktree; all attributed tests throw MODULE_NOT_FOUND without a prior npm run build:lib. CI's pretest hook covers this, but isolated runs and fresh worktrees fail confusingly.

ADR-1143 (Claude Orchestration) — provisional, no implementation. Proposed and blocked on ADR-857; the capability directory does not exist, so the primary surface has zero coverage. Findings are provisional pending ADR-857.


Weak tests

Retire

Only entries whose adversarial verification confirmed finalVerdict === "retire". 5 tests survived verification as truly worthless.

File Location Why worthless Confirmed by verification
tests/enh-2790-skill-consolidation.test.cjs Lines 127–157 — 4 "has a name: field" spot-checks Assert only fm.name.length > 0; command-contract.test.cjs already enforces the stricter /^gsd[:-]/ prefix on all 67 files. Renaming gsd-capture→capture (the exact regression) passes these but is caught by command-contract. Cannot go red for the bug class the ADR exists to prevent. Yes — verified command-contract.test.cjs:44-54 covers these 4 files more strictly; deletion creates no coverage gap.
tests/command-routing-hub.test.cjs Lines 134–137 — "constructs successfully with only cjsRegistry" Duplicate of the construction test at 71–75; asserts only typeof hub.dispatch === 'function' with no dispatch. createHub does zero registry-dependent branching at construction. Yes — verified identical assertion; populated-registry dispatch path covered by the happy-path block at line 142.
tests/command-routing-hub.test.cjs Lines 515–519 — "ERROR_KINDS.UnknownCommand === result.kind" The identical assertion appears at 9 other sites (222, 234, 245, 256, 360, 393, 410, 422, 518). Same code path covered at 414–424. No unique setup/edge/field. Yes — verified superset coverage by lines 384–424 + the constant-stability test at 521–526.
tests/no-cjs-sdk-handsync-tooling.test.cjs All 3 tests (lines 26–53) Assert absence of lint-shared-module-handsync.cjs / allowlist JSON that never existed on main (CONTEXT.md:693 confirms). Can only go red if someone manually drops those exact files. No behavioral coverage. Yes — verified the real retired artifacts (cjs-sdk-bridge.cjs, sdk/) are covered by bug-190; these guard ADR-text-only artifacts.
tests/runtime-artifact-layout.test.cjs Lines 319–323 — "cline has one skills kind (#782)" Default-scope call is behaviorally identical to the explicit-global test at 249–265, which asserts a strict superset (kinds.length, kind, destSubpath, prefix, stage). Yes — verified default scope = global (source line 430); also independently covered by bug-782-cline-skills-emission.test.cjs:629-636.

Redesign / improve

All redesign + keep-but-improve verdicts, plus the 6 retire-candidates that verification downgraded to redesign. (Representative selection across ADRs; the full set is large — these are the highest-signal entries and every downgraded retire-candidate.)

Downgraded retire-candidates (verification overruled the retire verdict — do NOT delete)

File Location Problem What the rewritten test should assert
tests/bug-3135-capture-backlog-workflow.test.cjs Lines 136–182 broad @-ref regression The block IS redundant (only checks /workflows/ refs, misses 24 files with references//templates/ refs that command-contract Rule 4 catches) — delete ONLY that block. But the FILE's lines 1–132 test unique add-backlog.md content invariants with no equivalent elsewhere. Delete lines 134–182 only. Retain content-contract assertions: add-backlog.md uses phase.next-decimal, writes ROADMAP before mkdir, 999.x scheme; capture.md routes to add-backlog.md.
tests/review-reviewer-selection.test.cjs Lines 115–122 result-shape test Shape-only on { detected: [] }. But the empty-detected call exercises the no_config_all_detected else-branch that no other test asserts (existing test uses ['gemini']). Replace shape assertions with behavioral: source === 'no_config_all_detected', selected/errors/warnings === [] for the empty-detected fallback.
tests/review-reviewer-selection.test.cjs Lines 17–27 KNOWN_REVIEWER_SLUGS Tests a compiled-CJS constant's shape. Verification found the slug membership DOES drive resolveReviewerSelection's warn/drop logic at runtime — it's the only guard against silent slug-list drift. Collapse to one behavioral test: resolveReviewerSelection with a config'd known slug yields no warning + slug in selected; an unknown slug yields a warning.
tests/enh-191-retire-sdk-package.test.cjs Lines 52–58 AGENTS.md absence Off-topic for SDK retirement, but NOT valueless: bin/install.js:~10963 writes a root AGENTS.md on local Copilot install; this is the only guard against that artifact committing. Extract to a repo-layout governance test (e.g. tests/repo-layout.test.cjs); rename to state the real invariant (no ad-hoc root instruction file vs CONTEXT.md/ADRs); link the copilot install path.
tests/capability-registry.test.cjs Lines 558–584 drift test Steps 2–3 are a file-I/O tautology and skip the real --check path (stripGeneratedComment + normalizeLineEndings). But it is the only unit-level coverage of the drift-comparison pipeline. Call the actual comparison expression: assert a stale version survives both stripGeneratedComment and normalizeLineEndings and is detected; assert a comment-only-timestamp change is NOT flagged after stripping.
tests/verify-test-quality.test.cjs 3 describes (disabled / circular / assertion-strength detection) Self-referential: define a regex inline, write a matching fixture, assert the regex matches — cannot go red for any production change. The detector lives as prose in verify-phase (workflow, retired in #1892). Replace with a structural guard reading gsd-core/workflows/verify-phase (retired in #1892): assert the audit_test_quality step tag exists and contains the skip-pattern, circular-detection, and assertion-strength-table markers.

Other redesign / keep-but-improve (high-signal selection)

File Location Problem What the rewritten test should assert
tests/research-cli.test.cjs Line 423 Pass-always assert.ok(true) placeholder; the 2-package arg-parse regression is undefended. Extract the flag parser; call with ['--bad-flag','pkgA','pkgB'] and assert unknown-flag error + that pkgA/pkgB are not consumed as flag values. (No network.)
tests/worktree-baseref-install.test.cjs Line 444 Pass-always assert.ok(true); the null-settings #683 guard is never exercised. assert.doesNotThrow(() => applyWorktreeBaseRef(null, tmpDir)) and assert no worktree.baseRef is written.
tests/bug-260-worktree-path-guard.test.cjs Line 338 Environment-dependent branch silently passes via assert.ok(true) when the traversal path resolves inside the worktree — a security guard goes untested on macOS/private-symlink hosts. Construct the traversal target in a separate tmpdir outside the worktree; remove the if/else; unconditionally assert result.status === 2 / decision block.
tests/eslint-rules.test.cjs 24 trailing assert.ok(true) calls RuleTester already throws on failure; the trailing literal asserts nothing and inflates count. Delete the assert.ok(true) lines; optionally add assert.strictEqual(typeof rule.create, 'function') per describe to confirm the module loaded.
tests/phase.test.cjs Lines 4343–4368 (last_updated refresh) Math.abs(new Date() - updatedAt) < 60_000 wall-clock window — flaky under loaded CI. Assert last_updated is a valid ISO timestamp ≠ the stale seed value AND its date portion equals today's UTC date.
tests/bug-3707-locked-worktree-cleanup.test.cjs 7 Date.now() stale-mtime sites Real-clock coupling for stale-lock boundary; violates the ADR-456 clock-seam mandate. Pin a constant epoch (PINNED_NOW_MS) for mtime fixtures; pass it to the SUT as a clock parameter where supported.
tests/context-utilization.property.test.cjs Line 187 Math.floor(Math.random()*…) inside fc.property breaks reproducibility under the pinned seed. Replace with fc.integer({ min: 0, max: contextWindow }); split a shape test from a value-correctness test at known ratios.
tests/research-provider.property.test.cjs Single test Only "never throws + valid type"; inverting HIGH/LOW classification passes. Keep robustness property; add example-based unit tests for classification boundaries; add module to mutation COVERED.
tests/research-store.property.test.cjs All 3 tests Shape/determinism only; cache-key collisions or over-normalization invisible. Add a collision property (distinct inputs → distinct keys) + boundary examples (npm/lodash ≠ npm/react).
tests/install.test.cjs Lines 683–712 (Kilo source-grep) Reads bin/install.js as a string and .includes() internal symbol names — source-grep, breaks on rename, passes on dead code. Invoke install.js with --kilo in a tmpdir; assert on the installed config/manifest content (or export+call the kilo-detection logic).
tests/sh-hook-paths.test.cjs Tests 2–4 (lines 119–194) Source-grep variable-name scans of install.js/runtime-hooks-surface.cts; behavior already covered by Test 1's behavioral call. Call buildHookCommand per .sh hook for linux+win32; assert non-empty, bash runner, absolute quoted configDir prefix.
tests/clusters.test.cjs Lines 50–58 `assert.ok(utilitySize >= skills.length
tests/model-profiles.test.cjs Lines 115–122 Test titled "resolution order: override > profile > default" exercises no override and asserts nothing about override behavior. Use a temp project with config.model_overrides, call resolveModelInternal, assert override > profile > default explicitly.
tests/active-workstream-store.unit.test.cjs Lines 361–364 `assert.ok(key === null
tests/feat-488-effort-sync.test.cjs Lines 177–219 Does not redirect GSD_HOME; resolved effort depends on the dev's real ~/.gsd; degrades to a type-only check. Set GSD_HOME=tmpHome; assert result.synced === 1 and changes[0].to === 'low'.
tests/bug-492-effort-manifest-fallback.test.cjs Lines 20–48 Mutates the module-level CANONICAL_CONFIG_DEFAULTS singleton — independence violation; races under parallel runs. Drive the same path via a tmpHome ~/.gsd/defaults.json rather than mutating shared module state.
tests/issue-766-plugin-manifest.test.cjs Section C, "claude plugin validate --strict" Permanently skipped when the claude binary is absent (i.e. on CI) — the only full-schema gate never runs. Install the claude CLI in a CI step, OR validate plugin.json against a snapshotted JSON schema unconditionally.
tests/cross-ai-execution.test.cjs Entire file Misattributed to ADR-15; tests a different feature (workflow.cross_ai_execution). Also a shared-mutable let content independence issue. Leave the file for its real feature; create tests/adr-15-plan-strategy-seam.test.cjs for the actual converge seam; hoist content to a single module-level read.
tests/issue-844-manifest-version-sync.test.cjs Describe A (33–127) Order-dependent shared tmpRoot across setup/assert/cleanup tests. Refactor to a single test with per-test before()/after() tmpRoot allocation.
tests/plan-review-convergence.test.cjs Lines 688–692 Four workflow.includes(verdict) whole-file substring checks — pass even if verdicts move to dead/comment text. Extract the source-grounding section slice; assert each verdict + its severity mapping (AMBIGUOUS→MEDIUM, UNCHECKABLE→INFO) inside that slice.
tests/feat-3594-parser-property-style.test.cjs Lines 119–135 assert.ok(elapsedMs < 2000) wall-clock; escapes no-elapsed-assertion via the variable name. Replace with a call-count metric (instrument char comparisons, assert O(n)); or raise bound to 10000ms as a stopgap.
tests/core.test.cjs Lines 1834–1879 (reapStaleTempFiles) Writes to the global shared os.tmpdir()/gsd; races with temp-subdir.test.cjs under concurrency. Use a per-test isolated tmpdir via the existing { dir } option; clean up in afterEach.

Prioritized action plan

P0 — do first (blast radius / false-green)

  1. Fix recursive test discovery in scripts/run-tests.cjs (replace flat readdirSync with the already-present findTestFilesRecursive). This alone reactivates ADR-227's entire UUID-validation suite (tests/observability/, tests/dispatch/) — currently silently excluded from npm test and CI.
  2. Raise Stryker break from 50 → 80 in stryker.config.mjs (ADR-456 mandate). Move any module that cannot meet 80% into UNMUTATED with a documented per-entry reason rather than diluting the global floor.
  3. Add tests for ADR-218 (zero coverage today): extract the release-version regex and npm-duplicate precheck into a testable .cjs helper (or a source-text-is-the-product workflow assertion on release.yml), and assert 1.01.0/01.0.0 are rejected, valid versions pass, and the duplicate-version step exists and is wired before the publish jobs.
  4. Retire the 5 verified-worthless tests (see Retire table): enh-2790 name spot-checks (127–157), command-routing-hub lines 134–137 and 515–519, no-cjs-sdk-handsync-tooling (all 3), runtime-artifact-layout cline edge-case (319–323).
  5. Redesign the 6 downgraded retire-candidates (do NOT delete): bug-3135 (delete only the broad block, keep content tests), both review-reviewer-selection tests, enh-191 AGENTS.md (extract to governance test), capability-registry drift test, verify-test-quality (replace with structural workflow guard).
  6. Close the ADR-0006 critical gap: add a regression test that init execute-phase/plan-phase/phase-op/milestone-op return workstream-scoped paths under GSD_WORKSTREAM, plus a guard against the banned .planning/projects/<project> form.

P1 — high priority (rigor enforcement + concentrated weakness)

  1. Bring core.cts and config-loader.cts into the Stryker COVERED set (smallest high-value modules first); write their *.unit.test.cjs as a prerequisite.
  2. Promote no-source-grep to error and apply it to tests/**, splitting product-file reads (.md/.json, allowed) from source-file grep (.cjs/.js/.ts, blocked). Then fix the surfaced violations (install.test.cjs Kilo block, sh-hook-paths.test.cjs Tests 2–4).
  3. Add the allow-test-rule tracking-issue lint gate + run the one-time sweep to attach #NNN (or an approved category tag) to the 332 untracked exemptions.
  4. Eliminate the pass-always placeholders: redesign research-cli.test.cjs:423, worktree-baseref-install.test.cjs:444, the bug-260 environment-dependent branch, and strip the 24 assert.ok(true) lines in eslint-rules.test.cjs. Add a no-tautological-assert ESLint rule to prevent recurrence.
  5. Add no-only-tests RuleTester coverage and extend no-elapsed-assertion to catch Date.now()-start and computed-variable (elapsedMs) patterns.
  6. Add a branch-coverage floor (--branches 60) and a per-file branch floor for the UNMUTATED critical modules.
  7. De-flake the clock-coupled tests: phase.test.cjs:4362 (60s window → ISO-date assertion), bug-3707 (pin epoch), context-utilization.property.test.cjs:187 (fc.integer not Math.random).
  8. Add the ADR-15 primary-surface tests (/gsd-progress --next --auto --converge) and re-attribute cross-ai-execution.test.cjs to its real feature.

P2 — follow-up (robustness & hygiene)

  1. Fail-on-zero-executed in run-tests.cjs (exit 1 on empty suite by default; gate the allowance behind an env flag).
  2. Isolate the shared GSD_TEMP_DIR reap tests (core.test.cjs:1834-1879, temp-subdir.test.cjs) via the existing { dir } option + afterEach cleanup.
  3. Add build-artifact before() guards to artifact-dependent suites (ADR-3524 subjects, ADR-0011 family) so a clean worktree fails with a clear "run npm run build:lib" message instead of a top-level crash.
  4. Strengthen the strong-but-incomplete ADRs: ADR-443 end-to-end effort propagation, ADR-0656 slopcheck "cannot-lower" + scrape-dispatch, ADR-0009's 3 absent ADR-listed test files, ADR-766 agents-absence assertion + post-#770 hook drift.
  5. Add .gitignore ↔ eslint.config.mjs ignore-list parity test (ADR-457) so a migration that updates one list but not the other is caught.

Capability-cluster items (ADR-894/959/1016/1143) are deliberately deferred — they are coupled to ADR-857 (worked elsewhere) and their findings are provisional. Re-run this audit slice once ADR-857 lands.