golem15/msd-core
3
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

fix: recover silently-excluded test dirs + test-architecture audit hardening (#1195)

Browse Source 
* fix: recurse test discovery so subdir test suites actually run

scripts/run-tests.cjs discovered tests with a flat readdirSync(testDir),
silently excluding tests/observability/ (4 files), tests/dispatch/ (1) and
tests/installer-migrations/ (1) — 94 passing tests — from `npm test` and all
CI lanes. Walk the tree recursively (relative subpaths preserved), classify
suites by basename, and add a fail-on-zero-executed guard for suite/default
runs (escape hatch GSD_ALLOW_EMPTY_SUITE=1) while preserving the empty
--files/--files-from path the CI inert lane relies on.

Unit suite 735 -> 741 files; surfaces ADR-227's observability/dispatch seam.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: retire 5 verified-worthless tests

Adversarial verification confirmed these 5 prove nothing — their coverage is
provided more strictly elsewhere:
- enh-2790 'has a name: field' spot-checks (command-contract enforces /^gsd[:-]/)
- command-routing-hub duplicate construct + duplicate ERROR_KINDS assertions
- no-cjs-sdk-handsync-tooling (guarded files that never existed on main; bug-190
  covers the real retired SDK artifacts)
- runtime-artifact-layout cline edge case (subsumed by the explicit-global test
  and bug-782-cline-skills-emission)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: add ADR-218 release version-validation coverage

ADR-218 (reject leading-zero versions like 1.01.0; npm duplicate pre-check) had
zero tests — the logic lived only in release.yml bash. Add a test that extracts
the actual rejection regexes from the workflow and exercises them against a
boundary table (leading-zero/malformed rejected, valid accepted) plus structural
wiring assertions. Goes red if the regex is reverted to [0-9]+.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: redesign weak tests into behavioral, deterministic assertions

Per the ADR test audit, rewrite 27 weak test files (test-only, no source
changes) so each can go red for the defect it guards:
- kill pass-always assert.ok(true) placeholders (research-cli, worktree-baseref,
  bug-260 security guard, eslint-rules x24, clusters '|| true')
- replace source-text grep with behavioral calls (install Kilo, sh-hook-paths,
  plan-review-convergence) and add a repo-layout governance test
- de-flake real-clock/Math.random coupling (phase last_updated, bug-3707 mtime,
  context-utilization property, feat-3594)
- fix independence/shared-state violations (bug-492 singleton, issue-844 tmpRoot,
  core reapStaleTempFiles, active-workstream TTY, feat-488 GSD_HOME)
- strengthen property/shape-only tests (research-provider/store classification +
  collision) and unconditional plugin.json schema validation (issue-766)

Verified: all 28 files run together 1220 pass / 0 fail / 1 skip.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: add no-tautological-assert lint rule, error in test suite

New custom ESLint rule (eslint-rules/no-tautological-assert.cjs) bans asserts
that can never fail: assert(true)/assert.ok(<always-truthy literal>),
'cond || true' inside an assert, and equality asserts comparing two identical
literals. Wired as error on tests/**; full sweep confirmed zero existing
violations so the suite stays green. Prevents the placeholder-assert regressions
the audit redesigns just removed. RuleTester coverage added (6 valid, 8 invalid).

Note: no-only-tests was already enforced via eslint-plugin-no-only-tests, so no
duplicate rule was added.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: gate new allow-test-rule exemptions to require an issue ref

ADR-456 requires any allow-test-rule exemption added after the ADR to carry a
tracking issue number, but nothing enforced it. New ratchet gate
(scripts/lint-allow-test-rule-refs.cjs, wired into lint:ci) fails when a NEW
allow-test-rule comment lacks a #NNN/URL reference; the 323 existing untracked
exemptions are grandfathered in an allowlist that ratchets down as they gain
refs. Red-green verified (novel untracked offender fails; compliant passes).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: add ADR test-audit evidence report (#1192)

Full risk-first qa-test-architect audit of the ADR portfolio (37 ADRs + 4
platform lenses, adversarial verification of retire verdicts) that drove the
P0 discovery fix, ADR-218 coverage, 5 retires, 27 redesigns, and the two new
lint gates. Filed as point-in-time evidence under docs/issueevidence/, named
for tracking issue #1192.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: replace pre-existing raw NUL byte with escape in feat-3594 fixture

feat-3594's null-byte parser fixture contained a literal NUL byte (pre-existing
on next at b10e5681 — confirmed: base blob has 1 NUL, this fix has 0), which
made git treat the file as binary and would break grep/editors. Switch to the
\x00 escape; the runtime string value (a real NUL in the parser input) is
unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: address adversarial-review findings

Codex adversarial pass over the branch:
- capability-registry drift test no longer mutates the committed generated
  capability-registry.cjs in place (concurrency hazard) — uses in-memory
  checkPipeline comparison instead.
- allow-test-rule ratchet now detects exemptions in ALL comment forms (block
  /* */ too, matching no-source-grep) so a block comment can't bypass it;
  one newly-surfaced pre-existing offender grandfathered (323->324).
- install.test Kilo case asserts on what install(false,'kilo') actually writes
  rather than manually calling configureKiloPermissions (masked the call site).
- issue-766 drops the undeclared transitive ajv dep for explicit structural
  assertions from the schema fixture.
- adr-218 test notes the hotfix leading-zero gap is tracked in #1186.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: address code-review findings (subdir discovery, rule + test gaps)

xhigh code review surfaced 15 confirmed issues, all fixed:
- run-tests.cjs --files now resolves subdir tests by bare basename + handles
  Windows backslash paths (ambiguous basenames error clearly).
- affected-tests-lib.cjs listTestFiles made recursive — the targeted CI lane was
  silently dropping changed subdir tests (same false-green class the audit fixed).
- no-tautological-assert now catches 'true || cond' and empty []/{}  equality.
- verify-test-quality: restore provenance-classification coverage, tighten the
  writeFile circular-detection check, guard the module-level file read.
- sh-hook-paths: cover the global-install .sh delegation branch (#2045 guard).
- active-workstream null-guard runs deterministically (no longer skipped on TTY).
- adr-218 structural guards tightened (major/minor leading-zero; needs: membership).
- repo-layout AGENTS.md guard no longer false-alarms on equivalent refactors.
- cross-ai ordering guard fails red when the step is missing.
- issue-766 parses required fields from the schema fixture (auto-enforced).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test: stub USERPROFILE alongside HOME in feat-488 (Windows parity)

The feat-488 redesign stubbed process.env.HOME but not USERPROFILE; os.homedir()
resolves from USERPROFILE on Windows, so the home stub was not hermetic there —
caught by windows-test-parity-guard (stubsHomeNoUserProfile). Save/set/restore
USERPROFILE symmetrically with HOME (delete-if-originally-undefined).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: reconcile allow-test-rule allowlist after rebase onto next

Rebasing onto current next pulled in merged PR #1170, which added
inventory-headings-countfree.test.cjs (a baseline allow-test-rule exemption) and
deleted inventory-counts.test.cjs. Grandfather the former and prune the latter so
the ratchet matches the merged tree. No new debt from this PR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-06-13 23:35:08 -04:00
committed by GitHub
parent 3ebd13c45d
commit 5fa4dcd78c
44 changed files with 3958 additions and 757 deletions
Download Patch File Download Diff File Expand all files Collapse all files

223
docs/issueevidence/1192-adr-test-audit-2026-06-13.md Normal file
View File

@@ -0,0 +1,223 @@
# ADR Test-Coverage Audit — 2026-06-13
Lead QA architect final report. Scope: 38 ADRs in `docs/adr/`, evaluated with the qa-test-architect methodology, plus 4 platform-capability lenses over the test infrastructure. Every RETIRE verdict was independently re-checked by an adversarial verification pass.
> **Scope note.** ADR-857 (capability system core) was intentionally excluded from this audit — it is being worked by another instance. The capability cluster (ADR-894, ADR-959, ADR-1016, ADR-1143) is *coupled to* ADR-857, so all findings for those four ADRs are **provisional** and may shift once ADR-857 lands.
---
## Executive summary
The test portfolio is, on the whole, healthier than its volume suggests: the deterministic seams (dispatch hub, model catalog, planning-path resolution, installer migration, shell-command projection) are genuinely behavioral and well-covered, and the project has built real test-rigor infrastructure (an injectable clock seam, fast-check with a pinned seed, Stryker mutation testing wired into CI, and four custom ESLint rules). The dominant problem is not absent tests — it is **enforcement gaps that let the rigor infrastructure run as theater**, and a long tail of **weak tests concentrated in three families**: source-text-grep assertions, tautological `assert.ok(true)` placeholders, and architecture-retirement guards that can never go red.
Five headline problems demand attention first: (1) **Stryker mutation testing covers only ~4–6% of the lib surface** and excludes every high-risk module (state, phase, verify, init, commands), with the break threshold set to 50% instead of the ADR-456-mandated 80% — the mutation gate trivially passes for PRs touching those modules. (2) **332 of 335 `allow-test-rule` exemptions carry no tracking issue**, directly violating ADR-456's delete-bad-tests policy. (3) **A subdirectory blind spot in `scripts/run-tests.cjs` silently excludes all of `tests/observability/` and `tests/dispatch/` from `npm test`** — meaning ADR-227's entire UUID-validation seam is unguarded by any continuously-executed test. (4) **Multiple ADRs marked Accepted or Proposed have zero attributed tests or test files that cannot run** because their source modules are gitignored build artifacts absent from a clean worktree (ADR-218, ADR-0011 family). (5) A scatter of **vacuous and self-referential tests** (`verify-test-quality.test.cjs`, `assert.ok(true)` placeholders, environment-dependent path-guard branches) that provide false confidence on security-relevant code.
**Do first:** raise the Stryker break threshold and bring `core.cts`/`config-loader.cts` into the mutation-covered set (P0); fix the `run-tests.cjs` non-recursive file scan so observability/dispatch tests actually execute (P0); retire the 5 verified-worthless tests and redesign the 6 downgraded retire-candidates (P0/P1).
---
## Testing-platform robustness
Synthesis of the four platform lenses: `adr-456-test-rigor`, `coverage-mutation`, `anti-pattern-sweep`, `determinism-flake`.
### (a) What the platform CAN do today
- **Deterministic clock seam** (`tests/helpers/clock.cjs`, `makeFakeClock`) — well-designed and correctly used by lock/state tests (`clock-seam.test.cjs`, `perf-407`, `bug-474`). Eliminates wall-clock races in the lock subsystem.
- **Property-based testing** via fast-check with a pinned seed (`tests/helpers/fast-check-setup.cjs`, seed=42) — 9 `*.property.test.cjs` files exist and run reproducibly.
- **Mutation testing** — Stryker is configured (`stryker.config.mjs`) and wired into a PR-gating workflow (`.github/workflows/mutation.yml`) that shards per changed module.
- **Custom ESLint rules** — `no-source-grep`, `no-magic-sleep-in-tests`, `no-elapsed-assertion`, `no-raw-rmsync-in-tests`, plus `no-only-tests` — exist and are unit-tested via RuleTester (`tests/eslint-rules.test.cjs`).
- **Env hermeticity** — `scripts/run-tests.cjs` deletes `GSD_PROJECT`/`GSD_WORKSTREAM` before spawning child test processes (mitigates the ambient-env hazard recorded in MEMORY.md).
- **Line-coverage gate** — `c8 --lines 70` (lib) enforced in CI as part of the unit lane.
### (b) ENFORCED vs aspirational
| Mechanism | Status | Note |
|---|---|---|
| Clock seam in lock/state tests | **Enforced** | Used correctly; but not mandated by lint — `bug-3707` and `phase.test.cjs:4362` still use real `Date.now()` |
| fast-check seed pinned | **Enforced** | seed=42; but no lint rule requires `*.property.test.cjs` to import the setup, and `context-utilization.property.test.cjs` uses `Math.random()` inside `fc.property`, breaking reproducibility |
| Stryker mutation gate | **Aspirational** | Runs in CI, but break=50% (not 80%) and covers 6/109 modules; trivially passes for excluded modules |
| Line coverage 70% | **Enforced** | But satisfiable via CLI integration tests that execute lines without asserting sub-behaviors; no branch-coverage floor |
| `no-magic-sleep-in-tests` | **Enforced** (`error`) | Promoted to error |
| `no-raw-rmsync-in-tests` | **Enforced** (`error`) | |
| `no-only-tests` | **Aspirational** | Wired at `error` but has zero RuleTester coverage and no integration test |
| `no-source-grep` | **Aspirational** | At `warn` only, and NOT applied to `tests/**` at all — the file-level `allow-test-rule` bypass blanket-exempts source-grep in test files |
| `no-elapsed-assertion` | **Aspirational** | At `warn`; only matches `.elapsed/.duration/.took/.ms` properties, missing `Date.now()-start` and computed-variable patterns (e.g. `elapsedMs`) |
| `allow-test-rule` tracking-issue requirement | **Not enforced** | 332/335 exemptions lack an issue number; no CI gate prevents new untracked exemptions |
| Subdirectory test discovery | **Broken** | `run-tests.cjs` uses flat `readdirSync(testDir)`; `tests/observability/` and `tests/dispatch/` are silently never run by `npm test` |
| Fail-on-zero-executed | **Not enforced** | `run-tests.cjs` exits 0 when a suite resolves to zero files |
### (c) Prioritized capability upgrades
| Capability | Current state | Recommendation | Priority |
|---|---|---|---|
| Recursive test discovery | Flat `readdirSync` skips `tests/observability/`, `tests/dispatch/` | Replace with the existing `findTestFilesRecursive` (already in the same file) | **P0** |
| Stryker break threshold | `break: 50` in `stryker.config.mjs` | Raise to `80` per ADR-456; move sub-80% modules to UNMUTATED with documented reasons | **P0** |
| Mutation coverage breadth | 6/109 modules covered; 12 critical modules excluded | Graduate one module/release cycle starting with `core.cts`, `config-loader.cts`; require any module with a `*.unit/property.test.cjs` to be in COVERED or UNMUTATED | **P0** |
| `allow-test-rule` audit | 332/335 untracked exemptions | Lint gate requiring either an approved category tag or `#NNN`; one-time tracking-issue sweep | **P1** |
| `no-source-grep` in test files | Not applied to `tests/**`; warn-only | Apply at `error` to `tests/**`; split rule to allow product-file (`.md/.json`) reads but block source-file (`.cjs/.js/.ts`) grep assertions | **P1** |
| Branch-coverage floor | Only line coverage gated | Add `--branches 60`; per-file branch floor for the UNMUTATED critical modules | **P1** |
| `no-elapsed-assertion` reach | Property-name match only | Extend to `Date.now()-start` binary expressions and identifiers ending `Ms/Elapsed/Duration` in assert args | **P1** |
| `no-only-tests` coverage | Wired `error`, untested | Add RuleTester valid/invalid cases + an integration assertion it fires | **P1** |
| `no-tautological-assert` rule | None | New ~30-line local rule flagging `assert.ok(true)`/`assert.ok(1)` in `tests/**` | **P1** |
| Fail-on-zero-executed | Exits 0 on empty suite | Exit 1 by default; gate the empty-suite allowance behind `RUN_TESTS_ALLOW_EMPTY_SUITE=1` | **P2** |
| Shared-`GSD_TEMP_DIR` isolation | reap tests write to global `os.tmpdir()/gsd` | Redirect via the existing `dir` option to per-test tmpdirs; lint against writing to the shared path without cleanup | **P2** |
| Build-artifact preflight in artifact-dependent tests | Top-level `require()` of gitignored `.cjs` hard-crashes on clean worktree | `before()` `existsSync` guard with a clear "run npm run build:lib" message | **P2** |
### ADR-456 compliance — per mandate
ADR-456 defines four test-rigor mandates. Compliance is **partial on all four**:
- **(a) Concurrency via injectable clock seam, not real OS races** — *Substantially met.* The clock seam exists and the lock subsystem uses it. **Gaps:** `bug-3707-locked-worktree-cleanup.test.cjs` computes stale-lock mtimes from real `Date.now()` (7 sites); `phase.test.cjs:4362` asserts a 60-second wall-clock window; `core.test.cjs` `timeAgo` boundary tests use real clock at the 5s bucket edge. No lint rule mandates the seam.
- **(b) fast-check + Stryker antagonistic tier, 80% kill threshold** — *Largely aspirational.* fast-check is wired with a pinned seed, but Stryker covers ~4–6% of source lines, excludes all high-risk modules, and the break threshold is **50%, not the mandated 80%**. The antagonistic tier exists in name but does not guard the dangerous surface.
- **(c) Assert on typed `--json` fields and exported registries, not rendered text or source literals** — *Partially met, weakly enforced.* New tests largely follow it, but `no-source-grep` is `warn`-only and not applied to test files, so source-grep assertions persist (e.g. `install.test.cjs` `.includes()` on `bin/install.js`; `sh-hook-paths.test.cjs` variable-name scans). Several CLI tests still assert on `stdout.includes()` substrings rather than parsed JSON.
- **(d) Delete pass-always / vacuous / source-grep / elapsed / real-race / permanent-allow-rule tests** — *Not met.* Concrete pass-always tests survive (`research-cli.test.cjs:423`, `worktree-baseref-install.test.cjs:444`, 24 `assert.ok(true)` in `eslint-rules.test.cjs`); 332/335 `allow-test-rule` exemptions are effectively permanent and untracked, directly contradicting the policy clause requiring `// allow-test-rule: see #NNN` on post-ADR exemptions.
---
## Per-ADR coverage
| ADR | Status | #Tests | Happy | Boundary | Negative | Independence | Verdict |
|---|---|---|---|---|---|---|---|
| 0001 Dispatch Policy Module | Accepted | 12 | covered | partial | covered | covered | Core protected; SDK bridge / exit-code mapping unverified |
| 0002 Command Contract Validation | Accepted | 6 | covered | partial | missing | covered | Integration-strong; helper unit tests + lint negatives missing |
| 0003 Model Catalog Module | Accepted | 13 | covered | partial | covered | covered | CJS solid; SDK-parity invariant untested |
| 0004 Worktree/Workstream Seam | Accepted | 13 | covered | partial | covered | covered | Substantially protected; cross-path naming + .planning/.lock 2-proc gaps |
| 0005 SDK Architecture Seam Map | Superseded (0174) | 13 | partial | partial | partial | covered | Retirement guards OK; thin-adapter enforcement absent |
| 0006 Planning-Path Projection | Accepted | 6 | partial | partial | covered | covered | **Critical gap:** init-handler `planningPaths()` consumption untested |
| 0007 SDK Package Seam | Superseded (0174) | 10 | covered | partial | partial | covered | Retirement guarded; `classifyPromptUserAction` boundary untested |
| 0008 Installer Migration Module | Accepted | 16 | covered | partial | covered | covered | Rich suite; rewrite-json apply + local-scope gaps |
| 0009 Shell Command Projection | Accepted | 21 | covered | partial | covered | covered | Strong; 3 ADR-listed test files absent; source-grep in `sh-hook-paths` |
| 0010a Skill Surface Budget | Proposed (impl w/ gaps) | 8 | covered | partial | partial | covered | ~75% protected; `stageCommandsForRuntimeFlat` + migration one-shot untested |
| 0010b File Operation Engine | Superseded (0009) | 10 | covered | partial | covered | covered | Canonical seam solid; divergent `atomicWriteFileSync` + path-containment gaps |
| 0011a Skill Surface Budget | Accepted | 12 | covered | partial | partial | covered | Broad; **source modules absent from worktree → MODULE_NOT_FOUND**; cyclic-requires untested |
| 0011b Review Default Reviewers (PRD) | Draft | 5 | partial | partial | partial | covered | infos-path + `--all`+flags edge untested; build-artifact dependency |
| 0011c Review Default Reviewers | Proposed | 4 | covered | partial | partial | covered | Mid-tier risks (infos, empty-selection) missing; build-artifact dependency |
| 0012 Command Routing Hub | Superseded (0174) | 5 | covered | partial | covered | covered | Hub well-covered; `uat-passed` adapter arg-shape gap; 3 redundant tests |
| 15 Autonomous Cross-AI Convergence | Proposed | 8 | partial | partial | partial | partial | **Primary surface (`/gsd-progress --next --auto --converge`) has zero coverage** |
| 22 Plan Drift Guard | Proposed | 6 | partial | partial | covered | covered | Severity mapping, authority auto-upgrade, rung≥3 hard-block untested |
| 58 Runtime Install Policy | Accepted | 7 | covered | partial | partial | covered | Strong projection coverage; purity guarantee + sandboxTier-invalid untested |
| 0174 Retire gsd-sdk Package | Accepted | 12 | covered | covered | covered | covered | Strong & behavioral; init-composer trace + tsc-output invariant gaps |
| 218 Release Version Validation | Accepted | **0** | missing | missing | missing | na | **ZERO TESTS — entirely unprotected (regex lives only in release.yml bash)** |
| 227 Input Validation Shape | Accepted | 5 | covered | covered | partial | partial | **All 5 test files silently excluded from npm test (subdir blind spot)** |
| 230 Next Integration Branch | Proposed | 5 | partial | partial | partial | covered | Most critical seams untested; 2 files misattributed (unrelated features) |
| 415 Stale-Base Token Reintro | Accepted | 12 | covered | partial | partial | covered | Token reintro guarded; propagator + ruleset-strict-mode untested |
| 443 Unified Effort / Fast-Mode | Proposed | 12 | covered | covered | covered | partial | Resolver exceptionally covered; **end-to-end effort propagation untested** |
| 452 ESLint Lint Harness | Accepted | 2 | partial | partial | partial | covered | Local rules covered; `no-only-tests` + plugin-n + script-retirement untested |
| 456 Test Rigor Architecture | Accepted | 19 | covered | partial | partial | covered | Clock seam strong; mutation breadth + warn→error + allow-rule tracking gaps |
| 457 Generated CJS Single Source | Accepted | 8 | covered | partial | partial | covered | Value-baking solid; .gitignore↔eslint sync + emit-path gaps |
| 550 Spec-Phase Probe Contract | Accepted | 6 | covered | covered | covered | covered | Core seam strong; Decisions 4(b–d)/6 deferred to #644 (untested by design) |
| 0656 Research Module Seam | Accepted | 9 | covered | covered | covered | covered | Strong; slopcheck "cannot-lower" + scrape-dispatch gaps |
| 660 Release From Next Head | Proposed | 13 | partial | partial | covered | covered | Scripts covered; orchestration (release.yml/auto-backmerge) largely untested |
| 766 Claude Plugin Manifest | Accepted | 3 | covered | partial | partial | partial | `agents`-absence + post-#770 event drift untested; CI-skipped schema validate |
| 894 Capability Decl Format | Proposed (coupled 857) | 5 | covered | partial | covered | partial | *Provisional.* Strong; `when`-ownership, tier-monotone std→full gaps |
| 959 Capability Command Contribution | Proposed (coupled 857) | 6 | covered | partial | covered | covered | *Provisional.* Strong; proto-pollution family-name + case-removal gaps |
| 1016 Runtime Capability Descriptor | Proposed (coupled 857) | 9 | covered | partial | covered | covered | *Provisional.* Dense; sandboxTier fail-loud + 5f required-axes gaps |
| 1143 Claude Orchestration | Proposed (coupled 857) | 9 | missing | missing | missing | na | *Provisional.* No impl; primary surface entirely untested |
| 3524 CJS-SDK Hard Seam | Superseded (0174) | 12 | partial | partial | covered | covered | Drift-bugs guarded; all subjects gitignored artifacts (crash on clean worktree) |
| 3660 Runtime Artifact Layout | Accepted | 6 | covered | partial | partial | covered | Core protected; `findInstallSourceRoot` fail-path + agent `tools:`-strip gaps |
### ADRs with zero tests or notable gaps
**ADR-218 (Release Version Validation) — ZERO attributed tests.** This is the most acute hole. Both ADR decisions — the leading-zero-rejecting format regex (`^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.0$`) and the npm duplicate-version precheck — live exclusively in a bash `run:` block in `.github/workflows/release.yml`. No test exercises either. The `1.01.0` incident that motivated the ADR caused an irrecoverable production divergence, and nothing prevents a reversion to the old broken regex. `issue-version-gate.test.cjs` is a superficially-related but wholly separate feature. **This is a critical gap in a high-blast-radius zone.**
**ADR-227 (Input Validation Shape) — 5 tests that never run.** All five attributed files live in `tests/observability/` and `tests/dispatch/`, which `scripts/run-tests.cjs:269` never enters (flat `readdirSync`). The tests are individually high quality (8 boundary/negative UUID-v4 cases, real file I/O, per-test isolation) but are silently excluded from `npm test`, CI, and gsd-test. ADR-227's two-layer validation guarantee — the correlation-poisoning and log-blowup classes it was written to prevent — is effectively unguarded until the runner is fixed.
**ADR-0006 (Planning-Path Projection) — critical handler-consumption gap.** The ADR's single most important invariant — that `cmdInitExecutePhase/PlanPhase/PhaseOp/MilestoneOp` route through `planningPaths().planning` rather than flat `path.join(cwd, '.planning')` — has no regression test. The banned `.planning/projects/<project>` form also has no guard.
**ADR-15 (Autonomous Cross-AI Convergence) — primary surface untested.** The ADR's designated primary operator surface, `/gsd-progress --next --auto --converge`, is both unimplemented and untested; the existing `cross-ai-execution.test.cjs` tests a *different* feature (execute-phase delegation) and is misattributed.
**ADR-0011a / 0011b / 0011c (Skill Surface / Review Default Reviewers).** Source modules (`install-profiles.cjs`, `surface.cjs`, `clusters.cjs`, `review-reviewer-selection.cjs`, `config-schema.cjs`) are gitignored build artifacts absent from a clean worktree; all attributed tests throw `MODULE_NOT_FOUND` without a prior `npm run build:lib`. CI's pretest hook covers this, but isolated runs and fresh worktrees fail confusingly.
**ADR-1143 (Claude Orchestration) — provisional, no implementation.** Proposed and blocked on ADR-857; the capability directory does not exist, so the primary surface has zero coverage. Findings are provisional pending ADR-857.
---
## Weak tests
### Retire
Only entries whose adversarial verification confirmed `finalVerdict === "retire"`. **5 tests survived verification as truly worthless.**
| File | Location | Why worthless | Confirmed by verification |
|---|---|---|---|
| `tests/enh-2790-skill-consolidation.test.cjs` | Lines 127–157 — 4 "has a name: field" spot-checks | Assert only `fm.name.length > 0`; `command-contract.test.cjs` already enforces the stricter `/^gsd[:-]/` prefix on all 67 files. Renaming `gsd:capture`→`capture` (the exact regression) passes these but is caught by command-contract. Cannot go red for the bug class the ADR exists to prevent. | **Yes** — verified `command-contract.test.cjs:44-54` covers these 4 files more strictly; deletion creates no coverage gap. |
| `tests/command-routing-hub.test.cjs` | Lines 134–137 — "constructs successfully with only cjsRegistry" | Duplicate of the construction test at 71–75; asserts only `typeof hub.dispatch === 'function'` with no dispatch. `createHub` does zero registry-dependent branching at construction. | **Yes** — verified identical assertion; populated-registry dispatch path covered by the happy-path block at line 142. |
| `tests/command-routing-hub.test.cjs` | Lines 515–519 — "ERROR_KINDS.UnknownCommand === result.kind" | The identical assertion appears at 9 other sites (222, 234, 245, 256, 360, 393, 410, 422, 518). Same code path covered at 414–424. No unique setup/edge/field. | **Yes** — verified superset coverage by lines 384–424 + the constant-stability test at 521–526. |
| `tests/no-cjs-sdk-handsync-tooling.test.cjs` | All 3 tests (lines 26–53) | Assert absence of `lint-shared-module-handsync.cjs` / allowlist JSON that **never existed on main** (CONTEXT.md:693 confirms). Can only go red if someone manually drops those exact files. No behavioral coverage. | **Yes** — verified the real retired artifacts (`cjs-sdk-bridge.cjs`, `sdk/`) are covered by `bug-190`; these guard ADR-text-only artifacts. |
| `tests/runtime-artifact-layout.test.cjs` | Lines 319–323 — "cline has one skills kind (#782)" | Default-scope call is behaviorally identical to the explicit-global test at 249–265, which asserts a strict superset (kinds.length, kind, destSubpath, prefix, stage). | **Yes** — verified default scope = global (source line 430); also independently covered by `bug-782-cline-skills-emission.test.cjs:629-636`. |
### Redesign / improve
All `redesign` + `keep-but-improve` verdicts, **plus the 6 retire-candidates that verification downgraded** to redesign. (Representative selection across ADRs; the full set is large — these are the highest-signal entries and every downgraded retire-candidate.)
#### Downgraded retire-candidates (verification overruled the retire verdict — do NOT delete)
| File | Location | Problem | What the rewritten test should assert |
|---|---|---|---|
| `tests/bug-3135-capture-backlog-workflow.test.cjs` | Lines 136–182 broad @-ref regression | The block IS redundant (only checks `/workflows/` refs, misses 24 files with `references/`/`templates/` refs that command-contract Rule 4 catches) — delete ONLY that block. But the FILE's lines 1–132 test unique `add-backlog.md` content invariants with no equivalent elsewhere. | Delete lines 134–182 only. Retain content-contract assertions: `add-backlog.md` uses `phase.next-decimal`, writes ROADMAP before mkdir, 999.x scheme; `capture.md` routes to `add-backlog.md`. |
| `tests/review-reviewer-selection.test.cjs` | Lines 115–122 result-shape test | Shape-only on `{ detected: [] }`. But the empty-detected call exercises the `no_config_all_detected` else-branch that no other test asserts (existing test uses `['gemini']`). | Replace shape assertions with behavioral: `source === 'no_config_all_detected'`, `selected/errors/warnings === []` for the empty-detected fallback. |
| `tests/review-reviewer-selection.test.cjs` | Lines 17–27 KNOWN_REVIEWER_SLUGS | Tests a compiled-CJS constant's shape. Verification found the slug membership DOES drive `resolveReviewerSelection`'s warn/drop logic at runtime — it's the only guard against silent slug-list drift. | Collapse to one behavioral test: `resolveReviewerSelection` with a config'd known slug yields no warning + slug in `selected`; an unknown slug yields a warning. |
| `tests/enh-191-retire-sdk-package.test.cjs` | Lines 52–58 AGENTS.md absence | Off-topic for SDK retirement, but NOT valueless: `bin/install.js:~10963` writes a root `AGENTS.md` on local Copilot install; this is the only guard against that artifact committing. | Extract to a repo-layout governance test (e.g. `tests/repo-layout.test.cjs`); rename to state the real invariant (no ad-hoc root instruction file vs CONTEXT.md/ADRs); link the copilot install path. |
| `tests/capability-registry.test.cjs` | Lines 558–584 drift test | Steps 2–3 are a file-I/O tautology and skip the real `--check` path (`stripGeneratedComment` + `normalizeLineEndings`). But it is the only unit-level coverage of the drift-comparison pipeline. | Call the actual comparison expression: assert a stale version survives both `stripGeneratedComment` and `normalizeLineEndings` and is detected; assert a comment-only-timestamp change is NOT flagged after stripping. |
| `tests/verify-test-quality.test.cjs` | 3 describes (disabled / circular / assertion-strength detection) | Self-referential: define a regex inline, write a matching fixture, assert the regex matches — cannot go red for any production change. The detector lives as prose in `verify-phase.md`. | Replace with a structural guard reading `gsd-core/workflows/verify-phase.md`: assert the `audit_test_quality` step tag exists and contains the skip-pattern, circular-detection, and assertion-strength-table markers. |
#### Other redesign / keep-but-improve (high-signal selection)
| File | Location | Problem | What the rewritten test should assert |
|---|---|---|---|
| `tests/research-cli.test.cjs` | Line 423 | Pass-always `assert.ok(true)` placeholder; the 2-package arg-parse regression is undefended. | Extract the flag parser; call with `['--bad-flag','pkgA','pkgB']` and assert unknown-flag error + that `pkgA/pkgB` are not consumed as flag values. (No network.) |
| `tests/worktree-baseref-install.test.cjs` | Line 444 | Pass-always `assert.ok(true)`; the null-settings #683 guard is never exercised. | `assert.doesNotThrow(() => applyWorktreeBaseRef(null, tmpDir))` and assert no `worktree.baseRef` is written. |
| `tests/bug-260-worktree-path-guard.test.cjs` | Line 338 | Environment-dependent branch silently passes via `assert.ok(true)` when the traversal path resolves inside the worktree — a security guard goes untested on macOS/private-symlink hosts. | Construct the traversal target in a separate tmpdir outside the worktree; remove the if/else; unconditionally assert `result.status === 2` / decision `block`. |
| `tests/eslint-rules.test.cjs` | 24 trailing `assert.ok(true)` calls | RuleTester already throws on failure; the trailing literal asserts nothing and inflates count. | Delete the `assert.ok(true)` lines; optionally add `assert.strictEqual(typeof rule.create, 'function')` per describe to confirm the module loaded. |
| `tests/phase.test.cjs` | Lines 4343–4368 (`last_updated` refresh) | `Math.abs(new Date() - updatedAt) < 60_000` wall-clock window — flaky under loaded CI. | Assert `last_updated` is a valid ISO timestamp ≠ the stale seed value AND its date portion equals today's UTC date. |
| `tests/bug-3707-locked-worktree-cleanup.test.cjs` | 7 `Date.now()` stale-mtime sites | Real-clock coupling for stale-lock boundary; violates the ADR-456 clock-seam mandate. | Pin a constant epoch (`PINNED_NOW_MS`) for mtime fixtures; pass it to the SUT as a clock parameter where supported. |
| `tests/context-utilization.property.test.cjs` | Line 187 | `Math.floor(Math.random()*…)` inside `fc.property` breaks reproducibility under the pinned seed. | Replace with `fc.integer({ min: 0, max: contextWindow })`; split a shape test from a value-correctness test at known ratios. |
| `tests/research-provider.property.test.cjs` | Single test | Only "never throws + valid type"; inverting HIGH/LOW classification passes. | Keep robustness property; add example-based unit tests for classification boundaries; add module to mutation COVERED. |
| `tests/research-store.property.test.cjs` | All 3 tests | Shape/determinism only; cache-key collisions or over-normalization invisible. | Add a collision property (distinct inputs → distinct keys) + boundary examples (`npm/lodash` ≠ `npm/react`). |
| `tests/install.test.cjs` | Lines 683–712 (Kilo source-grep) | Reads `bin/install.js` as a string and `.includes()` internal symbol names — source-grep, breaks on rename, passes on dead code. | Invoke `install.js` with `--kilo` in a tmpdir; assert on the installed config/manifest content (or export+call the kilo-detection logic). |
| `tests/sh-hook-paths.test.cjs` | Tests 2–4 (lines 119–194) | Source-grep variable-name scans of `install.js`/`runtime-hooks-surface.cts`; behavior already covered by Test 1's behavioral call. | Call `buildHookCommand` per `.sh` hook for linux+win32; assert non-empty, `bash` runner, absolute quoted configDir prefix. |
| `tests/clusters.test.cjs` | Lines 50–58 | `assert.ok(utilitySize >= skills.length || true)` — `|| true` makes it vacuous. | Remove `|| true`; assert a meaningful lower bound (`utilitySize >= 5`). |
| `tests/model-profiles.test.cjs` | Lines 115–122 | Test titled "resolution order: override > profile > default" exercises no override and asserts nothing about override behavior. | Use a temp project with `config.model_overrides`, call `resolveModelInternal`, assert override > profile > default explicitly. |
| `tests/active-workstream-store.unit.test.cjs` | Lines 361–364 | `assert.ok(key === null || typeof key === 'string')` — disjunction always passes; env-dependent. | Clear `TTY`/`SSH_TTY` in `beforeEach`; assert `strictEqual(key, null)`, or skip when a real TTY is present. |
| `tests/feat-488-effort-sync.test.cjs` | Lines 177–219 | Does not redirect `GSD_HOME`; resolved effort depends on the dev's real `~/.gsd`; degrades to a type-only check. | Set `GSD_HOME=tmpHome`; assert `result.synced === 1` and `changes[0].to === 'low'`. |
| `tests/bug-492-effort-manifest-fallback.test.cjs` | Lines 20–48 | Mutates the module-level `CANONICAL_CONFIG_DEFAULTS` singleton — independence violation; races under parallel runs. | Drive the same path via a tmpHome `~/.gsd/defaults.json` rather than mutating shared module state. |
| `tests/issue-766-plugin-manifest.test.cjs` | Section C, "claude plugin validate --strict" | Permanently skipped when the `claude` binary is absent (i.e. on CI) — the only full-schema gate never runs. | Install the claude CLI in a CI step, OR validate `plugin.json` against a snapshotted JSON schema unconditionally. |
| `tests/cross-ai-execution.test.cjs` | Entire file | Misattributed to ADR-15; tests a different feature (`workflow.cross_ai_execution`). Also a shared-mutable `let content` independence issue. | Leave the file for its real feature; create `tests/adr-15-plan-strategy-seam.test.cjs` for the actual converge seam; hoist `content` to a single module-level read. |
| `tests/issue-844-manifest-version-sync.test.cjs` | Describe A (33–127) | Order-dependent shared `tmpRoot` across setup/assert/cleanup tests. | Refactor to a single test with per-test `before()/after()` tmpRoot allocation. |
| `tests/plan-review-convergence.test.cjs` | Lines 688–692 | Four `workflow.includes(verdict)` whole-file substring checks — pass even if verdicts move to dead/comment text. | Extract the source-grounding section slice; assert each verdict + its severity mapping (AMBIGUOUS→MEDIUM, UNCHECKABLE→INFO) inside that slice. |
| `tests/feat-3594-parser-property-style.test.cjs` | Lines 119–135 | `assert.ok(elapsedMs < 2000)` wall-clock; escapes `no-elapsed-assertion` via the variable name. | Replace with a call-count metric (instrument char comparisons, assert O(n)); or raise bound to 10000ms as a stopgap. |
| `tests/core.test.cjs` | Lines 1834–1879 (`reapStaleTempFiles`) | Writes to the global shared `os.tmpdir()/gsd`; races with `temp-subdir.test.cjs` under concurrency. | Use a per-test isolated tmpdir via the existing `{ dir }` option; clean up in `afterEach`. |
---
## Prioritized action plan
### P0 — do first (blast radius / false-green)
1. **Fix recursive test discovery** in `scripts/run-tests.cjs` (replace flat `readdirSync` with the already-present `findTestFilesRecursive`). This alone reactivates ADR-227's entire UUID-validation suite (`tests/observability/`, `tests/dispatch/`) — currently silently excluded from `npm test` and CI.
2. **Raise Stryker `break` from 50 → 80** in `stryker.config.mjs` (ADR-456 mandate). Move any module that cannot meet 80% into `UNMUTATED` with a documented per-entry reason rather than diluting the global floor.
3. **Add tests for ADR-218** (zero coverage today): extract the release-version regex and npm-duplicate precheck into a testable `.cjs` helper (or a `source-text-is-the-product` workflow assertion on `release.yml`), and assert `1.01.0`/`01.0.0` are rejected, valid versions pass, and the duplicate-version step exists and is wired before the publish jobs.
4. **Retire the 5 verified-worthless tests** (see Retire table): `enh-2790` name spot-checks (127–157), `command-routing-hub` lines 134–137 and 515–519, `no-cjs-sdk-handsync-tooling` (all 3), `runtime-artifact-layout` cline edge-case (319–323).
5. **Redesign the 6 downgraded retire-candidates** (do NOT delete): `bug-3135` (delete only the broad block, keep content tests), both `review-reviewer-selection` tests, `enh-191` AGENTS.md (extract to governance test), `capability-registry` drift test, `verify-test-quality` (replace with structural workflow guard).
6. **Close the ADR-0006 critical gap**: add a regression test that `init execute-phase/plan-phase/phase-op/milestone-op` return workstream-scoped paths under `GSD_WORKSTREAM`, plus a guard against the banned `.planning/projects/<project>` form.
### P1 — high priority (rigor enforcement + concentrated weakness)
7. **Bring `core.cts` and `config-loader.cts` into the Stryker COVERED set** (smallest high-value modules first); write their `*.unit.test.cjs` as a prerequisite.
8. **Promote `no-source-grep` to `error` and apply it to `tests/**`**, splitting product-file reads (`.md/.json`, allowed) from source-file grep (`.cjs/.js/.ts`, blocked). Then fix the surfaced violations (`install.test.cjs` Kilo block, `sh-hook-paths.test.cjs` Tests 2–4).
9. **Add the `allow-test-rule` tracking-issue lint gate** + run the one-time sweep to attach `#NNN` (or an approved category tag) to the 332 untracked exemptions.
10. **Eliminate the pass-always placeholders**: redesign `research-cli.test.cjs:423`, `worktree-baseref-install.test.cjs:444`, the `bug-260` environment-dependent branch, and strip the 24 `assert.ok(true)` lines in `eslint-rules.test.cjs`. Add a `no-tautological-assert` ESLint rule to prevent recurrence.
11. **Add `no-only-tests` RuleTester coverage** and extend `no-elapsed-assertion` to catch `Date.now()-start` and computed-variable (`elapsedMs`) patterns.
12. **Add a branch-coverage floor** (`--branches 60`) and a per-file branch floor for the UNMUTATED critical modules.
13. **De-flake the clock-coupled tests**: `phase.test.cjs:4362` (60s window → ISO-date assertion), `bug-3707` (pin epoch), `context-utilization.property.test.cjs:187` (`fc.integer` not `Math.random`).
14. **Add the ADR-15 primary-surface tests** (`/gsd-progress --next --auto --converge`) and re-attribute `cross-ai-execution.test.cjs` to its real feature.
### P2 — follow-up (robustness & hygiene)
15. **Fail-on-zero-executed** in `run-tests.cjs` (exit 1 on empty suite by default; gate the allowance behind an env flag).
16. **Isolate the shared `GSD_TEMP_DIR` reap tests** (`core.test.cjs:1834-1879`, `temp-subdir.test.cjs`) via the existing `{ dir }` option + `afterEach` cleanup.
17. **Add build-artifact `before()` guards** to artifact-dependent suites (ADR-3524 subjects, ADR-0011 family) so a clean worktree fails with a clear "run npm run build:lib" message instead of a top-level crash.
18. **Strengthen the strong-but-incomplete ADRs**: ADR-443 end-to-end effort propagation, ADR-0656 slopcheck "cannot-lower" + scrape-dispatch, ADR-0009's 3 absent ADR-listed test files, ADR-766 `agents`-absence assertion + post-#770 hook drift.
19. **Add `.gitignore` ↔ `eslint.config.mjs` ignore-list parity test** (ADR-457) so a migration that updates one list but not the other is caught.
> Capability-cluster items (ADR-894/959/1016/1143) are deliberately deferred — they are coupled to ADR-857 (worked elsewhere) and their findings are provisional. Re-run this audit slice once ADR-857 lands.