* fix(#2903): use the command form that actually works in reader-facing docs Docs told readers to type the colon form, which no runtime registers -- 18 of 19 runtimes use slash-hyphen and the 19th uses shell-var -- so anyone copying an example got an unrecognized command. Swept 178 occurrences across 53 files, locale mirrors included so they do not re-diverge from English. The colon form is a source-authoring token, not a user-facing one: install-time converters key on it to produce the hyphen form runtimes actually register. So the sweep is scoped, and three things are deliberately left alone: - ADRs, which are a historical record; editing their prose falsifies what was written at the time. - The legacy release-notes archive, pending a maintainer decision on whether it follows the same historical carve-out. Excluding it keeps a later reversal additive rather than a revert. - Source artifacts under commands, workflows and agents, where the colon form is load-bearing. Rewriting those would break the installed-skill guarantee across every runtime -- the single largest hazard here. The plugin namespace form is a real, separate token and survives untouched. Adds a lint enforcing exactly that boundary, since the correct form genuinely differs by directory and nothing previously caught the drift. Also fixes a hardcoded colon form in the capability-matrix generator. The sweep alone would have left the generated matrix disagreeing with the template that produces it, so the fix is at the source and the output regenerated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#2903): stop the sweep misquoting source frontmatter Adversarial review caught three lines where the sweep rewrote a citation of the literal YAML name: key from a source command file. That key genuinely is the colon form -- this change's own carve-out logic says source-authoring tokens keep it -- so the docs ended up misquoting the real files. One of the three is an acceptance-checklist assertion, which the sweep turned into a false statement. Restored the three citations to match their sources verbatim, surgically: where a line carried both a name: citation and a real reader-facing slash command, only the citation reverted and the command stayed corrected. The guard needed the same distinction, or it would have flagged the restoration and reddened the build: a gsd:<cmd> token preceded by name: is a citation of a source token and is now permitted. The exemption is deliberately narrow -- a bare gsd:<cmd> anywhere else still fails -- with a test pinning that narrowness. Also makes the detection case-insensitive. Review found /GSD:next slipped through silently; no such casing exists in the tree today, so this closes a latent gap rather than fixing a live one. Swept the whole tree for further corrupted citations: none beyond the three. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#2903): retire the stale-next invariant and sweep next like every other command Maintainer decision on a genuine conflict between two contracts. Invariant #3054 banned the literal /gsd-next from user-facing docs because it named a retired workflow-advance command. But commands/gsd/next.md is a live command -- the state-aware smart-entry launcher -- and this issue requires docs to use the hyphen form every runtime actually registers. Both could not hold for this one command, so docs had been sidestepping the ban by keeping the colon form, which is exactly the defect this issue exists to remove. FEATURES.md already recorded the reassignment: the hyphen form "is not the retired workflow-advance command; it is reserved for the state-aware smart-entry launcher. Workflow advancement remains under /gsd-progress --next." With that reassignment the invariant's premise is obsolete and the guard now contradicts the documented command form, so it is retired with a comment recording why rather than deleted silently. next is now swept like every other command, and the earlier exemption added to the new guard is removed so nothing is special-cased. Four citations of the literal name: frontmatter key stay in colon form, because the source file really does carry name: gsd:next and a doc quoting it must reproduce it verbatim. Two of those lines were reworded to say which side is the frontmatter key and which is the slash command, since they previously conflated the two. Verified the retired scan would now genuinely fail against this tree -- the conflict was real and resolved, not dodged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(#2903): backfill changeset pr number Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
35 KiB
ADR Test-Coverage Audit — 2026-06-13
Lead QA architect final report. Scope: 38 ADRs in docs/adr/, evaluated with the qa-test-architect methodology, plus 4 platform-capability lenses over the test infrastructure. Every RETIRE verdict was independently re-checked by an adversarial verification pass.
Scope note. ADR-857 (capability system core) was intentionally excluded from this audit — it is being worked by another instance. The capability cluster (ADR-894, ADR-959, ADR-1016, ADR-1143) is coupled to ADR-857, so all findings for those four ADRs are provisional and may shift once ADR-857 lands.
Executive summary
The test portfolio is, on the whole, healthier than its volume suggests: the deterministic seams (dispatch hub, model catalog, planning-path resolution, installer migration, shell-command projection) are genuinely behavioral and well-covered, and the project has built real test-rigor infrastructure (an injectable clock seam, fast-check with a pinned seed, Stryker mutation testing wired into CI, and four custom ESLint rules). The dominant problem is not absent tests — it is enforcement gaps that let the rigor infrastructure run as theater, and a long tail of weak tests concentrated in three families: source-text-grep assertions, tautological assert.ok(true) placeholders, and architecture-retirement guards that can never go red.
Five headline problems demand attention first: (1) Stryker mutation testing covers only ~4–6% of the lib surface and excludes every high-risk module (state, phase, verify, init, commands), with the break threshold set to 50% instead of the ADR-456-mandated 80% — the mutation gate trivially passes for PRs touching those modules. (2) 332 of 335 allow-test-rule exemptions carry no tracking issue, directly violating ADR-456's delete-bad-tests policy. (3) A subdirectory blind spot in scripts/run-tests.cjs silently excludes all of tests/observability/ and tests/dispatch/ from npm test — meaning ADR-227's entire UUID-validation seam is unguarded by any continuously-executed test. (4) Multiple ADRs marked Accepted or Proposed have zero attributed tests or test files that cannot run because their source modules are gitignored build artifacts absent from a clean worktree (ADR-218, ADR-0011 family). (5) A scatter of vacuous and self-referential tests (verify-test-quality.test.cjs, assert.ok(true) placeholders, environment-dependent path-guard branches) that provide false confidence on security-relevant code.
Do first: raise the Stryker break threshold and bring core.cts/config-loader.cts into the mutation-covered set (P0); fix the run-tests.cjs non-recursive file scan so observability/dispatch tests actually execute (P0); retire the 5 verified-worthless tests and redesign the 6 downgraded retire-candidates (P0/P1).
Testing-platform robustness
Synthesis of the four platform lenses: adr-456-test-rigor, coverage-mutation, anti-pattern-sweep, determinism-flake.
(a) What the platform CAN do today
- Deterministic clock seam (
tests/helpers/clock.cjs,makeFakeClock) — well-designed and correctly used by lock/state tests (clock-seam.test.cjs,perf-407,bug-474). Eliminates wall-clock races in the lock subsystem. - Property-based testing via fast-check with a pinned seed (
tests/helpers/fast-check-setup.cjs, seed=42) — 9*.property.test.cjsfiles exist and run reproducibly. - Mutation testing — Stryker is configured (
stryker.config.mjs) and wired into a PR-gating workflow (.github/workflows/mutation.yml) that shards per changed module. - Custom ESLint rules —
no-source-grep,no-magic-sleep-in-tests,no-elapsed-assertion,no-raw-rmsync-in-tests, plusno-only-tests— exist and are unit-tested via RuleTester (tests/eslint-rules.test.cjs). - Env hermeticity —
scripts/run-tests.cjsdeletesGSD_PROJECT/GSD_WORKSTREAMbefore spawning child test processes (mitigates the ambient-env hazard recorded in MEMORY.md). - Line-coverage gate —
c8 --lines 70(lib) enforced in CI as part of the unit lane.
(b) ENFORCED vs aspirational
| Mechanism | Status | Note |
|---|---|---|
| Clock seam in lock/state tests | Enforced | Used correctly; but not mandated by lint — bug-3707 and phase.test.cjs:4362 still use real Date.now() |
| fast-check seed pinned | Enforced | seed=42; but no lint rule requires *.property.test.cjs to import the setup, and context-utilization.property.test.cjs uses Math.random() inside fc.property, breaking reproducibility |
| Stryker mutation gate | Aspirational | Runs in CI, but break=50% (not 80%) and covers 6/109 modules; trivially passes for excluded modules |
| Line coverage 70% | Enforced | But satisfiable via CLI integration tests that execute lines without asserting sub-behaviors; no branch-coverage floor |
no-magic-sleep-in-tests |
Enforced (error) |
Promoted to error |
no-raw-rmsync-in-tests |
Enforced (error) |
|
no-only-tests |
Aspirational | Wired at error but has zero RuleTester coverage and no integration test |
no-source-grep |
Aspirational | At warn only, and NOT applied to tests/** at all — the file-level allow-test-rule bypass blanket-exempts source-grep in test files |
no-elapsed-assertion |
Aspirational | At warn; only matches .elapsed/.duration/.took/.ms properties, missing Date.now()-start and computed-variable patterns (e.g. elapsedMs) |
allow-test-rule tracking-issue requirement |
Not enforced | 332/335 exemptions lack an issue number; no CI gate prevents new untracked exemptions |
| Subdirectory test discovery | Broken | run-tests.cjs uses flat readdirSync(testDir); tests/observability/ and tests/dispatch/ are silently never run by npm test |
| Fail-on-zero-executed | Not enforced | run-tests.cjs exits 0 when a suite resolves to zero files |
(c) Prioritized capability upgrades
| Capability | Current state | Recommendation | Priority |
|---|---|---|---|
| Recursive test discovery | Flat readdirSync skips tests/observability/, tests/dispatch/ |
Replace with the existing findTestFilesRecursive (already in the same file) |
P0 |
| Stryker break threshold | break: 50 in stryker.config.mjs |
Raise to 80 per ADR-456; move sub-80% modules to UNMUTATED with documented reasons |
P0 |
| Mutation coverage breadth | 6/109 modules covered; 12 critical modules excluded | Graduate one module/release cycle starting with core.cts, config-loader.cts; require any module with a *.unit/property.test.cjs to be in COVERED or UNMUTATED |
P0 |
allow-test-rule audit |
332/335 untracked exemptions | Lint gate requiring either an approved category tag or #NNN; one-time tracking-issue sweep |
P1 |
no-source-grep in test files |
Not applied to tests/**; warn-only |
Apply at error to tests/**; split rule to allow product-file (.md/.json) reads but block source-file (.cjs/.js/.ts) grep assertions |
P1 |
| Branch-coverage floor | Only line coverage gated | Add --branches 60; per-file branch floor for the UNMUTATED critical modules |
P1 |
no-elapsed-assertion reach |
Property-name match only | Extend to Date.now()-start binary expressions and identifiers ending Ms/Elapsed/Duration in assert args |
P1 |
no-only-tests coverage |
Wired error, untested |
Add RuleTester valid/invalid cases + an integration assertion it fires | P1 |
no-tautological-assert rule |
None | New ~30-line local rule flagging assert.ok(true)/assert.ok(1) in tests/** |
P1 |
| Fail-on-zero-executed | Exits 0 on empty suite | Exit 1 by default; gate the empty-suite allowance behind RUN_TESTS_ALLOW_EMPTY_SUITE=1 |
P2 |
Shared-GSD_TEMP_DIR isolation |
reap tests write to global os.tmpdir()/gsd |
Redirect via the existing dir option to per-test tmpdirs; lint against writing to the shared path without cleanup |
P2 |
| Build-artifact preflight in artifact-dependent tests | Top-level require() of gitignored .cjs hard-crashes on clean worktree |
before() existsSync guard with a clear "run npm run build:lib" message |
P2 |
ADR-456 compliance — per mandate
ADR-456 defines four test-rigor mandates. Compliance is partial on all four:
- (a) Concurrency via injectable clock seam, not real OS races — Substantially met. The clock seam exists and the lock subsystem uses it. Gaps:
bug-3707-locked-worktree-cleanup.test.cjscomputes stale-lock mtimes from realDate.now()(7 sites);phase.test.cjs:4362asserts a 60-second wall-clock window;core.test.cjstimeAgoboundary tests use real clock at the 5s bucket edge. No lint rule mandates the seam. - (b) fast-check + Stryker antagonistic tier, 80% kill threshold — Largely aspirational. fast-check is wired with a pinned seed, but Stryker covers ~4–6% of source lines, excludes all high-risk modules, and the break threshold is 50%, not the mandated 80%. The antagonistic tier exists in name but does not guard the dangerous surface.
- (c) Assert on typed
--jsonfields and exported registries, not rendered text or source literals — Partially met, weakly enforced. New tests largely follow it, butno-source-grepiswarn-only and not applied to test files, so source-grep assertions persist (e.g.install.test.cjs.includes()onbin/install.js;sh-hook-paths.test.cjsvariable-name scans). Several CLI tests still assert onstdout.includes()substrings rather than parsed JSON. - (d) Delete pass-always / vacuous / source-grep / elapsed / real-race / permanent-allow-rule tests — Not met. Concrete pass-always tests survive (
research-cli.test.cjs:423,worktree-baseref-install.test.cjs:444, 24assert.ok(true)ineslint-rules.test.cjs); 332/335allow-test-ruleexemptions are effectively permanent and untracked, directly contradicting the policy clause requiring// allow-test-rule: see #NNNon post-ADR exemptions.
Per-ADR coverage
| ADR | Status | #Tests | Happy | Boundary | Negative | Independence | Verdict |
|---|---|---|---|---|---|---|---|
| 0001 Dispatch Policy Module | Accepted | 12 | covered | partial | covered | covered | Core protected; SDK bridge / exit-code mapping unverified |
| 0002 Command Contract Validation | Accepted | 6 | covered | partial | missing | covered | Integration-strong; helper unit tests + lint negatives missing |
| 0003 Model Catalog Module | Accepted | 13 | covered | partial | covered | covered | CJS solid; SDK-parity invariant untested |
| 0004 Worktree/Workstream Seam | Accepted | 13 | covered | partial | covered | covered | Substantially protected; cross-path naming + .planning/.lock 2-proc gaps |
| 0005 SDK Architecture Seam Map | Superseded (0174) | 13 | partial | partial | partial | covered | Retirement guards OK; thin-adapter enforcement absent |
| 0006 Planning-Path Projection | Accepted | 6 | partial | partial | covered | covered | Critical gap: init-handler planningPaths() consumption untested |
| 0007 SDK Package Seam | Superseded (0174) | 10 | covered | partial | partial | covered | Retirement guarded; classifyPromptUserAction boundary untested |
| 0008 Installer Migration Module | Accepted | 16 | covered | partial | covered | covered | Rich suite; rewrite-json apply + local-scope gaps |
| 0009 Shell Command Projection | Accepted | 21 | covered | partial | covered | covered | Strong; 3 ADR-listed test files absent; source-grep in sh-hook-paths |
| 0010a Skill Surface Budget | Proposed (impl w/ gaps) | 8 | covered | partial | partial | covered | ~75% protected; stageCommandsForRuntimeFlat + migration one-shot untested |
| 0010b File Operation Engine | Superseded (0009) | 10 | covered | partial | covered | covered | Canonical seam solid; divergent atomicWriteFileSync + path-containment gaps |
| 0011a Skill Surface Budget | Accepted | 12 | covered | partial | partial | covered | Broad; source modules absent from worktree → MODULE_NOT_FOUND; cyclic-requires untested |
| 0011b Review Default Reviewers (PRD) | Draft | 5 | partial | partial | partial | covered | infos-path + --all+flags edge untested; build-artifact dependency |
| 0011c Review Default Reviewers | Proposed | 4 | covered | partial | partial | covered | Mid-tier risks (infos, empty-selection) missing; build-artifact dependency |
| 0012 Command Routing Hub | Superseded (0174) | 5 | covered | partial | covered | covered | Hub well-covered; uat-passed adapter arg-shape gap; 3 redundant tests |
| 15 Autonomous Cross-AI Convergence | Proposed | 8 | partial | partial | partial | partial | Primary surface (/gsd-progress --next --auto --converge) has zero coverage |
| 22 Plan Drift Guard | Proposed | 6 | partial | partial | covered | covered | Severity mapping, authority auto-upgrade, rung≥3 hard-block untested |
| 58 Runtime Install Policy | Accepted | 7 | covered | partial | partial | covered | Strong projection coverage; purity guarantee + sandboxTier-invalid untested |
| 0174 Retire gsd-sdk Package | Accepted | 12 | covered | covered | covered | covered | Strong & behavioral; init-composer trace + tsc-output invariant gaps |
| 218 Release Version Validation | Accepted | 0 | missing | missing | missing | na | ZERO TESTS — entirely unprotected (regex lives only in release.yml bash) |
| 227 Input Validation Shape | Accepted | 5 | covered | covered | partial | partial | All 5 test files silently excluded from npm test (subdir blind spot) |
| 230 Next Integration Branch | Proposed | 5 | partial | partial | partial | covered | Most critical seams untested; 2 files misattributed (unrelated features) |
| 415 Stale-Base Token Reintro | Accepted | 12 | covered | partial | partial | covered | Token reintro guarded; propagator + ruleset-strict-mode untested |
| 443 Unified Effort / Fast-Mode | Proposed | 12 | covered | covered | covered | partial | Resolver exceptionally covered; end-to-end effort propagation untested |
| 452 ESLint Lint Harness | Accepted | 2 | partial | partial | partial | covered | Local rules covered; no-only-tests + plugin-n + script-retirement untested |
| 456 Test Rigor Architecture | Accepted | 19 | covered | partial | partial | covered | Clock seam strong; mutation breadth + warn→error + allow-rule tracking gaps |
| 457 Generated CJS Single Source | Accepted | 8 | covered | partial | partial | covered | Value-baking solid; .gitignore↔eslint sync + emit-path gaps |
| 550 Spec-Phase Probe Contract | Accepted | 6 | covered | covered | covered | covered | Core seam strong; Decisions 4(b–d)/6 deferred to #644 (untested by design) |
| 0656 Research Module Seam | Accepted | 9 | covered | covered | covered | covered | Strong; slopcheck "cannot-lower" + scrape-dispatch gaps |
| 660 Release From Next Head | Proposed | 13 | partial | partial | covered | covered | Scripts covered; orchestration (release.yml/auto-backmerge) largely untested |
| 766 Claude Plugin Manifest | Accepted | 3 | covered | partial | partial | partial | agents-absence + post-#770 event drift untested; CI-skipped schema validate |
| 894 Capability Decl Format | Proposed (coupled 857) | 5 | covered | partial | covered | partial | Provisional. Strong; when-ownership, tier-monotone std→full gaps |
| 959 Capability Command Contribution | Proposed (coupled 857) | 6 | covered | partial | covered | covered | Provisional. Strong; proto-pollution family-name + case-removal gaps |
| 1016 Runtime Capability Descriptor | Proposed (coupled 857) | 9 | covered | partial | covered | covered | Provisional. Dense; sandboxTier fail-loud + 5f required-axes gaps |
| 1143 Claude Orchestration | Proposed (coupled 857) | 9 | missing | missing | missing | na | Provisional. No impl; primary surface entirely untested |
| 3524 CJS-SDK Hard Seam | Superseded (0174) | 12 | partial | partial | covered | covered | Drift-bugs guarded; all subjects gitignored artifacts (crash on clean worktree) |
| 3660 Runtime Artifact Layout | Accepted | 6 | covered | partial | partial | covered | Core protected; findInstallSourceRoot fail-path + agent tools:-strip gaps |
ADRs with zero tests or notable gaps
ADR-218 (Release Version Validation) — ZERO attributed tests. This is the most acute hole. Both ADR decisions — the leading-zero-rejecting format regex (^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.0$) and the npm duplicate-version precheck — live exclusively in a bash run: block in .github/workflows/release.yml. No test exercises either. The 1.01.0 incident that motivated the ADR caused an irrecoverable production divergence, and nothing prevents a reversion to the old broken regex. issue-version-gate.test.cjs is a superficially-related but wholly separate feature. This is a critical gap in a high-blast-radius zone.
ADR-227 (Input Validation Shape) — 5 tests that never run. All five attributed files live in tests/observability/ and tests/dispatch/, which scripts/run-tests.cjs:269 never enters (flat readdirSync). The tests are individually high quality (8 boundary/negative UUID-v4 cases, real file I/O, per-test isolation) but are silently excluded from npm test, CI, and gsd-test. ADR-227's two-layer validation guarantee — the correlation-poisoning and log-blowup classes it was written to prevent — is effectively unguarded until the runner is fixed.
ADR-0006 (Planning-Path Projection) — critical handler-consumption gap. The ADR's single most important invariant — that cmdInitExecutePhase/PlanPhase/PhaseOp/MilestoneOp route through planningPaths().planning rather than flat path.join(cwd, '.planning') — has no regression test. The banned .planning/projects/<project> form also has no guard.
ADR-15 (Autonomous Cross-AI Convergence) — primary surface untested. The ADR's designated primary operator surface, /gsd-progress --next --auto --converge, is both unimplemented and untested; the existing cross-ai-execution.test.cjs tests a different feature (execute-phase delegation) and is misattributed.
ADR-0011a / 0011b / 0011c (Skill Surface / Review Default Reviewers). Source modules (install-profiles.cjs, surface.cjs, clusters.cjs, review-reviewer-selection.cjs, config-schema.cjs) are gitignored build artifacts absent from a clean worktree; all attributed tests throw MODULE_NOT_FOUND without a prior npm run build:lib. CI's pretest hook covers this, but isolated runs and fresh worktrees fail confusingly.
ADR-1143 (Claude Orchestration) — provisional, no implementation. Proposed and blocked on ADR-857; the capability directory does not exist, so the primary surface has zero coverage. Findings are provisional pending ADR-857.
Weak tests
Retire
Only entries whose adversarial verification confirmed finalVerdict === "retire". 5 tests survived verification as truly worthless.
| File | Location | Why worthless | Confirmed by verification |
|---|---|---|---|
tests/enh-2790-skill-consolidation.test.cjs |
Lines 127–157 — 4 "has a name: field" spot-checks | Assert only fm.name.length > 0; command-contract.test.cjs already enforces the stricter /^gsd[:-]/ prefix on all 67 files. Renaming gsd-capture→capture (the exact regression) passes these but is caught by command-contract. Cannot go red for the bug class the ADR exists to prevent. |
Yes — verified command-contract.test.cjs:44-54 covers these 4 files more strictly; deletion creates no coverage gap. |
tests/command-routing-hub.test.cjs |
Lines 134–137 — "constructs successfully with only cjsRegistry" | Duplicate of the construction test at 71–75; asserts only typeof hub.dispatch === 'function' with no dispatch. createHub does zero registry-dependent branching at construction. |
Yes — verified identical assertion; populated-registry dispatch path covered by the happy-path block at line 142. |
tests/command-routing-hub.test.cjs |
Lines 515–519 — "ERROR_KINDS.UnknownCommand === result.kind" | The identical assertion appears at 9 other sites (222, 234, 245, 256, 360, 393, 410, 422, 518). Same code path covered at 414–424. No unique setup/edge/field. | Yes — verified superset coverage by lines 384–424 + the constant-stability test at 521–526. |
tests/no-cjs-sdk-handsync-tooling.test.cjs |
All 3 tests (lines 26–53) | Assert absence of lint-shared-module-handsync.cjs / allowlist JSON that never existed on main (CONTEXT.md:693 confirms). Can only go red if someone manually drops those exact files. No behavioral coverage. |
Yes — verified the real retired artifacts (cjs-sdk-bridge.cjs, sdk/) are covered by bug-190; these guard ADR-text-only artifacts. |
tests/runtime-artifact-layout.test.cjs |
Lines 319–323 — "cline has one skills kind (#782)" | Default-scope call is behaviorally identical to the explicit-global test at 249–265, which asserts a strict superset (kinds.length, kind, destSubpath, prefix, stage). | Yes — verified default scope = global (source line 430); also independently covered by bug-782-cline-skills-emission.test.cjs:629-636. |
Redesign / improve
All redesign + keep-but-improve verdicts, plus the 6 retire-candidates that verification downgraded to redesign. (Representative selection across ADRs; the full set is large — these are the highest-signal entries and every downgraded retire-candidate.)
Downgraded retire-candidates (verification overruled the retire verdict — do NOT delete)
| File | Location | Problem | What the rewritten test should assert |
|---|---|---|---|
tests/bug-3135-capture-backlog-workflow.test.cjs |
Lines 136–182 broad @-ref regression | The block IS redundant (only checks /workflows/ refs, misses 24 files with references//templates/ refs that command-contract Rule 4 catches) — delete ONLY that block. But the FILE's lines 1–132 test unique add-backlog.md content invariants with no equivalent elsewhere. |
Delete lines 134–182 only. Retain content-contract assertions: add-backlog.md uses phase.next-decimal, writes ROADMAP before mkdir, 999.x scheme; capture.md routes to add-backlog.md. |
tests/review-reviewer-selection.test.cjs |
Lines 115–122 result-shape test | Shape-only on { detected: [] }. But the empty-detected call exercises the no_config_all_detected else-branch that no other test asserts (existing test uses ['gemini']). |
Replace shape assertions with behavioral: source === 'no_config_all_detected', selected/errors/warnings === [] for the empty-detected fallback. |
tests/review-reviewer-selection.test.cjs |
Lines 17–27 KNOWN_REVIEWER_SLUGS | Tests a compiled-CJS constant's shape. Verification found the slug membership DOES drive resolveReviewerSelection's warn/drop logic at runtime — it's the only guard against silent slug-list drift. |
Collapse to one behavioral test: resolveReviewerSelection with a config'd known slug yields no warning + slug in selected; an unknown slug yields a warning. |
tests/enh-191-retire-sdk-package.test.cjs |
Lines 52–58 AGENTS.md absence | Off-topic for SDK retirement, but NOT valueless: bin/install.js:~10963 writes a root AGENTS.md on local Copilot install; this is the only guard against that artifact committing. |
Extract to a repo-layout governance test (e.g. tests/repo-layout.test.cjs); rename to state the real invariant (no ad-hoc root instruction file vs CONTEXT.md/ADRs); link the copilot install path. |
tests/capability-registry.test.cjs |
Lines 558–584 drift test | Steps 2–3 are a file-I/O tautology and skip the real --check path (stripGeneratedComment + normalizeLineEndings). But it is the only unit-level coverage of the drift-comparison pipeline. |
Call the actual comparison expression: assert a stale version survives both stripGeneratedComment and normalizeLineEndings and is detected; assert a comment-only-timestamp change is NOT flagged after stripping. |
tests/verify-test-quality.test.cjs |
3 describes (disabled / circular / assertion-strength detection) | Self-referential: define a regex inline, write a matching fixture, assert the regex matches — cannot go red for any production change. The detector lives as prose in verify-phase.md. |
Replace with a structural guard reading gsd-core/workflows/verify-phase.md: assert the audit_test_quality step tag exists and contains the skip-pattern, circular-detection, and assertion-strength-table markers. |
Other redesign / keep-but-improve (high-signal selection)
| File | Location | Problem | What the rewritten test should assert |
|---|---|---|---|
tests/research-cli.test.cjs |
Line 423 | Pass-always assert.ok(true) placeholder; the 2-package arg-parse regression is undefended. |
Extract the flag parser; call with ['--bad-flag','pkgA','pkgB'] and assert unknown-flag error + that pkgA/pkgB are not consumed as flag values. (No network.) |
tests/worktree-baseref-install.test.cjs |
Line 444 | Pass-always assert.ok(true); the null-settings #683 guard is never exercised. |
assert.doesNotThrow(() => applyWorktreeBaseRef(null, tmpDir)) and assert no worktree.baseRef is written. |
tests/bug-260-worktree-path-guard.test.cjs |
Line 338 | Environment-dependent branch silently passes via assert.ok(true) when the traversal path resolves inside the worktree — a security guard goes untested on macOS/private-symlink hosts. |
Construct the traversal target in a separate tmpdir outside the worktree; remove the if/else; unconditionally assert result.status === 2 / decision block. |
tests/eslint-rules.test.cjs |
24 trailing assert.ok(true) calls |
RuleTester already throws on failure; the trailing literal asserts nothing and inflates count. | Delete the assert.ok(true) lines; optionally add assert.strictEqual(typeof rule.create, 'function') per describe to confirm the module loaded. |
tests/phase.test.cjs |
Lines 4343–4368 (last_updated refresh) |
Math.abs(new Date() - updatedAt) < 60_000 wall-clock window — flaky under loaded CI. |
Assert last_updated is a valid ISO timestamp ≠ the stale seed value AND its date portion equals today's UTC date. |
tests/bug-3707-locked-worktree-cleanup.test.cjs |
7 Date.now() stale-mtime sites |
Real-clock coupling for stale-lock boundary; violates the ADR-456 clock-seam mandate. | Pin a constant epoch (PINNED_NOW_MS) for mtime fixtures; pass it to the SUT as a clock parameter where supported. |
tests/context-utilization.property.test.cjs |
Line 187 | Math.floor(Math.random()*…) inside fc.property breaks reproducibility under the pinned seed. |
Replace with fc.integer({ min: 0, max: contextWindow }); split a shape test from a value-correctness test at known ratios. |
tests/research-provider.property.test.cjs |
Single test | Only "never throws + valid type"; inverting HIGH/LOW classification passes. | Keep robustness property; add example-based unit tests for classification boundaries; add module to mutation COVERED. |
tests/research-store.property.test.cjs |
All 3 tests | Shape/determinism only; cache-key collisions or over-normalization invisible. | Add a collision property (distinct inputs → distinct keys) + boundary examples (npm/lodash ≠ npm/react). |
tests/install.test.cjs |
Lines 683–712 (Kilo source-grep) | Reads bin/install.js as a string and .includes() internal symbol names — source-grep, breaks on rename, passes on dead code. |
Invoke install.js with --kilo in a tmpdir; assert on the installed config/manifest content (or export+call the kilo-detection logic). |
tests/sh-hook-paths.test.cjs |
Tests 2–4 (lines 119–194) | Source-grep variable-name scans of install.js/runtime-hooks-surface.cts; behavior already covered by Test 1's behavioral call. |
Call buildHookCommand per .sh hook for linux+win32; assert non-empty, bash runner, absolute quoted configDir prefix. |
tests/clusters.test.cjs |
Lines 50–58 | `assert.ok(utilitySize >= skills.length | |
tests/model-profiles.test.cjs |
Lines 115–122 | Test titled "resolution order: override > profile > default" exercises no override and asserts nothing about override behavior. | Use a temp project with config.model_overrides, call resolveModelInternal, assert override > profile > default explicitly. |
tests/active-workstream-store.unit.test.cjs |
Lines 361–364 | `assert.ok(key === null | |
tests/feat-488-effort-sync.test.cjs |
Lines 177–219 | Does not redirect GSD_HOME; resolved effort depends on the dev's real ~/.gsd; degrades to a type-only check. |
Set GSD_HOME=tmpHome; assert result.synced === 1 and changes[0].to === 'low'. |
tests/bug-492-effort-manifest-fallback.test.cjs |
Lines 20–48 | Mutates the module-level CANONICAL_CONFIG_DEFAULTS singleton — independence violation; races under parallel runs. |
Drive the same path via a tmpHome ~/.gsd/defaults.json rather than mutating shared module state. |
tests/issue-766-plugin-manifest.test.cjs |
Section C, "claude plugin validate --strict" | Permanently skipped when the claude binary is absent (i.e. on CI) — the only full-schema gate never runs. |
Install the claude CLI in a CI step, OR validate plugin.json against a snapshotted JSON schema unconditionally. |
tests/cross-ai-execution.test.cjs |
Entire file | Misattributed to ADR-15; tests a different feature (workflow.cross_ai_execution). Also a shared-mutable let content independence issue. |
Leave the file for its real feature; create tests/adr-15-plan-strategy-seam.test.cjs for the actual converge seam; hoist content to a single module-level read. |
tests/issue-844-manifest-version-sync.test.cjs |
Describe A (33–127) | Order-dependent shared tmpRoot across setup/assert/cleanup tests. |
Refactor to a single test with per-test before()/after() tmpRoot allocation. |
tests/plan-review-convergence.test.cjs |
Lines 688–692 | Four workflow.includes(verdict) whole-file substring checks — pass even if verdicts move to dead/comment text. |
Extract the source-grounding section slice; assert each verdict + its severity mapping (AMBIGUOUS→MEDIUM, UNCHECKABLE→INFO) inside that slice. |
tests/feat-3594-parser-property-style.test.cjs |
Lines 119–135 | assert.ok(elapsedMs < 2000) wall-clock; escapes no-elapsed-assertion via the variable name. |
Replace with a call-count metric (instrument char comparisons, assert O(n)); or raise bound to 10000ms as a stopgap. |
tests/core.test.cjs |
Lines 1834–1879 (reapStaleTempFiles) |
Writes to the global shared os.tmpdir()/gsd; races with temp-subdir.test.cjs under concurrency. |
Use a per-test isolated tmpdir via the existing { dir } option; clean up in afterEach. |
Prioritized action plan
P0 — do first (blast radius / false-green)
- Fix recursive test discovery in
scripts/run-tests.cjs(replace flatreaddirSyncwith the already-presentfindTestFilesRecursive). This alone reactivates ADR-227's entire UUID-validation suite (tests/observability/,tests/dispatch/) — currently silently excluded fromnpm testand CI. - Raise Stryker
breakfrom 50 → 80 instryker.config.mjs(ADR-456 mandate). Move any module that cannot meet 80% intoUNMUTATEDwith a documented per-entry reason rather than diluting the global floor. - Add tests for ADR-218 (zero coverage today): extract the release-version regex and npm-duplicate precheck into a testable
.cjshelper (or asource-text-is-the-productworkflow assertion onrelease.yml), and assert1.01.0/01.0.0are rejected, valid versions pass, and the duplicate-version step exists and is wired before the publish jobs. - Retire the 5 verified-worthless tests (see Retire table):
enh-2790name spot-checks (127–157),command-routing-hublines 134–137 and 515–519,no-cjs-sdk-handsync-tooling(all 3),runtime-artifact-layoutcline edge-case (319–323). - Redesign the 6 downgraded retire-candidates (do NOT delete):
bug-3135(delete only the broad block, keep content tests), bothreview-reviewer-selectiontests,enh-191AGENTS.md (extract to governance test),capability-registrydrift test,verify-test-quality(replace with structural workflow guard). - Close the ADR-0006 critical gap: add a regression test that
init execute-phase/plan-phase/phase-op/milestone-opreturn workstream-scoped paths underGSD_WORKSTREAM, plus a guard against the banned.planning/projects/<project>form.
P1 — high priority (rigor enforcement + concentrated weakness)
- Bring
core.ctsandconfig-loader.ctsinto the Stryker COVERED set (smallest high-value modules first); write their*.unit.test.cjsas a prerequisite. - Promote
no-source-greptoerrorand apply it totests/**, splitting product-file reads (.md/.json, allowed) from source-file grep (.cjs/.js/.ts, blocked). Then fix the surfaced violations (install.test.cjsKilo block,sh-hook-paths.test.cjsTests 2–4). - Add the
allow-test-ruletracking-issue lint gate + run the one-time sweep to attach#NNN(or an approved category tag) to the 332 untracked exemptions. - Eliminate the pass-always placeholders: redesign
research-cli.test.cjs:423,worktree-baseref-install.test.cjs:444, thebug-260environment-dependent branch, and strip the 24assert.ok(true)lines ineslint-rules.test.cjs. Add ano-tautological-assertESLint rule to prevent recurrence. - Add
no-only-testsRuleTester coverage and extendno-elapsed-assertionto catchDate.now()-startand computed-variable (elapsedMs) patterns. - Add a branch-coverage floor (
--branches 60) and a per-file branch floor for the UNMUTATED critical modules. - De-flake the clock-coupled tests:
phase.test.cjs:4362(60s window → ISO-date assertion),bug-3707(pin epoch),context-utilization.property.test.cjs:187(fc.integernotMath.random). - Add the ADR-15 primary-surface tests (
/gsd-progress --next --auto --converge) and re-attributecross-ai-execution.test.cjsto its real feature.
P2 — follow-up (robustness & hygiene)
- Fail-on-zero-executed in
run-tests.cjs(exit 1 on empty suite by default; gate the allowance behind an env flag). - Isolate the shared
GSD_TEMP_DIRreap tests (core.test.cjs:1834-1879,temp-subdir.test.cjs) via the existing{ dir }option +afterEachcleanup. - Add build-artifact
before()guards to artifact-dependent suites (ADR-3524 subjects, ADR-0011 family) so a clean worktree fails with a clear "run npm run build:lib" message instead of a top-level crash. - Strengthen the strong-but-incomplete ADRs: ADR-443 end-to-end effort propagation, ADR-0656 slopcheck "cannot-lower" + scrape-dispatch, ADR-0009's 3 absent ADR-listed test files, ADR-766
agents-absence assertion + post-#770 hook drift. - Add
.gitignore↔eslint.config.mjsignore-list parity test (ADR-457) so a migration that updates one list but not the other is caught.
Capability-cluster items (ADR-894/959/1016/1143) are deliberately deferred — they are coupled to ADR-857 (worked elsewhere) and their findings are provisional. Re-run this audit slice once ADR-857 lands.