# Testing Suites This project's `tests/` directory uses **filename suffix markers** to group tests into named suites. The harness `scripts/run-tests.cjs` filters by suite when given `--suite `. Without a flag it runs every `*.test.cjs` file (the historical default — unchanged). > Tracked by issue [#3597](https://github.com/open-gsd/gsd-core/issues/3597). ## Suites | Suite | Filename pattern | What goes here | |---|---|---| | `unit` | `*.test.cjs` (no other marker) | Default fast lane. Pure logic, no network, no external processes beyond `gsd-tools`. Most tests live here. | | `integration` | `*.integration.test.cjs` | Cross-module flows: full installer end-to-end, multi-tool orchestration, anything that crosses two or more bin entry points. | | `install` | `*.install.test.cjs` | Tests that perform a real install/uninstall against a sandbox project. Slower; PR CI skips these on PRs and runs them on `main` push only. | | `security` | `*.security.test.cjs` | Adversarial input, prompt-injection guards, fixture-driven hostile-payload sweeps. | | `slow` | `*.slow.test.cjs` | Anything that routinely takes >5s wall-clock or holds significant memory. | | `all` | (any) | Explicit alias for "no filter". Equivalent to running with no `--suite` flag. | ## How to place a new test 1. Pick the most specific bucket above. 2. Name the file with the matching suffix: `tests/..test.cjs`. 3. If unsure, leave the suffix off — the file lands in `unit`, the default fast lane. Examples: - `tests/agent-frontmatter.test.cjs` — `unit` - `tests/prompt-injection-guards.security.test.cjs` — `security` - `tests/installer-end-to-end.install.test.cjs` — `install` - `tests/sdk-mutation-stress.slow.test.cjs` — `slow` The suite-suffix convention was chosen over a directory layout (`tests/security/`) so the 545+ existing test files don't need to move. Existing files all classify as `unit` until someone explicitly retags them. ## Regression tests **Do not create new top-level `tests/bug-NNNN-*.test.cjs` files.** Add the regression case to the owning module's main test file instead (e.g. a `describe('regressions')` block in `tests/.test.cjs`). `node --test` spawns one child process per FILE, so file count — not test count — is the unit of CI overhead, and it is worst on Windows lanes where every spawn is Defender-scanned. The 2026-06 CI audit found 244 one-off `bug-*` files (~38% of the suite). That population is grandfathered in `scripts/lint-regression-test-names.allowlist.json` and enforced by an identity ratchet (`npm run lint:regression-names`, part of `npm run lint:ci`): - A **new** `bug-*` file fails CI — fold it into the owning module's file. - **Deleting/consolidating** a grandfathered file requires pruning its allowlist entry, so the baseline only ever shrinks. - **Inherited drift** (the failure names files your PR didn't add — e.g. the base branch merged `bug-*` files without feeding the allowlist, or you rebased and carried a pre-rebase allowlist): run `node scripts/lint-regression-test-names.cjs --update` and commit the regenerated allowlist. Snapshot artifacts like this allowlist (and `docs/INVENTORY.md`) must be regenerated **after** rebasing, never carried through a rebase. The ratchet deliberately covers only `bug-*`. Files named `feat-NNNN-*` / `enh-NNNN-*` are *feature* test files — one (or one per suite) per feature is the sanctioned layout (see the #443 strategy below), not a one-off regression pattern. If `issue-*`/`perf-*` one-offs start accumulating the same way `bug-*` did, extend the ratchet's regex and regenerate the allowlist. ## Workflow & agent size budget > Tracked by issue [#1074](https://github.com/open-gsd/gsd-core/issues/1074). > Bytes (not lines) per [#717](https://github.com/open-gsd/gsd-core/issues/717); > LF-normalized per [#683](https://github.com/open-gsd/gsd-core/issues/683). Workflow files (`gsd-core/workflows/*.md`) and agent files (`agents/gsd-*.md`) both ship in the installed runtime and are loaded into context — workflows on every command, agents on every subagent dispatch — so their byte size is a real cost. Two sibling guards (`tests/workflow-size-budget.test.cjs` and `tests/agent-size-budget.test.cjs`) keep that cost from creeping up invisibly, sharing one byte-counter (`measureMdFiles`). Growth is caught by two independent layers: | Layer | What it does | Where | |---|---|---| | **Differential attribution size ratchet** (primary, #2724 / ADR-2719 §4) | The same computed-attribution check that replaced the golden-install-parity fixtures also reports growth in any `gsd-core/workflows/*.md` or `agents/gsd-*.md` file, with the exact byte delta, comparing PR HEAD against `next`. Unacknowledged growth is a hard failure; shrinkage needs no acknowledgment. No committed snapshot — nothing to regenerate by hand. | `tests/emitted-attribution.test.cjs` (real-tree test) via `tests/helpers/emitted-diff.cjs` | | **Loose tier hard caps** (backstop) | Absolute outer red lines per tier — workflows: `XL ≤ 98304`, `LARGE ≤ 61440`, `DEFAULT ≤ 40960` bytes; agents: `XL ≤ 57344`, `LARGE ≤ 49152`, `DEFAULT ≤ 24576` bytes. A cap is **never raised** when a file approaches it: crossing it means *extract*, not bump. Independent of the ratchet above — unaffected by #2724. | `XL/LARGE/DEFAULT_CAP` in each guard file | `discuss-phase.md` additionally has a thin-dispatcher target of `< 32000` bytes (the discuss-phase progressive-disclosure split, #717). A net-new agent is DEFAULT-tier and already bounded by the DEFAULT cap — no separate new-agent cap is needed. (This tier-cap machinery is distinct from the separate 45 KB-*char* extraction-evidence threshold on `gsd-planner` enforced by `tests/planner-decomposition.test.cjs` — that one proves mode sections were extracted; this one bounds total agent bytes.) ### How-to: a workflow or agent grew and CI is red The differential attribution check reports the file and the byte delta. To resolve: 1. **Justify the growth in your PR** (a sentence in the description is enough) — the acknowledgment entry (below) is the review record that the larger size was a deliberate, seen decision, not silent drift. 2. **Add an acknowledgment entry** in `tests/emitted-drift-ack.json` naming the file and the reason, per `CONTEXT.md`'s `### Emitted Artifact Provenance` entry. This is deliberately a committed file, not a flag — the entry appears in your PR diff, so touching it *is* the visible signal. 3. **Or shrink it instead of acknowledging.** Prefer extraction when the growth is incidental: for a workflow, move per-mode bodies to `workflows//modes/`, templates to `workflows//templates/`, and shared prose to `gsd-core/references/`; for an agent, lift shared boilerplate into `gsd-core/references/` and `@`-reference it — then load it **LAZILY**. Do *not* convert them to eager `@-required_reading` includes: that shrinks the file's bytes without shrinking loaded context, so it games the guard while making the real cost worse. See `workflows/discuss-phase/` for the progressive-disclosure pattern. If a hard cap (not the ratchet) is what failed, an acknowledgment will **not** help — that is the signal to extract, per step 3. ### Reference | Artifact | Role | |---|---| | `scripts/workflow-size.cjs` | Single source of truth — LF-normalized byte counter (`lfByteCount`) + generic `measureMdFiles(dir, predicate)` (backs both workflows and agents) + workflow enumeration (`listWorkflowStems`, `measureWorkflows`). Imported by both guards and by `tests/helpers/emitted-runtime.cjs`'s `currentSizes()` so they can never measure differently. | | `tests/emitted-attribution.test.cjs` + `tests/helpers/emitted-diff.cjs` | The differential attribution check and its size ratchet (ADR-2719). The sole mechanism for both emitted-content propagation AND per-file size growth as of #2724. | | `tests/emitted-drift-ack.json` | Committed acknowledgment file for unattributable emitted-content ripples and for size growth. Absent = no acks; its presence is the alarm. | | `npm run regen:derived` | Runs every remaining generator in dependency order (build → registry → ADR index → capability matrix → inventory manifest → manifest versions → `tests/fixtures/install-tree/*.json`). | | `tests/workflow-size-budget.test.cjs` | The workflow tier hard-cap guards, plus the `discuss-phase` progressive-disclosure checks. | | `tests/agent-size-budget.test.cjs` | The agent tier hard-cap guards (the agent analog). | `tests/workflow-size-baseline.json`, `tests/agent-size-baseline.json`, `tests/fixtures/golden-install-parity/*.json`, `scripts/update-size-baseline.cjs` (`npm run size:baseline`), and `scripts/git-merge-regen-driver.cjs` (`npm run setup:merge-driver`) are all removed by [#2724](https://github.com/open-gsd/gsd-core/issues/2724): they were pure functions of the source tree, conflicted on every merge that touched them, and their functions are now served by the differential attribution check above. `tests/fixtures/install-tree/*.json` is the one artifact family that stays committed and normally-merged (ADR-2719 §7) — it conflicts on 0 of 7, its diffs are readable, and it preserves "the installer stopped shipping X" as a hard absolute failure with no attribution reasoning involved. Regenerate it with `npm run gen:install-tree` (folded into `npm run regen:derived`). ## Running suites locally ```bash npm test # everything (backcompat — same as before) npm run test:unit # only unit npm run test:integration # only integration npm run test:install # only install npm run test:security # only security npm run test:slow # only slow npm run test:coverage # backcompat — coverage over EVERY test npm run test:coverage:unit # fast coverage signal — only unit suite npm run test:coverage:all # alias for test:coverage ``` Direct harness invocation also works: ```bash node scripts/run-tests.cjs --suite security node scripts/run-tests.cjs --suite=security node scripts/run-tests.cjs --files "tests/command-contract.test.cjs tests/core.test.cjs" node scripts/run-tests.cjs --files-from .ci-selected-tests.txt ``` `npm run test:affected` (scripts/run-affected-tests.cjs) is a **local-only** convenience that selects tests via the `require()` dependency graph of your working-tree diff. CI does not use it — CI selection is the rule table in `scripts/ci-test-scope.cjs`, which is the authoritative mapping. If the two disagree, trust (and fix) the rule table. Unknown suites exit non-zero with the list of valid suites. Empty suites (e.g. `--suite security` before any security-tagged file exists) exit `0` with a `no tests in suite "..."` notice on stderr so CI lanes don't go red while a suite is being populated. ## CI matrix The `Tests` workflow runs every PR through a scoped gate generated by `scripts/ci-test-scope.cjs`. | Lane | Node 22 | Node 24 | |---|---|---| | `ubuntu-latest` | scoped tests | unit + integration + security | | `windows-latest` | — | scoped Windows/path/shell tests | | `macos-latest` | full parity when required | full parity when required | - **Node 22** is the `engines.node` floor (`>=22.0.0`) — must stay green. - **Node 24** is the default development lane. - **Scoped tests** are selected from the changed paths, plus a small CLI/package smoke set. They are for confidence on the affected surface, not for counting tests. The default PR gate runs the broad `unit` (under the c8 coverage gate), `integration`, and `security` suites once on Ubuntu / Node 24, scoped tests on Ubuntu / Node 22, and scoped tests on Windows / Node 24. "Scoped" means the diff-selected list from the rule table — not the full suite and not a fixed smoke set (the fixed smoke list is only the empty-selection fallback). The Windows lane's list is the Windows-sensitive subset of the selection, plus **every changed test file, unconditionally** (the #494 invariant, narrowed): a modified test is exercised on the divergent OS before merge at per-file cost, without paying for the three full parity lanes. PRs touching workflow, package, test-runner, install, release, or Windows-sensitive surfaces also run the full parity matrix on macOS and the older Windows runtime, plus `install` and `slow` on the primary Ubuntu lane. Everything (including the full parity matrix) runs on every push to `next`, which covers the residual macOS / Windows-Node-22 cross-product for scoped PRs. Coverage runs inside the Ubuntu / Node 24 full lane (not a separate job — that duplicated the entire unit run) and stays single-lane because multiplying coverage across OS/runtime lanes adds cost without improving the threshold signal. Note the gate's deliberate blind spot: it measures `gsd-core/bin/lib/*.cjs` only — `scripts/`, `hooks/`, and `bin/` are unenforced, and `stryker.config.mjs` additionally excludes ~48% of lib lines from mutation testing (see the UNMUTATED list there). Widening either gate is tracked work, not an accident to "fix" silently by raising thresholds. To inspect the scope locally: ```bash npm run ci:test-scope -- --files "commands/gsd/plan-phase.md" node scripts/ci-test-scope.cjs --base origin/next --head HEAD ``` ## Chunk packing and the test timing table `scripts/run-tests.cjs` does not hand the whole selected file list to one `node --test` process. It packs the files into **chunks**, each spawned separately, because Windows caps a command line at 32,767 characters and because each chunk gets its own 600s timeout (`RUN_TESTS_CHUNK_TIMEOUT_MS`) and a fresh process, which bounds memory pressure. How files are distributed across those chunks decides whether the slowest chunk sits near that timeout while the others idle. The packer weights each file by its **measured duration**, read from `tests/test-timings.json`, and places files with LPT (longest-processing-time-first: heaviest file first, each into the currently lightest chunk). Before #2456 the weight was guessed from the filename, which mis-ranked files badly enough that the slowest chunk ran ~3.9x the lightest. ### Reference | Knob | Default | Meaning | |---|---|---| | `RUN_TESTS_MAX_FILES_PER_CHUNK` | `60` | Per-chunk weight budget. Weights are normalized so an **average-cost** file weighs 1, so this still reads as "about 60 average files". | | `RUN_TESTS_MAX_CMDLINE_CHARS` | `28000` | argv ceiling per chunk, with headroom under the Windows 32,767 limit. | | `RUN_TESTS_TIMINGS_FILE` | `tests/test-timings.json` | Path to the timing table. Tests override it to inject a synthetic cost profile. | | `RUN_TESTS_CHUNK_TIMEOUT_MS` | `600000` | Per-chunk timeout. | The timing table is **advisory and deliberately un-gated**. There is no `--check` mode and no CI lint that fails on staleness, because timing data legitimately varies run to run. A file missing from the table falls back to the table's median weight, and a missing or unparseable table falls back to uniform weight — so drift costs chunk *balance*, never a red build. A count-based floor additionally guarantees the packer never produces fewer chunks than plain count-based packing would, so a badly stale table cannot collapse the suite into a few fat chunks. ### How-to: regenerate the timing table Regenerate when the suite's cost profile has visibly drifted — after adding or removing expensive tests, not on a schedule. The input is a `node:test` reporter event stream from a `gsd-test` run: ```bash node scripts/gen-test-timings.cjs \ ~/.local/state/gsd-test/runs//test-events-linux-node22.jsonl \ ~/.local/state/gsd-test/runs//test-events-linux-node24.jsonl ``` Pass every lane you have. A file's recorded time is the **max** across the supplied streams, not the mean: the packer exists to keep the *slowest* lane's slowest chunk away from the timeout, so the conservative bound is the right one. Keys are sorted so a regeneration diff shows only the files whose cost moved. ## Best practices for forward-compat (Node 24/26) - Use `process.execPath` when spawning Node in tests so each matrix lane exercises the lane's Node version. - Avoid stack-trace or error-message prose assertions. Assert `err.code`, structured JSON fields, or enums — Node minor releases routinely tweak error wording. - Prefer `node:test`, `node:assert/strict`, and `node:test` mocks. No external test frameworks. - Coverage uses `c8` and propagates `NODE_V8_COVERAGE` through the harness's child process. --- ## Test strategy: #443 effort + fast_mode engine > Feature: unified cross-provider effort and fast_mode knobs (issue #443). > Test files: `tests/model-resolver.test.cjs` (unit), > `tests/model-resolver.test.cjs` (integration). ### Testing pyramid | Layer | File | What it covers | |---|---|---| | **Unit** | `feat-443-effort-fast-mode.test.cjs` | Pure logic: cascade rules, clamping, escalation math, malformed config handling, schema key validation. No CLI subprocess. | | **Integration** | `feat-443-effort-fast-mode.integration.test.cjs` | Architecture-level invariants: cross-provider validity, totality across the 33-agent registry, CLI JSON contract, config round-trip, fast-mode honesty. Real subprocesses via `runGsdTools`. | | **E2E** *(pending)* | *(not yet wired)* | Propagation layer: effort frontmatter / `CLAUDE_CODE_EFFORT_LEVEL` env actually reaching a spawned Claude Code subagent. See "Gaps" below. | ### Architectural invariants Each invariant exists to prevent a specific class of production failure. #### (a) Cross-provider validity **What:** `renderEffortForRuntime(runtime, universalEffort).value` must always be a member of the runtime's real provider enum. Ground-truth enums are defined as local constants in the test — not sourced from the implementation. ``` PROVIDER_EFFORT_ENUMS = { claude: Set { 'low', 'medium', 'high', 'xhigh', 'max' } // Anthropic output_config.effort codex: Set { 'minimal', 'low', 'medium', 'high', 'xhigh' } // OpenAI model_reasoning_effort } ``` **Why:** Passing a value outside these sets results in a 400 from the real API. The clamping logic (`max -> xhigh` for codex; `minimal -> low` for claude) must hold for every cell of the VALID_EFFORTS × runtimes matrix. #### (b) Param/channel contract **What:** Each runtime exposes a stable `param` string (the native API field name) and `channel` (how the value is propagated). Unknown runtimes return `param: null, channel: null` and pass the effort value through unchanged. **Why:** Callers read `.param` to construct the dispatch payload. A regression here would silently drop effort from subagent invocations. #### (c) Resolve-execution JSON contract **What:** The `gsd-tools resolve-execution ` command emits a JSON object with all eight keys present and typed correctly: `model` (string), `profile` (string), `effort` (VALID_EFFORTS member), `effort_rendered` (string), `effort_param` (string|null), `effort_propagation` (string|null), `fast_mode` (boolean), `fast_mode_supported` (boolean). **Why:** Orchestrators and workflow dispatchers parse this JSON. A missing or mistyped field silently breaks downstream consumers. #### (d) Totality across the real registry **What:** For every agent in the 33-agent registry, `resolveEffortInternal` returns a VALID_EFFORTS member (never undefined/null), `resolveFastModeInternal` returns a strict boolean, and `renderEffortForRuntime('claude', effort)` stays within the claude provider enum. **Why:** A catalog addition that introduces a missing `routingTier` mapping would otherwise produce `undefined` and propagate silently. #### (e) Fast-mode honesty invariant **What:** When the runtime is `claude`, `fast_mode_supported` in resolve-execution output is always `false`, regardless of the fast_mode config. `RUNTIMES_WITH_FAST_MODE` contains only `'api'`. **Why:** Claude Code's `/fast` toggle is session-level only. Emitting `fast_mode: true` as frontmatter on a Claude subagent is a silent no-op. Advertising `fast_mode_supported: true` for claude would cause orchestrators to believe the knob was wired when it is not. #### (f) Precedence first-valid-wins **What:** Both effort and fast_mode use a layered cascade. The test table covers all four effort layers (invocation override → agent_overrides → routing_tier_defaults → default) and all five fast_mode layers, including the case where an invalid value at a higher layer correctly falls through. **Why:** Silent precedence bugs (e.g., a numeric value in agent_overrides not being rejected) would override intentional user config. #### (g) Dynamic-routing composition **What:** `resolveEffortForTier` escalates effort by attempt number independently of the model tier mapping. The test verifies the effort ladder (`low -> medium -> high -> xhigh -> max`), the `max` clamp, the `max_escalations` cap, and that `escalate_on_failure: false` suppresses escalation entirely. **Why:** Effort escalation and model escalation share configuration (`dynamic_routing`) but must operate independently; coupling them would cause over-escalation or under-escalation. #### (h) Config-tooling round-trip **What:** `gsd-tools config-set` accepts all new key namespaces (`effort.default`, `effort.routing_tier_defaults.`, `effort.agent_overrides.`, `fast_mode.enabled`, `fast_mode.routing_tier_defaults.`, `fast_mode.agent_overrides.`) without an "Unknown config key" error, and values set via `config-set` are reflected in `resolve-execution` output. **Why:** The schema validation gate (`VALID_CONFIG_KEYS` + `DYNAMIC_KEY_PATTERNS`) is separate from the resolver logic. A key missing from the schema would produce a silent write failure and appear as a bug only at runtime. ### Coverage targets | Suite | Target | |---|---| | Unit | Every cascade rule, every fallthrough, every clamp. All function branches in `resolveEffortInternal`, `resolveFastModeInternal`, `resolveEffortForTier`, `renderEffortForRuntime`. | | Integration | All 8 architectural invariants. All 33 registered agents. All 6 provider × effort combinations for the valid-enum check. Full config-set key namespace. | ### Gaps / not yet covered **E2E orchestrator-spawn-propagation layer (pending follow-up wiring):** The integration tests verify that GSD resolves and renders effort values correctly. They do NOT verify that the rendered values actually reach a spawned Claude Code or Codex subagent at runtime. Specifically uncovered: - `CLAUDE_CODE_EFFORT_LEVEL` env var being set and read by a spawned claude subprocess - `output_config.effort` frontmatter key surviving the AGENTS.md template substitution - `model_reasoning_effort` field surviving serialization into a Codex API request body - Fast-mode `speed: "fast"` field reaching an `api`-runtime request when `fast_mode_supported: true` These require spawning real subagents (or stubs thereof) and asserting on the process environment / request payload — a scope that belongs in a future E2E suite under `*.slow.test.cjs` or dedicated fixture-driven integration work.