The changeset for this PR is typed `Added`, and CONTRIBUTING requires a docs/ change for that type. The only docs/ file in the diff was CONTEXT-INDEX.json -- a GENERATED index -- so the Docs Required gate passed while no human-readable documentation existed for either new variable. A gate satisfied by a generated artifact is satisfied vacuously. docs/TESTING-SUITES.md now carries a section on the guard: what it watches and why it is ownership-scoped rather than whole-root, the two env vars in a table, why the default is report-only and what the promotion condition is, and what each violation label means (including that UNVERIFIED is not clean). GSD_SKIP_LIVE_CONFIG_GUARD is named explicitly because it is a bypass on a safety check. An undocumented bypass is one people eventually set without knowing what they turned off. A test asserts both variables appear in that doc -- checked as permitted by local/no-source-grep before writing it, rather than assumed forbidden. It fails when the section is removed, so the doc cannot rot back to the state the review found. Addresses review finding: Major 4.
503 lines
28 KiB
Markdown
503 lines
28 KiB
Markdown
# Testing Suites
|
||
|
||
This project's `tests/` directory uses **filename suffix markers** to group tests into named suites. The harness `scripts/run-tests.cjs` filters by suite when given `--suite <name>`. Without a flag it runs every `*.test.cjs` file (the historical default — unchanged).
|
||
|
||
> Tracked by issue [#3597](https://github.com/open-gsd/gsd-core/issues/3597).
|
||
|
||
## Suites
|
||
|
||
| Suite | Filename pattern | What goes here |
|
||
|---|---|---|
|
||
| `unit` | `*.test.cjs` (no other marker) | Default fast lane. Pure logic, no network, no external processes beyond `gsd-tools`. Most tests live here. |
|
||
| `integration` | `*.integration.test.cjs` | Cross-module flows: full installer end-to-end, multi-tool orchestration, anything that crosses two or more bin entry points. |
|
||
| `install` | `*.install.test.cjs` | Tests that perform a real install/uninstall against a sandbox project. Slower; PR CI skips these on PRs and runs them on `main` push only. |
|
||
| `security` | `*.security.test.cjs` | Adversarial input, prompt-injection guards, fixture-driven hostile-payload sweeps. |
|
||
| `slow` | `*.slow.test.cjs` | Anything that routinely takes >5s wall-clock or holds significant memory. |
|
||
| `qa` | `*.qa.test.cjs` | End-to-end walks that drive the real `gsd-tools` binary across multiple loop steps against one accumulating temp project, with invariant oracles after every step. Slower than `unit`; excluded from the fast lane. |
|
||
| `all` | (any) | Explicit alias for "no filter". Equivalent to running with no `--suite` flag. |
|
||
|
||
## How to place a new test
|
||
|
||
1. Pick the most specific bucket above.
|
||
2. Name the file with the matching suffix: `tests/<feature>.<suite>.test.cjs`.
|
||
3. If unsure, leave the suffix off — the file lands in `unit`, the default fast lane.
|
||
|
||
Examples:
|
||
- `tests/agent-frontmatter.test.cjs` — `unit`
|
||
- `tests/prompt-injection-guards.security.test.cjs` — `security`
|
||
- `tests/installer-end-to-end.install.test.cjs` — `install`
|
||
- `tests/sdk-mutation-stress.slow.test.cjs` — `slow`
|
||
- `tests/loop-walk.qa.test.cjs` — `qa`
|
||
|
||
The suite-suffix convention was chosen over a directory layout (`tests/security/`) so the 545+ existing test files don't need to move. Existing files all classify as `unit` until someone explicitly retags them.
|
||
|
||
## Regression tests
|
||
|
||
**Do not create new top-level `tests/bug-NNNN-*.test.cjs` files.** Add the
|
||
regression case to the owning module's main test file instead (e.g. a
|
||
`describe('regressions')` block in `tests/<module>.test.cjs`).
|
||
|
||
`node --test` spawns one child process per FILE, so file count — not test
|
||
count — is the unit of CI overhead, and it is worst on Windows lanes where
|
||
every spawn is Defender-scanned. The 2026-06 CI audit found 244 one-off
|
||
`bug-*` files (~38% of the suite). That population is grandfathered in
|
||
`scripts/lint-regression-test-names.allowlist.json` and enforced by an
|
||
identity ratchet (`npm run lint:regression-names`, part of `npm run lint:ci`):
|
||
|
||
- A **new** `bug-*` file fails CI — fold it into the owning module's file.
|
||
- **Deleting/consolidating** a grandfathered file requires pruning its
|
||
allowlist entry, so the baseline only ever shrinks.
|
||
- **Inherited drift** (the failure names files your PR didn't add — e.g. the
|
||
base branch merged `bug-*` files without feeding the allowlist, or you
|
||
rebased and carried a pre-rebase allowlist): run
|
||
`node scripts/lint-regression-test-names.cjs --update` and commit the
|
||
regenerated allowlist. Snapshot artifacts like this allowlist (and
|
||
`docs/INVENTORY.md`) must be regenerated **after** rebasing, never carried
|
||
through a rebase.
|
||
|
||
The ratchet deliberately covers only `bug-*`. Files named `feat-NNNN-*` /
|
||
`enh-NNNN-*` are *feature* test files — one (or one per suite) per feature is
|
||
the sanctioned layout (see the #443 strategy below), not a one-off regression
|
||
pattern. If `issue-*`/`perf-*` one-offs start accumulating the same way
|
||
`bug-*` did, extend the ratchet's regex and regenerate the allowlist.
|
||
|
||
## Workflow & agent size budget
|
||
|
||
> Tracked by issue [#1074](https://github.com/open-gsd/gsd-core/issues/1074).
|
||
> Bytes (not lines) per [#717](https://github.com/open-gsd/gsd-core/issues/717);
|
||
> LF-normalized per [#683](https://github.com/open-gsd/gsd-core/issues/683).
|
||
|
||
Workflow files (`gsd-core/workflows/*.md`) and agent files (`agents/gsd-*.md`)
|
||
both ship in the installed runtime and are loaded into context — workflows on
|
||
every command, agents on every subagent dispatch — so their byte size is a real
|
||
cost. Two sibling guards (`tests/workflow-size-budget.test.cjs` and
|
||
`tests/agent-size-budget.test.cjs`) keep that cost from creeping up invisibly,
|
||
sharing one byte-counter (`measureMdFiles`). Growth is caught by two independent
|
||
layers:
|
||
|
||
| Layer | What it does | Where |
|
||
|---|---|---|
|
||
| **Differential attribution size ratchet** (primary, #2724 / ADR-2719 §4) | The same computed-attribution check that replaced the golden-install-parity fixtures also reports growth in any `gsd-core/workflows/*.md` or `agents/gsd-*.md` file, with the exact byte delta, comparing PR HEAD against `next`. Unacknowledged growth is a hard failure; shrinkage needs no acknowledgment. No committed snapshot — nothing to regenerate by hand. | `tests/emitted-attribution.test.cjs` (real-tree test) via `tests/helpers/emitted-diff.cjs` |
|
||
| **Loose tier hard caps** (backstop) | Absolute outer red lines per tier — workflows: `XL ≤ 98304`, `LARGE ≤ 61440`, `DEFAULT ≤ 40960` bytes; agents: `XL ≤ 57344`, `LARGE ≤ 49152`, `DEFAULT ≤ 24576` bytes. A cap is **never raised** when a file approaches it: crossing it means *extract*, not bump. Independent of the ratchet above — unaffected by #2724. | `XL/LARGE/DEFAULT_CAP` in each guard file |
|
||
|
||
`discuss-phase.md` additionally has a thin-dispatcher target of `< 32000` bytes
|
||
(the discuss-phase progressive-disclosure split, #717). A net-new agent is
|
||
DEFAULT-tier and already bounded by the DEFAULT cap — no separate new-agent cap
|
||
is needed. (This tier-cap machinery is distinct from the separate 45 KB-*char*
|
||
extraction-evidence threshold on `gsd-planner` enforced by
|
||
`tests/planner-decomposition.test.cjs` — that one proves mode sections were
|
||
extracted; this one bounds total agent bytes.)
|
||
|
||
### How-to: a workflow or agent grew and CI is red
|
||
|
||
The differential attribution check reports the file and the byte delta. To resolve:
|
||
|
||
1. **Justify the growth in your PR** (a sentence in the description is enough) —
|
||
the acknowledgment entry (below) is the review record that the larger size
|
||
was a deliberate, seen decision, not silent drift.
|
||
2. **Add an acknowledgment entry** in `tests/emitted-drift-ack.json` naming the
|
||
file and the reason, per `CONTEXT.md`'s `### Emitted Artifact Provenance`
|
||
entry. This is deliberately a committed file, not a flag — the entry appears
|
||
in your PR diff, so touching it *is* the visible signal.
|
||
3. **Or shrink it instead of acknowledging.** Prefer extraction when the growth
|
||
is incidental: for a workflow, move per-mode bodies to
|
||
`workflows/<name>/modes/`, templates to `workflows/<name>/templates/`, and
|
||
shared prose to `gsd-core/references/`; for an agent, lift shared boilerplate
|
||
into `gsd-core/references/` and `@`-reference it — then load it **LAZILY**. Do
|
||
*not* convert them to eager `@-required_reading` includes: that shrinks the
|
||
file's bytes without shrinking loaded context, so it games the guard while
|
||
making the real cost worse. See `workflows/discuss-phase/` for the
|
||
progressive-disclosure pattern.
|
||
|
||
If a hard cap (not the ratchet) is what failed, an acknowledgment will **not**
|
||
help — that is the signal to extract, per step 3.
|
||
|
||
### Reference
|
||
|
||
| Artifact | Role |
|
||
|---|---|
|
||
| `scripts/workflow-size.cjs` | Single source of truth — LF-normalized byte counter (`lfByteCount`) + generic `measureMdFiles(dir, predicate)` (backs both workflows and agents) + workflow enumeration (`listWorkflowStems`, `measureWorkflows`). Imported by both guards and by `tests/helpers/emitted-runtime.cjs`'s `currentSizes()` so they can never measure differently. |
|
||
| `tests/emitted-attribution.test.cjs` + `tests/helpers/emitted-diff.cjs` | The differential attribution check and its size ratchet (ADR-2719). The sole mechanism for both emitted-content propagation AND per-file size growth as of #2724. |
|
||
| `tests/emitted-drift-ack.json` | Committed acknowledgment file for unattributable emitted-content ripples and for size growth. Absent = no acks; its presence is the alarm. |
|
||
| `npm run regen:derived` | Runs every remaining generator in dependency order (build → registry → ADR index → capability matrix → inventory manifest → manifest versions → `tests/fixtures/install-tree/*.json`). |
|
||
| `tests/workflow-size-budget.test.cjs` | The workflow tier hard-cap guards, plus the `discuss-phase` progressive-disclosure checks. |
|
||
| `tests/agent-size-budget.test.cjs` | The agent tier hard-cap guards (the agent analog). |
|
||
|
||
`tests/workflow-size-baseline.json`, `tests/agent-size-baseline.json`,
|
||
`tests/fixtures/golden-install-parity/*.json`, `scripts/update-size-baseline.cjs`
|
||
(`npm run size:baseline`), and `scripts/git-merge-regen-driver.cjs`
|
||
(`npm run setup:merge-driver`) are all removed by
|
||
[#2724](https://github.com/open-gsd/gsd-core/issues/2724): they were pure
|
||
functions of the source tree, conflicted on every merge that touched them, and
|
||
their functions are now served by the differential attribution check above.
|
||
`tests/fixtures/install-tree/*.json` is the one artifact family that stays
|
||
committed and normally-merged (ADR-2719 §7) — it conflicts on 0 of 7, its diffs
|
||
are readable, and it preserves "the installer stopped shipping X" as a hard
|
||
absolute failure with no attribution reasoning involved. Regenerate it with
|
||
`npm run gen:install-tree` (folded into `npm run regen:derived`).
|
||
|
||
## The QA smell ratchet
|
||
|
||
`tests/loop-walk.qa.test.cjs` (the `qa` suite) is the QA-walk harness's own
|
||
self-test. Separately, `scripts/qa-smell-ratchet.cjs` drives that same harness
|
||
end to end against the real `gsd-tools` binary and turns its findings into a
|
||
CI gate — run it with `npm run lint:qa-smells`.
|
||
|
||
The harness's oracles (`tests/qa/oracles.cjs`) distinguish two severities:
|
||
|
||
- A **violation** is the engine breaking a documented contract. It always
|
||
fails the build — baseline or no baseline, acknowledged or not.
|
||
- A **smell** is legal-but-questionable behavior. A smell **never fails a
|
||
build on its own merits**. What fails is the absence of a decision about
|
||
it: an **unacknowledged NEW smell**, or a **STALE** entry in
|
||
`tests/qa/smell-baseline.json` (one that stopped firing — the baseline is
|
||
shrink-only, so a fixed or changed scenario must be pruned, not left
|
||
behind).
|
||
|
||
Every smell must terminate in exactly one of TWO states — there is no third
|
||
"accepted with a good explanation" state:
|
||
|
||
1. **REAL** — an assigned defect. File it, then acknowledge the smell with an
|
||
entry (baseline entry or `tests/qa/smell-acks/` fragment) carrying that
|
||
`issue` number.
|
||
2. **FALSE POSITIVE** — the oracle itself is wrong. Fix the oracle
|
||
(`tests/qa/oracles.cjs`) so it stops firing. It is NEVER baselined.
|
||
|
||
When the ratchet reports a NEW smell, there are exactly two legitimate
|
||
responses — fix the detector, or file a defect and cite its issue number:
|
||
|
||
1. **Fix the underlying behavior (or the oracle, if it's a false positive)**
|
||
so the smell stops firing.
|
||
2. **File a defect and acknowledge it** by adding a fragment under
|
||
`tests/qa/smell-acks/` — the ratchet's failure output prints a paste-ready
|
||
skeleton naming the required `key`, `id`, `scenario`, and `issue` fields.
|
||
`issue` MUST be a positive integer naming the tracking issue; a free-text
|
||
`reason` may accompany it as an optional human note but can NEVER
|
||
substitute for `issue` — "write an explanation" is not a way to acknowledge
|
||
a smell. See `tests/qa/smell-acks/README.md` for the full shape and
|
||
lifecycle.
|
||
|
||
Run `node scripts/qa-smell-ratchet.cjs --update` to regenerate
|
||
`tests/qa/smell-baseline.json` from the current run, folding in any acked
|
||
fragments and pruning stale entries. `--update` never invents an issue
|
||
number: a genuinely new smell is written with `issue: null` and a TODO
|
||
`reason`, and the very next plain (non-`--update`) run REJECTS that entry —
|
||
forcing a human to triage it before it can ship. The baseline only ever
|
||
shrinks: growth happens by adding an acknowledgment carrying a real issue
|
||
number (a reviewable diff), never by widening the generator's tolerance and
|
||
never by prose alone.
|
||
|
||
## Running suites locally
|
||
|
||
```bash
|
||
npm test # everything (backcompat — same as before)
|
||
npm run test:unit # only unit
|
||
npm run test:integration # only integration
|
||
npm run test:install # only install
|
||
npm run test:security # only security
|
||
npm run test:slow # only slow
|
||
|
||
npm run test:coverage # backcompat — coverage over EVERY test
|
||
npm run test:coverage:unit # fast coverage signal — only unit suite
|
||
npm run test:coverage:all # alias for test:coverage
|
||
```
|
||
|
||
Direct harness invocation also works:
|
||
|
||
```bash
|
||
node scripts/run-tests.cjs --suite security
|
||
node scripts/run-tests.cjs --suite=security
|
||
node scripts/run-tests.cjs --files "tests/command-contract.test.cjs tests/core.test.cjs"
|
||
node scripts/run-tests.cjs --files-from .ci-selected-tests.txt
|
||
```
|
||
|
||
`npm run test:affected` (scripts/run-affected-tests.cjs) is a **local-only**
|
||
convenience that selects tests via the `require()` dependency graph of your
|
||
working-tree diff. CI does not use it — CI selection is the rule table in
|
||
`scripts/ci-test-scope.cjs`, which is the authoritative mapping. If the two
|
||
disagree, trust (and fix) the rule table.
|
||
|
||
Unknown suites exit non-zero with the list of valid suites. Empty suites (e.g. `--suite security` before any security-tagged file exists) exit `0` with a `no tests in suite "..."` notice on stderr so CI lanes don't go red while a suite is being populated.
|
||
|
||
## The live-config hermeticity guard
|
||
|
||
Every `run-tests.cjs` invocation snapshots GSD's own install footprint in each
|
||
live runtime config directory before the suite and re-checks it afterwards. It
|
||
exists because the failure it catches is silent by construction: a test that
|
||
resolves a config directory from the ambient environment instead of a sandbox
|
||
writes into *your real* `~/.claude` (or `$GSD_HOME/.gsd`, or a Kimi
|
||
`config.toml`), and nothing reports it. CI cannot catch this class at all —
|
||
CI never has `CLAUDE_CONFIG_DIR` and friends set.
|
||
|
||
The guard watches only what GSD unambiguously owns — its top-level install
|
||
footprint plus `gsd-`-prefixed children of directories shared with the host
|
||
agent — never whole config roots, because a host agent legitimately writing
|
||
`history.jsonl` mid-run would make the guard cry wolf, and a guard that cries
|
||
wolf gets switched off.
|
||
|
||
Two environment variables control it:
|
||
|
||
| Variable | Effect |
|
||
|---|---|
|
||
| `GSD_STRICT_LIVE_CONFIG_GUARD=1` | A detected write **fails the run**. Set on the Linux/macOS lanes of every CI job that runs the suite. |
|
||
| `GSD_SKIP_LIVE_CONFIG_GUARD=1` | Skips the check entirely. |
|
||
|
||
Unset, the guard **reports and does not fail** — deliberately, not timidly. On
|
||
its first CI run it surfaced pre-existing leaks on the Windows lane, where
|
||
`os.homedir()` reads `USERPROFILE` and ~190 test sites sandbox `HOME` alone.
|
||
Those are real and worth fixing, but they are a different defect class, and a
|
||
brand-new gate that instantly reds an unrelated lane gets reverted rather than
|
||
obeyed. Windows lanes therefore stay report-only until that sweep lands; this
|
||
repo has the pattern already, in the `local/no-source-grep` ESLint rule that
|
||
shipped at `warn` and was promoted to `error` after its cleanup (ADR 452).
|
||
|
||
`GSD_SKIP_LIVE_CONFIG_GUARD` is a bypass on a safety check, so it is documented
|
||
here rather than left to be discovered in the source: an undocumented bypass is
|
||
one people eventually set without knowing what they turned off. If you need it
|
||
routinely, that is a bug report, not a workflow.
|
||
|
||
Reported paths are labelled `CREATED`, `MODIFIED`, `DELETED`, or `UNVERIFIED`.
|
||
`UNVERIFIED` means a scan bound was hit and the path could not be attested
|
||
either way — it is never the same as clean.
|
||
|
||
## CI matrix
|
||
|
||
The `Tests` workflow runs every PR through a scoped gate generated by
|
||
`scripts/ci-test-scope.cjs`.
|
||
|
||
| Lane | Node 22 | Node 24 |
|
||
|---|---|---|
|
||
| `ubuntu-latest` | scoped tests | unit + integration + security |
|
||
| `windows-latest` | — | scoped Windows/path/shell tests |
|
||
| `macos-latest` | full parity when required | full parity when required |
|
||
|
||
- **Node 22** is the `engines.node` floor (`>=22.0.0`) — must stay green.
|
||
- **Node 24** is the default development lane.
|
||
- **Scoped tests** are selected from the changed paths, plus a small CLI/package
|
||
smoke set. They are for confidence on the affected surface, not for counting
|
||
tests.
|
||
|
||
The default PR gate runs the broad `unit` (under the c8 coverage gate),
|
||
`integration`, and `security` suites once on Ubuntu / Node 24, scoped tests on
|
||
Ubuntu / Node 22, and scoped tests on Windows / Node 24. "Scoped" means the
|
||
diff-selected list from the rule table — not the full suite and not a fixed
|
||
smoke set (the fixed smoke list is only the empty-selection fallback). The
|
||
Windows lane's list is the Windows-sensitive subset of the selection, plus
|
||
**every changed test file, unconditionally** (the #494 invariant, narrowed): a
|
||
modified test is exercised on the divergent OS before merge at per-file cost,
|
||
without paying for the three full parity lanes.
|
||
|
||
PRs touching workflow, package, test-runner, install, release, or
|
||
Windows-sensitive surfaces also run the full parity matrix on macOS and the
|
||
older Windows runtime, plus `install` and `slow` on the primary Ubuntu lane.
|
||
Everything (including the full parity matrix) runs on every push to `next`,
|
||
which covers the residual macOS / Windows-Node-22 cross-product for scoped PRs.
|
||
|
||
Coverage runs inside the Ubuntu / Node 24 full lane (not a separate job — that
|
||
duplicated the entire unit run) and stays single-lane because multiplying
|
||
coverage across OS/runtime lanes adds cost without improving the threshold
|
||
signal. Note the gate's deliberate blind spot: it measures
|
||
`gsd-core/bin/lib/*.cjs` only — `scripts/`, `hooks/`, and `bin/` are
|
||
unenforced, and `stryker.config.mjs` additionally excludes ~48% of lib lines
|
||
from mutation testing (see the UNMUTATED list there). Widening either gate is
|
||
tracked work, not an accident to "fix" silently by raising thresholds.
|
||
|
||
To inspect the scope locally:
|
||
|
||
```bash
|
||
npm run ci:test-scope -- --files "commands/gsd/plan-phase.md"
|
||
node scripts/ci-test-scope.cjs --base origin/next --head HEAD
|
||
```
|
||
|
||
## Chunk packing and the test timing table
|
||
|
||
`scripts/run-tests.cjs` does not hand the whole selected file list to one
|
||
`node --test` process. It packs the files into **chunks**, each spawned
|
||
separately, because Windows caps a command line at 32,767 characters and because
|
||
each chunk gets its own 600s timeout (`RUN_TESTS_CHUNK_TIMEOUT_MS`) and a fresh
|
||
process, which bounds memory pressure.
|
||
|
||
How files are distributed across those chunks decides whether the slowest chunk
|
||
sits near that timeout while the others idle. The packer weights each file by its
|
||
**measured duration**, read from `tests/test-timings.json`, and places files with
|
||
LPT (longest-processing-time-first: heaviest file first, each into the currently
|
||
lightest chunk). Before #2456 the weight was guessed from the filename, which
|
||
mis-ranked files badly enough that the slowest chunk ran ~3.9x the lightest.
|
||
|
||
### Reference
|
||
|
||
| Knob | Default | Meaning |
|
||
|---|---|---|
|
||
| `RUN_TESTS_MAX_FILES_PER_CHUNK` | `60` | Per-chunk weight budget. Weights are normalized so an **average-cost** file weighs 1, so this still reads as "about 60 average files". |
|
||
| `RUN_TESTS_MAX_CMDLINE_CHARS` | `28000` | argv ceiling per chunk, with headroom under the Windows 32,767 limit. |
|
||
| `RUN_TESTS_TIMINGS_FILE` | `tests/test-timings.json` | Path to the timing table. Tests override it to inject a synthetic cost profile. |
|
||
| `RUN_TESTS_CHUNK_TIMEOUT_MS` | `600000` | Per-chunk timeout. |
|
||
|
||
The timing table is **advisory and deliberately un-gated**. There is no `--check`
|
||
mode and no CI lint that fails on staleness, because timing data legitimately
|
||
varies run to run. A file missing from the table falls back to the table's median
|
||
weight, and a missing or unparseable table falls back to uniform weight — so
|
||
drift costs chunk *balance*, never a red build. A count-based floor additionally
|
||
guarantees the packer never produces fewer chunks than plain count-based packing
|
||
would, so a badly stale table cannot collapse the suite into a few fat chunks.
|
||
|
||
### How-to: regenerate the timing table
|
||
|
||
Regenerate when the suite's cost profile has visibly drifted — after adding or
|
||
removing expensive tests, not on a schedule. The input is a `node:test` reporter
|
||
event stream from a `gsd-test` run:
|
||
|
||
```bash
|
||
node scripts/gen-test-timings.cjs \
|
||
~/.local/state/gsd-test/runs/<run-id>/test-events-linux-node22.jsonl \
|
||
~/.local/state/gsd-test/runs/<run-id>/test-events-linux-node24.jsonl
|
||
```
|
||
|
||
Pass every lane you have. A file's recorded time is the **max** across the
|
||
supplied streams, not the mean: the packer exists to keep the *slowest* lane's
|
||
slowest chunk away from the timeout, so the conservative bound is the right one.
|
||
Keys are sorted so a regeneration diff shows only the files whose cost moved.
|
||
|
||
## Best practices for forward-compat (Node 24/26)
|
||
|
||
- Use `process.execPath` when spawning Node in tests so each matrix lane exercises the lane's Node version.
|
||
- Avoid stack-trace or error-message prose assertions. Assert `err.code`, structured JSON fields, or enums — Node minor releases routinely tweak error wording.
|
||
- Prefer `node:test`, `node:assert/strict`, and `node:test` mocks. No external test frameworks.
|
||
- Coverage uses `c8` and propagates `NODE_V8_COVERAGE` through the harness's child process.
|
||
|
||
---
|
||
|
||
## Test strategy: #443 effort + fast_mode engine
|
||
|
||
> Feature: unified cross-provider effort and fast_mode knobs (issue #443).
|
||
> Test files: `tests/model-resolver.test.cjs` (unit),
|
||
> `tests/model-resolver.test.cjs` (integration).
|
||
|
||
### Testing pyramid
|
||
|
||
| Layer | File | What it covers |
|
||
|---|---|---|
|
||
| **Unit** | `feat-443-effort-fast-mode.test.cjs` | Pure logic: cascade rules, clamping, escalation math, malformed config handling, schema key validation. No CLI subprocess. |
|
||
| **Integration** | `feat-443-effort-fast-mode.integration.test.cjs` | Architecture-level invariants: cross-provider validity, totality across the 33-agent registry, CLI JSON contract, config round-trip, fast-mode honesty. Real subprocesses via `runGsdTools`. |
|
||
| **E2E** *(pending)* | *(not yet wired)* | Propagation layer: effort frontmatter / `CLAUDE_CODE_EFFORT_LEVEL` env actually reaching a spawned Claude Code subagent. See "Gaps" below. |
|
||
|
||
### Architectural invariants
|
||
|
||
Each invariant exists to prevent a specific class of production failure.
|
||
|
||
#### (a) Cross-provider validity
|
||
|
||
**What:** `renderEffortForRuntime(runtime, universalEffort).value` must always
|
||
be a member of the runtime's real provider enum. Ground-truth enums are defined
|
||
as local constants in the test — not sourced from the implementation.
|
||
|
||
```
|
||
PROVIDER_EFFORT_ENUMS = {
|
||
claude: Set { 'low', 'medium', 'high', 'xhigh', 'max' } // Anthropic output_config.effort
|
||
codex: Set { 'minimal', 'low', 'medium', 'high', 'xhigh' } // OpenAI model_reasoning_effort
|
||
}
|
||
```
|
||
|
||
**Why:** Passing a value outside these sets results in a 400 from the real API.
|
||
The clamping logic (`max -> xhigh` for codex; `minimal -> low` for claude) must
|
||
hold for every cell of the VALID_EFFORTS × runtimes matrix.
|
||
|
||
#### (b) Param/channel contract
|
||
|
||
**What:** Each runtime exposes a stable `param` string (the native API field
|
||
name) and `channel` (how the value is propagated). Unknown runtimes return
|
||
`param: null, channel: null` and pass the effort value through unchanged.
|
||
|
||
**Why:** Callers read `.param` to construct the dispatch payload. A regression
|
||
here would silently drop effort from subagent invocations.
|
||
|
||
#### (c) Resolve-execution JSON contract
|
||
|
||
**What:** The `gsd-tools resolve-execution <agent>` command emits a JSON object
|
||
with all eight keys present and typed correctly: `model` (string), `profile`
|
||
(string), `effort` (VALID_EFFORTS member), `effort_rendered` (string),
|
||
`effort_param` (string|null), `effort_propagation` (string|null), `fast_mode`
|
||
(boolean), `fast_mode_supported` (boolean).
|
||
|
||
**Why:** Orchestrators and workflow dispatchers parse this JSON. A missing or
|
||
mistyped field silently breaks downstream consumers.
|
||
|
||
#### (d) Totality across the real registry
|
||
|
||
**What:** For every agent in the 33-agent registry, `resolveEffortInternal`
|
||
returns a VALID_EFFORTS member (never undefined/null), `resolveFastModeInternal`
|
||
returns a strict boolean, and `renderEffortForRuntime('claude', effort)` stays
|
||
within the claude provider enum.
|
||
|
||
**Why:** A catalog addition that introduces a missing `routingTier` mapping
|
||
would otherwise produce `undefined` and propagate silently.
|
||
|
||
#### (e) Fast-mode honesty invariant
|
||
|
||
**What:** When the runtime is `claude`, `fast_mode_supported` in
|
||
resolve-execution output is always `false`, regardless of the fast_mode config.
|
||
`RUNTIMES_WITH_FAST_MODE` contains only `'api'`.
|
||
|
||
**Why:** Claude Code's `/fast` toggle is session-level only. Emitting
|
||
`fast_mode: true` as frontmatter on a Claude subagent is a silent no-op.
|
||
Advertising `fast_mode_supported: true` for claude would cause orchestrators to
|
||
believe the knob was wired when it is not.
|
||
|
||
#### (f) Precedence first-valid-wins
|
||
|
||
**What:** Both effort and fast_mode use a layered cascade. The test table covers
|
||
all four effort layers (invocation override → agent_overrides →
|
||
routing_tier_defaults → default) and all five fast_mode layers, including the
|
||
case where an invalid value at a higher layer correctly falls through.
|
||
|
||
**Why:** Silent precedence bugs (e.g., a numeric value in agent_overrides not
|
||
being rejected) would override intentional user config.
|
||
|
||
#### (g) Dynamic-routing composition
|
||
|
||
**What:** `resolveEffortForTier` escalates effort by attempt number
|
||
independently of the model tier mapping. The test verifies the effort ladder
|
||
(`low -> medium -> high -> xhigh -> max`), the `max` clamp, the
|
||
`max_escalations` cap, and that `escalate_on_failure: false` suppresses
|
||
escalation entirely.
|
||
|
||
**Why:** Effort escalation and model escalation share configuration
|
||
(`dynamic_routing`) but must operate independently; coupling them would cause
|
||
over-escalation or under-escalation.
|
||
|
||
#### (h) Config-tooling round-trip
|
||
|
||
**What:** `gsd-tools config-set` accepts all new key namespaces
|
||
(`effort.default`, `effort.routing_tier_defaults.<tier>`,
|
||
`effort.agent_overrides.<agent>`, `fast_mode.enabled`,
|
||
`fast_mode.routing_tier_defaults.<tier>`, `fast_mode.agent_overrides.<agent>`)
|
||
without an "Unknown config key" error, and values set via `config-set` are
|
||
reflected in `resolve-execution` output.
|
||
|
||
**Why:** The schema validation gate (`VALID_CONFIG_KEYS` + `DYNAMIC_KEY_PATTERNS`)
|
||
is separate from the resolver logic. A key missing from the schema would produce
|
||
a silent write failure and appear as a bug only at runtime.
|
||
|
||
### Coverage targets
|
||
|
||
| Suite | Target |
|
||
|---|---|
|
||
| Unit | Every cascade rule, every fallthrough, every clamp. All function branches in `resolveEffortInternal`, `resolveFastModeInternal`, `resolveEffortForTier`, `renderEffortForRuntime`. |
|
||
| Integration | All 8 architectural invariants. All 33 registered agents. All 6 provider × effort combinations for the valid-enum check. Full config-set key namespace. |
|
||
|
||
### Gaps / not yet covered
|
||
|
||
**E2E orchestrator-spawn-propagation layer (pending follow-up wiring):**
|
||
The integration tests verify that GSD resolves and renders effort values
|
||
correctly. They do NOT verify that the rendered values actually reach a spawned
|
||
Claude Code or Codex subagent at runtime. Specifically uncovered:
|
||
|
||
- `CLAUDE_CODE_EFFORT_LEVEL` env var being set and read by a spawned claude subprocess
|
||
- `output_config.effort` frontmatter key surviving the AGENTS.md template substitution
|
||
- `model_reasoning_effort` field surviving serialization into a Codex API request body
|
||
- Fast-mode `speed: "fast"` field reaching an `api`-runtime request when `fast_mode_supported: true`
|
||
|
||
These require spawning real subagents (or stubs thereof) and asserting on the
|
||
process environment / request payload — a scope that belongs in a future E2E
|
||
suite under `*.slow.test.cjs` or dedicated fixture-driven integration work.
|