dc3c81e93d
* test(#3412): failing-first suite for the pattern-construction seam Phase 1 of epic #3212 (ADR-3212 §1/§2/§7). Tests only — src/pattern.cts and eslint-rules/no-adhoc-regex-escape.cjs do not exist yet, so both suites fail with MODULE_NOT_FOUND, which is the intended RED. Locks the measured behavior rather than the assumed behavior: RegExp.escape hex-escapes the leading character of nearly every string ("abc" -> "\x61bc"), so the suite asserts match-equivalence against an inlined historical oracle (the implementation being deleted) rather than byte-equivalence of pattern text — 200 seeded fast-check runs plus a fixed corpus, 0 mismatches. Also locks the latent character-class range bug this phase fixes as a side effect: a hyphen-bearing value interpolated into [...] currently forms a real range and matches an unintended character; post-migration it must not. * chore(#3412): src/pattern.cts owns runtime-value regex construction Phase 1 of epic #3212 (ADR-3212 §1/§2/§6/§7). Adds the pattern seam delegating to the built-in RegExp.escape, deletes every hand-rolled copy, and raises the Node floor to the Active LTS line. The census was low, three times over. ADR-3212 counted 10 copies; a graph query found 12; the new lint rule — once live — found 27 more. The difference is that the census counted named helper FUNCTIONS while the rule counts the escape SHAPE, so inline .replace(<class>, '\$&') copies were never in scope. ADR §1's actual requirement is that no module outside the seam escapes a value for regex use, so all of them are, and CLAUDE.md's no-defer rule makes them this change's work. Fourth consecutive epic here whose copy count was low — the argument for ADR-3180 Amendment 3's "state N found by the guard" rule. Also corrected mid-implementation: the survey reported phase-id.cts's escapeRegex had 0 external importers. It had 8 production importers, making its removal a public-surface change to an ADR-2121-owned module and requiring an update to that ADR's locked-surface test. Blast radius revised Medium-High -> High. RegExp.escape is match-equivalent but NOT text-equivalent: it hex-escapes the leading char of nearly every string ("abc" -> "\x61bc"). Equivalence is proven by a seeded fast-check property test against the deleted implementation as oracle. It also fixes a latent bug: a hyphen-bearing value interpolated into a character class previously formed a real range and matched an unintended character. Node floor 22 -> 24 (RegExp.escape is Node 24+), across engines, .nvmrc, package-lock, 9 CI matrix entries, and 5 docs. The aggregate `required-tests` context is unchanged and no job was added or removed, so branch protection cannot be orphaned by the dropped lanes. Enforced by eslint-rules/no-adhoc-regex-escape.cjs (shape-matched, with structural provenance for reviewed pattern-fragment constants rather than a name heuristic) plus a whole-tree companion guard covering the directories ESLint's globs miss. * fix(#3412): close the _SOURCE guard evasion, correct two false claims Three findings from the orthogonal review pass, all fixed. 1. The ESLint rule's `_SOURCE` provenance fallback was pure identifier- name matching with no binding check, so `new RegExp(userInput_SOURCE)` — a function parameter — sailed past the guard. That is the same rename-evasion class issue #3410 documents, reopened by the very fallback meant to complement the structural check. Now bound to the identifier's actual binding kind: import, require-derived const, or module-scope const; parameters, `let`/`var`, and unresolvable bindings fail closed. Four RuleTester cases cover the evasion and prove the legitimate cross-module case still passes. 2. src/pattern.cts's own header carried the stale pre-correction counts (12 copies / 17 call sites) while CONTEXT.md and the design doc carried the corrected ones (~39 / ~44) — a self-contradiction inside the PR whose entire purpose is deleting divergent copies. Rewritten, preserving the durable lesson: a named-function census cannot see inline copies; only a shape-matching guard can. 3. The claim that all deleted copies threw TypeError on non-string was false. phase-id.cts's copy — the one with 8 external importers — did String(value).replace(...) and never threw. The seam's locked signature does not coerce, so this is a real, now-disclosed behavior change rather than the pure preservation the tests asserted. Audited all 32 invocations across the 8 importers and 6 in-file callers: every one is safe by construction (upstream truthy guard or a string-producing derivation), verified by runtime probe against the compiled modules rather than by TS compilation, which cannot see a runtime undefined. Corrected the false claim in both the test comment and the design doc, and added it to Known limits. * docs(#3412): add Changed changeset for the Node 24 floor The only user-visible break in this phase. The escape-behavior change is internal and match-equivalent, so it carries no user-facing note. * fix(#3412): resolve the seam's require graph in script fixtures and packaging Checkpoint 2 came back red with 90 failures on the node24 lane. Three distinct defects, all introduced by routing scripts/ through the new pattern seam, none reproducible by any local gate: 1. ~82 failures — tests/adr-index-gate.test.cjs and tests/removed-but-needed-lint.test.cjs copy a scripts/*.cjs into an mkdtemp fixture and spawn it there (necessary: those scripts resolve their scan root from __dirname/.., so running the real script would scan the real repo). Each harness hand-listed the dependencies to copy alongside. Adding require('../gsd-core/bin/lib/pattern.cjs') to gen-adr-index.cjs made both lists silently incomplete -> MODULE_NOT_FOUND, plus 17 downstream 'did not emit parseable JSON' failures from the same crash. Fixed as a class, not an instance: new tests/helpers/copy-script- fixture.cjs walks a script's transitive static relative-require graph and copies it, so dependencies are derived and never re-declared. It throws (naming the unbuilt artifact) instead of letting the child die with a bare MODULE_NOT_FOUND. Verified for all four seam-consuming scripts: gen-adr-index, lint-removed-but-needed, gen-loop-host- contract, sync-runtime-launcher. 2. 2 failures — scripts/ ships wholesale but eslint-rules/ does not, so the new scripts/lint-no-adhoc-regex-escape.cjs would be MODULE_NOT_FOUND in a published install (#2858 guard). Excluded from the tarball, matching the existing precedent for gen-emitted- baseline.cjs, which is excluded for the identical reason, and locked with a test modeled on that one. Confirmed against a real npm pack: 890 files, 0 from eslint-rules/, and gsd-core/bin/lib/pattern.cjs present (so the other four scripts' requires are legitimate). 3. 6 failures — tests/phase-id.test.cjs asserted the literal escaped source text ('0*29', 'PROJ-42'). RegExp.escape is match-equivalent to the retired hand-rolled escaper but NOT text-equivalent: it hex- escapes the leading character and all hyphens ('0*\x329', '\x50ROJ\x2d42'). Verified NOT a behavior change — 576 match decisions across all three real interpolation prefixes, zero divergence. Those tests now compile each source into the same heading regex src/roadmap.cts's searchPhaseInContent builds and assert what matches and what does not, including the 'i'-flag canonicalization the hex escape has to preserve. Re-pinning the new literals would have rebuilt the same brittleness one layer down. Adds a test for the property the escape exists for: a dot in '1.2' must not act as a wildcard. Also shares one definition of 'a require' between the packaging guard and the fixture copier, so the two cannot disagree about what they scan. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#3412): refuse to copy a fixture dependency outside the fixture root copyScriptWithDeps resolved each relative require and joined the repo-relative result onto fixtureRoot. A require resolving OUTSIDE the repo yields a '../'-prefixed relative path, so path.join climbed out of the fixture and wrote into the surrounding temp dir (verified: repoRoot=/repo + depAbs=/etc/passwd wrote /tmp/etc/passwd). No script in the tree does this today, so this closes an available escape rather than an active one. Refuses via the existing unresolved- require path so the failure names the offending specifier. Covered by a negative proof that the guard fires and that nothing lands outside the fixture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#3412): parse requires instead of pattern-matching them; restore the foreign-prefix contract Applies all findings from the second orthogonal review round, re-run because real code changed after round 1. HIGH (security) — extractRequires stripped BLOCK comments before LINE comments, so a '//' comment containing '/*' opened a phantom block comment, and a '//' inside a string literal truncated the line. Both hid real requires: 'const u="http://x"; require("./real.cjs")' returned [], and four real requires in gsd-core/bin/gsd-tools.cjs were invisible. Replaced with a real AST parse via espree. This is ADR-3212's own Decision 4 — tokenizer-first for stateful grammars — applied to the case it describes; comment/string/regex nesting is exactly such a grammar, which is why the regex version was wrong. The function was moved byte-identical out of the #2858 packaging guard, so the bug PRE-DATES this branch and has been a live blind spot there: a shipped script could have required an unshipped path undetected. Fixing it makes that guard strictly stronger than on next. espree is promoted from a transitive eslint dependency to an explicit devDependency rather than relying on hoisting. The script parse attempt sets ecmaFeatures.globalReturn because Node wraps CommonJS bodies in a function, making a top-level return legal — scripts/check-coverage-gate .cjs relies on it, and without the flag the guard throws on a file it is supposed to scan. Verified 0 unparseable across all 324 .cjs/.js under scripts/, bin/, and gsd-core/bin/, and 0 new violations against a real npm pack, so the exact extractor does not newly fail the guard. MEDIUM (security) — the repo-containment check guarded dependencies but not the entry path. One escapesContainment predicate now guards both. LOW (security) — containment was lexical while fs follows symlinks, and a directory symlink could mint a fresh dedupe key per level. realpath now resolves both repoRoot and each dependency before the decision, and the realpath-derived path is the dedupe key. Destination layout still uses the original repo-relative path, so copied trees are unchanged. MAJOR (standards) — the round-1 behavioral rewrite of phase-id tests lost the foreign-prefix contract: every assertion was satisfied by an impl returning [A-Z]+\x2d42, i.e. ANY project code — the exact #3599 bug class the exact-source prevents. The literal assertions it replaced were catching this. Now asserts the compiled regex REJECTS a different prefix with the same number. MAJOR (standards) — the test hand-duplicated production's heading regex with no parity guard (CLAUDE.md's 'Generative Fix Divergence'). Removed the parallel surface instead of policing it: src/roadmap.cts exports buildPhaseHeadingRegex, searchPhaseInContent calls it, the test imports it. Byte-identical .source and .flags verified for both escaped forms. MINOR — '..foo' no longer false-flagged as an escape; the inverted spurious-vs-missing doc claim corrected; the dead allow-test-rule header removed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(#3412): backfill changeset pr number to 3416 * fix(#3412): make the escape guard's own regex linear, reword an injection-scan collision Two CI failures on PR #3416, both in code this branch added. CodeQL js/redos (high) — REPLACE_CALL_RE's outer alternation let a bracket run be consumed EITHER by the character-class branch OR one character at a time by the trailing catch-all, so a failing match explored both parses of every pair. Measured on the real regex: n=26 -> 204ms, n=28 -> 791ms, n=30 -> 3475ms, a clean 2^n. This script scans repo source, so a file with a long bracket run after '.replace(/' would hang CI outright — a guard against undisciplined pattern construction was itself the worst pattern in the diff. Fixed the way ADR-3212 already prescribes: the catch-all branch now excludes '[' and ']' so a bracket can only be consumed by the class branch (this is what makes it linear), and every quantifier is bounded (the locked bounded-quantifiers decision) as a second line of defense. Now 0ms at n=2000. Disclosed coverage tradeoff, recorded at the constant: a regex literal with a BARE unescaped ']' outside a class is no longer matched by this backstop. No census shape has that form, and the AST rule remains the primary detector. Verified the guard did not go blind doing it: a real census-shape violation is still reported, and an allow-adhoc-regex-escape suppression comment is still honored. Regression test drives the exported findViolations on a 2000-repetition adversarial input and asserts the RESULT. It makes no wall-clock assertion — elapsed-time tests are forbidden — so a regression surfaces as a harness timeout, which is the correct signal. Prompt injection scan — 'must not act as a regex wildcard' in a test comment matched the scanner's jailbreak pattern act\s+as\s+(a|an|if| my). Reworded to 'behave as'. Deliberately NOT allowlisted: silencing a whole test file over one phrase would blunt the scanner permanently, and the comment has nothing to do with injection. Neither failure was reachable from the remote runner — CodeQL and the injection scan are not in that matrix, so the sha it passed was green and still wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: sim <sim@local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
547 lines
30 KiB
Markdown
547 lines
30 KiB
Markdown
# Testing Suites
|
||
|
||
This project's `tests/` directory uses **filename suffix markers** to group tests into named suites. The harness `scripts/run-tests.cjs` filters by suite when given `--suite <name>`. Without a flag it runs every `*.test.cjs` file (the historical default — unchanged).
|
||
|
||
> Tracked by issue [#3597](https://github.com/open-gsd/gsd-core/issues/3597).
|
||
|
||
## Suites
|
||
|
||
| Suite | Filename pattern | What goes here |
|
||
|---|---|---|
|
||
| `unit` | `*.test.cjs` (no other marker) | Default fast lane. Pure logic, no network, no external processes beyond `gsd-tools`. Most tests live here. |
|
||
| `integration` | `*.integration.test.cjs` | Cross-module flows: full installer end-to-end, multi-tool orchestration, anything that crosses two or more bin entry points. |
|
||
| `install` | `*.install.test.cjs` | Tests that perform a real install/uninstall against a sandbox project. Slower; PR CI skips these on PRs and runs them on `main` push only. |
|
||
| `security` | `*.security.test.cjs` | Adversarial input, prompt-injection guards, fixture-driven hostile-payload sweeps. |
|
||
| `slow` | `*.slow.test.cjs` | Anything that routinely takes >5s wall-clock or holds significant memory. |
|
||
| `qa` | `*.qa.test.cjs` | End-to-end walks that drive the real `gsd-tools` binary across multiple loop steps against one accumulating temp project, with invariant oracles after every step. Slower than `unit`; excluded from the fast lane. |
|
||
| `all` | (any) | Explicit alias for "no filter". Equivalent to running with no `--suite` flag. |
|
||
|
||
## How to place a new test
|
||
|
||
1. Pick the most specific bucket above.
|
||
2. Name the file with the matching suffix: `tests/<feature>.<suite>.test.cjs`.
|
||
3. If unsure, leave the suffix off — the file lands in `unit`, the default fast lane.
|
||
|
||
Examples:
|
||
- `tests/agent-frontmatter.test.cjs` — `unit`
|
||
- `tests/prompt-injection-guards.security.test.cjs` — `security`
|
||
- `tests/installer-end-to-end.install.test.cjs` — `install`
|
||
- `tests/sdk-mutation-stress.slow.test.cjs` — `slow`
|
||
- `tests/loop-walk.qa.test.cjs` — `qa`
|
||
|
||
The suite-suffix convention was chosen over a directory layout (`tests/security/`) so the 545+ existing test files don't need to move. Existing files all classify as `unit` until someone explicitly retags them.
|
||
|
||
## Regression tests
|
||
|
||
**Do not create new top-level `tests/bug-NNNN-*.test.cjs`,
|
||
`tests/fix-NNNN-*.test.cjs`, or `tests/issue-NNNN-*.test.cjs` files.** Add the
|
||
regression case to the owning module's main test file instead (e.g. a
|
||
`describe('regressions')` block in `tests/<module>.test.cjs`).
|
||
|
||
`node --test` spawns one child process per FILE, so file count — not test
|
||
count — is the unit of CI overhead, and it is worst on Windows lanes where
|
||
every spawn is Defender-scanned. The 2026-06 CI audit found 244 one-off
|
||
`bug-*` files (~38% of the suite). That population is grandfathered in
|
||
`scripts/lint-regression-test-names.allowlist.json` and enforced by an
|
||
identity ratchet (`npm run lint:regression-names`, part of `npm run lint:ci`),
|
||
which also bans `fix-*` and `issue-*` NNNN-prefixed files the same way:
|
||
|
||
- A **new** `bug-*`, `fix-*`, or `issue-*` NNNN-prefixed file fails CI — fold
|
||
it into the owning module's file.
|
||
- **Deleting/consolidating** a grandfathered file requires pruning its
|
||
allowlist entry, so the baseline only ever shrinks.
|
||
- **Inherited drift** (the failure names files your PR didn't add — e.g. the
|
||
base branch merged `bug-*`/`fix-*`/`issue-*` files without feeding the
|
||
allowlist, or you rebased and carried a pre-rebase allowlist): run
|
||
`node scripts/lint-regression-test-names.cjs --update` and commit the
|
||
regenerated allowlist. Snapshot artifacts like this allowlist (and
|
||
`docs/INVENTORY.md`) must be regenerated **after** rebasing, never carried
|
||
through a rebase.
|
||
|
||
### A folded suite may appear only once per host
|
||
|
||
When a standalone file is folded into its owning module's test file, the moved
|
||
suite is wrapped in a self-contained block carrying a marker:
|
||
|
||
```javascript
|
||
// ────────────────────────────────────────────────────────────────────────
|
||
// Folded from tests/bug-376-claude-js-hook-gsd-rewriter.test.cjs — …
|
||
// ────────────────────────────────────────────────────────────────────────
|
||
{
|
||
const { describe: __foldDescribe } = require('node:test');
|
||
__foldDescribe("folded:bug-376-claude-js-hook-gsd-rewriter (…)", () => { … });
|
||
}
|
||
```
|
||
|
||
Because the block is self-contained, a **second verbatim copy in the same host
|
||
parses, registers, and passes — twice.** Nothing in a green suite reports it.
|
||
[#3271](https://github.com/open-gsd/gsd-core/issues/3271) found 25 such copies
|
||
(~5,800 lines) across three install suites, all from a single stale-base
|
||
re-application during the consolidation epic. The cost is not only wasted CI on
|
||
every lane: it is a `DEFECT.GENERATIVE-FIX` trap, because a contributor fixing
|
||
one of those regressions edits the copy they found and leaves the other
|
||
asserting the old behavior, with the suite still green.
|
||
|
||
`local/no-duplicate-fold-marker` (`eslint-rules/no-duplicate-fold-marker.cjs`,
|
||
error under `tests/**/*.cjs`) reports the second and every later occurrence of a
|
||
`folded:<marker>` title in one file, naming the line the first occurrence sits
|
||
on. **When it fires, delete the copy it points at** — the two blocks are the
|
||
same suite, so the fix is removal, never an `eslint-disable`.
|
||
|
||
It keys on the whitespace-delimited token after `folded:`, which matters in both
|
||
directions. A narrower key that stops at `.` would collide
|
||
`feat-443-effort-fast-mode.integration` with `feat-443-effort-fast-mode` — two
|
||
genuinely distinct suites that coexist in `tests/model-resolver.test.cjs`. Keying
|
||
on the *whole* title instead would let a re-fold under a different batch label
|
||
slip through, which is exactly the shape #3271 took. Titles without a `folded:`
|
||
prefix, non-literal titles, and the same marker appearing in two *different* host
|
||
files are all left alone.
|
||
|
||
The ratchet deliberately covers only `bug-*`. Files named `feat-NNNN-*` /
|
||
`enh-NNNN-*` are *feature* test files — one (or one per suite) per feature is
|
||
the sanctioned layout (see the #443 strategy below), not a one-off regression
|
||
pattern. If `issue-*`/`perf-*` one-offs start accumulating the same way
|
||
`bug-*` did, extend the ratchet's regex and regenerate the allowlist.
|
||
|
||
## Workflow & agent size budget
|
||
|
||
> Tracked by issue [#1074](https://github.com/open-gsd/gsd-core/issues/1074).
|
||
> Bytes (not lines) per [#717](https://github.com/open-gsd/gsd-core/issues/717);
|
||
> LF-normalized per [#683](https://github.com/open-gsd/gsd-core/issues/683).
|
||
|
||
Workflow files (`gsd-core/workflows/*.md`) and agent files (`agents/gsd-*.md`)
|
||
both ship in the installed runtime and are loaded into context — workflows on
|
||
every command, agents on every subagent dispatch — so their byte size is a real
|
||
cost. Two sibling guards (`tests/workflow-size-budget.test.cjs` and
|
||
`tests/agent-size-budget.test.cjs`) keep that cost from creeping up invisibly,
|
||
sharing one byte-counter (`measureMdFiles`). Growth is caught by two independent
|
||
layers:
|
||
|
||
| Layer | What it does | Where |
|
||
|---|---|---|
|
||
| **Differential attribution size ratchet** (primary, #2724 / ADR-2719 §4) | The same computed-attribution check that replaced the golden-install-parity fixtures also reports growth in any `gsd-core/workflows/*.md` or `agents/gsd-*.md` file, with the exact byte delta, comparing PR HEAD against `next`. Unacknowledged growth is a hard failure; shrinkage needs no acknowledgment. No committed snapshot — nothing to regenerate by hand. | `tests/emitted-attribution.test.cjs` (real-tree test) via `tests/helpers/emitted-diff.cjs` |
|
||
| **Loose tier hard caps** (backstop) | Absolute outer red lines per tier — workflows: `XL ≤ 98304`, `LARGE ≤ 61440`, `DEFAULT ≤ 40960` bytes; agents: `XL ≤ 57344`, `LARGE ≤ 49152`, `DEFAULT ≤ 24576` bytes. A cap is **never raised** when a file approaches it: crossing it means *extract*, not bump. Independent of the ratchet above — unaffected by #2724. | `XL/LARGE/DEFAULT_CAP` in each guard file |
|
||
|
||
`discuss-phase.md` additionally has a thin-dispatcher target of `< 32000` bytes
|
||
(the discuss-phase progressive-disclosure split, #717). A net-new agent is
|
||
DEFAULT-tier and already bounded by the DEFAULT cap — no separate new-agent cap
|
||
is needed. (This tier-cap machinery is distinct from the separate 45 KB-*char*
|
||
extraction-evidence threshold on `gsd-planner` enforced by
|
||
`tests/planner-decomposition.test.cjs` — that one proves mode sections were
|
||
extracted; this one bounds total agent bytes.)
|
||
|
||
### How-to: a workflow or agent grew and CI is red
|
||
|
||
The differential attribution check reports the file and the byte delta. To resolve:
|
||
|
||
1. **Justify the growth in your PR** (a sentence in the description is enough) —
|
||
the acknowledgment entry (below) is the review record that the larger size
|
||
was a deliberate, seen decision, not silent drift.
|
||
2. **Add an acknowledgment entry** in `tests/emitted-drift-ack.json` naming the
|
||
file and the reason, per `CONTEXT.md`'s `### Emitted Artifact Provenance`
|
||
entry. This is deliberately a committed file, not a flag — the entry appears
|
||
in your PR diff, so touching it *is* the visible signal.
|
||
3. **Or shrink it instead of acknowledging.** Prefer extraction when the growth
|
||
is incidental: for a workflow, move per-mode bodies to
|
||
`workflows/<name>/modes/`, templates to `workflows/<name>/templates/`, and
|
||
shared prose to `gsd-core/references/`; for an agent, lift shared boilerplate
|
||
into `gsd-core/references/` and `@`-reference it — then load it **LAZILY**. Do
|
||
*not* convert them to eager `@-required_reading` includes: that shrinks the
|
||
file's bytes without shrinking loaded context, so it games the guard while
|
||
making the real cost worse. See `workflows/discuss-phase/` for the
|
||
progressive-disclosure pattern.
|
||
|
||
If a hard cap (not the ratchet) is what failed, an acknowledgment will **not**
|
||
help — that is the signal to extract, per step 3.
|
||
|
||
### Reference
|
||
|
||
| Artifact | Role |
|
||
|---|---|
|
||
| `scripts/workflow-size.cjs` | Single source of truth — LF-normalized byte counter (`lfByteCount`) + generic `measureMdFiles(dir, predicate)` (backs both workflows and agents) + workflow enumeration (`listWorkflowStems`, `measureWorkflows`). Imported by both guards and by `tests/helpers/emitted-runtime.cjs`'s `currentSizes()` so they can never measure differently. |
|
||
| `tests/emitted-attribution.test.cjs` + `tests/helpers/emitted-diff.cjs` | The differential attribution check and its size ratchet (ADR-2719). The sole mechanism for both emitted-content propagation AND per-file size growth as of #2724. |
|
||
| `tests/emitted-drift-ack.json` | Committed acknowledgment file for unattributable emitted-content ripples and for size growth. Absent = no acks; its presence is the alarm. |
|
||
| `npm run regen:derived` | Runs every remaining generator in dependency order (build → registry → ADR index → capability matrix → inventory manifest → manifest versions → `tests/fixtures/install-tree/*.json`). |
|
||
| `tests/workflow-size-budget.test.cjs` | The workflow tier hard-cap guards, plus the `discuss-phase` progressive-disclosure checks. |
|
||
| `tests/agent-size-budget.test.cjs` | The agent tier hard-cap guards (the agent analog). |
|
||
|
||
`tests/workflow-size-baseline.json`, `tests/agent-size-baseline.json`,
|
||
`tests/fixtures/golden-install-parity/*.json`, `scripts/update-size-baseline.cjs`
|
||
(`npm run size:baseline`), and `scripts/git-merge-regen-driver.cjs`
|
||
(`npm run setup:merge-driver`) are all removed by
|
||
[#2724](https://github.com/open-gsd/gsd-core/issues/2724): they were pure
|
||
functions of the source tree, conflicted on every merge that touched them, and
|
||
their functions are now served by the differential attribution check above.
|
||
`tests/fixtures/install-tree/*.json` is the one artifact family that stays
|
||
committed and normally-merged (ADR-2719 §7) — it conflicts on 0 of 7, its diffs
|
||
are readable, and it preserves "the installer stopped shipping X" as a hard
|
||
absolute failure with no attribution reasoning involved. Regenerate it with
|
||
`npm run gen:install-tree` (folded into `npm run regen:derived`).
|
||
|
||
## The QA smell ratchet
|
||
|
||
`tests/loop-walk.qa.test.cjs` (the `qa` suite) is the QA-walk harness's own
|
||
self-test. Separately, `scripts/qa-smell-ratchet.cjs` drives that same harness
|
||
end to end against the real `gsd-tools` binary and turns its findings into a
|
||
CI gate — run it with `npm run lint:qa-smells`.
|
||
|
||
The harness's oracles (`tests/qa/oracles.cjs`) distinguish two severities:
|
||
|
||
- A **violation** is the engine breaking a documented contract. It always
|
||
fails the build — baseline or no baseline, acknowledged or not.
|
||
- A **smell** is legal-but-questionable behavior. A smell **never fails a
|
||
build on its own merits**. What fails is the absence of a decision about
|
||
it: an **unacknowledged NEW smell**, or a **STALE** entry in
|
||
`tests/qa/smell-baseline.json` (one that stopped firing — the baseline is
|
||
shrink-only, so a fixed or changed scenario must be pruned, not left
|
||
behind).
|
||
|
||
Every smell must terminate in exactly one of TWO states — there is no third
|
||
"accepted with a good explanation" state:
|
||
|
||
1. **REAL** — an assigned defect. File it, then acknowledge the smell with an
|
||
entry (baseline entry or `tests/qa/smell-acks/` fragment) carrying that
|
||
`issue` number.
|
||
2. **FALSE POSITIVE** — the oracle itself is wrong. Fix the oracle
|
||
(`tests/qa/oracles.cjs`) so it stops firing. It is NEVER baselined.
|
||
|
||
When the ratchet reports a NEW smell, there are exactly two legitimate
|
||
responses — fix the detector, or file a defect and cite its issue number:
|
||
|
||
1. **Fix the underlying behavior (or the oracle, if it's a false positive)**
|
||
so the smell stops firing.
|
||
2. **File a defect and acknowledge it** by adding a fragment under
|
||
`tests/qa/smell-acks/` — the ratchet's failure output prints a paste-ready
|
||
skeleton naming the required `key`, `id`, `scenario`, and `issue` fields.
|
||
`issue` MUST be a positive integer naming the tracking issue; a free-text
|
||
`reason` may accompany it as an optional human note but can NEVER
|
||
substitute for `issue` — "write an explanation" is not a way to acknowledge
|
||
a smell. See `tests/qa/smell-acks/README.md` for the full shape and
|
||
lifecycle.
|
||
|
||
Run `node scripts/qa-smell-ratchet.cjs --update` to regenerate
|
||
`tests/qa/smell-baseline.json` from the current run, folding in any acked
|
||
fragments and pruning stale entries. `--update` never invents an issue
|
||
number: a genuinely new smell is written with `issue: null` and a TODO
|
||
`reason`, and the very next plain (non-`--update`) run REJECTS that entry —
|
||
forcing a human to triage it before it can ship. The baseline only ever
|
||
shrinks: growth happens by adding an acknowledgment carrying a real issue
|
||
number (a reviewable diff), never by widening the generator's tolerance and
|
||
never by prose alone.
|
||
|
||
## Running suites locally
|
||
|
||
```bash
|
||
npm test # everything (backcompat — same as before)
|
||
npm run test:unit # only unit
|
||
npm run test:integration # only integration
|
||
npm run test:install # only install
|
||
npm run test:security # only security
|
||
npm run test:slow # only slow
|
||
|
||
npm run test:coverage # backcompat — coverage over EVERY test
|
||
npm run test:coverage:unit # fast coverage signal — only unit suite
|
||
npm run test:coverage:all # alias for test:coverage
|
||
```
|
||
|
||
Direct harness invocation also works:
|
||
|
||
```bash
|
||
node scripts/run-tests.cjs --suite security
|
||
node scripts/run-tests.cjs --suite=security
|
||
node scripts/run-tests.cjs --files "tests/command-contract.test.cjs tests/core.test.cjs"
|
||
node scripts/run-tests.cjs --files-from .ci-selected-tests.txt
|
||
```
|
||
|
||
`npm run test:affected` (scripts/run-affected-tests.cjs) is a **local-only**
|
||
convenience that selects tests via the `require()` dependency graph of your
|
||
working-tree diff. CI does not use it — CI selection is the rule table in
|
||
`scripts/ci-test-scope.cjs`, which is the authoritative mapping. If the two
|
||
disagree, trust (and fix) the rule table.
|
||
|
||
Unknown suites exit non-zero with the list of valid suites. Empty suites (e.g. `--suite security` before any security-tagged file exists) exit `0` with a `no tests in suite "..."` notice on stderr so CI lanes don't go red while a suite is being populated.
|
||
|
||
## The live-config hermeticity guard
|
||
|
||
Every `run-tests.cjs` invocation snapshots GSD's own install footprint in each
|
||
live runtime config directory before the suite and re-checks it afterwards. It
|
||
exists because the failure it catches is silent by construction: a test that
|
||
resolves a config directory from the ambient environment instead of a sandbox
|
||
writes into *your real* `~/.claude` (or `$GSD_HOME/.gsd`, or a Kimi
|
||
`config.toml`), and nothing reports it. CI cannot catch this class at all —
|
||
CI never has `CLAUDE_CONFIG_DIR` and friends set.
|
||
|
||
The guard watches only what GSD unambiguously owns — its top-level install
|
||
footprint plus `gsd-`-prefixed children of directories shared with the host
|
||
agent — never whole config roots, because a host agent legitimately writing
|
||
`history.jsonl` mid-run would make the guard cry wolf, and a guard that cries
|
||
wolf gets switched off.
|
||
|
||
Two environment variables control it:
|
||
|
||
| Variable | Effect |
|
||
|---|---|
|
||
| `GSD_STRICT_LIVE_CONFIG_GUARD=1` | A detected write **fails the run**. Set on the Linux/macOS lanes of every CI job that runs the suite. |
|
||
| `GSD_SKIP_LIVE_CONFIG_GUARD=1` | Skips the check entirely. |
|
||
|
||
Unset, the guard **reports and does not fail** — deliberately, not timidly. On
|
||
its first CI run it surfaced pre-existing leaks on the Windows lane, where
|
||
`os.homedir()` reads `USERPROFILE` and ~190 test sites sandbox `HOME` alone.
|
||
Those are real and worth fixing, but they are a different defect class, and a
|
||
brand-new gate that instantly reds an unrelated lane gets reverted rather than
|
||
obeyed. Windows lanes therefore stay report-only until that sweep lands; this
|
||
repo has the pattern already, in the `local/no-source-grep` ESLint rule that
|
||
shipped at `warn` and was promoted to `error` after its cleanup (ADR 452).
|
||
|
||
`GSD_SKIP_LIVE_CONFIG_GUARD` is a bypass on a safety check, so it is documented
|
||
here rather than left to be discovered in the source: an undocumented bypass is
|
||
one people eventually set without knowing what they turned off. If you need it
|
||
routinely, that is a bug report, not a workflow.
|
||
|
||
Reported paths are labelled `CREATED`, `MODIFIED`, `DELETED`, or `UNVERIFIED`.
|
||
`UNVERIFIED` means a scan bound was hit and the path could not be attested
|
||
either way — it is never the same as clean.
|
||
|
||
## CI matrix
|
||
|
||
The `Tests` workflow runs every PR through a scoped gate generated by
|
||
`scripts/ci-test-scope.cjs`.
|
||
|
||
All lanes run on **Node 24** — the `engines.node` floor (`>=24.0.0`) and the
|
||
only supported runtime.
|
||
|
||
| Lane | Scope |
|
||
|---|---|
|
||
| `ubuntu-latest` (scoped) | scoped tests — fast PR signal |
|
||
| `ubuntu-latest` (full, sharded) | unit + integration + security |
|
||
| `windows-latest` | scoped Windows/path/shell tests |
|
||
| `macos-latest` | full parity when required |
|
||
|
||
- **Scoped tests** are selected from the changed paths, plus a small CLI/package
|
||
smoke set. They are for confidence on the affected surface, not for counting
|
||
tests.
|
||
|
||
The default PR gate runs the broad `unit` (under the c8 coverage gate),
|
||
`integration`, and `security` suites once on Ubuntu / Node 24, scoped tests on
|
||
a second Ubuntu / Node 24 lane, and scoped tests on Windows / Node 24.
|
||
"Scoped" means the diff-selected list from the rule table — not the full suite
|
||
and not a fixed smoke set (the fixed smoke list is only the empty-selection
|
||
fallback). The Windows lane's list is the Windows-sensitive subset of the
|
||
selection, plus **every changed test file, unconditionally** (the #494
|
||
invariant, narrowed): a modified test is exercised on the divergent OS before
|
||
merge at per-file cost, without paying for the three full parity lanes.
|
||
|
||
PRs touching workflow, package, test-runner, install, release, or
|
||
Windows-sensitive surfaces also run the full parity matrix on macOS and the
|
||
older Windows runtime, plus `install` and `slow` on the primary Ubuntu lane.
|
||
Everything (including the full parity matrix) runs on every push to `next`,
|
||
which covers the residual macOS / Windows cross-product for scoped PRs.
|
||
|
||
Coverage runs inside the Ubuntu / Node 24 full lane (not a separate job — that
|
||
duplicated the entire unit run) and stays single-lane because multiplying
|
||
coverage across OS/runtime lanes adds cost without improving the threshold
|
||
signal. Note the gate's deliberate blind spot: it measures
|
||
`gsd-core/bin/lib/*.cjs` only — `scripts/`, `hooks/`, and `bin/` are
|
||
unenforced, and `stryker.config.mjs` additionally excludes ~48% of lib lines
|
||
from mutation testing (see the UNMUTATED list there). Widening either gate is
|
||
tracked work, not an accident to "fix" silently by raising thresholds.
|
||
|
||
To inspect the scope locally:
|
||
|
||
```bash
|
||
npm run ci:test-scope -- --files "commands/gsd/plan-phase.md"
|
||
node scripts/ci-test-scope.cjs --base origin/next --head HEAD
|
||
```
|
||
|
||
## Chunk packing and the test timing table
|
||
|
||
`scripts/run-tests.cjs` does not hand the whole selected file list to one
|
||
`node --test` process. It packs the files into **chunks**, each spawned
|
||
separately, because Windows caps a command line at 32,767 characters and because
|
||
each chunk gets its own 600s timeout (`RUN_TESTS_CHUNK_TIMEOUT_MS`) and a fresh
|
||
process, which bounds memory pressure.
|
||
|
||
How files are distributed across those chunks decides whether the slowest chunk
|
||
sits near that timeout while the others idle. The packer weights each file by its
|
||
**measured duration**, read from `tests/test-timings.json`, and places files with
|
||
LPT (longest-processing-time-first: heaviest file first, each into the currently
|
||
lightest chunk). Before #2456 the weight was guessed from the filename, which
|
||
mis-ranked files badly enough that the slowest chunk ran ~3.9x the lightest.
|
||
|
||
### Reference
|
||
|
||
| Knob | Default | Meaning |
|
||
|---|---|---|
|
||
| `RUN_TESTS_MAX_FILES_PER_CHUNK` | `60` | Per-chunk weight budget. Weights are normalized so an **average-cost** file weighs 1, so this still reads as "about 60 average files". |
|
||
| `RUN_TESTS_MAX_CMDLINE_CHARS` | `28000` | argv ceiling per chunk, with headroom under the Windows 32,767 limit. |
|
||
| `RUN_TESTS_TIMINGS_FILE` | `tests/test-timings.json` | Path to the timing table. Tests override it to inject a synthetic cost profile. |
|
||
| `RUN_TESTS_CHUNK_TIMEOUT_MS` | `600000` | Per-chunk timeout. |
|
||
|
||
The timing table is **advisory and deliberately un-gated**. There is no `--check`
|
||
mode and no CI lint that fails on staleness, because timing data legitimately
|
||
varies run to run. A file missing from the table falls back to the table's median
|
||
weight, and a missing or unparseable table falls back to uniform weight — so
|
||
drift costs chunk *balance*, never a red build. A count-based floor additionally
|
||
guarantees the packer never produces fewer chunks than plain count-based packing
|
||
would, so a badly stale table cannot collapse the suite into a few fat chunks.
|
||
|
||
### How-to: regenerate the timing table
|
||
|
||
Regenerate when the suite's cost profile has visibly drifted — after adding or
|
||
removing expensive tests, not on a schedule. The input is a `node:test` reporter
|
||
event stream from a `gsd-test` run:
|
||
|
||
```bash
|
||
node scripts/gen-test-timings.cjs \
|
||
~/.local/state/gsd-test/runs/<run-id>/test-events-linux-node24.jsonl \
|
||
~/.local/state/gsd-test/runs/<run-id>/test-events-macos-node24.jsonl
|
||
```
|
||
|
||
Pass every lane you have. A file's recorded time is the **max** across the
|
||
supplied streams, not the mean: the packer exists to keep the *slowest* lane's
|
||
slowest chunk away from the timeout, so the conservative bound is the right one.
|
||
Keys are sorted so a regeneration diff shows only the files whose cost moved.
|
||
|
||
## Best practices for forward-compat (Node 24/26)
|
||
|
||
- Use `process.execPath` when spawning Node in tests so each matrix lane exercises the lane's Node version.
|
||
- Avoid stack-trace or error-message prose assertions. Assert `err.code`, structured JSON fields, or enums — Node minor releases routinely tweak error wording.
|
||
- Prefer `node:test`, `node:assert/strict`, and `node:test` mocks. No external test frameworks.
|
||
- Coverage uses `c8` and propagates `NODE_V8_COVERAGE` through the harness's child process.
|
||
|
||
---
|
||
|
||
## Test strategy: #443 effort + fast_mode engine
|
||
|
||
> Feature: unified cross-provider effort and fast_mode knobs (issue #443).
|
||
> Test files: `tests/model-resolver.test.cjs` (unit),
|
||
> `tests/model-resolver.test.cjs` (integration).
|
||
|
||
### Testing pyramid
|
||
|
||
| Layer | File | What it covers |
|
||
|---|---|---|
|
||
| **Unit** | `feat-443-effort-fast-mode.test.cjs` | Pure logic: cascade rules, clamping, escalation math, malformed config handling, schema key validation. No CLI subprocess. |
|
||
| **Integration** | `feat-443-effort-fast-mode.integration.test.cjs` | Architecture-level invariants: cross-provider validity, totality across the 33-agent registry, CLI JSON contract, config round-trip, fast-mode honesty. Real subprocesses via `runGsdTools`. |
|
||
| **E2E** *(pending)* | *(not yet wired)* | Propagation layer: effort frontmatter / `CLAUDE_CODE_EFFORT_LEVEL` env actually reaching a spawned Claude Code subagent. See "Gaps" below. |
|
||
|
||
### Architectural invariants
|
||
|
||
Each invariant exists to prevent a specific class of production failure.
|
||
|
||
#### (a) Cross-provider validity
|
||
|
||
**What:** `renderEffortForRuntime(runtime, universalEffort).value` must always
|
||
be a member of the runtime's real provider enum. Ground-truth enums are defined
|
||
as local constants in the test — not sourced from the implementation.
|
||
|
||
```
|
||
PROVIDER_EFFORT_ENUMS = {
|
||
claude: Set { 'low', 'medium', 'high', 'xhigh', 'max' } // Anthropic output_config.effort
|
||
codex: Set { 'minimal', 'low', 'medium', 'high', 'xhigh' } // OpenAI model_reasoning_effort
|
||
}
|
||
```
|
||
|
||
**Why:** Passing a value outside these sets results in a 400 from the real API.
|
||
The clamping logic (`max -> xhigh` for codex; `minimal -> low` for claude) must
|
||
hold for every cell of the VALID_EFFORTS × runtimes matrix.
|
||
|
||
#### (b) Param/channel contract
|
||
|
||
**What:** Each runtime exposes a stable `param` string (the native API field
|
||
name) and `channel` (how the value is propagated). Unknown runtimes return
|
||
`param: null, channel: null` and pass the effort value through unchanged.
|
||
|
||
**Why:** Callers read `.param` to construct the dispatch payload. A regression
|
||
here would silently drop effort from subagent invocations.
|
||
|
||
#### (c) Resolve-execution JSON contract
|
||
|
||
**What:** The `gsd-tools resolve-execution <agent>` command emits a JSON object
|
||
with all eight keys present and typed correctly: `model` (string), `profile`
|
||
(string), `effort` (VALID_EFFORTS member), `effort_rendered` (string),
|
||
`effort_param` (string|null), `effort_propagation` (string|null), `fast_mode`
|
||
(boolean), `fast_mode_supported` (boolean).
|
||
|
||
**Why:** Orchestrators and workflow dispatchers parse this JSON. A missing or
|
||
mistyped field silently breaks downstream consumers.
|
||
|
||
#### (d) Totality across the real registry
|
||
|
||
**What:** For every agent in the 33-agent registry, `resolveEffortInternal`
|
||
returns a VALID_EFFORTS member (never undefined/null), `resolveFastModeInternal`
|
||
returns a strict boolean, and `renderEffortForRuntime('claude', effort)` stays
|
||
within the claude provider enum.
|
||
|
||
**Why:** A catalog addition that introduces a missing `routingTier` mapping
|
||
would otherwise produce `undefined` and propagate silently.
|
||
|
||
#### (e) Fast-mode honesty invariant
|
||
|
||
**What:** When the runtime is `claude`, `fast_mode_supported` in
|
||
resolve-execution output is always `false`, regardless of the fast_mode config.
|
||
`RUNTIMES_WITH_FAST_MODE` contains only `'api'`.
|
||
|
||
**Why:** Claude Code's `/fast` toggle is session-level only. Emitting
|
||
`fast_mode: true` as frontmatter on a Claude subagent is a silent no-op.
|
||
Advertising `fast_mode_supported: true` for claude would cause orchestrators to
|
||
believe the knob was wired when it is not.
|
||
|
||
#### (f) Precedence first-valid-wins
|
||
|
||
**What:** Both effort and fast_mode use a layered cascade. The test table covers
|
||
all four effort layers (invocation override → agent_overrides →
|
||
routing_tier_defaults → default) and all five fast_mode layers, including the
|
||
case where an invalid value at a higher layer correctly falls through.
|
||
|
||
**Why:** Silent precedence bugs (e.g., a numeric value in agent_overrides not
|
||
being rejected) would override intentional user config.
|
||
|
||
#### (g) Dynamic-routing composition
|
||
|
||
**What:** `resolveEffortForTier` escalates effort by attempt number
|
||
independently of the model tier mapping. The test verifies the effort ladder
|
||
(`low -> medium -> high -> xhigh -> max`), the `max` clamp, the
|
||
`max_escalations` cap, and that `escalate_on_failure: false` suppresses
|
||
escalation entirely.
|
||
|
||
**Why:** Effort escalation and model escalation share configuration
|
||
(`dynamic_routing`) but must operate independently; coupling them would cause
|
||
over-escalation or under-escalation.
|
||
|
||
#### (h) Config-tooling round-trip
|
||
|
||
**What:** `gsd-tools config-set` accepts all new key namespaces
|
||
(`effort.default`, `effort.routing_tier_defaults.<tier>`,
|
||
`effort.agent_overrides.<agent>`, `fast_mode.enabled`,
|
||
`fast_mode.routing_tier_defaults.<tier>`, `fast_mode.agent_overrides.<agent>`)
|
||
without an "Unknown config key" error, and values set via `config-set` are
|
||
reflected in `resolve-execution` output.
|
||
|
||
**Why:** The schema validation gate (`VALID_CONFIG_KEYS` + `DYNAMIC_KEY_PATTERNS`)
|
||
is separate from the resolver logic. A key missing from the schema would produce
|
||
a silent write failure and appear as a bug only at runtime.
|
||
|
||
### Coverage targets
|
||
|
||
| Suite | Target |
|
||
|---|---|
|
||
| Unit | Every cascade rule, every fallthrough, every clamp. All function branches in `resolveEffortInternal`, `resolveFastModeInternal`, `resolveEffortForTier`, `renderEffortForRuntime`. |
|
||
| Integration | All 8 architectural invariants. All 33 registered agents. All 6 provider × effort combinations for the valid-enum check. Full config-set key namespace. |
|
||
|
||
### Gaps / not yet covered
|
||
|
||
**E2E orchestrator-spawn-propagation layer (pending follow-up wiring):**
|
||
The integration tests verify that GSD resolves and renders effort values
|
||
correctly. They do NOT verify that the rendered values actually reach a spawned
|
||
Claude Code or Codex subagent at runtime. Specifically uncovered:
|
||
|
||
- `CLAUDE_CODE_EFFORT_LEVEL` env var being set and read by a spawned claude subprocess
|
||
- `output_config.effort` frontmatter key surviving the AGENTS.md template substitution
|
||
- `model_reasoning_effort` field surviving serialization into a Codex API request body
|
||
- Fast-mode `speed: "fast"` field reaching an `api`-runtime request when `fast_mode_supported: true`
|
||
|
||
These require spawning real subagents (or stubs thereof) and asserting on the
|
||
process environment / request payload — a scope that belongs in a future E2E
|
||
suite under `*.slow.test.cjs` or dedicated fixture-driven integration work.
|