9dd2c87c6c
* ci: streamline PR pipeline gates * test: annotate ci-test-scope stderr assertions for lint rule --------- Co-authored-by: Colin <colin@solvely.net>
97 lines
4.8 KiB
Markdown
97 lines
4.8 KiB
Markdown
# Testing Suites
|
|
|
|
This project's `tests/` directory uses **filename suffix markers** to group tests into named suites. The harness `scripts/run-tests.cjs` filters by suite when given `--suite <name>`. Without a flag it runs every `*.test.cjs` file (the historical default — unchanged).
|
|
|
|
> Tracked by issue [#3597](https://github.com/open-gsd/get-shit-done-redux/issues/3597).
|
|
|
|
## Suites
|
|
|
|
| Suite | Filename pattern | What goes here |
|
|
|---|---|---|
|
|
| `unit` | `*.test.cjs` (no other marker) | Default fast lane. Pure logic, no network, no external processes beyond `gsd-tools`. Most tests live here. |
|
|
| `integration` | `*.integration.test.cjs` | Cross-module flows: full installer end-to-end, multi-tool orchestration, anything that crosses two or more bin entry points. |
|
|
| `install` | `*.install.test.cjs` | Tests that perform a real install/uninstall against a sandbox project. Slower; PR CI skips these on PRs and runs them on `main` push only. |
|
|
| `security` | `*.security.test.cjs` | Adversarial input, prompt-injection guards, fixture-driven hostile-payload sweeps. |
|
|
| `slow` | `*.slow.test.cjs` | Anything that routinely takes >5s wall-clock or holds significant memory. |
|
|
| `all` | (any) | Explicit alias for "no filter". Equivalent to running with no `--suite` flag. |
|
|
|
|
## How to place a new test
|
|
|
|
1. Pick the most specific bucket above.
|
|
2. Name the file with the matching suffix: `tests/<feature>.<suite>.test.cjs`.
|
|
3. If unsure, leave the suffix off — the file lands in `unit`, the default fast lane.
|
|
|
|
Examples:
|
|
- `tests/agent-frontmatter.test.cjs` — `unit`
|
|
- `tests/prompt-injection-guards.security.test.cjs` — `security`
|
|
- `tests/installer-end-to-end.install.test.cjs` — `install`
|
|
- `tests/sdk-mutation-stress.slow.test.cjs` — `slow`
|
|
|
|
The suite-suffix convention was chosen over a directory layout (`tests/security/`) so the 545+ existing test files don't need to move. Existing files all classify as `unit` until someone explicitly retags them.
|
|
|
|
## Running suites locally
|
|
|
|
```bash
|
|
npm test # everything (backcompat — same as before)
|
|
npm run test:unit # only unit
|
|
npm run test:integration # only integration
|
|
npm run test:install # only install
|
|
npm run test:security # only security
|
|
npm run test:slow # only slow
|
|
|
|
npm run test:coverage # backcompat — coverage over EVERY test
|
|
npm run test:coverage:unit # fast coverage signal — only unit suite
|
|
npm run test:coverage:all # alias for test:coverage
|
|
```
|
|
|
|
Direct harness invocation also works:
|
|
|
|
```bash
|
|
node scripts/run-tests.cjs --suite security
|
|
node scripts/run-tests.cjs --suite=security
|
|
node scripts/run-tests.cjs --files "tests/command-contract.test.cjs tests/core.test.cjs"
|
|
node scripts/run-tests.cjs --files-from .ci-selected-tests.txt
|
|
```
|
|
|
|
Unknown suites exit non-zero with the list of valid suites. Empty suites (e.g. `--suite security` before any security-tagged file exists) exit `0` with a `no tests in suite "..."` notice on stderr so CI lanes don't go red while a suite is being populated.
|
|
|
|
## CI matrix
|
|
|
|
The `Tests` workflow runs every PR through a scoped gate generated by
|
|
`scripts/ci-test-scope.cjs`.
|
|
|
|
| Lane | Node 22 | Node 24 |
|
|
|---|---|---|
|
|
| `ubuntu-latest` | scoped tests | unit + integration + security |
|
|
| `windows-latest` | — | scoped Windows/path/shell tests |
|
|
| `macos-latest` | full parity when required | full parity when required |
|
|
|
|
- **Node 22** is the `engines.node` floor (`>=22.0.0`) — must stay green.
|
|
- **Node 24** is the default development lane.
|
|
- **Scoped tests** are selected from the changed paths, plus a small CLI/package
|
|
smoke set. They are for confidence on the affected surface, not for counting
|
|
tests.
|
|
|
|
The default PR gate runs the broad `unit`, `integration`, and `security` suites
|
|
once on Ubuntu / Node 24, scoped smoke on Ubuntu / Node 22, scoped
|
|
Windows/path/shell tests on Windows / Node 24, and unit coverage once on Ubuntu /
|
|
Node 24. PRs touching workflow, package, test-runner, install, release, or
|
|
Windows-sensitive surfaces also run the full parity matrix on macOS and the
|
|
older Windows runtime, plus `install` and `slow` on the primary Ubuntu lane.
|
|
Coverage stays single-lane because multiplying coverage across OS/runtime lanes
|
|
adds cost without improving the threshold signal.
|
|
|
|
To inspect the scope locally:
|
|
|
|
```bash
|
|
npm run ci:test-scope -- --files "commands/gsd/plan-phase.md"
|
|
node scripts/ci-test-scope.cjs --base origin/next --head HEAD
|
|
```
|
|
|
|
## Best practices for forward-compat (Node 24/26)
|
|
|
|
- Use `process.execPath` when spawning Node in tests so each matrix lane exercises the lane's Node version.
|
|
- Avoid stack-trace or error-message prose assertions. Assert `err.code`, structured JSON fields, or enums — Node minor releases routinely tweak error wording.
|
|
- Prefer `node:test`, `node:assert/strict`, and `node:test` mocks. No external test frameworks.
|
|
- Coverage uses `c8` and propagates `NODE_V8_COVERAGE` through the harness's child process.
|