diff --git a/docs/TESTING-SUITES.md b/docs/TESTING-SUITES.md index aa126fdf2..fa4e199cb 100644 --- a/docs/TESTING-SUITES.md +++ b/docs/TESTING-SUITES.md @@ -219,6 +219,47 @@ disagree, trust (and fix) the rule table. Unknown suites exit non-zero with the list of valid suites. Empty suites (e.g. `--suite security` before any security-tagged file exists) exit `0` with a `no tests in suite "..."` notice on stderr so CI lanes don't go red while a suite is being populated. +## The live-config hermeticity guard + +Every `run-tests.cjs` invocation snapshots GSD's own install footprint in each +live runtime config directory before the suite and re-checks it afterwards. It +exists because the failure it catches is silent by construction: a test that +resolves a config directory from the ambient environment instead of a sandbox +writes into *your real* `~/.claude` (or `$GSD_HOME/.gsd`, or a Kimi +`config.toml`), and nothing reports it. CI cannot catch this class at all — +CI never has `CLAUDE_CONFIG_DIR` and friends set. + +The guard watches only what GSD unambiguously owns — its top-level install +footprint plus `gsd-`-prefixed children of directories shared with the host +agent — never whole config roots, because a host agent legitimately writing +`history.jsonl` mid-run would make the guard cry wolf, and a guard that cries +wolf gets switched off. + +Two environment variables control it: + +| Variable | Effect | +|---|---| +| `GSD_STRICT_LIVE_CONFIG_GUARD=1` | A detected write **fails the run**. Set on the Linux/macOS lanes of every CI job that runs the suite. | +| `GSD_SKIP_LIVE_CONFIG_GUARD=1` | Skips the check entirely. | + +Unset, the guard **reports and does not fail** — deliberately, not timidly. On +its first CI run it surfaced pre-existing leaks on the Windows lane, where +`os.homedir()` reads `USERPROFILE` and ~190 test sites sandbox `HOME` alone. +Those are real and worth fixing, but they are a different defect class, and a +brand-new gate that instantly reds an unrelated lane gets reverted rather than +obeyed. Windows lanes therefore stay report-only until that sweep lands; this +repo has the pattern already, in the `local/no-source-grep` ESLint rule that +shipped at `warn` and was promoted to `error` after its cleanup (ADR 452). + +`GSD_SKIP_LIVE_CONFIG_GUARD` is a bypass on a safety check, so it is documented +here rather than left to be discovered in the source: an undocumented bypass is +one people eventually set without knowing what they turned off. If you need it +routinely, that is a bug report, not a workflow. + +Reported paths are labelled `CREATED`, `MODIFIED`, `DELETED`, or `UNVERIFIED`. +`UNVERIFIED` means a scan bound was hit and the path could not be attested +either way — it is never the same as clean. + ## CI matrix The `Tests` workflow runs every PR through a scoped gate generated by diff --git a/tests/live-config-guard.test.cjs b/tests/live-config-guard.test.cjs index 4e79bd046..6514949ea 100644 --- a/tests/live-config-guard.test.cjs +++ b/tests/live-config-guard.test.cjs @@ -606,6 +606,17 @@ describe('#2665 round 4: CI wires the guard to strict mode', () => { // green. Windows lanes are deliberately report-only until the documented // pre-existing USERPROFILE leak class is swept (SEVERITY note in // scripts/live-config-guard.cjs) — so the assertion is per-OS, not global. + test('both guard env vars are documented for humans, not just in the source', () => { + // A skip-switch on a safety guard has to be discoverable: an undocumented + // bypass is one people eventually set without knowing what they turned off. + // The Docs Required gate gets satisfied by ANY docs/ file in the diff -- + // including a generated index -- so it cannot stand in for this. + const doc = fs.readFileSync(path.join(__dirname, '..', 'docs', 'TESTING-SUITES.md'), 'utf8'); + for (const v of ['GSD_STRICT_LIVE_CONFIG_GUARD', 'GSD_SKIP_LIVE_CONFIG_GUARD']) { + assert.ok(doc.includes(v), `${v} must be documented in docs/TESTING-SUITES.md`); + } + }); + // DERIVED, not hand-listed. The previous version named three jobs as literals, // so it could not see a FOURTH lane that runs the suite — and there was one: // qa-loop-walk reaches run-tests.cjs through `npm run test:qa` and escaped