# ADR-2966: Test the five-step loop as a continuous walk, not isolated points

- **Status:** Accepted
- **Date:** 2026-08-01
- **Issue:** [#2966](https://github.com/open-gsd/gsd-core/issues/2966)
- **Supersedes:** —
- **Relationship to prior work:** Extends `RULESET.TESTS.feedback-loop-convergence` (`CONTEXT.md`) from its existing single instance, `tests/estimate-loop-convergence.test.cjs`, to the full five-step loop.

## Context

131 of roughly 705 test files already drive the real `msd-tools` binary through `runMsdTools` in `tests/helpers.cjs` — CLI-altitude testing is solved and well trodden. What no existing test does is carry one project's accumulating state across all five loop steps in a single run. Every loop test today asserts a step in isolation, against state fabricated for that step alone. `CONTEXT.md`'s `RULESET.TESTS.feedback-loop-convergence` already names the alternative — assert convergence across an accumulating sequence — and one instance of it exists for estimation (`tests/estimate-loop-convergence.test.cjs`). This ADR extends that pattern to the loop itself.

## Decision

### 1. Walk the loop as a continuous trajectory

Build a walk driver, `tests/qa/loop-walk.cjs`, that runs the five loop steps in order against one project's accumulating on-disk state, rather than five independent point tests each seeded from scratch. This is `RULESET.TESTS.feedback-loop-convergence` generalized from estimation to the loop, anchored to the existing convergence instance rather than invented fresh.

### 2. Layer over `tests/helpers.cjs`; do not build a second substrate

`runMsdTools` already invokes the real subprocess with a 60s timeout and env injection, and 131 files depend on it. The walk driver is built strictly on top of it. The rejected alternative — a standalone harness re-implementing subprocess invocation — was considered and rejected as duplication (Gall's Law: a working complex system evolves from a working simple one).

### 3. The stub agent seam, and its limit stated plainly

`msd-tools init new-project` (and the other loop init commands) write nothing — they are read-only context projections; the agent is the one that authors artifacts (STATE.md, phase files, etc.). A scripted stub writer standing in for the agent is therefore faithful to the real division of labor, and the entire engine — routing, state transitions, gates — runs for real behind it.

The limit is stated plainly and not glossed over: this walk proves the engine's state machine is coherent across all five steps. It proves nothing about whether a real LLM agent emits artifacts of the shape the walk's fixtures assume.

### 4. Oracles read a typed IR only, never rendered text

`tests/qa/oracles.cjs`'s ten invariant oracles (seven `SEVERITY.VIOLATION` oracles plus three `SEVERITY.SMELL` oracles — see §5) assert exclusively against `tests/qa/result.cjs`'s typed `RunResult` IR, never against raw stdout/stderr text. This is forced, not stylistic: `CONTRIBUTING.md` → "Prohibited: Raw Text Matching on Test Outputs" and `RULESET.TESTS.no-source-grep.tmp-file-traps` both ban it. The same constraint governs the tree-idempotence oracle: it compares `fs.statSync` facts (`size`, `mtimeMs`, `isFile()`) across runs, never content hashing or reading the SUT's own tmp-file output, which would trip the identical lint.

### 5. Findings carry severity; the harness reports evidence, it does not adjudicate

`tests/qa/oracles.cjs`'s `check(ctx)` outcomes carry a `severity`:

- `SEVERITY.VIOLATION` — the documented contract is broken. Fails `runOracles(ctx).failed` and therefore fails the build.
- `SEVERITY.SMELL` — legal under today's implementation but structurally questionable. Recorded in `.smells`, never folded into `.failed`, and never fails a build.

Rationale: an oracle set derived from current behavior can only ever confirm current behavior — that framing turns the harness into a conformance test for the status quo. Severity is what lets the harness say "this works and is still wrong," which is the sentence a quality tool must be able to form. The maintainer decides which smells become fixes; the tool's job is to make the trade visible instead of invisible.

Consequence, stated as an explicit safety property: `runOracles(ctx).failed` is a getter that deliberately aliases `violations` only — it never includes `smells`. Adding a smell oracle, or a new smell case to an existing oracle, can therefore never redden CI. The severity split is what makes it safe to keep adding observational oracles without turning every new observation into a build break.

### 6. `output({error: …})` is not changed here — its cost is now visible on every occurrence

42 call sites use `output({error: …})`: a JSON payload carrying an `error` key on stdout, exit 0. This is deliberately distinct from `error(msg, reason)`, which writes `{ok:false, reason, message}` to stderr and exits 1. `get_impact` rates `cmdStateSnapshot` — the function this idiom threads through — **CRITICAL** (55 affected symbols, 23 processes). Normalizing all 42 sites to the hard-failure path was considered and rejected: it would flip exit codes 0 → 1 across a seam that workflows and agents currently treat as a soft signal — Hyrum's Law applies directly, since the exit-0 behavior is observable and already depended on.

This is not left alone because it is judged fine. It is not changed here because a blast radius of that size warrants a separate, deliberate decision, not one folded into a QA-harness ADR — and the `soft-error-exit-zero` smell (§5) now fires on every occurrence so the cost stays visible instead of quietly disappearing back into the status quo. The concrete cost for a caller: a shell caller's `if ! cmd; then` is blind to a failure reported through a payload key with exit 0 — the process exits 0, so the conditional never trips. The walk's `RunResult` IR types this shape explicitly (`kind: 'soft-error'`) so it can be reported as a smell rather than silently swallowed.

### 7. The `docs/json-errors.md` vs. `src/cli-exit.cts` contract conflict is surfaced, not resolved

`docs/json-errors.md` states that every error emits exactly one JSON line to stderr and exits 1. `src/cli-exit.cts`'s `runMain` returns before reaching the json-error branch for `ExitError`, so CLI usage errors deliberately emit plain text instead. Both behaviors are current and they contradict each other.

This ADR does not pick a side — resolving it means changing every `ExitError` throw site, which is a separate decision with its own blast radius. What changes is how the conflict is surfaced: the `contract-conflict` smell (§5) fires on every occurrence in json-error mode and quotes both sides in its detail — the `docs/json-errors.md` sentence and the `src/cli-exit.cts` `runMain` early-return that produces the breach — so a reader gets the evidence needed to decide, not a bare note that a conflict exists. This is layered on top of `json-contract`, which already reports the same observation as a VIOLATION of the documented contract (`RunResult.kind: 'unstructured-error'` on rows the doc says should be structured); `contract-conflict` adds the reason it is ambiguous rather than a clear-cut bug.

### 8. Fixture provenance is honest, not borrowed

Scenario artifacts live at `tests/qa/fixtures/`, template-derived, carrying an explicit provenance comment. They deliberately do **not** live at `tests/fixtures/representative/`, which asserts real-user provenance this repo cannot substantiate — no `.planning/` directory exists in this repo and no real loop-artifact fixtures exist to source from. Per #2371, template-derived provenance is adequate for happy-path and sequence scenarios, because templates are exactly what agents are instructed to emit. It is **not** adequate for a negative fixture asserting that the engine correctly rejects malformed input — a rejection test needs to be grounded in what the engine's grammar actually forbids, not a plausible guess at malformed shape. Perturbations (`tests/qa/mutations.cjs`) are exempt from this constraint: a CRLF/BOM/truncate transform is drawn from a generic corruption catalog, not from the engine's grammar, so it carries no provenance claim to substantiate.

### 9. New `qa` test-suite marker, excluded from the default `unit` lane

`scripts/run-tests.cjs` gains a `qa` suite marker for `tests/loop-walk.qa.test.cjs` and its dependents, excluded from the default `unit` lane. The walk's subprocess fan-out is deliberately bounded and run sequentially within a file: `RULESET.HARNESS.test-memory-guard` denies node spawns above 4 GiB aggregate RSS, and parallelizing scenario runs would risk crossing it.

### What the first run found

Running the walk driver against the real engine — not a mock — produced 0 violations and three smell classes on this first pass:

- `value-hygiene` — `init` returns `agents_dir` pointing outside the project tree.
- `untyped-success` — `smart-entry` emits prose unconditionally (`msd-core/bin/lib/smart-entry.cjs:577-587`), so the routing oracle cannot assert on it until a `--json` mode exists.
- `soft-error-exit-zero` — `state-snapshot` reports a missing STATE.md through a payload key with exit 0.

These are observations for a maintainer to weigh, not defects this PR fixes.

## Consequences

- **Positive — makes easy.** A defect that only manifests as accumulated state drifting across loop steps (a stale pointer, a field one step wrote that a later step misreads) is now catchable by a single walk instead of requiring a human to hand-construct multi-step state. The `RunResult` IR gives every future loop-facing test a typed, lint-compliant surface to assert against instead of re-deriving raw-text parsing per test. The convergence pattern (`RULESET.TESTS.feedback-loop-convergence`) now has a second, load-bearing instance beyond estimation, making it a demonstrated pattern rather than a one-off.
- **Positive — anti-vacuity guard.** The end-to-end test (`tests/loop-walk.qa.test.cjs`) asserts the walk produces at least one smell, because a QA harness reporting nothing on a first run against a real engine is more likely mis-specified than the engine is perfect. It deliberately does **not** assert exact smell ids or counts — pinning those would re-freeze current behavior, the same failure mode the severity split (§5) exists to avoid.
- **Negative — makes harder.** The walk is one more thing to keep in sync with the five-step loop's actual shape; `LOOP_HOST_CONTRACT` is consumed from the generated module rather than re-listed, specifically to prevent this from becoming a second copy that drifts (per the design's known-defect gauntlet). The `qa` suite's sequential-only execution bounds wall-clock cost as scenario count grows.
- **Known limits, stated rather than hidden.**
  - The stub agent is not an LLM. The walk validates the engine's state machine; it says nothing about whether a real agent produces artifacts of the assumed shape.
  - `ship` has no runnable predicate evaluator (`tests/loop-hooks-ship-pre-e2e.test.cjs`: "enforcement is ship.md prose only"), so the `ship` step is asserted at contract-shape level only, not behaviorally.
  - `smart-entry` emits prose unconditionally on success (`msd-core/bin/lib/smart-entry.cjs:577-587`), which is one of several untyped-success commands (`config-path`, `audit-open`, `skill-manifest`, and others). The routing oracle cannot assert on `smart-entry`'s routing decision until it gains a `--json` mode; that is future work, not covered here.
  - The open `docs/json-errors.md` vs. `src/cli-exit.cts` contract conflict (§6) remains unresolved by this ADR.

## References

- `CONTEXT.md` → `RULESET.TESTS.feedback-loop-convergence`
- `tests/estimate-loop-convergence.test.cjs` — the existing convergence instance this ADR generalizes
- `tests/helpers.cjs` — `runMsdTools`, the substrate the walk is layered over
- `tests/qa/loop-walk.cjs`, `tests/qa/result.cjs`, `tests/qa/oracles.cjs`, `tests/qa/mutations.cjs`, `tests/qa/fixtures/**`, `tests/qa/scenarios/*.json`
- `tests/loop-walk.qa.test.cjs`
- `src/io.cts` — `output` (`output({error})` soft-failure idiom, 42 sites, `CRITICAL` per `get_impact`), `error`
- `src/state.cts:1388` — `cmdStateSnapshot`
- `src/cli-exit.cts` — `runMain`, the `ExitError` early-return that produces the §6 conflict
- `docs/json-errors.md`
- `CONTRIBUTING.md` → "Prohibited: Raw Text Matching on Test Outputs"
- `msd-core/bin/lib/smart-entry.cjs:577-587`
- Issue #2371 (fixture provenance standard)
- `.msd/phase/test-2966-loop-qa-walk/40-design.md` — the design this ADR records