c933184b97
* test(#3172): failing-first suite for the stated failing-direction probe Pins the <fails_when> pairing walk, placeholder denylist, MISSING sentinel exemption, degraded-read contract, CLI arm and the plan-authoring contract text. RED by construction: the module exports it requires do not exist yet. Executed on the remote runner. * feat(#3172): require a stated failing direction for every automated acceptance command Every runnable <automated> command now carries a <fails_when> sibling naming what output constitutes failure. A command with no expressible failure mode is not an acceptance test: it reads as rigour and is not falsifiable. - verify-command-grounding gains a failing-direction probe sharing the existing <automated> grammar, MISSING sentinel and walk guard rather than copying them - gsd-tools check verify-failure-directions <N> backs it; plan-phase dispatches it and hands the JSON to gsd-plan-checker check 8f - Dimension 8 detail extracted to references to stay under the agent size cap Verified on the remote runner. * fix(#3172): close four review findings in the failing-direction probe - MISSING_SENTINEL_RE matched an env-var assignment prefix (MISSING=1 cmd), so a real command was exempted from the new blocking gate. Tightened the SHARED constant rather than adding a second copy. - Both token regexes scanned to EOF on unclosed openers (O(n^2), 1562ms at 40k). Bodies are now non-crossing; 1ms, byte-identical on well-formed input. The pre-existing AUTOMATED_BLOCK_RE carried the same defect and is fixed here too. - probePhaseFailingDirections reported status 'ok' when one plan was unreadable, conflating 'could not look' with 'nothing to report'. - Extracted the phase-resolution block both check arms had copied verbatim. Also corrects a docs/AGENTS.md dimension list stale since #2401. Verified on the remote runner. * fix(#3172): project the planner rule onto the spawn contract, settle emitted bookkeeping The remote runner refuted the planner-side edit. agents/gsd-planner.md is frozen under a 49152-LF-char cap asserted by four suites and sat at 49,146 — six chars of headroom — so the +537 of authoring rule blew it. #3297/#3645 already settled where such a rule goes: the planner spawn contract in plan-phase.md, beside <tracked_source_paths>. The agent file is reverted to origin/next verbatim. - plan-phase.md gains <failing_direction_contract>; tests row 30 now asserts the contract there and row 30b guards the freeze in both directions - plan-phase.md growth acknowledged by APPENDING to the 3409 fragment, per the precedent that two ack sources may never name the same path - install-tree fixtures regenerated for the three new reference files Verified on the remote runner. * chore(#3172): backfill PR number into the changeset fragment pr:0 -> pr:3825 now that the PR exists. --------- Co-authored-by: sim <sim@local>
79 lines
4.2 KiB
Markdown
79 lines
4.2 KiB
Markdown
# Check 8f — Stated Failing Direction (#3172)
|
||
|
||
> Reference file for gsd-plan-checker agent. Loaded on-demand via `@` reference.
|
||
|
||
**Question:** For each runnable `<automated>` command, does the plan say what output constitutes
|
||
failure?
|
||
|
||
Checks 8a–8d ask whether an acceptance command is *present*, and Verify Command Path
|
||
Resolvability asks whether its target *resolves*. This one asks whether the command is
|
||
**falsifiable at all**. A command with no expressible failure mode is not an acceptance test —
|
||
it reads as rigour and delivers none.
|
||
|
||
#3172: a planner emitted 21 `<automated>` commands that could not run. Cargo exited non-zero, so
|
||
that instance failed loudly — luck, not design. The same class of error with a command that
|
||
exits 0 on a no-op passes green and silently. Requiring a stated failing direction is the only
|
||
shape that catches the silent case, because it forces the plan to name the failure signal rather
|
||
than assume the command has one.
|
||
|
||
## The contract
|
||
|
||
```xml
|
||
<verify>
|
||
<automated>npm --prefix apps/api test -- auth.spec.ts</automated>
|
||
<fails_when>non-zero exit, or "0 passed" in the summary line</fails_when>
|
||
</verify>
|
||
```
|
||
|
||
Within one `<task>`, each `<fails_when>` binds to the nearest **preceding** `<automated>`; a
|
||
command's binding statement is the **first** one that follows it. N runnable commands need N
|
||
statements.
|
||
|
||
## Do not hand-reason this
|
||
|
||
`gsd-core/workflows/plan-phase.md` runs the deterministic probe **before** spawning this checker
|
||
and interpolates the result into the verification prompt as `{FAILING_DIRECTIONS}`, inside a
|
||
`<failing_direction_probe>` block. This check reads that already-supplied JSON — it never invokes
|
||
`gsd_run check verify-failure-directions` itself. If `{FAILING_DIRECTIONS}` is absent from the
|
||
prompt, treat this check as silent (nothing to check) rather than trying to run the probe.
|
||
|
||
The probe never executes command text (PLAN.md is untrusted, LLM-authored). It is a **presence**
|
||
recognizer: it proves a statement exists and is not a placeholder. It does not judge whether the
|
||
statement names the *right* signal — that judgment is yours, below.
|
||
|
||
## Process — act on `severity` only
|
||
|
||
| `severity` | `status` | Action |
|
||
|---|---|---|
|
||
| `blocker` | `missing` | **BLOCKER** — quote the `command` verbatim: it has no stated failing direction |
|
||
| `blocker` | `empty` | **BLOCKER** — a `<fails_when>` is present but blank |
|
||
| `blocker` | `placeholder` | **BLOCKER** — quote the placeholder text; `TBD` is not a failure signal |
|
||
| `warning` | `orphan` | **WARNING** — a `<fails_when>` that follows no command; it satisfies nothing |
|
||
| `none` | `ok` / `sentinel` | silent |
|
||
|
||
Rules:
|
||
|
||
- **Report, never prescribe.** State which command has no stated failure mode. Do **not** author
|
||
the statement for the planner. A prescribed statement is copied verbatim and carries zero
|
||
information — that reproduces #3172 one level up, exactly as the #2401 probe refuses to
|
||
prescribe a replacement path.
|
||
- `status: sentinel` is a `MISSING — Wave 0 must create …` placeholder command. It is not
|
||
runnable, so it has no failure mode to state. **Not a finding.** Say nothing; checks 8a/8d
|
||
own it.
|
||
- A non-empty `readError` means the probe **could not look**. Report that as a WARNING in its own
|
||
words; it is not a clean bill of health.
|
||
- **Your added judgment, on `ok` rows only:** a statement that is present, non-empty and
|
||
non-placeholder can still be vacuous — *"the command fails"*, *"it doesn't work"*, *"an error
|
||
occurs"* restate the word "failure" without naming an observable signal. Raise those as a
|
||
**WARNING**, naming what a usable statement looks like (an exit code, a string in the output, a
|
||
missing line). Do **not** escalate a vacuous statement to BLOCKER: the deterministic layer owns
|
||
the blockers so that a BLOCKER is always reproducible, and prose judgment stays advisory.
|
||
- **Length is not a signal.** `<fails_when>non-zero exit</fails_when>` is a complete failing
|
||
direction. There is no minimum length, word count, or required keyword.
|
||
|
||
## Not in scope
|
||
|
||
A command that runs successfully and asserts nothing (a test-name filter matching zero tests and
|
||
exiting 0) is the adjacent **vacuous pass** problem. It is explicitly out of scope for this
|
||
check — see the issue's "Out of scope". Do not report it here.
|