* test(#3172): failing-first suite for the stated failing-direction probe Pins the <fails_when> pairing walk, placeholder denylist, MISSING sentinel exemption, degraded-read contract, CLI arm and the plan-authoring contract text. RED by construction: the module exports it requires do not exist yet. Executed on the remote runner. * feat(#3172): require a stated failing direction for every automated acceptance command Every runnable <automated> command now carries a <fails_when> sibling naming what output constitutes failure. A command with no expressible failure mode is not an acceptance test: it reads as rigour and is not falsifiable. - verify-command-grounding gains a failing-direction probe sharing the existing <automated> grammar, MISSING sentinel and walk guard rather than copying them - gsd-tools check verify-failure-directions <N> backs it; plan-phase dispatches it and hands the JSON to gsd-plan-checker check 8f - Dimension 8 detail extracted to references to stay under the agent size cap Verified on the remote runner. * fix(#3172): close four review findings in the failing-direction probe - MISSING_SENTINEL_RE matched an env-var assignment prefix (MISSING=1 cmd), so a real command was exempted from the new blocking gate. Tightened the SHARED constant rather than adding a second copy. - Both token regexes scanned to EOF on unclosed openers (O(n^2), 1562ms at 40k). Bodies are now non-crossing; 1ms, byte-identical on well-formed input. The pre-existing AUTOMATED_BLOCK_RE carried the same defect and is fixed here too. - probePhaseFailingDirections reported status 'ok' when one plan was unreadable, conflating 'could not look' with 'nothing to report'. - Extracted the phase-resolution block both check arms had copied verbatim. Also corrects a docs/AGENTS.md dimension list stale since #2401. Verified on the remote runner. * fix(#3172): project the planner rule onto the spawn contract, settle emitted bookkeeping The remote runner refuted the planner-side edit. agents/gsd-planner.md is frozen under a 49152-LF-char cap asserted by four suites and sat at 49,146 — six chars of headroom — so the +537 of authoring rule blew it. #3297/#3645 already settled where such a rule goes: the planner spawn contract in plan-phase.md, beside <tracked_source_paths>. The agent file is reverted to origin/next verbatim. - plan-phase.md gains <failing_direction_contract>; tests row 30 now asserts the contract there and row 30b guards the freeze in both directions - plan-phase.md growth acknowledged by APPENDING to the 3409 fragment, per the precedent that two ack sources may never name the same path - install-tree fixtures regenerated for the three new reference files Verified on the remote runner. * chore(#3172): backfill PR number into the changeset fragment pr:0 -> pr:3825 now that the PR exists. --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
78
gsd-core/references/failing-direction.md
Normal file
78
gsd-core/references/failing-direction.md
Normal file
@@ -0,0 +1,78 @@
|
||||
# Check 8f — Stated Failing Direction (#3172)
|
||||
|
||||
> Reference file for gsd-plan-checker agent. Loaded on-demand via `@` reference.
|
||||
|
||||
**Question:** For each runnable `<automated>` command, does the plan say what output constitutes
|
||||
failure?
|
||||
|
||||
Checks 8a–8d ask whether an acceptance command is *present*, and Verify Command Path
|
||||
Resolvability asks whether its target *resolves*. This one asks whether the command is
|
||||
**falsifiable at all**. A command with no expressible failure mode is not an acceptance test —
|
||||
it reads as rigour and delivers none.
|
||||
|
||||
#3172: a planner emitted 21 `<automated>` commands that could not run. Cargo exited non-zero, so
|
||||
that instance failed loudly — luck, not design. The same class of error with a command that
|
||||
exits 0 on a no-op passes green and silently. Requiring a stated failing direction is the only
|
||||
shape that catches the silent case, because it forces the plan to name the failure signal rather
|
||||
than assume the command has one.
|
||||
|
||||
## The contract
|
||||
|
||||
```xml
|
||||
<verify>
|
||||
<automated>npm --prefix apps/api test -- auth.spec.ts</automated>
|
||||
<fails_when>non-zero exit, or "0 passed" in the summary line</fails_when>
|
||||
</verify>
|
||||
```
|
||||
|
||||
Within one `<task>`, each `<fails_when>` binds to the nearest **preceding** `<automated>`; a
|
||||
command's binding statement is the **first** one that follows it. N runnable commands need N
|
||||
statements.
|
||||
|
||||
## Do not hand-reason this
|
||||
|
||||
`gsd-core/workflows/plan-phase.md` runs the deterministic probe **before** spawning this checker
|
||||
and interpolates the result into the verification prompt as `{FAILING_DIRECTIONS}`, inside a
|
||||
`<failing_direction_probe>` block. This check reads that already-supplied JSON — it never invokes
|
||||
`gsd_run check verify-failure-directions` itself. If `{FAILING_DIRECTIONS}` is absent from the
|
||||
prompt, treat this check as silent (nothing to check) rather than trying to run the probe.
|
||||
|
||||
The probe never executes command text (PLAN.md is untrusted, LLM-authored). It is a **presence**
|
||||
recognizer: it proves a statement exists and is not a placeholder. It does not judge whether the
|
||||
statement names the *right* signal — that judgment is yours, below.
|
||||
|
||||
## Process — act on `severity` only
|
||||
|
||||
| `severity` | `status` | Action |
|
||||
|---|---|---|
|
||||
| `blocker` | `missing` | **BLOCKER** — quote the `command` verbatim: it has no stated failing direction |
|
||||
| `blocker` | `empty` | **BLOCKER** — a `<fails_when>` is present but blank |
|
||||
| `blocker` | `placeholder` | **BLOCKER** — quote the placeholder text; `TBD` is not a failure signal |
|
||||
| `warning` | `orphan` | **WARNING** — a `<fails_when>` that follows no command; it satisfies nothing |
|
||||
| `none` | `ok` / `sentinel` | silent |
|
||||
|
||||
Rules:
|
||||
|
||||
- **Report, never prescribe.** State which command has no stated failure mode. Do **not** author
|
||||
the statement for the planner. A prescribed statement is copied verbatim and carries zero
|
||||
information — that reproduces #3172 one level up, exactly as the #2401 probe refuses to
|
||||
prescribe a replacement path.
|
||||
- `status: sentinel` is a `MISSING — Wave 0 must create …` placeholder command. It is not
|
||||
runnable, so it has no failure mode to state. **Not a finding.** Say nothing; checks 8a/8d
|
||||
own it.
|
||||
- A non-empty `readError` means the probe **could not look**. Report that as a WARNING in its own
|
||||
words; it is not a clean bill of health.
|
||||
- **Your added judgment, on `ok` rows only:** a statement that is present, non-empty and
|
||||
non-placeholder can still be vacuous — *"the command fails"*, *"it doesn't work"*, *"an error
|
||||
occurs"* restate the word "failure" without naming an observable signal. Raise those as a
|
||||
**WARNING**, naming what a usable statement looks like (an exit code, a string in the output, a
|
||||
missing line). Do **not** escalate a vacuous statement to BLOCKER: the deterministic layer owns
|
||||
the blockers so that a BLOCKER is always reproducible, and prose judgment stays advisory.
|
||||
- **Length is not a signal.** `<fails_when>non-zero exit</fails_when>` is a complete failing
|
||||
direction. There is no minimum length, word count, or required keyword.
|
||||
|
||||
## Not in scope
|
||||
|
||||
A command that runs successfully and asserts nothing (a test-name filter matching zero tests and
|
||||
exiting 0) is the adjacent **vacuous pass** problem. It is explicitly out of scope for this
|
||||
check — see the issue's "Out of scope". Do not report it here.
|
||||
74
gsd-core/references/nyquist-compliance.md
Normal file
74
gsd-core/references/nyquist-compliance.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Dimension 8 — Nyquist Compliance (checks 8a–8e)
|
||||
|
||||
> Reference file for gsd-plan-checker agent. Loaded on-demand via `@` reference.
|
||||
> Extracted from `agents/gsd-plan-checker.md` in #3172 to stay under the agent's
|
||||
> LARGE size cap. Checks 8a-8e are moved unchanged; the only edit is to the
|
||||
> Dimension 8 Output table, which gains a `Failing Direction` column and a
|
||||
> stated/runnable tally so check 8f's result is reported alongside them.
|
||||
> Check 8f itself lives in its own reference:
|
||||
> @gsd-core/references/failing-direction.md
|
||||
|
||||
**Skip if:** `workflow.nyquist_validation` is explicitly set to `false` in config.json
|
||||
(absent key = enabled), phase has no RESEARCH.md, or RESEARCH.md has no "Validation
|
||||
Architecture" section. Output: "Dimension 8: SKIPPED (nyquist_validation disabled or not
|
||||
applicable)"
|
||||
|
||||
## Check 8e — VALIDATION.md Existence (Gate)
|
||||
|
||||
Before running checks 8a-8d, verify VALIDATION.md exists:
|
||||
|
||||
```bash
|
||||
ls "${PHASE_DIR}"/*-VALIDATION.md 2>/dev/null
|
||||
```
|
||||
|
||||
**If missing:** **BLOCKING FAIL** — "VALIDATION.md not found for phase {N}. Re-run
|
||||
`/gsd:plan-phase {N} --research` to regenerate."
|
||||
Skip checks 8a-8d entirely. Report Dimension 8 as FAIL with this single issue.
|
||||
|
||||
**If exists:** Proceed to checks 8a-8d.
|
||||
|
||||
## Check 8a — Automated Verify Presence
|
||||
|
||||
For each `<task>` in each plan:
|
||||
- `<verify>` must contain `<automated>` command, OR a Wave 0 dependency that creates the test
|
||||
first
|
||||
- If `<automated>` is absent with no Wave 0 dependency → **BLOCKING FAIL**
|
||||
- If `<automated>` says "MISSING", a Wave 0 task must reference the same test file path →
|
||||
**BLOCKING FAIL** if link broken
|
||||
|
||||
## Check 8b — Feedback Latency Assessment
|
||||
|
||||
For each `<automated>` command:
|
||||
- Full E2E suite (playwright, cypress, selenium) → **WARNING** — suggest faster unit/smoke test
|
||||
- Watch mode flags (`--watchAll`) → **BLOCKING FAIL**
|
||||
- Delays > 30 seconds → **WARNING**
|
||||
|
||||
## Check 8c — Sampling Continuity
|
||||
|
||||
Map tasks to waves. Per wave, any consecutive window of 3 implementation tasks must have ≥2
|
||||
with `<automated>` verify. 3 consecutive without → **BLOCKING FAIL**.
|
||||
|
||||
## Check 8d — Wave 0 Completeness
|
||||
|
||||
For each `<automated>MISSING</automated>` reference:
|
||||
- Wave 0 task must exist with matching `<files>` path
|
||||
- Wave 0 plan must execute before dependent task
|
||||
- Missing match → **BLOCKING FAIL**
|
||||
|
||||
## Dimension 8 Output
|
||||
|
||||
```
|
||||
## Dimension 8: Nyquist Compliance
|
||||
|
||||
| Task | Plan | Wave | Automated Command | Failing Direction | Status |
|
||||
|------|------|------|-------------------|-------------------|--------|
|
||||
| {task} | {plan} | {wave} | `{command}` | `{fails_when}` / ❌ | ✅ / ❌ |
|
||||
|
||||
Sampling: Wave {N}: {X}/{Y} verified → ✅ / ❌
|
||||
Wave 0: {test file} → ✅ present / ❌ MISSING
|
||||
Failing directions: {stated}/{runnable} → ✅ / ❌
|
||||
Overall: ✅ PASS / ❌ FAIL
|
||||
```
|
||||
|
||||
If FAIL: return to planner with specific fixes. Same revision loop as other dimensions
|
||||
(max 3 loops).
|
||||
53
gsd-core/references/planner-failing-direction.md
Normal file
53
gsd-core/references/planner-failing-direction.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# Stated Failing Direction (#3172)
|
||||
|
||||
> Reference file for the gsd-planner agent. Loaded on-demand via `@` reference from the
|
||||
> `<failing_direction_contract>` block of the planner spawn prompt in
|
||||
> `gsd-core/workflows/plan-phase.md` — NOT from `agents/gsd-planner.md`, which is frozen
|
||||
> under a 49152-LF-char cap, so planner-side rules are projected onto its spawn contract
|
||||
> (the #3297 / #3645 precedent).
|
||||
|
||||
**Every runnable `<automated>` command needs a `<fails_when>` sibling naming what output
|
||||
constitutes failure.** A command with no expressible failure mode is not an acceptance test.
|
||||
|
||||
```xml
|
||||
<verify>
|
||||
<automated>npm --prefix apps/api test -- auth.spec.ts</automated>
|
||||
<fails_when>non-zero exit, or "0 passed" in the summary line</fails_when>
|
||||
</verify>
|
||||
```
|
||||
|
||||
**Why.** #3172: six plans shipped 21 `<automated>` commands that could not run at all — a
|
||||
`--lib` target against a binary-only package. They sat inside the very blocks that decide whether
|
||||
work is done, so the acceptance criteria for those plans were improvised at execution time by
|
||||
three separate executors instead of reviewed at planning time. Cargo happened to exit non-zero,
|
||||
so it failed loudly. The identical mistake with a command that exits 0 on a no-op — a test-name
|
||||
filter matching nothing — passes green and silently. Naming the failure signal is what makes the
|
||||
difference visible while you are still authoring the plan.
|
||||
|
||||
**The authoring test, applied to yourself:** *if this command were silently doing nothing, what
|
||||
in its output would tell me?* If you cannot answer, you do not yet have an acceptance command —
|
||||
you have a command. Fix the command, do not invent a statement for it.
|
||||
|
||||
## Rules
|
||||
|
||||
- **One statement per runnable command**, placed immediately after it. Within a task, each
|
||||
`<fails_when>` binds to the nearest preceding `<automated>`, and the first statement after a
|
||||
command is the binding one. Two commands need two statements.
|
||||
- **Name an observable signal**, not the word "failure". `non-zero exit`, `"0 passed" in the
|
||||
summary`, `the coverage line is absent`, `stderr contains "ECONNREFUSED"` are signals. *"the
|
||||
command fails"*, *"it doesn't work"*, *"an error occurs"* are restatements and will be flagged.
|
||||
- **Short is fine.** `non-zero exit` is complete. There is no minimum length and no required
|
||||
keyword.
|
||||
- **`TBD`, `TODO`, `N/A`, `none`, `unknown`, `?`, `-` are rejected outright** as whole values.
|
||||
A statement you cannot write is a command you should not ship.
|
||||
- **Any characters are safe.** `exit code > 0`, `stderr contains "FAIL" && exit != 0` are ordinary
|
||||
prose here — that is exactly why this is an element and not an attribute.
|
||||
- **The `MISSING — Wave 0 must create …` sentinel is exempt.** It is not a runnable command, so
|
||||
it has no failure mode to state. Do not attach a `<fails_when>` to one.
|
||||
|
||||
## Where the failing direction comes from
|
||||
|
||||
Prefer the signal the tool actually emits over one you imagine. When
|
||||
`prior_verify_commands` supplies a command a prior phase already proved, the failure signal that
|
||||
command produces is the one to state — you have seen its output. When you author a new command,
|
||||
name the signal from the tool's documented output shape, not from a guess about it.
|
||||
@@ -766,6 +766,31 @@ inherited paths: fix a mirror path, never inherit. Submodule files: check
|
||||
from within the submodule.
|
||||
</tracked_source_paths>
|
||||
|
||||
<failing_direction_contract>
|
||||
**Stated failing direction (#3172):** Every runnable `<automated>` verify command
|
||||
you write MUST be followed by a `<fails_when>` sibling naming what output
|
||||
constitutes failure — an exit code, a string in the output, a missing line. A
|
||||
command with no expressible failure mode is not an acceptance test.
|
||||
|
||||
```xml
|
||||
<verify>
|
||||
<automated>npm --prefix apps/api test -- auth.spec.ts</automated>
|
||||
<fails_when>non-zero exit, or "0 passed" in the summary line</fails_when>
|
||||
</verify>
|
||||
```
|
||||
|
||||
One statement per runnable command, placed immediately after it: within a task
|
||||
each `<fails_when>` binds to the nearest preceding `<automated>`, and the first
|
||||
statement after a command is the binding one. Name an OBSERVABLE signal, never
|
||||
the word "failure" — `non-zero exit` is complete, `the command fails` is a
|
||||
restatement. `TBD`/`TODO`/`N/A`/`none`/`unknown`/`?`/`-` are rejected outright as
|
||||
whole values. The `MISSING — Wave 0 …` sentinel is exempt: it is not runnable, so
|
||||
it has no failure mode to state. Ask yourself: if this command were silently
|
||||
doing nothing, what in its output would tell me? If you cannot answer, fix the
|
||||
command — do not invent a statement for it.
|
||||
Rules + worked examples: @gsd-core/references/planner-failing-direction.md
|
||||
</failing_direction_contract>
|
||||
|
||||
**Project instructions:** Read ./CLAUDE.md or ./.claude/CLAUDE.md if either exists — follow project-specific guidelines
|
||||
**Project skills:** Check .claude/skills/ or .agents/skills/ directory (if either exists) — read SKILL.md files, plans should account for project skill rules
|
||||
|
||||
@@ -991,13 +1016,17 @@ Display banner:
|
||||
◆ Spawning plan checker... (runs in a subagent — no output until it returns, ~1–5 min; expected, not a freeze)
|
||||
```
|
||||
|
||||
**Verify-command path probe (#2401).** Before spawning, run the deterministic resolvability
|
||||
probe and hand its JSON to the checker. It never executes command text and never prescribes a
|
||||
replacement path — it reports which `<automated>` targets resolve, which do not, and which it
|
||||
refused to guess at. Handing it over is what stops the checker hand-reasoning the filesystem.
|
||||
**Verify-command probes (#2401, #3172).** Before spawning, run both deterministic probes and
|
||||
hand their JSON to the checker. The first resolves each `<automated>` command's target; the
|
||||
second reports which runnable commands carry a `<fails_when>` statement naming their failure
|
||||
signal. Neither executes command text, and neither prescribes a replacement — the first reports
|
||||
which `<automated>` targets resolve, which do not, and which it refused to guess at; the second
|
||||
reports which commands state a failure signal and never authors one. Handing both over is what
|
||||
stops the checker hand-reasoning the filesystem or the plans.
|
||||
|
||||
```bash
|
||||
VERIFY_PATHS=$(gsd_run check verify-command-paths "${PHASE}" --raw)
|
||||
FAILING_DIRECTIONS=$(gsd_run check verify-failure-directions "${PHASE}" --raw)
|
||||
```
|
||||
|
||||
Checker prompt:
|
||||
@@ -1032,6 +1061,18 @@ replacement path.
|
||||
```
|
||||
</verify_command_path_probe>
|
||||
|
||||
<failing_direction_probe>
|
||||
**Deterministic failing-direction probe (#3172)** — already run; do NOT re-derive these verdicts
|
||||
by re-reading the plans yourself. Act on `severity` per check 8f: `blocker` → BLOCKER,
|
||||
`warning` → WARNING, `none` → silent. `status: sentinel` is a Wave-0 `MISSING` placeholder and is
|
||||
not a finding. A non-empty `readError` means the probe could not look — a WARNING, not a pass.
|
||||
Quote the command that has no stated failure mode; never author the statement for the planner.
|
||||
|
||||
```json
|
||||
{FAILING_DIRECTIONS}
|
||||
```
|
||||
</failing_direction_probe>
|
||||
|
||||
<review_incorporation_verification>
|
||||
**If Mode is reviews:** Read REVIEWS.md and verify each current actionable review finding is visible in executable PLAN.md content or explicitly deferred/rejected in the relevant PLAN.md. A finding remains actionable if it requires a concrete plan task, `<action>`, `<acceptance_criteria>`, `<verify>`, `must_haves`, threat-model item, stale-path correction, or execution contract change before /gsd:execute-phase runs.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user