* chore(#1892): delete orphan verify-phase workflow, migrate live gates to verifier reference * test(#1892): retarget structural suites from verify-phase.md to verifier-phase-gates.md * chore(#1892): reword retired-workflow mentions for removed-but-needed lint * test(#1892): correct stale surface labels in retargeted suites * docs(#1892): add verifier-phase-gates row to locale inventories * chore(#3421): backfill changeset pr number --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
@@ -36,7 +36,7 @@ Configuration options for `.planning/` directory behavior.
|
||||
| `git.quick_branch_template` | `null` | Optional branch template for quick-task runs |
|
||||
| `workflow.use_worktrees` | `true` | Whether executor agents run in isolated git worktrees. Set to `false` to disable worktrees — agents execute sequentially on the main working tree instead. Recommended for solo developers or when worktree merges cause issues. Note: if your branch is ahead of `origin/HEAD` (a diverged milestone or feature branch), GSD auto-degrades to sequential and prints a warning; set `worktree.baseRef:"head"` in `.claude/settings.local.json` to restore parallel execution. See the branch-divergence note below. |
|
||||
| `workflow.subagent_timeout` | `300000` | Timeout in milliseconds for parallel subagent tasks (e.g. codebase mapping). Increase for large codebases or slower models. Default: 300000 (5 minutes). |
|
||||
| `workflow.test_command` | `null` | Custom shell command run as the regression/test gate by verify-phase, execute-phase, audit-fix, and post-merge-gate. When unset, GSD auto-detects (Makefile / package.json / Cargo.toml / go.mod / pyproject.toml). Example: `npm test`. |
|
||||
| `workflow.test_command` | `null` | Custom shell command run as the regression/test gate by execute-phase, audit-fix, and post-merge-gate. When unset, GSD auto-detects (Makefile / package.json / Cargo.toml / go.mod / pyproject.toml). Example: `npm test`. |
|
||||
| `workflow.build_command` | `null` | Custom shell command run as the build gate by the post-merge gate. When unset, the build step is skipped/auto-detected. Example: `npm run build`. |
|
||||
| `workflow.inline_plan_threshold` | `2` | Plans with this many tasks or fewer execute inline (Pattern C) instead of spawning a subagent. Avoids ~14K token spawn overhead for small plans. Set to `0` to always spawn subagents. |
|
||||
| `manager.flags.discuss` | `""` | Flags passed to `/gsd:discuss-phase` when dispatched from manager (e.g. `"--auto --analyze"`) |
|
||||
@@ -266,7 +266,7 @@ Set via `workflow.*` namespace in config.json (e.g., `"workflow": { "research":
|
||||
| `workflow.skip_discuss` | boolean | `false` | `true`, `false` | Skip discuss phase entirely |
|
||||
| `workflow.use_worktrees` | boolean | `true` | `true`, `false` | Run executor agents in isolated git worktrees |
|
||||
| `workflow.subagent_timeout` | number | `300000` | Any positive integer (ms) | Timeout for parallel subagent tasks (default: 5 minutes) |
|
||||
| `workflow.test_command` | string\|null | `null` | Any shell command | Regression/test gate command run by verify-phase, execute-phase, audit-fix, and post-merge-gate. Unset → GSD auto-detects (Makefile / package.json / Cargo.toml / go.mod / pyproject.toml). |
|
||||
| `workflow.test_command` | string\|null | `null` | Any shell command | Regression/test gate command run by execute-phase, audit-fix, and post-merge-gate. Unset → GSD auto-detects (Makefile / package.json / Cargo.toml / go.mod / pyproject.toml). |
|
||||
| `workflow.build_command` | string\|null | `null` | Any shell command | Build gate command run by the post-merge gate. Unset → build step auto-detected/skipped. |
|
||||
| `workflow.mvp_mode` | boolean | `false` | `true`, `false` | Persist the MVP-mode flag in config so every phase defaults to MVP framing without requiring `--mvp` on the CLI. Resolved via the chain: `--mvp` CLI flag → ROADMAP.md `**Mode:** mvp` field → this config value → `false`. When `true`, the planner, executor, verifier, and discovery surfaces (progress, stats, graphify) all treat the phase as an MVP vertical slice (UI → API → DB) of one user-visible capability. |
|
||||
| `workflow.context_guard_mode` | string | `"warn"` | `"auto"`, `"warn"`, `"off"` | Context exhaustion guard mode for `execute-phase`. Before each wave, the orchestrator self-assesses context pressure using degradation signals from `context-budget.md`. `"warn"` (default): emit a warning and recommend `/gsd:pause-work` when POOR tier is detected. `"auto"`: automatically invoke `/gsd:pause-work` before the next wave when POOR tier is detected. `"off"`: disable the guard. The guard is heuristic — no programmatic context-% API exists. |
|
||||
|
||||
175
gsd-core/references/verifier-phase-gates.md
Normal file
175
gsd-core/references/verifier-phase-gates.md
Normal file
@@ -0,0 +1,175 @@
|
||||
# Verifier Phase Gates
|
||||
|
||||
> Loaded eagerly by `agents/gsd-verifier.md` (`<required_reading>`). Carries the three
|
||||
> verification-time gates that lived in the retired `verify-phase` workflow
|
||||
> (#1892 / epic #1891 F7): decision-coverage validation (#2492), the test-quality audit,
|
||||
> and infrastructure-phase human-verification scoping (#2504). Run each at its named
|
||||
> agent step; `gsd_run` is the launcher shim defined in the agent's own Step 1 block.
|
||||
|
||||
## verify_decisions — Decision Coverage Gate (run after Step 6, requirements coverage)
|
||||
|
||||
<step name="verify_decisions">
|
||||
**Decision coverage validation gate (issue #2492).**
|
||||
|
||||
After requirements coverage, also check that each trackable CONTEXT.md
|
||||
`<decisions>` entry shows up somewhere in the shipped artifacts (plans,
|
||||
SUMMARY.md, files modified by the phase, or recent commit subjects on the
|
||||
phase branch).
|
||||
|
||||
This gate is **non-blocking / warning only** by deliberate asymmetry with
|
||||
the plan-phase translation gate. The plan-phase gate already blocked at
|
||||
translation time, so by the time verification runs every decision has
|
||||
either been translated or explicitly deferred. This gate's job is to
|
||||
surface decisions that *were* translated but vanished during execution —
|
||||
that's a soft signal because "honors a decision" is a fuzzy substring
|
||||
heuristic, and we don't want a paraphrase miss to fail an otherwise good
|
||||
phase.
|
||||
|
||||
**Skip if** `workflow.context_coverage_gate` is explicitly set to `false`
|
||||
(absent key = enabled). Also skip cleanly when CONTEXT.md is missing or has
|
||||
no `<decisions>` block.
|
||||
|
||||
```bash
|
||||
GATE_CFG=$(gsd_run query config-get workflow.context_coverage_gate 2>/dev/null || echo "true")
|
||||
if [ "$GATE_CFG" != "false" ]; then
|
||||
CONTEXT_PATH=$(ls "${PHASE_DIR}"/*-CONTEXT.md 2>/dev/null | head -1) # #2962: not a for-glob (zsh aborts)
|
||||
DECISION_RESULT=$(gsd_run query check.decision-coverage-verify "${PHASE_DIR}" "${CONTEXT_PATH}")
|
||||
fi
|
||||
```
|
||||
|
||||
The handler returns JSON `{ skipped, blocking: false, total, honored,
|
||||
not_honored: [...], message }`.
|
||||
|
||||
**Reporting:** Append the handler's `message` (a `### Decision Coverage`
|
||||
section) to VERIFICATION.md regardless of outcome — even when all
|
||||
decisions are honored, recording the count helps reviewers spot drift over
|
||||
time. Set `decision_coverage` in the verification result to
|
||||
`{honored, total, not_honored: [...]}` so downstream tooling can read it.
|
||||
|
||||
**Status impact:** none. The decision gate does NOT influence the
|
||||
`gaps_found` / `human_needed` / `passed` decision tree in Step 9. Its
|
||||
findings are warnings the user reviews and may act on by re-opening the
|
||||
phase or by acknowledging the decision was abandoned intentionally.
|
||||
</step>
|
||||
|
||||
## audit_test_quality (run after Step 7b, alongside anti-patterns)
|
||||
|
||||
<step name="audit_test_quality">
|
||||
**Verify that tests PROVE what they claim to prove.**
|
||||
|
||||
This step catches test-level deceptions that pass all prior checks: files exist, are substantive, are wired, and tests pass — but the tests don't actually validate the requirement.
|
||||
|
||||
**1. Identify requirement-linked test files**
|
||||
|
||||
From PLAN and SUMMARY files, map each requirement to the test files that are supposed to prove it.
|
||||
|
||||
**2. Disabled test scan**
|
||||
|
||||
For ALL test files linked to requirements, search for disabled/skipped patterns:
|
||||
|
||||
```bash
|
||||
grep -rn -E "it\.skip|describe\.skip|test\.skip|xit\(|xdescribe\(|xtest\(|@pytest\.mark\.skip|@unittest\.skip|#\[ignore\]|\.pending|it\.todo|test\.todo" "$TEST_FILE"
|
||||
```
|
||||
|
||||
**Rule:** A disabled test linked to a requirement = requirement NOT tested.
|
||||
- 🛑 BLOCKER if the disabled test is the only test proving that requirement
|
||||
- ⚠️ WARNING if other active tests also cover the requirement
|
||||
|
||||
**3. Circular test detection**
|
||||
|
||||
Search for scripts/utilities that generate expected values by running the system under test:
|
||||
|
||||
```bash
|
||||
grep -rn -E "writeFileSync|writeFile|fs\.write|open\(.*w\)" "$TEST_DIRS"
|
||||
```
|
||||
|
||||
For each match, check if it also imports the system/service/module being tested. If a script both imports the system-under-test AND writes expected output values → CIRCULAR.
|
||||
|
||||
**Circular test indicators:**
|
||||
- Script imports a service AND writes to fixture files
|
||||
- Expected values have comments like "computed from engine", "captured from baseline"
|
||||
- Script filename contains "capture", "baseline", "generate", "snapshot" in test context
|
||||
- Expected values were added in the same commit as the test assertions
|
||||
|
||||
**Rule:** A test comparing system output against values generated by the same system is circular. It proves consistency, not correctness.
|
||||
|
||||
**4. Expected value provenance** (for comparison/parity/migration requirements)
|
||||
|
||||
When a requirement demands comparison with an external source ("identical to X", "matches Y", "same output as Z"):
|
||||
|
||||
- Is the external source actually invoked or referenced in the test pipeline?
|
||||
- Do fixture files contain data sourced from the external system?
|
||||
- Or do all expected values come from the new system itself or from mathematical formulas?
|
||||
|
||||
**Provenance classification:**
|
||||
- VALID: Expected value from external/legacy system output, manual capture, or independent oracle
|
||||
- PARTIAL: Expected value from mathematical derivation (proves formula, not system match)
|
||||
- CIRCULAR: Expected value from the system being tested
|
||||
- UNKNOWN: No provenance information — treat as SUSPECT
|
||||
|
||||
**5. Assertion strength**
|
||||
|
||||
For each test linked to a requirement, classify the strongest assertion:
|
||||
|
||||
| Level | Examples | Proves |
|
||||
|-------|---------|--------|
|
||||
| Existence | `toBeDefined()`, `!= null` | Something returned |
|
||||
| Type | `typeof x === 'number'` | Correct shape |
|
||||
| Status | `code === 200` | No error |
|
||||
| Value | `toEqual(expected)`, `toBeCloseTo(x)` | Specific value |
|
||||
| Behavioral | Multi-step workflow assertions | End-to-end correctness |
|
||||
|
||||
If a requirement demands value-level or behavioral-level proof and the test only has existence/type/status assertions → INSUFFICIENT.
|
||||
|
||||
**6. Coverage quantity**
|
||||
|
||||
If a requirement specifies a quantity of test cases (e.g., "30 calculations"), check if the actual number of active (non-skipped) test cases meets the requirement.
|
||||
|
||||
**Reporting — add to VERIFICATION.md:**
|
||||
|
||||
```markdown
|
||||
### Test Quality Audit
|
||||
|
||||
| Test File | Linked Req | Active | Skipped | Circular | Assertion Level | Verdict |
|
||||
|-----------|-----------|--------|---------|----------|-----------------|---------|
|
||||
|
||||
**Disabled tests on requirements:** {N} → {BLOCKER if any req has ONLY disabled tests}
|
||||
**Circular patterns detected:** {N} → {BLOCKER if any}
|
||||
**Insufficient assertions:** {N} → {WARNING}
|
||||
```
|
||||
|
||||
**Impact on status:** Any BLOCKER from test quality audit → overall status = `gaps_found` (Step 9 rule 1), regardless of other checks passing.
|
||||
</step>
|
||||
|
||||
## identify_human_verification — infrastructure/foundation scoping (apply at Step 8)
|
||||
|
||||
**First: determine if this is an infrastructure/foundation phase.**
|
||||
|
||||
Infrastructure and foundation phases — code foundations, database schema, internal APIs, data models, build tooling, CI/CD, internal service integrations — have no user-facing elements by definition. For these phases:
|
||||
|
||||
- Do NOT invent artificial manual steps (e.g., "manually run git commits", "manually invoke methods", "manually check database state").
|
||||
- Mark human verification as **N/A** with rationale: "Infrastructure/foundation phase — no user-facing elements to test manually."
|
||||
- Set `human_verification: []` and do **not** produce a `human_needed` status solely due to lack of user-facing features.
|
||||
- Only add human verification items if the phase goal or success criteria explicitly describe something a user would interact with (UI, CLI command output visible to end users, external service UX).
|
||||
- **Exception — behavior-unverified truths still count.** A truth marked ⚠️ PRESENT_BEHAVIOR_UNVERIFIED (a state transition or a cancellation/cleanup/ordering invariant with no test exercising it) is a behavioral-evidence gap, not an artificial user-facing step. Record it in `behavior_unverified_items` and emit a human-verification item for it **even on an infrastructure/foundation phase** — these invariants are exactly where infra phases hide runtime state leaks. Such a truth drives `human_needed`; the auto-pass-UAT shortcut applies only to the absence of user-facing UX, never to a behavior-unverified invariant.
|
||||
|
||||
**How to determine if a phase is infrastructure/foundation:**
|
||||
- Phase goal or name contains: "foundation", "infrastructure", "schema", "database", "internal API", "data model", "scaffolding", "pipeline", "tooling", "CI", "migrations", "service layer", "backend", "core library"
|
||||
- Phase success criteria describe only technical artifacts (files exist, tests pass, schema is valid) with no user interaction required
|
||||
- There is no UI, CLI output visible to end users, or real-time behavior to observe
|
||||
|
||||
**If the phase IS infrastructure/foundation:** auto-pass UAT — skip the human verification items list entirely, **except any ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truth (see exception above), which still emits a human-verification item and drives `human_needed`.** Log:
|
||||
|
||||
```markdown
|
||||
## Human Verification
|
||||
|
||||
N/A — Infrastructure/foundation phase with no user-facing elements.
|
||||
All acceptance criteria are verifiable programmatically.
|
||||
```
|
||||
|
||||
**If the phase IS user-facing:** only flag items that genuinely require a human — per the Step 8 always/uncertain lists already in the agent. Do not invent steps.
|
||||
|
||||
## Lazy references
|
||||
|
||||
- **Per-stack verification patterns:** before Step 4 (artifact verification) on an unfamiliar stack, Read `~/.claude/gsd-core/references/verification-patterns.md` — the grep catalog for React/Next.js components, API routes, database schema, and the universal stub patterns. Read it lazily (only the sections for the stack under verification); it is too large to load wholesale on every run.
|
||||
- **Canonical report shape:** the emitted VERIFICATION.md follows `@~/.claude/gsd-core/templates/verification-report.md` — the template whose Guidelines and row shapes `src/uat.cts` treats as canonical when consuming verification output.
|
||||
Reference in New Issue
Block a user