* test(#3909): failing-first suite for the fabricated probe fallbacks Binds the four fabrication sites found by executing the surfaces (ADR-3889 failure class (c)), each with a positive control so an over-firing fix goes red: - the blocking api-coverage.verify-pre gate certifying "no external-API integration" from a zero-byte phase scope - the assumption-delta query route scanning an unresolvable phase section as the empty string and reporting it as an examined negative - both capability fragments' probe fallbacks, which append a fabricated verdict rather than replacing, and fire on the legitimate exit-1 negative Verification runs on the remote runner. Refs #3909 * enhance(#3909): a probe that could not run no longer asserts a verdict ADR-3889 Phase 5. Four sites turned a failed or unexamined probe into a confident negative; each now reports what it could not establish. - check api-coverage.verify-pre: a phase with no plan body and no roadmap section ran detection over zero bytes and PASSED the blocking seal gate, certifying "no external-API integration" from input it never read. It now holds with scope_unavailable. The discriminator is bytes examined, never signals found, so a phase whose plans are real and simply carry no API vocabulary passes exactly as before. - query assumption-delta scan: an unresolvable phase section was scanned as the empty string and reported as an examined negative. It now returns {skipped, reason: phase_unresolved}, still at exit 0 — an ADR-2980 degraded result in the payload, leaving the gsd-tools exit projection to P8. - both capability fragments: `|| echo '{"detected":false}'` appended rather than replaced, and fired on the legitimate exit-1 negative, so a correct answer and an honest skip both arrived as two concatenated objects. They now keep the probe's own payload and manufacture only an explicit probe_unavailable skip when the probe produced nothing at all. Every registered outcome is more restrictive on a blocking gate, so this can turn a false green red and never a red green. Docs: FEATURES 156, CONFIGURATION (both keys), references/api-coverage.md seal-time outcome table, and a new how-to for the reason-code vocabulary. Verification runs on the remote runner. Closes #3909 * test(#3909): correct the stale unknown-phase assertion `unknown phase → detected:false, no throw (graceful)` scanned phase 999 against a two-phase roadmap and asserted `detected === false`. That pinned the fabrication as intended behavior: the phase does not exist, so the detector was handed the empty string and its "no core assumption changed" answer described nothing that was ever read. It now asserts the skipped-with-reason shape. The graceful-degradation contract the test was actually protecting — the query succeeds and does not throw on an unknown phase — is unchanged. Found by code review, not by the author. Refs #3909 * docs(#3909): author the FEATURES entry in its generator source `docs/FEATURES.md` is generated by `scripts/gen-features.cjs` from the per-feature fragments in `docs/features/`. The API-coverage entry was edited in the generated file, so the next regeneration silently dropped it. The text now lives in `docs/features/api-coverage-gate.md` and `docs/FEATURES.md` is regenerated from it, leaving the shipped file byte-identical and its content actually derivable. Caught by `lint:generated-sync`. Refs #3909 * test(#3909): bind the skip to "not found", and pin the discriminator The first verification run went red on one case, and the case was wrong rather than the code. `getRoadmapPhaseWithFallback` returns `null` for an unknown phase and for a missing ROADMAP.md, but for a section whose body is whitespace-only it returns the heading line alone — which is not empty. So a body-less section WAS found, and reporting `detected:false` over its heading is a real negative, not a fabrication. The test had assumed the resolver yielded `''` there. Correcting the test rather than the resolver keeps `skipped` bound to the distinction the issue asks for — found versus not found — and avoids diverging `assumption-delta scan` from `roadmap.get-phase`, which the fragment documents as sharing one resolver. Also adds the seeded property the test matrix had promised: for any plan body, the scope read back is whitespace-only exactly when the body was. That pins the gate's discriminator to bytes examined, so it cannot quietly become "no signals found", across unicode whitespace and CRLF. `docs/INVENTORY.md` picks up the reference doc's new seal-time outcome table — surfaced by the co-change gate, not by a lint failure. Refs #3909 * chore(#3909): backfill the changeset PR number Refs #3909 --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
@@ -395,7 +395,7 @@ All workflow toggles follow the **absent = enabled** pattern. If a key is missin
|
||||
| `workflow.nyquist_validation` | boolean | `true` | Test coverage mapping during plan-phase research |
|
||||
| `workflow.ui_phase` | boolean | `true` | Generate UI design contracts for frontend phases |
|
||||
| `workflow.ui_safety_gate` | boolean | `true` | Prompt to run /gsd-ui-phase for frontend phases during plan-phase |
|
||||
| `workflow.assumption_delta` | boolean | `true` | Advisory architecture checkpoint during planning. When a phase makes something **plural, optional, or chosen** that used to be **singular, required, or derived** (e.g. a second auth method, a required field becoming optional, a constant becoming a parameter), the planner is prompted to re-ask whether the primary key / identity model still names the right thing (promote the new general representation vs. add it alongside). Non-blocking; fires only on a detected signal. Bare "or" is intentionally excluded (prose false-positives). Inspect a phase with `gsd_run query assumption-delta scan <phase>`. Added in #1561 |
|
||||
| `workflow.assumption_delta` | boolean | `true` | Advisory architecture checkpoint during planning. When a phase makes something **plural, optional, or chosen** that used to be **singular, required, or derived** (e.g. a second auth method, a required field becoming optional, a constant becoming a parameter), the planner is prompted to re-ask whether the primary key / identity model still names the right thing (promote the new general representation vs. add it alongside). Non-blocking; fires only on a detected signal. Bare "or" is intentionally excluded (prose false-positives). Inspect a phase with `gsd_run query assumption-delta scan <phase>`. Added in #1561. A phase section that cannot be resolved returns `{"skipped":true,"reason":"phase_unresolved"}` rather than a fabricated `detected:false` (#3909) |
|
||||
| `workflow.ui_review` | boolean | `true` | Run visual quality audit (`/gsd-ui-review`) after phase execution in autonomous mode. When `false`, the UI audit step is skipped. |
|
||||
| `workflow.live_dom_uat` | boolean | `false` | **Default-off.** Enable live-DOM verification (#2856). When `true`, a `gsd-dom-verifier` step runs after each execution wave and writes `{phase}-DOM-VERIFY.md`, and the orchestrator's automated UI verification will additionally consider `mcp__chrome-devtools__*` / `mcp__claude-in-chrome__*` when present. Browser reach is confined to `gsd-dom-verifier` — `gsd-executor`'s tool surface is unchanged in every configuration. Presence of a browser MCP server is **not** sufficient on its own: a server configured for unrelated work is never driven unless this key is on. The pre-existing `mcp__playwright__*` path is unaffected by this key. Note `chrome-devtools-mcp` holds an exclusive browser-profile lock, so concurrent waves need `--isolated` on **your** MCP server registration — GSD cannot pass it. See [Enable live-DOM verification](how-to/enable-live-dom-verification.md). |
|
||||
| `workflow.node_repair` | boolean | `true` | Autonomous task repair on verification failure |
|
||||
@@ -428,7 +428,7 @@ All workflow toggles follow the **absent = enabled** pattern. If a key is missin
|
||||
| `workflow.cross_ai_timeout` | number | `300` | Timeout in seconds for cross-AI execution commands. Prevents runaway external processes. Added in v1.36 |
|
||||
| `workflow.test_gate_timeout` | number | `600` | Wall-clock timeout (seconds) for a verification test gate; a watch-mode runner (vitest/jest) that never exits is aborted after this budget instead of hanging the orchestrator (#1857) |
|
||||
| `workflow.ai_integration_phase` | boolean | `true` | Enable the `/gsd-ai-integration-phase` command. When `false`, the command exits with a configuration gate message |
|
||||
| `workflow.api_coverage_gate` | boolean | `true` | Require an explicit API-coverage decision before a phase that integrates an external API/SDK/service can seal. At `plan:pre` the planner is prompted to produce a `COVERAGE.md` matrix (full coverage by default, every opt-out reasoned); at `verify:pre` a blocking gate fails the seal unless the matrix is complete. Independent of `ai_integration_phase` (#1562) |
|
||||
| `workflow.api_coverage_gate` | boolean | `true` | Require an explicit API-coverage decision before a phase that integrates an external API/SDK/service can seal. At `plan:pre` the planner is prompted to produce a `COVERAGE.md` matrix (full coverage by default, every opt-out reasoned); at `verify:pre` a blocking gate fails the seal unless the matrix is complete. Independent of `ai_integration_phase` (#1562). A phase whose scope cannot be established at all (no plan body and no roadmap section) is held rather than passed, reporting `scope_unavailable` — see [Resolve a skipped capability probe](how-to/resolve-a-skipped-capability-probe.md) (#3909) |
|
||||
| `workflow.auto_prune_state` | boolean | `false` | When `true`, automatically prune stale entries from STATE.md at phase boundaries instead of prompting |
|
||||
| `workflow.pattern_mapper` | boolean | `true` | Run the `gsd-pattern-mapper` agent between research and planning to map new files to existing codebase analogs |
|
||||
| `workflow.subagent_timeout` | number | `300000` | Timeout in milliseconds for parallel subagent tasks (e.g. codebase mapping). Increase for large codebases or slower models. Default: 300000 (5 minutes) |
|
||||
|
||||
@@ -3412,6 +3412,12 @@ The load-bearing wire is the `plan-phase` lift into `must_haves.prohibitions`, s
|
||||
|
||||
**Purpose:** A phase that integrates an external API, SDK, or service can no longer seal verification without a decided coverage matrix (#1562).
|
||||
|
||||
**Behavior:** At seal time the gate reads the phase scope — the plan bodies, falling back to this phase's ROADMAP section — and runs the deterministic detector over it. An integration signal without a `COVERAGE.md` matrix blocks the seal; no signal passes.
|
||||
|
||||
**Unestablished scope is not a negative verdict (#3909).** A phase with no plan body *and* no roadmap section gives the detector nothing to examine. The gate used to run detection over zero bytes and pass, certifying "no external-API integration" from a probe that never looked. It now holds the seal instead, reporting `scope_unavailable: true`. A phase whose plans are real and simply contain no API vocabulary is unaffected — the discriminator is *bytes examined*, never *signals found*.
|
||||
|
||||
**Breaking change:** a phase that previously sealed because its detector could not establish a scope is now correctly held. Add the phase plan, or record a reasoned `No external API integration: <reason>` declaration in `COVERAGE.md`. See [Resolve a skipped capability probe](how-to/resolve-a-skipped-capability-probe.md).
|
||||
|
||||
---
|
||||
|
||||
### 157. State Rebuild & Configurable Graph Path
|
||||
|
||||
@@ -375,7 +375,7 @@ Full roster at `gsd-core/references/*.md`. References are shared knowledge docum
|
||||
| `autonomous-ui-design-contract.md` | Autonomous-mode step 3a.5 — resolve whether a frontend phase needs a UI-SPEC.md and generate one through active `plan:pre` hooks; always non-blocking. |
|
||||
| `ios-scaffold.md` | iOS application scaffolding patterns. |
|
||||
| `ai-evals.md` | AI evaluation design reference for `/gsd-ai-integration-phase`. |
|
||||
| `api-coverage.md` | API-coverage gate reference (full-coverage-by-default) for the `ai-integration` capability's `verify:pre` blocking gate (#1562) — matrix format, trigger, tuning, detector CLI. |
|
||||
| `api-coverage.md` | API-coverage gate reference (full-coverage-by-default) for the `ai-integration` capability's `verify:pre` blocking gate (#1562) — matrix format, trigger, tuning, detector CLI, and the seal-time outcome table naming every pass/block arm including `scope_unavailable` (#3909). |
|
||||
| `ai-frameworks.md` | AI framework decision-matrix reference for `gsd-framework-selector`. |
|
||||
| `executor-examples.md` | Worked examples for the gsd-executor agent. |
|
||||
| `doc-conflict-engine.md` | Shared conflict-detection contract for ingest/import workflows. |
|
||||
|
||||
@@ -30,6 +30,7 @@ Language versions: [English](README.md) · [Português (pt-BR)](pt-BR/README.md)
|
||||
- [State a failing direction](how-to/state-a-failing-direction.md) — say what output constitutes failure for an `<automated>` verify command, and migrate a phase planned before the rule
|
||||
- [Resolve a contract-drift finding](how-to/resolve-contract-drift-findings.md) — bring an agent's completion contract, read-tag gate, or deleted-file test reference back into agreement with the registry
|
||||
- [Resolve unreachable-guard findings](how-to/resolve-unreachable-guard-findings.md) — fix shell guards whose fallback arm cannot run, and tell "nothing to report" apart from "could not look"
|
||||
- [Resolve a skipped capability probe](how-to/resolve-a-skipped-capability-probe.md) — act on a coverage gate that held your phase for an unestablished scope, or a planning checkpoint that reported `skipped` instead of a verdict
|
||||
- [Diagnose which gsd-tools is running](how-to/diagnose-a-foreign-gsd-tools.md) — tell this package's tool apart from the predecessor's colliding binary and from a gsd-core too old to identify itself
|
||||
- [Resolve an ESLint glob-coverage finding](how-to/resolve-eslint-coverage-findings.md) — bring a source file that matches no lint rule under coverage, or record a reasoned exemption
|
||||
- [Read the statusline freshness marker](how-to/read-the-statusline-freshness-marker.md) — turn on `state ~N commits back`, and tell "STATE.md is fresh" apart from "freshness could not be established"
|
||||
|
||||
@@ -7,3 +7,9 @@ group: v1.7.0 Features
|
||||
**Command:** `/gsd-verify-work`
|
||||
|
||||
**Purpose:** A phase that integrates an external API, SDK, or service can no longer seal verification without a decided coverage matrix (#1562).
|
||||
|
||||
**Behavior:** At seal time the gate reads the phase scope — the plan bodies, falling back to this phase's ROADMAP section — and runs the deterministic detector over it. An integration signal without a `COVERAGE.md` matrix blocks the seal; no signal passes.
|
||||
|
||||
**Unestablished scope is not a negative verdict (#3909).** A phase with no plan body *and* no roadmap section gives the detector nothing to examine. The gate used to run detection over zero bytes and pass, certifying "no external-API integration" from a probe that never looked. It now holds the seal instead, reporting `scope_unavailable: true`. A phase whose plans are real and simply contain no API vocabulary is unaffected — the discriminator is *bytes examined*, never *signals found*.
|
||||
|
||||
**Breaking change:** a phase that previously sealed because its detector could not establish a scope is now correctly held. Add the phase plan, or record a reasoned `No external API integration: <reason>` declaration in `COVERAGE.md`. See [Resolve a skipped capability probe](how-to/resolve-a-skipped-capability-probe.md).
|
||||
|
||||
116
docs/how-to/resolve-a-skipped-capability-probe.md
Normal file
116
docs/how-to/resolve-a-skipped-capability-probe.md
Normal file
@@ -0,0 +1,116 @@
|
||||
# Resolve a skipped capability probe
|
||||
|
||||
A phase-scope probe — the API-coverage detector or the assumption-delta
|
||||
detector — needs real text to examine before it can assert a verdict. When it
|
||||
gets none, it says so instead of guessing. This page covers reading that
|
||||
signal and clearing it.
|
||||
|
||||
## What you saw
|
||||
|
||||
One of two things, depending on which surface you hit:
|
||||
|
||||
- **The seal gate held your phase.** `verify:pre` reported a block with
|
||||
`scope_unavailable: true` instead of the usual "no external-API integration
|
||||
detected" pass.
|
||||
- **A planning checkpoint reported `skipped` instead of a verdict.** The
|
||||
`assumption-delta` or `api-coverage` plan-time checkpoint printed
|
||||
`{"skipped":true,"reason":"..."}` and produced no decision either way.
|
||||
|
||||
Both are the same underlying fix (#3909): a probe that never examined real
|
||||
input used to fabricate `detected:false`, which reads as "nothing here" when
|
||||
the true answer is "nobody looked." Now it says which one happened.
|
||||
|
||||
## The reason-code table
|
||||
|
||||
| `reason` | What it means | What actually happened | Remedy |
|
||||
|---|---|---|---|
|
||||
| `scope_unavailable` | The seal-time gate found no phase scope at all | No plan body (`.planning/phases/<N>/*-PLAN.md`) *and* no ROADMAP section for the phase | Add the plan or roadmap section, or write a reasoned `No external API integration: <reason>` declaration to `COVERAGE.md` |
|
||||
| `phase_unresolved` | The `assumption-delta scan` query could not resolve a phase section to scan | No `ROADMAP.md`, an unknown phase number, or a phase section with no body | If you expected a real scan, fix the phase reference or roadmap section; otherwise no action — the checkpoint correctly stays silent |
|
||||
| `probe_unavailable` | The detector process itself produced no output | The probe crashed, was not found, or its stdout was empty for a reason unrelated to input content | Check that `gsd-core/bin/lib/api-coverage.cjs` or `gsd-core/bin/lib/assumption-delta.cjs` runs standalone; re-run the checkpoint once the probe itself is healthy |
|
||||
| `no_input` | The detector ran but stdin was empty or whitespace-only | The phase scope resolved to nothing (empty plan body and empty roadmap section) | Same as `scope_unavailable` — give the detector something to read |
|
||||
| `stdin_error` | The detector could not read stdin at all | A pipe/read failure upstream of the detector, not an empty-input case | Re-run; if it recurs, the caller constructing `$SCOPE` is the thing to fix, not the detector |
|
||||
|
||||
## The seal gate held my phase
|
||||
|
||||
1. Confirm the reason directly:
|
||||
|
||||
```bash
|
||||
gsd_run check api-coverage.verify-pre <phase> --raw
|
||||
```
|
||||
|
||||
Look for `"scope_unavailable": true` in the output. That confirms this is
|
||||
the fail-closed arm, not a real "integration detected" block.
|
||||
|
||||
2. Check whether the phase actually has scope to read:
|
||||
|
||||
```bash
|
||||
ls .planning/phases/<N>/*-PLAN.md 2>/dev/null
|
||||
gsd_run query roadmap.get-phase <phase>
|
||||
```
|
||||
|
||||
If both come back empty, the gate is correct — there is genuinely nothing
|
||||
for the detector to examine.
|
||||
|
||||
3. Resolve it one of two ways:
|
||||
- **Add the missing scope.** Write the phase plan, or add the phase's
|
||||
section to `ROADMAP.md`, then re-run detection.
|
||||
- **Record the reasoned declaration anyway.** If the phase truly has no
|
||||
plan body worth writing (rare), put the human decision directly in
|
||||
`COVERAGE.md`:
|
||||
|
||||
```markdown
|
||||
No external API integration: <one-line reason>.
|
||||
```
|
||||
|
||||
This is the same reasoned overrule the gate already accepts when a
|
||||
detector *does* find (or falsely flags) a signal — see
|
||||
[`gsd-core/references/api-coverage.md`](../../gsd-core/references/api-coverage.md#declaring-no-external-api-integration-2365).
|
||||
|
||||
4. Re-run the gate to confirm it clears:
|
||||
|
||||
```bash
|
||||
gsd_run check api-coverage.verify-pre <phase> --raw
|
||||
```
|
||||
|
||||
**Turning the gate off does not answer the question.** Setting
|
||||
`workflow.api_coverage_gate: false` in `.planning/config.json` silences the
|
||||
hold, but the phase's API surface is still undecided — you have just stopped
|
||||
being told. Prefer resolving the scope or writing the declaration.
|
||||
|
||||
## A checkpoint reported skipped
|
||||
|
||||
The `assumption-delta` and `api-coverage` plan-time checkpoints are advisory:
|
||||
when they cannot resolve a phase section, they skip rather than fire, and that
|
||||
is correct, non-blocking behavior. You do not need to do anything except
|
||||
notice that the phase was not actually cleared by a real scan — a `skipped`
|
||||
payload carries no `detected` key and is not the same as "no signal found."
|
||||
If you expected a real scan and got a skip instead, treat it like the
|
||||
`phase_unresolved` row above: check that the phase reference and roadmap
|
||||
section actually exist.
|
||||
|
||||
## Why this is not just a stricter gate
|
||||
|
||||
A probe reporting `detected:false` from an input it never read is a false
|
||||
negative on a *blocking* gate — the one direction a gate must never fail
|
||||
silently. A false positive here costs one line in `COVERAGE.md`; a false
|
||||
negative lets a real external-API integration seal with an undecided surface,
|
||||
discovered later by a user who reasonably expected it to work. This change can
|
||||
turn a false green red. It can never turn a red green — every phase that
|
||||
passed because a signal was genuinely absent from scope the detector actually
|
||||
read still passes, unchanged.
|
||||
|
||||
## Telling "nothing to report" apart from "could not look"
|
||||
|
||||
This is the recurring distinction across this doc set (see also
|
||||
[Resolve unreachable-guard findings](resolve-unreachable-guard-findings.md) and
|
||||
[Consume the planning snapshot](consume-the-planning-snapshot.md)). A verdict
|
||||
of `detected: false` or `passed: true` means the detector examined real text
|
||||
and found nothing. A `skipped` payload or a `scope_unavailable` block means
|
||||
the detector examined nothing at all. Only the first is good news; the second
|
||||
is a request for more scope, not a clean bill of health.
|
||||
|
||||
## Related
|
||||
|
||||
- [`gsd-core/references/api-coverage.md`](../../gsd-core/references/api-coverage.md) — the full API-coverage gate reference, including the seal-time outcome table
|
||||
- [Resolve unreachable-guard findings](resolve-unreachable-guard-findings.md) — the same "nothing to report vs. could not look" distinction, one layer down
|
||||
- [Consume the planning snapshot](consume-the-planning-snapshot.md) — the `scope` field's version of this same rule
|
||||
Reference in New Issue
Block a user