enhance(#3909): a probe that could not run no longer asserts a verdict (#3944)

* test(#3909): failing-first suite for the fabricated probe fallbacks

Binds the four fabrication sites found by executing the surfaces (ADR-3889
failure class (c)), each with a positive control so an over-firing fix goes red:

- the blocking api-coverage.verify-pre gate certifying "no external-API
  integration" from a zero-byte phase scope
- the assumption-delta query route scanning an unresolvable phase section as
  the empty string and reporting it as an examined negative
- both capability fragments' probe fallbacks, which append a fabricated
  verdict rather than replacing, and fire on the legitimate exit-1 negative

Verification runs on the remote runner.

Refs #3909

* enhance(#3909): a probe that could not run no longer asserts a verdict

ADR-3889 Phase 5. Four sites turned a failed or unexamined probe into a
confident negative; each now reports what it could not establish.

- check api-coverage.verify-pre: a phase with no plan body and no roadmap
  section ran detection over zero bytes and PASSED the blocking seal gate,
  certifying "no external-API integration" from input it never read. It now
  holds with scope_unavailable. The discriminator is bytes examined, never
  signals found, so a phase whose plans are real and simply carry no API
  vocabulary passes exactly as before.
- query assumption-delta scan: an unresolvable phase section was scanned as
  the empty string and reported as an examined negative. It now returns
  {skipped, reason: phase_unresolved}, still at exit 0 — an ADR-2980 degraded
  result in the payload, leaving the gsd-tools exit projection to P8.
- both capability fragments: `|| echo '{"detected":false}'` appended rather
  than replaced, and fired on the legitimate exit-1 negative, so a correct
  answer and an honest skip both arrived as two concatenated objects. They now
  keep the probe's own payload and manufacture only an explicit
  probe_unavailable skip when the probe produced nothing at all.

Every registered outcome is more restrictive on a blocking gate, so this can
turn a false green red and never a red green.

Docs: FEATURES 156, CONFIGURATION (both keys), references/api-coverage.md
seal-time outcome table, and a new how-to for the reason-code vocabulary.

Verification runs on the remote runner.

Closes #3909

* test(#3909): correct the stale unknown-phase assertion

`unknown phase → detected:false, no throw (graceful)` scanned phase 999
against a two-phase roadmap and asserted `detected === false`. That pinned
the fabrication as intended behavior: the phase does not exist, so the
detector was handed the empty string and its "no core assumption changed"
answer described nothing that was ever read.

It now asserts the skipped-with-reason shape. The graceful-degradation
contract the test was actually protecting — the query succeeds and does not
throw on an unknown phase — is unchanged.

Found by code review, not by the author.

Refs #3909

* docs(#3909): author the FEATURES entry in its generator source

`docs/FEATURES.md` is generated by `scripts/gen-features.cjs` from the
per-feature fragments in `docs/features/`. The API-coverage entry was edited
in the generated file, so the next regeneration silently dropped it.

The text now lives in `docs/features/api-coverage-gate.md` and
`docs/FEATURES.md` is regenerated from it, leaving the shipped file
byte-identical and its content actually derivable.

Caught by `lint:generated-sync`.

Refs #3909

* test(#3909): bind the skip to "not found", and pin the discriminator

The first verification run went red on one case, and the case was wrong
rather than the code.

`getRoadmapPhaseWithFallback` returns `null` for an unknown phase and for a
missing ROADMAP.md, but for a section whose body is whitespace-only it returns
the heading line alone — which is not empty. So a body-less section WAS found,
and reporting `detected:false` over its heading is a real negative, not a
fabrication. The test had assumed the resolver yielded `''` there.

Correcting the test rather than the resolver keeps `skipped` bound to the
distinction the issue asks for — found versus not found — and avoids diverging
`assumption-delta scan` from `roadmap.get-phase`, which the fragment documents
as sharing one resolver.

Also adds the seeded property the test matrix had promised: for any plan body,
the scope read back is whitespace-only exactly when the body was. That pins the
gate's discriminator to bytes examined, so it cannot quietly become "no signals
found", across unicode whitespace and CRLF.

`docs/INVENTORY.md` picks up the reference doc's new seal-time outcome table —
surfaced by the co-change gate, not by a lint failure.

Refs #3909

* chore(#3909): backfill the changeset PR number

Refs #3909

---------

Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
2026-08-27 15:50:12 -04:00
committed by GitHub
parent 9410f7e6e6
commit 03b7125293
16 changed files with 755 additions and 14 deletions

View File

@@ -395,7 +395,7 @@ All workflow toggles follow the **absent = enabled** pattern. If a key is missin
| `workflow.nyquist_validation` | boolean | `true` | Test coverage mapping during plan-phase research |
| `workflow.ui_phase` | boolean | `true` | Generate UI design contracts for frontend phases |
| `workflow.ui_safety_gate` | boolean | `true` | Prompt to run /gsd-ui-phase for frontend phases during plan-phase |
| `workflow.assumption_delta` | boolean | `true` | Advisory architecture checkpoint during planning. When a phase makes something **plural, optional, or chosen** that used to be **singular, required, or derived** (e.g. a second auth method, a required field becoming optional, a constant becoming a parameter), the planner is prompted to re-ask whether the primary key / identity model still names the right thing (promote the new general representation vs. add it alongside). Non-blocking; fires only on a detected signal. Bare "or" is intentionally excluded (prose false-positives). Inspect a phase with `gsd_run query assumption-delta scan <phase>`. Added in #1561 |
| `workflow.assumption_delta` | boolean | `true` | Advisory architecture checkpoint during planning. When a phase makes something **plural, optional, or chosen** that used to be **singular, required, or derived** (e.g. a second auth method, a required field becoming optional, a constant becoming a parameter), the planner is prompted to re-ask whether the primary key / identity model still names the right thing (promote the new general representation vs. add it alongside). Non-blocking; fires only on a detected signal. Bare "or" is intentionally excluded (prose false-positives). Inspect a phase with `gsd_run query assumption-delta scan <phase>`. Added in #1561. A phase section that cannot be resolved returns `{"skipped":true,"reason":"phase_unresolved"}` rather than a fabricated `detected:false` (#3909) |
| `workflow.ui_review` | boolean | `true` | Run visual quality audit (`/gsd-ui-review`) after phase execution in autonomous mode. When `false`, the UI audit step is skipped. |
| `workflow.live_dom_uat` | boolean | `false` | **Default-off.** Enable live-DOM verification (#2856). When `true`, a `gsd-dom-verifier` step runs after each execution wave and writes `{phase}-DOM-VERIFY.md`, and the orchestrator's automated UI verification will additionally consider `mcp__chrome-devtools__*` / `mcp__claude-in-chrome__*` when present. Browser reach is confined to `gsd-dom-verifier` — `gsd-executor`'s tool surface is unchanged in every configuration. Presence of a browser MCP server is **not** sufficient on its own: a server configured for unrelated work is never driven unless this key is on. The pre-existing `mcp__playwright__*` path is unaffected by this key. Note `chrome-devtools-mcp` holds an exclusive browser-profile lock, so concurrent waves need `--isolated` on **your** MCP server registration — GSD cannot pass it. See [Enable live-DOM verification](how-to/enable-live-dom-verification.md). |
| `workflow.node_repair` | boolean | `true` | Autonomous task repair on verification failure |
@@ -428,7 +428,7 @@ All workflow toggles follow the **absent = enabled** pattern. If a key is missin
| `workflow.cross_ai_timeout` | number | `300` | Timeout in seconds for cross-AI execution commands. Prevents runaway external processes. Added in v1.36 |
| `workflow.test_gate_timeout` | number | `600` | Wall-clock timeout (seconds) for a verification test gate; a watch-mode runner (vitest/jest) that never exits is aborted after this budget instead of hanging the orchestrator (#1857) |
| `workflow.ai_integration_phase` | boolean | `true` | Enable the `/gsd-ai-integration-phase` command. When `false`, the command exits with a configuration gate message |
| `workflow.api_coverage_gate` | boolean | `true` | Require an explicit API-coverage decision before a phase that integrates an external API/SDK/service can seal. At `plan:pre` the planner is prompted to produce a `COVERAGE.md` matrix (full coverage by default, every opt-out reasoned); at `verify:pre` a blocking gate fails the seal unless the matrix is complete. Independent of `ai_integration_phase` (#1562) |
| `workflow.api_coverage_gate` | boolean | `true` | Require an explicit API-coverage decision before a phase that integrates an external API/SDK/service can seal. At `plan:pre` the planner is prompted to produce a `COVERAGE.md` matrix (full coverage by default, every opt-out reasoned); at `verify:pre` a blocking gate fails the seal unless the matrix is complete. Independent of `ai_integration_phase` (#1562). A phase whose scope cannot be established at all (no plan body and no roadmap section) is held rather than passed, reporting `scope_unavailable` — see [Resolve a skipped capability probe](how-to/resolve-a-skipped-capability-probe.md) (#3909) |
| `workflow.auto_prune_state` | boolean | `false` | When `true`, automatically prune stale entries from STATE.md at phase boundaries instead of prompting |
| `workflow.pattern_mapper` | boolean | `true` | Run the `gsd-pattern-mapper` agent between research and planning to map new files to existing codebase analogs |
| `workflow.subagent_timeout` | number | `300000` | Timeout in milliseconds for parallel subagent tasks (e.g. codebase mapping). Increase for large codebases or slower models. Default: 300000 (5 minutes) |