From 03b712529333eda8a8e023be405374b9b2fd6426 Mon Sep 17 00:00:00 2001 From: Tom Boucher Date: Thu, 27 Aug 2026 15:50:12 -0400 Subject: [PATCH] enhance(#3909): a probe that could not run no longer asserts a verdict (#3944) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(#3909): failing-first suite for the fabricated probe fallbacks Binds the four fabrication sites found by executing the surfaces (ADR-3889 failure class (c)), each with a positive control so an over-firing fix goes red: - the blocking api-coverage.verify-pre gate certifying "no external-API integration" from a zero-byte phase scope - the assumption-delta query route scanning an unresolvable phase section as the empty string and reporting it as an examined negative - both capability fragments' probe fallbacks, which append a fabricated verdict rather than replacing, and fire on the legitimate exit-1 negative Verification runs on the remote runner. Refs #3909 * enhance(#3909): a probe that could not run no longer asserts a verdict ADR-3889 Phase 5. Four sites turned a failed or unexamined probe into a confident negative; each now reports what it could not establish. - check api-coverage.verify-pre: a phase with no plan body and no roadmap section ran detection over zero bytes and PASSED the blocking seal gate, certifying "no external-API integration" from input it never read. It now holds with scope_unavailable. The discriminator is bytes examined, never signals found, so a phase whose plans are real and simply carry no API vocabulary passes exactly as before. - query assumption-delta scan: an unresolvable phase section was scanned as the empty string and reported as an examined negative. It now returns {skipped, reason: phase_unresolved}, still at exit 0 — an ADR-2980 degraded result in the payload, leaving the gsd-tools exit projection to P8. - both capability fragments: `|| echo '{"detected":false}'` appended rather than replaced, and fired on the legitimate exit-1 negative, so a correct answer and an honest skip both arrived as two concatenated objects. They now keep the probe's own payload and manufacture only an explicit probe_unavailable skip when the probe produced nothing at all. Every registered outcome is more restrictive on a blocking gate, so this can turn a false green red and never a red green. Docs: FEATURES 156, CONFIGURATION (both keys), references/api-coverage.md seal-time outcome table, and a new how-to for the reason-code vocabulary. Verification runs on the remote runner. Closes #3909 * test(#3909): correct the stale unknown-phase assertion `unknown phase → detected:false, no throw (graceful)` scanned phase 999 against a two-phase roadmap and asserted `detected === false`. That pinned the fabrication as intended behavior: the phase does not exist, so the detector was handed the empty string and its "no core assumption changed" answer described nothing that was ever read. It now asserts the skipped-with-reason shape. The graceful-degradation contract the test was actually protecting — the query succeeds and does not throw on an unknown phase — is unchanged. Found by code review, not by the author. Refs #3909 * docs(#3909): author the FEATURES entry in its generator source `docs/FEATURES.md` is generated by `scripts/gen-features.cjs` from the per-feature fragments in `docs/features/`. The API-coverage entry was edited in the generated file, so the next regeneration silently dropped it. The text now lives in `docs/features/api-coverage-gate.md` and `docs/FEATURES.md` is regenerated from it, leaving the shipped file byte-identical and its content actually derivable. Caught by `lint:generated-sync`. Refs #3909 * test(#3909): bind the skip to "not found", and pin the discriminator The first verification run went red on one case, and the case was wrong rather than the code. `getRoadmapPhaseWithFallback` returns `null` for an unknown phase and for a missing ROADMAP.md, but for a section whose body is whitespace-only it returns the heading line alone — which is not empty. So a body-less section WAS found, and reporting `detected:false` over its heading is a real negative, not a fabrication. The test had assumed the resolver yielded `''` there. Correcting the test rather than the resolver keeps `skipped` bound to the distinction the issue asks for — found versus not found — and avoids diverging `assumption-delta scan` from `roadmap.get-phase`, which the fragment documents as sharing one resolver. Also adds the seeded property the test matrix had promised: for any plan body, the scope read back is whitespace-only exactly when the body was. That pins the gate's discriminator to bytes examined, so it cannot quietly become "no signals found", across unicode whitespace and CRLF. `docs/INVENTORY.md` picks up the reference doc's new seal-time outcome table — surfaced by the co-change gate, not by a lint failure. Refs #3909 * chore(#3909): backfill the changeset PR number Refs #3909 --------- Co-authored-by: sim --- .changeset/plucky-hawks-sing.md | 5 + .../fragments/api-coverage-plan-pre.md | 10 +- .../assumption-delta/fragments/plan-pre.md | 7 +- docs/CONFIGURATION.md | 4 +- docs/FEATURES.md | 6 + docs/INVENTORY.md | 2 +- docs/README.md | 1 + docs/features/api-coverage-gate.md | 6 + .../resolve-a-skipped-capability-probe.md | 116 +++++++ gsd-core/bin/gsd-tools.cjs | 13 +- gsd-core/bin/lib/capability-registry.cjs | 8 +- gsd-core/references/api-coverage.md | 21 +- src/check-command-router.cts | 27 ++ tests/api-coverage-gate-e2e.test.cjs | 160 ++++++++++ .../assumption-delta-checkpoint-e2e.test.cjs | 93 +++++- tests/capability-probe-fallback.test.cjs | 290 ++++++++++++++++++ 16 files changed, 755 insertions(+), 14 deletions(-) create mode 100644 .changeset/plucky-hawks-sing.md create mode 100644 docs/how-to/resolve-a-skipped-capability-probe.md create mode 100644 tests/capability-probe-fallback.test.cjs diff --git a/.changeset/plucky-hawks-sing.md b/.changeset/plucky-hawks-sing.md new file mode 100644 index 000000000..7bad0cb91 --- /dev/null +++ b/.changeset/plucky-hawks-sing.md @@ -0,0 +1,5 @@ +--- +type: Changed +pr: 3944 +--- +**The API-coverage seal gate no longer clears a phase it never examined** — a phase with no plan body and no roadmap section previously ran the detector over zero bytes and sealed as "no external-API integration"; it is now held with `scope_unavailable`, and the assumption-delta checkpoint reports `skipped` instead of a fabricated `detected:false` when it cannot resolve a phase section. (#3909) diff --git a/capabilities/ai-integration/fragments/api-coverage-plan-pre.md b/capabilities/ai-integration/fragments/api-coverage-plan-pre.md index 2cfd6fb8c..5d3d10b60 100644 --- a/capabilities/ai-integration/fragments/api-coverage-plan-pre.md +++ b/capabilities/ai-integration/fragments/api-coverage-plan-pre.md @@ -22,9 +22,17 @@ scope (the concatenation of this phase's ROADMAP section + the PLAN body): ```bash SCOPE="$(cat "${PHASE_DIR}"/*-PLAN.md 2>/dev/null) $(gsd_run query roadmap.get-phase "${PHASE}" 2>/dev/null || true)" -API_COVERAGE_JSON=$(printf '%s' "$SCOPE" | node gsd-core/bin/lib/api-coverage.cjs --json 2>/dev/null || echo '{"detected":false,"signals":[]}') +API_COVERAGE_JSON=$(printf '%s' "$SCOPE" | node gsd-core/bin/lib/api-coverage.cjs --json 2>/dev/null) || true +[ -n "$API_COVERAGE_JSON" ] || API_COVERAGE_JSON='{"skipped":true,"reason":"probe_unavailable"}' ``` +The `|| true` neutralizes the assignment's status without discarding the +detector's own payload: the detector exits **1** for a real "no integration" +verdict, so treating any non-zero exit as failure would throw away a correct +answer. Emptiness — not exit status — is what proves the probe never ran, and +the second line is the only place the fragment manufactures a payload of its +own — one that records the *absence* of a verdict rather than asserting one. + The detector's exit code and `--json` payload now distinguish a real negative from an unexamined input (ADR-3889 Phase 3, #3907): empty/whitespace-only `$SCOPE` or a stdin read failure emit `{"skipped":true,"reason":"no_input"| diff --git a/capabilities/assumption-delta/fragments/plan-pre.md b/capabilities/assumption-delta/fragments/plan-pre.md index 2864d5c2a..f2cc2c0e2 100644 --- a/capabilities/assumption-delta/fragments/plan-pre.md +++ b/capabilities/assumption-delta/fragments/plan-pre.md @@ -11,10 +11,11 @@ Most quietly-imported architectural debt does not come from a missing upfront de The detector is a deterministic scan over the phase scope text. It strips fenced code blocks first, so a trigger word that appears only inside a code snippet does not fire. It returns a typed result: `{ detected, signals[], terms }`. Resolve it through the `assumption-delta scan` query (same phase-section resolver as `roadmap.get-phase`): ```bash -ASSUMPTION_DELTA_JSON=$(gsd_run query assumption-delta scan "${PHASE}" --json 2>/dev/null || echo '{"detected":false,"signals":[],"terms":{}}') +ASSUMPTION_DELTA_JSON=$(gsd_run query assumption-delta scan "${PHASE}" --json 2>/dev/null) || true +[ -n "$ASSUMPTION_DELTA_JSON" ] || ASSUMPTION_DELTA_JSON='{"skipped":true,"reason":"probe_unavailable"}' ``` -> If the phase section cannot be resolved (no `ROADMAP.md` / unknown phase), the query emits `{ "detected": false, ... }` — the checkpoint does not fire. Do not block on it. +> If the phase section cannot be resolved (no `ROADMAP.md` / unknown phase, or a section with no body), the query emits `{ "skipped": true, "reason": "phase_unresolved" }` — **not** `detected:false`. A probe that never had input does not get to assert that this phase changes no core assumption. The checkpoint does not fire either way; the difference is that a skip is now distinguishable from a real negative. Do not block on it. > > Optional tuning — pass `--terms ` to replace the curated pluralization cues for this project (the `optional`/`chosen` cues keep their defaults): `gsd_run query assumption-delta scan "${PHASE}" --json --terms second,alternative,fallback`. @@ -22,6 +23,8 @@ ASSUMPTION_DELTA_JSON=$(gsd_run query assumption-delta scan "${PHASE}" --json 2> Read `ASSUMPTION_DELTA_JSON`. Act on `detected` only — do **not** pattern-match the human prose. +**If `skipped` is `true`:** the detector never examined a phase section — it could not resolve one (`phase_unresolved`) or could not run at all (`probe_unavailable`). Skip the checkpoint for this run rather than asserting a verdict about input that was never examined; do not raise it with the user. **Check for `skipped` before reading `detected`** — a skipped payload carries no `detected` key, and treating its absence as `false` re-creates the fabrication this branch exists to prevent. + **If `detected` is `false`:** this phase does not change a core assumption. Skip the checkpoint entirely and continue planning. Do not raise it with the user. **If `detected` is `true`:** a core assumption may have lost its monopoly. The `signals[]` array tells you which family fired: diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 51c452220..d427a9cc4 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -395,7 +395,7 @@ All workflow toggles follow the **absent = enabled** pattern. If a key is missin | `workflow.nyquist_validation` | boolean | `true` | Test coverage mapping during plan-phase research | | `workflow.ui_phase` | boolean | `true` | Generate UI design contracts for frontend phases | | `workflow.ui_safety_gate` | boolean | `true` | Prompt to run /gsd-ui-phase for frontend phases during plan-phase | -| `workflow.assumption_delta` | boolean | `true` | Advisory architecture checkpoint during planning. When a phase makes something **plural, optional, or chosen** that used to be **singular, required, or derived** (e.g. a second auth method, a required field becoming optional, a constant becoming a parameter), the planner is prompted to re-ask whether the primary key / identity model still names the right thing (promote the new general representation vs. add it alongside). Non-blocking; fires only on a detected signal. Bare "or" is intentionally excluded (prose false-positives). Inspect a phase with `gsd_run query assumption-delta scan `. Added in #1561 | +| `workflow.assumption_delta` | boolean | `true` | Advisory architecture checkpoint during planning. When a phase makes something **plural, optional, or chosen** that used to be **singular, required, or derived** (e.g. a second auth method, a required field becoming optional, a constant becoming a parameter), the planner is prompted to re-ask whether the primary key / identity model still names the right thing (promote the new general representation vs. add it alongside). Non-blocking; fires only on a detected signal. Bare "or" is intentionally excluded (prose false-positives). Inspect a phase with `gsd_run query assumption-delta scan `. Added in #1561. A phase section that cannot be resolved returns `{"skipped":true,"reason":"phase_unresolved"}` rather than a fabricated `detected:false` (#3909) | | `workflow.ui_review` | boolean | `true` | Run visual quality audit (`/gsd-ui-review`) after phase execution in autonomous mode. When `false`, the UI audit step is skipped. | | `workflow.live_dom_uat` | boolean | `false` | **Default-off.** Enable live-DOM verification (#2856). When `true`, a `gsd-dom-verifier` step runs after each execution wave and writes `{phase}-DOM-VERIFY.md`, and the orchestrator's automated UI verification will additionally consider `mcp__chrome-devtools__*` / `mcp__claude-in-chrome__*` when present. Browser reach is confined to `gsd-dom-verifier` — `gsd-executor`'s tool surface is unchanged in every configuration. Presence of a browser MCP server is **not** sufficient on its own: a server configured for unrelated work is never driven unless this key is on. The pre-existing `mcp__playwright__*` path is unaffected by this key. Note `chrome-devtools-mcp` holds an exclusive browser-profile lock, so concurrent waves need `--isolated` on **your** MCP server registration — GSD cannot pass it. See [Enable live-DOM verification](how-to/enable-live-dom-verification.md). | | `workflow.node_repair` | boolean | `true` | Autonomous task repair on verification failure | @@ -428,7 +428,7 @@ All workflow toggles follow the **absent = enabled** pattern. If a key is missin | `workflow.cross_ai_timeout` | number | `300` | Timeout in seconds for cross-AI execution commands. Prevents runaway external processes. Added in v1.36 | | `workflow.test_gate_timeout` | number | `600` | Wall-clock timeout (seconds) for a verification test gate; a watch-mode runner (vitest/jest) that never exits is aborted after this budget instead of hanging the orchestrator (#1857) | | `workflow.ai_integration_phase` | boolean | `true` | Enable the `/gsd-ai-integration-phase` command. When `false`, the command exits with a configuration gate message | -| `workflow.api_coverage_gate` | boolean | `true` | Require an explicit API-coverage decision before a phase that integrates an external API/SDK/service can seal. At `plan:pre` the planner is prompted to produce a `COVERAGE.md` matrix (full coverage by default, every opt-out reasoned); at `verify:pre` a blocking gate fails the seal unless the matrix is complete. Independent of `ai_integration_phase` (#1562) | +| `workflow.api_coverage_gate` | boolean | `true` | Require an explicit API-coverage decision before a phase that integrates an external API/SDK/service can seal. At `plan:pre` the planner is prompted to produce a `COVERAGE.md` matrix (full coverage by default, every opt-out reasoned); at `verify:pre` a blocking gate fails the seal unless the matrix is complete. Independent of `ai_integration_phase` (#1562). A phase whose scope cannot be established at all (no plan body and no roadmap section) is held rather than passed, reporting `scope_unavailable` — see [Resolve a skipped capability probe](how-to/resolve-a-skipped-capability-probe.md) (#3909) | | `workflow.auto_prune_state` | boolean | `false` | When `true`, automatically prune stale entries from STATE.md at phase boundaries instead of prompting | | `workflow.pattern_mapper` | boolean | `true` | Run the `gsd-pattern-mapper` agent between research and planning to map new files to existing codebase analogs | | `workflow.subagent_timeout` | number | `300000` | Timeout in milliseconds for parallel subagent tasks (e.g. codebase mapping). Increase for large codebases or slower models. Default: 300000 (5 minutes) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 5f31a4afa..42258c28e 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -3412,6 +3412,12 @@ The load-bearing wire is the `plan-phase` lift into `must_haves.prohibitions`, s **Purpose:** A phase that integrates an external API, SDK, or service can no longer seal verification without a decided coverage matrix (#1562). +**Behavior:** At seal time the gate reads the phase scope — the plan bodies, falling back to this phase's ROADMAP section — and runs the deterministic detector over it. An integration signal without a `COVERAGE.md` matrix blocks the seal; no signal passes. + +**Unestablished scope is not a negative verdict (#3909).** A phase with no plan body *and* no roadmap section gives the detector nothing to examine. The gate used to run detection over zero bytes and pass, certifying "no external-API integration" from a probe that never looked. It now holds the seal instead, reporting `scope_unavailable: true`. A phase whose plans are real and simply contain no API vocabulary is unaffected — the discriminator is *bytes examined*, never *signals found*. + +**Breaking change:** a phase that previously sealed because its detector could not establish a scope is now correctly held. Add the phase plan, or record a reasoned `No external API integration: ` declaration in `COVERAGE.md`. See [Resolve a skipped capability probe](how-to/resolve-a-skipped-capability-probe.md). + --- ### 157. State Rebuild & Configurable Graph Path diff --git a/docs/INVENTORY.md b/docs/INVENTORY.md index 2d1c4768e..89f42a921 100644 --- a/docs/INVENTORY.md +++ b/docs/INVENTORY.md @@ -375,7 +375,7 @@ Full roster at `gsd-core/references/*.md`. References are shared knowledge docum | `autonomous-ui-design-contract.md` | Autonomous-mode step 3a.5 — resolve whether a frontend phase needs a UI-SPEC.md and generate one through active `plan:pre` hooks; always non-blocking. | | `ios-scaffold.md` | iOS application scaffolding patterns. | | `ai-evals.md` | AI evaluation design reference for `/gsd-ai-integration-phase`. | -| `api-coverage.md` | API-coverage gate reference (full-coverage-by-default) for the `ai-integration` capability's `verify:pre` blocking gate (#1562) — matrix format, trigger, tuning, detector CLI. | +| `api-coverage.md` | API-coverage gate reference (full-coverage-by-default) for the `ai-integration` capability's `verify:pre` blocking gate (#1562) — matrix format, trigger, tuning, detector CLI, and the seal-time outcome table naming every pass/block arm including `scope_unavailable` (#3909). | | `ai-frameworks.md` | AI framework decision-matrix reference for `gsd-framework-selector`. | | `executor-examples.md` | Worked examples for the gsd-executor agent. | | `doc-conflict-engine.md` | Shared conflict-detection contract for ingest/import workflows. | diff --git a/docs/README.md b/docs/README.md index b00cd448c..b96624c24 100644 --- a/docs/README.md +++ b/docs/README.md @@ -30,6 +30,7 @@ Language versions: [English](README.md) · [Português (pt-BR)](pt-BR/README.md) - [State a failing direction](how-to/state-a-failing-direction.md) — say what output constitutes failure for an `` verify command, and migrate a phase planned before the rule - [Resolve a contract-drift finding](how-to/resolve-contract-drift-findings.md) — bring an agent's completion contract, read-tag gate, or deleted-file test reference back into agreement with the registry - [Resolve unreachable-guard findings](how-to/resolve-unreachable-guard-findings.md) — fix shell guards whose fallback arm cannot run, and tell "nothing to report" apart from "could not look" +- [Resolve a skipped capability probe](how-to/resolve-a-skipped-capability-probe.md) — act on a coverage gate that held your phase for an unestablished scope, or a planning checkpoint that reported `skipped` instead of a verdict - [Diagnose which gsd-tools is running](how-to/diagnose-a-foreign-gsd-tools.md) — tell this package's tool apart from the predecessor's colliding binary and from a gsd-core too old to identify itself - [Resolve an ESLint glob-coverage finding](how-to/resolve-eslint-coverage-findings.md) — bring a source file that matches no lint rule under coverage, or record a reasoned exemption - [Read the statusline freshness marker](how-to/read-the-statusline-freshness-marker.md) — turn on `state ~N commits back`, and tell "STATE.md is fresh" apart from "freshness could not be established" diff --git a/docs/features/api-coverage-gate.md b/docs/features/api-coverage-gate.md index 19631e68d..99bee272b 100644 --- a/docs/features/api-coverage-gate.md +++ b/docs/features/api-coverage-gate.md @@ -7,3 +7,9 @@ group: v1.7.0 Features **Command:** `/gsd-verify-work` **Purpose:** A phase that integrates an external API, SDK, or service can no longer seal verification without a decided coverage matrix (#1562). + +**Behavior:** At seal time the gate reads the phase scope — the plan bodies, falling back to this phase's ROADMAP section — and runs the deterministic detector over it. An integration signal without a `COVERAGE.md` matrix blocks the seal; no signal passes. + +**Unestablished scope is not a negative verdict (#3909).** A phase with no plan body *and* no roadmap section gives the detector nothing to examine. The gate used to run detection over zero bytes and pass, certifying "no external-API integration" from a probe that never looked. It now holds the seal instead, reporting `scope_unavailable: true`. A phase whose plans are real and simply contain no API vocabulary is unaffected — the discriminator is *bytes examined*, never *signals found*. + +**Breaking change:** a phase that previously sealed because its detector could not establish a scope is now correctly held. Add the phase plan, or record a reasoned `No external API integration: ` declaration in `COVERAGE.md`. See [Resolve a skipped capability probe](how-to/resolve-a-skipped-capability-probe.md). diff --git a/docs/how-to/resolve-a-skipped-capability-probe.md b/docs/how-to/resolve-a-skipped-capability-probe.md new file mode 100644 index 000000000..dbc195a36 --- /dev/null +++ b/docs/how-to/resolve-a-skipped-capability-probe.md @@ -0,0 +1,116 @@ +# Resolve a skipped capability probe + +A phase-scope probe — the API-coverage detector or the assumption-delta +detector — needs real text to examine before it can assert a verdict. When it +gets none, it says so instead of guessing. This page covers reading that +signal and clearing it. + +## What you saw + +One of two things, depending on which surface you hit: + +- **The seal gate held your phase.** `verify:pre` reported a block with + `scope_unavailable: true` instead of the usual "no external-API integration + detected" pass. +- **A planning checkpoint reported `skipped` instead of a verdict.** The + `assumption-delta` or `api-coverage` plan-time checkpoint printed + `{"skipped":true,"reason":"..."}` and produced no decision either way. + +Both are the same underlying fix (#3909): a probe that never examined real +input used to fabricate `detected:false`, which reads as "nothing here" when +the true answer is "nobody looked." Now it says which one happened. + +## The reason-code table + +| `reason` | What it means | What actually happened | Remedy | +|---|---|---|---| +| `scope_unavailable` | The seal-time gate found no phase scope at all | No plan body (`.planning/phases//*-PLAN.md`) *and* no ROADMAP section for the phase | Add the plan or roadmap section, or write a reasoned `No external API integration: ` declaration to `COVERAGE.md` | +| `phase_unresolved` | The `assumption-delta scan` query could not resolve a phase section to scan | No `ROADMAP.md`, an unknown phase number, or a phase section with no body | If you expected a real scan, fix the phase reference or roadmap section; otherwise no action — the checkpoint correctly stays silent | +| `probe_unavailable` | The detector process itself produced no output | The probe crashed, was not found, or its stdout was empty for a reason unrelated to input content | Check that `gsd-core/bin/lib/api-coverage.cjs` or `gsd-core/bin/lib/assumption-delta.cjs` runs standalone; re-run the checkpoint once the probe itself is healthy | +| `no_input` | The detector ran but stdin was empty or whitespace-only | The phase scope resolved to nothing (empty plan body and empty roadmap section) | Same as `scope_unavailable` — give the detector something to read | +| `stdin_error` | The detector could not read stdin at all | A pipe/read failure upstream of the detector, not an empty-input case | Re-run; if it recurs, the caller constructing `$SCOPE` is the thing to fix, not the detector | + +## The seal gate held my phase + +1. Confirm the reason directly: + + ```bash + gsd_run check api-coverage.verify-pre --raw + ``` + + Look for `"scope_unavailable": true` in the output. That confirms this is + the fail-closed arm, not a real "integration detected" block. + +2. Check whether the phase actually has scope to read: + + ```bash + ls .planning/phases//*-PLAN.md 2>/dev/null + gsd_run query roadmap.get-phase + ``` + + If both come back empty, the gate is correct — there is genuinely nothing + for the detector to examine. + +3. Resolve it one of two ways: + - **Add the missing scope.** Write the phase plan, or add the phase's + section to `ROADMAP.md`, then re-run detection. + - **Record the reasoned declaration anyway.** If the phase truly has no + plan body worth writing (rare), put the human decision directly in + `COVERAGE.md`: + + ```markdown + No external API integration: . + ``` + + This is the same reasoned overrule the gate already accepts when a + detector *does* find (or falsely flags) a signal — see + [`gsd-core/references/api-coverage.md`](../../gsd-core/references/api-coverage.md#declaring-no-external-api-integration-2365). + +4. Re-run the gate to confirm it clears: + + ```bash + gsd_run check api-coverage.verify-pre --raw + ``` + +**Turning the gate off does not answer the question.** Setting +`workflow.api_coverage_gate: false` in `.planning/config.json` silences the +hold, but the phase's API surface is still undecided — you have just stopped +being told. Prefer resolving the scope or writing the declaration. + +## A checkpoint reported skipped + +The `assumption-delta` and `api-coverage` plan-time checkpoints are advisory: +when they cannot resolve a phase section, they skip rather than fire, and that +is correct, non-blocking behavior. You do not need to do anything except +notice that the phase was not actually cleared by a real scan — a `skipped` +payload carries no `detected` key and is not the same as "no signal found." +If you expected a real scan and got a skip instead, treat it like the +`phase_unresolved` row above: check that the phase reference and roadmap +section actually exist. + +## Why this is not just a stricter gate + +A probe reporting `detected:false` from an input it never read is a false +negative on a *blocking* gate — the one direction a gate must never fail +silently. A false positive here costs one line in `COVERAGE.md`; a false +negative lets a real external-API integration seal with an undecided surface, +discovered later by a user who reasonably expected it to work. This change can +turn a false green red. It can never turn a red green — every phase that +passed because a signal was genuinely absent from scope the detector actually +read still passes, unchanged. + +## Telling "nothing to report" apart from "could not look" + +This is the recurring distinction across this doc set (see also +[Resolve unreachable-guard findings](resolve-unreachable-guard-findings.md) and +[Consume the planning snapshot](consume-the-planning-snapshot.md)). A verdict +of `detected: false` or `passed: true` means the detector examined real text +and found nothing. A `skipped` payload or a `scope_unavailable` block means +the detector examined nothing at all. Only the first is good news; the second +is a request for more scope, not a clean bill of health. + +## Related + +- [`gsd-core/references/api-coverage.md`](../../gsd-core/references/api-coverage.md) — the full API-coverage gate reference, including the seal-time outcome table +- [Resolve unreachable-guard findings](resolve-unreachable-guard-findings.md) — the same "nothing to report vs. could not look" distinction, one layer down +- [Consume the planning snapshot](consume-the-planning-snapshot.md) — the `scope` field's version of this same rule diff --git a/gsd-core/bin/gsd-tools.cjs b/gsd-core/bin/gsd-tools.cjs index 058c101f4..8e73ee6a0 100755 --- a/gsd-core/bin/gsd-tools.cjs +++ b/gsd-core/bin/gsd-tools.cjs @@ -2419,8 +2419,19 @@ function dispatchOverlayCapabilityCommand({ command, args, cwd, raw, error, load .filter((t) => t.length > 0); termsOverride = list.length > 0 ? { pluralization: list } : undefined; } + // An unresolvable phase section is not a negative verdict. Feeding + // `''` to the detector reported "examined, found nothing" for a + // probe that never had input — no ROADMAP.md, or a phase number + // absent from it, both read as a confident `detected:false` + // (ADR-3889 failure class (c), #3909). Exit stays 0: this is an + // ADR-2980 degraded result carried in the payload, and ADR-3889 P8 + // pins the gsd-tools exit projection at v1. const section = roadmap.getRoadmapPhaseWithFallback(cwd, phaseNum); - const result = detectAssumptionDelta(section ?? '', termsOverride); + if (typeof section !== 'string' || section.trim() === '') { + output({ skipped: true, reason: 'phase_unresolved' }, raw); + return; + } + const result = detectAssumptionDelta(section, termsOverride); output(result, raw); return; } diff --git a/gsd-core/bin/lib/capability-registry.cjs b/gsd-core/bin/lib/capability-registry.cjs index 69b9debbe..9eb399dda 100644 --- a/gsd-core/bin/lib/capability-registry.cjs +++ b/gsd-core/bin/lib/capability-registry.cjs @@ -68,7 +68,7 @@ const capabilities = { "into": "planner", "fragment": { "path": "fragments/api-coverage-plan-pre.md", - "inline": "# API Coverage Decision Checkpoint\n\n> Full API Coverage by Default — Opt Out, Never Opt In. Fires when a phase\n> integrates an external API / SDK / service. Most non-API phases will not fire\n> it — that is the point.\n\n## Why this exists\n\n\"We integrated the API\" too often silently means \"we integrated whatever the\nfirst use case exercised.\" Every un-built capability is then an invisible hole,\ndiscovered later by a user who reasonably expected it to work. The phase sealed\ngreen because its tasks completed; nobody decided the gaps were acceptable,\nbecause nobody enumerated them. This checkpoint makes the surface **visible and\ndecided** before the phase can seal.\n\n## Detect whether this phase integrates an external API\n\nThe detector is a deterministic scan over the phase scope. It strips fenced\ncode blocks first, so a trigger term inside a code snippet does not fire. It\nreturns a typed result: `{ detected, signals[], terms }`. Run it on the phase\nscope (the concatenation of this phase's ROADMAP section + the PLAN body):\n\n```bash\nSCOPE=\"$(cat \"${PHASE_DIR}\"/*-PLAN.md 2>/dev/null) $(gsd_run query roadmap.get-phase \"${PHASE}\" 2>/dev/null || true)\"\nAPI_COVERAGE_JSON=$(printf '%s' \"$SCOPE\" | node gsd-core/bin/lib/api-coverage.cjs --json 2>/dev/null || echo '{\"detected\":false,\"signals\":[]}')\n```\n\nThe detector's exit code and `--json` payload now distinguish a real negative\nfrom an unexamined input (ADR-3889 Phase 3, #3907): empty/whitespace-only\n`$SCOPE` or a stdin read failure emit `{\"skipped\":true,\"reason\":\"no_input\"|\n\"stdin_error\"}` — no `detected` key at all. **Check for `skipped` before\nreading `detected`**: a `skipped` payload is not a confirmed \"no API\nintegration\" verdict, it means the detector never examined real input. Do not\ntreat it as `detected:false`. Read `API_COVERAGE_JSON.detected` only when\n`skipped` is absent — act on it only, do **not** pattern-match the prose\nyourself.\n\n**If `skipped` is `true`:** the detector could not establish a scope (empty\n`$SCOPE`) or failed to run (stdin read error). Skip the checkpoint for this\nrun rather than asserting a verdict about input that was never examined; do\nnot raise it with the user.\n\n**If `detected` is `false`:** this phase does not integrate an external API. Skip\nthe checkpoint entirely and continue planning. Do not raise it with the user.\n\n**If `detected` is `true`:** an external-API integration is in scope. You MUST\nproduce a **coverage matrix** before the plan is finalized.\n\n**If `detected` is `true` but the phase genuinely integrates no external API**\n(the detector is deterministic, not infallible — confirm by re-reading the phase\nscope, not by preference): do NOT fabricate a matrix row for a capability that\ndoes not exist. Write a reasoned declaration to `${PHASE_DIR}/COVERAGE.md`\ninstead:\n\n```markdown\nNo external API integration: .\n```\n\nThe reason is required, exactly like an `OPT-OUT` reason. The seal-time gate\naccepts this declaration in place of a matrix.\n\n## Produce the coverage matrix\n\nEnumerate the external API's full **capability surface** — the verb/endpoint/method\nlist (e.g. for a music service: `search`, `play`, `pause`, `skip`, `set_volume`,\n`get_playlist`, `create_playlist`, `add_to_playlist`, …). For each capability\nrecord a decision, starting from **full coverage** as the default:\n\n| capability | decision | reason |\n|---|---|---|\n| `` | `INTEGRATE` \\| `OPT-OUT` | `` |\n\nRules:\n\n- **`INTEGRATE` is the default.** Every capability starts as INTEGRATE; the\n matrix is the *subtraction record*.\n- **Every `OPT-OUT` MUST carry a one-line reason** (`not needed`, `not needed\n yet`, `explicitly out of scope`, …). An opt-out without a reason is an\n un-decided hole — the exact failure mode this gate exists to close.\n- **A second integration against the same need** (e.g. a second platform for the\n same capability) starts from the **same full-coverage baseline** as the first.\n Do not carry over the first integration's opt-outs silently — re-decide each\n capability for the new surface, so a first-class/fallback asymmetry cannot\n accumulate.\n\nWrite the matrix to `${PHASE_DIR}/COVERAGE.md` (canonical markdown-table form):\n\n```markdown\n# API Coverage — \n\n> Full coverage by default. Opt-outs are explicit, reasoned decisions.\n\n| capability | decision | reason |\n|---|---|---|\n| search | INTEGRATE | |\n| playlists | INTEGRATE | |\n| skip | OPT-OUT | not needed yet — tracked for follow-up phase |\n```\n\nA fenced ` ```coverage ` JSON block is also accepted for machine-generated\nmatrices; the markdown table is preferred (human-editable, diff-friendly).\n\n## The seal-time gate\n\nThis checkpoint is enforced. At `verify:pre` the `api-coverage.verify-pre` gate\nruns `check api-coverage.verify-pre `:\n\n- If `COVERAGE.md` exists, it is validated — every row needs a valid decision and\n every `OPT-OUT` a reason. A malformed/partial matrix **blocks the seal**. A\n reasoned `No external API integration: …` declaration (and no rows) passes.\n- If `COVERAGE.md` is absent, the detector runs again over the phase scope. If a\n strong external-API-integration signal is found, the seal is **blocked** until a\n matrix is produced. If no signal is found, the phase is treated as a non-API\n phase and the seal proceeds.\n\nSo: an API-integrating phase cannot seal without a decided matrix. Produce it at\nplan time; do not leave it for seal time.\n\n## Tuning the vocabulary (optional)\n\nThe trigger vocabulary is a curated, additive-only set in\n`gsd-core/bin/lib/api-coverage.cjs` (`DEFAULT_API_COVERAGE_TERMS`). To widen it\nfor a project, override at the call site:\n\n```bash\nprintf '%s' \"$SCOPE\" | node gsd-core/bin/lib/api-coverage.cjs --json \\\n --verbs integrate,wrap,connect,embed --nouns api,sdk,rest,grpc,webhook,plugin\n```\n\nThe whole checkpoint is toggleable via `workflow.api_coverage_gate` in\n`.planning/config.json`.\n" + "inline": "# API Coverage Decision Checkpoint\n\n> Full API Coverage by Default — Opt Out, Never Opt In. Fires when a phase\n> integrates an external API / SDK / service. Most non-API phases will not fire\n> it — that is the point.\n\n## Why this exists\n\n\"We integrated the API\" too often silently means \"we integrated whatever the\nfirst use case exercised.\" Every un-built capability is then an invisible hole,\ndiscovered later by a user who reasonably expected it to work. The phase sealed\ngreen because its tasks completed; nobody decided the gaps were acceptable,\nbecause nobody enumerated them. This checkpoint makes the surface **visible and\ndecided** before the phase can seal.\n\n## Detect whether this phase integrates an external API\n\nThe detector is a deterministic scan over the phase scope. It strips fenced\ncode blocks first, so a trigger term inside a code snippet does not fire. It\nreturns a typed result: `{ detected, signals[], terms }`. Run it on the phase\nscope (the concatenation of this phase's ROADMAP section + the PLAN body):\n\n```bash\nSCOPE=\"$(cat \"${PHASE_DIR}\"/*-PLAN.md 2>/dev/null) $(gsd_run query roadmap.get-phase \"${PHASE}\" 2>/dev/null || true)\"\nAPI_COVERAGE_JSON=$(printf '%s' \"$SCOPE\" | node gsd-core/bin/lib/api-coverage.cjs --json 2>/dev/null) || true\n[ -n \"$API_COVERAGE_JSON\" ] || API_COVERAGE_JSON='{\"skipped\":true,\"reason\":\"probe_unavailable\"}'\n```\n\nThe `|| true` neutralizes the assignment's status without discarding the\ndetector's own payload: the detector exits **1** for a real \"no integration\"\nverdict, so treating any non-zero exit as failure would throw away a correct\nanswer. Emptiness — not exit status — is what proves the probe never ran, and\nthe second line is the only place the fragment manufactures a payload of its\nown — one that records the *absence* of a verdict rather than asserting one.\n\nThe detector's exit code and `--json` payload now distinguish a real negative\nfrom an unexamined input (ADR-3889 Phase 3, #3907): empty/whitespace-only\n`$SCOPE` or a stdin read failure emit `{\"skipped\":true,\"reason\":\"no_input\"|\n\"stdin_error\"}` — no `detected` key at all. **Check for `skipped` before\nreading `detected`**: a `skipped` payload is not a confirmed \"no API\nintegration\" verdict, it means the detector never examined real input. Do not\ntreat it as `detected:false`. Read `API_COVERAGE_JSON.detected` only when\n`skipped` is absent — act on it only, do **not** pattern-match the prose\nyourself.\n\n**If `skipped` is `true`:** the detector could not establish a scope (empty\n`$SCOPE`) or failed to run (stdin read error). Skip the checkpoint for this\nrun rather than asserting a verdict about input that was never examined; do\nnot raise it with the user.\n\n**If `detected` is `false`:** this phase does not integrate an external API. Skip\nthe checkpoint entirely and continue planning. Do not raise it with the user.\n\n**If `detected` is `true`:** an external-API integration is in scope. You MUST\nproduce a **coverage matrix** before the plan is finalized.\n\n**If `detected` is `true` but the phase genuinely integrates no external API**\n(the detector is deterministic, not infallible — confirm by re-reading the phase\nscope, not by preference): do NOT fabricate a matrix row for a capability that\ndoes not exist. Write a reasoned declaration to `${PHASE_DIR}/COVERAGE.md`\ninstead:\n\n```markdown\nNo external API integration: .\n```\n\nThe reason is required, exactly like an `OPT-OUT` reason. The seal-time gate\naccepts this declaration in place of a matrix.\n\n## Produce the coverage matrix\n\nEnumerate the external API's full **capability surface** — the verb/endpoint/method\nlist (e.g. for a music service: `search`, `play`, `pause`, `skip`, `set_volume`,\n`get_playlist`, `create_playlist`, `add_to_playlist`, …). For each capability\nrecord a decision, starting from **full coverage** as the default:\n\n| capability | decision | reason |\n|---|---|---|\n| `` | `INTEGRATE` \\| `OPT-OUT` | `` |\n\nRules:\n\n- **`INTEGRATE` is the default.** Every capability starts as INTEGRATE; the\n matrix is the *subtraction record*.\n- **Every `OPT-OUT` MUST carry a one-line reason** (`not needed`, `not needed\n yet`, `explicitly out of scope`, …). An opt-out without a reason is an\n un-decided hole — the exact failure mode this gate exists to close.\n- **A second integration against the same need** (e.g. a second platform for the\n same capability) starts from the **same full-coverage baseline** as the first.\n Do not carry over the first integration's opt-outs silently — re-decide each\n capability for the new surface, so a first-class/fallback asymmetry cannot\n accumulate.\n\nWrite the matrix to `${PHASE_DIR}/COVERAGE.md` (canonical markdown-table form):\n\n```markdown\n# API Coverage — \n\n> Full coverage by default. Opt-outs are explicit, reasoned decisions.\n\n| capability | decision | reason |\n|---|---|---|\n| search | INTEGRATE | |\n| playlists | INTEGRATE | |\n| skip | OPT-OUT | not needed yet — tracked for follow-up phase |\n```\n\nA fenced ` ```coverage ` JSON block is also accepted for machine-generated\nmatrices; the markdown table is preferred (human-editable, diff-friendly).\n\n## The seal-time gate\n\nThis checkpoint is enforced. At `verify:pre` the `api-coverage.verify-pre` gate\nruns `check api-coverage.verify-pre `:\n\n- If `COVERAGE.md` exists, it is validated — every row needs a valid decision and\n every `OPT-OUT` a reason. A malformed/partial matrix **blocks the seal**. A\n reasoned `No external API integration: …` declaration (and no rows) passes.\n- If `COVERAGE.md` is absent, the detector runs again over the phase scope. If a\n strong external-API-integration signal is found, the seal is **blocked** until a\n matrix is produced. If no signal is found, the phase is treated as a non-API\n phase and the seal proceeds.\n\nSo: an API-integrating phase cannot seal without a decided matrix. Produce it at\nplan time; do not leave it for seal time.\n\n## Tuning the vocabulary (optional)\n\nThe trigger vocabulary is a curated, additive-only set in\n`gsd-core/bin/lib/api-coverage.cjs` (`DEFAULT_API_COVERAGE_TERMS`). To widen it\nfor a project, override at the call site:\n\n```bash\nprintf '%s' \"$SCOPE\" | node gsd-core/bin/lib/api-coverage.cjs --json \\\n --verbs integrate,wrap,connect,embed --nouns api,sdk,rest,grpc,webhook,plugin\n```\n\nThe whole checkpoint is toggleable via `workflow.api_coverage_gate` in\n`.planning/config.json`.\n" }, "produces": [ "COVERAGE.md" @@ -280,7 +280,7 @@ const capabilities = { "into": "planner", "fragment": { "path": "fragments/plan-pre.md", - "inline": "# Assumption-Delta Architecture Checkpoint\n\n> Advisory, non-blocking. Fires **only** when the phase scope shows a singular→plural / required→optional / derived→chosen transition. When it fires, it surfaces ONE identity-model question before the plan is finalized. Most phases will not fire it — that is the point.\n\n## Why this exists\n\nMost quietly-imported architectural debt does not come from a missing upfront design phase. It comes at the *seam*: a later phase introduces a second case (a second platform, auth method, tenant, region, source of truth) and nobody re-asks whether the original abstraction still names the right thing. The phase that adds the second case is exactly the 20-minute conversation that prevents an afternoon of later cleanup.\n\n## Run the detector\n\nThe detector is a deterministic scan over the phase scope text. It strips fenced code blocks first, so a trigger word that appears only inside a code snippet does not fire. It returns a typed result: `{ detected, signals[], terms }`. Resolve it through the `assumption-delta scan` query (same phase-section resolver as `roadmap.get-phase`):\n\n```bash\nASSUMPTION_DELTA_JSON=$(gsd_run query assumption-delta scan \"${PHASE}\" --json 2>/dev/null || echo '{\"detected\":false,\"signals\":[],\"terms\":{}}')\n```\n\n> If the phase section cannot be resolved (no `ROADMAP.md` / unknown phase), the query emits `{ \"detected\": false, ... }` — the checkpoint does not fire. Do not block on it.\n>\n> Optional tuning — pass `--terms ` to replace the curated pluralization cues for this project (the `optional`/`chosen` cues keep their defaults): `gsd_run query assumption-delta scan \"${PHASE}\" --json --terms second,alternative,fallback`.\n\n## Decision branch\n\nRead `ASSUMPTION_DELTA_JSON`. Act on `detected` only — do **not** pattern-match the human prose.\n\n**If `detected` is `false`:** this phase does not change a core assumption. Skip the checkpoint entirely and continue planning. Do not raise it with the user.\n\n**If `detected` is `true`:** a core assumption may have lost its monopoly. The `signals[]` array tells you which family fired:\n\n| `kind` | What changed | The question to answer |\n|---|---|---|\n| `pluralization` | A second X was introduced where there was one (second platform / auth method / tenant / region / source of truth) | Does the current primary key / identity model still name the right noun? |\n| `optional` | A required / `only` field became optional | Is the field still the right anchor, or has the anchor moved? |\n| `chosen` | A derived value became chosen, or a constant became a parameter | Has a configuration decision become a modeling decision? |\n\nBefore finalizing the plan, answer this for the user and record the decision explicitly:\n\n> **Promote vs. add-alongside.** The usual correct move when a generalization occurs is to **promote** the new general representation to the primary and **demote** the old specific one to a detail of one variant — *not* to add the new one alongside the still-required old one. Adding alongside silently contradicts the generalized intent (a later variant that does not fit the old primary can be stored but never confirmed as a default).\n\nRecord the outcome in the PLAN.md front matter / a `` block:\n\n- The **noun** that is now primary (the generalized identity).\n- The **decision**: `promote` | `add-alongside` | `no-change`, with a one-line rationale.\n- If `add-alongside`: call it out as accepted debt and note what would force a later promote.\n\n## Optional companion: an invariant test\n\nWhen `detected` is `true`, suggest (do not require) a contract/invariant test that encodes the now-generalized intent — e.g. *\"every confirmed default round-trips through the primary use-path, for every supported variant.\"* That test goes red the instant a future phase reintroduces the singular assumption, so the regression cannot land silently. If the user accepts, add the test as a task in the plan.\n\n## Tuning the vocabulary (optional)\n\nThe trigger vocabulary is a curated, additive-only set in `gsd-core/bin/lib/assumption-delta.cjs` (`DEFAULT_ASSUMPTION_DELTA_TERMS`). Bare \"or\" is intentionally excluded — it is too common in prose and would make the gate fire constantly. To widen or narrow the cues for a project, override at the call site with `--terms ` (replaces the pluralization cues; `optional`/`chosen` keep defaults). The whole checkpoint is toggleable via `workflow.assumption_delta` in `.planning/config.json`.\n\nThis checkpoint is advisory: it informs and records; it never blocks the phase.\n" + "inline": "# Assumption-Delta Architecture Checkpoint\n\n> Advisory, non-blocking. Fires **only** when the phase scope shows a singular→plural / required→optional / derived→chosen transition. When it fires, it surfaces ONE identity-model question before the plan is finalized. Most phases will not fire it — that is the point.\n\n## Why this exists\n\nMost quietly-imported architectural debt does not come from a missing upfront design phase. It comes at the *seam*: a later phase introduces a second case (a second platform, auth method, tenant, region, source of truth) and nobody re-asks whether the original abstraction still names the right thing. The phase that adds the second case is exactly the 20-minute conversation that prevents an afternoon of later cleanup.\n\n## Run the detector\n\nThe detector is a deterministic scan over the phase scope text. It strips fenced code blocks first, so a trigger word that appears only inside a code snippet does not fire. It returns a typed result: `{ detected, signals[], terms }`. Resolve it through the `assumption-delta scan` query (same phase-section resolver as `roadmap.get-phase`):\n\n```bash\nASSUMPTION_DELTA_JSON=$(gsd_run query assumption-delta scan \"${PHASE}\" --json 2>/dev/null) || true\n[ -n \"$ASSUMPTION_DELTA_JSON\" ] || ASSUMPTION_DELTA_JSON='{\"skipped\":true,\"reason\":\"probe_unavailable\"}'\n```\n\n> If the phase section cannot be resolved (no `ROADMAP.md` / unknown phase, or a section with no body), the query emits `{ \"skipped\": true, \"reason\": \"phase_unresolved\" }` — **not** `detected:false`. A probe that never had input does not get to assert that this phase changes no core assumption. The checkpoint does not fire either way; the difference is that a skip is now distinguishable from a real negative. Do not block on it.\n>\n> Optional tuning — pass `--terms ` to replace the curated pluralization cues for this project (the `optional`/`chosen` cues keep their defaults): `gsd_run query assumption-delta scan \"${PHASE}\" --json --terms second,alternative,fallback`.\n\n## Decision branch\n\nRead `ASSUMPTION_DELTA_JSON`. Act on `detected` only — do **not** pattern-match the human prose.\n\n**If `skipped` is `true`:** the detector never examined a phase section — it could not resolve one (`phase_unresolved`) or could not run at all (`probe_unavailable`). Skip the checkpoint for this run rather than asserting a verdict about input that was never examined; do not raise it with the user. **Check for `skipped` before reading `detected`** — a skipped payload carries no `detected` key, and treating its absence as `false` re-creates the fabrication this branch exists to prevent.\n\n**If `detected` is `false`:** this phase does not change a core assumption. Skip the checkpoint entirely and continue planning. Do not raise it with the user.\n\n**If `detected` is `true`:** a core assumption may have lost its monopoly. The `signals[]` array tells you which family fired:\n\n| `kind` | What changed | The question to answer |\n|---|---|---|\n| `pluralization` | A second X was introduced where there was one (second platform / auth method / tenant / region / source of truth) | Does the current primary key / identity model still name the right noun? |\n| `optional` | A required / `only` field became optional | Is the field still the right anchor, or has the anchor moved? |\n| `chosen` | A derived value became chosen, or a constant became a parameter | Has a configuration decision become a modeling decision? |\n\nBefore finalizing the plan, answer this for the user and record the decision explicitly:\n\n> **Promote vs. add-alongside.** The usual correct move when a generalization occurs is to **promote** the new general representation to the primary and **demote** the old specific one to a detail of one variant — *not* to add the new one alongside the still-required old one. Adding alongside silently contradicts the generalized intent (a later variant that does not fit the old primary can be stored but never confirmed as a default).\n\nRecord the outcome in the PLAN.md front matter / a `` block:\n\n- The **noun** that is now primary (the generalized identity).\n- The **decision**: `promote` | `add-alongside` | `no-change`, with a one-line rationale.\n- If `add-alongside`: call it out as accepted debt and note what would force a later promote.\n\n## Optional companion: an invariant test\n\nWhen `detected` is `true`, suggest (do not require) a contract/invariant test that encodes the now-generalized intent — e.g. *\"every confirmed default round-trips through the primary use-path, for every supported variant.\"* That test goes red the instant a future phase reintroduces the singular assumption, so the regression cannot land silently. If the user accepts, add the test as a task in the plan.\n\n## Tuning the vocabulary (optional)\n\nThe trigger vocabulary is a curated, additive-only set in `gsd-core/bin/lib/assumption-delta.cjs` (`DEFAULT_ASSUMPTION_DELTA_TERMS`). Bare \"or\" is intentionally excluded — it is too common in prose and would make the gate fire constantly. To widen or narrow the cues for a project, override at the call site with `--terms ` (replaces the pluralization cues; `optional`/`chosen` keep defaults). The whole checkpoint is toggleable via `workflow.assumption_delta` in `.planning/config.json`.\n\nThis checkpoint is advisory: it informs and records; it never blocks the phase.\n" }, "produces": [], "consumes": [ @@ -4256,7 +4256,7 @@ const byLoopPoint = { "into": "planner", "fragment": { "path": "fragments/api-coverage-plan-pre.md", - "inline": "# API Coverage Decision Checkpoint\n\n> Full API Coverage by Default — Opt Out, Never Opt In. Fires when a phase\n> integrates an external API / SDK / service. Most non-API phases will not fire\n> it — that is the point.\n\n## Why this exists\n\n\"We integrated the API\" too often silently means \"we integrated whatever the\nfirst use case exercised.\" Every un-built capability is then an invisible hole,\ndiscovered later by a user who reasonably expected it to work. The phase sealed\ngreen because its tasks completed; nobody decided the gaps were acceptable,\nbecause nobody enumerated them. This checkpoint makes the surface **visible and\ndecided** before the phase can seal.\n\n## Detect whether this phase integrates an external API\n\nThe detector is a deterministic scan over the phase scope. It strips fenced\ncode blocks first, so a trigger term inside a code snippet does not fire. It\nreturns a typed result: `{ detected, signals[], terms }`. Run it on the phase\nscope (the concatenation of this phase's ROADMAP section + the PLAN body):\n\n```bash\nSCOPE=\"$(cat \"${PHASE_DIR}\"/*-PLAN.md 2>/dev/null) $(gsd_run query roadmap.get-phase \"${PHASE}\" 2>/dev/null || true)\"\nAPI_COVERAGE_JSON=$(printf '%s' \"$SCOPE\" | node gsd-core/bin/lib/api-coverage.cjs --json 2>/dev/null || echo '{\"detected\":false,\"signals\":[]}')\n```\n\nThe detector's exit code and `--json` payload now distinguish a real negative\nfrom an unexamined input (ADR-3889 Phase 3, #3907): empty/whitespace-only\n`$SCOPE` or a stdin read failure emit `{\"skipped\":true,\"reason\":\"no_input\"|\n\"stdin_error\"}` — no `detected` key at all. **Check for `skipped` before\nreading `detected`**: a `skipped` payload is not a confirmed \"no API\nintegration\" verdict, it means the detector never examined real input. Do not\ntreat it as `detected:false`. Read `API_COVERAGE_JSON.detected` only when\n`skipped` is absent — act on it only, do **not** pattern-match the prose\nyourself.\n\n**If `skipped` is `true`:** the detector could not establish a scope (empty\n`$SCOPE`) or failed to run (stdin read error). Skip the checkpoint for this\nrun rather than asserting a verdict about input that was never examined; do\nnot raise it with the user.\n\n**If `detected` is `false`:** this phase does not integrate an external API. Skip\nthe checkpoint entirely and continue planning. Do not raise it with the user.\n\n**If `detected` is `true`:** an external-API integration is in scope. You MUST\nproduce a **coverage matrix** before the plan is finalized.\n\n**If `detected` is `true` but the phase genuinely integrates no external API**\n(the detector is deterministic, not infallible — confirm by re-reading the phase\nscope, not by preference): do NOT fabricate a matrix row for a capability that\ndoes not exist. Write a reasoned declaration to `${PHASE_DIR}/COVERAGE.md`\ninstead:\n\n```markdown\nNo external API integration: .\n```\n\nThe reason is required, exactly like an `OPT-OUT` reason. The seal-time gate\naccepts this declaration in place of a matrix.\n\n## Produce the coverage matrix\n\nEnumerate the external API's full **capability surface** — the verb/endpoint/method\nlist (e.g. for a music service: `search`, `play`, `pause`, `skip`, `set_volume`,\n`get_playlist`, `create_playlist`, `add_to_playlist`, …). For each capability\nrecord a decision, starting from **full coverage** as the default:\n\n| capability | decision | reason |\n|---|---|---|\n| `` | `INTEGRATE` \\| `OPT-OUT` | `` |\n\nRules:\n\n- **`INTEGRATE` is the default.** Every capability starts as INTEGRATE; the\n matrix is the *subtraction record*.\n- **Every `OPT-OUT` MUST carry a one-line reason** (`not needed`, `not needed\n yet`, `explicitly out of scope`, …). An opt-out without a reason is an\n un-decided hole — the exact failure mode this gate exists to close.\n- **A second integration against the same need** (e.g. a second platform for the\n same capability) starts from the **same full-coverage baseline** as the first.\n Do not carry over the first integration's opt-outs silently — re-decide each\n capability for the new surface, so a first-class/fallback asymmetry cannot\n accumulate.\n\nWrite the matrix to `${PHASE_DIR}/COVERAGE.md` (canonical markdown-table form):\n\n```markdown\n# API Coverage — \n\n> Full coverage by default. Opt-outs are explicit, reasoned decisions.\n\n| capability | decision | reason |\n|---|---|---|\n| search | INTEGRATE | |\n| playlists | INTEGRATE | |\n| skip | OPT-OUT | not needed yet — tracked for follow-up phase |\n```\n\nA fenced ` ```coverage ` JSON block is also accepted for machine-generated\nmatrices; the markdown table is preferred (human-editable, diff-friendly).\n\n## The seal-time gate\n\nThis checkpoint is enforced. At `verify:pre` the `api-coverage.verify-pre` gate\nruns `check api-coverage.verify-pre `:\n\n- If `COVERAGE.md` exists, it is validated — every row needs a valid decision and\n every `OPT-OUT` a reason. A malformed/partial matrix **blocks the seal**. A\n reasoned `No external API integration: …` declaration (and no rows) passes.\n- If `COVERAGE.md` is absent, the detector runs again over the phase scope. If a\n strong external-API-integration signal is found, the seal is **blocked** until a\n matrix is produced. If no signal is found, the phase is treated as a non-API\n phase and the seal proceeds.\n\nSo: an API-integrating phase cannot seal without a decided matrix. Produce it at\nplan time; do not leave it for seal time.\n\n## Tuning the vocabulary (optional)\n\nThe trigger vocabulary is a curated, additive-only set in\n`gsd-core/bin/lib/api-coverage.cjs` (`DEFAULT_API_COVERAGE_TERMS`). To widen it\nfor a project, override at the call site:\n\n```bash\nprintf '%s' \"$SCOPE\" | node gsd-core/bin/lib/api-coverage.cjs --json \\\n --verbs integrate,wrap,connect,embed --nouns api,sdk,rest,grpc,webhook,plugin\n```\n\nThe whole checkpoint is toggleable via `workflow.api_coverage_gate` in\n`.planning/config.json`.\n" + "inline": "# API Coverage Decision Checkpoint\n\n> Full API Coverage by Default — Opt Out, Never Opt In. Fires when a phase\n> integrates an external API / SDK / service. Most non-API phases will not fire\n> it — that is the point.\n\n## Why this exists\n\n\"We integrated the API\" too often silently means \"we integrated whatever the\nfirst use case exercised.\" Every un-built capability is then an invisible hole,\ndiscovered later by a user who reasonably expected it to work. The phase sealed\ngreen because its tasks completed; nobody decided the gaps were acceptable,\nbecause nobody enumerated them. This checkpoint makes the surface **visible and\ndecided** before the phase can seal.\n\n## Detect whether this phase integrates an external API\n\nThe detector is a deterministic scan over the phase scope. It strips fenced\ncode blocks first, so a trigger term inside a code snippet does not fire. It\nreturns a typed result: `{ detected, signals[], terms }`. Run it on the phase\nscope (the concatenation of this phase's ROADMAP section + the PLAN body):\n\n```bash\nSCOPE=\"$(cat \"${PHASE_DIR}\"/*-PLAN.md 2>/dev/null) $(gsd_run query roadmap.get-phase \"${PHASE}\" 2>/dev/null || true)\"\nAPI_COVERAGE_JSON=$(printf '%s' \"$SCOPE\" | node gsd-core/bin/lib/api-coverage.cjs --json 2>/dev/null) || true\n[ -n \"$API_COVERAGE_JSON\" ] || API_COVERAGE_JSON='{\"skipped\":true,\"reason\":\"probe_unavailable\"}'\n```\n\nThe `|| true` neutralizes the assignment's status without discarding the\ndetector's own payload: the detector exits **1** for a real \"no integration\"\nverdict, so treating any non-zero exit as failure would throw away a correct\nanswer. Emptiness — not exit status — is what proves the probe never ran, and\nthe second line is the only place the fragment manufactures a payload of its\nown — one that records the *absence* of a verdict rather than asserting one.\n\nThe detector's exit code and `--json` payload now distinguish a real negative\nfrom an unexamined input (ADR-3889 Phase 3, #3907): empty/whitespace-only\n`$SCOPE` or a stdin read failure emit `{\"skipped\":true,\"reason\":\"no_input\"|\n\"stdin_error\"}` — no `detected` key at all. **Check for `skipped` before\nreading `detected`**: a `skipped` payload is not a confirmed \"no API\nintegration\" verdict, it means the detector never examined real input. Do not\ntreat it as `detected:false`. Read `API_COVERAGE_JSON.detected` only when\n`skipped` is absent — act on it only, do **not** pattern-match the prose\nyourself.\n\n**If `skipped` is `true`:** the detector could not establish a scope (empty\n`$SCOPE`) or failed to run (stdin read error). Skip the checkpoint for this\nrun rather than asserting a verdict about input that was never examined; do\nnot raise it with the user.\n\n**If `detected` is `false`:** this phase does not integrate an external API. Skip\nthe checkpoint entirely and continue planning. Do not raise it with the user.\n\n**If `detected` is `true`:** an external-API integration is in scope. You MUST\nproduce a **coverage matrix** before the plan is finalized.\n\n**If `detected` is `true` but the phase genuinely integrates no external API**\n(the detector is deterministic, not infallible — confirm by re-reading the phase\nscope, not by preference): do NOT fabricate a matrix row for a capability that\ndoes not exist. Write a reasoned declaration to `${PHASE_DIR}/COVERAGE.md`\ninstead:\n\n```markdown\nNo external API integration: .\n```\n\nThe reason is required, exactly like an `OPT-OUT` reason. The seal-time gate\naccepts this declaration in place of a matrix.\n\n## Produce the coverage matrix\n\nEnumerate the external API's full **capability surface** — the verb/endpoint/method\nlist (e.g. for a music service: `search`, `play`, `pause`, `skip`, `set_volume`,\n`get_playlist`, `create_playlist`, `add_to_playlist`, …). For each capability\nrecord a decision, starting from **full coverage** as the default:\n\n| capability | decision | reason |\n|---|---|---|\n| `` | `INTEGRATE` \\| `OPT-OUT` | `` |\n\nRules:\n\n- **`INTEGRATE` is the default.** Every capability starts as INTEGRATE; the\n matrix is the *subtraction record*.\n- **Every `OPT-OUT` MUST carry a one-line reason** (`not needed`, `not needed\n yet`, `explicitly out of scope`, …). An opt-out without a reason is an\n un-decided hole — the exact failure mode this gate exists to close.\n- **A second integration against the same need** (e.g. a second platform for the\n same capability) starts from the **same full-coverage baseline** as the first.\n Do not carry over the first integration's opt-outs silently — re-decide each\n capability for the new surface, so a first-class/fallback asymmetry cannot\n accumulate.\n\nWrite the matrix to `${PHASE_DIR}/COVERAGE.md` (canonical markdown-table form):\n\n```markdown\n# API Coverage — \n\n> Full coverage by default. Opt-outs are explicit, reasoned decisions.\n\n| capability | decision | reason |\n|---|---|---|\n| search | INTEGRATE | |\n| playlists | INTEGRATE | |\n| skip | OPT-OUT | not needed yet — tracked for follow-up phase |\n```\n\nA fenced ` ```coverage ` JSON block is also accepted for machine-generated\nmatrices; the markdown table is preferred (human-editable, diff-friendly).\n\n## The seal-time gate\n\nThis checkpoint is enforced. At `verify:pre` the `api-coverage.verify-pre` gate\nruns `check api-coverage.verify-pre `:\n\n- If `COVERAGE.md` exists, it is validated — every row needs a valid decision and\n every `OPT-OUT` a reason. A malformed/partial matrix **blocks the seal**. A\n reasoned `No external API integration: …` declaration (and no rows) passes.\n- If `COVERAGE.md` is absent, the detector runs again over the phase scope. If a\n strong external-API-integration signal is found, the seal is **blocked** until a\n matrix is produced. If no signal is found, the phase is treated as a non-API\n phase and the seal proceeds.\n\nSo: an API-integrating phase cannot seal without a decided matrix. Produce it at\nplan time; do not leave it for seal time.\n\n## Tuning the vocabulary (optional)\n\nThe trigger vocabulary is a curated, additive-only set in\n`gsd-core/bin/lib/api-coverage.cjs` (`DEFAULT_API_COVERAGE_TERMS`). To widen it\nfor a project, override at the call site:\n\n```bash\nprintf '%s' \"$SCOPE\" | node gsd-core/bin/lib/api-coverage.cjs --json \\\n --verbs integrate,wrap,connect,embed --nouns api,sdk,rest,grpc,webhook,plugin\n```\n\nThe whole checkpoint is toggleable via `workflow.api_coverage_gate` in\n`.planning/config.json`.\n" }, "produces": [ "COVERAGE.md" @@ -4273,7 +4273,7 @@ const byLoopPoint = { "into": "planner", "fragment": { "path": "fragments/plan-pre.md", - "inline": "# Assumption-Delta Architecture Checkpoint\n\n> Advisory, non-blocking. Fires **only** when the phase scope shows a singular→plural / required→optional / derived→chosen transition. When it fires, it surfaces ONE identity-model question before the plan is finalized. Most phases will not fire it — that is the point.\n\n## Why this exists\n\nMost quietly-imported architectural debt does not come from a missing upfront design phase. It comes at the *seam*: a later phase introduces a second case (a second platform, auth method, tenant, region, source of truth) and nobody re-asks whether the original abstraction still names the right thing. The phase that adds the second case is exactly the 20-minute conversation that prevents an afternoon of later cleanup.\n\n## Run the detector\n\nThe detector is a deterministic scan over the phase scope text. It strips fenced code blocks first, so a trigger word that appears only inside a code snippet does not fire. It returns a typed result: `{ detected, signals[], terms }`. Resolve it through the `assumption-delta scan` query (same phase-section resolver as `roadmap.get-phase`):\n\n```bash\nASSUMPTION_DELTA_JSON=$(gsd_run query assumption-delta scan \"${PHASE}\" --json 2>/dev/null || echo '{\"detected\":false,\"signals\":[],\"terms\":{}}')\n```\n\n> If the phase section cannot be resolved (no `ROADMAP.md` / unknown phase), the query emits `{ \"detected\": false, ... }` — the checkpoint does not fire. Do not block on it.\n>\n> Optional tuning — pass `--terms ` to replace the curated pluralization cues for this project (the `optional`/`chosen` cues keep their defaults): `gsd_run query assumption-delta scan \"${PHASE}\" --json --terms second,alternative,fallback`.\n\n## Decision branch\n\nRead `ASSUMPTION_DELTA_JSON`. Act on `detected` only — do **not** pattern-match the human prose.\n\n**If `detected` is `false`:** this phase does not change a core assumption. Skip the checkpoint entirely and continue planning. Do not raise it with the user.\n\n**If `detected` is `true`:** a core assumption may have lost its monopoly. The `signals[]` array tells you which family fired:\n\n| `kind` | What changed | The question to answer |\n|---|---|---|\n| `pluralization` | A second X was introduced where there was one (second platform / auth method / tenant / region / source of truth) | Does the current primary key / identity model still name the right noun? |\n| `optional` | A required / `only` field became optional | Is the field still the right anchor, or has the anchor moved? |\n| `chosen` | A derived value became chosen, or a constant became a parameter | Has a configuration decision become a modeling decision? |\n\nBefore finalizing the plan, answer this for the user and record the decision explicitly:\n\n> **Promote vs. add-alongside.** The usual correct move when a generalization occurs is to **promote** the new general representation to the primary and **demote** the old specific one to a detail of one variant — *not* to add the new one alongside the still-required old one. Adding alongside silently contradicts the generalized intent (a later variant that does not fit the old primary can be stored but never confirmed as a default).\n\nRecord the outcome in the PLAN.md front matter / a `` block:\n\n- The **noun** that is now primary (the generalized identity).\n- The **decision**: `promote` | `add-alongside` | `no-change`, with a one-line rationale.\n- If `add-alongside`: call it out as accepted debt and note what would force a later promote.\n\n## Optional companion: an invariant test\n\nWhen `detected` is `true`, suggest (do not require) a contract/invariant test that encodes the now-generalized intent — e.g. *\"every confirmed default round-trips through the primary use-path, for every supported variant.\"* That test goes red the instant a future phase reintroduces the singular assumption, so the regression cannot land silently. If the user accepts, add the test as a task in the plan.\n\n## Tuning the vocabulary (optional)\n\nThe trigger vocabulary is a curated, additive-only set in `gsd-core/bin/lib/assumption-delta.cjs` (`DEFAULT_ASSUMPTION_DELTA_TERMS`). Bare \"or\" is intentionally excluded — it is too common in prose and would make the gate fire constantly. To widen or narrow the cues for a project, override at the call site with `--terms ` (replaces the pluralization cues; `optional`/`chosen` keep defaults). The whole checkpoint is toggleable via `workflow.assumption_delta` in `.planning/config.json`.\n\nThis checkpoint is advisory: it informs and records; it never blocks the phase.\n" + "inline": "# Assumption-Delta Architecture Checkpoint\n\n> Advisory, non-blocking. Fires **only** when the phase scope shows a singular→plural / required→optional / derived→chosen transition. When it fires, it surfaces ONE identity-model question before the plan is finalized. Most phases will not fire it — that is the point.\n\n## Why this exists\n\nMost quietly-imported architectural debt does not come from a missing upfront design phase. It comes at the *seam*: a later phase introduces a second case (a second platform, auth method, tenant, region, source of truth) and nobody re-asks whether the original abstraction still names the right thing. The phase that adds the second case is exactly the 20-minute conversation that prevents an afternoon of later cleanup.\n\n## Run the detector\n\nThe detector is a deterministic scan over the phase scope text. It strips fenced code blocks first, so a trigger word that appears only inside a code snippet does not fire. It returns a typed result: `{ detected, signals[], terms }`. Resolve it through the `assumption-delta scan` query (same phase-section resolver as `roadmap.get-phase`):\n\n```bash\nASSUMPTION_DELTA_JSON=$(gsd_run query assumption-delta scan \"${PHASE}\" --json 2>/dev/null) || true\n[ -n \"$ASSUMPTION_DELTA_JSON\" ] || ASSUMPTION_DELTA_JSON='{\"skipped\":true,\"reason\":\"probe_unavailable\"}'\n```\n\n> If the phase section cannot be resolved (no `ROADMAP.md` / unknown phase, or a section with no body), the query emits `{ \"skipped\": true, \"reason\": \"phase_unresolved\" }` — **not** `detected:false`. A probe that never had input does not get to assert that this phase changes no core assumption. The checkpoint does not fire either way; the difference is that a skip is now distinguishable from a real negative. Do not block on it.\n>\n> Optional tuning — pass `--terms ` to replace the curated pluralization cues for this project (the `optional`/`chosen` cues keep their defaults): `gsd_run query assumption-delta scan \"${PHASE}\" --json --terms second,alternative,fallback`.\n\n## Decision branch\n\nRead `ASSUMPTION_DELTA_JSON`. Act on `detected` only — do **not** pattern-match the human prose.\n\n**If `skipped` is `true`:** the detector never examined a phase section — it could not resolve one (`phase_unresolved`) or could not run at all (`probe_unavailable`). Skip the checkpoint for this run rather than asserting a verdict about input that was never examined; do not raise it with the user. **Check for `skipped` before reading `detected`** — a skipped payload carries no `detected` key, and treating its absence as `false` re-creates the fabrication this branch exists to prevent.\n\n**If `detected` is `false`:** this phase does not change a core assumption. Skip the checkpoint entirely and continue planning. Do not raise it with the user.\n\n**If `detected` is `true`:** a core assumption may have lost its monopoly. The `signals[]` array tells you which family fired:\n\n| `kind` | What changed | The question to answer |\n|---|---|---|\n| `pluralization` | A second X was introduced where there was one (second platform / auth method / tenant / region / source of truth) | Does the current primary key / identity model still name the right noun? |\n| `optional` | A required / `only` field became optional | Is the field still the right anchor, or has the anchor moved? |\n| `chosen` | A derived value became chosen, or a constant became a parameter | Has a configuration decision become a modeling decision? |\n\nBefore finalizing the plan, answer this for the user and record the decision explicitly:\n\n> **Promote vs. add-alongside.** The usual correct move when a generalization occurs is to **promote** the new general representation to the primary and **demote** the old specific one to a detail of one variant — *not* to add the new one alongside the still-required old one. Adding alongside silently contradicts the generalized intent (a later variant that does not fit the old primary can be stored but never confirmed as a default).\n\nRecord the outcome in the PLAN.md front matter / a `` block:\n\n- The **noun** that is now primary (the generalized identity).\n- The **decision**: `promote` | `add-alongside` | `no-change`, with a one-line rationale.\n- If `add-alongside`: call it out as accepted debt and note what would force a later promote.\n\n## Optional companion: an invariant test\n\nWhen `detected` is `true`, suggest (do not require) a contract/invariant test that encodes the now-generalized intent — e.g. *\"every confirmed default round-trips through the primary use-path, for every supported variant.\"* That test goes red the instant a future phase reintroduces the singular assumption, so the regression cannot land silently. If the user accepts, add the test as a task in the plan.\n\n## Tuning the vocabulary (optional)\n\nThe trigger vocabulary is a curated, additive-only set in `gsd-core/bin/lib/assumption-delta.cjs` (`DEFAULT_ASSUMPTION_DELTA_TERMS`). Bare \"or\" is intentionally excluded — it is too common in prose and would make the gate fire constantly. To widen or narrow the cues for a project, override at the call site with `--terms ` (replaces the pluralization cues; `optional`/`chosen` keep defaults). The whole checkpoint is toggleable via `workflow.assumption_delta` in `.planning/config.json`.\n\nThis checkpoint is advisory: it informs and records; it never blocks the phase.\n" }, "produces": [], "consumes": [ diff --git a/gsd-core/references/api-coverage.md b/gsd-core/references/api-coverage.md index 7d30ba287..ecb05950d 100644 --- a/gsd-core/references/api-coverage.md +++ b/gsd-core/references/api-coverage.md @@ -53,7 +53,26 @@ compound modifiers ("Resolver-only API"), and first-party-qualified services `capabilities/ai-integration/fragments/api-coverage-plan-pre.md`. 2. **Seal time (`verify:pre`).** The blocking `api-coverage.verify-pre` gate runs `check api-coverage.verify-pre ` and blocks unless a valid - matrix exists (or no integration is detected). + matrix exists (or no integration is detected in a scope it could actually + read). + +### Seal-time outcomes + +| Condition | Verdict | +|---|---| +| Valid `COVERAGE.md` matrix | pass | +| `No external API integration: ` declaration | pass (the reasoned human overrule) | +| Malformed / partial matrix | **block** | +| No matrix, integration signal found | **block** | +| No matrix, no signal, scope was read | pass | +| No matrix, **scope could not be established** | **block** — `scope_unavailable: true` (#3909) | +| Phase token unresolvable | **block** | +| Plan file exists but unreadable | **block** | +| No `.planning/phases` tree at all | pass — not a GSD project layout | + +The last four are the fail-closed arms: the gate refuses to certify "no +external-API integration" from scope it did not read. A phase with real plans +and no API vocabulary is the fifth row, and is unaffected. ## The coverage matrix format diff --git a/src/check-command-router.cts b/src/check-command-router.cts index 77f124bc3..46ffcb9fb 100644 --- a/src/check-command-router.cts +++ b/src/check-command-router.cts @@ -1505,6 +1505,33 @@ function cmdApiCoverageVerifyPre(projectDir: string, args: string[], raw: boolea ); return; } + // An EMPTY scope is not a negative verdict. This gate's neighbouring arms + // already fail closed (unresolvable phase → block; unreadable plan → block), + // but a phase with no plan body AND no roadmap section fell through to + // detection over zero bytes and CERTIFIED "no external-API integration" — + // clearing a blocking seal gate on a probe that examined nothing + // (ADR-3889 failure class (c), #3909). The discriminator is BYTES EXAMINED, + // never SIGNALS FOUND: a phase with real plans and no API vocabulary still + // reaches the pass below unchanged. + if (scope.text.trim() === '') { + output( + { + block: true, + passed: false, + coverage_present: false, + detected: false, + scope_unavailable: true, + message: + 'api-coverage: the phase scope is empty — no plan body and no roadmap section were ' + + 'found, so nothing was examined. Refusing to certify no external-API integration ' + + 'from an unestablished scope. Add the phase plan, or add a COVERAGE.md declaration.', + }, + raw, + undefined, + ); + return; + } + const detection = detectApiIntegration(scope.text); if (detection.detected) { // Surface only verb/noun (typed, bounded) — NOT raw prose snippets — so the diff --git a/tests/api-coverage-gate-e2e.test.cjs b/tests/api-coverage-gate-e2e.test.cjs index 4cb629afb..809912728 100644 --- a/tests/api-coverage-gate-e2e.test.cjs +++ b/tests/api-coverage-gate-e2e.test.cjs @@ -16,6 +16,7 @@ const { describe, test, afterEach } = require('node:test'); const assert = require('node:assert/strict'); +const fc = require('fast-check'); const fs = require('node:fs'); const os = require('node:os'); const path = require('node:path'); @@ -369,3 +370,162 @@ describe('readPhaseScope — fail-closed on a real read failure (#2365 review)', assert.strictEqual(res.text, ''); }); }); + +// ─── Unestablished scope is not a negative verdict (#3909, ADR-3889 P5) ─────── +// The gate's two neighbouring arms already fail closed (unresolvable phase → +// block; unreadable plan → block). Certifying "no external-API integration" +// from ZERO examined bytes was the remaining fabrication: nothing was read, yet +// the blocking seal gate cleared. The discriminator is BYTES EXAMINED, never +// SIGNALS FOUND — a phase with real plans and no API vocabulary must keep +// passing exactly as it did before. +describe('api-coverage.verify-pre — unestablished scope blocks (#3909)', () => { + let tmpDir; + afterEach(() => { if (tmpDir) { cleanup(tmpDir); tmpDir = null; } }); + + function gateJson(phaseDir) { + return JSON.parse(runGate(tmpDir, phaseDir).output); + } + + test('CONTROL: a real plan with no API vocabulary still passes', () => { + tmpDir = makeProject({ api_coverage_gate: true }); + const phaseDir = makePhaseDir(tmpDir, '01-refactor'); + writePlan(phaseDir, '01-PLAN.md', '# Plan\nRefactor the internal state machine and rename fields.'); + const j = gateJson(phaseDir); + assert.strictEqual(j.block, false, 'an examined phase with no signal must still pass'); + assert.strictEqual(j.passed, true); + assert.strictEqual(j.detected, false); + assert.strictEqual( + j.scope_unavailable, + undefined, + 'an examined scope must NOT be reported as unavailable', + ); + }); + + test('a phase dir with no plans and no roadmap section BLOCKS instead of certifying', () => { + tmpDir = makeProject({ api_coverage_gate: true }); + const phaseDir = makePhaseDir(tmpDir, '02-empty'); + const j = gateJson(phaseDir); + assert.strictEqual(j.block, true, 'zero examined bytes must not clear the blocking seal gate'); + assert.strictEqual(j.passed, false); + assert.strictEqual(j.scope_unavailable, true, 'the reason must be reported, not implied'); + assert.strictEqual( + j.detected, + false, + 'detected stays false — nothing was found because nothing was examined', + ); + assert.match(j.message, /scope/i); + }); + + test('BOUNDARY: whitespace-only scope is unestablished', () => { + tmpDir = makeProject({ api_coverage_gate: true }); + const phaseDir = makePhaseDir(tmpDir, '03-ws'); + writePlan(phaseDir, '01-PLAN.md', ' \n\t\n '); + const j = gateJson(phaseDir); + assert.strictEqual(j.block, true, 'whitespace is not examined content'); + assert.strictEqual(j.scope_unavailable, true); + }); + + test('BOUNDARY: CRLF-only scope is unestablished', () => { + tmpDir = makeProject({ api_coverage_gate: true }); + const phaseDir = makePhaseDir(tmpDir, '04-crlf'); + writePlan(phaseDir, '01-PLAN.md', '\r\n\r\n'); + const j = gateJson(phaseDir); + assert.strictEqual(j.block, true, 'CRLF-only content carries no scope'); + assert.strictEqual(j.scope_unavailable, true); + }); + + test('BOUNDARY: a single non-whitespace character IS examined content', () => { + tmpDir = makeProject({ api_coverage_gate: true }); + const phaseDir = makePhaseDir(tmpDir, '05-one'); + writePlan(phaseDir, '01-PLAN.md', 'x'); + const j = gateJson(phaseDir); + assert.strictEqual(j.block, false, 'one byte of real scope is examined — normal detection applies'); + assert.strictEqual(j.scope_unavailable, undefined); + }); + + test('a COVERAGE.md matrix short-circuits before scope is ever read', () => { + tmpDir = makeProject({ api_coverage_gate: true }); + const phaseDir = makePhaseDir(tmpDir, '06-matrix'); + // No plans at all — scope would be unestablished, but the matrix decides. + writeCoverage(phaseDir, [ + '# API Coverage — Stripe', + '', + '| capability | decision | reason |', + '|---|---|---|', + '| charge | INTEGRATE | |', + ].join('\n')); + const j = gateJson(phaseDir); + assert.strictEqual(j.block, false, 'a valid matrix passes regardless of scope readability'); + assert.strictEqual(j.coverage_present, true); + assert.strictEqual(j.scope_unavailable, undefined); + }); + + test('a no-integration DECLARATION passes even with an unestablished scope', () => { + tmpDir = makeProject({ api_coverage_gate: true }); + const phaseDir = makePhaseDir(tmpDir, '07-decl'); + writeCoverage(phaseDir, 'No external API integration: pure internal refactor.\n'); + const j = gateJson(phaseDir); + assert.strictEqual(j.block, false, 'the human declaration is the reasoned overrule'); + assert.strictEqual(j.none_declared, true); + assert.strictEqual(j.scope_unavailable, undefined); + }); + + test('a project with no .planning/phases tree still fails OPEN (not a scope block)', () => { + const noPhases = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-apicov-nophases-')); + try { + fs.mkdirSync(path.join(noPhases, '.planning'), { recursive: true }); + fs.writeFileSync( + path.join(noPhases, '.planning', 'config.json'), + JSON.stringify({ workflow: { api_coverage_gate: true } }), + 'utf8', + ); + const r = runTools(['check', 'api-coverage.verify-pre', '01-x', '--raw'], noPhases); + const j = JSON.parse(r.output); + assert.strictEqual(j.block, false, 'a non-GSD layout must stay fail-open'); + assert.strictEqual(j.passed, true); + assert.strictEqual( + j.scope_unavailable, + undefined, + 'the new block must not widen to swallow the non-GSD-project pass', + ); + } finally { + cleanup(noPhases); + } + }); +}); + +// ─── Property: the scope discriminator is emptiness, nothing else (#3909) ───── +// The gate reports `scope_unavailable` exactly when the scope it read is +// whitespace-only. This pins that predicate against arbitrary input — unicode +// whitespace, CRLF, strings that merely look blank — so the discriminator can +// never quietly become "no signals found" instead of "nothing examined". +describe('api-coverage scope emptiness — properties (#3909)', () => { + let tmpDir; + afterEach(() => { if (tmpDir) { cleanup(tmpDir); tmpDir = null; } }); + + test('P1: the scope read back is whitespace-only iff the plan body was', () => { + fc.assert( + fc.property(fc.string(), (body) => { + tmpDir = makeProject({ api_coverage_gate: true }); + try { + const phaseDir = makePhaseDir(tmpDir, '01-prop'); + writePlan(phaseDir, '01-PLAN.md', body); + // No ROADMAP.md is written, so the roadmap fallback contributes + // nothing and the plan body is the entire scope. + const res = readPhaseScope(tmpDir, phaseDir, '01'); + assert.strictEqual(res.readError, null, 'a written plan file must always be readable'); + assert.strictEqual( + res.text.trim() === '', + body.trim() === '', + 'the gate blocks on an unestablished scope, so its emptiness test must agree ' + + 'with the emptiness of what the author actually wrote', + ); + } finally { + cleanup(tmpDir); + tmpDir = null; + } + }), + { numRuns: 200, seed: 3909 }, + ); + }); +}); diff --git a/tests/assumption-delta-checkpoint-e2e.test.cjs b/tests/assumption-delta-checkpoint-e2e.test.cjs index b19895d7d..18b55c0c7 100644 --- a/tests/assumption-delta-checkpoint-e2e.test.cjs +++ b/tests/assumption-delta-checkpoint-e2e.test.cjs @@ -127,11 +127,19 @@ describe('assumption-delta scan query — phase-section detection (#1561)', () = assert.deepStrictEqual(parsed.signals, []); }); - test('unknown phase → detected:false, no throw (graceful)', () => { + test('unknown phase → skipped, no throw (graceful) (#3909)', () => { tmpDir = makeRoadmapProject(); const r = scanQuery(tmpDir, 999, ['--json']); assert.ok(r.success, `scan should succeed on unknown phase. stderr: ${r.error}`); - assert.strictEqual(JSON.parse(r.output).detected, false); + const parsed = JSON.parse(r.output); + // This asserted `detected:false` until #3909. That was the fabrication + // itself pinned as intended behavior: phase 999 does not exist, so the + // detector was handed the empty string and its "no core assumption + // changed" answer described nothing. The graceful-degradation contract + // the test was protecting (succeeds, does not throw) is unchanged. + assert.strictEqual(parsed.skipped, true); + assert.strictEqual(parsed.reason, 'phase_unresolved'); + assert.strictEqual(parsed.detected, undefined); }); test('--terms override narrows the vocabulary (custom cue fires, default cue does not)', () => { @@ -226,3 +234,84 @@ describe('assumption-delta capability — plan:pre render-hooks wiring (#1561)', ); }); }); + +// ─── An unresolved phase section is not a negative verdict (#3909, P5) ──────── +// `routeAssumptionDelta` fed `detectAssumptionDelta(section ?? '')`, so a phase +// section that could not be resolved at all (no ROADMAP.md, unknown phase +// number) was scanned as the empty string and reported as an examined-and- +// negative result. That is a fabricated verdict: the probe never had input. +// The route now reports the skipped-with-reason shape instead. Exit code stays +// 0 — this is an ADR-2980 degraded result reported through the payload, and +// ADR-3889 P8 pins the gsd-tools projection at v1. +describe('query assumption-delta scan — unresolved phase reports skipped (#3909)', () => { + let tmpDir; + afterEach(() => { if (tmpDir) { cleanup(tmpDir); tmpDir = null; } }); + + function writeRoadmap(dir, body) { + fs.writeFileSync(path.join(dir, '.planning', 'ROADMAP.md'), body, 'utf8'); + } + + function scan(cwd, phase, extra = []) { + const r = runTools(['query', 'assumption-delta', 'scan', phase, '--json', ...extra], cwd); + return { raw: r, json: JSON.parse(r.output) }; + } + + test('CONTROL: a resolved section with no cues still reports detected:false', () => { + tmpDir = makeProject({ workflow: { assumption_delta: true } }); + writeRoadmap(tmpDir, '# Roadmap\n\n### Phase 01: Cleanup\n\nRefactor the internal state machine.\n'); + const { raw, json } = scan(tmpDir, '01'); + assert.strictEqual(raw.exitCode, 0); + assert.strictEqual(json.detected, false, 'an examined section with no cues is a real negative'); + assert.strictEqual(json.skipped, undefined, 'a real scan must NOT be reported as skipped'); + }); + + test('CONTROL: a resolved section with a pluralization cue still detects', () => { + tmpDir = makeProject({ workflow: { assumption_delta: true } }); + writeRoadmap(tmpDir, '# Roadmap\n\n### Phase 01: Auth\n\nAdd a second authentication method.\n'); + const { json } = scan(tmpDir, '01'); + assert.strictEqual(json.detected, true); + assert.strictEqual(json.skipped, undefined); + }); + + test('no ROADMAP.md at all reports skipped, not detected:false', () => { + tmpDir = makeProject({ workflow: { assumption_delta: true } }); + const { raw, json } = scan(tmpDir, '01'); + assert.strictEqual(json.skipped, true, 'a probe with no input must not assert a verdict'); + assert.strictEqual(json.reason, 'phase_unresolved'); + assert.strictEqual( + json.detected, + undefined, + 'a skipped payload must carry no detected key — that is what makes it distinguishable', + ); + assert.strictEqual(raw.exitCode, 0, 'degraded result stays exit 0 (ADR-2980; P8 pins v1)'); + }); + + test('a ROADMAP.md that does not contain the phase reports skipped', () => { + tmpDir = makeProject({ workflow: { assumption_delta: true } }); + writeRoadmap(tmpDir, '# Roadmap\n\n### Phase 01: Auth\n\nAdd a second authentication method.\n'); + const { json } = scan(tmpDir, '99'); + assert.strictEqual(json.skipped, true, 'an unknown phase number resolves to nothing'); + assert.strictEqual(json.reason, 'phase_unresolved'); + }); + + test('BOUNDARY: a resolved but body-less section is EXAMINED, not skipped', () => { + tmpDir = makeProject({ workflow: { assumption_delta: true } }); + writeRoadmap(tmpDir, '# Roadmap\n\n### Phase 01: Empty\n\n \n\t\n\n### Phase 02: Next\n\nBody.\n'); + const { json } = scan(tmpDir, '01'); + // The resolver returns the heading line alone for a body-less section, so + // the phase WAS found and the detector did examine what the roadmap holds + // for it. `skipped` is reserved for the cases where the resolver returns + // null — an unknown phase number, or no ROADMAP.md at all. Pinning that + // line here keeps the distinction the issue actually asks for (found vs + // not found) from drifting into a vaguer "looks empty to me". + assert.strictEqual(json.skipped, undefined, 'a resolved section is not a skip'); + assert.strictEqual(json.detected, false, 'the heading alone carries no cues — a real negative'); + }); + + test('--terms does not rescue an unresolved phase', () => { + tmpDir = makeProject({ workflow: { assumption_delta: true } }); + const { json } = scan(tmpDir, '01', ['--terms', 'second,alternative']); + assert.strictEqual(json.skipped, true, 'a vocabulary override cannot manufacture a scan'); + assert.strictEqual(json.reason, 'phase_unresolved'); + }); +}); diff --git a/tests/capability-probe-fallback.test.cjs b/tests/capability-probe-fallback.test.cjs new file mode 100644 index 000000000..b41183fd6 --- /dev/null +++ b/tests/capability-probe-fallback.test.cjs @@ -0,0 +1,290 @@ +'use strict'; + +/** + * The capability fragments' probe fallbacks must be honest (#3909, ADR-3889 P5). + * + * Two capability fragments carry a shell snippet that runs a detector and + * captures its JSON. Those snippets used `… 2>/dev/null || echo '{"detected":false}'`, + * which FABRICATES a negative verdict whenever the probe exits non-zero. That is + * wrong three separate ways, and all three are covered here: + * + * 1. `||` fires on exit 1 — which ADR-3889 P3 made the LEGITIMATE negative — + * so a correct "no integration" answer got a second object appended to it. + * 2. `$( )` captures the whole compound's stdout, so the fallback APPENDS + * rather than replaces: an honest `{"skipped":true}` was immediately + * contradicted by a fabricated `{"detected":false}` in the same string. + * 3. A probe that genuinely could not run produced a clean, confident, + * wrong `detected:false`. + * + * BEHAVIORAL, not source-grep: each test extracts the fragment's own fenced + * bash block, executes it under `bash` with the surrounding contract stubbed + * (`gsd_run`, `PHASE_DIR`, `PHASE`), and asserts on the captured variable. + */ + +const { describe, test, afterEach } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); + +const { cleanup } = require('./helpers.cjs'); +const { runNode, OUTCOME } = require('./helpers/process-seam.cjs'); +const { splitLines } = require('../gsd-core/bin/lib/text-lines.cjs'); + +const REPO_ROOT = path.join(__dirname, '..'); +const API_FRAGMENT = path.join( + REPO_ROOT, 'capabilities', 'ai-integration', 'fragments', 'api-coverage-plan-pre.md'); +const DELTA_FRAGMENT = path.join( + REPO_ROOT, 'capabilities', 'assumption-delta', 'fragments', 'plan-pre.md'); +const TOOLS_PATH = path.join(REPO_ROOT, 'gsd-core', 'bin', 'gsd-tools.cjs'); + +/** + * Pull the fragment's probe snippet out of its markdown: the first fenced + * ```bash block that assigns `varName`. The block is returned verbatim so the + * test executes exactly the bytes the planner is handed. + */ +function extractProbeBlock(fragmentPath, varName) { + const md = splitLines(fs.readFileSync(fragmentPath, 'utf8')); + const fences = []; + let current = null; + for (const line of md) { + if (current === null) { + if (line.trim() === '```bash') { + current = []; + } + continue; + } + if (line.trim() === '```') { + fences.push(current.join('\n')); + current = null; + continue; + } + current.push(line); + } + const block = fences.find((body) => body.includes(`${varName}=`)); + assert.ok( + block, + `${path.basename(fragmentPath)} must contain a fenced bash block assigning ${varName}`, + ); + return block; +} + +/** + * Run a fragment snippet under bash and return the captured variable's value. + * `prelude` stubs the surrounding workflow contract; `cwd` decides whether the + * detector module is reachable (an unreachable one is how "the probe could not + * launch" is simulated — no chmod, no monkeypatch, just a different cwd). + */ +function runSnippet({ block, varName, prelude, cwd }) { + const script = `set -u\n${prelude}\n${block}\nprintf '%s' "\${${varName}}"\n`; + const scriptFile = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-frag-')), 'run.sh'); + fs.writeFileSync(scriptFile, script, 'utf8'); + try { + const r = runNode(['-e', ` + const { spawnSync } = require('node:child_process'); + const r = spawnSync('bash', [process.argv[1]], { cwd: process.argv[2], encoding: 'utf8', timeout: 60000 }); + process.stdout.write(r.stdout || ''); + `, scriptFile, cwd], { cwd, timeoutMs: 60000 }); + assert.strictEqual(r.outcome, OUTCOME.EXITED, `snippet runner outcome: ${r.outcome}`); + return r.stdout; + } finally { + cleanup(path.dirname(scriptFile)); + } +} + +/** Assert the captured value is exactly ONE JSON object, and return it. */ +function parseSingleObject(captured, what) { + assert.notStrictEqual(captured.trim(), '', `${what}: the fragment captured nothing at all`); + let parsed; + try { + parsed = JSON.parse(captured); + } catch (err) { + assert.fail( + `${what}: the fragment produced text that is not a single JSON object — ` + + `a concatenated fallback is exactly this failure. Captured: ${JSON.stringify(captured)} ` + + `(${err.message})`, + ); + } + assert.strictEqual(typeof parsed, 'object', `${what}: payload must be an object`); + assert.notStrictEqual(parsed, null, `${what}: payload must not be null`); + return parsed; +} + +// ─── api-coverage fragment ──────────────────────────────────────────────────── + +describe('api-coverage fragment probe fallback is honest (#3909)', () => { + let tmpDir; + afterEach(() => { if (tmpDir) { cleanup(tmpDir); tmpDir = null; } }); + + // The fragment's snippet builds SCOPE from `${PHASE_DIR}/*-PLAN.md` plus a + // `gsd_run query roadmap.get-phase` call. Both are stubbed so the test + // controls exactly what reaches the detector. + function preludeFor(planBody) { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-apifrag-')); + if (planBody !== null) { + fs.writeFileSync(path.join(tmpDir, '01-PLAN.md'), planBody, 'utf8'); + } + return [ + `PHASE_DIR=${JSON.stringify(tmpDir)}`, + 'PHASE=01', + 'gsd_run() { return 0; }', + ].join('\n'); + } + + function capture(planBody, { cwd = REPO_ROOT } = {}) { + return runSnippet({ + block: extractProbeBlock(API_FRAGMENT, 'API_COVERAGE_JSON'), + varName: 'API_COVERAGE_JSON', + prelude: preludeFor(planBody), + cwd, + }); + } + + test('CONTROL: a scope with API vocabulary captures a detected verdict', () => { + const j = parseSingleObject( + capture('# Plan\nIntegrate the Stripe API and wrap its SDK.'), 'detected case'); + assert.strictEqual(j.detected, true); + assert.strictEqual(j.skipped, undefined); + }); + + test('a LEGITIMATE negative (probe exit 1) is captured as ONE valid object', () => { + // Regression: the detector exits 1 for a real negative, so `|| echo …` + // fired on the success path and appended a second object. + const j = parseSingleObject( + capture('# Plan\nRefactor the internal state machine.'), 'legit negative'); + assert.strictEqual(j.detected, false, 'a real negative must survive intact'); + assert.strictEqual(j.skipped, undefined, 'a real negative is not a skip'); + }); + + test('an HONEST skip (empty scope) is not contradicted by a fabricated verdict', () => { + const j = parseSingleObject(capture(null), 'empty scope'); + assert.strictEqual(j.skipped, true, 'an unexamined scope must report skipped'); + assert.strictEqual( + j.detected, + undefined, + 'the skip must not carry a detected key — that contradiction is the defect', + ); + }); + + test('a probe that CANNOT LAUNCH reports skipped, never detected:false', () => { + // cwd without the module → node fails, stdout empty. The fragment must not + // manufacture a verdict from that. + const away = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-nomodule-')); + try { + const j = parseSingleObject( + capture('# Plan\nIntegrate the Stripe API.', { cwd: away }), 'probe unavailable'); + assert.strictEqual(j.skipped, true, 'a probe that could not run must report skipped'); + assert.strictEqual(j.reason, 'probe_unavailable'); + assert.strictEqual( + j.detected, + undefined, + 'asserting detected:false from a probe that never ran is the bug this closes', + ); + } finally { + cleanup(away); + } + }); +}); + +// ─── assumption-delta fragment ──────────────────────────────────────────────── + +describe('assumption-delta fragment probe fallback is honest (#3909)', () => { + let tmpDir; + afterEach(() => { if (tmpDir) { cleanup(tmpDir); tmpDir = null; } }); + + function projectWithRoadmap(body) { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-deltafrag-')); + fs.mkdirSync(path.join(tmpDir, '.planning'), { recursive: true }); + if (body !== null) { + fs.writeFileSync(path.join(tmpDir, '.planning', 'ROADMAP.md'), body, 'utf8'); + } + return tmpDir; + } + + // `gsd_run` is the workflow launcher's shell function; stub it to the real + // CLI so the snippet exercises the genuine query route. + function realGsdRunPrelude(projectDir) { + return [ + 'PHASE=01', + `gsd_run() { ( cd ${JSON.stringify(projectDir)} && ` + + `node ${JSON.stringify(TOOLS_PATH)} "$@" ); }`, + ].join('\n'); + } + + function capture(prelude) { + return runSnippet({ + block: extractProbeBlock(DELTA_FRAGMENT, 'ASSUMPTION_DELTA_JSON'), + varName: 'ASSUMPTION_DELTA_JSON', + prelude, + cwd: REPO_ROOT, + }); + } + + test('CONTROL: a resolved section with a cue captures a detected verdict', () => { + const dir = projectWithRoadmap( + '# Roadmap\n\n### Phase 01: Auth\n\nAdd a second authentication method.\n'); + const j = parseSingleObject(capture(realGsdRunPrelude(dir)), 'detected case'); + assert.strictEqual(j.detected, true); + assert.strictEqual(j.skipped, undefined); + }); + + test('CONTROL: a resolved section with no cue captures ONE valid negative', () => { + const dir = projectWithRoadmap( + '# Roadmap\n\n### Phase 01: Cleanup\n\nRefactor the internal state machine.\n'); + const j = parseSingleObject(capture(realGsdRunPrelude(dir)), 'legit negative'); + assert.strictEqual(j.detected, false); + assert.strictEqual(j.skipped, undefined); + }); + + test('an unresolvable phase captures skipped, not a fabricated negative', () => { + const dir = projectWithRoadmap(null); + const j = parseSingleObject(capture(realGsdRunPrelude(dir)), 'unresolved phase'); + assert.strictEqual(j.skipped, true); + assert.strictEqual(j.detected, undefined); + }); + + test('a probe that CANNOT LAUNCH reports skipped, never detected:false', () => { + const j = parseSingleObject( + capture(['PHASE=01', 'gsd_run() { return 127; }'].join('\n')), 'probe unavailable'); + assert.strictEqual(j.skipped, true, 'a launcher that failed must not yield a verdict'); + assert.strictEqual(j.reason, 'probe_unavailable'); + assert.strictEqual(j.detected, undefined); + }); +}); + +// ─── Parity: one vocabulary across both fragments ───────────────────────────── + +describe('both fragments share one skipped-with-reason vocabulary (#3909)', () => { + // Generative-fix divergence guard: two surfaces adopting one convention must + // fail this test the moment they drift apart. + test('both fragments emit the same probe-unavailable reason token', () => { + const results = [ + { name: 'api-coverage', block: extractProbeBlock(API_FRAGMENT, 'API_COVERAGE_JSON') }, + { name: 'assumption-delta', block: extractProbeBlock(DELTA_FRAGMENT, 'ASSUMPTION_DELTA_JSON') }, + ].map(({ name, block }) => { + const varName = name === 'api-coverage' ? 'API_COVERAGE_JSON' : 'ASSUMPTION_DELTA_JSON'; + const captured = runSnippet({ + block, + varName, + // Force the unavailable path for both: no PHASE_DIR contents, and a + // launcher/cwd that cannot produce output. + prelude: [ + 'PHASE_DIR=/nonexistent-phase-dir-3909', + 'PHASE=01', + 'gsd_run() { return 127; }', + ].join('\n'), + cwd: os.tmpdir(), + }); + return { name, payload: parseSingleObject(captured, name) }; + }); + + for (const { name, payload } of results) { + assert.strictEqual(payload.skipped, true, `${name} must report skipped when the probe cannot run`); + } + assert.strictEqual( + results[0].payload.reason, + results[1].payload.reason, + 'the two fragments must not invent different reason tokens for the same condition', + ); + }); +});