03b7125293
* test(#3909): failing-first suite for the fabricated probe fallbacks Binds the four fabrication sites found by executing the surfaces (ADR-3889 failure class (c)), each with a positive control so an over-firing fix goes red: - the blocking api-coverage.verify-pre gate certifying "no external-API integration" from a zero-byte phase scope - the assumption-delta query route scanning an unresolvable phase section as the empty string and reporting it as an examined negative - both capability fragments' probe fallbacks, which append a fabricated verdict rather than replacing, and fire on the legitimate exit-1 negative Verification runs on the remote runner. Refs #3909 * enhance(#3909): a probe that could not run no longer asserts a verdict ADR-3889 Phase 5. Four sites turned a failed or unexamined probe into a confident negative; each now reports what it could not establish. - check api-coverage.verify-pre: a phase with no plan body and no roadmap section ran detection over zero bytes and PASSED the blocking seal gate, certifying "no external-API integration" from input it never read. It now holds with scope_unavailable. The discriminator is bytes examined, never signals found, so a phase whose plans are real and simply carry no API vocabulary passes exactly as before. - query assumption-delta scan: an unresolvable phase section was scanned as the empty string and reported as an examined negative. It now returns {skipped, reason: phase_unresolved}, still at exit 0 — an ADR-2980 degraded result in the payload, leaving the gsd-tools exit projection to P8. - both capability fragments: `|| echo '{"detected":false}'` appended rather than replaced, and fired on the legitimate exit-1 negative, so a correct answer and an honest skip both arrived as two concatenated objects. They now keep the probe's own payload and manufacture only an explicit probe_unavailable skip when the probe produced nothing at all. Every registered outcome is more restrictive on a blocking gate, so this can turn a false green red and never a red green. Docs: FEATURES 156, CONFIGURATION (both keys), references/api-coverage.md seal-time outcome table, and a new how-to for the reason-code vocabulary. Verification runs on the remote runner. Closes #3909 * test(#3909): correct the stale unknown-phase assertion `unknown phase → detected:false, no throw (graceful)` scanned phase 999 against a two-phase roadmap and asserted `detected === false`. That pinned the fabrication as intended behavior: the phase does not exist, so the detector was handed the empty string and its "no core assumption changed" answer described nothing that was ever read. It now asserts the skipped-with-reason shape. The graceful-degradation contract the test was actually protecting — the query succeeds and does not throw on an unknown phase — is unchanged. Found by code review, not by the author. Refs #3909 * docs(#3909): author the FEATURES entry in its generator source `docs/FEATURES.md` is generated by `scripts/gen-features.cjs` from the per-feature fragments in `docs/features/`. The API-coverage entry was edited in the generated file, so the next regeneration silently dropped it. The text now lives in `docs/features/api-coverage-gate.md` and `docs/FEATURES.md` is regenerated from it, leaving the shipped file byte-identical and its content actually derivable. Caught by `lint:generated-sync`. Refs #3909 * test(#3909): bind the skip to "not found", and pin the discriminator The first verification run went red on one case, and the case was wrong rather than the code. `getRoadmapPhaseWithFallback` returns `null` for an unknown phase and for a missing ROADMAP.md, but for a section whose body is whitespace-only it returns the heading line alone — which is not empty. So a body-less section WAS found, and reporting `detected:false` over its heading is a real negative, not a fabrication. The test had assumed the resolver yielded `''` there. Correcting the test rather than the resolver keeps `skipped` bound to the distinction the issue asks for — found versus not found — and avoids diverging `assumption-delta scan` from `roadmap.get-phase`, which the fragment documents as sharing one resolver. Also adds the seeded property the test matrix had promised: for any plan body, the scope read back is whitespace-only exactly when the body was. That pins the gate's discriminator to bytes examined, so it cannot quietly become "no signals found", across unicode whitespace and CRLF. `docs/INVENTORY.md` picks up the reference doc's new seal-time outcome table — surfaced by the co-change gate, not by a lint failure. Refs #3909 * chore(#3909): backfill the changeset PR number Refs #3909 --------- Co-authored-by: sim <sim@local>
140 lines
6.4 KiB
Markdown
140 lines
6.4 KiB
Markdown
# API Coverage Decision Checkpoint
|
|
|
|
> Full API Coverage by Default — Opt Out, Never Opt In. Fires when a phase
|
|
> integrates an external API / SDK / service. Most non-API phases will not fire
|
|
> it — that is the point.
|
|
|
|
## Why this exists
|
|
|
|
"We integrated the API" too often silently means "we integrated whatever the
|
|
first use case exercised." Every un-built capability is then an invisible hole,
|
|
discovered later by a user who reasonably expected it to work. The phase sealed
|
|
green because its tasks completed; nobody decided the gaps were acceptable,
|
|
because nobody enumerated them. This checkpoint makes the surface **visible and
|
|
decided** before the phase can seal.
|
|
|
|
## Detect whether this phase integrates an external API
|
|
|
|
The detector is a deterministic scan over the phase scope. It strips fenced
|
|
code blocks first, so a trigger term inside a code snippet does not fire. It
|
|
returns a typed result: `{ detected, signals[], terms }`. Run it on the phase
|
|
scope (the concatenation of this phase's ROADMAP section + the PLAN body):
|
|
|
|
```bash
|
|
SCOPE="$(cat "${PHASE_DIR}"/*-PLAN.md 2>/dev/null) $(gsd_run query roadmap.get-phase "${PHASE}" 2>/dev/null || true)"
|
|
API_COVERAGE_JSON=$(printf '%s' "$SCOPE" | node gsd-core/bin/lib/api-coverage.cjs --json 2>/dev/null) || true
|
|
[ -n "$API_COVERAGE_JSON" ] || API_COVERAGE_JSON='{"skipped":true,"reason":"probe_unavailable"}'
|
|
```
|
|
|
|
The `|| true` neutralizes the assignment's status without discarding the
|
|
detector's own payload: the detector exits **1** for a real "no integration"
|
|
verdict, so treating any non-zero exit as failure would throw away a correct
|
|
answer. Emptiness — not exit status — is what proves the probe never ran, and
|
|
the second line is the only place the fragment manufactures a payload of its
|
|
own — one that records the *absence* of a verdict rather than asserting one.
|
|
|
|
The detector's exit code and `--json` payload now distinguish a real negative
|
|
from an unexamined input (ADR-3889 Phase 3, #3907): empty/whitespace-only
|
|
`$SCOPE` or a stdin read failure emit `{"skipped":true,"reason":"no_input"|
|
|
"stdin_error"}` — no `detected` key at all. **Check for `skipped` before
|
|
reading `detected`**: a `skipped` payload is not a confirmed "no API
|
|
integration" verdict, it means the detector never examined real input. Do not
|
|
treat it as `detected:false`. Read `API_COVERAGE_JSON.detected` only when
|
|
`skipped` is absent — act on it only, do **not** pattern-match the prose
|
|
yourself.
|
|
|
|
**If `skipped` is `true`:** the detector could not establish a scope (empty
|
|
`$SCOPE`) or failed to run (stdin read error). Skip the checkpoint for this
|
|
run rather than asserting a verdict about input that was never examined; do
|
|
not raise it with the user.
|
|
|
|
**If `detected` is `false`:** this phase does not integrate an external API. Skip
|
|
the checkpoint entirely and continue planning. Do not raise it with the user.
|
|
|
|
**If `detected` is `true`:** an external-API integration is in scope. You MUST
|
|
produce a **coverage matrix** before the plan is finalized.
|
|
|
|
**If `detected` is `true` but the phase genuinely integrates no external API**
|
|
(the detector is deterministic, not infallible — confirm by re-reading the phase
|
|
scope, not by preference): do NOT fabricate a matrix row for a capability that
|
|
does not exist. Write a reasoned declaration to `${PHASE_DIR}/COVERAGE.md`
|
|
instead:
|
|
|
|
```markdown
|
|
No external API integration: <one-line reason — what the phase touches instead>.
|
|
```
|
|
|
|
The reason is required, exactly like an `OPT-OUT` reason. The seal-time gate
|
|
accepts this declaration in place of a matrix.
|
|
|
|
## Produce the coverage matrix
|
|
|
|
Enumerate the external API's full **capability surface** — the verb/endpoint/method
|
|
list (e.g. for a music service: `search`, `play`, `pause`, `skip`, `set_volume`,
|
|
`get_playlist`, `create_playlist`, `add_to_playlist`, …). For each capability
|
|
record a decision, starting from **full coverage** as the default:
|
|
|
|
| capability | decision | reason |
|
|
|---|---|---|
|
|
| `<capability-id>` | `INTEGRATE` \| `OPT-OUT` | `<one-line reason if OPT-OUT>` |
|
|
|
|
Rules:
|
|
|
|
- **`INTEGRATE` is the default.** Every capability starts as INTEGRATE; the
|
|
matrix is the *subtraction record*.
|
|
- **Every `OPT-OUT` MUST carry a one-line reason** (`not needed`, `not needed
|
|
yet`, `explicitly out of scope`, …). An opt-out without a reason is an
|
|
un-decided hole — the exact failure mode this gate exists to close.
|
|
- **A second integration against the same need** (e.g. a second platform for the
|
|
same capability) starts from the **same full-coverage baseline** as the first.
|
|
Do not carry over the first integration's opt-outs silently — re-decide each
|
|
capability for the new surface, so a first-class/fallback asymmetry cannot
|
|
accumulate.
|
|
|
|
Write the matrix to `${PHASE_DIR}/COVERAGE.md` (canonical markdown-table form):
|
|
|
|
```markdown
|
|
# API Coverage — <service>
|
|
|
|
> Full coverage by default. Opt-outs are explicit, reasoned decisions.
|
|
|
|
| capability | decision | reason |
|
|
|---|---|---|
|
|
| search | INTEGRATE | |
|
|
| playlists | INTEGRATE | |
|
|
| skip | OPT-OUT | not needed yet — tracked for follow-up phase |
|
|
```
|
|
|
|
A fenced ` ```coverage ` JSON block is also accepted for machine-generated
|
|
matrices; the markdown table is preferred (human-editable, diff-friendly).
|
|
|
|
## The seal-time gate
|
|
|
|
This checkpoint is enforced. At `verify:pre` the `api-coverage.verify-pre` gate
|
|
runs `check api-coverage.verify-pre <phase-dir>`:
|
|
|
|
- If `COVERAGE.md` exists, it is validated — every row needs a valid decision and
|
|
every `OPT-OUT` a reason. A malformed/partial matrix **blocks the seal**. A
|
|
reasoned `No external API integration: …` declaration (and no rows) passes.
|
|
- If `COVERAGE.md` is absent, the detector runs again over the phase scope. If a
|
|
strong external-API-integration signal is found, the seal is **blocked** until a
|
|
matrix is produced. If no signal is found, the phase is treated as a non-API
|
|
phase and the seal proceeds.
|
|
|
|
So: an API-integrating phase cannot seal without a decided matrix. Produce it at
|
|
plan time; do not leave it for seal time.
|
|
|
|
## Tuning the vocabulary (optional)
|
|
|
|
The trigger vocabulary is a curated, additive-only set in
|
|
`gsd-core/bin/lib/api-coverage.cjs` (`DEFAULT_API_COVERAGE_TERMS`). To widen it
|
|
for a project, override at the call site:
|
|
|
|
```bash
|
|
printf '%s' "$SCOPE" | node gsd-core/bin/lib/api-coverage.cjs --json \
|
|
--verbs integrate,wrap,connect,embed --nouns api,sdk,rest,grpc,webhook,plugin
|
|
```
|
|
|
|
The whole checkpoint is toggleable via `workflow.api_coverage_gate` in
|
|
`.planning/config.json`.
|