Files

Tom Boucher 6addeccd19 feat(ai-integration): API-coverage verify:pre gate (#1562 )

Full API Coverage by Default — Opt Out, Never Opt In. A phase that integrates
an external API/SDK/service can no longer seal without a decided coverage matrix.

- src/api-coverage.cts: deterministic detector (compound verb+noun signal +
  <Service> API/SDK surface; stopword-guarded; strips fenced code) + matrix
  parse/validate/render with field-length caps.
- check api-coverage.verify-pre: blocking seal-time gate; phase arg resolved as
  a token under .planning/phases/ only (traversal-neutralized); validates
  COVERAGE.md or blocks iff a strong integration signal is detected and no
  matrix exists; fail-closed when phases tree exists but phase unresolvable.
- capabilities/ai-integration: workflow.api_coverage_gate config key (default
  true), plan:pre contribution, blocking verify:pre gate. Data-driven.
- gsd-core/workflows/verify-work.md: generic verify:pre gate dispatch.
- Tests: detector FP/FN + matrix validation + fast-check bijection; gate e2e.
  Code+security review findings fixed (stopword FP, scope containment, pipe/cap
  rejection, prompt-injection message hygiene).
- Regenerated registry/matrix/loop-host-contract/goldens/baseline + docs.

Closes #1562

2026-07-07 15:11:12 -04:00

4.5 KiB

Raw Blame History

API Coverage Gate (Full Coverage by Default — Opt Out, Never Opt In)

Reference for the api-coverage gate on the ai-integration capability (#1562). Config key: workflow.api_coverage_gate (default true). Gate point: verify:pre.

The problem this closes

"We integrated the API" too often silently means "we integrated whatever the first use case exercised." Every un-built capability is then an invisible hole, discovered later by a user who reasonably expected it to work. The phase sealed green because its tasks completed — nobody decided the gaps were acceptable, because nobody enumerated them.

This gate makes the API surface visible and decided before the phase can seal. Full coverage is the default starting position; the coverage matrix is the subtraction record. Every gap is an explicit, reasoned opt-out rather than a surprise.

When it fires

The gate runs at verify:pre (before /gsd:verify-work begins UAT). A phase is treated as an external-API integration when either:

a COVERAGE.md matrix is present in the phase directory (the planner produced one at plan:pre), or
the phase scope shows a strong external-API-integration signal (an integration verb co-occurring with an external-API noun, or an explicit <Service> API|SDK|REST|GraphQL surface) and no matrix yet exists.

Non-API phases (refactors, bug fixes, internal-only work, features that merely mention an existing internal API) do not fire the gate — the trigger requires a compound signal, so a bare word like "api" in "the public API of UserController" is intentionally ignored.

The two touch points

Plan time (plan:pre). A contribution to the planner prompts it to run the deterministic detector over the phase scope and, when an integration is detected, produce COVERAGE.md. See capabilities/ai-integration/fragments/api-coverage-plan-pre.md.
Seal time (verify:pre). The blocking api-coverage.verify-pre gate runs check api-coverage.verify-pre <phase-dir> and blocks unless a valid matrix exists (or no integration is detected).

The coverage matrix format

Canonical form — a markdown table (human-editable, diff-friendly):

# API Coverage — <service>

> Full coverage by default. Opt-outs are explicit, reasoned decisions.

| capability | decision | reason |
|---|---|---|
| search | INTEGRATE | |
| playlists | INTEGRATE | |
| skip | OPT-OUT | not needed yet — tracked for follow-up phase |

INTEGRATE is the default. Every capability starts as INTEGRATE.
Every OPT-OUT MUST carry a one-line reason (not needed, not needed yet, explicitly out of scope, …). An opt-out without a reason is an un-decided hole — the exact failure mode this gate exists to close.
A fenced ```coverage JSON block ([{"capability":…,"decision":…,"reason":…}]) is also accepted for machine-generated matrices.

Rules enforced at seal time: the matrix must be non-empty; every capability name must be non-empty and unique; every decision must be INTEGRATE or OPT-OUT; every OPT-OUT must have a reason. Violations block the seal with a precise error.

A second integration against the same need

A second platform for an existing capability (e.g. adding YouTube alongside Spotify for media playback) starts from the same full-coverage baseline as the first. Do not carry over the first integration's opt-outs silently — re-decide each capability for the new surface, so a first-class/fallback asymmetry cannot accumulate into a later user-facing bug.

The matrix persists

COVERAGE.md is a phase artifact. A future phase that extends the same integration starts from the recorded surface and decisions rather than from zero — the matrix is the durable subtraction record.

Tuning

Disable entirely: set workflow.api_coverage_gate: false in .planning/config.json (the gate unregisters from verify:pre).
Widen the trigger vocabulary: the detector accepts --verbs / --nouns overrides (see capabilities/ai-integration/fragments/api-coverage-plan-pre.md). The default vocabulary is additive-only.

Detector CLI

echo "$PHASE_SCOPE" | node gsd-core/bin/lib/api-coverage.cjs --json
# exit 0 = integration detected, 1 = none, 2 = startup error

The detector is a pure function (detectApiIntegration → { detected, signals, terms }) shared by the plan-time prompt and the seal-time gate, so the low-false-positive guarantee is testable rather than a judgment call.

4.5 KiB Raw Blame History