feat(spec-phase): prohibition probe — surface "must-NOT" constraints (#644) (#1149)

Adds the spec-time prohibition probe (spec-phase Step 5.6) — the second adapter of the probe-core resolution model. Surfaces unwritten must-NOT constraints as negative SPEC acceptance criteria with test/judgment verification tiers; fail-closed at verify time. Per ADR-550. Closes #644.
This commit is contained in:
Rezolv
2026-06-14 21:29:11 -04:00
committed by GitHub
parent 9a4a448c27
commit 395fb519e7
29 changed files with 1503 additions and 11 deletions

View File

@@ -0,0 +1,14 @@
{
"items": [
{
"requirement_id": "R1",
"category": "values",
"status": "resolved",
"verification": "judgment",
"resolution": null,
"reason": null,
"statement": "MUST NOT use shaming, guilt, or loss-aversion streak framing (e.g. \"Don't lose your streak!\") — the reminder must encourage without penalty framing"
}
],
"coverage": { "applicable": 1, "resolved": 1, "unresolved": 0, "byVerification": { "test": 0, "judgment": 1 } }
}

View File

@@ -0,0 +1,4 @@
{
"items": [],
"coverage": { "applicable": 0, "resolved": 0, "unresolved": 0, "byVerification": { "test": 0, "judgment": 0 } }
}

View File

@@ -0,0 +1,32 @@
{
"items": [
{
"requirement_id": "R1",
"category": "fairness",
"status": "resolved",
"verification": "judgment",
"resolution": null,
"reason": null,
"statement": "MUST NOT use protected attributes (race, gender, age, national origin) or their proxies (zip code, name) in the loan decision or rate"
},
{
"requirement_id": "R1",
"category": "privacy",
"status": "resolved",
"verification": "test",
"resolution": null,
"reason": null,
"statement": "MUST NOT store raw PII / financial secrets (SSN, full account or card numbers) in plaintext in the audit log"
},
{
"requirement_id": "R1",
"category": "transparency",
"status": "resolved",
"verification": "judgment",
"resolution": null,
"reason": null,
"statement": "MUST NOT mislead or omit the true rate/APR/terms in the explanation; an adverse decision must state the real principal reason (adverse-action)"
}
],
"coverage": { "applicable": 3, "resolved": 3, "unresolved": 0, "byVerification": { "test": 1, "judgment": 2 } }
}

View File

@@ -0,0 +1,248 @@
# Prohibition-Probe — Spec-Completeness Must-NOT Reference
Shared reference for the spec/requirements phase. Companion to
`@~/.claude/gsd-core/references/edge-probe.md`: `edge-probe` reaches the
**data/behavior-shape axis** (boundaries, adjacency, encoding, ordering) — the things a
feature must *do*. This reference reaches the orthogonal **must-NOT axis** (product, values,
safety, ethics) — the things a feature must *never silently become*. The edge-probe caught
0/8 of these in controlled testing because it is the wrong instrument: a shape taxonomy
cannot surface "the reminder must not shame the user." Walk each requirement through the
two-stage recall→precision protocol below and resolve each surfaced prohibition to exactly
one state.
This doc is written in generic `requirements → checks → verifier` terms with no
tool-specific vocabulary, so it is portable: copy it into any spec/requirements process.
A short mapping table at the end binds it to common host structures.
## Why front-of-pipeline
A goal-backward verifier only checks assertions that exist; an assertion only exists for a
requirement that was written down. The class of constraint this probe targets — the
*"must-NOT"* the author assumed but never wrote — is invisible to the verifier in exactly
the same way an omitted edge is, but with a sharper failure mode: a `✅ done` that means
"the code matches the words in the spec" can still ship a feature that does what the author
explicitly would *not* want. The manipulative-streak reminder, the loan model that proxies
on zip code, the audit log that stores raw SSN — each one passes a literal spec while
violating the intent. The fix is not a better verifier; it is **spec completeness**: surface
the omitted prohibition into an explicit, checkable acceptance criterion *before* any code
exists, after which the verifier reliably enforces it.
The technique is adversarial elicitation, not deterministic computation. Unlike the edge
taxonomy (a closed eight categories a classifier can apply), the recall stage is inherently
model-driven: it asks an open question and reads prose. There is **no compiled
`prohibition-probe.cjs` engine** — the recall stage is an LLM prose pass, and only the
schema/projection layer is real code (ADR-550 Decision 7b). Building a deterministic
recall adapter would be the scope-creep the maintainer flags.
## Inputs
A list of requirements, each a `{ id, text }` record where `text` is a testable statement.
There is no shape override and no taxonomy classifier — the recall stage reads the prose
directly and the precision stage filters its raw output. The probe runs **after** the
edge-probe in the spec phase, over the same requirement list.
## Two-stage protocol (recall → precision)
The probe is a two-pass pipeline per requirement. Stage 1 maximizes recall (cast wide);
Stage 2 restores precision (drop the noise). Running them in this order — wide then narrow —
is what keeps the surfaced list both complete and short.
**Stage 1 — Recall (adversarial probe).** Ask the single adversarial question of each
requirement:
> *What could this feature silently become that the author would NOT want, but the spec
> does not forbid?*
This question is model-robust (17/17 holistic surfacing including smaller models in the N18
experiment). It deliberately over-produces: ~10 raw candidates per requirement, including
routine engineering items. That over-production is intentional — recall first.
**Stage 2 — Precision (one-pass classifier).** Filter the raw Stage-1 list in a single pass.
The rule is a drop/keep split:
- **DROP routine-engineering items** — anything that is a normal correctness or hygiene
concern rather than an intent constraint: "must not mutate its input", "must not throw on
empty list", "must return a primitive not an object", "must not leak a file handle". These
belong to the edge-probe or to ordinary code review, not here.
- **KEEP values / safety / ethics items** — anything that, if violated, makes the feature do
something the author would object to on product, fairness, privacy, transparency, or
safety grounds: "must not use shaming framing", "must not proxy on protected attributes",
"must not store raw PII in plaintext".
This collapses the raw ~10 to ~2–3 genuine prohibitions (GT 5/5, 0 false positives on the
N18 eight-spec battery). A requirement that yields zero kept prohibitions emits an empty
list — that is the correct precision outcome for a pure utility, not a failure.
## Canon-referral (do not mint canon items)
Some kept candidates are not bespoke at all — they are **canon** security/compliance
constraints that a dedicated tool already owns. Do NOT mint a prohibition for them. Instead
emit a one-line breadcrumb and stop:
- OWASP / prototype-pollution / path-traversal / injection → breadcrumb to `/gsd:secure-phase`
and `eslint` (security plugins), not a minted prohibition.
- GDPR / data-retention / consent → breadcrumb to `/gsd:secure-phase`.
- Generic fairness/bias canon → breadcrumb to `/gsd:secure-phase`.
The breadcrumb reads like: *"prototype-pollution is canon — covered by /gsd:secure-phase +
eslint; not minted here."* This keeps the surfaced list to the ~2–3 **bespoke** items that
no other tool would catch — the manipulative-framing prohibition, the product-specific
fairness constraint — which is the whole value of the probe. Minting canon items both
duplicates other tooling and drowns the bespoke signal (ADR-550 Decision 6).
## Resolution states
Each surfaced prohibition carries two orthogonal axes — a resolution **lifecycle** and, when
resolved, a **verification** tier (ADR-550 Decision 7, the shared probe-core model; the
lifecycle is identical to the edge-probe, the verification tiers differ):
- **status** — `resolved | dismissed | unresolved` (IDENTICAL to the edge-probe):
- **resolved** — the prohibition is addressed; *how* it is addressed is the verification tier.
- **dismissed** — not a genuine prohibition for this feature, accompanied by a required,
non-empty reason string. "N/A — this utility has no user-facing surface, no values
constraint applies" is valid; silence is not. The reason string is the audit trail.
- **unresolved** — carried forward and flagged; the author chose not to resolve it yet.
- **verification** (only when `status` is `resolved`; `null` otherwise) — `test | judgment`
(this REPLACES the edge-probe's `explicit | backstop`):
- **test** — the prohibition can be mechanically checked (a negative test, a lint rule, an
assertion that the audit log contains no raw SSN). A checkable assertion exists.
- **judgment** — the prohibition is real but cannot be reduced to a mechanical test (a
human/LLM judgment that the framing is not manipulative). It records intent and routes
to a judgment-based review rather than a green/red test.
Splitting these axes keeps the lifecycle enum free of a verification fact and lets the
prohibition adapter declare `test | judgment` without forking the shared lifecycle enum that
the edge-probe's `explicit | backstop` also uses.
## Output schema
The probe emits, per kept prohibition, an item of the form:
```
{ requirement_id, category, status, verification, resolution, reason, statement }
```
where `statement` is the must-NOT sentence and `category` is the values/safety/ethics class
(`values`, `fairness`, `privacy`, `transparency`, `safety`, …), plus a coverage summary:
```
coverage: { applicable, resolved, unresolved, byVerification: { test, judgment } }
```
`applicable` is the number of kept prohibitions, `resolved` = closed (`resolved` +
`dismissed`) status items, `unresolved` is the remainder, and `byVerification` breaks the
`resolved`-status items down by tier (`{ test, judgment }`). This JSON is the stable contract
both the reference implementation and any third-party port emit.
## Generic mapping (requirements → checks → verifier)
| Host structure | "requirement" | a `resolved`/`test` prohibition becomes | a `resolved`/`judgment` prohibition becomes |
|----------------|---------------|------------------------------------------|----------------------------------------------|
| GSD SPEC | a SPEC Requirement | a SPEC acceptance criterion (marked prohibition) that `plan-phase` lifts into `must_haves.prohibitions` | a `must_haves.prohibitions` item routed to judgment review |
| Gherkin feature | a Scenario | a negative `Then` assertion / tagged negative scenario | a tagged scenario routed to manual review |
| OpenAPI operation | an operation | a contract test asserting the forbidden behavior never occurs | a documented constraint flagged for review |
| Docstring contract | a documented behavior | a negative assertion in the contract test | a documented must-NOT for reviewers |
The portable invariant: a `resolved`/`test` prohibition produces **a checkable negative the
verifier iterates over** (GSD: a `must_haves.prohibitions` item with a test); a
`resolved`/`judgment` prohibition produces a recorded intent routed to judgment review. An
`unresolved` prohibition is an explicit assumption the downstream planner must surface, not
silently drop.
## Worked example (streak-reminder)
A single requirement to send a daily habit reminder. The edge-probe sees a `stateful`
requirement and asks about idempotency; the prohibition-probe asks the adversarial question
and surfaces what the reminder must never *become*. Stage 1 over-produces ("must not spam",
"must not throw on a deleted habit", "must not use shaming framing"); Stage 2 drops the
routine-engineering items and keeps the one genuine values prohibition:
```json prohibition-probe:01-streak-reminder/expected.json
{
"items": [
{
"requirement_id": "R1",
"category": "values",
"status": "resolved",
"verification": "judgment",
"resolution": null,
"reason": null,
"statement": "MUST NOT use shaming, guilt, or loss-aversion streak framing (e.g. \"Don't lose your streak!\") — the reminder must encourage without penalty framing"
}
],
"coverage": { "applicable": 1, "resolved": 1, "unresolved": 0, "byVerification": { "test": 0, "judgment": 1 } }
}
```
The kept prohibition is `judgment`-tier: "manipulative framing" cannot be reduced to a
mechanical test, so it records intent and routes to judgment review — but it is now an
explicit acceptance criterion the spec must clear, not an unwritten assumption.
## Worked example (clean-utility)
A pure utility requirement — "deduplicate a list of integers" — has no user-facing surface,
no values/safety/ethics dimension. Stage 1 still over-produces ("must not mutate the input",
"must not change order"), but every candidate is routine engineering that Stage 2 drops (and
the edge-probe already owns). The correct precision outcome is an empty prohibition list — a
zero, not a false positive:
```json prohibition-probe:02-clean-utility/expected.json
{
"items": [],
"coverage": { "applicable": 0, "resolved": 0, "unresolved": 0, "byVerification": { "test": 0, "judgment": 0 } }
}
```
This is the precision discipline that keeps the probe from crying wolf: a utility with no
intent surface produces zero prohibitions, so a non-empty list always carries signal.
## Worked example (multi-prohibition)
A loan-decision requirement is the high-stakes case: it surfaces several distinct
prohibitions across categories. Stage 1 produces a long list including canon items
(prototype-pollution, generic GDPR retention) that canon-referral breadcrumbs out; Stage 2
keeps the three bespoke values/safety items — a `fairness` constraint, a `privacy` constraint
(`test`-tier, mechanically checkable against the audit log), and a `transparency` constraint:
```json prohibition-probe:03-multi-prohibition/expected.json
{
"items": [
{
"requirement_id": "R1",
"category": "fairness",
"status": "resolved",
"verification": "judgment",
"resolution": null,
"reason": null,
"statement": "MUST NOT use protected attributes (race, gender, age, national origin) or their proxies (zip code, name) in the loan decision or rate"
},
{
"requirement_id": "R1",
"category": "privacy",
"status": "resolved",
"verification": "test",
"resolution": null,
"reason": null,
"statement": "MUST NOT store raw PII / financial secrets (SSN, full account or card numbers) in plaintext in the audit log"
},
{
"requirement_id": "R1",
"category": "transparency",
"status": "resolved",
"verification": "judgment",
"resolution": null,
"reason": null,
"statement": "MUST NOT mislead or omit the true rate/APR/terms in the explanation; an adverse decision must state the real principal reason (adverse-action)"
}
],
"coverage": { "applicable": 3, "resolved": 3, "unresolved": 0, "byVerification": { "test": 1, "judgment": 2 } }
}
```
The `privacy` row is `test`-tier — "no raw SSN in the audit log" is a mechanical assertion —
while `fairness` and `transparency` are `judgment`-tier. The byVerification rollup
`{ test: 1, judgment: 2 }` is the count-preserved breakdown of the three `resolved`-status
items. Each worked-example block above is kept byte-for-byte (parsed-JSON) identical to its
fixture under `gsd-core/references/prohibition-probe-fixtures/` by
`tests/prohibition-probe.docs-fixtures.test.cjs`, so the doc and the reference data cannot
silently drift.