* feat(verify-phase): honest verifier — abstain (insufficient_spec) on non-inferable backstop truths (#1154) Carry the edge-probe's existing `backstop` (non-inferable) tier through the plan-phase projection as a structured flat-scalar marker instead of a prose parenthetical, and make verify-phase abstain -> human_needed (never silent-pass) on a backstop truth it cannot confirm with explicit evidence. Truth-axis mirror of #644's prohibition judgment-tier (ADR-550 D4). Engine (deterministic, CI-tested per ADR-550 D5 — never the LLM verdict): - src/probe-core.cts: truthStatement/truthVerification normalizers, projectTruths (conservative serializer), dispositionForUnverifiableTruth (backstop+no-evidence -> unverified/flagged/insufficient_spec; backstop+evidence -> green; inferable -> green, the over-abstention guard). - src/roadmap.cts: coerceTruthToString now reads `statement` first so an object-form backstop truth is surfaced, not dropped (Hyrum backward-compat for truth-readers). Workflow/agent/docs: plan-phase emits the structured marker (flat scalar, ADR-550 #1278); verify-phase + gsd-verifier add the abstain arm; new references/honest-verifier.md; FEATURES/COMMANDS document insufficient_spec; ADR-550 amended (truth-axis D4 mirror). Decisions adopted (trek-e review): insufficient_spec feeds existing human_needed with a distinguishable reason (no new VERIFIER_STATUS); changeset Changed; round-trip parity test; abstain-on-unconfirmed-backstop regression test red-first. Implementation notes (deviations from the issue's proposed file list, verified live): - frontmatter.cts needs no change — its flat parser already round-trips object-form truths. - verify.cts needs no change — it grades artifacts/key_links structurally; truths are LLM-graded at the workflow layer, so consumption lives there + the deterministic helper. - No CJS<->SDK hand-sync — the SDK seam was retired (ADR-0174); src/*.cts is sole source. Regenerated artifacts: golden-install-parity fixtures, INVENTORY-MANIFEST, size baselines. * chore(#1154): add changeset (Changed) for honest verifier User-facing changelog fragment for #1738. Typed `Changed` (not `Added`) per trek-e review condition 3 — the verify behavior shifts for backstop-bearing specs (a confident silent `passed` becomes `human_needed`), which is user-visible even though the schema marker is additive. * docs(#1154): score-formula also excludes abstained insufficient_spec truths (review nit-1) trek-e review nit: the verify-phase score sentence said PRESENT_BEHAVIOR_UNVERIFIED truths were "the only ones excluded" from verified_truths. Post-#1154 an abstained `insufficient_spec` backstop truth is also excluded (it is not ✓ VERIFIED and routes to human_needed). Behavior was already correct; this tightens the wording. Regenerated golden-install-parity fixtures + workflow-size baseline for the touched verify-phase.md. (Nit-2 — a dedicated insufficient_spec_items frontmatter list — is intentionally not taken: the current design is ADR-550-D4-conformant, the abstain cause rides as a distinguishable report reason, and adding it would exceed the approved scope.) --------- Co-authored-by: Tom Boucher <trekkie@nomorestars.com>
This commit is contained in:
Rezolv
@@ -315,6 +315,8 @@ For browser-backed UAT, use a configured browser MCP server. The current Open GS
|
||||
|
||||
**Coverage-aware UAT routing (#1602).** When a SUMMARY.md carries a `coverage:` frontmatter block, `verify-work` classifies each deliverable deterministically instead of prompting for every prose bullet: deliverables proven by passing tests are auto-passed (recorded with `source: automated`, no prompt) and only judgment-dependent deliverables are presented for human sign-off. SUMMARYs without a `coverage:` block fall back to the previous prose-based extraction unchanged. See the [`coverage:` block reference](#summary-coverage-block) below.
|
||||
|
||||
**Honest verifier — `insufficient_spec` abstention (#1154).** A `must_haves.truths` item carrying the `verification: backstop` marker (a *non-inferable* check the edge-probe surfaced at spec time) is graded specially: if the verifier cannot confirm it with **explicit evidence** (a passing wired held-out/property-based test, or a directly-observed behavior), it **abstains** — the item is reported `unverified — held-out test recommended` and the phase verdict becomes `human_needed` (with reason `insufficient_spec`, distinct from ordinary manual-UAT `human_needed`), **never a silent `passed`**. Autonomous runs complete with "N unverified non-inferable checks" rather than hard-halting; interactive runs route the item to the end-of-phase human checkpoint. Abstention is exogenous (driven by the `backstop` tag, never a self-judged "abstain if unsure") and an inferable truth is never abstained. Reliable on capable verifier tiers (`sonnet`+); the budget `haiku` tier degrades toward current behavior. See [Honest Verifier](../gsd-core/references/honest-verifier.md).
|
||||
|
||||
#### SUMMARY `coverage:` block
|
||||
|
||||
A SUMMARY.md may carry an optional `coverage:` frontmatter block — a list of per-deliverable entries that joins requirements → tests → verification status:
|
||||
|
||||
@@ -3115,7 +3115,9 @@ When a requirement's prose matches **no** shape cue, the probe does not silently
|
||||
|
||||
The resolved edges populate a `## Edge Coverage` section in `SPEC.md`. Unresolved *applicable* edges trigger a soft gate (Resolve / Write-anyway-flagged / Keep-probing) rather than a hard block. Under `--auto`, the probe **never auto-dismisses** — it auto-covers where a defensible criterion exists, otherwise auto-backstops, and logs `[auto] edge coverage: C covered, B backstop, U unresolved`. The one exception is an `unclassified` candidate: `--auto` leaves it **`unresolved`** (surfaced as a flagged assumption), never auto-`backstop` — a missing shape is not evidence an edge exists, so minting a held-out edge obligation would be a false claim.
|
||||
|
||||
The load-bearing wire is the `plan-phase` lift: `covered` and `backstop` edges become `must_haves.truths` the verifier can check, so the section is not merely documentation.
|
||||
The load-bearing wire is the `plan-phase` lift: `covered` and `backstop` edges become `must_haves.truths` the verifier can check, so the section is not merely documentation. A `backstop` edge is lifted as a **structured non-inferable marker** (`{ statement, verification: backstop }`, a flat scalar — not a prose note), which the **honest verifier** then consumes (see below) — closing the loop the edge-probe opened.
|
||||
|
||||
**Honest verifier — abstention on non-inferable checks (#1154).** A non-inferable (`backstop`) truth is one whose correct behavior is not derivable from the spec alone, so the verifier cannot self-detect the gap and would confidently false-pass it (~100% of the time). At verify time, a `backstop` truth the verifier cannot confirm with **explicit evidence** (a passing wired held-out/property-based test, or a directly-observed behavior) **abstains** → `human_needed` with reason `insufficient_spec` (reported as `unverified — held-out test recommended`), **never a silent `passed`**. This is the verify-time, truth-axis mirror of the prohibition judgment-tier disposition (ADR-550 D4): exogenous (driven by the `backstop` tag, never a self-judged "abstain if unsure"), routing-not-diagnosis (the held-out test carries the omitted rule), and capable-tier dependent (reliable on `sonnet`+; the budget `haiku` tier degrades toward current behavior). An inferable truth is never abstained (the over-abstention guard). Reference: [Honest Verifier](../gsd-core/references/honest-verifier.md).
|
||||
|
||||
**Requirements:**
|
||||
- REQ-EDGE-01: The edge pass MUST run after the ambiguity gate and emit a `## Edge Coverage` SPEC section.
|
||||
@@ -3125,6 +3127,8 @@ The load-bearing wire is the `plan-phase` lift: `covered` and `backstop` edges b
|
||||
- REQ-EDGE-05: `--auto` MUST never auto-dismiss — auto-cover or auto-backstop only.
|
||||
- REQ-EDGE-06: `plan-phase` MUST lift `covered` criteria and `backstop` notes into `must_haves.truths`.
|
||||
- REQ-EDGE-07: A requirement whose prose matches no shape cue MUST surface an `unclassified — review manually` candidate (never silently dropped); `--auto` MUST leave it `unresolved`, never auto-`backstop`.
|
||||
- REQ-EDGE-08: `plan-phase` MUST lift a `backstop` edge into `must_haves.truths` as a structured flat-scalar marker (`{ statement, verification: backstop }`), never a prose parenthetical.
|
||||
- REQ-HONEST-01: At verify time a `backstop` truth that cannot be confirmed with explicit evidence MUST abstain → `human_needed` (reason `insufficient_spec`), never `passed`; an inferable truth MUST never be abstained (over-abstention guard); abstention MUST be exogenous (driven by the `backstop` tag, not self-judgment).
|
||||
|
||||
**Reference:** [Edge Probe](../gsd-core/references/edge-probe.md)
|
||||
|
||||
|
||||
@@ -222,6 +222,7 @@
|
||||
"gates.md",
|
||||
"git-integration.md",
|
||||
"git-planning-commit.md",
|
||||
"honest-verifier.md",
|
||||
"ios-scaffold.md",
|
||||
"loop-hook-dispatch.md",
|
||||
"mandatory-initial-read.md",
|
||||
|
||||
@@ -308,6 +308,7 @@ Full roster at `gsd-core/references/*.md`. References are shared knowledge docum
|
||||
| `domain-probes.md` | Domain-specific probing questions for discuss-phase. |
|
||||
| `edge-probe.md` | Spec-phase edge-completeness probe — 8-category edge taxonomy, shape classification, and the `requirements → checks → verifier` resolution model (Step 5.5). |
|
||||
| `prohibition-probe.md` | Spec-phase prohibition-completeness probe — the two-stage adversarial-recall → precision protocol that surfaces the unwritten *must-NOT* constraints (values/safety/ethics), with status×verification (`test`/`judgment`) tiering and canon-referral breadcrumbs (Step 5.6); second adapter of the `probe-core` resolution model. |
|
||||
| `honest-verifier.md` | Verify-time abstention on non-inferable (`backstop`) truths — the truth-axis mirror of the prohibition judgment-tier disposition (ADR-550 D4): a `backstop` truth the verifier can't confirm with explicit evidence abstains → `human_needed` (reason `insufficient_spec`), never a silent pass (#1154). |
|
||||
| `gate-prompts.md` | Gate/checkpoint prompt templates. |
|
||||
| `loop-hook-dispatch.md` | Generic dispatch contract for consuming `gsd_run loop render-hooks <point> --raw` output in any host-loop workflow — envelope shape, per-kind dispatch rules (contribution/step/gate), and liveness banner. |
|
||||
| `scout-codebase.md` | Phase-type→codebase-map selection table for discuss-phase scout step (extracted via the discuss-phase/modes progressive-disclosure split, #717). |
|
||||
|
||||
@@ -143,6 +143,22 @@ This ratifies the **deterministic SOURCE** for the test-tier `CheckDescriptor` t
|
||||
|
||||
Net effect on D3: the prohibition-item shape is extended with three optional, backward-compatible flat-scalar keys that give the test-tier locate a deterministic spec-phase source; the contract's CI-testable surface (D5) gains the projection round-trip parity (CHK-03), the fail-closed guard (CHK-06), and the byte-stable backward-compat fixture (CHK-07). The decision also lives in `src/probe-core.cts` / `src/prohibition-enforcement.cts` comments, the `verify-phase.md` / `spec-phase.md` prose, and the #1278 changeset.
|
||||
|
||||
## Addendum (2026-06-25, #1154) — honest verifier: the truth-axis disposition mirror of D4
|
||||
|
||||
This records the **truth-axis half of Decision 4** that the original ADR deliberately scoped out. D3 left `truths` untouched (no `polarity` field — that is a prohibition concern), and the Lineage note parked the N17 abstention experiment as the verify-time half of the **prohibition** judgment-tier only. D7a, however, already gives the **edge** axis an orthogonal `verification` tier (`explicit | backstop`), and `plan-phase` already lifts a `backstop` edge into `must_haves.truths`. Until now that tier was flattened to a prose parenthetical at the projection, so the verifier had nothing structured to branch on and graded a `backstop` truth `passed` like any inferable one — confidently false-passing a non-inferable check ~100% of the time (the exact "verifier reach = spec reach" failure ADR-857 names). This addendum closes that gap by giving the `backstop` **truth** tier the same abstain-and-flag disposition D4 gave the prohibition `judgment` tier — **opposite polarity (must-HAVE under-specified vs must-NOT irreducible), same never-silent-pass machinery.** It cross-references **ADR-857** (the verifier↔predicate contract this rides is core, non-toggleable substrate, graded exogenously — `:65`); this completes the already-endorsed edge branch of that rail and adds no parallel mechanism.
|
||||
|
||||
1. **The D3 truth-item shape gains an OPTIONAL flat-scalar `verification` marker.** A `must_haves.truths` item is normally a plain string (an inferable truth — today's shape, unchanged). A **non-inferable** truth MAY instead be an object item carrying `statement` + a flat scalar `verification: backstop`. The marker is additive and default-absent: a string truth, or an object with no marker, behaves byte-identically to today (Hyrum's Law backward-compat). This extends the **Decision 3 truth shape** the same way #1278 extended the prohibition shape — it does **not** add a `polarity` field (D3's "truths untouched" holds); `verification` is D7a's pre-existing orthogonal axis, now carried through to the truth projection.
|
||||
|
||||
2. **Flat scalars — NOT a nested object (load-bearing, #1278 precedent).** The marker is a flat `verification:` continuation key on the truth item, never a nested object — the shared flat `parseMustHavesBlock` round-trips it with **no parser change** (the round-trip parity test is the proof; the untouched frontmatter suite confirms `truths`/`artifacts`/`key_links`/`prohibitions` readers stay regression-free).
|
||||
|
||||
3. **Deterministic disposition + projection.** `projectTruths` (`src/probe-core.cts`) emits the flat-scalar marker ONLY for a `backstop` truth and collapses every inferable truth to a bare string (conservative serializer). `dispositionForUnverifiableTruth(truth, { evidence })` is the pure, fail-closed verdict: a `backstop` truth with no **explicit evidence** (a passing wired held-out/property-based test, or a directly-observed behavior) → `{ status: 'unverified', flagged: true, reason: 'insufficient_spec' }`, **never green**; with evidence → green; any non-`backstop` truth → green (the over-abstention guard). No LLM judgment is tested (D5) — the helper owns routing once evidence-existence is known; the LLM verifier's only job is to decide whether explicit evidence exists.
|
||||
|
||||
4. **`insufficient_spec` feeds the EXISTING `human_needed` outcome — no new verifier status (maintainer Decision 1).** Abstention reuses the locked 3-value `VERIFIER_STATUSES` (`['passed','gaps_found','human_needed']`, `src/verification.cts`) with **zero change** and no new downstream routing in `ship`/`execute-phase`. The abstain cause rides as a **distinguishable report reason** (`human_needed` + `reason: insufficient_spec`) so it is never conflated with an ordinary manual-UAT `human_needed` — asserted in a test (review condition-1 caveat). *Interactive:* the item routes to the end-of-phase human checkpoint. *Autonomous (AFK):* a prominent `unverified — held-out test recommended` flag; completion reads "complete with N unverified non-inferable checks" — never a silent pass, never a hard halt (the D4 guarantee, now on the truth axis).
|
||||
|
||||
5. **Two measured properties define the design (maintainer Decision 2; caveats to record).** *Exogenous, not endogenous:* abstention is triggered by the external `backstop` tag, never a self-judged "abstain if unsure" — endogenous abstention was measured near-useless on true blind spots (100% → 67% vs exogenous 100% → 17%; N17). *Routing, not diagnosis:* the verdict does not name the omitted rule (the held-out test carries it). **Evidence honesty:** N17 is n=27, 1 rep — **direction-finding, not powered**; the effect is large and monotone but real-world precision depends on the edge-probe's *true* `backstop` recall/precision (the experiment modeled a perfect tagger), which is why the over-abstention guard and the capable-tier requirement are load-bearing acceptance criteria. **Model-tier coupling:** abstention is reliable on the default `gsd-verifier` tier (`sonnet`+); the budget tier (`haiku`) heeds the tag only inconsistently and degrades toward current behavior — captured as a documented cost (and a test) so a tier regression is caught, not discovered in production.
|
||||
|
||||
Net effect: the truth-axis `backstop` tier gains the verify-time disposition D4 gave the prohibition judgment tier; the contract's CI-testable surface (D5) gains the truth-axis projection round-trip parity and the abstain-on-unconfirmed-backstop regression. The decision also lives in `src/probe-core.cts` comments, `gsd-core/references/honest-verifier.md`, the `plan-phase.md` / `verify-phase.md` / `agents/gsd-verifier.md` prose, and the #1154 changeset.
|
||||
|
||||
## Addendum (2026-06-22) — Alternatives considered (recall / representation / packaging side)
|
||||
|
||||
This consolidates the spec-phase-side rejected and deferred alternatives for the probe family,
|
||||
|
||||
Reference in New Issue
Block a user