feat(#1154): honest verifier — abstain (insufficient_spec) on non-inferable backstop truths (#1738)

* feat(verify-phase): honest verifier — abstain (insufficient_spec) on non-inferable backstop truths (#1154)

Carry the edge-probe's existing `backstop` (non-inferable) tier through the
plan-phase projection as a structured flat-scalar marker instead of a prose
parenthetical, and make verify-phase abstain -> human_needed (never silent-pass)
on a backstop truth it cannot confirm with explicit evidence. Truth-axis mirror
of #644's prohibition judgment-tier (ADR-550 D4).

Engine (deterministic, CI-tested per ADR-550 D5 — never the LLM verdict):
- src/probe-core.cts: truthStatement/truthVerification normalizers, projectTruths
  (conservative serializer), dispositionForUnverifiableTruth (backstop+no-evidence
  -> unverified/flagged/insufficient_spec; backstop+evidence -> green; inferable
  -> green, the over-abstention guard).
- src/roadmap.cts: coerceTruthToString now reads `statement` first so an object-form
  backstop truth is surfaced, not dropped (Hyrum backward-compat for truth-readers).

Workflow/agent/docs: plan-phase emits the structured marker (flat scalar, ADR-550
#1278); verify-phase + gsd-verifier add the abstain arm; new references/honest-verifier.md;
FEATURES/COMMANDS document insufficient_spec; ADR-550 amended (truth-axis D4 mirror).

Decisions adopted (trek-e review): insufficient_spec feeds existing human_needed with a
distinguishable reason (no new VERIFIER_STATUS); changeset Changed; round-trip parity
test; abstain-on-unconfirmed-backstop regression test red-first.

Implementation notes (deviations from the issue's proposed file list, verified live):
- frontmatter.cts needs no change — its flat parser already round-trips object-form truths.
- verify.cts needs no change — it grades artifacts/key_links structurally; truths are
  LLM-graded at the workflow layer, so consumption lives there + the deterministic helper.
- No CJS<->SDK hand-sync — the SDK seam was retired (ADR-0174); src/*.cts is sole source.

Regenerated artifacts: golden-install-parity fixtures, INVENTORY-MANIFEST, size baselines.

* chore(#1154): add changeset (Changed) for honest verifier

User-facing changelog fragment for #1738. Typed `Changed` (not `Added`) per
trek-e review condition 3 — the verify behavior shifts for backstop-bearing specs
(a confident silent `passed` becomes `human_needed`), which is user-visible even
though the schema marker is additive.

* docs(#1154): score-formula also excludes abstained insufficient_spec truths (review nit-1)

trek-e review nit: the verify-phase score sentence said PRESENT_BEHAVIOR_UNVERIFIED
truths were "the only ones excluded" from verified_truths. Post-#1154 an abstained
`insufficient_spec` backstop truth is also excluded (it is not ✓ VERIFIED and routes
to human_needed). Behavior was already correct; this tightens the wording.
Regenerated golden-install-parity fixtures + workflow-size baseline for the touched
verify-phase.md. (Nit-2 — a dedicated insufficient_spec_items frontmatter list — is
intentionally not taken: the current design is ADR-550-D4-conformant, the abstain
cause rides as a distinguishable report reason, and adding it would exceed the
approved scope.)

---------

Co-authored-by: Tom Boucher <trekkie@nomorestars.com>

gsd-core/references/honest-verifier.md

@@ -0,0 +1,105 @@
+# Honest Verifier — Abstention on Non-Inferable Checks
+
+Shared reference for the **verify** phase. The verify-time companion to the spec-time
+`@~/.claude/gsd-core/references/edge-probe.md` (which *classifies* non-inferable checks) and
+`@~/.claude/gsd-core/references/prohibition-probe.md` (whose judgment-tier disposition this mirrors).
+This doc is written in generic `spec → predicate → verifier` terms with no tool-specific vocabulary,
+so it is portable: copy it into any verification process.
+
+## The problem it solves
+
+A verifier is trustworthy on **inferable** checks — defects determined by the stated spec. On a
+**non-inferable** check the correct answer is *not derivable from the spec alone* (e.g. "does `[1,2]`
+touching `[2,3]` merge?", "is a 'character' a grapheme or a code unit?"). On these the verifier *does
+not know that it does not know*: measured behavior is a **confident PASS on the blind-spot check ~100%
+of the time** (mean confidence ~0.93), because a model cannot self-detect a gap it does not perceive.
+
+The edge-probe already detects these at spec time and tags them `verification: backstop` (ADR-550
+D7a). The honest verifier consumes that tag so the verifier **abstains** instead of confidently
+false-passing — converting a silent false-pass (the worst failure: you don't know to look) into an
+explicit, actionable "write a held-out test." Measured: the confident-false-pass rate on the blind
+spot drops **100% → 17%** (N17).
+
+## The two properties that define the design
+
+1. **Exogenous, not endogenous.** The trigger is the *external tag* (`backstop`), never the verifier's
+   self-judgment. Asking the verifier to "abstain if unsure" barely moves the number (100% → 67%) and
+   only on ambiguity it already notices; on a true blind spot it stays confidently wrong. A confidence
+   gate cannot reach a blind spot the model does not feel — so there is **no "are you sure?" prompt**;
+   routing is on the pre-existing tag only.
+2. **Routing, not diagnosis.** The verifier need not name the omitted rule (if it could, it wouldn't
+   be a blind spot). In testing, verifiers abstained correctly while citing the *wrong* edge. The
+   honest verdict requires only "I was told this is under-specified and I cannot rule it out." The
+   omitted rule is carried by a human-authored held-out test, not by the verifier.
+
+## The disposition (the protocol)
+
+For each `must_haves.truths` item:
+
+| Item | Confirmable with explicit evidence? | Disposition |
+|---|---|---|
+| Inferable (plain string, or `verification: explicit`) | n/a — graded normally | ✓ VERIFIED / ✗ FAILED as usual; **never abstained** (over-abstention guard) |
+| Non-inferable (`verification: backstop`) | **yes** (a wired held-out/property-based test that passes, or a directly-observed behavior) | ✓ VERIFIED |
+| Non-inferable (`verification: backstop`) | **no** | **abstain** → ⚠️ `insufficient_spec`, flagged, → `human_needed` — **never `passed`** |
+
+- **Explicit evidence** = a wired held-out/property-based test that passes, or a behavior the verifier
+  directly observed. Symbol presence + wiring is **not** explicit evidence for a non-inferable truth.
+- **Never silent, never a hard halt.** *Interactive:* the abstained item routes to the end-of-phase
+  human checkpoint. *Autonomous (AFK):* it produces a prominent `unverified — held-out test
+  recommended` flag and the completion line reads "complete with N unverified non-inferable checks";
+  the run neither silently passes the blind spot nor hard-halts.
+- **Distinguishable reason.** The abstain disposition carries `reason: insufficient_spec` so the
+  `human_needed` outcome is never conflated with an ordinary manual-UAT `human_needed`.
+
+This is the verify-time half of ADR-550 Decision 4 (the never-silent-pass disposition), applied to the
+edge `backstop` truth tier instead of the prohibition judgment tier — the same machinery, opposite
+polarity (must-HAVE under-specified vs must-NOT irreducible).
+
+## Deterministic engine surface
+
+The CI-testable surface is the **deterministic disposition + projection**, never the LLM's judgment
+(ADR-550 D5 — a test asserting the model's verdict is vacuous and rejected). In `probe-core`:
+
+- `truthStatement(t)` / `truthVerification(t)` — normalizers; read a truth's statement and tier from
+  either the plain-string or object form (a truth-reader MUST normalize, never assume a string).
+- `projectTruths(items)` — conservative serializer: a `backstop` truth → flat-scalar object
+  `{ statement, verification: backstop }`; every inferable truth → a bare string.
+- `dispositionForUnverifiableTruth(truth, { evidence })` → `{ status, flagged, tier, reason }`:
+  `backstop` + no evidence → `unverified`/`flagged`/`insufficient_spec`; `backstop` + evidence →
+  `green`; non-`backstop` → `green` (over-abstention guard).
+
+## Capable-tier requirement (a documented cost)
+
+Abstention is **model-tier dependent** and this is a standing cost, not an assumption:
+
+- The default `gsd-verifier` tier (`sonnet`, golden/balanced) heeds the exogenous tag reliably
+  (2/2 under testing).
+- The **budget tier (`haiku`)** is the least flag-responsive (1/2, inconsistent) and **degrades toward
+  current behavior** (confident false-pass). Run honest-verifier on a capable tier; treat the budget
+  tier as best-effort. Re-validate when the `gsd-verifier` model tier changes or a new budget model is
+  adopted (captured as a test so a tier regression is caught, not discovered in production).
+
+## Evidence and scope (stated honestly)
+
+- **Evidence strength.** N17 is n=27 verdicts (3 models × 3 conditions × 3 tasks), 1 rep —
+  **direction-finding, not powered.** The blind-spot effect is large and monotone
+  (100% → 67% → 17%); the two costs are clean single events (a *false* tag made the strongest model
+  over-abstain on a real spec-determined bug; the weakest tier was flag-deaf) and they name exactly
+  the failure modes the over-abstention guard and the capable-tier requirement defend against.
+- **Tag-precision coupling.** Quality is bounded by the edge-probe's `backstop` recall/precision — a
+  false non-inferable flag causes over-abstention. Positive coupling: improving the probe (#1110)
+  improves this for free. It adds no independent burden.
+- **Explicit non-goals.** Does NOT identify the omitted rule; does NOT recalibrate decisive verdicts;
+  does NOT defend against *malicious compliance* (a self-graded review rationalizing away its own
+  findings). It raises the floor on *honest* uncertainty about non-inferable checks — that is the
+  whole claim.
+
+## Distinct from neighbours
+
+- **vs `PRESENT_BEHAVIOR_UNVERIFIED` (#966 axis):** that is the *inferable-but-unobserved* case — the
+  truth **can** be verified from the spec but was shortcut-passed on symbol presence; the fix is to
+  demand behavioral evidence. Honest-verifier is the *non-inferable* case — the truth **cannot** be
+  verified from the spec at all; the fix is to abstain and route to a held-out test. Orthogonal axes
+  (insufficient *evidence* vs insufficient *spec*); both feed the same `human_needed` sink.
+- **vs prohibition judgment-tier (#644):** that disposes **must-NOT** constraints; honest-verifier
+  disposes **non-inferable positive truths**. Opposite polarity, same never-silent disposition.

gsd-core/workflows/plan-phase.md

@@ -912,7 +912,7 @@ Output consumed by /gsd:execute-phase. Plans need:
 - Tasks in XML format with read_first and acceptance_criteria fields (MANDATORY on every task)
 - Verification criteria
 - must_haves for goal-backward verification
-- If the SPEC has an `## Edge Coverage` section, lift every `covered` edge's acceptance criterion into `must_haves.truths`, and every `backstop` edge into `must_haves.truths` as a non-inferable check (note it needs a held-out/property-based test). `unresolved` edges are explicit assumptions — surface them in the plan, do not silently drop them.
+- If the SPEC has an `## Edge Coverage` section, lift every `covered` edge's acceptance criterion into `must_haves.truths` as a plain string, and every `backstop` edge **as a structured flat-scalar marker** — an object item `{ statement: <the check>, verification: backstop }`, NOT a prose note (the verifier branches deterministically on the `verification: backstop` field; a parenthetical is unparseable — the #1110 fragility). Use a flat scalar `verification:` continuation key, never a nested object (ADR-550 #1278). At verify time a `backstop` truth the verifier cannot confirm with explicit evidence abstains → `human_needed` (reason `insufficient_spec`), never a silent pass (#1154; see `references/honest-verifier.md`). `unresolved` edges are explicit assumptions — surface them in the plan, do not silently drop them.
 - If the SPEC has a `## Prohibitions` section, lift every resolved prohibition into the `must_haves.prohibitions:` sibling block (NOT `truths` — ADR-550 D3) carrying `statement` + `status` + `verification`; unresolved prohibitions are explicit assumptions — surface them in the plan, do not silently drop them. A prohibition is a must-NOT (negative) check that belongs in its own `must_haves.prohibitions` block. Never place a must-NOT under `must_haves.truths` — that block keeps positive-observable semantics only.
 - **"Artifacts this phase produces" section (MANDATORY)** — list every symbol this phase creates: decorators, classes, functions, CLI flags, struct/dataclass fields, new file paths. The plan-review-convergence source-grounding pass reads this section to exclude newly-created symbols from drift verification; omitting it causes new symbols to be flagged for acknowledgement.
 </downstream_consumer>

gsd-core/workflows/verify-phase.md

@@ -117,6 +117,8 @@ For each truth: identify supporting artifacts → check artifact status → chec
 
 **Behavior-dependent truths:** when a truth asserts a state transition or a cancellation/cleanup/ordering invariant, symbol presence + wiring is necessary but not sufficient — the code can be present and wired yet still leak state on the path the invariant covers. Mark such a truth ✓ VERIFIED only when a pre-existing test exercises the transition/invariant and passes (one named test, never the full suite); otherwise mark it ⚠️ PRESENT_BEHAVIOR_UNVERIFIED, emit a human-verification item, and exclude it from the verified score.
 
+**Non-inferable (`backstop`) truths (#1154):** a `must_haves.truths` item in object form `{ statement, verification: backstop }` is non-inferable — the correct behavior is not derivable from the spec alone, so the verifier cannot self-detect the gap and would false-pass it confidently. Branch on the `verification: backstop` field (read via `truthVerification()`, never prose): if confirmable with **explicit evidence** (a passing wired held-out/property test, or a directly-observed behavior) → ✓ VERIFIED; otherwise **abstain** — mark ⚠️ `insufficient_spec`, emit an `unverified — held-out test recommended` human-verification item, exclude from the verified score (routes to `human_needed`). Exogenous only (never a self-judged "abstain if unsure"); an inferable truth is never abstained. See `references/honest-verifier.md`.
+
 **Example:** Truth "User can see existing messages" depends on Chat.tsx (renders), /api/chat GET (provides), Message model (schema). If Chat.tsx is a stub or API returns hardcoded [] → FAILED. If all exist, are substantive, and connected → VERIFIED.
 </step>
 
@@ -488,17 +490,22 @@ Classify status using this decision tree IN ORDER (most restrictive first):
    - **judgment-tier, autonomous run** (non-authoritative LLM-judge verdict): emit the `unverified-prohibition — human review recommended` flag and classify → **human_needed** (autonomous completion reads "complete with N flagged prohibitions"; never a silent pass, never a hard halt).
    - **judgment-tier, interactive run**: route to the end-of-phase human checkpoint → **human_needed**.
 
-3. IF the previous step produced ANY human verification items — this includes every ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truth:
+2b. IF any `must_haves.truths` item carries the `verification: backstop` marker (#1154 — the verify-time truth-axis mirror of ADR-550 D4) AND the verifier cannot confirm it with **explicit evidence** (a wired held-out/property-based test that PASSES, or a directly-observed behavior — i.e. `dispositionForUnverifiableTruth()` returns `status: 'unverified'`, `flagged: true`, `reason: 'insufficient_spec'`):
+   - **abstain → human_needed**, NEVER `passed` and never silently graded green. Emit a prominent `unverified — held-out test recommended` flag carrying the distinguishable `reason: insufficient_spec` (so it is not conflated with ordinary manual-UAT `human_needed`).
+   - *Autonomous run:* record it and continue — completion reads "complete with N unverified non-inferable checks"; never a hard halt of an AFK run. *Interactive run:* route to the end-of-phase human checkpoint.
+   - **Exogenous only:** abstention fires SOLELY on the `backstop` tag, never a self-judged "abstain if unsure" (N17). An **inferable** truth is NEVER abstained (over-abstention guard); a `backstop` truth WITH a passing wired held-out test reaches **passed**. Reliable on capable tiers (`sonnet`+); the budget `haiku` tier degrades — see `references/honest-verifier.md`.
+
+3. IF the previous step produced ANY human verification items — this includes every ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truth and every abstained `insufficient_spec` backstop truth:
    → **human_needed** (even if all other truths VERIFIED)
 
-4. IF all checks pass AND no human verification items AND no flagged prohibitions:
+4. IF all checks pass AND no human verification items AND no flagged prohibitions AND no abstained (`insufficient_spec`) truths:
    → **passed**
 
-**passed is ONLY valid when no human verification items AND no flagged prohibitions exist.** A prohibition (must-NOT) can never be silently absorbed into a `passed` verdict — that is the core failure mode ADR-550 D4 forbids.
+**passed is ONLY valid when no human verification items, no flagged prohibitions, AND no abstained `insufficient_spec` truths exist.** Neither a prohibition (must-NOT) nor an unconfirmable non-inferable truth can ever be silently absorbed into a `passed` verdict — that is the core failure mode ADR-550 D4 forbids (now closed on both the prohibition and truth axes).
 
 A ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truth is never FAILED and never VERIFIED: it does not trigger gaps_found (the code is present and wired) and is not counted as verified (its runtime behavior was not exercised). It routes through the existing human_needed sink — no new overall status.
 
-**Score:** `verified_truths / total_truths` — `verified_truths` counts ✓ VERIFIED truths plus PASSED (override) truths; ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truths are the only ones excluded, reported separately as the `behavior_unverified` count. A headline N/N therefore certifies behavioral evidence for every behavior-dependent truth, not merely symbol presence.
+**Score:** `verified_truths / total_truths` — `verified_truths` counts ✓ VERIFIED truths plus PASSED (override) truths; excluded are ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truths (the `behavior_unverified` count) and abstained ⚠️ `insufficient_spec` backstop truths (#1154) — both are not ✓ VERIFIED and both route to `human_needed`. A headline N/N therefore certifies behavioral evidence for every behavior-dependent truth and explicit evidence for every non-inferable one, not merely symbol presence.
 </step>
 
 <step name="filter_deferred_items">