* test(#1955): failing-first contract for verifier coincidental-reliance advisory * test(#1955): anchor coincidental-reliance assertions on the frontmatter block * feat(#1955): flag coincidental reliance in goal-backward verification * chore(#1955): correct stale workflow tier high-water comment * fix(#1955): close the verify-phase divergence and state the endogeneity limit * docs(#1955): backfill changeset pr number --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
@@ -319,6 +319,9 @@ Two further dimensions carry no number: **Verify Command Format Sanity** and
|
||||
- **Test quality audit** (v1.32): verifies that tests prove what they claim by checking for disabled/skipped tests on requirements, circular test patterns (system generating its own expected values), assertion strength (existence vs. value vs. behavioral), and expected value provenance. Blockers from test quality audit override an otherwise passing verification
|
||||
- Runs the full workspace test suite at most once per verification — proves a test *exists* by enumeration and that it *passes* via a single named test, never re-running the whole suite per must-have.
|
||||
- **Behavior-dependent calibration (#966):** a must-have that asserts a state transition or a cancellation/cleanup/ordering invariant is marked `⚠️ PRESENT_BEHAVIOR_UNVERIFIED` (not `VERIFIED`) when no test exercises it — excluded from the `verified_truths` score, counted in the `behavior_unverified` frontmatter field, and routed to human verification, so a clean `N/N` certifies behavioral evidence rather than mere symbol presence.
|
||||
- **Coincidental-reliance advisory (#1955):** a truth that reaches `✓ VERIFIED` is additionally asked *why* it holds. When the recorded evidence shows the truth holding for an incidental reason — `undeclared-precondition`, `incidental-ordering`, or `fixture-only` — the verdict is qualified as `✓ VERIFIED (coincidental-reliance)` and the truth is listed in the `coincidental_reliance_items` frontmatter field with what to harden. This is **advisory**: the base `✓ VERIFIED` token is unchanged, the truth still counts toward `verified_truths`, the overall `status` is unaffected, and no human-verification item is emitted — a passing phase still passes. It classifies evidence the verifier already gathered rather than asking it to rate its own confidence — but it is honestly an **endogenous** check, and `gsd-core/references/honest-verifier.md` records that endogenous gates are measurably weaker than the exogenous `backstop` tag it routes on. Advisory status is the consequence, not a coincidence: a miss costs exactly today's behaviour (a plain `✓ VERIFIED`) and a false positive costs one line of prose, never a failed phase, so a weaker mechanism is affordable here in a way it would not be on a pass/fail axis. Its precision is unmeasured. It complements the two existing axes: `PRESENT_BEHAVIOR_UNVERIFIED` is *no* behavioral evidence, `insufficient_spec` is an under-specified truth, and this is evidence that exists and passes for the wrong reason.
|
||||
|
||||
The advisory is carried by two surfaces. `agents/gsd-verifier.md` (Step 3, sub-step 5c) holds the detection rule for the subagent path. The non-subagent path, `gsd-core/workflows/verify-phase.md`, receives it through the eager `@`-import of `gsd-core/templates/verification-report.md`, whose `## Guidelines` carry the same instruction — `verify-phase.md` itself is deliberately **not** edited, because it sits 29 bytes under the DEFAULT tier hard cap in `tests/workflow-size-budget.test.cjs` and absorbing rubric prose there requires a lazy extraction first.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -204,7 +204,7 @@ actuals:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Purpose** | Phase goal verification report. Checks `must_haves.truths`, `must_haves.artifacts`, and `must_haves.key_links` from all plans against the actual codebase after execution. Records `status: passed \| gaps_found \| human_needed`. A truth whose correctness depends on runtime behaviour — a state transition or a cancellation/cleanup/ordering invariant — is marked `⚠️ PRESENT_BEHAVIOR_UNVERIFIED` (not `VERIFIED`) when no test exercises it: it is excluded from `score`, counted in the `behavior_unverified` frontmatter field, and routed to `human_needed`, so a behaviour-dependent gap can no longer count toward a clean N/N. |
|
||||
| **Purpose** | Phase goal verification report. Checks `must_haves.truths`, `must_haves.artifacts`, and `must_haves.key_links` from all plans against the actual codebase after execution. Records `status: passed \| gaps_found \| human_needed`. A truth whose correctness depends on runtime behaviour — a state transition or a cancellation/cleanup/ordering invariant — is marked `⚠️ PRESENT_BEHAVIOR_UNVERIFIED` (not `VERIFIED`) when no test exercises it: it is excluded from `score`, counted in the `behavior_unverified` frontmatter field, and routed to `human_needed`, so a behaviour-dependent gap can no longer count toward a clean N/N. A truth that *does* reach `✓ VERIFIED` but holds for an incidental reason (`undeclared-precondition`, `incidental-ordering`, `fixture-only`) is qualified `✓ VERIFIED (coincidental-reliance)` and listed in `coincidental_reliance_items` — **advisory only**: it still counts toward `score`, does not change `status`, and emits no human-verification item (#1955). |
|
||||
| **Produced by** | `/gsd-verify-work` (or the verify step within `/gsd-execute-phase`). |
|
||||
| **Consumed by** | `plan-phase` closed-phase gate (a `status: passed` VERIFICATION.md marks the phase `Complete` and blocks replanning without `--force`); `/gsd-progress`; human review. |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user