* enh(#966): gate behavior-dependent truths on behavioral evidence in gsd-verifier Introduce a per-truth PRESENT_BEHAVIOR_UNVERIFIED state for must-haves that assert a state transition or a cancellation/cleanup/ordering invariant whose only evidence is symbol presence + wiring. Such truths are excluded from the verified_truths score, reported as a behavior_unverified count, recorded in an always-on behavior_unverified_items frontmatter list, and routed to the existing human_needed sink — so a clean N/N can no longer be reached on symbol presence alone. The overall-status vocabulary and the src/verification.cts seam are unchanged (the new state is per-truth only); gaps_found keeps decision-tree precedence; override-passed truths still count toward verified_truths. Mirrors the calibration into the shipped verify-phase.md workflow (with an infra/foundation carve-out), the VERIFICATION.md templates, and docs (planning-artifacts.md, AGENTS.md). gsd-verifier.md kept under its 48KB LARGE cap; size baselines regenerated. Closes #966 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(#966): add changeset fragment for gsd-verifier behavior-unverified calibration Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
5
.changeset/tidy-lemurs-travel.md
Normal file
5
.changeset/tidy-lemurs-travel.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
type: Changed
|
||||
pr: 1271
|
||||
---
|
||||
**`gsd-verifier` no longer marks behavior-dependent must-haves `VERIFIED` on symbol presence alone** — a truth that asserts a state transition or a cancellation/cleanup/ordering invariant is marked `PRESENT_BEHAVIOR_UNVERIFIED` when no test exercises it: excluded from the `verified_truths` score, reported as a `behavior_unverified` count, and routed to human verification, so a clean N/N now certifies behavioral evidence rather than mere symbol presence. (#966)
|
||||
@@ -180,21 +180,28 @@ For each truth, determine if codebase enables it.
|
||||
|
||||
**Verification status:**
|
||||
|
||||
- ✓ VERIFIED: All supporting artifacts pass all checks
|
||||
- ✓ VERIFIED: All supporting artifacts pass all checks — and, for a behavior-dependent truth, a behavioral test exercises the asserted behavior (see below)
|
||||
- ⚠️ PRESENT_BEHAVIOR_UNVERIFIED: Supporting artifacts are present and wired, but the truth asserts runtime behavior that no test exercises — present, not behaviorally proven. Routes to human verification (Step 8) and does NOT count toward the verified score (Step 9).
|
||||
- ✗ FAILED: One or more artifacts missing, stub, or unwired
|
||||
- ? UNCERTAIN: Can't verify programmatically (needs human)
|
||||
|
||||
**Behavior-dependent truths.** A truth is *behavior-dependent* when its correctness hinges on runtime behavior grep/presence checks cannot see — a **state transition** or a **cancellation / cleanup / ordering invariant** (e.g. "cancels the in-flight task and bumps the generation counter", "resets the busy flag on abort", "rolls back on failure"). For these, symbol presence + wiring is *necessary but not sufficient*: the code can be present and wired yet still leak state on the very path the invariant covers.
|
||||
|
||||
For each truth:
|
||||
|
||||
1. Identify supporting artifacts
|
||||
2. Check artifact status (Step 4)
|
||||
3. Check wiring status (Step 5)
|
||||
4. **Before marking FAIL:** Check for override (Step 3b)
|
||||
5. Determine truth status
|
||||
4. **Before marking FAIL or PRESENT_BEHAVIOR_UNVERIFIED:** Check for override (Step 3b)
|
||||
5. **Classify behavior-dependence.** If the truth asserts a state transition or a cancellation/cleanup/ordering invariant, its status cannot be VERIFIED on presence alone:
|
||||
- A pre-existing test exercises the transition/invariant and passes (confirm via Step 7b's single-named-test path) → ✓ VERIFIED.
|
||||
- No such test exists, or it can't run without a server/state mutation → ⚠️ PRESENT_BEHAVIOR_UNVERIFIED. Emit a human-verification item (Step 8) and do not count it toward the verified score (Step 9).
|
||||
- An accepted override (Step 3b) carries the truth as PASSED (override), exactly as it does for a FAILED truth.
|
||||
6. Determine truth status
|
||||
|
||||
## Step 3b: Check Verification Overrides
|
||||
|
||||
Before marking any must-have as FAILED, check the VERIFICATION.md frontmatter for an `overrides:` entry that matches this must-have.
|
||||
Before marking any must-have as FAILED or ⚠️ PRESENT_BEHAVIOR_UNVERIFIED, check the VERIFICATION.md frontmatter for an `overrides:` entry that matches this must-have.
|
||||
|
||||
**Override check procedure:**
|
||||
|
||||
@@ -204,12 +211,12 @@ Before marking any must-have as FAILED, check the VERIFICATION.md frontmatter fo
|
||||
4. Key technical terms (file paths, component names, API endpoints) have higher weight
|
||||
|
||||
**If override found:**
|
||||
- Mark as `PASSED (override)` instead of FAIL
|
||||
- Mark as `PASSED (override)` instead of FAIL/PRESENT_BEHAVIOR_UNVERIFIED
|
||||
- Evidence: `Override: {reason} — accepted by {accepted_by} on {accepted_at}`
|
||||
- Count toward passing score, not failing score
|
||||
- Count toward passing score (`verified_truths`), not failing score
|
||||
|
||||
**If no override found:**
|
||||
- Mark as FAILED as normal
|
||||
- Mark as FAILED (or ⚠️ PRESENT_BEHAVIOR_UNVERIFIED, per Step 3 step 5) as normal
|
||||
- Consider suggesting an override if the failure looks intentional (alternative implementation exists)
|
||||
|
||||
**Suggesting overrides:** When a must-have FAILs but evidence shows an alternative implementation that achieves the same intent, include an override suggestion in the report:
|
||||
@@ -461,6 +468,8 @@ Anti-pattern scanning (Step 7) checks for code smells. Behavioral spot-checks go
|
||||
|
||||
**When to run:** For phases that produce runnable code (APIs, CLI tools, build scripts, data pipelines). Skip for documentation-only or config-only phases.
|
||||
|
||||
**Behavioral evidence for behavior-dependent truths (Step 3).** When a truth asserts a state transition or a cancellation/cleanup/ordering invariant, the single named test below is what upgrades it from ⚠️ PRESENT_BEHAVIOR_UNVERIFIED to ✓ VERIFIED. Run only the one named test that exercises the transition/invariant — never the full suite (per #25/#753). If no such test exists, leave the truth ⚠️ PRESENT_BEHAVIOR_UNVERIFIED and route it to human verification (Step 8); do not mark it VERIFIED on presence.
|
||||
|
||||
**How:**
|
||||
|
||||
1. **Identify checkable behaviors** from must-haves truths. Select 2-4 that can be tested with a single command:
|
||||
@@ -548,6 +557,8 @@ done
|
||||
|
||||
**Needs human if uncertain:** Complex wiring grep can't trace, dynamic state behavior, edge cases.
|
||||
|
||||
**Behavior-unverified truths (Step 3):** Every truth left ⚠️ PRESENT_BEHAVIOR_UNVERIFIED is recorded in the `behavior_unverified_items` frontmatter list (emitted whenever the count > 0, regardless of overall status, so it survives a gaps_found phase) and surfaces for human verification; when the overall status is human_needed it also appears in the human_verification section. Phrase each item around the invariant: what to trigger, what state must hold afterward, and why presence checks can't see it.
|
||||
|
||||
**Harvest deferred items from PLAN.md (#3309 / `workflow.human_verify_mode = end-of-phase`):** Scan every PLAN file in the phase for `<verify><human-check>` blocks on `auto` tasks. These are verification items the planner deliberately deferred from `checkpoint:human-verify` to end-of-phase to avoid the executor cold-start cost. Each block has the same shape used by the planner:
|
||||
|
||||
```xml
|
||||
@@ -579,18 +590,30 @@ Classify status using this decision tree IN ORDER (most restrictive first):
|
||||
1. IF any truth FAILED, artifact MISSING/STUB, key link NOT_WIRED, or blocker anti-pattern found:
|
||||
→ **status: gaps_found**
|
||||
|
||||
2. IF Step 8 produced ANY human verification items (section is non-empty):
|
||||
2. IF Step 8 produced ANY human verification items (section is non-empty) — this includes every ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truth from Step 3:
|
||||
→ **status: human_needed**
|
||||
(Even if all truths are VERIFIED and score is N/N — human items take priority)
|
||||
(Even if all other truths are VERIFIED — human items take priority)
|
||||
|
||||
3. IF all truths VERIFIED, all artifacts pass, all links WIRED, no blockers, AND no human verification items:
|
||||
→ **status: passed**
|
||||
|
||||
**passed is ONLY valid when the human verification section is empty.** If you identified items requiring human testing in Step 8, status MUST be human_needed.
|
||||
**passed is ONLY valid when the human verification section is empty.** If Step 8 produced any items — including any truth left ⚠️ PRESENT_BEHAVIOR_UNVERIFIED — the status is not `passed`: it is `human_needed`, or `gaps_found` when rule 1 also fires (the ordered tree keeps gaps_found's precedence).
|
||||
|
||||
**A ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truth is never FAILED and never VERIFIED.** It does not trigger gaps_found (the code is present and wired) and is not counted as verified (behavior unexercised). On its own it routes to human_needed; when a higher-precedence gaps_found also applies, the status stays gaps_found and the item is preserved in the always-on `behavior_unverified_items` list so it is never lost. Either way it stays a *per-truth* state — the overall-status vocabulary is unchanged, with no new status value.
|
||||
|
||||
> **Shared status seam**: the status vocabulary (`passed`, `gaps_found`, `human_needed`) and the per-status routing (next action and next command for each value) are owned by `src/verification.cts` via `gsd_run query verification.status`. This agent is the single emitter of the frontmatter status field; consumers (ship.md, execute-phase.md) read routing from that query instead of re-deriving it.
|
||||
|
||||
**Score:** `verified_truths / total_truths`
|
||||
**Score (presence- vs behavior-verified split):**
|
||||
|
||||
- `verified_truths` counts ✓ VERIFIED truths plus PASSED (override) truths (Step 3b). For a behavior-dependent truth, VERIFIED means a behavioral test passed, not just that symbols are present.
|
||||
- ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truths are the *only* ones excluded from `verified_truths`; they are reported separately as `behavior_unverified`.
|
||||
|
||||
```text
|
||||
score: verified_truths / total_truths # e.g. 6/7
|
||||
behavior_unverified: P # truths present + wired but behavior not exercised
|
||||
```
|
||||
|
||||
A headline N/N therefore certifies that every behavior-dependent truth had behavioral evidence — a clean score can no longer be reached on symbol presence alone.
|
||||
|
||||
## Step 9b: Filter Deferred Items
|
||||
|
||||
@@ -699,6 +722,7 @@ phase: XX-name
|
||||
verified: YYYY-MM-DDTHH:MM:SSZ
|
||||
status: passed | gaps_found | human_needed
|
||||
score: N/M must-haves verified
|
||||
behavior_unverified: 0 # Count of ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truths (present + wired, behavior not exercised); each is detailed in behavior_unverified_items below (and in human_verification when status is human_needed)
|
||||
overrides_applied: 0 # Count of PASSED (override) items included in score
|
||||
overrides: # Only if overrides exist — carried forward or newly added
|
||||
- must_have: "Must-have text that was overridden"
|
||||
@@ -725,6 +749,11 @@ deferred: # Only if deferred items exist (Step 9b)
|
||||
- truth: "Observable truth addressed in a later phase"
|
||||
addressed_in: "Phase N"
|
||||
evidence: "Matching goal or success criteria text"
|
||||
behavior_unverified_items: # Only if behavior_unverified > 0 — emitted regardless of overall status, so these survive a gaps_found phase
|
||||
- truth: "Observable truth whose state transition or cancellation/cleanup/ordering invariant no test exercises"
|
||||
test: "What to trigger"
|
||||
expected: "What state must hold afterward"
|
||||
why_human: "Why presence checks can't see it"
|
||||
human_verification: # Only if status: human_needed
|
||||
- test: "What to do"
|
||||
expected: "What should happen"
|
||||
@@ -746,8 +775,9 @@ human_verification: # Only if status: human_needed
|
||||
| --- | ------- | ---------- | -------------- |
|
||||
| 1 | {truth} | ✓ VERIFIED | {evidence} |
|
||||
| 2 | {truth} | ✗ FAILED | {what's wrong} |
|
||||
| 3 | {truth} | ⚠️ PRESENT_BEHAVIOR_UNVERIFIED | {present + wired; no test exercises the transition/invariant — see Human Verification} |
|
||||
|
||||
**Score:** {N}/{M} truths verified
|
||||
**Score:** {N}/{M} truths verified ({P} present, behavior-unverified)
|
||||
|
||||
### Deferred Items
|
||||
|
||||
@@ -834,7 +864,7 @@ Structured gaps in VERIFICATION.md frontmatter for `/gsd:plan-phase --gaps`.
|
||||
|
||||
{If human_needed:}
|
||||
### Human Verification Required
|
||||
{N} items need human testing:
|
||||
{N} items need human testing (including {P} present-but-behavior-unverified truths — code wired, transition/invariant not exercised by a test):
|
||||
1. **{Test name}** — {what to do}
|
||||
- Expected: {what should happen}
|
||||
|
||||
@@ -857,6 +887,8 @@ Automated checks passed. Awaiting human verification.
|
||||
|
||||
**Keep verification fast.** Use grep/file checks, not running the app.
|
||||
|
||||
**Presence is not behavior.** Grep/file checks prove a symbol is present and wired — they do not prove a state transition or a cancellation/cleanup/ordering invariant holds at runtime. For a behavior-dependent truth, require a passing behavioral test (Step 7b's single named test) or mark it ⚠️ PRESENT_BEHAVIOR_UNVERIFIED and route to human verification. Never let symbol presence alone produce a VERIFIED on a behavior-dependent truth.
|
||||
|
||||
**DO NOT commit.** Leave committing to the orchestrator.
|
||||
|
||||
</critical_rules>
|
||||
|
||||
@@ -300,6 +300,7 @@ GSD uses a multi-agent architecture where thin orchestrators (workflow files) sp
|
||||
- Milestone scope filtering: gaps addressed in later phases are marked as "deferred", not reported as failures (v1.32)
|
||||
- **Test quality audit** (v1.32): verifies that tests prove what they claim by checking for disabled/skipped tests on requirements, circular test patterns (system generating its own expected values), assertion strength (existence vs. value vs. behavioral), and expected value provenance. Blockers from test quality audit override an otherwise passing verification
|
||||
- Runs the full workspace test suite at most once per verification — proves a test *exists* by enumeration and that it *passes* via a single named test, never re-running the whole suite per must-have.
|
||||
- **Behavior-dependent calibration (#966):** a must-have that asserts a state transition or a cancellation/cleanup/ordering invariant is marked `⚠️ PRESENT_BEHAVIOR_UNVERIFIED` (not `VERIFIED`) when no test exercises it — excluded from the `verified_truths` score, counted in the `behavior_unverified` frontmatter field, and routed to human verification, so a clean `N/N` certifies behavioral evidence rather than mere symbol presence.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -183,7 +183,7 @@ See [PLAN.md schema](plan-md.md) for the full field reference.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Purpose** | Phase goal verification report. Checks `must_haves.truths`, `must_haves.artifacts`, and `must_haves.key_links` from all plans against the actual codebase after execution. Records `status: passed | gaps_found | human_needed`. |
|
||||
| **Purpose** | Phase goal verification report. Checks `must_haves.truths`, `must_haves.artifacts`, and `must_haves.key_links` from all plans against the actual codebase after execution. Records `status: passed \| gaps_found \| human_needed`. A truth whose correctness depends on runtime behaviour — a state transition or a cancellation/cleanup/ordering invariant — is marked `⚠️ PRESENT_BEHAVIOR_UNVERIFIED` (not `VERIFIED`) when no test exercises it: it is excluded from `score`, counted in the `behavior_unverified` frontmatter field, and routed to `human_needed`, so a behaviour-dependent gap can no longer count toward a clean N/N. |
|
||||
| **Produced by** | `/gsd-verify-work` (or the verify step within `/gsd-execute-phase`). |
|
||||
| **Consumed by** | `plan-phase` closed-phase gate (a `status: passed` VERIFICATION.md marks the phase `Complete` and blocks replanning without `--force`); `/gsd-progress`; human review. |
|
||||
|
||||
|
||||
@@ -12,6 +12,12 @@ phase: XX-name
|
||||
verified: YYYY-MM-DDTHH:MM:SSZ
|
||||
status: passed | gaps_found | human_needed
|
||||
score: N/M must-haves verified
|
||||
behavior_unverified: 0 # Count of ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truths (present + wired, behavior not exercised)
|
||||
behavior_unverified_items: # Only if behavior_unverified > 0 — the truths above as structured items; emitted regardless of overall status
|
||||
- truth: "Observable truth whose state transition or cancellation/cleanup/ordering invariant no test exercises"
|
||||
test: "What to trigger"
|
||||
expected: "What state must hold afterward"
|
||||
why_human: "Why presence checks can't see it"
|
||||
---
|
||||
|
||||
# Phase {X}: {Name} Verification Report
|
||||
@@ -28,9 +34,10 @@ score: N/M must-haves verified
|
||||
|---|-------|--------|----------|
|
||||
| 1 | {truth from must_haves} | ✓ VERIFIED | {what confirmed it} |
|
||||
| 2 | {truth from must_haves} | ✗ FAILED | {what's wrong} |
|
||||
| 3 | {truth from must_haves} | ? UNCERTAIN | {why can't verify} |
|
||||
| 3 | {truth from must_haves} | ⚠️ PRESENT_BEHAVIOR_UNVERIFIED | {present + wired; transition/invariant not exercised by a test — see Human Verification} |
|
||||
| 4 | {truth from must_haves} | ? UNCERTAIN | {why can't verify} |
|
||||
|
||||
**Score:** {N}/{M} truths verified
|
||||
**Score:** {N}/{M} truths verified ({P} present, behavior-unverified)
|
||||
|
||||
### Required Artifacts
|
||||
|
||||
@@ -161,11 +168,17 @@ None — all verifiable items checked programmatically.
|
||||
|
||||
## Guidelines
|
||||
|
||||
**Status values:**
|
||||
**Status values (overall, frontmatter `status:`):**
|
||||
- `passed` — All must-haves verified, no blockers
|
||||
- `gaps_found` — One or more critical gaps found
|
||||
- `human_needed` — Automated checks pass but human verification required
|
||||
|
||||
**Per-truth states (Observable Truths `Status` column):**
|
||||
- `✓ VERIFIED` — supporting artifacts pass all checks; for a behavior-dependent truth, a behavioral test exercised the asserted behavior
|
||||
- `⚠️ PRESENT_BEHAVIOR_UNVERIFIED` — present + wired, but a state transition or cancellation/cleanup/ordering invariant was not exercised by any test. Counts toward `behavior_unverified`, routes to human verification, and is *excluded* from the verified score. Per-truth only — on its own the overall `status:` becomes `human_needed` (unless a higher-precedence `gaps_found` also applies); the item is preserved in `behavior_unverified_items` regardless.
|
||||
- `✗ FAILED` — artifact missing, stub, or unwired
|
||||
- `? UNCERTAIN` — can't verify programmatically
|
||||
|
||||
**Evidence types:**
|
||||
- For EXISTS: "File at path, exports X"
|
||||
- For SUBSTANTIVE: "N lines, has patterns X, Y, Z"
|
||||
|
||||
@@ -101,10 +101,12 @@ If no must_haves in frontmatter AND no Success Criteria in ROADMAP:
|
||||
<step name="verify_truths">
|
||||
For each observable truth, determine if the codebase enables it.
|
||||
|
||||
**Status:** ✓ VERIFIED (all supporting artifacts pass) | ✗ FAILED (artifact missing/stub/unwired) | ? UNCERTAIN (needs human)
|
||||
**Status:** ✓ VERIFIED (all supporting artifacts pass — and, for a behavior-dependent truth, a behavioral test exercises the asserted behavior) | ⚠️ PRESENT_BEHAVIOR_UNVERIFIED (present + wired, but a state transition or cancellation/cleanup/ordering invariant is exercised by no test — routes to human verification, excluded from the score) | ✗ FAILED (artifact missing/stub/unwired) | ? UNCERTAIN (needs human)
|
||||
|
||||
For each truth: identify supporting artifacts → check artifact status → check wiring → determine truth status.
|
||||
|
||||
**Behavior-dependent truths:** when a truth asserts a state transition or a cancellation/cleanup/ordering invariant, symbol presence + wiring is necessary but not sufficient — the code can be present and wired yet still leak state on the path the invariant covers. Mark such a truth ✓ VERIFIED only when a pre-existing test exercises the transition/invariant and passes (one named test, never the full suite); otherwise mark it ⚠️ PRESENT_BEHAVIOR_UNVERIFIED, emit a human-verification item, and exclude it from the verified score.
|
||||
|
||||
**Example:** Truth "User can see existing messages" depends on Chat.tsx (renders), /api/chat GET (provides), Message model (schema). If Chat.tsx is a stub or API returns hardcoded [] → FAILED. If all exist, are substantive, and connected → VERIFIED.
|
||||
</step>
|
||||
|
||||
@@ -440,13 +442,14 @@ Infrastructure and foundation phases — code foundations, database schema, inte
|
||||
- Mark human verification as **N/A** with rationale: "Infrastructure/foundation phase — no user-facing elements to test manually."
|
||||
- Set `human_verification: []` and do **not** produce a `human_needed` status solely due to lack of user-facing features.
|
||||
- Only add human verification items if the phase goal or success criteria explicitly describe something a user would interact with (UI, CLI command output visible to end users, external service UX).
|
||||
- **Exception — behavior-unverified truths still count.** A truth marked ⚠️ PRESENT_BEHAVIOR_UNVERIFIED (a state transition or a cancellation/cleanup/ordering invariant with no test exercising it) is a behavioral-evidence gap, not an artificial user-facing step. Record it in `behavior_unverified_items` and emit a human-verification item for it **even on an infrastructure/foundation phase** — these invariants are exactly where infra phases hide runtime state leaks. Such a truth drives `human_needed`; the auto-pass-UAT shortcut applies only to the absence of user-facing UX, never to a behavior-unverified invariant.
|
||||
|
||||
**How to determine if a phase is infrastructure/foundation:**
|
||||
- Phase goal or name contains: "foundation", "infrastructure", "schema", "database", "internal API", "data model", "scaffolding", "pipeline", "tooling", "CI", "migrations", "service layer", "backend", "core library"
|
||||
- Phase success criteria describe only technical artifacts (files exist, tests pass, schema is valid) with no user interaction required
|
||||
- There is no UI, CLI output visible to end users, or real-time behavior to observe
|
||||
|
||||
**If the phase IS infrastructure/foundation:** auto-pass UAT — skip the human verification items list entirely. Log:
|
||||
**If the phase IS infrastructure/foundation:** auto-pass UAT — skip the human verification items list entirely, **except any ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truth (see exception above), which still emits a human-verification item and drives `human_needed`.** Log:
|
||||
|
||||
```markdown
|
||||
## Human Verification
|
||||
@@ -475,15 +478,17 @@ Classify status using this decision tree IN ORDER (most restrictive first):
|
||||
- **judgment-tier, autonomous run** (non-authoritative LLM-judge verdict): emit the `unverified-prohibition — human review recommended` flag and classify → **human_needed** (autonomous completion reads "complete with N flagged prohibitions"; never a silent pass, never a hard halt).
|
||||
- **judgment-tier, interactive run**: route to the end-of-phase human checkpoint → **human_needed**.
|
||||
|
||||
3. IF the previous step produced ANY human verification items:
|
||||
→ **human_needed** (even if all truths VERIFIED and score is N/N)
|
||||
3. IF the previous step produced ANY human verification items — this includes every ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truth:
|
||||
→ **human_needed** (even if all other truths VERIFIED)
|
||||
|
||||
4. IF all checks pass AND no human verification items AND no flagged prohibitions:
|
||||
→ **passed**
|
||||
|
||||
**passed is ONLY valid when no human verification items AND no flagged prohibitions exist.** A prohibition (must-NOT) can never be silently absorbed into a `passed` verdict — that is the core failure mode ADR-550 D4 forbids.
|
||||
|
||||
**Score:** `verified_truths / total_truths`
|
||||
A ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truth is never FAILED and never VERIFIED: it does not trigger gaps_found (the code is present and wired) and is not counted as verified (its runtime behavior was not exercised). It routes through the existing human_needed sink — no new overall status.
|
||||
|
||||
**Score:** `verified_truths / total_truths` — `verified_truths` counts ✓ VERIFIED truths plus PASSED (override) truths; ⚠️ PRESENT_BEHAVIOR_UNVERIFIED truths are the only ones excluded, reported separately as the `behavior_unverified` count. A headline N/N therefore certifies behavioral evidence for every behavior-dependent truth, not merely symbol presence.
|
||||
</step>
|
||||
|
||||
<step name="filter_deferred_items">
|
||||
|
||||
@@ -32,5 +32,5 @@
|
||||
"gsd-ui-checker.md": 11088,
|
||||
"gsd-ui-researcher.md": 19272,
|
||||
"gsd-user-profiler.md": 8516,
|
||||
"gsd-verifier.md": 43346
|
||||
"gsd-verifier.md": 48859
|
||||
}
|
||||
|
||||
129
tests/verifier-behavior-unverified.test.cjs
Normal file
129
tests/verifier-behavior-unverified.test.cjs
Normal file
@@ -0,0 +1,129 @@
|
||||
'use strict';
|
||||
|
||||
// Issue #966 — behavior-dependent must-haves must not pass on symbol presence.
|
||||
// Content-assertion contract for the gsd-verifier agent: the
|
||||
// PRESENT_BEHAVIOR_UNVERIFIED per-truth state, its routing to human_needed,
|
||||
// the behavior-verified score split, and the parity invariant that the new
|
||||
// per-truth state never leaks into the overall-status vocabulary.
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const ROOT = path.join(__dirname, '..');
|
||||
const verifierPath = path.join(ROOT, 'agents', 'gsd-verifier.md');
|
||||
const verifier = fs.readFileSync(verifierPath, 'utf-8');
|
||||
const standaloneTemplatePath = path.join(ROOT, 'gsd-core', 'templates', 'verification-report.md');
|
||||
const standalone = fs.readFileSync(standaloneTemplatePath, 'utf-8');
|
||||
|
||||
test('Step 3 defines the PRESENT_BEHAVIOR_UNVERIFIED per-truth state', () => {
|
||||
assert.match(verifier, /PRESENT_BEHAVIOR_UNVERIFIED/);
|
||||
assert.match(verifier, /present[^\n]*wired|wired[^\n]*present/i);
|
||||
});
|
||||
|
||||
test('behavior-dependent trigger names transition + cancellation/cleanup/ordering invariants', () => {
|
||||
assert.match(verifier, /state transition/i);
|
||||
assert.match(verifier, /cancellation|cleanup|ordering/i);
|
||||
assert.match(verifier, /invariant/i);
|
||||
});
|
||||
|
||||
test('PRESENT_BEHAVIOR_UNVERIFIED routes to human verification', () => {
|
||||
assert.match(verifier, /PRESENT_BEHAVIOR_UNVERIFIED[\s\S]{0,400}?human/i);
|
||||
});
|
||||
|
||||
test('PRESENT_BEHAVIOR_UNVERIFIED is the only truth excluded from the verified score', () => {
|
||||
assert.match(
|
||||
verifier,
|
||||
/(do not count it toward the verified score)|(only[^\n]*excluded from `verified_truths`)|(only truths excluded)/i,
|
||||
);
|
||||
});
|
||||
|
||||
test('score still credits PASSED (override) truths (override contract preserved)', () => {
|
||||
// The Step 9 score definition must count override-passed truths in verified_truths.
|
||||
assert.match(
|
||||
verifier,
|
||||
/verified_truths[\s\S]{0,200}?PASSED \(override\)/,
|
||||
'Step 9 score must count PASSED (override) truths in verified_truths',
|
||||
);
|
||||
});
|
||||
|
||||
test('behavior-unverified truths get a structured frontmatter list that survives gaps_found', () => {
|
||||
assert.match(verifier, /behavior_unverified_items/);
|
||||
// and it must NOT be gated only to human_needed (must mention it is emitted regardless of status / when count > 0)
|
||||
assert.match(
|
||||
verifier,
|
||||
/behavior_unverified_items[\s\S]{0,160}?(regardless of (overall )?status|count > 0)/i,
|
||||
);
|
||||
});
|
||||
|
||||
test('Step 9 / template carry the behavior_unverified score-split field', () => {
|
||||
assert.match(verifier, /behavior_unverified/);
|
||||
});
|
||||
|
||||
test('critical_rules calibrates "presence is not behavior" without dropping the speed guard', () => {
|
||||
assert.match(verifier, /presence is not behavior/i);
|
||||
assert.match(verifier, /Keep verification fast/);
|
||||
});
|
||||
|
||||
test('PARITY: per-truth state never leaks into the overall-status vocabulary', () => {
|
||||
assert.doesNotMatch(verifier, /→ \*\*status:\s*present_behavior_unverified\*\*/i);
|
||||
const unionLines = verifier.match(/^status:\s+[a-z_]+(?:\s*\|\s*[a-z_]+)+\s*$/gm) || [];
|
||||
assert.ok(unionLines.length > 0, 'expected at least one status union line');
|
||||
for (const line of unionLines) {
|
||||
assert.doesNotMatch(line, /present_behavior_unverified/i);
|
||||
// The real invariant is that the per-truth state is NOT in the union (above).
|
||||
// Membership (order-independent) avoids brittleness on a future legitimate reorder.
|
||||
for (const s of ['passed', 'gaps_found', 'human_needed']) {
|
||||
assert.ok(line.includes(s), `status union must still contain ${s}: ${line}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('overall-status enum in verification.cts is unchanged (no per-truth leak)', () => {
|
||||
const cts = fs.readFileSync(path.join(ROOT, 'src', 'verification.cts'), 'utf-8');
|
||||
const m = cts.match(/VERIFIER_STATUSES[^=]*=\s*\[([^\]]*)\]/);
|
||||
assert.ok(m, 'VERIFIER_STATUSES array must be present');
|
||||
assert.doesNotMatch(m[1], /present_behavior_unverified/i);
|
||||
for (const s of ['passed', 'gaps_found', 'human_needed']) {
|
||||
assert.match(m[1], new RegExp(`'${s}'`));
|
||||
}
|
||||
});
|
||||
|
||||
test('VERIFICATION.md templates carry behavior_unverified + the new truth-state', () => {
|
||||
assert.match(verifier, /behavior_unverified/);
|
||||
assert.match(standalone, /PRESENT_BEHAVIOR_UNVERIFIED/);
|
||||
assert.match(standalone, /behavior_unverified/);
|
||||
assert.match(verifier, /behavior_unverified_items/);
|
||||
assert.match(standalone, /behavior_unverified_items/);
|
||||
});
|
||||
|
||||
const verifyPhase = fs.readFileSync(path.join(ROOT, 'gsd-core', 'workflows', 'verify-phase.md'), 'utf-8');
|
||||
const planningArtifacts = fs.readFileSync(path.join(ROOT, 'docs', 'reference', 'planning-artifacts.md'), 'utf-8');
|
||||
|
||||
test('shipped verify-phase workflow mirrors the behavior-unverified calibration', () => {
|
||||
assert.match(verifyPhase, /PRESENT_BEHAVIOR_UNVERIFIED/);
|
||||
assert.match(verifyPhase, /behavior_unverified/);
|
||||
assert.match(verifyPhase, /state transition/i);
|
||||
});
|
||||
|
||||
test('planning-artifacts reference documents the behavior-unverified calibration', () => {
|
||||
assert.match(planningArtifacts, /PRESENT_BEHAVIOR_UNVERIFIED/);
|
||||
assert.match(planningArtifacts, /behavior_unverified/);
|
||||
});
|
||||
|
||||
test('Step 9 keeps gaps_found precedence and preserves behavior-unverified items', () => {
|
||||
assert.match(verifier, /gaps_found's precedence|gaps_found[\s\S]{0,160}?precedence/i);
|
||||
assert.match(verifier, /behavior_unverified_items[\s\S]{0,120}?(never lost|survive|regardless)/i);
|
||||
});
|
||||
|
||||
test('shipped workflow flags behavior-unverified truths even on infrastructure phases', () => {
|
||||
assert.match(
|
||||
verifyPhase,
|
||||
/PRESENT_BEHAVIOR_UNVERIFIED[\s\S]{0,400}?infrastructure|infrastructure[\s\S]{0,400}?PRESENT_BEHAVIOR_UNVERIFIED/i,
|
||||
);
|
||||
});
|
||||
|
||||
test('standalone template per-truth guideline respects gaps_found precedence', () => {
|
||||
assert.match(standalone, /becomes `human_needed`[\s\S]{0,80}?gaps_found/i);
|
||||
});
|
||||
@@ -85,6 +85,6 @@
|
||||
"undo.md": 10431,
|
||||
"update.md": 21053,
|
||||
"validate-phase.md": 10745,
|
||||
"verify-phase.md": 30875,
|
||||
"verify-phase.md": 33161,
|
||||
"verify-work.md": 31157
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user