fix(verification): keep passed uat results across verifier re-runs
A phase whose human UAT passed could loop forever between execute-phase and verify-work. A gap-closure plan, or a later phase editing a covered file, made the report stale; verify-work only recorded a passed UAT against human_needed; the re-run verifier could not see the UAT file and re-emitted human_needed; and execute-phase's human_needed branch then rewrote *-UAT.md with every row back to [pending]. - verification.uat-evidence: recorded UAT rows plus the covered implementation files changed since the UAT. The verifier (Step 8b) treats a row that passed on unchanged code as verified and re-lists only affected rows, each with a retest_reason. - verification.seed-uat: merges the report's human items into *-UAT.md. Existing rows are kept byte-for-byte; a pass is reset only with a recorded reason. - verification.canonicalize-uat: the single human_needed -> passed flip, gated on the uat-only row predicate; refuses a stale report. - verification.status reports stale_reason; init.progress lists every executed-but-stale phase in reverify_phases. - verify-work and progress re-run the verifier themselves for a stale report instead of routing to execute-phase; execute-phase records a gap-closure checkpoint's UAT answers before dispatching the verifier. Fail-closed properties are unchanged: a malformed fingerprint and a real change to covered code still read stale. Emitted-Drift-Ack-Growth: execute-phase.md — verify_phase_goal gains the gap-closure uat-record pointer and the merge-not-overwrite human_needed branch; the uat template it replaced moved into verification.seed-uat Emitted-Drift-Ack-Growth: msd-verifier.md — new step 8b pointer to the lazily loaded verifier-uat-evidence reference plus the human_verified frontmatter key Emitted-Drift-Ack-Growth: progress.md — route v.stale now re-verifies every stale executed phase in place instead of naming one command
This commit is contained in:
@@ -1189,6 +1189,8 @@ If `section_manifest` is `null` or `"regression-gate"` is in its `included` list
|
||||
<step name="verify_phase_goal">
|
||||
Verify phase achieved its GOAL, not just completed tasks.
|
||||
|
||||
**Gap-closure runs:** if this run executed a `gap_closure: true` plan, first follow `execute-phase/steps/gap-closure-uat-record.md` — the human's checkpoint results go into `*-UAT.md` BEFORE the verifier is dispatched, so the same run can reach `passed`.
|
||||
|
||||
```bash
|
||||
VERIFIER_SKILLS=$(msd_run query agent-skills msd-verifier)
|
||||
```
|
||||
@@ -1208,6 +1210,7 @@ Create VERIFICATION.md.
|
||||
Read these files before verification:
|
||||
- {phase_dir}/*-PLAN.md (All plans — understand intent, check must_haves)
|
||||
- {phase_dir}/*-SUMMARY.md (All summaries — cross-reference claimed vs actual)
|
||||
- {phase_dir}/*-UAT.md (If present: recorded human UAT — evidence, verifier Step 8b)
|
||||
- {requirements_path} (Requirement traceability)
|
||||
${CONTEXT_WINDOW >= 500000 ? `- {phase_dir}/*-CONTEXT.md (User decisions — verify they were honored)
|
||||
- {phase_dir}/*-RESEARCH.md (Known pitfalls — check for traps)
|
||||
@@ -1235,60 +1238,30 @@ Route on `$STATUS`: if `passed`, proceed to update_roadmap. Otherwise keep the p
|
||||
|
||||
**If human_needed:**
|
||||
|
||||
**Step A: Persist human verification items as UAT file.**
|
||||
**Step A: Merge human verification items into the UAT file — never overwrite it.**
|
||||
|
||||
Create `{phase_dir}/{phase_num}-UAT.md` using UAT template format:
|
||||
|
||||
```markdown
|
||||
---
|
||||
status: testing
|
||||
phase: {phase_num}-{phase_name}
|
||||
source: [{phase_num}-VERIFICATION.md]
|
||||
started: [now ISO]
|
||||
updated: [now ISO]
|
||||
---
|
||||
|
||||
## Current Test
|
||||
|
||||
number: 1
|
||||
name: {first human_verification item description}
|
||||
expected: |
|
||||
{expected behavior from VERIFICATION.md}
|
||||
awaiting: user response
|
||||
|
||||
## Tests
|
||||
|
||||
{For each human_verification item from VERIFICATION.md:}
|
||||
|
||||
### {N}. {item description}
|
||||
expected: {expected behavior from VERIFICATION.md}
|
||||
result: [pending]
|
||||
|
||||
## Summary
|
||||
|
||||
total: {count}
|
||||
passed: 0
|
||||
issues: 0
|
||||
pending: {count}
|
||||
skipped: 0
|
||||
blocked: 0
|
||||
|
||||
## Gaps
|
||||
```
|
||||
|
||||
Commit the file:
|
||||
```bash
|
||||
msd_run query commit "test({phase_num}): persist human verification items as UAT" --files "{phase_dir}/{phase_num}-UAT.md"
|
||||
SEED=$(msd_run query verification.seed-uat "$PHASE_DIR")
|
||||
SEED_UAT=$(printf '%s' "$SEED" | jq -r '.uat_file // empty')
|
||||
SEED_PENDING=$(printf '%s' "$SEED" | jq -r '.pending // 0')
|
||||
```
|
||||
|
||||
`verification.seed-uat` creates `{phase_num}-UAT.md` when none exists, and otherwise MERGES: every existing row and its recorded result is kept, only unseen items are appended as `[pending]`, and a passing row is reset only when the verifier recorded a `retest_reason` for it. Do NOT hand-write or regenerate the UAT file. When `.changed` is `true`, commit it:
|
||||
|
||||
```bash
|
||||
msd_run query commit "test({phase_num}): persist human verification items as UAT" --files "$SEED_UAT"
|
||||
```
|
||||
|
||||
**If `SEED_PENDING` is `0`** — every UAT row already has a result, so no human test is owed. Do NOT present the items as tests to run. Say so, and hand off: `/msd:verify-work {X} ${MSD_WS}` records the pass against the report at once (`verification.canonicalize-uat`) and completes the phase without re-asking a single row. Skip Step B.
|
||||
|
||||
**Step B: Present to user**:
|
||||
|
||||
```
|
||||
## ◷ Phase {X}: {Name} — Human Verification Needed
|
||||
|
||||
All automated checks passed. {N} item(s) require human testing before this phase can be marked complete:
|
||||
All automated checks passed. {SEED_PENDING} item(s) require human testing before this phase can be marked complete:
|
||||
|
||||
{From VERIFICATION.md human_verification section}
|
||||
{The pending rows of {phase_num}-UAT.md — rows that already passed are NOT re-tested}
|
||||
|
||||
Tests saved to `{phase_num}-UAT.md`.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user