fix(verification): keep passed uat results across verifier re-runs #2

Merged

jakub merged 2 commits from fix/uat-verification-stale-loop into next 2026-10-11 10:44:26 +00:00

Conversation 0 Commits 2 Files Changed 29 +1659 -129

jakub commented 2026-10-10 23:49:51 +00:00

Owner

Copy Link

Problem

A phase whose human UAT passed could not reach passed. Found on summercms.go (Phase 16.2, and earlier Phase 9): the user was sent round /msd-execute-phase N → /msd-verify-work N → /msd-progress repeatedly, and each pass threw away the UAT they had just completed.

Five causes, all confirmed against the source:

1. Gap closure makes the report stale by construction. allCurrentArtifactsCovered() requires every live PLAN/SUMMARY to be in covered_files; a gap-closure plan always adds ones the earlier report could not list.
2. stale outranks human_needed. verify-work only recorded a passed UAT against a human_needed status, so a passed UAT could never be recorded against a stale report.
3. The verifier ignored UAT, and execute-phase then destroyed it. The verifier never read *-UAT.md, so it re-emitted the same human items; the human_needed branch then rewrote the UAT file with every row [pending].
4. Cross-phase drift. A later phase editing a shared covered file re-staled finished phases, and progress walked the user back through them one at a time.
5. A gap plan ending in a human checkpoint never closed verification. Nothing recorded the checkpoint answers or re-verified with them.

Fix

Three new msd-tools verbs in src/uat-evidence.cts, under the verification family:

Verb	What it does
verification.uat-evidence	Read-only. The recorded UAT rows, plus which covered implementation files changed after the UAT was recorded (commit time when clean, mtime otherwise — the same clock as #2348).
verification.seed-uat	Merges the report's human_verification items into *-UAT.md. Existing rows are kept byte-for-byte; unseen items are appended as [pending]; a pass is reset only when the verifier recorded a retest_reason, which is written into the row.
verification.canonicalize-uat	The single human_needed → passed flip, gated on the uat-only row predicate (#4663). Refuses any other status, so a stale report is never flipped.

Also:

• verification.status reports stale_reason (uncovered_artifacts, covered_changed, fingerprint_malformed, summary_newer) and raw_status on a stale result.
• init.progress lists every executed-but-stale phase in reverify_phases.

Prompt changes:

• msd-verifier — new Step 8b (references/verifier-uat-evidence.md): a human item whose UAT row passed on unchanged code is verified by that row; a row whose tested code changed is re-listed with a retest_reason. Unsure → re-test.
• execute-phase — the human_needed branch calls verification.seed-uat instead of hand-writing the UAT file; a gap-closure run records its checkpoint's UAT answers before the verifier is dispatched (steps/gap-closure-uat-record.md).
• verify-work — a stale report is re-verified in-session, before canonicalization, instead of routing to execute-phase.
• progress — Route V.stale re-verifies every stale executed phase as one action.
• references/stale-reverification.md is the one shared re-verification procedure for all three workflows.

Cross-phase drift decision (cause 4)

Drift detection is kept: a covered file edited by a later phase still makes the earlier report stale. What changed is the cost. Progress offers all such phases as a single re-verify action and runs the verifiers itself, and the human is only asked to re-test rows the changed files bear on.

Safety properties kept

• A malformed or incomplete fingerprint still reads stale (#4155).
• A real change to covered code still reads stale, and the affected UAT row is marked for re-test rather than silently passed.
• The human_needed branch of execute-phase still never completes the phase directly (issue #38): when every UAT row already passes it hands off to verify-work, which records the pass without re-asking anything.

Tests

New tests/uat-evidence.test.cjs (24 tests), including the five acceptance scenarios:

• gap-closure fixture: gap plan + passed UAT reaches passed without re-testing
• no-clobber: passed rows and their reported: evidence are left byte-identical
• real change still stales, and the affected row is marked for re-test
• loop guard: after UAT passes, progress does not route the phase anywhere
• cross-phase drift: finished phases surface as one re-verify action and keep their UAT

These simulate the verifier by writing the report it would write. The verifier's own judgement of which rows a changed file affects is prompt text and is not exercised by a test.

The second commit updates the text-pinning tests in verify-work-auto-transition.test.cjs and progress-forensic.test.cjs and refreshes the compact-content benchmark baseline.

Known state of the suite

Full-suite comparison against next shows no failures introduced by this branch. Failures that exist on next independently of this work include one test in verification-status.test.cjs and three CI lint steps (lint-allow-test-rule-refs, lint-example-parser-parity, lint-docs-guard-registration).

Notes for the reviewer

• No version bump; the changeset fragment carries the changelog entry.
• Three shipped prompt files grew; each has an Emitted-Drift-Ack-Growth trailer on the fix commit.

🤖 Generated with Claude Code

## Problem A phase whose human UAT passed could not reach `passed`. Found on summercms.go (Phase 16.2, and earlier Phase 9): the user was sent round `/msd-execute-phase N` → `/msd-verify-work N` → `/msd-progress` repeatedly, and each pass threw away the UAT they had just completed. Five causes, all confirmed against the source: 1. **Gap closure makes the report stale by construction.** `allCurrentArtifactsCovered()` requires every live PLAN/SUMMARY to be in `covered_files`; a gap-closure plan always adds ones the earlier report could not list. 2. **`stale` outranks `human_needed`.** `verify-work` only recorded a passed UAT against a `human_needed` status, so a passed UAT could never be recorded against a stale report. 3. **The verifier ignored UAT, and execute-phase then destroyed it.** The verifier never read `*-UAT.md`, so it re-emitted the same human items; the `human_needed` branch then rewrote the UAT file with every row `[pending]`. 4. **Cross-phase drift.** A later phase editing a shared covered file re-staled finished phases, and progress walked the user back through them one at a time. 5. **A gap plan ending in a human checkpoint never closed verification.** Nothing recorded the checkpoint answers or re-verified with them. ## Fix Three new `msd-tools` verbs in `src/uat-evidence.cts`, under the `verification` family: | Verb | What it does | |---|---| | `verification.uat-evidence` | Read-only. The recorded UAT rows, plus which covered implementation files changed after the UAT was recorded (commit time when clean, mtime otherwise — the same clock as #2348). | | `verification.seed-uat` | Merges the report's `human_verification` items into `*-UAT.md`. Existing rows are kept byte-for-byte; unseen items are appended as `[pending]`; a pass is reset only when the verifier recorded a `retest_reason`, which is written into the row. | | `verification.canonicalize-uat` | The single `human_needed` → `passed` flip, gated on the uat-only row predicate (#4663). Refuses any other status, so a stale report is never flipped. | Also: - `verification.status` reports `stale_reason` (`uncovered_artifacts`, `covered_changed`, `fingerprint_malformed`, `summary_newer`) and `raw_status` on a stale result. - `init.progress` lists every executed-but-stale phase in `reverify_phases`. Prompt changes: - **`msd-verifier`** — new Step 8b (`references/verifier-uat-evidence.md`): a human item whose UAT row passed on unchanged code is verified by that row; a row whose tested code changed is re-listed with a `retest_reason`. Unsure → re-test. - **`execute-phase`** — the `human_needed` branch calls `verification.seed-uat` instead of hand-writing the UAT file; a gap-closure run records its checkpoint's UAT answers before the verifier is dispatched (`steps/gap-closure-uat-record.md`). - **`verify-work`** — a stale report is re-verified in-session, before canonicalization, instead of routing to `execute-phase`. - **`progress`** — Route V.stale re-verifies every stale executed phase as one action. - `references/stale-reverification.md` is the one shared re-verification procedure for all three workflows. ## Cross-phase drift decision (cause 4) Drift detection is kept: a covered file edited by a later phase still makes the earlier report `stale`. What changed is the cost. Progress offers all such phases as a single re-verify action and runs the verifiers itself, and the human is only asked to re-test rows the changed files bear on. ## Safety properties kept - A malformed or incomplete fingerprint still reads `stale` (#4155). - A real change to covered code still reads `stale`, and the affected UAT row is marked for re-test rather than silently passed. - The `human_needed` branch of `execute-phase` still never completes the phase directly (issue #38): when every UAT row already passes it hands off to `verify-work`, which records the pass without re-asking anything. ## Tests New `tests/uat-evidence.test.cjs` (24 tests), including the five acceptance scenarios: - gap-closure fixture: gap plan + passed UAT reaches `passed` without re-testing - no-clobber: passed rows and their `reported:` evidence are left byte-identical - real change still stales, and the affected row is marked for re-test - loop guard: after UAT passes, progress does not route the phase anywhere - cross-phase drift: finished phases surface as one re-verify action and keep their UAT These simulate the verifier by writing the report it would write. The verifier's own judgement of which rows a changed file affects is prompt text and is not exercised by a test. The second commit updates the text-pinning tests in `verify-work-auto-transition.test.cjs` and `progress-forensic.test.cjs` and refreshes the compact-content benchmark baseline. ## Known state of the suite Full-suite comparison against `next` shows no failures introduced by this branch. Failures that exist on `next` independently of this work include one test in `verification-status.test.cjs` and three CI lint steps (`lint-allow-test-rule-refs`, `lint-example-parser-parity`, `lint-docs-guard-registration`). ## Notes for the reviewer - No version bump; the changeset fragment carries the changelog entry. - Three shipped prompt files grew; each has an `Emitted-Drift-Ack-Growth` trailer on the fix commit. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

jakub added 2 commits 2026-10-10 23:49:51 +00:00

fix(verification): keep passed uat results across verifier re-runs ... aa03fc7fa9

A phase whose human UAT passed could loop forever between execute-phase
and verify-work. A gap-closure plan, or a later phase editing a covered
file, made the report stale; verify-work only recorded a passed UAT
against human_needed; the re-run verifier could not see the UAT file and
re-emitted human_needed; and execute-phase's human_needed branch then
rewrote *-UAT.md with every row back to [pending].

- verification.uat-evidence: recorded UAT rows plus the covered
  implementation files changed since the UAT. The verifier (Step 8b)
  treats a row that passed on unchanged code as verified and re-lists
  only affected rows, each with a retest_reason.
- verification.seed-uat: merges the report's human items into *-UAT.md.
  Existing rows are kept byte-for-byte; a pass is reset only with a
  recorded reason.
- verification.canonicalize-uat: the single human_needed -> passed flip,
  gated on the uat-only row predicate; refuses a stale report.
- verification.status reports stale_reason; init.progress lists every
  executed-but-stale phase in reverify_phases.
- verify-work and progress re-run the verifier themselves for a stale
  report instead of routing to execute-phase; execute-phase records a
  gap-closure checkpoint's UAT answers before dispatching the verifier.

Fail-closed properties are unchanged: a malformed fingerprint and a real
change to covered code still read stale.

Emitted-Drift-Ack-Growth: execute-phase.md — verify_phase_goal gains the gap-closure uat-record pointer and the merge-not-overwrite human_needed branch; the uat template it replaced moved into verification.seed-uat
Emitted-Drift-Ack-Growth: msd-verifier.md — new step 8b pointer to the lazily loaded verifier-uat-evidence reference plus the human_verified frontmatter key
Emitted-Drift-Ack-Growth: progress.md — route v.stale now re-verifies every stale executed phase in place instead of naming one command

test: align pinned workflow text with the uat canonicalize seam ... 7b50026075

verify-work now calls verification.canonicalize-uat instead of
hand-rolling the status flip, and progress's Route V.stale re-verifies
in place. The flip's behavior is covered in tests/uat-evidence.test.cjs;
these tests keep the workflow's side of the contract. Refreshes the
compact-content benchmark baseline for the moved bytes.

jakub merged commit 157d5d9d31 into next 2026-10-11 10:44:26 +00:00

jakub referenced this issue from a commit 2026-10-11 10:44:27 +00:00

Merge pull request 'fix(verification): keep passed uat results across verifier re-runs' (#2) from fix/uat-verification-stale-loop into next

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

260 jakub lucas

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: golem15/msd-core#2