fix(verification): keep passed uat results across verifier re-runs
A phase whose human UAT passed could loop forever between execute-phase and verify-work. A gap-closure plan, or a later phase editing a covered file, made the report stale; verify-work only recorded a passed UAT against human_needed; the re-run verifier could not see the UAT file and re-emitted human_needed; and execute-phase's human_needed branch then rewrote *-UAT.md with every row back to [pending]. - verification.uat-evidence: recorded UAT rows plus the covered implementation files changed since the UAT. The verifier (Step 8b) treats a row that passed on unchanged code as verified and re-lists only affected rows, each with a retest_reason. - verification.seed-uat: merges the report's human items into *-UAT.md. Existing rows are kept byte-for-byte; a pass is reset only with a recorded reason. - verification.canonicalize-uat: the single human_needed -> passed flip, gated on the uat-only row predicate; refuses a stale report. - verification.status reports stale_reason; init.progress lists every executed-but-stale phase in reverify_phases. - verify-work and progress re-run the verifier themselves for a stale report instead of routing to execute-phase; execute-phase records a gap-closure checkpoint's UAT answers before dispatching the verifier. Fail-closed properties are unchanged: a malformed fingerprint and a real change to covered code still read stale. Emitted-Drift-Ack-Growth: execute-phase.md — verify_phase_goal gains the gap-closure uat-record pointer and the merge-not-overwrite human_needed branch; the uat template it replaced moved into verification.seed-uat Emitted-Drift-Ack-Growth: msd-verifier.md — new step 8b pointer to the lazily loaded verifier-uat-evidence reference plus the human_verified frontmatter key Emitted-Drift-Ack-Growth: progress.md — route v.stale now re-verifies every stale executed phase in place instead of naming one command
This commit is contained in:
Jakub Zych
5
.changeset/nimble-deer-purr.md
Normal file
5
.changeset/nimble-deer-purr.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
type: Fixed
|
||||
pr: 2
|
||||
---
|
||||
**A passed UAT is no longer discarded or re-requested after gap closure** — a phase whose human UAT passed could loop forever between `/msd-execute-phase` and `/msd-verify-work`: a gap-closure plan (or a later phase editing a covered file) made the report `stale`, the re-run verifier could not see the UAT file, and execute-phase then rewrote `*-UAT.md` with every row back to `[pending]`. The verifier now reads recorded UAT results as evidence (`verification.uat-evidence`) and re-requests only rows whose tested code changed; the UAT file is merged, never overwritten (`verification.seed-uat`); `verify-work` and `progress` re-run the verifier themselves for a stale report instead of routing to `execute-phase`; and `progress` offers every stale-but-executed phase as one re-verify action. `verification.status` now reports `stale_reason`. A real change to covered code still makes the report stale.
|
||||
1
.gitignore
vendored
1
.gitignore
vendored
@@ -312,6 +312,7 @@ build/
|
||||
/msd-core/bin/lib/uat.cjs
|
||||
/msd-core/bin/lib/coverage.cjs
|
||||
/msd-core/bin/lib/uat-predicate.cjs
|
||||
/msd-core/bin/lib/uat-evidence.cjs
|
||||
/msd-core/bin/lib/workstream.cjs
|
||||
/msd-core/bin/lib/roadmap.cjs
|
||||
/msd-core/bin/lib/audit.cjs
|
||||
|
||||
@@ -537,6 +537,10 @@ Merge those harvested items into the same human verification list as your own an
|
||||
**Why human:** {Why can't verify programmatically}
|
||||
```
|
||||
|
||||
## Step 8b: Apply Recorded UAT Evidence
|
||||
|
||||
If the phase directory holds a `*-UAT.md`, read and follow `~/.claude/msd-core/references/verifier-uat-evidence.md` before Step 9. A human item whose UAT row already passed on unchanged code is verified by that row and leaves the human verification list; a re-listed row that had passed carries a `retest_reason`. Never re-request a UAT the change did not touch.
|
||||
|
||||
## Step 9: Determine Overall Status
|
||||
|
||||
Classify status using this decision tree IN ORDER (most restrictive first):
|
||||
@@ -722,6 +726,7 @@ coincidental_reliance_items: # Only if a ✓ VERIFIED truth holds incidentally
|
||||
- truth: "Observable truth that holds incidentally"
|
||||
reason: undeclared-precondition | incidental-ordering | fixture-only
|
||||
harden: "Precondition/ordering to declare or enforce"
|
||||
human_verified: # Only if Step 8b settled items from *-UAT.md — shape in verifier-uat-evidence.md
|
||||
human_verification: # Only if status: human_needed
|
||||
- test: "What to do"
|
||||
expected: "What should happen"
|
||||
|
||||
@@ -323,6 +323,7 @@
|
||||
"sketch-variant-patterns.md",
|
||||
"specless-probe-fallback.md",
|
||||
"spidr-splitting.md",
|
||||
"stale-reverification.md",
|
||||
"tdd.md",
|
||||
"thinking-models-debug.md",
|
||||
"thinking-models-execution.md",
|
||||
@@ -340,6 +341,7 @@
|
||||
"verification-patterns.md",
|
||||
"verifier-evidence-gate.md",
|
||||
"verifier-phase-gates.md",
|
||||
"verifier-uat-evidence.md",
|
||||
"verifier-wiring-patterns.md",
|
||||
"verify-command-path-resolvability.md",
|
||||
"verify-mvp-mode.md",
|
||||
@@ -565,6 +567,7 @@
|
||||
"template.cjs",
|
||||
"text-lines.cjs",
|
||||
"token-scanner.cjs",
|
||||
"uat-evidence.cjs",
|
||||
"uat-predicate.cjs",
|
||||
"uat.cjs",
|
||||
"ui-consideration-probe.cjs",
|
||||
@@ -652,6 +655,7 @@
|
||||
"execute-phase/steps/executor-isolation-dispatch.md",
|
||||
"execute-phase/steps/executor-progress-policy.md",
|
||||
"execute-phase/steps/gap-closure-artifacts.md",
|
||||
"execute-phase/steps/gap-closure-uat-record.md",
|
||||
"execute-phase/steps/partial-wave.md",
|
||||
"execute-phase/steps/per-plan-executor-routing.md",
|
||||
"execute-phase/steps/per-plan-worktree-gate.md",
|
||||
|
||||
@@ -347,6 +347,8 @@ Full roster at `msd-core/references/*.md`. References are shared knowledge docum
|
||||
| `verification-overrides.md` | Per-artifact verification override rules. |
|
||||
| `verifier-phase-gates.md` | Verifier-time gates eagerly imported by `msd-verifier` (migrated from the retired `verify-phase` workflow, #1892): decision-coverage validation (#2492), test-quality audit, and infrastructure-phase human-verification scoping (#2504). |
|
||||
| `verifier-evidence-gate.md` | Re-verification convergence gate loaded by `msd-verifier` (#3304): a Step 7 anti-pattern blocker that is neither a carried-forward gap nor a regression needs deterministic evidence to stay blocking, else it downgrades to advisory. |
|
||||
| `verifier-uat-evidence.md` | UAT-evidence gate loaded by `msd-verifier` at Step 8b when the phase has a `*-UAT.md`: a human item whose UAT row passed on unchanged code is verified by that row; a row whose tested code changed is re-listed with a `retest_reason`. |
|
||||
| `stale-reverification.md` | The single procedure for refreshing a `stale` VERIFICATION.md on a fully summarized phase (verifier re-run, `verification.seed-uat` merge, fresh-status routing); shared by `execute-phase`, `verify-work` and `progress`. |
|
||||
| `planning-config.md` | Full config schema and behavior. |
|
||||
| `phase-id-convention.md` | Canonical bracket phase-ID grammar card and display/disk translation reference (ADR-612). |
|
||||
| `security-asvs-levels.md` | OWASP ASVS level definitions for MSD threat modeling — per-level planner disposition rigor and auditor verification depth (L1 opportunistic, L2 standard, L3 comprehensive). |
|
||||
@@ -709,6 +711,7 @@ Full listing: `msd-core/bin/lib/*.cjs`.
|
||||
| `normalize-test-command.cjs` | Normalizes a resolved test command to a one-shot form so a watch-mode runner (vitest/jest) cannot hang a verification gate (#1857); shared by all three live test-command gates (regression, post-merge, audit-fix) |
|
||||
| `uat.cjs` | UAT file parsing, verification debt tracking, audit-uat support |
|
||||
| `uat-predicate.cjs` | UAT-passed predicate — markdown-aware evaluation of HUMAN-UAT results; returns pass only when all required checks pass; ignores false-positive contexts (frontmatter, fenced code, blockquotes, HTML comments) |
|
||||
| `uat-evidence.cjs` | UAT-evidence seam — `verification.uat-evidence` (recorded UAT rows plus covered files changed since), `verification.seed-uat` (merge the report's human items into `*-UAT.md` without overwriting results) and `verification.canonicalize-uat` (the single `human_needed` → `passed` flip, gated on the UAT-row predicate) |
|
||||
| `ui-consideration-probe.cjs` | Spec-completeness UI-consideration probe (compiled from `src/ui-consideration-probe.cts`, gitignored) — the third adapter of the `probe-core` resolution model (ADR-550 Decision 7): element-kind classification, applicable-category relevance filter, consideration proposal, `proposeElements`/`autoResolve` (propose-then-confirm + the `--auto` never-dismiss floor), and the `{explicit, backstop}` validators; delegates merge/rollup/CLI to `probe-core`; exports `classifyElement`, `applicableCategories`, `proposeConsiderations`, `proposeElements`, `autoResolve`, `analyzeCoverage`, `UI_TAXONOMY` (#1867) |
|
||||
| `ui-safety-gate.cjs` | Shell-free word-boundary UI token detector (#3706, #3718); reads phase-section text from stdin, exits 0 (UI found), 1 (no UI — real input, examined), NO_INPUT (empty/whitespace-only stdin) or UNAVAILABLE (stdin read failed) — the latter two are registry codes (ADR-3889 Phase 3, #3907); also deployed to `msd-core/bin/lib/` so the MSD installer ships it to `$RUNTIME_DIR` (#448) |
|
||||
| `ui-frontend-evidence.cjs` | Static frontend-evidence detector (compiled from `src/ui-frontend-evidence.cts`, gitignored) — plan-time structural corroboration for `computeUiPlanGate` (#3312): a `package.json` UI-framework dependency or a component-framework file (`*.tsx/*.jsx/*.vue/*.svelte`) in the tree, so a UI-token match on a hyphenated proper noun (e.g. repo `dashboard-financeiro`) cannot block planning in a repo with no frontend; mirrors the post-wave `computeUiSafetyGate` git-diff corroboration |
|
||||
|
||||
@@ -333,6 +333,7 @@ export default tseslint.config(
|
||||
'msd-core/bin/lib/uat.cjs',
|
||||
'msd-core/bin/lib/coverage.cjs',
|
||||
'msd-core/bin/lib/uat-predicate.cjs',
|
||||
'msd-core/bin/lib/uat-evidence.cjs',
|
||||
'msd-core/bin/lib/workstream.cjs',
|
||||
'msd-core/bin/lib/roadmap.cjs',
|
||||
'msd-core/bin/lib/audit.cjs',
|
||||
|
||||
@@ -326,6 +326,7 @@ const evalMod = require('./lib/eval.cjs');
|
||||
const { routeVerificationCommand } = require('./lib/verification-command-router.cjs');
|
||||
const { routePlanningCommand } = require('./lib/planning-command-router.cjs');
|
||||
const verification = require('./lib/verification.cjs');
|
||||
const uatEvidence = require('./lib/uat-evidence.cjs');
|
||||
const { routeInitCommand } = require('./lib/init-command-router.cjs');
|
||||
// Stale-bake guard (#1688): warns once when model config changed since agents
|
||||
// were last baked on static-frontmatter runtimes (codex/opencode). Lazy-required
|
||||
@@ -1094,6 +1095,7 @@ function dispatchOverlayCapabilityCommand({ command, args, cwd, raw, error, load
|
||||
function routeVerification({ args, cwd, raw, error }) {
|
||||
routeVerificationCommand({
|
||||
verification,
|
||||
uatEvidence,
|
||||
args,
|
||||
cwd,
|
||||
raw,
|
||||
|
||||
88
msd-core/references/stale-reverification.md
Normal file
88
msd-core/references/stale-reverification.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# Stale Re-verification
|
||||
|
||||
> The ONE procedure for refreshing a `stale` VERIFICATION.md when every plan in the phase
|
||||
> already has a SUMMARY. Loaded by `execute-phase` (`steps/stale-reverification.md`),
|
||||
> `verify-work` (its completion gate) and `progress` (Route V.stale), so the three cannot
|
||||
> drift apart. Nothing here executes a plan, and nothing here discards a recorded UAT
|
||||
> result. `msd_run` is the launcher shim of the loading workflow.
|
||||
|
||||
`verification.status` reports `stale` with a `stale_reason`:
|
||||
|
||||
| `stale_reason` | What happened |
|
||||
|---|---|
|
||||
| `uncovered_artifacts` | The phase gained a PLAN/SUMMARY the report never declared — every gap-closure cycle does this. |
|
||||
| `covered_changed` | A covered file changed after the verifier ran — possibly edited by a LATER phase. |
|
||||
| `fingerprint_malformed` / `summary_newer` | The fingerprint is unusable, or a legacy report predates a SUMMARY. |
|
||||
|
||||
Every reason has the same remedy: the verifier looks again. What must NOT happen is the
|
||||
human being asked to repeat a UAT the change did not touch.
|
||||
|
||||
## 1. Re-run the verifier
|
||||
|
||||
Dispatch `msd-verifier` with the same brief execute-phase's `verify_phase_goal` step uses. A
|
||||
workflow that did not load execute-phase's init resolves the inputs itself:
|
||||
|
||||
```bash
|
||||
PHASE_INIT=$(msd_run query init.execute-phase "{phase}") # phase_dir, phase_req_ids, requirements_path, verifier_model
|
||||
PHASE_GOAL=$(msd_run query roadmap.get-phase "{phase}" --pick goal)
|
||||
VERIFIER_SKILLS=$(msd_run query agent-skills msd-verifier)
|
||||
```
|
||||
|
||||
```
|
||||
Agent(
|
||||
description="Re-verify phase {phase_number} (stale report)",
|
||||
prompt="Verify phase {phase_number} goal achievement.
|
||||
Phase directory: {phase_dir}
|
||||
Phase goal: {PHASE_GOAL}
|
||||
Phase requirement IDs: {phase_req_ids}
|
||||
Check must_haves against actual codebase. The previous VERIFICATION.md is stale — regenerate it.
|
||||
A recorded human UAT is evidence: apply Step 8b before deciding the status.
|
||||
|
||||
<required_reading>
|
||||
- {phase_dir}/*-PLAN.md
|
||||
- {phase_dir}/*-SUMMARY.md
|
||||
- {phase_dir}/*-UAT.md (if present)
|
||||
- {requirements_path}
|
||||
</required_reading>
|
||||
|
||||
${VERIFIER_SKILLS}",
|
||||
subagent_type="msd-verifier",
|
||||
model="{verifier_model}"
|
||||
)
|
||||
```
|
||||
|
||||
The verifier reads the phase's `*-UAT.md` itself (`verification.uat-evidence`): a human item
|
||||
whose UAT row passed on unchanged code comes back verified, and only a row whose tested code
|
||||
changed comes back for re-test.
|
||||
|
||||
Re-verifying several phases (progress's batch): dispatch one verifier per phase, in
|
||||
parallel, and run steps 2–3 for each as it returns.
|
||||
|
||||
## 2. Merge human items into the UAT file — never overwrite it
|
||||
|
||||
```bash
|
||||
SEED=$(msd_run query verification.seed-uat "$PHASE_DIR")
|
||||
```
|
||||
|
||||
Run this only when the fresh status is `human_needed`. It keeps every existing row and
|
||||
result, appends unseen items as `[pending]`, and resets a passing row only when the
|
||||
verifier recorded a `retest_reason` (written into the row). When `.changed` is `true`,
|
||||
commit `.uat_file`:
|
||||
|
||||
```bash
|
||||
msd_run query commit "test({phase_num}): merge human verification items into UAT" --files "$(printf '%s' "$SEED" | jq -r '.uat_file')"
|
||||
```
|
||||
|
||||
## 3. Route on the fresh status
|
||||
|
||||
```bash
|
||||
STATUS=$(msd_run query verification.status "$PHASE_DIR" --pick status)
|
||||
```
|
||||
|
||||
| Fresh status | Then |
|
||||
|---|---|
|
||||
| `passed` | Done — the phase is verified again. No human was asked anything. |
|
||||
| `human_needed`, `SEED.pending` is `0` | Every UAT row may already pass — record it: `msd_run query verification.canonicalize-uat "$PHASE_DIR"`. `canonicalized: true` → `passed`, no test re-run. Otherwise present its `blockers` and route to `/msd:verify-work {phase}`. |
|
||||
| `human_needed`, `SEED.pending` > `0` | Only the pending rows need a human — `/msd:verify-work {phase}` resumes at the first one. |
|
||||
| `gaps_found` | A real regression: `/msd:plan-phase {phase} --gaps`. |
|
||||
| `stale` again | Stop and present the report — never loop on it (#4623 covers what the digest hashes). |
|
||||
58
msd-core/references/verifier-uat-evidence.md
Normal file
58
msd-core/references/verifier-uat-evidence.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Verifier UAT Evidence
|
||||
|
||||
> Loaded by `agents/msd-verifier.md` at Step 8b, only when the phase directory holds a
|
||||
> `*-UAT.md`. A human who already ran a check is evidence. Re-asking for it on every
|
||||
> verifier re-run — after a gap-closure plan, or after a covered file drifted — is the
|
||||
> loop this gate exists to end. `msd_run` is the launcher shim from the agent's Step 1.
|
||||
|
||||
## 1. Read the evidence
|
||||
|
||||
Pass the ROOT-relative implementation files you are about to declare in `covered_files`:
|
||||
|
||||
```bash
|
||||
msd_run query verification.uat-evidence "$PHASE_DIR" {impl file}...
|
||||
```
|
||||
|
||||
The result carries `rows` (each UAT row: `test`, `name`, `result`, `passing`, `expected`),
|
||||
`recorded_at` (when the UAT file last changed), and `changed_since_uat` (covered
|
||||
implementation files modified, or gone, AFTER the UAT was recorded). An empty `uat_file`
|
||||
means there is no UAT evidence — skip this gate.
|
||||
|
||||
## 2. Settle each Step 8 human item against its row
|
||||
|
||||
Match by meaning, not by wording: a row covers an item when a human performing the row's
|
||||
check would have observed what the item asks for.
|
||||
|
||||
| Row for the item | `changed_since_uat` | Outcome |
|
||||
|---|---|---|
|
||||
| `passing: true` | empty, or no changed file bears on what the row exercised | **Verified by human UAT.** Remove the item from the human verification list. Record it under `human_verified` (below) and mark the truth `✓ VERIFIED (human UAT, row N)`. |
|
||||
| `passing: true` | a changed file bears on what the row exercised | **Re-test.** Keep the item in `human_verification` with `retest_reason` naming the file and `recorded_at`. Use the row's exact `name` as the item's `test`. |
|
||||
| not passing (`pending`, `issue`, `blocked`, `skipped`, `missing`) | any | Ordinary human item. Use the row's exact `name` as `test`; no `retest_reason`. |
|
||||
| no row | any | Ordinary new human item. |
|
||||
|
||||
Rules:
|
||||
|
||||
- **Unsure whether a changed file affects a row → re-test.** A changed file you cannot
|
||||
rule out is a changed file that bears on it. Real change still costs a re-test.
|
||||
- **A pass is only ever withdrawn with a reason.** `retest_reason` is what lets the UAT
|
||||
writer reset that row; an item re-listed without one leaves the recorded pass in place.
|
||||
- A `⚠️ PRESENT_BEHAVIOR_UNVERIFIED` truth whose row passed on unchanged code has directly
|
||||
observed behavior: it is VERIFIED by that row and leaves `behavior_unverified_items`.
|
||||
- Step 9 then runs on what is LEFT. If every human item was verified by UAT, the human
|
||||
verification section is empty and the ordinary tree yields `passed` (or `gaps_found`).
|
||||
- Never write or edit `*-UAT.md`. The orchestrator merges your `human_verification` items
|
||||
into it with `verification.seed-uat`, which keeps every recorded result.
|
||||
|
||||
## 3. Frontmatter
|
||||
|
||||
```yaml
|
||||
human_verified: # Only if UAT evidence settled at least one item
|
||||
- test: "Exact UAT row name"
|
||||
uat_row: 3
|
||||
recorded_at: "ISO time from uat-evidence"
|
||||
human_verification: # Only if status: human_needed
|
||||
- test: "Exact UAT row name"
|
||||
expected: "What should happen"
|
||||
why_human: "Why can't verify programmatically"
|
||||
retest_reason: "src/auth/session.ts changed after the UAT pass (2026-10-05T…)" # ONLY for a row that had passed
|
||||
```
|
||||
@@ -1189,6 +1189,8 @@ If `section_manifest` is `null` or `"regression-gate"` is in its `included` list
|
||||
<step name="verify_phase_goal">
|
||||
Verify phase achieved its GOAL, not just completed tasks.
|
||||
|
||||
**Gap-closure runs:** if this run executed a `gap_closure: true` plan, first follow `execute-phase/steps/gap-closure-uat-record.md` — the human's checkpoint results go into `*-UAT.md` BEFORE the verifier is dispatched, so the same run can reach `passed`.
|
||||
|
||||
```bash
|
||||
VERIFIER_SKILLS=$(msd_run query agent-skills msd-verifier)
|
||||
```
|
||||
@@ -1208,6 +1210,7 @@ Create VERIFICATION.md.
|
||||
Read these files before verification:
|
||||
- {phase_dir}/*-PLAN.md (All plans — understand intent, check must_haves)
|
||||
- {phase_dir}/*-SUMMARY.md (All summaries — cross-reference claimed vs actual)
|
||||
- {phase_dir}/*-UAT.md (If present: recorded human UAT — evidence, verifier Step 8b)
|
||||
- {requirements_path} (Requirement traceability)
|
||||
${CONTEXT_WINDOW >= 500000 ? `- {phase_dir}/*-CONTEXT.md (User decisions — verify they were honored)
|
||||
- {phase_dir}/*-RESEARCH.md (Known pitfalls — check for traps)
|
||||
@@ -1235,60 +1238,30 @@ Route on `$STATUS`: if `passed`, proceed to update_roadmap. Otherwise keep the p
|
||||
|
||||
**If human_needed:**
|
||||
|
||||
**Step A: Persist human verification items as UAT file.**
|
||||
**Step A: Merge human verification items into the UAT file — never overwrite it.**
|
||||
|
||||
Create `{phase_dir}/{phase_num}-UAT.md` using UAT template format:
|
||||
|
||||
```markdown
|
||||
---
|
||||
status: testing
|
||||
phase: {phase_num}-{phase_name}
|
||||
source: [{phase_num}-VERIFICATION.md]
|
||||
started: [now ISO]
|
||||
updated: [now ISO]
|
||||
---
|
||||
|
||||
## Current Test
|
||||
|
||||
number: 1
|
||||
name: {first human_verification item description}
|
||||
expected: |
|
||||
{expected behavior from VERIFICATION.md}
|
||||
awaiting: user response
|
||||
|
||||
## Tests
|
||||
|
||||
{For each human_verification item from VERIFICATION.md:}
|
||||
|
||||
### {N}. {item description}
|
||||
expected: {expected behavior from VERIFICATION.md}
|
||||
result: [pending]
|
||||
|
||||
## Summary
|
||||
|
||||
total: {count}
|
||||
passed: 0
|
||||
issues: 0
|
||||
pending: {count}
|
||||
skipped: 0
|
||||
blocked: 0
|
||||
|
||||
## Gaps
|
||||
```
|
||||
|
||||
Commit the file:
|
||||
```bash
|
||||
msd_run query commit "test({phase_num}): persist human verification items as UAT" --files "{phase_dir}/{phase_num}-UAT.md"
|
||||
SEED=$(msd_run query verification.seed-uat "$PHASE_DIR")
|
||||
SEED_UAT=$(printf '%s' "$SEED" | jq -r '.uat_file // empty')
|
||||
SEED_PENDING=$(printf '%s' "$SEED" | jq -r '.pending // 0')
|
||||
```
|
||||
|
||||
`verification.seed-uat` creates `{phase_num}-UAT.md` when none exists, and otherwise MERGES: every existing row and its recorded result is kept, only unseen items are appended as `[pending]`, and a passing row is reset only when the verifier recorded a `retest_reason` for it. Do NOT hand-write or regenerate the UAT file. When `.changed` is `true`, commit it:
|
||||
|
||||
```bash
|
||||
msd_run query commit "test({phase_num}): persist human verification items as UAT" --files "$SEED_UAT"
|
||||
```
|
||||
|
||||
**If `SEED_PENDING` is `0`** — every UAT row already has a result, so no human test is owed. Do NOT present the items as tests to run. Say so, and hand off: `/msd:verify-work {X} ${MSD_WS}` records the pass against the report at once (`verification.canonicalize-uat`) and completes the phase without re-asking a single row. Skip Step B.
|
||||
|
||||
**Step B: Present to user**:
|
||||
|
||||
```
|
||||
## ◷ Phase {X}: {Name} — Human Verification Needed
|
||||
|
||||
All automated checks passed. {N} item(s) require human testing before this phase can be marked complete:
|
||||
All automated checks passed. {SEED_PENDING} item(s) require human testing before this phase can be marked complete:
|
||||
|
||||
{From VERIFICATION.md human_verification section}
|
||||
{The pending rows of {phase_num}-UAT.md — rows that already passed are NOT re-tested}
|
||||
|
||||
Tests saved to `{phase_num}-UAT.md`.
|
||||
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
Apply response_language to all user-facing prose — narration between tool calls, status updates, progress notes, and findings included; preserve code, paths, and identifiers.
|
||||
|
||||
<step name="gap_closure_uat_record">
|
||||
A gap-closure plan often ends in a human checkpoint that re-runs the UAT checks which failed.
|
||||
The human answers inside this run. If those answers are not written down before the verifier
|
||||
is dispatched, the verifier sees no evidence, reports `human_needed` again, and the user is
|
||||
sent to repeat a UAT they finished minutes ago.
|
||||
|
||||
**Skip if** no `gap_closure: true` plan executed in this run, or none of them had a
|
||||
`checkpoint:human-verify` / `checkpoint:human-action` task whose response reported on UAT
|
||||
checks.
|
||||
|
||||
**1. Record each reported outcome in the phase's `*-UAT.md`.** For every UAT row the human
|
||||
reported on at the checkpoint, edit ONLY that row:
|
||||
|
||||
- passed → `result: pass`, plus `reported: "{verbatim response}"` and
|
||||
`retested_by: {gap plan id}`
|
||||
- failed → `result: issue`, `reported: "{verbatim response}"`, `severity: {inferred}`
|
||||
|
||||
Leave every other row untouched. Do not invent results for rows the human did not mention.
|
||||
Recompute the `## Summary` counters; set frontmatter `status: complete` when no row is
|
||||
`[pending]`, and refresh `updated:`.
|
||||
|
||||
**2. Resolve the gaps the plan closed.** For each `## Gaps` entry named in an executed plan's
|
||||
`gap_ids` whose row now passes: `status: resolved`, with `resolved_by` / `resolved_at`.
|
||||
|
||||
**3. Commit before verifying:**
|
||||
|
||||
```bash
|
||||
_MSD_SHIM_NAME="msd-tools.cjs"; _MSD_RUNTIME_ROOT="${RUNTIME_DIR:-$(git rev-parse --show-toplevel 2>/dev/null || pwd)}"; MSD_TOOLS="${_MSD_RUNTIME_ROOT}/msd-core/bin/${_MSD_SHIM_NAME}"; _msd_at() { for _p; do if [ -f "$_p" ]; then MSD_TOOLS="$_p"; return 0; fi; done; return 1; }; _msd_id_ok() { case "$("$1" runtime-identity --raw 2>/dev/null || true)" in '{"packageName":"@golem15/msd-core"'*'}') return 0;; *) return 1;; esac; }; _msd_homes() { _msd_at "${CLAUDE_CONFIG_DIR:-$HOME/.claude}/msd-core/bin/${_MSD_SHIM_NAME}" "${CURSOR_CONFIG_DIR:-$HOME/.cursor}/msd-core/bin/${_MSD_SHIM_NAME}" "${CODEX_HOME:-$HOME/.codex}/msd-core/bin/${_MSD_SHIM_NAME}" "${GEMINI_CONFIG_DIR:-$HOME/.gemini}/msd-core/bin/${_MSD_SHIM_NAME}" "${GROK_AGENTS_HOME:-$HOME/.agents}/msd-core/bin/${_MSD_SHIM_NAME}" "${ANTIGRAVITY_CONFIG_DIR:-$HOME/.gemini/antigravity}/msd-core/bin/${_MSD_SHIM_NAME}" "${OPENCODE_CONFIG_DIR:-${XDG_CONFIG_HOME:-$HOME/.config}/opencode}/msd-core/bin/${_MSD_SHIM_NAME}"; }; if _msd_at "${_MSD_RUNTIME_ROOT}/msd-core/bin/${_MSD_SHIM_NAME}" "${_MSD_RUNTIME_ROOT}/.claude/msd-core/bin/${_MSD_SHIM_NAME}" "${_MSD_RUNTIME_ROOT}/.codex/msd-core/bin/${_MSD_SHIM_NAME}"; then msd_run() { node "$MSD_TOOLS" "$@"; }; elif _msd_homes; then msd_run() { node "$MSD_TOOLS" "$@"; }; elif unset -f msd_run; _G="$(command -v msd_run)"; [ -n "$_G" ] && _msd_id_ok "$_G"; then MSD_TOOLS="$_G"; msd_run() { "$MSD_TOOLS" "$@"; }; else echo "ERROR: msd-tools.cjs not found at $MSD_TOOLS and no identity-proving msd_run is on PATH. Run: npx -y @golem15/msd-core@latest --claude --local" >&2; exit 1; fi; MSD_IDENTITY_STATUS=unverified; _msd_id_ok msd_run && MSD_IDENTITY_STATUS=ok; export MSD_IDENTITY_STATUS; [ "$MSD_IDENTITY_STATUS" = ok ] || echo "WARNING: \"$MSD_TOOLS\" did not prove it is @golem15/msd-core - it is either a different package or an @golem15/msd-core older than the runtime-identity verb. See docs/how-to/diagnose-a-foreign-msd-tools.md" >&2; if [ -n "${CLAUDE_ENV_FILE:-}" ] && [ -n "${MSD_TOOLS:-}" ]; then printf "export PATH='%s':\"\$PATH\"\n" "${MSD_TOOLS%/*}" >> "$CLAUDE_ENV_FILE" 2>/dev/null || true; fi
|
||||
msd_run query commit "test({phase_num}): record gap-closure UAT results" --files "{phase_dir}/{phase_num}-UAT.md"
|
||||
```
|
||||
|
||||
The verifier dispatched next reads these rows as evidence (`verification.uat-evidence`), so a
|
||||
gap closure whose UAT passed ends this run at `passed` — no second command.
|
||||
</step>
|
||||
@@ -1,9 +1,10 @@
|
||||
Apply response_language to all user-facing prose — narration between tool calls, status updates, progress notes, and findings included; preserve code, paths, and identifiers.
|
||||
|
||||
<step name="stale_reverification">
|
||||
Covered source files changed after the verifier last ran — the recorded digest no longer
|
||||
matches, so the report cannot be trusted until the verifier re-runs (#4682). The plans are all
|
||||
summarized: there is no wave work to do.
|
||||
The report no longer matches the phase — a covered file changed, or the phase gained a
|
||||
PLAN/SUMMARY the report never declared (`verification.status` names which in `stale_reason`)
|
||||
— so it cannot be trusted until the verifier re-runs (#4682). The plans are all summarized:
|
||||
there is no wave work to do, and no plan will execute.
|
||||
|
||||
Report:
|
||||
```
|
||||
@@ -19,6 +20,10 @@ digest. This holds whether or not the phase was already marked complete — a st
|
||||
marked phase is refreshed the same way. `verification.status`'s `next_command` routes here for
|
||||
exactly this state.
|
||||
|
||||
Recorded UAT results survive this re-run: the verifier reads `*-UAT.md` as evidence, and the
|
||||
`human_needed` branch merges into it (`verification.seed-uat`) instead of rewriting it — the
|
||||
shared procedure is `msd-core/references/stale-reverification.md`.
|
||||
|
||||
Never silently proceed past a stale gate: if the re-run verifier still produces a stale report,
|
||||
stop and present it (#4623 covers what the digest hashes).
|
||||
</step>
|
||||
|
||||
@@ -321,7 +321,7 @@ VERIFICATION_STATUS=$(printf '%s' "$VERIFICATION" | jq -r '.status' 2>/dev/null
|
||||
VERIFICATION_NEXT_ACTION=$(printf '%s' "$VERIFICATION" | jq -r '.next_action' 2>/dev/null || echo "")
|
||||
```
|
||||
|
||||
Track: `verification_status` — the `.status` field (`passed | stale | gaps_found | human_needed | missing | unknown`). The query/projection handles a missing VERIFICATION.md (`missing`), unexpected values, and stale verification (`stale`, when summaries are newer than verification). Only `passed` routes as phase complete (Step 3); every other status routes back to close verification debt (Step 2).
|
||||
Track: `verification_status` — the `.status` field (`passed | stale | gaps_found | human_needed | missing | unknown`). The query/projection handles a missing VERIFICATION.md (`missing`), unexpected values, and stale verification (`stale` — the report no longer matches the phase; `stale_reason` says why). Only `passed` routes as phase complete (Step 3); every other status routes back to close verification debt (Step 2).
|
||||
|
||||
**Step 2: Route based on counts**
|
||||
|
||||
@@ -532,12 +532,22 @@ VERIFICATION.md has an unexpected status. The phase is implementation-complete,
|
||||
|
||||
**Route V.stale: verification is stale**
|
||||
|
||||
VERIFICATION.md has `status: passed`, but one or more SUMMARY.md files are newer than the verification report. The phase is implementation-complete, not phase-complete.
|
||||
Every plan is summarized; the only open item is verification bookkeeping. The report no longer matches the phase — a gap-closure plan it never saw, or a covered file edited since (often by a LATER phase). No plan needs to execute, so do NOT route to `/msd:execute-phase`: re-run the verifier here.
|
||||
|
||||
```bash
|
||||
REVERIFY=$(printf '%s' "$INIT" | jq -c '.reverify_phases // []')
|
||||
```
|
||||
|
||||
`reverify_phases` lists EVERY phase in this state, not just the current one. Handle them as ONE action:
|
||||
|
||||
```
|
||||
`/msd:verify-work {phase} ${MSD_WS}` — re-run verification against the latest summaries
|
||||
## Re-verifying {N} phase(s): {numbers}
|
||||
|
||||
Their reports are stale ({stale_reason per phase}). Re-running the verifier — no plans will execute, and recorded UAT results are kept.
|
||||
```
|
||||
|
||||
Read and follow `msd-core/references/stale-reverification.md` for each listed phase (verifiers in parallel). Then re-run `init.progress` and route again from Step 2 — a phase the verifier returned as `passed` is complete; one with rows genuinely needing a re-test goes to **Route V.human**, naming only those rows.
|
||||
|
||||
---
|
||||
|
||||
**Route V.gaps: verification found gaps (gaps_found)**
|
||||
|
||||
@@ -652,44 +652,45 @@ If an active secure-phase step hook exists AND `SECURITY_FILE` exists: check fro
|
||||
|
||||
If no active secure-phase step hook exists OR (`SECURITY_FILE` exists AND `threats_open` is `0`):
|
||||
|
||||
If execution verification is waiting only on human UAT and this session recorded zero issues, canonicalize the report before the shared completion predicate. (#4663) Zero issues is NOT pass evidence on its own — blocked rows are not issues by this workflow's own rule, so a session that observed nothing (0 passed / 0 issues / N blocked) must NOT flip the report. The flip runs the SAME UAT-row predicate the phase-close uses, in its `--uat-only` form: it skips the verification-status blockers (the report still reads `human_needed` at this point — the full predicate could never pass here), and `passed` means at least one UAT check passed with no row pending/blocked/failed or skipped without a reason. The flagged transition-gate call below stays the final say on canonical verification:
|
||||
Read the canonical verification status first:
|
||||
|
||||
```bash
|
||||
PHASE_DIR=$(printf '%s' "$INIT" | jq -r '.phase_dir // empty')
|
||||
VERIFICATION_FILE=$(msd_run query verification.resolve-file "$PHASE_DIR" --raw 2>/dev/null)
|
||||
VERIFICATION_STATUS=$(msd_run query verification.status "$PHASE_DIR" 2>/dev/null)
|
||||
VERIFICATION_STATUS_VALUE=$(printf '%s' "$VERIFICATION_STATUS" | jq -r '.status // empty' 2>/dev/null || echo "")
|
||||
PHASE_VERIFICATION_STATUS="$VERIFICATION_STATUS_VALUE"
|
||||
if [ "$VERIFICATION_STATUS_VALUE" = "human_needed" ]; then
|
||||
UAT_PRECHECK=$(msd_run phase uat-passed "{phase}" --uat-only 2>/dev/null)
|
||||
UAT_PRECHECK_PASSED=$(printf '%s' "$UAT_PRECHECK" | jq -r '.passed // false' 2>/dev/null || echo "false")
|
||||
if [ "$UAT_PRECHECK_PASSED" = "true" ]; then
|
||||
msd_run query frontmatter.set "$VERIFICATION_FILE" --field status --value passed
|
||||
else
|
||||
UAT_BLOCKERS=$(printf '%s' "$UAT_PRECHECK" | jq -r '.blockers | length' 2>/dev/null)
|
||||
[ -n "$UAT_BLOCKERS" ] || UAT_BLOCKERS="?"
|
||||
echo "NOT canonicalizing: ${UAT_BLOCKERS} UAT row(s) blocked or not passing; verification stays human_needed. Resolve or pass them, then re-run /msd:verify-work {phase}." >&2
|
||||
fi
|
||||
fi
|
||||
PHASE_VERIFICATION_STATUS=$(printf '%s' "$VERIFICATION_STATUS" | jq -r '.status // empty' 2>/dev/null || echo "")
|
||||
```
|
||||
|
||||
If `PHASE_VERIFICATION_STATUS` is `stale`, the covered source files changed after the verifier
|
||||
last ran — re-run the VERIFIER, not this workflow (`/msd:verify-work` never rewrites
|
||||
VERIFICATION.md; its only write is the human_needed canonicalization, #4663). Spawn the
|
||||
verifier for this phase exactly as execute-phase's `verify_phase_goal` step does (subagent
|
||||
`msd-verifier`; phase directory, goal, requirement IDs, and all SUMMARYs in
|
||||
`<required_reading>`), then re-read `verification.status` and continue at the fresh/passed
|
||||
case below. (#4682)
|
||||
If `PHASE_VERIFICATION_STATUS` is `stale`, the report no longer matches the phase — a gap-closure
|
||||
plan added a PLAN/SUMMARY it never declared, or a covered file changed (`stale_reason`). Re-run the
|
||||
VERIFIER here, in this session — do NOT send the user to `/msd:execute-phase` for a phase whose
|
||||
plans are all summarized. Read and follow `msd-core/references/stale-reverification.md`: dispatch
|
||||
`msd-verifier` exactly as execute-phase's `verify_phase_goal` step does, merge any human items
|
||||
into the UAT file with `verification.seed-uat` (recorded results are kept), then re-read
|
||||
`verification.status` into `PHASE_VERIFICATION_STATUS`. (#4682)
|
||||
|
||||
- Fresh status `human_needed` with new `[pending]` rows → go to `resume_from_file`; only those
|
||||
rows are tested.
|
||||
- Fresh status still `stale`, or `gaps_found` → stop and present it:
|
||||
|
||||
```
|
||||
Verification is stale: covered source files changed after the verifier last ran.
|
||||
Verification could not be refreshed: {fresh status}
|
||||
|
||||
Blocking completion:
|
||||
verification is stale
|
||||
{verification.status next_action}
|
||||
```
|
||||
|
||||
- Re-run the verifier for phase {phase} (dispatch `msd-verifier` as in execute-phase's
|
||||
verify_phase_goal step) to regenerate VERIFICATION.md with a fresh digest, then re-run
|
||||
`/msd:verify-work {phase}`
|
||||
- Otherwise continue below.
|
||||
|
||||
If execution verification is waiting only on human UAT, canonicalize the report before the shared completion predicate. (#4663) Zero issues is NOT pass evidence on its own — blocked rows are not issues by this workflow's own rule, so a session that observed nothing (0 passed / 0 issues / N blocked) must NOT flip the report. `verification.canonicalize-uat` is the single seam for the flip: it acts only on a `human_needed` report (never a `stale` one) and runs the SAME UAT-row predicate the phase-close uses, in its uat-only form — at least one UAT check passed with no row pending/blocked/failed or skipped without a reason. The flagged transition-gate call below stays the final say on canonical verification:
|
||||
|
||||
```bash
|
||||
CANON=$(msd_run query verification.canonicalize-uat "$PHASE_DIR" 2>/dev/null)
|
||||
CANON_REASON=$(printf '%s' "$CANON" | jq -r '.reason // empty' 2>/dev/null || echo "")
|
||||
if [ "$CANON_REASON" = "uat_not_passed" ]; then
|
||||
UAT_BLOCKERS=$(printf '%s' "$CANON" | jq -r '.blockers | length' 2>/dev/null)
|
||||
[ -n "$UAT_BLOCKERS" ] || UAT_BLOCKERS="?"
|
||||
echo "NOT canonicalizing: ${UAT_BLOCKERS} UAT row(s) blocked or not passing; verification stays human_needed. Resolve or pass them, then re-run /msd:verify-work {phase}." >&2
|
||||
fi
|
||||
```
|
||||
|
||||
Otherwise, check the shared UAT-plus-verification completion predicate before transition:
|
||||
@@ -708,8 +709,8 @@ All UAT tests passed, but phase advancement is blocked until canonical verificat
|
||||
Blocking completion:
|
||||
{PHASE_COMPLETE_BLOCKERS}
|
||||
|
||||
- `/msd:execute-phase {phase}` — regenerate execution verification
|
||||
- `/msd:verify-work {phase}` — resume UAT if blockers remain
|
||||
- `/msd:plan-phase {phase} --gaps` — if verification reports gaps
|
||||
```
|
||||
|
||||
**Auto-transition: mark phase complete in ROADMAP.md and STATE.md**
|
||||
|
||||
18
src/init.cts
18
src/init.cts
@@ -272,6 +272,12 @@ interface PhaseCompletionProjection {
|
||||
* complete) or ran to completion.
|
||||
*/
|
||||
verification_stale_check_indeterminate: boolean;
|
||||
/**
|
||||
* WHY the report reads `stale` (`readVerificationStatus`'s `stale_reason`),
|
||||
* '' for every other status. Lets progress tell a gap-closure plan the
|
||||
* report never saw from a covered file drifting under a finished phase.
|
||||
*/
|
||||
verification_stale_reason: string;
|
||||
}
|
||||
|
||||
|
||||
@@ -330,6 +336,7 @@ function buildPhaseCompletionProjection(
|
||||
// internal staleness check could not run to completion.
|
||||
verification_stale_check_indeterminate: 'staleCheckIndeterminate' in verificationStatus
|
||||
&& verificationStatus.staleCheckIndeterminate === true,
|
||||
verification_stale_reason: verificationStatus.stale_reason ?? '',
|
||||
};
|
||||
}
|
||||
|
||||
@@ -3872,6 +3879,17 @@ function cmdInitProgress(cwd: string, raw: boolean, options: Record<string, unkn
|
||||
phases,
|
||||
phase_count: phases.length,
|
||||
completed_count: phases.filter((p) => p['status'] === 'complete').length,
|
||||
// Every phase whose plans are all summarized and whose ONLY open item is a
|
||||
// stale report — one verifier re-run each, no plan executes. Progress
|
||||
// offers these as a single "re-verify" action instead of walking the user
|
||||
// back through them one execute-phase route per session.
|
||||
reverify_phases: phases
|
||||
.filter((p) => p['implementation_complete'] === true && p['verification_status'] === 'stale')
|
||||
.map((p) => ({
|
||||
number: p['number'],
|
||||
directory: p['directory'],
|
||||
stale_reason: p['verification_stale_reason'],
|
||||
})),
|
||||
in_progress_count: phases.filter((p) =>
|
||||
['executed', 'in_progress'].includes(p['status'] as string),
|
||||
).length,
|
||||
|
||||
735
src/uat-evidence.cts
Normal file
735
src/uat-evidence.cts
Normal file
@@ -0,0 +1,735 @@
|
||||
/**
|
||||
* UAT Evidence — the seam that lets a recorded human UAT survive a verifier
|
||||
* re-run.
|
||||
*
|
||||
* A phase's `*-VERIFICATION.md` goes `stale` whenever the verifier has to look
|
||||
* again: a gap-closure plan added a PLAN/SUMMARY it never declared, or a
|
||||
* covered source file changed. Re-running the verifier is the right remedy —
|
||||
* but before this module nothing carried the human's UAT results across that
|
||||
* re-run. The verifier could not see `*-UAT.md`, so it re-emitted the same
|
||||
* `human_needed` items, and execute-phase's `human_needed` branch then wrote a
|
||||
* fresh `*-UAT.md` over the completed one, every row back to `[pending]`. The
|
||||
* user was asked for the same UAT again, indefinitely.
|
||||
*
|
||||
* Two verbs close that loop, both under the `verification` family:
|
||||
*
|
||||
* - `verification.uat-evidence <phaseDir> [files…]` (read-only) — the
|
||||
* recorded UAT rows, plus which covered implementation files changed
|
||||
* AFTER the UAT was recorded. The verifier is an LLM, not a clock: this
|
||||
* does the deterministic part so the agent only decides which rows a
|
||||
* changed file actually affects.
|
||||
* - `verification.seed-uat <phaseDir>` — writes the report's
|
||||
* `human_verification` items into `*-UAT.md` by MERGING: existing rows and
|
||||
* their results are kept byte-for-byte, only unseen items are appended as
|
||||
* `[pending]`, and a passing row is reset only when the verifier recorded
|
||||
* a `retest_reason` for it (the reason is written into the row).
|
||||
*
|
||||
* - `verification.canonicalize-uat <phaseDir>` — the ONE place a
|
||||
* `human_needed` report becomes `passed` on the strength of a passed UAT
|
||||
* (#4663). verify-work, execute-phase and progress all call it, so the
|
||||
* rule cannot be re-derived differently per workflow.
|
||||
*
|
||||
* The verifier remains the only emitter of a verdict it reached itself; the
|
||||
* canonicalization records the human's half of a `human_needed` verdict, and
|
||||
* refuses every other status — a `stale` report is never flipped.
|
||||
*
|
||||
* ADR-457 build-at-publish: compiled by tsc to msd-core/bin/lib/uat-evidence.cjs.
|
||||
*/
|
||||
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { findProjectRoot } from './project-root.cjs';
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- io.cjs is an export= CommonJS module
|
||||
import io = require('./io.cjs');
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- phase-id.cjs is an export= CommonJS module
|
||||
import phaseId = require('./phase-id.cjs');
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- frontmatter.cjs is an export= CommonJS module
|
||||
import frontmatterMod = require('./frontmatter.cjs');
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- core-utils.cjs is an export= CommonJS module
|
||||
import coreUtilsMod = require('./core-utils.cjs');
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- verification.cjs is an export= CommonJS module
|
||||
import verification = require('./verification.cjs');
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- uat-predicate.cjs is an export= CommonJS module
|
||||
import uatPredicate = require('./uat-predicate.cjs');
|
||||
|
||||
const { output, error } = io;
|
||||
const { evaluateUatPassed } = uatPredicate;
|
||||
const { extractPhaseToken } = phaseId;
|
||||
const { extractFrontmatter, frontmatterListEntries, spliceFrontmatter } = frontmatterMod;
|
||||
const { normalizeLineEndings } = coreUtilsMod;
|
||||
const {
|
||||
resolveUatFile,
|
||||
resolveVerificationFile,
|
||||
readVerificationStatus,
|
||||
defaultPhaseCleanCommitTimesMs,
|
||||
canonicalizeCoveredFiles,
|
||||
parseFingerprintFileArgs,
|
||||
} = verification;
|
||||
|
||||
// ─── Types ────────────────────────────────────────────────────────────────────
|
||||
|
||||
interface UatRow {
|
||||
test: number;
|
||||
name: string;
|
||||
/** Lowercased `result:` value without brackets; `missing` when the row has none. */
|
||||
result: string;
|
||||
passing: boolean;
|
||||
expected: string;
|
||||
/** Present when a previous seed reset this row for re-test. */
|
||||
retest_reason: string;
|
||||
/** Line indexes into the scanned document (internal to the merge). */
|
||||
headingLine: number;
|
||||
resultLine: number;
|
||||
endLine: number;
|
||||
}
|
||||
|
||||
interface HumanItem {
|
||||
test: string;
|
||||
expected: string;
|
||||
why_human: string;
|
||||
/** Set by the verifier ONLY when it re-lists an item whose UAT row already passed. */
|
||||
retest_reason: string;
|
||||
}
|
||||
|
||||
type PublicRow = Pick<UatRow, 'test' | 'name' | 'result' | 'passing' | 'expected' | 'retest_reason'>;
|
||||
|
||||
interface UatEvidence {
|
||||
uat_file: string;
|
||||
uat_status: string;
|
||||
/** ISO time the UAT file last changed (commit time when clean, else mtime); '' when no UAT file. */
|
||||
recorded_at: string;
|
||||
rows: PublicRow[];
|
||||
/** Implementation files compared against `recorded_at` (`.planning/` paths are never compared). */
|
||||
checked_files: string[];
|
||||
changed_since_uat: Array<{ file: string; reason: 'modified' | 'missing' }>;
|
||||
/** True only when a UAT file exists and no checked file changed after it. */
|
||||
code_unchanged_since_uat: boolean;
|
||||
}
|
||||
|
||||
interface SeedUatResult {
|
||||
uat_file: string;
|
||||
created: boolean;
|
||||
changed: boolean;
|
||||
kept: Array<{ test: number; name: string; result: string }>;
|
||||
appended: Array<{ test: number; name: string }>;
|
||||
retested: Array<{ test: number; name: string; previous_result: string; reason: string }>;
|
||||
pending: number;
|
||||
/** Every row passes — verify-work can canonicalize without asking the human anything. */
|
||||
all_passing: boolean;
|
||||
}
|
||||
|
||||
type CleanCommitTimesFn = (dir: string, files: string[]) => Map<string, number>;
|
||||
|
||||
// ─── Row scanning ─────────────────────────────────────────────────────────────
|
||||
|
||||
const PASSING_RESULTS: ReadonlySet<string> = new Set(['pass', 'passed']);
|
||||
const HEADING_RE = /^###\s*(\d+)\.\s*(.+)$/;
|
||||
const SECTION_RE = /^##\s+\S/;
|
||||
const RESULT_RE = /^result:[ \t]*\[?([\w-]+)\]?/i;
|
||||
const FENCE_RE = /^\s{0,3}(`{3,}|~{3,})/;
|
||||
|
||||
/** Index of the first body line — the line after a byte-0 frontmatter block, or 0. */
|
||||
function bodyStart(lines: readonly string[]): number {
|
||||
if (lines[0] !== '---') return 0;
|
||||
for (let i = 1; i < lines.length; i++) {
|
||||
if (/^---[ \t]*$/.test(lines[i])) return i + 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Scan a `*-UAT.md` body for `### N. Name` test rows, with the line positions
|
||||
* the merge needs. Headings and `result:` lines inside fenced code or an HTML
|
||||
* comment are not rows — the same contexts `uat-predicate.cts` discards, so a
|
||||
* row this scan would rewrite is always one the completion predicate reads.
|
||||
*/
|
||||
function scanUatRows(lines: readonly string[]): UatRow[] {
|
||||
const rows: UatRow[] = [];
|
||||
let current: UatRow | null = null;
|
||||
let fence: string | null = null;
|
||||
let inComment = false;
|
||||
|
||||
const close = (endLine: number): void => {
|
||||
if (current) {
|
||||
current.endLine = endLine;
|
||||
rows.push(current);
|
||||
current = null;
|
||||
}
|
||||
};
|
||||
|
||||
for (let i = bodyStart(lines); i < lines.length; i++) {
|
||||
const line = lines[i];
|
||||
if (inComment) {
|
||||
if (line.includes('-->')) inComment = false;
|
||||
continue;
|
||||
}
|
||||
const fenceMatch = FENCE_RE.exec(line);
|
||||
if (fence) {
|
||||
if (fenceMatch && fenceMatch[1][0] === fence[0] && fenceMatch[1].length >= fence.length) fence = null;
|
||||
continue;
|
||||
}
|
||||
if (fenceMatch) {
|
||||
fence = fenceMatch[1];
|
||||
continue;
|
||||
}
|
||||
if (line.includes('<!--') && !line.includes('-->', line.indexOf('<!--'))) {
|
||||
inComment = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
const heading = HEADING_RE.exec(line);
|
||||
if (heading) {
|
||||
close(i);
|
||||
current = {
|
||||
test: Number.parseInt(heading[1], 10),
|
||||
name: heading[2].trim(),
|
||||
result: 'missing',
|
||||
passing: false,
|
||||
expected: '',
|
||||
retest_reason: '',
|
||||
headingLine: i,
|
||||
resultLine: -1,
|
||||
endLine: lines.length,
|
||||
};
|
||||
continue;
|
||||
}
|
||||
if (SECTION_RE.test(line)) {
|
||||
close(i);
|
||||
continue;
|
||||
}
|
||||
if (!current) continue;
|
||||
|
||||
// FIRST match wins for every key, matching uat-predicate's contract.
|
||||
const result = RESULT_RE.exec(line);
|
||||
if (result && current.resultLine === -1) {
|
||||
current.result = result[1].toLowerCase();
|
||||
current.passing = PASSING_RESULTS.has(current.result);
|
||||
current.resultLine = i;
|
||||
continue;
|
||||
}
|
||||
const expected = /^expected:[ \t]*(.*)$/.exec(line);
|
||||
if (expected && current.expected === '') {
|
||||
const inline = expected[1].trim();
|
||||
if (inline === '|' || inline === '>' || inline === '') {
|
||||
const block: string[] = [];
|
||||
for (let j = i + 1; j < lines.length && /^\s+\S/.test(lines[j]); j++) block.push(lines[j].trim());
|
||||
current.expected = block.join(' ');
|
||||
} else {
|
||||
current.expected = inline;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const retest = /^retest_reason:[ \t]*(.*)$/.exec(line);
|
||||
if (retest && current.retest_reason === '') current.retest_reason = unquote(retest[1].trim());
|
||||
}
|
||||
close(lines.length);
|
||||
return rows;
|
||||
}
|
||||
|
||||
function unquote(value: string): string {
|
||||
if (value.length >= 2 && value.startsWith('"') && value.endsWith('"')) {
|
||||
return value.slice(1, -1).replace(/\\"/g, '"').replace(/\\\\/g, '\\');
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function quote(value: string): string {
|
||||
return `"${value.replace(/\\/g, '\\\\').replace(/"/g, '\\"')}"`;
|
||||
}
|
||||
|
||||
/** One line, whitespace collapsed — a heading or a quoted reason must never span lines. */
|
||||
function oneLine(value: string): string {
|
||||
return value.replace(/\s+/g, ' ').trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* The key a verifier item and a UAT row are matched on: lowercase, punctuation
|
||||
* stripped, whitespace collapsed — the normalization `verification-overrides`
|
||||
* already uses to match a must-have across verifier runs.
|
||||
*/
|
||||
function matchKey(name: string): string {
|
||||
return name
|
||||
.toLowerCase()
|
||||
.replace(/[^\p{L}\p{N}]+/gu, ' ')
|
||||
.trim();
|
||||
}
|
||||
|
||||
function publicRow(row: UatRow): PublicRow {
|
||||
return {
|
||||
test: row.test,
|
||||
name: row.name,
|
||||
result: row.result,
|
||||
passing: row.passing,
|
||||
expected: row.expected,
|
||||
retest_reason: row.retest_reason,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Phase file resolution ────────────────────────────────────────────────────
|
||||
|
||||
interface PhaseFiles {
|
||||
uatFile: string | null;
|
||||
verificationFile: string | null;
|
||||
}
|
||||
|
||||
function resolvePhaseFiles(phaseDir: string): PhaseFiles {
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = fs.readdirSync(phaseDir);
|
||||
} catch {
|
||||
return { uatFile: null, verificationFile: null };
|
||||
}
|
||||
const phaseDirName = path.basename(phaseDir);
|
||||
const options = { allowBare: true, phaseToken: extractPhaseToken(phaseDirName), phaseDirName };
|
||||
return {
|
||||
uatFile: resolveUatFile(entries, options),
|
||||
verificationFile: resolveVerificationFile(entries, options),
|
||||
};
|
||||
}
|
||||
|
||||
function readNormalized(filePath: string): string | null {
|
||||
try {
|
||||
return normalizeLineEndings(fs.readFileSync(filePath, 'utf-8'));
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function stringField(entry: unknown, key: string): string {
|
||||
if (entry === null || typeof entry !== 'object') return '';
|
||||
const value = (entry as Record<string, unknown>)[key];
|
||||
return typeof value === 'string' ? value.trim() : '';
|
||||
}
|
||||
|
||||
/** The report's `human_verification` items; an item without a `test` is not an item. */
|
||||
function readHumanItems(verificationContent: string): HumanItem[] {
|
||||
const entries = frontmatterListEntries(verificationContent, 'human_verification') ?? [];
|
||||
const items: HumanItem[] = [];
|
||||
for (const entry of entries) {
|
||||
const test = oneLine(stringField(entry, 'test'));
|
||||
if (test.length === 0) continue;
|
||||
items.push({
|
||||
test,
|
||||
expected: stringField(entry, 'expected'),
|
||||
why_human: stringField(entry, 'why_human'),
|
||||
retest_reason: oneLine(stringField(entry, 'retest_reason')),
|
||||
});
|
||||
}
|
||||
return items;
|
||||
}
|
||||
|
||||
// ─── uat-evidence ─────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Read the recorded UAT rows for a phase and report which implementation files
|
||||
* changed after the UAT was recorded.
|
||||
*
|
||||
* `files` defaults to the current report's own `covered_files`. Paths under
|
||||
* `.planning/` are never compared: they are bookkeeping (the SUMMARY of the
|
||||
* very plan whose checkpoint WAS the UAT is written after it), not the code a
|
||||
* human tested.
|
||||
*
|
||||
* Time is commit time for a committed-and-clean file, mtime otherwise — the
|
||||
* same clock the legacy staleness check uses (#2348), so a fresh clone and the
|
||||
* machine that recorded the UAT agree. A file that no longer exists counts as
|
||||
* changed: fail closed, the same direction as the fingerprint.
|
||||
*/
|
||||
function readUatEvidence(
|
||||
phaseDir: string,
|
||||
files?: readonly string[],
|
||||
cleanCommitTimesMs: CleanCommitTimesFn = defaultPhaseCleanCommitTimesMs,
|
||||
): UatEvidence {
|
||||
const empty: UatEvidence = {
|
||||
uat_file: '',
|
||||
uat_status: '',
|
||||
recorded_at: '',
|
||||
rows: [],
|
||||
checked_files: [],
|
||||
changed_since_uat: [],
|
||||
code_unchanged_since_uat: false,
|
||||
};
|
||||
const { uatFile, verificationFile } = resolvePhaseFiles(phaseDir);
|
||||
if (!uatFile) return empty;
|
||||
const uatPath = path.join(phaseDir, uatFile);
|
||||
const content = readNormalized(uatPath);
|
||||
if (content === null) return empty;
|
||||
|
||||
let declared: readonly string[] = files ?? [];
|
||||
if (files === undefined && verificationFile) {
|
||||
const report = readNormalized(path.join(phaseDir, verificationFile));
|
||||
const covered = report === null ? undefined : extractFrontmatter(report)['covered_files'];
|
||||
if (Array.isArray(covered)) declared = covered.filter((f): f is string => typeof f === 'string');
|
||||
}
|
||||
const checked = (canonicalizeCoveredFiles(declared)).filter(
|
||||
(f) => f !== '.planning' && !f.startsWith('.planning/') && !f.startsWith('../') && !path.isAbsolute(f),
|
||||
);
|
||||
|
||||
const projectRoot = findProjectRoot(phaseDir);
|
||||
const uatRel = path.relative(projectRoot, uatPath).replace(/\\/g, '/');
|
||||
const commitTimes = cleanCommitTimesMs(projectRoot, [uatRel, ...checked]);
|
||||
const timeOf = (rel: string): number | null => {
|
||||
const committed = commitTimes.get(rel);
|
||||
if (committed !== undefined) return committed;
|
||||
try {
|
||||
return fs.statSync(path.resolve(projectRoot, rel)).mtimeMs;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
const recordedAt = timeOf(uatRel);
|
||||
const changed: UatEvidence['changed_since_uat'] = [];
|
||||
for (const file of checked) {
|
||||
const changedAt = timeOf(file);
|
||||
if (changedAt === null) changed.push({ file, reason: 'missing' });
|
||||
else if (recordedAt === null || changedAt > recordedAt) changed.push({ file, reason: 'modified' });
|
||||
}
|
||||
|
||||
const status = extractFrontmatter(content, uatPath)['status'];
|
||||
return {
|
||||
uat_file: uatPath,
|
||||
uat_status: typeof status === 'string' ? status.trim() : '',
|
||||
recorded_at: recordedAt === null ? '' : new Date(recordedAt).toISOString(),
|
||||
rows: scanUatRows(content.split('\n')).map(publicRow),
|
||||
checked_files: checked,
|
||||
changed_since_uat: changed,
|
||||
code_unchanged_since_uat: changed.length === 0,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── seed-uat ─────────────────────────────────────────────────────────────────
|
||||
|
||||
function renderRow(test: number, item: HumanItem): string[] {
|
||||
const expected = item.expected.split('\n').map((l) => l.trim()).filter((l) => l.length > 0);
|
||||
const lines = [`### ${test}. ${item.test}`];
|
||||
if (expected.length <= 1) lines.push(`expected: ${expected[0] ?? ''}`);
|
||||
else lines.push('expected: |', ...expected.map((l) => ` ${l}`));
|
||||
lines.push('result: [pending]', '');
|
||||
return lines;
|
||||
}
|
||||
|
||||
function countResults(rows: readonly UatRow[]): Record<string, number> {
|
||||
const count = (pred: (r: UatRow) => boolean): number => rows.filter(pred).length;
|
||||
return {
|
||||
total: rows.length,
|
||||
passed: count((r) => r.passing),
|
||||
issues: count((r) => r.result === 'issue'),
|
||||
pending: count((r) => r.result === 'pending'),
|
||||
skipped: count((r) => r.result === 'skipped'),
|
||||
blocked: count((r) => r.result === 'blocked'),
|
||||
};
|
||||
}
|
||||
|
||||
/** Rewrite the `## Summary` counters that already exist; never invent a section. */
|
||||
function refreshSummary(lines: string[], rows: readonly UatRow[]): void {
|
||||
const counts = countResults(rows);
|
||||
const start = lines.findIndex((l, i) => i >= bodyStart(lines) && /^##\s+Summary\s*$/.test(l));
|
||||
if (start === -1) return;
|
||||
for (let i = start + 1; i < lines.length && !SECTION_RE.test(lines[i]); i++) {
|
||||
const m = /^(total|passed|issues|pending|skipped|blocked):/.exec(lines[i]);
|
||||
if (m) lines[i] = `${m[1]}: ${counts[m[1]]}`;
|
||||
}
|
||||
}
|
||||
|
||||
/** A re-opened session is `testing` again; only the two frontmatter lines that say so change. */
|
||||
function reopenFrontmatter(lines: string[], nowIso: string): void {
|
||||
const end = bodyStart(lines);
|
||||
for (let i = 1; i < end - 1; i++) {
|
||||
if (/^status:/.test(lines[i])) lines[i] = 'status: testing';
|
||||
else if (/^updated:/.test(lines[i])) lines[i] = `updated: ${nowIso}`;
|
||||
}
|
||||
}
|
||||
|
||||
function newUatDocument(phaseLabel: string, sourceFile: string, items: readonly HumanItem[], nowIso: string): string {
|
||||
const first = items[0];
|
||||
const lines = [
|
||||
'---',
|
||||
'status: testing',
|
||||
`phase: ${phaseLabel}`,
|
||||
`source: [${sourceFile}]`,
|
||||
`started: ${nowIso}`,
|
||||
`updated: ${nowIso}`,
|
||||
'---',
|
||||
'',
|
||||
'## Current Test',
|
||||
'',
|
||||
'number: 1',
|
||||
`name: ${first.test}`,
|
||||
'expected: |',
|
||||
...first.expected.split('\n').map((l) => ` ${l.trim()}`),
|
||||
'awaiting: user response',
|
||||
'',
|
||||
'## Tests',
|
||||
'',
|
||||
];
|
||||
items.forEach((item, i) => lines.push(...renderRow(i + 1, item)));
|
||||
lines.push(
|
||||
'## Summary',
|
||||
'',
|
||||
`total: ${items.length}`,
|
||||
'passed: 0',
|
||||
'issues: 0',
|
||||
`pending: ${items.length}`,
|
||||
'skipped: 0',
|
||||
'blocked: 0',
|
||||
'',
|
||||
'## Gaps',
|
||||
'',
|
||||
);
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge the report's `human_verification` items into the phase's `*-UAT.md`.
|
||||
*
|
||||
* - no UAT file → create one, every item `[pending]`.
|
||||
* - item matches no row → appended as a new `[pending]` row.
|
||||
* - item matches a row → the row is KEPT exactly as recorded, unless the
|
||||
* row passes AND the item carries a `retest_reason`: then (and only then)
|
||||
* the row is reset to `[pending]`, with `previous_result` and the reason
|
||||
* written into the row. A pass is never reset without a recorded reason.
|
||||
* - row matches no item → kept. The verifier no longer asking for a check
|
||||
* is not a reason to forget that the human ran it.
|
||||
*
|
||||
* When nothing is appended or reset the file is not written at all.
|
||||
*
|
||||
* Returns `null` when the phase has no verification report to seed from.
|
||||
*/
|
||||
function seedUatFromVerification(phaseDir: string, opts: { now?: Date } = {}): SeedUatResult | null {
|
||||
const { uatFile, verificationFile } = resolvePhaseFiles(phaseDir);
|
||||
if (!verificationFile) return null;
|
||||
const report = readNormalized(path.join(phaseDir, verificationFile));
|
||||
if (report === null) return null;
|
||||
const items = readHumanItems(report);
|
||||
const nowIso = (opts.now ?? new Date()).toISOString();
|
||||
|
||||
if (!uatFile) {
|
||||
const result: SeedUatResult = {
|
||||
uat_file: '',
|
||||
created: false,
|
||||
changed: false,
|
||||
kept: [],
|
||||
appended: [],
|
||||
retested: [],
|
||||
pending: 0,
|
||||
all_passing: false,
|
||||
};
|
||||
if (items.length === 0) return result;
|
||||
const fileName = verificationFile.replace(/VERIFICATION\.md$/, 'UAT.md');
|
||||
const phase = extractFrontmatter(report)['phase'];
|
||||
const phaseLabel = typeof phase === 'string' && phase.trim() ? phase.trim() : path.basename(phaseDir);
|
||||
const uatPath = path.join(phaseDir, fileName);
|
||||
fs.writeFileSync(uatPath, newUatDocument(phaseLabel, verificationFile, items, nowIso), 'utf-8');
|
||||
return {
|
||||
...result,
|
||||
uat_file: uatPath,
|
||||
created: true,
|
||||
changed: true,
|
||||
appended: items.map((item, i) => ({ test: i + 1, name: item.test })),
|
||||
pending: items.length,
|
||||
};
|
||||
}
|
||||
|
||||
const uatPath = path.join(phaseDir, uatFile);
|
||||
const content = readNormalized(uatPath);
|
||||
if (content === null) return null;
|
||||
const lines = content.split('\n');
|
||||
const rows = scanUatRows(lines);
|
||||
const byKey = new Map<string, UatRow>();
|
||||
for (const row of rows) if (!byKey.has(matchKey(row.name))) byKey.set(matchKey(row.name), row);
|
||||
|
||||
const appended: SeedUatResult['appended'] = [];
|
||||
const retested: SeedUatResult['retested'] = [];
|
||||
const resetRows = new Set<UatRow>();
|
||||
const toAppend: HumanItem[] = [];
|
||||
for (const item of items) {
|
||||
const key = matchKey(item.test);
|
||||
const row = byKey.get(key);
|
||||
if (!row) {
|
||||
if (!toAppend.some((queued) => matchKey(queued.test) === key)) toAppend.push(item);
|
||||
} else if (row.passing && item.retest_reason.length > 0 && !resetRows.has(row)) {
|
||||
resetRows.add(row);
|
||||
retested.push({ test: row.test, name: row.name, previous_result: row.result, reason: item.retest_reason });
|
||||
}
|
||||
}
|
||||
|
||||
// Edit bottom-up so earlier line indexes stay valid.
|
||||
const nextTest = rows.reduce((max, r) => Math.max(max, r.test), 0) + 1;
|
||||
if (toAppend.length > 0) {
|
||||
const body = bodyStart(lines);
|
||||
const isSection = (re: RegExp) => (l: string, i: number) => i >= body && re.test(l);
|
||||
let insertAt = lines.findIndex(isSection(/^##\s+Summary\s*$/));
|
||||
if (insertAt === -1) insertAt = lines.findIndex(isSection(/^##\s+Gaps\s*$/));
|
||||
const lastRow = rows[rows.length - 1];
|
||||
if (insertAt === -1 || (lastRow && insertAt < lastRow.headingLine)) insertAt = lines.length;
|
||||
const block: string[] = [];
|
||||
if (insertAt > 0 && lines[insertAt - 1].trim() !== '') block.push('');
|
||||
toAppend.forEach((item, i) => {
|
||||
block.push(...renderRow(nextTest + i, item));
|
||||
appended.push({ test: nextTest + i, name: item.test });
|
||||
});
|
||||
lines.splice(insertAt, 0, ...block);
|
||||
}
|
||||
for (const row of [...resetRows].sort((a, b) => b.resultLine - a.resultLine)) {
|
||||
const reason = retested.find((r) => r.test === row.test && r.name === row.name)?.reason ?? '';
|
||||
lines.splice(
|
||||
row.resultLine,
|
||||
1,
|
||||
'result: [pending]',
|
||||
`previous_result: ${row.result}`,
|
||||
`retest_reason: ${quote(reason)}`,
|
||||
);
|
||||
}
|
||||
|
||||
const changed = appended.length > 0 || retested.length > 0;
|
||||
let finalRows = rows;
|
||||
if (changed) {
|
||||
finalRows = scanUatRows(lines);
|
||||
refreshSummary(lines, finalRows);
|
||||
reopenFrontmatter(lines, nowIso);
|
||||
fs.writeFileSync(uatPath, lines.join('\n'), 'utf-8');
|
||||
}
|
||||
return {
|
||||
uat_file: uatPath,
|
||||
created: false,
|
||||
changed,
|
||||
kept: rows.filter((r) => !resetRows.has(r)).map((r) => ({ test: r.test, name: r.name, result: r.result })),
|
||||
appended,
|
||||
retested,
|
||||
pending: finalRows.filter((r) => r.result === 'pending').length,
|
||||
all_passing: finalRows.length > 0 && finalRows.every((r) => r.passing),
|
||||
};
|
||||
}
|
||||
|
||||
// ─── canonicalize-uat ─────────────────────────────────────────────────────────
|
||||
|
||||
/** Frozen reason enum for `canonicalizeHumanNeeded` — callers branch on these, never on prose. */
|
||||
const CANONICALIZE_REASON = Object.freeze({
|
||||
UAT_PASSED: 'uat_passed',
|
||||
NOT_HUMAN_NEEDED: 'not_human_needed',
|
||||
UAT_NOT_PASSED: 'uat_not_passed',
|
||||
REPORT_UNWRITABLE: 'report_unwritable',
|
||||
} as const);
|
||||
type CanonicalizeReason = (typeof CANONICALIZE_REASON)[keyof typeof CANONICALIZE_REASON];
|
||||
|
||||
interface CanonicalizeResult {
|
||||
canonicalized: boolean;
|
||||
/** The verification status AFTER this call. */
|
||||
status: string;
|
||||
reason: CanonicalizeReason;
|
||||
/** The UAT rows that refused the flip (`uat_not_passed` only). */
|
||||
blockers: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Promote a `human_needed` report to `passed` when — and only when — the
|
||||
* human UAT passed (#4663).
|
||||
*
|
||||
* The status is read through `readVerificationStatus`, so staleness and
|
||||
* `gaps_found` outrank the raw frontmatter exactly as they do everywhere
|
||||
* else: a `stale` report is refused (`not_human_needed`) and must be
|
||||
* re-verified first. The UAT verdict is the SAME row predicate the
|
||||
* phase-close uses, in its `uat-only` form: at least one check passed and no
|
||||
* row is pending, blocked, failed, or skipped without a deferral reason. Zero
|
||||
* issues is not pass evidence on its own.
|
||||
*/
|
||||
function canonicalizeHumanNeeded(phaseDir: string): CanonicalizeResult {
|
||||
const status = (readVerificationStatus(phaseDir) as { status: string }).status;
|
||||
if (status !== 'human_needed') {
|
||||
return { canonicalized: false, status, reason: CANONICALIZE_REASON.NOT_HUMAN_NEEDED, blockers: [] };
|
||||
}
|
||||
const uat = evaluateUatPassed(phaseDir, { policy: { uatOnly: true } });
|
||||
if (!uat.passed) {
|
||||
return {
|
||||
canonicalized: false,
|
||||
status,
|
||||
reason: CANONICALIZE_REASON.UAT_NOT_PASSED,
|
||||
blockers: uat.blockers,
|
||||
};
|
||||
}
|
||||
const { verificationFile } = resolvePhaseFiles(phaseDir);
|
||||
try {
|
||||
const reportPath = path.join(phaseDir, verificationFile as string);
|
||||
const content = fs.readFileSync(reportPath, 'utf-8');
|
||||
const fm = extractFrontmatter(content, reportPath);
|
||||
fm['status'] = 'passed';
|
||||
fs.writeFileSync(reportPath, spliceFrontmatter(content, fm), 'utf-8');
|
||||
} catch {
|
||||
return { canonicalized: false, status, reason: CANONICALIZE_REASON.REPORT_UNWRITABLE, blockers: [] };
|
||||
}
|
||||
return { canonicalized: true, status: 'passed', reason: CANONICALIZE_REASON.UAT_PASSED, blockers: [] };
|
||||
}
|
||||
|
||||
// ─── CLI handlers ─────────────────────────────────────────────────────────────
|
||||
|
||||
function resolvePhaseDirArg(cwd: string, phaseDirArg: string | undefined, verb: string): string | null {
|
||||
if (!phaseDirArg) {
|
||||
error(`phase directory required for ${verb}`);
|
||||
return null;
|
||||
}
|
||||
const phaseDir = path.resolve(cwd, phaseDirArg);
|
||||
let isDir = false;
|
||||
try {
|
||||
isDir = fs.statSync(phaseDir).isDirectory();
|
||||
} catch {
|
||||
// not found → not a directory
|
||||
}
|
||||
if (!isDir) {
|
||||
error(`phase directory not found: ${phaseDirArg}`);
|
||||
return null;
|
||||
}
|
||||
return phaseDir;
|
||||
}
|
||||
|
||||
/**
|
||||
* `verification.uat-evidence <phaseDir> [files… | --files a,b]` — read-only.
|
||||
* With no files, compares the current report's own `covered_files`.
|
||||
*/
|
||||
function cmdVerificationUatEvidence(
|
||||
cwd: string,
|
||||
phaseDirArg: string | undefined,
|
||||
fileArgs: readonly string[],
|
||||
raw: boolean,
|
||||
): void {
|
||||
const phaseDir = resolvePhaseDirArg(cwd, phaseDirArg, 'verification.uat-evidence');
|
||||
if (phaseDir === null) return;
|
||||
const parsed = parseFingerprintFileArgs(fileArgs);
|
||||
if ('error' in parsed) {
|
||||
error(parsed.error.replace(/verification\.fingerprint/g, 'verification.uat-evidence'));
|
||||
return;
|
||||
}
|
||||
output(readUatEvidence(phaseDir, parsed.files.length > 0 ? parsed.files : undefined), raw);
|
||||
}
|
||||
|
||||
/** `verification.seed-uat <phaseDir>` — merge the report's human items into `*-UAT.md`. */
|
||||
function cmdVerificationSeedUat(cwd: string, phaseDirArg: string | undefined, raw: boolean): void {
|
||||
const phaseDir = resolvePhaseDirArg(cwd, phaseDirArg, 'verification.seed-uat');
|
||||
if (phaseDir === null) return;
|
||||
const result = seedUatFromVerification(phaseDir);
|
||||
if (result === null) {
|
||||
error('no readable *-VERIFICATION.md in the phase directory — verification.seed-uat merges that report\'s human_verification items');
|
||||
return;
|
||||
}
|
||||
output(result, raw, result.uat_file);
|
||||
}
|
||||
|
||||
/** `verification.canonicalize-uat <phaseDir>` — record a passed UAT against a `human_needed` report. */
|
||||
function cmdVerificationCanonicalizeUat(cwd: string, phaseDirArg: string | undefined, raw: boolean): void {
|
||||
const phaseDir = resolvePhaseDirArg(cwd, phaseDirArg, 'verification.canonicalize-uat');
|
||||
if (phaseDir === null) return;
|
||||
const result = canonicalizeHumanNeeded(phaseDir);
|
||||
output(result, raw, result.status);
|
||||
}
|
||||
|
||||
export = {
|
||||
CANONICALIZE_REASON,
|
||||
canonicalizeHumanNeeded,
|
||||
cmdVerificationCanonicalizeUat,
|
||||
scanUatRows,
|
||||
readHumanItems,
|
||||
readUatEvidence,
|
||||
seedUatFromVerification,
|
||||
cmdVerificationUatEvidence,
|
||||
cmdVerificationSeedUat,
|
||||
};
|
||||
@@ -21,8 +21,16 @@ interface VerificationModule {
|
||||
cmdVerificationFingerprint(cwd: string, phaseDirArg: string | undefined, files: string[], raw: boolean): void;
|
||||
}
|
||||
|
||||
/** UAT-evidence verbs live in their own module (it imports `verification`, not the reverse). */
|
||||
interface UatEvidenceModule {
|
||||
cmdVerificationUatEvidence(cwd: string, phaseDirArg: string | undefined, files: string[], raw: boolean): void;
|
||||
cmdVerificationSeedUat(cwd: string, phaseDirArg: string | undefined, raw: boolean): void;
|
||||
cmdVerificationCanonicalizeUat(cwd: string, phaseDirArg: string | undefined, raw: boolean): void;
|
||||
}
|
||||
|
||||
interface RouteVerificationCommandOptions {
|
||||
verification: VerificationModule;
|
||||
uatEvidence: UatEvidenceModule;
|
||||
args: string[];
|
||||
cwd: string;
|
||||
raw: boolean;
|
||||
@@ -31,10 +39,11 @@ interface RouteVerificationCommandOptions {
|
||||
|
||||
// ─── Implementation ───────────────────────────────────────────────────────────
|
||||
|
||||
const VERIFICATION_SUBCOMMANDS = ['status', 'resolve-file', 'fingerprint'];
|
||||
const VERIFICATION_SUBCOMMANDS = ['status', 'resolve-file', 'fingerprint', 'uat-evidence', 'seed-uat', 'canonicalize-uat'];
|
||||
|
||||
function routeVerificationCommand({
|
||||
verification,
|
||||
uatEvidence,
|
||||
args,
|
||||
cwd,
|
||||
raw,
|
||||
@@ -51,6 +60,9 @@ function routeVerificationCommand({
|
||||
status: () => verification.cmdVerificationStatus(cwd, args[2], raw),
|
||||
'resolve-file': () => verification.cmdVerificationResolveFile(cwd, args[2], raw),
|
||||
fingerprint: () => verification.cmdVerificationFingerprint(cwd, args[2], args.slice(3), raw),
|
||||
'uat-evidence': () => uatEvidence.cmdVerificationUatEvidence(cwd, args[2], args.slice(3), raw),
|
||||
'seed-uat': () => uatEvidence.cmdVerificationSeedUat(cwd, args[2], raw),
|
||||
'canonicalize-uat': () => uatEvidence.cmdVerificationCanonicalizeUat(cwd, args[2], raw),
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
@@ -118,7 +118,11 @@ const VERIFICATION_ROUTING_TABLE: Record<string, VerificationRoute> = {
|
||||
// here was an advice loop. execute-phase resumes at the verification
|
||||
// gates and re-runs the verifier (its resume tree routes a stale report
|
||||
// to re-verification), which regenerates VERIFICATION.md and its digest.
|
||||
next_action: 'Verification is stale — covered source files changed after the verifier last ran. Re-run execute-phase for this phase: it resumes at the verification gates and re-runs the verifier, regenerating VERIFICATION.md and its digest. verify-work alone cannot refresh a stale report.',
|
||||
// the action is named for what it is — a verifier re-run, not an
|
||||
// execution. No plan re-runs, and recorded UAT results are kept: the
|
||||
// verifier reads them (`verification.uat-evidence`) and the UAT writer
|
||||
// merges instead of overwriting (`verification.seed-uat`).
|
||||
next_action: 'Verification is stale — the report no longer matches the phase. Re-run the verifier: execute-phase runs only plans that have no SUMMARY (none, when every plan is summarized), then resumes at the verification gates and regenerates VERIFICATION.md and its digest. Recorded UAT results are kept, not re-requested.',
|
||||
next_command: 'execute-phase',
|
||||
},
|
||||
// INTERNAL SENTINEL: constructed when no *-VERIFICATION.md file exists or when
|
||||
@@ -865,8 +869,42 @@ interface VerificationStatusResult {
|
||||
* `gaps_found` short-circuit above it, or no verification file at all).
|
||||
*/
|
||||
staleCheckIndeterminate?: boolean;
|
||||
/**
|
||||
* WHY a `stale` result is stale — present only when `status` is
|
||||
* `stale`. Every reason still means "re-run the verifier"; the split lets
|
||||
* a caller say what happened (a gap-closure plan the report never saw is
|
||||
* not the same event as a covered source file drifting under a finished
|
||||
* phase) and lets progress batch the drift case across phases.
|
||||
*/
|
||||
stale_reason?: StaleReason;
|
||||
/**
|
||||
* the report's own frontmatter `status` underneath a `stale` result —
|
||||
* present only when `status` is `stale`. `human_needed` here means a UAT
|
||||
* is still owed (or already recorded) once the verifier has re-run.
|
||||
*/
|
||||
raw_status?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* the closed set of reasons a report reads `stale`.
|
||||
* - `fingerprint_malformed` — the `covered_files`/`covered_digest` pair is
|
||||
* incomplete, wrong-shaped, or names an unknown version (fail closed, #4155).
|
||||
* - `covered_changed` — a declared covered file's bytes changed, or
|
||||
* the file is gone.
|
||||
* - `uncovered_artifacts` — every declared file still matches, but the
|
||||
* phase directory holds a PLAN/SUMMARY the report never declared (the
|
||||
* shape every gap-closure cycle produces).
|
||||
* - `summary_newer` — legacy (pre-#4155) report with a SUMMARY
|
||||
* changed after it.
|
||||
*/
|
||||
type StaleReason = 'fingerprint_malformed' | 'covered_changed' | 'uncovered_artifacts' | 'summary_newer';
|
||||
const STALE_REASONS: ReadonlyArray<StaleReason> = [
|
||||
'fingerprint_malformed',
|
||||
'covered_changed',
|
||||
'uncovered_artifacts',
|
||||
'summary_newer',
|
||||
];
|
||||
|
||||
function findStaleVerificationSummary(
|
||||
phaseDir: string,
|
||||
fsImpl: FsLike = defaultFsImpl,
|
||||
@@ -1096,6 +1134,7 @@ function readVerificationStatus(
|
||||
|
||||
let staleCheckIndeterminate = false;
|
||||
let isStale: boolean;
|
||||
let staleReason: StaleReason = 'summary_newer';
|
||||
if (declaresFingerprint) {
|
||||
// Stated directly rather than relying on `null !== coveredDigestVal`
|
||||
// being true whenever the pair is malformed: `!hasWellFormedFingerprint`
|
||||
@@ -1115,11 +1154,22 @@ function readVerificationStatus(
|
||||
hasWellFormedFingerprint && typeof coveredDigestVal === 'string'
|
||||
? parseFingerprintVersion(coveredDigestVal)
|
||||
: null;
|
||||
isStale =
|
||||
!hasWellFormedFingerprint ||
|
||||
storedVersion === null ||
|
||||
computeCoveredDigest(findProjectRoot(phaseDir), coveredFilesVal, storedVersion, { phaseDir }) !== coveredDigestVal ||
|
||||
!allCurrentArtifactsCovered(phaseDir, coveredFilesVal);
|
||||
// same three checks, same order, same short-circuit — only named, so
|
||||
// the result can say which one fired.
|
||||
if (!hasWellFormedFingerprint || storedVersion === null) {
|
||||
isStale = true;
|
||||
staleReason = 'fingerprint_malformed';
|
||||
} else if (
|
||||
computeCoveredDigest(findProjectRoot(phaseDir), coveredFilesVal, storedVersion, { phaseDir }) !== coveredDigestVal
|
||||
) {
|
||||
isStale = true;
|
||||
staleReason = 'covered_changed';
|
||||
} else if (!allCurrentArtifactsCovered(phaseDir, coveredFilesVal)) {
|
||||
isStale = true;
|
||||
staleReason = 'uncovered_artifacts';
|
||||
} else {
|
||||
isStale = false;
|
||||
}
|
||||
} else {
|
||||
const staleCheck = findStaleVerificationSummary(
|
||||
phaseDir,
|
||||
@@ -1144,6 +1194,8 @@ function readVerificationStatus(
|
||||
// the verifier, regenerating VERIFICATION.md and its digest — the same
|
||||
// routing the `missing` sentinel has used since #2868.
|
||||
next_command: projectNextCommand('execute-phase', runtime, phaseArg),
|
||||
stale_reason: staleReason,
|
||||
raw_status: rawStatus,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1470,6 +1522,8 @@ function cmdVerificationFingerprint(
|
||||
export = {
|
||||
VERIFIER_STATUSES,
|
||||
VERIFICATION_ROUTING_TABLE,
|
||||
STALE_REASONS,
|
||||
canonicalizeCoveredFiles,
|
||||
defaultPhaseCleanCommitTimesMs,
|
||||
resolveVerificationFile,
|
||||
resolveUatFile,
|
||||
|
||||
3
tests/fixtures/install-tree/antigravity.json
vendored
3
tests/fixtures/install-tree/antigravity.json
vendored
@@ -366,6 +366,7 @@
|
||||
"msd-core/references/sketch-variant-patterns.md",
|
||||
"msd-core/references/specless-probe-fallback.md",
|
||||
"msd-core/references/spidr-splitting.md",
|
||||
"msd-core/references/stale-reverification.md",
|
||||
"msd-core/references/tdd.md",
|
||||
"msd-core/references/thinking-models-debug.md",
|
||||
"msd-core/references/thinking-models-execution.md",
|
||||
@@ -383,6 +384,7 @@
|
||||
"msd-core/references/verification-patterns.md",
|
||||
"msd-core/references/verifier-evidence-gate.md",
|
||||
"msd-core/references/verifier-phase-gates.md",
|
||||
"msd-core/references/verifier-uat-evidence.md",
|
||||
"msd-core/references/verifier-wiring-patterns.md",
|
||||
"msd-core/references/verify-command-path-resolvability.md",
|
||||
"msd-core/references/verify-mvp-mode.md",
|
||||
@@ -485,6 +487,7 @@
|
||||
"msd-core/workflows/execute-phase/steps/executor-isolation-dispatch.md",
|
||||
"msd-core/workflows/execute-phase/steps/executor-progress-policy.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-artifacts.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-uat-record.md",
|
||||
"msd-core/workflows/execute-phase/steps/partial-wave.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-executor-routing.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-worktree-gate.md",
|
||||
|
||||
@@ -272,6 +272,7 @@
|
||||
"msd-core/references/sketch-variant-patterns.md",
|
||||
"msd-core/references/specless-probe-fallback.md",
|
||||
"msd-core/references/spidr-splitting.md",
|
||||
"msd-core/references/stale-reverification.md",
|
||||
"msd-core/references/tdd.md",
|
||||
"msd-core/references/thinking-models-debug.md",
|
||||
"msd-core/references/thinking-models-execution.md",
|
||||
@@ -289,6 +290,7 @@
|
||||
"msd-core/references/verification-patterns.md",
|
||||
"msd-core/references/verifier-evidence-gate.md",
|
||||
"msd-core/references/verifier-phase-gates.md",
|
||||
"msd-core/references/verifier-uat-evidence.md",
|
||||
"msd-core/references/verifier-wiring-patterns.md",
|
||||
"msd-core/references/verify-command-path-resolvability.md",
|
||||
"msd-core/references/verify-mvp-mode.md",
|
||||
@@ -391,6 +393,7 @@
|
||||
"msd-core/workflows/execute-phase/steps/executor-isolation-dispatch.md",
|
||||
"msd-core/workflows/execute-phase/steps/executor-progress-policy.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-artifacts.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-uat-record.md",
|
||||
"msd-core/workflows/execute-phase/steps/partial-wave.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-executor-routing.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-worktree-gate.md",
|
||||
|
||||
3
tests/fixtures/install-tree/claude.json
vendored
3
tests/fixtures/install-tree/claude.json
vendored
@@ -336,6 +336,7 @@
|
||||
"msd-core/references/sketch-variant-patterns.md",
|
||||
"msd-core/references/specless-probe-fallback.md",
|
||||
"msd-core/references/spidr-splitting.md",
|
||||
"msd-core/references/stale-reverification.md",
|
||||
"msd-core/references/tdd.md",
|
||||
"msd-core/references/thinking-models-debug.md",
|
||||
"msd-core/references/thinking-models-execution.md",
|
||||
@@ -353,6 +354,7 @@
|
||||
"msd-core/references/verification-patterns.md",
|
||||
"msd-core/references/verifier-evidence-gate.md",
|
||||
"msd-core/references/verifier-phase-gates.md",
|
||||
"msd-core/references/verifier-uat-evidence.md",
|
||||
"msd-core/references/verifier-wiring-patterns.md",
|
||||
"msd-core/references/verify-command-path-resolvability.md",
|
||||
"msd-core/references/verify-mvp-mode.md",
|
||||
@@ -455,6 +457,7 @@
|
||||
"msd-core/workflows/execute-phase/steps/executor-isolation-dispatch.md",
|
||||
"msd-core/workflows/execute-phase/steps/executor-progress-policy.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-artifacts.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-uat-record.md",
|
||||
"msd-core/workflows/execute-phase/steps/partial-wave.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-executor-routing.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-worktree-gate.md",
|
||||
|
||||
3
tests/fixtures/install-tree/codex.json
vendored
3
tests/fixtures/install-tree/codex.json
vendored
@@ -365,6 +365,7 @@
|
||||
"msd-core/references/sketch-variant-patterns.md",
|
||||
"msd-core/references/specless-probe-fallback.md",
|
||||
"msd-core/references/spidr-splitting.md",
|
||||
"msd-core/references/stale-reverification.md",
|
||||
"msd-core/references/tdd.md",
|
||||
"msd-core/references/thinking-models-debug.md",
|
||||
"msd-core/references/thinking-models-execution.md",
|
||||
@@ -382,6 +383,7 @@
|
||||
"msd-core/references/verification-patterns.md",
|
||||
"msd-core/references/verifier-evidence-gate.md",
|
||||
"msd-core/references/verifier-phase-gates.md",
|
||||
"msd-core/references/verifier-uat-evidence.md",
|
||||
"msd-core/references/verifier-wiring-patterns.md",
|
||||
"msd-core/references/verify-command-path-resolvability.md",
|
||||
"msd-core/references/verify-mvp-mode.md",
|
||||
@@ -484,6 +486,7 @@
|
||||
"msd-core/workflows/execute-phase/steps/executor-isolation-dispatch.md",
|
||||
"msd-core/workflows/execute-phase/steps/executor-progress-policy.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-artifacts.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-uat-record.md",
|
||||
"msd-core/workflows/execute-phase/steps/partial-wave.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-executor-routing.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-worktree-gate.md",
|
||||
|
||||
3
tests/fixtures/install-tree/cursor.json
vendored
3
tests/fixtures/install-tree/cursor.json
vendored
@@ -339,6 +339,7 @@
|
||||
"msd-core/references/sketch-variant-patterns.md",
|
||||
"msd-core/references/specless-probe-fallback.md",
|
||||
"msd-core/references/spidr-splitting.md",
|
||||
"msd-core/references/stale-reverification.md",
|
||||
"msd-core/references/tdd.md",
|
||||
"msd-core/references/thinking-models-debug.md",
|
||||
"msd-core/references/thinking-models-execution.md",
|
||||
@@ -356,6 +357,7 @@
|
||||
"msd-core/references/verification-patterns.md",
|
||||
"msd-core/references/verifier-evidence-gate.md",
|
||||
"msd-core/references/verifier-phase-gates.md",
|
||||
"msd-core/references/verifier-uat-evidence.md",
|
||||
"msd-core/references/verifier-wiring-patterns.md",
|
||||
"msd-core/references/verify-command-path-resolvability.md",
|
||||
"msd-core/references/verify-mvp-mode.md",
|
||||
@@ -458,6 +460,7 @@
|
||||
"msd-core/workflows/execute-phase/steps/executor-isolation-dispatch.md",
|
||||
"msd-core/workflows/execute-phase/steps/executor-progress-policy.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-artifacts.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-uat-record.md",
|
||||
"msd-core/workflows/execute-phase/steps/partial-wave.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-executor-routing.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-worktree-gate.md",
|
||||
|
||||
3
tests/fixtures/install-tree/opencode.json
vendored
3
tests/fixtures/install-tree/opencode.json
vendored
@@ -437,6 +437,7 @@
|
||||
"msd-core/references/sketch-variant-patterns.md",
|
||||
"msd-core/references/specless-probe-fallback.md",
|
||||
"msd-core/references/spidr-splitting.md",
|
||||
"msd-core/references/stale-reverification.md",
|
||||
"msd-core/references/tdd.md",
|
||||
"msd-core/references/thinking-models-debug.md",
|
||||
"msd-core/references/thinking-models-execution.md",
|
||||
@@ -454,6 +455,7 @@
|
||||
"msd-core/references/verification-patterns.md",
|
||||
"msd-core/references/verifier-evidence-gate.md",
|
||||
"msd-core/references/verifier-phase-gates.md",
|
||||
"msd-core/references/verifier-uat-evidence.md",
|
||||
"msd-core/references/verifier-wiring-patterns.md",
|
||||
"msd-core/references/verify-command-path-resolvability.md",
|
||||
"msd-core/references/verify-mvp-mode.md",
|
||||
@@ -556,6 +558,7 @@
|
||||
"msd-core/workflows/execute-phase/steps/executor-isolation-dispatch.md",
|
||||
"msd-core/workflows/execute-phase/steps/executor-progress-policy.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-artifacts.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-uat-record.md",
|
||||
"msd-core/workflows/execute-phase/steps/partial-wave.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-executor-routing.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-worktree-gate.md",
|
||||
|
||||
3
tests/fixtures/install-tree/zcode.json
vendored
3
tests/fixtures/install-tree/zcode.json
vendored
@@ -397,6 +397,7 @@
|
||||
"msd-core/references/sketch-variant-patterns.md",
|
||||
"msd-core/references/specless-probe-fallback.md",
|
||||
"msd-core/references/spidr-splitting.md",
|
||||
"msd-core/references/stale-reverification.md",
|
||||
"msd-core/references/tdd.md",
|
||||
"msd-core/references/thinking-models-debug.md",
|
||||
"msd-core/references/thinking-models-execution.md",
|
||||
@@ -414,6 +415,7 @@
|
||||
"msd-core/references/verification-patterns.md",
|
||||
"msd-core/references/verifier-evidence-gate.md",
|
||||
"msd-core/references/verifier-phase-gates.md",
|
||||
"msd-core/references/verifier-uat-evidence.md",
|
||||
"msd-core/references/verifier-wiring-patterns.md",
|
||||
"msd-core/references/verify-command-path-resolvability.md",
|
||||
"msd-core/references/verify-mvp-mode.md",
|
||||
@@ -516,6 +518,7 @@
|
||||
"msd-core/workflows/execute-phase/steps/executor-isolation-dispatch.md",
|
||||
"msd-core/workflows/execute-phase/steps/executor-progress-policy.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-artifacts.md",
|
||||
"msd-core/workflows/execute-phase/steps/gap-closure-uat-record.md",
|
||||
"msd-core/workflows/execute-phase/steps/partial-wave.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-executor-routing.md",
|
||||
"msd-core/workflows/execute-phase/steps/per-plan-worktree-gate.md",
|
||||
|
||||
500
tests/uat-evidence.test.cjs
Normal file
500
tests/uat-evidence.test.cjs
Normal file
@@ -0,0 +1,500 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Tests for the UAT-evidence seam (src/uat-evidence.cts) and the stale-reason
|
||||
* split on readVerificationStatus.
|
||||
*
|
||||
* The bug: a phase whose human UAT passed could not reach `passed`. A
|
||||
* gap-closure plan (or a later phase editing a covered file) made the report
|
||||
* `stale`; verify-work only recorded a passed UAT against `human_needed`; the
|
||||
* re-run verifier could not see the UAT file and re-emitted `human_needed`;
|
||||
* and execute-phase's human_needed branch rewrote `*-UAT.md` with every row
|
||||
* `[pending]`. Each acceptance scenario below simulates the verifier re-run by
|
||||
* writing the report a verifier would write, then drives the real verbs.
|
||||
*
|
||||
* All assertions are on structured values (typed results, `--json` CLI output,
|
||||
* parsed UAT rows) — never on rendered prose.
|
||||
*/
|
||||
|
||||
const { describe, test, beforeEach, afterEach } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const { createTempProject, cleanup, runMsdTools } = require('./helpers.cjs');
|
||||
const {
|
||||
readVerificationStatus,
|
||||
computeCoveredDigest,
|
||||
STALE_REASONS,
|
||||
} = require('../msd-core/bin/lib/verification.cjs');
|
||||
const {
|
||||
CANONICALIZE_REASON,
|
||||
canonicalizeHumanNeeded,
|
||||
readUatEvidence,
|
||||
seedUatFromVerification,
|
||||
scanUatRows,
|
||||
} = require('../msd-core/bin/lib/uat-evidence.cjs');
|
||||
|
||||
const PHASE = '03-demo';
|
||||
const ITEMS = [
|
||||
{ test: 'Login form renders', expected: 'Two fields and a submit button' },
|
||||
{ test: 'Logout clears the session', expected: 'Redirected to /login' },
|
||||
];
|
||||
|
||||
let root;
|
||||
let phaseDir;
|
||||
|
||||
function rel(...segments) {
|
||||
return ['.planning', 'phases', PHASE, ...segments].join('/');
|
||||
}
|
||||
|
||||
function writePlan(n) {
|
||||
fs.writeFileSync(path.join(phaseDir, `03-${n}-PLAN.md`), `plan ${n}\n`);
|
||||
fs.writeFileSync(path.join(phaseDir, `03-${n}-SUMMARY.md`), `summary ${n}\n`);
|
||||
}
|
||||
|
||||
function artifacts(...plans) {
|
||||
return plans.flatMap((n) => [rel(`03-${n}-PLAN.md`), rel(`03-${n}-SUMMARY.md`)]);
|
||||
}
|
||||
|
||||
/** Write the report a verifier run would write over `covered`, digest computed for real. */
|
||||
function writeReport({ status, covered, items = [] }) {
|
||||
const digest = computeCoveredDigest(root, covered, undefined, { phaseDir });
|
||||
assert.ok(digest, 'fixture covered files must fingerprint');
|
||||
const lines = [
|
||||
'---',
|
||||
`phase: ${PHASE}`,
|
||||
`status: ${status}`,
|
||||
`covered_files: [${covered.join(', ')}]`,
|
||||
`covered_digest: "${digest}"`,
|
||||
];
|
||||
if (items.length > 0) {
|
||||
lines.push('human_verification:');
|
||||
for (const item of items) {
|
||||
lines.push(` - test: "${item.test}"`, ` expected: "${item.expected}"`, ' why_human: "visual"');
|
||||
if (item.retest_reason) lines.push(` retest_reason: "${item.retest_reason}"`);
|
||||
}
|
||||
}
|
||||
lines.push('---', '', '# Verification', '');
|
||||
fs.writeFileSync(path.join(phaseDir, '03-VERIFICATION.md'), lines.join('\n'));
|
||||
}
|
||||
|
||||
function writePassedUat() {
|
||||
fs.writeFileSync(
|
||||
path.join(phaseDir, '03-UAT.md'),
|
||||
[
|
||||
'---',
|
||||
'status: complete',
|
||||
`phase: ${PHASE}`,
|
||||
'source: [03-VERIFICATION.md]',
|
||||
'started: 2026-10-01T10:00:00.000Z',
|
||||
'updated: 2026-10-01T11:00:00.000Z',
|
||||
'---',
|
||||
'',
|
||||
'## Current Test',
|
||||
'',
|
||||
'[testing complete]',
|
||||
'',
|
||||
'## Tests',
|
||||
'',
|
||||
'### 1. Login form renders',
|
||||
'expected: Two fields and a submit button',
|
||||
'result: pass',
|
||||
'reported: "both fields there, button works"',
|
||||
'',
|
||||
'### 2. Logout clears the session',
|
||||
'expected: Redirected to /login',
|
||||
'result: pass',
|
||||
'reported: "landed on /login"',
|
||||
'',
|
||||
'## Summary',
|
||||
'',
|
||||
'total: 2',
|
||||
'passed: 2',
|
||||
'issues: 0',
|
||||
'pending: 0',
|
||||
'skipped: 0',
|
||||
'blocked: 0',
|
||||
'',
|
||||
'## Gaps',
|
||||
'',
|
||||
'- truth: "Logout clears the session"',
|
||||
' status: resolved',
|
||||
' test: 2',
|
||||
'',
|
||||
].join('\n'),
|
||||
);
|
||||
}
|
||||
|
||||
function uatBytes() {
|
||||
return fs.readFileSync(path.join(phaseDir, '03-UAT.md'), 'utf-8');
|
||||
}
|
||||
|
||||
function uatRows() {
|
||||
return scanUatRows(uatBytes().split('\n'));
|
||||
}
|
||||
|
||||
/** Pin a file's mtime (the fixture is not a git repo, so mtime is the clock). */
|
||||
function setMtime(file, isoTime) {
|
||||
const t = new Date(isoTime);
|
||||
fs.utimesSync(path.join(root, file), t, t);
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
root = createTempProject('msd-uat-evidence-');
|
||||
phaseDir = path.join(root, '.planning', 'phases', PHASE);
|
||||
fs.mkdirSync(phaseDir, { recursive: true });
|
||||
fs.mkdirSync(path.join(root, 'src'), { recursive: true });
|
||||
fs.writeFileSync(path.join(root, 'src', 'auth.js'), 'export const login = 1;\n');
|
||||
fs.writeFileSync(path.join(root, 'src', 'billing.js'), 'export const bill = 1;\n');
|
||||
writePlan('01');
|
||||
writePlan('02');
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
cleanup(root);
|
||||
});
|
||||
|
||||
describe('readVerificationStatus — stale_reason', () => {
|
||||
test('a fresh report carries no stale_reason', () => {
|
||||
writeReport({ status: 'passed', covered: [...artifacts('01', '02'), 'src/auth.js'] });
|
||||
const result = readVerificationStatus(phaseDir);
|
||||
assert.equal(result.status, 'passed');
|
||||
assert.equal('stale_reason' in result, false);
|
||||
assert.equal('raw_status' in result, false);
|
||||
});
|
||||
|
||||
test('a PLAN/SUMMARY the report never declared reads uncovered_artifacts', () => {
|
||||
writeReport({ status: 'human_needed', covered: [...artifacts('01', '02'), 'src/auth.js'], items: ITEMS });
|
||||
writePlan('03');
|
||||
const result = readVerificationStatus(phaseDir);
|
||||
assert.equal(result.status, 'stale');
|
||||
assert.equal(result.stale_reason, 'uncovered_artifacts');
|
||||
assert.equal(result.raw_status, 'human_needed');
|
||||
});
|
||||
|
||||
test('an edited covered file reads covered_changed', () => {
|
||||
writeReport({ status: 'passed', covered: [...artifacts('01', '02'), 'src/auth.js'] });
|
||||
fs.writeFileSync(path.join(root, 'src', 'auth.js'), 'export const login = 2;\n');
|
||||
const result = readVerificationStatus(phaseDir);
|
||||
assert.equal(result.status, 'stale');
|
||||
assert.equal(result.stale_reason, 'covered_changed');
|
||||
assert.equal(result.raw_status, 'passed');
|
||||
});
|
||||
|
||||
test('a malformed fingerprint still fails closed, as fingerprint_malformed', () => {
|
||||
fs.writeFileSync(
|
||||
path.join(phaseDir, '03-VERIFICATION.md'),
|
||||
['---', 'status: passed', 'covered_files: [src/auth.js]', '---', ''].join('\n'),
|
||||
);
|
||||
const result = readVerificationStatus(phaseDir);
|
||||
assert.equal(result.status, 'stale');
|
||||
assert.equal(result.stale_reason, 'fingerprint_malformed');
|
||||
});
|
||||
|
||||
test('every emitted reason belongs to the frozen STALE_REASONS set', () => {
|
||||
assert.deepEqual([...STALE_REASONS].sort(), [
|
||||
'covered_changed',
|
||||
'fingerprint_malformed',
|
||||
'summary_newer',
|
||||
'uncovered_artifacts',
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('verification.seed-uat — merge, never overwrite', () => {
|
||||
test('creates the UAT file with every item pending when none exists', () => {
|
||||
writeReport({ status: 'human_needed', covered: artifacts('01', '02'), items: ITEMS });
|
||||
const seed = seedUatFromVerification(phaseDir);
|
||||
assert.equal(seed.created, true);
|
||||
assert.equal(seed.pending, 2);
|
||||
assert.deepEqual(seed.appended, [
|
||||
{ test: 1, name: 'Login form renders' },
|
||||
{ test: 2, name: 'Logout clears the session' },
|
||||
]);
|
||||
assert.deepEqual(uatRows().map((r) => r.result), ['pending', 'pending']);
|
||||
});
|
||||
|
||||
test('no-clobber: passed rows and their reported evidence are left byte-identical', () => {
|
||||
writeReport({ status: 'human_needed', covered: artifacts('01', '02'), items: ITEMS });
|
||||
writePassedUat();
|
||||
const before = uatBytes();
|
||||
const seed = seedUatFromVerification(phaseDir);
|
||||
assert.equal(seed.changed, false);
|
||||
assert.equal(seed.created, false);
|
||||
assert.equal(seed.pending, 0);
|
||||
assert.equal(seed.all_passing, true);
|
||||
assert.equal(uatBytes(), before);
|
||||
});
|
||||
|
||||
test('an unseen item is appended as pending; existing results are kept', () => {
|
||||
writePassedUat();
|
||||
writeReport({
|
||||
status: 'human_needed',
|
||||
covered: artifacts('01', '02'),
|
||||
items: [...ITEMS, { test: 'Password reset email arrives', expected: 'Email within a minute' }],
|
||||
});
|
||||
const seed = seedUatFromVerification(phaseDir, { now: new Date('2026-10-10T00:00:00.000Z') });
|
||||
assert.deepEqual(seed.appended, [{ test: 3, name: 'Password reset email arrives' }]);
|
||||
assert.deepEqual(seed.kept.map((r) => r.result), ['pass', 'pass']);
|
||||
assert.equal(seed.pending, 1);
|
||||
assert.deepEqual(uatRows().map((r) => [r.test, r.result]), [[1, 'pass'], [2, 'pass'], [3, 'pending']]);
|
||||
const fm = runMsdTools(['frontmatter', 'get', rel('03-UAT.md')], root);
|
||||
const parsed = JSON.parse(fm.output);
|
||||
assert.equal(parsed.status, 'testing');
|
||||
assert.equal(parsed.updated, '2026-10-10T00:00:00.000Z');
|
||||
});
|
||||
|
||||
test('item matching ignores case and punctuation', () => {
|
||||
writePassedUat();
|
||||
writeReport({
|
||||
status: 'human_needed',
|
||||
covered: artifacts('01', '02'),
|
||||
items: [{ test: 'login form renders.', expected: 'x' }],
|
||||
});
|
||||
const seed = seedUatFromVerification(phaseDir);
|
||||
assert.equal(seed.changed, false);
|
||||
assert.deepEqual(seed.appended, []);
|
||||
});
|
||||
|
||||
test('a passing row is reset only with a recorded retest_reason, written into the row', () => {
|
||||
writePassedUat();
|
||||
writeReport({
|
||||
status: 'human_needed',
|
||||
covered: artifacts('01', '02'),
|
||||
items: [
|
||||
{ ...ITEMS[0], retest_reason: 'src/auth.js changed after the UAT pass' },
|
||||
ITEMS[1],
|
||||
],
|
||||
});
|
||||
const seed = seedUatFromVerification(phaseDir);
|
||||
assert.deepEqual(seed.retested, [
|
||||
{ test: 1, name: 'Login form renders', previous_result: 'pass', reason: 'src/auth.js changed after the UAT pass' },
|
||||
]);
|
||||
assert.equal(seed.pending, 1);
|
||||
const rows = uatRows();
|
||||
assert.equal(rows[0].result, 'pending');
|
||||
assert.equal(rows[0].retest_reason, 'src/auth.js changed after the UAT pass');
|
||||
assert.equal(rows[1].result, 'pass');
|
||||
});
|
||||
|
||||
test('a row inside a fenced block is not a row', () => {
|
||||
const rows = scanUatRows(
|
||||
['## Tests', '', '```', '### 9. Fake', 'result: pass', '```', '', '### 1. Real', 'result: [pending]', ''],
|
||||
);
|
||||
assert.deepEqual(rows.map((r) => [r.test, r.result]), [[1, 'pending']]);
|
||||
});
|
||||
|
||||
test('returns null when the phase has no verification report', () => {
|
||||
assert.equal(seedUatFromVerification(phaseDir), null);
|
||||
});
|
||||
});
|
||||
|
||||
describe('verification.canonicalize-uat — the single flip seam', () => {
|
||||
test('flips human_needed to passed when every UAT row passed', () => {
|
||||
writeReport({ status: 'human_needed', covered: artifacts('01', '02'), items: ITEMS });
|
||||
writePassedUat();
|
||||
const result = canonicalizeHumanNeeded(phaseDir);
|
||||
assert.deepEqual(result, {
|
||||
canonicalized: true,
|
||||
status: 'passed',
|
||||
reason: CANONICALIZE_REASON.UAT_PASSED,
|
||||
blockers: [],
|
||||
});
|
||||
assert.equal(readVerificationStatus(phaseDir).status, 'passed');
|
||||
});
|
||||
|
||||
test('refuses while a UAT row is pending, and names the blocking rows (#4663)', () => {
|
||||
writeReport({ status: 'human_needed', covered: artifacts('01', '02'), items: ITEMS });
|
||||
seedUatFromVerification(phaseDir);
|
||||
const result = canonicalizeHumanNeeded(phaseDir);
|
||||
assert.equal(result.canonicalized, false);
|
||||
assert.equal(result.reason, CANONICALIZE_REASON.UAT_NOT_PASSED);
|
||||
assert.equal(result.blockers.length, 2);
|
||||
assert.equal(readVerificationStatus(phaseDir).status, 'human_needed');
|
||||
});
|
||||
|
||||
test('refuses when there is no UAT evidence at all (zero issues is not a pass)', () => {
|
||||
writeReport({ status: 'human_needed', covered: artifacts('01', '02'), items: ITEMS });
|
||||
const result = canonicalizeHumanNeeded(phaseDir);
|
||||
assert.equal(result.canonicalized, false);
|
||||
assert.equal(result.reason, CANONICALIZE_REASON.UAT_NOT_PASSED);
|
||||
});
|
||||
|
||||
test('never flips a stale report, even with a fully passed UAT', () => {
|
||||
writeReport({ status: 'human_needed', covered: artifacts('01', '02'), items: ITEMS });
|
||||
writePassedUat();
|
||||
writePlan('03');
|
||||
const result = canonicalizeHumanNeeded(phaseDir);
|
||||
assert.equal(result.canonicalized, false);
|
||||
assert.equal(result.status, 'stale');
|
||||
assert.equal(result.reason, CANONICALIZE_REASON.NOT_HUMAN_NEEDED);
|
||||
});
|
||||
|
||||
test('CLI: verification canonicalize-uat emits the typed result', () => {
|
||||
writeReport({ status: 'human_needed', covered: artifacts('01', '02'), items: ITEMS });
|
||||
writePassedUat();
|
||||
const run = runMsdTools(['verification', 'canonicalize-uat', rel()], root);
|
||||
assert.equal(run.success, true);
|
||||
assert.equal(JSON.parse(run.output).reason, 'uat_passed');
|
||||
});
|
||||
});
|
||||
|
||||
describe('acceptance — a passed UAT is neither discarded nor re-requested', () => {
|
||||
test('gap-closure fixture: gap plan + passed UAT reaches passed without re-testing', () => {
|
||||
writeReport({ status: 'human_needed', covered: [...artifacts('01', '02'), 'src/auth.js'], items: ITEMS });
|
||||
writePlan('03'); // the gap-closure plan and its summary
|
||||
writePassedUat(); // the human re-ran UAT at the gap plan's checkpoint
|
||||
assert.equal(readVerificationStatus(phaseDir).stale_reason, 'uncovered_artifacts');
|
||||
|
||||
// Worst-case verifier re-run: it covers plan 03 but re-lists both human
|
||||
// items without consulting the UAT file (no retest_reason).
|
||||
writeReport({ status: 'human_needed', covered: [...artifacts('01', '02', '03'), 'src/auth.js'], items: ITEMS });
|
||||
const before = uatBytes();
|
||||
const seed = seedUatFromVerification(phaseDir);
|
||||
assert.equal(seed.pending, 0);
|
||||
assert.equal(uatBytes(), before);
|
||||
|
||||
assert.equal(canonicalizeHumanNeeded(phaseDir).canonicalized, true);
|
||||
const status = readVerificationStatus(phaseDir);
|
||||
assert.equal(status.status, 'passed');
|
||||
assert.equal(status.next_command, '');
|
||||
assert.deepEqual(uatRows().map((r) => r.result), ['pass', 'pass']);
|
||||
});
|
||||
|
||||
test('real change still stales, and the affected row is marked for re-test', () => {
|
||||
const covered = [...artifacts('01', '02'), 'src/auth.js', 'src/billing.js'];
|
||||
writeReport({ status: 'human_needed', covered, items: ITEMS });
|
||||
writePassedUat();
|
||||
assert.equal(canonicalizeHumanNeeded(phaseDir).canonicalized, true);
|
||||
setMtime(rel('03-UAT.md'), '2026-10-01T11:00:00.000Z');
|
||||
setMtime('src/billing.js', '2026-09-30T00:00:00.000Z');
|
||||
|
||||
fs.writeFileSync(path.join(root, 'src', 'auth.js'), 'export const login = 2;\n');
|
||||
setMtime('src/auth.js', '2026-10-05T00:00:00.000Z');
|
||||
|
||||
const status = readVerificationStatus(phaseDir);
|
||||
assert.equal(status.status, 'stale');
|
||||
assert.equal(status.stale_reason, 'covered_changed');
|
||||
|
||||
const evidence = readUatEvidence(phaseDir);
|
||||
assert.deepEqual(evidence.changed_since_uat, [{ file: 'src/auth.js', reason: 'modified' }]);
|
||||
assert.equal(evidence.code_unchanged_since_uat, false);
|
||||
assert.deepEqual(evidence.checked_files, ['src/auth.js', 'src/billing.js']);
|
||||
assert.deepEqual(evidence.rows.map((r) => r.passing), [true, true]);
|
||||
|
||||
// The verifier re-lists only the row the changed file bears on.
|
||||
writeReport({
|
||||
status: 'human_needed',
|
||||
covered,
|
||||
items: [{ ...ITEMS[0], retest_reason: 'src/auth.js changed after the UAT pass' }],
|
||||
});
|
||||
const seed = seedUatFromVerification(phaseDir);
|
||||
assert.deepEqual(seed.retested.map((r) => r.test), [1]);
|
||||
assert.deepEqual(uatRows().map((r) => r.result), ['pending', 'pass']);
|
||||
|
||||
const refused = canonicalizeHumanNeeded(phaseDir);
|
||||
assert.equal(refused.canonicalized, false);
|
||||
assert.equal(refused.reason, CANONICALIZE_REASON.UAT_NOT_PASSED);
|
||||
assert.equal(readVerificationStatus(phaseDir).status, 'human_needed');
|
||||
});
|
||||
|
||||
test('a deleted covered file counts as changed since the UAT', () => {
|
||||
writePassedUat();
|
||||
const evidence = readUatEvidence(phaseDir, ['src/auth.js', 'src/gone.js', rel('03-01-SUMMARY.md')]);
|
||||
assert.deepEqual(evidence.checked_files, ['src/auth.js', 'src/gone.js']);
|
||||
assert.deepEqual(
|
||||
evidence.changed_since_uat.filter((c) => c.reason === 'missing'),
|
||||
[{ file: 'src/gone.js', reason: 'missing' }],
|
||||
);
|
||||
});
|
||||
|
||||
test('no UAT file is no evidence', () => {
|
||||
const evidence = readUatEvidence(phaseDir, ['src/auth.js']);
|
||||
assert.equal(evidence.uat_file, '');
|
||||
assert.equal(evidence.code_unchanged_since_uat, false);
|
||||
assert.deepEqual(evidence.rows, []);
|
||||
});
|
||||
|
||||
test('loop guard: after UAT passes, progress does not route the phase anywhere', () => {
|
||||
writeReport({ status: 'human_needed', covered: [...artifacts('01', '02'), 'src/auth.js'], items: ITEMS });
|
||||
writePassedUat();
|
||||
canonicalizeHumanNeeded(phaseDir);
|
||||
|
||||
const run = runMsdTools(['init', 'progress'], root);
|
||||
assert.equal(run.success, true);
|
||||
const progress = JSON.parse(run.output);
|
||||
const phase = progress.phases.find((p) => p.number === '03');
|
||||
assert.equal(phase.verification_status, 'passed');
|
||||
assert.equal(phase.phase_complete, true);
|
||||
assert.equal(phase.verification_next_command, '');
|
||||
assert.equal(phase.verification_stale_reason, '');
|
||||
assert.deepEqual(progress.reverify_phases, []);
|
||||
});
|
||||
|
||||
test('cross-phase drift: finished phases surface as ONE re-verify action and keep their UAT', () => {
|
||||
// Phase 03 passed with UAT; a second finished phase shares src/auth.js.
|
||||
const covered03 = [...artifacts('01', '02'), 'src/auth.js', 'src/billing.js'];
|
||||
writeReport({ status: 'human_needed', covered: covered03, items: ITEMS });
|
||||
writePassedUat();
|
||||
canonicalizeHumanNeeded(phaseDir);
|
||||
setMtime(rel('03-UAT.md'), '2026-10-01T11:00:00.000Z');
|
||||
setMtime('src/auth.js', '2026-09-30T00:00:00.000Z');
|
||||
|
||||
const otherDir = path.join(root, '.planning', 'phases', '04-other');
|
||||
fs.mkdirSync(otherDir, { recursive: true });
|
||||
fs.writeFileSync(path.join(otherDir, '04-01-PLAN.md'), 'plan\n');
|
||||
fs.writeFileSync(path.join(otherDir, '04-01-SUMMARY.md'), 'summary\n');
|
||||
const covered04 = ['.planning/phases/04-other/04-01-PLAN.md', '.planning/phases/04-other/04-01-SUMMARY.md', 'src/billing.js'];
|
||||
fs.writeFileSync(
|
||||
path.join(otherDir, '04-VERIFICATION.md'),
|
||||
[
|
||||
'---',
|
||||
'status: passed',
|
||||
`covered_files: [${covered04.join(', ')}]`,
|
||||
`covered_digest: "${computeCoveredDigest(root, covered04, undefined, { phaseDir: otherDir })}"`,
|
||||
'---',
|
||||
'',
|
||||
].join('\n'),
|
||||
);
|
||||
|
||||
// A later phase edits the shared framework file.
|
||||
fs.writeFileSync(path.join(root, 'src', 'billing.js'), 'export const bill = 2;\n');
|
||||
setMtime('src/billing.js', '2026-10-08T00:00:00.000Z');
|
||||
|
||||
const progress = JSON.parse(runMsdTools(['init', 'progress'], root).output);
|
||||
assert.deepEqual(
|
||||
progress.reverify_phases.map((p) => [p.number, p.stale_reason]),
|
||||
[['03', 'covered_changed'], ['04', 'covered_changed']],
|
||||
);
|
||||
|
||||
// Evidence names only the drifted file; both UAT rows still stand.
|
||||
const evidence = readUatEvidence(phaseDir);
|
||||
assert.deepEqual(evidence.changed_since_uat, [{ file: 'src/billing.js', reason: 'modified' }]);
|
||||
assert.deepEqual(evidence.rows.map((r) => r.passing), [true, true]);
|
||||
|
||||
// The verifier finds neither row exercises billing: it re-verifies to
|
||||
// `passed` with no human items. The UAT file is untouched and nothing is
|
||||
// left to re-verify for phase 03.
|
||||
const before = uatBytes();
|
||||
writeReport({ status: 'passed', covered: covered03 });
|
||||
assert.equal(seedUatFromVerification(phaseDir).changed, false);
|
||||
assert.equal(uatBytes(), before);
|
||||
const after = JSON.parse(runMsdTools(['init', 'progress'], root).output);
|
||||
assert.deepEqual(after.reverify_phases.map((p) => p.number), ['04']);
|
||||
});
|
||||
|
||||
test('CLI: verification uat-evidence and seed-uat emit JSON; unknown flags are usage errors', () => {
|
||||
writeReport({ status: 'human_needed', covered: [...artifacts('01', '02'), 'src/auth.js'], items: ITEMS });
|
||||
const seed = runMsdTools(['verification', 'seed-uat', rel()], root);
|
||||
assert.equal(seed.success, true);
|
||||
assert.equal(JSON.parse(seed.output).created, true);
|
||||
|
||||
const evidence = runMsdTools(['verification', 'uat-evidence', rel(), '--files', 'src/auth.js,src/billing.js'], root);
|
||||
assert.equal(evidence.success, true);
|
||||
assert.deepEqual(JSON.parse(evidence.output).checked_files, ['src/auth.js', 'src/billing.js']);
|
||||
|
||||
const bad = runMsdTools(['verification', 'uat-evidence', rel(), '--nope'], root);
|
||||
assert.equal(bad.success, false);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user