Merge pull request #1212 from jecanore/feat/audit-uat
feat: add verification debt tracking and /gsd:audit-uat command
This commit is contained in:
Tom Boucher
109
get-shit-done/workflows/audit-uat.md
Normal file
109
get-shit-done/workflows/audit-uat.md
Normal file
@@ -0,0 +1,109 @@
|
||||
<purpose>
|
||||
Cross-phase audit of all UAT and verification files. Finds every outstanding item (pending, skipped, blocked, human_needed), optionally verifies against the codebase to detect stale docs, and produces a prioritized human test plan.
|
||||
</purpose>
|
||||
|
||||
<process>
|
||||
|
||||
<step name="initialize">
|
||||
Run the CLI audit:
|
||||
|
||||
```bash
|
||||
AUDIT=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" audit-uat --raw)
|
||||
```
|
||||
|
||||
Parse JSON for `results` array and `summary` object.
|
||||
|
||||
If `summary.total_items` is 0:
|
||||
```
|
||||
## All Clear
|
||||
|
||||
No outstanding UAT or verification items found across all phases.
|
||||
All tests are passing, resolved, or diagnosed with fix plans.
|
||||
```
|
||||
Stop here.
|
||||
</step>
|
||||
|
||||
<step name="categorize">
|
||||
Group items by what's actionable NOW vs. what needs prerequisites:
|
||||
|
||||
**Testable Now** (no external dependencies):
|
||||
- `pending` — tests never run
|
||||
- `human_uat` — human verification items
|
||||
- `skipped_unresolved` — skipped without clear blocking reason
|
||||
|
||||
**Needs Prerequisites:**
|
||||
- `server_blocked` — needs external server running
|
||||
- `device_needed` — needs physical device (not simulator)
|
||||
- `build_needed` — needs release/preview build
|
||||
- `third_party` — needs external service configuration
|
||||
|
||||
For each item in "Testable Now", use Grep/Read to check if the underlying feature still exists in the codebase:
|
||||
- If the test references a component/function that no longer exists → mark as `stale`
|
||||
- If the test references code that has been significantly rewritten → mark as `needs_update`
|
||||
- Otherwise → mark as `active`
|
||||
</step>
|
||||
|
||||
<step name="present">
|
||||
Present the audit report:
|
||||
|
||||
```
|
||||
## UAT Audit Report
|
||||
|
||||
**{total_items} outstanding items across {total_files} files in {phase_count} phases**
|
||||
|
||||
### Testable Now ({count})
|
||||
|
||||
| # | Phase | Test | Description | Status |
|
||||
|---|-------|------|-------------|--------|
|
||||
| 1 | {phase} | {test_name} | {expected} | {active/stale/needs_update} |
|
||||
...
|
||||
|
||||
### Needs Prerequisites ({count})
|
||||
|
||||
| # | Phase | Test | Blocked By | Description |
|
||||
|---|-------|------|------------|-------------|
|
||||
| 1 | {phase} | {test_name} | {category} | {expected} |
|
||||
...
|
||||
|
||||
### Stale (can be closed) ({count})
|
||||
|
||||
| # | Phase | Test | Why Stale |
|
||||
|---|-------|------|-----------|
|
||||
| 1 | {phase} | {test_name} | {reason} |
|
||||
...
|
||||
|
||||
---
|
||||
|
||||
## Recommended Actions
|
||||
|
||||
1. **Close stale items:** `/gsd:verify-work {phase}` — mark stale tests as resolved
|
||||
2. **Run active tests:** Human UAT test plan below
|
||||
3. **When prerequisites met:** Retest blocked items with `/gsd:verify-work {phase}`
|
||||
```
|
||||
</step>
|
||||
|
||||
<step name="test_plan">
|
||||
Generate a human UAT test plan for "Testable Now" + "active" items only:
|
||||
|
||||
Group by what can be tested together (same screen, same feature, same prerequisite):
|
||||
|
||||
```
|
||||
## Human UAT Test Plan
|
||||
|
||||
### Group 1: {category — e.g., "Billing Flow"}
|
||||
Prerequisites: {what needs to be running/configured}
|
||||
|
||||
1. **{Test name}** (Phase {N})
|
||||
- Navigate to: {where}
|
||||
- Do: {action}
|
||||
- Expected: {expected behavior}
|
||||
|
||||
2. **{Test name}** (Phase {N})
|
||||
...
|
||||
|
||||
### Group 2: {category}
|
||||
...
|
||||
```
|
||||
</step>
|
||||
|
||||
</process>
|
||||
@@ -525,6 +525,51 @@ grep "^status:" "$PHASE_DIR"/*-VERIFICATION.md | cut -d: -f2 | tr -d ' '
|
||||
| `gaps_found` | Present gap summary, offer `/gsd:plan-phase {phase} --gaps` |
|
||||
|
||||
**If human_needed:**
|
||||
|
||||
**Step A: Persist human verification items as UAT file.**
|
||||
|
||||
Create `{phase_dir}/{phase_num}-HUMAN-UAT.md` using UAT template format:
|
||||
|
||||
```markdown
|
||||
---
|
||||
status: partial
|
||||
phase: {phase_num}-{phase_name}
|
||||
source: [{phase_num}-VERIFICATION.md]
|
||||
started: [now ISO]
|
||||
updated: [now ISO]
|
||||
---
|
||||
|
||||
## Current Test
|
||||
|
||||
[awaiting human testing]
|
||||
|
||||
## Tests
|
||||
|
||||
{For each human_verification item from VERIFICATION.md:}
|
||||
|
||||
### {N}. {item description}
|
||||
expected: {expected behavior from VERIFICATION.md}
|
||||
result: [pending]
|
||||
|
||||
## Summary
|
||||
|
||||
total: {count}
|
||||
passed: 0
|
||||
issues: 0
|
||||
pending: {count}
|
||||
skipped: 0
|
||||
blocked: 0
|
||||
|
||||
## Gaps
|
||||
```
|
||||
|
||||
Commit the file:
|
||||
```bash
|
||||
node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" commit "test({phase_num}): persist human verification items as UAT" --files "{phase_dir}/{phase_num}-HUMAN-UAT.md"
|
||||
```
|
||||
|
||||
**Step B: Present to user:**
|
||||
|
||||
```
|
||||
## ✓ Phase {X}: {Name} — Human Verification Required
|
||||
|
||||
@@ -532,9 +577,15 @@ All automated checks passed. {N} items need human testing:
|
||||
|
||||
{From VERIFICATION.md human_verification section}
|
||||
|
||||
Items saved to `{phase_num}-HUMAN-UAT.md` — they will appear in `/gsd:progress` and `/gsd:audit-uat`.
|
||||
|
||||
"approved" → continue | Report issues → gap closure
|
||||
```
|
||||
|
||||
**If user says "approved":** Proceed to `update_roadmap`. The HUMAN-UAT.md file persists with `status: partial` and will surface in future progress checks until the user runs `/gsd:verify-work` on it.
|
||||
|
||||
**If user reports issues:** Proceed to gap closure as currently implemented.
|
||||
|
||||
**If gaps_found:**
|
||||
```
|
||||
## ⚠ Phase {X}: {Name} — Gaps Found
|
||||
@@ -572,8 +623,18 @@ The CLI handles:
|
||||
- Updating plan count to final
|
||||
- Advancing STATE.md to next phase
|
||||
- Updating REQUIREMENTS.md traceability
|
||||
- Scanning for verification debt (returns `warnings` array)
|
||||
|
||||
Extract from result: `next_phase`, `next_phase_name`, `is_last_phase`.
|
||||
Extract from result: `next_phase`, `next_phase_name`, `is_last_phase`, `warnings`, `has_warnings`.
|
||||
|
||||
**If has_warnings is true:**
|
||||
```
|
||||
## Phase {X} marked complete with {N} warnings:
|
||||
|
||||
{list each warning}
|
||||
|
||||
These items are tracked and will appear in `/gsd:progress` and `/gsd:audit-uat`.
|
||||
```
|
||||
|
||||
```bash
|
||||
node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" commit "docs(phase-{X}): complete phase execution" --files .planning/ROADMAP.md .planning/STATE.md .planning/REQUIREMENTS.md {phase_dir}/*-VERIFICATION.md
|
||||
|
||||
@@ -373,6 +373,17 @@ Capture a forward-looking idea with trigger conditions for automatic surfacing.
|
||||
|
||||
Usage: `/gsd:plant-seed "add real-time notifications when we build the events system"`
|
||||
|
||||
---
|
||||
|
||||
**`/gsd:audit-uat`**
|
||||
Cross-phase audit of all outstanding UAT and verification items.
|
||||
- Scans every phase for pending, skipped, blocked, and human_needed items
|
||||
- Cross-references against codebase to detect stale documentation
|
||||
- Produces prioritized human test plan grouped by testability
|
||||
- Use before starting a new milestone to clear verification debt
|
||||
|
||||
Usage: `/gsd:audit-uat`
|
||||
|
||||
### Milestone Auditing
|
||||
|
||||
**`/gsd:audit-milestone [version]`**
|
||||
|
||||
@@ -150,17 +150,47 @@ State: "This phase has {X} plans, {Y} summaries."
|
||||
Check for UAT.md files with status "diagnosed" (has gaps needing fixes).
|
||||
|
||||
```bash
|
||||
# Check for diagnosed UAT with gaps
|
||||
grep -l "status: diagnosed" .planning/phases/[current-phase-dir]/*-UAT.md 2>/dev/null
|
||||
# Check for diagnosed UAT with gaps or partial (incomplete) testing
|
||||
grep -l "status: diagnosed\|status: partial" .planning/phases/[current-phase-dir]/*-UAT.md 2>/dev/null
|
||||
```
|
||||
|
||||
Track:
|
||||
- `uat_with_gaps`: UAT.md files with status "diagnosed" (gaps need fixing)
|
||||
- `uat_partial`: UAT.md files with status "partial" (incomplete testing)
|
||||
|
||||
**Step 1.6: Cross-phase health check**
|
||||
|
||||
Scan ALL phases in the current milestone for outstanding verification debt using the CLI (which respects milestone boundaries via `getMilestonePhaseFilter`):
|
||||
|
||||
```bash
|
||||
DEBT=$(node "$HOME/.claude/get-shit-done/bin/gsd-tools.cjs" audit-uat --raw 2>/dev/null)
|
||||
```
|
||||
|
||||
Parse JSON for `summary.total_items` and `summary.total_files`.
|
||||
|
||||
Track: `outstanding_debt` — `summary.total_items` from the audit.
|
||||
|
||||
**If outstanding_debt > 0:** Add a warning section to the progress report output (in the `report` step), placed between "## What's Next" and the route suggestion:
|
||||
|
||||
```markdown
|
||||
## Verification Debt ({N} files across prior phases)
|
||||
|
||||
| Phase | File | Issue |
|
||||
|-------|------|-------|
|
||||
| {phase} | {filename} | {pending_count} pending, {skipped_count} skipped, {blocked_count} blocked |
|
||||
| {phase} | {filename} | human_needed — {count} items |
|
||||
|
||||
Review: `/gsd:audit-uat` — full cross-phase audit
|
||||
Resume testing: `/gsd:verify-work {phase}` — retest specific phase
|
||||
```
|
||||
|
||||
This is a WARNING, not a blocker — routing proceeds normally. The debt is visible so the user can make an informed choice.
|
||||
|
||||
**Step 2: Route based on counts**
|
||||
|
||||
| Condition | Meaning | Action |
|
||||
|-----------|---------|--------|
|
||||
| uat_partial > 0 | UAT testing incomplete | Go to **Route E.2** |
|
||||
| uat_with_gaps > 0 | UAT gaps need fix plans | Go to **Route E** |
|
||||
| summaries < plans | Unexecuted plans exist | Go to **Route A** |
|
||||
| summaries = plans AND plans > 0 | Phase complete | Go to Step 3 |
|
||||
@@ -260,6 +290,32 @@ UAT.md exists with gaps (diagnosed issues). User needs to plan fixes.
|
||||
|
||||
---
|
||||
|
||||
**Route E.2: UAT testing incomplete (partial)**
|
||||
|
||||
UAT.md exists with `status: partial` — testing session ended before all items resolved.
|
||||
|
||||
```
|
||||
---
|
||||
|
||||
## Incomplete UAT Testing
|
||||
|
||||
**{phase_num}-UAT.md** has {N} unresolved tests (pending, blocked, or skipped).
|
||||
|
||||
`/gsd:verify-work {phase}` — resume testing from where you left off
|
||||
|
||||
<sub>`/clear` first → fresh context window</sub>
|
||||
|
||||
---
|
||||
|
||||
**Also available:**
|
||||
- `/gsd:audit-uat` — full cross-phase UAT audit
|
||||
- `/gsd:execute-phase {phase}` — execute phase plans
|
||||
|
||||
---
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Step 3: Check milestone status (only when phase complete)**
|
||||
|
||||
Read ROADMAP.md and identify:
|
||||
|
||||
@@ -74,6 +74,30 @@ cat .planning/config.json 2>/dev/null
|
||||
|
||||
</config-check>
|
||||
|
||||
**Check for verification debt in this phase:**
|
||||
|
||||
```bash
|
||||
# Count outstanding items in current phase
|
||||
OUTSTANDING=""
|
||||
for f in .planning/phases/XX-current/*-UAT.md .planning/phases/XX-current/*-VERIFICATION.md; do
|
||||
[ -f "$f" ] || continue
|
||||
grep -q "result: pending\|result: blocked\|status: partial\|status: human_needed\|status: diagnosed" "$f" && OUTSTANDING="$OUTSTANDING\n$(basename $f)"
|
||||
done
|
||||
```
|
||||
|
||||
**If OUTSTANDING is not empty:**
|
||||
|
||||
Append to the completion confirmation message (regardless of mode):
|
||||
|
||||
```
|
||||
Outstanding verification items in this phase:
|
||||
{list filenames}
|
||||
|
||||
These will carry forward as debt. Review: `/gsd:audit-uat`
|
||||
```
|
||||
|
||||
This does NOT block transition — it ensures the user sees the debt before confirming.
|
||||
|
||||
**If all plans complete:**
|
||||
|
||||
<if mode="yolo">
|
||||
|
||||
@@ -231,6 +231,29 @@ result: skipped
|
||||
reason: [user's reason if provided]
|
||||
```
|
||||
|
||||
**If response indicates blocked:**
|
||||
- "blocked", "can't test - server not running", "need physical device", "need release build"
|
||||
- Or any response containing: "server", "blocked", "not running", "physical device", "release build"
|
||||
|
||||
Infer blocked_by tag from response:
|
||||
- Contains: server, not running, gateway, API → `server`
|
||||
- Contains: physical, device, hardware, real phone → `physical-device`
|
||||
- Contains: release, preview, build, EAS → `release-build`
|
||||
- Contains: stripe, twilio, third-party, configure → `third-party`
|
||||
- Contains: depends on, prior phase, prerequisite → `prior-phase`
|
||||
- Default: `other`
|
||||
|
||||
Update Tests section:
|
||||
```
|
||||
### {N}. {name}
|
||||
expected: {expected}
|
||||
result: blocked
|
||||
blocked_by: {inferred tag}
|
||||
reason: "{verbatim user response}"
|
||||
```
|
||||
|
||||
Note: Blocked tests do NOT go into the Gaps section (they aren't code issues — they're prerequisite gates).
|
||||
|
||||
**If response is anything else:**
|
||||
- Treat as issue description
|
||||
|
||||
@@ -293,8 +316,24 @@ Proceed to `present_test`.
|
||||
<step name="complete_session">
|
||||
**Complete testing and commit:**
|
||||
|
||||
**Determine final status:**
|
||||
|
||||
Count results:
|
||||
- `pending_count`: tests with `result: [pending]`
|
||||
- `blocked_count`: tests with `result: blocked`
|
||||
- `skipped_no_reason`: tests with `result: skipped` and no `reason` field
|
||||
|
||||
```
|
||||
if pending_count > 0 OR blocked_count > 0 OR skipped_no_reason > 0:
|
||||
status: partial
|
||||
# Session ended but not all tests resolved
|
||||
else:
|
||||
status: complete
|
||||
# All tests have a definitive result (pass, issue, or skipped-with-reason)
|
||||
```
|
||||
|
||||
Update frontmatter:
|
||||
- status: complete
|
||||
- status: {computed status}
|
||||
- updated: [now]
|
||||
|
||||
Clear Current Test section:
|
||||
|
||||
Reference in New Issue
Block a user