Merge pull request #420 from davesienkowski/fix/executor-completion-verification
fix(executor): add completion verification to prevent hallucinated success (#315)
This commit is contained in:
@@ -678,6 +678,45 @@ During execution, these authentication requirements were handled:
|
||||
|
||||
</summary_creation>
|
||||
|
||||
<self_check>
|
||||
After writing SUMMARY.md, verify your own claims before proceeding.
|
||||
|
||||
**1. Check created files exist:**
|
||||
|
||||
Parse `key-files.created` from the SUMMARY frontmatter. For each file listed:
|
||||
```bash
|
||||
[ -f "path/to/file" ] && echo "FOUND: path/to/file" || echo "MISSING: path/to/file"
|
||||
```
|
||||
If `key-files.created` is empty (docs-only plans), skip this check.
|
||||
|
||||
**2. Check commits exist:**
|
||||
|
||||
Parse commit hashes from the "Task Commits" section. Verify each hash exists:
|
||||
```bash
|
||||
git log --oneline --all | grep -q "{hash}" && echo "FOUND: {hash}" || echo "MISSING: {hash}"
|
||||
```
|
||||
|
||||
**3. Append self-check result to SUMMARY.md:**
|
||||
|
||||
If ANY file or commit is missing, append to SUMMARY.md:
|
||||
```markdown
|
||||
## Self-Check: FAILED
|
||||
|
||||
Missing files:
|
||||
- path/to/missing-file.ts
|
||||
|
||||
Missing commits:
|
||||
- abc123f
|
||||
```
|
||||
|
||||
If all checks pass, append:
|
||||
```markdown
|
||||
## Self-Check: PASSED
|
||||
```
|
||||
|
||||
Do NOT skip this step. Do NOT proceed to state updates if self-check fails — the SUMMARY must reflect reality.
|
||||
</self_check>
|
||||
|
||||
<state_updates>
|
||||
After creating SUMMARY.md, update STATE.md.
|
||||
|
||||
|
||||
@@ -76,7 +76,7 @@ Phase: $ARGUMENTS
|
||||
For each wave in order:
|
||||
- Spawn `gsd-executor` for each plan in wave (parallel Task calls)
|
||||
- Wait for completion (Task blocks)
|
||||
- Verify SUMMARYs created
|
||||
- Verify SUMMARYs created and spot-check claims
|
||||
- Proceed to next wave
|
||||
|
||||
5. **Aggregate results**
|
||||
|
||||
@@ -337,6 +337,20 @@ Execute each wave in sequence. Autonomous plans within a wave run in parallel **
|
||||
- Read SUMMARY.md to extract what was built
|
||||
- Note any issues or deviations
|
||||
|
||||
**Spot-check claims before trusting SUMMARY:**
|
||||
|
||||
For each completed plan's SUMMARY.md:
|
||||
- Pick the first 2 files from `key-files.created` frontmatter — verify they exist on disk with `[ -f ]`
|
||||
- Check `git log --oneline --all --grep="{phase}-{plan}"` returns at least 1 commit
|
||||
- Check SUMMARY.md for `## Self-Check: FAILED` marker
|
||||
|
||||
If ANY spot-check fails:
|
||||
- Do NOT proceed silently
|
||||
- Report which plan failed verification and what was missing
|
||||
- Route to failure handler (step 4): ask user "Retry plan?" or "Continue with remaining waves?"
|
||||
|
||||
If spot-checks pass: proceed normally.
|
||||
|
||||
**Output:**
|
||||
```
|
||||
---
|
||||
|
||||
@@ -460,6 +460,13 @@ For Pattern A (fully autonomous) and Pattern C (decision-dependent), skip this s
|
||||
|
||||
D. Report completion
|
||||
|
||||
E. Self-check the aggregated SUMMARY:
|
||||
- Parse `key-files.created` from the SUMMARY frontmatter
|
||||
- Verify first 2 files exist on disk with `[ -f ]`
|
||||
- Check `git log --oneline --all --grep="{phase}-{plan}"` returns at least 1 commit
|
||||
- If any check fails: append `## Self-Check: FAILED` section to SUMMARY listing missing items
|
||||
- If all pass: append `## Self-Check: PASSED`
|
||||
|
||||
**Example execution trace:**
|
||||
|
||||
````
|
||||
|
||||
Reference in New Issue
Block a user