Merge pull request #420 from davesienkowski/fix/executor-completion-verification
fix(executor): add completion verification to prevent hallucinated success (#315)
This commit is contained in:
@@ -678,6 +678,45 @@ During execution, these authentication requirements were handled:
|
|||||||
|
|
||||||
</summary_creation>
|
</summary_creation>
|
||||||
|
|
||||||
|
<self_check>
|
||||||
|
After writing SUMMARY.md, verify your own claims before proceeding.
|
||||||
|
|
||||||
|
**1. Check created files exist:**
|
||||||
|
|
||||||
|
Parse `key-files.created` from the SUMMARY frontmatter. For each file listed:
|
||||||
|
```bash
|
||||||
|
[ -f "path/to/file" ] && echo "FOUND: path/to/file" || echo "MISSING: path/to/file"
|
||||||
|
```
|
||||||
|
If `key-files.created` is empty (docs-only plans), skip this check.
|
||||||
|
|
||||||
|
**2. Check commits exist:**
|
||||||
|
|
||||||
|
Parse commit hashes from the "Task Commits" section. Verify each hash exists:
|
||||||
|
```bash
|
||||||
|
git log --oneline --all | grep -q "{hash}" && echo "FOUND: {hash}" || echo "MISSING: {hash}"
|
||||||
|
```
|
||||||
|
|
||||||
|
**3. Append self-check result to SUMMARY.md:**
|
||||||
|
|
||||||
|
If ANY file or commit is missing, append to SUMMARY.md:
|
||||||
|
```markdown
|
||||||
|
## Self-Check: FAILED
|
||||||
|
|
||||||
|
Missing files:
|
||||||
|
- path/to/missing-file.ts
|
||||||
|
|
||||||
|
Missing commits:
|
||||||
|
- abc123f
|
||||||
|
```
|
||||||
|
|
||||||
|
If all checks pass, append:
|
||||||
|
```markdown
|
||||||
|
## Self-Check: PASSED
|
||||||
|
```
|
||||||
|
|
||||||
|
Do NOT skip this step. Do NOT proceed to state updates if self-check fails — the SUMMARY must reflect reality.
|
||||||
|
</self_check>
|
||||||
|
|
||||||
<state_updates>
|
<state_updates>
|
||||||
After creating SUMMARY.md, update STATE.md.
|
After creating SUMMARY.md, update STATE.md.
|
||||||
|
|
||||||
|
|||||||
@@ -76,7 +76,7 @@ Phase: $ARGUMENTS
|
|||||||
For each wave in order:
|
For each wave in order:
|
||||||
- Spawn `gsd-executor` for each plan in wave (parallel Task calls)
|
- Spawn `gsd-executor` for each plan in wave (parallel Task calls)
|
||||||
- Wait for completion (Task blocks)
|
- Wait for completion (Task blocks)
|
||||||
- Verify SUMMARYs created
|
- Verify SUMMARYs created and spot-check claims
|
||||||
- Proceed to next wave
|
- Proceed to next wave
|
||||||
|
|
||||||
5. **Aggregate results**
|
5. **Aggregate results**
|
||||||
|
|||||||
@@ -337,6 +337,20 @@ Execute each wave in sequence. Autonomous plans within a wave run in parallel **
|
|||||||
- Read SUMMARY.md to extract what was built
|
- Read SUMMARY.md to extract what was built
|
||||||
- Note any issues or deviations
|
- Note any issues or deviations
|
||||||
|
|
||||||
|
**Spot-check claims before trusting SUMMARY:**
|
||||||
|
|
||||||
|
For each completed plan's SUMMARY.md:
|
||||||
|
- Pick the first 2 files from `key-files.created` frontmatter — verify they exist on disk with `[ -f ]`
|
||||||
|
- Check `git log --oneline --all --grep="{phase}-{plan}"` returns at least 1 commit
|
||||||
|
- Check SUMMARY.md for `## Self-Check: FAILED` marker
|
||||||
|
|
||||||
|
If ANY spot-check fails:
|
||||||
|
- Do NOT proceed silently
|
||||||
|
- Report which plan failed verification and what was missing
|
||||||
|
- Route to failure handler (step 4): ask user "Retry plan?" or "Continue with remaining waves?"
|
||||||
|
|
||||||
|
If spot-checks pass: proceed normally.
|
||||||
|
|
||||||
**Output:**
|
**Output:**
|
||||||
```
|
```
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -460,6 +460,13 @@ For Pattern A (fully autonomous) and Pattern C (decision-dependent), skip this s
|
|||||||
|
|
||||||
D. Report completion
|
D. Report completion
|
||||||
|
|
||||||
|
E. Self-check the aggregated SUMMARY:
|
||||||
|
- Parse `key-files.created` from the SUMMARY frontmatter
|
||||||
|
- Verify first 2 files exist on disk with `[ -f ]`
|
||||||
|
- Check `git log --oneline --all --grep="{phase}-{plan}"` returns at least 1 commit
|
||||||
|
- If any check fails: append `## Self-Check: FAILED` section to SUMMARY listing missing items
|
||||||
|
- If all pass: append `## Self-Check: PASSED`
|
||||||
|
|
||||||
**Example execution trace:**
|
**Example execution trace:**
|
||||||
|
|
||||||
````
|
````
|
||||||
|
|||||||
Reference in New Issue
Block a user