fix: add executor stall recovery contract (#3329)

* fix: add executor stall recovery contract

* chore: add changeset for executor recovery

* chore: keep execute phase within size budget
This commit is contained in:
Tom Boucher
2026-05-09 23:28:52 -04:00
committed by GitHub
parent dc003610e2
commit e7942c21b3
9 changed files with 161 additions and 0 deletions

View File

@@ -81,6 +81,8 @@ Read worktree config:
```bash
USE_WORKTREES=$(gsd-sdk query config-get workflow.use_worktrees 2>/dev/null || echo "true")
EXECUTOR_STALL_INTERVAL_MINUTES=$(gsd-sdk query config-get executor.stall_detect_interval_minutes 2>/dev/null || echo "5")
EXECUTOR_STALL_THRESHOLD_MINUTES=$(gsd-sdk query config-get executor.stall_threshold_minutes 2>/dev/null || echo "10")
```
If the project uses git submodules, worktree isolation is unsafe **only when a plan touches a submodule path** — the executor commit protocol cannot correctly handle submodule commits inside isolated worktrees. The previous behavior unconditionally disabled worktree isolation whenever `.gitmodules` existed, which penalised every plan in a submodule project even when the plan was nowhere near a submodule. Compute submodule paths once and intersect them per-plan with the plan's declared `files_modified` frontmatter.
@@ -146,6 +148,22 @@ MVP_MODE=$(gsd-sdk query phase.mvp-mode "${PHASE_NUMBER}" $MVP_FLAG_ARG --pick a
TDD_MODE=$(gsd-sdk query config-get workflow.tdd_mode 2>/dev/null || echo "false")
```
<step name="safe_resume_gate">
Before trusting `STATE.md` or dispatching any executor, derive `CURRENT_PLAN_ID`
from the active incomplete plan in `INIT`, then search recent history:
```bash
CURRENT_PLAN_ID="{phase_number}-{plan_padded}"
SUMMARY_PATH="{phase_dir}/{plan_padded}-SUMMARY.md"
PLAN_COMMITS=$(git log --oneline --grep="${CURRENT_PLAN_ID}" -30)
```
If production commits exist and `SUMMARY.md is missing`, stop before spawning a
new executor; continuing risks duplicate work and stale `STATE.md`/ROADMAP progress.
Offer these recovery options:
- `close out manually` — inspect commits, write SUMMARY.md, then update STATE/ROADMAP.
- `re-execute from scratch` — revert or supersede partial commits before dispatch.
- `mark-and-skip` — record the anomaly and move on only with explicit confirmation.
</step>
**MVP+TDD gate.** Task-scoped enforcement runs inside plan execution (immediately before each implementation step), where `TASK_FILE`, `PLAN_ID`, and `TASK_ID` are defined. Keep the same predicate and RED-commit contract:
```bash
if [ "$MVP_MODE" = "true" ] && [ "$TDD_MODE" = "true" ]; then
@@ -494,6 +512,8 @@ increases monotonically across waves. `{status}` is `complete` (success),
Before spawning, capture the current HEAD:
```bash
EXPECTED_BASE=$(git rev-parse HEAD)
DISPATCH_TS=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
EXPECTED_BRANCH=$(git rev-parse --abbrev-ref HEAD)
```
**Sequential dispatch for parallel execution (waves with 2+ agents):**
@@ -666,6 +686,7 @@ increases monotonically across waves. `{status}` is `complete` (success),
# For each plan in this wave, check if the executor finished:
SUMMARY_EXISTS=$(test -f "{phase_dir}/{plan_number}-{plan_padded}-SUMMARY.md" && echo "true" || echo "false")
COMMITS_FOUND=$(git log --oneline --all --grep="{phase_number}-{plan_padded}" --since="1 hour ago" | head -1)
COMMITS_SINCE_DISPATCH=$(git log "${EXPECTED_BRANCH}" --since="${DISPATCH_TS}" --oneline | head -1)
```
**If SUMMARY.md exists AND commits are found:** The agent completed successfully —
@@ -676,6 +697,13 @@ increases monotonically across waves. `{status}` is `complete` (success),
activity. If commits are still appearing, wait longer. If no activity, report
the plan as failed and route to the failure handler in step 6.
**Configurable stall surveillance (#3212):** Every `${EXECUTOR_STALL_INTERVAL_MINUTES}`
minutes while waiting, inspect `git log "${EXPECTED_BRANCH}" --since="${DISPATCH_TS}"`
for activity. If no completion signal, no SUMMARY.md, and no expected-branch
commits appear for `${EXECUTOR_STALL_THRESHOLD_MINUTES}` minutes, pause and
ask for one recovery path: `continue waiting`, `kill and retry`, or
`kill and switch to inline execution`.
**This fallback applies automatically to all runtimes.** Claude Code's Agent() normally
returns synchronously, but the fallback ensures resilience if it doesn't.