Files
msd-core/gsd-core/references/execute-phase-quota-recovery.md
Tom Boucher 455ad49ae3 feat(#2296): config-gated provider escalation on quota-exceeded (#2458)
* test(#2296): failing-first coverage for provider escalation on quota-exceeded

Covers the provider-escalation ladder layered onto EXEC.CLASSIFY: back-compat
(no escalation block without --failure-class), cap boundaries at
min(max_escalations, list length) at limit-1/limit/limit+1, opt-in gating,
malformed/hostile provider_escalation config, the --failure-class CLI negative
matrix, config-key registration, and a fast-check budget-limit property.

Red until the resolver, CLI flag, and manifest key land.

Refs #2296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(#2296): config-gated provider escalation on quota-exceeded

The dynamic_routing tier ladder escalates within one provider, which does not
help when that provider is what ran out of quota. Add an opt-in provider ladder
layered on the existing EXEC.CLASSIFY seam.

- model-resolver: resolveProviderEscalation walks dynamic_routing.provider_escalation
  capped at min(max_escalations, list length), reporting from/to/attempted/exhausted.
  Invalid entries are dropped (ADR 227 shape validation). Stays a leaf module —
  the quota-class policy decision is the caller's, per the CONTEXT.md contract.
- agent-command-router: export a frozen AGENT_FAILURE_CLASSES so the new CLI
  validator cannot drift from the classifier that produces the values.
- resolve-execution: --failure-class flag; emits an escalation block ONLY when
  passed, so the existing JSON contract is byte-identical for every caller.
- config-schema.manifest: register dynamic_routing.provider_escalation.
- execute-phase step 7.1: auto-escalate, honor Retry-After, fail loudly naming
  every model tried once the ladder is spent.

Refs #2296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2296): extract quota recovery to a reference fragment; regen goldens

The step 7.1a addition pushed gsd-core/workflows/execute-phase.md from 93390 to
95111 LF bytes, past the frozen ADR-857 Phase 6 ceiling (hard <93600, margin
<=93400) asserted by tests/fix-2285-claude-orchestration-wiring.test.cjs. The
base sat 10 bytes under the margin, so no inline wording would have fit.

That gate's own rationale is that optional-feature detail belongs in a fragment,
not the host loop. Moved BOTH the new provider-escalation branch and the
pre-existing manual recovery prompt into
gsd-core/references/execute-phase-quota-recovery.md, leaving step 7.1 as a
one-line pointer. execute-phase.md is now 92880 bytes — 510 SMALLER than base.

Also regenerates the fixtures that legitimately moved because three shipped
files changed (gsd-tools.cjs, config-schema.manifest.json, execute-phase.md):
golden-install-parity + install-tree for all 16 runtimes, INVENTORY.md +
INVENTORY-MANIFEST.json for the new reference, and the workflow size baseline.

Refs #2296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(#2351): make the C1 orphan-reaping test load-independent

tests/run-with-timeout.test.cjs C1 asserted the child heartbeat file exists
after a 1s group-kill window, but the child only wrote it on the first 100ms
setInterval tick. Nothing synchronized the two: on a loaded container the group
is SIGKILLed before that tick lands, the file never appears, and the assertion
fails for a reason unrelated to reaping. Observed failing on both linux-node22
and linux-node24.

The behavior actually under test is the FREEZE assertion (heartbeat stops
advancing => descendant was reaped, not orphaned). That is unaffected by
sampling once more at t=0.

Child now writes its first heartbeat synchronously at startup before arming the
interval, and the kill window widens 1s -> 3s to cover child boot under load.
Both remove the timing dependency; neither weakens what the test proves.

Refs #2296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(#2296): backfill pr:2458 in .changeset/rapid-jays-bark.md

* chore(#2296): regenerate fixtures after rebase onto #2402

The rebase conflicted on the generated golden-install-parity fixtures and
workflow-size-baseline.json because #2402 (b6e6a22fc) regenerated the same
artifacts. Conflict resolution picked a side to unblock the rebase; a true
regeneration on the combined tree then produced further drift, confirming the
resolved content was stale and would have dropped #2402's fixture changes.

Regenerated goldens, install-tree, size baseline, and INVENTORY-MANIFEST from
the merged tree. docs/INVENTORY.md keeps BOTH new reference rows.

execute-phase.md is 92782 LF bytes with both #2402's and this PR's extractions
applied — under the frozen ceiling (hard <93600, margin <=93400).

Refs #2296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 14:59:30 -04:00

2.3 KiB

Step 7.1 detail — class == "quota-exceeded" recovery.

Do not offer "retry now". Run the step-5 spot-check first; if SUMMARY.md is missing but commits exist, route to safe-resume (state.verify-against-disk) instead of an immediate redispatch.

7.1a — provider escalation (#2296, opt-in). A heavier tier on the same throttled provider is still throttled, so when dynamic_routing.provider_escalation is configured GSD swaps PROVIDER rather than waiting for a reset. QUOTA_ATTEMPT starts at 1 on the first quota failure of this phase and increments on each subsequent one.

ESC_JSON=$(gsd_run query resolve-execution gsd-executor --attempt "${QUOTA_ATTEMPT:-1}" --failure-class quota-exceeded)
ESCALATED=$(echo "$ESC_JSON" | jq -r '.escalation.escalated')
EXHAUSTED=$(echo "$ESC_JSON" | jq -r '.escalation.exhausted')
ESC_FROM=$(echo "$ESC_JSON" | jq -r '.escalation.from')
ESC_TO=$(echo "$ESC_JSON" | jq -r '.escalation.to')
ESC_TRIED=$(echo "$ESC_JSON" | jq -r '.escalation.attempted | join(" -> ")')
  • ESCALATED == "true" — log the switch, honor the provider's own backoff (sleep "$RETRY_AFTER" when RETRY_AFTER is set), then re-dispatch the failed plan with executor_model overridden to $ESC_TO and QUOTA_ATTEMPT incremented. Do not prompt — this is the configured, opt-in path.

    ⚡ Provider quota hit — escalating model: {ESC_FROM} → {ESC_TO}
      Runtime sentinel: {SENTINEL}
      {RETRY_HINT}
    
  • EXHAUSTED == "true" — the ladder is spent. Fail loudly naming every model tried, then fall through to the manual options below. Never silently retry the last one.

    ⛔ Provider escalation exhausted — tried: {ESC_TRIED}
    
  • ESCALATED == "false" and not exhausted — escalation is not configured for this project; use the manual path below. This is the default.

7.1b — manual recovery (default when escalation is not configured).

⚠ Plan {plan_id} terminated by provider quota / rate limit
  Runtime sentinel: {SENTINEL}
  {RETRY_HINT}
  Partial commits on worktree branch: {N}
  SUMMARY.md present: {yes|no}
  1. Wait for quota reset, then resume (recommended)
2. Switch to a different runtime / model and resume
3. Abort phase and report partial state

Re-run /gsd:execute-phase after the quota resets for Option 1.