feat(#2296): config-gated provider escalation on quota-exceeded (#2458)

* test(#2296): failing-first coverage for provider escalation on quota-exceeded

Covers the provider-escalation ladder layered onto EXEC.CLASSIFY: back-compat
(no escalation block without --failure-class), cap boundaries at
min(max_escalations, list length) at limit-1/limit/limit+1, opt-in gating,
malformed/hostile provider_escalation config, the --failure-class CLI negative
matrix, config-key registration, and a fast-check budget-limit property.

Red until the resolver, CLI flag, and manifest key land.

Refs #2296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(#2296): config-gated provider escalation on quota-exceeded

The dynamic_routing tier ladder escalates within one provider, which does not
help when that provider is what ran out of quota. Add an opt-in provider ladder
layered on the existing EXEC.CLASSIFY seam.

- model-resolver: resolveProviderEscalation walks dynamic_routing.provider_escalation
  capped at min(max_escalations, list length), reporting from/to/attempted/exhausted.
  Invalid entries are dropped (ADR 227 shape validation). Stays a leaf module —
  the quota-class policy decision is the caller's, per the CONTEXT.md contract.
- agent-command-router: export a frozen AGENT_FAILURE_CLASSES so the new CLI
  validator cannot drift from the classifier that produces the values.
- resolve-execution: --failure-class flag; emits an escalation block ONLY when
  passed, so the existing JSON contract is byte-identical for every caller.
- config-schema.manifest: register dynamic_routing.provider_escalation.
- execute-phase step 7.1: auto-escalate, honor Retry-After, fail loudly naming
  every model tried once the ladder is spent.

Refs #2296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(#2296): extract quota recovery to a reference fragment; regen goldens

The step 7.1a addition pushed gsd-core/workflows/execute-phase.md from 93390 to
95111 LF bytes, past the frozen ADR-857 Phase 6 ceiling (hard <93600, margin
<=93400) asserted by tests/fix-2285-claude-orchestration-wiring.test.cjs. The
base sat 10 bytes under the margin, so no inline wording would have fit.

That gate's own rationale is that optional-feature detail belongs in a fragment,
not the host loop. Moved BOTH the new provider-escalation branch and the
pre-existing manual recovery prompt into
gsd-core/references/execute-phase-quota-recovery.md, leaving step 7.1 as a
one-line pointer. execute-phase.md is now 92880 bytes — 510 SMALLER than base.

Also regenerates the fixtures that legitimately moved because three shipped
files changed (gsd-tools.cjs, config-schema.manifest.json, execute-phase.md):
golden-install-parity + install-tree for all 16 runtimes, INVENTORY.md +
INVENTORY-MANIFEST.json for the new reference, and the workflow size baseline.

Refs #2296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(#2351): make the C1 orphan-reaping test load-independent

tests/run-with-timeout.test.cjs C1 asserted the child heartbeat file exists
after a 1s group-kill window, but the child only wrote it on the first 100ms
setInterval tick. Nothing synchronized the two: on a loaded container the group
is SIGKILLed before that tick lands, the file never appears, and the assertion
fails for a reason unrelated to reaping. Observed failing on both linux-node22
and linux-node24.

The behavior actually under test is the FREEZE assertion (heartbeat stops
advancing => descendant was reaped, not orphaned). That is unaffected by
sampling once more at t=0.

Child now writes its first heartbeat synchronously at startup before arming the
interval, and the kill window widens 1s -> 3s to cover child boot under load.
Both remove the timing dependency; neither weakens what the test proves.

Refs #2296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(#2296): backfill pr:2458 in .changeset/rapid-jays-bark.md

* chore(#2296): regenerate fixtures after rebase onto #2402

The rebase conflicted on the generated golden-install-parity fixtures and
workflow-size-baseline.json because #2402 (b6e6a22fc) regenerated the same
artifacts. Conflict resolution picked a side to unblock the rebase; a true
regeneration on the combined tree then produced further drift, confirming the
resolved content was stale and would have dropped #2402's fixture changes.

Regenerated goldens, install-tree, size baseline, and INVENTORY-MANIFEST from
the merged tree. docs/INVENTORY.md keeps BOTH new reference rows.

execute-phase.md is 92782 LF bytes with both #2402's and this PR's extractions
applied — under the frozen ceiling (hard <93600, margin <=93400).

Refs #2296

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-07-20 14:59:30 -04:00
committed by GitHub
parent b6e6a22fce
commit 455ad49ae3
51 changed files with 811 additions and 84 deletions

View File

@@ -295,7 +295,7 @@ const { routePhasesCommand } = require('./lib/phases-command-router.cjs');
const { routeValidateCommand } = require('./lib/validate-command-router.cjs');
const { routeRoadmapCommand } = require('./lib/roadmap-command-router.cjs');
const { routeCapabilityCommand } = require('./lib/capability-command-router.cjs');
const { routeAgentCommand } = require('./lib/agent-command-router.cjs');
const { routeAgentCommand, AGENT_FAILURE_CLASSES } = require('./lib/agent-command-router.cjs');
const smartEntryMod = require('./lib/smart-entry.cjs');
const { routeCheckCommand } = require('./lib/check-command-router.cjs');
const { routeTaskCommand } = require('./lib/task-command-router.cjs');
@@ -609,7 +609,20 @@ function dispatchOverlayCapabilityCommand({ command, args, cwd, raw, error, load
let effortOverride;
let fastModeOverride;
let attempt;
let failureClass;
const positionals = [];
// #2296: the valid classes come from the classifier's own frozen enum, so
// this validator can never drift from what `agent classify-failure` emits.
const validFailureClasses = Object.values(AGENT_FAILURE_CLASSES);
const setFailureClass = (v) => {
if (!validFailureClasses.includes(v)) {
error(
`--failure-class must be one of: ${validFailureClasses.join(', ')}`,
ERROR_REASON.USAGE,
);
}
failureClass = v;
};
for (let i = 0; i < execArgs.length; i++) {
const a = execArgs[i];
if (a.startsWith('--effort=')) {
@@ -628,6 +641,10 @@ function dispatchOverlayCapabilityCommand({ command, args, cwd, raw, error, load
attempt = n;
continue;
}
if (a.startsWith('--failure-class=')) {
setFailureClass(a.slice('--failure-class='.length));
continue;
}
if (a === '--effort') {
const val = execArgs[i + 1];
if (val === undefined || val.startsWith('--')) error('Missing value for --effort', ERROR_REASON.USAGE);
@@ -651,6 +668,13 @@ function dispatchOverlayCapabilityCommand({ command, args, cwd, raw, error, load
i++;
continue;
}
if (a === '--failure-class') {
const val = execArgs[i + 1];
if (val === undefined || val.startsWith('--')) error('Missing value for --failure-class', ERROR_REASON.USAGE);
setFailureClass(val);
i++;
continue;
}
if (a === '--raw') continue;
if (a.startsWith('-')) error(`Unknown flag for resolve-execution: ${a}`, ERROR_REASON.USAGE);
positionals.push(a);
@@ -662,6 +686,7 @@ function dispatchOverlayCapabilityCommand({ command, args, cwd, raw, error, load
effortOverride,
fastModeOverride,
attempt,
failureClass,
});
}

View File

@@ -149,8 +149,8 @@
},
{
"topLevel": "dynamic_routing",
"source": "^dynamic_routing\\.(enabled|escalate_on_failure|max_escalations|tier_models\\.(light|standard|heavy))$",
"description": "dynamic_routing.<enabled|escalate_on_failure|max_escalations|tier_models.<light|standard|heavy>>"
"source": "^dynamic_routing\\.(enabled|escalate_on_failure|max_escalations|provider_escalation|tier_models\\.(light|standard|heavy))$",
"description": "dynamic_routing.<enabled|escalate_on_failure|max_escalations|provider_escalation|tier_models.<light|standard|heavy>>"
},
{
"topLevel": "model_overrides",

View File

@@ -0,0 +1,55 @@
**Step 7.1 detail — `class == "quota-exceeded"` recovery.**
Do not offer "retry now". Run the step-5 spot-check first; if SUMMARY.md is missing but
commits exist, route to safe-resume (`state.verify-against-disk`) instead of an immediate
redispatch.
**7.1a — provider escalation (#2296, opt-in).** A heavier tier on the same throttled
provider is still throttled, so when `dynamic_routing.provider_escalation` is configured
GSD swaps PROVIDER rather than waiting for a reset. `QUOTA_ATTEMPT` starts at 1 on the
first quota failure of this phase and increments on each subsequent one.
```bash
ESC_JSON=$(gsd_run query resolve-execution gsd-executor --attempt "${QUOTA_ATTEMPT:-1}" --failure-class quota-exceeded)
ESCALATED=$(echo "$ESC_JSON" | jq -r '.escalation.escalated')
EXHAUSTED=$(echo "$ESC_JSON" | jq -r '.escalation.exhausted')
ESC_FROM=$(echo "$ESC_JSON" | jq -r '.escalation.from')
ESC_TO=$(echo "$ESC_JSON" | jq -r '.escalation.to')
ESC_TRIED=$(echo "$ESC_JSON" | jq -r '.escalation.attempted | join(" -> ")')
```
- **`ESCALATED == "true"`** — log the switch, honor the provider's own backoff
(`sleep "$RETRY_AFTER"` when `RETRY_AFTER` is set), then re-dispatch the failed plan with
`executor_model` overridden to `$ESC_TO` and `QUOTA_ATTEMPT` incremented. Do not prompt —
this is the configured, opt-in path.
```text
⚡ Provider quota hit — escalating model: {ESC_FROM} → {ESC_TO}
Runtime sentinel: {SENTINEL}
{RETRY_HINT}
```
- **`EXHAUSTED == "true"`** — the ladder is spent. Fail loudly naming every model tried,
then fall through to the manual options below. Never silently retry the last one.
```text
⛔ Provider escalation exhausted — tried: {ESC_TRIED}
```
- **`ESCALATED == "false"` and not exhausted** — escalation is not configured for this
project; use the manual path below. This is the default.
**7.1b — manual recovery (default when escalation is not configured).**
```text
⚠ Plan {plan_id} terminated by provider quota / rate limit
Runtime sentinel: {SENTINEL}
{RETRY_HINT}
Partial commits on worktree branch: {N}
SUMMARY.md present: {yes|no}
1. Wait for quota reset, then resume (recommended)
2. Switch to a different runtime / model and resume
3. Abort phase and report partial state
```
Re-run `/gsd:execute-phase` after the quota resets for Option 1.

View File

@@ -1024,24 +1024,14 @@ increases monotonically across waves. `{status}` is `complete` (success),
if [ -n "$RETRY_AFTER" ]; then RETRY_HINT=" Provider hinted retry-after: ${RETRY_AFTER}s"; else RETRY_HINT=""; fi
```
One classifier branch handles sentinels across Claude/Copilot/Codex/Gemini. Reference: `docs/research/provider-rate-limit-signals.md`.
**Step 7.1 — `class == "quota-exceeded"`:**
Do not offer "retry now". Run step-5 spot-check first; if SUMMARY.md is missing but commits exist, route to safe-resume (`state.verify-against-disk`) instead of immediate redispatch.
```text
⚠ Plan {plan_id} terminated by provider quota / rate limit
Runtime sentinel: {SENTINEL}
{RETRY_HINT}
Partial commits on worktree branch: {N}
SUMMARY.md present: {yes|no}
1. Wait for quota reset, then resume (recommended)
2. Switch to a different runtime / model and resume
3. Abort phase and report partial state
```
Re-run `/gsd:execute-phase` after quota reset for Option 1.
**Step 7.1 — `class == "quota-exceeded"`:** follow the quota-recovery fragment below.
**Step 7.2 — `class == "classify-handoff-bug"`:**
If error contains `classifyHandoffIfNeeded is not defined`, treat as Claude runtime bug. Run the same step-5 spot-checks; PASS => treat as success, FAIL => fall through.
**Step 7.3 — `class == "unknown-failure"`:**
Report failed plan and ask Continue/Stop; continuing may cascade into dependent plan failures.
@~/.claude/gsd-core/references/execute-phase-quota-recovery.md
@~/.claude/gsd-core/references/execute-phase-between-wave-reset.md
8. **Execute checkpoint plans between waves** — see `<checkpoint_handling>`.