feat(#2296): config-gated provider escalation on quota-exceeded (#2458)

* test(#2296): failing-first coverage for provider escalation on quota-exceeded Covers the provider-escalation ladder layered onto EXEC.CLASSIFY: back-compat (no escalation block without --failure-class), cap boundaries at min(max_escalations, list length) at limit-1/limit/limit+1, opt-in gating, malformed/hostile provider_escalation config, the --failure-class CLI negative matrix, config-key registration, and a fast-check budget-limit property. Red until the resolver, CLI flag, and manifest key land. Refs #2296 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(#2296): config-gated provider escalation on quota-exceeded The dynamic_routing tier ladder escalates within one provider, which does not help when that provider is what ran out of quota. Add an opt-in provider ladder layered on the existing EXEC.CLASSIFY seam. - model-resolver: resolveProviderEscalation walks dynamic_routing.provider_escalation capped at min(max_escalations, list length), reporting from/to/attempted/exhausted. Invalid entries are dropped (ADR 227 shape validation). Stays a leaf module — the quota-class policy decision is the caller's, per the CONTEXT.md contract. - agent-command-router: export a frozen AGENT_FAILURE_CLASSES so the new CLI validator cannot drift from the classifier that produces the values. - resolve-execution: --failure-class flag; emits an escalation block ONLY when passed, so the existing JSON contract is byte-identical for every caller. - config-schema.manifest: register dynamic_routing.provider_escalation. - execute-phase step 7.1: auto-escalate, honor Retry-After, fail loudly naming every model tried once the ladder is spent. Refs #2296 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#2296): extract quota recovery to a reference fragment; regen goldens The step 7.1a addition pushed gsd-core/workflows/execute-phase.md from 93390 to 95111 LF bytes, past the frozen ADR-857 Phase 6 ceiling (hard <93600, margin <=93400) asserted by tests/fix-2285-claude-orchestration-wiring.test.cjs. The base sat 10 bytes under the margin, so no inline wording would have fit. That gate's own rationale is that optional-feature detail belongs in a fragment, not the host loop. Moved BOTH the new provider-escalation branch and the pre-existing manual recovery prompt into gsd-core/references/execute-phase-quota-recovery.md, leaving step 7.1 as a one-line pointer. execute-phase.md is now 92880 bytes — 510 SMALLER than base. Also regenerates the fixtures that legitimately moved because three shipped files changed (gsd-tools.cjs, config-schema.manifest.json, execute-phase.md): golden-install-parity + install-tree for all 16 runtimes, INVENTORY.md + INVENTORY-MANIFEST.json for the new reference, and the workflow size baseline. Refs #2296 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(#2351): make the C1 orphan-reaping test load-independent tests/run-with-timeout.test.cjs C1 asserted the child heartbeat file exists after a 1s group-kill window, but the child only wrote it on the first 100ms setInterval tick. Nothing synchronized the two: on a loaded container the group is SIGKILLed before that tick lands, the file never appears, and the assertion fails for a reason unrelated to reaping. Observed failing on both linux-node22 and linux-node24. The behavior actually under test is the FREEZE assertion (heartbeat stops advancing => descendant was reaped, not orphaned). That is unaffected by sampling once more at t=0. Child now writes its first heartbeat synchronously at startup before arming the interval, and the kill window widens 1s -> 3s to cover child boot under load. Both remove the timing dependency; neither weakens what the test proves. Refs #2296 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(#2296): backfill pr:2458 in .changeset/rapid-jays-bark.md * chore(#2296): regenerate fixtures after rebase onto #2402 The rebase conflicted on the generated golden-install-parity fixtures and workflow-size-baseline.json because #2402 (b6e6a22fc) regenerated the same artifacts. Conflict resolution picked a side to unblock the rebase; a true regeneration on the combined tree then produced further drift, confirming the resolved content was stale and would have dropped #2402's fixture changes. Regenerated goldens, install-tree, size baseline, and INVENTORY-MANIFEST from the merged tree. docs/INVENTORY.md keeps BOTH new reference rows. execute-phase.md is 92782 LF bytes with both #2402's and this PR's extractions applied — under the frozen ceiling (hard <93600, margin <=93400). Refs #2296 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 14:59:30 -04:00
parent b6e6a22fce
commit 455ad49ae3
51 changed files with 811 additions and 84 deletions
--- a/src/agent-command-router.cts
+++ b/src/agent-command-router.cts
@@ -36,6 +36,21 @@ interface RouteAgentCommandOptions {

 // ─── Constants ────────────────────────────────────────────────────────────────

+/**
+ * #2296 — The runtime enum of failure classes `classifyAgentFailure` can emit.
+ *
+ * `AgentFailureResult`'s class strings are TypeScript types, which erase at
+ * runtime. Any second surface that needs to validate a class (the
+ * `resolve-execution --failure-class` flag) would otherwise have to re-declare
+ * the literals, giving two lists that can silently diverge. This frozen enum is
+ * the single runtime source both surfaces consume.
+ */
+const AGENT_FAILURE_CLASSES = Object.freeze({
+  QUOTA_EXCEEDED: 'quota-exceeded',
+  CLASSIFY_HANDOFF_BUG: 'classify-handoff-bug',
+  UNKNOWN_FAILURE: 'unknown-failure',
+} as const);
+
 const QUOTA_SENTINELS: string[] = [
  '429',
  'usage_limit_reached',
@@ -65,26 +80,26 @@ function classifyAgentFailure(body: unknown): AgentFailureResult {
  // eslint-disable-next-line @typescript-eslint/no-base-to-string
  const normalized = String(body ?? '').toLowerCase();
  if (normalized.trim() === '') {
-    return { class: 'unknown-failure' };
+    return { class: AGENT_FAILURE_CLASSES.UNKNOWN_FAILURE };
  }

  for (const sentinel of QUOTA_SENTINELS) {
    if (normalized.includes(sentinel)) {
      const retryAfterSeconds = parseRetryAfter(body);
      return retryAfterSeconds === undefined
-        ? { class: 'quota-exceeded', sentinel }
-        : { class: 'quota-exceeded', sentinel, retryAfterSeconds };
+        ? { class: AGENT_FAILURE_CLASSES.QUOTA_EXCEEDED, sentinel }
+        : { class: AGENT_FAILURE_CLASSES.QUOTA_EXCEEDED, sentinel, retryAfterSeconds };
    }
  }

  if (normalized.includes(CLASSIFY_HANDOFF_SENTINEL)) {
    return {
-      class: 'classify-handoff-bug',
+      class: AGENT_FAILURE_CLASSES.CLASSIFY_HANDOFF_BUG,
      sentinel: CLASSIFY_HANDOFF_SENTINEL,
    };
  }

-  return { class: 'unknown-failure' };
+  return { class: AGENT_FAILURE_CLASSES.UNKNOWN_FAILURE };
 }

 function routeAgentCommand({ args, raw }: RouteAgentCommandOptions): void {
@@ -98,6 +113,7 @@ function routeAgentCommand({ args, raw }: RouteAgentCommandOptions): void {
 }

 export = {
+  AGENT_FAILURE_CLASSES,
  classifyAgentFailure,
  routeAgentCommand,
 };
--- a/src/commands.cts
+++ b/src/commands.cts
@@ -30,7 +30,10 @@ import roadmapParserMod = require('./roadmap-parser.cjs');
 const { extractCurrentMilestone, stripShippedMilestones: _stripShippedMilestones, getMilestoneInfo, getMilestonePhaseFilter, getRoadmapPhaseInternal } = roadmapParserMod;
 // eslint-disable-next-line @typescript-eslint/no-require-imports
 import modelResolverMod = require('./model-resolver.cjs');
-const { resolveModelInternal, resolveModelForTier, resolveEffortInternal, resolveFastModeInternal, resolveEffortForTier, resolveGranularityInternal, assertValidGranularityOverride } = modelResolverMod;
+const { resolveModelInternal, resolveModelForTier, resolveProviderEscalation, resolveEffortInternal, resolveFastModeInternal, resolveEffortForTier, resolveGranularityInternal, assertValidGranularityOverride } = modelResolverMod;
+// eslint-disable-next-line @typescript-eslint/no-require-imports
+import agentCommandRouterMod = require('./agent-command-router.cjs');
+const { AGENT_FAILURE_CLASSES } = agentCommandRouterMod;
 import { renderEffortForRuntime, RUNTIMES_WITH_FAST_MODE } from './model-catalog.cjs';
 // eslint-disable-next-line @typescript-eslint/no-require-imports
 import planningWorkspace = require('./planning-workspace.cjs');
@@ -482,9 +485,10 @@ function cmdResolveGranularity(cwd: string, phaseType: string | undefined, raw:
 *   { model, profile, effort, effort_rendered, effort_param, effort_propagation,
 *     fast_mode, fast_mode_supported, [unknown_agent] }
 *
- * Flags: --effort <level>, --fast-mode <true|false>, --attempt <n>
+ * Flags: --effort <level>, --fast-mode <true|false>, --attempt <n>,
+ *        --failure-class <class> (#2296)
 */
-function cmdResolveExecution(cwd: string, agentType: string | undefined, raw: boolean, opts?: { effortOverride?: string; fastModeOverride?: boolean; attempt?: number }): void {
+function cmdResolveExecution(cwd: string, agentType: string | undefined, raw: boolean, opts?: { effortOverride?: string; fastModeOverride?: boolean; attempt?: number; failureClass?: string }): void {
  if (!agentType) {
    error('agent-type required');
  }
@@ -499,10 +503,23 @@ function cmdResolveExecution(cwd: string, agentType: string | undefined, raw: bo
  // including dynamic_routing-enabled users who don't pass --attempt), and only an
  // explicit attempt routes through the tier ladder. resolveModelForTier itself
  // still falls back to resolveModelInternal when dynamic_routing is off.
-  const model = (opts.attempt !== undefined && opts.attempt !== null)
+  let model = (opts.attempt !== undefined && opts.attempt !== null)
    ? resolveModelForTier(cwd, agentType!, opts.attempt)
    : resolveModelInternal(cwd, agentType!);

+  // #2296: when the caller reports WHY the previous attempt failed, consult the
+  // provider-escalation ladder. Only a quota/rate-limit class warrants it — a
+  // heavier tier on the same throttled provider is still throttled, so this
+  // ladder swaps providers instead. Gated on an explicit --failure-class so the
+  // JSON contract is byte-identical for every existing caller.
+  let escalation: Record<string, unknown> | undefined;
+  if (opts.failureClass !== undefined) {
+    const applicable = opts.failureClass === AGENT_FAILURE_CLASSES.QUOTA_EXCEEDED;
+    const resolved = resolveProviderEscalation(cwd, agentType!, opts.attempt, applicable);
+    if (resolved.escalated) model = resolved.to;
+    escalation = { class: opts.failureClass, ...resolved };
+  }
+
  const effortOpts: Record<string, unknown> = {};
  if (typeof opts.effortOverride === 'string') effortOpts['override'] = opts.effortOverride;

@@ -532,6 +549,7 @@ function cmdResolveExecution(cwd: string, agentType: string | undefined, raw: bo
    fast_mode_supported: fastModeSupported,
  };
  if (!agentModels) result['unknown_agent'] = true;
+  if (escalation) result['escalation'] = escalation;
  output(result, raw, effort);
 }

--- a/src/model-resolver.cts
+++ b/src/model-resolver.cts
@@ -505,6 +505,101 @@ function resolveModelForTier(cwd: string, agentType: string, attempt?: number):
  return alias;
 }

+/**
+ * #2296 — Outcome of consulting the provider-escalation ladder for one attempt.
+ *
+ * `from`/`to` describe the PROVIDER ladder only; they are equal whenever the
+ * ladder was not consulted or had nothing to offer.
+ */
+interface ProviderEscalationResult {
+  from: string;
+  to: string;
+  escalated: boolean;
+  exhausted: boolean;
+  attempted: string[];
+  index: number;
+}
+
+/**
+ * Keep only usable model ids: non-empty strings. A malformed config can put
+ * anything in here (nulls, numbers, blank strings), and a blank model id would
+ * resolve to an unusable agent invocation rather than failing visibly. Invalid
+ * entries are dropped and the surviving order is preserved, so the ladder stays
+ * predictable (ADR 227 — validate shape, not just type).
+ */
+function sanitizeProviderEscalation(raw: unknown): string[] {
+  if (!Array.isArray(raw)) return [];
+  return raw.filter((entry): entry is string => typeof entry === 'string' && entry.trim().length > 0);
+}
+
+/**
+ * #2296 — Resolve the model for one attempt of the PROVIDER escalation ladder.
+ *
+ * The tier ladder (`resolveModelForTier`) escalates within one provider's
+ * `tier_models`, which does not help when that provider is the thing that is
+ * throttled. This walks `dynamic_routing.provider_escalation` instead: an
+ * ordered list of alternative model ids, capped by
+ * `min(max_escalations, list length)`.
+ *
+ * `applicable` is the caller's policy decision (only a quota-exceeded
+ * classification should consult this ladder). It is a parameter rather than a
+ * class check here so this module keeps depending only on leaf modules, per
+ * CONTEXT.md's Model Resolution module contract.
+ *
+ * Attempt 0 — and every non-applicable call — stays on the source model.
+ * `exhausted` reports that the ladder is spent so the caller can fail loudly
+ * naming every model it tried.
+ */
+function resolveProviderEscalation(
+  cwd: string,
+  agentType: string,
+  attempt: number | undefined,
+  applicable: boolean,
+): ProviderEscalationResult {
+  // The model that would be used with no provider escalation at all.
+  const from = resolveModelForTier(cwd, agentType, 0);
+  const stay = (exhausted = false): ProviderEscalationResult => ({
+    from,
+    to: from,
+    escalated: false,
+    exhausted,
+    attempted: [from],
+    index: 0,
+  });
+
+  if (!applicable) return stay();
+
+  const config = loadConfig(cwd);
+  const dr = config['dynamic_routing'] as Record<string, unknown> | null | undefined;
+  if (!dr || typeof dr !== 'object' || dr['enabled'] !== true) return stay();
+  if (dr['escalate_on_failure'] === false) return stay();
+
+  const list = sanitizeProviderEscalation(dr['provider_escalation']);
+  if (list.length === 0) return stay();
+
+  // Same default and same validity rule as the tier ladder above — a negative or
+  // non-integer max_escalations is invalid config, not a request for zero.
+  const maxEscalations = Number.isInteger(dr['max_escalations']) && (dr['max_escalations'] as number) >= 0
+    ? (dr['max_escalations'] as number)
+    : 1;
+  const cap = Math.min(maxEscalations, list.length);
+  // An explicit cap of 0 means the ladder exists but is spent before it starts.
+  if (cap === 0) return stay(true);
+
+  const attemptN = Number.isInteger(attempt) && (attempt as number) > 0 ? (attempt as number) : 0;
+  if (attemptN === 0) return stay();
+
+  const index = Math.min(attemptN, cap);
+  return {
+    from,
+    to: list[index - 1],
+    escalated: true,
+    exhausted: attemptN > cap,
+    attempted: [from, ...list.slice(0, index)],
+    index,
+  };
+}
+
 // ─── #443 — Unified effort + fast_mode resolvers ─────────────────────────────

 const VALID_EFFORTS = ['minimal', 'low', 'medium', 'high', 'xhigh', 'max'];
@@ -691,6 +786,7 @@ export = {
  resolveGranularityInternal,
  assertValidGranularityOverride,
  resolveModelForTier,
+  resolveProviderEscalation,
  VALID_EFFORTS,
  EFFORT_SET,
  nextEffort,