* feat(execute-phase): classify quota/rate-limit failures across runtimes (#3095) Dispatched executor subagents that die from provider quota or rate-limit errors currently look identical to a crashed agent to the orchestrator — so step 7's recovery prompt offers "retry now" when the right action is "wait for reset and resume". This adds a runtime-agnostic classifier and wires execute-phase step 7 to it. - `agent.classify-failure` SDK query returns `{class: 'quota-exceeded' | 'classify-handoff-bug' | 'unknown-failure', sentinel?, retryAfterSeconds?}`. Sentinels cover Claude Code (`usage limit`, `429`), Copilot CLI (`rate_limit`, `user_weekly_rate_limited`), Codex (`usage_limit_reached`, `too many requests`), and Gemini (`RESOURCE_EXHAUSTED`, `exceeded your`). - `execute-phase.md` step 7 now branches on the class. Quota-exceeded presents a wait-for-reset prompt and points at the safe-resume gate landing in #3212 instead of re-dispatching a fresh executor. - `docs/research/provider-rate-limit-signals.md` records the proactive (header / SDK event) signals each provider exposes and the upstream Claude Code / Copilot / Codex issues blocking hook-side detection — the forward path once host runtimes surface them. Resume-from-partial-worktree and context-load metrics from the original report are deliberately out of scope; they overlap #3212's `state.verify-against-disk` work already in flight. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(execute): render quota retry hint and refresh alias artifacts * fix(workflow): restore slash namespace and execute-phase size budget --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
5
.changeset/3095-quota-failure-classification.md
Normal file
5
.changeset/3095-quota-failure-classification.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
type: Added
|
||||
pr: 3095
|
||||
---
|
||||
**Distinct quota / rate-limit failure class for dispatched executor subagents** — `execute-phase` step 7 now routes provider quota and rate-limit terminations through a separate recovery branch instead of the generic "real failure" prompt. A new SDK query `agent.classify-failure` (`gsd-sdk query agent.classify-failure -- "<body>"`) classifies the agent return body as `quota-exceeded`, `classify-handoff-bug`, or `unknown-failure`, extracting `retryAfterSeconds` when the provider echoes one. Sentinels cover every runtime GSD dispatches into: Claude Code (`usage limit`, `429`), Copilot CLI (`rate_limit`, `user_weekly_rate_limited`), Codex (`usage_limit_reached`, `too many requests`), and Gemini (`RESOURCE_EXHAUSTED`, `exceeded your`). The recovery prompt for `quota-exceeded` offers wait-for-reset and resume as the first option rather than retry-now, which the runtime would reject again until the quota window resets. The resume path itself relies on the safe-resume gate landing in #3212. (#3095)
|
||||
61
docs/research/provider-rate-limit-signals.md
Normal file
61
docs/research/provider-rate-limit-signals.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# Provider rate-limit signals across executor runtimes
|
||||
|
||||
**Status:** research note — informs #3095 reactive classification and points at the proactive path forward.
|
||||
|
||||
GSD dispatches executor subagents into one of four host runtimes today: Claude Code, GitHub Copilot CLI, OpenAI Codex CLI, and Google Gemini CLI. Each provider exposes rate-limit information at three different layers — pre-warning, post-mortem error body, and underlying HTTP transport — but the *host runtime* (the CLI that wraps the provider for us) gates how much of that surfaces to GSD's orchestrator.
|
||||
|
||||
The reactive classifier shipped with #3095 (`agent.classify-failure`) parses post-mortem error bodies. This note records the proactive signals that exist at the provider layer but are not yet surfaced to orchestrators by the host runtimes — i.e. the forward path once host runtimes expose them to hooks.
|
||||
|
||||
## Anthropic / Claude Code
|
||||
|
||||
| Layer | Signal | Available to GSD today? |
|
||||
|--------------|---------------------------------------------------------------------------------------------------|----------------------------------------------------------|
|
||||
| HTTP headers | `anthropic-ratelimit-requests-remaining`, `anthropic-ratelimit-tokens-remaining`, `retry-after` on 429 | No — Claude Code does not forward these to hooks ([#33820](https://github.com/anthropics/claude-code/issues/33820), [#22407](https://github.com/anthropics/claude-code/issues/22407)) |
|
||||
| Agent SDK | `RateLimitEvent` with `status` transitions: `allowed` → `allowed_warning` → `rejected` | Only when callers use the Agent SDK directly |
|
||||
| Plan usage | Max plan session / weekly usage | No — Claude Code SDK does not expose Max-plan limits ([#32796](https://github.com/anthropics/claude-code/issues/32796)) |
|
||||
| Error body | `"You've hit your org's monthly usage limit"`, `429`, `rate_limit_error` | Yes — parsed by `agent.classify-failure` |
|
||||
|
||||
## GitHub Copilot CLI
|
||||
|
||||
| Layer | Signal | Available to GSD today? |
|
||||
|--------------|---------------------------------------------------------------------------------------------------------|-------------------------|
|
||||
| Pre-warning | CLI displays a warning when approaching a limit (per [GitHub Copilot usage-limits docs](https://docs.github.com/en/copilot/concepts/usage-limits)) | Visible in subprocess stdout, not as a structured signal |
|
||||
| Error body | `"hit a rate limit"`, `"exceeded your Copilot token usage"`, `rate_limited`, `user_weekly_rate_limited` | Yes — parsed by `agent.classify-failure` |
|
||||
|
||||
## OpenAI Codex CLI
|
||||
|
||||
| Layer | Signal | Available to GSD today? |
|
||||
|--------------|-----------------------------------------------------------------------|-------------------------|
|
||||
| HTTP headers | `x-ratelimit-remaining-requests`, `x-ratelimit-remaining-tokens` | No — Codex CLI does not forward these to hooks |
|
||||
| Error body | `429`, `usage_limit_reached`, `"exceeded your current quota"`, `Too Many Requests` ([openai/codex#9135](https://github.com/openai/codex/issues/9135)) | Yes — parsed by `agent.classify-failure` |
|
||||
|
||||
## Google Gemini CLI
|
||||
|
||||
| Layer | Signal | Available to GSD today? |
|
||||
|--------------|---------------------------------------------------------------------------|-------------------------|
|
||||
| Error body | `RESOURCE_EXHAUSTED`, `"exceeded your current quota"`, `429` | Yes — parsed by `agent.classify-failure` |
|
||||
| Pre-warning | None documented at the CLI layer | n/a |
|
||||
|
||||
## Forward path
|
||||
|
||||
When the host runtimes (Claude Code, Copilot CLI, Codex CLI) start exposing rate-limit headers and SDK events to hooks — which is the active ask in [anthropics/claude-code#33820](https://github.com/anthropics/claude-code/issues/33820), [#22407](https://github.com/anthropics/claude-code/issues/22407), and [#32796](https://github.com/anthropics/claude-code/issues/32796) — GSD should:
|
||||
|
||||
1. Read `requests-remaining` / `tokens-remaining` in a PreToolUse / SessionStart hook and surface a soft warning to the user before dispatching a new wave when the values fall below a configurable threshold.
|
||||
2. Treat `RateLimitEvent.status == "allowed_warning"` (Anthropic Agent SDK) as a checkpoint signal in long-running executors — emit a partial SUMMARY and let the orchestrator pick up after reset.
|
||||
3. Combine the proactive headers with `executor.stall_threshold_minutes` (#3329) so the orchestrator does not wait the full stall interval when the runtime has already signalled `rejected`.
|
||||
|
||||
Until then, `agent.classify-failure` is the actionable boundary: post-mortem error-body sentinels, with a quota-distinct recovery prompt in `execute-phase` step 7.
|
||||
|
||||
## Sources
|
||||
|
||||
- [Anthropic Claude API — Rate limits](https://platform.claude.com/docs/en/api/rate-limits)
|
||||
- [anthropics/claude-code#33820 — Expose API rate-limit response headers to hooks and status line scripts](https://github.com/anthropics/claude-code/issues/33820)
|
||||
- [anthropics/claude-code#22407 — Feature Request: Include rate limit info in statusline data](https://github.com/anthropics/claude-code/issues/22407)
|
||||
- [anthropics/claude-code#32796 — Expose Max plan usage limits via Claude Code API/SDK](https://github.com/anthropics/claude-code/issues/32796)
|
||||
- [anthropics/anthropic-sdk-typescript#450 — API calls missing Rate Limit Response Headers](https://github.com/anthropics/anthropic-sdk-typescript/issues/450)
|
||||
- [GitHub Copilot — Rate limits documentation](https://docs.github.com/en/copilot/concepts/usage-limits)
|
||||
- [github/copilot-cli#2742 — Persistent Global 429 Rate Limit on Paid Pro+ Account](https://github.com/github/copilot-cli/issues/2742)
|
||||
- [OpenAI — Error codes](https://developers.openai.com/api/docs/guides/error-codes)
|
||||
- [openai/codex#9135 — improper 429 error near the end of a 5-hour window](https://github.com/openai/codex/issues/9135)
|
||||
- [Gemini API — Rate limits](https://ai.google.dev/gemini-api/docs/rate-limits)
|
||||
- [google-gemini/gemini-cli#6986 — stuck in resource exhausted loop](https://github.com/google-gemini/gemini-cli/issues/6986)
|
||||
@@ -553,6 +553,13 @@ const ROADMAP_COMMAND_ALIASES = [
|
||||
];
|
||||
|
||||
const NON_FAMILY_COMMAND_ALIASES = [
|
||||
{
|
||||
"canonical": "agent.classify-failure",
|
||||
"aliases": [
|
||||
"agent classify-failure"
|
||||
],
|
||||
"mutation": false
|
||||
},
|
||||
{
|
||||
"canonical": "check-commit",
|
||||
"aliases": [],
|
||||
@@ -828,4 +835,4 @@ module.exports = {
|
||||
PHASES_SUBCOMMANDS,
|
||||
VALIDATE_SUBCOMMANDS,
|
||||
ROADMAP_SUBCOMMANDS,
|
||||
};
|
||||
};
|
||||
|
||||
@@ -1005,46 +1005,46 @@ increases monotonically across waves. `{status}` is `complete` (success),
|
||||
---
|
||||
```
|
||||
|
||||
- Bad: "Wave 2 complete. Proceeding to Wave 3."
|
||||
- Good: "Terrain system complete — 3 biome types, height-based texturing, physics collision meshes. Vehicle physics (Wave 3) can now reference ground surfaces."
|
||||
|
||||
7. **Handle failures:**
|
||||
|
||||
**Known Claude Code bug (classifyHandoffIfNeeded):** If an agent reports "failed" with error containing `classifyHandoffIfNeeded is not defined`, this is a Claude Code runtime bug — not a GSD or agent issue. The error fires in the completion handler AFTER all tool calls finish. In this case: run the same spot-checks as step 5 (SUMMARY.md exists, git commits present, no Self-Check: FAILED). If spot-checks PASS → treat as **successful**. If spot-checks FAIL → treat as real failure below.
|
||||
|
||||
For real failures: report which plan failed → ask "Continue?" or "Stop?" → if continue, dependent plans may also fail. If stop, partial completion report.
|
||||
**Step 7.0 — classify before branching (#3095):**
|
||||
```bash
|
||||
CLASS_JSON=$(gsd-sdk query agent.classify-failure -- "$AGENT_RETURN_BODY")
|
||||
CLASS=$(echo "$CLASS_JSON" | jq -r '.class')
|
||||
SENTINEL=$(echo "$CLASS_JSON" | jq -r '.sentinel // empty')
|
||||
RETRY_AFTER=$(echo "$CLASS_JSON" | jq -r '.retryAfterSeconds // empty')
|
||||
if [ -n "$RETRY_AFTER" ]; then RETRY_HINT=" Provider hinted retry-after: ${RETRY_AFTER}s"; else RETRY_HINT=""; fi
|
||||
```
|
||||
One classifier branch handles sentinels across Claude/Copilot/Codex/Gemini. Reference: `docs/research/provider-rate-limit-signals.md`.
|
||||
**Step 7.1 — `class == "quota-exceeded"`:**
|
||||
Do not offer "retry now". Run step-5 spot-check first; if SUMMARY.md is missing but commits exist, route to safe-resume (`state.verify-against-disk`) instead of immediate redispatch.
|
||||
```text
|
||||
⚠ Plan {plan_id} terminated by provider quota / rate limit
|
||||
Runtime sentinel: {SENTINEL}
|
||||
{RETRY_HINT}
|
||||
Partial commits on worktree branch: {N}
|
||||
SUMMARY.md present: {yes|no}
|
||||
1. Wait for quota reset, then resume (recommended)
|
||||
2. Switch to a different runtime / model and resume
|
||||
3. Abort phase and report partial state
|
||||
```
|
||||
Re-run `/gsd:execute-phase` after quota reset for Option 1.
|
||||
**Step 7.2 — `class == "classify-handoff-bug"`:**
|
||||
If error contains `classifyHandoffIfNeeded is not defined`, treat as Claude runtime bug. Run the same step-5 spot-checks; PASS => treat as success, FAIL => fall through.
|
||||
**Step 7.3 — `class == "unknown-failure"`:**
|
||||
Report failed plan and ask Continue/Stop; continuing may cascade into dependent plan failures.
|
||||
|
||||
7b. **Pre-wave dependency check (waves 2+ only):**
|
||||
|
||||
Before spawning wave N+1, for each plan in the upcoming wave:
|
||||
```bash
|
||||
gsd-sdk query verify.key-links {phase_dir}/{plan}-PLAN.md
|
||||
```
|
||||
|
||||
If any key-link from a PRIOR wave's artifact fails verification:
|
||||
|
||||
## Cross-Plan Wiring Gap
|
||||
|
||||
| Plan | Link | From | Expected Pattern | Status |
|
||||
|------|------|------|-----------------|--------|
|
||||
| {plan} | {via} | {from} | {pattern} | NOT FOUND |
|
||||
|
||||
Wave {N} artifacts may not be properly wired. Options:
|
||||
1. Investigate and fix before continuing
|
||||
2. Continue (may cause cascading failures in wave {N+1})
|
||||
|
||||
Key-links referencing files in the CURRENT (upcoming) wave are skipped.
|
||||
|
||||
Before wave N+1, run `gsd-sdk query verify.key-links {phase_dir}/{plan}-PLAN.md` for each upcoming plan.
|
||||
If any PRIOR-wave artifact link fails, present:
|
||||
- `## Cross-Plan Wiring Gap` with plan/link/from/pattern rows
|
||||
- Options: investigate+fix before continue, or continue with cascade risk
|
||||
Skip key-links that reference files in the CURRENT (upcoming) wave.
|
||||
8. **Execute checkpoint plans between waves** — see `<checkpoint_handling>`.
|
||||
|
||||
9. **Proceed to next wave.**
|
||||
</step>
|
||||
|
||||
<step name="checkpoint_handling">
|
||||
Plans with `autonomous: false` require user interaction.
|
||||
|
||||
**Auto-mode checkpoint handling:**
|
||||
|
||||
Read auto-advance config (chain flag OR user preference — same boolean as `check.auto-mode`):
|
||||
```bash
|
||||
AUTO_MODE=$(gsd-sdk query check auto-mode --pick active 2>/dev/null || echo "false")
|
||||
@@ -1785,6 +1785,7 @@ For 1M+ context models, consider:
|
||||
</context_efficiency>
|
||||
|
||||
<failure_handling>
|
||||
- **Quota / rate-limit (any runtime — #3095):** Agent return body contains a sentinel like `usage limit`, `rate limit`, `429`, `too many requests`, `RESOURCE_EXHAUSTED`, `usage_limit_reached`. Route via `gsd-sdk query agent.classify-failure` → `class: "quota-exceeded"`. Do not offer retry-now; the right action is wait-for-reset and resume.
|
||||
- **classifyHandoffIfNeeded false failure:** Agent reports "failed" but error is `classifyHandoffIfNeeded is not defined` → Claude Code bug, not GSD. Spot-check (SUMMARY exists, commits present) → if pass, treat as success
|
||||
- **Agent fails mid-plan:** Missing SUMMARY.md → report, ask user how to proceed
|
||||
- **Dependency chain breaks:** Wave 1 fails → Wave 2 dependents likely fail → user chooses attempt or skip
|
||||
|
||||
157
sdk/src/query/agent-failure-classifier.test.ts
Normal file
157
sdk/src/query/agent-failure-classifier.test.ts
Normal file
@@ -0,0 +1,157 @@
|
||||
/**
|
||||
* Unit tests for `classifyAgentFailure` (#3095 — execute-phase quota-kill classification).
|
||||
*
|
||||
* The orchestrator dispatches executor subagents and receives a free-text return body
|
||||
* when an agent finishes or fails. Today the body is parsed only for the
|
||||
* `classifyHandoffIfNeeded is not defined` Claude Code runtime bug (treated as
|
||||
* possible-success), and everything else falls through to a generic
|
||||
* "real failure" branch. Quota / rate-limit terminations look identical to a
|
||||
* crashed agent to the orchestrator, so the recovery prompt is wrong
|
||||
* (offers "retry now" when the right action is "wait and resume").
|
||||
*
|
||||
* This classifier returns a structured class plus the matched sentinel so the
|
||||
* workflow can route to a quota-distinct recovery prompt.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { classifyAgentFailure } from './agent-failure-classifier.js';
|
||||
|
||||
describe('classifyAgentFailure', () => {
|
||||
describe('quota-exceeded class', () => {
|
||||
it('matches the exact org-monthly-limit message from the #3095 report', () => {
|
||||
const body = "You've hit your org's monthly usage limit";
|
||||
const result = classifyAgentFailure(body);
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
expect(result.sentinel).toBe('usage limit');
|
||||
});
|
||||
|
||||
it('matches "rate limit" sentinel', () => {
|
||||
const result = classifyAgentFailure('Anthropic API rate limit hit; retry later.');
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
expect(result.sentinel).toBe('rate limit');
|
||||
});
|
||||
|
||||
it('matches "quota" sentinel', () => {
|
||||
const result = classifyAgentFailure('Your monthly quota has been exhausted.');
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
expect(result.sentinel).toBe('quota');
|
||||
});
|
||||
|
||||
it('matches HTTP 429 sentinel', () => {
|
||||
const result = classifyAgentFailure('Request failed: HTTP 429 Too Many Requests');
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
// 429 wins over the secondary "too many requests" tag (more specific).
|
||||
expect(result.sentinel).toBe('429');
|
||||
});
|
||||
|
||||
it('matches "too many requests" when 429 not present (Codex/OpenAI variant)', () => {
|
||||
const result = classifyAgentFailure('Too Many Requests — please retry later');
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
expect(result.sentinel).toBe('too many requests');
|
||||
});
|
||||
|
||||
it('matches "usage_limit_reached" (Codex CLI sentinel)', () => {
|
||||
const result = classifyAgentFailure('agent failed: usage_limit_reached');
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
expect(result.sentinel).toBe('usage_limit_reached');
|
||||
});
|
||||
|
||||
it('matches "RESOURCE_EXHAUSTED" (Gemini CLI sentinel)', () => {
|
||||
const result = classifyAgentFailure(
|
||||
'Error: RESOURCE_EXHAUSTED — You exceeded your current quota',
|
||||
);
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
// First-match precedence per SENTINEL_ORDER — quota wins here.
|
||||
expect(['quota', 'resource_exhausted']).toContain(result.sentinel);
|
||||
});
|
||||
|
||||
it('matches "user_weekly_rate_limited" (Copilot CLI sentinel)', () => {
|
||||
const result = classifyAgentFailure('Server Error: user_weekly_rate_limited');
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
// The high-priority "rate limit" tag also appears as a substring here;
|
||||
// either is acceptable as long as classification is quota-exceeded.
|
||||
expect(result.sentinel).toBeDefined();
|
||||
});
|
||||
|
||||
it('matches "exceeded your" generic quota phrasing (OpenAI / Gemini)', () => {
|
||||
const result = classifyAgentFailure(
|
||||
"You exceeded your current quota, please check your plan and billing details.",
|
||||
);
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
});
|
||||
|
||||
it('matches case-insensitively', () => {
|
||||
const result = classifyAgentFailure('USAGE LIMIT REACHED');
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
expect(result.sentinel).toBe('usage limit');
|
||||
});
|
||||
|
||||
it('extracts retry-after seconds when present in the body', () => {
|
||||
const body = 'rate_limit_error: retry-after: 3600';
|
||||
const result = classifyAgentFailure(body);
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
expect(result.retryAfterSeconds).toBe(3600);
|
||||
});
|
||||
|
||||
it('extracts retry-after seconds from header-style header value', () => {
|
||||
// Some runtimes echo the raw Retry-After header value.
|
||||
const body = 'Failed: 429 Too Many Requests (Retry-After: 60)';
|
||||
const result = classifyAgentFailure(body);
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
expect(result.retryAfterSeconds).toBe(60);
|
||||
});
|
||||
|
||||
it('does not parse retry-after when token is embedded in another word', () => {
|
||||
const body = 'noretry-after: 3600 quota exceeded';
|
||||
const result = classifyAgentFailure(body);
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
expect(result.retryAfterSeconds).toBeUndefined();
|
||||
});
|
||||
|
||||
it('returns undefined retryAfterSeconds when no hint present', () => {
|
||||
const result = classifyAgentFailure("You've hit your org's monthly usage limit");
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
expect(result.retryAfterSeconds).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('classify-handoff-bug class', () => {
|
||||
it('preserves the existing classifyHandoffIfNeeded pass-through', () => {
|
||||
// Keep the existing Claude Code runtime bug path callable through the
|
||||
// same classifier so step 7 has a single dispatch point.
|
||||
const body = 'ReferenceError: classifyHandoffIfNeeded is not defined';
|
||||
const result = classifyAgentFailure(body);
|
||||
expect(result.class).toBe('classify-handoff-bug');
|
||||
expect(result.sentinel).toBe('classifyhandoffifneeded is not defined');
|
||||
});
|
||||
});
|
||||
|
||||
describe('unknown-failure class (fallback)', () => {
|
||||
it('returns unknown-failure for a generic crash', () => {
|
||||
const result = classifyAgentFailure('Error: something blew up');
|
||||
expect(result.class).toBe('unknown-failure');
|
||||
expect(result.sentinel).toBeUndefined();
|
||||
});
|
||||
|
||||
it('returns unknown-failure for empty input', () => {
|
||||
const result = classifyAgentFailure('');
|
||||
expect(result.class).toBe('unknown-failure');
|
||||
});
|
||||
|
||||
it('returns unknown-failure for whitespace-only input', () => {
|
||||
const result = classifyAgentFailure(' \n\t ');
|
||||
expect(result.class).toBe('unknown-failure');
|
||||
});
|
||||
});
|
||||
|
||||
describe('precedence', () => {
|
||||
it('quota sentinel wins over classify-handoff sentinel when both present', () => {
|
||||
// The runtime bug is a post-completion handler crash; if the underlying
|
||||
// cause was quota, that is the actionable signal — surface it.
|
||||
const body =
|
||||
"You've hit your org's monthly usage limit\nReferenceError: classifyHandoffIfNeeded is not defined";
|
||||
const result = classifyAgentFailure(body);
|
||||
expect(result.class).toBe('quota-exceeded');
|
||||
});
|
||||
});
|
||||
});
|
||||
105
sdk/src/query/agent-failure-classifier.ts
Normal file
105
sdk/src/query/agent-failure-classifier.ts
Normal file
@@ -0,0 +1,105 @@
|
||||
/**
|
||||
* Classify the free-text return body from a dispatched executor subagent into
|
||||
* an actionable failure class for the orchestrator's recovery router.
|
||||
*
|
||||
* Context (#3095): the orchestrator currently treats every non-success agent
|
||||
* return as a generic "real failure" except for the Claude Code
|
||||
* `classifyHandoffIfNeeded is not defined` runtime bug. Quota / rate-limit
|
||||
* terminations look identical to a crashed agent — but the right user
|
||||
* response is "wait for reset and resume", not "retry now or abort".
|
||||
*
|
||||
* Sentinel coverage spans the runtimes GSD supports as executor targets:
|
||||
* - Anthropic / Claude Code — "usage limit", "rate limit", "quota", "429", "retry-after"
|
||||
* - GitHub Copilot CLI — "rate limit", "rate_limited", "user_weekly_rate_limited"
|
||||
* - OpenAI Codex CLI — "429", "usage_limit_reached", "too many requests"
|
||||
* - Google Gemini CLI — "RESOURCE_EXHAUSTED", "exceeded your", "quota"
|
||||
*
|
||||
* See docs/research/provider-rate-limit-signals.md for the proactive (header
|
||||
* / SDK event) signals the orchestrator could use once the host runtime
|
||||
* (Claude Code, Copilot, Codex) exposes them to hooks.
|
||||
*/
|
||||
|
||||
export type AgentFailureClass =
|
||||
| 'quota-exceeded'
|
||||
| 'classify-handoff-bug'
|
||||
| 'unknown-failure';
|
||||
|
||||
export interface AgentFailureClassification {
|
||||
class: AgentFailureClass;
|
||||
/** Lower-cased substring that matched, if any. Useful for log lines. */
|
||||
sentinel?: string;
|
||||
/** Seconds the runtime asked us to wait, parsed from "retry-after: N". */
|
||||
retryAfterSeconds?: number;
|
||||
}
|
||||
|
||||
// Order matters: the first match wins, so list the most specific / most
|
||||
// actionable sentinels first. "429" beats "too many requests"; "quota" beats
|
||||
// "RESOURCE_EXHAUSTED" because "quota" is the universal token. All matches
|
||||
// are case-insensitive (sentinel value below is the lower-cased form).
|
||||
const QUOTA_SENTINELS: ReadonlyArray<string> = [
|
||||
'429',
|
||||
'usage_limit_reached',
|
||||
'usage limit',
|
||||
'rate limit',
|
||||
'rate-limited',
|
||||
// `rate_limit` (stem) covers `rate_limited`, `rate_limit_error`,
|
||||
// `rate_limit_exceeded`, and Copilot's `user_weekly_rate_limited`.
|
||||
'rate_limit',
|
||||
'resource_exhausted',
|
||||
'quota',
|
||||
'too many requests',
|
||||
'exceeded your',
|
||||
];
|
||||
|
||||
const CLASSIFY_HANDOFF_SENTINEL = 'classifyhandoffifneeded is not defined';
|
||||
|
||||
function parseRetryAfter(body: string): number | undefined {
|
||||
// Match "retry-after: N" or "Retry-After: N" with optional surrounding
|
||||
// punctuation. Captures integer seconds only — HTTP-date form is rare in
|
||||
// agent return bodies and not worth the surface area.
|
||||
const match = body.match(/\bretry[-_ ]after[:\s]+(\d+)\b/i);
|
||||
if (!match) return undefined;
|
||||
const seconds = Number.parseInt(match[1]!, 10);
|
||||
return Number.isFinite(seconds) ? seconds : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Query-handler wrapper for `agent.classify-failure`. Reads the body to
|
||||
* classify from the joined positional args (typed via `--`) so workflow
|
||||
* shell snippets can pass it as `gsd-sdk query agent.classify-failure -- "$BODY"`.
|
||||
*/
|
||||
export async function agentClassifyFailure(
|
||||
args: string[],
|
||||
): Promise<{ data: AgentFailureClassification }> {
|
||||
const body = args.join(' ');
|
||||
return { data: classifyAgentFailure(body) };
|
||||
}
|
||||
|
||||
export function classifyAgentFailure(body: string): AgentFailureClassification {
|
||||
const normalized = body.toLowerCase();
|
||||
|
||||
if (normalized.trim() === '') {
|
||||
return { class: 'unknown-failure' };
|
||||
}
|
||||
|
||||
// Quota sentinels take precedence over the classifyHandoff runtime bug:
|
||||
// a quota-kill that *also* crashed the completion handler is still a quota
|
||||
// event, and the recovery path differs (wait vs. spot-check-and-treat-as-OK).
|
||||
for (const sentinel of QUOTA_SENTINELS) {
|
||||
if (normalized.includes(sentinel)) {
|
||||
const retryAfterSeconds = parseRetryAfter(body);
|
||||
return retryAfterSeconds === undefined
|
||||
? { class: 'quota-exceeded', sentinel }
|
||||
: { class: 'quota-exceeded', sentinel, retryAfterSeconds };
|
||||
}
|
||||
}
|
||||
|
||||
if (normalized.includes(CLASSIFY_HANDOFF_SENTINEL)) {
|
||||
return {
|
||||
class: 'classify-handoff-bug',
|
||||
sentinel: CLASSIFY_HANDOFF_SENTINEL,
|
||||
};
|
||||
}
|
||||
|
||||
return { class: 'unknown-failure' };
|
||||
}
|
||||
@@ -104,6 +104,7 @@ export interface NonFamilyCommandAlias {
|
||||
}
|
||||
|
||||
export const NON_FAMILY_COMMAND_ALIASES: readonly NonFamilyCommandAlias[] = [
|
||||
{ canonical: 'agent.classify-failure', aliases: ['agent classify-failure'], mutation: false },
|
||||
{ canonical: 'check-commit', aliases: [], mutation: true },
|
||||
{ canonical: 'check.decision-coverage-plan', aliases: ['check decision-coverage-plan'], mutation: false },
|
||||
{ canonical: 'check.decision-coverage-verify', aliases: ['check decision-coverage-verify'], mutation: false },
|
||||
|
||||
@@ -80,4 +80,6 @@ export const NON_FAMILY_COMMAND_MANIFEST: readonly NonFamilyCommandManifestEntry
|
||||
{ canonical: 'generate-claude-md', aliases: [], mutation: true, outputMode: 'json' },
|
||||
|
||||
{ canonical: 'verify-summary', aliases: ['verify.summary', 'verify summary'], mutation: false, outputMode: 'raw' },
|
||||
|
||||
{ canonical: 'agent.classify-failure', aliases: ['agent classify-failure'], mutation: false, outputMode: 'json' },
|
||||
] as const;
|
||||
|
||||
@@ -24,6 +24,7 @@ import { checkCompletion } from './check-completion.js';
|
||||
import { checkGates } from './check-gates.js';
|
||||
import { checkVerificationStatus } from './check-verification-status.js';
|
||||
import { checkShipReady } from './check-ship-ready.js';
|
||||
import { agentClassifyFailure } from './agent-failure-classifier.js';
|
||||
|
||||
export const FOUNDATION_STATIC_CATALOG: ReadonlyArray<readonly [string, QueryHandler]> = [
|
||||
['generate-slug', generateSlug],
|
||||
@@ -97,4 +98,6 @@ export const DECISION_ROUTING_STATIC_CATALOG: ReadonlyArray<readonly [string, Qu
|
||||
['check.ship-ready', checkShipReady],
|
||||
['check ship-ready', checkShipReady],
|
||||
['commands', commandsList],
|
||||
['agent.classify-failure', agentClassifyFailure],
|
||||
['agent classify-failure', agentClassifyFailure],
|
||||
] as const;
|
||||
|
||||
Reference in New Issue
Block a user