From e1d661ece064c890adb0ef7416bc924301b66e34 Mon Sep 17 00:00:00 2001 From: Tom Boucher Date: Sat, 2 May 2026 14:26:35 -0400 Subject: [PATCH] feat(#3024): dynamic routing with failure-tier escalation (#3031) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(#3024): dynamic routing with failure-tier escalation Adds a `dynamic_routing` block to .planning/config.json that lets the resolver start agents on a cheap tier and escalate one tier up when the orchestrator detects a soft failure (verification inconclusive, plan-check FLAG, etc.). Solves the "pay Opus rates as insurance" anti-pattern by making escalation observed-quality-driven. Architecture: - AGENT_DEFAULT_TIERS map (light/standard/heavy) — every agent in MODEL_PROFILES declares a default tier; tests assert coverage so adding a new agent without updating the map fails CI. - nextTier(currentTier) helper — light → standard → heavy → heavy (heavy stays at heavy; can't go further). - resolveModelForTier(cwd, agentType, attempt) — new resolver. The orchestrator tracks the attempt counter and passes 0 for the first spawn, 1+ on escalation. The resolver caps internally at max_escalations so the orchestrator can blindly bump the counter. - Schema validation: dynamic_routing.enabled / escalate_on_failure / max_escalations / tier_models.. Unknown tiers and unknown sub-keys rejected at config-set time. - SDK schema mirror updated to keep CJS/SDK in lockstep (#2653). Resolution precedence (highest → lowest): 1. model_overrides[] (full IDs accepted) 2. dynamic_routing.tier_models[] (NEW; escalation-aware) 3. models[] (#3023 phase-type map) 4. model_profile (per-agent column) 5. Runtime default Backward compatibility: dynamic_routing is disabled by default (enabled: false or block omitted). resolveModelForTier short- circuits to resolveModelInternal in that case, so callers can adopt unconditionally without breaking existing behavior. This PR delivers the JS-layer infrastructure: schema + tier map + resolver. Orchestrator adoption (workflow markdown updates that detect soft failures and call resolveModelForTier with attempt+1) is incremental follow-up — verifier / plan-checker / integration- checker each adopt the protocol when ready. Tests (23 cases, all structural-IR — no stdout grep): - Schema invariants: AGENT_DEFAULT_TIERS coverage, VALID_AGENT_TIERS exact match, every assignment uses a valid tier - nextTier helper: light→standard→heavy→heavy, null on invalid input - Disabled mode: no block + enabled:false both no-op (back-compat) - Enabled mode: attempt=0 returns default tier model, attempt=1 escalates, beyond max_escalations caps, heavy agents stay heavy, default max_escalations=1 when omitted - Precedence: per-agent override beats dynamic_routing, dynamic_routing beats phase-type models - Validation: every settings key accepted, unknown tiers/sub-keys rejected, bare `dynamic_routing` rejected as config-set target Documentation: - get-shit-done/references/model-profiles.md — full reference section - docs/CONFIGURATION.md — full settings table + escalation flow - docs/USER-GUIDE.md — task-oriented "Cheap-by-default" section - docs/FEATURES.md — config row cross-link Verification: - 23/23 pass on regression test - 6843/6843 full suite (23 net new from 6820) - lint-no-source-grep clean (376 test files) - SDK schema mirror keeps CJS/SDK in sync per #2653 parity test Closes #3024 * fix(#3024): honor escalate_on_failure:false + 3 CR follow-ups CodeRabbit on PR #3031 (4 findings — 1 Major + 2 Minor + 1 Nitpick): 1. **Major (inline)** — get-shit-done/bin/lib/core.cjs:1668 resolveModelForTier ignored dynamic_routing.escalate_on_failure. When the user set it to false, escalation should be disabled, but the resolver only checked attempt/max_escalations. An orchestrator that always passes attempt+1 on retry would silently escalate despite the user opting out. Fix: gate effectiveAttempt on `dr.escalate_on_failure !== false` so false short-circuits every attempt back to the default tier. 2. **Minor (inline)** — docs/CONFIGURATION.md:123-126 The dynamic_routing rows in the Core Settings table had 4 cells instead of 5 (missing the Options column), breaking the table structure. Added explicit Options values for enabled / escalate_on_failure / max_escalations rows. 3. **Minor (outside-diff)** — references/model-profiles.md:179-195 "Resolution Logic" sketch was pre-#3024 and didn't include dynamic_routing in the precedence ladder. Updated to a 6-step block with dynamic_routing at step 3 (between override and phase-type). 4. **Nitpick** — tests/feat-3024-dynamic-routing.test.cjs:189+ Tests used `if (lightAgent) { ... }` guards that silent-pass when AGENT_DEFAULT_TIERS drifts. Replaced all 5 conditional skips with `assert.ok(lightAgent, '...')` preconditions so a tier-mapping change surfaces as a test failure. Plus: 2 new regression tests for the Major fix: - escalate_on_failure:false caps every attempt at default tier - escalate_on_failure:true (explicit) still escalates normally Verification: - 25/25 pass on regression test (23 prior + 2 escalate_on_failure) - 6845/6845 full suite (2 net new) - lint-no-source-grep clean * docs(#3024): align precedence + add fence language tags (CR follow-up) CodeRabbit (3 minor): 1. docs/CONFIGURATION.md:691 — "Per-Phase-Type Models → Resolution precedence" was a 4-step block written pre-#3024; readers got contradictory rules between the per-phase-type section and the later dynamic_routing section. Updated to the same 5-step ladder with dynamic_routing at step 2, and noted that dynamic_routing is disabled by default so this section's behavior is unchanged when the kill-switch is off. 2. docs/CONFIGURATION.md:770 — escalation-flow code fence missing language tag (MD040). Added `text`. 3. references/model-profiles.md:184 — resolution-ladder code fence missing language tag (MD040). Added `text`. No code changes; docs only. Verification: regression test still 25/25. * docs(#3024): clarify precedence prose — five layers, not four (CR nitpick) CodeRabbit nitpick: the "Per-Phase-Type Models → Resolution precedence" prose said "The four layers compose..." but the ladder above lists five (including Runtime default). Also "dynamic_routing escalates per-attempt above all of them" misreads as suggesting dynamic_routing wins over model_overrides — actually overrides still win at step 1. Reworded top-down so the precedence direction is unambiguous: - model_profile = base - models = phase-level override - dynamic_routing = per-attempt escalation - model_overrides = per-agent exception (top) - runtime default = fallback No code changes; docs only. * docs(#3024): note escalate_on_failure:false in escalation-flow diagram (CR) CodeRabbit nitpick: the escalation-flow diagram in docs/CONFIGURATION.md described the soft-failure → respawn → tier_models[next_tier_up] path, but didn't surface the `dynamic_routing.escalate_on_failure: false` kill-switch right next to it. Users reading the flow diagram (which is the canonical place to understand attempt behavior) wouldn't see that the kill-switch overrides the soft-failure branch. Added a one-paragraph note immediately after the flow listing, before the tier-sequence example, so the kill-switch is visible exactly where users decide whether escalation will happen. No code changes; docs only. --- .changeset/dynamic-routing.md | 5 + docs/CONFIGURATION.md | 97 +++++- docs/FEATURES.md | 1 + docs/USER-GUIDE.md | 30 ++ get-shit-done/bin/lib/config-schema.cjs | 5 + get-shit-done/bin/lib/core.cjs | 97 +++++- get-shit-done/bin/lib/model-profiles.cjs | 63 ++++ get-shit-done/references/model-profiles.md | 61 +++- sdk/src/query/config-schema.ts | 6 + tests/feat-3024-dynamic-routing.test.cjs | 366 +++++++++++++++++++++ 10 files changed, 717 insertions(+), 14 deletions(-) create mode 100644 .changeset/dynamic-routing.md create mode 100644 tests/feat-3024-dynamic-routing.test.cjs diff --git a/.changeset/dynamic-routing.md b/.changeset/dynamic-routing.md new file mode 100644 index 000000000..095ef84cb --- /dev/null +++ b/.changeset/dynamic-routing.md @@ -0,0 +1,5 @@ +--- +type: Added +pr: TBD +--- +**`dynamic_routing` block in `.planning/config.json` for failure-tier escalation (#3024).** Each agent declares a default tier (`light` / `standard` / `heavy`); when `dynamic_routing.enabled: true`, the resolver picks `tier_models[default_tier]` for the first spawn and escalates one tier up on orchestrator-detected soft failure (capped by `max_escalations`). Disabled by default — fully backward compatible. Composes with `model_overrides` (higher precedence) and `models.` (lower) for full cost-control flexibility. Adds new resolver `resolveModelForTier(cwd, agent, attempt)` to `core.cjs` for orchestrator integration. diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 240524b99..8c0590d1a 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -17,6 +17,7 @@ GSD stores project settings in `.planning/config.json`. Created during `/gsd-new "model_profile": "balanced", "model_overrides": {}, "models": {}, + "dynamic_routing": null, "planning": { "commit_docs": true, "search_gitignored": false, @@ -119,6 +120,10 @@ GSD stores project settings in `.planning/config.json`. Created during `/gsd-new | `runtime` | string | `claude`, `codex`, or any string | (none) | Active runtime for [runtime-aware profile resolution](#runtime-aware-profiles-2517). When set, profile tiers (opus/sonnet/haiku) resolve to runtime-native model IDs. Today only the Codex install path emits per-agent model IDs from this resolver; other runtimes (`opencode`, `gemini`, `qwen`, `copilot`, …) consume the resolver at spawn time and gain dedicated install-path support in [#2612](https://github.com/gsd-build/get-shit-done/issues/2612). When unset (default), behavior is unchanged from prior versions. Added in v1.39 | | `model_profile_overrides..` | string \| object | per-runtime tier override | (none) | Override the runtime-aware tier mapping for a specific `(runtime, tier)`. Tier is one of `opus`, `sonnet`, `haiku`. Value is either a model ID string (e.g. `"gpt-5-pro"`) or `{ model, reasoning_effort }`. See [Runtime-Aware Profiles](#runtime-aware-profiles-2517). Added in v1.39 | | `models.` | enum | `opus`, `sonnet`, `haiku`, `inherit` | (none) | Per-phase-type model tier. Six accepted slots: `planning`, `discuss`, `research`, `execution`, `verification`, `completion`. Lets you tune at the phase level ("Opus for planning, Sonnet for the rest") without learning agent names. Resolves between `model_overrides` (higher) and `model_profile` (lower); see [Per-Phase-Type Models](#per-phase-type-models-models--added-in-v140). Added in v1.40 ([#3023](https://github.com/gsd-build/get-shit-done/pull/3030)) | +| `dynamic_routing.enabled` | boolean | `true`, `false` | `false` | Master switch for [dynamic routing with failure-tier escalation](#dynamic-routing-with-failure-tier-escalation-dynamic_routing--added-in-v140). When `true`, agents resolve to `tier_models[default_tier]` and escalate one tier up on orchestrator-detected soft failure. Added in v1.40 ([#3024](https://github.com/gsd-build/get-shit-done/pull/3031)) | +| `dynamic_routing.tier_models.` | enum | `opus`, `sonnet`, `haiku` | (none) | Tier alias for `light`, `standard`, or `heavy`. Used when `dynamic_routing.enabled: true`. Added in v1.40 | +| `dynamic_routing.escalate_on_failure` | boolean | `true`, `false` | `true` | When `false`, escalation is disabled even if `enabled: true` — every attempt uses the default tier. Added in v1.40 | +| `dynamic_routing.max_escalations` | integer | `0`, `1`, `2`, … | `1` | Hard cap on retries per agent invocation. Beyond the cap the resolver returns the cap-tier model. Added in v1.40 | | `project_code` | string | any short string | (none) | Prefix for phase directory names (e.g., `"ABC"` produces `ABC-01-setup/`). Added in v1.31 | | `response_language` | string | language code | (none) | Language for agent responses (e.g., `"pt"`, `"ko"`, `"ja"`). Propagates to all spawned agents for cross-phase language consistency. Added in v1.32 | | `context_window` | number | any integer | `200000` | Context window size in tokens. Set `1000000` for 1M-context models (e.g., `claude-opus-4-7[1m]`). Values `>= 500000` enable adaptive context enrichment (full-body reads of prior SUMMARY.md, deeper anti-pattern reads). Configured via `/gsd-settings-advanced`. | @@ -683,14 +688,15 @@ for the change to take effect. See issue #2256. #### Resolution precedence (highest → lowest) -``` -1. model_overrides[] ← per-agent; full IDs accepted; targeted exception -2. models[] ← coarse phase-level tier -3. model_profile (per-agent col) ← global tier strategy -4. Runtime default ← when nothing else applies +```text +1. model_overrides[] ← per-agent; full IDs; targeted exception +2. dynamic_routing.tier_models[] ← when enabled (see §Dynamic Routing) +3. models[] ← coarse phase-level tier (this section) +4. model_profile (per-agent col) ← global tier strategy +5. Runtime default ← when nothing else applies ``` -The three layers compose: `models` defaults a phase, and `model_overrides` carves an exception out of it. In the example above, all five research agents resolve to `sonnet` *except* `gsd-codebase-mapper`, which the per-agent override pins to `haiku`. +The five layers compose top-down: `model_profile` is the base tier, `models[]` overrides at the phase level, `dynamic_routing` (when enabled) escalates per-attempt on soft failure, `model_overrides[]` carves per-agent exceptions at the top, and the runtime default applies when nothing else does. In the example above, all five research agents resolve to `sonnet` *except* `gsd-codebase-mapper`, which the per-agent override pins to `haiku`. `dynamic_routing` is disabled by default — when off (`enabled: false` or block omitted), this section's behavior is unchanged from today. #### Accepted values @@ -728,6 +734,85 @@ $ gsd config-set models.research sonnet Direct edits to `.planning/config.json` are looser — the resolver simply ignores values it doesn't recognize and falls through to the profile tier — so a typo doesn't silently break tier resolution. +### Dynamic Routing with Failure-Tier Escalation (`dynamic_routing`) — added in v1.40 + +> Start cheap, escalate only when the agent fails the gate. Added in [#3024](https://github.com/gsd-build/get-shit-done/pull/3031). + +`dynamic_routing` lets you pay for the cheap tier by default and only escalate to the more expensive tier when the orchestrator detects a soft failure (verification inconclusive, plan-check FLAG, etc.). + +```json +{ + "dynamic_routing": { + "enabled": true, + "tier_models": { + "light": "haiku", + "standard": "sonnet", + "heavy": "opus" + }, + "escalate_on_failure": true, + "max_escalations": 1 + } +} +``` + +#### Agent default tiers + +Each agent in `MODEL_PROFILES` declares one of three default tiers. The resolver picks `tier_models[default_tier]` for the first attempt. + +| Tier | Agents | Use case | +|---|---|---| +| `light` | gsd-codebase-mapper, gsd-pattern-mapper, gsd-research-synthesizer, gsd-plan-checker, gsd-integration-checker, gsd-nyquist-auditor, gsd-ui-checker, gsd-ui-auditor, gsd-doc-verifier | Cheap/fast — pure mappers, scanners, low-stakes audits | +| `standard` | gsd-executor, gsd-phase-researcher, gsd-project-researcher, gsd-verifier, gsd-doc-writer, gsd-ui-researcher | Default workhorse — research, writing, primary verification | +| `heavy` | gsd-planner, gsd-roadmapper, gsd-debugger | Deep reasoning — already at top, can't escalate further | + +#### Escalation flow + +```text +1. Orchestrator spawns agent → resolver returns tier_models[default_tier] +2. Soft failure? + ├─ no → ✓ done (cheap path) + └─ yes → orchestrator re-spawns at attempt+1 + → resolver returns tier_models[next_tier_up] + → cap at max_escalations +3. Hard failure (exception/crash) → bypass escalation, surface immediately +``` + +If `dynamic_routing.escalate_on_failure: false`, soft failures do **not** advance the tier — every respawn keeps using `tier_models[default_tier]` regardless of the attempt counter. The kill-switch overrides the soft-failure branch above. + +`light → standard → heavy → heavy` (heavy stays at heavy; can't go further). + +#### Resolution precedence (highest → lowest) + +1. **`model_overrides[]`** — full IDs accepted; targeted exception +2. **`dynamic_routing.tier_models[]`** (when `enabled: true`) +3. **`models[]`** — coarse phase-level (#3023) +4. **`model_profile`** — per-agent column from active profile +5. **Runtime default** + +The `dynamic_routing` block is **disabled by default** — `enabled: false` (or omitting the block) preserves today's static resolution exactly. + +#### Settings + +| Key | Type | Default | Description | +|---|---|---|---| +| `dynamic_routing.enabled` | boolean | `false` | Master switch. When `true`, the dynamic-routing resolver is used for tier selection. | +| `dynamic_routing.tier_models.light` | enum | (none) | Tier alias for the light tier. Typically `haiku`. | +| `dynamic_routing.tier_models.standard` | enum | (none) | Tier alias for standard. Typically `sonnet`. | +| `dynamic_routing.tier_models.heavy` | enum | (none) | Tier alias for heavy. Typically `opus`. | +| `dynamic_routing.escalate_on_failure` | boolean | `true` | When false, escalation is disabled (every attempt uses the default tier). | +| `dynamic_routing.max_escalations` | integer | `1` | Hard cap on retries per agent invocation. Prevents runaway loops. | + +#### When to use which + +| You want | Use | +|---|---| +| One tier strategy across all agents | `model_profile` | +| Coarse phase-level tuning | `models.` | +| Per-agent precision (full IDs) | `model_overrides` | +| **Cheap-by-default, escalate only on failure** | **`dynamic_routing`** | + +`dynamic_routing` is structurally a *cost lever*: you pay Opus rates only for the hard cases that warrant Opus. Compose with `model_overrides` for per-agent exceptions (override always wins). + ### Non-Claude Runtimes (Codex, OpenCode, Gemini CLI, Kilo) When GSD is installed for a non-Claude runtime, the installer automatically sets `resolve_model_ids: "omit"` in `~/.gsd/defaults.json`. This causes GSD to return an empty model parameter for all agents, so each agent uses whatever model the runtime is configured with. No additional setup is needed for the default case. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 83fd3eddc..5f301dc2c 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -924,6 +924,7 @@ continues. Drift detection cannot fail verification. | `granularity` | enum | `standard` | `coarse`, `standard`, or `fine` | | `model_profile` | enum | `balanced` | `quality`, `balanced`, `budget`, or `inherit` | | `models.` | enum | (none) | Per-phase-type tier override (`planning`, `discuss`, `research`, `execution`, `verification`, `completion`). Values: `opus`, `sonnet`, `haiku`, `inherit`. Coarse phase-level tuning that wins over `model_profile` but loses to per-agent `model_overrides`. See [CONFIGURATION.md](CONFIGURATION.md#per-phase-type-models-models--added-in-v140). Added in v1.40 | +| `dynamic_routing.enabled` | boolean | `false` | Master switch for failure-tier escalation. When `true`, agents resolve to `tier_models[default_tier]` and escalate one tier on orchestrator-detected soft failure. Capped by `max_escalations`. See [CONFIGURATION.md](CONFIGURATION.md#dynamic-routing-with-failure-tier-escalation-dynamic_routing--added-in-v140). Added in v1.40 | | `workflow.research` | boolean | `true` | Domain research before planning | | `workflow.plan_check` | boolean | `true` | Plan verification loop | | `workflow.verifier` | boolean | `true` | Post-execution verification | diff --git a/docs/USER-GUIDE.md b/docs/USER-GUIDE.md index 7afb9185d..b37a62799 100644 --- a/docs/USER-GUIDE.md +++ b/docs/USER-GUIDE.md @@ -1047,6 +1047,36 @@ That gives sonnet to all research agents *except* the codebase mapper, which run For the full mapping table and resolution-precedence rules, see [Per-Phase-Type Models](CONFIGURATION.md#per-phase-type-models-models--added-in-v140) in the configuration reference. +### Cheap-by-default with `dynamic_routing` — added in v1.40 + +If you've been paying Opus rates everywhere as insurance against a single hard verification, dynamic routing flips it: every agent starts on a cheaper tier and escalates only when the orchestrator marks a soft failure (verification inconclusive, plan-check FLAG, etc.). + +```json +{ + "dynamic_routing": { + "enabled": true, + "tier_models": { + "light": "haiku", + "standard": "sonnet", + "heavy": "opus" + }, + "escalate_on_failure": true, + "max_escalations": 1 + } +} +``` + +Each agent has a default tier (`light`, `standard`, or `heavy`). On the first attempt, GSD picks `tier_models[default_tier]`. If the orchestrator detects a soft failure, it re-spawns once at the next tier up. `max_escalations` caps total retries so a runaway loop can't burn through your budget. + +Concretely: +- `gsd-codebase-mapper` (default `light`) → first attempt = `haiku`. If escalated → `sonnet`. +- `gsd-verifier` (default `standard`) → first attempt = `sonnet`. If escalated → `opus`. +- `gsd-planner` (default `heavy`) → always `opus`. No tier above; can't escalate further. + +To turn it off, set `dynamic_routing.enabled: false` (the default) — behavior is identical to today. + +For the full agent → tier mapping and resolution-precedence rules, see [Dynamic Routing](CONFIGURATION.md#dynamic-routing-with-failure-tier-escalation-dynamic_routing--added-in-v140) in the configuration reference. + ### Using Non-Claude Runtimes (Codex, OpenCode, Gemini CLI, Kilo) If you installed GSD for a non-Claude runtime, the installer already configured model resolution so all agents use the runtime's default model. No manual setup is needed. Specifically, the installer sets `resolve_model_ids: "omit"` in your config, which tells GSD to skip Anthropic model ID resolution and let the runtime choose its own default model. diff --git a/get-shit-done/bin/lib/config-schema.cjs b/get-shit-done/bin/lib/config-schema.cjs index 42e5671f9..2fec078c0 100644 --- a/get-shit-done/bin/lib/config-schema.cjs +++ b/get-shit-done/bin/lib/config-schema.cjs @@ -90,6 +90,11 @@ const DYNAMIC_KEY_PATTERNS = [ // precedence over phase-type at resolve time. { topLevel: 'models', test: (k) => /^models\.(planning|discuss|research|execution|verification|completion)$/.test(k), description: 'models.' }, + // #3024 — dynamic routing block. Three top-level scalar settings + // plus a tier_models sub-block keyed by light/standard/heavy. + { topLevel: 'dynamic_routing', + test: (k) => /^dynamic_routing\.(enabled|escalate_on_failure|max_escalations|tier_models\.(light|standard|heavy))$/.test(k), + description: 'dynamic_routing.>' }, ]; /** diff --git a/get-shit-done/bin/lib/core.cjs b/get-shit-done/bin/lib/core.cjs index 1435d03b9..a51b80f09 100644 --- a/get-shit-done/bin/lib/core.cjs +++ b/get-shit-done/bin/lib/core.cjs @@ -6,7 +6,7 @@ const fs = require('fs'); const os = require('os'); const path = require('path'); const { execSync, execFileSync, spawnSync } = require('child_process'); -const { MODEL_PROFILES, AGENT_TO_PHASE_TYPE, VALID_PHASE_TYPES } = require('./model-profiles.cjs'); +const { MODEL_PROFILES, AGENT_TO_PHASE_TYPE, VALID_PHASE_TYPES, AGENT_DEFAULT_TIERS, VALID_AGENT_TIERS, nextTier } = require('./model-profiles.cjs'); // Compatibility shim: new imports should use planning-workspace.cjs directly. const { planningDir, @@ -506,6 +506,11 @@ function loadConfig(cwd) { // resolveModelInternal. Defaults to null so configs without it // behave exactly as today. models: parsed.models || null, + // #3024 — dynamic routing block. When `enabled: true`, the + // resolveModelForTier() resolver picks tier_models[default_tier] + // for the agent and escalates one tier per attempt up to + // max_escalations. Disabled by default for backward compat. + dynamic_routing: parsed.dynamic_routing || null, // #2517 — runtime-aware profiles. `runtime` defaults to null (back-compat). // When null, resolveModelInternal preserves today's Claude-native behavior. // NOTE: `runtime` and `model_profile_overrides` are intentionally read @@ -567,6 +572,7 @@ function loadConfig(cwd) { subagent_timeout: globalDefaults.subagent_timeout ?? defaults.subagent_timeout, model_overrides: globalDefaults.model_overrides || null, models: globalDefaults.models || null, + dynamic_routing: globalDefaults.dynamic_routing || null, agent_skills: globalDefaults.agent_skills || {}, response_language: globalDefaults.response_language || null, }; @@ -1599,6 +1605,94 @@ function resolveModelInternal(cwd, agentType) { return alias; } +/** + * #3024 — Resolve a model for a specific dynamic-routing attempt. + * + * The orchestrator (workflow agent) tracks the attempt counter. On + * the first spawn, it calls with attempt=0. If the orchestrator detects + * a soft failure (verification inconclusive, plan-check FLAG, etc.), + * it re-spawns with attempt=1, which escalates the agent's tier one + * step up. `max_escalations` caps how many escalations are allowed. + * + * Resolution precedence (highest → lowest): + * 1. config.model_overrides[agent] (full IDs accepted) + * 2. dynamic_routing.tier_models[escalated_tier] (when enabled) + * 3. models[phase_type] / model_profile (existing chain via + * resolveModelInternal) + * + * When dynamic_routing is null/disabled, this function is identical + * to resolveModelInternal — orchestrators can call it unconditionally + * without breaking back-compat. + * + * @param {string} cwd - Project directory. + * @param {string} agentType - Agent name (e.g. 'gsd-verifier'). + * @param {number} [attempt=0] - 0 for first spawn; 1+ for escalation. + * Capped internally at max_escalations. + * @returns {string} Model alias (opus/sonnet/haiku) or full ID. + */ +function resolveModelForTier(cwd, agentType, attempt) { + const config = loadConfig(cwd); + const attemptN = Number.isInteger(attempt) && attempt > 0 ? attempt : 0; + + // Per-agent override always wins — same as resolveModelInternal step 1. + // User-supplied full IDs bypass the entire tier mechanism. + const override = config.model_overrides?.[agentType]; + if (override) return override; + + const dr = config.dynamic_routing; + // Disabled / missing / non-object → fall back to the existing resolver. + if (!dr || typeof dr !== 'object' || dr.enabled !== true) { + return resolveModelInternal(cwd, agentType); + } + + const tierModels = dr.tier_models; + if (!tierModels || typeof tierModels !== 'object') { + // tier_models missing — can't dynamic-route; fall back. + return resolveModelInternal(cwd, agentType); + } + + const defaultTier = AGENT_DEFAULT_TIERS[agentType]; + if (!defaultTier || !VALID_AGENT_TIERS.has(defaultTier)) { + // Unmapped agent — no default tier; fall back so we don't silently + // pick the wrong model. + return resolveModelInternal(cwd, agentType); + } + + // Cap effective escalation at max_escalations (default 1). Beyond + // the cap, the resolver returns the model for the cap level so the + // orchestrator can log "max escalations reached" without burning + // further budget. + // + // CR Major (#3031): `escalate_on_failure: false` is the kill-switch + // for escalation — when false, every attempt resolves to the default + // tier regardless of the attempt counter. Without this guard, an + // orchestrator that blindly bumps the counter on retry would silently + // escalate even though the user opted out. + const maxEscalations = Number.isInteger(dr.max_escalations) && dr.max_escalations >= 0 + ? dr.max_escalations + : 1; + const escalationEnabled = dr.escalate_on_failure !== false; + const effectiveAttempt = escalationEnabled + ? Math.min(attemptN, maxEscalations) + : 0; + + // Walk the escalation chain N times from the default tier. + let tier = defaultTier; + for (let i = 0; i < effectiveAttempt; i += 1) { + const next = nextTier(tier); + if (!next || next === tier) break; // already at top + tier = next; + } + + const alias = tierModels[tier]; + if (typeof alias !== 'string' || alias.length === 0) { + // Misconfigured tier_models — missing slot. Fall back rather + // than emit an empty model id. + return resolveModelInternal(cwd, agentType); + } + return alias; +} + /** * #2517 — Resolve runtime-specific reasoning_effort for an agent. * Returns null unless: @@ -1959,6 +2053,7 @@ module.exports = { getArchivedPhaseDirs, getRoadmapPhaseInternal, resolveModelInternal, + resolveModelForTier, resolveReasoningEffortInternal, RUNTIME_PROFILE_MAP, RUNTIMES_WITH_REASONING_EFFORT, diff --git a/get-shit-done/bin/lib/model-profiles.cjs b/get-shit-done/bin/lib/model-profiles.cjs index a314d30b7..719e6bb63 100644 --- a/get-shit-done/bin/lib/model-profiles.cjs +++ b/get-shit-done/bin/lib/model-profiles.cjs @@ -83,6 +83,66 @@ const VALID_PHASE_TYPES = new Set([ 'planning', 'discuss', 'research', 'execution', 'verification', 'completion', ]); +/** + * #3024 — Per-agent default tier for dynamic routing. + * + * Each agent declares a default routing tier (light/standard/heavy) + * that the dynamic-routing resolver uses to pick from + * `dynamic_routing.tier_models[tier]` on the first attempt. On + * orchestrator-detected soft failure, the resolver escalates to the + * next tier up (capped at `max_escalations`). + * + * Tier semantics: + * - light: cheap/fast — pure mappers/scanners, low-stakes verifiers + * - standard: default workhorse — most researchers/writers/checkers + * - heavy: deep reasoning — planners/debuggers; can't escalate further + * + * Adding a new agent to MODEL_PROFILES requires adding an entry here too; + * tests/feat-3024-dynamic-routing.test.cjs asserts coverage. + */ +const AGENT_DEFAULT_TIERS = { + // Heavy — deep reasoning, planning, hard debugging + 'gsd-planner': 'heavy', + 'gsd-roadmapper': 'heavy', + 'gsd-debugger': 'heavy', + // Standard — default workhorse: research, writing, primary verification + 'gsd-executor': 'standard', + 'gsd-phase-researcher': 'standard', + 'gsd-project-researcher': 'standard', + 'gsd-verifier': 'standard', + 'gsd-doc-writer': 'standard', + 'gsd-ui-researcher': 'standard', + // Light — fast scanners, structural mappers, low-stakes audits + 'gsd-codebase-mapper': 'light', + 'gsd-pattern-mapper': 'light', + 'gsd-research-synthesizer': 'light', + 'gsd-plan-checker': 'light', + 'gsd-integration-checker': 'light', + 'gsd-nyquist-auditor': 'light', + 'gsd-ui-checker': 'light', + 'gsd-ui-auditor': 'light', + 'gsd-doc-verifier': 'light', +}; + +/** + * The three valid agent tier slots for dynamic routing. Used to + * validate `dynamic_routing.tier_models.` keys at config-set + * time and the AGENT_DEFAULT_TIERS values at startup. + */ +const VALID_AGENT_TIERS = new Set(['light', 'standard', 'heavy']); + +/** + * Tier escalation order: light → standard → heavy. + * `nextTier(currentTier)` returns the tier one step up. `heavy` stays + * at heavy (no tier above). Returns null for invalid input so callers + * can detect mis-config rather than silently degrade. + */ +const _TIER_ESCALATION = { light: 'standard', standard: 'heavy', heavy: 'heavy' }; +function nextTier(currentTier) { + if (typeof currentTier !== 'string') return null; + return _TIER_ESCALATION[currentTier] || null; +} + /** * Formats the agent-to-model mapping as a human-readable table (in string format). * @@ -125,6 +185,9 @@ module.exports = { VALID_PROFILES, AGENT_TO_PHASE_TYPE, VALID_PHASE_TYPES, + AGENT_DEFAULT_TIERS, + VALID_AGENT_TIERS, + nextTier, formatAgentToModelMapAsTable, getAgentToModelMapForProfile, }; diff --git a/get-shit-done/references/model-profiles.md b/get-shit-done/references/model-profiles.md index 0367e46a1..e19ca169d 100644 --- a/get-shit-done/references/model-profiles.md +++ b/get-shit-done/references/model-profiles.md @@ -133,22 +133,69 @@ If you're using Claude Code with OpenRouter, a local model, or any non-Anthropic Without `inherit`, GSD's default `balanced` profile spawns specific Anthropic models (`opus`, `sonnet`, `haiku`) for each agent type, which can result in additional API costs through your non-Anthropic provider. +## Dynamic Routing with Failure-Tier Escalation (#3024) + +When `dynamic_routing.enabled = true` in `.planning/config.json`, the resolver picks a model from a tier-mapped table based on the agent's *default tier* (light / standard / heavy) and escalates to the next tier up on orchestrator-detected soft failure. + +```json +{ + "dynamic_routing": { + "enabled": true, + "tier_models": { + "light": "haiku", + "standard": "sonnet", + "heavy": "opus" + }, + "escalate_on_failure": true, + "max_escalations": 1 + } +} +``` + +**Agent default tiers** (each agent in `MODEL_PROFILES` declares one): + +| Tier | Agents | Use case | +|---|---|---| +| `light` | gsd-codebase-mapper, gsd-pattern-mapper, gsd-research-synthesizer, gsd-plan-checker, gsd-integration-checker, gsd-nyquist-auditor, gsd-ui-checker, gsd-ui-auditor, gsd-doc-verifier | Cheap/fast — pure mappers, scanners, low-stakes audits | +| `standard` | gsd-executor, gsd-phase-researcher, gsd-project-researcher, gsd-verifier, gsd-doc-writer, gsd-ui-researcher | Default workhorse — research, writing, primary verification | +| `heavy` | gsd-planner, gsd-roadmapper, gsd-debugger | Deep reasoning — already at top, can't escalate further | + +**Escalation flow** (orchestrator-driven): + +1. Orchestrator spawns agent with `attempt: 0` → resolver returns `tier_models[default_tier]` +2. If orchestrator marks the result a soft failure, it re-spawns with `attempt: 1` → resolver returns `tier_models[next_tier_up]` +3. `max_escalations` caps total retries (default 1). Beyond the cap the resolver returns the cap-tier model so the orchestrator can log without burning further budget. +4. Hard failures (exceptions) bypass escalation and surface immediately. + +**Precedence with other tier sources** (highest → lowest): + +1. `model_overrides[]` — full ID, always wins +2. `dynamic_routing.tier_models[escalated_tier]` — when `enabled: true` +3. `models[]` — coarse phase-level (#3023) +4. `model_profile` — global tier strategy + +When `dynamic_routing.enabled = false` (default), behavior is identical to today. + ## Resolution Logic -Orchestrators resolve model before spawning: +Orchestrators resolve model before spawning. The full precedence ladder +is (highest → lowest): -``` +```text 1. Read .planning/config.json -2. Check model_overrides for agent-specific override -3. If no override, check models[phase_type] for a phase-type tier +2. Check model_overrides[] (full IDs accepted; targeted exceptions) +3. If dynamic_routing.enabled, return tier_models[escalated_tier] + (see §Dynamic Routing — escalation steps tier up per attempt counter) +4. If no dynamic_routing match, check models[phase_type] for a phase-type tier (see §Per-Phase-Type Model Map for the agent → phase-type mapping) -4. If no phase-type slot, look up agent in profile table -5. Pass model parameter to Task call +5. If no phase-type slot, look up agent in profile table +6. Pass model parameter to Task call ``` The same precedence applies to `reasoning_effort` resolution on runtimes that support it (Codex), so `model` and `reasoning_effort` always derive -from the same tier source — a `models[phase_type]` override flips both. +from the same tier source — a `models[phase_type]` or +`dynamic_routing` override flips both. ## Per-Agent Overrides diff --git a/sdk/src/query/config-schema.ts b/sdk/src/query/config-schema.ts index 9e6b0d7be..0029efa6a 100644 --- a/sdk/src/query/config-schema.ts +++ b/sdk/src/query/config-schema.ts @@ -117,6 +117,12 @@ export const DYNAMIC_KEY_PATTERNS: readonly DynamicKeyPattern[] = [ description: 'models.', test: (k) => /^models\.(planning|discuss|research|execution|verification|completion)$/.test(k), }, + // #3024 — dynamic routing with failure-tier escalation + { + source: '^dynamic_routing\\.(enabled|escalate_on_failure|max_escalations|tier_models\\.(light|standard|heavy))$', + description: 'dynamic_routing.>', + test: (k) => /^dynamic_routing\.(enabled|escalate_on_failure|max_escalations|tier_models\.(light|standard|heavy))$/.test(k), + }, ]; /** Returns true if keyPath is a valid config key (exact or dynamic pattern). */ diff --git a/tests/feat-3024-dynamic-routing.test.cjs b/tests/feat-3024-dynamic-routing.test.cjs new file mode 100644 index 000000000..22feb23d6 --- /dev/null +++ b/tests/feat-3024-dynamic-routing.test.cjs @@ -0,0 +1,366 @@ +/** + * Feature test for issue #3024 — dynamic routing with failure-tier escalation. + * + * Adds a `dynamic_routing` block to .planning/config.json: + * + * { + * "dynamic_routing": { + * "enabled": true, + * "tier_models": { + * "light": "haiku", + * "standard": "sonnet", + * "heavy": "opus" + * }, + * "escalate_on_failure": true, + * "max_escalations": 1 + * } + * } + * + * Each agent has a default tier (light/standard/heavy). When dynamic + * routing is enabled, the resolver picks `tier_models[default_tier]` + * for the first attempt. On orchestrator-detected soft failure, the + * orchestrator calls the resolver again with `attempt: 1`, which + * returns the next tier up (capped at `max_escalations`). + * + * This PR delivers the JS-layer infrastructure: schema + tier map + + * resolver + escalation helpers. Orchestrator adoption is incremental + * follow-up — this PR's contract is the resolver function and the + * config it consumes. + * + * Resolution precedence (highest → lowest): + * 1. model_overrides[agent] (full IDs accepted; targeted) + * 2. dynamic_routing.tier_models[tier] (NEW; escalation-aware) + * 3. models[phase_type] (#3023; coarse phase-level) + * 4. model_profile (per-agent column) + * 5. Runtime default + * + * Tests are typed-IR / structural — assert on the value returned by + * resolveModelForTier or isValidConfigKey, not stdout/grep. + */ + +'use strict'; + +process.env.GSD_TEST_MODE = '1'; + +const { test, describe, beforeEach, afterEach } = require('node:test'); +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const path = require('node:path'); +const os = require('node:os'); + +const { + resolveModelInternal, + resolveModelForTier, +} = require('../get-shit-done/bin/lib/core.cjs'); +const { + AGENT_DEFAULT_TIERS, + VALID_AGENT_TIERS, + MODEL_PROFILES, + nextTier, +} = require('../get-shit-done/bin/lib/model-profiles.cjs'); +const { isValidConfigKey } = require('../get-shit-done/bin/lib/config-schema.cjs'); + +function makeTmp(prefix) { + return fs.mkdtempSync(path.join(os.tmpdir(), `gsd-3024-${prefix}-`)); +} +function writeConfig(dir, config) { + const planningDir = path.join(dir, '.planning'); + fs.mkdirSync(planningDir, { recursive: true }); + fs.writeFileSync(path.join(planningDir, 'config.json'), JSON.stringify(config, null, 2)); +} +function rmr(p) { try { fs.rmSync(p, { recursive: true, force: true }); } catch { /* noop */ } } + +// ─── Schema: AGENT_DEFAULT_TIERS coverage + valid tier set ────────────────── + +describe('#3024 schema: every agent has a default tier (light/standard/heavy)', () => { + test('AGENT_DEFAULT_TIERS exported as a non-empty object', () => { + assert.equal(typeof AGENT_DEFAULT_TIERS, 'object'); + assert.ok(AGENT_DEFAULT_TIERS !== null); + assert.ok(Object.keys(AGENT_DEFAULT_TIERS).length > 0); + }); + + test('VALID_AGENT_TIERS exposes exactly {light, standard, heavy}', () => { + assert.deepStrictEqual([...VALID_AGENT_TIERS].sort(), ['heavy', 'light', 'standard']); + }); + + test('every agent in MODEL_PROFILES has a default tier', () => { + const missing = Object.keys(MODEL_PROFILES).filter((a) => !AGENT_DEFAULT_TIERS[a]); + assert.deepStrictEqual(missing, []); + }); + + test('every assigned tier is one of the three valid tiers', () => { + const invalid = Object.entries(AGENT_DEFAULT_TIERS).filter( + ([, t]) => !VALID_AGENT_TIERS.has(t) + ); + assert.deepStrictEqual(invalid, []); + }); +}); + +// ─── nextTier helper ──────────────────────────────────────────────────────── + +describe('#3024 nextTier helper', () => { + test('exported as a function', () => { + assert.equal(typeof nextTier, 'function'); + }); + + test('light → standard → heavy → heavy (caps at heavy)', () => { + assert.equal(nextTier('light'), 'standard'); + assert.equal(nextTier('standard'), 'heavy'); + assert.equal(nextTier('heavy'), 'heavy', 'already at top — stays at heavy'); + }); + + test('returns null for invalid input', () => { + assert.equal(nextTier('jumbo'), null); + assert.equal(nextTier(null), null); + assert.equal(nextTier(undefined), null); + }); +}); + +// ─── Resolver behavior: dynamic routing, disabled mode ────────────────────── + +describe('#3024 resolveModelForTier: disabled mode is a no-op (acceptance criterion 1)', () => { + let projectDir; + beforeEach(() => { projectDir = makeTmp('disabled'); }); + afterEach(() => { rmr(projectDir); }); + + test('exported as a function', () => { + assert.equal(typeof resolveModelForTier, 'function'); + }); + + test('with no dynamic_routing block, falls back to resolveModelInternal', () => { + writeConfig(projectDir, { model_profile: 'balanced' }); + // resolveModelForTier with attempt=0 must match resolveModelInternal. + const baseline = resolveModelInternal(projectDir, 'gsd-phase-researcher'); + assert.equal(resolveModelForTier(projectDir, 'gsd-phase-researcher', 0), baseline); + }); + + test('with dynamic_routing.enabled=false, attempt argument is ignored — same as resolveModelInternal', () => { + writeConfig(projectDir, { + model_profile: 'balanced', + dynamic_routing: { + enabled: false, + tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' }, + }, + }); + const baseline = resolveModelInternal(projectDir, 'gsd-phase-researcher'); + // attempt=0 and attempt=1 both ignored when disabled + assert.equal(resolveModelForTier(projectDir, 'gsd-phase-researcher', 0), baseline); + assert.equal(resolveModelForTier(projectDir, 'gsd-phase-researcher', 1), baseline); + }); +}); + +// ─── Resolver behavior: dynamic routing, enabled ──────────────────────────── + +describe('#3024 resolveModelForTier: enabled mode picks tier_models[default_tier]', () => { + let projectDir; + beforeEach(() => { projectDir = makeTmp('enabled'); }); + afterEach(() => { rmr(projectDir); }); + + test('attempt=0 returns tier_models[agent_default_tier] (acceptance criterion 2)', () => { + writeConfig(projectDir, { + model_profile: 'balanced', + dynamic_routing: { + enabled: true, + tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' }, + }, + }); + // gsd-codebase-mapper has light default tier per AGENT_DEFAULT_TIERS. + // CR nitpick (#3031): assert preconditions explicitly so a tier + // re-mapping in AGENT_DEFAULT_TIERS surfaces as a test failure + // instead of a silent skip. + assert.equal(AGENT_DEFAULT_TIERS['gsd-codebase-mapper'], 'light', + 'gsd-codebase-mapper expected to be light tier'); + assert.equal(resolveModelForTier(projectDir, 'gsd-codebase-mapper', 0), 'haiku'); + assert.equal(AGENT_DEFAULT_TIERS['gsd-planner'], 'heavy', + 'gsd-planner expected to be heavy tier'); + assert.equal(resolveModelForTier(projectDir, 'gsd-planner', 0), 'opus'); + }); + + test('attempt=1 escalates to next tier up (acceptance criterion 3)', () => { + writeConfig(projectDir, { + model_profile: 'balanced', + dynamic_routing: { + enabled: true, + tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' }, + escalate_on_failure: true, + max_escalations: 1, + }, + }); + // For an agent with default tier 'light', attempt=1 should give 'standard' tier model. + const lightAgent = Object.entries(AGENT_DEFAULT_TIERS).find(([, t]) => t === 'light')?.[0]; + assert.ok(lightAgent, 'AGENT_DEFAULT_TIERS must contain at least one light agent'); + assert.equal(resolveModelForTier(projectDir, lightAgent, 0), 'haiku'); + assert.equal(resolveModelForTier(projectDir, lightAgent, 1), 'sonnet'); + // For a 'standard' agent, attempt=1 should give 'heavy' model. + const stdAgent = Object.entries(AGENT_DEFAULT_TIERS).find(([, t]) => t === 'standard')?.[0]; + assert.ok(stdAgent, 'AGENT_DEFAULT_TIERS must contain at least one standard agent'); + assert.equal(resolveModelForTier(projectDir, stdAgent, 0), 'sonnet'); + assert.equal(resolveModelForTier(projectDir, stdAgent, 1), 'opus'); + }); + + test('attempts beyond max_escalations cap at the highest reachable tier (acceptance criterion 4)', () => { + writeConfig(projectDir, { + model_profile: 'balanced', + dynamic_routing: { + enabled: true, + tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' }, + escalate_on_failure: true, + max_escalations: 1, // cap at 1 escalation total + }, + }); + const lightAgent = Object.entries(AGENT_DEFAULT_TIERS).find(([, t]) => t === 'light')?.[0]; + assert.ok(lightAgent, 'AGENT_DEFAULT_TIERS must contain at least one light agent'); + // attempts beyond max_escalations should not exceed max_escalations' + // tier — i.e. attempt=2 with max=1 = same as attempt=1. + assert.equal(resolveModelForTier(projectDir, lightAgent, 2), 'sonnet', + 'attempt=2 with max_escalations=1 caps at attempt=1 tier'); + assert.equal(resolveModelForTier(projectDir, lightAgent, 5), 'sonnet'); + }); + + test('"heavy" agents stay at heavy (no tier above)', () => { + writeConfig(projectDir, { + model_profile: 'balanced', + dynamic_routing: { + enabled: true, + tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' }, + escalate_on_failure: true, + max_escalations: 2, + }, + }); + const heavyAgent = Object.entries(AGENT_DEFAULT_TIERS).find(([, t]) => t === 'heavy')?.[0]; + assert.ok(heavyAgent, 'AGENT_DEFAULT_TIERS must contain at least one heavy agent'); + assert.equal(resolveModelForTier(projectDir, heavyAgent, 0), 'opus'); + // Already at heavy — escalation cannot go higher. + assert.equal(resolveModelForTier(projectDir, heavyAgent, 1), 'opus'); + assert.equal(resolveModelForTier(projectDir, heavyAgent, 5), 'opus'); + }); + + test('default max_escalations is 1 when omitted', () => { + writeConfig(projectDir, { + model_profile: 'balanced', + dynamic_routing: { + enabled: true, + tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' }, + // max_escalations omitted — default to 1 + }, + }); + const lightAgent = Object.entries(AGENT_DEFAULT_TIERS).find(([, t]) => t === 'light')?.[0]; + assert.ok(lightAgent, 'AGENT_DEFAULT_TIERS must contain at least one light agent'); + // attempt=1 escalates; attempt=2 should cap at attempt=1 (default max=1) + assert.equal(resolveModelForTier(projectDir, lightAgent, 1), 'sonnet'); + assert.equal(resolveModelForTier(projectDir, lightAgent, 2), 'sonnet'); + }); + + // ─── CR Major (#3031): escalate_on_failure: false honored ────────────── + + test('escalate_on_failure:false disables escalation even when attempt > 0 (CR Major)', () => { + // Pre-fix bug: an orchestrator that always passes attempt+1 on retry + // would silently escalate even though the user opted out via + // escalate_on_failure:false. The kill-switch must short-circuit + // every attempt back to the default tier. + writeConfig(projectDir, { + model_profile: 'balanced', + dynamic_routing: { + enabled: true, + tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' }, + escalate_on_failure: false, // ← kill-switch + max_escalations: 5, + }, + }); + const lightAgent = Object.entries(AGENT_DEFAULT_TIERS).find(([, t]) => t === 'light')?.[0]; + assert.ok(lightAgent, 'AGENT_DEFAULT_TIERS must contain at least one light agent'); + // Every attempt must resolve to the default (light → haiku), + // regardless of how high the orchestrator bumped the counter. + assert.equal(resolveModelForTier(projectDir, lightAgent, 0), 'haiku'); + assert.equal(resolveModelForTier(projectDir, lightAgent, 1), 'haiku', + 'escalate_on_failure:false must not escalate even at attempt=1'); + assert.equal(resolveModelForTier(projectDir, lightAgent, 5), 'haiku'); + }); + + test('escalate_on_failure:true (explicit) escalates normally', () => { + // Sanity: explicit true matches the default truthy behavior. + writeConfig(projectDir, { + model_profile: 'balanced', + dynamic_routing: { + enabled: true, + tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' }, + escalate_on_failure: true, + max_escalations: 1, + }, + }); + const lightAgent = Object.entries(AGENT_DEFAULT_TIERS).find(([, t]) => t === 'light')?.[0]; + assert.ok(lightAgent); + assert.equal(resolveModelForTier(projectDir, lightAgent, 1), 'sonnet'); + }); +}); + +// ─── Resolver precedence ──────────────────────────────────────────────────── + +describe('#3024 precedence: per-agent override > dynamic_routing > models > profile', () => { + let projectDir; + beforeEach(() => { projectDir = makeTmp('precedence'); }); + afterEach(() => { rmr(projectDir); }); + + test('per-agent model_overrides beats dynamic_routing (acceptance criterion: override wins)', () => { + writeConfig(projectDir, { + model_profile: 'balanced', + model_overrides: { 'gsd-codebase-mapper': 'openai/gpt-5' }, + dynamic_routing: { + enabled: true, + tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' }, + }, + }); + // Per-agent override always wins, even at escalated attempt. + assert.equal(resolveModelForTier(projectDir, 'gsd-codebase-mapper', 0), 'openai/gpt-5'); + assert.equal(resolveModelForTier(projectDir, 'gsd-codebase-mapper', 1), 'openai/gpt-5'); + }); + + test('dynamic_routing beats phase-type models (#3023)', () => { + writeConfig(projectDir, { + model_profile: 'balanced', + models: { research: 'opus' }, // phase-type would say opus + dynamic_routing: { + enabled: true, + tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' }, + }, + }); + // gsd-codebase-mapper is research phase-type; phase-type would give 'opus', + // but dynamic routing (light default → haiku) wins. + if (AGENT_DEFAULT_TIERS['gsd-codebase-mapper'] === 'light') { + assert.equal(resolveModelForTier(projectDir, 'gsd-codebase-mapper', 0), 'haiku'); + } + }); +}); + +// ─── Schema validation ────────────────────────────────────────────────────── + +describe('#3024 config-schema: dynamic_routing.* validation', () => { + test('dynamic_routing.enabled is a valid config key', () => { + assert.equal(isValidConfigKey('dynamic_routing.enabled'), true); + }); + + test('dynamic_routing.escalate_on_failure is a valid config key', () => { + assert.equal(isValidConfigKey('dynamic_routing.escalate_on_failure'), true); + }); + + test('dynamic_routing.max_escalations is a valid config key', () => { + assert.equal(isValidConfigKey('dynamic_routing.max_escalations'), true); + }); + + test('dynamic_routing.tier_models. for each valid tier', () => { + for (const t of ['light', 'standard', 'heavy']) { + assert.equal(isValidConfigKey(`dynamic_routing.tier_models.${t}`), true); + } + }); + + test('unknown tier in tier_models is rejected', () => { + assert.equal(isValidConfigKey('dynamic_routing.tier_models.jumbo'), false); + assert.equal(isValidConfigKey('dynamic_routing.tier_models.medium'), false); + }); + + test('unknown dynamic_routing.* keys are rejected', () => { + assert.equal(isValidConfigKey('dynamic_routing.foo'), false); + assert.equal(isValidConfigKey('dynamic_routing'), false, + 'bare dynamic_routing (no field) must not be a config-set target'); + }); +});