* feat(#3024): dynamic routing with failure-tier escalation Adds a `dynamic_routing` block to .planning/config.json that lets the resolver start agents on a cheap tier and escalate one tier up when the orchestrator detects a soft failure (verification inconclusive, plan-check FLAG, etc.). Solves the "pay Opus rates as insurance" anti-pattern by making escalation observed-quality-driven. Architecture: - AGENT_DEFAULT_TIERS map (light/standard/heavy) — every agent in MODEL_PROFILES declares a default tier; tests assert coverage so adding a new agent without updating the map fails CI. - nextTier(currentTier) helper — light → standard → heavy → heavy (heavy stays at heavy; can't go further). - resolveModelForTier(cwd, agentType, attempt) — new resolver. The orchestrator tracks the attempt counter and passes 0 for the first spawn, 1+ on escalation. The resolver caps internally at max_escalations so the orchestrator can blindly bump the counter. - Schema validation: dynamic_routing.enabled / escalate_on_failure / max_escalations / tier_models.<light|standard|heavy>. Unknown tiers and unknown sub-keys rejected at config-set time. - SDK schema mirror updated to keep CJS/SDK in lockstep (#2653). Resolution precedence (highest → lowest): 1. model_overrides[<agent>] (full IDs accepted) 2. dynamic_routing.tier_models[<tier>] (NEW; escalation-aware) 3. models[<phase_type>] (#3023 phase-type map) 4. model_profile (per-agent column) 5. Runtime default Backward compatibility: dynamic_routing is disabled by default (enabled: false or block omitted). resolveModelForTier short- circuits to resolveModelInternal in that case, so callers can adopt unconditionally without breaking existing behavior. This PR delivers the JS-layer infrastructure: schema + tier map + resolver. Orchestrator adoption (workflow markdown updates that detect soft failures and call resolveModelForTier with attempt+1) is incremental follow-up — verifier / plan-checker / integration- checker each adopt the protocol when ready. Tests (23 cases, all structural-IR — no stdout grep): - Schema invariants: AGENT_DEFAULT_TIERS coverage, VALID_AGENT_TIERS exact match, every assignment uses a valid tier - nextTier helper: light→standard→heavy→heavy, null on invalid input - Disabled mode: no block + enabled:false both no-op (back-compat) - Enabled mode: attempt=0 returns default tier model, attempt=1 escalates, beyond max_escalations caps, heavy agents stay heavy, default max_escalations=1 when omitted - Precedence: per-agent override beats dynamic_routing, dynamic_routing beats phase-type models - Validation: every settings key accepted, unknown tiers/sub-keys rejected, bare `dynamic_routing` rejected as config-set target Documentation: - get-shit-done/references/model-profiles.md — full reference section - docs/CONFIGURATION.md — full settings table + escalation flow - docs/USER-GUIDE.md — task-oriented "Cheap-by-default" section - docs/FEATURES.md — config row cross-link Verification: - 23/23 pass on regression test - 6843/6843 full suite (23 net new from 6820) - lint-no-source-grep clean (376 test files) - SDK schema mirror keeps CJS/SDK in sync per #2653 parity test Closes #3024 * fix(#3024): honor escalate_on_failure:false + 3 CR follow-ups CodeRabbit on PR #3031 (4 findings — 1 Major + 2 Minor + 1 Nitpick): 1. **Major (inline)** — get-shit-done/bin/lib/core.cjs:1668 resolveModelForTier ignored dynamic_routing.escalate_on_failure. When the user set it to false, escalation should be disabled, but the resolver only checked attempt/max_escalations. An orchestrator that always passes attempt+1 on retry would silently escalate despite the user opting out. Fix: gate effectiveAttempt on `dr.escalate_on_failure !== false` so false short-circuits every attempt back to the default tier. 2. **Minor (inline)** — docs/CONFIGURATION.md:123-126 The dynamic_routing rows in the Core Settings table had 4 cells instead of 5 (missing the Options column), breaking the table structure. Added explicit Options values for enabled / escalate_on_failure / max_escalations rows. 3. **Minor (outside-diff)** — references/model-profiles.md:179-195 "Resolution Logic" sketch was pre-#3024 and didn't include dynamic_routing in the precedence ladder. Updated to a 6-step block with dynamic_routing at step 3 (between override and phase-type). 4. **Nitpick** — tests/feat-3024-dynamic-routing.test.cjs:189+ Tests used `if (lightAgent) { ... }` guards that silent-pass when AGENT_DEFAULT_TIERS drifts. Replaced all 5 conditional skips with `assert.ok(lightAgent, '...')` preconditions so a tier-mapping change surfaces as a test failure. Plus: 2 new regression tests for the Major fix: - escalate_on_failure:false caps every attempt at default tier - escalate_on_failure:true (explicit) still escalates normally Verification: - 25/25 pass on regression test (23 prior + 2 escalate_on_failure) - 6845/6845 full suite (2 net new) - lint-no-source-grep clean * docs(#3024): align precedence + add fence language tags (CR follow-up) CodeRabbit (3 minor): 1. docs/CONFIGURATION.md:691 — "Per-Phase-Type Models → Resolution precedence" was a 4-step block written pre-#3024; readers got contradictory rules between the per-phase-type section and the later dynamic_routing section. Updated to the same 5-step ladder with dynamic_routing at step 2, and noted that dynamic_routing is disabled by default so this section's behavior is unchanged when the kill-switch is off. 2. docs/CONFIGURATION.md:770 — escalation-flow code fence missing language tag (MD040). Added `text`. 3. references/model-profiles.md:184 — resolution-ladder code fence missing language tag (MD040). Added `text`. No code changes; docs only. Verification: regression test still 25/25. * docs(#3024): clarify precedence prose — five layers, not four (CR nitpick) CodeRabbit nitpick: the "Per-Phase-Type Models → Resolution precedence" prose said "The four layers compose..." but the ladder above lists five (including Runtime default). Also "dynamic_routing escalates per-attempt above all of them" misreads as suggesting dynamic_routing wins over model_overrides — actually overrides still win at step 1. Reworded top-down so the precedence direction is unambiguous: - model_profile = base - models = phase-level override - dynamic_routing = per-attempt escalation - model_overrides = per-agent exception (top) - runtime default = fallback No code changes; docs only. * docs(#3024): note escalate_on_failure:false in escalation-flow diagram (CR) CodeRabbit nitpick: the escalation-flow diagram in docs/CONFIGURATION.md described the soft-failure → respawn → tier_models[next_tier_up] path, but didn't surface the `dynamic_routing.escalate_on_failure: false` kill-switch right next to it. Users reading the flow diagram (which is the canonical place to understand attempt behavior) wouldn't see that the kill-switch overrides the soft-failure branch. Added a one-paragraph note immediately after the flow listing, before the tier-sequence example, so the kill-switch is visible exactly where users decide whether escalation will happen. No code changes; docs only.
194 lines
8.8 KiB
JavaScript
194 lines
8.8 KiB
JavaScript
/**
|
|
* Mapping of GSD agent to model for each profile.
|
|
*
|
|
* Should be in sync with the profiles table in `get-shit-done/references/model-profiles.md`. But
|
|
* possibly worth making this the single source of truth at some point, and removing the markdown
|
|
* reference table in favor of programmatically determining the model to use for an agent (which
|
|
* would be faster, use fewer tokens, and be less error-prone).
|
|
*/
|
|
const MODEL_PROFILES = {
|
|
'gsd-planner': { quality: 'opus', balanced: 'opus', budget: 'sonnet', adaptive: 'opus' },
|
|
'gsd-roadmapper': { quality: 'opus', balanced: 'sonnet', budget: 'sonnet', adaptive: 'sonnet' },
|
|
'gsd-executor': { quality: 'opus', balanced: 'sonnet', budget: 'sonnet', adaptive: 'sonnet' },
|
|
'gsd-phase-researcher': { quality: 'opus', balanced: 'sonnet', budget: 'haiku', adaptive: 'sonnet' },
|
|
'gsd-project-researcher': { quality: 'opus', balanced: 'sonnet', budget: 'haiku', adaptive: 'sonnet' },
|
|
'gsd-research-synthesizer': { quality: 'sonnet', balanced: 'sonnet', budget: 'haiku', adaptive: 'haiku' },
|
|
'gsd-debugger': { quality: 'opus', balanced: 'sonnet', budget: 'sonnet', adaptive: 'opus' },
|
|
'gsd-codebase-mapper': { quality: 'sonnet', balanced: 'haiku', budget: 'haiku', adaptive: 'haiku' },
|
|
'gsd-verifier': { quality: 'sonnet', balanced: 'sonnet', budget: 'haiku', adaptive: 'sonnet' },
|
|
'gsd-plan-checker': { quality: 'sonnet', balanced: 'sonnet', budget: 'haiku', adaptive: 'haiku' },
|
|
'gsd-integration-checker': { quality: 'sonnet', balanced: 'sonnet', budget: 'haiku', adaptive: 'haiku' },
|
|
'gsd-nyquist-auditor': { quality: 'sonnet', balanced: 'sonnet', budget: 'haiku', adaptive: 'haiku' },
|
|
'gsd-pattern-mapper': { quality: 'sonnet', balanced: 'sonnet', budget: 'haiku', adaptive: 'haiku' },
|
|
'gsd-ui-researcher': { quality: 'opus', balanced: 'sonnet', budget: 'haiku', adaptive: 'sonnet' },
|
|
'gsd-ui-checker': { quality: 'sonnet', balanced: 'sonnet', budget: 'haiku', adaptive: 'haiku' },
|
|
'gsd-ui-auditor': { quality: 'sonnet', balanced: 'sonnet', budget: 'haiku', adaptive: 'haiku' },
|
|
'gsd-doc-writer': { quality: 'opus', balanced: 'sonnet', budget: 'haiku', adaptive: 'sonnet' },
|
|
'gsd-doc-verifier': { quality: 'sonnet', balanced: 'sonnet', budget: 'haiku', adaptive: 'haiku' },
|
|
};
|
|
const VALID_PROFILES = [...Object.keys(MODEL_PROFILES['gsd-planner']), 'inherit'];
|
|
|
|
/**
|
|
* #3023 — Phase-type → agent mapping table.
|
|
*
|
|
* Lets users tune model selection at the *phase-type* level (planning,
|
|
* research, execution, verification, ...) instead of per-agent. Maps
|
|
* each agent in MODEL_PROFILES to exactly one phase-type so resolution
|
|
* is deterministic.
|
|
*
|
|
* Adding a new agent to MODEL_PROFILES requires adding an entry here too;
|
|
* tests/feat-3023-phase-type-models.test.cjs asserts coverage.
|
|
*
|
|
* Phase-type semantics:
|
|
* - planning: produces the plan (PLAN.md, ROADMAP.md, patterns)
|
|
* - discuss: collaborative scoping (no subagent today; reserved
|
|
* for orchestrators that may spawn one)
|
|
* - research: gathers external/codebase context (RESEARCH.md)
|
|
* - execution: implements the plan or writes user-facing artifacts
|
|
* - verification: checks correctness (VERIFICATION.md, audits)
|
|
* - completion: post-execution wrap-up (no subagent today; reserved)
|
|
*/
|
|
const AGENT_TO_PHASE_TYPE = {
|
|
// Planning — produces the plan / roadmap / pattern map
|
|
'gsd-planner': 'planning',
|
|
'gsd-roadmapper': 'planning',
|
|
'gsd-pattern-mapper': 'planning',
|
|
// Research — external/codebase information gathering
|
|
'gsd-phase-researcher': 'research',
|
|
'gsd-project-researcher': 'research',
|
|
'gsd-research-synthesizer': 'research',
|
|
'gsd-codebase-mapper': 'research',
|
|
'gsd-ui-researcher': 'research',
|
|
// Execution — implementation, debugging, doc writing
|
|
'gsd-executor': 'execution',
|
|
'gsd-debugger': 'execution',
|
|
'gsd-doc-writer': 'execution',
|
|
// Verification — correctness checks, audits, gap analysis
|
|
'gsd-verifier': 'verification',
|
|
'gsd-plan-checker': 'verification',
|
|
'gsd-integration-checker': 'verification',
|
|
'gsd-nyquist-auditor': 'verification',
|
|
'gsd-ui-checker': 'verification',
|
|
'gsd-ui-auditor': 'verification',
|
|
'gsd-doc-verifier': 'verification',
|
|
};
|
|
|
|
/**
|
|
* The six phase-type slots accepted in `.planning/config.json` `models`
|
|
* block. `discuss` and `completion` are reserved — no current agent maps
|
|
* to them today — so users can pre-configure those slots without
|
|
* breaking validation when an orchestrator starts honoring them.
|
|
*/
|
|
const VALID_PHASE_TYPES = new Set([
|
|
'planning', 'discuss', 'research', 'execution', 'verification', 'completion',
|
|
]);
|
|
|
|
/**
|
|
* #3024 — Per-agent default tier for dynamic routing.
|
|
*
|
|
* Each agent declares a default routing tier (light/standard/heavy)
|
|
* that the dynamic-routing resolver uses to pick from
|
|
* `dynamic_routing.tier_models[tier]` on the first attempt. On
|
|
* orchestrator-detected soft failure, the resolver escalates to the
|
|
* next tier up (capped at `max_escalations`).
|
|
*
|
|
* Tier semantics:
|
|
* - light: cheap/fast — pure mappers/scanners, low-stakes verifiers
|
|
* - standard: default workhorse — most researchers/writers/checkers
|
|
* - heavy: deep reasoning — planners/debuggers; can't escalate further
|
|
*
|
|
* Adding a new agent to MODEL_PROFILES requires adding an entry here too;
|
|
* tests/feat-3024-dynamic-routing.test.cjs asserts coverage.
|
|
*/
|
|
const AGENT_DEFAULT_TIERS = {
|
|
// Heavy — deep reasoning, planning, hard debugging
|
|
'gsd-planner': 'heavy',
|
|
'gsd-roadmapper': 'heavy',
|
|
'gsd-debugger': 'heavy',
|
|
// Standard — default workhorse: research, writing, primary verification
|
|
'gsd-executor': 'standard',
|
|
'gsd-phase-researcher': 'standard',
|
|
'gsd-project-researcher': 'standard',
|
|
'gsd-verifier': 'standard',
|
|
'gsd-doc-writer': 'standard',
|
|
'gsd-ui-researcher': 'standard',
|
|
// Light — fast scanners, structural mappers, low-stakes audits
|
|
'gsd-codebase-mapper': 'light',
|
|
'gsd-pattern-mapper': 'light',
|
|
'gsd-research-synthesizer': 'light',
|
|
'gsd-plan-checker': 'light',
|
|
'gsd-integration-checker': 'light',
|
|
'gsd-nyquist-auditor': 'light',
|
|
'gsd-ui-checker': 'light',
|
|
'gsd-ui-auditor': 'light',
|
|
'gsd-doc-verifier': 'light',
|
|
};
|
|
|
|
/**
|
|
* The three valid agent tier slots for dynamic routing. Used to
|
|
* validate `dynamic_routing.tier_models.<tier>` keys at config-set
|
|
* time and the AGENT_DEFAULT_TIERS values at startup.
|
|
*/
|
|
const VALID_AGENT_TIERS = new Set(['light', 'standard', 'heavy']);
|
|
|
|
/**
|
|
* Tier escalation order: light → standard → heavy.
|
|
* `nextTier(currentTier)` returns the tier one step up. `heavy` stays
|
|
* at heavy (no tier above). Returns null for invalid input so callers
|
|
* can detect mis-config rather than silently degrade.
|
|
*/
|
|
const _TIER_ESCALATION = { light: 'standard', standard: 'heavy', heavy: 'heavy' };
|
|
function nextTier(currentTier) {
|
|
if (typeof currentTier !== 'string') return null;
|
|
return _TIER_ESCALATION[currentTier] || null;
|
|
}
|
|
|
|
/**
|
|
* Formats the agent-to-model mapping as a human-readable table (in string format).
|
|
*
|
|
* @param {Object<string, string>} agentToModelMap - A mapping from agent to model
|
|
* @returns {string} A formatted table string
|
|
*/
|
|
function formatAgentToModelMapAsTable(agentToModelMap) {
|
|
const agentWidth = Math.max('Agent'.length, ...Object.keys(agentToModelMap).map((a) => a.length));
|
|
const modelWidth = Math.max(
|
|
'Model'.length,
|
|
...Object.values(agentToModelMap).map((m) => m.length)
|
|
);
|
|
const sep = '─'.repeat(agentWidth + 2) + '┼' + '─'.repeat(modelWidth + 2);
|
|
const header = ' ' + 'Agent'.padEnd(agentWidth) + ' │ ' + 'Model'.padEnd(modelWidth);
|
|
let agentToModelTable = header + '\n' + sep + '\n';
|
|
for (const [agent, model] of Object.entries(agentToModelMap)) {
|
|
agentToModelTable += ' ' + agent.padEnd(agentWidth) + ' │ ' + model.padEnd(modelWidth) + '\n';
|
|
}
|
|
return agentToModelTable;
|
|
}
|
|
|
|
/**
|
|
* Returns a mapping from agent to model for the given model profile.
|
|
*
|
|
* @param {string} normalizedProfile - The normalized (lowercase and trimmed) profile name
|
|
* @returns {Object<string, string>} A mapping from agent to model for the given profile
|
|
*/
|
|
function getAgentToModelMapForProfile(normalizedProfile) {
|
|
const agentToModelMap = {};
|
|
for (const [agent, profileToModelMap] of Object.entries(MODEL_PROFILES)) {
|
|
agentToModelMap[agent] = normalizedProfile === 'inherit'
|
|
? 'inherit'
|
|
: profileToModelMap[normalizedProfile];
|
|
}
|
|
return agentToModelMap;
|
|
}
|
|
|
|
module.exports = {
|
|
MODEL_PROFILES,
|
|
VALID_PROFILES,
|
|
AGENT_TO_PHASE_TYPE,
|
|
VALID_PHASE_TYPES,
|
|
AGENT_DEFAULT_TIERS,
|
|
VALID_AGENT_TIERS,
|
|
nextTier,
|
|
formatAgentToModelMapAsTable,
|
|
getAgentToModelMapForProfile,
|
|
};
|