diff --git a/.changeset/kind-sloths-swim.md b/.changeset/kind-sloths-swim.md new file mode 100644 index 000000000..5ff41a8b2 --- /dev/null +++ b/.changeset/kind-sloths-swim.md @@ -0,0 +1,5 @@ +--- +type: Fixed +pr: 4396 +--- +**Explicit model pins now hold on the Claude runtime** — set `model_profile_overrides.claude.` (e.g. pin the opus tier to `claude-opus-4-7`) and the resolver silently returned the bare tier alias anyway, and a fully-qualified Claude model ID in `model_overrides` was warn-dropped to tier resolution even though the configuration docs promise any fully-qualified model ID is valid; both are now resolved as configured (values naming the current tier default still collapse to their alias, so nothing changes for unpinned installs), and the docs now state the claude-runtime pin contract including the `fable` alias. (#4192) diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index 02e7f5700..9b098536f 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -1564,7 +1564,9 @@ Override specific agents without changing the entire profile: } ``` -Valid override values: `opus`, `sonnet`, `haiku`, `inherit`, or any fully-qualified model ID (e.g., `"openai/o3"`, `"google/gemini-2.5-pro"`). +Valid override values: `opus`, `sonnet`, `haiku`, `fable`, `inherit`, or any fully-qualified model ID (e.g., `"openai/o3"`, `"google/gemini-2.5-pro"`). + +On the Claude runtime, fully-qualified Claude model IDs are honored as explicit generation pins (#4192): an ID that names the current tier default (e.g. `"claude-sonnet-5"`) collapses to its tier alias — the same model in the form Claude Code's Agent tool always accepts — while any other ID (e.g. `"claude-opus-4-7"`) is resolved verbatim, so the pinned generation is what `resolve-model` reports. GSD emits a warn-once stderr breadcrumb for verbatim pins, because Claude Code setups whose Agent tool accepts only tier aliases will not honor a full ID. `fable` is a Claude Code Agent-tool alias, not a GSD profile tier: it is valid in `model_overrides` but has no column in the profile table. `model_overrides` can be set in either `.planning/config.json` (per-project) or `~/.gsd/defaults.json` (global). Per-project entries win on conflict and @@ -2095,15 +2097,20 @@ When `runtime` is set, profile tiers (`opus`/`sonnet`/`haiku`) resolve to runtim This resolves `gsd-planner` → `gpt-5.6-sol` (xhigh), `gsd-executor` → `gpt-5.6-terra` (medium), `gsd-codebase-mapper` → `gpt-5.6-luna` (medium). Codex skills pass each resolved `model` and `reasoning_effort` to `spawn_agent` when its visible schema advertises the corresponding field; otherwise they omit the field and inherit the session/static agent configuration. -**Claude example** — explicit opt-in resolves to full Claude IDs (no `resolve_model_ids: true` needed): +**Claude example** — pin a tier's generation without giving up tier-based profiles (#4192): ```json { "runtime": "claude", - "model_profile": "quality" + "model_profile": "quality", + "model_profile_overrides": { + "claude": { "opus": "claude-opus-4-7" } + } } ``` +On the Claude runtime, tier resolution stays on Claude Code's adaptive tier aliases (`opus` / `sonnet` / `haiku`) unless you override a tier. An override value that names the current tier default collapses back to its alias (the same model in the always-accepted form); any other value — a pinned older generation such as `claude-opus-4-7`, a bare alias repointing the tier, or a non-Anthropic model id — is resolved verbatim, so `resolve-model` reports exactly what the profile pins. Setting `runtime: "claude"` alone (no overrides) changes nothing: aliases resolve exactly as they do with the key absent. `resolve_model_ids: true` remains the global switch for materializing full IDs on every agent. + **Per-runtime overrides** — replace one or more tier defaults: ```json @@ -2122,8 +2129,8 @@ This resolves `gsd-planner` → `gpt-5.6-sol` (xhigh), `gsd-executor` → `gpt-5 **Precedence (highest to lowest):** 1. `model_overrides[]` — explicit per-agent ID always wins. -2. **Runtime-aware tier resolution** (this section) — when `runtime` is set and profile is not `inherit`. -3. `resolve_model_ids: "omit"` — returns empty string when no `runtime` is set. +2. **Runtime-aware tier resolution** (this section) — when `runtime` is set and profile is not `inherit`. On non-Claude runtimes this is the built-in tier map merged with your `model_profile_overrides`; on the Claude runtime it applies only the `model_profile_overrides.claude.` entry you set (#4192) — never the built-in defaults, so unpinned installs keep resolving aliases. +3. `resolve_model_ids: "omit"` — returns empty string when no `runtime` is set (an explicit project-level `"omit"` wins over a `claude` tier override too). 4. Claude-native default — `model_profile` tier as alias (current default). 5. `inherit` — propagates literal `inherit` for `Task(model="inherit")` semantics. diff --git a/docs/how-to/configure-model-profiles.md b/docs/how-to/configure-model-profiles.md index 5e10da375..fc720a256 100644 --- a/docs/how-to/configure-model-profiles.md +++ b/docs/how-to/configure-model-profiles.md @@ -51,7 +51,9 @@ If a single agent needs a different tier without changing the whole profile, use } ``` -Valid values: `opus`, `sonnet`, `haiku`, `inherit`, or any fully-qualified model ID (e.g. `"openai/o3"`, `"google/gemini-2.5-pro"`). +Valid values: `opus`, `sonnet`, `haiku`, `fable`, `inherit`, or any fully-qualified model ID (e.g. `"openai/o3"`, `"google/gemini-2.5-pro"`). + +On the Claude runtime, fully-qualified Claude model IDs act as explicit generation pins (#4192): an ID naming the current tier default (e.g. `"claude-sonnet-5"`) resolves to its tier alias — the same model in the form Claude Code's Agent tool always accepts — while any other ID (e.g. `"claude-opus-4-7"`) resolves verbatim, with a warn-once stderr note that setups accepting only tier aliases will not honor a full ID. `fable` is a Claude Code Agent-tool alias, not a GSD profile tier: valid here, but it has no column in the profile table. To pin a generation for a whole tier instead of one agent, use `model_profile_overrides` (see below). `model_overrides` can be set per-project in `.planning/config.json` or globally in `~/.gsd/defaults.json`. Per-project entries win on conflict; non-conflicting global entries are preserved. @@ -303,8 +305,9 @@ When multiple layers apply, the resolver picks the highest-priority entry: 1. model_overrides[] — per-agent; full IDs; targeted exception 2. dynamic_routing.tier_models[] — when enabled; escalates on soft failure 3. models[] — coarse phase-level tier -4. model_profile (per-agent column) — global tier strategy -5. Runtime default — when nothing else applies +4. model_profile_overrides.. — per-tier model override (#4192: honored on the claude runtime too) +5. model_profile (per-agent column) — global tier strategy +6. Runtime default — when nothing else applies ``` --- @@ -317,6 +320,7 @@ When multiple layers apply, the resolver picks the highest-priority entry: | Coarse phase-level tuning ("Opus for planning") | `models.` | | Per-agent precision ("force Haiku on the codebase mapper") | `model_overrides[]` | | A fully-qualified model ID for a specific agent | `model_overrides[]: "openai/gpt-5"` | +| Pin a tier's generation on Claude Code (e.g. executor stays on Opus 4.7) | `model_profile_overrides.claude.: "claude-opus-4-7"` | | Start cheap, escalate only on failure | `dynamic_routing` | | All agents follow the session model (non-Anthropic provider) | `model_profile: "inherit"` | diff --git a/gsd-core/references/model-profiles.md b/gsd-core/references/model-profiles.md index 9743ac5bc..e6034872e 100644 --- a/gsd-core/references/model-profiles.md +++ b/gsd-core/references/model-profiles.md @@ -58,6 +58,10 @@ Model profiles control which Claude model each GSD agent uses. This allows balan 3. **Profile table** — the per-agent column from the active `model_profile` 4. **Runtime default** — when nothing else applies +Steps 2–4 select the *tier*; a `model_profile_overrides..` entry then +maps that tier to a concrete model (#4192 — honored on the claude runtime as well, so +pinning composes with tiering instead of replacing it). + ### Why two layers above the profile? - **Profile** is a global tier strategy (everyone runs balanced). @@ -215,8 +219,11 @@ is (highest → lowest): (see §Dynamic Routing — escalation steps tier up per attempt counter) 4. If no dynamic_routing match, check models[phase_type] for a phase-type tier (see §Per-Phase-Type Model Map for the agent → phase-type mapping) -5. If no phase-type slot, look up agent in profile table -6. Pass model parameter to Task call +5. Check model_profile_overrides.. for a per-tier model override + (honored on the claude runtime too — #4192; verbatim unless it maps to the + current tier alias) +6. If no phase-type slot, look up agent in profile table +7. Pass model parameter to Task call ``` `model` and `effort` resolve through different mechanisms at different @@ -246,7 +253,9 @@ Override specific agents without changing the entire profile: } ``` -Overrides take precedence over the profile. Valid values: `opus`, `sonnet`, `haiku`, `inherit`, or any fully-qualified model ID (e.g., `"o3"`, `"openai/o3"`, `"google/gemini-2.5-pro"`). +Overrides take precedence over the profile. Valid values: `opus`, `sonnet`, `haiku`, `fable`, `inherit`, or any fully-qualified model ID (e.g., `"o3"`, `"openai/o3"`, `"google/gemini-2.5-pro"`). `fable` is a Claude Code Agent-tool alias, not a GSD profile tier — it has no column in the profile table above. + +On the Claude runtime, fully-qualified Claude model IDs are honored as explicit generation pins (#4192): an ID that names the current tier default (e.g. `"claude-sonnet-5"`) resolves to its tier alias — the same model in the form the Agent tool always accepts — while any other ID (e.g. `"claude-opus-4-7"`) resolves verbatim, with a warn-once stderr note that setups accepting only tier aliases will not honor a full ID. To pin a generation for a whole tier rather than one agent, set `model_profile_overrides.claude.` (see docs/CONFIGURATION.md — Runtime-Aware Profiles). ## Switching Profiles diff --git a/src/model-resolver.cts b/src/model-resolver.cts index 680bde926..939b55707 100644 --- a/src/model-resolver.cts +++ b/src/model-resolver.cts @@ -157,6 +157,65 @@ function _resolveRuntimeTier(config: Record, tier: string): Tie }); } +/** + * #4192 — Resolve the claude-runtime TIER OVERRIDE model for (config, tier). + * + * Step 3's runtime-aware resolution deliberately skips the claude runtime to + * preserve the alias-native posture (#1156/#2297): with no user override, the + * resolver must keep returning bare tier aliases, and the builtin claude tier + * map (`opus → claude-opus-4-8`, …) must never force full-ID emission on every + * default install. But `model_profile_overrides..` is a + * documented override point (docs/CONFIGURATION.md § Runtime-Aware Profiles) + * that `workflows/settings-advanced.md` actively writes for claude-runtime + * users — and #4192 Finding 1 measured the key inert on this runtime. + * + * This helper reads ONLY the user's override entry for the effective claude + * runtime and tier — never the builtin claude tier map — so an install with no + * override is byte-identical to before the fix. The runtime is resolved the + * same way steps 1-3 resolve it (config['runtime'], defaulting to 'claude'), + * NOT via resolveActiveRuntime (GSD_RUNTIME/marker): the value policy must + * key off the config the operator wrote, matching mapClaudeOverrideForRuntime. + * + * Value policy mirrors the model_overrides path (#2041/#4192): an override + * value that maps to a current tier alias collapses to that alias + * (byte-equivalent resolution, alias-form emission); anything else — a pinned + * older generation (`claude-opus-4-7`), a bare alias/tier repoint (`sonnet`), + * or a non-Claude vendor id (`openai/o3`) — is emitted verbatim as pinned. + * Malformed entries (no usable `model` string) return null so the caller falls + * through to normal alias resolution (ADR-443 D1: invalid values fall through). + */ +function resolveClaudeTierOverrideModel( + configRuntime: string | null | undefined, + tier: string | null | undefined, + overrides: Record | null | undefined, +): string | null { + if (!tier || tier === 'inherit') return null; + const effectiveRuntime = configRuntime || 'claude'; + if (effectiveRuntime !== 'claude') return null; // non-claude runtimes resolve at step 3 + const overridesMap = overrides as Record> | null | undefined; + if (!overridesMap || typeof overridesMap !== 'object') return null; + // Own-property guards throughout: both levels are config-supplied plain + // objects, so a prototype-chain key ("constructor", "toString") must not + // resolve an inherited member instead of falling through (same hardening as + // every other config-keyed lookup in this module). + const runtimeEntry = Object.hasOwn(overridesMap, effectiveRuntime) + ? overridesMap[effectiveRuntime] + : undefined; + if (!runtimeEntry || typeof runtimeEntry !== 'object') return null; + const userRaw = Object.hasOwn(runtimeEntry, tier) ? runtimeEntry[tier] : undefined; + if (userRaw === undefined || userRaw === null) return null; + const entry: Record = typeof userRaw === 'string' + ? { model: userRaw } + : (userRaw as Record); + if (!entry || typeof entry !== 'object') return null; + const model = entry['model']; + if (typeof model !== 'string' || model.length === 0) return null; + if (Object.hasOwn(CLAUDE_POLICY_ID_TO_ALIAS, model)) { + return CLAUDE_POLICY_ID_TO_ALIAS[model]; + } + return model; +} + // Reverse of the Claude tier-default IDs, plus the Fable alias which Claude // Code's Agent tool accepts but which is not a GSD model-profile tier (#1133). const CLAUDE_POLICY_ID_TO_ALIAS: Record = { @@ -189,7 +248,10 @@ function _resetModelPolicyWarningCacheForTests(): void { _modelPolicyUnmappableWarned.clear(); } -// Dedupe stderr warnings for unmappable model_overrides Claude IDs (#2041). +// Dedupe stderr warnings for unmappable model_overrides Claude IDs (#2041 / +// #4192). #2041 originally warned that such a value was being DROPPED to tier +// resolution; #4192 keeps the warn-once breadcrumb but changes the behavior to +// a verbatim pass-through, so the text now describes the pass-through. const _modelOverrideUnmappableWarned = new Set(); function warnModelOverrideUnmappable(agentType: string, overrideValue: string): void { const key = `${agentType}::${overrideValue}`; @@ -200,8 +262,9 @@ function warnModelOverrideUnmappable(agentType: string, overrideValue: string): // model's JSON result is parsed from stdout. const safe = overrideValue.length > 64 ? overrideValue.slice(0, 64) + '…' : overrideValue; process.stderr.write( - `gsd: warning — model_overrides value "${safe}" for ${agentType} ` + - `has no Claude agent alias; falling through to tier resolution.\n`, + `gsd: warning — model_overrides value "${safe}" for ${agentType} is a fully-qualified ` + + `Claude model ID with no tier alias; passing it through verbatim. Claude Code setups ` + + `whose Agent tool accepts only tier aliases will not honor it. (#4192)\n`, ); } @@ -213,12 +276,24 @@ function _resetModelOverrideWarningCacheForTests(): void { /** * #2041 — Map a `model_overrides` value to its Claude Agent-tool alias on the * claude runtime, mirroring the `model_policy` path (#1144). Claude Code's - * Agent tool `model` parameter documents only tier aliases (opus/sonnet/haiku/ - * fable); a full Claude model ID returned verbatim is silently dropped by the - * spawner. Returns the value to return verbatim, or null to signal "fall - * through to normal tier/dynamic-routing resolution" (used when a Claude full - * ID has no alias — matches model_policy's warn-and-fall-through). Non-Claude - * runtimes and non-Claude values always pass through verbatim. + * Agent tool `model` parameter documents tier aliases (opus/sonnet/haiku/ + * fable) as the always-accepted form. Returns the value to emit verbatim, or + * null to signal "fall through to normal tier/dynamic-routing resolution". + * Non-Claude runtimes and non-Claude values always pass through verbatim. + * + * #4192 — an unmappable `claude-*` value (a pinned generation that is not the + * current catalog default, e.g. `claude-opus-4-7`) is now PASSED THROUGH + * VERBATIM with a warn-once stderr breadcrumb, instead of being dropped to + * tier resolution. #2041's drop was correct when the value was plausibly a + * mis-typed current default, but for an explicit pin it silently UNPINNED the + * operator's choice — the resolver would report a tier the config never asked + * for, the exact "profile can misrepresent what actually runs" defect #4192 + * files. The documented contract ("any fully-qualified model ID", + * docs/CONFIGURATION.md § Per-Agent Overrides, + * gsd-core/references/model-profiles.md § Per-Agent Overrides) is restored: + * the pin is resolved as configured. Values that DO map to a current tier + * alias still collapse to that alias — byte-equivalent resolution, the #2041 + * protection preserved — and a mappable pin never warns. * * Hardening (code+security review): a `typeof` guard preserves the pre-fix * no-crash behavior if a malformed config surfaces a non-string value, and an @@ -243,8 +318,9 @@ function mapClaudeOverrideForRuntime( } if (CLAUDE_AGENT_ALIASES.has(override)) return override; if (override.startsWith('claude-')) { + // #4192: explicit generation pin — resolve as configured (see docblock). warnModelOverrideUnmappable(agentType, override); - return null; + return override; } return override; } @@ -507,6 +583,23 @@ function resolveModelInternal(cwd: string, agentType: string): string { return ''; } + // 4.5. Claude-runtime tier override (#4192 Finding 1). Sits AFTER the omit + // gate so an explicit project `resolve_model_ids:"omit"` still wins (#2297: + // explicit project omit is honored regardless of runtime), and BEFORE the + // alias return so a pinned generation is not re-collapsed to a tier alias or + // re-materialized to the LATEST catalog id by step 5's + // `resolve_model_ids:true` path. Fires ONLY when the user wrote a + // `model_profile_overrides.claude.` entry for this tier — see + // resolveClaudeTierOverrideModel for why the builtin map stays out. + if (tier && tier !== 'inherit') { + const claudeOverrideModel = resolveClaudeTierOverrideModel( + configRuntime, + tier, + config['model_profile_overrides'] as Record | null | undefined, + ); + if (claudeOverrideModel !== null) return claudeOverrideModel; + } + // 5. Profile lookup (Claude-native default). if (!agentModels) { return profile === 'quality' ? 'opus' diff --git a/tests/commands.test.cjs b/tests/commands.test.cjs index a299cdc1a..76a5e18fb 100644 --- a/tests/commands.test.cjs +++ b/tests/commands.test.cjs @@ -1415,6 +1415,38 @@ describe('resolve-model command', () => { assert.ok(output.model, 'should resolve a model'); }); + // #4192 (AC1, behavioral): a claude-runtime tier override must change the + // resolved output relative to the no-override control — the CLI surface the + // orchestrator reads is where the documented contract is observable. + test('claude-runtime tier override changes resolved output vs control (#4192)', () => { + fs.writeFileSync(path.join(tmpDir, '.planning', 'config.json'), JSON.stringify({ + model_profile: 'balanced', + model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } }, + })); + const pinned = runGsdTools('resolve-model gsd-planner', tmpDir); + assert.ok(pinned.success, `Command failed: ${pinned.error}`); + assert.strictEqual(JSON.parse(pinned.output).model, 'claude-opus-4-7'); + + fs.writeFileSync(path.join(tmpDir, '.planning', 'config.json'), JSON.stringify({ + model_profile: 'balanced', + })); + const control = runGsdTools('resolve-model gsd-planner', tmpDir); + assert.ok(control.success, `Command failed: ${control.error}`); + assert.strictEqual(JSON.parse(control.output).model, 'opus'); + }); + + // #4192 (AC2, behavioral): a per-agent fully-qualified Claude model ID is + // resolved as configured through the CLI surface. + test('per-agent fully-qualified claude ID resolves as configured (#4192)', () => { + fs.writeFileSync(path.join(tmpDir, '.planning', 'config.json'), JSON.stringify({ + model_profile: 'balanced', + model_overrides: { 'gsd-debugger': 'claude-opus-4-7' }, + })); + const result = runGsdTools('resolve-model gsd-debugger', tmpDir); + assert.ok(result.success, `Command failed: ${result.error}`); + assert.strictEqual(JSON.parse(result.output).model, 'claude-opus-4-7'); + }); + // #443: resolve-model now emits unified `effort` instead of `reasoning_effort`. // reasoning_effort was flavor-text (resolved but consumed by nobody); effort is // the wired, config-driven universal effort string for all runtimes. diff --git a/tests/model-resolver.test.cjs b/tests/model-resolver.test.cjs index 9e09dc9ee..98bb569fa 100644 --- a/tests/model-resolver.test.cjs +++ b/tests/model-resolver.test.cjs @@ -4123,16 +4123,19 @@ describe('#2041 model_overrides: Claude full ID → alias on claude runtime', () assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-executor'), 'claude-sonnet-5'); }); - // AC5: unmappable Claude full ID warns once + falls through to tier alias - test('model_overrides unmappable claude ID (claude-opus-4-5) falls through to tier alias on claude', () => { + // AC5 (#4192 revision): an unmappable Claude full ID — an explicit + // generation pin — is passed through VERBATIM with a warn-once breadcrumb, + // instead of being dropped to tier resolution (which silently unpinned the + // operator's explicit choice; see #4192 Finding 2). + test('model_overrides unmappable claude ID (claude-opus-4-5) passes through verbatim on claude', () => { resetRuntimeWarningCaches(); writeConfig(tmpDir, { runtime: 'claude', model_profile: 'balanced', model_overrides: { 'gsd-planner': 'claude-opus-4-5' }, }); - // gsd-planner balanced → opus tier; claude-opus-4-5 has no alias → warn + fall through → 'opus' - assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus'); + // gsd-planner balanced → opus tier; claude-opus-4-5 has no alias → warn + verbatim pin + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'claude-opus-4-5'); }); test('model_overrides unmappable claude ID emits a stderr warning exactly once (dedupe)', () => { @@ -4173,19 +4176,19 @@ describe('#2041 model_overrides: Claude full ID → alias on claude runtime', () assert.strictEqual(resolveModelForTier(tmpDir, 'gsd-executor', 0), 'claude-sonnet-5'); }); - // MEDIUM-1 (review): exercise the unmappable-override fall-through branch in - // resolveModelForTier (closes the mutation-score gap — a future refactor that - // accidentally returned the verbatim override instead of falling through - // would otherwise survive the suite). - test('resolveModelForTier unmappable claude ID falls through to tier alias on claude', () => { + // MEDIUM-1 (review, #4192 revision): exercise the unmappable-override + // branch in resolveModelForTier (keeps the mutation-score gap closed — the + // verbatim-pin return must survive a future refactor on the escalation path + // too, not just resolveModelInternal). + test('resolveModelForTier unmappable claude ID passes through verbatim on claude', () => { resetRuntimeWarningCaches(); writeConfig(tmpDir, { runtime: 'claude', model_profile: 'balanced', model_overrides: { 'gsd-planner': 'claude-opus-4-5' }, }); - // unmappable override → fall through → no dynamic_routing → resolveModelInternal → 'opus' - assert.strictEqual(resolveModelForTier(tmpDir, 'gsd-planner', 0), 'opus'); + // unmappable override → no dynamic_routing → resolveModelInternal → verbatim pin + assert.strictEqual(resolveModelForTier(tmpDir, 'gsd-planner', 0), 'claude-opus-4-5'); }); // LOW-2 (review): pin the case-sensitive contract — a case-variant like @@ -6321,3 +6324,346 @@ describe('#3007 PROPERTY: renderEffortForRuntime never renders a level the model ); }); }); + +// ─── #4192: claude-runtime generation pinning via explicit overrides ────────── +// +// Confirmed-bug scope (maintainer triage): Findings 1 and 2 — documented +// behavior the resolver does not implement on the claude runtime. +// +// F1 — model_profile_overrides.claude. was inert: step 3 of +// resolveModelInternal gated runtime-aware tier resolution on +// `configRuntime !== 'claude'`, so the only reader of the key was never +// consulted on claude, while settings-advanced.md writes it for +// claude-runtime users. +// F2 — fully-qualified claude-* IDs in model_overrides were warn-dropped to +// tier resolution (mapClaudeOverrideForRuntime unmappable branch, +// #2041), while the configuration reference and the shipped +// model-profiles reference both document "any fully-qualified model +// ID" as valid. +// +// Agreed contract (AC2, pinned here): an explicit pin is RESOLVED AS +// CONFIGURED. A claude-* value that maps to a current tier alias still +// collapses to that alias (the #2041 protection — byte-equivalent resolution); +// an unmappable one (a pinned older generation) is returned verbatim with a +// warn-once breadcrumb, because dropping it would silently unpin the operator's +// explicit choice — the exact "profile misrepresents what runs" defect of +// #4192. Unpinned resolution is byte-stable (control rows below). +describe('#4192 model_profile_overrides.claude.*: tier overrides honor pins on the claude runtime', () => { + const { createTempDir, resetRuntimeWarningCaches } = require('./helpers.cjs'); + let tmpDir; + const make = () => createTempDir('gsd-4192-tier-override-'); + const write = (cfg) => fs.writeFileSync( + path.join(tmpDir, '.planning', 'config.json'), JSON.stringify(cfg, null, 2), 'utf-8'); + + beforeEach(() => { + tmpDir = make(); + fs.mkdirSync(path.join(tmpDir, '.planning'), { recursive: true }); + resetRuntimeWarningCaches(); + }); + afterEach(() => { + cleanup(tmpDir); + resetRuntimeWarningCaches(); + }); + + // Row 1 — REGRESSION (failing-first): pinned generation honored, implicit claude runtime. + test('claude.opus = "claude-opus-4-7" pins the opus tier (implicit claude runtime)', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'claude-opus-4-7'); + }); + + // Row 2 — same with an explicit runtime key. + test('claude.opus pin honored with explicit runtime: "claude"', () => { + write({ + runtime: 'claude', + model_profile: 'balanced', + model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'claude-opus-4-7'); + }); + + // Row 3 — mappable override collapses to the alias (form parity with #2041 step 1). + test('claude.sonnet = "claude-sonnet-5" resolves to the "sonnet" alias', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { sonnet: 'claude-sonnet-5' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-executor'), 'sonnet'); + }); + + // Row 4 — fable-valued override maps through the fable alias. + test('claude.opus = "claude-fable-5" resolves to "fable"', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { opus: 'claude-fable-5' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'fable'); + }); + + // Row 5 — bare-alias / tier-repoint override passes through verbatim. + test('claude.opus = "sonnet" repoints the tier at the sonnet alias', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { opus: 'sonnet' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'sonnet'); + }); + + // Row 6 — non-Claude ID override passes through verbatim (docs: any fully-qualified ID). + test('claude.haiku = "openai/gpt-4o-mini" passes through verbatim on claude', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { haiku: 'openai/gpt-4o-mini' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-codebase-mapper'), 'openai/gpt-4o-mini'); + }); + + // Row 7 — object-form override (settings workflow accepts {model, reasoning_effort}). + test('claude.opus = { model: "claude-opus-4-7" } object form pins the tier', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { opus: { model: 'claude-opus-4-7' } } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'claude-opus-4-7'); + }); + + // Row 8 — CONTROL (AC1): no override → byte-identical alias resolution. + test('no model_profile_overrides → alias resolution unchanged', () => { + write({ model_profile: 'balanced' }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus'); + }); + + // Row 9 — CONTROL: overrides for another runtime never apply to claude. + test('codex-only overrides are inert on the claude runtime', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { codex: { opus: 'gpt-5-pro' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus'); + }); + + // Row 10 — CONTROL: override for a different tier than the agent's is inert for that agent. + test('claude.sonnet override does not touch an opus-tier agent', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { sonnet: 'claude-sonnet-4-6' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus'); + }); + + // Row 11 — CONTROL: inherit profile is immune to tier overrides. + test('model_profile: "inherit" + claude.opus pin → "inherit"', () => { + write({ + model_profile: 'inherit', + model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'inherit'); + }); + + // Row 12 — CONTROL: explicit project resolve_model_ids:"omit" beats the override (#2297). + test('project resolve_model_ids: "omit" + claude.opus pin → empty string', () => { + write({ + model_profile: 'balanced', + resolve_model_ids: 'omit', + model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), ''); + }); + + // Row 13 — CONTROL: model_overrides still wins over the tier override. + test('model_overrides beats model_profile_overrides.claude', () => { + write({ + model_profile: 'balanced', + model_overrides: { 'gsd-planner': 'haiku' }, + model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'haiku'); + }); + + // Row 15 — CONTROL: object override without a model key degrades to the alias. + test('claude.opus = { reasoning_effort } (no model) falls through to the alias', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { opus: { reasoning_effort: 'high' } } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus'); + }); + + // Row 16 — CONTROL: non-string/non-object value degrades to the alias. + test('claude.opus = 42 (malformed value) falls through to the alias', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { opus: 42 } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus'); + }); + + // Row 17 — CONTROL: empty-string value degrades to the alias. + test('claude.opus = "" falls through to the alias', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { opus: '' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus'); + }); + + // Row 26 — ADVERSARIAL: prototype-chain keys in the override map must not leak. + test('"constructor" as a claude override key does not resolve an inherited member', () => { + write({ + model_profile: 'balanced', + model_profile_overrides: { claude: { constructor: 'claude-opus-4-7' } }, + }); + // 'constructor' is not a tier; resolution must ignore it entirely and land + // on the profile alias, never on Function.prototype's members. + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus'); + }); + + // Row 30 — the pin wins over resolve_model_ids:true alias materialization. + test('claude.opus pin beats resolve_model_ids: true materialization', () => { + write({ + model_profile: 'balanced', + resolve_model_ids: true, + model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'claude-opus-4-7'); + }); +}); + +describe('#4192 model_overrides: fully-qualified claude IDs resolve as configured', () => { + const { createTempDir, resetRuntimeWarningCaches } = require('./helpers.cjs'); + let tmpDir; + const make = () => createTempDir('gsd-4192-agent-override-'); + const write = (cfg) => fs.writeFileSync( + path.join(tmpDir, '.planning', 'config.json'), JSON.stringify(cfg, null, 2), 'utf-8'); + + beforeEach(() => { + tmpDir = make(); + fs.mkdirSync(path.join(tmpDir, '.planning'), { recursive: true }); + resetRuntimeWarningCaches(); + }); + afterEach(() => { + cleanup(tmpDir); + resetRuntimeWarningCaches(); + }); + + // Row 18 — REGRESSION (failing-first): pinned generation honored, implicit claude runtime. + test('model_overrides "claude-opus-4-7" resolves verbatim (implicit claude runtime)', () => { + write({ + model_profile: 'balanced', + model_overrides: { 'gsd-debugger': 'claude-opus-4-7' }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-debugger'), 'claude-opus-4-7'); + }); + + // Row 18b — explicit runtime key. + test('model_overrides "claude-opus-4-7" resolves verbatim with runtime: "claude"', () => { + write({ + runtime: 'claude', + model_profile: 'balanced', + model_overrides: { 'gsd-debugger': 'claude-opus-4-7' }, + }); + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-debugger'), 'claude-opus-4-7'); + }); + + // Row 21 — warn-once breadcrumb on an unmappable pin (visibility, not a drop). + test('unmappable pin emits exactly one pass-through stderr warning (dedupe)', () => { + write({ + runtime: 'claude', + model_profile: 'balanced', + model_overrides: { 'gsd-debugger': 'claude-opus-4-7' }, + }); + const writes = []; + const original = process.stderr.write.bind(process.stderr); + process.stderr.write = (chunk) => { writes.push(String(chunk)); return true; }; + try { + resolveModelInternal(tmpDir, 'gsd-debugger'); + resolveModelInternal(tmpDir, 'gsd-debugger'); // dedupe must suppress + } finally { + process.stderr.write = original; + } + const warnings = writes.filter((w) => w.includes('model_overrides') && w.includes('claude-opus-4-7')); + assert.strictEqual(warnings.length, 1, + `expected exactly one override warning, got ${warnings.length}: ${JSON.stringify(writes)}`); + // The warning must describe pass-through, not a fall-through that no longer happens. + assert.ok(!warnings[0].includes('falling through'), + `warning must not claim a fall-through: ${warnings[0]}`); + }); + + // Row 21b — no warning for a value that needs no breadcrumb (mappable / non-claude). + test('mappable ID resolution emits no model_overrides warning', () => { + write({ + runtime: 'claude', + model_profile: 'balanced', + model_overrides: { 'gsd-debugger': 'claude-sonnet-5' }, + }); + const writes = []; + const original = process.stderr.write.bind(process.stderr); + process.stderr.write = (chunk) => { writes.push(String(chunk)); return true; }; + try { + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-debugger'), 'sonnet'); + } finally { + process.stderr.write = original; + } + assert.strictEqual(writes.filter((w) => w.includes('model_overrides')).length, 0); + }); + + // Row 22 — escalation path parity. + test('resolveModelForTier returns the pinned generation verbatim', () => { + write({ + runtime: 'claude', + model_profile: 'balanced', + model_overrides: { 'gsd-debugger': 'claude-opus-4-7' }, + }); + assert.strictEqual(resolveModelForTier(tmpDir, 'gsd-debugger', 0), 'claude-opus-4-7'); + }); + + // Row 24 — tier honesty signal unchanged (AC3): a raw pin carries no tier. + test('resolveTierFromConfig reports "unknown" for a raw pinned generation', () => { + write({ + runtime: 'claude', + model_profile: 'balanced', + model_overrides: { 'gsd-debugger': 'claude-opus-4-7' }, + }); + assert.strictEqual(resolveTierFromConfig( + JSON.parse(fs.readFileSync(path.join(tmpDir, '.planning', 'config.json'), 'utf-8')), + 'gsd-debugger'), 'unknown'); + }); + + // Row 25 — ADVERSARIAL: prototype-chain agentType must not leak through overrides. + test('agentType "toString" against model_overrides: {} stays on the unknown-agent path', () => { + write({ + model_profile: 'balanced', + model_overrides: {}, + }); + // Unknown agent + balanced profile → the hardcoded fallback alias, never + // an inherited Function.prototype member. + assert.strictEqual(resolveModelInternal(tmpDir, 'toString'), 'sonnet'); + }); + + // Row 28 — an oversized pin value survives resolution; any warning stays capped. + test('oversized unmappable pin resolves verbatim and warning text is capped at 64 chars', () => { + const longPin = 'claude-opus-' + '9'.repeat(80); + write({ + runtime: 'claude', + model_profile: 'balanced', + model_overrides: { 'gsd-debugger': longPin }, + }); + const writes = []; + const original = process.stderr.write.bind(process.stderr); + process.stderr.write = (chunk) => { writes.push(String(chunk)); return true; }; + try { + assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-debugger'), longPin); + } finally { + process.stderr.write = original; + } + const warnings = writes.filter((w) => w.includes('model_overrides')); + assert.strictEqual(warnings.length, 1); + // The rendered value inside the warning is the 64-char cap + ellipsis, not the full pin. + assert.ok(!warnings[0].includes(longPin), + `warning must not contain the uncapped pin: ${warnings[0]}`); + assert.ok(warnings[0].includes('claude-opus-' + '9'.repeat(52) + '…'), + `warning must contain the capped pin render: ${warnings[0]}`); + }); +});