* fix(#4192): honor explicit model pins on the claude runtime Two documented model-configuration contracts did not hold on the claude runtime (confirmed-bug scope from the issue triage): Finding 1 — model_profile_overrides.claude.<tier> was inert. Step 3 of resolveModelInternal gated runtime-aware tier resolution on configRuntime !== 'claude', so the key's only reader was never consulted, while workflows/settings-advanced.md writes it for claude-runtime users. A new step 4.5 resolves ONLY the user's override entry (never the builtin claude tier map, so unpinned installs keep resolving aliases). An override value that maps to a current tier alias collapses to that alias (byte-equivalent, the #2041 protection); anything else — a pinned older generation, a bare alias repoint, a non-Anthropic id — resolves verbatim. It sits after the resolve_model_ids:'omit' gate so an explicit project omit still wins (#2297) and before the alias return so resolve_model_ids:true cannot re-materialize the pin to the latest id. Finding 2 — fully-qualified claude-* ids in model_overrides were warn-dropped to tier resolution (mapClaudeOverrideForRuntime unmappable branch, #2041), while the docs promise any fully-qualified model id is valid. The unmappable branch now passes the pin through verbatim with a warn-once breadcrumb (text describes the pass-through). Dropping it silently unpinned the operator's explicit choice — the exact 'profile can misrepresent what actually runs' defect of #4192. Mappable ids and non-claude values behave exactly as before; resolveModelForTier shares the mapping; the tier honesty signal is unchanged (raw ids still report 'unknown'); the model_policy path is untouched. Docs updated to the agreed contract (CONFIGURATION.md false 'Claude example' corrected; how-to + shipped reference document the pin semantics, the fable alias, and the tier-override composition). * test(#4192): pin explicit model pin resolution on the claude runtime 28 failing-first rows across the resolver seam and the resolve-model CLI: pinned-generation fidelity (tier override + per-agent verbatim pins, object form, explicit runtime), unpinned controls byte-stable (no override, other runtime/tier, inherit, project omit, precedence), adversarial rows (prototype-chain keys, malformed values, warn-once dedupe, 64-char stderr cap), and behavioral AC1/AC2 rows through runGsdTools. The stale #2041 fall-through assertions now pin the pass-through contract; mappable-id collapse assertions unchanged. * chore(#4192): add changeset fragment * chore(#4192): backfill PR number in changeset fragment --------- Co-authored-by: ZCode <zcode@localhost>
This commit is contained in:
Tom Boucher
5
.changeset/kind-sloths-swim.md
Normal file
5
.changeset/kind-sloths-swim.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
type: Fixed
|
||||
pr: 4396
|
||||
---
|
||||
**Explicit model pins now hold on the Claude runtime** — set `model_profile_overrides.claude.<tier>` (e.g. pin the opus tier to `claude-opus-4-7`) and the resolver silently returned the bare tier alias anyway, and a fully-qualified Claude model ID in `model_overrides` was warn-dropped to tier resolution even though the configuration docs promise any fully-qualified model ID is valid; both are now resolved as configured (values naming the current tier default still collapse to their alias, so nothing changes for unpinned installs), and the docs now state the claude-runtime pin contract including the `fable` alias. (#4192)
|
||||
@@ -1564,7 +1564,9 @@ Override specific agents without changing the entire profile:
|
||||
}
|
||||
```
|
||||
|
||||
Valid override values: `opus`, `sonnet`, `haiku`, `inherit`, or any fully-qualified model ID (e.g., `"openai/o3"`, `"google/gemini-2.5-pro"`).
|
||||
Valid override values: `opus`, `sonnet`, `haiku`, `fable`, `inherit`, or any fully-qualified model ID (e.g., `"openai/o3"`, `"google/gemini-2.5-pro"`).
|
||||
|
||||
On the Claude runtime, fully-qualified Claude model IDs are honored as explicit generation pins (#4192): an ID that names the current tier default (e.g. `"claude-sonnet-5"`) collapses to its tier alias — the same model in the form Claude Code's Agent tool always accepts — while any other ID (e.g. `"claude-opus-4-7"`) is resolved verbatim, so the pinned generation is what `resolve-model` reports. GSD emits a warn-once stderr breadcrumb for verbatim pins, because Claude Code setups whose Agent tool accepts only tier aliases will not honor a full ID. `fable` is a Claude Code Agent-tool alias, not a GSD profile tier: it is valid in `model_overrides` but has no column in the profile table.
|
||||
|
||||
`model_overrides` can be set in either `.planning/config.json` (per-project)
|
||||
or `~/.gsd/defaults.json` (global). Per-project entries win on conflict and
|
||||
@@ -2095,15 +2097,20 @@ When `runtime` is set, profile tiers (`opus`/`sonnet`/`haiku`) resolve to runtim
|
||||
|
||||
This resolves `gsd-planner` → `gpt-5.6-sol` (xhigh), `gsd-executor` → `gpt-5.6-terra` (medium), `gsd-codebase-mapper` → `gpt-5.6-luna` (medium). Codex skills pass each resolved `model` and `reasoning_effort` to `spawn_agent` when its visible schema advertises the corresponding field; otherwise they omit the field and inherit the session/static agent configuration.
|
||||
|
||||
**Claude example** — explicit opt-in resolves to full Claude IDs (no `resolve_model_ids: true` needed):
|
||||
**Claude example** — pin a tier's generation without giving up tier-based profiles (#4192):
|
||||
|
||||
```json
|
||||
{
|
||||
"runtime": "claude",
|
||||
"model_profile": "quality"
|
||||
"model_profile": "quality",
|
||||
"model_profile_overrides": {
|
||||
"claude": { "opus": "claude-opus-4-7" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
On the Claude runtime, tier resolution stays on Claude Code's adaptive tier aliases (`opus` / `sonnet` / `haiku`) unless you override a tier. An override value that names the current tier default collapses back to its alias (the same model in the always-accepted form); any other value — a pinned older generation such as `claude-opus-4-7`, a bare alias repointing the tier, or a non-Anthropic model id — is resolved verbatim, so `resolve-model` reports exactly what the profile pins. Setting `runtime: "claude"` alone (no overrides) changes nothing: aliases resolve exactly as they do with the key absent. `resolve_model_ids: true` remains the global switch for materializing full IDs on every agent.
|
||||
|
||||
**Per-runtime overrides** — replace one or more tier defaults:
|
||||
|
||||
```json
|
||||
@@ -2122,8 +2129,8 @@ This resolves `gsd-planner` → `gpt-5.6-sol` (xhigh), `gsd-executor` → `gpt-5
|
||||
**Precedence (highest to lowest):**
|
||||
|
||||
1. `model_overrides[<agent>]` — explicit per-agent ID always wins.
|
||||
2. **Runtime-aware tier resolution** (this section) — when `runtime` is set and profile is not `inherit`.
|
||||
3. `resolve_model_ids: "omit"` — returns empty string when no `runtime` is set.
|
||||
2. **Runtime-aware tier resolution** (this section) — when `runtime` is set and profile is not `inherit`. On non-Claude runtimes this is the built-in tier map merged with your `model_profile_overrides`; on the Claude runtime it applies only the `model_profile_overrides.claude.<tier>` entry you set (#4192) — never the built-in defaults, so unpinned installs keep resolving aliases.
|
||||
3. `resolve_model_ids: "omit"` — returns empty string when no `runtime` is set (an explicit project-level `"omit"` wins over a `claude` tier override too).
|
||||
4. Claude-native default — `model_profile` tier as alias (current default).
|
||||
5. `inherit` — propagates literal `inherit` for `Task(model="inherit")` semantics.
|
||||
|
||||
|
||||
@@ -51,7 +51,9 @@ If a single agent needs a different tier without changing the whole profile, use
|
||||
}
|
||||
```
|
||||
|
||||
Valid values: `opus`, `sonnet`, `haiku`, `inherit`, or any fully-qualified model ID (e.g. `"openai/o3"`, `"google/gemini-2.5-pro"`).
|
||||
Valid values: `opus`, `sonnet`, `haiku`, `fable`, `inherit`, or any fully-qualified model ID (e.g. `"openai/o3"`, `"google/gemini-2.5-pro"`).
|
||||
|
||||
On the Claude runtime, fully-qualified Claude model IDs act as explicit generation pins (#4192): an ID naming the current tier default (e.g. `"claude-sonnet-5"`) resolves to its tier alias — the same model in the form Claude Code's Agent tool always accepts — while any other ID (e.g. `"claude-opus-4-7"`) resolves verbatim, with a warn-once stderr note that setups accepting only tier aliases will not honor a full ID. `fable` is a Claude Code Agent-tool alias, not a GSD profile tier: valid here, but it has no column in the profile table. To pin a generation for a whole tier instead of one agent, use `model_profile_overrides` (see below).
|
||||
|
||||
`model_overrides` can be set per-project in `.planning/config.json` or globally in `~/.gsd/defaults.json`. Per-project entries win on conflict; non-conflicting global entries are preserved.
|
||||
|
||||
@@ -303,8 +305,9 @@ When multiple layers apply, the resolver picks the highest-priority entry:
|
||||
1. model_overrides[<agent>] — per-agent; full IDs; targeted exception
|
||||
2. dynamic_routing.tier_models[<tier>] — when enabled; escalates on soft failure
|
||||
3. models[<phase_type>] — coarse phase-level tier
|
||||
4. model_profile (per-agent column) — global tier strategy
|
||||
5. Runtime default — when nothing else applies
|
||||
4. model_profile_overrides.<runtime>.<tier> — per-tier model override (#4192: honored on the claude runtime too)
|
||||
5. model_profile (per-agent column) — global tier strategy
|
||||
6. Runtime default — when nothing else applies
|
||||
```
|
||||
|
||||
---
|
||||
@@ -317,6 +320,7 @@ When multiple layers apply, the resolver picks the highest-priority entry:
|
||||
| Coarse phase-level tuning ("Opus for planning") | `models.<phase_type>` |
|
||||
| Per-agent precision ("force Haiku on the codebase mapper") | `model_overrides[<agent>]` |
|
||||
| A fully-qualified model ID for a specific agent | `model_overrides[<agent>]: "openai/gpt-5"` |
|
||||
| Pin a tier's generation on Claude Code (e.g. executor stays on Opus 4.7) | `model_profile_overrides.claude.<tier>: "claude-opus-4-7"` |
|
||||
| Start cheap, escalate only on failure | `dynamic_routing` |
|
||||
| All agents follow the session model (non-Anthropic provider) | `model_profile: "inherit"` |
|
||||
|
||||
|
||||
@@ -58,6 +58,10 @@ Model profiles control which Claude model each GSD agent uses. This allows balan
|
||||
3. **Profile table** — the per-agent column from the active `model_profile`
|
||||
4. **Runtime default** — when nothing else applies
|
||||
|
||||
Steps 2–4 select the *tier*; a `model_profile_overrides.<runtime>.<tier>` entry then
|
||||
maps that tier to a concrete model (#4192 — honored on the claude runtime as well, so
|
||||
pinning composes with tiering instead of replacing it).
|
||||
|
||||
### Why two layers above the profile?
|
||||
|
||||
- **Profile** is a global tier strategy (everyone runs balanced).
|
||||
@@ -215,8 +219,11 @@ is (highest → lowest):
|
||||
(see §Dynamic Routing — escalation steps tier up per attempt counter)
|
||||
4. If no dynamic_routing match, check models[phase_type] for a phase-type tier
|
||||
(see §Per-Phase-Type Model Map for the agent → phase-type mapping)
|
||||
5. If no phase-type slot, look up agent in profile table
|
||||
6. Pass model parameter to Task call
|
||||
5. Check model_profile_overrides.<runtime>.<tier> for a per-tier model override
|
||||
(honored on the claude runtime too — #4192; verbatim unless it maps to the
|
||||
current tier alias)
|
||||
6. If no phase-type slot, look up agent in profile table
|
||||
7. Pass model parameter to Task call
|
||||
```
|
||||
|
||||
`model` and `effort` resolve through different mechanisms at different
|
||||
@@ -246,7 +253,9 @@ Override specific agents without changing the entire profile:
|
||||
}
|
||||
```
|
||||
|
||||
Overrides take precedence over the profile. Valid values: `opus`, `sonnet`, `haiku`, `inherit`, or any fully-qualified model ID (e.g., `"o3"`, `"openai/o3"`, `"google/gemini-2.5-pro"`).
|
||||
Overrides take precedence over the profile. Valid values: `opus`, `sonnet`, `haiku`, `fable`, `inherit`, or any fully-qualified model ID (e.g., `"o3"`, `"openai/o3"`, `"google/gemini-2.5-pro"`). `fable` is a Claude Code Agent-tool alias, not a GSD profile tier — it has no column in the profile table above.
|
||||
|
||||
On the Claude runtime, fully-qualified Claude model IDs are honored as explicit generation pins (#4192): an ID that names the current tier default (e.g. `"claude-sonnet-5"`) resolves to its tier alias — the same model in the form the Agent tool always accepts — while any other ID (e.g. `"claude-opus-4-7"`) resolves verbatim, with a warn-once stderr note that setups accepting only tier aliases will not honor a full ID. To pin a generation for a whole tier rather than one agent, set `model_profile_overrides.claude.<tier>` (see docs/CONFIGURATION.md — Runtime-Aware Profiles).
|
||||
|
||||
## Switching Profiles
|
||||
|
||||
|
||||
@@ -157,6 +157,65 @@ function _resolveRuntimeTier(config: Record<string, unknown>, tier: string): Tie
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* #4192 — Resolve the claude-runtime TIER OVERRIDE model for (config, tier).
|
||||
*
|
||||
* Step 3's runtime-aware resolution deliberately skips the claude runtime to
|
||||
* preserve the alias-native posture (#1156/#2297): with no user override, the
|
||||
* resolver must keep returning bare tier aliases, and the builtin claude tier
|
||||
* map (`opus → claude-opus-4-8`, …) must never force full-ID emission on every
|
||||
* default install. But `model_profile_overrides.<runtime>.<tier>` is a
|
||||
* documented override point (docs/CONFIGURATION.md § Runtime-Aware Profiles)
|
||||
* that `workflows/settings-advanced.md` actively writes for claude-runtime
|
||||
* users — and #4192 Finding 1 measured the key inert on this runtime.
|
||||
*
|
||||
* This helper reads ONLY the user's override entry for the effective claude
|
||||
* runtime and tier — never the builtin claude tier map — so an install with no
|
||||
* override is byte-identical to before the fix. The runtime is resolved the
|
||||
* same way steps 1-3 resolve it (config['runtime'], defaulting to 'claude'),
|
||||
* NOT via resolveActiveRuntime (GSD_RUNTIME/marker): the value policy must
|
||||
* key off the config the operator wrote, matching mapClaudeOverrideForRuntime.
|
||||
*
|
||||
* Value policy mirrors the model_overrides path (#2041/#4192): an override
|
||||
* value that maps to a current tier alias collapses to that alias
|
||||
* (byte-equivalent resolution, alias-form emission); anything else — a pinned
|
||||
* older generation (`claude-opus-4-7`), a bare alias/tier repoint (`sonnet`),
|
||||
* or a non-Claude vendor id (`openai/o3`) — is emitted verbatim as pinned.
|
||||
* Malformed entries (no usable `model` string) return null so the caller falls
|
||||
* through to normal alias resolution (ADR-443 D1: invalid values fall through).
|
||||
*/
|
||||
function resolveClaudeTierOverrideModel(
|
||||
configRuntime: string | null | undefined,
|
||||
tier: string | null | undefined,
|
||||
overrides: Record<string, unknown> | null | undefined,
|
||||
): string | null {
|
||||
if (!tier || tier === 'inherit') return null;
|
||||
const effectiveRuntime = configRuntime || 'claude';
|
||||
if (effectiveRuntime !== 'claude') return null; // non-claude runtimes resolve at step 3
|
||||
const overridesMap = overrides as Record<string, Record<string, unknown>> | null | undefined;
|
||||
if (!overridesMap || typeof overridesMap !== 'object') return null;
|
||||
// Own-property guards throughout: both levels are config-supplied plain
|
||||
// objects, so a prototype-chain key ("constructor", "toString") must not
|
||||
// resolve an inherited member instead of falling through (same hardening as
|
||||
// every other config-keyed lookup in this module).
|
||||
const runtimeEntry = Object.hasOwn(overridesMap, effectiveRuntime)
|
||||
? overridesMap[effectiveRuntime]
|
||||
: undefined;
|
||||
if (!runtimeEntry || typeof runtimeEntry !== 'object') return null;
|
||||
const userRaw = Object.hasOwn(runtimeEntry, tier) ? runtimeEntry[tier] : undefined;
|
||||
if (userRaw === undefined || userRaw === null) return null;
|
||||
const entry: Record<string, unknown> = typeof userRaw === 'string'
|
||||
? { model: userRaw }
|
||||
: (userRaw as Record<string, unknown>);
|
||||
if (!entry || typeof entry !== 'object') return null;
|
||||
const model = entry['model'];
|
||||
if (typeof model !== 'string' || model.length === 0) return null;
|
||||
if (Object.hasOwn(CLAUDE_POLICY_ID_TO_ALIAS, model)) {
|
||||
return CLAUDE_POLICY_ID_TO_ALIAS[model];
|
||||
}
|
||||
return model;
|
||||
}
|
||||
|
||||
// Reverse of the Claude tier-default IDs, plus the Fable alias which Claude
|
||||
// Code's Agent tool accepts but which is not a GSD model-profile tier (#1133).
|
||||
const CLAUDE_POLICY_ID_TO_ALIAS: Record<string, string> = {
|
||||
@@ -189,7 +248,10 @@ function _resetModelPolicyWarningCacheForTests(): void {
|
||||
_modelPolicyUnmappableWarned.clear();
|
||||
}
|
||||
|
||||
// Dedupe stderr warnings for unmappable model_overrides Claude IDs (#2041).
|
||||
// Dedupe stderr warnings for unmappable model_overrides Claude IDs (#2041 /
|
||||
// #4192). #2041 originally warned that such a value was being DROPPED to tier
|
||||
// resolution; #4192 keeps the warn-once breadcrumb but changes the behavior to
|
||||
// a verbatim pass-through, so the text now describes the pass-through.
|
||||
const _modelOverrideUnmappableWarned = new Set<string>();
|
||||
function warnModelOverrideUnmappable(agentType: string, overrideValue: string): void {
|
||||
const key = `${agentType}::${overrideValue}`;
|
||||
@@ -200,8 +262,9 @@ function warnModelOverrideUnmappable(agentType: string, overrideValue: string):
|
||||
// model's JSON result is parsed from stdout.
|
||||
const safe = overrideValue.length > 64 ? overrideValue.slice(0, 64) + '…' : overrideValue;
|
||||
process.stderr.write(
|
||||
`gsd: warning — model_overrides value "${safe}" for ${agentType} ` +
|
||||
`has no Claude agent alias; falling through to tier resolution.\n`,
|
||||
`gsd: warning — model_overrides value "${safe}" for ${agentType} is a fully-qualified ` +
|
||||
`Claude model ID with no tier alias; passing it through verbatim. Claude Code setups ` +
|
||||
`whose Agent tool accepts only tier aliases will not honor it. (#4192)\n`,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -213,12 +276,24 @@ function _resetModelOverrideWarningCacheForTests(): void {
|
||||
/**
|
||||
* #2041 — Map a `model_overrides` value to its Claude Agent-tool alias on the
|
||||
* claude runtime, mirroring the `model_policy` path (#1144). Claude Code's
|
||||
* Agent tool `model` parameter documents only tier aliases (opus/sonnet/haiku/
|
||||
* fable); a full Claude model ID returned verbatim is silently dropped by the
|
||||
* spawner. Returns the value to return verbatim, or null to signal "fall
|
||||
* through to normal tier/dynamic-routing resolution" (used when a Claude full
|
||||
* ID has no alias — matches model_policy's warn-and-fall-through). Non-Claude
|
||||
* runtimes and non-Claude values always pass through verbatim.
|
||||
* Agent tool `model` parameter documents tier aliases (opus/sonnet/haiku/
|
||||
* fable) as the always-accepted form. Returns the value to emit verbatim, or
|
||||
* null to signal "fall through to normal tier/dynamic-routing resolution".
|
||||
* Non-Claude runtimes and non-Claude values always pass through verbatim.
|
||||
*
|
||||
* #4192 — an unmappable `claude-*` value (a pinned generation that is not the
|
||||
* current catalog default, e.g. `claude-opus-4-7`) is now PASSED THROUGH
|
||||
* VERBATIM with a warn-once stderr breadcrumb, instead of being dropped to
|
||||
* tier resolution. #2041's drop was correct when the value was plausibly a
|
||||
* mis-typed current default, but for an explicit pin it silently UNPINNED the
|
||||
* operator's choice — the resolver would report a tier the config never asked
|
||||
* for, the exact "profile can misrepresent what actually runs" defect #4192
|
||||
* files. The documented contract ("any fully-qualified model ID",
|
||||
* docs/CONFIGURATION.md § Per-Agent Overrides,
|
||||
* gsd-core/references/model-profiles.md § Per-Agent Overrides) is restored:
|
||||
* the pin is resolved as configured. Values that DO map to a current tier
|
||||
* alias still collapse to that alias — byte-equivalent resolution, the #2041
|
||||
* protection preserved — and a mappable pin never warns.
|
||||
*
|
||||
* Hardening (code+security review): a `typeof` guard preserves the pre-fix
|
||||
* no-crash behavior if a malformed config surfaces a non-string value, and an
|
||||
@@ -243,8 +318,9 @@ function mapClaudeOverrideForRuntime(
|
||||
}
|
||||
if (CLAUDE_AGENT_ALIASES.has(override)) return override;
|
||||
if (override.startsWith('claude-')) {
|
||||
// #4192: explicit generation pin — resolve as configured (see docblock).
|
||||
warnModelOverrideUnmappable(agentType, override);
|
||||
return null;
|
||||
return override;
|
||||
}
|
||||
return override;
|
||||
}
|
||||
@@ -507,6 +583,23 @@ function resolveModelInternal(cwd: string, agentType: string): string {
|
||||
return '';
|
||||
}
|
||||
|
||||
// 4.5. Claude-runtime tier override (#4192 Finding 1). Sits AFTER the omit
|
||||
// gate so an explicit project `resolve_model_ids:"omit"` still wins (#2297:
|
||||
// explicit project omit is honored regardless of runtime), and BEFORE the
|
||||
// alias return so a pinned generation is not re-collapsed to a tier alias or
|
||||
// re-materialized to the LATEST catalog id by step 5's
|
||||
// `resolve_model_ids:true` path. Fires ONLY when the user wrote a
|
||||
// `model_profile_overrides.claude.<tier>` entry for this tier — see
|
||||
// resolveClaudeTierOverrideModel for why the builtin map stays out.
|
||||
if (tier && tier !== 'inherit') {
|
||||
const claudeOverrideModel = resolveClaudeTierOverrideModel(
|
||||
configRuntime,
|
||||
tier,
|
||||
config['model_profile_overrides'] as Record<string, unknown> | null | undefined,
|
||||
);
|
||||
if (claudeOverrideModel !== null) return claudeOverrideModel;
|
||||
}
|
||||
|
||||
// 5. Profile lookup (Claude-native default).
|
||||
if (!agentModels) {
|
||||
return profile === 'quality' ? 'opus'
|
||||
|
||||
@@ -1415,6 +1415,38 @@ describe('resolve-model command', () => {
|
||||
assert.ok(output.model, 'should resolve a model');
|
||||
});
|
||||
|
||||
// #4192 (AC1, behavioral): a claude-runtime tier override must change the
|
||||
// resolved output relative to the no-override control — the CLI surface the
|
||||
// orchestrator reads is where the documented contract is observable.
|
||||
test('claude-runtime tier override changes resolved output vs control (#4192)', () => {
|
||||
fs.writeFileSync(path.join(tmpDir, '.planning', 'config.json'), JSON.stringify({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } },
|
||||
}));
|
||||
const pinned = runGsdTools('resolve-model gsd-planner', tmpDir);
|
||||
assert.ok(pinned.success, `Command failed: ${pinned.error}`);
|
||||
assert.strictEqual(JSON.parse(pinned.output).model, 'claude-opus-4-7');
|
||||
|
||||
fs.writeFileSync(path.join(tmpDir, '.planning', 'config.json'), JSON.stringify({
|
||||
model_profile: 'balanced',
|
||||
}));
|
||||
const control = runGsdTools('resolve-model gsd-planner', tmpDir);
|
||||
assert.ok(control.success, `Command failed: ${control.error}`);
|
||||
assert.strictEqual(JSON.parse(control.output).model, 'opus');
|
||||
});
|
||||
|
||||
// #4192 (AC2, behavioral): a per-agent fully-qualified Claude model ID is
|
||||
// resolved as configured through the CLI surface.
|
||||
test('per-agent fully-qualified claude ID resolves as configured (#4192)', () => {
|
||||
fs.writeFileSync(path.join(tmpDir, '.planning', 'config.json'), JSON.stringify({
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-debugger': 'claude-opus-4-7' },
|
||||
}));
|
||||
const result = runGsdTools('resolve-model gsd-debugger', tmpDir);
|
||||
assert.ok(result.success, `Command failed: ${result.error}`);
|
||||
assert.strictEqual(JSON.parse(result.output).model, 'claude-opus-4-7');
|
||||
});
|
||||
|
||||
// #443: resolve-model now emits unified `effort` instead of `reasoning_effort`.
|
||||
// reasoning_effort was flavor-text (resolved but consumed by nobody); effort is
|
||||
// the wired, config-driven universal effort string for all runtimes.
|
||||
|
||||
@@ -4123,16 +4123,19 @@ describe('#2041 model_overrides: Claude full ID → alias on claude runtime', ()
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-executor'), 'claude-sonnet-5');
|
||||
});
|
||||
|
||||
// AC5: unmappable Claude full ID warns once + falls through to tier alias
|
||||
test('model_overrides unmappable claude ID (claude-opus-4-5) falls through to tier alias on claude', () => {
|
||||
// AC5 (#4192 revision): an unmappable Claude full ID — an explicit
|
||||
// generation pin — is passed through VERBATIM with a warn-once breadcrumb,
|
||||
// instead of being dropped to tier resolution (which silently unpinned the
|
||||
// operator's explicit choice; see #4192 Finding 2).
|
||||
test('model_overrides unmappable claude ID (claude-opus-4-5) passes through verbatim on claude', () => {
|
||||
resetRuntimeWarningCaches();
|
||||
writeConfig(tmpDir, {
|
||||
runtime: 'claude',
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-planner': 'claude-opus-4-5' },
|
||||
});
|
||||
// gsd-planner balanced → opus tier; claude-opus-4-5 has no alias → warn + fall through → 'opus'
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus');
|
||||
// gsd-planner balanced → opus tier; claude-opus-4-5 has no alias → warn + verbatim pin
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'claude-opus-4-5');
|
||||
});
|
||||
|
||||
test('model_overrides unmappable claude ID emits a stderr warning exactly once (dedupe)', () => {
|
||||
@@ -4173,19 +4176,19 @@ describe('#2041 model_overrides: Claude full ID → alias on claude runtime', ()
|
||||
assert.strictEqual(resolveModelForTier(tmpDir, 'gsd-executor', 0), 'claude-sonnet-5');
|
||||
});
|
||||
|
||||
// MEDIUM-1 (review): exercise the unmappable-override fall-through branch in
|
||||
// resolveModelForTier (closes the mutation-score gap — a future refactor that
|
||||
// accidentally returned the verbatim override instead of falling through
|
||||
// would otherwise survive the suite).
|
||||
test('resolveModelForTier unmappable claude ID falls through to tier alias on claude', () => {
|
||||
// MEDIUM-1 (review, #4192 revision): exercise the unmappable-override
|
||||
// branch in resolveModelForTier (keeps the mutation-score gap closed — the
|
||||
// verbatim-pin return must survive a future refactor on the escalation path
|
||||
// too, not just resolveModelInternal).
|
||||
test('resolveModelForTier unmappable claude ID passes through verbatim on claude', () => {
|
||||
resetRuntimeWarningCaches();
|
||||
writeConfig(tmpDir, {
|
||||
runtime: 'claude',
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-planner': 'claude-opus-4-5' },
|
||||
});
|
||||
// unmappable override → fall through → no dynamic_routing → resolveModelInternal → 'opus'
|
||||
assert.strictEqual(resolveModelForTier(tmpDir, 'gsd-planner', 0), 'opus');
|
||||
// unmappable override → no dynamic_routing → resolveModelInternal → verbatim pin
|
||||
assert.strictEqual(resolveModelForTier(tmpDir, 'gsd-planner', 0), 'claude-opus-4-5');
|
||||
});
|
||||
|
||||
// LOW-2 (review): pin the case-sensitive contract — a case-variant like
|
||||
@@ -6321,3 +6324,346 @@ describe('#3007 PROPERTY: renderEffortForRuntime never renders a level the model
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// ─── #4192: claude-runtime generation pinning via explicit overrides ──────────
|
||||
//
|
||||
// Confirmed-bug scope (maintainer triage): Findings 1 and 2 — documented
|
||||
// behavior the resolver does not implement on the claude runtime.
|
||||
//
|
||||
// F1 — model_profile_overrides.claude.<tier> was inert: step 3 of
|
||||
// resolveModelInternal gated runtime-aware tier resolution on
|
||||
// `configRuntime !== 'claude'`, so the only reader of the key was never
|
||||
// consulted on claude, while settings-advanced.md writes it for
|
||||
// claude-runtime users.
|
||||
// F2 — fully-qualified claude-* IDs in model_overrides were warn-dropped to
|
||||
// tier resolution (mapClaudeOverrideForRuntime unmappable branch,
|
||||
// #2041), while the configuration reference and the shipped
|
||||
// model-profiles reference both document "any fully-qualified model
|
||||
// ID" as valid.
|
||||
//
|
||||
// Agreed contract (AC2, pinned here): an explicit pin is RESOLVED AS
|
||||
// CONFIGURED. A claude-* value that maps to a current tier alias still
|
||||
// collapses to that alias (the #2041 protection — byte-equivalent resolution);
|
||||
// an unmappable one (a pinned older generation) is returned verbatim with a
|
||||
// warn-once breadcrumb, because dropping it would silently unpin the operator's
|
||||
// explicit choice — the exact "profile misrepresents what runs" defect of
|
||||
// #4192. Unpinned resolution is byte-stable (control rows below).
|
||||
describe('#4192 model_profile_overrides.claude.*: tier overrides honor pins on the claude runtime', () => {
|
||||
const { createTempDir, resetRuntimeWarningCaches } = require('./helpers.cjs');
|
||||
let tmpDir;
|
||||
const make = () => createTempDir('gsd-4192-tier-override-');
|
||||
const write = (cfg) => fs.writeFileSync(
|
||||
path.join(tmpDir, '.planning', 'config.json'), JSON.stringify(cfg, null, 2), 'utf-8');
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = make();
|
||||
fs.mkdirSync(path.join(tmpDir, '.planning'), { recursive: true });
|
||||
resetRuntimeWarningCaches();
|
||||
});
|
||||
afterEach(() => {
|
||||
cleanup(tmpDir);
|
||||
resetRuntimeWarningCaches();
|
||||
});
|
||||
|
||||
// Row 1 — REGRESSION (failing-first): pinned generation honored, implicit claude runtime.
|
||||
test('claude.opus = "claude-opus-4-7" pins the opus tier (implicit claude runtime)', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'claude-opus-4-7');
|
||||
});
|
||||
|
||||
// Row 2 — same with an explicit runtime key.
|
||||
test('claude.opus pin honored with explicit runtime: "claude"', () => {
|
||||
write({
|
||||
runtime: 'claude',
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'claude-opus-4-7');
|
||||
});
|
||||
|
||||
// Row 3 — mappable override collapses to the alias (form parity with #2041 step 1).
|
||||
test('claude.sonnet = "claude-sonnet-5" resolves to the "sonnet" alias', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { sonnet: 'claude-sonnet-5' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-executor'), 'sonnet');
|
||||
});
|
||||
|
||||
// Row 4 — fable-valued override maps through the fable alias.
|
||||
test('claude.opus = "claude-fable-5" resolves to "fable"', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { opus: 'claude-fable-5' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'fable');
|
||||
});
|
||||
|
||||
// Row 5 — bare-alias / tier-repoint override passes through verbatim.
|
||||
test('claude.opus = "sonnet" repoints the tier at the sonnet alias', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { opus: 'sonnet' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'sonnet');
|
||||
});
|
||||
|
||||
// Row 6 — non-Claude ID override passes through verbatim (docs: any fully-qualified ID).
|
||||
test('claude.haiku = "openai/gpt-4o-mini" passes through verbatim on claude', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { haiku: 'openai/gpt-4o-mini' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-codebase-mapper'), 'openai/gpt-4o-mini');
|
||||
});
|
||||
|
||||
// Row 7 — object-form override (settings workflow accepts {model, reasoning_effort}).
|
||||
test('claude.opus = { model: "claude-opus-4-7" } object form pins the tier', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { opus: { model: 'claude-opus-4-7' } } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'claude-opus-4-7');
|
||||
});
|
||||
|
||||
// Row 8 — CONTROL (AC1): no override → byte-identical alias resolution.
|
||||
test('no model_profile_overrides → alias resolution unchanged', () => {
|
||||
write({ model_profile: 'balanced' });
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus');
|
||||
});
|
||||
|
||||
// Row 9 — CONTROL: overrides for another runtime never apply to claude.
|
||||
test('codex-only overrides are inert on the claude runtime', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { codex: { opus: 'gpt-5-pro' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus');
|
||||
});
|
||||
|
||||
// Row 10 — CONTROL: override for a different tier than the agent's is inert for that agent.
|
||||
test('claude.sonnet override does not touch an opus-tier agent', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { sonnet: 'claude-sonnet-4-6' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus');
|
||||
});
|
||||
|
||||
// Row 11 — CONTROL: inherit profile is immune to tier overrides.
|
||||
test('model_profile: "inherit" + claude.opus pin → "inherit"', () => {
|
||||
write({
|
||||
model_profile: 'inherit',
|
||||
model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'inherit');
|
||||
});
|
||||
|
||||
// Row 12 — CONTROL: explicit project resolve_model_ids:"omit" beats the override (#2297).
|
||||
test('project resolve_model_ids: "omit" + claude.opus pin → empty string', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
resolve_model_ids: 'omit',
|
||||
model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), '');
|
||||
});
|
||||
|
||||
// Row 13 — CONTROL: model_overrides still wins over the tier override.
|
||||
test('model_overrides beats model_profile_overrides.claude', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-planner': 'haiku' },
|
||||
model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'haiku');
|
||||
});
|
||||
|
||||
// Row 15 — CONTROL: object override without a model key degrades to the alias.
|
||||
test('claude.opus = { reasoning_effort } (no model) falls through to the alias', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { opus: { reasoning_effort: 'high' } } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus');
|
||||
});
|
||||
|
||||
// Row 16 — CONTROL: non-string/non-object value degrades to the alias.
|
||||
test('claude.opus = 42 (malformed value) falls through to the alias', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { opus: 42 } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus');
|
||||
});
|
||||
|
||||
// Row 17 — CONTROL: empty-string value degrades to the alias.
|
||||
test('claude.opus = "" falls through to the alias', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { opus: '' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus');
|
||||
});
|
||||
|
||||
// Row 26 — ADVERSARIAL: prototype-chain keys in the override map must not leak.
|
||||
test('"constructor" as a claude override key does not resolve an inherited member', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_profile_overrides: { claude: { constructor: 'claude-opus-4-7' } },
|
||||
});
|
||||
// 'constructor' is not a tier; resolution must ignore it entirely and land
|
||||
// on the profile alias, never on Function.prototype's members.
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'opus');
|
||||
});
|
||||
|
||||
// Row 30 — the pin wins over resolve_model_ids:true alias materialization.
|
||||
test('claude.opus pin beats resolve_model_ids: true materialization', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
resolve_model_ids: true,
|
||||
model_profile_overrides: { claude: { opus: 'claude-opus-4-7' } },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-planner'), 'claude-opus-4-7');
|
||||
});
|
||||
});
|
||||
|
||||
describe('#4192 model_overrides: fully-qualified claude IDs resolve as configured', () => {
|
||||
const { createTempDir, resetRuntimeWarningCaches } = require('./helpers.cjs');
|
||||
let tmpDir;
|
||||
const make = () => createTempDir('gsd-4192-agent-override-');
|
||||
const write = (cfg) => fs.writeFileSync(
|
||||
path.join(tmpDir, '.planning', 'config.json'), JSON.stringify(cfg, null, 2), 'utf-8');
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = make();
|
||||
fs.mkdirSync(path.join(tmpDir, '.planning'), { recursive: true });
|
||||
resetRuntimeWarningCaches();
|
||||
});
|
||||
afterEach(() => {
|
||||
cleanup(tmpDir);
|
||||
resetRuntimeWarningCaches();
|
||||
});
|
||||
|
||||
// Row 18 — REGRESSION (failing-first): pinned generation honored, implicit claude runtime.
|
||||
test('model_overrides "claude-opus-4-7" resolves verbatim (implicit claude runtime)', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-debugger': 'claude-opus-4-7' },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-debugger'), 'claude-opus-4-7');
|
||||
});
|
||||
|
||||
// Row 18b — explicit runtime key.
|
||||
test('model_overrides "claude-opus-4-7" resolves verbatim with runtime: "claude"', () => {
|
||||
write({
|
||||
runtime: 'claude',
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-debugger': 'claude-opus-4-7' },
|
||||
});
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-debugger'), 'claude-opus-4-7');
|
||||
});
|
||||
|
||||
// Row 21 — warn-once breadcrumb on an unmappable pin (visibility, not a drop).
|
||||
test('unmappable pin emits exactly one pass-through stderr warning (dedupe)', () => {
|
||||
write({
|
||||
runtime: 'claude',
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-debugger': 'claude-opus-4-7' },
|
||||
});
|
||||
const writes = [];
|
||||
const original = process.stderr.write.bind(process.stderr);
|
||||
process.stderr.write = (chunk) => { writes.push(String(chunk)); return true; };
|
||||
try {
|
||||
resolveModelInternal(tmpDir, 'gsd-debugger');
|
||||
resolveModelInternal(tmpDir, 'gsd-debugger'); // dedupe must suppress
|
||||
} finally {
|
||||
process.stderr.write = original;
|
||||
}
|
||||
const warnings = writes.filter((w) => w.includes('model_overrides') && w.includes('claude-opus-4-7'));
|
||||
assert.strictEqual(warnings.length, 1,
|
||||
`expected exactly one override warning, got ${warnings.length}: ${JSON.stringify(writes)}`);
|
||||
// The warning must describe pass-through, not a fall-through that no longer happens.
|
||||
assert.ok(!warnings[0].includes('falling through'),
|
||||
`warning must not claim a fall-through: ${warnings[0]}`);
|
||||
});
|
||||
|
||||
// Row 21b — no warning for a value that needs no breadcrumb (mappable / non-claude).
|
||||
test('mappable ID resolution emits no model_overrides warning', () => {
|
||||
write({
|
||||
runtime: 'claude',
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-debugger': 'claude-sonnet-5' },
|
||||
});
|
||||
const writes = [];
|
||||
const original = process.stderr.write.bind(process.stderr);
|
||||
process.stderr.write = (chunk) => { writes.push(String(chunk)); return true; };
|
||||
try {
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-debugger'), 'sonnet');
|
||||
} finally {
|
||||
process.stderr.write = original;
|
||||
}
|
||||
assert.strictEqual(writes.filter((w) => w.includes('model_overrides')).length, 0);
|
||||
});
|
||||
|
||||
// Row 22 — escalation path parity.
|
||||
test('resolveModelForTier returns the pinned generation verbatim', () => {
|
||||
write({
|
||||
runtime: 'claude',
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-debugger': 'claude-opus-4-7' },
|
||||
});
|
||||
assert.strictEqual(resolveModelForTier(tmpDir, 'gsd-debugger', 0), 'claude-opus-4-7');
|
||||
});
|
||||
|
||||
// Row 24 — tier honesty signal unchanged (AC3): a raw pin carries no tier.
|
||||
test('resolveTierFromConfig reports "unknown" for a raw pinned generation', () => {
|
||||
write({
|
||||
runtime: 'claude',
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-debugger': 'claude-opus-4-7' },
|
||||
});
|
||||
assert.strictEqual(resolveTierFromConfig(
|
||||
JSON.parse(fs.readFileSync(path.join(tmpDir, '.planning', 'config.json'), 'utf-8')),
|
||||
'gsd-debugger'), 'unknown');
|
||||
});
|
||||
|
||||
// Row 25 — ADVERSARIAL: prototype-chain agentType must not leak through overrides.
|
||||
test('agentType "toString" against model_overrides: {} stays on the unknown-agent path', () => {
|
||||
write({
|
||||
model_profile: 'balanced',
|
||||
model_overrides: {},
|
||||
});
|
||||
// Unknown agent + balanced profile → the hardcoded fallback alias, never
|
||||
// an inherited Function.prototype member.
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'toString'), 'sonnet');
|
||||
});
|
||||
|
||||
// Row 28 — an oversized pin value survives resolution; any warning stays capped.
|
||||
test('oversized unmappable pin resolves verbatim and warning text is capped at 64 chars', () => {
|
||||
const longPin = 'claude-opus-' + '9'.repeat(80);
|
||||
write({
|
||||
runtime: 'claude',
|
||||
model_profile: 'balanced',
|
||||
model_overrides: { 'gsd-debugger': longPin },
|
||||
});
|
||||
const writes = [];
|
||||
const original = process.stderr.write.bind(process.stderr);
|
||||
process.stderr.write = (chunk) => { writes.push(String(chunk)); return true; };
|
||||
try {
|
||||
assert.strictEqual(resolveModelInternal(tmpDir, 'gsd-debugger'), longPin);
|
||||
} finally {
|
||||
process.stderr.write = original;
|
||||
}
|
||||
const warnings = writes.filter((w) => w.includes('model_overrides'));
|
||||
assert.strictEqual(warnings.length, 1);
|
||||
// The rendered value inside the warning is the 64-char cap + ellipsis, not the full pin.
|
||||
assert.ok(!warnings[0].includes(longPin),
|
||||
`warning must not contain the uncapped pin: ${warnings[0]}`);
|
||||
assert.ok(warnings[0].includes('claude-opus-' + '9'.repeat(52) + '…'),
|
||||
`warning must contain the capped pin render: ${warnings[0]}`);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user