Files
msd-core/tests/feat-443-effort-fast-mode.integration.test.cjs
Tom Boucher 5ca646f015 feat(#443): unified cross-provider effort controls + fast-mode-aware routing (#463)
* test(#443): RED unified effort + fast_mode + resolve-execution

All 68 tests failing as expected — no implementation yet.
Covers: effort cascade (tier defaults, overrides, invalid fallthrough),
fast_mode cascade (boolean-only, tier defaults), resolveEffortForTier
escalation, renderEffortForRuntime clamping, resolve-execution CLI,
config schema new keys, QA hostile-input matrix.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(#443): unified cross-provider effort + fast_mode knobs and resolve-execution query

Adds config-driven effort control (universal ladder: minimal<low<medium<high<xhigh<max)
and fast_mode propagation knobs, with per-runtime rendering that clamps the unique
tail values (max=Anthropic-only clamps to xhigh on Codex; minimal=Codex-only clamps
to low on Claude).

Key changes:
- config-schema.manifest.json: add effort.default, fast_mode.enabled as validKeys;
  add 4 dynamicKeyPatterns for effort.routing_tier_defaults, effort.agent_overrides,
  fast_mode.routing_tier_defaults, fast_mode.agent_overrides; fix stale _comment
- config-defaults.manifest.json: add effort and fast_mode blocks with tier defaults
- model-catalog.cjs: add EFFORT_RENDERING map, renderEffortForRuntime(), RUNTIMES_WITH_FAST_MODE
- model-profiles.cjs: re-export new catalog exports
- core.cjs: add resolveEffortInternal, resolveFastModeInternal, resolveEffortForTier,
  VALID_EFFORTS, EFFORT_SET, nextEffort; pass effort/fast_mode through loadConfig
- commands.cjs: replace reasoning_effort in cmdResolveModel with unified effort;
  add cmdResolveExecution (superset command with effort_rendered, effort_param,
  effort_propagation, fast_mode, fast_mode_supported)
- gsd-tools.cjs: add resolve-execution case with --effort/--fast-mode/--attempt flags
- tests/feat-443: 69 tests covering cascade, rendering, escalation, CLI, schema, QA matrix
- tests/commands.test.cjs: convert 3 reasoning_effort assertions to unified effort
- docs/CONFIGURATION.md: document effort + fast_mode + resolve-execution sections
- settings-advanced.md: list new effort/fast_mode keys in confirmation table

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(#443): remove dead catalog effort lane; unify codex effort through renderEffortForRuntime

- Remove resolveReasoningEffortInternal (catalog-driven effort function) from
  core.cjs and its export; remove from commands.cjs destructure import
- Convert tests/issue-2517-runtime-aware-profiles.test.cjs: all 11 effort
  assertions now use resolveEffortInternal + renderEffortForRuntime; Claude
  effort is first-class (output_config.effort); unknown runtimes assert param===null
- Convert tests/feat-3023-model-phase-types.test.cjs: replace the entire
  resolveReasoningEffortInternal describe with unified effort assertions;
  effort derives from AGENT_DEFAULT_TIERS routing tier, not phase-type tier

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(#443): ADR for unified cross-provider effort + fast-mode routing

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* test(#443): architecture-level QA invariants + test-strategy doc

Add 48-test integration suite (feat-443-effort-fast-mode.integration.test.cjs)
covering 8 architectural invariants: cross-provider validity (never emit a value
the real API would 400 on), param/channel contract stability, resolve-execution
JSON contract (all 8 keys + correct types), totality across the full 33-agent
registry, fast-mode honesty (claude always fast_mode_supported=false), precedence
first-valid-wins matrix for both effort and fast_mode cascades, dynamic-routing
composition (effort escalation independent of model tier), and config-set round-trip
for all new effort/* and fast_mode/* key namespaces. Append test-strategy section
with invariant rationale and E2E gap documentation to docs/TESTING-SUITES.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(#443): add failing install-wiring tests for effort per-runtime injection (RED)

TDD RED: 10 failing tests covering:
- Claude .md gets effort: injected per tier (planner=xhigh, mapper=low, executor=high)
- Gemini .md does NOT get effort: (already passing — Gemini-safe)
- Codex .toml gets model_reasoning_effort via unified resolver
- Config-driven: effort.agent_overrides drives both Claude .md and Codex .toml
- Source purity: agents/*.md have no effort: key (already passing)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(#443): wire effort per-runtime at install (Claude .md frontmatter + Codex .toml unified)

- Import AGENT_DEFAULT_TIERS and renderEffortForRuntime from model-catalog.cjs
- Add readGsdEffectiveEffortConfig(targetDir): reads merged effort config from
  .planning/config.json (per-project wins) + ~/.gsd/defaults.json (global fallback),
  same probe pattern as readGsdRuntimeProfileResolver
- Add resolveInstallTimeEffort(effortCfg, agentName): pure function matching
  resolveEffortInternal() precedence (agent_overrides > routing_tier_defaults > default > 'high')
  without loadConfig side-effects (no sub-repo detection, no migration writes)
- Claude agent copy loop: inject `effort: <value>` into frontmatter ONLY for
  runtime === 'claude'; all other .md runtimes (Gemini, Qwen, Hermes, etc.) stay
  effort-free (Gemini-safe source contract preserved in agents/*.md)
- generateCodexAgentToml: add effortCfg param; emit model_reasoning_effort from
  unified resolver (replaces old catalog entry.reasoning_effort); Codex clamps
  max → xhigh via renderEffortForRuntime('codex', ...)
- installCodexConfig: pass readGsdEffectiveEffortConfig(targetDir) to
  generateCodexAgentToml so per-project config wins for Codex .toml too
- Update failing tests to GREEN: 12/12 pass; all 17 install tests pass;
  2847/2848 unit tests pass (1 pre-existing failure: policy-shell-pinning)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(#443): source install effort defaults from manifest (kill drift) + guard test

Replace hardcoded _GSD_EFFORT_MANIFEST_TIER_DEFAULTS and the 'high' fallback in
resolveInstallTimeEffort with values read from config-defaults.manifest.json at
module init, using the same __dirname-relative path install.js already uses for
all shared manifests. Add feat-443-effort-defaults-drift.test.cjs to assert
equality between install.js's runtime constants and the manifest on every CI run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(#443): reconcile Codex TOML tests with unified effort design

The #443 unified effort resolver makes generateCodexAgentToml always emit
model_reasoning_effort (driven by resolveInstallTimeEffort, not model_profile_overrides).
The test 'generated TOML omits reasoning_effort when runtime has none' had an
obsolete premise — model_profile_overrides.reasoning_effort:'' no longer suppresses
unified effort. Convert it to assert the new invariant: Codex TOML always carries a
valid model_reasoning_effort from the agent's routing tier (xhigh for gsd-planner,
a heavy-tier agent), while model_profile_overrides model override is still respected.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(#443): make install.js effort resolution lazy (no load-time side effects breaking launcher-parity)

Replace module-load-time IIFE + hard throw (config-defaults.manifest.json read)
and top-level require of model-catalog.cjs with a lazy _getGsdEffortCatalog()
getter that initialises on first call from resolveInstallTimeEffort /
generateCodexAgentToml / Claude .md effort injection.  Requiring install.js in
unrelated test contexts (e.g. runtime-launcher-parity) no longer triggers
manifest IO or throws, eliminating the load-time side effect that changed
subprocess exit codes / stderr on the bench.

Drift-guard exports (_GSD_EFFORT_MANIFEST_TIER_DEFAULTS / _GSD_EFFORT_MANIFEST_DEFAULT)
preserved as lazy getter properties on module.exports so feat-443-effort-defaults-drift
still validates them without forcing eager load.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(#443): isolate install-wiring test HOME to stop \$HOME/.claude pollution breaking launcher-parity

runGlobalInstall() now redirects HOME to a per-call isolated tmpdir in addition
to the existing runtime-specific env-var redirects (CLAUDE_CONFIG_DIR,
GEMINI_CONFIG_DIR, CODEX_HOME). This ensures install.js code that uses
os.homedir() directly — including the ~/.cache/gsd update-check deletion,
~/.gsd/defaults.json reads, and any HOME-relative npm subprocess writes —
never touches the real \$HOME during the test.

Without the HOME isolation the install test (which is new to this branch and
is now picked up by Docker's raw \`tests/*.test.cjs\` glob) could write or
delete files under the real \$HOME, causing runtime-launcher-parity test (D)
to fail: (D) asserts a loud non-zero exit when \$RUNTIME_DIR/gsd-tools.cjs is
absent and gsd-tools is not on PATH, but the launcher's \$HOME/.claude fallback
arm succeeds if \$HOME/.claude/get-shit-done/bin/gsd-tools.cjs exists.

Also sets GSD_SKIP_STALE_SDK_CHECK=1 to suppress the \`npm ls -g\` subprocess
that the global installer spawns — irrelevant to effort-wiring assertions,
slow, and potentially writes to ~/.npm cache.

All 12 feat-443 install-wiring assertions preserved. Drift-guard 5/5. Unit
suite 2848/2850 (pre-existing policy-shell-pinning.test.cjs failure on next).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore(#443): add changeset fragment for effort + fast-mode routing

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(#443): set GSD_TEST_MODE before requiring install.js in drift-guard test to prevent HOME leak

Without GSD_TEST_MODE=1, require('bin/install.js') runs the module's main
install block (guarded by !GSD_TEST_MODE), performing a real global Claude
install into $HOME/.claude/. On CI ubuntu where node is on standard PATH,
the launcher's $HOME/.claude fallback arm then finds gsd-tools.cjs, causing
runtime-launcher-parity test (D) to exit zero when it must exit non-zero.

Root cause: feat-443-effort-defaults-drift.test.cjs (unit suite) runs
alphabetically before runtime-launcher-parity.test.cjs in the same node
--test invocation. Each runs in a separate worker process but shares the
same HOME. The drift test's install leaks gsd-tools.cjs into that HOME,
then the launcher test's bash subprocess finds it via the $HOME/.claude arm.

Fix: add process.env.GSD_TEST_MODE = '1' at the top of the drift-guard
test, before the require(installPath) call. This matches the pattern used
by feat-443-effort-fast-mode.test.cjs and feat-443-effort-install-wiring
.install.test.cjs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(#443): deterministic resolve-execution arg parsing + validate install-time effort (Codex adversarial findings)

Finding 1: resolve-execution --effort low gsd-planner misrouted 'low' as the agent.
Replace find(non-dash) with a proper flag-consuming loop that collects a single
positional; validate missing/extra positionals and malformed --attempt values.

Finding 2: resolveInstallTimeEffort returned unvalidated effort strings (e.g. "ultra")
verbatim. Each precedence layer now checks GSD_EFFORT_SET (imported once from
core.cjs) before accepting a value, mirroring resolveEffortInternal exactly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(#443): newline-agnostic effort frontmatter injection (Windows CRLF) + CRLF-safe assertions

Extracts injectEffortFrontmatter(content, effortValue) pure helper that detects
EOL (LF vs CRLF) from the opening '---' line and inserts 'effort: <value>'
before the closing '---' delimiter using the same EOL as the surrounding
frontmatter. Regex now uses /^---\r?\n([\s\S]*?)^---\r?$/m instead of the
LF-only /^(---\n[\s\S]*?)(---)(\n|$)/ that silently skipped CRLF files on
Windows (git core.autocrlf=true checkout).

Also adds 7 unit tests covering LF, CRLF, idempotency, no-frontmatter, and
complex frontmatter cases. Exports injectEffortFrontmatter from module.exports.

Fixes 6 CI failures in tests/feat-443-effort-install-wiring.install.test.cjs
on windows-latest runners (lines 138, 145, 152, 261, 345, 356).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: CI Rebase Check <ci@gsd-redux>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-29 10:32:58 -04:00

719 lines
30 KiB
JavaScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
'use strict';
/**
* Architecture-level QA for issue #443 — unified effort + fast_mode engine.
*
* Integration suite (*.integration.test.cjs): cross-module flows that exercise
* real CLI invocations via runGsdTools, the full 33-agent registry, and the
* config round-trip through config-set -> resolve-execution.
*
* INVARIANTS tested here (each is also documented in docs/TESTING-SUITES.md):
*
* (a) CROSS-PROVIDER VALIDITY — renderEffortForRuntime never emits a value
* that the real provider API would 400 on. Ground-truth provider enums are
* defined as local constants (not sourced from the implementation).
*
* (b) PARAM/CHANNEL CONTRACT — each runtime exposes a stable parameter name
* and propagation channel.
*
* (c) RESOLVE-EXECUTION JSON CONTRACT — the CLI command emits a stable JSON
* shape with all required keys and correct types.
*
* (d) TOTALITY across the real 33-agent registry — every agent produces a
* valid effort value; none returns undefined/null.
*
* (e) FAST-MODE HONESTY INVARIANT — claude runtime always reports
* fast_mode_supported=false (emitting fast_mode frontmatter is a silent
* no-op for Claude Code subagents).
*
* (f) PRECEDENCE MATRIX — first-valid-wins for both effort and fast_mode
* cascades, including invalid values correctly falling through.
*
* (g) DYNAMIC-ROUTING COMPOSITION — resolveEffortForTier escalates
* independently of model tier logic; clamps at 'max'; respects
* max_escalations; disabled when escalate_on_failure=false.
*
* (h) CONFIG-TOOLING ROUND-TRIP — config-set accepts all new effort/fast_mode
* key paths (schema validation passes); values survive round-trip through
* resolve-execution.
*/
process.env.GSD_TEST_MODE = '1';
const { describe, test, before, after, beforeEach, afterEach } = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const { createTempProject, cleanup, runGsdTools } = require('./helpers.cjs');
const {
resolveEffortInternal,
resolveFastModeInternal,
resolveEffortForTier,
VALID_EFFORTS,
} = require('../get-shit-done/bin/lib/core.cjs');
const {
renderEffortForRuntime,
RUNTIMES_WITH_FAST_MODE,
catalog,
} = require('../get-shit-done/bin/lib/model-catalog.cjs');
// ─────────────────────────────────────────────────────────────────────────────
// Ground-truth provider enums (defined HERE, not sourced from the implementation).
// These are the exact values the real APIs accept — using a value outside these
// sets would result in a 400 response from the provider.
//
// Sources:
// Anthropic: output_config.effort — https://docs.anthropic.com (Claude API)
// OpenAI: model_reasoning_effort — https://platform.openai.com/docs (Codex)
// ─────────────────────────────────────────────────────────────────────────────
const PROVIDER_EFFORT_ENUMS = {
claude: new Set(['low', 'medium', 'high', 'xhigh', 'max']),
codex: new Set(['minimal', 'low', 'medium', 'high', 'xhigh']),
};
// Helper: write config.json into a temp project
function writeConfig(dir, config) {
const planningDir = path.join(dir, '.planning');
fs.mkdirSync(planningDir, { recursive: true });
fs.writeFileSync(path.join(planningDir, 'config.json'), JSON.stringify(config, null, 2));
}
// ─── (a) CROSS-PROVIDER VALIDITY INVARIANT ───────────────────────────────────
describe('#443 integration (a): cross-provider validity invariant', () => {
// For every universal effort × every provider runtime, the rendered value
// must be a member of that provider's real API enum.
test('all VALID_EFFORTS render within provider enums for claude and codex', () => {
for (const universalEffort of VALID_EFFORTS) {
for (const [runtime, providerEnum] of Object.entries(PROVIDER_EFFORT_ENUMS)) {
const rendered = renderEffortForRuntime(runtime, universalEffort);
assert.ok(
providerEnum.has(rendered.value),
`render('${runtime}', '${universalEffort}').value = '${rendered.value}' is NOT in the ` +
`${runtime} provider enum ${[...providerEnum].join('|')} — real API would 400`
);
}
}
});
// Documented clamps must hold exactly
test("render('codex','max').value === 'xhigh' (max is Anthropic-only)", () => {
assert.strictEqual(renderEffortForRuntime('codex', 'max').value, 'xhigh');
});
test("render('claude','minimal').value === 'low' (minimal is Codex-only)", () => {
assert.strictEqual(renderEffortForRuntime('claude', 'minimal').value, 'low');
});
// Common levels must pass through unchanged on BOTH providers
test('common levels (low/medium/high/xhigh) pass through unchanged on claude', () => {
for (const level of ['low', 'medium', 'high', 'xhigh']) {
assert.strictEqual(
renderEffortForRuntime('claude', level).value,
level,
`claude: level '${level}' should pass through unchanged`
);
}
});
test('common levels (low/medium/high/xhigh) pass through unchanged on codex', () => {
for (const level of ['low', 'medium', 'high', 'xhigh']) {
assert.strictEqual(
renderEffortForRuntime('codex', level).value,
level,
`codex: level '${level}' should pass through unchanged`
);
}
});
});
// ─── (b) PARAM/CHANNEL CONTRACT ──────────────────────────────────────────────
describe('#443 integration (b): param/channel contract', () => {
test("claude: param is always 'output_config.effort'", () => {
for (const effort of VALID_EFFORTS) {
const r = renderEffortForRuntime('claude', effort);
assert.strictEqual(r.param, 'output_config.effort',
`claude param must be 'output_config.effort' for effort '${effort}'`);
}
});
test("codex: param is always 'model_reasoning_effort'", () => {
for (const effort of VALID_EFFORTS) {
const r = renderEffortForRuntime('codex', effort);
assert.strictEqual(r.param, 'model_reasoning_effort',
`codex param must be 'model_reasoning_effort' for effort '${effort}'`);
}
});
test('claude channel is stable: frontmatter', () => {
for (const effort of VALID_EFFORTS) {
assert.strictEqual(renderEffortForRuntime('claude', effort).channel, 'frontmatter');
}
});
test('codex channel is stable: api', () => {
for (const effort of VALID_EFFORTS) {
assert.strictEqual(renderEffortForRuntime('codex', effort).channel, 'api');
}
});
test("unknown runtimes (gemini, qwen, 'mystery'): param===null, value passes through", () => {
for (const runtime of ['gemini', 'qwen', 'mystery']) {
for (const effort of VALID_EFFORTS) {
const r = renderEffortForRuntime(runtime, effort);
assert.strictEqual(r.param, null, `${runtime}: param must be null`);
assert.strictEqual(r.channel, null, `${runtime}: channel must be null`);
assert.strictEqual(r.value, effort, `${runtime}: value must pass through unchanged`);
}
}
});
});
// ─── (c) RESOLVE-EXECUTION JSON CONTRACT ─────────────────────────────────────
describe('#443 integration (c): resolve-execution JSON contract', () => {
let tmpDir;
beforeEach(() => { tmpDir = createTempProject(); });
afterEach(() => { cleanup(tmpDir); });
function assertFullContract(output, label) {
assert.ok(typeof output.model === 'string' && output.model.length > 0,
`${label}: model must be a non-empty string`);
assert.ok(typeof output.profile === 'string' && output.profile.length > 0,
`${label}: profile must be a non-empty string`);
assert.ok(VALID_EFFORTS.includes(output.effort),
`${label}: effort '${output.effort}' must be a member of VALID_EFFORTS`);
assert.ok(typeof output.effort_rendered === 'string' && output.effort_rendered.length > 0,
`${label}: effort_rendered must be a non-empty string`);
assert.ok(output.effort_param === null || typeof output.effort_param === 'string',
`${label}: effort_param must be string or null`);
assert.ok(output.effort_propagation === null || typeof output.effort_propagation === 'string',
`${label}: effort_propagation must be string or null`);
assert.ok(typeof output.fast_mode === 'boolean',
`${label}: fast_mode must be a boolean`);
assert.ok(typeof output.fast_mode_supported === 'boolean',
`${label}: fast_mode_supported must be a boolean`);
}
test('gsd-planner (default claude runtime): full contract + known-agent shape', () => {
const result = runGsdTools(['resolve-execution', 'gsd-planner'], tmpDir, { HOME: tmpDir });
assert.ok(result.success, `Command failed: ${result.error}`);
const output = JSON.parse(result.output);
assertFullContract(output, 'gsd-planner/claude');
assert.strictEqual(output.effort_param, 'output_config.effort');
assert.strictEqual(output.effort_propagation, 'frontmatter');
assert.strictEqual(output.fast_mode_supported, false);
// known agent must NOT have unknown_agent:true
assert.ok(!output.unknown_agent, 'known agent must not have unknown_agent:true');
});
test('codex runtime: full contract + effort_param=model_reasoning_effort', () => {
writeConfig(tmpDir, { runtime: 'codex' });
const result = runGsdTools(['resolve-execution', 'gsd-planner'], tmpDir, { HOME: tmpDir });
assert.ok(result.success, `Command failed: ${result.error}`);
const output = JSON.parse(result.output);
assertFullContract(output, 'gsd-planner/codex');
assert.strictEqual(output.effort_param, 'model_reasoning_effort');
assert.strictEqual(output.fast_mode_supported, false);
});
test('gemini runtime: full contract + effort_param===null (no effort wire)', () => {
writeConfig(tmpDir, { runtime: 'gemini' });
const result = runGsdTools(['resolve-execution', 'gsd-planner'], tmpDir, { HOME: tmpDir });
assert.ok(result.success, `Command failed: ${result.error}`);
const output = JSON.parse(result.output);
assertFullContract(output, 'gsd-planner/gemini');
assert.strictEqual(output.effort_param, null);
assert.strictEqual(output.effort_propagation, null);
assert.strictEqual(output.fast_mode_supported, false);
});
test('unknown agent: full contract + unknown_agent===true', () => {
const result = runGsdTools(['resolve-execution', 'unknown-agent-xyz'], tmpDir, { HOME: tmpDir });
assert.ok(result.success, `Command failed: ${result.error}`);
const output = JSON.parse(result.output);
assertFullContract(output, 'unknown-agent-xyz');
assert.strictEqual(output.unknown_agent, true, 'unknown agent must have unknown_agent:true');
});
});
// ─── (d) TOTALITY across the real 33-agent registry ──────────────────────────
describe('#443 integration (d): totality across real registry', () => {
let tmpDir;
before(() => { tmpDir = createTempProject(); });
after(() => { cleanup(tmpDir); });
const registeredAgents = Object.keys(catalog.agents);
// Confirm we're covering the full registry — snapshot the count so a
// catalog shrink is caught by this assertion.
test(`registry has at least 33 agents (currently ${registeredAgents.length})`, () => {
assert.ok(registeredAgents.length >= 33,
`Expected at least 33 agents in registry, got ${registeredAgents.length}`);
});
test(`all ${registeredAgents.length} agents: resolveEffortInternal returns a VALID_EFFORTS member`, () => {
const effortSet = new Set(VALID_EFFORTS);
const bad = [];
for (const agent of registeredAgents) {
const effort = resolveEffortInternal(tmpDir, agent);
if (effort === undefined || effort === null || !effortSet.has(effort)) {
bad.push(`${agent}: got ${JSON.stringify(effort)}`);
}
}
assert.strictEqual(bad.length, 0,
`Agents with invalid effort:\n${bad.join('\n')}`);
});
test(`all ${registeredAgents.length} agents: resolveFastModeInternal returns strict boolean`, () => {
const bad = [];
for (const agent of registeredAgents) {
const fm = resolveFastModeInternal(tmpDir, agent);
if (typeof fm !== 'boolean') {
bad.push(`${agent}: got ${JSON.stringify(fm)} (${typeof fm})`);
}
}
assert.strictEqual(bad.length, 0,
`Agents with non-boolean fast_mode:\n${bad.join('\n')}`);
});
test(`all ${registeredAgents.length} agents: renderEffortForRuntime('claude', effort) stays in claude enum`, () => {
const claudeEnum = PROVIDER_EFFORT_ENUMS.claude;
const bad = [];
for (const agent of registeredAgents) {
const effort = resolveEffortInternal(tmpDir, agent);
const rendered = renderEffortForRuntime('claude', effort);
if (!claudeEnum.has(rendered.value)) {
bad.push(`${agent}: effort=${effort} rendered=${rendered.value} not in claude enum`);
}
}
assert.strictEqual(bad.length, 0,
`Agents producing invalid claude effort:\n${bad.join('\n')}`);
});
});
// ─── (e) FAST-MODE HONESTY INVARIANT ─────────────────────────────────────────
describe('#443 integration (e): fast-mode honesty invariant', () => {
let tmpDir;
beforeEach(() => { tmpDir = createTempProject(); });
afterEach(() => { cleanup(tmpDir); });
// Sample of agents across all tiers to prove the invariant is not agent-specific
const testAgents = ['gsd-planner', 'gsd-executor', 'gsd-codebase-mapper', 'gsd-verifier'];
test('claude runtime: fast_mode_supported is ALWAYS false regardless of fast_mode config', () => {
const configs = [
{},
{ fast_mode: { enabled: true } },
{ fast_mode: { routing_tier_defaults: { heavy: true } } },
{ fast_mode: { agent_overrides: { 'gsd-planner': true } } },
];
for (const config of configs) {
writeConfig(tmpDir, config);
for (const agent of testAgents) {
const result = runGsdTools(['resolve-execution', agent], tmpDir, { HOME: tmpDir });
assert.ok(result.success, `Command failed for ${agent}: ${result.error}`);
const output = JSON.parse(result.output);
assert.strictEqual(output.fast_mode_supported, false,
`claude/${agent}: fast_mode_supported must be false (Claude has no per-subagent fast-mode mechanism); config=${JSON.stringify(config)}`);
}
}
});
test("RUNTIMES_WITH_FAST_MODE.has('api') === true (api is the only fast-mode capable runtime)", () => {
assert.ok(RUNTIMES_WITH_FAST_MODE.has('api'),
"RUNTIMES_WITH_FAST_MODE must include 'api' — this is the only runtime with per-call fast_mode support");
});
test("RUNTIMES_WITH_FAST_MODE.has('claude') === false (claude fast-mode is session-level only)", () => {
assert.ok(!RUNTIMES_WITH_FAST_MODE.has('claude'),
"RUNTIMES_WITH_FAST_MODE must NOT include 'claude' — emitting fast_mode frontmatter on a Claude subagent is a silent no-op");
});
test("RUNTIMES_WITH_FAST_MODE.has('codex') === false", () => {
assert.ok(!RUNTIMES_WITH_FAST_MODE.has('codex'),
"codex does not support per-call fast_mode");
});
test("RUNTIMES_WITH_FAST_MODE.has('gemini') === false", () => {
assert.ok(!RUNTIMES_WITH_FAST_MODE.has('gemini'),
"gemini does not support per-call fast_mode");
});
});
// ─── (f) PRECEDENCE MATRIX ───────────────────────────────────────────────────
describe('#443 integration (f): precedence matrix (property/table-driven)', () => {
let tmpDir;
beforeEach(() => { tmpDir = createTempProject(); });
afterEach(() => { cleanup(tmpDir); });
// Effort: first-valid-wins from highest precedence to lowest
// 1. opts.override (invocation)
// 2. effort.agent_overrides.<agent>
// 3. effort.routing_tier_defaults.<tier>
// 4. effort.default
// 5. manifest tier default
// 6. hardcoded 'high'
const effortPrecedenceTable = [
{
label: 'layer 1 (invocation override) beats all',
config: {
effort: {
agent_overrides: { 'gsd-planner': 'low' },
routing_tier_defaults: { heavy: 'medium' },
default: 'xhigh',
},
},
opts: { override: 'minimal' },
expected: 'minimal',
},
{
label: 'layer 2 (agent_override) beats tier default and default',
config: {
effort: {
agent_overrides: { 'gsd-planner': 'low' },
routing_tier_defaults: { heavy: 'medium' },
default: 'xhigh',
},
},
opts: {},
expected: 'low',
},
{
label: 'layer 3 (routing_tier_defaults) beats effort.default',
config: {
effort: {
routing_tier_defaults: { heavy: 'medium' },
default: 'xhigh',
},
},
opts: {},
expected: 'medium',
},
{
label: 'layer 4 (effort.default) when no tier default set',
config: {
effort: { default: 'low' },
},
opts: {},
expected: 'low',
},
{
label: 'invalid layer 1 (turbo) falls through to layer 2 (agent_override)',
config: {
effort: { agent_overrides: { 'gsd-planner': 'medium' } },
},
opts: { override: 'turbo' },
expected: 'medium',
},
{
label: 'invalid layer 2 (agent_override=123 numeric) falls through to tier default',
config: {
effort: {
agent_overrides: { 'gsd-planner': 123 },
routing_tier_defaults: { heavy: 'high' },
},
},
opts: {},
expected: 'high',
},
{
label: 'invalid tier default (turbo) falls through to effort.default',
config: {
effort: {
routing_tier_defaults: { heavy: 'turbo' },
default: 'low',
},
},
opts: {},
expected: 'low',
},
];
for (const row of effortPrecedenceTable) {
test(`effort precedence: ${row.label}`, () => {
writeConfig(tmpDir, row.config);
const result = resolveEffortInternal(tmpDir, 'gsd-planner', row.opts);
assert.strictEqual(result, row.expected,
`Expected '${row.expected}', got '${result}' — config: ${JSON.stringify(row.config)}`);
});
}
// fast_mode precedence:
// 1. opts.override (strict boolean only)
// 2. fast_mode.agent_overrides.<agent> (strict boolean only)
// 3. fast_mode.routing_tier_defaults.<tier> (strict boolean only)
// 4. fast_mode.enabled (strict boolean only)
// 5. false
const fastModePrecedenceTable = [
{
label: 'layer 1 (opts.override=false) beats enabled=true',
config: { fast_mode: { enabled: true } },
opts: { override: false },
expected: false,
},
{
label: 'layer 2 (agent_override=true) beats tier default',
config: {
fast_mode: {
agent_overrides: { 'gsd-planner': true },
routing_tier_defaults: { heavy: false },
enabled: false,
},
},
opts: {},
expected: true,
},
{
label: 'layer 3 (tier default=true) beats enabled=false',
config: {
fast_mode: {
routing_tier_defaults: { heavy: true },
enabled: false,
},
},
opts: {},
expected: true,
},
{
label: 'layer 4 (enabled=true) when no tier/agent overrides',
config: { fast_mode: { enabled: true } },
opts: {},
expected: true,
},
{
label: 'layer 5 (default false) when all absent',
config: {},
opts: {},
expected: false,
},
{
label: 'string "true" in opts.override is NOT accepted (falls through)',
config: { fast_mode: { enabled: true } },
// override must be strict boolean; string falls through to next layer
opts: { override: 'true' },
// 'true' as string is not boolean -> falls through to tier default
// gsd-planner is heavy; no tier default set; falls to enabled=true
expected: true,
},
{
label: 'string "true" in agent_overrides is NOT accepted',
config: {
fast_mode: {
agent_overrides: { 'gsd-planner': 'true' },
enabled: false,
},
},
opts: {},
// string 'true' is not boolean -> fall through to tier default -> enabled=false -> false
expected: false,
},
];
for (const row of fastModePrecedenceTable) {
test(`fast_mode precedence: ${row.label}`, () => {
writeConfig(tmpDir, row.config);
const result = resolveFastModeInternal(tmpDir, 'gsd-planner', row.opts);
assert.strictEqual(result, row.expected,
`Expected ${row.expected}, got ${result} — config: ${JSON.stringify(row.config)}`);
});
}
});
// ─── (g) DYNAMIC-ROUTING COMPOSITION ─────────────────────────────────────────
describe('#443 integration (g): dynamic-routing composition', () => {
let tmpDir;
beforeEach(() => { tmpDir = createTempProject(); });
afterEach(() => { cleanup(tmpDir); });
const dynamicRoutingBase = {
dynamic_routing: {
enabled: true,
tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' },
escalate_on_failure: true,
max_escalations: 4,
},
effort: { routing_tier_defaults: { light: 'low' } },
};
test('resolveEffortForTier escalates independently of model resolution', () => {
writeConfig(tmpDir, dynamicRoutingBase);
const effort0 = resolveEffortForTier(tmpDir, 'gsd-codebase-mapper', 0);
const effort1 = resolveEffortForTier(tmpDir, 'gsd-codebase-mapper', 1);
const effort2 = resolveEffortForTier(tmpDir, 'gsd-codebase-mapper', 2);
assert.strictEqual(effort0, 'low');
assert.strictEqual(effort1, 'medium');
assert.strictEqual(effort2, 'high');
// Verify the effort ladder steps up correctly without asserting model value
// (model timing is a separate concern from effort escalation)
assert.notStrictEqual(effort0, effort1, 'effort should escalate at attempt 1');
assert.notStrictEqual(effort1, effort2, 'effort should escalate at attempt 2');
});
test('escalate_on_failure=false: attempt is ignored for effort', () => {
writeConfig(tmpDir, {
...dynamicRoutingBase,
dynamic_routing: {
...dynamicRoutingBase.dynamic_routing,
escalate_on_failure: false,
},
});
const e0 = resolveEffortForTier(tmpDir, 'gsd-codebase-mapper', 0);
const e1 = resolveEffortForTier(tmpDir, 'gsd-codebase-mapper', 1);
const e3 = resolveEffortForTier(tmpDir, 'gsd-codebase-mapper', 3);
assert.strictEqual(e0, e1, 'effort must not escalate when escalate_on_failure=false');
assert.strictEqual(e0, e3, 'effort must not escalate when escalate_on_failure=false');
});
test('escalation clamps at "max" regardless of attempt number', () => {
writeConfig(tmpDir, {
dynamic_routing: {
enabled: true,
tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' },
escalate_on_failure: true,
max_escalations: 99,
},
effort: { default: 'max' },
});
// Any large attempt number — result must never exceed 'max'
const r = resolveEffortForTier(tmpDir, 'gsd-planner', 50);
assert.strictEqual(r, 'max', `Effort must clamp at 'max', got '${r}'`);
const EFFORT_LADDER = VALID_EFFORTS;
const maxIdx = EFFORT_LADDER.indexOf('max');
const rIdx = EFFORT_LADDER.indexOf(r);
assert.ok(rIdx <= maxIdx, 'Effort must not exceed the max position in the ladder');
});
test('respects max_escalations cap: attempt beyond cap gives same as cap', () => {
writeConfig(tmpDir, {
dynamic_routing: {
enabled: true,
tier_models: { light: 'haiku', standard: 'sonnet', heavy: 'opus' },
escalate_on_failure: true,
max_escalations: 1,
},
effort: { routing_tier_defaults: { light: 'low' } },
});
const atCap = resolveEffortForTier(tmpDir, 'gsd-codebase-mapper', 1); // 1 escalation
const beyond = resolveEffortForTier(tmpDir, 'gsd-codebase-mapper', 5); // capped at 1
assert.strictEqual(atCap, beyond,
'Effort beyond max_escalations must be same as at cap');
assert.strictEqual(atCap, 'medium', 'low + 1 escalation = medium');
});
test('dynamic_routing disabled: resolveEffortForTier ignores attempt', () => {
writeConfig(tmpDir, {
effort: { routing_tier_defaults: { light: 'low' } },
});
const e0 = resolveEffortForTier(tmpDir, 'gsd-codebase-mapper', 0);
const e5 = resolveEffortForTier(tmpDir, 'gsd-codebase-mapper', 5);
assert.strictEqual(e0, e5, 'Effort must not change when dynamic_routing is disabled');
assert.strictEqual(e0, 'low');
});
});
// ─── (h) CONFIG-TOOLING ROUND-TRIP ───────────────────────────────────────────
describe('#443 integration (h): config-tooling round-trip', () => {
let tmpDir;
beforeEach(() => { tmpDir = createTempProject(); });
afterEach(() => { cleanup(tmpDir); });
test('config-set effort.default then resolve-execution reflects new value', () => {
const setResult = runGsdTools(['config-set', 'effort.default', 'low'], tmpDir, { HOME: tmpDir });
assert.ok(setResult.success, `config-set effort.default failed: ${setResult.error}`);
const execResult = runGsdTools(['resolve-execution', 'unknown-agent-xyz'], tmpDir, { HOME: tmpDir });
assert.ok(execResult.success, `resolve-execution failed: ${execResult.error}`);
const output = JSON.parse(execResult.output);
// unknown agent falls through to effort.default
assert.strictEqual(output.effort, 'low',
`Expected effort='low' after config-set, got '${output.effort}'`);
});
test('config-set effort.routing_tier_defaults.heavy then resolve-execution uses it', () => {
const setResult = runGsdTools(
['config-set', 'effort.routing_tier_defaults.heavy', 'medium'],
tmpDir, { HOME: tmpDir }
);
assert.ok(setResult.success, `config-set failed: ${setResult.error}`);
const execResult = runGsdTools(['resolve-execution', 'gsd-planner'], tmpDir, { HOME: tmpDir });
assert.ok(execResult.success, `resolve-execution failed: ${execResult.error}`);
const output = JSON.parse(execResult.output);
// gsd-planner is heavy; tier default now overridden to medium
assert.strictEqual(output.effort, 'medium',
`Expected effort='medium' after routing_tier_defaults override, got '${output.effort}'`);
});
test('config-set effort.agent_overrides.<agent> wins over tier default', () => {
// Set tier default first, then per-agent override
runGsdTools(['config-set', 'effort.routing_tier_defaults.heavy', 'medium'], tmpDir, { HOME: tmpDir });
const setResult = runGsdTools(
['config-set', 'effort.agent_overrides.gsd-planner', 'xhigh'],
tmpDir, { HOME: tmpDir }
);
assert.ok(setResult.success, `config-set agent_overrides failed: ${setResult.error}`);
const execResult = runGsdTools(['resolve-execution', 'gsd-planner'], tmpDir, { HOME: tmpDir });
assert.ok(execResult.success, `resolve-execution failed: ${execResult.error}`);
const output = JSON.parse(execResult.output);
assert.strictEqual(output.effort, 'xhigh',
`Expected agent_overrides to win (xhigh), got '${output.effort}'`);
});
test('config-set fast_mode.enabled true then resolve-execution reflects fast_mode=true', () => {
const setResult = runGsdTools(['config-set', 'fast_mode.enabled', 'true'], tmpDir, { HOME: tmpDir });
assert.ok(setResult.success, `config-set fast_mode.enabled failed: ${setResult.error}`);
const execResult = runGsdTools(['resolve-execution', 'gsd-planner'], tmpDir, { HOME: tmpDir });
assert.ok(execResult.success, `resolve-execution failed: ${execResult.error}`);
const output = JSON.parse(execResult.output);
assert.strictEqual(output.fast_mode, true,
`Expected fast_mode=true after config-set, got ${output.fast_mode}`);
// fast_mode_supported stays false (claude runtime)
assert.strictEqual(output.fast_mode_supported, false);
});
test('config-set fast_mode.agent_overrides.<agent> true reflects in output', () => {
const setResult = runGsdTools(
['config-set', 'fast_mode.agent_overrides.gsd-codebase-mapper', 'true'],
tmpDir, { HOME: tmpDir }
);
assert.ok(setResult.success, `config-set failed: ${setResult.error}`);
const execResult = runGsdTools(['resolve-execution', 'gsd-codebase-mapper'], tmpDir, { HOME: tmpDir });
assert.ok(execResult.success, `resolve-execution failed: ${execResult.error}`);
const output = JSON.parse(execResult.output);
assert.strictEqual(output.fast_mode, true,
`Expected fast_mode=true for agent-specific override`);
});
// Prove the config-set commands accept all the new key namespaces (schema validation)
test('config-set accepts all effort/* and fast_mode/* key namespaces without error', () => {
const keysToTest = [
['effort.default', 'high'],
['effort.routing_tier_defaults.light', 'low'],
['effort.routing_tier_defaults.standard', 'medium'],
['effort.routing_tier_defaults.heavy', 'xhigh'],
['effort.agent_overrides.gsd-executor', 'high'],
['fast_mode.enabled', 'false'],
['fast_mode.routing_tier_defaults.light', 'false'],
['fast_mode.routing_tier_defaults.standard', 'false'],
['fast_mode.routing_tier_defaults.heavy', 'false'],
['fast_mode.agent_overrides.gsd-verifier', 'false'],
];
for (const [key, val] of keysToTest) {
const r = runGsdTools(['config-set', key, val], tmpDir, { HOME: tmpDir });
assert.ok(r.success, `config-set '${key}' '${val}' should succeed, got: ${r.error}`);
}
});
});