f7df920681f233ae0fe064ee659550bdf41ff708
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tom Boucher
|
004e9dd741 |
fix(#3007): resolve Codex reasoning effort per model and make every clamp visible (#3765)
* test(#3007): failing-first suite for per-model Codex effort capability RED by construction. Binds to behavior renderEffortForRuntime does not yet have: an optional third `model` argument, a per-model advertised-level table, `max` passing through instead of clamping to `xhigh`, `minimal` clamping to `low`, `ultra` rejected outright, and clamp visibility (`requested`/`clamped`/ `reason`) so a downgrade is legible from resolver output rather than silent. Two of these pin defects that exist on next today: - `max` is discarded. Both Codex models whose catalog entries are retrievable (sol, luna) advertise `max`; GSD clamps it to `xhigh` and reports nothing. - `minimal` is emitted to a model that refuses it. providerPresets.openai. haiku.low pairs gpt-5.6-luna with reasoning_effort "minimal", and luna's advertised floor is `low`. GSD is sending a value into a document Codex itself validates. The parity test is what pins that fixed, and it names the offending path/model/effort when it trips. Also corrects tests/model-resolver.test.cjs:351, which asserted renderEffortForRuntime('codex','max').value === 'xhigh' -- the defect pinned as though it were a contract. ADR-443 recorded "Codex has no max" as fact and it was true when written; Codex has since added both `max` and `ultra`. That is a stale premise, so the assertion is corrected here rather than worked around. The property test asserts the invariant the whole change exists for: a rendered effort is always a level the target model actually advertises, or an explicit rejection. There is no third outcome. * fix(#3007): resolve Codex effort per model, and make every clamp visible Codex declares supported_reasoning_levels per MODEL and validates against it, so a single per-runtime capability set cannot be right for all of them. GSD's was wrong in both directions at once. `max` reaches Codex now. ADR-443 recorded "Codex has no max" as fact and clamped max -> xhigh on that basis; it was accurate when written, and Codex has since added both `max` and `ultra`. Every Codex model whose catalog entry is retrievable advertises `max`, so the clamp was discarding a level the provider supports, silently, on the most-used path. `minimal` stops reaching Codex. No Codex model advertises it -- both retrievable entries floor at `low` -- yet providerPresets.openai.haiku.low paired gpt-5.6-luna with reasoning_effort "minimal". GSD was writing a value the receiver validates and refuses into a file the receiver reads. Being unconservative in what you send is the half of Postel's rule with no defensible reading, so that preset is corrected and a parity test pins it. `ultra` is refused rather than laddered. Codex's own catalog calls it "Maximum reasoning with automatic task delegation": at ultra, effective_multi_agent_mode returns Proactive and Codex spawns sub-agents on its own initiative, underneath GSD's orchestration rather than inside it (#2167). It is a mode switch, not a reasoning depth, so it is not added to the universal ladder -- which stays provider-agnostic by ADR-443's design -- and it is rejected even for gpt-5.6-sol, which does advertise it. Clamping it down to `max` was considered and rejected: that silently discards what the user actually asked for. Clamping is now visible. RenderedEffort carries requested/clamped/reason and resolve-execution surfaces them. The previous table clamped correctly but invisibly, so a user asking for `max` on Codex had no way to find out they were getting `xhigh` -- exactly the failure mode the robustness principle's modern critique warns about, and why "be liberal" has to mean "liberal and loud". Also closes a latent trap found while reviewing the implementation: the clamp-up loop walks the ladder upward, and for a future model advertising `ultra` but not `max` it would have selected `ultra` as the clamp target -- re-entering by the back door the mode the rejection above exists to keep out. A clamp may never produce a value that a direct request for that value would refuse. Unreachable with today's catalog, which is why no test caught it; a test now asserts the invariant directly. Signature stability is preserved: the third `model` argument is optional and the two-argument form still resolves, against the family baseline. That form's BEHAVIOR does change for `max` and `minimal`, and it must -- keeping the old answer would have fixed the defect only where a model happened to be threaded through and left it live everywhere else. tests/model-resolver.test.cjs:351 asserted the defect as if it were a contract and is corrected here rather than worked around. * fix(#3007): close every review finding on the Codex effort alignment Two isolated reviewers, correctness and security. Both found the same two blockers, and the per-model work was inert on every surface that matters until this commit. BLOCKER — resolve-execution never passed the model and discarded the clamp. cmdResolveExecution called the two-argument form and emitted only effort_rendered/effort_param/effort_propagation, so the per-model table was unreachable from production code (tests were its only caller) and requested/ clamped/reason were computed and thrown away. Requested outcome 3 names "the effective rendered effort in resolver output" specifically, so the feature was unmet on the exact surface the issue asks for. Now passes the resolved model and emits effort_requested / effort_clamped / effort_clamp_reason, flat, matching the existing key convention rather than introducing a nested object. BLOCKER — the docs described output that did not exist. CONFIGURATION.md showed a nested {"effort": ...} sample; the real result is flat and those keys were absent entirely. A reference doc asserting a JSON path a reader can copy is worse than no doc. Corrected against the actual emitted key set. MAJOR — the argv channel still shipped both original defects. EFFORT_ARGV.codex kept minimal in its supported set and still clamped max down to xhigh, so the invocation-time and install-time channels disagreed about the same runtime's capability: --host codex with max emitted xhigh while the generated TOML said max. This is the repo's documented generative-fix-divergence class, so both tables now cross-reference each other and a parity test fails if they ever diverge again. MAJOR — malformed catalog data failed OPEN and could crash the CLI. A null _baseline became an EMPTY Set that is nonetheless truthy, so the nullish fallback never fired and every effort rendered as null. And a non-array value made the Set constructor throw at module load — model-catalog.cjs is required across the whole CLI, so one bad JSON value killed every command, not just codex effort. Guarded on size and filtered to array values; both degrade to the hardcoded baseline. MAJOR — value widened to a nullable string with two consumers left behind. runtime-artifact-conversion passed it straight into injectEffortFrontmatter (a null effort key in generated frontmatter); install-effort-resolver still declared a non-nullable return, a structural lie that silently defeated null checking. Both corrected, both omitting the key on null — the same posture as 'inherit', where omission means "follow the host default". MAJOR — the per-model table is inert today, and the docs now say so. All three shipped models advertise the same usable range and ultra (sol's only differentiator) is rejected for every model, so no observable output differs by model. The table stays because Codex declares capability per model and the sets are free to diverge — a single per-runtime assumption is precisely what went stale and produced this issue — but overselling it as a visible per-model feature would have been the same class of error as the doc blocker above. Tests: three passed under a full revert and are strengthened rather than deleted, since each guards a real contract (#3533's inherit rule, the undeclared-host rule, off-ladder handling) — they now also assert the clamp-visibility fields, which only exist after this change. The fast-check property is kept for its shrinking, and a deterministic nested loop over the full cross-product now sits beside it so coverage is exhaustive rather than sampled. Also folded in earlier: bin/install.js generated the Codex TOML with the two-arg form and would have written a literal null reasoning effort on the ultra path; CONTEXT.md's Model Catalog Module glossary entry now records CODEX_MODEL_EFFORT. The installer defect was found by the co-change gate, not by a reviewer — install.js is a historical co-change partner of model-catalog.cts that this diff had not touched. * test(#3007): correct assertions that pinned Codex's stale effort premise Thirteen pre-existing tests encoded "Codex has no max" as fact and failed on the shipped commit. Every one is a stale pin, not a defect: each was probed against the built module before its expectation was changed, and none failed for a reason other than this premise correction. Kept as its own commit per CONTRIBUTING — a test-fixture correction made stale by a production change must not ride inside another commit, because the release-sdk hotfix cherry-pick filter routes by subject prefix and a correction buried under the wrong prefix ships a half-state (v1.42.3, #3621). The most valuable one was tests/model-resolver.test.cjs's cross-provider validity invariant, which hardcoded the Codex enum as `minimal|low|medium|high|xhigh` and failed with "real API would 400". That message is now false in both directions: Codex accepts `max`, and rejects `minimal`, which no model advertises. The enum is corrected to `low|medium|high|xhigh|max` and the guard is kept intact — it is exactly the "would the real API refuse this" check worth having, and it was right to fail here. It simply carried the stale fact in its own fixture. Test NAMES were corrected alongside their assertions wherever the name asserted the old behavior — "max is Anthropic-only", "max clamps to xhigh", "minimal passthrough". A renamed test that still claims the old thing is worse than a failing one, and a green test whose name states a falsehood is how the next reader inherits the wrong premise. Both channels are covered: install-time (renderEffortForRuntime, and the generated .toml in install-runtime-artifacts) and invocation-time argv (effort-surface-axis). They were deliberately brought into agreement in this change, so their assertions had to move together. Each site carries a #3007 comment recording that Codex gained max/ultra and that capability is declared per model, so a future reader can tell this was a deliberate premise correction rather than a test bent to fit an implementation. * test(#3007): separate the effort-precedence case from the clamp case The previous stale-assertion pass over-corrected one test. It saw `effort: { default: 'max' }` on codex expecting `effort_rendered: 'xhigh'`, assumed the xhigh came from the max→xhigh clamp #3007 removes, renamed it to "max passes through" and changed the expectation to `max`. The remote runner disagreed. Reproduced against the real CLI: with that config and `gsd-planner`, the resolver emits `effort: "xhigh"`, `effort_requested: "xhigh"`, `effort_clamped: false`. The xhigh is produced by effort-resolution PRECEDENCE — gsd-planner is heavy/opus tier and its routing-tier default outranks `effort.default` — so `max` never reaches the renderer at all. The test says nothing about clamping and never did; it only looked like a clamp pin because both mechanisms happened to yield the same string. Restored to `xhigh` and renamed to say what it actually tests. It now also asserts `effort_clamped === false` and `effort_requested === 'xhigh'`, which is what makes it impossible to mistake for a clamp pin again: those two fields prove the value is what the resolver produced rather than something the renderer downgraded. Before #3007 there was no way to tell the two apart from the output — which is precisely why the previous pass could not tell them apart either. Added the test that was actually missing: `effort.agent_overrides`, which outranks the tier default, so the requested level genuinely reaches the renderer and `max` survives to `effort_rendered` end-to-end through the real CLI. Verified by probe before asserting. One test now pins the precedence rule and the other pins the #3007 behavior, and neither can be read as the other. That the clamp-visibility fields are what resolved this is a small argument for having added them. * chore(#3007): backfill changeset pr number to 3765 * test(#3007): put model-catalog under the mutation gate The Stryker shard showed as `skipping` on this PR despite the diff rewriting model-catalog's effort logic. That was legitimate, not a detection bug: `model-catalog` was never in scripts/mutation-matrix.cjs's COVERED map, so the whole module — including everything #3007 touches — sat entirely outside mutation scoring with has_work "false". Registered, with a dedicated spawn-free surface. tests/model-catalog.unit.test.cjs is new: 44 in-process tests, no runGsdTools, no child process, no filesystem, no temp dirs. That shape is not stylistic — it is the #2790 precedent this file already documents. Stryker's command runner treats a whole `node --test <file>` invocation as ONE test costing whatever its slowest case costs, and re-runs it per mutant, so pointing a shard at tests/model-resolver.test.cjs (which uses runGsdTools throughout) would reproduce exactly the 15-minute shard-cap cancellation #2790 hit. The integration file is unaffected and keeps running in full in the normal test job. Coverage spans the module rather than only the diff, because the score is measured over the whole file: effort rendering across every model and ladder level in both channels, the prototype-chain host guard, the exported enums and maps, isAnthropicFlavoredModel's provider namespacings, the profile projections, nextTier, and mergeEffortTierDefaults. The last two were nearly left out and are worth naming — every uncovered exported function is score given away, and mergeEffortTierDefaults turned out to have a genuinely interesting contract (#3531: a partial override merges over the built-ins rather than replacing them, and isValid gates the VALUE, not the tier name, so an unknown tier key is still merged in). Every expectation was probed against the built module before being asserted. minScore is 1 and that is a PLACEHOLDER, flagged as such in the registry comment. Floors in this repo are measured, not chosen — the existing entries sit at 94, 75 and 56 — and they can only be measured in CI, because mutation shards run `node --test`, which is hard-blocked locally. The first CI run on this branch reports the real number and the floor gets ratcheted to it before merge. A placeholder of 1 reaching `next` would make the gate decorative: it would pass whether or not a single mutant is ever killed. Note the target is "never regress from measured", not a fixed 80 — planning-inspect sits at 56 and is documented as an accepted ratchet candidate. * test(#3007): bootstrap model-catalog's mutation floor legally The placeholder floor was structurally illegal and the remote run said so. tests/mutation-matrix-ratchet.test.cjs guards the guard: every COVERED module must carry a matching RATCHET_BASELINE entry in the same diff, minScore must EQUAL that baseline, and it must be at least 50. `minScore: 1` failed all three. That is the ratchet working exactly as intended — a floor nobody can satisfy accidentally is the point of it. Bootstrapped at 50 in both places. Fifty is not a measured score and the comment says so plainly: it is the minimum the guard permits, and it coincides with Stryker's own configured `break` threshold, so it is the lowest legal starting point for a module that has never been measured. It still must be ratcheted to floor(measured) - 1 before this PR merges. Also corrected a real defect in the file's own instructions. "HOW TO UPDATE" step 1 read "Run the per-module Stryker shard locally" — which cannot be done here, and which the same file contradicts eighty lines further down, where the #2790 scores are recorded as "not a local run; mutation shards run `node --test`, hard-blocked in this repo's local environment". stryker.config.mjs confirms the command runner invokes `node --test` once per mutant, and .claude/hooks/block-local-node-test.sh denies exactly that. So the documented first step sends the next contributor at a wall. Rewritten to describe the path that works — push, read the measured score off the CI shard, then set the floor and its baseline together in one diff — and to say why local measurement is not available, so nobody rediscovers it the slow way. GOODHART SAFETY is untouched. The two-step is inherent to the environment rather than a shortcut: a floor cannot be measured before the first CI run exists, and the guard rightly refuses to accept an unmeasured one below its minimum. * test(#3007): ratchet model-catalog's mutation floor to its measured score The shard ran in CI and reported 59.62% — 248 mutants killed, 168 survived, no timeouts, no errors (run 32605073352, job 97108869486). Floor set to 58 per this file's own rule, minScore = floor(measured) - 1, which is the same arithmetic every sibling entry used: 57.03 to 56, 76.58 to 75, 95.65 to 94. Both halves moved together, because the ratchet guard asserts minScore equals its RATCHET_BASELINE entry and would reject them drifting apart. The spawn-free unit surface is vindicated by the clock: 57 seconds, against a 15-minute shard cap and a 9m46s frontmatter shard in the same run. That was the whole reason for creating tests/model-catalog.unit.test.cjs rather than pointing the shard at tests/model-resolver.test.cjs — #2790 recorded shards being CANCELLED at that cap when they targeted a runGsdTools-heavy integration file. The registry comment is rewritten rather than deleted. It previously warned that the floor was provisional and must not ship that way; leaving that text next to a measured floor would make the file lie in the other direction. It now records the measurement the way the sibling entries do, including that 59.62 sits below TARGET (80) and is therefore a ratchet candidate like planning-inspect at 56 — comfortably clear of its own floor with real room to grow. Raise it as the tests improve; never lower it. Worth stating plainly: 168 surviving mutants is not a clean bill of health. It is an honest floor for a module that had NO mutation coverage at all an hour ago, and it is now pinned so it cannot silently regress. --------- Co-authored-by: sim <sim@local> |
||
|
Tom Boucher
|
8526bd46f8 |
enhance(#2475): scope ADR-443 item 1 to the operator surface and ratify the ADR (#3688)
* test(#2475): widen the item-1 effort-caller guard to both CLI argument shapes The guard matched only `resolve-execution ... --effort\s`, but the CLI also accepts `--effort=<level>` (gsd-core/bin/gsd-tools.cjs). A workflow written with the equals form was a live invocation-override caller the guard passed silently, along with `--effort` at end-of-input. Lift the matcher to a shared predicate and assert it directly against every shape the CLI accepts, plus the decoys it must not fire on (--effortless, a bare --effort with no resolve-execution, item 6's --attempt caller, a call and flag split across lines). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#2475): scope ADR-443 item 1 to the operator surface and ratify the ADR ADR-443 sat at Proposed on one condition: its Decision item 1 orchestrator invocation override needed a caller in shipped orchestration. Per the maintainer's ruling, take unblock path (b) for item 1 only -- record that the override is an operator-facing CLI surface, deliberately not driven by shipped orchestration, and ratify. The ADR's own path (b) wording is not adopted verbatim: it says the scope is limited to static install-time propagation, which is false on both counts -- item 6 has a live caller (#2296) and #2481 delivered a live invocation-time argv channel. Only one precedence step is narrowed. No consumer was invented to clear the gate: nobody has asked for a per-run effort override, and #2475's actual complaint is already closed by the cascade-to-argv path. The amendment states explicitly that --effort remains supported and is not deprecated, so the scoping is not read as dead code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(#2475): correct two bare-`gsd` invocation examples to `gsd_run` There is no `gsd` binary -- package.json exposes gsd-core, gsd-tools, gsd_run and gsd-mcp-server. Both sites presented a command that cannot run as written. One is in this branch's own new ADR-443 amendment; the other is a pre-existing error in the docs/CONFIGURATION.md assumption_delta row, fixed here rather than deferred. No translated copy carries either line, so no i18n drift is created. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#2475): record the guard/CLI divergence risk on the item-1 matcher The predicate independently models gsd-tools.cjs's argument parser rather than sharing a constant with it, so a third --effort spelling would leave the guard reporting green while ADR-443's ratifying invariant silently stopped holding. Name that risk where the next editor will meet it. Also restores the bounded-prose rationale that was attached to the eslint directive removed in 39793079c -- the directive went unused once the regex moved to a const, but the reasoning it carried is still worth having. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: sim <sim@local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
sim
|
1bc7f7e6b0 |
test(#3339): fold the state/phase/dispatch & model-profile issue-* cluster — Wave 7
Folds 9 legacy issue-*.test.cjs regression files (140 test() blocks) into their module's main suite, per H3 (#3315) of the test-hygiene epic (#3053). LAST of 4 issue-* waves — closes out the 74-file fix-*/issue-* backlog (pending BUG_FILE_RE extension, held for a follow-up commit until Wave 6 is confirmed merged, per the epic's own zero-backlog precondition). - issue-2828-flat-roadmap-total-phases.test.cjs (1) + issue-3204-state- writer-phase-count.test.cjs (21): both target state-document.cjs buildStateFrontmatter via different CLI entrypoints — merged jointly into state-document.test.cjs, 0 dropped. - issue-2945-phase-complete-checkbox-rollback.test.cjs (4) + issue-2949- phase-complete-stage3-sentinel.test.cjs (4): both target phase.cts cmdPhaseComplete; issue explicitly warned of overlap — verified disjoint fixtures/assertions, 0 dropped, merged into phase.test.cjs. - issue-2927-reviewer-lane-overlay-invocation.test.cjs (10) merged into review-lane-descriptor.test.cjs. - issue-2939-dispatch-flatten-maxdepth.test.cjs (9) merged into host-integration.test.cjs, 2 dropped as verified exact duplicates. - issue-2977-frontmatter-bom.test.cjs (5) merged into frontmatter.test.cjs. - issue-2045-third-party-skills-surface.test.cjs (6) merged into capability-loader.test.cjs. - issue-2517-runtime-aware-profiles.test.cjs (80, the largest single fold in the epic) merged into model-resolver.test.cjs, 1 dropped as a verified true duplicate (checked against src/model-resolver.cts logic, not just title similarity). Fixed a genuine eslint irregular-whitespace finding: a literal BOM character embedded in a doc comment (pre-existing content from the original #2977 source, illustrating what a BOM looks like) — replaced with a readable U+FEFF notation. 3 stale doc references found and fixed (docs/adr/2313, 3180, 443). Zero net test-coverage loss. No production code changed. |
||
|
Tom Boucher
|
09b535ac00 |
feat(#2481): add a negotiated effortSurface axis and wire invocation-time effort
ADR-1239 gains a ninth negotiated axis, effortSurface (argv | none), declaring how
a host accepts reasoning effort. ADR-443 is amended in the same change because its
recorded deferral is what the axis resolves: its Unblock condition offered paths
(a) and (b) and stated the choice was 'a maintainer call this file records but does
not make'. Path (a) is selected and satisfied here.
Before this, effort reached a runtime only through install-time channels
(EFFORT_RENDERING's frontmatter/api), so reviewer CLIs spawned as subprocesses
silently inherited whatever effort sat in the user's own global CLI config. The
review lane now resolves one universal effort through the ADR-443 cascade and
renders it per host through the negotiated descriptor.
Every per-host value is documentation-sourced, never inferred:
- claude argv -- verified via 'claude --help' (--effort <level>)
- opencode argv -- verified via 'opencode run --help' (--variant)
- codex argv -- codex-rs/exec/src/cli.rs: model_reasoning_effort is NOT a CLI
flag (config.toml key only), so the global -c override is the
only argv route
- 15 hosts undocumented -- their docs state no reasoning setting; the sentinel
fails closed rather than inheriting a profile baseline
No config-file vocabulary member: the only host that ever had one (Gemini CLI's
thinkingConfig) was removed as a sunset runtime by
|
||
|
Tom Boucher
|
67a9243cf1 |
chore(#2356): make the ADR index a generated artifact and enforce ADR lifecycle invariants (#2367)
* chore: rebuild ADR index as a generated artifact and enforce lifecycle invariants
The ADR index in docs/adr/README.md was hand-maintained with nothing checking
it, and had drifted to 40 of 65 ADRs. The absent rows included the entire
capability family (857/894/959/1016/1143/1213/1244) and ADR-1239 (EoS) itself,
so the decisions a reader most needed were the ones they could not find.
Make the index a derived artifact, matching the repo's existing generated-file
idiom (lint:generated-sync), and enforce the corpus' lifecycle invariants:
- scripts/gen-adr-index.cjs generates the index between markers and validates
the status vocabulary (Accepted/Proposed/Superseded/Legacy/Retired),
successor links, id/filename agreement, and supersession symmetry.
- Wire --check into lint:generated-sync so drift fails CI.
Correct the lifecycle metadata the gate surfaced, without flipping any status:
- ADR-1239 (EoS) declared it subsumed ADR-1016/58/3660/894; none recorded it.
Add reciprocal "Subsumed by" pointers + dated amendments. Subsumption keeps
the target Accepted -- these are live adapters, not dead decisions.
- ADR-857/894 carry dated status caveats: they read Proposed while the
capability system shipped and epic #857 is closed. Ratification is a
maintainer act and is deliberately left open.
- Link ADR-0005/0007/0012/3524 -> ADR-0174 and ADR-0010 -> ADR-0009; record
the reciprocal Supersedes on ADR-0009.
- ADR-218 declared itself "ADR-0175" -- an unfinished rename.
- The 0011 PRD moves from the non-canonical "Draft" to "Legacy".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: capture stderr via spawnSync; record ADR-0010 draft supersession
Two fixes surfaced by the first gsd-test run and by regenerating the index:
- tests/adr-index-gate.test.cjs used execFileSync, which only surfaces stderr
through the thrown error on non-zero exit. The `--write` path exits 0 while
reporting outstanding violations on stderr, so the helper always saw ''.
spawnSync captures both streams on both outcomes.
- The hand-maintained index recorded 0010-skill-surface-budget-module.md as
"earlier draft superseded by ADR-0011" while the file itself still said
Proposed. Deriving the index from the files would have dropped that
assertion and resurrected a superseded draft as a live decision, so it is
recorded at its source, with the reciprocal Supersedes on ADR-0011.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: drop the dead sdk/ model-catalog candidate retired by ADR-0174
src/model-catalog.cts resolved model-catalog.json through three candidates, the
second being sdk/shared/model-catalog.json three levels up. That was the legacy
source-repo fallback kept by the #3288 fix ("check the co-located path FIRST,
before the legacy source-repo path").
ADR-0174 then retired the @opengsd/gsd-sdk package boundary and deleted the sdk/
tree (
|
||
|
Tom Boucher
|
85ed50cc4f |
test(#1972): consolidate 94 command/module regression tests into subject suites
Fold 94 issue-named command/module regression files into the canonical test file that owns each subject-under-test, across 52 existing suites (state, config, frontmatter, roadmap-parser, capability-registry, shell-command-projection-dispatch, plan-phase-drift-guard, health-validation, runtime-converters, commands, etc.). Verbatim block-scoped describe wrappers; 881 subtests conserved 1:1. No new test files. Host-env pre-check (per B2): the only GSD_WORKSTREAM/GSD_PROJECT-touching destinations (intel, planning-workspace) clear those vars hermetically, so folded CLI tests are safe. Regenerates regression-name allowlist (222->162), ratchets file-count allowlist across 8 buckets (validate entry removed after dropping <=2), makes 34 relocated allow-test-rule exemptions issue-ref-compliant (ADR-456; prunes 34 stale ids). Repoints CONTEXT.md + ADR-0002/443/1235/3524 test-file references. lint:ci green. Part of epic #1969. Closes #1972. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Tom Boucher
|
5ca646f015 |
feat(#443): unified cross-provider effort controls + fast-mode-aware routing (#463)
* test(#443): RED unified effort + fast_mode + resolve-execution All 68 tests failing as expected — no implementation yet. Covers: effort cascade (tier defaults, overrides, invalid fallthrough), fast_mode cascade (boolean-only, tier defaults), resolveEffortForTier escalation, renderEffortForRuntime clamping, resolve-execution CLI, config schema new keys, QA hostile-input matrix. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(#443): unified cross-provider effort + fast_mode knobs and resolve-execution query Adds config-driven effort control (universal ladder: minimal<low<medium<high<xhigh<max) and fast_mode propagation knobs, with per-runtime rendering that clamps the unique tail values (max=Anthropic-only clamps to xhigh on Codex; minimal=Codex-only clamps to low on Claude). Key changes: - config-schema.manifest.json: add effort.default, fast_mode.enabled as validKeys; add 4 dynamicKeyPatterns for effort.routing_tier_defaults, effort.agent_overrides, fast_mode.routing_tier_defaults, fast_mode.agent_overrides; fix stale _comment - config-defaults.manifest.json: add effort and fast_mode blocks with tier defaults - model-catalog.cjs: add EFFORT_RENDERING map, renderEffortForRuntime(), RUNTIMES_WITH_FAST_MODE - model-profiles.cjs: re-export new catalog exports - core.cjs: add resolveEffortInternal, resolveFastModeInternal, resolveEffortForTier, VALID_EFFORTS, EFFORT_SET, nextEffort; pass effort/fast_mode through loadConfig - commands.cjs: replace reasoning_effort in cmdResolveModel with unified effort; add cmdResolveExecution (superset command with effort_rendered, effort_param, effort_propagation, fast_mode, fast_mode_supported) - gsd-tools.cjs: add resolve-execution case with --effort/--fast-mode/--attempt flags - tests/feat-443: 69 tests covering cascade, rendering, escalation, CLI, schema, QA matrix - tests/commands.test.cjs: convert 3 reasoning_effort assertions to unified effort - docs/CONFIGURATION.md: document effort + fast_mode + resolve-execution sections - settings-advanced.md: list new effort/fast_mode keys in confirmation table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(#443): remove dead catalog effort lane; unify codex effort through renderEffortForRuntime - Remove resolveReasoningEffortInternal (catalog-driven effort function) from core.cjs and its export; remove from commands.cjs destructure import - Convert tests/issue-2517-runtime-aware-profiles.test.cjs: all 11 effort assertions now use resolveEffortInternal + renderEffortForRuntime; Claude effort is first-class (output_config.effort); unknown runtimes assert param===null - Convert tests/feat-3023-model-phase-types.test.cjs: replace the entire resolveReasoningEffortInternal describe with unified effort assertions; effort derives from AGENT_DEFAULT_TIERS routing tier, not phase-type tier Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(#443): ADR for unified cross-provider effort + fast-mode routing Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(#443): architecture-level QA invariants + test-strategy doc Add 48-test integration suite (feat-443-effort-fast-mode.integration.test.cjs) covering 8 architectural invariants: cross-provider validity (never emit a value the real API would 400 on), param/channel contract stability, resolve-execution JSON contract (all 8 keys + correct types), totality across the full 33-agent registry, fast-mode honesty (claude always fast_mode_supported=false), precedence first-valid-wins matrix for both effort and fast_mode cascades, dynamic-routing composition (effort escalation independent of model tier), and config-set round-trip for all new effort/* and fast_mode/* key namespaces. Append test-strategy section with invariant rationale and E2E gap documentation to docs/TESTING-SUITES.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(#443): add failing install-wiring tests for effort per-runtime injection (RED) TDD RED: 10 failing tests covering: - Claude .md gets effort: injected per tier (planner=xhigh, mapper=low, executor=high) - Gemini .md does NOT get effort: (already passing — Gemini-safe) - Codex .toml gets model_reasoning_effort via unified resolver - Config-driven: effort.agent_overrides drives both Claude .md and Codex .toml - Source purity: agents/*.md have no effort: key (already passing) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(#443): wire effort per-runtime at install (Claude .md frontmatter + Codex .toml unified) - Import AGENT_DEFAULT_TIERS and renderEffortForRuntime from model-catalog.cjs - Add readGsdEffectiveEffortConfig(targetDir): reads merged effort config from .planning/config.json (per-project wins) + ~/.gsd/defaults.json (global fallback), same probe pattern as readGsdRuntimeProfileResolver - Add resolveInstallTimeEffort(effortCfg, agentName): pure function matching resolveEffortInternal() precedence (agent_overrides > routing_tier_defaults > default > 'high') without loadConfig side-effects (no sub-repo detection, no migration writes) - Claude agent copy loop: inject `effort: <value>` into frontmatter ONLY for runtime === 'claude'; all other .md runtimes (Gemini, Qwen, Hermes, etc.) stay effort-free (Gemini-safe source contract preserved in agents/*.md) - generateCodexAgentToml: add effortCfg param; emit model_reasoning_effort from unified resolver (replaces old catalog entry.reasoning_effort); Codex clamps max → xhigh via renderEffortForRuntime('codex', ...) - installCodexConfig: pass readGsdEffectiveEffortConfig(targetDir) to generateCodexAgentToml so per-project config wins for Codex .toml too - Update failing tests to GREEN: 12/12 pass; all 17 install tests pass; 2847/2848 unit tests pass (1 pre-existing failure: policy-shell-pinning) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(#443): source install effort defaults from manifest (kill drift) + guard test Replace hardcoded _GSD_EFFORT_MANIFEST_TIER_DEFAULTS and the 'high' fallback in resolveInstallTimeEffort with values read from config-defaults.manifest.json at module init, using the same __dirname-relative path install.js already uses for all shared manifests. Add feat-443-effort-defaults-drift.test.cjs to assert equality between install.js's runtime constants and the manifest on every CI run. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(#443): reconcile Codex TOML tests with unified effort design The #443 unified effort resolver makes generateCodexAgentToml always emit model_reasoning_effort (driven by resolveInstallTimeEffort, not model_profile_overrides). The test 'generated TOML omits reasoning_effort when runtime has none' had an obsolete premise — model_profile_overrides.reasoning_effort:'' no longer suppresses unified effort. Convert it to assert the new invariant: Codex TOML always carries a valid model_reasoning_effort from the agent's routing tier (xhigh for gsd-planner, a heavy-tier agent), while model_profile_overrides model override is still respected. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(#443): make install.js effort resolution lazy (no load-time side effects breaking launcher-parity) Replace module-load-time IIFE + hard throw (config-defaults.manifest.json read) and top-level require of model-catalog.cjs with a lazy _getGsdEffortCatalog() getter that initialises on first call from resolveInstallTimeEffort / generateCodexAgentToml / Claude .md effort injection. Requiring install.js in unrelated test contexts (e.g. runtime-launcher-parity) no longer triggers manifest IO or throws, eliminating the load-time side effect that changed subprocess exit codes / stderr on the bench. Drift-guard exports (_GSD_EFFORT_MANIFEST_TIER_DEFAULTS / _GSD_EFFORT_MANIFEST_DEFAULT) preserved as lazy getter properties on module.exports so feat-443-effort-defaults-drift still validates them without forcing eager load. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(#443): isolate install-wiring test HOME to stop \$HOME/.claude pollution breaking launcher-parity runGlobalInstall() now redirects HOME to a per-call isolated tmpdir in addition to the existing runtime-specific env-var redirects (CLAUDE_CONFIG_DIR, GEMINI_CONFIG_DIR, CODEX_HOME). This ensures install.js code that uses os.homedir() directly — including the ~/.cache/gsd update-check deletion, ~/.gsd/defaults.json reads, and any HOME-relative npm subprocess writes — never touches the real \$HOME during the test. Without the HOME isolation the install test (which is new to this branch and is now picked up by Docker's raw \`tests/*.test.cjs\` glob) could write or delete files under the real \$HOME, causing runtime-launcher-parity test (D) to fail: (D) asserts a loud non-zero exit when \$RUNTIME_DIR/gsd-tools.cjs is absent and gsd-tools is not on PATH, but the launcher's \$HOME/.claude fallback arm succeeds if \$HOME/.claude/get-shit-done/bin/gsd-tools.cjs exists. Also sets GSD_SKIP_STALE_SDK_CHECK=1 to suppress the \`npm ls -g\` subprocess that the global installer spawns — irrelevant to effort-wiring assertions, slow, and potentially writes to ~/.npm cache. All 12 feat-443 install-wiring assertions preserved. Drift-guard 5/5. Unit suite 2848/2850 (pre-existing policy-shell-pinning.test.cjs failure on next). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(#443): add changeset fragment for effort + fast-mode routing Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(#443): set GSD_TEST_MODE before requiring install.js in drift-guard test to prevent HOME leak Without GSD_TEST_MODE=1, require('bin/install.js') runs the module's main install block (guarded by !GSD_TEST_MODE), performing a real global Claude install into $HOME/.claude/. On CI ubuntu where node is on standard PATH, the launcher's $HOME/.claude fallback arm then finds gsd-tools.cjs, causing runtime-launcher-parity test (D) to exit zero when it must exit non-zero. Root cause: feat-443-effort-defaults-drift.test.cjs (unit suite) runs alphabetically before runtime-launcher-parity.test.cjs in the same node --test invocation. Each runs in a separate worker process but shares the same HOME. The drift test's install leaks gsd-tools.cjs into that HOME, then the launcher test's bash subprocess finds it via the $HOME/.claude arm. Fix: add process.env.GSD_TEST_MODE = '1' at the top of the drift-guard test, before the require(installPath) call. This matches the pattern used by feat-443-effort-fast-mode.test.cjs and feat-443-effort-install-wiring .install.test.cjs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(#443): deterministic resolve-execution arg parsing + validate install-time effort (Codex adversarial findings) Finding 1: resolve-execution --effort low gsd-planner misrouted 'low' as the agent. Replace find(non-dash) with a proper flag-consuming loop that collects a single positional; validate missing/extra positionals and malformed --attempt values. Finding 2: resolveInstallTimeEffort returned unvalidated effort strings (e.g. "ultra") verbatim. Each precedence layer now checks GSD_EFFORT_SET (imported once from core.cjs) before accepting a value, mirroring resolveEffortInternal exactly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(#443): newline-agnostic effort frontmatter injection (Windows CRLF) + CRLF-safe assertions Extracts injectEffortFrontmatter(content, effortValue) pure helper that detects EOL (LF vs CRLF) from the opening '---' line and inserts 'effort: <value>' before the closing '---' delimiter using the same EOL as the surrounding frontmatter. Regex now uses /^---\r?\n([\s\S]*?)^---\r?$/m instead of the LF-only /^(---\n[\s\S]*?)(---)(\n|$)/ that silently skipped CRLF files on Windows (git core.autocrlf=true checkout). Also adds 7 unit tests covering LF, CRLF, idempotency, no-frontmatter, and complex frontmatter cases. Exports injectEffortFrontmatter from module.exports. Fixes 6 CI failures in tests/feat-443-effort-install-wiring.install.test.cjs on windows-latest runners (lines 138, 145, 152, 261, 345, 356). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: CI Rebase Check <ci@gsd-redux> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |