6f99e493e7
* test(#4395): prove the manager spawns its debugger without blocking
Failing-first regression coverage for #4395.
debug.md:209 mandates the orchestrator to session-manager spawn carry
run_in_background: false, and says why outright: "Claude Code backgrounds
subagents by default, and only that flag makes the spawn return the
compact session summary directly" (#2196).
The session-manager to debugger spawn, one level down, carries no flag.
Measured: run_in_background appears nowhere under agents/ -- only in
gsd-core/workflows/. So by the rule #2196 itself states, that spawn is
backgrounded, Step 3 ("Handle Agent Return") has no return to inspect,
the manager emits CONTINUE_REQUIRED, the orchestrator auto-resumes per
#2257/#3448, and a second detached debugger races the first on
.planning/debug/<slug>.md.
Row 4 is the load-bearing one: it closes the CLASS by requiring every
subagent spawn under agents/ to declare run_in_background explicitly, so
the next agent that spawns one has to decide rather than inherit a silent
host default. It is scoped to agents/ precisely so it cannot misfire on
the workflows that deliberately use true for parallel fan-out.
Rows 5-7 are pins, not fixes: the #2196 mandate one level up, Step 2 as
the single spawn-format source that the eight continuation sites delegate
to, and the survival of CONTINUE_REQUIRED (which has a legitimate trigger
unrelated to this defect).
Red round: 4 of 7 rows fail. Row 3 needed hardening first -- asserting
only that the two variants AGREE passed vacuously, because two missing
flags are also equal; it now asserts each is present before comparing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(#4395): make the manager's own debugger spawn blocking
The orchestrator-to-manager hop already requires a blocking spawn and says
why (#2196, debug.md:209): Claude Code backgrounds subagents by default,
and only run_in_background=false makes the spawn return its summary. The
manager-to-debugger hop, one level down, carried no flag -- measured,
run_in_background appeared nowhere under agents/ at all.
So that spawn was backgrounded. Step 3 ("Handle Agent Return") opens
"Inspect the return output for the structured return header" -- with
nothing to inspect, the manager correctly declined to fabricate a terminal
summary and returned CONTINUE_REQUIRED; the orchestrator correctly
auto-resumed (#2257/#3448); the resumed manager reached Step 2 and spawned
a SECOND detached debugger. Both then raced on .planning/debug/<slug>.md.
Every observable in the report follows with no further assumption,
including the count: the reporter saw exactly three collisions in one
invocation, and debug.md:251 caps auto-resumes at three per slug -- one
collision per cycle.
Fixed at the cause, in both shipped variants, kept byte-consistent. The
eight continuation sites say "see Step 2 format", so they inherit it.
The issue offered two remedies. The second -- have the auto-resume path
reconcile a still-running debugger before spawning another -- is not taken:
it treats the symptom, and needs machinery that does not exist (no portable
way to enumerate or stop another runtime's live agents, plus an in-flight
sentinel with staleness and recovery rules, or an orphaned marker deadlocks
the session permanently). With the spawn blocking, the manager cannot reach
Step 4 while a debugger is live, so such a guard would also be unreachable.
#2257, #3448, the anti-loop heuristic, the cap of three, and the
CONTINUE_REQUIRED shape are all correct and untouched. CONTINUE_REQUIRED
keeps its legitimate trigger: the manager genuinely exhausting its own turn
budget mid-investigation.
Also corrects the red-round test to the canonical CALL form. debug.md
writes run_in_background=false inside Agent(...) and run_in_background:
false in prose; the first draft asserted the prose form, which the shipped
call would never have matched.
Emitted-Drift-Ack-Growth: gsd-debug-session-manager.md — the blocking spawn flag plus the note recording why an unstated flag produced colliding debuggers
Emitted-Drift-Ack-Growth: gsd-debug-session-manager.compact.md — same change as its full sibling, kept byte-consistent with it
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(#4395): add changeset fragment
pr:0 placeholder is backfilled with the real number once the PR exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(#4395): refresh the variant benchmark baseline
Two entries move.
gsd-debug-session-manager.md 4766/4477 -> 4938/4649 is this change: the
blocking-spawn flag plus its explanatory note, added to BOTH variants to
keep them byte-consistent, so the compact sibling grows by the same amount
and the pair's reduction ratio dips 6.06 -> 5.85. The compact file remains
strictly smaller than its canonical sibling, which is what the variant
guard's size check actually requires.
gsd-code-fixer.md 10741 -> 10740 is NOT from this branch -- the file is
untouched here. It has scored 10740 since f334f277dd (#4324) reworded a
line without refreshing this fixture, so the stale number is sitting on
next. Fixed here rather than deferred; the #4350 branch carries the
identical one-token correction, so whichever lands first makes the other a
no-op.
Refreshed with the variant script's own --write.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(#4395): state the blocking rule agent-wide and correct two overclaims
Review round. Three substantive corrections, one of them to a claim I made
in the previous commit message.
1. The blocking rule is now stated AGENT-WIDE, not per-call. My earlier
claim that "the eight continuation sites say 'see Step 2 format', so
they inherit it" was false: exactly ONE of them names Step 2 (the
compact variant says "Step 2 format" without the "see", which is why
the first draft of the test matched it zero times there). The other
sites inherit only because Step 2 holds the sole Agent() spawn literal
in each file. Both variants now say so outright, and the test pins the
sole-literal invariant in BOTH variants rather than the prose wording
in one.
2. debug.md's two auto-resume buckets now restate run_in_background=false
for the re-spawn. "The same session_params" does not carry it --
session_params is prompt content, not the spawn flag.
3. The test extractor now also matches single-line Agent(...) calls. The
class guard was blind to exactly the shape a future offender is most
likely to take.
Also corrects the diagnosis: remedy 2 is NOT unreachable once the spawn
blocks. This agent's own retained CONTINUE_REQUIRED trigger is "turn
budget exhausted WHILE THE DEBUGGER IS STILL INVESTIGATING", so a harness
turn cutoff mid-wait still double-spawns; debug.md:209 names the same
class from the other side. What this fix removes is the SYSTEMATIC case --
every invocation, because the spawn was always backgrounded. The residual
turn-cutoff window survives, bounded by the existing three-resume cap, and
is stated in the PR rather than denied.
Emitted-Drift-Ack-Growth: gsd-debug-session-manager.md — blocking spawn flag, the note recording why an unstated flag produced colliding debuggers, and the agent-wide restatement the per-site inheritance actually depends on
Emitted-Drift-Ack-Growth: gsd-debug-session-manager.compact.md — same change as its full sibling, kept byte-consistent with it
Emitted-Drift-Ack-Growth: debug.md — both auto-resume buckets restate the spawn flag, since session_params does not carry it
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* chore(#4395): backfill the changeset PR number
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: sim <sim@local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
177 lines
4.3 KiB
JSON
177 lines
4.3 KiB
JSON
{
|
|
"schema_version": 1,
|
|
"generated_by": "scripts/benchmark-compact-content-variants.cjs",
|
|
"tokenizer": {
|
|
"name": "gpt-tokenizer",
|
|
"version": "4.0.0"
|
|
},
|
|
"label": "PROXY-TOKENIZER DELTA — gpt-tokenizer is a stand-in; Anthropic publishes no tokenizer for Claude 3+. The on/off COMPARISON is exact under this pinned tokenizer; absolute counts are not Claude's real token counts.",
|
|
"pairs": {
|
|
"agents/gsd-advisor-researcher.md": {
|
|
"offTokens": 1087,
|
|
"onTokens": 853,
|
|
"reductionPct": 21.53
|
|
},
|
|
"agents/gsd-ai-researcher.md": {
|
|
"offTokens": 1446,
|
|
"onTokens": 1341,
|
|
"reductionPct": 7.26
|
|
},
|
|
"agents/gsd-assumptions-analyzer.md": {
|
|
"offTokens": 1063,
|
|
"onTokens": 842,
|
|
"reductionPct": 20.79
|
|
},
|
|
"agents/gsd-code-fixer.md": {
|
|
"offTokens": 10740,
|
|
"onTokens": 6690,
|
|
"reductionPct": 37.71
|
|
},
|
|
"agents/gsd-code-reviewer.md": {
|
|
"offTokens": 4408,
|
|
"onTokens": 3601,
|
|
"reductionPct": 18.31
|
|
},
|
|
"agents/gsd-codebase-mapper.md": {
|
|
"offTokens": 5357,
|
|
"onTokens": 4834,
|
|
"reductionPct": 9.76
|
|
},
|
|
"agents/gsd-debug-session-manager.md": {
|
|
"offTokens": 5027,
|
|
"onTokens": 4738,
|
|
"reductionPct": 5.75
|
|
},
|
|
"agents/gsd-doc-classifier.md": {
|
|
"offTokens": 2901,
|
|
"onTokens": 2340,
|
|
"reductionPct": 19.34
|
|
},
|
|
"agents/gsd-doc-synthesizer.md": {
|
|
"offTokens": 3191,
|
|
"onTokens": 2768,
|
|
"reductionPct": 13.26
|
|
},
|
|
"agents/gsd-doc-verifier.md": {
|
|
"offTokens": 2996,
|
|
"onTokens": 2667,
|
|
"reductionPct": 10.98
|
|
},
|
|
"agents/gsd-doc-writer.md": {
|
|
"offTokens": 9040,
|
|
"onTokens": 5588,
|
|
"reductionPct": 38.19
|
|
},
|
|
"agents/gsd-dom-verifier.md": {
|
|
"offTokens": 1733,
|
|
"onTokens": 1431,
|
|
"reductionPct": 17.43
|
|
},
|
|
"agents/gsd-domain-researcher.md": {
|
|
"offTokens": 1621,
|
|
"onTokens": 1332,
|
|
"reductionPct": 17.83
|
|
},
|
|
"agents/gsd-eval-auditor.md": {
|
|
"offTokens": 2941,
|
|
"onTokens": 2808,
|
|
"reductionPct": 4.52
|
|
},
|
|
"agents/gsd-eval-planner.md": {
|
|
"offTokens": 1674,
|
|
"onTokens": 1567,
|
|
"reductionPct": 6.39
|
|
},
|
|
"agents/gsd-framework-selector.md": {
|
|
"offTokens": 1513,
|
|
"onTokens": 1095,
|
|
"reductionPct": 27.63
|
|
},
|
|
"agents/gsd-integration-checker.md": {
|
|
"offTokens": 4105,
|
|
"onTokens": 2377,
|
|
"reductionPct": 42.1
|
|
},
|
|
"agents/gsd-intel-updater.md": {
|
|
"offTokens": 4353,
|
|
"onTokens": 4115,
|
|
"reductionPct": 5.47
|
|
},
|
|
"agents/gsd-mempalace-curator.md": {
|
|
"offTokens": 1160,
|
|
"onTokens": 985,
|
|
"reductionPct": 15.09
|
|
},
|
|
"agents/gsd-nyquist-auditor.md": {
|
|
"offTokens": 1768,
|
|
"onTokens": 1654,
|
|
"reductionPct": 6.45
|
|
},
|
|
"agents/gsd-pattern-mapper.md": {
|
|
"offTokens": 3155,
|
|
"onTokens": 2670,
|
|
"reductionPct": 15.37
|
|
},
|
|
"agents/gsd-project-researcher.md": {
|
|
"offTokens": 5464,
|
|
"onTokens": 5135,
|
|
"reductionPct": 6.02
|
|
},
|
|
"agents/gsd-research-synthesizer.md": {
|
|
"offTokens": 3122,
|
|
"onTokens": 2917,
|
|
"reductionPct": 6.57
|
|
},
|
|
"agents/gsd-roadmapper.md": {
|
|
"offTokens": 6008,
|
|
"onTokens": 4774,
|
|
"reductionPct": 20.54
|
|
},
|
|
"agents/gsd-security-auditor.md": {
|
|
"offTokens": 2201,
|
|
"onTokens": 1969,
|
|
"reductionPct": 10.54
|
|
},
|
|
"agents/gsd-ui-auditor.md": {
|
|
"offTokens": 4145,
|
|
"onTokens": 3926,
|
|
"reductionPct": 5.28
|
|
},
|
|
"agents/gsd-ui-checker.md": {
|
|
"offTokens": 4499,
|
|
"onTokens": 3048,
|
|
"reductionPct": 32.25
|
|
},
|
|
"agents/gsd-ui-researcher.md": {
|
|
"offTokens": 5432,
|
|
"onTokens": 4815,
|
|
"reductionPct": 11.36
|
|
},
|
|
"agents/gsd-user-profiler.md": {
|
|
"offTokens": 1859,
|
|
"onTokens": 1481,
|
|
"reductionPct": 20.33
|
|
},
|
|
"gsd-core/templates/summary.md": {
|
|
"offTokens": 2938,
|
|
"onTokens": 2299,
|
|
"reductionPct": 21.75
|
|
},
|
|
"gsd-core/templates/user-setup.md": {
|
|
"offTokens": 2061,
|
|
"onTokens": 1363,
|
|
"reductionPct": 33.87
|
|
},
|
|
"gsd-core/workflows/help/modes/full.md": {
|
|
"offTokens": 9950,
|
|
"onTokens": 6579,
|
|
"reductionPct": 33.88
|
|
}
|
|
},
|
|
"aggregate": {
|
|
"offTokens": 118958,
|
|
"onTokens": 94602,
|
|
"reductionPct": 20.47
|
|
}
|
|
}
|