feat(3081): auto-trim review prompts for small-context model reviewers (#3708)

* feat(3081): auto-trim review prompts for small-context model reviewers

Adds review.max_prompt_tokens and review.max_prompt_tokens_per_reviewer
config keys. When configured, the /gsd-review workflow deterministically
trims the assembled prompt before sending to each reviewer (drop CONTEXT
→ RESEARCH → REQUIREMENTS; head-shrink PROJECT.md; tail-truncate PLANs
proportionally; reserve disclosure-note tokens upfront). Trim metadata
is recorded in REVIEWS.md frontmatter. Reviewer is skipped with a
warning if even the minimum review set exceeds the budget.

Closes #3081

* fix(3081): register prompt-budget in SDK query registry and update inventory manifest

review.md references `gsd-sdk query prompt-budget` at three call sites, but the
command had no handler in the SDK registry — failing the registry-integration
drift-guard test on all 6 CI matrix legs. Added a native TypeScript SDK handler
(sdk/src/query/prompt-budget.ts) that ports the applyBudget logic from the CJS
module, registered it in DOMAIN_STATIC_CATALOG, and regenerated
docs/INVENTORY-MANIFEST.json to include the new cli_modules/prompt-budget.cjs entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(3081): bump ws to 8.20.1 and allowlist prompt-budget sibling pair

Two additional CI failures after the registry fix:

1. ws moderate CVE (GHSA-58qx-3vcg-4xpx, uninitialized memory disclosure):
   The advisory covers ws >=8.0.0 <8.20.1. Both root and sdk/package.json
   pinned ^8.20.0 which resolved to 8.20.0. Bumped both to 8.20.1 to clear
   the npm audit drift-guard test (bug-3588-npm-audit-clean.test.cjs).

2. lint-shared-module-handsync detected the new prompt-budget.ts / prompt-budget.cjs
   sibling pair without an allowlist entry. Added a cooperatingSiblings entry
   to scripts/shared-module-handsync-allowlist.json with classification and
   justification matching the established pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(3081): align prompt-budget skip semantics across CJS and SDK dispatch paths

Replace brittle `[ $EXIT -eq 2 ]` guards with `[ $EXIT -ne 0 ]` in all three
local-reviewer blocks (Ollama, LM Studio, llama.cpp) in workflows/review.md.
Any non-zero exit from prompt-budget now triggers a skip with a descriptive
warning — exit 2/11 prints "budget too small", any other non-zero prints
"unexpected exit code". This ensures the SDK bridge dispatch path (exit 11
via GSDError(Blocked)) triggers the same skip as the CJS path (exit 2).

The SDK handler (sdk/src/query/prompt-budget.ts) already writes both metadata
and prompt files before throwing, so no change needed there.

The Ollama block also gains the missing OLLAMA_SKIP guard so the reviewer
invocation is actually skipped (previously the block only suppressed the
OLLAMA_PROMPT_FILE update but still ran the curl invocation).

SDK integration path (hardFailed via GSDError(Blocked) → exit 11) is covered
by handler unit tests in tests/prompt-budget.test.cjs; no gsd-sdk-*.test.cjs
exercising the full bridge dispatch for this command exists yet — that gap
remains and is documented here.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix prompt-budget trim ordering and review guard follow-ups

* perf: optimize prompt-budget and dedup reviewer trim workflow

* fix(3708): drop source-grep theater tests to satisfy lint-no-source-grep

All four test files added in commit 2df566ed were pure source-grep theater:
they read .cjs / .ts / .md source files and asserted that specific string
literals were present or absent. None exercised runtime behaviour.

Deleted:
- tests/gsd-tools-memory-optimizer.test.cjs   — 7 includes() on gsd-tools.cjs
- tests/prompt-budget-hotpath-optimizer.test.cjs — includes() on prompt-budget.cjs + .ts
- tests/prompt-budget-io-optimizer.test.cjs   — includes() on prompt-budget.ts + gsd-tools.cjs
- tests/review-workflow-budget-dedup.test.cjs — includes() on review.md

Behavioural coverage for the prompt-budget feature already exists in
tests/prompt-budget.test.cjs and tests/prompt-budget-cli.test.cjs (also
added by this PR). No replacement tests needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(3708): correct budget-pressure threshold and minSet accounting

Two bugs in applyBudget caused premature trimming and false hard-fails:

1. UNNEEDED_TRIM: budgetUnderPressure compared baseTokens against
   effectiveBudget - NOTE_RESERVE_TOKENS, triggering trim pressure 80
   tokens before the budget was actually exceeded. Fix: compare against
   effectiveBudget directly; NOTE_RESERVE_TOKENS are still reserved in
   contentBudget once real pressure is confirmed.

2. FALSE_HARDFAIL: minSet included NOTE_RESERVE_TOKENS unconditionally,
   treating the note as mandatory even when no trim would occur and no
   note would be injected. Fix: exclude NOTE_RESERVE_TOKENS from minSet;
   a prompt that fits untrimmed needs no note and must not hard-fail.

Both fixes applied in CJS and TypeScript implementations. Two regression
tests added (cycles 11 and 12) that reproduce each case behaviorally.

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-05-18 23:13:09 -04:00
committed by GitHub
parent e15348ea43
commit 6a5fa59129
21 changed files with 2008 additions and 33 deletions

View File

@@ -173,6 +173,39 @@ Output your review in markdown format.
```
Write to a temp file: `/tmp/gsd-review-prompt-{phase}.md`
Also write individual section files so the budget tool can re-trim per reviewer:
```bash
# Write individual section files for per-reviewer budget trimming
# These are always written so reviewers with a budget can invoke prompt-budget
cp "$INSTRUCTIONS_BLOCK_FILE" "/tmp/gsd-review-${PHASE}-instructions.md"
cp "$ROADMAP_SECTION_FILE" "/tmp/gsd-review-${PHASE}-roadmap.md"
# Plan files: copy each PLAN.md to a predictable numbered path
PLAN_INDEX=0
for PLAN_FILE in "${PHASE_DIR}"/*-PLAN.md; do
PADDED_IDX=$(printf '%02d' "$PLAN_INDEX")
cp "$PLAN_FILE" "/tmp/gsd-review-${PHASE}-plan-${PADDED_IDX}.md"
PLAN_INDEX=$((PLAN_INDEX + 1))
done
# Optional section files (only if content was included in the combined prompt)
if [ -f ".planning/PROJECT.md" ]; then
cp .planning/PROJECT.md "/tmp/gsd-review-${PHASE}-project.md"
fi
if ls "${PHASE_DIR}/"*"-CONTEXT.md" >/dev/null 2>&1; then
cat "${PHASE_DIR}/"*"-CONTEXT.md" > "/tmp/gsd-review-${PHASE}-context.md"
fi
if ls "${PHASE_DIR}/"*"-RESEARCH.md" >/dev/null 2>&1; then
cat "${PHASE_DIR}/"*"-RESEARCH.md" > "/tmp/gsd-review-${PHASE}-research.md"
fi
if [ -f ".planning/REQUIREMENTS.md" ]; then
cp .planning/REQUIREMENTS.md "/tmp/gsd-review-${PHASE}-requirements.md"
fi
```
Note: The variable names above (`INSTRUCTIONS_BLOCK_FILE`, `ROADMAP_SECTION_FILE`, `PHASE_DIR`, `PHASE`) reference the variables already established during prompt assembly. In practice the AI implementing this step writes the instruction and roadmap blocks to temp files while assembling the combined prompt, then copies those same temp files to the per-reviewer section paths. If the assembled prompt was built inline (string concatenation rather than file-by-file), write each section to the corresponding path after writing the combined file.
</step>
<step name="invoke_reviewers">
@@ -256,13 +289,74 @@ fi
Read host and model from config. All three local backends share the same `/v1/chat/completions` endpoint — only host and model differ. Use `jq --rawfile` to safely encode the multi-line prompt as JSON without shell-escaping issues.
```bash
# Shared helper: apply prompt-budget trimming for local reviewers
prepare_trimmed_prompt_for_reviewer() {
REVIEWER_KEY="$1"
REVIEWER_BUDGET="$2"
OUTPUT_PROMPT="$3"
OUTPUT_META="$4"
[ -z "$REVIEWER_BUDGET" ] && return 0
[ "$REVIEWER_BUDGET" = "null" ] && return 0
[ "$REVIEWER_BUDGET" = "0" ] && return 0
PLAN_FILE_ARGS=""
for p in /tmp/gsd-review-{phase}-plan-*.md; do
[ -f "$p" ] && PLAN_FILE_ARGS="$PLAN_FILE_ARGS --plan-file $p"
done
PROJECT_ARG=""
[ -f "/tmp/gsd-review-{phase}-project.md" ] && PROJECT_ARG="--project-file /tmp/gsd-review-{phase}-project.md"
CONTEXT_ARG=""
[ -f "/tmp/gsd-review-{phase}-context.md" ] && CONTEXT_ARG="--context-file /tmp/gsd-review-{phase}-context.md"
RESEARCH_ARG=""
[ -f "/tmp/gsd-review-{phase}-research.md" ] && RESEARCH_ARG="--research-file /tmp/gsd-review-{phase}-research.md"
REQUIREMENTS_ARG=""
[ -f "/tmp/gsd-review-{phase}-requirements.md" ] && REQUIREMENTS_ARG="--requirements-file /tmp/gsd-review-{phase}-requirements.md"
gsd-sdk query prompt-budget \
--budget "$REVIEWER_BUDGET" \
--instructions-file "/tmp/gsd-review-{phase}-instructions.md" \
--roadmap-file "/tmp/gsd-review-{phase}-roadmap.md" \
$PLAN_FILE_ARGS $PROJECT_ARG $CONTEXT_ARG $RESEARCH_ARG $REQUIREMENTS_ARG \
--output-prompt "$OUTPUT_PROMPT" \
--output-metadata "$OUTPUT_META"
return $?
}
# Resolve prompt budget for Ollama: per-reviewer override > global default > null
OLLAMA_REVIEWER_BUDGET=$(gsd-sdk query config-get review.max_prompt_tokens_per_reviewer.ollama 2>/dev/null | jq -r '.' 2>/dev/null || echo "null")
if [ -z "$OLLAMA_REVIEWER_BUDGET" ] || [ "$OLLAMA_REVIEWER_BUDGET" = "null" ]; then
OLLAMA_REVIEWER_BUDGET=$(gsd-sdk query config-get review.max_prompt_tokens 2>/dev/null | jq -r '.' 2>/dev/null || echo "null")
fi
# Apply budget trim for Ollama if a budget is configured
OLLAMA_PROMPT_FILE="/tmp/gsd-review-prompt-{phase}.md"
OLLAMA_SKIP=0
if [ -n "$OLLAMA_REVIEWER_BUDGET" ] && [ "$OLLAMA_REVIEWER_BUDGET" != "null" ] && [ "$OLLAMA_REVIEWER_BUDGET" != "0" ]; then
OLLAMA_TRIMMED_PROMPT="/tmp/gsd-review-prompt-{phase}-ollama.md"
OLLAMA_TRIM_META="/tmp/gsd-review-prompt-{phase}-ollama.metadata.json"
prepare_trimmed_prompt_for_reviewer "ollama" "$OLLAMA_REVIEWER_BUDGET" "$OLLAMA_TRIMMED_PROMPT" "$OLLAMA_TRIM_META"
OLLAMA_EXIT=$?
if [ $OLLAMA_EXIT -ne 0 ]; then
if [ $OLLAMA_EXIT -eq 2 ] || [ $OLLAMA_EXIT -eq 11 ]; then
echo "WARNING: prompt budget for ollama (${OLLAMA_REVIEWER_BUDGET} tokens) is too small for the minimum review set. Skipping Ollama reviewer." >&2
else
echo "WARNING: prompt-budget returned unexpected exit code ${OLLAMA_EXIT} for ollama. Skipping Ollama reviewer." >&2
fi
OLLAMA_SKIP=1
else
OLLAMA_PROMPT_FILE="$OLLAMA_TRIMMED_PROMPT"
fi
fi
if [ "$OLLAMA_SKIP" != "1" ]; then
OLLAMA_HOST=$(gsd-sdk query config-get review.ollama_host 2>/dev/null | jq -r '.' 2>/dev/null || echo "")
if [ -z "$OLLAMA_HOST" ] || [ "$OLLAMA_HOST" = "null" ]; then OLLAMA_HOST="http://localhost:11434"; fi
OLLAMA_MODEL=$(gsd-sdk query config-get review.models.ollama 2>/dev/null | jq -r '.' 2>/dev/null || echo "")
if [ -z "$OLLAMA_MODEL" ] || [ "$OLLAMA_MODEL" = "null" ]; then
OLLAMA_MODEL=$(curl -s --max-time 2 "${OLLAMA_HOST}/v1/models" 2>/dev/null | jq -r '.data[0].id // "llama3"' 2>/dev/null || echo "llama3")
fi
jq -n --rawfile content /tmp/gsd-review-prompt-{phase}.md \
jq -n --rawfile content "$OLLAMA_PROMPT_FILE" \
--arg model "$OLLAMA_MODEL" \
'{model: $model, messages: [{role: "user", content: $content}]}' | \
curl -s --max-time 120 -X POST "${OLLAMA_HOST}/v1/chat/completions" \
@@ -272,17 +366,45 @@ jq -n --rawfile content /tmp/gsd-review-prompt-{phase}.md \
if [ ! -s /tmp/gsd-review-ollama-{phase}.md ]; then
echo "Ollama review failed or returned empty output." > /tmp/gsd-review-ollama-{phase}.md
fi
fi
```
**LM Studio (local, OpenAI-compatible):**
```bash
# Resolve prompt budget for LM Studio: per-reviewer override > global default > null
LM_STUDIO_REVIEWER_BUDGET=$(gsd-sdk query config-get review.max_prompt_tokens_per_reviewer.lm_studio 2>/dev/null | jq -r '.' 2>/dev/null || echo "null")
if [ -z "$LM_STUDIO_REVIEWER_BUDGET" ] || [ "$LM_STUDIO_REVIEWER_BUDGET" = "null" ]; then
LM_STUDIO_REVIEWER_BUDGET=$(gsd-sdk query config-get review.max_prompt_tokens 2>/dev/null | jq -r '.' 2>/dev/null || echo "null")
fi
# Apply budget trim for LM Studio if a budget is configured
LM_STUDIO_PROMPT_FILE="/tmp/gsd-review-prompt-{phase}.md"
LM_STUDIO_SKIP=0
if [ -n "$LM_STUDIO_REVIEWER_BUDGET" ] && [ "$LM_STUDIO_REVIEWER_BUDGET" != "null" ] && [ "$LM_STUDIO_REVIEWER_BUDGET" != "0" ]; then
LM_STUDIO_TRIMMED_PROMPT="/tmp/gsd-review-prompt-{phase}-lm_studio.md"
LM_STUDIO_TRIM_META="/tmp/gsd-review-prompt-{phase}-lm_studio.metadata.json"
prepare_trimmed_prompt_for_reviewer "lm_studio" "$LM_STUDIO_REVIEWER_BUDGET" "$LM_STUDIO_TRIMMED_PROMPT" "$LM_STUDIO_TRIM_META"
LM_STUDIO_EXIT=$?
if [ $LM_STUDIO_EXIT -ne 0 ]; then
if [ $LM_STUDIO_EXIT -eq 2 ] || [ $LM_STUDIO_EXIT -eq 11 ]; then
echo "WARNING: prompt budget for lm_studio (${LM_STUDIO_REVIEWER_BUDGET} tokens) is too small for the minimum review set. Skipping LM Studio reviewer." >&2
else
echo "WARNING: prompt-budget returned unexpected exit code ${LM_STUDIO_EXIT} for lm_studio. Skipping LM Studio reviewer." >&2
fi
LM_STUDIO_SKIP=1
else
LM_STUDIO_PROMPT_FILE="$LM_STUDIO_TRIMMED_PROMPT"
fi
fi
if [ "$LM_STUDIO_SKIP" != "1" ]; then
LM_STUDIO_HOST=$(gsd-sdk query config-get review.lm_studio_host 2>/dev/null | jq -r '.' 2>/dev/null || echo "")
if [ -z "$LM_STUDIO_HOST" ] || [ "$LM_STUDIO_HOST" = "null" ]; then LM_STUDIO_HOST="http://localhost:1234"; fi
LM_STUDIO_MODEL=$(gsd-sdk query config-get review.models.lm_studio 2>/dev/null | jq -r '.' 2>/dev/null || echo "")
if [ -z "$LM_STUDIO_MODEL" ] || [ "$LM_STUDIO_MODEL" = "null" ]; then
LM_STUDIO_MODEL=$(curl -s --max-time 2 "${LM_STUDIO_HOST}/v1/models" 2>/dev/null | jq -r '.data[0].id // "local-model"' 2>/dev/null || echo "local-model")
fi
LM_STUDIO_RESPONSE=$(jq -n --rawfile content /tmp/gsd-review-prompt-{phase}.md \
LM_STUDIO_RESPONSE=$(jq -n --rawfile content "$LM_STUDIO_PROMPT_FILE" \
--arg model "$LM_STUDIO_MODEL" \
'{model: $model, messages: [{role: "user", content: $content}]}' | \
curl -s --max-time 120 -X POST "${LM_STUDIO_HOST}/v1/chat/completions" \
@@ -297,17 +419,45 @@ if [ -n "$LM_STUDIO_CONTENT" ]; then
else
echo "Warning: LM Studio returned empty content — skipping review." >&2
fi
fi
```
**llama.cpp (local, OpenAI-compatible):**
```bash
# Resolve prompt budget for llama.cpp: per-reviewer override > global default > null
LLAMA_CPP_REVIEWER_BUDGET=$(gsd-sdk query config-get review.max_prompt_tokens_per_reviewer.llama_cpp 2>/dev/null | jq -r '.' 2>/dev/null || echo "null")
if [ -z "$LLAMA_CPP_REVIEWER_BUDGET" ] || [ "$LLAMA_CPP_REVIEWER_BUDGET" = "null" ]; then
LLAMA_CPP_REVIEWER_BUDGET=$(gsd-sdk query config-get review.max_prompt_tokens 2>/dev/null | jq -r '.' 2>/dev/null || echo "null")
fi
# Apply budget trim for llama.cpp if a budget is configured
LLAMA_CPP_PROMPT_FILE="/tmp/gsd-review-prompt-{phase}.md"
LLAMA_CPP_SKIP=0
if [ -n "$LLAMA_CPP_REVIEWER_BUDGET" ] && [ "$LLAMA_CPP_REVIEWER_BUDGET" != "null" ] && [ "$LLAMA_CPP_REVIEWER_BUDGET" != "0" ]; then
LLAMA_CPP_TRIMMED_PROMPT="/tmp/gsd-review-prompt-{phase}-llama_cpp.md"
LLAMA_CPP_TRIM_META="/tmp/gsd-review-prompt-{phase}-llama_cpp.metadata.json"
prepare_trimmed_prompt_for_reviewer "llama_cpp" "$LLAMA_CPP_REVIEWER_BUDGET" "$LLAMA_CPP_TRIMMED_PROMPT" "$LLAMA_CPP_TRIM_META"
LLAMA_CPP_EXIT=$?
if [ $LLAMA_CPP_EXIT -ne 0 ]; then
if [ $LLAMA_CPP_EXIT -eq 2 ] || [ $LLAMA_CPP_EXIT -eq 11 ]; then
echo "WARNING: prompt budget for llama_cpp (${LLAMA_CPP_REVIEWER_BUDGET} tokens) is too small for the minimum review set. Skipping llama.cpp reviewer." >&2
else
echo "WARNING: prompt-budget returned unexpected exit code ${LLAMA_CPP_EXIT} for llama_cpp. Skipping llama.cpp reviewer." >&2
fi
LLAMA_CPP_SKIP=1
else
LLAMA_CPP_PROMPT_FILE="$LLAMA_CPP_TRIMMED_PROMPT"
fi
fi
if [ "$LLAMA_CPP_SKIP" != "1" ]; then
LLAMA_CPP_HOST=$(gsd-sdk query config-get review.llama_cpp_host 2>/dev/null | jq -r '.' 2>/dev/null || echo "")
if [ -z "$LLAMA_CPP_HOST" ] || [ "$LLAMA_CPP_HOST" = "null" ]; then LLAMA_CPP_HOST="http://localhost:8080"; fi
LLAMA_CPP_MODEL=$(gsd-sdk query config-get review.models.llama_cpp 2>/dev/null | jq -r '.' 2>/dev/null || echo "")
if [ -z "$LLAMA_CPP_MODEL" ] || [ "$LLAMA_CPP_MODEL" = "null" ]; then
LLAMA_CPP_MODEL=$(curl -s --max-time 2 "${LLAMA_CPP_HOST}/v1/models" 2>/dev/null | jq -r '.data[0].id // "local-model"' 2>/dev/null || echo "local-model")
fi
LLAMA_CPP_CONTENT=$(jq -n --rawfile content /tmp/gsd-review-prompt-{phase}.md \
LLAMA_CPP_CONTENT=$(jq -n --rawfile content "$LLAMA_CPP_PROMPT_FILE" \
--arg model "$LLAMA_CPP_MODEL" \
'{model: $model, messages: [{role: "user", content: $content}]}' | \
curl -s --max-time 120 -X POST "${LLAMA_CPP_HOST}/v1/chat/completions" \
@@ -318,6 +468,7 @@ if [ -n "$LLAMA_CPP_CONTENT" ]; then
else
echo "Warning: llama.cpp returned empty content — skipping review." >&2
fi
fi
```
If a CLI or local server fails, log the error and continue with remaining reviewers.
@@ -336,12 +487,24 @@ Display progress:
<step name="write_reviews">
Combine all review responses into `{phase_dir}/{padded_phase}-REVIEWS.md`:
After all reviewers complete, collect trim metadata files written during the run. For each reviewer that was trimmed (i.e. a `.metadata.json` file exists and `hardFailed` or `omitted` is non-empty, or `projectMdShrunk` is true, or `planTruncationPct > 0`), include a `trimmed_reviewers` block in the frontmatter. Omit the key entirely if no reviewer was trimmed.
```markdown
---
phase: {N}
reviewers: [gemini, claude, codex, coderabbit, opencode, qwen, cursor, ollama, lm_studio, llama_cpp] # populate at runtime with only the reviewers actually invoked
reviewed_at: {ISO timestamp}
plans_reviewed: [{list of PLAN.md files}]
trimmed_reviewers: # only present if at least one reviewer was trimmed
ollama:
budget: 6000
effective_budget: 5400
estimated_tokens: 5380
omitted: [context, research]
project_md_shrunk: true
plan_truncation_pct: 22
hard_failed: false
note_injected: true
---
# Cross-AI Plan Review — Phase {N}