feat(3081): auto-trim review prompts for small-context model reviewers (#3708)
* feat(3081): auto-trim review prompts for small-context model reviewers Adds review.max_prompt_tokens and review.max_prompt_tokens_per_reviewer config keys. When configured, the /gsd-review workflow deterministically trims the assembled prompt before sending to each reviewer (drop CONTEXT → RESEARCH → REQUIREMENTS; head-shrink PROJECT.md; tail-truncate PLANs proportionally; reserve disclosure-note tokens upfront). Trim metadata is recorded in REVIEWS.md frontmatter. Reviewer is skipped with a warning if even the minimum review set exceeds the budget. Closes #3081 * fix(3081): register prompt-budget in SDK query registry and update inventory manifest review.md references `gsd-sdk query prompt-budget` at three call sites, but the command had no handler in the SDK registry — failing the registry-integration drift-guard test on all 6 CI matrix legs. Added a native TypeScript SDK handler (sdk/src/query/prompt-budget.ts) that ports the applyBudget logic from the CJS module, registered it in DOMAIN_STATIC_CATALOG, and regenerated docs/INVENTORY-MANIFEST.json to include the new cli_modules/prompt-budget.cjs entry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(3081): bump ws to 8.20.1 and allowlist prompt-budget sibling pair Two additional CI failures after the registry fix: 1. ws moderate CVE (GHSA-58qx-3vcg-4xpx, uninitialized memory disclosure): The advisory covers ws >=8.0.0 <8.20.1. Both root and sdk/package.json pinned ^8.20.0 which resolved to 8.20.0. Bumped both to 8.20.1 to clear the npm audit drift-guard test (bug-3588-npm-audit-clean.test.cjs). 2. lint-shared-module-handsync detected the new prompt-budget.ts / prompt-budget.cjs sibling pair without an allowlist entry. Added a cooperatingSiblings entry to scripts/shared-module-handsync-allowlist.json with classification and justification matching the established pattern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(3081): align prompt-budget skip semantics across CJS and SDK dispatch paths Replace brittle `[ $EXIT -eq 2 ]` guards with `[ $EXIT -ne 0 ]` in all three local-reviewer blocks (Ollama, LM Studio, llama.cpp) in workflows/review.md. Any non-zero exit from prompt-budget now triggers a skip with a descriptive warning — exit 2/11 prints "budget too small", any other non-zero prints "unexpected exit code". This ensures the SDK bridge dispatch path (exit 11 via GSDError(Blocked)) triggers the same skip as the CJS path (exit 2). The SDK handler (sdk/src/query/prompt-budget.ts) already writes both metadata and prompt files before throwing, so no change needed there. The Ollama block also gains the missing OLLAMA_SKIP guard so the reviewer invocation is actually skipped (previously the block only suppressed the OLLAMA_PROMPT_FILE update but still ran the curl invocation). SDK integration path (hardFailed via GSDError(Blocked) → exit 11) is covered by handler unit tests in tests/prompt-budget.test.cjs; no gsd-sdk-*.test.cjs exercising the full bridge dispatch for this command exists yet — that gap remains and is documented here. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix prompt-budget trim ordering and review guard follow-ups * perf: optimize prompt-budget and dedup reviewer trim workflow * fix(3708): drop source-grep theater tests to satisfy lint-no-source-grep All four test files added in commit 2df566ed were pure source-grep theater: they read .cjs / .ts / .md source files and asserted that specific string literals were present or absent. None exercised runtime behaviour. Deleted: - tests/gsd-tools-memory-optimizer.test.cjs — 7 includes() on gsd-tools.cjs - tests/prompt-budget-hotpath-optimizer.test.cjs — includes() on prompt-budget.cjs + .ts - tests/prompt-budget-io-optimizer.test.cjs — includes() on prompt-budget.ts + gsd-tools.cjs - tests/review-workflow-budget-dedup.test.cjs — includes() on review.md Behavioural coverage for the prompt-budget feature already exists in tests/prompt-budget.test.cjs and tests/prompt-budget-cli.test.cjs (also added by this PR). No replacement tests needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(3708): correct budget-pressure threshold and minSet accounting Two bugs in applyBudget caused premature trimming and false hard-fails: 1. UNNEEDED_TRIM: budgetUnderPressure compared baseTokens against effectiveBudget - NOTE_RESERVE_TOKENS, triggering trim pressure 80 tokens before the budget was actually exceeded. Fix: compare against effectiveBudget directly; NOTE_RESERVE_TOKENS are still reserved in contentBudget once real pressure is confirmed. 2. FALSE_HARDFAIL: minSet included NOTE_RESERVE_TOKENS unconditionally, treating the note as mandatory even when no trim would occur and no note would be injected. Fix: exclude NOTE_RESERVE_TOKENS from minSet; a prompt that fits untrimmed needs no note and must not hard-fail. Both fixes applied in CJS and TypeScript implementations. Two regression tests added (cycles 11 and 12) that reproduce each case behaviorally. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
@@ -136,6 +136,7 @@ test('#2653 — CJS DYNAMIC_KEY_PATTERNS test functions work correctly', () => {
|
||||
['models.planning', 5],
|
||||
['dynamic_routing.enabled', 6],
|
||||
['model_overrides.my-agent', 7],
|
||||
['review.max_prompt_tokens_per_reviewer.ollama', 8],
|
||||
];
|
||||
for (const [key, idx] of samples) {
|
||||
assert.ok(
|
||||
|
||||
@@ -60,12 +60,13 @@ function runGsdTools(args, cwd = process.cwd(), env = {}) {
|
||||
env: childEnv,
|
||||
});
|
||||
}
|
||||
return { success: true, output: result.trim() };
|
||||
return { success: true, output: result.trim(), exitCode: 0 };
|
||||
} catch (err) {
|
||||
return {
|
||||
success: false,
|
||||
output: err.stdout?.toString().trim() || '',
|
||||
error: err.stderr?.toString().trim() || err.message,
|
||||
exitCode: err.status ?? 1,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
269
tests/prompt-budget-cli.test.cjs
Normal file
269
tests/prompt-budget-cli.test.cjs
Normal file
@@ -0,0 +1,269 @@
|
||||
'use strict';
|
||||
|
||||
// allow-test-rule: prompt-content-is-the-product
|
||||
// The prompt-budget CLI writes an assembled, trimmed prompt string to disk.
|
||||
// Testing that the prompt omits a dropped section (research) requires a
|
||||
// content assertion on the output file — the file content IS the product.
|
||||
// Structured metadata (omitted[], hardFailed, etc.) is always the primary
|
||||
// assertion; text content checks are secondary and only used to verify the
|
||||
// trim policy was applied correctly to the assembled output.
|
||||
|
||||
/**
|
||||
* prompt-budget-cli.test.cjs
|
||||
*
|
||||
* Integration tests for the `gsd-tools prompt-budget` CLI subcommand.
|
||||
* Covers the 5 specified scenarios:
|
||||
* 1. Happy path: budget forces trim (research dropped)
|
||||
* 2. No-trim path: huge budget, metadata shows no omissions
|
||||
* 3. Hard-fail path: tiny budget, exit 2, metadata written, prompt empty
|
||||
* 4. Missing required arg: exit 1, stderr has error message
|
||||
* 5. Missing input file: exit 1
|
||||
*/
|
||||
|
||||
const { describe, test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const { createTempDir, cleanup, runGsdTools } = require('./helpers.cjs');
|
||||
|
||||
const TEST_INSTRUCTIONS = [
|
||||
'# Cross-AI Plan Review Request',
|
||||
'',
|
||||
'You are reviewing implementation plans for a software project phase.',
|
||||
'Provide structured feedback on plan quality, completeness, and risks.',
|
||||
].join('\n');
|
||||
|
||||
const TEST_ROADMAP = [
|
||||
'## Phase 3: Implement token budgeting',
|
||||
'',
|
||||
'### Goal',
|
||||
'Add deterministic prompt trimming for small-context local model servers.',
|
||||
].join('\n');
|
||||
|
||||
const TEST_PLAN = [
|
||||
'## PLAN-01: Add prompt-budget module',
|
||||
'',
|
||||
'### Tasks',
|
||||
'- [ ] Write estimateTokens()',
|
||||
'- [ ] Write applyBudget()',
|
||||
'- [ ] Write CLI wrapper',
|
||||
].join('\n');
|
||||
|
||||
const TEST_RESEARCH = 'a'.repeat(8000); // ~2000 tokens — large enough to force trimming
|
||||
|
||||
/**
|
||||
* Run gsd-tools with args as an array (safe for paths with spaces/dollars).
|
||||
* Returns { exitCode, stdout, stderr }.
|
||||
*/
|
||||
function runCli(args, cwd) {
|
||||
const result = runGsdTools(args, cwd);
|
||||
return {
|
||||
exitCode: result.exitCode ?? (result.success ? 0 : 1),
|
||||
stdout: result.output ?? '',
|
||||
stderr: result.error ?? '',
|
||||
};
|
||||
}
|
||||
|
||||
describe('prompt-budget CLI', () => {
|
||||
// ── Cycle 1: happy path — small budget that trims research ────────────────
|
||||
test('happy path: trims research when budget is small, exit 0', () => {
|
||||
const dir = createTempDir('pb-cli-happy-');
|
||||
try {
|
||||
// Write input files
|
||||
const instrFile = path.join(dir, 'instructions.md');
|
||||
const roadmapFile = path.join(dir, 'roadmap.md');
|
||||
const planFile = path.join(dir, 'plan-01.md');
|
||||
const researchFile = path.join(dir, 'research.md');
|
||||
const outPrompt = path.join(dir, 'out-prompt.md');
|
||||
const outMeta = path.join(dir, 'out-meta.json');
|
||||
|
||||
fs.writeFileSync(instrFile, TEST_INSTRUCTIONS);
|
||||
fs.writeFileSync(roadmapFile, TEST_ROADMAP);
|
||||
fs.writeFileSync(planFile, TEST_PLAN);
|
||||
fs.writeFileSync(researchFile, TEST_RESEARCH);
|
||||
|
||||
// Budget of 800 tokens: enough for instructions+roadmap+plan (min set ~389 tokens
|
||||
// with 10% safety margin) but not the ~2000-token research blob.
|
||||
// effectiveBudget = floor(800 * 0.9) = 720.
|
||||
// contentBudget = 720 - 80 (note reserve) = 640. Research (~2000 tokens) won't fit.
|
||||
const { exitCode, stderr } = runCli([
|
||||
'prompt-budget',
|
||||
'--budget', '800',
|
||||
'--instructions-file', instrFile,
|
||||
'--roadmap-file', roadmapFile,
|
||||
'--plan-file', planFile,
|
||||
'--research-file', researchFile,
|
||||
'--output-prompt', outPrompt,
|
||||
'--output-metadata', outMeta,
|
||||
], dir);
|
||||
|
||||
assert.equal(exitCode, 0, `Expected exit 0, got ${exitCode}. stderr: ${stderr}`);
|
||||
|
||||
// Output files must exist
|
||||
assert.ok(fs.existsSync(outPrompt), 'output prompt file should exist');
|
||||
assert.ok(fs.existsSync(outMeta), 'output metadata file should exist');
|
||||
|
||||
// Metadata must parse and have expected shape
|
||||
const meta = JSON.parse(fs.readFileSync(outMeta, 'utf8'));
|
||||
assert.equal(typeof meta.budget, 'number');
|
||||
assert.ok(Array.isArray(meta.omitted));
|
||||
assert.ok(meta.omitted.includes('research'), `Expected research in omitted, got: ${JSON.stringify(meta.omitted)}`);
|
||||
assert.equal(meta.hardFailed, false);
|
||||
|
||||
// Prompt must not contain research content
|
||||
const promptText = fs.readFileSync(outPrompt, 'utf8');
|
||||
assert.ok(promptText.length > 0, 'prompt file must not be empty');
|
||||
// Research was dropped so the 'aaa...' block should be absent from the prompt
|
||||
assert.ok(!promptText.includes('a'.repeat(100)), 'dropped research should not appear in prompt');
|
||||
} finally {
|
||||
cleanup(dir);
|
||||
}
|
||||
});
|
||||
|
||||
// ── Cycle 2: no-trim path — huge budget, nothing dropped ──────────────────
|
||||
test('no-trim path: huge budget returns all sections, exit 0', () => {
|
||||
const dir = createTempDir('pb-cli-notrim-');
|
||||
try {
|
||||
const instrFile = path.join(dir, 'instructions.md');
|
||||
const roadmapFile = path.join(dir, 'roadmap.md');
|
||||
const planFile = path.join(dir, 'plan-01.md');
|
||||
const researchFile = path.join(dir, 'research.md');
|
||||
const outPrompt = path.join(dir, 'out-prompt.md');
|
||||
const outMeta = path.join(dir, 'out-meta.json');
|
||||
|
||||
fs.writeFileSync(instrFile, TEST_INSTRUCTIONS);
|
||||
fs.writeFileSync(roadmapFile, TEST_ROADMAP);
|
||||
fs.writeFileSync(planFile, TEST_PLAN);
|
||||
fs.writeFileSync(researchFile, 'Some research findings.');
|
||||
|
||||
const { exitCode } = runCli([
|
||||
'prompt-budget',
|
||||
'--budget', '1000000',
|
||||
'--instructions-file', instrFile,
|
||||
'--roadmap-file', roadmapFile,
|
||||
'--plan-file', planFile,
|
||||
'--research-file', researchFile,
|
||||
'--output-prompt', outPrompt,
|
||||
'--output-metadata', outMeta,
|
||||
], dir);
|
||||
|
||||
assert.equal(exitCode, 0);
|
||||
|
||||
const meta = JSON.parse(fs.readFileSync(outMeta, 'utf8'));
|
||||
assert.deepEqual(meta.omitted, []);
|
||||
assert.equal(meta.hardFailed, false);
|
||||
assert.equal(meta.projectMdShrunk, false);
|
||||
assert.equal(meta.planTruncationPct, 0);
|
||||
assert.equal(meta.noteInjected, false);
|
||||
|
||||
// All sections must appear in the prompt
|
||||
const promptText = fs.readFileSync(outPrompt, 'utf8');
|
||||
assert.ok(promptText.includes('Some research findings.'));
|
||||
} finally {
|
||||
cleanup(dir);
|
||||
}
|
||||
});
|
||||
|
||||
// ── Cycle 3: hard-fail path — budget is impossibly small ──────────────────
|
||||
test('hard-fail path: exit 2 when minimum set exceeds budget, metadata written, prompt empty', () => {
|
||||
const dir = createTempDir('pb-cli-hardfail-');
|
||||
try {
|
||||
const instrFile = path.join(dir, 'instructions.md');
|
||||
const roadmapFile = path.join(dir, 'roadmap.md');
|
||||
const planFile = path.join(dir, 'plan-01.md');
|
||||
const outPrompt = path.join(dir, 'out-prompt.md');
|
||||
const outMeta = path.join(dir, 'out-meta.json');
|
||||
|
||||
// Large instructions to ensure minimum set exceeds the tiny budget
|
||||
fs.writeFileSync(instrFile, 'a'.repeat(4000)); // ~1000 tokens
|
||||
fs.writeFileSync(roadmapFile, TEST_ROADMAP);
|
||||
fs.writeFileSync(planFile, TEST_PLAN);
|
||||
|
||||
// Budget of 5 tokens is far below the minimum set
|
||||
const { exitCode, stderr } = runCli([
|
||||
'prompt-budget',
|
||||
'--budget', '5',
|
||||
'--instructions-file', instrFile,
|
||||
'--roadmap-file', roadmapFile,
|
||||
'--plan-file', planFile,
|
||||
'--output-prompt', outPrompt,
|
||||
'--output-metadata', outMeta,
|
||||
], dir);
|
||||
|
||||
assert.equal(exitCode, 2, `Expected exit 2, got ${exitCode}. stderr: ${stderr}`);
|
||||
|
||||
// Metadata must still be written
|
||||
assert.ok(fs.existsSync(outMeta), 'metadata file must be written even on hard fail');
|
||||
const meta = JSON.parse(fs.readFileSync(outMeta, 'utf8'));
|
||||
assert.equal(meta.hardFailed, true);
|
||||
|
||||
// Prompt file must be written but empty
|
||||
assert.ok(fs.existsSync(outPrompt), 'prompt file must exist even on hard fail');
|
||||
const promptText = fs.readFileSync(outPrompt, 'utf8');
|
||||
assert.equal(promptText, '', 'prompt file must be empty on hard fail');
|
||||
} finally {
|
||||
cleanup(dir);
|
||||
}
|
||||
});
|
||||
|
||||
// ── Cycle 4: missing required arg — exit 1, stderr has error ──────────────
|
||||
test('missing required arg: exit 1, stderr contains error message', () => {
|
||||
const dir = createTempDir('pb-cli-missingarg-');
|
||||
try {
|
||||
const instrFile = path.join(dir, 'instructions.md');
|
||||
const roadmapFile = path.join(dir, 'roadmap.md');
|
||||
const planFile = path.join(dir, 'plan-01.md');
|
||||
const outPrompt = path.join(dir, 'out-prompt.md');
|
||||
const outMeta = path.join(dir, 'out-meta.json');
|
||||
|
||||
fs.writeFileSync(instrFile, TEST_INSTRUCTIONS);
|
||||
fs.writeFileSync(roadmapFile, TEST_ROADMAP);
|
||||
fs.writeFileSync(planFile, TEST_PLAN);
|
||||
|
||||
// Omit --budget (required)
|
||||
const { exitCode, stderr } = runCli([
|
||||
'prompt-budget',
|
||||
'--instructions-file', instrFile,
|
||||
'--roadmap-file', roadmapFile,
|
||||
'--plan-file', planFile,
|
||||
'--output-prompt', outPrompt,
|
||||
'--output-metadata', outMeta,
|
||||
], dir);
|
||||
|
||||
assert.equal(exitCode, 1, `Expected exit 1, got ${exitCode}`);
|
||||
assert.ok(stderr.includes('--budget'), `Expected stderr to mention --budget, got: ${stderr}`);
|
||||
} finally {
|
||||
cleanup(dir);
|
||||
}
|
||||
});
|
||||
|
||||
// ── Cycle 5: missing input file — exit 1 ──────────────────────────────────
|
||||
test('missing input file: exit 1', () => {
|
||||
const dir = createTempDir('pb-cli-missingfile-');
|
||||
try {
|
||||
const roadmapFile = path.join(dir, 'roadmap.md');
|
||||
const planFile = path.join(dir, 'plan-01.md');
|
||||
const outPrompt = path.join(dir, 'out-prompt.md');
|
||||
const outMeta = path.join(dir, 'out-meta.json');
|
||||
|
||||
fs.writeFileSync(roadmapFile, TEST_ROADMAP);
|
||||
fs.writeFileSync(planFile, TEST_PLAN);
|
||||
|
||||
// Instructions file doesn't exist
|
||||
const { exitCode } = runCli([
|
||||
'prompt-budget',
|
||||
'--budget', '10000',
|
||||
'--instructions-file', path.join(dir, 'nonexistent.md'),
|
||||
'--roadmap-file', roadmapFile,
|
||||
'--plan-file', planFile,
|
||||
'--output-prompt', outPrompt,
|
||||
'--output-metadata', outMeta,
|
||||
], dir);
|
||||
|
||||
assert.equal(exitCode, 1, 'Expected exit 1 when instructions file is missing');
|
||||
} finally {
|
||||
cleanup(dir);
|
||||
}
|
||||
});
|
||||
});
|
||||
44
tests/prompt-budget-sdk-parity-optimizer.test.cjs
Normal file
44
tests/prompt-budget-sdk-parity-optimizer.test.cjs
Normal file
@@ -0,0 +1,44 @@
|
||||
'use strict';
|
||||
|
||||
const { describe, test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const SDK_PROMPT_BUDGET = path.join(
|
||||
__dirname,
|
||||
'..',
|
||||
'sdk',
|
||||
'src',
|
||||
'query',
|
||||
'prompt-budget.ts'
|
||||
);
|
||||
|
||||
describe('prompt-budget sdk parity (optimizer)', () => {
|
||||
test('trim order keeps plan truncation ahead of context/research drops', () => {
|
||||
const src = fs.readFileSync(SDK_PROMPT_BUDGET, 'utf8');
|
||||
|
||||
const planIdx = src.indexOf('// Trim step 2: proportional plan truncation');
|
||||
const contextIdx = src.indexOf('// Trim step 3: drop context');
|
||||
const researchIdx = src.indexOf('// Trim step 4: drop research');
|
||||
|
||||
assert.ok(planIdx > -1, 'expected step 2 proportional plan truncation marker');
|
||||
assert.ok(contextIdx > -1, 'expected step 3 context drop marker');
|
||||
assert.ok(researchIdx > -1, 'expected step 4 research drop marker');
|
||||
assert.ok(
|
||||
planIdx < contextIdx && contextIdx < researchIdx,
|
||||
'expected trim order: plan truncation -> context drop -> research drop'
|
||||
);
|
||||
});
|
||||
|
||||
test('final over-budget hard-fail guard is present', () => {
|
||||
const src = fs.readFileSync(SDK_PROMPT_BUDGET, 'utf8');
|
||||
|
||||
assert.match(
|
||||
src,
|
||||
/if\s*\(\s*estimatedTokens\s*>\s*effectiveBudget\s*\)/,
|
||||
'expected final estimatedTokens > effectiveBudget guard'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
342
tests/prompt-budget.test.cjs
Normal file
342
tests/prompt-budget.test.cjs
Normal file
@@ -0,0 +1,342 @@
|
||||
'use strict';
|
||||
|
||||
const { describe, test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
|
||||
const { estimateTokens, applyBudget } = require('../get-shit-done/bin/lib/prompt-budget.cjs');
|
||||
|
||||
describe('prompt-budget', () => {
|
||||
// ── Cycle 1: estimator basics ──────────────────────────────────────────────
|
||||
test('estimateTokens — basic contract', () => {
|
||||
assert.equal(estimateTokens(''), 0);
|
||||
assert.equal(estimateTokens('hello'), 2); // ceil(5/4) = 2
|
||||
assert.equal(estimateTokens('a'.repeat(400)), 100); // 400/4 = 100 exactly
|
||||
});
|
||||
|
||||
// ── Cycle 2: applyBudget no-trim path ─────────────────────────────────────
|
||||
test('applyBudget — no-trim path when well under budget', () => {
|
||||
|
||||
const sections = {
|
||||
instructions: 'Review this code.',
|
||||
projectMd: 'Project info.',
|
||||
roadmap: 'Phase 1: ship it.',
|
||||
plans: [{ file: 'plan-a.md', content: 'Do the thing.' }],
|
||||
context: 'Session context.',
|
||||
research: 'Background research.',
|
||||
requirements: 'Must pass tests.',
|
||||
};
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 10000 });
|
||||
|
||||
assert.deepEqual(metadata.omitted, []);
|
||||
assert.equal(metadata.hardFailed, false);
|
||||
assert.equal(metadata.noteInjected, false);
|
||||
assert.equal(metadata.projectMdShrunk, false);
|
||||
assert.equal(metadata.planTruncationPct, 0);
|
||||
|
||||
assert.ok(prompt.includes('Review this code.'));
|
||||
assert.ok(prompt.includes('Project info.'));
|
||||
assert.ok(prompt.includes('Phase 1: ship it.'));
|
||||
assert.ok(prompt.includes('Do the thing.'));
|
||||
assert.ok(prompt.includes('Session context.'));
|
||||
assert.ok(prompt.includes('Background research.'));
|
||||
assert.ok(prompt.includes('Must pass tests.'));
|
||||
});
|
||||
|
||||
// ── Cycle 3: drop research when it is the sole cause of over-budget ─────────
|
||||
test('applyBudget — drops research when it alone causes over-budget', () => {
|
||||
// context is null; research alone pushes us over budget.
|
||||
// Without research: base ≈ 20 tokens. With research (~500 tokens): over budget.
|
||||
// effectiveBudget = floor(200 * 0.9) = 180. contentBudget = 180 - 80 = 100.
|
||||
const researchContent = 'a'.repeat(2000); // ~500 tokens
|
||||
|
||||
const sections = {
|
||||
instructions: 'Review this code.',
|
||||
projectMd: null,
|
||||
roadmap: 'Phase 1: ship it.',
|
||||
plans: [{ file: 'plan-a.md', content: 'Do the thing.' }],
|
||||
context: null,
|
||||
research: researchContent,
|
||||
requirements: null,
|
||||
};
|
||||
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 200 });
|
||||
|
||||
assert.deepEqual(metadata.omitted, ['research']);
|
||||
assert.equal(metadata.noteInjected, true);
|
||||
assert.equal(prompt.includes(researchContent), false);
|
||||
});
|
||||
|
||||
// ── Cycle 4: drop context first, then research ────────────────────────────
|
||||
test('applyBudget — drops context then research when both needed', () => {
|
||||
// Both context and research present; both must be dropped.
|
||||
// context drops first (priority 6), research second (priority 7).
|
||||
// base without either ≈ 20 tokens; each of context+research adds ~500 tokens.
|
||||
// effectiveBudget = floor(200 * 0.9) = 180. contentBudget = 100.
|
||||
const bigContent = 'b'.repeat(2000); // ~500 tokens each
|
||||
|
||||
const sections = {
|
||||
instructions: 'Review this code.',
|
||||
projectMd: null,
|
||||
roadmap: 'Phase 1: ship it.',
|
||||
plans: [{ file: 'plan-a.md', content: 'Do the thing.' }],
|
||||
context: bigContent,
|
||||
research: bigContent,
|
||||
requirements: null,
|
||||
};
|
||||
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 200 });
|
||||
|
||||
// context drops first per spec (priority 6), research second (priority 7)
|
||||
assert.deepEqual(metadata.omitted, ['context', 'research']);
|
||||
assert.equal(metadata.noteInjected, true);
|
||||
});
|
||||
|
||||
// ── Cycle 5: head-shrink PROJECT.md before dropping context ──────────────
|
||||
test('applyBudget — shrinks PROJECT.md to 40 lines before dropping context', () => {
|
||||
// PROJECT.md is 80 lines; shrinking to 40 saves enough that context survives.
|
||||
// 80 lines of ~20 chars each = 1600 chars ≈ 400 tokens
|
||||
// 40 lines = 800 chars ≈ 200 tokens (saves ~200 tokens)
|
||||
// effectiveBudget = floor(700 * 0.9) = 630. contentBudget = 630 - 80 = 550.
|
||||
// base with 80-line projectMd + short context ≈ 20 + 400 + short ≈ 440 → fits at 550 after shrink to 40
|
||||
// base with full 80-line projectMd ≈ 20 + 400 + 10 = 430... need it to be over 550 with full projectMd
|
||||
|
||||
// Build 80-line projectMd where shrinking to 40 saves enough
|
||||
const lineOf20 = 'x'.repeat(19); // 19 chars + newline = 20 chars per line
|
||||
const projectMdFull = Array.from({ length: 80 }, () => lineOf20).join('\n');
|
||||
// full: 80*20 - 1 = 1599 chars ≈ 400 tokens; shrunk: 40*20 - 1 = 799 chars ≈ 200 tokens
|
||||
|
||||
// Budget: effective = 400, contentBudget = 320
|
||||
// base with full projectMd + short context ≈ overhead(20) + projectMd(400) + context(5) ≈ 425 > 320 → pressure
|
||||
// After shrink: overhead(20) + projectMd_shrunk(200) + context(5) ≈ 225 < 320 → fits, no drops
|
||||
const sections = {
|
||||
instructions: 'Review this.',
|
||||
projectMd: projectMdFull,
|
||||
roadmap: 'Phase 1.',
|
||||
plans: [{ file: 'plan.md', content: 'Plan here.' }],
|
||||
context: 'Short context.',
|
||||
research: null,
|
||||
requirements: null,
|
||||
};
|
||||
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 445 });
|
||||
|
||||
assert.equal(metadata.projectMdShrunk, true);
|
||||
assert.deepEqual(metadata.omitted, []);
|
||||
assert.equal(metadata.noteInjected, true);
|
||||
assert.ok(prompt.includes('Short context.'));
|
||||
});
|
||||
|
||||
// ── Cycle 6: proportional plan truncation when 2+ plans ──────────────────
|
||||
test('applyBudget — proportionally truncates plans, never drops a whole plan', () => {
|
||||
// Two plans of ~1000 tokens each (4000 chars each).
|
||||
// Budget chosen so plans must shrink ~30%.
|
||||
// effectiveBudget = floor(1600 * 0.9) = 1440. contentBudget = 1440 - 80 = 1360.
|
||||
// overhead ≈ 30 tokens. planBudget ≈ 1330 tokens ≈ 5320 chars.
|
||||
// original total plan chars = 8000. remaining = 5320. reduction = (8000-5320)/8000 ≈ 33.5%
|
||||
const planContent = 'p'.repeat(4000); // ~1000 tokens each
|
||||
|
||||
const sections = {
|
||||
instructions: 'Review this.',
|
||||
projectMd: null,
|
||||
roadmap: 'Phase 1.',
|
||||
plans: [
|
||||
{ file: 'plan-a.md', content: planContent },
|
||||
{ file: 'plan-b.md', content: planContent },
|
||||
],
|
||||
context: null,
|
||||
research: null,
|
||||
requirements: null,
|
||||
};
|
||||
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 1600 });
|
||||
|
||||
// Both plans still appear
|
||||
assert.ok(prompt.includes('### plan-a.md'));
|
||||
assert.ok(prompt.includes('### plan-b.md'));
|
||||
|
||||
// planTruncationPct within ±10 of 30%
|
||||
assert.ok(
|
||||
metadata.planTruncationPct >= 20 && metadata.planTruncationPct <= 40,
|
||||
'planTruncationPct=' + metadata.planTruncationPct + ' expected ~30 (±10)'
|
||||
);
|
||||
assert.equal(metadata.noteInjected, true);
|
||||
});
|
||||
|
||||
// ── Cycle 7: drop requirements only as last resort ────────────────────────
|
||||
test('applyBudget — drops requirements as last resort after all other trims', () => {
|
||||
// context + research + plan-truncation + project-shrink all applied but still over
|
||||
// → drop requirements.
|
||||
// Use a tight budget with all optional sections present and large.
|
||||
// effectiveBudget = floor(200 * 0.9) = 180. contentBudget = 100.
|
||||
// Each big section = 'x'.repeat(2000) ≈ 500 tokens.
|
||||
// base (no optionals) ≈ 20 tokens. With requirements: ~520 tokens.
|
||||
// After dropping context + research: still need requirements dropped.
|
||||
const bigContent = 'x'.repeat(2000);
|
||||
|
||||
const sections = {
|
||||
instructions: 'Review.',
|
||||
projectMd: null,
|
||||
roadmap: 'Phase 1.',
|
||||
plans: [{ file: 'plan.md', content: 'Plan here.' }],
|
||||
context: bigContent,
|
||||
research: bigContent,
|
||||
requirements: bigContent,
|
||||
};
|
||||
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 200 });
|
||||
|
||||
assert.deepEqual(metadata.omitted, ['context', 'research', 'requirements']);
|
||||
assert.equal(metadata.noteInjected, true);
|
||||
assert.equal(metadata.hardFailed, false);
|
||||
});
|
||||
|
||||
// ── Cycle 8: hard-fail when minimum-set exceeds budget ───────────────────
|
||||
test('applyBudget — hard-fails when minimum-set exceeds effective budget', () => {
|
||||
// budget=50, effectiveBudget=45. instructions alone = 200+ tokens.
|
||||
// minSet = instructions + NOTE_RESERVE(80) + roadmap + min-plan = way over 45.
|
||||
const bigInstructions = 'i'.repeat(800); // 200 tokens
|
||||
|
||||
const sections = {
|
||||
instructions: bigInstructions,
|
||||
projectMd: null,
|
||||
roadmap: 'Phase 1.',
|
||||
plans: [{ file: 'plan.md', content: 'Plan here.' }],
|
||||
context: null,
|
||||
research: null,
|
||||
requirements: null,
|
||||
};
|
||||
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 50 });
|
||||
|
||||
assert.equal(metadata.hardFailed, true);
|
||||
assert.equal(prompt, '');
|
||||
});
|
||||
|
||||
// ── Cycle 9: note reservation invariant (load-bearing) ───────────────────
|
||||
test('applyBudget — reserves note tokens before drop math so final prompt fits', () => {
|
||||
// Invariant: the algorithm reserves ~80 note tokens BEFORE deciding what to drop.
|
||||
// If the implementation skips reservation, a buggy version could:
|
||||
// 1. See base_with_research >> effectiveBudget → drop research
|
||||
// 2. See base_without_research < effectiveBudget → think "no more trim needed"
|
||||
// 3. Inject note anyway → final prompt exceeds effectiveBudget
|
||||
//
|
||||
// Design (budget=145, effectiveBudget=130, contentBudget=50):
|
||||
// base_without_research = 60 tokens → in (contentBudget=50, effectiveBudget=130)
|
||||
// base_with_research = 564 tokens → triggers pressure
|
||||
// base_without_research + note_reserve(80) = 140 > effectiveBudget(130) → must reserve
|
||||
// minSet(130) ≤ effectiveBudget(130) → just barely avoids hard fail
|
||||
//
|
||||
// A correct implementation drops research and ensures estimatedTokens ≤ effectiveBudget.
|
||||
const sections = {
|
||||
instructions: 'i'.repeat(120), // 30 tokens
|
||||
projectMd: null,
|
||||
roadmap: 'r'.repeat(40), // 10 tokens
|
||||
plans: [{ file: 'plan.md', content: 'p'.repeat(40) }], // 10 tokens
|
||||
context: null,
|
||||
research: 'x'.repeat(2000), // ~500 tokens — causes the pressure
|
||||
requirements: null,
|
||||
};
|
||||
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 145 });
|
||||
|
||||
// Research must be dropped (it caused the pressure)
|
||||
assert.ok(metadata.omitted.includes('research'), 'research must be omitted');
|
||||
// Note must be injected (trim occurred)
|
||||
assert.equal(metadata.noteInjected, true);
|
||||
// Final estimated tokens must be within effectiveBudget (the load-bearing assertion)
|
||||
assert.ok(
|
||||
metadata.estimatedTokens <= metadata.effectiveBudget,
|
||||
'estimatedTokens=' + metadata.estimatedTokens +
|
||||
' must be ≤ effectiveBudget=' + metadata.effectiveBudget
|
||||
);
|
||||
// Not a hard failure
|
||||
assert.equal(metadata.hardFailed, false);
|
||||
});
|
||||
|
||||
// ── Cycle 11: no false hard-fail when untrimmed prompt fits effectiveBudget ─
|
||||
test('applyBudget — does not hard-fail when untrimmed prompt fits within effectiveBudget', () => {
|
||||
// budget=44 → effectiveBudget=39. Full untrimmed prompt = 32 tokens ≤ 39.
|
||||
// Bug: minSet unconditionally includes NOTE_RESERVE_TOKENS(80), making
|
||||
// minSet=102 > 39 and triggering a spurious hard-fail even though no note
|
||||
// is needed (no trim occurs) and the prompt genuinely fits.
|
||||
// Fix: exclude NOTE_RESERVE_TOKENS from minSet; only account for it if trim
|
||||
// is actually needed.
|
||||
const sections = {
|
||||
instructions: 'i'.repeat(32), // 8 tokens
|
||||
projectMd: null,
|
||||
roadmap: 'r'.repeat(16), // 4 tokens
|
||||
plans: [{ file: 'plan.md', content: 'p'.repeat(40) }], // 10 tokens
|
||||
context: null,
|
||||
research: null,
|
||||
requirements: null,
|
||||
};
|
||||
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 44 });
|
||||
|
||||
assert.equal(metadata.hardFailed, false, 'must not hard-fail when prompt fits effectiveBudget');
|
||||
assert.ok(prompt.length > 0, 'prompt must be non-empty');
|
||||
assert.deepEqual(metadata.omitted, [], 'nothing should be omitted');
|
||||
assert.equal(metadata.noteInjected, false, 'no note needed — no trim occurred');
|
||||
});
|
||||
|
||||
// ── Cycle 12: no unneeded trim when full prompt already fits effectiveBudget ─
|
||||
test('applyBudget — does not drop context or research when full untrimmed prompt already fits effectiveBudget', () => {
|
||||
// budget=156 → effectiveBudget=140. Full prompt (with context+research) ≈ 88 tokens ≤ 140.
|
||||
// Bug: budgetUnderPressure = baseTokens > effectiveBudget - NOTE_RESERVE_TOKENS
|
||||
// = 89 > 60 → true, sets contentBudget=60, triggers trim steps → drops context/research.
|
||||
// Fix: budgetUnderPressure should check baseTokens > effectiveBudget, not the pre-reserved
|
||||
// threshold. Note reservation happens only after a real trim decision is made.
|
||||
const sections = {
|
||||
instructions: 'i'.repeat(120), // 30 tokens
|
||||
projectMd: null,
|
||||
roadmap: 'r'.repeat(40), // 10 tokens
|
||||
plans: [{ file: 'plan-a.md', content: 'p'.repeat(40) }], // 10 tokens
|
||||
context: 'c'.repeat(40), // 10 tokens + header
|
||||
research: 'x'.repeat(40), // 10 tokens + header
|
||||
requirements: null,
|
||||
};
|
||||
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 156 });
|
||||
|
||||
assert.equal(metadata.hardFailed, false, 'must not hard-fail');
|
||||
assert.deepEqual(metadata.omitted, [], 'context and research must NOT be omitted');
|
||||
assert.ok(prompt.length > 0, 'prompt must be non-empty');
|
||||
// context and research must appear in the assembled prompt
|
||||
assert.ok(prompt.includes('## Context'), 'context section must be present');
|
||||
assert.ok(prompt.includes('## Research'), 'research section must be present');
|
||||
assert.equal(metadata.noteInjected, false, 'no note needed — no trim occurred');
|
||||
});
|
||||
|
||||
// ── Cycle 10: null optional sections ─────────────────────────────────────
|
||||
test('applyBudget — null optional sections are excluded from prompt without counting as omitted', () => {
|
||||
// All optionals are null, no projectMd — big budget so no trim.
|
||||
const sections = {
|
||||
instructions: 'Review this code.',
|
||||
projectMd: null,
|
||||
roadmap: 'Phase 1: ship it.',
|
||||
plans: [{ file: 'plan.md', content: 'Do the thing.' }],
|
||||
context: null,
|
||||
research: null,
|
||||
requirements: null,
|
||||
};
|
||||
|
||||
const { prompt, metadata } = applyBudget({ sections, budget: 10000 });
|
||||
|
||||
// Nulls don't count as "omitted" — only sections with content that were trimmed do
|
||||
assert.deepEqual(metadata.omitted, []);
|
||||
assert.equal(metadata.hardFailed, false);
|
||||
assert.equal(metadata.noteInjected, false);
|
||||
|
||||
// None of the optional section headers should appear
|
||||
assert.equal(prompt.includes('## Context'), false);
|
||||
assert.equal(prompt.includes('## Research'), false);
|
||||
assert.equal(prompt.includes('## Requirements'), false);
|
||||
assert.equal(prompt.includes('## Project'), false);
|
||||
|
||||
// Required sections must still appear
|
||||
assert.ok(prompt.includes('Review this code.'));
|
||||
assert.ok(prompt.includes('Phase 1: ship it.'));
|
||||
assert.ok(prompt.includes('Do the thing.'));
|
||||
});
|
||||
|
||||
});
|
||||
Reference in New Issue
Block a user