* test(#2929): capture prompt-budget parity corpus pre-refactor Phase 2 of epic #1671 generalizes prompt-budget's trim ladder into a shared context-composer seam. Its success condition is that review-prompt output does not change, and the only authority on "did not change" is the behavior that shipped before the refactor. Capture that behavior now, while it is still the live implementation. 47 characterization cases, every `expected` value computed by executing the current implementation rather than hand-authored — the independence CONTRIBUTING.md "Fixture provenance (#2371)" asks for. A corpus is only worth what it can detect, so this one was validated by mutation rather than assumed. Five deliberate defects were injected and each must be caught by at least one case: - the note reserve deducted unconditionally instead of only under pressure - the pressure test relaxed from `>` to `>=` - a no-op head-shrink still setting the shrunk flag - the per-plan floor dropped from the proportional share - drop order reversed Two of those exposed real holes in the first cut of this corpus, and the cases that close them exist because of it: - `>=` was caught by NOTHING. At exact cap the only trimmable fragment was a floored plan group, and the 1024-char floor absorbed the entire trim, so the mutation was byte-invisible. A3b/A3c put a droppable at exactly the cap, which makes the strict inequality observable as context kept vs omitted. - No case reached proportional-truncate at all — B6 and B7 both hard-failed the min-set pre-check first, leaving planTruncationPct at 0 across every case and the floor semantics entirely unexercised. Rebudgeted to 700 and 1100 so the min-set fits and the truncate step is actually reached; they now record 40.20% and 48.80%. The A4/A10 families sweep the pressure boundary from both sides, which is where this function has regressed before: CONTEXT.md's LEARNING.prompt-budget.boundary-gap records PR #3708 shipping two regressions that only fired when the baseline sat inside the NOTE_RESERVE_TOKENS band, because the suite paired a trivially-fitting budget with a trivially-overflowing one and never sampled between them. A4 pins that nothing is trimmed from the cap down to 81 tokens under it; A10 pins that pressure fires at +1. Together with A3b/A3c they satisfy row (d) of RULESET.TESTS.boundary-coverage.fixtures. Two facts the corpus establishes that the design notes had wrong: - "" and null sections are NOT distinguished. applyBudget uses truthy checks throughout, so an empty-string section is treated as absent: not rendered, not dropped, never recorded in `omitted`. B13b pins this while the ladder is actively trimming, where only the non-empty `research` is dropped. - Sizing matters. B12/B13 were first written at a budget where both hard-failed the min-set check and returned "", so comparing them compared two empty strings and proved nothing. Committed as its own commit, ahead of the refactor, and regenerated against the pre-refactor implementation, so the oracle is demonstrably independent of the change it will adjudicate. Refs #2929 * refactor(#2929): extract the context-composer seam from prompt-budget Epic #1671 needs prompt-budget's budget-trimming logic for a second consumer — per-runtime artifact emission — but it is walled inside the cross-AI review pipeline. Lift it into a shared seam so later phases can call it, without changing what the review pipeline emits. ADR-1671 specifies the composer as "priority + binary-search cutoff to a per-runtime budget". Read against the code it generalizes, that contract cannot express the thing being generalized. applyBudget is not a cutoff: it is a fixed five-step ladder in which each section carries its own shrink strategy, and only three of its eight sections are ever dropped. PROJECT.md is head-shrunk to N lines; plans are proportionally tail-truncated with a per-plan 1024-byte floor; instructions and roadmap are never touched at all. A cutoff composer sorts by priority and discards the tail — it has no way to say "shrink this one", "truncate that one but never below 1 KB each", or "these three are the only droppables, in this order". Building to the literal contract and routing prompt-budget through it would have silently changed review-prompt output, which is the one outcome this phase forbids. So shrink strategies are the core abstraction here, and cutoff becomes one strategy among them — the right one for per-runtime emission in Phases 3-4, not for this ladder. That is an elaboration of the ADR's intent, not a departure from it, and ADR-1671 is updated to say so. Three decisions worth stating: - The composer DECIDES; the caller RENDERS. composeWithinBudget returns a plan of surviving fragments and never a string. assemblePrompt's rendering is prompt-shaped (`## Roadmap`, `### <file>`, the note in position two), and owning it in the composer would force emission to adopt prompt-shaped rendering. The split is what lets one seam serve both consumers. - The budget unit is INJECTED via `measure(text)`. prompt-budget passes its chars/4 estimator; emission will pass a byte counter, which ADR-1671 requires for emission caps. The existing code converts a token budget to a character budget with a hardcoded `* 4`; that assumption is now an explicit `charsPerUnit` inverse, which is precisely what a byte unit needs in order to reuse this. - The entry point is `composeWithinBudget`, not `applyBudget`. That name already exists twice — src/prompt-budget.cts and src/graphify.cts, the latter being an unrelated graph-edge budget. A third would make every symbol search in this repo ambiguous, and it already misresolves: preflight and impact queries for "applyBudget" return graphify's. Behavior is unchanged and proven so: all 47 characterization cases reproduce byte-identically, and the corpus is mutation-validated rather than merely green (see the preceding commit). prompt-budget.cts drops from 436 to 343 lines and from eighteen mutable accumulators to two, both inside a helper copied verbatim. estimateTokens deliberately stays in prompt-budget and keeps its exact math: src/phase-estimation.cts re-exports it as measureTokens, and CONTEXT.md pins plan estimates and recorded actuals to that same scale, so moving or changing it would silently break the calibration loop. Refs #2929 * docs(#2929): document the context-composer seam and amend ADR-1671 Adds the INVENTORY row, the CONTEXT.md glossary entry (a PR gate for new domain modules), and a mutation-matrix entry for the new module. The ADR amendment is the substantive part. ADR-1671 specified the composer as "priority + binary-search cutoff to a per-runtime budget". Implementing Phase 2 established that a cutoff alone cannot express the function the platform generalizes, so the ADR now records shrink strategies as the core abstraction with cutoff as one strategy among them, reserved for per-runtime emission in Phases 3-4. Recording it in the ADR matters because Phases 3-6 are planned against that contract and would otherwise be planned against a mechanism that does not work. The mutation-matrix entry is not bookkeeping. Stryker scores per module against a named .cjs, so relocating the ladder out of prompt-budget.cjs would leave the extracted code unmeasured while prompt-budget's own score floated free of the logic it used to cover. context-composer gets its own entry at the same floor. Refs #2929 * test(#2929): pin the effectiveBudget rounding mode in the parity corpus An isolated correctness review found a real blind spot: mutating `Math.floor` to `Math.round` in the effectiveBudget calculation failed ZERO of the 47 corpus cases. Every (budget, safetyMarginPct) pair in the generator happened to produce a whole number, so floor, round and ceil all agreed and the rounding mode was entirely unpinned by a corpus whose whole job is to pin observable behavior. Three cases fix that by straddling the .5 boundary: A11 95 * 0.90 = 85.5 floor 85, round 86 -> the two disagree A12 97 * 0.90 = 87.3 floor and round agree; ceil (88) does not A13 93 * 0.85 = 79.05 same guard at a non-multiple-of-10 margin, so the margin arithmetic is exercised and not just the budget A11 alone catches the round mutation; all three catch ceil. Regenerated against the pre-refactor implementation (`git show 9557f8552:src/prompt-budget.cts`), so the expanded corpus keeps the independence property the original capture had. The corpus is now mutation-validated against seven injected defects, every one caught: unconditional note reserve, `>` relaxed to `>=`, no-op head-shrink setting its flag, the truncate floor ignored, drop order reversed, and both rounding-mode changes. Refs #2929 * feat(#2929): flexReserve floors and the byte-stable isolate prefix Two of issue #2929's "Done when" items were unimplemented rather than deferred, and an isolated review flagged them alongside my own audit. Both are part of ADR-1671's composer contract, so shipping the seam without them would have left Phases 3-4 building against a contract that does not exist yet. flexReserve is a per-fragment floor in measure units that every strategy must respect, which is what makes it different from the pre-existing floorChars: that one is a chars-denominated detail of proportional-truncate alone and is retained unchanged. A floored fragment is never dropped, is never head-shrunk below its floor, and raises its own proportional cap. A fragment already smaller than its floor is untouchable outright. Metadata gains `floored`, listing the ids whose floor actually prevented a trim — a guarantee no caller can observe is a guarantee no test can hold you to. isolate marks the byte-stable canonical prefix the ADR calls for: never trimmed, never dropped, but still counted, because a prefix excluded from accounting would silently under-count real context. Metadata gains `isolatePrefix` so a caller can hash or assert on the exact bytes. Declaring an isolate fragment after a non-isolate one throws: a prefix that is not at the front is not a prefix, and accepting it would make the cross-runtime stability claim meaningless. Adds tests/context-composer.test.cjs for the exact new semantics and tests/context-composer.property.test.cjs for the five invariants, including the budget-monotonicity property the issue names explicitly. Both are registered in the mutation matrix, since coverage does not migrate with relocated code. prompt-budget uses neither feature, and its output is unchanged: all 50 corpus cases still reproduce byte-identically. Refs #2929 * chore(#2929): allowlist the prompt-budget parity suite The parity corpus needs its own test file and that makes prompt-budget a three-file module against a limit of two. The lint offers consolidation or an allowlist entry with justification; the entry is the right call here. Consolidation would mean folding the characterization suite into prompt-budget.test.cjs, which is the one thing that should not happen to it. The parity suite is a distinct concern with a distinct lifecycle: it is generated rather than hand-written, it is named by scripts/mutation-matrix.cjs as its own scoring target, and its failure means something categorically different from a unit-test failure — not "this behavior is wrong" but "observable output moved". Burying it inside a general unit file would obscure exactly that signal. The allowlist is an identity ratchet, so this entry pins today's three exact filenames: adding a fourth still fails, and dropping back to two requires removing the entry. Refs #2929 * fix(#2929): register the new module with two gates it was missing The remote matrix caught three defects that no local check could, because the local runner is blocked in this repo and these suites had therefore never executed. Eight failures, identical on node22 and node24, so nothing environment-shaped. Two are the new-module ripple. A net-new src/*.cts lands in six places and this change had reached four of them — .gitignore, INVENTORY, the manifest, and the CONTEXT.md glossary — while missing the ESLint ignore list (tsc OUTPUTS must not be linted; repo-invariants asserts linted-xor-ignored) and the mutation ratchet baseline (a deliberate review-visible mirror of the matrix floors, which every COVERED module must carry). Both are now registered, the ratchet at the same floor of 66 the matrix declares. The third was a test asserting an outcome it had made impossible. It set budget:1 alongside a 400-char required fragment, so the group budget came out at -99 and the proportional-truncate step was skipped entirely — the deliberate "non-positive group budget is skipped, never clamped" rule inherited from the original ladder. Nothing was trimmed, and the test then asserted a truncation. Rebudgeted so the step actually runs, with the arithmetic written out in a comment so the next reader does not have to re-derive why 120 rather than 80. Fixing that surfaced a genuine bug in the composer. `floored` is documented as recording fragments whose flexReserve prevented a trim that would otherwise have happened, but the push sat in the else-branch of "content did not change", so it only fired when nothing was trimmed at all. A fragment truncated to a reserve-raised cap has also had a trim prevented — 40 characters' worth in the test above — and was silently absent from the field that exists to make the guarantee observable. The condition was already right; it was in the wrong branch. Now recorded on both paths: a drop prevented outright, and a truncation capped higher than the share alone would have allowed. Parity is unaffected — prompt-budget never sets flexReserve, so the branch is unreachable from every corpus path, and all 50 cases still match. Refs #2929 * chore(#2929): backfill changeset PR number (#2958) * chore(#2929): correct the corpus case count in the changeset fragment --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
645
scripts/gen-prompt-budget-parity-corpus.cjs
Normal file
645
scripts/gen-prompt-budget-parity-corpus.cjs
Normal file
@@ -0,0 +1,645 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* gen-prompt-budget-parity-corpus.cjs
|
||||
*
|
||||
* Captures a characterization corpus of `prompt-budget.applyBudget`'s observable
|
||||
* output, for issue #2929 (epic #1671 Phase 2 — the `context-composer` seam).
|
||||
*
|
||||
* WHY THIS EXISTS
|
||||
* ---------------
|
||||
* Phase 2 generalizes `applyBudget`'s trim ladder into a shared composer seam.
|
||||
* Its success condition is that `prompt-budget`'s output does not change. The only
|
||||
* authority on "did not change" is the behavior that shipped BEFORE the refactor,
|
||||
* so this corpus is generated against the pre-refactor implementation and committed
|
||||
* as a frozen fixture. `tests/prompt-budget-parity.test.cjs` then replays every case
|
||||
* and asserts byte-identical `prompt` plus field-identical `metadata`.
|
||||
*
|
||||
* FIXTURE PROVENANCE (CONTRIBUTING.md "Fixture provenance (#2371)")
|
||||
* -----------------------------------------------------------------
|
||||
* That rule forbids fixtures derived from the gate's own writer, grammar, or the
|
||||
* author's mental model, because such a fixture can only confirm what the author
|
||||
* already believed. This corpus is expressly NOT hand-authored: every `expected`
|
||||
* value is computed by executing the shipped implementation, which predates the
|
||||
* composer and knows nothing about it. What IS author-chosen is the set of INPUTS —
|
||||
* so the inputs are derived from the design's parity-critical behavior table rather
|
||||
* than from intuition, and `tests/prompt-budget-parity.test.cjs` additionally proves
|
||||
* the corpus is non-vacuous by mutating the ladder and requiring the corpus to fail.
|
||||
* Disclosed here rather than assumed.
|
||||
*
|
||||
* USAGE
|
||||
* node scripts/gen-prompt-budget-parity-corpus.cjs --write # regenerate
|
||||
* node scripts/gen-prompt-budget-parity-corpus.cjs --check # drift-guard (exit 1 on drift)
|
||||
*
|
||||
* `--check` is the intentional-change gate: after Phase 2 lands, any diff here means
|
||||
* review-prompt output moved, which is a user-visible change requiring justification.
|
||||
*/
|
||||
|
||||
'use strict';
|
||||
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const REPO_ROOT = path.resolve(__dirname, '..');
|
||||
const OUT_PATH = path.join(REPO_ROOT, 'tests', 'fixtures', 'prompt-budget-parity', 'corpus.json');
|
||||
const IMPL_PATH = path.join(REPO_ROOT, 'gsd-core', 'bin', 'lib', 'prompt-budget.cjs');
|
||||
|
||||
/** Corpus schema version. Bump only on a deliberate shape change. */
|
||||
const CORPUS_VERSION = 1;
|
||||
|
||||
// ─── Deterministic content builders ──────────────────────────────────────────
|
||||
// estimateTokens() is Math.ceil(len / 4), so a string of length 4*N measures
|
||||
// exactly N tokens. Every size below is therefore exact, not approximate — which
|
||||
// is what makes the cap-1 / cap / cap+1 rows land ON the boundary rather than near it.
|
||||
|
||||
/** A filler string measuring exactly `tokens` tokens under chars/4. */
|
||||
function tokensOf(tokens, fill = 'x') {
|
||||
return fill.repeat(tokens * 4);
|
||||
}
|
||||
|
||||
/** `lines` newline-separated lines, each `width` chars wide. */
|
||||
function linesOf(lines, width = 19) {
|
||||
return Array.from({ length: lines }, (_, i) => String(i).padEnd(width, '-')).join('\n');
|
||||
}
|
||||
|
||||
// ─── Case definitions ────────────────────────────────────────────────────────
|
||||
// Each case is { name, why, sections, budget, options }.
|
||||
// `why` names the design/test-matrix row the case exists to pin.
|
||||
|
||||
function buildCases() {
|
||||
const cases = [];
|
||||
const add = (c) => cases.push(c);
|
||||
|
||||
// Base shape reused by the boundary family. With safetyMarginPct 0 the effective
|
||||
// budget equals the budget, so totals can be placed exactly on the cap.
|
||||
// staticBase = est(instructions) + est('## Roadmap\n\n') + est(roadmap)
|
||||
// + est('## Plans\n\n') + est('### a.md\n\n')
|
||||
const boundaryBase = (planTokens) => ({
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(planTokens, 'P') }],
|
||||
});
|
||||
|
||||
// ── A. Budget boundary (RULESET.TESTS.boundary-coverage.fixtures a/b/c) ────
|
||||
add({
|
||||
name: 'A1-exact-cap',
|
||||
why: 'A1/row 3 — total measures exactly effectiveBudget; must NOT be treated as pressure',
|
||||
sections: boundaryBase(71),
|
||||
budget: 100,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'A2-one-under-cap',
|
||||
why: 'A2/row 5 — one token under; no pressure, no reserve deducted',
|
||||
sections: boundaryBase(70),
|
||||
budget: 100,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'A3-one-over-cap',
|
||||
why: 'A3/row 4 — one token over; pressure engages the ladder',
|
||||
sections: boundaryBase(72),
|
||||
budget: 100,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
|
||||
// A1 pins exact-cap as "no pressure", but with only a floored plan group present that
|
||||
// assertion is not load-bearing: the 1024-char floor absorbs any trim, so relaxing the
|
||||
// pressure test from `>` to `>=` produces byte-identical output and the corpus cannot
|
||||
// see the difference. This case gives the ladder something DROPPABLE at exactly the cap,
|
||||
// so `>` vs `>=` becomes observable as context surviving vs being omitted.
|
||||
// staticBase 29 + plan 20 = 49, context wrapper 3, so est(context) 148 puts the
|
||||
// baseline at exactly 200 = effectiveBudget.
|
||||
add({
|
||||
name: 'A3b-exact-cap-with-droppable-present',
|
||||
why: 'exact-cap discriminator — baseline lands exactly ON effectiveBudget with a droppable available. Pins that `baseline > effectiveBudget` is STRICT: at equality nothing is dropped. Mutating the comparison to >= flips context from kept to omitted, which A1 alone cannot detect.',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(20, 'P') }],
|
||||
context: tokensOf(148, 'C'),
|
||||
},
|
||||
budget: 200,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'A3c-one-under-cap-with-droppable-present',
|
||||
why: 'companion to A3b — one token under the cap with the same droppable available. Together they bracket the strict-inequality boundary from below.',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(20, 'P') }],
|
||||
context: tokensOf(147, 'C'),
|
||||
},
|
||||
budget: 200,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
|
||||
// Row (d) of RULESET.TESTS.boundary-coverage.fixtures: baseline within
|
||||
// reserve-distance of the limit. NOTE_RESERVE_TOKENS is 80, and this is the exact
|
||||
// band (effectiveBudget - 80, effectiveBudget] that PR #3708 shipped two
|
||||
// regressions in (LEARNING.prompt-budget.boundary-gap). Swept token by token.
|
||||
// Budget 200 (not 100) is required here, not incidental: the band is
|
||||
// NOTE_RESERVE_TOKENS (80) wide, so the baseline needs >=81 tokens of headroom
|
||||
// beneath the cap to be swept. staticBase is 29 tokens, so planTokens = 171 - delta
|
||||
// places the baseline exactly `delta` under effectiveBudget and stays positive
|
||||
// across the whole sweep. delta 80 and 81 sit just OUTSIDE the band deliberately —
|
||||
// they are the negative controls that prove the band's edge is where it is claimed.
|
||||
for (const delta of [0, 1, 2, 39, 40, 79, 80, 81]) {
|
||||
add({
|
||||
name: `A4-reserve-band-minus-${delta}`,
|
||||
why: `boundary-coverage.fixtures row (d) — baseline at effectiveBudget-${delta}. Pins the NEGATIVE control: within NOTE_RESERVE_TOKENS distance of the cap, NOTHING may be trimmed. Early-pressure firing here is the exact PR #3708 regression.`,
|
||||
sections: boundaryBase(171 - delta),
|
||||
budget: 200,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
}
|
||||
|
||||
// Mirror of A4 on the OTHER side of the cap. A4 proves the reserve is NOT deducted
|
||||
// at or below the cap; this proves it IS deducted above it, and that the ladder
|
||||
// relieves pressure by dropping. Without both sides the family pins only half the
|
||||
// boundary, and an "always reserve" regression would still pass A4.
|
||||
// staticBase 29 + planContent 20 = 49, and the context wrapper costs 3, so
|
||||
// est(context) = 148 + delta places the baseline exactly `delta` OVER effectiveBudget.
|
||||
for (const delta of [1, 2, 40, 79, 80, 81]) {
|
||||
add({
|
||||
name: `A10-pressure-band-plus-${delta}`,
|
||||
why: `boundary-coverage.fixtures row (d), above-cap mirror — baseline at effectiveBudget+${delta}; pressure fires, NOTE_RESERVE_TOKENS is deducted, and context is dropped`,
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(20, 'P') }],
|
||||
context: tokensOf(148 + delta, 'C'),
|
||||
},
|
||||
budget: 200,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
}
|
||||
|
||||
add({
|
||||
name: 'A5-safety-margin-default',
|
||||
why: 'A4 — default 10% margin; effectiveBudget = floor(budget * 0.9)',
|
||||
sections: boundaryBase(60),
|
||||
budget: 100,
|
||||
options: {},
|
||||
});
|
||||
|
||||
// The rounding mode of effectiveBudget was unpinned: every other case in this file
|
||||
// uses a (budget, safetyMarginPct) pair whose product is a whole number, so
|
||||
// Math.floor and Math.round agree and mutating one to the other changed nothing.
|
||||
// These three straddle .5 in both directions so the mode is observable.
|
||||
// 95 * 0.90 = 85.5 -> floor 85, round 86 (disagree)
|
||||
// 97 * 0.90 = 87.3 -> floor 87, round 87 (agree; guards against a ceil mutation)
|
||||
// 93 * 0.85 = 79.05 -> floor 79, round 79 (agree; second ceil guard, different margin)
|
||||
add({
|
||||
name: 'A11-fractional-effective-budget-half',
|
||||
why: 'pins the ROUNDING MODE of effectiveBudget. 95 * (1 - 10/100) = 85.5, where Math.floor (85) and Math.round (86) disagree. Without a fractional case the mode is unpinned and floor->round is a byte-invisible mutation, which an isolated review confirmed against the 47-case corpus.',
|
||||
sections: boundaryBase(56),
|
||||
budget: 95,
|
||||
options: { safetyMarginPct: 10 },
|
||||
});
|
||||
add({
|
||||
name: 'A12-fractional-effective-budget-below-half',
|
||||
why: 'companion to A11 below the .5 point: 97 * 0.90 = 87.3, where floor and round agree but Math.ceil (88) would not. Guards the mutation A11 does not cover.',
|
||||
sections: boundaryBase(58),
|
||||
budget: 97,
|
||||
options: { safetyMarginPct: 10 },
|
||||
});
|
||||
add({
|
||||
name: 'A13-fractional-effective-budget-odd-margin',
|
||||
why: 'third rounding guard at a non-multiple-of-10 margin: 93 * (1 - 15/100) = 79.05. Exercises the margin arithmetic itself, not just the budget, since safetyMarginPct is caller-supplied.',
|
||||
sections: boundaryBase(50),
|
||||
budget: 93,
|
||||
options: { safetyMarginPct: 15 },
|
||||
});
|
||||
add({
|
||||
name: 'A6-safety-margin-zero',
|
||||
why: 'A5 — 0% margin means the full budget is usable',
|
||||
sections: boundaryBase(60),
|
||||
budget: 100,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'A7-safety-margin-hundred',
|
||||
why: 'A5 — 100% margin drives effectiveBudget to 0, forcing the min-set hard fail',
|
||||
sections: boundaryBase(10),
|
||||
budget: 100,
|
||||
options: { safetyMarginPct: 100 },
|
||||
});
|
||||
add({
|
||||
name: 'A8-budget-zero',
|
||||
why: 'A6 — non-positive budget must hard-fail cleanly, never negative-length slice',
|
||||
sections: boundaryBase(10),
|
||||
budget: 0,
|
||||
options: {},
|
||||
});
|
||||
add({
|
||||
name: 'A9-budget-negative',
|
||||
why: 'A6 — negative budget must not crash',
|
||||
sections: boundaryBase(10),
|
||||
budget: -50,
|
||||
options: {},
|
||||
});
|
||||
|
||||
// ── B. Shrink strategies ──────────────────────────────────────────────────
|
||||
add({
|
||||
name: 'B3-head-shrink-engaged',
|
||||
why: 'B3/row 7 — projectMd longer than projectMdHeadLines under pressure; shrunk flag true',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(40, 'P') }],
|
||||
projectMd: linesOf(120),
|
||||
},
|
||||
budget: 200,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B4-head-shrink-noop',
|
||||
why: 'B4/row 7+9 — fewer lines than the cap; text unchanged and shrunk flag MUST stay false',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(40, 'P') }],
|
||||
projectMd: linesOf(5),
|
||||
},
|
||||
budget: 60,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B5-head-shrink-zero-lines',
|
||||
why: 'B5/row 8 — projectMdHeadLines <= 0 yields the empty string',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(40, 'P') }],
|
||||
projectMd: linesOf(120),
|
||||
},
|
||||
budget: 60,
|
||||
options: { safetyMarginPct: 0, projectMdHeadLines: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B6-plan-truncate-share-above-floor',
|
||||
why: 'B6/row 13 — proportional share ABOVE the floor, so the share (not the floor) decides. Budget 700: min-set is 5+5+256+256=522 which fits, so this reaches the truncate step instead of hard-failing the way the original budget-400 version did. groupBudget 598 of 1000 => charsBudget 2392, share 1196 per plan > floor 1024.',
|
||||
sections: {
|
||||
instructions: tokensOf(5, 'I'),
|
||||
roadmap: tokensOf(5, 'R'),
|
||||
plans: [
|
||||
{ file: 'a.md', content: tokensOf(500, 'A') },
|
||||
{ file: 'b.md', content: tokensOf(500, 'B') },
|
||||
],
|
||||
},
|
||||
budget: 700,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B7-plan-truncate-floor-binding',
|
||||
why: 'B7/row 10 — share falls BELOW the 1024-char floor so the floor wins, and the resulting total deliberately EXCEEDS the group budget. Budget 1100: min-set 5+5+4*256=1034 fits; groupBudget 992 => charsBudget 3968 => share 992 < 1024, so every plan is held at the floor. This is the case that proves a floor is a per-fragment guarantee, not a budget cap.',
|
||||
sections: {
|
||||
instructions: tokensOf(5, 'I'),
|
||||
roadmap: tokensOf(5, 'R'),
|
||||
plans: [
|
||||
{ file: 'a.md', content: tokensOf(500, 'A') },
|
||||
{ file: 'b.md', content: tokensOf(500, 'B') },
|
||||
{ file: 'c.md', content: tokensOf(500, 'C') },
|
||||
{ file: 'd.md', content: tokensOf(500, 'D') },
|
||||
],
|
||||
},
|
||||
budget: 1100,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B8-zero-sized-plans',
|
||||
why: 'B8/row 11 — total original chars 0; no divide-by-zero, pct stays 0',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [
|
||||
{ file: 'a.md', content: '' },
|
||||
{ file: 'b.md', content: '' },
|
||||
],
|
||||
context: tokensOf(200, 'C'),
|
||||
},
|
||||
budget: 60,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B9-nonpositive-plan-budget',
|
||||
why: 'B9/row 12 — computed plan budget <= 0 skips the truncate step ENTIRELY rather than clamping',
|
||||
sections: boundaryBase(72),
|
||||
budget: 100,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B11-drop-order-all-three',
|
||||
why: 'B11/row 14 — droppables leave in declared order: context, research, requirements',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(30, 'P') }],
|
||||
context: tokensOf(300, 'C'),
|
||||
research: tokensOf(300, 'S'),
|
||||
requirements: tokensOf(300, 'Q'),
|
||||
},
|
||||
budget: 200,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B11b-drop-context-only',
|
||||
why: 'B11 — pressure relieved after the first drop; research and requirements survive',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(30, 'P') }],
|
||||
context: tokensOf(300, 'C'),
|
||||
research: tokensOf(10, 'S'),
|
||||
requirements: tokensOf(10, 'Q'),
|
||||
},
|
||||
budget: 300,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B12-droppables-absent',
|
||||
why: 'B12/row 15 — null droppables are not dropped and never appear in omitted. Budget 400 (not 120) so this actually RENDERS: at 120 the min-set pre-check hard-fails and the case would compare two empty strings.',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(300, 'P') }],
|
||||
context: null,
|
||||
research: null,
|
||||
requirements: null,
|
||||
},
|
||||
budget: 400,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B13-droppables-empty-string',
|
||||
why: 'B13/row 23 — empty-string droppables. Paired with B12 to pin whether "" and null are distinguished. They are NOT: applyBudget uses truthy checks throughout, so "" behaves as absent. This case records that fact rather than the design doc\'s claim that they differ.',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(300, 'P') }],
|
||||
context: '',
|
||||
research: '',
|
||||
requirements: '',
|
||||
},
|
||||
budget: 400,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'B13b-empty-string-droppable-under-pressure',
|
||||
why: 'B13 companion — an empty-string droppable while the ladder is ACTIVELY running. Pins that "" is never recorded in omitted even under real pressure, which B12/B13 (no pressure) cannot show. research is real and IS dropped, so the ladder is demonstrably engaged.',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(20, 'P') }],
|
||||
context: '',
|
||||
research: tokensOf(200, 'S'),
|
||||
requirements: '',
|
||||
},
|
||||
budget: 200,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
|
||||
// ── C. Note reserve + hard-fail paths ─────────────────────────────────────
|
||||
add({
|
||||
name: 'C1-no-trim-no-note',
|
||||
why: 'C1/row 1 — everything fits; no note, noteInjected false',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(10, 'P') }],
|
||||
projectMd: linesOf(3),
|
||||
context: tokensOf(5, 'C'),
|
||||
research: tokensOf(5, 'S'),
|
||||
requirements: tokensOf(5, 'Q'),
|
||||
},
|
||||
budget: 1000,
|
||||
options: {},
|
||||
});
|
||||
add({
|
||||
name: 'C4-min-set-hard-fail',
|
||||
why: 'C4/row 16 — min-set pre-check exceeds budget; empty output and estimatedTokens 0',
|
||||
sections: {
|
||||
instructions: tokensOf(500, 'I'),
|
||||
roadmap: tokensOf(500, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(500, 'P') }],
|
||||
},
|
||||
budget: 100,
|
||||
options: {},
|
||||
});
|
||||
add({
|
||||
name: 'C5-post-assembly-hard-fail',
|
||||
why: 'C5/row 17 — survives the min-set check but the assembled prompt still overflows; real measured size reported',
|
||||
sections: {
|
||||
instructions: tokensOf(80, 'I'),
|
||||
roadmap: tokensOf(80, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(2000, 'P') }],
|
||||
},
|
||||
budget: 200,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'C7-custom-note-template',
|
||||
why: 'note template placeholders {budget} {omittedList} {planTruncationPct} all substitute',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(30, 'P') }],
|
||||
context: tokensOf(300, 'C'),
|
||||
},
|
||||
budget: 200,
|
||||
options: {
|
||||
safetyMarginPct: 0,
|
||||
noteTemplate: 'TRIMMED b={budget} o={omittedList} p={planTruncationPct}',
|
||||
},
|
||||
});
|
||||
add({
|
||||
name: 'C8-omitted-list-none',
|
||||
why: 'renderNote emits the literal "none" when a trim occurred but nothing was dropped',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(30, 'P') }],
|
||||
projectMd: linesOf(120),
|
||||
},
|
||||
budget: 120,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
|
||||
// ── D. Shape + encoding ───────────────────────────────────────────────────
|
||||
add({
|
||||
name: 'D1-all-sections-present',
|
||||
why: 'full assembly order: instructions, note, roadmap, project, plans, context, research, requirements',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [
|
||||
{ file: 'one.md', content: tokensOf(10, '1') },
|
||||
{ file: 'two.md', content: tokensOf(10, '2') },
|
||||
],
|
||||
projectMd: linesOf(4),
|
||||
context: tokensOf(10, 'C'),
|
||||
research: tokensOf(10, 'S'),
|
||||
requirements: tokensOf(10, 'Q'),
|
||||
},
|
||||
budget: 5000,
|
||||
options: {},
|
||||
});
|
||||
add({
|
||||
name: 'D2-no-optional-sections',
|
||||
why: 'minimum viable shape — only the two never-droppables plus one plan',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a.md', content: tokensOf(10, 'P') }],
|
||||
},
|
||||
budget: 5000,
|
||||
options: {},
|
||||
});
|
||||
add({
|
||||
name: 'D3-zero-plans',
|
||||
why: 'empty plans array — headers still assembled, reduce over [] is 0',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [],
|
||||
},
|
||||
budget: 5000,
|
||||
options: {},
|
||||
});
|
||||
add({
|
||||
name: 'D4-crlf-content',
|
||||
why: 'row 25 — CRLF measured and sliced identically to LF under the token unit',
|
||||
sections: {
|
||||
instructions: 'alpha\r\nbeta\r\ngamma',
|
||||
roadmap: 'one\r\ntwo\r\nthree',
|
||||
plans: [{ file: 'a.md', content: Array.from({ length: 60 }, (_, i) => `line ${i}`).join('\r\n') }],
|
||||
projectMd: Array.from({ length: 80 }, (_, i) => `p${i}`).join('\r\n'),
|
||||
},
|
||||
budget: 120,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'D5-unicode-content',
|
||||
why: 'row 26 — multibyte and astral content; chars/4 counts UTF-16 units, and slice may split a surrogate pair (pinned as current behavior)',
|
||||
sections: {
|
||||
instructions: 'héllo wörld ünicode',
|
||||
roadmap: '日本語のロードマップ',
|
||||
plans: [{ file: 'ünï.md', content: '🎉'.repeat(400) }],
|
||||
context: 'Ω'.repeat(200),
|
||||
},
|
||||
budget: 150,
|
||||
options: { safetyMarginPct: 0 },
|
||||
});
|
||||
add({
|
||||
name: 'D6-plan-filename-affects-budget',
|
||||
why: 'S5 — per-plan wrapper cost "### <file>\\n\\n" counts toward staticBaseTokens',
|
||||
sections: {
|
||||
instructions: tokensOf(10, 'I'),
|
||||
roadmap: tokensOf(10, 'R'),
|
||||
plans: [{ file: 'a-very-long-plan-filename-that-costs-tokens.md', content: tokensOf(10, 'P') }],
|
||||
},
|
||||
budget: 5000,
|
||||
options: {},
|
||||
});
|
||||
|
||||
return cases;
|
||||
}
|
||||
|
||||
// ─── Capture ─────────────────────────────────────────────────────────────────
|
||||
|
||||
function capture() {
|
||||
if (!fs.existsSync(IMPL_PATH)) {
|
||||
throw new Error(
|
||||
`prompt-budget.cjs not found at ${IMPL_PATH}. Run "npm run build:lib" first ` +
|
||||
'(the .cjs is a gitignored tsc output of src/prompt-budget.cts).'
|
||||
);
|
||||
}
|
||||
const { applyBudget } = require(IMPL_PATH);
|
||||
|
||||
const cases = buildCases();
|
||||
|
||||
const seen = new Set();
|
||||
for (const c of cases) {
|
||||
if (seen.has(c.name)) throw new Error(`duplicate corpus case name: ${c.name}`);
|
||||
seen.add(c.name);
|
||||
}
|
||||
|
||||
return {
|
||||
version: CORPUS_VERSION,
|
||||
description:
|
||||
'Characterization corpus of prompt-budget.applyBudget captured from the pre-refactor ' +
|
||||
'implementation. Oracle for issue #2929 (epic #1671 Phase 2). Regenerate with ' +
|
||||
'node scripts/gen-prompt-budget-parity-corpus.cjs --write',
|
||||
cases: cases.map((c) => {
|
||||
const result = applyBudget({
|
||||
sections: c.sections,
|
||||
budget: c.budget,
|
||||
options: c.options,
|
||||
});
|
||||
return {
|
||||
name: c.name,
|
||||
why: c.why,
|
||||
input: { sections: c.sections, budget: c.budget, options: c.options },
|
||||
expected: { prompt: result.prompt, metadata: result.metadata },
|
||||
};
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
function serialize(corpus) {
|
||||
return `${JSON.stringify(corpus, null, 2)}\n`;
|
||||
}
|
||||
|
||||
// ─── CLI ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
function main(argv) {
|
||||
const write = argv.includes('--write');
|
||||
const check = argv.includes('--check');
|
||||
|
||||
if (write === check) {
|
||||
process.stderr.write('usage: gen-prompt-budget-parity-corpus.cjs (--write | --check)\n');
|
||||
return 2;
|
||||
}
|
||||
|
||||
const serialized = serialize(capture());
|
||||
|
||||
if (write) {
|
||||
fs.mkdirSync(path.dirname(OUT_PATH), { recursive: true });
|
||||
fs.writeFileSync(OUT_PATH, serialized);
|
||||
const n = JSON.parse(serialized).cases.length;
|
||||
process.stdout.write(`wrote ${path.relative(REPO_ROOT, OUT_PATH)} (${n} cases)\n`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!fs.existsSync(OUT_PATH)) {
|
||||
process.stderr.write(
|
||||
`MISSING ${path.relative(REPO_ROOT, OUT_PATH)}\n` +
|
||||
'Run: node scripts/gen-prompt-budget-parity-corpus.cjs --write\n'
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
const committed = fs.readFileSync(OUT_PATH, 'utf8');
|
||||
if (committed !== serialized) {
|
||||
process.stderr.write(
|
||||
`DRIFT in ${path.relative(REPO_ROOT, OUT_PATH)}\n\n` +
|
||||
"prompt-budget's observable output no longer matches the committed corpus.\n" +
|
||||
'This corpus is the parity oracle for #2929: a diff here means review-prompt\n' +
|
||||
'output MOVED. That is user-visible.\n\n' +
|
||||
'If the change was NOT intended, fix the code — do not regenerate.\n' +
|
||||
'If it WAS intended, regenerate and justify the diff in the PR:\n' +
|
||||
' node scripts/gen-prompt-budget-parity-corpus.cjs --write\n'
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
|
||||
process.stdout.write(`ok ${path.relative(REPO_ROOT, OUT_PATH)}\n`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
process.exitCode = main(process.argv.slice(2));
|
||||
}
|
||||
|
||||
module.exports = { buildCases, capture, serialize, OUT_PATH, CORPUS_VERSION };
|
||||
@@ -116,6 +116,14 @@
|
||||
"host-integration-descriptors.test.cjs"
|
||||
],
|
||||
"issue": "1684"
|
||||
},
|
||||
"prompt-budget": {
|
||||
"files": [
|
||||
"prompt-budget-cli.test.cjs",
|
||||
"prompt-budget-parity.test.cjs",
|
||||
"prompt-budget.test.cjs"
|
||||
],
|
||||
"issue": "2929"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -130,6 +130,19 @@ const COVERED = {
|
||||
// minScore raised to TARGET (80) — module now meets ADR-456 goal.
|
||||
minScore: 80,
|
||||
},
|
||||
// context-composer: extracted from prompt-budget by #2929. Needs its own entry because
|
||||
// mutation coverage does not migrate with relocated code — scoring only prompt-budget.cjs
|
||||
// would leave the extracted ladder unmeasured.
|
||||
'context-composer': {
|
||||
cjs: 'gsd-core/bin/lib/context-composer.cjs',
|
||||
tests: [
|
||||
'tests/prompt-budget-parity.test.cjs',
|
||||
'tests/prompt-budget.unit.test.cjs',
|
||||
'tests/context-composer.test.cjs',
|
||||
'tests/context-composer.property.test.cjs',
|
||||
],
|
||||
minScore: 66,
|
||||
},
|
||||
'prompt-budget': {
|
||||
cjs: 'gsd-core/bin/lib/prompt-budget.cjs',
|
||||
tests: [
|
||||
|
||||
Reference in New Issue
Block a user