enhance(#2671): brand raw vs calibrated token types so double-application is a compile error (#2676)

* test(#2671): add failing-first brand-typing compile fixtures

* feat(#2671): brand raw vs calibrated token types

* refactor(#2671): hoist type-compile into a before() hook

Two review responses:

- The fixture compile ran in the describe() body, so it executed at
  collection time even when the block was filtered out, and a failed
  precondition collapsed eight independent assertions into one opaque
  describe-level failure. A before() hook is this repo's documented
  idiom and preserves per-test granularity.

- parseTokensFlag now records WHY it returns an unbranded number: it
  validates the magnitude of --tokens, but the basis is decided by
  --calibrated, so branding here would be wrong for half its callers.
  The assertion belongs to cmdEstimateCheck, its only caller.

* test(#2671): pin each brand diagnostic to its OFFENDING marker

Adversarial review demonstrated that asserting only exactly-one-diagnostic-
at-code-N is not airtight. Repairing a fixture's brand violation while
injecting an unrelated error of the same code (a string passed as the
budget argument) still yielded exactly one TS2345, so the fixture would
have reported green while no longer testing its regression at all.

Each bad-* fixture now routes its violating value through a const named
OFFENDING, and the test asserts the diagnostic's start offset falls inside
that node — located through the AST, so it survives reformatting and never
pattern-matches source text. Replaying the proof-of-concept against the new
assertion rejects it: the diagnostic lands on the budget literal, not the
marker.

Also corrects a doc comment that claimed the program type-checks all of
src/; it covers phase-estimation.cts and its transitive dependencies.

* chore(#2671): backfill changeset PR number (#2676)
This commit is contained in:
Tom Boucher
2026-07-26 21:42:50 -04:00
committed by GitHub
parent 0d08c32048
commit a5633bb32f
13 changed files with 460 additions and 16 deletions

45
tests/fixtures/brand-typing/README.md vendored Normal file
View File

@@ -0,0 +1,45 @@
# Brand-typing compile fixtures (#2671)
These `.cts` files are **compiler inputs, not runtime code**. They are deliberately
excluded from `tsconfig.json` / `tsconfig.build.json` (both include `src/**/*.cts`
only), so they never enter `npm run build:lib` and never emit a `.cjs`. `tests/` is
not in the package's `files` list either, so they do not ship.
`tests/phase-estimation.test.cjs` compiles them in-process with the TypeScript
compiler API, using the repo's real `tsconfig.build.json` options, and asserts on
the returned **diagnostic objects** (`code`, `file`, `start`) — never on rendered
compiler prose.
| Fixture | Must | Guards |
|---|---|---|
| `ok-correct-composition.cts` | compile clean | the positive control — proves the harness, imports, and option set are sound |
| `bad-double-calibration.cts` | fail | #2631 — applying the factor to an already-corrected figure (factor²) |
| `bad-raw-against-budget.cts` | fail | comparing an uncorrected projection against the smart-zone budget |
| `bad-calibrated-as-sample-basis.cts` | fail | #2632 — calibrating against the emitted figure instead of the raw basis |
| `bad-rebrand-calibrated-as-raw.cts` | fail | laundering a corrected figure back into the raw basis |
| `bad-unbranded-number-as-raw.cts` | fail | proves the brand is not vacuously `number` |
## The three rules that keep these non-vacuous
A negative-compile test is worthless if it can pass for the wrong reason. Three
independent checks prevent that, and each exists because of a demonstrated failure:
1. **The positive control must compile clean.** Otherwise a broken import path or
an unusable option set would make every `bad-*` fixture "fail correctly" while
testing nothing.
2. **No diagnostic may originate outside this directory.** Otherwise a real compile
error in `src/` could hide inside fixture noise.
3. **Each `bad-*` fixture routes its violating value through a const named
`OFFENDING`, and the diagnostic must land on that node.** Code-and-count alone
is not enough — an adversarial review proved that a fixture whose brand
violation had been *repaired*, but which gained an unrelated error of the same
code (a string passed as the budget), still produced "exactly one TS2345" and
would have reported green while no longer testing its regression at all.
Each `bad-*` fixture therefore contains **exactly one** deliberate type error, on
its `OFFENDING` marker. Adding a second error, or moving the violation off the
marker, breaks the contract on purpose — add a new fixture instead.
TypeScript reports an object-literal property mismatch on the property *name*
rather than its initializer, so the marker check also accepts the name of a
property initialized from `OFFENDING` (see `bad-calibrated-as-sample-basis.cts`).

View File

@@ -0,0 +1,29 @@
/**
* MUST NOT COMPILE (#2671) — the #2632 defect: the self-defeating loop.
*
* Calibration must measure actual/raw. Measuring actual/calibrated makes the
* loop un-correct itself: once the correction works the observed ratio
* approaches 1, which drags the median back toward 1. Simulated over 10 phases
* against a true 2x miss it oscillates and settles near 1.41 instead of
* converging on 2.0 (ADR-2629 Decision 4, amended #2632).
*
* `OFFENDING` is the marker the test pins the diagnostic to — see the README.
* TypeScript reports an object-literal property mismatch on the property NAME,
* so the test accepts that span too.
*/
import estimation = require('../../../src/phase-estimation.cjs');
const estimate: estimation.PhaseEstimate = {
tokens: estimation.asCalibratedTokens(100000),
rawTokens: estimation.asRawTokens(50000),
tasks: 5,
confidence: 'med',
};
const OFFENDING = estimate.tokens;
export const sample: estimation.CalibrationSample = {
estimateTokens: OFFENDING,
actualTokens: 74000,
};

View File

@@ -0,0 +1,17 @@
/**
* MUST NOT COMPILE (#2671) — the #2631 defect in its pure form.
*
* `gsd-planner` emitted an already-calibrated figure; `gsd-plan-checker` fed it
* to a parameter named `rawTokens`, which applied the factor a second time. The
* effective correction became factor^2 — with the [0.5, 3.0] clamp, anywhere
* from 4x under to 9x over — and it was invisible below 3 samples because there
* factor === 1 and 1^2 === 1.
*
* `OFFENDING` is the marker the test pins the diagnostic to — see the README.
*/
import estimation = require('../../../src/phase-estimation.cjs');
const OFFENDING = estimation.applyCalibration(estimation.asRawTokens(50000), 2);
export const squared = estimation.applyCalibration(OFFENDING, 2);

View File

@@ -0,0 +1,16 @@
/**
* MUST NOT COMPILE (#2671) — an uncorrected projection compared to the budget.
*
* The smart-zone verdict is only meaningful against the project's corrected
* figure. Classifying the raw projection reports the estimator's bias as if it
* were the phase's size, and silently under-reports every over-budget phase on
* a project whose factor is above 1.
*
* `OFFENDING` is the marker the test pins the diagnostic to — see the README.
*/
import estimation = require('../../../src/phase-estimation.cjs');
const OFFENDING = estimation.asRawTokens(50000);
export const verdict = estimation.classifyAgainstBudget(OFFENDING, 100000);

View File

@@ -0,0 +1,20 @@
/**
* MUST NOT COMPILE (#2671) — laundering a corrected figure back into the basis.
*
* Without this guard the brand would be decorative: `asRawTokens()` would accept
* any number, so re-labelling a calibrated figure as raw would restore both
* shipped defects through the front door.
*
* The one legitimate crossover — a pre-#2632 plan whose `tokens` IS the raw
* projection because no factor had been applied yet — lives in
* `calibrationBasis()` behind an explicit, commented assertion, so it stays a
* single auditable line rather than an open door.
*
* `OFFENDING` is the marker the test pins the diagnostic to — see the README.
*/
import estimation = require('../../../src/phase-estimation.cjs');
const OFFENDING = estimation.asCalibratedTokens(100000);
export const relabelled = estimation.asRawTokens(OFFENDING);

View File

@@ -0,0 +1,16 @@
/**
* MUST NOT COMPILE (#2671) — proves the brand is not vacuously `number`.
*
* If `RawTokens` were a plain alias for `number`, every other fixture here would
* still compile and the whole guard would be theatre. A bare number carries no
* basis, so it must pass through `asRawTokens` / `asCalibratedTokens` and the
* caller must state which one it is.
*
* `OFFENDING` is the marker the test pins the diagnostic to — see the README.
*/
import estimation = require('../../../src/phase-estimation.cjs');
const OFFENDING = 50000;
export const calibrated = estimation.applyCalibration(OFFENDING, 2);

View File

@@ -0,0 +1,37 @@
/**
* POSITIVE CONTROL (#2671) — the correct composition must compile clean.
*
* Without this, every `bad-*` fixture could be passing for the wrong reason (a
* typo, a bad import path, a mis-built compiler option set) and the suite would
* still be green. This file is what makes the negative fixtures non-vacuous.
*/
import estimation = require('../../../src/phase-estimation.cjs');
const FACTOR = 2;
const BUDGET = 100000;
// argv/disk figure -> caller asserts the basis -> correction -> budget verdict.
const raw = estimation.asRawTokens(50000);
const calibrated = estimation.applyCalibration(raw, FACTOR);
export const verdict = estimation.classifyAgainstBudget(calibrated, BUDGET);
// `estimate-check --calibrated`: the caller states the factor is already applied,
// so the figure goes straight to the budget without a second correction.
export const preCalibratedVerdict = estimation.classifyAgainstBudget(
estimation.asCalibratedTokens(50000),
BUDGET,
);
// ADR-2629 Decision 4: the calibration denominator is the RAW basis.
const estimate: estimation.PhaseEstimate = {
tokens: estimation.asCalibratedTokens(100000),
rawTokens: estimation.asRawTokens(50000),
tasks: 5,
confidence: 'med',
};
export const sample: estimation.CalibrationSample = {
estimateTokens: estimation.calibrationBasis(estimate),
actualTokens: 74000,
};

View File

@@ -15,7 +15,7 @@
* limit-1 / limit / limit+1 per RULESET.TESTS.boundary-coverage.fixtures.
*/
const { describe, test } = require('node:test');
const { describe, test, before } = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
@@ -733,3 +733,156 @@ describe('estimate-check --calibrated', () => {
assert.equal(pre.over_budget, false, 'the honest figure is under budget');
});
});
// ─── branded raw-vs-calibrated basis (#2671) ───────────────────────────────
describe('RawTokens / CalibratedTokens brands', () => {
// The behavioural guards above pin the two shipped defects (#2631 factor^2,
// #2632 self-defeating loop) at the CLI surface. Both were composition errors
// between individually-correct functions, and ~26,800 unit/boundary/property
// tests were green for both. This block asserts the stronger property: with
// the brands in place the wrong composition is not merely wrong, it is
// UNREPRESENTABLE — `npm run build:lib` refuses it.
//
// The oracle is the TypeScript compiler, driven in-process through its API
// (no subprocess, so no timeout and no spawn flake) against the repo's REAL
// tsconfig.build.json options — the same strictness the publish build uses.
// Assertions are on the returned diagnostic OBJECTS (`code`, `file`), never
// on rendered compiler prose.
const ts = require('typescript');
const REPO_ROOT = path.join(__dirname, '..');
const FIXTURE_DIR = path.join(__dirname, 'fixtures', 'brand-typing');
/** TS "argument of type X is not assignable to parameter of type Y". */
const TS_ARG_NOT_ASSIGNABLE = 2345;
/** TS "type X is not assignable to type Y" (object-literal property). */
const TS_TYPE_NOT_ASSIGNABLE = 2322;
const CASES = [
{ fixture: 'ok-correct-composition.cts', expected: null },
{ fixture: 'bad-double-calibration.cts', expected: TS_ARG_NOT_ASSIGNABLE },
{ fixture: 'bad-raw-against-budget.cts', expected: TS_ARG_NOT_ASSIGNABLE },
{ fixture: 'bad-calibrated-as-sample-basis.cts', expected: TS_TYPE_NOT_ASSIGNABLE },
{ fixture: 'bad-rebrand-calibrated-as-raw.cts', expected: TS_ARG_NOT_ASSIGNABLE },
{ fixture: 'bad-unbranded-number-as-raw.cts', expected: TS_ARG_NOT_ASSIGNABLE },
];
/**
* Every `bad-*` fixture routes its violating value through a const with this
* name, and the test asserts the diagnostic lands ON that node.
*
* Code-and-count alone is NOT enough: a fixture that stops exercising its
* brand violation but acquires an unrelated error of the same code still
* yields "exactly one TS2345" and would report green while testing nothing.
* That was demonstrated against an earlier version of this block, so the
* position check is a regression guard, not a precaution.
*/
const OFFENDING = 'OFFENDING';
/**
* Spans a diagnostic is allowed to occupy: any occurrence of the marker
* identifier, plus — because TypeScript reports an object-literal property
* mismatch on the property NAME rather than its initializer — the name of any
* property initialized from the marker. Located through the AST, so this
* survives reformatting and never pattern-matches source text.
*/
const markerSpans = (sourceFile) => {
const spans = [];
const visit = (node) => {
if (ts.isIdentifier(node) && node.text === OFFENDING) {
spans.push([node.getStart(sourceFile), node.getEnd()]);
} else if (ts.isPropertyAssignment(node)
&& ts.isIdentifier(node.initializer)
&& node.initializer.text === OFFENDING) {
spans.push([node.name.getStart(sourceFile), node.name.getEnd()]);
}
ts.forEachChild(node, visit);
};
visit(sourceFile);
return spans;
};
/**
* Compile every fixture in ONE program and bucket the diagnostics by source
* file. The program covers `phase-estimation.cts` and its transitive
* dependencies — not all of `src/`, which `npm run build:lib` gates
* separately — which is what makes the "no foreign diagnostics" assertion
* below meaningful: anything outside the fixture directory is a real compile
* error in the module under test.
*/
let byFixture;
let foreign;
let sourceFileOf;
before(() => {
const configPath = path.join(REPO_ROOT, 'tsconfig.build.json');
const readConfig = ts.readConfigFile(configPath, ts.sys.readFile);
assert.equal(readConfig.error, undefined, 'tsconfig.build.json must parse');
const parsed = ts.parseJsonConfigFileContent(readConfig.config, ts.sys, REPO_ROOT);
assert.deepEqual(parsed.errors, [], 'tsconfig.build.json must yield usable compiler options');
const options = {
...parsed.options,
// The fixtures live outside `src/`, so the emit-shaped settings have to go.
// Everything that governs STRICTNESS is inherited untouched — that is the
// whole point of reading the real config instead of hand-rolling options.
noEmit: true,
rootDir: undefined,
outDir: undefined,
incremental: false,
tsBuildInfoFile: undefined,
};
const roots = CASES.map((c) => path.join(FIXTURE_DIR, c.fixture));
const program = ts.createProgram(roots, options);
byFixture = new Map(CASES.map((c) => [c.fixture, []]));
foreign = [];
sourceFileOf = new Map(
CASES.map((c) => [c.fixture, program.getSourceFile(path.join(FIXTURE_DIR, c.fixture))]),
);
for (const diagnostic of ts.getPreEmitDiagnostics(program)) {
const name = diagnostic.file === undefined ? null : path.basename(diagnostic.file.fileName);
if (name !== null && byFixture.has(name)) byFixture.get(name).push(diagnostic);
else foreign.push(diagnostic);
}
});
test('the module and its real build options compile clean', () => {
// A diagnostic outside the fixture directory means `src/` itself is broken,
// or the harness picked up the wrong options. Either way the negative cases
// below would be passing for the wrong reason.
assert.deepEqual(foreign.map((d) => d.code), [],
'no diagnostic may originate outside tests/fixtures/brand-typing/');
});
test('the correct composition compiles — the positive control', () => {
// This is what makes every "must not compile" case non-vacuous: it proves
// the fixture imports resolve and the option set is usable, so a diagnostic
// in a bad-* fixture is the brand rejecting rather than a broken harness.
assert.deepEqual(byFixture.get('ok-correct-composition.cts').map((d) => d.code), []);
});
for (const { fixture, expected } of CASES.filter((c) => c.expected !== null)) {
test(`${fixture} is a compile error on its ${OFFENDING} marker`, () => {
const diagnostics = byFixture.get(fixture);
assert.equal(diagnostics.length, 1,
`${fixture} must produce exactly one diagnostic — see the fixture README`);
assert.equal(diagnostics[0].code, expected);
// The diagnostic must land on the marker. Without this a fixture that
// stopped exercising its brand violation, but gained an unrelated error
// of the same code, would still pass.
const spans = markerSpans(sourceFileOf.get(fixture));
assert.ok(spans.length > 0, `${fixture} must declare a ${OFFENDING} marker`);
const start = diagnostics[0].start;
assert.ok(
spans.some(([from, to]) => start >= from && start < to),
`${fixture}: diagnostic at offset ${start} is not on the ${OFFENDING} marker `
+ `(marker spans: ${JSON.stringify(spans)}) — the fixture is failing for the wrong reason`,
);
});
}
});