enhance(#2671): brand raw vs calibrated token types so double-application is a compile error (#2676)
* test(#2671): add failing-first brand-typing compile fixtures * feat(#2671): brand raw vs calibrated token types * refactor(#2671): hoist type-compile into a before() hook Two review responses: - The fixture compile ran in the describe() body, so it executed at collection time even when the block was filtered out, and a failed precondition collapsed eight independent assertions into one opaque describe-level failure. A before() hook is this repo's documented idiom and preserves per-test granularity. - parseTokensFlag now records WHY it returns an unbranded number: it validates the magnitude of --tokens, but the basis is decided by --calibrated, so branding here would be wrong for half its callers. The assertion belongs to cmdEstimateCheck, its only caller. * test(#2671): pin each brand diagnostic to its OFFENDING marker Adversarial review demonstrated that asserting only exactly-one-diagnostic- at-code-N is not airtight. Repairing a fixture's brand violation while injecting an unrelated error of the same code (a string passed as the budget argument) still yielded exactly one TS2345, so the fixture would have reported green while no longer testing its regression at all. Each bad-* fixture now routes its violating value through a const named OFFENDING, and the test asserts the diagnostic's start offset falls inside that node — located through the AST, so it survives reformatting and never pattern-matches source text. Replaying the proof-of-concept against the new assertion rejects it: the diagnostic lands on the budget literal, not the marker. Also corrects a doc comment that claimed the program type-checks all of src/; it covers phase-estimation.cts and its transitive dependencies. * chore(#2671): backfill changeset PR number (#2676)
This commit is contained in:
45
tests/fixtures/brand-typing/README.md
vendored
Normal file
45
tests/fixtures/brand-typing/README.md
vendored
Normal file
@@ -0,0 +1,45 @@
|
||||
# Brand-typing compile fixtures (#2671)
|
||||
|
||||
These `.cts` files are **compiler inputs, not runtime code**. They are deliberately
|
||||
excluded from `tsconfig.json` / `tsconfig.build.json` (both include `src/**/*.cts`
|
||||
only), so they never enter `npm run build:lib` and never emit a `.cjs`. `tests/` is
|
||||
not in the package's `files` list either, so they do not ship.
|
||||
|
||||
`tests/phase-estimation.test.cjs` compiles them in-process with the TypeScript
|
||||
compiler API, using the repo's real `tsconfig.build.json` options, and asserts on
|
||||
the returned **diagnostic objects** (`code`, `file`, `start`) — never on rendered
|
||||
compiler prose.
|
||||
|
||||
| Fixture | Must | Guards |
|
||||
|---|---|---|
|
||||
| `ok-correct-composition.cts` | compile clean | the positive control — proves the harness, imports, and option set are sound |
|
||||
| `bad-double-calibration.cts` | fail | #2631 — applying the factor to an already-corrected figure (factor²) |
|
||||
| `bad-raw-against-budget.cts` | fail | comparing an uncorrected projection against the smart-zone budget |
|
||||
| `bad-calibrated-as-sample-basis.cts` | fail | #2632 — calibrating against the emitted figure instead of the raw basis |
|
||||
| `bad-rebrand-calibrated-as-raw.cts` | fail | laundering a corrected figure back into the raw basis |
|
||||
| `bad-unbranded-number-as-raw.cts` | fail | proves the brand is not vacuously `number` |
|
||||
|
||||
## The three rules that keep these non-vacuous
|
||||
|
||||
A negative-compile test is worthless if it can pass for the wrong reason. Three
|
||||
independent checks prevent that, and each exists because of a demonstrated failure:
|
||||
|
||||
1. **The positive control must compile clean.** Otherwise a broken import path or
|
||||
an unusable option set would make every `bad-*` fixture "fail correctly" while
|
||||
testing nothing.
|
||||
2. **No diagnostic may originate outside this directory.** Otherwise a real compile
|
||||
error in `src/` could hide inside fixture noise.
|
||||
3. **Each `bad-*` fixture routes its violating value through a const named
|
||||
`OFFENDING`, and the diagnostic must land on that node.** Code-and-count alone
|
||||
is not enough — an adversarial review proved that a fixture whose brand
|
||||
violation had been *repaired*, but which gained an unrelated error of the same
|
||||
code (a string passed as the budget), still produced "exactly one TS2345" and
|
||||
would have reported green while no longer testing its regression at all.
|
||||
|
||||
Each `bad-*` fixture therefore contains **exactly one** deliberate type error, on
|
||||
its `OFFENDING` marker. Adding a second error, or moving the violation off the
|
||||
marker, breaks the contract on purpose — add a new fixture instead.
|
||||
|
||||
TypeScript reports an object-literal property mismatch on the property *name*
|
||||
rather than its initializer, so the marker check also accepts the name of a
|
||||
property initialized from `OFFENDING` (see `bad-calibrated-as-sample-basis.cts`).
|
||||
29
tests/fixtures/brand-typing/bad-calibrated-as-sample-basis.cts
vendored
Normal file
29
tests/fixtures/brand-typing/bad-calibrated-as-sample-basis.cts
vendored
Normal file
@@ -0,0 +1,29 @@
|
||||
/**
|
||||
* MUST NOT COMPILE (#2671) — the #2632 defect: the self-defeating loop.
|
||||
*
|
||||
* Calibration must measure actual/raw. Measuring actual/calibrated makes the
|
||||
* loop un-correct itself: once the correction works the observed ratio
|
||||
* approaches 1, which drags the median back toward 1. Simulated over 10 phases
|
||||
* against a true 2x miss it oscillates and settles near 1.41 instead of
|
||||
* converging on 2.0 (ADR-2629 Decision 4, amended #2632).
|
||||
*
|
||||
* `OFFENDING` is the marker the test pins the diagnostic to — see the README.
|
||||
* TypeScript reports an object-literal property mismatch on the property NAME,
|
||||
* so the test accepts that span too.
|
||||
*/
|
||||
|
||||
import estimation = require('../../../src/phase-estimation.cjs');
|
||||
|
||||
const estimate: estimation.PhaseEstimate = {
|
||||
tokens: estimation.asCalibratedTokens(100000),
|
||||
rawTokens: estimation.asRawTokens(50000),
|
||||
tasks: 5,
|
||||
confidence: 'med',
|
||||
};
|
||||
|
||||
const OFFENDING = estimate.tokens;
|
||||
|
||||
export const sample: estimation.CalibrationSample = {
|
||||
estimateTokens: OFFENDING,
|
||||
actualTokens: 74000,
|
||||
};
|
||||
17
tests/fixtures/brand-typing/bad-double-calibration.cts
vendored
Normal file
17
tests/fixtures/brand-typing/bad-double-calibration.cts
vendored
Normal file
@@ -0,0 +1,17 @@
|
||||
/**
|
||||
* MUST NOT COMPILE (#2671) — the #2631 defect in its pure form.
|
||||
*
|
||||
* `gsd-planner` emitted an already-calibrated figure; `gsd-plan-checker` fed it
|
||||
* to a parameter named `rawTokens`, which applied the factor a second time. The
|
||||
* effective correction became factor^2 — with the [0.5, 3.0] clamp, anywhere
|
||||
* from 4x under to 9x over — and it was invisible below 3 samples because there
|
||||
* factor === 1 and 1^2 === 1.
|
||||
*
|
||||
* `OFFENDING` is the marker the test pins the diagnostic to — see the README.
|
||||
*/
|
||||
|
||||
import estimation = require('../../../src/phase-estimation.cjs');
|
||||
|
||||
const OFFENDING = estimation.applyCalibration(estimation.asRawTokens(50000), 2);
|
||||
|
||||
export const squared = estimation.applyCalibration(OFFENDING, 2);
|
||||
16
tests/fixtures/brand-typing/bad-raw-against-budget.cts
vendored
Normal file
16
tests/fixtures/brand-typing/bad-raw-against-budget.cts
vendored
Normal file
@@ -0,0 +1,16 @@
|
||||
/**
|
||||
* MUST NOT COMPILE (#2671) — an uncorrected projection compared to the budget.
|
||||
*
|
||||
* The smart-zone verdict is only meaningful against the project's corrected
|
||||
* figure. Classifying the raw projection reports the estimator's bias as if it
|
||||
* were the phase's size, and silently under-reports every over-budget phase on
|
||||
* a project whose factor is above 1.
|
||||
*
|
||||
* `OFFENDING` is the marker the test pins the diagnostic to — see the README.
|
||||
*/
|
||||
|
||||
import estimation = require('../../../src/phase-estimation.cjs');
|
||||
|
||||
const OFFENDING = estimation.asRawTokens(50000);
|
||||
|
||||
export const verdict = estimation.classifyAgainstBudget(OFFENDING, 100000);
|
||||
20
tests/fixtures/brand-typing/bad-rebrand-calibrated-as-raw.cts
vendored
Normal file
20
tests/fixtures/brand-typing/bad-rebrand-calibrated-as-raw.cts
vendored
Normal file
@@ -0,0 +1,20 @@
|
||||
/**
|
||||
* MUST NOT COMPILE (#2671) — laundering a corrected figure back into the basis.
|
||||
*
|
||||
* Without this guard the brand would be decorative: `asRawTokens()` would accept
|
||||
* any number, so re-labelling a calibrated figure as raw would restore both
|
||||
* shipped defects through the front door.
|
||||
*
|
||||
* The one legitimate crossover — a pre-#2632 plan whose `tokens` IS the raw
|
||||
* projection because no factor had been applied yet — lives in
|
||||
* `calibrationBasis()` behind an explicit, commented assertion, so it stays a
|
||||
* single auditable line rather than an open door.
|
||||
*
|
||||
* `OFFENDING` is the marker the test pins the diagnostic to — see the README.
|
||||
*/
|
||||
|
||||
import estimation = require('../../../src/phase-estimation.cjs');
|
||||
|
||||
const OFFENDING = estimation.asCalibratedTokens(100000);
|
||||
|
||||
export const relabelled = estimation.asRawTokens(OFFENDING);
|
||||
16
tests/fixtures/brand-typing/bad-unbranded-number-as-raw.cts
vendored
Normal file
16
tests/fixtures/brand-typing/bad-unbranded-number-as-raw.cts
vendored
Normal file
@@ -0,0 +1,16 @@
|
||||
/**
|
||||
* MUST NOT COMPILE (#2671) — proves the brand is not vacuously `number`.
|
||||
*
|
||||
* If `RawTokens` were a plain alias for `number`, every other fixture here would
|
||||
* still compile and the whole guard would be theatre. A bare number carries no
|
||||
* basis, so it must pass through `asRawTokens` / `asCalibratedTokens` and the
|
||||
* caller must state which one it is.
|
||||
*
|
||||
* `OFFENDING` is the marker the test pins the diagnostic to — see the README.
|
||||
*/
|
||||
|
||||
import estimation = require('../../../src/phase-estimation.cjs');
|
||||
|
||||
const OFFENDING = 50000;
|
||||
|
||||
export const calibrated = estimation.applyCalibration(OFFENDING, 2);
|
||||
37
tests/fixtures/brand-typing/ok-correct-composition.cts
vendored
Normal file
37
tests/fixtures/brand-typing/ok-correct-composition.cts
vendored
Normal file
@@ -0,0 +1,37 @@
|
||||
/**
|
||||
* POSITIVE CONTROL (#2671) — the correct composition must compile clean.
|
||||
*
|
||||
* Without this, every `bad-*` fixture could be passing for the wrong reason (a
|
||||
* typo, a bad import path, a mis-built compiler option set) and the suite would
|
||||
* still be green. This file is what makes the negative fixtures non-vacuous.
|
||||
*/
|
||||
|
||||
import estimation = require('../../../src/phase-estimation.cjs');
|
||||
|
||||
const FACTOR = 2;
|
||||
const BUDGET = 100000;
|
||||
|
||||
// argv/disk figure -> caller asserts the basis -> correction -> budget verdict.
|
||||
const raw = estimation.asRawTokens(50000);
|
||||
const calibrated = estimation.applyCalibration(raw, FACTOR);
|
||||
export const verdict = estimation.classifyAgainstBudget(calibrated, BUDGET);
|
||||
|
||||
// `estimate-check --calibrated`: the caller states the factor is already applied,
|
||||
// so the figure goes straight to the budget without a second correction.
|
||||
export const preCalibratedVerdict = estimation.classifyAgainstBudget(
|
||||
estimation.asCalibratedTokens(50000),
|
||||
BUDGET,
|
||||
);
|
||||
|
||||
// ADR-2629 Decision 4: the calibration denominator is the RAW basis.
|
||||
const estimate: estimation.PhaseEstimate = {
|
||||
tokens: estimation.asCalibratedTokens(100000),
|
||||
rawTokens: estimation.asRawTokens(50000),
|
||||
tasks: 5,
|
||||
confidence: 'med',
|
||||
};
|
||||
|
||||
export const sample: estimation.CalibrationSample = {
|
||||
estimateTokens: estimation.calibrationBasis(estimate),
|
||||
actualTokens: 74000,
|
||||
};
|
||||
@@ -15,7 +15,7 @@
|
||||
* limit-1 / limit / limit+1 per RULESET.TESTS.boundary-coverage.fixtures.
|
||||
*/
|
||||
|
||||
const { describe, test } = require('node:test');
|
||||
const { describe, test, before } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
@@ -733,3 +733,156 @@ describe('estimate-check --calibrated', () => {
|
||||
assert.equal(pre.over_budget, false, 'the honest figure is under budget');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── branded raw-vs-calibrated basis (#2671) ───────────────────────────────
|
||||
|
||||
describe('RawTokens / CalibratedTokens brands', () => {
|
||||
// The behavioural guards above pin the two shipped defects (#2631 factor^2,
|
||||
// #2632 self-defeating loop) at the CLI surface. Both were composition errors
|
||||
// between individually-correct functions, and ~26,800 unit/boundary/property
|
||||
// tests were green for both. This block asserts the stronger property: with
|
||||
// the brands in place the wrong composition is not merely wrong, it is
|
||||
// UNREPRESENTABLE — `npm run build:lib` refuses it.
|
||||
//
|
||||
// The oracle is the TypeScript compiler, driven in-process through its API
|
||||
// (no subprocess, so no timeout and no spawn flake) against the repo's REAL
|
||||
// tsconfig.build.json options — the same strictness the publish build uses.
|
||||
// Assertions are on the returned diagnostic OBJECTS (`code`, `file`), never
|
||||
// on rendered compiler prose.
|
||||
const ts = require('typescript');
|
||||
|
||||
const REPO_ROOT = path.join(__dirname, '..');
|
||||
const FIXTURE_DIR = path.join(__dirname, 'fixtures', 'brand-typing');
|
||||
|
||||
/** TS "argument of type X is not assignable to parameter of type Y". */
|
||||
const TS_ARG_NOT_ASSIGNABLE = 2345;
|
||||
/** TS "type X is not assignable to type Y" (object-literal property). */
|
||||
const TS_TYPE_NOT_ASSIGNABLE = 2322;
|
||||
|
||||
const CASES = [
|
||||
{ fixture: 'ok-correct-composition.cts', expected: null },
|
||||
{ fixture: 'bad-double-calibration.cts', expected: TS_ARG_NOT_ASSIGNABLE },
|
||||
{ fixture: 'bad-raw-against-budget.cts', expected: TS_ARG_NOT_ASSIGNABLE },
|
||||
{ fixture: 'bad-calibrated-as-sample-basis.cts', expected: TS_TYPE_NOT_ASSIGNABLE },
|
||||
{ fixture: 'bad-rebrand-calibrated-as-raw.cts', expected: TS_ARG_NOT_ASSIGNABLE },
|
||||
{ fixture: 'bad-unbranded-number-as-raw.cts', expected: TS_ARG_NOT_ASSIGNABLE },
|
||||
];
|
||||
|
||||
/**
|
||||
* Every `bad-*` fixture routes its violating value through a const with this
|
||||
* name, and the test asserts the diagnostic lands ON that node.
|
||||
*
|
||||
* Code-and-count alone is NOT enough: a fixture that stops exercising its
|
||||
* brand violation but acquires an unrelated error of the same code still
|
||||
* yields "exactly one TS2345" and would report green while testing nothing.
|
||||
* That was demonstrated against an earlier version of this block, so the
|
||||
* position check is a regression guard, not a precaution.
|
||||
*/
|
||||
const OFFENDING = 'OFFENDING';
|
||||
|
||||
/**
|
||||
* Spans a diagnostic is allowed to occupy: any occurrence of the marker
|
||||
* identifier, plus — because TypeScript reports an object-literal property
|
||||
* mismatch on the property NAME rather than its initializer — the name of any
|
||||
* property initialized from the marker. Located through the AST, so this
|
||||
* survives reformatting and never pattern-matches source text.
|
||||
*/
|
||||
const markerSpans = (sourceFile) => {
|
||||
const spans = [];
|
||||
const visit = (node) => {
|
||||
if (ts.isIdentifier(node) && node.text === OFFENDING) {
|
||||
spans.push([node.getStart(sourceFile), node.getEnd()]);
|
||||
} else if (ts.isPropertyAssignment(node)
|
||||
&& ts.isIdentifier(node.initializer)
|
||||
&& node.initializer.text === OFFENDING) {
|
||||
spans.push([node.name.getStart(sourceFile), node.name.getEnd()]);
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
};
|
||||
visit(sourceFile);
|
||||
return spans;
|
||||
};
|
||||
|
||||
/**
|
||||
* Compile every fixture in ONE program and bucket the diagnostics by source
|
||||
* file. The program covers `phase-estimation.cts` and its transitive
|
||||
* dependencies — not all of `src/`, which `npm run build:lib` gates
|
||||
* separately — which is what makes the "no foreign diagnostics" assertion
|
||||
* below meaningful: anything outside the fixture directory is a real compile
|
||||
* error in the module under test.
|
||||
*/
|
||||
let byFixture;
|
||||
let foreign;
|
||||
let sourceFileOf;
|
||||
|
||||
before(() => {
|
||||
const configPath = path.join(REPO_ROOT, 'tsconfig.build.json');
|
||||
const readConfig = ts.readConfigFile(configPath, ts.sys.readFile);
|
||||
assert.equal(readConfig.error, undefined, 'tsconfig.build.json must parse');
|
||||
|
||||
const parsed = ts.parseJsonConfigFileContent(readConfig.config, ts.sys, REPO_ROOT);
|
||||
assert.deepEqual(parsed.errors, [], 'tsconfig.build.json must yield usable compiler options');
|
||||
|
||||
const options = {
|
||||
...parsed.options,
|
||||
// The fixtures live outside `src/`, so the emit-shaped settings have to go.
|
||||
// Everything that governs STRICTNESS is inherited untouched — that is the
|
||||
// whole point of reading the real config instead of hand-rolling options.
|
||||
noEmit: true,
|
||||
rootDir: undefined,
|
||||
outDir: undefined,
|
||||
incremental: false,
|
||||
tsBuildInfoFile: undefined,
|
||||
};
|
||||
|
||||
const roots = CASES.map((c) => path.join(FIXTURE_DIR, c.fixture));
|
||||
const program = ts.createProgram(roots, options);
|
||||
|
||||
byFixture = new Map(CASES.map((c) => [c.fixture, []]));
|
||||
foreign = [];
|
||||
sourceFileOf = new Map(
|
||||
CASES.map((c) => [c.fixture, program.getSourceFile(path.join(FIXTURE_DIR, c.fixture))]),
|
||||
);
|
||||
for (const diagnostic of ts.getPreEmitDiagnostics(program)) {
|
||||
const name = diagnostic.file === undefined ? null : path.basename(diagnostic.file.fileName);
|
||||
if (name !== null && byFixture.has(name)) byFixture.get(name).push(diagnostic);
|
||||
else foreign.push(diagnostic);
|
||||
}
|
||||
});
|
||||
|
||||
test('the module and its real build options compile clean', () => {
|
||||
// A diagnostic outside the fixture directory means `src/` itself is broken,
|
||||
// or the harness picked up the wrong options. Either way the negative cases
|
||||
// below would be passing for the wrong reason.
|
||||
assert.deepEqual(foreign.map((d) => d.code), [],
|
||||
'no diagnostic may originate outside tests/fixtures/brand-typing/');
|
||||
});
|
||||
|
||||
test('the correct composition compiles — the positive control', () => {
|
||||
// This is what makes every "must not compile" case non-vacuous: it proves
|
||||
// the fixture imports resolve and the option set is usable, so a diagnostic
|
||||
// in a bad-* fixture is the brand rejecting rather than a broken harness.
|
||||
assert.deepEqual(byFixture.get('ok-correct-composition.cts').map((d) => d.code), []);
|
||||
});
|
||||
|
||||
for (const { fixture, expected } of CASES.filter((c) => c.expected !== null)) {
|
||||
test(`${fixture} is a compile error on its ${OFFENDING} marker`, () => {
|
||||
const diagnostics = byFixture.get(fixture);
|
||||
assert.equal(diagnostics.length, 1,
|
||||
`${fixture} must produce exactly one diagnostic — see the fixture README`);
|
||||
assert.equal(diagnostics[0].code, expected);
|
||||
|
||||
// The diagnostic must land on the marker. Without this a fixture that
|
||||
// stopped exercising its brand violation, but gained an unrelated error
|
||||
// of the same code, would still pass.
|
||||
const spans = markerSpans(sourceFileOf.get(fixture));
|
||||
assert.ok(spans.length > 0, `${fixture} must declare a ${OFFENDING} marker`);
|
||||
const start = diagnostics[0].start;
|
||||
assert.ok(
|
||||
spans.some(([from, to]) => start >= from && start < to),
|
||||
`${fixture}: diagnostic at offset ${start} is not on the ${OFFENDING} marker `
|
||||
+ `(marker spans: ${JSON.stringify(spans)}) — the fixture is failing for the wrong reason`,
|
||||
);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user