chore(#597): replace count-based ratchet guards with AST lint + named-set allowlists (#603)

The windows-test-parity ratchet greps test source for fs.rmSync-without-
maxRetries (and six other Windows-portability anti-patterns), failing when an
integer offender COUNT exceeds a frozen baseline (rmSync: 95). A count ratchet
is a Goodhart metric: fixing one offender and adding another keeps the count
constant, so a new defect slips through green. Replace it — and every other
count ratchet in the repo — with a layered, masking-proof design.

Behavioral seam test
- tests/helpers-cleanup.test.cjs proves helpers.cleanup() carries the Windows
  EBUSY retry budget. cleanup() delegates retries to Node's fs.rmSync via
  maxRetries (it owns no loop), so the test asserts the option contract
  (recursive/force/maxRetries>0/retryDelay>0) + real-FS removal + the cwd-guard,
  rather than a loop that does not exist. The EBUSY risk is now tested ONCE at
  the helper, not approximated textually at every call site.

Write-time ESLint rule (AST-accurate, replaces the grep)
- eslint-rules/no-raw-rmsync-in-tests.cjs (error in tests/**/*.test.cjs) bans
  raw fs.rmSync, steering to cleanup(). Catches member, computed (fs['rmSync']),
  destructured and aliased forms; escape hatch is inline
  `// eslint-disable-next-line local/no-raw-rmsync-in-tests -- <reason>` only.
- Migrated 336 raw fs.rmSync teardown calls across ~116 test files to cleanup().
  ~18 genuinely load-bearing sites (mid-test SUT/fault-injection removals,
  error-swallowing or name-colliding local teardown helpers) keep the raw call
  with an inline eslint-disable + reason.

Shared anti-ratchet primitive
- scripts/lib/allowlist-ratchet.cjs:
  - assertWithinAllowlist: fails on NOVEL ids (new offender introduced) AND on
    STALE ids (a known offender was fixed but not pruned) — identity, not count,
    and a ratchet DOWN toward zero.
  - assertTightCeiling: a size/length budget whose ceiling must stay within a
    grace band of the high-water mark, so budgets may only tighten, never creep.

Ratchets converted onto the primitive
- windows-test-parity-guard.test.cjs: rmSync rule deleted (now ESLint-enforced);
  the remaining six patterns moved from integer baselines to named-set
  allowlists with ratchet-down.
- scripts/lint-test-file-count.{cjs,allowlist.json}: per-module integer counts →
  named filename sets (closes the swap-a-file-keep-the-count blind spot); a
  module dropping under cap now FAILS to force pruning its allowlist entry.
- enh-2790 skill-count `<= 63` → named skill allowlist (ratchets toward ~58).

Size budgets hardened (tighten-only)
- agent-size / workflow-size / feat-3039 help-tiered: ceilings lowered to the
  current high-water mark and an assertTightCeiling anti-creep check added per
  tier. Fixed external-contract limits (description ≤100 chars, agent ≤100 KB)
  are intentionally left as-is — they are not grandfathered creeping budgets.

No user-facing behavior change (tests + tooling only); no USER_FACING_PREFIXES
touched, so no changeset fragment is required.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-06-01 22:43:49 -04:00
committed by GitHub
parent a594f5175c
commit a28dcec981
130 changed files with 2030 additions and 586 deletions

View File

@@ -0,0 +1,136 @@
'use strict';
/**
* @file allowlist-ratchet.cjs
*
* Reusable "better than a count ratchet" primitives for CI guards.
*
* ## Motivation (issue #597)
*
* A count ratchet (`assert(offenders.length <= N)`) has a masking blind spot:
* fixing one offender and introducing a new one keeps the count constant, so a
* novel defect slips through green. These helpers enforce on IDENTITY instead,
* making every individual offender visible and requiring monotonic progress
* toward zero.
*
* ## Design
*
* Both functions are pure (no I/O, no global state). The `fail` callback is
* injected by the caller so the same logic can be used with `node:assert.fail`,
* a custom throw, or a message-collector in unit tests.
*/
/**
* Assert that `current` offenders are all within the known allowlist, and that
* every entry in the allowlist still offends (forcing the allowlist to shrink
* as defects are fixed).
*
* Fails when:
* - Any id in `current` is NOT in `known` → novel offender introduced.
* - Any id in `known` is NOT in `current` → stale allowlist entry must be
* pruned so the guard ratchets toward zero (the ratchet-DOWN direction).
*
* ## Masking blind spot this prevents (issue #597)
*
* A count ratchet (`assert(count <= N)`) allows one offender to be silently
* replaced by another while the count stays at N. By asserting on identity
* instead, every new offender is caught by name, and every fixed offender
* forces the allowlist to shrink.
*
* @param {object} opts
* @param {string} opts.label - Human-readable name for the guard (used
* in failure messages).
* @param {Iterable<string>} opts.current - The offending ids found in the
* current run.
* @param {Iterable<string>} opts.known - The allowlisted ids (baseline).
* @param {function(string): void} opts.fail - Callback invoked with a
* descriptive message on any violation.
* Pass `require('node:assert').fail`, a
* custom thrower, or a collector. The
* function is NOT imported here so callers
* control the failure mode.
* @param {string} [opts.pruneHint] - Optional hint appended to the stale-
* entry failure message (e.g. the name of
* the allowlist file to edit).
* @returns {{ novel: string[], stale: string[] }} Sorted arrays of novel ids
* (in current but not known) and stale ids (in known but not current).
*/
function assertWithinAllowlist({ label, current, known, fail, pruneHint }) {
const currentSet = new Set(current);
const knownSet = new Set(known);
const novel = [...currentSet].filter((id) => !knownSet.has(id)).sort();
const stale = [...knownSet].filter((id) => !currentSet.has(id)).sort();
if (novel.length > 0) {
const list = novel.map((id) => ` - ${id}`).join('\n');
fail(
`[${label}] ${novel.length} NEW offender(s) introduced — fix at the source; do not just add to the allowlist.\n${list}`
);
}
if (stale.length > 0) {
const list = stale.map((id) => ` - ${id}`).join('\n');
const hint = pruneHint ? `\n(${pruneHint})` : '';
fail(
`[${label}] ${stale.length} allowlisted id(s) no longer offend and MUST be pruned so the guard ratchets toward zero.${hint}\n${list}`
);
}
return { novel, stale };
}
/**
* Assert that an artifact's measured maximum stays within a declared ceiling,
* and that the ceiling itself does not creep above the high-water mark (budgets
* may only decrease, not increase over time).
*
* Fails when:
* - `actualMax > ceiling` → regression: artifact exceeds budget.
* - `ceiling - actualMax > grace` → ceiling sits too far above the measured
* value; tighten it toward `actualMax`.
*
* ## Masking blind spot this prevents (issue #597)
*
* A plain `assert(size <= ceiling)` with a ceiling set generously high allows
* the artifact to grow unchecked as long as it stays under the ceiling. The
* `grace` band forces the ceiling to stay close to the high-water mark,
* ensuring that any upward creep is immediately visible.
*
* @param {object} opts
* @param {string} opts.label - Human-readable name for the guard (used
* in failure messages).
* @param {number} opts.actualMax - The measured value (e.g. bundle size in
* bytes, line count).
* @param {number} opts.ceiling - The declared budget ceiling.
* @param {number} opts.grace - Maximum allowed slack (`ceiling -
* actualMax`) before the ceiling is
* considered too loose.
* @param {function(string): void} opts.fail - Callback invoked with a
* descriptive message on any violation.
* @returns {{ ok: boolean, slack: number }} Whether both checks passed and the
* current slack value.
*/
function assertTightCeiling({ label, actualMax, ceiling, grace, fail }) {
const slack = ceiling - actualMax;
let ok = true;
if (actualMax > ceiling) {
ok = false;
fail(
`[${label}] Regression: artifact value ${actualMax} exceeds budget ceiling ${ceiling}. ` +
`Raise the ceiling to at most ${actualMax} only if the increase is justified.`
);
} else if (slack > grace) {
ok = false;
fail(
`[${label}] Ceiling ${ceiling} sits too far above the high-water mark ${actualMax} ` +
`(slack ${slack} > grace ${grace}). Tighten the ceiling toward ${actualMax}. ` +
`Budgets may only decrease.`
);
}
return { ok, slack };
}
module.exports = { assertWithinAllowlist, assertTightCeiling };

View File

@@ -1,36 +1,131 @@
{
"_doc": "Baseline of modules currently exceeding the 2-test-file limit. Each entry locks in TODAY's count as the ceiling. Reductions are ratcheted automatically — when a cluster drops to ≤ 2, remove its entry. New entries require justification in PR description.",
"_doc": "Baseline of modules currently exceeding the 2-test-file limit. Each entry locks in TODAY's exact test filenames as the allowlisted set (identity ratchet). Adding a NEW test file to a capped module fails (novel). Removing one requires pruning this list (stale, ratchet-down). When a cluster drops to ≤ 2, remove its entry entirely. New entries require justification in PR description.",
"modules": {
"phase": { "current": 5, "issue": "3788" },
"worktree": { "current": 13, "issue": "TBD" },
"milestone": { "current": 10, "issue": "TBD" },
"roadmap": { "current": 9, "issue": "TBD" },
"verify": { "current": 10, "issue": "3767" },
"install": { "current": 9, "issue": "TBD" },
"init": { "current": 8, "issue": "TBD" },
"state": { "current": 11, "issue": "180" },
"config": { "current": 8, "issue": "TBD" },
"graphify": { "current": 7, "issue": "TBD" },
"progress": { "current": 6, "issue": "14" },
"cli": { "current": 5, "issue": "TBD" },
"surface": { "current": 5, "issue": "TBD" },
"commit": { "current": 4, "issue": "TBD" },
"frontmatter": { "current": 4, "issue": "TBD" },
"index": { "current": 5, "issue": "180" },
"intel": { "current": 4, "issue": "TBD" },
"mvp": { "current": 4, "issue": "TBD" },
"install-profiles": { "current": 4, "issue": "TBD" },
"audit": { "current": 4, "issue": "192" },
"audit-open": { "current": 3, "issue": "TBD" },
"config-schema": { "current": 3, "issue": "TBD" },
"profile": { "current": 3, "issue": "TBD" },
"prompt-budget": { "current": 3, "issue": "TBD" },
"uat": { "current": 3, "issue": "TBD" },
"validate": { "current": 3, "issue": "TBD" },
"workstream": { "current": 3, "issue": "TBD" },
"gsd-tools": { "current": 3, "issue": "TBD" },
"runtime-artifact-layout":{ "current": 3, "issue": "TBD" },
"security": { "current": 3, "issue": "TBD" },
"gsd-sdk": { "current": 3, "issue": "TBD" }
"audit": {
"files": [
"audit-fix-command.test.cjs",
"bug-2659-audit-open-crash.test.cjs",
"bug-2836-audit-open-summary-uat-drift.test.cjs",
"bug-2911-audit-open-output-shape.test.cjs"
],
"issue": "192"
},
"config": {
"files": [
"bug-2943-config-get-context-window-default.test.cjs",
"bug-321-config-defaults-clone-strategy.test.cjs",
"bug-3227-config-set-model-overrides.test.cjs",
"bug-442-config-dir-equals-in-path.test.cjs",
"config-field-docs.test.cjs",
"config-get-default.test.cjs",
"config-schema.property.test.cjs",
"config.test.cjs"
],
"issue": "TBD"
},
"graphify": {
"files": [
"graphify-auto-update.test.cjs",
"graphify-query.test.cjs",
"graphify-visualization.test.cjs",
"graphify.test.cjs"
],
"issue": "TBD"
},
"intel": {
"files": [
"bug-2351-intel-kilo-layout.test.cjs",
"bug-3290-intel-updater-layout-block.test.cjs",
"intel.test.cjs"
],
"issue": "TBD"
},
"milestone": {
"files": [
"milestone-archive.test.cjs",
"milestone-helper.test.cjs",
"milestone-prefixed-convention.test.cjs",
"milestone-summary.test.cjs",
"milestone.test.cjs"
],
"issue": "TBD"
},
"phase": {
"files": [
"bug-214-phase-researcher-write-truncation-contract.test.cjs",
"phase-dependency-levels.test.cjs",
"phase.test.cjs"
],
"issue": "597"
},
"roadmap": {
"files": [
"bug-2661-roadmap-sync-parallel.test.cjs",
"bug-3128-roadmap-plan-count-slug-layout.test.cjs",
"bug-3599-roadmap-get-phase-project-code-prefix.test.cjs",
"enh-2447-roadmap-wave-deps.test.cjs",
"roadmap-mode-field.test.cjs",
"roadmap-phase-fallback.test.cjs",
"roadmap.test.cjs"
],
"issue": "TBD"
},
"runtime-artifact-layout": {
"files": [
"runtime-artifact-layout-install-profiles.test.cjs",
"runtime-artifact-layout-surface.test.cjs",
"runtime-artifact-layout.test.cjs"
],
"issue": "TBD"
},
"security": {
"files": [
"security-prompt-injection.test.cjs",
"security-scan.test.cjs",
"security.test.cjs"
],
"issue": "TBD"
},
"state": {
"files": [
"bug-21-state-md-template-frontmatter.test.cjs",
"bug-2630-state-frontmatter-milestone-switch.test.cjs",
"bug-3127-state-begin-phase-idempotent.test.cjs",
"bug-3242-state-update-progress-trample.test.cjs",
"bug-3286-state-write-routing.test.cjs",
"bug-3454-state-dollar-backreference-growth.test.cjs",
"bug-397-state-preserve-executor-authored.test.cjs",
"state-acquirestatelock-non-eexist.test.cjs",
"state-prune.test.cjs",
"state.test.cjs"
],
"issue": "180"
},
"verify": {
"files": [
"bug-2969-verify-reapply-patches.test.cjs",
"bug-2994-verify-reapply-patches-installed-path.test.cjs",
"bug-3381-verify-work-workstream.test.cjs",
"bug-3657-verify-reapply-patches-pristine-drift.test.cjs",
"verify-health.test.cjs",
"verify-mvp-uat.test.cjs",
"verify-test-quality.test.cjs",
"verify-work-auto-transition.test.cjs",
"verify.test.cjs"
],
"issue": "3767"
},
"install": {
"files": [
"bug-410-install-defaults-test-mode-guard.test.cjs",
"install-minimal-hooks.test.cjs",
"install-path-detection.test.cjs",
"install-regressions.test.cjs",
"install-runtime-artifacts.test.cjs",
"install-update-marker.test.cjs",
"install.test.cjs"
],
"issue": "TBD"
}
}
}

View File

@@ -4,16 +4,19 @@
*
* Scans sdk/src/query/, sdk/src/, get-shit-done/bin/lib/, bin/ for production
* modules, then counts matching test files in tests/ and sdk/src (recursive). Cap is 2
* (primary + one integration). Over-limit clusters must be in the allowlist at
* their frozen count (ratchet: may only decrease). --json emits structured output.
* (primary + one integration). Over-limit clusters must be in the allowlist with the
* EXACT set of test filenames grandfathered (identity ratchet via allowlist-ratchet.cjs).
* Adding a new test file to a capped module is a novel offender; removing one requires
* pruning the allowlist entry (stale). --json emits structured output.
*
* Verdicts: OK_UNDER_LIMIT | OK_IN_ALLOWLIST | FAIL_EXCEEDS_LIMIT |
* FAIL_EXCEEDS_ALLOWLIST | HINT_CAN_REMOVE_FROM_ALLOWLIST
* FAIL_NOVEL_FILES | FAIL_STALE_ALLOWLIST
*/
'use strict';
const fs = require('fs');
const path = require('path');
const { assertWithinAllowlist } = require('./lib/allowlist-ratchet.cjs');
const ROOT = path.join(__dirname, '..');
const PROD_DIRS = [
@@ -30,11 +33,11 @@ const ALLOWLIST_PATH = path.join(__dirname, 'lint-test-file-count.allowlist.json
const MAX_FILES = 2;
const Verdict = Object.freeze({
OK_UNDER_LIMIT: 'OK_UNDER_LIMIT',
OK_IN_ALLOWLIST: 'OK_IN_ALLOWLIST',
FAIL_EXCEEDS_LIMIT: 'FAIL_EXCEEDS_LIMIT',
FAIL_EXCEEDS_ALLOWLIST: 'FAIL_EXCEEDS_ALLOWLIST',
HINT_CAN_REMOVE_FROM_ALLOWLIST: 'HINT_CAN_REMOVE_FROM_ALLOWLIST',
OK_UNDER_LIMIT: 'OK_UNDER_LIMIT',
OK_IN_ALLOWLIST: 'OK_IN_ALLOWLIST',
FAIL_EXCEEDS_LIMIT: 'FAIL_EXCEEDS_LIMIT',
FAIL_NOVEL_FILES: 'FAIL_NOVEL_FILES',
FAIL_STALE_ALLOWLIST: 'FAIL_STALE_ALLOWLIST',
});
function isTestFile(name) {
@@ -120,17 +123,62 @@ function loadAllowlist() {
catch (_) { return {}; }
}
/**
* Evaluate one module's test files against the allowlist.
*
* For modules under the default cap (≤ MAX_FILES): simple count check.
* For modules with an allowlist entry: identity check via assertWithinAllowlist.
* - count now ≤ MAX_FILES → FAIL_STALE_ALLOWLIST (all known files are stale; prune entry)
* - novel files (in current but not in known) → FAIL_NOVEL_FILES
* - stale files (in known but not in current) → FAIL_STALE_ALLOWLIST
* - exact match → OK_IN_ALLOWLIST
* For modules over the cap with no allowlist entry: FAIL_EXCEEDS_LIMIT.
*
* Returns { verdict, prefix, count, knownFiles, novel, stale, files }
*/
function evaluateLint({ prefix, testFiles, allowlist }) {
const count = testFiles.length;
const entry = allowlist[prefix];
const ceiling = entry ? entry.current : null;
const currentNames = testFiles.map(f => path.basename(f));
if (entry !== undefined) {
if (count <= MAX_FILES) return { verdict: Verdict.HINT_CAN_REMOVE_FROM_ALLOWLIST, prefix, count, ceiling, files: testFiles };
if (count <= ceiling) return { verdict: Verdict.OK_IN_ALLOWLIST, prefix, count, ceiling, files: testFiles };
return { verdict: Verdict.FAIL_EXCEEDS_ALLOWLIST, prefix, count, ceiling, files: testFiles };
const knownFiles = Array.isArray(entry.files) ? entry.files : [];
// If the module is now at or under the default cap, the entire allowlist entry is
// stale and must be removed — this is a ratchet-DOWN failure, not a hint.
// All known files are stale (the whole entry can go).
if (count <= MAX_FILES) {
return {
verdict: Verdict.FAIL_STALE_ALLOWLIST,
prefix, count,
knownFiles,
novel: [],
stale: knownFiles.slice().sort(),
files: testFiles,
};
}
// Identity check via assertWithinAllowlist
const messages = [];
const { novel, stale } = assertWithinAllowlist({
label: prefix,
current: currentNames,
known: knownFiles,
fail: (msg) => messages.push(msg),
pruneHint: 'scripts/lint-test-file-count.allowlist.json',
});
if (novel.length > 0) {
return { verdict: Verdict.FAIL_NOVEL_FILES, prefix, count, knownFiles, novel, stale, files: testFiles };
}
if (stale.length > 0) {
return { verdict: Verdict.FAIL_STALE_ALLOWLIST, prefix, count, knownFiles, novel, stale, files: testFiles };
}
return { verdict: Verdict.OK_IN_ALLOWLIST, prefix, count, knownFiles, novel: [], stale: [], files: testFiles };
}
if (count <= MAX_FILES) return { verdict: Verdict.OK_UNDER_LIMIT, prefix, count, ceiling: null, files: testFiles };
return { verdict: Verdict.FAIL_EXCEEDS_LIMIT, prefix, count, ceiling: null, files: testFiles };
if (count <= MAX_FILES) {
return { verdict: Verdict.OK_UNDER_LIMIT, prefix, count, knownFiles: null, novel: [], stale: [], files: testFiles };
}
return { verdict: Verdict.FAIL_EXCEEDS_LIMIT, prefix, count, knownFiles: null, novel: [], stale: [], files: testFiles };
}
function run() {
@@ -147,35 +195,42 @@ function run() {
}
const failures = results.filter(r =>
r.verdict === Verdict.FAIL_EXCEEDS_LIMIT || r.verdict === Verdict.FAIL_EXCEEDS_ALLOWLIST);
const hints = results.filter(r => r.verdict === Verdict.HINT_CAN_REMOVE_FROM_ALLOWLIST);
r.verdict === Verdict.FAIL_EXCEEDS_LIMIT ||
r.verdict === Verdict.FAIL_NOVEL_FILES ||
r.verdict === Verdict.FAIL_STALE_ALLOWLIST);
if (jsonMode) {
console.log(JSON.stringify({ ok: failures.length === 0, results, failures, hints }, null, 2));
console.log(JSON.stringify({ ok: failures.length === 0, results, failures, hints: [] }, null, 2));
process.exit(failures.length > 0 ? 1 : 0);
}
if (failures.length === 0) {
const inAllowlist = results.filter(r => r.verdict === Verdict.OK_IN_ALLOWLIST).length;
console.log(`ok lint-test-file-count: ${results.length} module(s) checked, 0 failures` +
(inAllowlist > 0 ? `, ${inAllowlist} allowlisted` : '') +
(hints.length > 0 ? `, ${hints.length} hint(s)` : ''));
for (const h of hints) {
console.log(` hint: "${h.prefix}" is allowlisted at ${h.ceiling} but now has ${h.count} — remove from allowlist`);
}
(inAllowlist > 0 ? `, ${inAllowlist} allowlisted` : ''));
process.exit(0);
}
process.stderr.write(`\nERROR lint-test-file-count: ${failures.length} module(s) exceed the test-file limit\n\n`);
for (const f of failures) {
const tag = f.verdict === Verdict.FAIL_EXCEEDS_LIMIT
? `${f.count} files (limit ${MAX_FILES})`
: `${f.count} files (allowlist ceiling ${f.ceiling})`;
process.stderr.write(` ${f.prefix}: ${tag}\n`);
for (const tf of f.files) process.stderr.write(` ${path.relative(ROOT, tf)}\n`);
if (f.verdict === Verdict.FAIL_EXCEEDS_LIMIT) {
process.stderr.write(` ${f.prefix}: ${f.count} files (limit ${MAX_FILES}) — not in allowlist\n`);
for (const tf of f.files) process.stderr.write(` ${path.relative(ROOT, tf)}\n`);
} else if (f.verdict === Verdict.FAIL_NOVEL_FILES) {
process.stderr.write(` ${f.prefix}: ${f.novel.length} NEW test file(s) not in allowlist\n`);
for (const n of f.novel) process.stderr.write(` + ${n}\n`);
} else if (f.verdict === Verdict.FAIL_STALE_ALLOWLIST) {
if (f.count <= MAX_FILES) {
const staleList = f.stale.join(', ');
process.stderr.write(` "${f.prefix}": now at ${f.count} file(s) (≤ ${MAX_FILES}) — remove its entry from the allowlist (stale: ${staleList})\n`);
} else {
process.stderr.write(` ${f.prefix}: ${f.stale.length} allowlisted file(s) no longer present — prune allowlist\n`);
for (const s of f.stale) process.stderr.write(` - ${s}\n`);
}
}
}
process.stderr.write('\nFix: consolidate test files per module (one primary + one integration).\n');
process.stderr.write('Or add the module to scripts/lint-test-file-count.allowlist.json with PR justification.\n\n');
process.stderr.write('Or update scripts/lint-test-file-count.allowlist.json with PR justification.\n\n');
process.exit(1);
}