fix: recover silently-excluded test dirs + test-architecture audit hardening (#1195)
* fix: recurse test discovery so subdir test suites actually run
scripts/run-tests.cjs discovered tests with a flat readdirSync(testDir),
silently excluding tests/observability/ (4 files), tests/dispatch/ (1) and
tests/installer-migrations/ (1) — 94 passing tests — from `npm test` and all
CI lanes. Walk the tree recursively (relative subpaths preserved), classify
suites by basename, and add a fail-on-zero-executed guard for suite/default
runs (escape hatch GSD_ALLOW_EMPTY_SUITE=1) while preserving the empty
--files/--files-from path the CI inert lane relies on.
Unit suite 735 -> 741 files; surfaces ADR-227's observability/dispatch seam.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: retire 5 verified-worthless tests
Adversarial verification confirmed these 5 prove nothing — their coverage is
provided more strictly elsewhere:
- enh-2790 'has a name: field' spot-checks (command-contract enforces /^gsd[:-]/)
- command-routing-hub duplicate construct + duplicate ERROR_KINDS assertions
- no-cjs-sdk-handsync-tooling (guarded files that never existed on main; bug-190
covers the real retired SDK artifacts)
- runtime-artifact-layout cline edge case (subsumed by the explicit-global test
and bug-782-cline-skills-emission)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: add ADR-218 release version-validation coverage
ADR-218 (reject leading-zero versions like 1.01.0; npm duplicate pre-check) had
zero tests — the logic lived only in release.yml bash. Add a test that extracts
the actual rejection regexes from the workflow and exercises them against a
boundary table (leading-zero/malformed rejected, valid accepted) plus structural
wiring assertions. Goes red if the regex is reverted to [0-9]+.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: redesign weak tests into behavioral, deterministic assertions
Per the ADR test audit, rewrite 27 weak test files (test-only, no source
changes) so each can go red for the defect it guards:
- kill pass-always assert.ok(true) placeholders (research-cli, worktree-baseref,
bug-260 security guard, eslint-rules x24, clusters '|| true')
- replace source-text grep with behavioral calls (install Kilo, sh-hook-paths,
plan-review-convergence) and add a repo-layout governance test
- de-flake real-clock/Math.random coupling (phase last_updated, bug-3707 mtime,
context-utilization property, feat-3594)
- fix independence/shared-state violations (bug-492 singleton, issue-844 tmpRoot,
core reapStaleTempFiles, active-workstream TTY, feat-488 GSD_HOME)
- strengthen property/shape-only tests (research-provider/store classification +
collision) and unconditional plugin.json schema validation (issue-766)
Verified: all 28 files run together 1220 pass / 0 fail / 1 skip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: add no-tautological-assert lint rule, error in test suite
New custom ESLint rule (eslint-rules/no-tautological-assert.cjs) bans asserts
that can never fail: assert(true)/assert.ok(<always-truthy literal>),
'cond || true' inside an assert, and equality asserts comparing two identical
literals. Wired as error on tests/**; full sweep confirmed zero existing
violations so the suite stays green. Prevents the placeholder-assert regressions
the audit redesigns just removed. RuleTester coverage added (6 valid, 8 invalid).
Note: no-only-tests was already enforced via eslint-plugin-no-only-tests, so no
duplicate rule was added.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: gate new allow-test-rule exemptions to require an issue ref
ADR-456 requires any allow-test-rule exemption added after the ADR to carry a
tracking issue number, but nothing enforced it. New ratchet gate
(scripts/lint-allow-test-rule-refs.cjs, wired into lint:ci) fails when a NEW
allow-test-rule comment lacks a #NNN/URL reference; the 323 existing untracked
exemptions are grandfathered in an allowlist that ratchets down as they gain
refs. Red-green verified (novel untracked offender fails; compliant passes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: add ADR test-audit evidence report (#1192)
Full risk-first qa-test-architect audit of the ADR portfolio (37 ADRs + 4
platform lenses, adversarial verification of retire verdicts) that drove the
P0 discovery fix, ADR-218 coverage, 5 retires, 27 redesigns, and the two new
lint gates. Filed as point-in-time evidence under docs/issueevidence/, named
for tracking issue #1192.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: replace pre-existing raw NUL byte with escape in feat-3594 fixture
feat-3594's null-byte parser fixture contained a literal NUL byte (pre-existing
on next at b10e5681 — confirmed: base blob has 1 NUL, this fix has 0), which
made git treat the file as binary and would break grep/editors. Switch to the
\x00 escape; the runtime string value (a real NUL in the parser input) is
unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: address adversarial-review findings
Codex adversarial pass over the branch:
- capability-registry drift test no longer mutates the committed generated
capability-registry.cjs in place (concurrency hazard) — uses in-memory
checkPipeline comparison instead.
- allow-test-rule ratchet now detects exemptions in ALL comment forms (block
/* */ too, matching no-source-grep) so a block comment can't bypass it;
one newly-surfaced pre-existing offender grandfathered (323->324).
- install.test Kilo case asserts on what install(false,'kilo') actually writes
rather than manually calling configureKiloPermissions (masked the call site).
- issue-766 drops the undeclared transitive ajv dep for explicit structural
assertions from the schema fixture.
- adr-218 test notes the hotfix leading-zero gap is tracked in #1186.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address code-review findings (subdir discovery, rule + test gaps)
xhigh code review surfaced 15 confirmed issues, all fixed:
- run-tests.cjs --files now resolves subdir tests by bare basename + handles
Windows backslash paths (ambiguous basenames error clearly).
- affected-tests-lib.cjs listTestFiles made recursive — the targeted CI lane was
silently dropping changed subdir tests (same false-green class the audit fixed).
- no-tautological-assert now catches 'true || cond' and empty []/{} equality.
- verify-test-quality: restore provenance-classification coverage, tighten the
writeFile circular-detection check, guard the module-level file read.
- sh-hook-paths: cover the global-install .sh delegation branch (#2045 guard).
- active-workstream null-guard runs deterministically (no longer skipped on TTY).
- adr-218 structural guards tightened (major/minor leading-zero; needs: membership).
- repo-layout AGENTS.md guard no longer false-alarms on equivalent refactors.
- cross-ai ordering guard fails red when the step is missing.
- issue-766 parses required fields from the schema fixture (auto-enforced).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: stub USERPROFILE alongside HOME in feat-488 (Windows parity)
The feat-488 redesign stubbed process.env.HOME but not USERPROFILE; os.homedir()
resolves from USERPROFILE on Windows, so the home stub was not hermetic there —
caught by windows-test-parity-guard (stubsHomeNoUserProfile). Save/set/restore
USERPROFILE symmetrically with HOME (delete-if-originally-undefined).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: reconcile allow-test-rule allowlist after rebase onto next
Rebasing onto current next pulled in merged PR #1170, which added
inventory-headings-countfree.test.cjs (a baseline allow-test-rule exemption) and
deleted inventory-counts.test.cjs. Grandfather the former and prune the latter so
the ratchet matches the merged tree. No new debt from this PR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
162
scripts/lint-allow-test-rule-refs.cjs
Normal file
162
scripts/lint-allow-test-rule-refs.cjs
Normal file
@@ -0,0 +1,162 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* lint-allow-test-rule-refs.cjs — enforce that NEW `allow-test-rule:` exemption
|
||||
* comments carry a tracking-issue reference.
|
||||
*
|
||||
* ## Why
|
||||
*
|
||||
* `allow-test-rule:` is an inline comment that disables the `no-source-grep`
|
||||
* ESLint rule for a whole test file. Today many such comments exist with no
|
||||
* issue reference, making it impossible to audit or revisit them. Per ADR-456
|
||||
* (docs/adr/456-test-rigor-architecture.md) every NEW exemption must carry a
|
||||
* `#NNN` issue reference or an https:// URL so the decision is traceable.
|
||||
*
|
||||
* ## What "compliant" means
|
||||
*
|
||||
* A compliant `allow-test-rule:` comment is one whose reason text (everything
|
||||
* after the colon) contains either:
|
||||
* - a `#\d+` token (e.g. `// allow-test-rule: see #1234`)
|
||||
* - an https?:// URL
|
||||
*
|
||||
* Any other comment is an OFFENDER.
|
||||
*
|
||||
* ## Grandfathering
|
||||
*
|
||||
* All pre-existing untracked exemptions are recorded in
|
||||
* scripts/lint-allow-test-rule-refs.allowlist.json (seeded at gate introduction
|
||||
* time). The identity ratchet (scripts/lib/allowlist-ratchet.cjs) means:
|
||||
* - A NEW non-compliant comment not in the allowlist → gate fails.
|
||||
* - A previously-offending comment that is now compliant → allowlist entry is
|
||||
* STALE and must be pruned (ratchet-down; the baseline only ever shrinks).
|
||||
*
|
||||
* ## Offender identifiers
|
||||
*
|
||||
* Identifiers are stable cross-rename-safe strings of the form:
|
||||
* `<repo-relative-path> :: <trimmed-reason>`
|
||||
*
|
||||
* e.g. `tests/foo.test.cjs :: source-text-is-the-product`
|
||||
*
|
||||
* If a file has multiple non-compliant comments with the SAME reason text, only
|
||||
* one identifier is recorded (deduped via Set).
|
||||
*
|
||||
* See docs/adr/456-test-rigor-architecture.md for the full policy.
|
||||
*/
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { assertWithinAllowlist } = require('./lib/allowlist-ratchet.cjs');
|
||||
const { ExitError, runMain } = require('./lib/cli-exit.cjs');
|
||||
|
||||
const ROOT = path.join(__dirname, '..');
|
||||
const TESTS_DIR = process.env.GSD_LINT_ALLOW_TEST_RULE_TESTS_DIR || path.join(ROOT, 'tests');
|
||||
const ALLOWLIST_PATH =
|
||||
process.env.GSD_LINT_ALLOW_TEST_RULE_ALLOWLIST ||
|
||||
path.join(__dirname, 'lint-allow-test-rule-refs.allowlist.json');
|
||||
|
||||
/**
|
||||
* Extracts the reason text after `allow-test-rule:` from a single line of source
|
||||
* text in any comment form that the no-source-grep ESLint rule honours.
|
||||
*
|
||||
* The ESLint rule tests `c.value` (AST comment node value, delimiters stripped)
|
||||
* with /allow-test-rule:\s*\S/, which fires on BOTH:
|
||||
* // allow-test-rule: <reason> (line comment)
|
||||
* /* allow-test-rule: <reason> * / (block comment, single-line)
|
||||
*
|
||||
* By scanning line-by-line and extracting everything after `allow-test-rule:` on
|
||||
* each line, we cover both forms without a cross-line regex (which was previously
|
||||
* matching arbitrary `/* ... * /` pairs spanning hundreds of lines, causing false
|
||||
* positives).
|
||||
*
|
||||
* The trailing `*\/` and whitespace are stripped so block-comment closers don't
|
||||
* bleed into the extracted reason.
|
||||
*/
|
||||
const ALLOW_TEST_RULE_LINE_RE = /allow-test-rule:\s*(.+)/;
|
||||
/** Matches a compliant issue reference or URL */
|
||||
const ISSUE_REF_RE = /#\d+|https?:\/\//;
|
||||
|
||||
/**
|
||||
* Recursively collect offender identifiers from all *.test.cjs files under dir.
|
||||
*
|
||||
* @param {string} dir absolute path to scan
|
||||
* @returns {string[]} sorted, deduped list of `<relpath> :: <reason>` strings
|
||||
*/
|
||||
function collectOffenders(dir) {
|
||||
const offenders = new Set();
|
||||
|
||||
function scan(current) {
|
||||
for (const entry of fs.readdirSync(current, { withFileTypes: true })) {
|
||||
const full = path.join(current, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
scan(full);
|
||||
} else if (entry.isFile() && entry.name.endsWith('.test.cjs')) {
|
||||
const relpath = path.relative(ROOT, full).split(path.sep).join('/');
|
||||
let content;
|
||||
try {
|
||||
content = fs.readFileSync(full, 'utf8');
|
||||
} catch {
|
||||
// skip unreadable files (e.g. binary)
|
||||
continue;
|
||||
}
|
||||
// Scan line-by-line. By testing each line for `allow-test-rule:` we
|
||||
// cover BOTH comment forms without a cross-line regex:
|
||||
// // allow-test-rule: <reason> ← line comment
|
||||
// /* allow-test-rule: <reason> */ ← single-line block comment
|
||||
//
|
||||
// For each matching line we extract the reason (everything after the
|
||||
// colon), then strip any trailing block-comment closer `*/` and
|
||||
// whitespace so the identifier stays clean.
|
||||
for (const line of content.split('\n')) {
|
||||
const m = ALLOW_TEST_RULE_LINE_RE.exec(line);
|
||||
if (!m) continue;
|
||||
// Strip trailing block-comment closer and whitespace if present
|
||||
const reason = m[1].replace(/\s*\*\/\s*$/, '').trim();
|
||||
if (!reason) continue;
|
||||
if (ISSUE_REF_RE.test(reason)) continue; // compliant — skip
|
||||
offenders.add(`${relpath} :: ${reason}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
scan(dir);
|
||||
return [...offenders].sort();
|
||||
}
|
||||
|
||||
function main() {
|
||||
const args = process.argv.slice(2);
|
||||
const unknown = args.filter((a) => a !== '--help');
|
||||
if (unknown.length > 0) {
|
||||
throw new ExitError(2, `lint-allow-test-rule-refs: unknown argument(s): ${unknown.join(', ')}`);
|
||||
}
|
||||
|
||||
const current = collectOffenders(TESTS_DIR);
|
||||
const known = JSON.parse(fs.readFileSync(ALLOWLIST_PATH, 'utf8'));
|
||||
|
||||
const failures = [];
|
||||
const { novel } = assertWithinAllowlist({
|
||||
label: 'allow-test-rule-refs',
|
||||
current,
|
||||
known,
|
||||
fail: (msg) => failures.push(msg),
|
||||
pruneHint: 'edit scripts/lint-allow-test-rule-refs.allowlist.json',
|
||||
});
|
||||
|
||||
if (failures.length > 0) {
|
||||
for (const msg of failures) process.stderr.write(`${msg}\n`);
|
||||
if (novel.length > 0) {
|
||||
process.stderr.write(
|
||||
'\nNew allow-test-rule exemption without an issue ref — add `see #NNN` per ADR-456' +
|
||||
' (docs/adr/456-test-rigor-architecture.md).\n'
|
||||
);
|
||||
}
|
||||
throw new ExitError(1);
|
||||
}
|
||||
|
||||
console.log(
|
||||
`ok lint-allow-test-rule-refs: ${current.length} grandfathered exemption(s) tracked, no novel untracked offenders`
|
||||
);
|
||||
}
|
||||
|
||||
runMain(main);
|
||||
Reference in New Issue
Block a user