2 Commits

Author	SHA1	Message	Date
Tom Boucher	bc557f6876	chore(#3520): ratchet on effective exemptions, track unverified separately (#3529) ... Phase 5 of #3464, following #3465, #3466, #3502 and #3508. Those cut the ceiling 305 -> 278, made the rule accurate, and ended file-wide amnesty. This one fixes the number itself. scripts/lint-allow-test-rule-refs.cjs counted FILES CONTAINING MARKER TEXT. Only 5 of those files carry a marker that actually suppresses a violation the rule detects, across 10 sites. The ratcheted number was ~98% noise, which is exactly why bumping it was frictionless: the metric was never coupled to the thing it claimed to govern. That is the whole complaint this epic opened with, stated precisely. Verified directly rather than assumed: only eslint-rules/no-source-grep.cjs functionally honors the marker. Four other rule files mention allow-test-rule in prose only, and no-raw-rmsync-in-tests.cjs:24 explicitly states it does not apply. So the large count was not legitimately large because several rules share the annotation. Now two numbers, only the first ratcheted: EFFECTIVE EXEMPTIONS -- markers that actually suppress a detected violation. 10 sites across 5 files. Tightly ratcheted in both directions, as before: over the ceiling fails, and slack beyond grace fails. UNVERIFIED MARKERS -- marker-bearing files with no detectable violation. 273 files. Reported and given a loose ceiling so the pool cannot silently balloon, but deliberately NOT tightly ratcheted, because shrinking it is a rule-coverage problem and not a delete-the-markers problem. A file with at least one effective site counts as effective and is not also counted as unverified; the two numbers never double-count. Reporting ONLY the effective count was considered and rejected. It would say five files and look excellent while being falsely reassuring, because "no detectable violation" is not "no violation". This phase's own measurement found two genuine source-greps that are unsuppressed AND undetected -- tests/security-prompt-injection.security.test.cjs:852 and tests/check-update-config-dir.test.cjs:91 -- each reading a real shipped file and text-searching it, invisible only because the path is bound to a separate const the rule never resolves back to its literal. Markers guarding that class count as zero-effective and would look vestigial. Trading a number that is too big and meaningless for one that is too small and falsely reassuring is not progress, so the script prints the known-limit caveat alongside the numbers and the two undetected violations are filed separately rather than lost. Single source of truth is structural, not a matter of discipline. The script does not re-implement detection or the site-scoped adjacency predicate -- that is the generative-fix-divergence class this repo has shipped before. The rule now exports MAX_MARKER_LOOKAHEAD_LINES, MARKER_COMMENT_RE, collectMarkerAndCommentLines and isSuppressedAt (extracted verbatim, no logic change), plus a default-off neutralizeSuppression option so the counter can enumerate every site through the real rule via ESLint's Linter API and then classify each with the rule's own predicate, replicating reportUnlessSuppressed's search-line-OR-read-line check exactly. Default rule behavior is byte-identical: tests/eslint-rules.test.cjs passes 168/168 unchanged. A parity test asserts the script's suppressed/not verdict equals the rule's own report/no-report outcome for every site in a fixture corpus. A real silent-failure bug surfaced and was fixed while building this: ESLint's flat-config Linter reports "No matching configuration" and returns ZERO messages for any filename resolving outside its cwd. That would have quietly misclassified every sandboxed test fixture as having no violations -- a test suite that passes while asserting nothing. Fixed by anchoring the Linter to the tests dir, with a defensive throw if it ever recurs. Two earlier claims of mine are corrected by this phase's measurement. Widening the source-dir allowlist to include hooks/ -- the "fifth blind spot" recorded in #3508 -- rescues ZERO sites; it is real in principle and has no practical effect, because the hooks reads that exist are missed for other reasons (.sh extension, identifier-indirection, dynamic filenames). And the #3508 correction that attributed those reads to the hooks/ gap rather than to variable indirection was itself incomplete: both are independently sufficient, so fixing either alone changes nothing. I accepted the reviewer's causal claim as uncritically as I had made my own. The unverified count is 273, not the ~289 in the phase design doc. That is legitimate drift -- the baseline was measured at fba7c9032 and other merged work has since removed markers. Left as measured rather than adjusted to match the document. Adversarial review found a BLOCKER in the first revision and it is fixed here. The counter walked only tests/**/*.test.cjs, but the rule is registered on tests/**/*.cjs -- every .cjs, not just test files -- plus scripts/**, eslint-rules/**, bin/lib/**, pi/**, examples/**, gsd-core/bin/** and three plugin globs. 35 non-.test.cjs files under tests/ were never walked, and tests/helpers/live-command-registry.cjs:1 carries a real marker that appeared in NEITHER reported number. A counter that undercounts is worse than the meaningless one it replaces, because it will be trusted. The scan set is now derived programmatically: the script dynamically imports eslint.config.mjs and extracts the `files` globs from every config block that enables local/no-source-grep. There is no hardcoded list to drift, which is the same divergence class the parity test already guards. Two further defects surfaced while fixing it. The silent-clean catch around linter.verify() was worse than reported -- ESLint signals a parse error by returning a message with fatal:true rather than throwing, so the original catch would not even have fired for the common case. Both paths now throw with the file path. That silent swallow was actively hiding a broken fixture in this suite's own tests: 'no marker here\n' is not valid JS and the case only "passed" because the parse failure was discarded. Fixed. And once the scan widened, the raw-text marker scanner started matching this tooling's own doc comments and RuleTester fixture strings, so marker extraction moved to AST comment nodes. That corrected three long-standing FALSE POSITIVES: allowlist entries for tests/eslint-rules.test.cjs that were never real markers, only fixture payload. Allowlist 137 -> 135: three false positives pruned, one real entry added for live-command-registry. The headline numbers are coincidentally unchanged (10/10 effective across 5 files, 273/280 unverified) but the composition is corrected: one real file gained, one phantom dropped. Verified by direct diff rather than inferred from the totals matching. The first remote run of this branch came back RED with 23 failures, all one cause, and it is fixed here. Classification drove ESLint's flat-config Linter, which resolves configuration relative to a cwd and refuses to lint any file outside it. The test harness writes fixtures into an OS temp dir, so every sandbox row hit "No matching configuration found" and tripped the defensive throw. It surfaced only on the container because the repo lives at /work there and the fixtures at /tmp, making the mismatch unmissable; a local run had reported the suite green, which it was not. Fixed by not depending on config *resolution* at all: the script now builds an eslintrc-format Linter and registers the rule directly with defineRule, since it already knows exactly which rule and options it wants. That removes the cwd-ancestor constraint entirely and makes repo files and out-of-tree fixtures classify identically. Verified out-of-tree explicitly, not just in-repo, because the local temp path shape is what hid the bug the first time. Parse failures still throw loudly -- that behavior is required and tested. Classifications are unchanged for real repo files (10/10 effective across 5, 273/280 unverified, 0 live), which is the check that the config swap did not quietly alter results. The second remote run cut the failures from 23 to 2, and the survivors were a genuinely different and more interesting defect: tests/packaging-shipped-scripts-require-only-shipped.test.cjs caught that scripts/ SHIPS in the published package while eslint-rules/ does not, so importing the rule for single-source-of-truth would MODULE_NOT_FOUND in a real consumer's install. That test statically extracts require() calls, so hiding the import inside a function would have dodged the check without fixing the problem. Resolved along the grain of existing convention rather than by weakening anything: package.json already excludes several repo-internal lint gates from the shipped set via `!scripts/...`, including `!scripts/lint-no-adhoc-regex-escape.cjs`, which is the same situation. This gate is CI-only and has no meaning in a consumer install, so it joins them. Verified with `npm pack --dry-run` that the .cjs is genuinely absent from the tarball (its inert JSON config files remain, and carry no requires). The third remote run failed on shard 3/3 across all three OSes with exit code NULL and empty output -- the spawned gate was killed by a timeout, not failing an assertion. Cause: replacing a raw text scan with a full ESLint Linter pass over every file in every registered glob took the gate from 0.57s to ~7-12s, and GitHub's runners are slower than the bench that had just passed it green. Fixed by narrowing the work rather than raising the timeout to hide it. Both numbers the gate computes are properties of files that CONTAIN a marker, so only those (~294) need linting; the repo-wide byte walk that finds them stays, since that was the undercount fix. Live violations in files carrying NO marker are already enforced by npm run lint over exactly these globs, so re-detecting them here was redundant. 2.87s now, from ~7s measured locally. That narrowing changes what one reported number means, so the wording changed with it: "live violations in marker-bearing files: 0 (unmarked files are enforced separately by npm run lint)". A number that quietly covers less than it reads is the failure this whole epic is about, so it is stated rather than left implicit, and the test row that asserted the old broader behavior was split -- an unmarked live violation now passes this gate (and is caught by eslint), while a live violation in a marker-bearing file still fails it. The repo-baseline test's timeout was also raised to 30s with a comment, since a gate that legitimately takes ~3s must not sit at a timeout close to its own runtime. Every other row keeps the shorter sandbox-scoped timeout. Closes #3520 Co-authored-by: sim <sim@local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-15 00:33:31 -04:00
Tom Boucher	5339dd60e5	feat(#3313): allow-test-rule total-file-count ratchet, F17 promotions (#3326) ... Extends lint-allow-test-rule-refs.cjs with a second, independent check alongside the existing uncited-citation identity ratchet: the total number of distinct test files carrying any allow-test-rule marker (cited or not) is now checked against a tight ceiling via the previously-unwired assertTightCeiling primitive (allowlist-ratchet.cjs, 0 prior callers). A cited exemption is legitimate under ADR-456 but nothing stopped the raw total from growing forever - this closes that gap without duplicating the file walk (both checks consume one shared walkTestFiles pass). Ceiling introduced at the exact measured high-water mark (314 files, grace 3) rather than a padded estimate, per "budgets may only decrease." Also lands the two F17 pieces (absorbed from the now-closed #1885) that had no precondition: - --max-warnings 0 added to lint/lint:ci - local/no-source-grep promoted warn->error in the scripts/bin/ eslint-rules glob block (already error in the tests/ glob) Both promotions were pre-verified against a zero-warning tree (fresh non-cached eslint run) before flipping, per the maintainer's clean- tree-first decision. Not included: local/no-elapsed-assertion promotion, which stays warn pending #3314 (H2) - 10 of 19 clock-touching src modules have no sanctioned time-control mechanism until ADR-456 is amended there. H1 of epic #3053, absorbing #1885 F17. Co-authored-by: sim <sim@local>	2026-08-10 12:23:01 -04:00