Phase 1 of #3464. Removes the `// allow-test-rule:` marker from 22 test files
where it is provably vestigial, and tightens the ratchet ceiling in
scripts/lint-allow-test-rule-refs.ceiling.json from 305 to 285.
Eligibility is decided by two independent AST discriminators, both
conservative (any doubt => keep):
(a) Read-target type. Every readFileSync/readFile call in the file resolves
statically to a prose/config extension (.md/.json/.yml/.yaml/.toml/.txt),
or the file performs no reads at all. Any read of a source extension
(.cjs/.js/.mjs/.ts/.cts/.mts/.jsx/.tsx), any dynamic/unresolvable path,
and any other extension all disqualify the file.
(b) Marker context. Every `allow-test-rule:` occurrence is a genuine comment
node, never string- or template-literal payload. A marker that lives
inside a RuleTester `code:` fixture is test DATA, not a suppression
directive; stripping it corrupts the test. tests/eslint-rules.test.cjs is
the one such fixture host and is deliberately untouched.
An earlier attempt at this phase classified markers by "strip it and see if
local/no-source-grep still passes" and was reverted in full before commit.
That oracle is unsound: the rule only fires on a literal .cjs/.js/.ts path
containing a quoted bin/lib/gsd-core/src segment, tracked one hop from the
binding, so files that genuinely source-grep real JavaScript pass it
silently -- tests/no-unbounded-spawn-allowlist.test.cjs (reads test sources
through a listTestFiles() walk) and tests/claude-imperative-reference.test.cjs
(matches bin/install.js through an intermediate variable) both cleared it
while being real source-greps. The rule's implementation is narrower than its
intent, so it cannot adjudicate whether an exemption is load-bearing.
Scope is limited to comment deletions: the diff over the test tree is 100%
line removals with zero insertions, and no executable line is altered.
On the ceiling value. The measured count at this HEAD is 283, so 285 leaves 2
slack -- deliberate, and well inside the documented grace band of 3. Pinning
the ceiling to the exact count makes this change effectively unmergeable: any
concurrent PR that lands one marker-bearing test file re-reds it. That race
fired twice while preparing this branch (once mid-rebase taking the count
304->305 on next, once between rebase and the verification run taking it
282->283), and it is the same race that broke next in #3461. A ceiling of
actual+2 preserves a merge window while still ratcheting 305 -> 285.
Known limit, disclosed rather than papered over: this clears 22 of 303
markers and does not reach #3464's trend-to-zero goal. Most of the remaining
markers sit on dynamic-path reads, commonly a hoisted `const p =
path.join(tmpDir, 'STATE.md')` whose target is prose but is unresolvable to
this classifier. A stricter one-hop const resolution would flip an estimated
95 more; that is deliberately left to a follow-up so it can be reviewed on
its own evidence.
Marker discovery reads bytes rather than shelling out to grep:
tests/security-prompt-injection.security.test.cjs carries a literal NUL byte
(an intentional injection fixture) that makes grep treat it as binary and skip
it, which is why the true marked-file count is 303 and not the 302 a shell
scan reports.
Closes#3465
Co-authored-by: sim <sim@local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
An allow-test-rule annotation citing a category that does not apply is worse
than no annotation, because it reads as reviewed. Eight were confirmed by
reading the assertions each one covered, and auditing the rest found five more
plus one refutation — a converter test whose wording described the wrong
mechanism while the covered assertion genuinely was deployed-text.
The instructive one used the CANONICAL string for the same mistake: STATE.md
command output labelled as a deployed artifact. A canonical string is not
evidence the category fits, which is why normalising strings alone would have
laundered the problem rather than fixed it. Every mapping the audit had inferred
rather than code-verified was spot-checked before rewriting, and the ones that
turned out not to fit were re-annotated rather than relabelled.
Fourteen STATE.md assertions had a typed extractor available all along and now
use it; their annotations came out because nothing needs exempting. Eight
assertions genuinely need a production change first — CLI stdout and stderr with
no structured mode — and are tagged pending-migration-to-typed-ir citing #3090,
which is what that category is for. It had zero real uses before this, while one
file carried a real citation to migration issue #2974 under a non-canonical tag.
Six annotations covered assertions that do no text matching at all. An exemption
for a violation that does not exist is noise that makes the real ones harder to
audit; those are removed.
atomic-write-coverage gains the annotation it always warranted — its own
docstring describes a structural-regression-guard while the file carried none.
Fifty-nine non-canonical strings across roughly thirty files are normalised, and
the allow-test-rule allowlist is regenerated to match. 472 annotations became
463: every one now uses a canonical category, and the two remaining
non-canonical strings are ESLint RuleTester fixtures, not annotations.
Refs #3057
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- rename tests/_ff_lint_violation.test.cjs -> .cjs so node --test does not execute the lint fixture (ENOENT on intentional lib/foo.cjs); scope local plugin to it so eslint . stays green (violation -> suppressedMessages, prover still proves it)
- migrate prohibition-probe.verify-tier.test.cjs A/B to inject proveFailFirst (FF-08: attestation alone no longer greens post-#1279)
- re-baseline verify-phase.md workflow size (35362 -> 35812) after the descriptor-shape prose grew
- changeset pr: TODO -> 1279 (issue number; update to PR number at open) so changeset + docs-required gates pass
Adversarial pre-submission review found the injected-runCheck tests masked a
non-functional real runner. Fixes:
- BL-01 (false green on vacuous test): the node-test runner now parses the TAP
summary and requires a NON-VACUOUS pass (>=1 test, >=1 pass, 0 fail) AND a
reported test named distinctly from the file — node --test counts an empty file
as one passing test, so counts alone could not catch it.
- SF-01 (lint anchor never greened): the lint-rule runner now runs the project
eslint as --format json and filters by ruleId, so plugin rules (local/*) load
via the flat config — bare --rule cannot load a plugin. local/no-source-grep
now genuinely greens (covered by a real, non-injected test).
- BL-02 (tautological fail-first): the runner no longer echoes the caller's
failFirst as if confirmed. failFirst is documented as caller-ATTESTED; the
producer requires attestation + a genuine non-vacuous pass. Machine-proven
fail-first (needs a violation fixture) is flagged as a tracked follow-up in
ADR-550, the changeset, FEATURES, the reference doc, and verify-phase.
- SF-02: added real-runner end-to-end tests (no injected runCheck) + pure,
exported parse/filter helpers (parseNodeTestSummary, tapTestNames,
eslintJsonHasRule, eslintFileResultCount) so the shipping branches are
mutation-pinned.
- NIT-01/02: LOCATE guard rejects empty-string rule and unknown kinds.
- Hardening: spawn checks with NODE_TEST_CONTEXT/NODE_OPTIONS scrubbed so an
ambient test-runner context cannot corrupt a verify-time result.
- Docs reconciled to the shipped behavior (no 'confirms fail-first' overclaim).
The default lint-rule runner passed check.target as BOTH the --rule id and the
eslint path, so it could never pass (eslint tried to lint a file named after the
rule). Add a distinct check.rule field (rule id) vs check.target (path to lint),
extract a pure exported buildLintArgs() so the mapping is mutation-testable
without spawning eslint, fail-closed on a lint-rule missing its rule id, and carry
the rule into enforcement evidence. Updates verify-phase descriptor docs.
- Extend prohibition-probe.verify-tier.test.cjs with the ENFORCEMENT half (#1259, ADR-550 D5d)
- Require the not-yet-built gsd-core/bin/lib/prohibition-enforcement.cjs (RED)
- Cover both wired-check kinds (node-test + no-source-grep lint-rule) and miss/fail hard-gate
- Typed-field assertions only; injected runCheck (no real subprocess)
- Keep the 2 original fail-closed tests verbatim
Adds the spec-time prohibition probe (spec-phase Step 5.6) — the second adapter of the probe-core resolution model. Surfaces unwritten must-NOT constraints as negative SPEC acceptance criteria with test/judgment verification tiers; fail-closed at verify time. Per ADR-550. Closes#644.
