diff --git a/.changeset/1279-machine-proven-fail-first.md b/.changeset/1279-machine-proven-fail-first.md index 79b9d331c..0a06345cb 100644 --- a/.changeset/1279-machine-proven-fail-first.md +++ b/.changeset/1279-machine-proven-fail-first.md @@ -1,10 +1,10 @@ --- type: Changed -pr: TODO +pr: 1279 --- **Test-tier prohibition fail-first is now MACHINE-PROVEN, not caller-attested** — the deferred literal `regression-must-fail-first` property of ADR-550 Decision 4 (the gap #1259 / PR #1273 left as a tracked follow-up) has landed. The `check prohibition-enforcement` producer (`src/prohibition-enforcement.cts`, compiled by `build:lib` to the gitignored `gsd-core/bin/lib/prohibition-enforcement.cjs`) no longer trusts the caller's `failFirst` attestation: before a clean, non-vacuous pass can dispose a `test`-tier prohibition green, the new `defaultProveFailFirst` prover independently RUNS the wired check against a KNOWN VIOLATION and confirms it goes RED. Attestation is gone from the green AND (`passed = proof.provenFailFirst === true && run.passed === true`); any other outcome — passes-on-violation, can't-prove, throws, times out, or no violation source — **hard-gates in BOTH interactive and autonomous modes** (ADR-550 D4 / D3). The evidence record gains a `failFirstProof` field recording HOW fail-first was proven (FF-07). A caller can no longer green a toothless check. The violation is sourced from a new descriptor field, **`CheckDescriptor.violationFixture`** — an author-supplied path to a known-bad subject. For a `lint-rule` the prover lints that fixture and requires the rule id to appear in the JSON report (the rule must have teeth); for a `node-test` the prover spawns the negative test with the subject injected through the **`GSD_PROHIB_SUBJECT`** env convention and requires the run to report `# fail >= 1` (the pure, mutation-pinned `isNodeTestRed` helper). The `lint-rule` path is fully shippable now and is dogfooded against the in-tree `local/no-source-grep` rule; the `node-test` path ships its mechanism (a fixture-bearing descriptor IS machine-proven) and is exercised by SYNTHETIC temp fixtures — there is no live in-tree `node --test` prohibition to dogfood. `CheckDescriptor.failFirst` is **DEMOTED, not removed** (FF-08): it is kept as a non-authoritative hint so the #1259 route-JSON shape and the `CheckDescriptor` type stay backward-compatible mid-migration, but no path greens on it alone. The green/fail-closed policy in `src/probe-core.cts` (`dispositionForProhibition`, reads only `evidence.length > 0`) is untouched; the evidence array shape is additive. This closes ADR-550's D5d follow-up — see the dated 2026-06-15 ADR-550 addendum. (#1279) -**PR-review flag — `GSD_PROHIB_SUBJECT` + `violationFixture` are PROPOSED, renamable conventions.** Both are net-new surface with ZERO live in-tree consumers (no in-tree node-test prohibition yet; node-test proof runs only on synthetic test fixtures, the real dogfood stays the lint-rule). They are forward-looking scaffolding, so a later rename — or replacing the env var with an argv — is a mechanical, zero-migration find/replace. Surfacing them here so the maintainer can **ratify, rename, or replace them at PR review** with no migration cost, exactly as #1278's ADR addendum was reviewed at PR time. The `failFirst` demotion is likewise open to weighing outright removal; the keep-as-hint rationale is recorded in the ADR addendum. +**PR-review flag — `GSD_PROHIB_SUBJECT` + `violationFixture` are PROPOSED, renamable conventions.** Both are net-new surface with ZERO live in-tree consumers (no in-tree node-test prohibition yet; node-test proof runs only on synthetic test fixtures, the real dogfood stays the lint-rule). They are forward-looking scaffolding, so a later rename — or replacing the env var with an argv — is a mechanical, zero-migration find/replace. Surfacing them here so the maintainer can **ratify, rename, or replace them at PR review** with no migration cost, exactly as #1278's ADR addendum was reviewed at PR time. The `failFirst` demotion is likewise open to weighing outright removal; the keep-as-hint rationale is recorded in the ADR addendum. (The `pr:` field above is set to the issue number #1279 because the PR is not yet open; update it to the actual PR number at open time.) diff --git a/eslint.config.mjs b/eslint.config.mjs index f9baa6efb..9f3df9be1 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -265,4 +265,19 @@ export default tseslint.config( 'no-irregular-whitespace': 'warn', }, }, + + // ── #1279 lint-rule fail-first fixture ────────────────────────────────────── + // `tests/_ff_lint_violation.cjs` is a PLAIN `.cjs` (NOT `*.test.cjs`) on purpose: it is a KNOWN + // `local/no-source-grep` violation that `defaultProveFailFirst` lints to machine-prove the rule + // has teeth, and it must stay OFF the `node --test` runner glob (executing it ENOENTs on the + // intentional `lib/foo.cjs` path). It still needs the `local` plugin registered so its inline + // `/* eslint-disable local/no-source-grep */` resolves (otherwise `eslint .` errors "rule not + // found") and the violation lands in `suppressedMessages` (which the prover reads), keeping the + // project's own `eslint .` green. (#1279) + { + files: ['tests/_ff_lint_violation.cjs'], + plugins: { local: localPlugin }, + languageOptions: { sourceType: 'commonjs', globals: { ...globals.node } }, + rules: { 'local/no-source-grep': 'error' }, + }, ); diff --git a/tests/_ff_lint_violation.test.cjs b/tests/_ff_lint_violation.cjs similarity index 74% rename from tests/_ff_lint_violation.test.cjs rename to tests/_ff_lint_violation.cjs index 63309041f..73a3eaf8b 100644 --- a/tests/_ff_lint_violation.test.cjs +++ b/tests/_ff_lint_violation.cjs @@ -4,13 +4,18 @@ // This file is a KNOWN `local/no-source-grep` violation used by `defaultProveFailFirst`'s lint-rule // path to machine-prove the rule has teeth (the fail-first proof). The exact form below is the ONLY // shape that fires `local/no-source-grep` under the project flat config (verified empirically, #1279 -// RESEARCH): a `tests/**/*.test.cjs` file whose `readFileSync` argument is the +// RESEARCH): a lint-scoped `.cjs` file under `tests/` whose `readFileSync` argument is the // `path.join('lib','foo.cjs')` form (a STANDALONE quoted source-dir token `'lib'` + a quoted // `.cjs` extension) followed by a text-search method (`.includes`) on the bound variable. // // - `'src/x.cjs'` as a single string literal does NOT fire (no standalone quoted dir token). // - a repo-root / tmp path does NOT fire (no source-dir token). // +// NAME: this is a PLAIN `.cjs` fixture, NOT `*.test.cjs`. `no-source-grep` has no filename guard, so +// it fires on any lint-scoped `.cjs`; the `.test.cjs` suffix would make the `node --test` runner +// EXECUTE it (ENOENT on `lib/foo.cjs` at module load → a real full-suite failure, #1279 wave 5). It +// is a lint fixture, never a test — keep it off the `.test.cjs` glob. +// // The leading `/* eslint-disable local/no-source-grep */` keeps the project's own `eslint .` green // (the violation lands in eslint's `suppressedMessages`, which the prover's `eslintJsonHasRule` // reads — an inline-disabled violation still proves the rule has teeth, #1259 B1). If the prover diff --git a/tests/prohibition-enforcement.test.cjs b/tests/prohibition-enforcement.test.cjs index aafe92279..764e59e57 100644 --- a/tests/prohibition-enforcement.test.cjs +++ b/tests/prohibition-enforcement.test.cjs @@ -536,7 +536,7 @@ describe('prohibition-enforcement REAL runner end-to-end (#1259)', () => { test('a clean in-tree target greens the lint-rule kind via the real eslint + real prover (SF-01: plugin loads)', () => { const enforce = require(ENFORCEMENT_LIB); // Migrated to the SHIPPING prover (#1279): the default real prover lints the committed - // `_ff_lint_violation.test.cjs` violationFixture (the rule fires -> fail-first proven) while the + // `_ff_lint_violation.cjs` violationFixture (the rule fires -> fail-first proven) while the // clean runCheck lints src/clock.cts (no violation -> non-vacuous pass). Both via real eslint. const result = enforce.runProhibitionEnforcement( TEST_TIER, @@ -545,7 +545,7 @@ describe('prohibition-enforcement REAL runner end-to-end (#1259)', () => { rule: 'local/no-source-grep', target: 'src/clock.cts', failFirst: true, - violationFixture: path.join('tests', '_ff_lint_violation.test.cjs'), + violationFixture: path.join('tests', '_ff_lint_violation.cjs'), }, { cwd: process.cwd() }, ); @@ -579,7 +579,7 @@ describe('prohibition-enforcement REAL runner end-to-end (#1259)', () => { test('FULL producer (real prover + real runner): lint-rule greens on a real no-source-grep violation fixture + clean target (FF-02/FF-10)', () => { const enforce = require(ENFORCEMENT_LIB); // No injected runCheck/proveFailFirst: the default prover lints the committed - // `_ff_lint_violation.test.cjs` (the rule fires -> fail-first proven) AND the default runner lints + // `_ff_lint_violation.cjs` (the rule fires -> fail-first proven) AND the default runner lints // the clean `src/clock.cts` (no violation -> non-vacuous pass). BOTH directions via real eslint. const result = enforce.runProhibitionEnforcement( TEST_TIER, @@ -587,7 +587,7 @@ describe('prohibition-enforcement REAL runner end-to-end (#1259)', () => { kind: 'lint-rule', rule: 'local/no-source-grep', target: 'src/clock.cts', - violationFixture: path.join('tests', '_ff_lint_violation.test.cjs'), + violationFixture: path.join('tests', '_ff_lint_violation.cjs'), }, { cwd: process.cwd() }, ); @@ -707,12 +707,12 @@ describe('prohibition-enforcement REAL runner end-to-end (#1259)', () => { // ─── #1279 defaultProveFailFirst REAL prover end-to-end (FF-02 / FF-03 / FF-05 / FF-06 / FF-07) ── // Exercises the SHIPPING default prover against REAL subprocesses (eslint + node --test) — the gap // that let #1259's BL-01/SF-01 slip past injected doubles. The lint-rule path dogfoods the committed -// `tests/_ff_lint_violation.test.cjs` fixture; the node-test path uses synthetic temp fixtures that +// `tests/_ff_lint_violation.cjs` fixture; the node-test path uses synthetic temp fixtures that // demonstrate the `GSD_PROHIB_SUBJECT` subject-injection convention. Typed-result assertions only. describe('prohibition-enforcement defaultProveFailFirst REAL prover (#1279)', () => { const fs = require('node:fs'); // The committed load-bearing lint fixture (a real, suppressed no-source-grep violation). - const LINT_FIXTURE = 'tests/_ff_lint_violation.test.cjs'; + const LINT_FIXTURE = 'tests/_ff_lint_violation.cjs'; test('exports the prover surface (defaultProveFailFirst + FailFirstProof-shaped result)', () => { const enforce = require(ENFORCEMENT_LIB); diff --git a/tests/prohibition-probe.verify-tier.test.cjs b/tests/prohibition-probe.verify-tier.test.cjs index e07f8116d..ef9ac82c6 100644 --- a/tests/prohibition-probe.verify-tier.test.cjs +++ b/tests/prohibition-probe.verify-tier.test.cjs @@ -70,7 +70,9 @@ describe('prohibition-probe verify-tier: test-tier fail-closed safety (PROB-12 / // non-green) in BOTH interactive and autonomous modes (ADR-550 D4 / D3). // // Typed-field assertions ONLY (status / flagged / tier / evidence / located / kind / mode-block). -// The check-runner is injected via options.runCheck so no real subprocess is spawned. +// The check-runner is injected via options.runCheck so no real subprocess is spawned; the GREEN +// cases (A/B) ALSO inject options.proveFailFirst since #1279 (FF-08) requires fail-first to be +// MACHINE-PROVEN — caller attestation (failFirst:true) alone no longer greens. describe('prohibition-probe verify-tier: test-tier ENFORCEMENT (REQ-PROHIB-07 / ADR-550 D5d)', () => { const testTierProhibition = { requirement_id: 'R1', @@ -83,12 +85,15 @@ describe('prohibition-probe verify-tier: test-tier ENFORCEMENT (REQ-PROHIB-07 / }; // Test A — node --test negative test that PASSES -> green + non-empty evidence. + // #1279 (FF-08): attestation alone no longer greens — the producer requires the fail-first to be + // MACHINE-PROVEN. With injected runCheck (no real subprocess) we also inject a proving prover so + // the GREEN direction is exercised; the hard-gate directions (C/D/D2 below) need no prover. test('A: wired node-test check that passes disposes green with non-empty enforcement evidence', () => { const enforce = require(ENFORCEMENT_LIB); const result = enforce.runProhibitionEnforcement( testTierProhibition, { kind: 'node-test', target: 'tests/some-negative.test.cjs', failFirst: true }, - { runCheck: () => ({ passed: true }) }, + { runCheck: () => ({ passed: true }), proveFailFirst: () => ({ provenFailFirst: true }) }, ); assert.ok(result && typeof result === 'object', 'result must be a structured object'); assert.equal(result.status, 'green', 'a passing wired node-test check must dispose green'); @@ -105,7 +110,7 @@ describe('prohibition-probe verify-tier: test-tier ENFORCEMENT (REQ-PROHIB-07 / const result = enforce.runProhibitionEnforcement( testTierProhibition, { kind: 'lint-rule', rule: 'local/no-source-grep', target: 'tests/', failFirst: true }, - { runCheck: () => ({ passed: true }) }, + { runCheck: () => ({ passed: true }), proveFailFirst: () => ({ provenFailFirst: true }) }, ); assert.equal(result.status, 'green', 'a passing wired lint-rule check must dispose green'); assert.equal(result.flagged, false, 'a green disposition must not be flagged'); diff --git a/tests/workflow-size-baseline.json b/tests/workflow-size-baseline.json index 43f85547b..d77e6d94b 100644 --- a/tests/workflow-size-baseline.json +++ b/tests/workflow-size-baseline.json @@ -85,6 +85,6 @@ "undo.md": 10431, "update.md": 21053, "validate-phase.md": 10745, - "verify-phase.md": 35362, + "verify-phase.md": 35812, "verify-work.md": 31157 }