fix(1259-01): address trek-e review — B1 (fatal/suppressed), B2 (bounded subprocess), M1/M2, minors
Maintainer CHANGES_REQUESTED (reviewede08667e5, pre-portability-fix): - B1: lint-rule no longer greens an unparseable target (eslintHasFatalError -> fail closed on any fatal/parse error) or an inline-suppressed violation (eslintJsonHasRule now scans suppressedMessages too). RED-first + real-runner repros. - B2: both child spawns get a bounded timeout (30s node / 60s eslint) + 16MiB maxBuffer; timeout fails closed. Injectable timeoutMs (positive-only — 0/negative can't disable the bound) enables a fast 1.5s hang test. - M1: scoped verify-phase.md — the check descriptor is author-supplied for now; filed #1278 for deterministic auto-locate of the descriptor (the locate half). - M2: added tests/prohibition-enforcement.property.test.cjs (fast-check fail-closed invariants; within the <=2-file budget). - m1: tapTestNames excludes # SKIP/# TODO; parseNodeTestSummary tracks # cancelled; isNonVacuousNodeTestPass requires cancelled===0. - m2: scoped the determinism claim to the decision/parse layer (real runner is env-dependent). - m3: filed #1279 for machine-proven fail-first (violation-fixture probe). - n1: -- before target in both arg builders (option-injection). n2: dropped dead token. - B3 (Windows npx) was already fixed in2af76306(pushed ~65s after the review). Verified node 22 + 24; size baseline regenerated for the verify-phase note.
This commit is contained in:
Dave
@@ -5,6 +5,6 @@ pr: 1273
|
||||
|
||||
**Test-tier prohibitions are now a real, provable gate instead of a permanent, unsatisfiable `gaps_found`** — the deferred ENFORCEMENT half of ADR-550 Decision 5d (the "heavy half" that #644 / PR #1149 deferred) has landed. A new deterministic `check prohibition-enforcement` sub-command (authored as `src/prohibition-enforcement.cts`, compiled by `build:lib` to the gitignored `gsd-core/bin/lib/prohibition-enforcement.cjs`) is the missing PRODUCER: it locates the wired mechanical check, runs it for a genuine **non-vacuous** pass, builds `enforcementEvidence`, and emits the `dispositionForProhibition()` verdict. The previously-unreachable green branch in `dispositionForProhibition()` is now reachable from the live pipeline — a test-tier prohibition with a genuinely-passing wired check disposes `green` and can reach `passed`, while a missing, non-attested, or non-passing check hard-gates (flagged, never green → `gaps_found`) in BOTH interactive and autonomous modes (ADR-550 D4 / D3). `verify-phase.md` wires the consumer; the green/fail-closed policy in `src/probe-core.cts` is untouched. Both wired-check kinds are accepted (ADR-550 D2): a `node --test` negative test (requiring a real reported test — an empty file, which `node --test` counts as one passing "test", does NOT green) AND a lint/AST rule run as `eslint --format json` filtered by `ruleId` (so plugin rules like `local/*` load — bare `--rule` cannot), anchored on the in-tree `local/no-source-grep` rule (dogfooding, ADR-550 D4). This enforcement seam is the concrete instance of ADR-857 open-question §147 and lands on the core verify rail, never in `capabilities/` (D6). (#1259)
|
||||
|
||||
**Honest scope — `failFirst` is caller-attested, not yet machine-proven.** This lands the *execution + non-vacuous-pass* half: the producer requires the caller to attest `failFirst: true` and the check to genuinely run and pass. It does NOT yet independently prove the check *fails-on-violation* (the literal `regression-must-fail-first` property) — cheap proof of that at verify time needs running the check against a known violation fixture, which is a **tracked follow-up**. The red-first property currently rests on caller attestation, surfaced transparently in the evidence record.
|
||||
**Honest scope — `failFirst` is caller-attested, not yet machine-proven.** This lands the *execution + non-vacuous-pass* half: the producer requires the caller to attest `failFirst: true` and the check to genuinely run and pass. It does NOT yet independently prove the check *fails-on-violation* (the literal `regression-must-fail-first` property) — cheap proof of that at verify time needs running the check against a known violation fixture, which is a **tracked follow-up (#1279)**. The red-first property currently rests on caller attestation, surfaced transparently in the evidence record.
|
||||
|
||||
**Correction to the issue body (#1259):** the issue's "96 invalid/error negative-proof cases" figure is wrong. For the `no-source-grep` anchor specifically, the genuine `regression-must-fail-first` proofs are its **two `invalid` cases** (the `.includes()` and `.match()` blocks) in `tests/eslint-rules.test.cjs` — not 96. The anchor argument is unaffected (those two cases ARE real fail-first proofs); only the count was off.
|
||||
|
||||
@@ -3180,6 +3180,6 @@ The load-bearing wire is the `plan-phase` lift into `must_haves.prohibitions`, s
|
||||
- REQ-PROHIB-04: `--auto` MUST never auto-dismiss.
|
||||
- REQ-PROHIB-05: `plan-phase` MUST lift resolved prohibitions into `must_haves.prohibitions` (never `truths`).
|
||||
- REQ-PROHIB-06: A well-formed but unwired `test`-tier prohibition MUST fail closed at verify time — never a silent pass.
|
||||
- REQ-PROHIB-07: A `test`-tier prohibition with a caller-attested, genuinely-passing (non-vacuous) wired mechanical check (a `node --test` negative test OR a lint/AST rule) MUST dispose green and be satisfiable; a missing, non-attested, or non-passing check MUST hard-gate (flagged, non-green) in both interactive and autonomous modes (#1259, ADR-550 D5d — the enforcement half; machine-proven fail-first is a tracked follow-up).
|
||||
- REQ-PROHIB-07: A `test`-tier prohibition with a caller-attested, genuinely-passing (non-vacuous) wired mechanical check (a `node --test` negative test OR a lint/AST rule) MUST dispose green and be satisfiable; a missing, non-attested, or non-passing check MUST hard-gate (flagged, non-green) in both interactive and autonomous modes (#1259, ADR-550 D5d — the enforcement half; machine-proven fail-first is tracked in #1279, deterministic descriptor auto-locate in #1278).
|
||||
|
||||
**Reference:** [Prohibition Probe](../gsd-core/references/prohibition-probe.md)
|
||||
|
||||
@@ -91,7 +91,7 @@ Decision 4 describes the `test`-tier as a "**Hard gate in both interactive and a
|
||||
|
||||
- A well-formed but **unwired** `test`-tier prohibition resolves via `dispositionForProhibition()` to `{ status: 'unverified', flagged: true }` — **provably never green** without explicit evidence (REQ-PROHIB-06). This is the load-bearing safety half and it holds today.
|
||||
- The **negative-test enforcement mechanism** — locating the wired mechanical check, running it for a genuine **non-vacuous** pass, and building the `enforcementEvidence` that flips a passing test-tier item green — **landed in #1259** as the deterministic `check prohibition-enforcement` sub-command (authored as `src/prohibition-enforcement.cts`, compiled by `build:lib` to the gitignored `gsd-core/bin/lib/prohibition-enforcement.cjs`). It accepts BOTH wired-check kinds — a `node --test` negative test (requiring a real reported test, not the empty file `node --test` would count as one passing "test") OR a lint/AST rule run through the project flat config as `eslint --format json` filtered by `ruleId` (so plugin rules like `local/*` load — bare `--rule` cannot) — and is anchored on the in-tree `local/no-source-grep` rule (dogfooding the existing must-NOT proof, ADR-550 D4; the #644 corpus had zero authored test-tier prohibitions, so no contrived consumer was minted). A passing wired check disposes green; a missing, non-attested, or genuinely-non-passing check hard-gates (flagged, non-green) in both interactive and autonomous modes.
|
||||
- **Honest scope — `failFirst` is caller-ATTESTED, not machine-proven (tracked follow-up).** What #1259 lands is the *execution + non-vacuous-pass* half: the producer requires the caller to attest `failFirst: true` and requires the check to genuinely run and pass. It does **not** yet independently prove the check *fails-on-violation* (the literal `regression-must-fail-first` property), because cheap proof of that at verify time needs running the check against a known **violation fixture** — deferred as a follow-up. Until then the red-first property rests on caller attestation, surfaced transparently in the evidence record. This closes the permanent-`gaps_found` dead-end with a genuinely-executed gate without overclaiming machine-proven fail-first.
|
||||
- **Honest scope — `failFirst` is caller-ATTESTED, not machine-proven (tracked follow-up).** What #1259 lands is the *execution + non-vacuous-pass* half: the producer requires the caller to attest `failFirst: true` and requires the check to genuinely run and pass. It does **not** yet independently prove the check *fails-on-violation* (the literal `regression-must-fail-first` property), because cheap proof of that at verify time needs running the check against a known **violation fixture** — deferred as a follow-up (#1279; the descriptor auto-locate half is #1278). Until then the red-first property rests on caller attestation, surfaced transparently in the evidence record. This closes the permanent-`gaps_found` dead-end with a genuinely-executed gate without overclaiming machine-proven fail-first.
|
||||
|
||||
Net effect on D4: the *guarantee* ("a `test`-tier prohibition is never a silent pass") was preserved through the fail-closed-now half and is now joined by the genuine-execution half — a test-tier prohibition with a passing, non-vacuous wired check can reach `green`/`passed`, and a missing/failing one hard-gates. The previously-unreachable green branch in `dispositionForProhibition()` is reachable from the live pipeline, and the fail-closed default backs every miss/fail. The one remaining gap to D4's literal intent — *machine-proven* fail-first — is documented above as a tracked follow-up. The decision also lives in `src/probe-core.cts` comments, `src/prohibition-enforcement.cts`, `verify-phase.md`, and the #644 / #1259 changesets.
|
||||
|
||||
|
||||
@@ -80,6 +80,8 @@ Aggregate all must_haves across plans for phase-level verification.
|
||||
- **`status: 'green'`, `flagged: false`** (a genuinely-passing wired negative test / lint rule, `located: true`, non-empty `evidence`) → the item is satisfiable → it can reach **passed**.
|
||||
- **missing, non-attested, or genuinely-non-passing check** (`located: false` OR `status: 'unverified'`, `flagged: true`) → **hard-gate**: disposes flagged-unverified, NEVER green, routing to `gaps_found` in BOTH interactive and autonomous modes (a failing mechanical check blocks even AFK; ADR-550 D4 / D3). The deterministic fail-closed default backing every miss/fail is `dispositionForProhibition()` in probe-core (`status: 'unverified'`, `flagged: true` on empty `enforcementEvidence`).
|
||||
|
||||
> **Descriptor authoring — current scope (#1259).** The `check` descriptor is **supplied by the phase author / verifier**; there is no projection field yet that deterministically derives `{ kind, target, rule, failFirst }` from a prohibition in `must_haves.prohibitions` (which carries only `{ statement, status, verification }`). So #1259 lands the **deterministic run+verdict half** (locate→run→evidence→disposition, all CI-testable) while the **locate→descriptor half is author-provided** for now. Deterministic auto-locate — so a wired passing test closes the gap with zero manual descriptor authoring — is a **tracked follow-up: #1278**. Until then, the green path requires the author to wire the descriptor explicitly.
|
||||
|
||||
**Option B: Use Success Criteria from ROADMAP.md**
|
||||
|
||||
If no must_haves in frontmatter (MUST_HAVES returns error or empty), check for Success Criteria:
|
||||
|
||||
@@ -29,10 +29,13 @@
|
||||
* `tsc -p tsconfig.build.json` (`npm run build:lib`) to the gitignored runtime artifact
|
||||
* `gsd-core/bin/lib/prohibition-enforcement.cjs`. Do NOT hand-write the `.cjs`; it is emitted.
|
||||
*
|
||||
* The function is PURE/deterministic (same input -> same output, no LLM, mutation-survivable): the
|
||||
* actual check execution is delegated to an injectable `runCheck` (defaults to a real runner) so
|
||||
* the contract is unit-testable without spawning a process — mirroring the injectable I/O pattern
|
||||
* in `runProbeCli` / `ProbeCliOptions`.
|
||||
* DETERMINISM SCOPE: the DECISION layer is pure/deterministic and no-LLM — given a `runCheck` result
|
||||
* the disposition is same-input-same-output and mutation-survivable, and the parse/filter helpers
|
||||
* (`parseNodeTestSummary`, `tapTestNames`, `eslintJsonHasRule`, `eslintHasFatalError`, …) are pure.
|
||||
* The DEFAULT REAL runner is NOT pure — it spawns `node --test` / eslint, so its result depends on the
|
||||
* environment (eslint version + flat config, node version, the target file). That is why the runner is
|
||||
* an injectable seam (`runCheck`): the contract is unit-tested against injected results, mirroring the
|
||||
* injectable I/O pattern in `runProbeCli` / `ProbeCliOptions`.
|
||||
*/
|
||||
|
||||
import fs from 'node:fs';
|
||||
@@ -90,6 +93,9 @@ export interface EnforcementOptions {
|
||||
mode?: string;
|
||||
/** Project root for the default real runner (defaults to process.cwd()). */
|
||||
cwd?: string;
|
||||
/** Override the per-kind subprocess timeout (ms); defaults to 30s (node-test) / 60s (eslint).
|
||||
* Injected in tests to prove the fail-closed-on-timeout bound without a 30s wait. */
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
/** The producer's verdict: the disposition PLUS the located/kind/evidence provenance. */
|
||||
@@ -100,17 +106,19 @@ export interface EnforcementResult extends ProhibitionDisposition {
|
||||
mode?: string;
|
||||
}
|
||||
|
||||
/** node --test argv. Forces the TAP reporter so the summary counts are parseable + version-stable. */
|
||||
/** node --test argv. Forces the TAP reporter so the summary counts are parseable + version-stable;
|
||||
* `--` before the target so a target starting with `-` is not parsed as a flag (option-injection). */
|
||||
export function buildNodeTestArgs(check: CheckDescriptor): string[] {
|
||||
return ['--test', '--test-reporter=tap', check.target];
|
||||
return ['--test', '--test-reporter=tap', '--', check.target];
|
||||
}
|
||||
|
||||
/** eslint argv (the args AFTER the eslint CLI path). Runs the project flat config so plugin rules
|
||||
* (e.g. `local/*`) load — `--rule` CANNOT load a plugin, so we lint the TARGET path as JSON and
|
||||
* filter by rule id. `--no-warn-ignored` makes an eslint-IGNORED target return `[]` (not a length-1
|
||||
* "File ignored" result) so an ignored path fails closed via the vacuity guard, not a false green. */
|
||||
* "File ignored" result) so an ignored path fails closed via the vacuity guard, not a false green.
|
||||
* `--` before the target so a target starting with `-` is not parsed as a flag (option-injection). */
|
||||
export function buildLintArgs(check: CheckDescriptor): string[] {
|
||||
return ['--no-warn-ignored', '--format', 'json', check.target];
|
||||
return ['--no-warn-ignored', '--format', 'json', '--', check.target];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -140,7 +148,7 @@ function baseOf(p: string): string {
|
||||
* (an empty / all-skipped / deleted-negative-test file exits 0 with `# tests 0`). Mutation-pinned by
|
||||
* unit tests so a threshold flip is caught.
|
||||
*/
|
||||
export function parseNodeTestSummary(out: string): { tests: number; pass: number; fail: number } {
|
||||
export function parseNodeTestSummary(out: string): { tests: number; pass: number; fail: number; cancelled: number } {
|
||||
const num = (re: RegExp): number => {
|
||||
const m = typeof out === 'string' ? out.match(re) : null;
|
||||
return m ? Number(m[1]) : 0;
|
||||
@@ -149,17 +157,22 @@ export function parseNodeTestSummary(out: string): { tests: number; pass: number
|
||||
tests: num(/^# tests (\d+)/m),
|
||||
pass: num(/^# pass (\d+)/m),
|
||||
fail: num(/^# fail (\d+)/m),
|
||||
cancelled: num(/^# cancelled (\d+)/m),
|
||||
};
|
||||
}
|
||||
|
||||
/** The names from TAP `ok N - <name>` / `not ok N - <name>` lines (directives like `# SKIP` stripped). */
|
||||
/** The names of REAL (run) tests from TAP `ok N - <name>` / `not ok N - <name>` lines. A line with a
|
||||
* `# SKIP` / `# TODO` directive is EXCLUDED — a skipped/todo negative test never executed, so it must
|
||||
* not count toward non-vacuity (#1259 m1). */
|
||||
export function tapTestNames(out: string): string[] {
|
||||
if (typeof out !== 'string') return [];
|
||||
const names: string[] = [];
|
||||
const re = /^(?:not )?ok \d+ - (.+)$/gm;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = re.exec(out)) !== null) {
|
||||
names.push(m[1].replace(/\s+#\s.*$/, '').trim());
|
||||
const rest = m[1];
|
||||
if (/\s#\s*(?:SKIP|TODO)\b/i.test(rest)) continue; // skipped/todo did not run
|
||||
names.push(rest.replace(/\s+#\s.*$/, '').trim());
|
||||
}
|
||||
return names;
|
||||
}
|
||||
@@ -178,7 +191,8 @@ export function tapTestNames(out: string): string[] {
|
||||
*/
|
||||
export function isNonVacuousNodeTestPass(out: string, target: string): boolean {
|
||||
const s = parseNodeTestSummary(out);
|
||||
if (!(s.tests >= 1 && s.pass >= 1 && s.fail === 0)) return false;
|
||||
// >=1 test, >=1 pass, ZERO failures AND ZERO cancelled (a cancelled run is not a clean pass, m1).
|
||||
if (!(s.tests >= 1 && s.pass >= 1 && s.fail === 0 && s.cancelled === 0)) return false;
|
||||
// Compare BASENAMES: node reports the file-test by varying path forms across OS / node version
|
||||
// (absolute, relative, normalized), so an exact-string compare misfires. A real test name (e.g.
|
||||
// "guards the must-NOT") has no separators, so its basename never equals the target file's.
|
||||
@@ -196,8 +210,37 @@ export function eslintFileResultCount(jsonText: string): number {
|
||||
}
|
||||
}
|
||||
|
||||
/** True if the eslint `--format json` report has ANY message for `rule`. Unparseable -> true
|
||||
* (fail-closed: treat an unreadable report as a violation rather than a silent pass). */
|
||||
/** Messages array of a single eslint file-result (empty if absent / wrong shape). */
|
||||
function eslintMessages(file: unknown, key: 'messages' | 'suppressedMessages'): Array<{ ruleId?: unknown; fatal?: unknown }> {
|
||||
return file && typeof file === 'object' && Array.isArray((file as Record<string, unknown>)[key])
|
||||
? ((file as Record<string, unknown>)[key] as Array<{ ruleId?: unknown; fatal?: unknown }>)
|
||||
: [];
|
||||
}
|
||||
|
||||
/**
|
||||
* True if the eslint `--format json` report has a FATAL / parse error — meaning the rule never got
|
||||
* to run on the target. A prohibition gate must fail closed on "the rule didn't execute" (#1259 B1),
|
||||
* NOT treat a length-1 fatal result as "clean". Unparseable report -> true (fail-closed).
|
||||
*/
|
||||
export function eslintHasFatalError(jsonText: string): boolean {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(jsonText);
|
||||
} catch {
|
||||
return true;
|
||||
}
|
||||
if (!Array.isArray(parsed)) return true;
|
||||
for (const file of parsed) {
|
||||
const fec = file && typeof file === 'object' ? (file as { fatalErrorCount?: unknown }).fatalErrorCount : undefined;
|
||||
if (typeof fec === 'number' && fec > 0) return true;
|
||||
if (eslintMessages(file, 'messages').some((m) => m && m.fatal === true)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** True if the eslint `--format json` report has ANY message for `rule` — in EITHER `messages` or
|
||||
* `suppressedMessages` (an inline `// eslint-disable` of the rule is still a violation, #1259 B1).
|
||||
* Unparseable -> true (fail-closed: treat an unreadable report as a violation, not a silent pass). */
|
||||
export function eslintJsonHasRule(jsonText: string, rule: string): boolean {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
@@ -207,11 +250,10 @@ export function eslintJsonHasRule(jsonText: string, rule: string): boolean {
|
||||
}
|
||||
if (!Array.isArray(parsed)) return true;
|
||||
for (const file of parsed) {
|
||||
const messages = file && typeof file === 'object' && Array.isArray((file as { messages?: unknown }).messages)
|
||||
? (file as { messages: Array<{ ruleId?: unknown }> }).messages
|
||||
: [];
|
||||
for (const msg of messages) {
|
||||
if (msg && typeof msg === 'object' && msg.ruleId === rule) return true;
|
||||
for (const key of ['messages', 'suppressedMessages'] as const) {
|
||||
for (const msg of eslintMessages(file, key)) {
|
||||
if (msg && typeof msg === 'object' && msg.ruleId === rule) return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
return false;
|
||||
@@ -246,7 +288,21 @@ function childEnv(): NodeJS.ProcessEnv {
|
||||
return env;
|
||||
}
|
||||
|
||||
function defaultRunCheck(check: CheckDescriptor, cwd: string): CheckRunResult {
|
||||
// Bounded subprocess limits (DEFECT.UNBOUNDED-SUBPROCESS): a stuck wired test / eslint must not hang
|
||||
// verify forever. On timeout `execFileSync` throws -> caught -> fail-closed (degraded, non-passing).
|
||||
// `maxBuffer` caps output so a runaway producer throws (safe direction) rather than OOMs the verifier.
|
||||
const NODE_TEST_TIMEOUT_MS = 30_000;
|
||||
const ESLINT_TIMEOUT_MS = 60_000;
|
||||
const CHECK_MAX_BUFFER = 16 * 1024 * 1024;
|
||||
|
||||
/** Resolve the effective timeout: only a POSITIVE override is honored — `0` (which Node treats as
|
||||
* "no timeout") or a negative value falls back to the bounded default, so the subprocess is ALWAYS
|
||||
* bounded (a `timeoutMs: 0` injection can never disable the bound). */
|
||||
function posTimeout(timeoutMs: number | undefined, def: number): number {
|
||||
return typeof timeoutMs === 'number' && timeoutMs > 0 ? timeoutMs : def;
|
||||
}
|
||||
|
||||
function defaultRunCheck(check: CheckDescriptor, cwd: string, timeoutMs?: number): CheckRunResult {
|
||||
try {
|
||||
if (check.kind === 'node-test') {
|
||||
let out = '';
|
||||
@@ -257,10 +313,12 @@ function defaultRunCheck(check: CheckDescriptor, cwd: string): CheckRunResult {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
windowsHide: true,
|
||||
env: childEnv(),
|
||||
timeout: posTimeout(timeoutMs, NODE_TEST_TIMEOUT_MS),
|
||||
maxBuffer: CHECK_MAX_BUFFER,
|
||||
});
|
||||
} catch (e) {
|
||||
// A failing test run exits non-zero (TAP still on stdout). Parse it: a real failure has
|
||||
// `# fail >= 1` -> non-vacuous check returns false. Missing node -> no stdout -> false.
|
||||
// A failing/timed-out run exits non-zero or is killed (partial TAP on stdout, no `# pass`
|
||||
// summary). Parse what we have: a real failure or timeout -> not a non-vacuous pass -> false.
|
||||
const stdout = e && typeof e === 'object' && 'stdout' in e ? (e as { stdout?: unknown }).stdout : '';
|
||||
out = typeof stdout === 'string' ? stdout : '';
|
||||
}
|
||||
@@ -277,14 +335,21 @@ function defaultRunCheck(check: CheckDescriptor, cwd: string): CheckRunResult {
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
windowsHide: true,
|
||||
env: childEnv(),
|
||||
timeout: posTimeout(timeoutMs, ESLINT_TIMEOUT_MS),
|
||||
maxBuffer: CHECK_MAX_BUFFER,
|
||||
});
|
||||
} catch (e) {
|
||||
// eslint exits non-zero when ANY error is present; the JSON report is still on stdout.
|
||||
// A timeout/kill leaves no parseable JSON -> eslintHasFatalError(unparseable) -> fail-closed.
|
||||
const stdout = e && typeof e === 'object' && 'stdout' in e ? (e as { stdout?: unknown }).stdout : '';
|
||||
json = typeof stdout === 'string' ? stdout : '';
|
||||
}
|
||||
// PASS requires: the target actually linted (>=1 file result), NO fatal/parse error (the rule
|
||||
// must have RUN — #1259 B1), and ZERO messages for the rule (in messages OR suppressedMessages).
|
||||
const lintedSomething = eslintFileResultCount(json) >= 1;
|
||||
return { passed: lintedSomething && !eslintJsonHasRule(json, check.rule as string) };
|
||||
return {
|
||||
passed: lintedSomething && !eslintHasFatalError(json) && !eslintJsonHasRule(json, check.rule as string),
|
||||
};
|
||||
}
|
||||
// Unknown kind — defensive; the LOCATE guard already rejects it.
|
||||
return { passed: false };
|
||||
@@ -327,7 +392,7 @@ export function runProhibitionEnforcement(
|
||||
return { ...disposition, located: false, kind: null, evidence: [], ...(mode ? { mode } : {}) };
|
||||
}
|
||||
|
||||
const runCheck = options.runCheck ?? ((toRun: CheckDescriptor) => defaultRunCheck(toRun, options.cwd ?? process.cwd()));
|
||||
const runCheck = options.runCheck ?? ((toRun: CheckDescriptor) => defaultRunCheck(toRun, options.cwd ?? process.cwd(), options.timeoutMs));
|
||||
|
||||
// (2) ATTEST fail-first (CALLER-DECLARED) + RUN. The caller must attest `failFirst: true` AND the
|
||||
// runner must observe a genuine NON-VACUOUS pass. The producer does NOT independently prove
|
||||
|
||||
60
tests/prohibition-enforcement.property.test.cjs
Normal file
60
tests/prohibition-enforcement.property.test.cjs
Normal file
@@ -0,0 +1,60 @@
|
||||
// Property-based tests for the prohibition-enforcement parsing/transform helpers (#1259, ADR-550 D5d).
|
||||
// RULESET.TESTS.property-based-testing: the producer is a parsing module (parseNodeTestSummary,
|
||||
// tapTestNames, eslintJsonHasRule, eslintHasFatalError, eslintFileResultCount), so it carries
|
||||
// fast-check invariants — especially the fail-closed safety invariants of the verify-time gate.
|
||||
'use strict';
|
||||
process.env.GSD_TEST_MODE = '1';
|
||||
|
||||
const { test, describe } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const path = require('node:path');
|
||||
const fc = require('./helpers/fast-check-setup.cjs');
|
||||
|
||||
const ENFORCEMENT_LIB = path.join(__dirname, '..', 'gsd-core', 'bin', 'lib', 'prohibition-enforcement.cjs');
|
||||
|
||||
describe('prohibition-enforcement properties (#1259)', () => {
|
||||
test('parseNodeTestSummary never throws and returns non-negative integer counts', () => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
fc.assert(fc.property(fc.string(), (s) => {
|
||||
const r = enforce.parseNodeTestSummary(s);
|
||||
for (const k of ['tests', 'pass', 'fail', 'cancelled']) {
|
||||
assert.ok(Number.isInteger(r[k]) && r[k] >= 0, `${k} is a non-negative integer`);
|
||||
}
|
||||
}));
|
||||
});
|
||||
|
||||
test('isNonVacuousNodeTestPass FAIL-CLOSED: a run with any failure or cancellation never greens', () => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
fc.assert(fc.property(
|
||||
fc.nat({ max: 50 }), fc.integer({ min: 1, max: 50 }), fc.nat({ max: 50 }), fc.string(),
|
||||
(tests, failOrCancel, pass, name) => {
|
||||
// A summary with fail>=1 (and, separately, cancelled>=1) must NEVER be a non-vacuous pass,
|
||||
// regardless of the reported test name.
|
||||
const failing = `ok 1 - ${name}\n# tests ${tests + 1}\n# pass ${pass}\n# fail ${failOrCancel}\n# cancelled 0\n`;
|
||||
assert.equal(enforce.isNonVacuousNodeTestPass(failing, 'neg.test.cjs'), false);
|
||||
const cancelled = `ok 1 - ${name}\n# tests ${tests + 1}\n# pass ${pass}\n# fail 0\n# cancelled ${failOrCancel}\n`;
|
||||
assert.equal(enforce.isNonVacuousNodeTestPass(cancelled, 'neg.test.cjs'), false);
|
||||
},
|
||||
));
|
||||
});
|
||||
|
||||
test('eslintJsonHasRule / eslintHasFatalError FAIL-CLOSED on any non-JSON / non-array input', () => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
fc.assert(fc.property(fc.string(), (s) => {
|
||||
// Only exercise strings that are NOT a valid JSON array (the unreadable-report branch).
|
||||
let isArray = false;
|
||||
try { isArray = Array.isArray(JSON.parse(s)); } catch { isArray = false; }
|
||||
fc.pre(!isArray);
|
||||
assert.equal(enforce.eslintJsonHasRule(s, 'local/no-source-grep'), true, 'unreadable report -> violation (fail-closed)');
|
||||
assert.equal(enforce.eslintHasFatalError(s), true, 'unreadable report -> fatal (fail-closed)');
|
||||
}));
|
||||
});
|
||||
|
||||
test('eslintFileResultCount never throws and is non-negative', () => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
fc.assert(fc.property(fc.string(), (s) => {
|
||||
const n = enforce.eslintFileResultCount(s);
|
||||
assert.ok(Number.isInteger(n) && n >= 0);
|
||||
}));
|
||||
});
|
||||
});
|
||||
@@ -1,7 +1,3 @@
|
||||
// allow-test-rule: runtime-contract-is-the-product (#1259) — the test-tier enforcement PRODUCER is
|
||||
// the deployed verify-time gate; these assertions pin its deterministic locate/fail-first/run/
|
||||
// evidence-construction contract to the code (ADR-550 D5d).
|
||||
//
|
||||
// Behavioral tests for the deterministic prohibition-enforcement producer (#1259, ADR-550 D5d
|
||||
// "heavy half"). Requires the BUILT gsd-core/bin/lib/prohibition-enforcement.cjs — authored as
|
||||
// src/prohibition-enforcement.cts and compiled by `npm run build:lib` (mirrors how the verify-tier
|
||||
@@ -221,16 +217,32 @@ describe('prohibition-enforcement: deterministic test-tier producer (#1259 / ADR
|
||||
// weakens "non-vacuous pass" or the ruleId filter is caught — the contract the injected-runner tests
|
||||
// above deliberately bypass.
|
||||
describe('prohibition-enforcement real-runner helpers (#1259)', () => {
|
||||
test('parseNodeTestSummary extracts the TAP tests/pass/fail counts', () => {
|
||||
test('parseNodeTestSummary extracts the TAP tests/pass/fail/cancelled counts', () => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
assert.deepEqual(enforce.parseNodeTestSummary('# tests 3\n# pass 2\n# fail 1\n'), { tests: 3, pass: 2, fail: 1 });
|
||||
assert.deepEqual(enforce.parseNodeTestSummary('no summary here'), { tests: 0, pass: 0, fail: 0 });
|
||||
assert.deepEqual(enforce.parseNodeTestSummary('# tests 3\n# pass 2\n# fail 1\n# cancelled 1\n'),
|
||||
{ tests: 3, pass: 2, fail: 1, cancelled: 1 });
|
||||
assert.deepEqual(enforce.parseNodeTestSummary('no summary here'), { tests: 0, pass: 0, fail: 0, cancelled: 0 });
|
||||
});
|
||||
|
||||
test('tapTestNames extracts ok/not-ok names, stripping directives', () => {
|
||||
test('tapTestNames EXCLUDES skipped/todo tests (they never ran, m1)', () => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
assert.deepEqual(enforce.tapTestNames('ok 1 - guards the must-NOT\nnot ok 2 - other # SKIP\n'),
|
||||
['guards the must-NOT', 'other']);
|
||||
assert.deepEqual(enforce.tapTestNames('ok 1 - guards the must-NOT\nok 2 - other # SKIP\nok 3 - later # TODO\n'),
|
||||
['guards the must-NOT'], 'a # SKIP / # TODO test is not a real run and must not count');
|
||||
});
|
||||
|
||||
test('isNonVacuousNodeTestPass: a SKIPPED negative test (file wrapper passes) is NOT a pass (m1)', () => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
// file wrapper + a skipped negative test: pass>=1 but the only named test is skipped -> vacuous.
|
||||
const skipped = 'ok 1 - empty.test.cjs\nok 2 - the negative test # SKIP\n# tests 2\n# pass 2\n# fail 0\n# cancelled 0\n';
|
||||
assert.equal(enforce.isNonVacuousNodeTestPass(skipped, 'empty.test.cjs'), false,
|
||||
'a skipped negative test never executed -> must not green');
|
||||
});
|
||||
|
||||
test('isNonVacuousNodeTestPass: a CANCELLED run is not a pass (m1)', () => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
const cancelled = 'ok 1 - guards\n# tests 1\n# pass 1\n# fail 0\n# cancelled 1\n';
|
||||
assert.equal(enforce.isNonVacuousNodeTestPass(cancelled, 'neg.test.cjs'), false,
|
||||
'a cancelled run is not a clean pass');
|
||||
});
|
||||
|
||||
test('isNonVacuousNodeTestPass: an empty file (node names the test after the file) is NOT a pass (BL-01)', () => {
|
||||
@@ -271,6 +283,22 @@ describe('prohibition-enforcement real-runner helpers (#1259)', () => {
|
||||
assert.equal(enforce.eslintFileResultCount('[]'), 0);
|
||||
assert.equal(enforce.eslintFileResultCount('garbage'), 0);
|
||||
});
|
||||
|
||||
test('eslintHasFatalError: a parse/fatal error must fail closed (B1)', () => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
const fatal = JSON.stringify([{ messages: [{ ruleId: null, fatal: true, severity: 2, message: 'Parsing error' }], fatalErrorCount: 1 }]);
|
||||
assert.equal(enforce.eslintHasFatalError(fatal), true, 'a fatal/parse error means the rule never ran -> fail closed');
|
||||
const clean = JSON.stringify([{ messages: [], fatalErrorCount: 0 }]);
|
||||
assert.equal(enforce.eslintHasFatalError(clean), false, 'a clean lint has no fatal error');
|
||||
assert.equal(enforce.eslintHasFatalError('not json'), true, 'an unreadable report is treated as fatal (fail closed)');
|
||||
});
|
||||
|
||||
test('eslintJsonHasRule also reads suppressedMessages — an inline-disabled violation still counts (B1)', () => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
const suppressed = JSON.stringify([{ messages: [], suppressedMessages: [{ ruleId: 'local/no-source-grep' }] }]);
|
||||
assert.equal(enforce.eslintJsonHasRule(suppressed, 'local/no-source-grep'), true,
|
||||
'a violation suppressed via // eslint-disable must NOT be treated as clean');
|
||||
});
|
||||
});
|
||||
|
||||
// ─── Real runner end-to-end (NO injected runCheck; #1259 SF-02 / BL-01 / SF-01) ──
|
||||
@@ -296,6 +324,23 @@ describe('prohibition-enforcement REAL runner end-to-end (#1259)', () => {
|
||||
assert.equal(result.evidence.length, 1);
|
||||
});
|
||||
|
||||
test('a HANGING node-test fails closed via the bounded timeout (B2: no unbounded subprocess)', (t) => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
const dir = createTempDir('prohib-hang-');
|
||||
t.after(() => cleanup(dir));
|
||||
const tf = path.join(dir, 'hang.test.cjs');
|
||||
// A test that never returns; the bounded timeout must kill it and dispose non-green.
|
||||
fs.writeFileSync(tf,
|
||||
"const { test } = require('node:test');\ntest('hangs forever', () => { while (true) {} });\n");
|
||||
const result = enforce.runProhibitionEnforcement(
|
||||
TEST_TIER,
|
||||
{ kind: 'node-test', target: tf, failFirst: true },
|
||||
{ cwd: dir, timeoutMs: 1500 },
|
||||
);
|
||||
assert.notEqual(result.status, 'green', 'a hung check must be killed and fail closed — never hang verify or green');
|
||||
assert.equal(result.located, true);
|
||||
});
|
||||
|
||||
test('an EMPTY node-test file (exit 0, zero tests) does NOT green via the real runner (BL-01)', (t) => {
|
||||
const enforce = require(ENFORCEMENT_LIB);
|
||||
const dir = createTempDir('prohib-real-empty-');
|
||||
|
||||
@@ -85,6 +85,6 @@
|
||||
"undo.md": 10431,
|
||||
"update.md": 21053,
|
||||
"validate-phase.md": 10745,
|
||||
"verify-phase.md": 32334,
|
||||
"verify-phase.md": 33076,
|
||||
"verify-work.md": 31157
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user