fa41bfec5cd556d80fa35cbb6a0e74f5f0c2ee10
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tom Boucher
|
fa41bfec5c |
enhance(#3942): the emitted-drift ack is PR-lifetime data — move it to a commit trailer (#3954)
* test(#3942): failing-first suite for the emitted-drift ack commit trailer Binds 37 input classes from the phase test matrix to the behavior ADR-3942 specifies, before any of it exists. Stubs return benign empty values rather than throwing, deliberately: several rows assert that something DOES throw (cap overflow, uncomputable commit range), and a throwing stub would turn those green for the wrong reason and destroy the red. The two rows that carry the design's load: - merge-base semantics. The range is $(git merge-base base HEAD)..HEAD, not base..HEAD, because changedPaths comes from `git diff base...HEAD` (three dot). Two-dot would let the ack set and the change set disagree about which commits are this PR's. The fixture forks a topic branch, puts a trailer on each side, and asserts only the topic-side trailer is in range. - fail-closed on an uncomputable range. With fragments a depth-1 checkout passes VACUOUSLY, every fragment reading as brand-new. With trailers the range cannot be computed at all, and returning an empty set would silently disarm the gate, so it must throw. The fixture builds a genuine shallow clone rather than simulating one. Also covers the self-inflicted case: this change's own documentation quotes the trailer syntax, so an example landing at the end of a commit message would arm a live acknowledgment keyed on the literal placeholder text. Keys carrying angle brackets or whitespace are rejected. Authored per the phase artifacts 40-design.md and 50-test-matrix.md. Not yet run on the remote runner — this commit exists to be tested. Refs #3942 * chore(#3942): move the emitted-drift ack to a commit trailer Implements ADR-3942, superseding ADR-2719 section 3 and its #2789 amendment. Sections 1, 2 and 4-7 are retained: the conservation law is unchanged, only the storage of its escape hatch moved off the working tree. An acknowledgment explains one PR's ripple, and the moment that PR merges the ripple is in the base, so it can never clear anything again. It was stored in permanent shared state anyway, and every consequence of that mismatch had to be built and then maintained. The chain is #2789 -> #2914 -> #3078 -> #3842 -> #3823 -> #3875, each fix generating the next defect, ending in a scheduled sweeper whose own first PR could not merge itself. Added parseAckTrailers + renderAckTrailer (pure) and readAckTrailers (IO shell), reading Emitted-Drift-Ack-Hash: / Emitted-Drift-Ack-Growth: trailers over the merge-base range. tests/emitted-ack-trailer.test.cjs, 37 cases, written failing-first and confirmed red before any of this existed. Changed diffEmitted takes two structurally distinct key-space maps instead of one shared paths map. That closes a latent defect: the spaces were separated by convention only, so a growth key satisfied a hash lookup by naming coincidence. staleAcks now reports which space a key was declared in. REMEDIATION teaches the trailer, per space, with its example rendered through renderAckTrailer so the taught grammar cannot drift from what the parser accepts. Removed the sweep workflow, the guard-no-ack-on-next job, the standalone linter and its lint:ci entry, the fragment directory and its three spent fragments, the legacy single-file union, and the baseAck/spentAcks mechanism -- spentness is now structural, not computed. Two range properties carry the design and are pinned by tests rather than asserted: the range is merge-base scoped, matching git diff base...HEAD, so an already-merged trailer is out of range by construction; and an uncomputable range throws instead of reading as zero acknowledgments, which is the inverse of the fragment guard's vacuous pass. Three deliberate observable changes, each disclosed in the changeset: the unread runtime field is gone, the legacy file is no longer read, and cross-space excusal no longer works. Ten open PRs carry fragments and will meet a modify/delete conflict. Measured before landing and accepted deliberately; the one-line migration is in the PR body. Verified: lint:ci exit 0. Remote runner to follow on this exact sha. Refs #3942 * fix(#3942): silent trailer collapse, lost coverage, and an unbounded cap Six findings from the orthogonal review round, all fixed in place. BLOCKER -- two trailers of the same name on one commit collapsed silently. readAckTrailers built `separator=1d` where git needs `separator=%x1d`: the `separator=` value inside a %(trailers:...) placeholder is itself a pretty-format string, so the bare hex was emitted as two literal characters and the split on \x1d never matched. Two same-name trailers therefore joined into one value with errors empty -- the first reason absorbing the second entry's key. Silent truncation, the exact class MAX_ACK_TRAILERS throws to prevent. Confirmed with od -c against real git output before and after. The failing-first matrix did not catch it because its "both spaces coexist" row uses Hash plus Growth -- different trailer NAMES -- so the value separator was never exercised. Two regression tests now cover same-name trailers directly. Coverage recovered: normalizeAckReason and INVISIBLE stayed on the live path via parseAckTrailers but lost every test when the old suite was pruned. Back under test against the current surface -- all six invisible codepoints individually, whitespace collapse, trim, CRLF, and two seeded fast-check properties. Dropping any single codepoint now fails. MAX_ACK_TRAILERS counted raw trailers before de-duplication, so one trailer carried forward across rebased commits counted once per commit and could throw on a legitimate branch. Now counts distinct entries; 100 identical repeats dedupe to one. diffEmitted validated baseline, current and changedPaths but not the new ackHash/ackGrowth, so a bad shape raised an unhandled TypeError instead of an error verdict -- the same defect shape this file documents for #2778. Docs: CONTRIBUTING and TESTING-SUITES were rewritten only in their first sections; the later passages still taught fragments, git rm and the deleted guard, contradicting the new text directly above them. Finished. Also extends lint-removed-but-needed to exempt docs/adr and docs/research. That gate fails on any docs mention of a file deleted in the same diff, which makes it impossible to document a deletion in the PR performing it -- an ADR's whole job is naming what it retired. Exemption is narrow and comes with a test proving the gate still fires for a live consumer elsewhere under docs/. A guard that cannot fail is worse than no guard. Maintainer-approved. CONTEXT.md names the retired machinery by role rather than by filename: its generated projection lands in docs/, which that gate does scan. Adds docs/how-to/acknowledge-emitted-drift.md. The required docs set is Reference and Explanation, so the task quadrant can be empty with every gate green -- and this change has a real multi-step journey, including the fragment migration ten open PRs now need. lint:ci exit 0. Refs #3942 * docs(#3942): correct the duplicate-trailer rule in CONTRIBUTING Both axes of the code review independently flagged the same passage, without seeing each other's output. It claimed two declarations of the same key are always "a hard, loudly-reported error, not a silent last-wins". That is only half true, and the missing half is the one contributors hit: identical declarations -- same key, same reason -- dedupe silently, because a trailer legitimately survives a rebase and reappears on every rebased commit. Failing there would red a branch for doing nothing wrong, which is exactly why the dedup exists. Only a same-key/different-reason pair errors, and that one is a genuine ambiguity about which explanation holds. As written, the paragraph told a contributor that a rebase-carried trailer breaks the gate -- the opposite of the behavior. CONTEXT.md's parallel entry already stated it correctly; this brings CONTRIBUTING into line. Doc-only, root-level markdown. Refs #3942 * chore(#3942): backfill changeset PR number to 3954 --------- Co-authored-by: sim <sim@local> |
||
|
Tom Boucher
|
b42cb4fb29 |
fix(#3597): count scenario expectation failures in the QA gate, and fix the workstream scope split it exposed (#3607)
* fix(#3597): count scenario expectation failures in the QA ratchet gate buildReport counts totals.violations as oracle violations PLUS scenario expectFailures, but collectFindings read only step.violations. A scenario whose declared expect failed therefore produced ok:false and violations:1 in the report while the ratchet printed "0 violations" and exited 0. multi-workstream has failed that way on every CI run since 2026-08-10, when #3217 (PR #3318) made computeProgressPercent withhold a percentage whose scope is not COMPLETE. The walk detected the change the day it landed; nothing was listening. - collectFindings returns a third bucket, expectationFailures, carrying no fingerprint so it can never be baselined or acked away - both modes of main() print and gate on it; the summary line reports it - guard runMain(main) behind require.main === module, so the QA suite can require the script to test collectFindings without running a real walk (that import side effect is why the gate logic had no test) - multi-workstream now asserts the true contract: phase_scope unreadable and percent null, per ADR-3180 7.6 rule 4 - the perturbation test asserts scenario ok, closing the test-side half Closes #3597 * fix(#3597): resolve the milestone window against the active workstream listMilestonePhaseDirs defaulted its ws option to null. planningDir treats undefined as "resolve the ambient workstream" and null as "force the project root", so that default suppressed the ambient resolution every other planning-path read uses. All 18 call sites derive phasesDir ambiently via planningPaths(cwd), so the counts came from the workstream while the milestone window came from the root .planning/ROADMAP.md — the exact numerator/denominator scope split ADR-3180 7.6 rule 3 forbids. workstream create migrates that root roadmap away, so the read threw and scope stayed UNREADABLE, and rule 4 then correctly withheld the percentage. Proof: with a workstream tree byte-unchanged, copying its own ROADMAP to the project root flipped --ws alpha progress from phase_scope:unreadable/percent:null to complete/100. This is the defect the loop QA walk was pointing at all along; the scenario expectation is restored to percent:100 rather than bent to match the bug. - pass ws through as undefined so ambient resolution applies - multi-workstream asserts phase_scope complete + percent 100 - regression test in completion-ratio-scope-withholding covers a workstream-only project with no root ROADMAP - replace the vacuous require.main test: runMain defers through a promise, so the in-process timing check passed against the unguarded file too; a child-process spawn now observes the guard for real - tie the oracle-violation test to expectationFailures, and cover the absent-key, multi-scenario and zero-step report shapes in parity - flatten scenario-authored strings before rendering them into the step summary and CI logs (forged markdown / ANSI injection) - widen the scenario contract assertions past perturbation-* so multi-workstream is actually covered test-side Closes #3597 * fix(#3597): flatten scenario-authored strings on the CI-log output path The step-summary path already routed findings through flattenUntrusted; the check-mode NEW-smell and STALE-entry console.error blocks, and the repro line in both printers, still interpolated raw. detail carries a scenario-authored expect[].path verbatim, and reason/scenario/id come from contributor-authored baseline and ack fragments validated only as non-empty strings. A crafted path could print a forged summary line into the CI log directly above the real one, plus ANSI repaint and unbounded length. Exit codes are unaffected — this is log spoofing, not gate bypass. * fix(#3597): refuse to archive on an unreadable milestone window; close review gaps Resolving the milestone window against the active workstream can leave the window UNREADABLE when that workstream has no ROADMAP of its own. getMilestonePhaseFilter throws, the window degrades to a pass-all fallback, and milestone complete would then move every phase dir -- breaking the guarantee stated at the archive site that no out-of-window directory is touched. milestone complete now refuses to archive when the window is UNREADABLE and reports the refusal; --dry-run previews the same refusal from the same shared derivation. The guard is scoped to UNREADABLE, not to every non-COMPLETE scope. A broader condition regressed ordinary root projects: the QA walk caught milestone-rollover leaving 01-parser on disk, which then tripped the #1447 abort in phases clear. UNSCOPED and TRUNCATED are pre-existing classifications and keep their existing behavior. Review fixes: - the workstream regression test asserted complete/100 but its fixture wrote no workstream STATE.md, so it resolved unscoped/null and the test failed; it now asserts a milestone and genuinely fails-first - the parity test hand-supplied totals.violations, hardcoding the very formula under test; at least one case now goes through the real buildReport - drop a vacuous qa-report.json assertion (jsonOut defaults to null, so no report is written by either shape) - buildRepro emitted a repo-relative binary path after cd-ing into a temp project, so every repro died with MODULE_NOT_FOUND; it now resolves an absolute path - flattenUntrusted truncated the repro to 300 chars, handing reviewers a command that looks complete and is not; length capping is now opt-out for repro while newline/control/backtick stripping still applies * chore(#3597): backfill changeset pr number (#3607) --------- Co-authored-by: sim <sim@local> |
||
|
Tom Boucher
|
5fd5c81042 |
test(#3055): add the process seam so a subprocess timeout is expressible as data (#3066)
* test(#3055): add the process seam and route runGsdTools through it Adds tests/helpers/process-seam.cjs — runNode/runGit/runHook over spawnSync, each returning a typed discriminated union { outcome, exitCode, stdout, stderr, timedOut, signal, killed, code }. Every call is timeout-bounded; there is no unbounded path. runGsdTools becomes an adapter over the seam. Its legacy { success, output, error, exitCode } shape and retry-once-on-kill behaviour are preserved byte-identically, so none of its 136 caller files change. Outcome discrimination was corrected against probed runtime behaviour rather than assumption: a timeout and a maxBuffer overflow are identical on both status (null) and signal (SIGTERM), and differ only by code (ETIMEDOUT vs ENOBUFS). Overflow is therefore classified before timeout. This fixes a live defect — the previous isKilled() treated an overflow as a kill, retried it for a second full 60s run, and then reported "host OOM or scheduler contention" for a child that had merely printed too much. Also widens the ESLint tests glob from tests/**/*.test.cjs to tests/**/*.cjs, which brought 31 previously unlinted shared helpers under the same rules their sibling test files already obey, and fixes the 5 violations that surfaced — including a bare npm invocation without shell:true in tests/helpers/emitted-runtime.cjs (DEFECT.WINDOWS-TEST-PORTABILITY), now routed through the existing portable runNpm helper. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#3055): migrate every local spawn wrapper onto the process seam Replaces the spawn body of all 25 local runHook/runGuard/runGate definitions with a call to tests/helpers/process-seam.cjs. Each wrapper keeps its name, parameter list, return shape and post-processing (JSON parse, ANSI strip, env sanitising, field extraction) — only the spawn mechanism changes, so no test assertion moves. The 4 bash-driven wrappers use the seam's explicit `interpreter` option rather than a fourth primitive; it is explicit rather than inferred from the file extension, because guessing an interpreter from a path fails silently when a script's name does not match its shebang. Seven wrappers were previously unbounded and now carry an explicit timeout sized to what each actually runs, not the seam default. Two of those seven (gsd-write-guard, lint-docs-command-form) were absent from the issue's inventory entirely and were found by scanning after the migration. Adds the CONTEXT.md `### Process seam` glossary entry and a CONTRIBUTING.md reference section covering the three primitives, the discriminated union, and the two rules the seam enforces. Scope disclosure recorded in the phase design notes: the issue scoped three identifier names. A scan for local helpers that spawn AND return the spawn result finds 113 across 82 names, 71 of them unbounded, plus 122 unbounded direct git call sites. This change bounds 25 of those. The remaining surface is the same defect class and is NOT closed by this PR. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#3055): classify an externally-killed child as KILLED, not EXITED Blocker found in this branch's own diff, independently confirmed by an isolated reviewer. A child killed by an external signal — a genuine bench OOM kill — makes spawnSync return { status: null, signal: 'SIGKILL' } with NO .error field. The seam's "no error implies EXITED" rule therefore classified it as a clean exit, and runGsdTools returned { success: false, exitCode: 1 } without retrying. That silently defeated the #969 kill-discrimination for precisely the case it was built for: the old isKilled() fired on `signal != null`, retried once, then threw a labelled resource-starvation error. A real OOM would have been reported as an ordinary assertion failure. Adds a fifth outcome, KILLED, for "no error but a signal is set", and makes the adapter retry on TIMED_OUT or KILLED — reproducing the old `killed || signal != null || code === 'ETIMEDOUT'` condition exactly. SPAWN_FAILED still does not retry (matching the old behaviour, where signal was null). BUFFER_OVERFLOW still does not retry, which remains a deliberate divergence: the old code retried it because signal was SIGTERM, burning a second 60s run on a child that had merely printed too much. All five outcomes verified against the live runtime rather than assumed: SIGKILL -> killed, exit 0/7 -> exited, timeout -> timed_out (ETIMEDOUT), >1MB stdout -> buffer_overflow (ENOBUFS). Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#3055): address standards-review findings on this branch Three findings from the standards axis of the review, all in this branch's own diff. The CONTEXT.md glossary entry this branch introduced was already stale on the branch's own last commit: it enumerated a 4-member OUTCOME while the code had 5, because the KILLED fix did not update it. That is precisely the drift the "module changes update Domain-terms" gate exists to catch, so the entry now lists all five and explains KILLED. api-coverage-gate-e2e compared an outcome against the raw string 'exited' rather than OUTCOME.EXITED, the only such outlier; the enum is now imported and used. A sweep for the other four outcome literals found no further comparison sites. Three call sites hand the literal bash flag '-c' to the seam's first parameter, which the JSDoc described as an absolute script path. Rather than add a fourth primitive, the contract is corrected to match reality: the parameter is renamed `target` and documented as the first argv element handed to the interpreter — normally a script path, but for an interpreter invoked with an inline program it may be that interpreter's own flag. No behaviour change. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#3055): assert the cross-platform timeout contract, not the macOS one The remote runner failed on both Linux lanes (node 22 and node 24, identical) while the same tests passed locally on macOS. Two assertions encoded a platform-specific behaviour as a cross-platform guarantee. When spawnSync times out, macOS preserves the child's partial stdout/stderr; Linux discards it and returns empty strings. Verified on node v26.5.1 both ways. The seam passes through whatever spawnSync hands it and cannot manufacture output that was discarded, so the production code was correct — the tests were wrong. Both tests now assert the guarantee the seam actually makes on every platform: outcome TIMED_OUT, timedOut true, and stdout/stderr always being strings rather than undefined or a Buffer. The partial-content assertions are retained behind an explicit process.platform === 'darwin' guard so the macOS coverage is not lost, and the first test is renamed to say what it now guarantees. This is the failure mode the remote matrix exists to catch: local macOS verification would have shipped it. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(#3055): classify a failed spawn as SPAWN_FAILED, not a timeout Windows CI caught two defects the Linux matrix could not. tests/context-predicates-query.test.cjs passes a 32K-char argv value. On Windows that exceeds the argv limit and spawnSync fails with code ENAMETOOLONG, signal null, status null. The seam's fallback rule — "otherwise, status === null implies TIMED_OUT" — swallowed it, so the adapter retried a spawn that can never succeed and then threw the resource-starvation error. The old isKilled() returned false for that shape and returned an ordinary failure result. TIMED_OUT is now identified positively: code === 'ETIMEDOUT' OR signal is set. Anything else carrying an error is SPAWN_FAILED, which covers ENAMETOOLONG, E2BIG, EACCES and ENOENT alike. The signal clause is what keeps a platform whose timeout errno differs classified correctly, so the greedy catch-all is no longer needed. The second defect is a contract regression I introduced and had claimed otherwise. That same test asserts `typeof r.exitCode === 'number'`, and toLegacyShape was returning null for BUFFER_OVERFLOW and SPAWN_FAILED, so the assertion failed on type. The old code returned `err.status ?? 1` on every non-retried failure path. The adapter now returns 1 again for both, and the comment claiming "never coerced to exitCode:1, unlike the pre-seam helper" is retracted: the seam keeps the richer truth (exitCode null plus a distinct outcome), the legacy adapter keeps the old numeric contract its callers actually depend on. Verified on this host: a 4MB argv yields E2BIG -> SPAWN_FAILED; ENOENT -> SPAWN_FAILED; timeout -> TIMED_OUT; >1MB stdout -> BUFFER_OVERFLOW; SIGKILL -> KILLED; clean exit -> EXITED. Refs #3051 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: sim <sim@local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom Boucher
|
33fd203ccd |
test(#2966): loop QA walk — drive real scenarios across all five loop steps (#2976)
* test(#2966): loop QA walk — drive real scenarios across all five loop steps Adds a headless walk that carries accumulating project state across discuss -> plan -> execute -> verify -> ship against one temp project, layered over the existing tests/helpers.cjs runGsdTools substrate. Findings carry severity. A violation breaks a stated contract and fails the build; a smell is legal under today's implementation but structurally questionable, is recorded, and never reddens CI. Without that split an oracle set derived from current behavior can only ever confirm current behavior -- the harness could not say "this works and is still wrong". The end-to-end test asserts the walk produces at least one smell: a QA harness that reports nothing on a first run against a real engine is far more likely mis-specified than the engine is perfect. It deliberately does not pin smell ids or counts, which would re-freeze current behavior. First run against the real engine: 0 violations, 3 smell classes -- init returns agents_dir outside the project tree; smart-entry emits prose unconditionally so routing cannot be asserted; state-snapshot reports a missing STATE.md through a payload key with exit 0. Also fixes tests/fixtures/index.cjs: createFixture with git:true and planning:false staged nothing, so the commit failed with "nothing to commit". That combination was unreachable until greenfield needed it. Extends RULESET.TESTS.feedback-loop-convergence from estimation to the loop itself. Design lock: docs/adr/2966-loop-qa-walk.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#2966): wire fault injection, make perturbations discriminating Independent review found tests/qa/mutations.cjs entirely unwired: 462 lines exercised only by their own unit tests, with no mutation hook in the scenario DSL and no scenario applying one, while the module header and the ADR described fault injection in the present tense. Dead code documented as live. Adds a `mutate` step field, three perturbation scenarios, and a wiring detector: a self-test scenario whose expectations are known-false and which MUST fail. The previous anti-vacuity check asserted only that the walk produced a smell, which passes on well-known engine behavior regardless of whether the harness wiring works. First perturbation attempt produced zero signal -- progress does not structurally parse ROADMAP.md, so a corrupted roadmap sailed through. A perturbation that cannot fail is the same defect in a new costume. Probes now target roadmap get-phase, and each mutated step runs a clean baseline first so `mutationObserved` records whether the corruption changed anything at all. Also clears four review findings: classify() returned PROSE for exit-0 with empty stdout; `warnings` was structurally unpopulatable on the success path (execFileSync discards it) and is now documented as error-path-only; read-only-idempotence passed vacuously when asked to check idempotence without the data to check it; the ADR miscounted the oracles. Discrimination matrix across 8 mutations x 6 commands: bom, duplicate-phase-id and escaped-pipes are absorbed silently by every probed surface, and progress / smart-entry / roadmap validate never reacted to any mutation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#2966): add path-containment guard for scenario-supplied targets Security review found scenario-supplied paths joined to the temp project with no containment check. step.mutate.target and agent.write keys were validated only as non-empty strings, so a target of ../../../../etc/hosts reached fs.unlinkSync / fs.writeFileSync / fs.symlinkSync outside the project. The symlink mutation was worst: it read the traversed file, wrote a sibling copy, deleted the original and symlinked it back. Not exploitable today -- all shipped scenarios target .planning/ROADMAP.md and scenarios are repo-committed, not runtime input. Fixed anyway: it is a live primitive any future scenario or copied helper can reach. Adds tests/qa/paths.cjs with resolveWithin(): rejects absolute paths, NUL bytes and empty input, normalizes separators unconditionally, and requires containment by path segment so a sibling like <base>-evil is not treated as inside. Non-existent targets resolve via nearest existing ancestor rather than falling back to a lexical compare. Scenario load now rejects traversing or absolute targets up front. oracles.cjs previously carried its own copy of the containment logic; both now share paths.cjs, since a duplicated containment check is exactly the divergence class this repo calls out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#2966): complete trajectory corpus, report emission, boundary-aware oracle Adds the remaining trajectories and drives all 11 mutations end-to-end. 20 scenarios, 72 steps, 0 violations, 25 smells. Adds qa-report.json with per-step verdicts and a copy-pasteable repro command, plus --keep / GSD_QA_KEEP=1 to preserve a failing tree. A repro line for a tree that was not preserved is marked NOT RUNNABLE rather than emitting a command pointing at a deleted directory. monotonic-progress is now boundary-aware. Two scenarios had been trimmed to stop the oracle complaining at a milestone rollover, which destroys the signal the trajectory exists to produce. Evidence: counters legitimately reset to zero at milestone complete, but the payload milestone_version lags until a new ROADMAP.md is written. So the oracle now scopes by milestone plus workstream, keeps a same-scope decrease as a violation, and records a boundary crossing as a smell. Both scenarios walk the real boundary again. Standards review fixes: oracle findings now carry a structured subject so tests assert on typed fields instead of substring-matching the free-form detail string, resolveWithin throws a typed EPATHESCAPE error, and the absolute-path predicate scenario.cjs had re-implemented now comes from paths.cjs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#2966): fix silently-vacuous fixtures and guard the class Every fixture carried its #2371 provenance comment BEFORE the frontmatter block, and extractFrontmatter returns {} when anything precedes the opening ---. So every scenario reading status/phase/name was operating on an empty object and reporting green. Nine fixtures repositioned; the comment stays, it just moves below the closing ---. Both UAT fixtures lacked a parser-recognized result block, so evaluateUatPassed saw checks.length===0 and could never return passed:true. The uat-fail-then-remediate scenario could not have proven a remediation. Its expect block only inspected blockers, which is empty before AND after, which is why the corpus never noticed. Both fixtures now carry real result blocks and the scenario asserts passed and no_uat_artifacts on each side of the flip. The actual deliverable is the guard: a fixture-integrity block asserting every fixture with a frontmatter shape parses to a non-empty object, that every fixture carries its provenance marker, and that the two UAT fixtures produce opposite verdicts through the real evaluateUatPassed. The first guard written required --- at byte 0, which would never have fired on the regression it exists to prevent; it was rewritten and proven by deliberately re-breaking a fixture. No engine defect here. no_uat_artifacts means no parsed check items, not no UAT files, and it was reporting correctly on fixtures that had none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#2966): make the walk report — smell ratchet, baseline, CI job The harness computed smells into a gitignored qa-report.json that nothing read. In CI it surfaced nothing at all: violations failed the build, but the half of the tool that says "this works and is still wrong" was inert. A QA tool nobody hears is decoration. Adds a ratchet on the same idiom this repo already uses three times over (the regression-test-name allowlist, the emitted-drift acks, the size baseline): a committed smell-baseline.json, per-PR acknowledgment fragments under tests/qa/smell-acks/, and a ratchet script wired into CI. The design invariant is preserved exactly. A smell still never fails a build on its own merits. What fails is an UNACKNOWLEDGED NEW smell -- the absence of a decision -- leaving an author two honest exits: fix it, or record a fragment with a real reason. An empty reason is rejected. The baseline is shrink-only, so a fixed smell must prune its entry. Violations remain unacknowledgeable. Fingerprints are composed only from stable fields (oracle id, scenario, argv, subject discriminator) -- never temp paths, timestamps or counts. Verified byte-identical across two runs in separate temp dirs; an unstable fingerprint would have false-positived every CI run. CI gains a qa-loop-walk job that runs the suite and the ratchet, uploads the report with `if: always()` (it matters most when it failed), and renders a summary a reviewer reads without downloading anything. Also fixes the report runner invoking main() unconditionally on require, so importing it double-ran every scenario and clobbered its own output. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#2966): every smell terminates in a defect or a fixed detector The baseline accepted a smell with a free-text reason. That is a mechanism for designing smells in -- an allowlist nobody revisits. The harness is brand new, so nothing it found is inherited legacy; every finding is a FIRST finding. Each must now terminate in exactly one of two states: REAL -> an assigned defect, entry carries the issue number FALSE POSITIVE -> the detector is wrong and gets fixed, never baselined There is no third "accepted with a good explanation" state, so the ratchet now requires a positive-integer `issue` on every entry. A reason may remain as a human note but can never substitute. `--update` refuses to invent issue numbers: a new smell is written with `issue: null` and a TODO, and the next plain run rejects it, forcing triage rather than accumulation. Working the 21 existing entries through that rule found 16 were my own detectors being wrong: value-hygiene (10) flagged $.agents_dir, a field whose entire contract is to point at the install tree outside any project. Fixed with a leaf-key allowlist of contractually-external fields, verified as the only such key in the init payload. Genuinely unexpected out-of-project paths still smell. monotonic-progress (6) fired on legitimate boundary crossings -- milestone v1.0 to v2.0, workstream beta to alpha -- and on one payload carrying no scope fields at all, where a change cannot even be known. Scope changes now reset silently and scope-less observations are skipped. The same-scope decrease remains a violation; that is the real invariant and is regression- guarded. The five survivors are real and now tracked: soft-error-exit-zero (#2980), untyped-success (#2979). Baseline 25 -> 5. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#2966): keep the ratchet out of the tarball, unpin the qa CI job The remote matrix returned failed -- 3 unique failures, identical on node22 and node24, both root causes in this branch's own diff. The ratchet lives under scripts/, which ships in the npm tarball, and it requires three modules under tests/, which does not. In a published install it is MODULE_NOT_FOUND at load. This is exactly the class the #2858 guard was added to catch, and it caught it. Fixed the way #2858 fixed the same shape for its own repo-only CI script: a targeted files[] negation, so the ratchet stays in the repo for CI and out of the tarball. Not solved by moving or inlining the required modules -- the ratchet must keep using the same code the harness uses, or the two drift. Verified both directions: the script is no longer in the pack list, and build-hooks.js, fix-slash-commands.cjs and gen-capability-registry.cjs are all still shipped. Over-negating there would have broken installs, since bin/install.js requires them. The qa-loop-walk job also carried CI_REBASE_BASE_SHA copied from a neighbouring job without the paired GSD_EMITTED_BASE, which the #2854 invariant forbids by name: diverging them makes the differential compare a tree against a baseline from a different commit. The job runs only the qa suite and the ratchet and invokes no emitted-attribution test, so it needs no rebase-pinned base at all -- the step was removed rather than paired. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(#2966): stop monotonic-progress going blind on scope-less payloads The full remote suite caught a false NEGATIVE I introduced while fixing a false positive. Silencing the boundary-crossing noise had made the oracle skip ANY observation lacking milestone fields -- so a minimal payload like {total_summaries: n} produced no violation at all, and the oracle stopped catching the exact defect it exists to catch. For a QA tool that is strictly worse than the noise it replaced. Scope is only indeterminate when the two observations DISAGREE about having it: both scoped, same scope, decrease -> VIOLATION both scoped, different scope -> reset silently NEITHER scoped, decrease -> VIOLATION (the regression) mixed -> skip the comparison Implementing the mixed case surfaced a second blind spot: advancing the reference point on a skipped pair lets a scope-less observation sitting between two same-scope ones mask a real decrease. Mixed now leaves the reference untouched. All four branches carry explicit coverage; only one did before, which is why this shipped. The self-test that failed was right and the code was wrong, so the code moved. Corpus behavior is unchanged: still 5 smells, 0 new, 0 stale, 0 violations. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |