f7df920681f233ae0fe064ee659550bdf41ff708
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tom Boucher
|
9448736872 |
fix(#3547): exercise the real global config-home shape in the install harness (#3567)
* test(#3547): failing-first regression for collapsed global install shape * fix(#3547): exercise the real global config-home shape in the install harness * test(#3547): align ripple suites with the real global install shape * test(#3547): update stale collapsed-shape pins in provenance and migration suites * fix(#3547): bump emitted-baseline schema version for the real install shape --------- Co-authored-by: sim <sim@local> |
||
|
Tom Boucher
|
aa19e3478c |
fix(#2854): pin the emitted gate to the base the tree was merged with (#2859)
* test(#2854): failing-first coverage for CI baseline export provenance Extracts the export decision out of main() behind injected IO so it is unit-testable, preserving today's export-whenever-present semantics, and adds the matrix that proves those semantics are wrong. The PR lane restores the emitted baseline keyed on the PR's recorded base sha while the gate resolves the base ref live, so the two drift whenever next advances mid-flight. The restore was published straight to GSD_EMITTED_BASELINE, where a mismatch is fatal, turning a recoverable cache into a hard failure on diffs that touched nothing related. Also renames the stale-env fixture from 'from-cache-restore.json' to an operator-pin name: that fixture asserted the exact conflation this bug is, documenting the defect as intended behavior. Refs #2854 * fix(#2854): validate a restored baseline before publishing it as an operator pin GSD_EMITTED_BASELINE is an operator pin: resolveBaseline() treats a mismatch there as a hard stop, because the operator said "use this one". CI published its cache restore to that same variable whenever the file merely existed, so a restore keyed on the PR's recorded base sha - while the gate resolves the base ref live - turned a recoverable cache into a fatal error whenever next advanced mid-flight. Required tests went red on diffs that touched nothing related, and named a test file the contributor never opened. The export step is the boundary, so it is the boundary that validates. It now publishes only a baseline already valid for the sha under test, judged by validateBaseline so the staleness rule keeps one definition. Anything refused is left to be found via the cache path, where a mismatch degrades to the in-job build exactly as ADR-2719 SS5 specifies. The operator hard stop is untouched, and the fast path still hits on a current cache. Also reports the sources actually reached rather than asserting all three ran: the failure message claimed an in-job build it had returned before calling, sending contributors after a rebuild that never happened. Fixes #2854 * fix(#2854): pin the emitted gate to the base the tree was actually merged with The differential compared a tree built on one commit against a baseline at a different one. "Rebase check" merges pull_request.base.sha, pinned by #2472 so all 12 matrix jobs agree on one tree, but resolveBase() fell through to origin/next, which fetch-depth 0 leaves at the live tip. Nothing set GSD_EMITTED_BASE, so whenever next advanced mid-flight the two disagreed. The cached baseline, keyed on base.sha, was correct for that tree and was rejected as STALE by a target that was not. Required tests went red on diffs that touched nothing related, naming a test file the contributor never opened. The near miss is the worse half: had resolution gotten past the baseline step, a baseline at the live tip would have attributed commits merged to next in between to the PR under test. The hard stop was shielding us from a wrong answer, so making it fall through would have made this worse. Pins GSD_EMITTED_BASE to the same expression as CI_REBASE_BASE_SHA in every rebase-merged job, with a parity test asserting the two cannot diverge. That parity check immediately caught a third lane, test-inert, that merges a pinned base and had been missed. Fixes #2854 * fix(#2854): keep the export step self-contained across the package boundary scripts/ ships in the npm tarball and tests/ does not, so requiring the validator across that boundary is MODULE_NOT_FOUND in a published install. The export step now reads the same GSD_EMITTED_BASE pin the gate resolves through, and applies a cheap self-contained precondition; validateBaseline remains the sole authority and still runs downstream on whatever is published, so there is no second opinion to drift. Reading the pin rather than re-deriving a base is the point: a second, divergent base lookup is exactly what caused this bug. The same hazard pre-exists in scripts/gen-emitted-baseline.cjs, which ships and requires three tests/ modules. Filed as #2858 rather than folded in: fixing it means relocating the shared helpers out of tests/ and updating every consumer, which would bury this change. Refs #2854 * fix(#2854): stop announcing the restored cache through the operator-pin door Two independent reviewers found the same blocker in the previous approach. Validating before publishing to GSD_EMITTED_BASELINE only narrowed the hole: the precondition gated on sha equality alone, so a document with a MATCHING sha but a wrong schema version or malformed manifests was still announced as an operator pin and still hard-stopped downstream. That reproduces this bug's own class, triggered by malformation instead of staleness. The step was never load-bearing. The cache restores to .gsd-cache/emitted-baseline.json, which is resolveBaseline's DEFAULT_CACHE_PATH and is read whether or not anything announces it. Publishing the same file to the pin door could only ever convert recoverable into fatal, so the step and its script are deleted rather than made cleverer. Every failure mode now degrades to the in-job build by construction, and validateBaseline is once again the only thing that judges a baseline. Coverage moves to where the behaviour lives: stale sha, wrong schema version, manifests array/absent, non-object documents, unreadable file, and the 39/40/41 hex boundary all assert degradation via the cache path. Adds the empty-pin case a reviewer flagged as untested - the pin is job-level env, so on push events it renders as an empty string, and only baseRefCandidates' truthy check keeps it out of the candidate list. Fixes #2854 --------- Co-authored-by: Test <test@example.com> |
||
|
Tom Boucher
|
9138271b5f |
test(#2723): differential emitted-attribution check, dual-run beside the golden (#2737)
* test(#2723): differential emitted-attribution check, dual-run beside the golden Phase 3 of #2719. The conservation law itself, running BESIDE golden-install-parity.test.cjs -- both green, fixtures untouched. Every emitted path whose hash moved between next HEAD and PR HEAD must be attributable, through the Phase 2 table, to a path the PR actually changed. Unattributable deltas fail with the paths NAMED. The only way through is a committed acknowledgment, never a flag -- a contributor facing a red gate sets a flag, which is what UPDATE_GOLDEN=1 is today. The central decision is that the law is a PURE function (no fs, git, installer, or clock), with I/O confined to a separate resolver. The naive one-big-integration-test shape would need ~38 installer spawns per assertion, so #2723's four failing-first criteria would not in practice have been written -- which is exactly how a phase ships promised-but-not-built. Pure, they are millisecond table tests, and the Stryker gate can actually bite. Buckets are conserved: every moved path lands in exactly one of attributed | unattributable | acked, property-tested at 400 runs. A path the provenance table cannot resolve surfaces as an error, never a silent skip. Asymmetries that are deliberate, each with a test: - an ADDED emitted key is a ripple too, not just a modified one - synthesized paths are exempt; code-derived ones are NOT (Phase 2 refused to mark them exempt precisely because exempt means permanently blind) - shrinkage needs no ack; growth does. Gating shrinkage would punish exactly what the size ratchet wants - a STALE ack is a hard failure -- an ack outliving its ripple pre-clears the next one on that path - a failed `git diff` is an explicit error, never an empty changedPaths set; reading it as "nothing changed" would make everything unattributable and produce a failure storm that reads like a real finding - prefix sources are SEGMENT-aware, so `agents/` does not attribute `agentsfoo/x.md` Baseline is cached, not committed, keyed on the next sha. A stale key is refused rather than used: absence fails loudly and gets fixed, whereas staleness produces a confident wrong answer. An explicitly pointed-at GSD_EMITTED_BASELINE that is stale is a hard stop; a stale cache falls through to the in-job build. No baseline-unavailable path returns -- ADR-2719 section 6 names that trap, since in node:test a bare return is a PASS. Both the conservation property and the staleness gate were mutation-verified (injecting a swallowed key fails 9 tests; disabling the staleness comparison fails 5). Fixtures, generators, the merge driver and the ADR status are untouched -- those are Phase 4 (#2724). Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * test(#2723): compute stale acks once, after the size pass Self-review defect found while the reviewers were running. `staleAcks` was computed twice: once between the hash pass and the size pass, then again after. Only the second value was returned, so the first was dead code -- and the dead one was placed where it would have been WRONG. An acknowledgment can be consumed by either a hash move or a size growth. Computing staleness before the size pass reports a legitimate growth ack as stale, which is a false failure that pushes a contributor to delete the very ack that is doing its job. Now computed once, after both passes, with a regression test. Verified by mutation: restoring the early computation fails 2 tests. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * test(#2723): wire the attribution check to the real tree, not just synthetic input An isolated reviewer caught that the first cut was INTERFACE-ONLY: nothing read the ack file from disk, nothing shelled git, nothing built real manifests. Every test was true of hand-built inputs and none of the repo, so the acceptance criterion "both this check and golden-install-parity green on the same tree" was trivially true rather than meaningfully true. That is the promised-but-not-built failure this epic keeps finding in its predecessors, recurring one phase later for the wiring itself. Taken, not argued. Adds tests/helpers/emitted-runtime.cjs -- the only module that touches git, disk, or the installer -- and an integration test that runs the same pure law against reality: - CURRENT side: 19 real installer spawns via runMinimalInstall + buildParityManifest, the same machinery the golden harness uses. - BASELINE side: `git show origin/next:<fixture>`. That is next's RECORDED emitted state and it costs nothing. Deliberately NOT the working-tree fixtures, which are whatever this PR's author regenerated -- comparing against those would be vacuous. Phase 4 deletes the fixtures and swaps in resolveBaseline's cache path, already implemented and tested. - changed paths from real `git diff --name-only origin/next...HEAD`, with the git subprocess bounded at 30s per CLAUDE.md's unbounded-subprocess rule. - the real tests/emitted-drift-ack.json (absent is legal; present-but-empty or unparseable throws rather than being read as absent). Verified it can actually fail: an uncommitted edit to a shipped workflow moves emitted output but never appears in the committed diff, and the check names all 18 affected emitted paths with the message format ADR-2719 §1 specifies. Restores clean. Also from review: - readAckFile now has a real test exercising the SUT across absent / valid / empty / unparseable / unreadable. The previous test asserted fs behaviour rather than SUT behaviour, because no SUT ack-reading path existed yet. - formatReport's sampleLimit gains true limit-1/limit/limit+1 coverage at 19/20/21. A test was previously NAMED "(limit+1)" while testing no numeric limit at all, which is worse than no coverage because it reads as covered. Windows uses an explicit t.skip (install output is platform-specific there, mirroring the golden harness) -- never a bare return, which node:test scores as a PASS. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * test(#2723): cover the claude-local manifest family in the real-tree check Isolated adversarial review, MAJOR. The real-tree wiring enumerated Object.keys(RUNTIME_META) -- 18 entries -- while the emitted manifest set has 19 families. The 19th is claude-local: claude is the reference host and the only runtime with a distinct LOCAL "legacy flat-commands" layout (commands/gsd-*.md + agents/gsd-*.md at project scope), which golden-install-parity.test.cjs guards with a hand-coded test outside its RUNTIME_META loop (#2086). The family was dropped from BOTH sides, so the test's own self-check (current.length === baseline.length) passed vacuously at 18 === 18. A PR changing Claude's local-scope output would have failed the golden while this check reported ok -- and that disagreement is precisely what the dual-run window is designed to surface as a provenance-table hole. A wiring omission masquerading as one is the worst available failure here, because it would have been read as evidence about Phase 2 rather than a bug in Phase 3. Fixed by deriving MANIFEST_FAMILIES explicitly (18 global + claude-local at local scope) instead of inferring the set from RUNTIME_META. The self-check is also repaired: it now asserts both sides against the INDEPENDENT EXPECTED_MANIFEST_COUNT from the Phase 2 table, and asserts claude-local specifically. Comparing the two sides to each other can never catch a family missing from both -- the assertion has to come from outside. Verified by mutation: removing claude-local again fails the test. Also from the same review: - sourceSatisfiedBy returns the matched source string, so an empty-string source would return '' and the caller's `if (hit)` would silently discard a real match. Unreachable today (every rule source is a non-empty template) but a footgun for the next rule author; now `!== null`. - the purity fixture used a single-element changedPaths array, so an in-place sort would have been invisible. Now three elements in unsorted order. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * fix(#2723): resolve the base ref tolerantly instead of hard-requiring origin/next The first matrix run failed on both linux lanes: differential attribution over the real tree cannot resolve origin/next (Command failed: git rev-parse origin/next) Not a flake, and not an environment excuse -- a real defect in this diff. The gsd-test runner shallow-clones and merges base+head, so no origin/* remote- tracking refs exist in the container. My own fail-loud path fired correctly; what was wrong was hard-depending on that ref existing. GitHub Actions has the same shape by default, which is exactly why changeset-required.yml carries an explicit `git fetch origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}"`. Now resolved through an ordered candidate list -- GSD_EMITTED_BASE (explicit lane override), then origin/$GITHUB_BASE_REF and $GITHUB_BASE_REF, then origin/next and next -- de-duplicated, each verified with `rev-parse --verify <ref>^{commit}`. When NO candidate resolves the test takes an explicit t.skip() naming every ref it tried and stating that the gate did not run here. That is the ADR-2719 section 6 distinction: t.skip is REPORTED as skipped, whereas a bare return is scored as a PASS. Hard-failing was the other option and is wrong -- it would make the suite permanently red wherever a base ref cannot exist by construction, which is a statement about the checkout, not a propagation finding. The candidate ordering is pinned by a unit test rather than left implicit, since the ordering IS the fix. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |