Files
msd-core/tests/helpers/emitted-baseline.cjs
Tom Boucher 9138271b5f test(#2723): differential emitted-attribution check, dual-run beside the golden (#2737)
* test(#2723): differential emitted-attribution check, dual-run beside the golden

Phase 3 of #2719. The conservation law itself, running BESIDE
golden-install-parity.test.cjs -- both green, fixtures untouched.

Every emitted path whose hash moved between next HEAD and PR HEAD must be
attributable, through the Phase 2 table, to a path the PR actually changed.
Unattributable deltas fail with the paths NAMED. The only way through is a
committed acknowledgment, never a flag -- a contributor facing a red gate
sets a flag, which is what UPDATE_GOLDEN=1 is today.

The central decision is that the law is a PURE function (no fs, git,
installer, or clock), with I/O confined to a separate resolver. The naive
one-big-integration-test shape would need ~38 installer spawns per assertion,
so #2723's four failing-first criteria would not in practice have been
written -- which is exactly how a phase ships promised-but-not-built. Pure,
they are millisecond table tests, and the Stryker gate can actually bite.

Buckets are conserved: every moved path lands in exactly one of
attributed | unattributable | acked, property-tested at 400 runs. A path the
provenance table cannot resolve surfaces as an error, never a silent skip.

Asymmetries that are deliberate, each with a test:
- an ADDED emitted key is a ripple too, not just a modified one
- synthesized paths are exempt; code-derived ones are NOT (Phase 2 refused to
  mark them exempt precisely because exempt means permanently blind)
- shrinkage needs no ack; growth does. Gating shrinkage would punish exactly
  what the size ratchet wants
- a STALE ack is a hard failure -- an ack outliving its ripple pre-clears the
  next one on that path
- a failed `git diff` is an explicit error, never an empty changedPaths set;
  reading it as "nothing changed" would make everything unattributable and
  produce a failure storm that reads like a real finding
- prefix sources are SEGMENT-aware, so `agents/` does not attribute
  `agentsfoo/x.md`

Baseline is cached, not committed, keyed on the next sha. A stale key is
refused rather than used: absence fails loudly and gets fixed, whereas
staleness produces a confident wrong answer. An explicitly pointed-at
GSD_EMITTED_BASELINE that is stale is a hard stop; a stale cache falls
through to the in-job build. No baseline-unavailable path returns -- ADR-2719
section 6 names that trap, since in node:test a bare return is a PASS.

Both the conservation property and the staleness gate were mutation-verified
(injecting a swallowed key fails 9 tests; disabling the staleness comparison
fails 5).

Fixtures, generators, the merge driver and the ADR status are untouched --
those are Phase 4 (#2724).

Refs #2719

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6

* test(#2723): compute stale acks once, after the size pass

Self-review defect found while the reviewers were running. `staleAcks` was
computed twice: once between the hash pass and the size pass, then again
after. Only the second value was returned, so the first was dead code -- and
the dead one was placed where it would have been WRONG.

An acknowledgment can be consumed by either a hash move or a size growth.
Computing staleness before the size pass reports a legitimate growth ack as
stale, which is a false failure that pushes a contributor to delete the very
ack that is doing its job.

Now computed once, after both passes, with a regression test. Verified by
mutation: restoring the early computation fails 2 tests.

Refs #2719

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6

* test(#2723): wire the attribution check to the real tree, not just synthetic input

An isolated reviewer caught that the first cut was INTERFACE-ONLY: nothing
read the ack file from disk, nothing shelled git, nothing built real
manifests. Every test was true of hand-built inputs and none of the repo, so
the acceptance criterion "both this check and golden-install-parity green on
the same tree" was trivially true rather than meaningfully true. That is the
promised-but-not-built failure this epic keeps finding in its predecessors,
recurring one phase later for the wiring itself. Taken, not argued.

Adds tests/helpers/emitted-runtime.cjs -- the only module that touches git,
disk, or the installer -- and an integration test that runs the same pure law
against reality:

- CURRENT side: 19 real installer spawns via runMinimalInstall +
  buildParityManifest, the same machinery the golden harness uses.
- BASELINE side: `git show origin/next:<fixture>`. That is next's RECORDED
  emitted state and it costs nothing. Deliberately NOT the working-tree
  fixtures, which are whatever this PR's author regenerated -- comparing
  against those would be vacuous. Phase 4 deletes the fixtures and swaps in
  resolveBaseline's cache path, already implemented and tested.
- changed paths from real `git diff --name-only origin/next...HEAD`, with the
  git subprocess bounded at 30s per CLAUDE.md's unbounded-subprocess rule.
- the real tests/emitted-drift-ack.json (absent is legal; present-but-empty
  or unparseable throws rather than being read as absent).

Verified it can actually fail: an uncommitted edit to a shipped workflow
moves emitted output but never appears in the committed diff, and the check
names all 18 affected emitted paths with the message format ADR-2719 §1
specifies. Restores clean.

Also from review:
- readAckFile now has a real test exercising the SUT across absent / valid /
  empty / unparseable / unreadable. The previous test asserted fs behaviour
  rather than SUT behaviour, because no SUT ack-reading path existed yet.
- formatReport's sampleLimit gains true limit-1/limit/limit+1 coverage at
  19/20/21. A test was previously NAMED "(limit+1)" while testing no numeric
  limit at all, which is worse than no coverage because it reads as covered.

Windows uses an explicit t.skip (install output is platform-specific there,
mirroring the golden harness) -- never a bare return, which node:test scores
as a PASS.

Refs #2719

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6

* test(#2723): cover the claude-local manifest family in the real-tree check

Isolated adversarial review, MAJOR. The real-tree wiring enumerated
Object.keys(RUNTIME_META) -- 18 entries -- while the emitted manifest set has
19 families. The 19th is claude-local: claude is the reference host and the
only runtime with a distinct LOCAL "legacy flat-commands" layout
(commands/gsd-*.md + agents/gsd-*.md at project scope), which
golden-install-parity.test.cjs guards with a hand-coded test outside its
RUNTIME_META loop (#2086).

The family was dropped from BOTH sides, so the test's own self-check
(current.length === baseline.length) passed vacuously at 18 === 18. A PR
changing Claude's local-scope output would have failed the golden while this
check reported ok -- and that disagreement is precisely what the dual-run
window is designed to surface as a provenance-table hole. A wiring omission
masquerading as one is the worst available failure here, because it would
have been read as evidence about Phase 2 rather than a bug in Phase 3.

Fixed by deriving MANIFEST_FAMILIES explicitly (18 global + claude-local at
local scope) instead of inferring the set from RUNTIME_META.

The self-check is also repaired: it now asserts both sides against the
INDEPENDENT EXPECTED_MANIFEST_COUNT from the Phase 2 table, and asserts
claude-local specifically. Comparing the two sides to each other can never
catch a family missing from both -- the assertion has to come from outside.
Verified by mutation: removing claude-local again fails the test.

Also from the same review:
- sourceSatisfiedBy returns the matched source string, so an empty-string
  source would return '' and the caller's `if (hit)` would silently discard a
  real match. Unreachable today (every rule source is a non-empty template)
  but a footgun for the next rule author; now `!== null`.
- the purity fixture used a single-element changedPaths array, so an in-place
  sort would have been invisible. Now three elements in unsorted order.

Refs #2719

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6

* fix(#2723): resolve the base ref tolerantly instead of hard-requiring origin/next

The first matrix run failed on both linux lanes:

  differential attribution over the real tree
  cannot resolve origin/next (Command failed: git rev-parse origin/next)

Not a flake, and not an environment excuse -- a real defect in this diff. The
gsd-test runner shallow-clones and merges base+head, so no origin/* remote-
tracking refs exist in the container. My own fail-loud path fired correctly;
what was wrong was hard-depending on that ref existing. GitHub Actions has the
same shape by default, which is exactly why changeset-required.yml carries an
explicit `git fetch origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}"`.

Now resolved through an ordered candidate list -- GSD_EMITTED_BASE (explicit
lane override), then origin/$GITHUB_BASE_REF and $GITHUB_BASE_REF, then
origin/next and next -- de-duplicated, each verified with
`rev-parse --verify <ref>^{commit}`.

When NO candidate resolves the test takes an explicit t.skip() naming every
ref it tried and stating that the gate did not run here. That is the
ADR-2719 section 6 distinction: t.skip is REPORTED as skipped, whereas a bare
return is scored as a PASS. Hard-failing was the other option and is wrong --
it would make the suite permanently red wherever a base ref cannot exist by
construction, which is a statement about the checkout, not a propagation
finding.

The candidate ordering is pinned by a unit test rather than left implicit,
since the ordering IS the fix.

Refs #2719

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 23:22:37 -04:00

193 lines
7.7 KiB
JavaScript

'use strict';
/**
* Baseline acquisition for the differential attribution check (ADR-2719 §5, #2723).
*
* The baseline is the emitted-manifest set built at `next` HEAD. It is CACHED, not
* committed — committing it would recreate the derived-state-in-git problem this epic
* exists to delete, and would double the rate `next` advances.
*
* ── The load-bearing part ────────────────────────────────────────────────────
* ADR-2719 §5: "keyed on the `next` sha the PR was merged with. That key discipline is
* the one thing that has to be exactly right — a stale baseline silently mis-attributes."
*
* Note the asymmetry that makes staleness worse than absence: a MISSING baseline fails
* loudly and gets fixed. A STALE one produces a confident wrong answer — it attributes
* deltas to the wrong commit's state, so real ripples read as explained. Every path
* through this module therefore fails closed on a key mismatch.
*
* ── No bare `return` anywhere ────────────────────────────────────────────────
* ADR-2719 §6 calls this out explicitly: in node:test a bare `return` is a PASS, not a
* skip, so a baseline-unavailable path that returns would make the whole gate fail open
* with nothing in CI to say so. This module returns an explicit {ok:false} result and
* the caller asserts on it.
*
* IO is INJECTED (readJson / exists / buildFallback) so every failure mode above is
* unit-testable without touching the filesystem or spawning 19 installers.
*/
/** Env var a CI cache-restore step points at the recovered baseline artifact. */
const BASELINE_ENV = 'GSD_EMITTED_BASELINE';
/** Default on-disk cache location, relative to the repo root. */
const DEFAULT_CACHE_PATH = '.gsd-cache/emitted-baseline.json';
/** Baseline artifact schema version — pinned so a format change fails loudly. */
const BASELINE_VERSION = 1;
/**
* Validate a baseline artifact's shape and freshness.
*
* @param {*} doc parsed artifact
* @param {string} expectedSha the `next` sha this PR is being evaluated against
* @param {string} source where it came from (named in every error)
* @returns {{ok: true, baseline: object, sizeBaseline: object|null, sha: string}
* |{ok: false, errors: string[]}}
*/
function validateBaseline(doc, expectedSha, source) {
const errors = [];
if (doc === null || doc === undefined) {
return { ok: false, errors: [`${source}: baseline is absent`] };
}
if (typeof doc !== 'object' || Array.isArray(doc)) {
// Same class as Phase 2's fixture loader: a document that parses but is not an
// object must never be read as "no entries", which would pass vacuously.
return {
ok: false,
errors: [`${source}: baseline must be a JSON object, got ${Array.isArray(doc) ? 'array' : typeof doc}`],
};
}
if (doc.version !== undefined && doc.version !== BASELINE_VERSION) {
errors.push(`${source}: unsupported baseline version ${JSON.stringify(doc.version)} (expected ${BASELINE_VERSION})`);
}
if (typeof doc.sha !== 'string' || !/^[0-9a-f]{40}$/.test(doc.sha)) {
errors.push(`${source}: baseline "sha" must be a 40-hex commit sha, got ${JSON.stringify(doc.sha)}`);
} else if (typeof expectedSha === 'string' && expectedSha !== '' && doc.sha !== expectedSha) {
// THE staleness gate. Never silently used.
errors.push(
`${source}: STALE baseline — built at ${doc.sha} but this PR is being evaluated ` +
`against next@${expectedSha}. A stale baseline mis-attributes silently, so it is ` +
'refused rather than used. Rebuild it, or let the in-job fallback run.',
);
}
if (!doc.manifests || typeof doc.manifests !== 'object' || Array.isArray(doc.manifests)) {
errors.push(`${source}: baseline "manifests" must be an object keyed by runtime`);
}
if (errors.length) return { ok: false, errors };
return {
ok: true,
baseline: doc.manifests,
sizeBaseline: (doc.sizes && typeof doc.sizes === 'object' && !Array.isArray(doc.sizes))
? doc.sizes
: null,
sha: doc.sha,
};
}
/**
* Resolve the baseline through the documented precedence, reporting WHICH step supplied
* it so a failure message can say where the answer came from.
*
* 1. `GSD_EMITTED_BASELINE` — explicit path (CI cache restore writes here)
* 2. the on-disk cache, validated against the expected sha
* 3. an in-job build at `origin/next` (slow fallback)
* 4. none → explicit failure (NEVER a silent pass)
*
* @param {object} opts
* @param {string} opts.expectedSha `next` sha under test
* @param {object} [opts.env] environment (injected)
* @param {string} [opts.cachePath]
* @param {function} opts.readJson (path) => parsed | null (null when absent)
* @param {function} [opts.buildFallback] () => artifact | null (the slow path)
* @returns {{ok: true, via: string, baseline: object, sizeBaseline: object|null, sha: string}
* |{ok: false, via: string, errors: string[]}}
*/
function resolveBaseline({
expectedSha,
env = process.env,
cachePath = DEFAULT_CACHE_PATH,
readJson,
buildFallback = null,
} = {}) {
if (typeof readJson !== 'function') {
return { ok: false, via: 'none', errors: ['resolveBaseline: readJson must be supplied'] };
}
const attempts = [];
const envPath = env && env[BASELINE_ENV];
if (envPath) {
let doc = null;
let readError = null;
try {
doc = readJson(envPath);
} catch (err) {
readError = `${BASELINE_ENV}=${envPath}: ${err.message}`;
}
if (readError) {
attempts.push(readError);
} else {
const v = validateBaseline(doc, expectedSha, `${BASELINE_ENV}=${envPath}`);
if (v.ok) return { ok: true, via: `env:${BASELINE_ENV}`, ...v };
attempts.push(...v.errors);
// An EXPLICITLY pointed-at baseline that is stale or malformed is a hard stop, not
// a reason to quietly fall through to a different one — the operator said "use this".
return { ok: false, via: `env:${BASELINE_ENV}`, errors: attempts };
}
}
let cacheDoc = null;
let cacheErr = null;
try {
cacheDoc = readJson(cachePath);
} catch (err) {
cacheErr = `${cachePath}: ${err.message}`;
}
if (cacheErr) {
attempts.push(cacheErr);
} else if (cacheDoc !== null && cacheDoc !== undefined) {
const v = validateBaseline(cacheDoc, expectedSha, cachePath);
if (v.ok) return { ok: true, via: `cache:${cachePath}`, ...v };
// A stale CACHE is recoverable: fall through to the build fallback, but keep the
// reason so the final message explains why the slow path ran.
attempts.push(...v.errors);
} else {
attempts.push(`${cachePath}: absent`);
}
if (typeof buildFallback === 'function') {
let built = null;
try {
built = buildFallback();
} catch (err) {
attempts.push(`in-job build at origin/next failed: ${err.message}`);
return { ok: false, via: 'build', errors: attempts };
}
const v = validateBaseline(built, expectedSha, 'in-job build at origin/next');
if (v.ok) return { ok: true, via: 'build', ...v };
attempts.push(...v.errors);
return { ok: false, via: 'build', errors: attempts };
}
attempts.push(
'no baseline available and no in-job build fallback was supplied. This is a hard ' +
'failure on purpose: a skipped propagation gate is worth less than a slow one ' +
'(ADR-2719 §6), and in node:test a bare `return` is a PASS, not a skip.',
);
return { ok: false, via: 'none', errors: attempts };
}
module.exports = {
BASELINE_ENV,
BASELINE_VERSION,
DEFAULT_CACHE_PATH,
validateBaseline,
resolveBaseline,
};