* test(#2723): differential emitted-attribution check, dual-run beside the golden Phase 3 of #2719. The conservation law itself, running BESIDE golden-install-parity.test.cjs -- both green, fixtures untouched. Every emitted path whose hash moved between next HEAD and PR HEAD must be attributable, through the Phase 2 table, to a path the PR actually changed. Unattributable deltas fail with the paths NAMED. The only way through is a committed acknowledgment, never a flag -- a contributor facing a red gate sets a flag, which is what UPDATE_GOLDEN=1 is today. The central decision is that the law is a PURE function (no fs, git, installer, or clock), with I/O confined to a separate resolver. The naive one-big-integration-test shape would need ~38 installer spawns per assertion, so #2723's four failing-first criteria would not in practice have been written -- which is exactly how a phase ships promised-but-not-built. Pure, they are millisecond table tests, and the Stryker gate can actually bite. Buckets are conserved: every moved path lands in exactly one of attributed | unattributable | acked, property-tested at 400 runs. A path the provenance table cannot resolve surfaces as an error, never a silent skip. Asymmetries that are deliberate, each with a test: - an ADDED emitted key is a ripple too, not just a modified one - synthesized paths are exempt; code-derived ones are NOT (Phase 2 refused to mark them exempt precisely because exempt means permanently blind) - shrinkage needs no ack; growth does. Gating shrinkage would punish exactly what the size ratchet wants - a STALE ack is a hard failure -- an ack outliving its ripple pre-clears the next one on that path - a failed `git diff` is an explicit error, never an empty changedPaths set; reading it as "nothing changed" would make everything unattributable and produce a failure storm that reads like a real finding - prefix sources are SEGMENT-aware, so `agents/` does not attribute `agentsfoo/x.md` Baseline is cached, not committed, keyed on the next sha. A stale key is refused rather than used: absence fails loudly and gets fixed, whereas staleness produces a confident wrong answer. An explicitly pointed-at GSD_EMITTED_BASELINE that is stale is a hard stop; a stale cache falls through to the in-job build. No baseline-unavailable path returns -- ADR-2719 section 6 names that trap, since in node:test a bare return is a PASS. Both the conservation property and the staleness gate were mutation-verified (injecting a swallowed key fails 9 tests; disabling the staleness comparison fails 5). Fixtures, generators, the merge driver and the ADR status are untouched -- those are Phase 4 (#2724). Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * test(#2723): compute stale acks once, after the size pass Self-review defect found while the reviewers were running. `staleAcks` was computed twice: once between the hash pass and the size pass, then again after. Only the second value was returned, so the first was dead code -- and the dead one was placed where it would have been WRONG. An acknowledgment can be consumed by either a hash move or a size growth. Computing staleness before the size pass reports a legitimate growth ack as stale, which is a false failure that pushes a contributor to delete the very ack that is doing its job. Now computed once, after both passes, with a regression test. Verified by mutation: restoring the early computation fails 2 tests. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * test(#2723): wire the attribution check to the real tree, not just synthetic input An isolated reviewer caught that the first cut was INTERFACE-ONLY: nothing read the ack file from disk, nothing shelled git, nothing built real manifests. Every test was true of hand-built inputs and none of the repo, so the acceptance criterion "both this check and golden-install-parity green on the same tree" was trivially true rather than meaningfully true. That is the promised-but-not-built failure this epic keeps finding in its predecessors, recurring one phase later for the wiring itself. Taken, not argued. Adds tests/helpers/emitted-runtime.cjs -- the only module that touches git, disk, or the installer -- and an integration test that runs the same pure law against reality: - CURRENT side: 19 real installer spawns via runMinimalInstall + buildParityManifest, the same machinery the golden harness uses. - BASELINE side: `git show origin/next:<fixture>`. That is next's RECORDED emitted state and it costs nothing. Deliberately NOT the working-tree fixtures, which are whatever this PR's author regenerated -- comparing against those would be vacuous. Phase 4 deletes the fixtures and swaps in resolveBaseline's cache path, already implemented and tested. - changed paths from real `git diff --name-only origin/next...HEAD`, with the git subprocess bounded at 30s per CLAUDE.md's unbounded-subprocess rule. - the real tests/emitted-drift-ack.json (absent is legal; present-but-empty or unparseable throws rather than being read as absent). Verified it can actually fail: an uncommitted edit to a shipped workflow moves emitted output but never appears in the committed diff, and the check names all 18 affected emitted paths with the message format ADR-2719 §1 specifies. Restores clean. Also from review: - readAckFile now has a real test exercising the SUT across absent / valid / empty / unparseable / unreadable. The previous test asserted fs behaviour rather than SUT behaviour, because no SUT ack-reading path existed yet. - formatReport's sampleLimit gains true limit-1/limit/limit+1 coverage at 19/20/21. A test was previously NAMED "(limit+1)" while testing no numeric limit at all, which is worse than no coverage because it reads as covered. Windows uses an explicit t.skip (install output is platform-specific there, mirroring the golden harness) -- never a bare return, which node:test scores as a PASS. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * test(#2723): cover the claude-local manifest family in the real-tree check Isolated adversarial review, MAJOR. The real-tree wiring enumerated Object.keys(RUNTIME_META) -- 18 entries -- while the emitted manifest set has 19 families. The 19th is claude-local: claude is the reference host and the only runtime with a distinct LOCAL "legacy flat-commands" layout (commands/gsd-*.md + agents/gsd-*.md at project scope), which golden-install-parity.test.cjs guards with a hand-coded test outside its RUNTIME_META loop (#2086). The family was dropped from BOTH sides, so the test's own self-check (current.length === baseline.length) passed vacuously at 18 === 18. A PR changing Claude's local-scope output would have failed the golden while this check reported ok -- and that disagreement is precisely what the dual-run window is designed to surface as a provenance-table hole. A wiring omission masquerading as one is the worst available failure here, because it would have been read as evidence about Phase 2 rather than a bug in Phase 3. Fixed by deriving MANIFEST_FAMILIES explicitly (18 global + claude-local at local scope) instead of inferring the set from RUNTIME_META. The self-check is also repaired: it now asserts both sides against the INDEPENDENT EXPECTED_MANIFEST_COUNT from the Phase 2 table, and asserts claude-local specifically. Comparing the two sides to each other can never catch a family missing from both -- the assertion has to come from outside. Verified by mutation: removing claude-local again fails the test. Also from the same review: - sourceSatisfiedBy returns the matched source string, so an empty-string source would return '' and the caller's `if (hit)` would silently discard a real match. Unreachable today (every rule source is a non-empty template) but a footgun for the next rule author; now `!== null`. - the purity fixture used a single-element changedPaths array, so an in-place sort would have been invisible. Now three elements in unsorted order. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * fix(#2723): resolve the base ref tolerantly instead of hard-requiring origin/next The first matrix run failed on both linux lanes: differential attribution over the real tree cannot resolve origin/next (Command failed: git rev-parse origin/next) Not a flake, and not an environment excuse -- a real defect in this diff. The gsd-test runner shallow-clones and merges base+head, so no origin/* remote- tracking refs exist in the container. My own fail-loud path fired correctly; what was wrong was hard-depending on that ref existing. GitHub Actions has the same shape by default, which is exactly why changeset-required.yml carries an explicit `git fetch origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}"`. Now resolved through an ordered candidate list -- GSD_EMITTED_BASE (explicit lane override), then origin/$GITHUB_BASE_REF and $GITHUB_BASE_REF, then origin/next and next -- de-duplicated, each verified with `rev-parse --verify <ref>^{commit}`. When NO candidate resolves the test takes an explicit t.skip() naming every ref it tried and stating that the gate did not run here. That is the ADR-2719 section 6 distinction: t.skip is REPORTED as skipped, whereas a bare return is scored as a PASS. Hard-failing was the other option and is wrong -- it would make the suite permanently red wherever a base ref cannot exist by construction, which is a statement about the checkout, not a propagation finding. The candidate ordering is pinned by a unit test rather than left implicit, since the ordering IS the fix. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
193 lines
7.7 KiB
JavaScript
193 lines
7.7 KiB
JavaScript
'use strict';
|
|
|
|
/**
|
|
* Baseline acquisition for the differential attribution check (ADR-2719 §5, #2723).
|
|
*
|
|
* The baseline is the emitted-manifest set built at `next` HEAD. It is CACHED, not
|
|
* committed — committing it would recreate the derived-state-in-git problem this epic
|
|
* exists to delete, and would double the rate `next` advances.
|
|
*
|
|
* ── The load-bearing part ────────────────────────────────────────────────────
|
|
* ADR-2719 §5: "keyed on the `next` sha the PR was merged with. That key discipline is
|
|
* the one thing that has to be exactly right — a stale baseline silently mis-attributes."
|
|
*
|
|
* Note the asymmetry that makes staleness worse than absence: a MISSING baseline fails
|
|
* loudly and gets fixed. A STALE one produces a confident wrong answer — it attributes
|
|
* deltas to the wrong commit's state, so real ripples read as explained. Every path
|
|
* through this module therefore fails closed on a key mismatch.
|
|
*
|
|
* ── No bare `return` anywhere ────────────────────────────────────────────────
|
|
* ADR-2719 §6 calls this out explicitly: in node:test a bare `return` is a PASS, not a
|
|
* skip, so a baseline-unavailable path that returns would make the whole gate fail open
|
|
* with nothing in CI to say so. This module returns an explicit {ok:false} result and
|
|
* the caller asserts on it.
|
|
*
|
|
* IO is INJECTED (readJson / exists / buildFallback) so every failure mode above is
|
|
* unit-testable without touching the filesystem or spawning 19 installers.
|
|
*/
|
|
|
|
/** Env var a CI cache-restore step points at the recovered baseline artifact. */
|
|
const BASELINE_ENV = 'GSD_EMITTED_BASELINE';
|
|
|
|
/** Default on-disk cache location, relative to the repo root. */
|
|
const DEFAULT_CACHE_PATH = '.gsd-cache/emitted-baseline.json';
|
|
|
|
/** Baseline artifact schema version — pinned so a format change fails loudly. */
|
|
const BASELINE_VERSION = 1;
|
|
|
|
/**
|
|
* Validate a baseline artifact's shape and freshness.
|
|
*
|
|
* @param {*} doc parsed artifact
|
|
* @param {string} expectedSha the `next` sha this PR is being evaluated against
|
|
* @param {string} source where it came from (named in every error)
|
|
* @returns {{ok: true, baseline: object, sizeBaseline: object|null, sha: string}
|
|
* |{ok: false, errors: string[]}}
|
|
*/
|
|
function validateBaseline(doc, expectedSha, source) {
|
|
const errors = [];
|
|
|
|
if (doc === null || doc === undefined) {
|
|
return { ok: false, errors: [`${source}: baseline is absent`] };
|
|
}
|
|
if (typeof doc !== 'object' || Array.isArray(doc)) {
|
|
// Same class as Phase 2's fixture loader: a document that parses but is not an
|
|
// object must never be read as "no entries", which would pass vacuously.
|
|
return {
|
|
ok: false,
|
|
errors: [`${source}: baseline must be a JSON object, got ${Array.isArray(doc) ? 'array' : typeof doc}`],
|
|
};
|
|
}
|
|
|
|
if (doc.version !== undefined && doc.version !== BASELINE_VERSION) {
|
|
errors.push(`${source}: unsupported baseline version ${JSON.stringify(doc.version)} (expected ${BASELINE_VERSION})`);
|
|
}
|
|
|
|
if (typeof doc.sha !== 'string' || !/^[0-9a-f]{40}$/.test(doc.sha)) {
|
|
errors.push(`${source}: baseline "sha" must be a 40-hex commit sha, got ${JSON.stringify(doc.sha)}`);
|
|
} else if (typeof expectedSha === 'string' && expectedSha !== '' && doc.sha !== expectedSha) {
|
|
// THE staleness gate. Never silently used.
|
|
errors.push(
|
|
`${source}: STALE baseline — built at ${doc.sha} but this PR is being evaluated ` +
|
|
`against next@${expectedSha}. A stale baseline mis-attributes silently, so it is ` +
|
|
'refused rather than used. Rebuild it, or let the in-job fallback run.',
|
|
);
|
|
}
|
|
|
|
if (!doc.manifests || typeof doc.manifests !== 'object' || Array.isArray(doc.manifests)) {
|
|
errors.push(`${source}: baseline "manifests" must be an object keyed by runtime`);
|
|
}
|
|
|
|
if (errors.length) return { ok: false, errors };
|
|
|
|
return {
|
|
ok: true,
|
|
baseline: doc.manifests,
|
|
sizeBaseline: (doc.sizes && typeof doc.sizes === 'object' && !Array.isArray(doc.sizes))
|
|
? doc.sizes
|
|
: null,
|
|
sha: doc.sha,
|
|
};
|
|
}
|
|
|
|
/**
|
|
* Resolve the baseline through the documented precedence, reporting WHICH step supplied
|
|
* it so a failure message can say where the answer came from.
|
|
*
|
|
* 1. `GSD_EMITTED_BASELINE` — explicit path (CI cache restore writes here)
|
|
* 2. the on-disk cache, validated against the expected sha
|
|
* 3. an in-job build at `origin/next` (slow fallback)
|
|
* 4. none → explicit failure (NEVER a silent pass)
|
|
*
|
|
* @param {object} opts
|
|
* @param {string} opts.expectedSha `next` sha under test
|
|
* @param {object} [opts.env] environment (injected)
|
|
* @param {string} [opts.cachePath]
|
|
* @param {function} opts.readJson (path) => parsed | null (null when absent)
|
|
* @param {function} [opts.buildFallback] () => artifact | null (the slow path)
|
|
* @returns {{ok: true, via: string, baseline: object, sizeBaseline: object|null, sha: string}
|
|
* |{ok: false, via: string, errors: string[]}}
|
|
*/
|
|
function resolveBaseline({
|
|
expectedSha,
|
|
env = process.env,
|
|
cachePath = DEFAULT_CACHE_PATH,
|
|
readJson,
|
|
buildFallback = null,
|
|
} = {}) {
|
|
if (typeof readJson !== 'function') {
|
|
return { ok: false, via: 'none', errors: ['resolveBaseline: readJson must be supplied'] };
|
|
}
|
|
|
|
const attempts = [];
|
|
|
|
const envPath = env && env[BASELINE_ENV];
|
|
if (envPath) {
|
|
let doc = null;
|
|
let readError = null;
|
|
try {
|
|
doc = readJson(envPath);
|
|
} catch (err) {
|
|
readError = `${BASELINE_ENV}=${envPath}: ${err.message}`;
|
|
}
|
|
if (readError) {
|
|
attempts.push(readError);
|
|
} else {
|
|
const v = validateBaseline(doc, expectedSha, `${BASELINE_ENV}=${envPath}`);
|
|
if (v.ok) return { ok: true, via: `env:${BASELINE_ENV}`, ...v };
|
|
attempts.push(...v.errors);
|
|
// An EXPLICITLY pointed-at baseline that is stale or malformed is a hard stop, not
|
|
// a reason to quietly fall through to a different one — the operator said "use this".
|
|
return { ok: false, via: `env:${BASELINE_ENV}`, errors: attempts };
|
|
}
|
|
}
|
|
|
|
let cacheDoc = null;
|
|
let cacheErr = null;
|
|
try {
|
|
cacheDoc = readJson(cachePath);
|
|
} catch (err) {
|
|
cacheErr = `${cachePath}: ${err.message}`;
|
|
}
|
|
if (cacheErr) {
|
|
attempts.push(cacheErr);
|
|
} else if (cacheDoc !== null && cacheDoc !== undefined) {
|
|
const v = validateBaseline(cacheDoc, expectedSha, cachePath);
|
|
if (v.ok) return { ok: true, via: `cache:${cachePath}`, ...v };
|
|
// A stale CACHE is recoverable: fall through to the build fallback, but keep the
|
|
// reason so the final message explains why the slow path ran.
|
|
attempts.push(...v.errors);
|
|
} else {
|
|
attempts.push(`${cachePath}: absent`);
|
|
}
|
|
|
|
if (typeof buildFallback === 'function') {
|
|
let built = null;
|
|
try {
|
|
built = buildFallback();
|
|
} catch (err) {
|
|
attempts.push(`in-job build at origin/next failed: ${err.message}`);
|
|
return { ok: false, via: 'build', errors: attempts };
|
|
}
|
|
const v = validateBaseline(built, expectedSha, 'in-job build at origin/next');
|
|
if (v.ok) return { ok: true, via: 'build', ...v };
|
|
attempts.push(...v.errors);
|
|
return { ok: false, via: 'build', errors: attempts };
|
|
}
|
|
|
|
attempts.push(
|
|
'no baseline available and no in-job build fallback was supplied. This is a hard ' +
|
|
'failure on purpose: a skipped propagation gate is worth less than a slow one ' +
|
|
'(ADR-2719 §6), and in node:test a bare `return` is a PASS, not a skip.',
|
|
);
|
|
return { ok: false, via: 'none', errors: attempts };
|
|
}
|
|
|
|
module.exports = {
|
|
BASELINE_ENV,
|
|
BASELINE_VERSION,
|
|
DEFAULT_CACHE_PATH,
|
|
validateBaseline,
|
|
resolveBaseline,
|
|
};
|