test(#2723): differential emitted-attribution check, dual-run beside the golden (#2737)

* test(#2723): differential emitted-attribution check, dual-run beside the golden

Phase 3 of #2719. The conservation law itself, running BESIDE
golden-install-parity.test.cjs -- both green, fixtures untouched.

Every emitted path whose hash moved between next HEAD and PR HEAD must be
attributable, through the Phase 2 table, to a path the PR actually changed.
Unattributable deltas fail with the paths NAMED. The only way through is a
committed acknowledgment, never a flag -- a contributor facing a red gate
sets a flag, which is what UPDATE_GOLDEN=1 is today.

The central decision is that the law is a PURE function (no fs, git,
installer, or clock), with I/O confined to a separate resolver. The naive
one-big-integration-test shape would need ~38 installer spawns per assertion,
so #2723's four failing-first criteria would not in practice have been
written -- which is exactly how a phase ships promised-but-not-built. Pure,
they are millisecond table tests, and the Stryker gate can actually bite.

Buckets are conserved: every moved path lands in exactly one of
attributed | unattributable | acked, property-tested at 400 runs. A path the
provenance table cannot resolve surfaces as an error, never a silent skip.

Asymmetries that are deliberate, each with a test:
- an ADDED emitted key is a ripple too, not just a modified one
- synthesized paths are exempt; code-derived ones are NOT (Phase 2 refused to
  mark them exempt precisely because exempt means permanently blind)
- shrinkage needs no ack; growth does. Gating shrinkage would punish exactly
  what the size ratchet wants
- a STALE ack is a hard failure -- an ack outliving its ripple pre-clears the
  next one on that path
- a failed `git diff` is an explicit error, never an empty changedPaths set;
  reading it as "nothing changed" would make everything unattributable and
  produce a failure storm that reads like a real finding
- prefix sources are SEGMENT-aware, so `agents/` does not attribute
  `agentsfoo/x.md`

Baseline is cached, not committed, keyed on the next sha. A stale key is
refused rather than used: absence fails loudly and gets fixed, whereas
staleness produces a confident wrong answer. An explicitly pointed-at
GSD_EMITTED_BASELINE that is stale is a hard stop; a stale cache falls
through to the in-job build. No baseline-unavailable path returns -- ADR-2719
section 6 names that trap, since in node:test a bare return is a PASS.

Both the conservation property and the staleness gate were mutation-verified
(injecting a swallowed key fails 9 tests; disabling the staleness comparison
fails 5).

Fixtures, generators, the merge driver and the ADR status are untouched --
those are Phase 4 (#2724).

Refs #2719

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6

* test(#2723): compute stale acks once, after the size pass

Self-review defect found while the reviewers were running. `staleAcks` was
computed twice: once between the hash pass and the size pass, then again
after. Only the second value was returned, so the first was dead code -- and
the dead one was placed where it would have been WRONG.

An acknowledgment can be consumed by either a hash move or a size growth.
Computing staleness before the size pass reports a legitimate growth ack as
stale, which is a false failure that pushes a contributor to delete the very
ack that is doing its job.

Now computed once, after both passes, with a regression test. Verified by
mutation: restoring the early computation fails 2 tests.

Refs #2719

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6

* test(#2723): wire the attribution check to the real tree, not just synthetic input

An isolated reviewer caught that the first cut was INTERFACE-ONLY: nothing
read the ack file from disk, nothing shelled git, nothing built real
manifests. Every test was true of hand-built inputs and none of the repo, so
the acceptance criterion "both this check and golden-install-parity green on
the same tree" was trivially true rather than meaningfully true. That is the
promised-but-not-built failure this epic keeps finding in its predecessors,
recurring one phase later for the wiring itself. Taken, not argued.

Adds tests/helpers/emitted-runtime.cjs -- the only module that touches git,
disk, or the installer -- and an integration test that runs the same pure law
against reality:

- CURRENT side: 19 real installer spawns via runMinimalInstall +
  buildParityManifest, the same machinery the golden harness uses.
- BASELINE side: `git show origin/next:<fixture>`. That is next's RECORDED
  emitted state and it costs nothing. Deliberately NOT the working-tree
  fixtures, which are whatever this PR's author regenerated -- comparing
  against those would be vacuous. Phase 4 deletes the fixtures and swaps in
  resolveBaseline's cache path, already implemented and tested.
- changed paths from real `git diff --name-only origin/next...HEAD`, with the
  git subprocess bounded at 30s per CLAUDE.md's unbounded-subprocess rule.
- the real tests/emitted-drift-ack.json (absent is legal; present-but-empty
  or unparseable throws rather than being read as absent).

Verified it can actually fail: an uncommitted edit to a shipped workflow
moves emitted output but never appears in the committed diff, and the check
names all 18 affected emitted paths with the message format ADR-2719 §1
specifies. Restores clean.

Also from review:
- readAckFile now has a real test exercising the SUT across absent / valid /
  empty / unparseable / unreadable. The previous test asserted fs behaviour
  rather than SUT behaviour, because no SUT ack-reading path existed yet.
- formatReport's sampleLimit gains true limit-1/limit/limit+1 coverage at
  19/20/21. A test was previously NAMED "(limit+1)" while testing no numeric
  limit at all, which is worse than no coverage because it reads as covered.

Windows uses an explicit t.skip (install output is platform-specific there,
mirroring the golden harness) -- never a bare return, which node:test scores
as a PASS.

Refs #2719

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6

* test(#2723): cover the claude-local manifest family in the real-tree check

Isolated adversarial review, MAJOR. The real-tree wiring enumerated
Object.keys(RUNTIME_META) -- 18 entries -- while the emitted manifest set has
19 families. The 19th is claude-local: claude is the reference host and the
only runtime with a distinct LOCAL "legacy flat-commands" layout
(commands/gsd-*.md + agents/gsd-*.md at project scope), which
golden-install-parity.test.cjs guards with a hand-coded test outside its
RUNTIME_META loop (#2086).

The family was dropped from BOTH sides, so the test's own self-check
(current.length === baseline.length) passed vacuously at 18 === 18. A PR
changing Claude's local-scope output would have failed the golden while this
check reported ok -- and that disagreement is precisely what the dual-run
window is designed to surface as a provenance-table hole. A wiring omission
masquerading as one is the worst available failure here, because it would
have been read as evidence about Phase 2 rather than a bug in Phase 3.

Fixed by deriving MANIFEST_FAMILIES explicitly (18 global + claude-local at
local scope) instead of inferring the set from RUNTIME_META.

The self-check is also repaired: it now asserts both sides against the
INDEPENDENT EXPECTED_MANIFEST_COUNT from the Phase 2 table, and asserts
claude-local specifically. Comparing the two sides to each other can never
catch a family missing from both -- the assertion has to come from outside.
Verified by mutation: removing claude-local again fails the test.

Also from the same review:
- sourceSatisfiedBy returns the matched source string, so an empty-string
  source would return '' and the caller's `if (hit)` would silently discard a
  real match. Unreachable today (every rule source is a non-empty template)
  but a footgun for the next rule author; now `!== null`.
- the purity fixture used a single-element changedPaths array, so an in-place
  sort would have been invisible. Now three elements in unsorted order.

Refs #2719

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6

* fix(#2723): resolve the base ref tolerantly instead of hard-requiring origin/next

The first matrix run failed on both linux lanes:

  differential attribution over the real tree
  cannot resolve origin/next (Command failed: git rev-parse origin/next)

Not a flake, and not an environment excuse -- a real defect in this diff. The
gsd-test runner shallow-clones and merges base+head, so no origin/* remote-
tracking refs exist in the container. My own fail-loud path fired correctly;
what was wrong was hard-depending on that ref existing. GitHub Actions has the
same shape by default, which is exactly why changeset-required.yml carries an
explicit `git fetch origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}"`.

Now resolved through an ordered candidate list -- GSD_EMITTED_BASE (explicit
lane override), then origin/$GITHUB_BASE_REF and $GITHUB_BASE_REF, then
origin/next and next -- de-duplicated, each verified with
`rev-parse --verify <ref>^{commit}`.

When NO candidate resolves the test takes an explicit t.skip() naming every
ref it tried and stating that the gate did not run here. That is the
ADR-2719 section 6 distinction: t.skip is REPORTED as skipped, whereas a bare
return is scored as a PASS. Hard-failing was the other option and is wrong --
it would make the suite permanently red wherever a base ref cannot exist by
construction, which is a statement about the checkout, not a propagation
finding.

The candidate ordering is pinned by a unit test rather than left implicit,
since the ordering IS the fix.

Refs #2719

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-07-27 23:22:37 -04:00
committed by GitHub
parent 1f6822ccba
commit 9138271b5f
5 changed files with 1533 additions and 1 deletions

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,752 @@
'use strict';
/**
* emitted-attribution.test.cjs — the differential attribution check (#2723,
* ADR-2719 §1/§3/§4/§5/§6, epic #2719 Phase 3).
*
* Runs BESIDE tests/golden-install-parity.test.cjs — both green, fixtures untouched.
* Any PR where the golden fails and this passes is a Phase 2 provenance-table hole;
* that disagreement is the entire point of the dual-run window, and Phase 4 (#2724)
* must not land until it has been observed on real PRs.
*
* The law: every emitted path whose hash moved between `next` HEAD and PR HEAD must be
* attributable — through the Phase 2 table — to a path the PR actually changed.
* Unattributable deltas fail with the paths NAMED. The only way through is a committed
* acknowledgment, never a flag (a contributor facing a red gate sets a flag, which is
* what UPDATE_GOLDEN=1 is today).
*
* Structure: the pure law is exercised against synthetic manifests, which is what makes
* the four failing-first criteria practical to assert at all — and then the final test
* runs that same law against the REAL tree: 19 actual installer spawns for the current
* side, `git show origin/next:<fixture>` for the baseline side, real `git diff` for the
* changed paths, and the real `tests/emitted-drift-ack.json`.
*
* That last test is load-bearing. Without it this file would be interface-only — every
* assertion true of hand-built inputs and none of the repo — which is the
* promised-but-not-built failure this epic keeps finding in its own predecessors.
* Verified by injecting an uncommitted edit to a shipped workflow: emitted output moves
* but the path never appears in `git diff origin/next...HEAD`, and the check names all
* 18 affected emitted paths.
*/
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const { execFileSync } = require('node:child_process');
const fc = require('fast-check');
const { cleanup } = require('./helpers.cjs');
const { BUILD_SCRIPT } = require('./helpers/install-shared.cjs');
const {
resolveChangedPaths,
resolveBase,
baseRefCandidates,
baselineManifestsAtRef,
baselineSizesAtRef,
currentManifests,
currentSizes,
readAckFile,
} = require('./helpers/emitted-runtime.cjs');
const { EXPECTED_MANIFEST_COUNT } = require('./helpers/emitted-provenance.cjs');
const {
ACK_VERSION,
sourceSatisfiedBy,
parseAck,
diffEmitted,
formatReport,
} = require('./helpers/emitted-diff.cjs');
const {
BASELINE_ENV,
BASELINE_VERSION,
resolveBaseline,
} = require('./helpers/emitted-baseline.cjs');
const SHA_A = 'a'.repeat(40);
const SHA_B = 'b'.repeat(40);
/** A real emitted key + its real source, so rows assert the shape production uses. */
const WORKFLOW_KEY = 'gsd-core/workflows/plan-phase.md';
const WORKFLOW_SRC = 'gsd-core/workflows/plan-phase.md';
const SKILL_KEY = 'skills/gsd-add-tests/SKILL.md';
const SKILL_SRC = 'commands/gsd/add-tests.md';
const mf = (obj) => ({ claude: obj });
// ─── Attribution: the conservation law ───────────────────────────────────────
test('unchanged hashes are not reported', () => {
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
current: mf({ [WORKFLOW_KEY]: 'aaa' }),
changedPaths: [],
});
assert.equal(r.moved, 0);
assert.equal(r.attributed.length, 0);
assert.equal(r.unattributable.length, 0);
assert.ok(r.ok);
});
test('a moved hash whose source changed is attributed', () => {
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
changedPaths: [WORKFLOW_SRC],
});
assert.equal(r.moved, 1);
assert.equal(r.unattributable.length, 0);
assert.equal(r.attributed.length, 1);
assert.equal(r.attributed[0].via, WORKFLOW_SRC);
assert.ok(r.ok);
});
test('a trailing-slash source entry matches by prefix, segment-aware', () => {
// Kimi's root agent aggregates all of agents/ — a Phase 2 prefix source.
assert.equal(sourceSatisfiedBy('agents/', new Set(['agents/gsd-planner.md'])), 'agents/gsd-planner.md');
// Hostile: a bare startsWith would over-attribute here. It must NOT match.
assert.equal(sourceSatisfiedBy('agents/', new Set(['agentsfoo/x.md'])), null);
// Exact entries compare exactly.
assert.equal(sourceSatisfiedBy('a/b.md', new Set(['a/b.md'])), 'a/b.md');
assert.equal(sourceSatisfiedBy('a/b.md', new Set(['a/b.md.bak'])), null);
const r = diffEmitted({
baseline: { kimi: { 'agents/gsd.yaml': 'aaa' } },
current: { kimi: { 'agents/gsd.yaml': 'bbb' } },
changedPaths: ['agents/gsd-planner.md'],
});
assert.equal(r.unattributable.length, 0, 'prefix source should attribute');
assert.equal(r.attributed[0].via, 'agents/gsd-planner.md');
});
test('a moved hash nothing explains is unattributable and named', () => {
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
changedPaths: ['README.md'],
});
assert.equal(r.unattributable.length, 1);
const u = r.unattributable[0];
assert.equal(u.rel, WORKFLOW_KEY);
assert.equal(u.runtime, 'claude');
assert.equal(u.ruleId, 'gsd-core-verbatim');
assert.deepEqual(u.expectedSources, [WORKFLOW_SRC]);
assert.ok(!r.ok);
// ADR-2719 §1 sells the design on this message — it is a deliverable.
const msg = formatReport(r);
assert.match(msg, /changed that nothing in this diff explains/);
assert.ok(msg.includes(WORKFLOW_KEY));
assert.ok(msg.includes(WORKFLOW_SRC), 'the message must say what WOULD have explained it');
});
test('synthesized paths are exempt from attribution', () => {
const r = diffEmitted({
baseline: mf({ 'gsd-core/VERSION': 'aaa' }),
current: mf({ 'gsd-core/VERSION': 'bbb' }),
changedPaths: [],
});
assert.equal(r.unattributable.length, 0, 'install-time state can never be unexplained');
assert.equal(r.attributed[0].via, '<synthesized: exempt>');
assert.ok(r.ok);
});
test('code-derived paths attribute to their emitting source file', () => {
// Phase 2 deliberately refused to mark these exempt; this is why.
const r = diffEmitted({
baseline: { cline: { '.clinerules/gsd.md': 'aaa' } },
current: { cline: { '.clinerules/gsd.md': 'bbb' } },
changedPaths: ['src/runtime-hooks-surface.cts'],
});
assert.equal(r.unattributable.length, 0);
assert.equal(r.attributed[0].via, 'src/runtime-hooks-surface.cts');
const blind = diffEmitted({
baseline: { cline: { '.clinerules/gsd.md': 'aaa' } },
current: { cline: { '.clinerules/gsd.md': 'bbb' } },
changedPaths: ['README.md'],
});
assert.equal(blind.unattributable.length, 1, 'had these been exempt, this ripple would be invisible forever');
});
test('an added emitted key is a ripple too', () => {
const r = diffEmitted({
baseline: mf({}),
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
changedPaths: ['README.md'],
});
assert.equal(r.moved, 1);
assert.equal(r.unattributable.length, 1);
assert.equal(r.unattributable[0].change, 'added');
});
test('a removed emitted key is reported', () => {
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
current: mf({}),
changedPaths: [WORKFLOW_SRC],
});
assert.equal(r.removed.length, 1);
assert.equal(r.removed[0].change, 'removed');
assert.equal(r.unattributable.length, 0, 'the deletion is explained by the source change');
});
test('moved hashes with no changed paths are all unattributable', () => {
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa', [SKILL_KEY]: 'ccc' }),
current: mf({ [WORKFLOW_KEY]: 'bbb', [SKILL_KEY]: 'ddd' }),
changedPaths: [],
});
assert.equal(r.unattributable.length, 2, 'emitted output moving with zero source changes is a real finding');
assert.ok(!r.ok);
});
test('a failed git diff is an error, not an empty change set', () => {
// Treating a git failure as "nothing changed" would make everything unattributable —
// a failure storm that reads exactly like a real finding.
const r = diffEmitted({ baseline: mf({}), current: mf({}), changedPaths: null });
assert.ok(!r.ok);
assert.match(r.errors.join('\n'), /changedPaths must be an array/);
});
test('an unattributable-by-table path surfaces as an error', () => {
const r = diffEmitted({
baseline: mf({ 'totally/unknown/thing.md': 'aaa' }),
current: mf({ 'totally/unknown/thing.md': 'bbb' }),
changedPaths: [],
});
assert.ok(!r.ok);
assert.match(r.errors.join('\n'), /no rule matches/);
assert.equal(r.unattributable.length, 0, 'a table hole is an error, not a silent skip');
});
// ─── Acknowledgment file ─────────────────────────────────────────────────────
test('an acked ripple passes and is echoed', () => {
const ack = { version: ACK_VERSION, paths: { [WORKFLOW_KEY]: { reason: 'converter change, #2723' } } };
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
changedPaths: ['README.md'],
ack,
});
assert.equal(r.unattributable.length, 0);
assert.equal(r.acked.length, 1);
assert.equal(r.acked[0].reason, 'converter change, #2723');
assert.ok(r.ok);
});
test('a stale ack entry fails', () => {
// An ack that outlives its ripple pre-clears the NEXT one on that path.
const ack = { version: ACK_VERSION, paths: { [WORKFLOW_KEY]: { reason: 'old' } } };
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
current: mf({ [WORKFLOW_KEY]: 'aaa' }),
changedPaths: [],
ack,
});
assert.deepEqual(r.staleAcks, [WORKFLOW_KEY]);
assert.ok(!r.ok);
assert.match(formatReport(r), /stale acknowledgment/);
});
test('an ack without a reason fails', () => {
for (const bad of [{ reason: '' }, { reason: ' ' }, {}, null, 42]) {
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
changedPaths: [],
ack: { version: ACK_VERSION, paths: { [WORKFLOW_KEY]: bad } },
});
assert.ok(!r.ok, `${JSON.stringify(bad)} must be rejected`);
assert.match(r.errors.join('\n'), /has no non-empty "reason"/);
}
});
test('an absent ack file means no acks', () => {
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
changedPaths: [WORKFLOW_SRC],
ack: null,
});
assert.equal(r.errors.length, 0);
assert.ok(r.ok, 'the healthy steady state is no ack file at all');
});
test('a live ack and a stale ack together: only the stale one is named', () => {
const ack = {
version: ACK_VERSION,
paths: {
[WORKFLOW_KEY]: { reason: 'live ripple' },
[SKILL_KEY]: { reason: 'stale' },
},
};
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa', [SKILL_KEY]: 'ccc' }),
current: mf({ [WORKFLOW_KEY]: 'bbb', [SKILL_KEY]: 'ccc' }),
changedPaths: [],
ack,
});
assert.deepEqual(r.staleAcks, [SKILL_KEY], 'the live one must not be named');
});
test('non-object ack JSON is rejected, not treated as empty', () => {
// Reading these as "no acks" would SILENTLY DISARM the gate — indistinguishable
// from a healthy run, which is the worst failure available here.
for (const bad of [0, 'a string', [], true]) {
const { errors } = parseAck(bad);
assert.ok(errors.length > 0, `${JSON.stringify(bad)} must be rejected`);
assert.match(errors.join('\n'), /must be a JSON object/);
}
assert.deepEqual(parseAck(null).errors, [], 'absent is legal');
assert.deepEqual(parseAck({}).errors, [], 'empty object is legal');
assert.equal(parseAck({ version: 99, paths: {} }).errors.length, 1, 'version drift is caught');
});
// ─── The acceptance criteria, failing-first ──────────────────────────────────
test('a ripple names the unexplained path and not the explained one', () => {
// #2723 AC: "edit one source file, corrupt an unrelated emitted file, assert the
// check names the unattributable paths."
const r = diffEmitted({
baseline: mf({ [WORKFLOW_KEY]: 'aaa', [SKILL_KEY]: 'ccc' }),
current: mf({ [WORKFLOW_KEY]: 'bbb', [SKILL_KEY]: 'ddd' }),
changedPaths: [WORKFLOW_SRC], // only the workflow source was edited
});
assert.equal(r.unattributable.length, 1);
assert.equal(r.unattributable[0].rel, SKILL_KEY, 'the unrelated emitted file is the finding');
assert.equal(r.attributed.length, 1);
assert.equal(r.attributed[0].rel, WORKFLOW_KEY, 'the explained one must NOT be reported');
assert.ok(!r.ok);
});
test('a converter change fails without an ack and passes with one', () => {
// #2723 AC: "simulate a legitimate converter change: assert it fails without an ack
// entry and passes with one." A converter edit moves emitted bytes for files whose
// sources nobody touched — ADR-2264's "~5% git cannot review".
const moved = {};
const base = {};
for (let i = 0; i < 25; i++) {
base[`skills/gsd-cmd-${i}/SKILL.md`] = `h${i}`;
moved[`skills/gsd-cmd-${i}/SKILL.md`] = `x${i}`;
}
const changedPaths = ['src/runtime-artifact-conversion.cts'];
const without = diffEmitted({ baseline: mf(base), current: mf(moved), changedPaths });
assert.equal(without.unattributable.length, 25);
assert.ok(!without.ok, 'a converter change must not pass silently');
const paths = {};
for (const rel of Object.keys(moved)) paths[rel] = { reason: 'converter rewrite, ADR-2719' };
const withAck = diffEmitted({
baseline: mf(base), current: mf(moved), changedPaths,
ack: { version: ACK_VERSION, paths },
});
assert.equal(withAck.unattributable.length, 0);
assert.equal(withAck.acked.length, 25);
assert.ok(withAck.ok);
});
test('growth is reported with its exact byte delta and needs an ack', () => {
// ADR-2719 must-have 6, added by an /adr-phase-coverage audit precisely because
// scope item 5 promised it and no criterion asserted it.
const sizeBaseline = { 'verify-work.md': 10000, 'plan-phase.md': 8000 };
const sizeCurrent = { 'verify-work.md': 11247, 'plan-phase.md': 8000 };
const without = diffEmitted({
baseline: mf({}), current: mf({}), changedPaths: [], sizeBaseline, sizeCurrent,
});
assert.equal(without.grown.length, 1);
assert.deepEqual(without.grown[0], {
name: 'verify-work.md', from: 10000, to: 11247, delta: 1247, acked: false,
});
assert.ok(!without.ok, 'unacked growth must block');
assert.match(formatReport(without), /verify-work\.md grew 1247 bytes/);
const withAck = diffEmitted({
baseline: mf({}), current: mf({}), changedPaths: [], sizeBaseline, sizeCurrent,
ack: { version: ACK_VERSION, paths: { 'verify-work.md': { reason: 'new UAT section' } } },
});
assert.equal(withAck.grown[0].acked, true);
assert.ok(withAck.ok);
});
test('an ack consumed by size growth alone is not reported as stale', () => {
// Ordering regression: stale-ack detection must run AFTER the size pass. Computing it
// between the hash pass and the size pass reports a legitimate growth ack as stale —
// a false failure that would push contributors to delete the very ack that is working.
const r = diffEmitted({
baseline: mf({}),
current: mf({}),
changedPaths: [],
sizeBaseline: { 'verify-work.md': 10000 },
sizeCurrent: { 'verify-work.md': 11247 },
ack: { version: ACK_VERSION, paths: { 'verify-work.md': { reason: 'new UAT section' } } },
});
assert.deepEqual(r.staleAcks, [], 'a growth-consumed ack is live, not stale');
assert.equal(r.grown[0].acked, true);
assert.ok(r.ok);
});
test('shrinkage is reported but needs no ack', () => {
const r = diffEmitted({
baseline: mf({}), current: mf({}), changedPaths: [],
sizeBaseline: { 'a.md': 9000 }, sizeCurrent: { 'a.md': 8000 },
});
assert.deepEqual(r.shrunk, [{ name: 'a.md', from: 9000, to: 8000, delta: 1000 }]);
assert.ok(r.ok, 'shrinkage is not creep — gating it would punish what the ratchet wants');
});
// ─── Baseline resolution + staleness ─────────────────────────────────────────
const goodBaseline = (sha) => ({
version: BASELINE_VERSION,
sha,
manifests: { claude: { [WORKFLOW_KEY]: 'aaa' } },
sizes: { 'plan-phase.md': 100 },
});
test('a stale baseline cache key is detected, not used', () => {
// ADR-2719 §5: the one thing that has to be exactly right.
const r = resolveBaseline({
expectedSha: SHA_A,
env: {},
cachePath: 'cache.json',
readJson: () => goodBaseline(SHA_B),
});
assert.ok(!r.ok);
assert.match(r.errors.join('\n'), /STALE baseline/);
assert.ok(r.errors.join('\n').includes(SHA_B) && r.errors.join('\n').includes(SHA_A));
});
test('a matching baseline sha is accepted', () => {
const r = resolveBaseline({
expectedSha: SHA_A, env: {}, cachePath: 'cache.json',
readJson: () => goodBaseline(SHA_A),
});
assert.ok(r.ok);
assert.equal(r.sha, SHA_A);
assert.equal(r.via, 'cache:cache.json');
assert.deepEqual(r.sizeBaseline, { 'plan-phase.md': 100 });
});
test('an unavailable baseline fails explicitly rather than skipping', () => {
// ADR-2719 §6 — in node:test a bare `return` is a PASS, which would fail the gate open.
const r = resolveBaseline({
expectedSha: SHA_A, env: {}, cachePath: 'cache.json',
readJson: () => null,
});
assert.ok(!r.ok);
assert.equal(r.via, 'none');
assert.match(r.errors.join('\n'), /bare `return` is a PASS/);
});
test('a malformed baseline is rejected', () => {
for (const bad of [0, 'str', [], true]) {
const r = resolveBaseline({
expectedSha: SHA_A, env: {}, cachePath: 'c.json', readJson: () => bad,
});
assert.ok(!r.ok, `${JSON.stringify(bad)} must be rejected`);
}
const noSha = resolveBaseline({
expectedSha: SHA_A, env: {}, cachePath: 'c.json',
readJson: () => ({ version: BASELINE_VERSION, manifests: {} }),
});
assert.match(noSha.errors.join('\n'), /must be a 40-hex commit sha/);
});
test('baseline resolution precedence is explicit and reported', () => {
// env wins over cache…
const viaEnv = resolveBaseline({
expectedSha: SHA_A,
env: { [BASELINE_ENV]: '/tmp/from-cache-restore.json' },
cachePath: 'cache.json',
readJson: (p) => (p === '/tmp/from-cache-restore.json' ? goodBaseline(SHA_A) : goodBaseline(SHA_B)),
});
assert.ok(viaEnv.ok);
assert.equal(viaEnv.via, `env:${BASELINE_ENV}`);
// …and an explicitly-pointed-at stale baseline is a hard stop, not a fall-through:
// the operator said "use this one".
const envStale = resolveBaseline({
expectedSha: SHA_A,
env: { [BASELINE_ENV]: '/tmp/x.json' },
cachePath: 'cache.json',
readJson: () => goodBaseline(SHA_B),
buildFallback: () => goodBaseline(SHA_A),
});
assert.ok(!envStale.ok, 'an explicit stale baseline must not silently fall through');
// a stale CACHE, by contrast, falls through to the build fallback
const viaBuild = resolveBaseline({
expectedSha: SHA_A, env: {}, cachePath: 'cache.json',
readJson: () => goodBaseline(SHA_B),
buildFallback: () => goodBaseline(SHA_A),
});
assert.ok(viaBuild.ok);
assert.equal(viaBuild.via, 'build');
});
test('base-ref candidates are ordered most-specific first and de-duplicated', () => {
// The gate went red on its first matrix run because it hard-depended on
// `origin/next`, which cannot exist in the gsd-test container (shallow clone +
// base/head merge, no remote-tracking refs). Candidate order is the fix, so it is
// pinned rather than left implicit.
assert.deepEqual(
baseRefCandidates({ GSD_EMITTED_BASE: 'abc123', GITHUB_BASE_REF: 'next' }),
['abc123', 'origin/next', 'next'],
'an explicit override wins, then the Actions base ref, then the defaults',
);
assert.deepEqual(
baseRefCandidates({ GITHUB_BASE_REF: 'release/1.9' }),
['origin/release/1.9', 'release/1.9', 'origin/next', 'next'],
'a non-next base ref is honored before falling back',
);
assert.deepEqual(
baseRefCandidates({}),
['origin/next', 'next'],
'with no env, the repo defaults are the only candidates',
);
// De-duplication matters: GITHUB_BASE_REF=next must not produce origin/next twice.
const dupes = baseRefCandidates({ GITHUB_BASE_REF: 'next' });
assert.equal(new Set(dupes).size, dupes.length);
});
test('an unreadable baseline surfaces an error', () => {
const r = resolveBaseline({
expectedSha: SHA_A, env: {}, cachePath: 'c.json',
readJson: () => { throw new Error('injected read failure'); },
});
assert.ok(!r.ok);
assert.match(r.errors.join('\n'), /injected read failure/);
});
test('readAckFile: absent is legal, malformed and unreadable are not', () => {
const tmp = fs.mkdtempSync(path.join(require('node:os').tmpdir(), 'gsd-ack-'));
try {
const ackPath = path.join(tmp, 'emitted-drift-ack.json');
// Absent == no acks. The healthy steady state.
assert.equal(readAckFile(ackPath), null);
// Present and valid.
fs.writeFileSync(ackPath, JSON.stringify({ version: ACK_VERSION, paths: {} }));
assert.deepEqual(readAckFile(ackPath), { version: ACK_VERSION, paths: {} });
// Present but empty — must NOT be read as absent.
fs.writeFileSync(ackPath, '');
assert.throws(() => readAckFile(ackPath), /present but empty/);
// Present but not JSON.
fs.writeFileSync(ackPath, '{not json');
assert.throws(() => readAckFile(ackPath), /not valid JSON/);
// Unreadable: monkeypatch the fs method, restore in `finally`. NEVER chmod 0o000 —
// root bypasses mode bits, so the test would silently pass with zero coverage in
// root Docker/CI. This exercises the SUT (readAckFile), not fs itself.
fs.writeFileSync(ackPath, JSON.stringify({ version: ACK_VERSION, paths: {} }));
const orig = fs.readFileSync;
try {
fs.readFileSync = () => { throw new Error('injected ack read failure'); };
assert.throws(() => readAckFile(ackPath), /injected ack read failure/);
} finally {
fs.readFileSync = orig;
}
// Restoration is real, not assumed.
assert.deepEqual(readAckFile(ackPath), { version: ACK_VERSION, paths: {} });
} finally {
cleanup(tmp);
}
});
test('formatReport truncation is exact at limit-1 / limit / limit+1', () => {
// sampleLimit gates a real branch. CLAUDE.md's boundary rule applies to it like any
// other limit; the earlier suite named a test "limit+1" that tested no numeric limit
// at all, which is worse than no coverage because it reads as covered.
const build = (n) => {
const baseline = {}; const current = {};
for (let i = 0; i < n; i++) {
const k = `gsd-core/workflows/w${String(i).padStart(3, '0')}.md`;
baseline[k] = 'a'; current[k] = 'b';
}
return diffEmitted({ baseline: mf(baseline), current: mf(current), changedPaths: [] });
};
const at19 = formatReport(build(19), { sampleLimit: 20 });
assert.ok(at19.includes('w018.md'), 'limit-1 lists every path');
assert.ok(!at19.includes('…and'), 'limit-1 must not truncate');
const at20 = formatReport(build(20), { sampleLimit: 20 });
assert.ok(at20.includes('w019.md'), 'at the limit the last path is listed');
assert.ok(!at20.includes('…and'), 'exactly at the limit must not truncate');
const at21 = formatReport(build(21), { sampleLimit: 20 });
assert.ok(at21.includes('…and 1 more'), 'limit+1 truncates and says how many were hidden');
assert.ok(!at21.includes('w020.md'), 'the 21st path is not listed');
});
// ─── Independence + purity ───────────────────────────────────────────────────
test('the differential covers every runtime present in either manifest', () => {
const baseline = { claude: { [WORKFLOW_KEY]: 'a' }, kimi: { [WORKFLOW_KEY]: 'a' } };
const current = { claude: { [WORKFLOW_KEY]: 'b' }, opencode: { [WORKFLOW_KEY]: 'c' } };
const r = diffEmitted({ baseline, current, changedPaths: [] });
const seen = new Set([...r.unattributable, ...r.attributed, ...r.removed].map((x) => x.runtime));
assert.deepEqual([...seen].sort(), ['claude', 'kimi', 'opencode'],
'a runtime present on only one side must still be evaluated');
});
test('diff is pure and repeatable', () => {
const args = {
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
// 3 elements in deliberately unsorted order, so an in-place sort would be visible.
changedPaths: ['zzz/last.md', WORKFLOW_SRC, 'aaa/first.md'],
};
const frozen = JSON.stringify(args);
const a = diffEmitted(args);
const b = diffEmitted(args);
assert.deepEqual(b, a);
assert.equal(JSON.stringify(args), frozen, 'inputs must not be mutated');
});
// ─── Property: conservation ──────────────────────────────────────────────────
test('property: every moved key lands in exactly one bucket', () => {
// The conservation law itself. A key silently dropped from all three buckets is a
// hole in the very invariant ADR-2719 asserts — and it is the failure a hand-written
// example set is least likely to find.
const keys = [WORKFLOW_KEY, SKILL_KEY, 'agents/gsd-planner.md', 'scripts/lib/cli-exit.cjs'];
const sources = { [WORKFLOW_KEY]: WORKFLOW_SRC, [SKILL_KEY]: SKILL_SRC,
'agents/gsd-planner.md': 'agents/gsd-planner.md', 'scripts/lib/cli-exit.cjs': 'scripts/lib/cli-exit.cjs' };
fc.assert(
fc.property(
fc.subarray(keys, { minLength: 1 }), // which keys move
fc.subarray(keys), // which sources the PR changed
fc.subarray(keys), // which keys are acked
(movedKeys, changedKeys, ackedKeys) => {
const baseline = {}; const current = {};
for (const k of keys) { baseline[k] = 'h'; current[k] = movedKeys.includes(k) ? 'x' : 'h'; }
const ackPaths = {};
for (const k of ackedKeys) ackPaths[k] = { reason: 'property' };
const r = diffEmitted({
baseline: mf(baseline),
current: mf(current),
changedPaths: changedKeys.map((k) => sources[k]),
ack: { version: ACK_VERSION, paths: ackPaths },
});
if (r.errors.length) return false;
const bucketed = [
...r.attributed.map((x) => x.rel),
...r.unattributable.map((x) => x.rel),
...r.acked.map((x) => x.rel),
];
// exactly-once, and exactly the moved set — no key invented, none dropped
return bucketed.length === movedKeys.length
&& new Set(bucketed).size === bucketed.length
&& movedKeys.every((k) => bucketed.includes(k));
},
),
{ numRuns: 400 },
);
});
// ─── The real thing: the law, run against the actual tree ───────────────────
//
// Everything above exercises the pure law against synthetic input, which is what makes
// the acceptance criteria practical to assert at all. This block is what stops the
// phase from being interface-only: it builds the CURRENT emitted manifests for real
// (one installer spawn per runtime), reads the BASELINE from `origin/next`, resolves
// the changed paths with real git, reads the real ack file, and runs the conservation
// law over all of it.
//
// Baseline source note: `git show origin/next:<fixture>` — next's RECORDED emitted
// state. Deliberately not the working-tree fixtures, which are whatever this PR's
// author regenerated; comparing against those would be vacuous. Phase 4 (#2724)
// deletes the fixtures and swaps in resolveBaseline's cache path, which is already
// implemented and tested above.
test('differential attribution over the real tree', { timeout: 900_000 }, async (t) => {
if (process.platform === 'win32') {
// Mirrors the golden harness: install output is platform-specific on Windows
// (backslash paths), so parity is asserted on macOS + Linux. An explicit t.skip,
// never a bare `return` — in node:test that would be a PASS (ADR-2719 §6).
t.skip('emitted parity is asserted on macOS + Linux; Windows install output is platform-specific');
return;
}
// hooks/dist is gitignored and built (DEFECT.HOOKS-DIST-SCOPED-CI): the scoped CI
// lane does not run build:hooks, so a real install there would emit no hooks/ dir.
// Build idempotently, exactly as the golden harness does.
execFileSync(process.execPath, [BUILD_SCRIPT], { encoding: 'utf-8', stdio: 'pipe', timeout: 120_000 });
// The base ref is not universally available. The gsd-test runner shallow-clones and
// merges base+head, so no `origin/*` remote-tracking ref exists in the container —
// this test went red on its first matrix run for exactly that reason, which is the
// resolver doing its job and the dependency being wrong.
//
// An explicit t.skip is the ADR-sanctioned response for a genuine environmental
// skip: it is REPORTED as skipped, unlike a bare `return`, which node:test scores as
// a PASS (ADR-2719 §6). Hard-failing instead would make the suite permanently red
// wherever a base ref cannot exist by construction, which is not a propagation
// finding — it is a statement about the checkout.
const resolved = resolveBase();
if (!resolved) {
t.skip(
'no base ref resolvable — tried ' + baseRefCandidates().join(', ') +
'. The differential gate did NOT run here. It binds in the CI test lanes, which ' +
'fetch the base ref explicitly; set GSD_EMITTED_BASE=<ref|sha> to run it elsewhere.',
);
return;
}
const { ref: base, sha: baseSha } = resolved;
assert.match(baseSha, /^[0-9a-f]{40}$/);
const baseline = baselineManifestsAtRef(base);
assert.ok(
baseline && Object.keys(baseline).length > 0,
`no baseline manifests found at ${base}. During the dual-run window these come from the ` +
'committed golden fixtures at that ref; after Phase 4 they come from the cached ' +
'baseline artifact via resolveBaseline().',
);
const changedPaths = resolveChangedPaths(base);
const ack = readAckFile();
const current = currentManifests();
// Assert against the INDEPENDENT expected count, not just baseline-vs-current.
// Comparing the two sides to each other cannot catch a family dropped from BOTH —
// which is exactly what happened with claude-local: 18 === 18 passed vacuously while
// the 19th family went unchecked.
assert.equal(
Object.keys(baseline).length,
EXPECTED_MANIFEST_COUNT,
`baseline must cover all ${EXPECTED_MANIFEST_COUNT} emitted manifest families`,
);
assert.equal(
Object.keys(current).length,
EXPECTED_MANIFEST_COUNT,
`current must cover all ${EXPECTED_MANIFEST_COUNT} emitted manifest families`,
);
assert.ok(baseline['claude-local'], 'the claude local-scope layout (#2086) must be covered');
assert.ok(current['claude-local'], 'the claude local-scope layout (#2086) must be covered');
const result = diffEmitted({
baseline,
current,
changedPaths,
ack,
sizeBaseline: baselineSizesAtRef(base),
sizeCurrent: currentSizes(),
});
assert.ok(
result.ok,
`emitted-attribution failed against ${base}@${baseSha.slice(0, 12)}:\n\n${formatReport(result)}`,
);
});

View File

@@ -0,0 +1,192 @@
'use strict';
/**
* Baseline acquisition for the differential attribution check (ADR-2719 §5, #2723).
*
* The baseline is the emitted-manifest set built at `next` HEAD. It is CACHED, not
* committed — committing it would recreate the derived-state-in-git problem this epic
* exists to delete, and would double the rate `next` advances.
*
* ── The load-bearing part ────────────────────────────────────────────────────
* ADR-2719 §5: "keyed on the `next` sha the PR was merged with. That key discipline is
* the one thing that has to be exactly right — a stale baseline silently mis-attributes."
*
* Note the asymmetry that makes staleness worse than absence: a MISSING baseline fails
* loudly and gets fixed. A STALE one produces a confident wrong answer — it attributes
* deltas to the wrong commit's state, so real ripples read as explained. Every path
* through this module therefore fails closed on a key mismatch.
*
* ── No bare `return` anywhere ────────────────────────────────────────────────
* ADR-2719 §6 calls this out explicitly: in node:test a bare `return` is a PASS, not a
* skip, so a baseline-unavailable path that returns would make the whole gate fail open
* with nothing in CI to say so. This module returns an explicit {ok:false} result and
* the caller asserts on it.
*
* IO is INJECTED (readJson / exists / buildFallback) so every failure mode above is
* unit-testable without touching the filesystem or spawning 19 installers.
*/
/** Env var a CI cache-restore step points at the recovered baseline artifact. */
const BASELINE_ENV = 'GSD_EMITTED_BASELINE';
/** Default on-disk cache location, relative to the repo root. */
const DEFAULT_CACHE_PATH = '.gsd-cache/emitted-baseline.json';
/** Baseline artifact schema version — pinned so a format change fails loudly. */
const BASELINE_VERSION = 1;
/**
* Validate a baseline artifact's shape and freshness.
*
* @param {*} doc parsed artifact
* @param {string} expectedSha the `next` sha this PR is being evaluated against
* @param {string} source where it came from (named in every error)
* @returns {{ok: true, baseline: object, sizeBaseline: object|null, sha: string}
* |{ok: false, errors: string[]}}
*/
function validateBaseline(doc, expectedSha, source) {
const errors = [];
if (doc === null || doc === undefined) {
return { ok: false, errors: [`${source}: baseline is absent`] };
}
if (typeof doc !== 'object' || Array.isArray(doc)) {
// Same class as Phase 2's fixture loader: a document that parses but is not an
// object must never be read as "no entries", which would pass vacuously.
return {
ok: false,
errors: [`${source}: baseline must be a JSON object, got ${Array.isArray(doc) ? 'array' : typeof doc}`],
};
}
if (doc.version !== undefined && doc.version !== BASELINE_VERSION) {
errors.push(`${source}: unsupported baseline version ${JSON.stringify(doc.version)} (expected ${BASELINE_VERSION})`);
}
if (typeof doc.sha !== 'string' || !/^[0-9a-f]{40}$/.test(doc.sha)) {
errors.push(`${source}: baseline "sha" must be a 40-hex commit sha, got ${JSON.stringify(doc.sha)}`);
} else if (typeof expectedSha === 'string' && expectedSha !== '' && doc.sha !== expectedSha) {
// THE staleness gate. Never silently used.
errors.push(
`${source}: STALE baseline — built at ${doc.sha} but this PR is being evaluated ` +
`against next@${expectedSha}. A stale baseline mis-attributes silently, so it is ` +
'refused rather than used. Rebuild it, or let the in-job fallback run.',
);
}
if (!doc.manifests || typeof doc.manifests !== 'object' || Array.isArray(doc.manifests)) {
errors.push(`${source}: baseline "manifests" must be an object keyed by runtime`);
}
if (errors.length) return { ok: false, errors };
return {
ok: true,
baseline: doc.manifests,
sizeBaseline: (doc.sizes && typeof doc.sizes === 'object' && !Array.isArray(doc.sizes))
? doc.sizes
: null,
sha: doc.sha,
};
}
/**
* Resolve the baseline through the documented precedence, reporting WHICH step supplied
* it so a failure message can say where the answer came from.
*
* 1. `GSD_EMITTED_BASELINE` — explicit path (CI cache restore writes here)
* 2. the on-disk cache, validated against the expected sha
* 3. an in-job build at `origin/next` (slow fallback)
* 4. none → explicit failure (NEVER a silent pass)
*
* @param {object} opts
* @param {string} opts.expectedSha `next` sha under test
* @param {object} [opts.env] environment (injected)
* @param {string} [opts.cachePath]
* @param {function} opts.readJson (path) => parsed | null (null when absent)
* @param {function} [opts.buildFallback] () => artifact | null (the slow path)
* @returns {{ok: true, via: string, baseline: object, sizeBaseline: object|null, sha: string}
* |{ok: false, via: string, errors: string[]}}
*/
function resolveBaseline({
expectedSha,
env = process.env,
cachePath = DEFAULT_CACHE_PATH,
readJson,
buildFallback = null,
} = {}) {
if (typeof readJson !== 'function') {
return { ok: false, via: 'none', errors: ['resolveBaseline: readJson must be supplied'] };
}
const attempts = [];
const envPath = env && env[BASELINE_ENV];
if (envPath) {
let doc = null;
let readError = null;
try {
doc = readJson(envPath);
} catch (err) {
readError = `${BASELINE_ENV}=${envPath}: ${err.message}`;
}
if (readError) {
attempts.push(readError);
} else {
const v = validateBaseline(doc, expectedSha, `${BASELINE_ENV}=${envPath}`);
if (v.ok) return { ok: true, via: `env:${BASELINE_ENV}`, ...v };
attempts.push(...v.errors);
// An EXPLICITLY pointed-at baseline that is stale or malformed is a hard stop, not
// a reason to quietly fall through to a different one — the operator said "use this".
return { ok: false, via: `env:${BASELINE_ENV}`, errors: attempts };
}
}
let cacheDoc = null;
let cacheErr = null;
try {
cacheDoc = readJson(cachePath);
} catch (err) {
cacheErr = `${cachePath}: ${err.message}`;
}
if (cacheErr) {
attempts.push(cacheErr);
} else if (cacheDoc !== null && cacheDoc !== undefined) {
const v = validateBaseline(cacheDoc, expectedSha, cachePath);
if (v.ok) return { ok: true, via: `cache:${cachePath}`, ...v };
// A stale CACHE is recoverable: fall through to the build fallback, but keep the
// reason so the final message explains why the slow path ran.
attempts.push(...v.errors);
} else {
attempts.push(`${cachePath}: absent`);
}
if (typeof buildFallback === 'function') {
let built = null;
try {
built = buildFallback();
} catch (err) {
attempts.push(`in-job build at origin/next failed: ${err.message}`);
return { ok: false, via: 'build', errors: attempts };
}
const v = validateBaseline(built, expectedSha, 'in-job build at origin/next');
if (v.ok) return { ok: true, via: 'build', ...v };
attempts.push(...v.errors);
return { ok: false, via: 'build', errors: attempts };
}
attempts.push(
'no baseline available and no in-job build fallback was supplied. This is a hard ' +
'failure on purpose: a skipped propagation gate is worth less than a slow one ' +
'(ADR-2719 §6), and in node:test a bare `return` is a PASS, not a skip.',
);
return { ok: false, via: 'none', errors: attempts };
}
module.exports = {
BASELINE_ENV,
BASELINE_VERSION,
DEFAULT_CACHE_PATH,
validateBaseline,
resolveBaseline,
};

View File

@@ -0,0 +1,319 @@
'use strict';
/**
* Differential emitted-artifact attribution — the conservation law (ADR-2719 §1,
* issue #2723, epic #2719 Phase 3).
*
* Given the emitted manifests at `next` HEAD and at PR HEAD, plus the repo paths the
* PR actually changed, decide which moved emitted paths are EXPLAINED by the diff and
* which are not. Unattributable deltas are a hard failure that names them; the only
* way through is a committed acknowledgment (`tests/emitted-drift-ack.json`).
*
* ── Why this module is pure ──────────────────────────────────────────────────
* No fs, no git, no installer, no clock. The naive shape — one integration test that
* builds 19 manifests at each end and asserts — cannot practically exercise the four
* failing-first criteria #2723 requires, so in practice they would not get written,
* which is precisely how a phase ships promised-but-not-built. Keeping the law pure
* makes every criterion a millisecond-scale table test, and makes the Stryker gate
* able to bite (a 20-branch pure function is mutation-testable; a 40-minute
* integration test is not).
*
* The expensive part — obtaining the manifests — lives in emitted-baseline.cjs.
*
* ── What this does NOT do ────────────────────────────────────────────────────
* It never re-derives a byte (ADR-2719 §1 is explicit that asserting
* `emitted == transform(source)` is the tautology ADR-2264's Amendment rejected).
* It constrains which keys may move. It also never mutates repo state: no
* regeneration, no auto-ack. `UPDATE_GOLDEN=1` is exactly the escape hatch this
* design removes.
*/
const { attributeEmittedPath } = require('./emitted-provenance.cjs');
/** Ack schema version. Pinned from day one: contributors hand-write this file, so its
* shape is public the moment it ships (Hyrum). Loosening later is easy; tightening is not. */
const ACK_VERSION = 1;
/**
* Does a changed repo path satisfy a provenance `sources` entry?
*
* A trailing `/` marks a PREFIX (Phase 2's SOURCE_PREFIX_SUFFIX contract) — e.g. Kimi's
* root agent aggregates all of `agents/`. Prefix matching is SEGMENT-AWARE on purpose:
* a bare `startsWith('agents/')` would also accept `agentsfoo/x.md` under a source of
* `agents`, and silently over-attribute. Exact entries compare exactly.
*/
function sourceSatisfiedBy(source, changedSet) {
if (source.endsWith('/')) {
for (const changed of changedSet) {
if (changed.startsWith(source)) return changed;
}
return null;
}
return changedSet.has(source) ? source : null;
}
/**
* Normalize + validate the acknowledgment document.
*
* Rejects a document that parses but is not a plain object. Treating `0` / `"s"` / `[]` /
* `null` / `true` as "no acks" would SILENTLY DISARM the gate — the single worst failure
* available here, because it looks identical to a healthy run.
*
* @returns {{ entries: Map<string, {reason: string, runtime?: string}>, errors: string[] }}
*/
function parseAck(doc, { source = 'emitted-drift-ack.json' } = {}) {
const errors = [];
const entries = new Map();
if (doc === null || doc === undefined) return { entries, errors }; // absent == no acks (legal)
if (typeof doc !== 'object' || Array.isArray(doc)) {
errors.push(
`${source}: must be a JSON object, got ${Array.isArray(doc) ? 'array' : typeof doc}`,
);
return { entries, errors };
}
if (doc.version !== undefined && doc.version !== ACK_VERSION) {
errors.push(`${source}: unsupported version ${JSON.stringify(doc.version)} (expected ${ACK_VERSION})`);
}
const paths = doc.paths;
if (paths === undefined) return { entries, errors }; // `{}` or `{version:1}` == no acks
if (paths === null || typeof paths !== 'object' || Array.isArray(paths)) {
errors.push(`${source}: "paths" must be an object of <emitted path> -> { reason }`);
return { entries, errors };
}
for (const [rel, value] of Object.entries(paths)) {
const reason = value && typeof value === 'object' ? value.reason : value;
if (typeof reason !== 'string' || reason.trim() === '') {
// "name them AND say why" is the contract (ADR-2719 §3). An ack with no reason
// is a silent regeneration wearing a declaration's clothes.
errors.push(`${source}: ack for "${rel}" has no non-empty "reason"`);
continue;
}
entries.set(rel, {
reason: reason.trim(),
runtime: value && typeof value === 'object' ? value.runtime : undefined,
});
}
return { entries, errors };
}
/**
* The conservation law.
*
* @param {object} opts
* @param {object} opts.baseline { [runtime]: { [rel]: hash } } at `next` HEAD
* @param {object} opts.current { [runtime]: { [rel]: hash } } at PR HEAD
* @param {string[]} opts.changedPaths repo paths the PR changed (git diff --name-only)
* @param {object} [opts.ack] parsed emitted-drift-ack.json document (or null)
* @param {object} [opts.sizeBaseline] { [name]: bytes } workflow/agent sizes at next
* @param {object} [opts.sizeCurrent] { [name]: bytes } workflow/agent sizes at PR HEAD
*
* @returns {{
* moved: number, attributed: Array, unattributable: Array, acked: Array,
* removed: Array, grown: Array, shrunk: Array, staleAcks: string[], errors: string[], ok: boolean
* }}
*/
function diffEmitted({
baseline,
current,
changedPaths,
ack = null,
sizeBaseline = null,
sizeCurrent = null,
} = {}) {
const errors = [];
if (!baseline || typeof baseline !== 'object' || Array.isArray(baseline)) {
errors.push('baseline manifest set must be an object keyed by runtime');
}
if (!current || typeof current !== 'object' || Array.isArray(current)) {
errors.push('current manifest set must be an object keyed by runtime');
}
if (!Array.isArray(changedPaths)) {
// NOT the same as an empty array. A failed `git diff` must never be read as
// "nothing changed" — that would make every moved hash unattributable and produce
// a failure storm that reads like a real finding.
errors.push('changedPaths must be an array (a failed git diff is an error, not an empty set)');
}
if (errors.length) {
return {
moved: 0, attributed: [], unattributable: [], acked: [], removed: [],
grown: [], shrunk: [], staleAcks: [], errors, ok: false,
};
}
const changedSet = new Set(changedPaths);
const { entries: ackEntries, errors: ackErrors } = parseAck(ack);
errors.push(...ackErrors);
const attributed = [];
const unattributable = [];
const acked = [];
const removed = [];
const usedAcks = new Set();
let moved = 0;
const runtimes = new Set([...Object.keys(baseline), ...Object.keys(current)]);
for (const runtime of [...runtimes].sort()) {
const before = baseline[runtime] || {};
const after = current[runtime] || {};
const keys = new Set([...Object.keys(before), ...Object.keys(after)]);
for (const rel of [...keys].sort()) {
const had = Object.prototype.hasOwnProperty.call(before, rel);
const has = Object.prototype.hasOwnProperty.call(after, rel);
if (had && has && before[rel] === after[rel]) continue; // unchanged — ignored
const change = !had ? 'added' : (!has ? 'removed' : 'modified');
moved++;
let attribution;
try {
attribution = attributeEmittedPath(rel, runtime);
} catch (err) {
// A path the Phase 2 table cannot resolve is surfaced, never silently skipped —
// otherwise a table hole becomes a blind spot in the differential too.
errors.push(`${runtime}: ${rel}: ${err.message}`);
continue;
}
const record = { runtime, rel, change, ruleId: attribution.ruleId, kind: attribution.kind };
if (change === 'removed') removed.push(record);
// Synthesized paths carry no repo source by definition, so a delta in them can
// never be "unexplained by the diff" — exempt, but still counted and reported.
if (attribution.kind === 'synthesized') {
attributed.push({ ...record, via: '<synthesized: exempt>' });
continue;
}
let via = null;
for (const source of attribution.sources) {
const hit = sourceSatisfiedBy(source, changedSet);
// `!== null`, not truthiness: an exact match returns the source string, and an
// empty-string source would return '' — falsy, so a real match would be
// silently discarded. Unreachable with today's rules (every source is a
// non-empty template) but it is a footgun for the next rule author.
if (hit !== null) { via = hit; break; }
}
if (via !== null) {
attributed.push({ ...record, via });
} else if (ackEntries.has(rel)) {
usedAcks.add(rel);
acked.push({ ...record, reason: ackEntries.get(rel).reason });
} else {
unattributable.push({ ...record, expectedSources: attribution.sources });
}
}
}
// ── Size ratchet, folded into the same machine (ADR-2719 §4, must-have 6) ──
// NOTE: stale-ack detection is computed AFTER this block, not before. An ack may be
// consumed by either a hash move or a size growth, so computing it earlier would
// report a size-growth ack as stale.
const grown = [];
const shrunk = [];
if (sizeBaseline && sizeCurrent) {
for (const name of Object.keys(sizeCurrent).sort()) {
if (!Object.prototype.hasOwnProperty.call(sizeBaseline, name)) continue;
const from = sizeBaseline[name];
const to = sizeCurrent[name];
if (to > from) {
// Growth needs the SAME acknowledgment. Anti-creep survives without pinning a
// number: "verify-work.md grew 1,247 bytes" beats a number moving in a 93-line map.
const isAcked = ackEntries.has(name);
if (isAcked) usedAcks.add(name);
grown.push({ name, from, to, delta: to - from, acked: isAcked });
} else if (to < from) {
// Shrinkage is not creep — reported, never gated.
shrunk.push({ name, from, to, delta: from - to });
}
}
}
const unackedGrowth = grown.filter((g) => !g.acked);
// An ack that outlives the ripple it explained is future blindness: it would silently
// pre-clear a NEW ripple on the same path. It must be deleted when the ripple is.
// Computed here, once, after BOTH the hash pass and the size pass have consumed acks.
const staleAcks = [...ackEntries.keys()].filter((rel) => !usedAcks.has(rel)).sort();
const ok = errors.length === 0
&& unattributable.length === 0
&& unackedGrowth.length === 0
&& staleAcks.length === 0;
return {
moved,
attributed,
unattributable,
acked,
removed,
grown,
shrunk,
staleAcks,
errors,
ok,
};
}
/**
* Render a report as the failure message ADR-2719 §1 specifies — it sells the whole
* design on this text, so it is a deliverable, not a detail.
*/
function formatReport(result, { sampleLimit = 20 } = {}) {
const parts = [];
if (result.errors.length) {
parts.push(`${result.errors.length} error(s):\n ${result.errors.slice(0, sampleLimit).join('\n ')}`);
}
if (result.unattributable.length) {
const list = result.unattributable.slice(0, sampleLimit)
.map((u) => ` ${u.runtime}: ${u.rel}\n rule ${u.ruleId}; expected a change under ${u.expectedSources.join(' or ')}`);
parts.push(
`${result.unattributable.length} emitted path(s) changed that nothing in this diff explains:\n${list.join('\n')}` +
(result.unattributable.length > sampleLimit
? `\n …and ${result.unattributable.length - sampleLimit} more`
: '') +
'\n\nIf this ripple is intended, record it in tests/emitted-drift-ack.json naming each\n' +
'path and why. Do NOT regenerate anything to silence this.',
);
}
const unackedGrowth = result.grown.filter((g) => !g.acked);
if (unackedGrowth.length) {
const list = unackedGrowth.slice(0, sampleLimit)
.map((g) => ` ${g.name} grew ${g.delta} bytes (${g.from} -> ${g.to})`);
parts.push(
`${unackedGrowth.length} file(s) grew without an acknowledgment:\n${list.join('\n')}`,
);
}
if (result.staleAcks.length) {
parts.push(
`${result.staleAcks.length} stale acknowledgment(s) — the ripple they explained is gone, ` +
'so they must be deleted (an ack that outlives its ripple pre-clears the next one):\n ' +
result.staleAcks.slice(0, sampleLimit).join('\n '),
);
}
return parts.join('\n\n');
}
module.exports = {
ACK_VERSION,
sourceSatisfiedBy,
parseAck,
diffEmitted,
formatReport,
};

View File

@@ -0,0 +1,269 @@
'use strict';
/**
* Real-world I/O shell for the differential attribution check (#2723, ADR-2719).
*
* `emitted-diff.cjs` holds the pure conservation law; this module is the only place
* that touches git, the filesystem, or the installer. Keeping them apart is what makes
* the law's acceptance criteria testable in milliseconds — but the shell still has to
* exist and actually run, or the phase ships as interface-only, which an isolated
* reviewer correctly called out on the first cut of this work.
*
* ── Baseline source during the dual-run window ───────────────────────────────
* The baseline is the emitted manifest set at `next` HEAD. During Phase 3 that is
* available for FREE and for REAL via `git show origin/next:<fixture>` — the committed
* golden fixtures ARE next's recorded emitted state, and CI keeps them current there.
* No worktree, no rebuild, no 19 installer spawns for the baseline side.
*
* Critically this is NOT the same as reading the fixtures from the WORKING TREE: those
* are whatever the PR author regenerated, so comparing against them would be vacuous
* (current vs. the author's own regeneration). Reading them at `origin/next` is what
* makes the comparison a real differential against upstream state.
*
* Phase 4 (#2724) deletes the fixtures, at which point `resolveBaseline`'s cache path
* (already implemented and tested in emitted-baseline.cjs) becomes the source. That
* swap is the only change Phase 4 needs here.
*
* The CURRENT side is built for real — 19 installer spawns via the same
* `runMinimalInstall` + `buildParityManifest` the golden harness uses. It is the
* expensive half on purpose: if a PR forgot to regenerate, current-real differs from
* next's recorded state and the attribution actually runs, which is the whole point.
*/
const fs = require('node:fs');
const path = require('node:path');
const { execFileSync } = require('node:child_process');
const { cleanup } = require('../helpers.cjs');
const {
RUNTIME_META,
runMinimalInstall,
buildParityManifest,
} = require('./install-shared.cjs');
const REPO_ROOT = path.join(__dirname, '..', '..');
const ACK_PATH = path.join(REPO_ROOT, 'tests', 'emitted-drift-ack.json');
const FIXTURE_SUBDIR = 'tests/fixtures/golden-install-parity';
/**
* The emitted manifest families, as (fixtureName -> install spec).
*
* NOT simply `Object.keys(RUNTIME_META)`: that has 18 entries while the fixture set has
* 19. The extra one is `claude-local` — claude is the reference host and the ONLY
* runtime with a distinct LOCAL "legacy flat-commands" layout (`commands/gsd-*.md` +
* `agents/gsd-*.md` at project scope), which `golden-install-parity.test.cjs` guards
* with a hand-coded test outside its RUNTIME_META loop (#2086).
*
* Enumerating from RUNTIME_META alone dropped that family from BOTH sides of the
* differential, so a same-count self-check (18 === 18) passed vacuously and a PR
* changing Claude's local-scope output would fail the golden while this check reported
* ok. That disagreement is exactly what the dual-run window is meant to surface as a
* provenance-table hole — so a wiring omission masquerading as one is the worst
* possible failure here. Derived explicitly, and asserted against the fixture count.
*/
const MANIFEST_FAMILIES = [
...Object.keys(RUNTIME_META).map((runtime) => ({ name: runtime, runtime, scope: 'global' })),
{ name: 'claude-local', runtime: 'claude', scope: 'local' },
];
/** Bounded git invocation. CLAUDE.md → KNOWN DEFECTS: every git subprocess needs a
* timeout (5-30s); an unbounded execFileSync is an indefinite hang, and it is how
* macOS CI silently stops reporting. */
const GIT_TIMEOUT_MS = 30_000;
function git(args, { cwd = REPO_ROOT } = {}) {
return execFileSync('git', args, {
cwd,
encoding: 'utf8',
timeout: GIT_TIMEOUT_MS,
maxBuffer: 64 * 1024 * 1024,
stdio: ['ignore', 'pipe', 'pipe'],
});
}
/**
* Repo paths the PR changed, via the three-dot form so the comparison is against the
* merge base rather than the tip of `base`.
*
* A git failure THROWS. It must never degrade to an empty array: reading "git broke" as
* "nothing changed" would make every moved hash unattributable and produce a failure
* storm that reads exactly like a real finding.
*/
function resolveChangedPaths(base = 'origin/next') {
let out;
try {
out = git(['diff', '--name-only', `${base}...HEAD`]);
} catch (err) {
throw new Error(
`emitted-attribution: could not resolve changed paths from "${base}...HEAD": ${err.message}. ` +
'This is a hard error on purpose — treating it as "no changes" would mark every ' +
'moved emitted path unattributable.',
);
}
return out.split('\n').map((l) => l.trim()).filter(Boolean);
}
/** Resolve `base` to a 40-hex sha, for the baseline cache-key discipline (ADR §5). */
function resolveBaseSha(base = 'origin/next') {
return git(['rev-parse', base]).trim();
}
/**
* Base-ref candidates, most-specific first.
*
* The differential needs a ref for `next`, and that ref is NOT universally present:
* - the gsd-test runner shallow-clones and merges base+head, so no `origin/*`
* remote-tracking refs exist in the container (verified: `git rev-parse
* origin/next` fails there, which is what turned this test red on its first run);
* - GitHub Actions' checkout does not create remote-tracking branches for OTHER
* branches by default, which is exactly why `changeset-required.yml` carries an
* explicit `git fetch origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}"` step.
*
* `GSD_EMITTED_BASE` lets a lane name the ref (or sha) directly. `GITHUB_BASE_REF` is
* set by Actions on pull_request events.
*/
function baseRefCandidates(env = process.env) {
const candidates = [];
if (env.GSD_EMITTED_BASE) candidates.push(env.GSD_EMITTED_BASE);
if (env.GITHUB_BASE_REF) {
candidates.push(`origin/${env.GITHUB_BASE_REF}`, env.GITHUB_BASE_REF);
}
candidates.push('origin/next', 'next');
return [...new Set(candidates)];
}
/**
* First candidate base ref that actually resolves, or null when none do.
*
* Returning null is NOT a pass — the caller turns it into an explicit `t.skip()` with
* the full candidate list in the message, so an environment where the gate did not run
* says so out loud. A bare `return` there would be a PASS (ADR-2719 §6), and a hard
* failure would make the suite permanently red in the gsd-test container, where no
* base ref can exist by construction.
*/
function resolveBase(env = process.env) {
for (const candidate of baseRefCandidates(env)) {
try {
const sha = git(['rev-parse', '--verify', `${candidate}^{commit}`]).trim();
if (/^[0-9a-f]{40}$/.test(sha)) return { ref: candidate, sha };
} catch { /* try the next candidate */ }
}
return null;
}
/**
* Emitted manifest set at `base`, read from the committed fixtures at that ref.
* Returns null when the fixtures are absent at `base` (i.e. after Phase 4's cutover),
* which is the signal to fall back to `resolveBaseline`'s cache path.
*/
function baselineManifestsAtRef(base = 'origin/next') {
const manifests = {};
let found = 0;
for (const { name } of MANIFEST_FAMILIES) {
let raw;
try {
raw = git(['show', `${base}:${FIXTURE_SUBDIR}/${name}.json`]);
} catch {
continue; // absent at that ref
}
let parsed;
try {
parsed = JSON.parse(raw);
} catch (err) {
throw new Error(`emitted-attribution: ${base}:${FIXTURE_SUBDIR}/${name}.json is not valid JSON: ${err.message}`);
}
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new Error(`emitted-attribution: ${base}:${FIXTURE_SUBDIR}/${name}.json must be an object of path->hash`);
}
manifests[name] = parsed;
found++;
}
return found === 0 ? null : manifests;
}
/** Size maps at `base`, for the ratchet half. Null when absent at that ref. */
function baselineSizesAtRef(base = 'origin/next') {
const sizes = {};
let found = 0;
for (const rel of ['tests/workflow-size-baseline.json', 'tests/agent-size-baseline.json']) {
try {
const parsed = JSON.parse(git(['show', `${base}:${rel}`]));
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
Object.assign(sizes, parsed);
found++;
}
} catch { /* absent at that ref */ }
}
return found === 0 ? null : sizes;
}
/**
* Build the CURRENT emitted manifest set for real — one installer spawn per runtime.
* This is the expensive, honest half: it reflects what the tree actually emits now,
* not what the author regenerated into a fixture.
*/
function currentManifests() {
const manifests = {};
for (const { name, runtime, scope } of MANIFEST_FAMILIES) {
const { configDir, root } = runMinimalInstall({ runtime, scope });
try {
manifests[name] = buildParityManifest(configDir, root);
} finally {
cleanup(root);
}
}
return manifests;
}
/** Current on-disk sizes for the workflow + agent families the ratchet covers. */
function currentSizes() {
const sizes = {};
for (const [dir, filter] of [
[path.join(REPO_ROOT, 'gsd-core', 'workflows'), (f) => f.endsWith('.md')],
[path.join(REPO_ROOT, 'agents'), (f) => f.endsWith('.md')],
]) {
if (!fs.existsSync(dir)) continue;
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
if (!entry.isFile() || !filter(entry.name)) continue;
sizes[entry.name] = fs.statSync(path.join(dir, entry.name)).size;
}
}
return sizes;
}
/**
* Read `tests/emitted-drift-ack.json`.
* Absent is legal and means "no acks" — its PRESENCE is the alarm (ADR §3).
* A present-but-unreadable or unparseable file THROWS: silently treating it as absent
* would disarm the gate in the one case where someone is actively using it.
*/
function readAckFile(ackPath = ACK_PATH) {
if (!fs.existsSync(ackPath)) return null;
const raw = fs.readFileSync(ackPath, 'utf8');
if (raw.trim() === '') {
throw new Error(`emitted-attribution: ${path.basename(ackPath)} is present but empty`);
}
try {
return JSON.parse(raw);
} catch (err) {
throw new Error(`emitted-attribution: ${path.basename(ackPath)} is not valid JSON: ${err.message}`);
}
}
module.exports = {
REPO_ROOT,
ACK_PATH,
FIXTURE_SUBDIR,
MANIFEST_FAMILIES,
GIT_TIMEOUT_MS,
git,
resolveChangedPaths,
resolveBaseSha,
baseRefCandidates,
resolveBase,
baselineManifestsAtRef,
baselineSizesAtRef,
currentManifests,
currentSizes,
readAckFile,
};