* test(#2723): differential emitted-attribution check, dual-run beside the golden Phase 3 of #2719. The conservation law itself, running BESIDE golden-install-parity.test.cjs -- both green, fixtures untouched. Every emitted path whose hash moved between next HEAD and PR HEAD must be attributable, through the Phase 2 table, to a path the PR actually changed. Unattributable deltas fail with the paths NAMED. The only way through is a committed acknowledgment, never a flag -- a contributor facing a red gate sets a flag, which is what UPDATE_GOLDEN=1 is today. The central decision is that the law is a PURE function (no fs, git, installer, or clock), with I/O confined to a separate resolver. The naive one-big-integration-test shape would need ~38 installer spawns per assertion, so #2723's four failing-first criteria would not in practice have been written -- which is exactly how a phase ships promised-but-not-built. Pure, they are millisecond table tests, and the Stryker gate can actually bite. Buckets are conserved: every moved path lands in exactly one of attributed | unattributable | acked, property-tested at 400 runs. A path the provenance table cannot resolve surfaces as an error, never a silent skip. Asymmetries that are deliberate, each with a test: - an ADDED emitted key is a ripple too, not just a modified one - synthesized paths are exempt; code-derived ones are NOT (Phase 2 refused to mark them exempt precisely because exempt means permanently blind) - shrinkage needs no ack; growth does. Gating shrinkage would punish exactly what the size ratchet wants - a STALE ack is a hard failure -- an ack outliving its ripple pre-clears the next one on that path - a failed `git diff` is an explicit error, never an empty changedPaths set; reading it as "nothing changed" would make everything unattributable and produce a failure storm that reads like a real finding - prefix sources are SEGMENT-aware, so `agents/` does not attribute `agentsfoo/x.md` Baseline is cached, not committed, keyed on the next sha. A stale key is refused rather than used: absence fails loudly and gets fixed, whereas staleness produces a confident wrong answer. An explicitly pointed-at GSD_EMITTED_BASELINE that is stale is a hard stop; a stale cache falls through to the in-job build. No baseline-unavailable path returns -- ADR-2719 section 6 names that trap, since in node:test a bare return is a PASS. Both the conservation property and the staleness gate were mutation-verified (injecting a swallowed key fails 9 tests; disabling the staleness comparison fails 5). Fixtures, generators, the merge driver and the ADR status are untouched -- those are Phase 4 (#2724). Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * test(#2723): compute stale acks once, after the size pass Self-review defect found while the reviewers were running. `staleAcks` was computed twice: once between the hash pass and the size pass, then again after. Only the second value was returned, so the first was dead code -- and the dead one was placed where it would have been WRONG. An acknowledgment can be consumed by either a hash move or a size growth. Computing staleness before the size pass reports a legitimate growth ack as stale, which is a false failure that pushes a contributor to delete the very ack that is doing its job. Now computed once, after both passes, with a regression test. Verified by mutation: restoring the early computation fails 2 tests. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * test(#2723): wire the attribution check to the real tree, not just synthetic input An isolated reviewer caught that the first cut was INTERFACE-ONLY: nothing read the ack file from disk, nothing shelled git, nothing built real manifests. Every test was true of hand-built inputs and none of the repo, so the acceptance criterion "both this check and golden-install-parity green on the same tree" was trivially true rather than meaningfully true. That is the promised-but-not-built failure this epic keeps finding in its predecessors, recurring one phase later for the wiring itself. Taken, not argued. Adds tests/helpers/emitted-runtime.cjs -- the only module that touches git, disk, or the installer -- and an integration test that runs the same pure law against reality: - CURRENT side: 19 real installer spawns via runMinimalInstall + buildParityManifest, the same machinery the golden harness uses. - BASELINE side: `git show origin/next:<fixture>`. That is next's RECORDED emitted state and it costs nothing. Deliberately NOT the working-tree fixtures, which are whatever this PR's author regenerated -- comparing against those would be vacuous. Phase 4 deletes the fixtures and swaps in resolveBaseline's cache path, already implemented and tested. - changed paths from real `git diff --name-only origin/next...HEAD`, with the git subprocess bounded at 30s per CLAUDE.md's unbounded-subprocess rule. - the real tests/emitted-drift-ack.json (absent is legal; present-but-empty or unparseable throws rather than being read as absent). Verified it can actually fail: an uncommitted edit to a shipped workflow moves emitted output but never appears in the committed diff, and the check names all 18 affected emitted paths with the message format ADR-2719 §1 specifies. Restores clean. Also from review: - readAckFile now has a real test exercising the SUT across absent / valid / empty / unparseable / unreadable. The previous test asserted fs behaviour rather than SUT behaviour, because no SUT ack-reading path existed yet. - formatReport's sampleLimit gains true limit-1/limit/limit+1 coverage at 19/20/21. A test was previously NAMED "(limit+1)" while testing no numeric limit at all, which is worse than no coverage because it reads as covered. Windows uses an explicit t.skip (install output is platform-specific there, mirroring the golden harness) -- never a bare return, which node:test scores as a PASS. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * test(#2723): cover the claude-local manifest family in the real-tree check Isolated adversarial review, MAJOR. The real-tree wiring enumerated Object.keys(RUNTIME_META) -- 18 entries -- while the emitted manifest set has 19 families. The 19th is claude-local: claude is the reference host and the only runtime with a distinct LOCAL "legacy flat-commands" layout (commands/gsd-*.md + agents/gsd-*.md at project scope), which golden-install-parity.test.cjs guards with a hand-coded test outside its RUNTIME_META loop (#2086). The family was dropped from BOTH sides, so the test's own self-check (current.length === baseline.length) passed vacuously at 18 === 18. A PR changing Claude's local-scope output would have failed the golden while this check reported ok -- and that disagreement is precisely what the dual-run window is designed to surface as a provenance-table hole. A wiring omission masquerading as one is the worst available failure here, because it would have been read as evidence about Phase 2 rather than a bug in Phase 3. Fixed by deriving MANIFEST_FAMILIES explicitly (18 global + claude-local at local scope) instead of inferring the set from RUNTIME_META. The self-check is also repaired: it now asserts both sides against the INDEPENDENT EXPECTED_MANIFEST_COUNT from the Phase 2 table, and asserts claude-local specifically. Comparing the two sides to each other can never catch a family missing from both -- the assertion has to come from outside. Verified by mutation: removing claude-local again fails the test. Also from the same review: - sourceSatisfiedBy returns the matched source string, so an empty-string source would return '' and the caller's `if (hit)` would silently discard a real match. Unreachable today (every rule source is a non-empty template) but a footgun for the next rule author; now `!== null`. - the purity fixture used a single-element changedPaths array, so an in-place sort would have been invisible. Now three elements in unsorted order. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 * fix(#2723): resolve the base ref tolerantly instead of hard-requiring origin/next The first matrix run failed on both linux lanes: differential attribution over the real tree cannot resolve origin/next (Command failed: git rev-parse origin/next) Not a flake, and not an environment excuse -- a real defect in this diff. The gsd-test runner shallow-clones and merges base+head, so no origin/* remote- tracking refs exist in the container. My own fail-loud path fired correctly; what was wrong was hard-depending on that ref existing. GitHub Actions has the same shape by default, which is exactly why changeset-required.yml carries an explicit `git fetch origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}"`. Now resolved through an ordered candidate list -- GSD_EMITTED_BASE (explicit lane override), then origin/$GITHUB_BASE_REF and $GITHUB_BASE_REF, then origin/next and next -- de-duplicated, each verified with `rev-parse --verify <ref>^{commit}`. When NO candidate resolves the test takes an explicit t.skip() naming every ref it tried and stating that the gate did not run here. That is the ADR-2719 section 6 distinction: t.skip is REPORTED as skipped, whereas a bare return is scored as a PASS. Hard-failing was the other option and is wrong -- it would make the suite permanently red wherever a base ref cannot exist by construction, which is a statement about the checkout, not a propagation finding. The candidate ordering is pinned by a unit test rather than left implicit, since the ordering IS the fix. Refs #2719 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W5kQs6ZufZDySC6zDJfYP6 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
File diff suppressed because one or more lines are too long
752
tests/emitted-attribution.test.cjs
Normal file
752
tests/emitted-attribution.test.cjs
Normal file
@@ -0,0 +1,752 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* emitted-attribution.test.cjs — the differential attribution check (#2723,
|
||||
* ADR-2719 §1/§3/§4/§5/§6, epic #2719 Phase 3).
|
||||
*
|
||||
* Runs BESIDE tests/golden-install-parity.test.cjs — both green, fixtures untouched.
|
||||
* Any PR where the golden fails and this passes is a Phase 2 provenance-table hole;
|
||||
* that disagreement is the entire point of the dual-run window, and Phase 4 (#2724)
|
||||
* must not land until it has been observed on real PRs.
|
||||
*
|
||||
* The law: every emitted path whose hash moved between `next` HEAD and PR HEAD must be
|
||||
* attributable — through the Phase 2 table — to a path the PR actually changed.
|
||||
* Unattributable deltas fail with the paths NAMED. The only way through is a committed
|
||||
* acknowledgment, never a flag (a contributor facing a red gate sets a flag, which is
|
||||
* what UPDATE_GOLDEN=1 is today).
|
||||
*
|
||||
* Structure: the pure law is exercised against synthetic manifests, which is what makes
|
||||
* the four failing-first criteria practical to assert at all — and then the final test
|
||||
* runs that same law against the REAL tree: 19 actual installer spawns for the current
|
||||
* side, `git show origin/next:<fixture>` for the baseline side, real `git diff` for the
|
||||
* changed paths, and the real `tests/emitted-drift-ack.json`.
|
||||
*
|
||||
* That last test is load-bearing. Without it this file would be interface-only — every
|
||||
* assertion true of hand-built inputs and none of the repo — which is the
|
||||
* promised-but-not-built failure this epic keeps finding in its own predecessors.
|
||||
* Verified by injecting an uncommitted edit to a shipped workflow: emitted output moves
|
||||
* but the path never appears in `git diff origin/next...HEAD`, and the check names all
|
||||
* 18 affected emitted paths.
|
||||
*/
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { execFileSync } = require('node:child_process');
|
||||
const fc = require('fast-check');
|
||||
|
||||
const { cleanup } = require('./helpers.cjs');
|
||||
const { BUILD_SCRIPT } = require('./helpers/install-shared.cjs');
|
||||
const {
|
||||
resolveChangedPaths,
|
||||
resolveBase,
|
||||
baseRefCandidates,
|
||||
baselineManifestsAtRef,
|
||||
baselineSizesAtRef,
|
||||
currentManifests,
|
||||
currentSizes,
|
||||
readAckFile,
|
||||
} = require('./helpers/emitted-runtime.cjs');
|
||||
|
||||
const { EXPECTED_MANIFEST_COUNT } = require('./helpers/emitted-provenance.cjs');
|
||||
const {
|
||||
ACK_VERSION,
|
||||
sourceSatisfiedBy,
|
||||
parseAck,
|
||||
diffEmitted,
|
||||
formatReport,
|
||||
} = require('./helpers/emitted-diff.cjs');
|
||||
|
||||
const {
|
||||
BASELINE_ENV,
|
||||
BASELINE_VERSION,
|
||||
resolveBaseline,
|
||||
} = require('./helpers/emitted-baseline.cjs');
|
||||
|
||||
const SHA_A = 'a'.repeat(40);
|
||||
const SHA_B = 'b'.repeat(40);
|
||||
|
||||
/** A real emitted key + its real source, so rows assert the shape production uses. */
|
||||
const WORKFLOW_KEY = 'gsd-core/workflows/plan-phase.md';
|
||||
const WORKFLOW_SRC = 'gsd-core/workflows/plan-phase.md';
|
||||
const SKILL_KEY = 'skills/gsd-add-tests/SKILL.md';
|
||||
const SKILL_SRC = 'commands/gsd/add-tests.md';
|
||||
|
||||
const mf = (obj) => ({ claude: obj });
|
||||
|
||||
// ─── Attribution: the conservation law ───────────────────────────────────────
|
||||
|
||||
test('unchanged hashes are not reported', () => {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
changedPaths: [],
|
||||
});
|
||||
assert.equal(r.moved, 0);
|
||||
assert.equal(r.attributed.length, 0);
|
||||
assert.equal(r.unattributable.length, 0);
|
||||
assert.ok(r.ok);
|
||||
});
|
||||
|
||||
test('a moved hash whose source changed is attributed', () => {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
|
||||
changedPaths: [WORKFLOW_SRC],
|
||||
});
|
||||
assert.equal(r.moved, 1);
|
||||
assert.equal(r.unattributable.length, 0);
|
||||
assert.equal(r.attributed.length, 1);
|
||||
assert.equal(r.attributed[0].via, WORKFLOW_SRC);
|
||||
assert.ok(r.ok);
|
||||
});
|
||||
|
||||
test('a trailing-slash source entry matches by prefix, segment-aware', () => {
|
||||
// Kimi's root agent aggregates all of agents/ — a Phase 2 prefix source.
|
||||
assert.equal(sourceSatisfiedBy('agents/', new Set(['agents/gsd-planner.md'])), 'agents/gsd-planner.md');
|
||||
// Hostile: a bare startsWith would over-attribute here. It must NOT match.
|
||||
assert.equal(sourceSatisfiedBy('agents/', new Set(['agentsfoo/x.md'])), null);
|
||||
// Exact entries compare exactly.
|
||||
assert.equal(sourceSatisfiedBy('a/b.md', new Set(['a/b.md'])), 'a/b.md');
|
||||
assert.equal(sourceSatisfiedBy('a/b.md', new Set(['a/b.md.bak'])), null);
|
||||
|
||||
const r = diffEmitted({
|
||||
baseline: { kimi: { 'agents/gsd.yaml': 'aaa' } },
|
||||
current: { kimi: { 'agents/gsd.yaml': 'bbb' } },
|
||||
changedPaths: ['agents/gsd-planner.md'],
|
||||
});
|
||||
assert.equal(r.unattributable.length, 0, 'prefix source should attribute');
|
||||
assert.equal(r.attributed[0].via, 'agents/gsd-planner.md');
|
||||
});
|
||||
|
||||
test('a moved hash nothing explains is unattributable and named', () => {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
|
||||
changedPaths: ['README.md'],
|
||||
});
|
||||
assert.equal(r.unattributable.length, 1);
|
||||
const u = r.unattributable[0];
|
||||
assert.equal(u.rel, WORKFLOW_KEY);
|
||||
assert.equal(u.runtime, 'claude');
|
||||
assert.equal(u.ruleId, 'gsd-core-verbatim');
|
||||
assert.deepEqual(u.expectedSources, [WORKFLOW_SRC]);
|
||||
assert.ok(!r.ok);
|
||||
|
||||
// ADR-2719 §1 sells the design on this message — it is a deliverable.
|
||||
const msg = formatReport(r);
|
||||
assert.match(msg, /changed that nothing in this diff explains/);
|
||||
assert.ok(msg.includes(WORKFLOW_KEY));
|
||||
assert.ok(msg.includes(WORKFLOW_SRC), 'the message must say what WOULD have explained it');
|
||||
});
|
||||
|
||||
test('synthesized paths are exempt from attribution', () => {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ 'gsd-core/VERSION': 'aaa' }),
|
||||
current: mf({ 'gsd-core/VERSION': 'bbb' }),
|
||||
changedPaths: [],
|
||||
});
|
||||
assert.equal(r.unattributable.length, 0, 'install-time state can never be unexplained');
|
||||
assert.equal(r.attributed[0].via, '<synthesized: exempt>');
|
||||
assert.ok(r.ok);
|
||||
});
|
||||
|
||||
test('code-derived paths attribute to their emitting source file', () => {
|
||||
// Phase 2 deliberately refused to mark these exempt; this is why.
|
||||
const r = diffEmitted({
|
||||
baseline: { cline: { '.clinerules/gsd.md': 'aaa' } },
|
||||
current: { cline: { '.clinerules/gsd.md': 'bbb' } },
|
||||
changedPaths: ['src/runtime-hooks-surface.cts'],
|
||||
});
|
||||
assert.equal(r.unattributable.length, 0);
|
||||
assert.equal(r.attributed[0].via, 'src/runtime-hooks-surface.cts');
|
||||
|
||||
const blind = diffEmitted({
|
||||
baseline: { cline: { '.clinerules/gsd.md': 'aaa' } },
|
||||
current: { cline: { '.clinerules/gsd.md': 'bbb' } },
|
||||
changedPaths: ['README.md'],
|
||||
});
|
||||
assert.equal(blind.unattributable.length, 1, 'had these been exempt, this ripple would be invisible forever');
|
||||
});
|
||||
|
||||
test('an added emitted key is a ripple too', () => {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({}),
|
||||
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
|
||||
changedPaths: ['README.md'],
|
||||
});
|
||||
assert.equal(r.moved, 1);
|
||||
assert.equal(r.unattributable.length, 1);
|
||||
assert.equal(r.unattributable[0].change, 'added');
|
||||
});
|
||||
|
||||
test('a removed emitted key is reported', () => {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
current: mf({}),
|
||||
changedPaths: [WORKFLOW_SRC],
|
||||
});
|
||||
assert.equal(r.removed.length, 1);
|
||||
assert.equal(r.removed[0].change, 'removed');
|
||||
assert.equal(r.unattributable.length, 0, 'the deletion is explained by the source change');
|
||||
});
|
||||
|
||||
test('moved hashes with no changed paths are all unattributable', () => {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa', [SKILL_KEY]: 'ccc' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'bbb', [SKILL_KEY]: 'ddd' }),
|
||||
changedPaths: [],
|
||||
});
|
||||
assert.equal(r.unattributable.length, 2, 'emitted output moving with zero source changes is a real finding');
|
||||
assert.ok(!r.ok);
|
||||
});
|
||||
|
||||
test('a failed git diff is an error, not an empty change set', () => {
|
||||
// Treating a git failure as "nothing changed" would make everything unattributable —
|
||||
// a failure storm that reads exactly like a real finding.
|
||||
const r = diffEmitted({ baseline: mf({}), current: mf({}), changedPaths: null });
|
||||
assert.ok(!r.ok);
|
||||
assert.match(r.errors.join('\n'), /changedPaths must be an array/);
|
||||
});
|
||||
|
||||
test('an unattributable-by-table path surfaces as an error', () => {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ 'totally/unknown/thing.md': 'aaa' }),
|
||||
current: mf({ 'totally/unknown/thing.md': 'bbb' }),
|
||||
changedPaths: [],
|
||||
});
|
||||
assert.ok(!r.ok);
|
||||
assert.match(r.errors.join('\n'), /no rule matches/);
|
||||
assert.equal(r.unattributable.length, 0, 'a table hole is an error, not a silent skip');
|
||||
});
|
||||
|
||||
// ─── Acknowledgment file ─────────────────────────────────────────────────────
|
||||
|
||||
test('an acked ripple passes and is echoed', () => {
|
||||
const ack = { version: ACK_VERSION, paths: { [WORKFLOW_KEY]: { reason: 'converter change, #2723' } } };
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
|
||||
changedPaths: ['README.md'],
|
||||
ack,
|
||||
});
|
||||
assert.equal(r.unattributable.length, 0);
|
||||
assert.equal(r.acked.length, 1);
|
||||
assert.equal(r.acked[0].reason, 'converter change, #2723');
|
||||
assert.ok(r.ok);
|
||||
});
|
||||
|
||||
test('a stale ack entry fails', () => {
|
||||
// An ack that outlives its ripple pre-clears the NEXT one on that path.
|
||||
const ack = { version: ACK_VERSION, paths: { [WORKFLOW_KEY]: { reason: 'old' } } };
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
changedPaths: [],
|
||||
ack,
|
||||
});
|
||||
assert.deepEqual(r.staleAcks, [WORKFLOW_KEY]);
|
||||
assert.ok(!r.ok);
|
||||
assert.match(formatReport(r), /stale acknowledgment/);
|
||||
});
|
||||
|
||||
test('an ack without a reason fails', () => {
|
||||
for (const bad of [{ reason: '' }, { reason: ' ' }, {}, null, 42]) {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
|
||||
changedPaths: [],
|
||||
ack: { version: ACK_VERSION, paths: { [WORKFLOW_KEY]: bad } },
|
||||
});
|
||||
assert.ok(!r.ok, `${JSON.stringify(bad)} must be rejected`);
|
||||
assert.match(r.errors.join('\n'), /has no non-empty "reason"/);
|
||||
}
|
||||
});
|
||||
|
||||
test('an absent ack file means no acks', () => {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
|
||||
changedPaths: [WORKFLOW_SRC],
|
||||
ack: null,
|
||||
});
|
||||
assert.equal(r.errors.length, 0);
|
||||
assert.ok(r.ok, 'the healthy steady state is no ack file at all');
|
||||
});
|
||||
|
||||
test('a live ack and a stale ack together: only the stale one is named', () => {
|
||||
const ack = {
|
||||
version: ACK_VERSION,
|
||||
paths: {
|
||||
[WORKFLOW_KEY]: { reason: 'live ripple' },
|
||||
[SKILL_KEY]: { reason: 'stale' },
|
||||
},
|
||||
};
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa', [SKILL_KEY]: 'ccc' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'bbb', [SKILL_KEY]: 'ccc' }),
|
||||
changedPaths: [],
|
||||
ack,
|
||||
});
|
||||
assert.deepEqual(r.staleAcks, [SKILL_KEY], 'the live one must not be named');
|
||||
});
|
||||
|
||||
test('non-object ack JSON is rejected, not treated as empty', () => {
|
||||
// Reading these as "no acks" would SILENTLY DISARM the gate — indistinguishable
|
||||
// from a healthy run, which is the worst failure available here.
|
||||
for (const bad of [0, 'a string', [], true]) {
|
||||
const { errors } = parseAck(bad);
|
||||
assert.ok(errors.length > 0, `${JSON.stringify(bad)} must be rejected`);
|
||||
assert.match(errors.join('\n'), /must be a JSON object/);
|
||||
}
|
||||
assert.deepEqual(parseAck(null).errors, [], 'absent is legal');
|
||||
assert.deepEqual(parseAck({}).errors, [], 'empty object is legal');
|
||||
assert.equal(parseAck({ version: 99, paths: {} }).errors.length, 1, 'version drift is caught');
|
||||
});
|
||||
|
||||
// ─── The acceptance criteria, failing-first ──────────────────────────────────
|
||||
|
||||
test('a ripple names the unexplained path and not the explained one', () => {
|
||||
// #2723 AC: "edit one source file, corrupt an unrelated emitted file, assert the
|
||||
// check names the unattributable paths."
|
||||
const r = diffEmitted({
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa', [SKILL_KEY]: 'ccc' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'bbb', [SKILL_KEY]: 'ddd' }),
|
||||
changedPaths: [WORKFLOW_SRC], // only the workflow source was edited
|
||||
});
|
||||
assert.equal(r.unattributable.length, 1);
|
||||
assert.equal(r.unattributable[0].rel, SKILL_KEY, 'the unrelated emitted file is the finding');
|
||||
assert.equal(r.attributed.length, 1);
|
||||
assert.equal(r.attributed[0].rel, WORKFLOW_KEY, 'the explained one must NOT be reported');
|
||||
assert.ok(!r.ok);
|
||||
});
|
||||
|
||||
test('a converter change fails without an ack and passes with one', () => {
|
||||
// #2723 AC: "simulate a legitimate converter change: assert it fails without an ack
|
||||
// entry and passes with one." A converter edit moves emitted bytes for files whose
|
||||
// sources nobody touched — ADR-2264's "~5% git cannot review".
|
||||
const moved = {};
|
||||
const base = {};
|
||||
for (let i = 0; i < 25; i++) {
|
||||
base[`skills/gsd-cmd-${i}/SKILL.md`] = `h${i}`;
|
||||
moved[`skills/gsd-cmd-${i}/SKILL.md`] = `x${i}`;
|
||||
}
|
||||
const changedPaths = ['src/runtime-artifact-conversion.cts'];
|
||||
|
||||
const without = diffEmitted({ baseline: mf(base), current: mf(moved), changedPaths });
|
||||
assert.equal(without.unattributable.length, 25);
|
||||
assert.ok(!without.ok, 'a converter change must not pass silently');
|
||||
|
||||
const paths = {};
|
||||
for (const rel of Object.keys(moved)) paths[rel] = { reason: 'converter rewrite, ADR-2719' };
|
||||
const withAck = diffEmitted({
|
||||
baseline: mf(base), current: mf(moved), changedPaths,
|
||||
ack: { version: ACK_VERSION, paths },
|
||||
});
|
||||
assert.equal(withAck.unattributable.length, 0);
|
||||
assert.equal(withAck.acked.length, 25);
|
||||
assert.ok(withAck.ok);
|
||||
});
|
||||
|
||||
test('growth is reported with its exact byte delta and needs an ack', () => {
|
||||
// ADR-2719 must-have 6, added by an /adr-phase-coverage audit precisely because
|
||||
// scope item 5 promised it and no criterion asserted it.
|
||||
const sizeBaseline = { 'verify-work.md': 10000, 'plan-phase.md': 8000 };
|
||||
const sizeCurrent = { 'verify-work.md': 11247, 'plan-phase.md': 8000 };
|
||||
|
||||
const without = diffEmitted({
|
||||
baseline: mf({}), current: mf({}), changedPaths: [], sizeBaseline, sizeCurrent,
|
||||
});
|
||||
assert.equal(without.grown.length, 1);
|
||||
assert.deepEqual(without.grown[0], {
|
||||
name: 'verify-work.md', from: 10000, to: 11247, delta: 1247, acked: false,
|
||||
});
|
||||
assert.ok(!without.ok, 'unacked growth must block');
|
||||
assert.match(formatReport(without), /verify-work\.md grew 1247 bytes/);
|
||||
|
||||
const withAck = diffEmitted({
|
||||
baseline: mf({}), current: mf({}), changedPaths: [], sizeBaseline, sizeCurrent,
|
||||
ack: { version: ACK_VERSION, paths: { 'verify-work.md': { reason: 'new UAT section' } } },
|
||||
});
|
||||
assert.equal(withAck.grown[0].acked, true);
|
||||
assert.ok(withAck.ok);
|
||||
});
|
||||
|
||||
test('an ack consumed by size growth alone is not reported as stale', () => {
|
||||
// Ordering regression: stale-ack detection must run AFTER the size pass. Computing it
|
||||
// between the hash pass and the size pass reports a legitimate growth ack as stale —
|
||||
// a false failure that would push contributors to delete the very ack that is working.
|
||||
const r = diffEmitted({
|
||||
baseline: mf({}),
|
||||
current: mf({}),
|
||||
changedPaths: [],
|
||||
sizeBaseline: { 'verify-work.md': 10000 },
|
||||
sizeCurrent: { 'verify-work.md': 11247 },
|
||||
ack: { version: ACK_VERSION, paths: { 'verify-work.md': { reason: 'new UAT section' } } },
|
||||
});
|
||||
assert.deepEqual(r.staleAcks, [], 'a growth-consumed ack is live, not stale');
|
||||
assert.equal(r.grown[0].acked, true);
|
||||
assert.ok(r.ok);
|
||||
});
|
||||
|
||||
test('shrinkage is reported but needs no ack', () => {
|
||||
const r = diffEmitted({
|
||||
baseline: mf({}), current: mf({}), changedPaths: [],
|
||||
sizeBaseline: { 'a.md': 9000 }, sizeCurrent: { 'a.md': 8000 },
|
||||
});
|
||||
assert.deepEqual(r.shrunk, [{ name: 'a.md', from: 9000, to: 8000, delta: 1000 }]);
|
||||
assert.ok(r.ok, 'shrinkage is not creep — gating it would punish what the ratchet wants');
|
||||
});
|
||||
|
||||
// ─── Baseline resolution + staleness ─────────────────────────────────────────
|
||||
|
||||
const goodBaseline = (sha) => ({
|
||||
version: BASELINE_VERSION,
|
||||
sha,
|
||||
manifests: { claude: { [WORKFLOW_KEY]: 'aaa' } },
|
||||
sizes: { 'plan-phase.md': 100 },
|
||||
});
|
||||
|
||||
test('a stale baseline cache key is detected, not used', () => {
|
||||
// ADR-2719 §5: the one thing that has to be exactly right.
|
||||
const r = resolveBaseline({
|
||||
expectedSha: SHA_A,
|
||||
env: {},
|
||||
cachePath: 'cache.json',
|
||||
readJson: () => goodBaseline(SHA_B),
|
||||
});
|
||||
assert.ok(!r.ok);
|
||||
assert.match(r.errors.join('\n'), /STALE baseline/);
|
||||
assert.ok(r.errors.join('\n').includes(SHA_B) && r.errors.join('\n').includes(SHA_A));
|
||||
});
|
||||
|
||||
test('a matching baseline sha is accepted', () => {
|
||||
const r = resolveBaseline({
|
||||
expectedSha: SHA_A, env: {}, cachePath: 'cache.json',
|
||||
readJson: () => goodBaseline(SHA_A),
|
||||
});
|
||||
assert.ok(r.ok);
|
||||
assert.equal(r.sha, SHA_A);
|
||||
assert.equal(r.via, 'cache:cache.json');
|
||||
assert.deepEqual(r.sizeBaseline, { 'plan-phase.md': 100 });
|
||||
});
|
||||
|
||||
test('an unavailable baseline fails explicitly rather than skipping', () => {
|
||||
// ADR-2719 §6 — in node:test a bare `return` is a PASS, which would fail the gate open.
|
||||
const r = resolveBaseline({
|
||||
expectedSha: SHA_A, env: {}, cachePath: 'cache.json',
|
||||
readJson: () => null,
|
||||
});
|
||||
assert.ok(!r.ok);
|
||||
assert.equal(r.via, 'none');
|
||||
assert.match(r.errors.join('\n'), /bare `return` is a PASS/);
|
||||
});
|
||||
|
||||
test('a malformed baseline is rejected', () => {
|
||||
for (const bad of [0, 'str', [], true]) {
|
||||
const r = resolveBaseline({
|
||||
expectedSha: SHA_A, env: {}, cachePath: 'c.json', readJson: () => bad,
|
||||
});
|
||||
assert.ok(!r.ok, `${JSON.stringify(bad)} must be rejected`);
|
||||
}
|
||||
const noSha = resolveBaseline({
|
||||
expectedSha: SHA_A, env: {}, cachePath: 'c.json',
|
||||
readJson: () => ({ version: BASELINE_VERSION, manifests: {} }),
|
||||
});
|
||||
assert.match(noSha.errors.join('\n'), /must be a 40-hex commit sha/);
|
||||
});
|
||||
|
||||
test('baseline resolution precedence is explicit and reported', () => {
|
||||
// env wins over cache…
|
||||
const viaEnv = resolveBaseline({
|
||||
expectedSha: SHA_A,
|
||||
env: { [BASELINE_ENV]: '/tmp/from-cache-restore.json' },
|
||||
cachePath: 'cache.json',
|
||||
readJson: (p) => (p === '/tmp/from-cache-restore.json' ? goodBaseline(SHA_A) : goodBaseline(SHA_B)),
|
||||
});
|
||||
assert.ok(viaEnv.ok);
|
||||
assert.equal(viaEnv.via, `env:${BASELINE_ENV}`);
|
||||
|
||||
// …and an explicitly-pointed-at stale baseline is a hard stop, not a fall-through:
|
||||
// the operator said "use this one".
|
||||
const envStale = resolveBaseline({
|
||||
expectedSha: SHA_A,
|
||||
env: { [BASELINE_ENV]: '/tmp/x.json' },
|
||||
cachePath: 'cache.json',
|
||||
readJson: () => goodBaseline(SHA_B),
|
||||
buildFallback: () => goodBaseline(SHA_A),
|
||||
});
|
||||
assert.ok(!envStale.ok, 'an explicit stale baseline must not silently fall through');
|
||||
|
||||
// a stale CACHE, by contrast, falls through to the build fallback
|
||||
const viaBuild = resolveBaseline({
|
||||
expectedSha: SHA_A, env: {}, cachePath: 'cache.json',
|
||||
readJson: () => goodBaseline(SHA_B),
|
||||
buildFallback: () => goodBaseline(SHA_A),
|
||||
});
|
||||
assert.ok(viaBuild.ok);
|
||||
assert.equal(viaBuild.via, 'build');
|
||||
});
|
||||
|
||||
test('base-ref candidates are ordered most-specific first and de-duplicated', () => {
|
||||
// The gate went red on its first matrix run because it hard-depended on
|
||||
// `origin/next`, which cannot exist in the gsd-test container (shallow clone +
|
||||
// base/head merge, no remote-tracking refs). Candidate order is the fix, so it is
|
||||
// pinned rather than left implicit.
|
||||
assert.deepEqual(
|
||||
baseRefCandidates({ GSD_EMITTED_BASE: 'abc123', GITHUB_BASE_REF: 'next' }),
|
||||
['abc123', 'origin/next', 'next'],
|
||||
'an explicit override wins, then the Actions base ref, then the defaults',
|
||||
);
|
||||
assert.deepEqual(
|
||||
baseRefCandidates({ GITHUB_BASE_REF: 'release/1.9' }),
|
||||
['origin/release/1.9', 'release/1.9', 'origin/next', 'next'],
|
||||
'a non-next base ref is honored before falling back',
|
||||
);
|
||||
assert.deepEqual(
|
||||
baseRefCandidates({}),
|
||||
['origin/next', 'next'],
|
||||
'with no env, the repo defaults are the only candidates',
|
||||
);
|
||||
// De-duplication matters: GITHUB_BASE_REF=next must not produce origin/next twice.
|
||||
const dupes = baseRefCandidates({ GITHUB_BASE_REF: 'next' });
|
||||
assert.equal(new Set(dupes).size, dupes.length);
|
||||
});
|
||||
|
||||
test('an unreadable baseline surfaces an error', () => {
|
||||
const r = resolveBaseline({
|
||||
expectedSha: SHA_A, env: {}, cachePath: 'c.json',
|
||||
readJson: () => { throw new Error('injected read failure'); },
|
||||
});
|
||||
assert.ok(!r.ok);
|
||||
assert.match(r.errors.join('\n'), /injected read failure/);
|
||||
});
|
||||
|
||||
test('readAckFile: absent is legal, malformed and unreadable are not', () => {
|
||||
const tmp = fs.mkdtempSync(path.join(require('node:os').tmpdir(), 'gsd-ack-'));
|
||||
try {
|
||||
const ackPath = path.join(tmp, 'emitted-drift-ack.json');
|
||||
|
||||
// Absent == no acks. The healthy steady state.
|
||||
assert.equal(readAckFile(ackPath), null);
|
||||
|
||||
// Present and valid.
|
||||
fs.writeFileSync(ackPath, JSON.stringify({ version: ACK_VERSION, paths: {} }));
|
||||
assert.deepEqual(readAckFile(ackPath), { version: ACK_VERSION, paths: {} });
|
||||
|
||||
// Present but empty — must NOT be read as absent.
|
||||
fs.writeFileSync(ackPath, '');
|
||||
assert.throws(() => readAckFile(ackPath), /present but empty/);
|
||||
|
||||
// Present but not JSON.
|
||||
fs.writeFileSync(ackPath, '{not json');
|
||||
assert.throws(() => readAckFile(ackPath), /not valid JSON/);
|
||||
|
||||
// Unreadable: monkeypatch the fs method, restore in `finally`. NEVER chmod 0o000 —
|
||||
// root bypasses mode bits, so the test would silently pass with zero coverage in
|
||||
// root Docker/CI. This exercises the SUT (readAckFile), not fs itself.
|
||||
fs.writeFileSync(ackPath, JSON.stringify({ version: ACK_VERSION, paths: {} }));
|
||||
const orig = fs.readFileSync;
|
||||
try {
|
||||
fs.readFileSync = () => { throw new Error('injected ack read failure'); };
|
||||
assert.throws(() => readAckFile(ackPath), /injected ack read failure/);
|
||||
} finally {
|
||||
fs.readFileSync = orig;
|
||||
}
|
||||
// Restoration is real, not assumed.
|
||||
assert.deepEqual(readAckFile(ackPath), { version: ACK_VERSION, paths: {} });
|
||||
} finally {
|
||||
cleanup(tmp);
|
||||
}
|
||||
});
|
||||
|
||||
test('formatReport truncation is exact at limit-1 / limit / limit+1', () => {
|
||||
// sampleLimit gates a real branch. CLAUDE.md's boundary rule applies to it like any
|
||||
// other limit; the earlier suite named a test "limit+1" that tested no numeric limit
|
||||
// at all, which is worse than no coverage because it reads as covered.
|
||||
const build = (n) => {
|
||||
const baseline = {}; const current = {};
|
||||
for (let i = 0; i < n; i++) {
|
||||
const k = `gsd-core/workflows/w${String(i).padStart(3, '0')}.md`;
|
||||
baseline[k] = 'a'; current[k] = 'b';
|
||||
}
|
||||
return diffEmitted({ baseline: mf(baseline), current: mf(current), changedPaths: [] });
|
||||
};
|
||||
|
||||
const at19 = formatReport(build(19), { sampleLimit: 20 });
|
||||
assert.ok(at19.includes('w018.md'), 'limit-1 lists every path');
|
||||
assert.ok(!at19.includes('…and'), 'limit-1 must not truncate');
|
||||
|
||||
const at20 = formatReport(build(20), { sampleLimit: 20 });
|
||||
assert.ok(at20.includes('w019.md'), 'at the limit the last path is listed');
|
||||
assert.ok(!at20.includes('…and'), 'exactly at the limit must not truncate');
|
||||
|
||||
const at21 = formatReport(build(21), { sampleLimit: 20 });
|
||||
assert.ok(at21.includes('…and 1 more'), 'limit+1 truncates and says how many were hidden');
|
||||
assert.ok(!at21.includes('w020.md'), 'the 21st path is not listed');
|
||||
});
|
||||
|
||||
// ─── Independence + purity ───────────────────────────────────────────────────
|
||||
|
||||
test('the differential covers every runtime present in either manifest', () => {
|
||||
const baseline = { claude: { [WORKFLOW_KEY]: 'a' }, kimi: { [WORKFLOW_KEY]: 'a' } };
|
||||
const current = { claude: { [WORKFLOW_KEY]: 'b' }, opencode: { [WORKFLOW_KEY]: 'c' } };
|
||||
const r = diffEmitted({ baseline, current, changedPaths: [] });
|
||||
const seen = new Set([...r.unattributable, ...r.attributed, ...r.removed].map((x) => x.runtime));
|
||||
assert.deepEqual([...seen].sort(), ['claude', 'kimi', 'opencode'],
|
||||
'a runtime present on only one side must still be evaluated');
|
||||
});
|
||||
|
||||
test('diff is pure and repeatable', () => {
|
||||
const args = {
|
||||
baseline: mf({ [WORKFLOW_KEY]: 'aaa' }),
|
||||
current: mf({ [WORKFLOW_KEY]: 'bbb' }),
|
||||
// 3 elements in deliberately unsorted order, so an in-place sort would be visible.
|
||||
changedPaths: ['zzz/last.md', WORKFLOW_SRC, 'aaa/first.md'],
|
||||
};
|
||||
const frozen = JSON.stringify(args);
|
||||
const a = diffEmitted(args);
|
||||
const b = diffEmitted(args);
|
||||
assert.deepEqual(b, a);
|
||||
assert.equal(JSON.stringify(args), frozen, 'inputs must not be mutated');
|
||||
});
|
||||
|
||||
// ─── Property: conservation ──────────────────────────────────────────────────
|
||||
|
||||
test('property: every moved key lands in exactly one bucket', () => {
|
||||
// The conservation law itself. A key silently dropped from all three buckets is a
|
||||
// hole in the very invariant ADR-2719 asserts — and it is the failure a hand-written
|
||||
// example set is least likely to find.
|
||||
const keys = [WORKFLOW_KEY, SKILL_KEY, 'agents/gsd-planner.md', 'scripts/lib/cli-exit.cjs'];
|
||||
const sources = { [WORKFLOW_KEY]: WORKFLOW_SRC, [SKILL_KEY]: SKILL_SRC,
|
||||
'agents/gsd-planner.md': 'agents/gsd-planner.md', 'scripts/lib/cli-exit.cjs': 'scripts/lib/cli-exit.cjs' };
|
||||
|
||||
fc.assert(
|
||||
fc.property(
|
||||
fc.subarray(keys, { minLength: 1 }), // which keys move
|
||||
fc.subarray(keys), // which sources the PR changed
|
||||
fc.subarray(keys), // which keys are acked
|
||||
(movedKeys, changedKeys, ackedKeys) => {
|
||||
const baseline = {}; const current = {};
|
||||
for (const k of keys) { baseline[k] = 'h'; current[k] = movedKeys.includes(k) ? 'x' : 'h'; }
|
||||
const ackPaths = {};
|
||||
for (const k of ackedKeys) ackPaths[k] = { reason: 'property' };
|
||||
|
||||
const r = diffEmitted({
|
||||
baseline: mf(baseline),
|
||||
current: mf(current),
|
||||
changedPaths: changedKeys.map((k) => sources[k]),
|
||||
ack: { version: ACK_VERSION, paths: ackPaths },
|
||||
});
|
||||
|
||||
if (r.errors.length) return false;
|
||||
const bucketed = [
|
||||
...r.attributed.map((x) => x.rel),
|
||||
...r.unattributable.map((x) => x.rel),
|
||||
...r.acked.map((x) => x.rel),
|
||||
];
|
||||
// exactly-once, and exactly the moved set — no key invented, none dropped
|
||||
return bucketed.length === movedKeys.length
|
||||
&& new Set(bucketed).size === bucketed.length
|
||||
&& movedKeys.every((k) => bucketed.includes(k));
|
||||
},
|
||||
),
|
||||
{ numRuns: 400 },
|
||||
);
|
||||
});
|
||||
|
||||
// ─── The real thing: the law, run against the actual tree ───────────────────
|
||||
//
|
||||
// Everything above exercises the pure law against synthetic input, which is what makes
|
||||
// the acceptance criteria practical to assert at all. This block is what stops the
|
||||
// phase from being interface-only: it builds the CURRENT emitted manifests for real
|
||||
// (one installer spawn per runtime), reads the BASELINE from `origin/next`, resolves
|
||||
// the changed paths with real git, reads the real ack file, and runs the conservation
|
||||
// law over all of it.
|
||||
//
|
||||
// Baseline source note: `git show origin/next:<fixture>` — next's RECORDED emitted
|
||||
// state. Deliberately not the working-tree fixtures, which are whatever this PR's
|
||||
// author regenerated; comparing against those would be vacuous. Phase 4 (#2724)
|
||||
// deletes the fixtures and swaps in resolveBaseline's cache path, which is already
|
||||
// implemented and tested above.
|
||||
|
||||
test('differential attribution over the real tree', { timeout: 900_000 }, async (t) => {
|
||||
if (process.platform === 'win32') {
|
||||
// Mirrors the golden harness: install output is platform-specific on Windows
|
||||
// (backslash paths), so parity is asserted on macOS + Linux. An explicit t.skip,
|
||||
// never a bare `return` — in node:test that would be a PASS (ADR-2719 §6).
|
||||
t.skip('emitted parity is asserted on macOS + Linux; Windows install output is platform-specific');
|
||||
return;
|
||||
}
|
||||
|
||||
// hooks/dist is gitignored and built (DEFECT.HOOKS-DIST-SCOPED-CI): the scoped CI
|
||||
// lane does not run build:hooks, so a real install there would emit no hooks/ dir.
|
||||
// Build idempotently, exactly as the golden harness does.
|
||||
execFileSync(process.execPath, [BUILD_SCRIPT], { encoding: 'utf-8', stdio: 'pipe', timeout: 120_000 });
|
||||
|
||||
// The base ref is not universally available. The gsd-test runner shallow-clones and
|
||||
// merges base+head, so no `origin/*` remote-tracking ref exists in the container —
|
||||
// this test went red on its first matrix run for exactly that reason, which is the
|
||||
// resolver doing its job and the dependency being wrong.
|
||||
//
|
||||
// An explicit t.skip is the ADR-sanctioned response for a genuine environmental
|
||||
// skip: it is REPORTED as skipped, unlike a bare `return`, which node:test scores as
|
||||
// a PASS (ADR-2719 §6). Hard-failing instead would make the suite permanently red
|
||||
// wherever a base ref cannot exist by construction, which is not a propagation
|
||||
// finding — it is a statement about the checkout.
|
||||
const resolved = resolveBase();
|
||||
if (!resolved) {
|
||||
t.skip(
|
||||
'no base ref resolvable — tried ' + baseRefCandidates().join(', ') +
|
||||
'. The differential gate did NOT run here. It binds in the CI test lanes, which ' +
|
||||
'fetch the base ref explicitly; set GSD_EMITTED_BASE=<ref|sha> to run it elsewhere.',
|
||||
);
|
||||
return;
|
||||
}
|
||||
const { ref: base, sha: baseSha } = resolved;
|
||||
assert.match(baseSha, /^[0-9a-f]{40}$/);
|
||||
|
||||
const baseline = baselineManifestsAtRef(base);
|
||||
assert.ok(
|
||||
baseline && Object.keys(baseline).length > 0,
|
||||
`no baseline manifests found at ${base}. During the dual-run window these come from the ` +
|
||||
'committed golden fixtures at that ref; after Phase 4 they come from the cached ' +
|
||||
'baseline artifact via resolveBaseline().',
|
||||
);
|
||||
|
||||
const changedPaths = resolveChangedPaths(base);
|
||||
const ack = readAckFile();
|
||||
const current = currentManifests();
|
||||
|
||||
// Assert against the INDEPENDENT expected count, not just baseline-vs-current.
|
||||
// Comparing the two sides to each other cannot catch a family dropped from BOTH —
|
||||
// which is exactly what happened with claude-local: 18 === 18 passed vacuously while
|
||||
// the 19th family went unchecked.
|
||||
assert.equal(
|
||||
Object.keys(baseline).length,
|
||||
EXPECTED_MANIFEST_COUNT,
|
||||
`baseline must cover all ${EXPECTED_MANIFEST_COUNT} emitted manifest families`,
|
||||
);
|
||||
assert.equal(
|
||||
Object.keys(current).length,
|
||||
EXPECTED_MANIFEST_COUNT,
|
||||
`current must cover all ${EXPECTED_MANIFEST_COUNT} emitted manifest families`,
|
||||
);
|
||||
assert.ok(baseline['claude-local'], 'the claude local-scope layout (#2086) must be covered');
|
||||
assert.ok(current['claude-local'], 'the claude local-scope layout (#2086) must be covered');
|
||||
|
||||
const result = diffEmitted({
|
||||
baseline,
|
||||
current,
|
||||
changedPaths,
|
||||
ack,
|
||||
sizeBaseline: baselineSizesAtRef(base),
|
||||
sizeCurrent: currentSizes(),
|
||||
});
|
||||
|
||||
assert.ok(
|
||||
result.ok,
|
||||
`emitted-attribution failed against ${base}@${baseSha.slice(0, 12)}:\n\n${formatReport(result)}`,
|
||||
);
|
||||
});
|
||||
192
tests/helpers/emitted-baseline.cjs
Normal file
192
tests/helpers/emitted-baseline.cjs
Normal file
@@ -0,0 +1,192 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Baseline acquisition for the differential attribution check (ADR-2719 §5, #2723).
|
||||
*
|
||||
* The baseline is the emitted-manifest set built at `next` HEAD. It is CACHED, not
|
||||
* committed — committing it would recreate the derived-state-in-git problem this epic
|
||||
* exists to delete, and would double the rate `next` advances.
|
||||
*
|
||||
* ── The load-bearing part ────────────────────────────────────────────────────
|
||||
* ADR-2719 §5: "keyed on the `next` sha the PR was merged with. That key discipline is
|
||||
* the one thing that has to be exactly right — a stale baseline silently mis-attributes."
|
||||
*
|
||||
* Note the asymmetry that makes staleness worse than absence: a MISSING baseline fails
|
||||
* loudly and gets fixed. A STALE one produces a confident wrong answer — it attributes
|
||||
* deltas to the wrong commit's state, so real ripples read as explained. Every path
|
||||
* through this module therefore fails closed on a key mismatch.
|
||||
*
|
||||
* ── No bare `return` anywhere ────────────────────────────────────────────────
|
||||
* ADR-2719 §6 calls this out explicitly: in node:test a bare `return` is a PASS, not a
|
||||
* skip, so a baseline-unavailable path that returns would make the whole gate fail open
|
||||
* with nothing in CI to say so. This module returns an explicit {ok:false} result and
|
||||
* the caller asserts on it.
|
||||
*
|
||||
* IO is INJECTED (readJson / exists / buildFallback) so every failure mode above is
|
||||
* unit-testable without touching the filesystem or spawning 19 installers.
|
||||
*/
|
||||
|
||||
/** Env var a CI cache-restore step points at the recovered baseline artifact. */
|
||||
const BASELINE_ENV = 'GSD_EMITTED_BASELINE';
|
||||
|
||||
/** Default on-disk cache location, relative to the repo root. */
|
||||
const DEFAULT_CACHE_PATH = '.gsd-cache/emitted-baseline.json';
|
||||
|
||||
/** Baseline artifact schema version — pinned so a format change fails loudly. */
|
||||
const BASELINE_VERSION = 1;
|
||||
|
||||
/**
|
||||
* Validate a baseline artifact's shape and freshness.
|
||||
*
|
||||
* @param {*} doc parsed artifact
|
||||
* @param {string} expectedSha the `next` sha this PR is being evaluated against
|
||||
* @param {string} source where it came from (named in every error)
|
||||
* @returns {{ok: true, baseline: object, sizeBaseline: object|null, sha: string}
|
||||
* |{ok: false, errors: string[]}}
|
||||
*/
|
||||
function validateBaseline(doc, expectedSha, source) {
|
||||
const errors = [];
|
||||
|
||||
if (doc === null || doc === undefined) {
|
||||
return { ok: false, errors: [`${source}: baseline is absent`] };
|
||||
}
|
||||
if (typeof doc !== 'object' || Array.isArray(doc)) {
|
||||
// Same class as Phase 2's fixture loader: a document that parses but is not an
|
||||
// object must never be read as "no entries", which would pass vacuously.
|
||||
return {
|
||||
ok: false,
|
||||
errors: [`${source}: baseline must be a JSON object, got ${Array.isArray(doc) ? 'array' : typeof doc}`],
|
||||
};
|
||||
}
|
||||
|
||||
if (doc.version !== undefined && doc.version !== BASELINE_VERSION) {
|
||||
errors.push(`${source}: unsupported baseline version ${JSON.stringify(doc.version)} (expected ${BASELINE_VERSION})`);
|
||||
}
|
||||
|
||||
if (typeof doc.sha !== 'string' || !/^[0-9a-f]{40}$/.test(doc.sha)) {
|
||||
errors.push(`${source}: baseline "sha" must be a 40-hex commit sha, got ${JSON.stringify(doc.sha)}`);
|
||||
} else if (typeof expectedSha === 'string' && expectedSha !== '' && doc.sha !== expectedSha) {
|
||||
// THE staleness gate. Never silently used.
|
||||
errors.push(
|
||||
`${source}: STALE baseline — built at ${doc.sha} but this PR is being evaluated ` +
|
||||
`against next@${expectedSha}. A stale baseline mis-attributes silently, so it is ` +
|
||||
'refused rather than used. Rebuild it, or let the in-job fallback run.',
|
||||
);
|
||||
}
|
||||
|
||||
if (!doc.manifests || typeof doc.manifests !== 'object' || Array.isArray(doc.manifests)) {
|
||||
errors.push(`${source}: baseline "manifests" must be an object keyed by runtime`);
|
||||
}
|
||||
|
||||
if (errors.length) return { ok: false, errors };
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
baseline: doc.manifests,
|
||||
sizeBaseline: (doc.sizes && typeof doc.sizes === 'object' && !Array.isArray(doc.sizes))
|
||||
? doc.sizes
|
||||
: null,
|
||||
sha: doc.sha,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the baseline through the documented precedence, reporting WHICH step supplied
|
||||
* it so a failure message can say where the answer came from.
|
||||
*
|
||||
* 1. `GSD_EMITTED_BASELINE` — explicit path (CI cache restore writes here)
|
||||
* 2. the on-disk cache, validated against the expected sha
|
||||
* 3. an in-job build at `origin/next` (slow fallback)
|
||||
* 4. none → explicit failure (NEVER a silent pass)
|
||||
*
|
||||
* @param {object} opts
|
||||
* @param {string} opts.expectedSha `next` sha under test
|
||||
* @param {object} [opts.env] environment (injected)
|
||||
* @param {string} [opts.cachePath]
|
||||
* @param {function} opts.readJson (path) => parsed | null (null when absent)
|
||||
* @param {function} [opts.buildFallback] () => artifact | null (the slow path)
|
||||
* @returns {{ok: true, via: string, baseline: object, sizeBaseline: object|null, sha: string}
|
||||
* |{ok: false, via: string, errors: string[]}}
|
||||
*/
|
||||
function resolveBaseline({
|
||||
expectedSha,
|
||||
env = process.env,
|
||||
cachePath = DEFAULT_CACHE_PATH,
|
||||
readJson,
|
||||
buildFallback = null,
|
||||
} = {}) {
|
||||
if (typeof readJson !== 'function') {
|
||||
return { ok: false, via: 'none', errors: ['resolveBaseline: readJson must be supplied'] };
|
||||
}
|
||||
|
||||
const attempts = [];
|
||||
|
||||
const envPath = env && env[BASELINE_ENV];
|
||||
if (envPath) {
|
||||
let doc = null;
|
||||
let readError = null;
|
||||
try {
|
||||
doc = readJson(envPath);
|
||||
} catch (err) {
|
||||
readError = `${BASELINE_ENV}=${envPath}: ${err.message}`;
|
||||
}
|
||||
if (readError) {
|
||||
attempts.push(readError);
|
||||
} else {
|
||||
const v = validateBaseline(doc, expectedSha, `${BASELINE_ENV}=${envPath}`);
|
||||
if (v.ok) return { ok: true, via: `env:${BASELINE_ENV}`, ...v };
|
||||
attempts.push(...v.errors);
|
||||
// An EXPLICITLY pointed-at baseline that is stale or malformed is a hard stop, not
|
||||
// a reason to quietly fall through to a different one — the operator said "use this".
|
||||
return { ok: false, via: `env:${BASELINE_ENV}`, errors: attempts };
|
||||
}
|
||||
}
|
||||
|
||||
let cacheDoc = null;
|
||||
let cacheErr = null;
|
||||
try {
|
||||
cacheDoc = readJson(cachePath);
|
||||
} catch (err) {
|
||||
cacheErr = `${cachePath}: ${err.message}`;
|
||||
}
|
||||
if (cacheErr) {
|
||||
attempts.push(cacheErr);
|
||||
} else if (cacheDoc !== null && cacheDoc !== undefined) {
|
||||
const v = validateBaseline(cacheDoc, expectedSha, cachePath);
|
||||
if (v.ok) return { ok: true, via: `cache:${cachePath}`, ...v };
|
||||
// A stale CACHE is recoverable: fall through to the build fallback, but keep the
|
||||
// reason so the final message explains why the slow path ran.
|
||||
attempts.push(...v.errors);
|
||||
} else {
|
||||
attempts.push(`${cachePath}: absent`);
|
||||
}
|
||||
|
||||
if (typeof buildFallback === 'function') {
|
||||
let built = null;
|
||||
try {
|
||||
built = buildFallback();
|
||||
} catch (err) {
|
||||
attempts.push(`in-job build at origin/next failed: ${err.message}`);
|
||||
return { ok: false, via: 'build', errors: attempts };
|
||||
}
|
||||
const v = validateBaseline(built, expectedSha, 'in-job build at origin/next');
|
||||
if (v.ok) return { ok: true, via: 'build', ...v };
|
||||
attempts.push(...v.errors);
|
||||
return { ok: false, via: 'build', errors: attempts };
|
||||
}
|
||||
|
||||
attempts.push(
|
||||
'no baseline available and no in-job build fallback was supplied. This is a hard ' +
|
||||
'failure on purpose: a skipped propagation gate is worth less than a slow one ' +
|
||||
'(ADR-2719 §6), and in node:test a bare `return` is a PASS, not a skip.',
|
||||
);
|
||||
return { ok: false, via: 'none', errors: attempts };
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
BASELINE_ENV,
|
||||
BASELINE_VERSION,
|
||||
DEFAULT_CACHE_PATH,
|
||||
validateBaseline,
|
||||
resolveBaseline,
|
||||
};
|
||||
319
tests/helpers/emitted-diff.cjs
Normal file
319
tests/helpers/emitted-diff.cjs
Normal file
@@ -0,0 +1,319 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Differential emitted-artifact attribution — the conservation law (ADR-2719 §1,
|
||||
* issue #2723, epic #2719 Phase 3).
|
||||
*
|
||||
* Given the emitted manifests at `next` HEAD and at PR HEAD, plus the repo paths the
|
||||
* PR actually changed, decide which moved emitted paths are EXPLAINED by the diff and
|
||||
* which are not. Unattributable deltas are a hard failure that names them; the only
|
||||
* way through is a committed acknowledgment (`tests/emitted-drift-ack.json`).
|
||||
*
|
||||
* ── Why this module is pure ──────────────────────────────────────────────────
|
||||
* No fs, no git, no installer, no clock. The naive shape — one integration test that
|
||||
* builds 19 manifests at each end and asserts — cannot practically exercise the four
|
||||
* failing-first criteria #2723 requires, so in practice they would not get written,
|
||||
* which is precisely how a phase ships promised-but-not-built. Keeping the law pure
|
||||
* makes every criterion a millisecond-scale table test, and makes the Stryker gate
|
||||
* able to bite (a 20-branch pure function is mutation-testable; a 40-minute
|
||||
* integration test is not).
|
||||
*
|
||||
* The expensive part — obtaining the manifests — lives in emitted-baseline.cjs.
|
||||
*
|
||||
* ── What this does NOT do ────────────────────────────────────────────────────
|
||||
* It never re-derives a byte (ADR-2719 §1 is explicit that asserting
|
||||
* `emitted == transform(source)` is the tautology ADR-2264's Amendment rejected).
|
||||
* It constrains which keys may move. It also never mutates repo state: no
|
||||
* regeneration, no auto-ack. `UPDATE_GOLDEN=1` is exactly the escape hatch this
|
||||
* design removes.
|
||||
*/
|
||||
|
||||
const { attributeEmittedPath } = require('./emitted-provenance.cjs');
|
||||
|
||||
/** Ack schema version. Pinned from day one: contributors hand-write this file, so its
|
||||
* shape is public the moment it ships (Hyrum). Loosening later is easy; tightening is not. */
|
||||
const ACK_VERSION = 1;
|
||||
|
||||
/**
|
||||
* Does a changed repo path satisfy a provenance `sources` entry?
|
||||
*
|
||||
* A trailing `/` marks a PREFIX (Phase 2's SOURCE_PREFIX_SUFFIX contract) — e.g. Kimi's
|
||||
* root agent aggregates all of `agents/`. Prefix matching is SEGMENT-AWARE on purpose:
|
||||
* a bare `startsWith('agents/')` would also accept `agentsfoo/x.md` under a source of
|
||||
* `agents`, and silently over-attribute. Exact entries compare exactly.
|
||||
*/
|
||||
function sourceSatisfiedBy(source, changedSet) {
|
||||
if (source.endsWith('/')) {
|
||||
for (const changed of changedSet) {
|
||||
if (changed.startsWith(source)) return changed;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
return changedSet.has(source) ? source : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize + validate the acknowledgment document.
|
||||
*
|
||||
* Rejects a document that parses but is not a plain object. Treating `0` / `"s"` / `[]` /
|
||||
* `null` / `true` as "no acks" would SILENTLY DISARM the gate — the single worst failure
|
||||
* available here, because it looks identical to a healthy run.
|
||||
*
|
||||
* @returns {{ entries: Map<string, {reason: string, runtime?: string}>, errors: string[] }}
|
||||
*/
|
||||
function parseAck(doc, { source = 'emitted-drift-ack.json' } = {}) {
|
||||
const errors = [];
|
||||
const entries = new Map();
|
||||
|
||||
if (doc === null || doc === undefined) return { entries, errors }; // absent == no acks (legal)
|
||||
|
||||
if (typeof doc !== 'object' || Array.isArray(doc)) {
|
||||
errors.push(
|
||||
`${source}: must be a JSON object, got ${Array.isArray(doc) ? 'array' : typeof doc}`,
|
||||
);
|
||||
return { entries, errors };
|
||||
}
|
||||
|
||||
if (doc.version !== undefined && doc.version !== ACK_VERSION) {
|
||||
errors.push(`${source}: unsupported version ${JSON.stringify(doc.version)} (expected ${ACK_VERSION})`);
|
||||
}
|
||||
|
||||
const paths = doc.paths;
|
||||
if (paths === undefined) return { entries, errors }; // `{}` or `{version:1}` == no acks
|
||||
if (paths === null || typeof paths !== 'object' || Array.isArray(paths)) {
|
||||
errors.push(`${source}: "paths" must be an object of <emitted path> -> { reason }`);
|
||||
return { entries, errors };
|
||||
}
|
||||
|
||||
for (const [rel, value] of Object.entries(paths)) {
|
||||
const reason = value && typeof value === 'object' ? value.reason : value;
|
||||
if (typeof reason !== 'string' || reason.trim() === '') {
|
||||
// "name them AND say why" is the contract (ADR-2719 §3). An ack with no reason
|
||||
// is a silent regeneration wearing a declaration's clothes.
|
||||
errors.push(`${source}: ack for "${rel}" has no non-empty "reason"`);
|
||||
continue;
|
||||
}
|
||||
entries.set(rel, {
|
||||
reason: reason.trim(),
|
||||
runtime: value && typeof value === 'object' ? value.runtime : undefined,
|
||||
});
|
||||
}
|
||||
|
||||
return { entries, errors };
|
||||
}
|
||||
|
||||
/**
|
||||
* The conservation law.
|
||||
*
|
||||
* @param {object} opts
|
||||
* @param {object} opts.baseline { [runtime]: { [rel]: hash } } at `next` HEAD
|
||||
* @param {object} opts.current { [runtime]: { [rel]: hash } } at PR HEAD
|
||||
* @param {string[]} opts.changedPaths repo paths the PR changed (git diff --name-only)
|
||||
* @param {object} [opts.ack] parsed emitted-drift-ack.json document (or null)
|
||||
* @param {object} [opts.sizeBaseline] { [name]: bytes } workflow/agent sizes at next
|
||||
* @param {object} [opts.sizeCurrent] { [name]: bytes } workflow/agent sizes at PR HEAD
|
||||
*
|
||||
* @returns {{
|
||||
* moved: number, attributed: Array, unattributable: Array, acked: Array,
|
||||
* removed: Array, grown: Array, shrunk: Array, staleAcks: string[], errors: string[], ok: boolean
|
||||
* }}
|
||||
*/
|
||||
function diffEmitted({
|
||||
baseline,
|
||||
current,
|
||||
changedPaths,
|
||||
ack = null,
|
||||
sizeBaseline = null,
|
||||
sizeCurrent = null,
|
||||
} = {}) {
|
||||
const errors = [];
|
||||
|
||||
if (!baseline || typeof baseline !== 'object' || Array.isArray(baseline)) {
|
||||
errors.push('baseline manifest set must be an object keyed by runtime');
|
||||
}
|
||||
if (!current || typeof current !== 'object' || Array.isArray(current)) {
|
||||
errors.push('current manifest set must be an object keyed by runtime');
|
||||
}
|
||||
if (!Array.isArray(changedPaths)) {
|
||||
// NOT the same as an empty array. A failed `git diff` must never be read as
|
||||
// "nothing changed" — that would make every moved hash unattributable and produce
|
||||
// a failure storm that reads like a real finding.
|
||||
errors.push('changedPaths must be an array (a failed git diff is an error, not an empty set)');
|
||||
}
|
||||
if (errors.length) {
|
||||
return {
|
||||
moved: 0, attributed: [], unattributable: [], acked: [], removed: [],
|
||||
grown: [], shrunk: [], staleAcks: [], errors, ok: false,
|
||||
};
|
||||
}
|
||||
|
||||
const changedSet = new Set(changedPaths);
|
||||
const { entries: ackEntries, errors: ackErrors } = parseAck(ack);
|
||||
errors.push(...ackErrors);
|
||||
|
||||
const attributed = [];
|
||||
const unattributable = [];
|
||||
const acked = [];
|
||||
const removed = [];
|
||||
const usedAcks = new Set();
|
||||
let moved = 0;
|
||||
|
||||
const runtimes = new Set([...Object.keys(baseline), ...Object.keys(current)]);
|
||||
|
||||
for (const runtime of [...runtimes].sort()) {
|
||||
const before = baseline[runtime] || {};
|
||||
const after = current[runtime] || {};
|
||||
const keys = new Set([...Object.keys(before), ...Object.keys(after)]);
|
||||
|
||||
for (const rel of [...keys].sort()) {
|
||||
const had = Object.prototype.hasOwnProperty.call(before, rel);
|
||||
const has = Object.prototype.hasOwnProperty.call(after, rel);
|
||||
|
||||
if (had && has && before[rel] === after[rel]) continue; // unchanged — ignored
|
||||
|
||||
const change = !had ? 'added' : (!has ? 'removed' : 'modified');
|
||||
moved++;
|
||||
|
||||
let attribution;
|
||||
try {
|
||||
attribution = attributeEmittedPath(rel, runtime);
|
||||
} catch (err) {
|
||||
// A path the Phase 2 table cannot resolve is surfaced, never silently skipped —
|
||||
// otherwise a table hole becomes a blind spot in the differential too.
|
||||
errors.push(`${runtime}: ${rel}: ${err.message}`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const record = { runtime, rel, change, ruleId: attribution.ruleId, kind: attribution.kind };
|
||||
|
||||
if (change === 'removed') removed.push(record);
|
||||
|
||||
// Synthesized paths carry no repo source by definition, so a delta in them can
|
||||
// never be "unexplained by the diff" — exempt, but still counted and reported.
|
||||
if (attribution.kind === 'synthesized') {
|
||||
attributed.push({ ...record, via: '<synthesized: exempt>' });
|
||||
continue;
|
||||
}
|
||||
|
||||
let via = null;
|
||||
for (const source of attribution.sources) {
|
||||
const hit = sourceSatisfiedBy(source, changedSet);
|
||||
// `!== null`, not truthiness: an exact match returns the source string, and an
|
||||
// empty-string source would return '' — falsy, so a real match would be
|
||||
// silently discarded. Unreachable with today's rules (every source is a
|
||||
// non-empty template) but it is a footgun for the next rule author.
|
||||
if (hit !== null) { via = hit; break; }
|
||||
}
|
||||
|
||||
if (via !== null) {
|
||||
attributed.push({ ...record, via });
|
||||
} else if (ackEntries.has(rel)) {
|
||||
usedAcks.add(rel);
|
||||
acked.push({ ...record, reason: ackEntries.get(rel).reason });
|
||||
} else {
|
||||
unattributable.push({ ...record, expectedSources: attribution.sources });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Size ratchet, folded into the same machine (ADR-2719 §4, must-have 6) ──
|
||||
// NOTE: stale-ack detection is computed AFTER this block, not before. An ack may be
|
||||
// consumed by either a hash move or a size growth, so computing it earlier would
|
||||
// report a size-growth ack as stale.
|
||||
const grown = [];
|
||||
const shrunk = [];
|
||||
if (sizeBaseline && sizeCurrent) {
|
||||
for (const name of Object.keys(sizeCurrent).sort()) {
|
||||
if (!Object.prototype.hasOwnProperty.call(sizeBaseline, name)) continue;
|
||||
const from = sizeBaseline[name];
|
||||
const to = sizeCurrent[name];
|
||||
if (to > from) {
|
||||
// Growth needs the SAME acknowledgment. Anti-creep survives without pinning a
|
||||
// number: "verify-work.md grew 1,247 bytes" beats a number moving in a 93-line map.
|
||||
const isAcked = ackEntries.has(name);
|
||||
if (isAcked) usedAcks.add(name);
|
||||
grown.push({ name, from, to, delta: to - from, acked: isAcked });
|
||||
} else if (to < from) {
|
||||
// Shrinkage is not creep — reported, never gated.
|
||||
shrunk.push({ name, from, to, delta: from - to });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const unackedGrowth = grown.filter((g) => !g.acked);
|
||||
|
||||
// An ack that outlives the ripple it explained is future blindness: it would silently
|
||||
// pre-clear a NEW ripple on the same path. It must be deleted when the ripple is.
|
||||
// Computed here, once, after BOTH the hash pass and the size pass have consumed acks.
|
||||
const staleAcks = [...ackEntries.keys()].filter((rel) => !usedAcks.has(rel)).sort();
|
||||
|
||||
const ok = errors.length === 0
|
||||
&& unattributable.length === 0
|
||||
&& unackedGrowth.length === 0
|
||||
&& staleAcks.length === 0;
|
||||
|
||||
return {
|
||||
moved,
|
||||
attributed,
|
||||
unattributable,
|
||||
acked,
|
||||
removed,
|
||||
grown,
|
||||
shrunk,
|
||||
staleAcks,
|
||||
errors,
|
||||
ok,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a report as the failure message ADR-2719 §1 specifies — it sells the whole
|
||||
* design on this text, so it is a deliverable, not a detail.
|
||||
*/
|
||||
function formatReport(result, { sampleLimit = 20 } = {}) {
|
||||
const parts = [];
|
||||
|
||||
if (result.errors.length) {
|
||||
parts.push(`${result.errors.length} error(s):\n ${result.errors.slice(0, sampleLimit).join('\n ')}`);
|
||||
}
|
||||
|
||||
if (result.unattributable.length) {
|
||||
const list = result.unattributable.slice(0, sampleLimit)
|
||||
.map((u) => ` ${u.runtime}: ${u.rel}\n rule ${u.ruleId}; expected a change under ${u.expectedSources.join(' or ')}`);
|
||||
parts.push(
|
||||
`${result.unattributable.length} emitted path(s) changed that nothing in this diff explains:\n${list.join('\n')}` +
|
||||
(result.unattributable.length > sampleLimit
|
||||
? `\n …and ${result.unattributable.length - sampleLimit} more`
|
||||
: '') +
|
||||
'\n\nIf this ripple is intended, record it in tests/emitted-drift-ack.json naming each\n' +
|
||||
'path and why. Do NOT regenerate anything to silence this.',
|
||||
);
|
||||
}
|
||||
|
||||
const unackedGrowth = result.grown.filter((g) => !g.acked);
|
||||
if (unackedGrowth.length) {
|
||||
const list = unackedGrowth.slice(0, sampleLimit)
|
||||
.map((g) => ` ${g.name} grew ${g.delta} bytes (${g.from} -> ${g.to})`);
|
||||
parts.push(
|
||||
`${unackedGrowth.length} file(s) grew without an acknowledgment:\n${list.join('\n')}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (result.staleAcks.length) {
|
||||
parts.push(
|
||||
`${result.staleAcks.length} stale acknowledgment(s) — the ripple they explained is gone, ` +
|
||||
'so they must be deleted (an ack that outlives its ripple pre-clears the next one):\n ' +
|
||||
result.staleAcks.slice(0, sampleLimit).join('\n '),
|
||||
);
|
||||
}
|
||||
|
||||
return parts.join('\n\n');
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
ACK_VERSION,
|
||||
sourceSatisfiedBy,
|
||||
parseAck,
|
||||
diffEmitted,
|
||||
formatReport,
|
||||
};
|
||||
269
tests/helpers/emitted-runtime.cjs
Normal file
269
tests/helpers/emitted-runtime.cjs
Normal file
@@ -0,0 +1,269 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Real-world I/O shell for the differential attribution check (#2723, ADR-2719).
|
||||
*
|
||||
* `emitted-diff.cjs` holds the pure conservation law; this module is the only place
|
||||
* that touches git, the filesystem, or the installer. Keeping them apart is what makes
|
||||
* the law's acceptance criteria testable in milliseconds — but the shell still has to
|
||||
* exist and actually run, or the phase ships as interface-only, which an isolated
|
||||
* reviewer correctly called out on the first cut of this work.
|
||||
*
|
||||
* ── Baseline source during the dual-run window ───────────────────────────────
|
||||
* The baseline is the emitted manifest set at `next` HEAD. During Phase 3 that is
|
||||
* available for FREE and for REAL via `git show origin/next:<fixture>` — the committed
|
||||
* golden fixtures ARE next's recorded emitted state, and CI keeps them current there.
|
||||
* No worktree, no rebuild, no 19 installer spawns for the baseline side.
|
||||
*
|
||||
* Critically this is NOT the same as reading the fixtures from the WORKING TREE: those
|
||||
* are whatever the PR author regenerated, so comparing against them would be vacuous
|
||||
* (current vs. the author's own regeneration). Reading them at `origin/next` is what
|
||||
* makes the comparison a real differential against upstream state.
|
||||
*
|
||||
* Phase 4 (#2724) deletes the fixtures, at which point `resolveBaseline`'s cache path
|
||||
* (already implemented and tested in emitted-baseline.cjs) becomes the source. That
|
||||
* swap is the only change Phase 4 needs here.
|
||||
*
|
||||
* The CURRENT side is built for real — 19 installer spawns via the same
|
||||
* `runMinimalInstall` + `buildParityManifest` the golden harness uses. It is the
|
||||
* expensive half on purpose: if a PR forgot to regenerate, current-real differs from
|
||||
* next's recorded state and the attribution actually runs, which is the whole point.
|
||||
*/
|
||||
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { execFileSync } = require('node:child_process');
|
||||
|
||||
const { cleanup } = require('../helpers.cjs');
|
||||
const {
|
||||
RUNTIME_META,
|
||||
runMinimalInstall,
|
||||
buildParityManifest,
|
||||
} = require('./install-shared.cjs');
|
||||
|
||||
const REPO_ROOT = path.join(__dirname, '..', '..');
|
||||
const ACK_PATH = path.join(REPO_ROOT, 'tests', 'emitted-drift-ack.json');
|
||||
const FIXTURE_SUBDIR = 'tests/fixtures/golden-install-parity';
|
||||
|
||||
/**
|
||||
* The emitted manifest families, as (fixtureName -> install spec).
|
||||
*
|
||||
* NOT simply `Object.keys(RUNTIME_META)`: that has 18 entries while the fixture set has
|
||||
* 19. The extra one is `claude-local` — claude is the reference host and the ONLY
|
||||
* runtime with a distinct LOCAL "legacy flat-commands" layout (`commands/gsd-*.md` +
|
||||
* `agents/gsd-*.md` at project scope), which `golden-install-parity.test.cjs` guards
|
||||
* with a hand-coded test outside its RUNTIME_META loop (#2086).
|
||||
*
|
||||
* Enumerating from RUNTIME_META alone dropped that family from BOTH sides of the
|
||||
* differential, so a same-count self-check (18 === 18) passed vacuously and a PR
|
||||
* changing Claude's local-scope output would fail the golden while this check reported
|
||||
* ok. That disagreement is exactly what the dual-run window is meant to surface as a
|
||||
* provenance-table hole — so a wiring omission masquerading as one is the worst
|
||||
* possible failure here. Derived explicitly, and asserted against the fixture count.
|
||||
*/
|
||||
const MANIFEST_FAMILIES = [
|
||||
...Object.keys(RUNTIME_META).map((runtime) => ({ name: runtime, runtime, scope: 'global' })),
|
||||
{ name: 'claude-local', runtime: 'claude', scope: 'local' },
|
||||
];
|
||||
|
||||
/** Bounded git invocation. CLAUDE.md → KNOWN DEFECTS: every git subprocess needs a
|
||||
* timeout (5-30s); an unbounded execFileSync is an indefinite hang, and it is how
|
||||
* macOS CI silently stops reporting. */
|
||||
const GIT_TIMEOUT_MS = 30_000;
|
||||
|
||||
function git(args, { cwd = REPO_ROOT } = {}) {
|
||||
return execFileSync('git', args, {
|
||||
cwd,
|
||||
encoding: 'utf8',
|
||||
timeout: GIT_TIMEOUT_MS,
|
||||
maxBuffer: 64 * 1024 * 1024,
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Repo paths the PR changed, via the three-dot form so the comparison is against the
|
||||
* merge base rather than the tip of `base`.
|
||||
*
|
||||
* A git failure THROWS. It must never degrade to an empty array: reading "git broke" as
|
||||
* "nothing changed" would make every moved hash unattributable and produce a failure
|
||||
* storm that reads exactly like a real finding.
|
||||
*/
|
||||
function resolveChangedPaths(base = 'origin/next') {
|
||||
let out;
|
||||
try {
|
||||
out = git(['diff', '--name-only', `${base}...HEAD`]);
|
||||
} catch (err) {
|
||||
throw new Error(
|
||||
`emitted-attribution: could not resolve changed paths from "${base}...HEAD": ${err.message}. ` +
|
||||
'This is a hard error on purpose — treating it as "no changes" would mark every ' +
|
||||
'moved emitted path unattributable.',
|
||||
);
|
||||
}
|
||||
return out.split('\n').map((l) => l.trim()).filter(Boolean);
|
||||
}
|
||||
|
||||
/** Resolve `base` to a 40-hex sha, for the baseline cache-key discipline (ADR §5). */
|
||||
function resolveBaseSha(base = 'origin/next') {
|
||||
return git(['rev-parse', base]).trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* Base-ref candidates, most-specific first.
|
||||
*
|
||||
* The differential needs a ref for `next`, and that ref is NOT universally present:
|
||||
* - the gsd-test runner shallow-clones and merges base+head, so no `origin/*`
|
||||
* remote-tracking refs exist in the container (verified: `git rev-parse
|
||||
* origin/next` fails there, which is what turned this test red on its first run);
|
||||
* - GitHub Actions' checkout does not create remote-tracking branches for OTHER
|
||||
* branches by default, which is exactly why `changeset-required.yml` carries an
|
||||
* explicit `git fetch origin "${BASE_REF}:refs/remotes/origin/${BASE_REF}"` step.
|
||||
*
|
||||
* `GSD_EMITTED_BASE` lets a lane name the ref (or sha) directly. `GITHUB_BASE_REF` is
|
||||
* set by Actions on pull_request events.
|
||||
*/
|
||||
function baseRefCandidates(env = process.env) {
|
||||
const candidates = [];
|
||||
if (env.GSD_EMITTED_BASE) candidates.push(env.GSD_EMITTED_BASE);
|
||||
if (env.GITHUB_BASE_REF) {
|
||||
candidates.push(`origin/${env.GITHUB_BASE_REF}`, env.GITHUB_BASE_REF);
|
||||
}
|
||||
candidates.push('origin/next', 'next');
|
||||
return [...new Set(candidates)];
|
||||
}
|
||||
|
||||
/**
|
||||
* First candidate base ref that actually resolves, or null when none do.
|
||||
*
|
||||
* Returning null is NOT a pass — the caller turns it into an explicit `t.skip()` with
|
||||
* the full candidate list in the message, so an environment where the gate did not run
|
||||
* says so out loud. A bare `return` there would be a PASS (ADR-2719 §6), and a hard
|
||||
* failure would make the suite permanently red in the gsd-test container, where no
|
||||
* base ref can exist by construction.
|
||||
*/
|
||||
function resolveBase(env = process.env) {
|
||||
for (const candidate of baseRefCandidates(env)) {
|
||||
try {
|
||||
const sha = git(['rev-parse', '--verify', `${candidate}^{commit}`]).trim();
|
||||
if (/^[0-9a-f]{40}$/.test(sha)) return { ref: candidate, sha };
|
||||
} catch { /* try the next candidate */ }
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Emitted manifest set at `base`, read from the committed fixtures at that ref.
|
||||
* Returns null when the fixtures are absent at `base` (i.e. after Phase 4's cutover),
|
||||
* which is the signal to fall back to `resolveBaseline`'s cache path.
|
||||
*/
|
||||
function baselineManifestsAtRef(base = 'origin/next') {
|
||||
const manifests = {};
|
||||
let found = 0;
|
||||
for (const { name } of MANIFEST_FAMILIES) {
|
||||
let raw;
|
||||
try {
|
||||
raw = git(['show', `${base}:${FIXTURE_SUBDIR}/${name}.json`]);
|
||||
} catch {
|
||||
continue; // absent at that ref
|
||||
}
|
||||
let parsed;
|
||||
try {
|
||||
parsed = JSON.parse(raw);
|
||||
} catch (err) {
|
||||
throw new Error(`emitted-attribution: ${base}:${FIXTURE_SUBDIR}/${name}.json is not valid JSON: ${err.message}`);
|
||||
}
|
||||
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||
throw new Error(`emitted-attribution: ${base}:${FIXTURE_SUBDIR}/${name}.json must be an object of path->hash`);
|
||||
}
|
||||
manifests[name] = parsed;
|
||||
found++;
|
||||
}
|
||||
return found === 0 ? null : manifests;
|
||||
}
|
||||
|
||||
/** Size maps at `base`, for the ratchet half. Null when absent at that ref. */
|
||||
function baselineSizesAtRef(base = 'origin/next') {
|
||||
const sizes = {};
|
||||
let found = 0;
|
||||
for (const rel of ['tests/workflow-size-baseline.json', 'tests/agent-size-baseline.json']) {
|
||||
try {
|
||||
const parsed = JSON.parse(git(['show', `${base}:${rel}`]));
|
||||
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
|
||||
Object.assign(sizes, parsed);
|
||||
found++;
|
||||
}
|
||||
} catch { /* absent at that ref */ }
|
||||
}
|
||||
return found === 0 ? null : sizes;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the CURRENT emitted manifest set for real — one installer spawn per runtime.
|
||||
* This is the expensive, honest half: it reflects what the tree actually emits now,
|
||||
* not what the author regenerated into a fixture.
|
||||
*/
|
||||
function currentManifests() {
|
||||
const manifests = {};
|
||||
for (const { name, runtime, scope } of MANIFEST_FAMILIES) {
|
||||
const { configDir, root } = runMinimalInstall({ runtime, scope });
|
||||
try {
|
||||
manifests[name] = buildParityManifest(configDir, root);
|
||||
} finally {
|
||||
cleanup(root);
|
||||
}
|
||||
}
|
||||
return manifests;
|
||||
}
|
||||
|
||||
/** Current on-disk sizes for the workflow + agent families the ratchet covers. */
|
||||
function currentSizes() {
|
||||
const sizes = {};
|
||||
for (const [dir, filter] of [
|
||||
[path.join(REPO_ROOT, 'gsd-core', 'workflows'), (f) => f.endsWith('.md')],
|
||||
[path.join(REPO_ROOT, 'agents'), (f) => f.endsWith('.md')],
|
||||
]) {
|
||||
if (!fs.existsSync(dir)) continue;
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
if (!entry.isFile() || !filter(entry.name)) continue;
|
||||
sizes[entry.name] = fs.statSync(path.join(dir, entry.name)).size;
|
||||
}
|
||||
}
|
||||
return sizes;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read `tests/emitted-drift-ack.json`.
|
||||
* Absent is legal and means "no acks" — its PRESENCE is the alarm (ADR §3).
|
||||
* A present-but-unreadable or unparseable file THROWS: silently treating it as absent
|
||||
* would disarm the gate in the one case where someone is actively using it.
|
||||
*/
|
||||
function readAckFile(ackPath = ACK_PATH) {
|
||||
if (!fs.existsSync(ackPath)) return null;
|
||||
const raw = fs.readFileSync(ackPath, 'utf8');
|
||||
if (raw.trim() === '') {
|
||||
throw new Error(`emitted-attribution: ${path.basename(ackPath)} is present but empty`);
|
||||
}
|
||||
try {
|
||||
return JSON.parse(raw);
|
||||
} catch (err) {
|
||||
throw new Error(`emitted-attribution: ${path.basename(ackPath)} is not valid JSON: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
REPO_ROOT,
|
||||
ACK_PATH,
|
||||
FIXTURE_SUBDIR,
|
||||
MANIFEST_FAMILIES,
|
||||
GIT_TIMEOUT_MS,
|
||||
git,
|
||||
resolveChangedPaths,
|
||||
resolveBaseSha,
|
||||
baseRefCandidates,
|
||||
resolveBase,
|
||||
baselineManifestsAtRef,
|
||||
baselineSizesAtRef,
|
||||
currentManifests,
|
||||
currentSizes,
|
||||
readAckFile,
|
||||
};
|
||||
Reference in New Issue
Block a user