* test(#2854): failing-first coverage for CI baseline export provenance Extracts the export decision out of main() behind injected IO so it is unit-testable, preserving today's export-whenever-present semantics, and adds the matrix that proves those semantics are wrong. The PR lane restores the emitted baseline keyed on the PR's recorded base sha while the gate resolves the base ref live, so the two drift whenever next advances mid-flight. The restore was published straight to GSD_EMITTED_BASELINE, where a mismatch is fatal, turning a recoverable cache into a hard failure on diffs that touched nothing related. Also renames the stale-env fixture from 'from-cache-restore.json' to an operator-pin name: that fixture asserted the exact conflation this bug is, documenting the defect as intended behavior. Refs #2854 * fix(#2854): validate a restored baseline before publishing it as an operator pin GSD_EMITTED_BASELINE is an operator pin: resolveBaseline() treats a mismatch there as a hard stop, because the operator said "use this one". CI published its cache restore to that same variable whenever the file merely existed, so a restore keyed on the PR's recorded base sha - while the gate resolves the base ref live - turned a recoverable cache into a fatal error whenever next advanced mid-flight. Required tests went red on diffs that touched nothing related, and named a test file the contributor never opened. The export step is the boundary, so it is the boundary that validates. It now publishes only a baseline already valid for the sha under test, judged by validateBaseline so the staleness rule keeps one definition. Anything refused is left to be found via the cache path, where a mismatch degrades to the in-job build exactly as ADR-2719 SS5 specifies. The operator hard stop is untouched, and the fast path still hits on a current cache. Also reports the sources actually reached rather than asserting all three ran: the failure message claimed an in-job build it had returned before calling, sending contributors after a rebuild that never happened. Fixes #2854 * fix(#2854): pin the emitted gate to the base the tree was actually merged with The differential compared a tree built on one commit against a baseline at a different one. "Rebase check" merges pull_request.base.sha, pinned by #2472 so all 12 matrix jobs agree on one tree, but resolveBase() fell through to origin/next, which fetch-depth 0 leaves at the live tip. Nothing set GSD_EMITTED_BASE, so whenever next advanced mid-flight the two disagreed. The cached baseline, keyed on base.sha, was correct for that tree and was rejected as STALE by a target that was not. Required tests went red on diffs that touched nothing related, naming a test file the contributor never opened. The near miss is the worse half: had resolution gotten past the baseline step, a baseline at the live tip would have attributed commits merged to next in between to the PR under test. The hard stop was shielding us from a wrong answer, so making it fall through would have made this worse. Pins GSD_EMITTED_BASE to the same expression as CI_REBASE_BASE_SHA in every rebase-merged job, with a parity test asserting the two cannot diverge. That parity check immediately caught a third lane, test-inert, that merges a pinned base and had been missed. Fixes #2854 * fix(#2854): keep the export step self-contained across the package boundary scripts/ ships in the npm tarball and tests/ does not, so requiring the validator across that boundary is MODULE_NOT_FOUND in a published install. The export step now reads the same GSD_EMITTED_BASE pin the gate resolves through, and applies a cheap self-contained precondition; validateBaseline remains the sole authority and still runs downstream on whatever is published, so there is no second opinion to drift. Reading the pin rather than re-deriving a base is the point: a second, divergent base lookup is exactly what caused this bug. The same hazard pre-exists in scripts/gen-emitted-baseline.cjs, which ships and requires three tests/ modules. Filed as #2858 rather than folded in: fixing it means relocating the shared helpers out of tests/ and updating every consumer, which would bury this change. Refs #2854 * fix(#2854): stop announcing the restored cache through the operator-pin door Two independent reviewers found the same blocker in the previous approach. Validating before publishing to GSD_EMITTED_BASELINE only narrowed the hole: the precondition gated on sha equality alone, so a document with a MATCHING sha but a wrong schema version or malformed manifests was still announced as an operator pin and still hard-stopped downstream. That reproduces this bug's own class, triggered by malformation instead of staleness. The step was never load-bearing. The cache restores to .gsd-cache/emitted-baseline.json, which is resolveBaseline's DEFAULT_CACHE_PATH and is read whether or not anything announces it. Publishing the same file to the pin door could only ever convert recoverable into fatal, so the step and its script are deleted rather than made cleverer. Every failure mode now degrades to the in-job build by construction, and validateBaseline is once again the only thing that judges a baseline. Coverage moves to where the behaviour lives: stale sha, wrong schema version, manifests array/absent, non-object documents, unreadable file, and the 39/40/41 hex boundary all assert degradation via the cache path. Adds the empty-pin case a reviewer flagged as untested - the pin is job-level env, so on push events it renders as an empty string, and only baseRefCandidates' truthy check keeps it out of the candidate list. Fixes #2854 --------- Co-authored-by: Test <test@example.com>
This commit is contained in:
Tom Boucher
48
.github/workflows/test.yml
vendored
48
.github/workflows/test.yml
vendored
@@ -119,6 +119,15 @@ jobs:
|
||||
timeout-minutes: 15
|
||||
env:
|
||||
GSD_PLUGIN_ROOT: .ci-gsd-plugin-root-disabled
|
||||
# #2854: pin the emitted gate's baseline to the SAME commit the tree was merged
|
||||
# with. "Rebase check" merges `pull_request.base.sha` (pinned by #2472 so all 12
|
||||
# matrix jobs agree on one tree), but `resolveBase()` otherwise falls through to
|
||||
# `origin/next`, which `fetch-depth: 0` leaves at the LIVE tip. Whenever `next`
|
||||
# advanced mid-flight the gate compared a tree built on base.sha against a
|
||||
# baseline at a newer commit — so the correctly-keyed cache was rejected as
|
||||
# "stale" and the run hard-failed on diffs that touched nothing related.
|
||||
# This must stay equal to CI_REBASE_BASE_SHA; a test asserts that parity.
|
||||
GSD_EMITTED_BASE: ${{ github.event.pull_request.base.sha }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -188,8 +197,16 @@ jobs:
|
||||
|
||||
# #2724 (ADR-2719 §5): the differential attribution check's baseline is cached,
|
||||
# not committed. This restores what the push-to-next job published, keyed on
|
||||
# the PR's base sha — the "next sha the PR was merged with" the ADR requires.
|
||||
# A miss is NOT a failure here: resolveBaseline() degrades to an in-job build.
|
||||
# the PR's base sha — the "next sha the PR was merged with" the ADR requires,
|
||||
# and the same commit `ci-rebase-check` merged into this tree.
|
||||
#
|
||||
# #2854: the restore lands on resolveBaseline()'s DEFAULT_CACHE_PATH, so it is
|
||||
# found without being announced. There used to be a step that published this
|
||||
# path to GSD_EMITTED_BASELINE — an OPERATOR PIN, where any rejection is a hard
|
||||
# stop rather than a fall-through. That converted every recoverable condition
|
||||
# (stale, malformed, wrong schema version) into a fatal CI failure on diffs that
|
||||
# touched nothing related. Restoring to the default path and saying nothing lets
|
||||
# a bad artifact degrade to the in-job build, which is what ADR-2719 §5 specifies.
|
||||
- name: Restore emitted-baseline cache
|
||||
if: github.event_name == 'pull_request'
|
||||
uses: actions/cache/restore@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
|
||||
@@ -197,13 +214,6 @@ jobs:
|
||||
path: .gsd-cache/emitted-baseline.json
|
||||
key: emitted-baseline-${{ github.event.pull_request.base.sha }}
|
||||
|
||||
- name: Export GSD_EMITTED_BASELINE when the cache restored
|
||||
if: github.event_name == 'pull_request'
|
||||
# No `shell:` override — H1 shell policy requires each job's native shell
|
||||
# (pwsh on Windows, zsh on macOS test-full, bash on Linux). A bare `node`
|
||||
# invocation has no shell-specific syntax, so it is valid under all three.
|
||||
run: node scripts/ci-export-emitted-baseline-env.cjs
|
||||
|
||||
- name: Prepare scoped test list
|
||||
if: matrix.scope != 'full'
|
||||
env:
|
||||
@@ -275,6 +285,10 @@ jobs:
|
||||
timeout-minutes: 15
|
||||
env:
|
||||
GSD_PLUGIN_ROOT: .ci-gsd-plugin-root-disabled
|
||||
# #2854: same pin as the other rebase-merged lanes. This lane runs a targeted
|
||||
# list that can include the emitted gate, and the invariant is easier to keep
|
||||
# with no exceptions: if a job merges a pinned base, the gate uses that base.
|
||||
GSD_EMITTED_BASE: ${{ github.event.pull_request.base.sha }}
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
@@ -342,6 +356,15 @@ jobs:
|
||||
timeout-minutes: 20
|
||||
env:
|
||||
GSD_PLUGIN_ROOT: .ci-gsd-plugin-root-disabled
|
||||
# #2854: pin the emitted gate's baseline to the SAME commit the tree was merged
|
||||
# with. "Rebase check" merges `pull_request.base.sha` (pinned by #2472 so all 12
|
||||
# matrix jobs agree on one tree), but `resolveBase()` otherwise falls through to
|
||||
# `origin/next`, which `fetch-depth: 0` leaves at the LIVE tip. Whenever `next`
|
||||
# advanced mid-flight the gate compared a tree built on base.sha against a
|
||||
# baseline at a newer commit — so the correctly-keyed cache was rejected as
|
||||
# "stale" and the run hard-failed on diffs that touched nothing related.
|
||||
# This must stay equal to CI_REBASE_BASE_SHA; a test asserts that parity.
|
||||
GSD_EMITTED_BASE: ${{ github.event.pull_request.base.sha }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -439,13 +462,6 @@ jobs:
|
||||
path: .gsd-cache/emitted-baseline.json
|
||||
key: emitted-baseline-${{ github.event.pull_request.base.sha }}
|
||||
|
||||
- name: Export GSD_EMITTED_BASELINE when the cache restored
|
||||
if: github.event_name == 'pull_request'
|
||||
# No `shell:` override — H1 shell policy requires each job's native shell
|
||||
# (pwsh on Windows, zsh on macOS test-full, bash on Linux). A bare `node`
|
||||
# invocation has no shell-specific syntax, so it is valid under all three.
|
||||
run: node scripts/ci-export-emitted-baseline-env.cjs
|
||||
|
||||
# The heavy unit suite is split across the 3 shards — each runs a
|
||||
# deterministic cost-balanced third of the sorted unit-file list (#2472). The
|
||||
# union of shards 1/3 + 2/3 + 3/3 is the full unit suite, so coverage is
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* ci-export-emitted-baseline-env.cjs — exports GSD_EMITTED_BASELINE for the
|
||||
* differential attribution check when its baseline cache was restored (#2724,
|
||||
* ADR-2719 §5).
|
||||
*
|
||||
* A plain `node <script>` invocation on purpose, not a shell script. The repo's H1
|
||||
* shell policy (scripts/workflow-policy.cjs) requires every CI step to run under its
|
||||
* job/matrix-declared NATIVE shell — pwsh on Windows, zsh on macOS (test-full's own
|
||||
* `shell: 'zsh {0}'`), bash on Linux — and a step that hardcodes `shell: bash`
|
||||
* violates that on the other two. A cross-platform Node script run via a bare
|
||||
* `node scripts/...` command line has no shell-specific syntax at all, so it is
|
||||
* valid unmodified under bash, zsh, AND pwsh — sidestepping the conflict instead of
|
||||
* picking one shell and being wrong on the other two.
|
||||
*/
|
||||
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const REPO_ROOT = path.join(__dirname, '..');
|
||||
const CACHE_PATH = path.join(REPO_ROOT, '.gsd-cache', 'emitted-baseline.json');
|
||||
|
||||
function main() {
|
||||
const githubEnv = process.env.GITHUB_ENV;
|
||||
|
||||
if (!fs.existsSync(CACHE_PATH)) {
|
||||
console.log(`ci-export-emitted-baseline-env: ${CACHE_PATH} absent (cache miss) — nothing to export`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!githubEnv) {
|
||||
// Not fatal: running this locally or outside Actions should not break anything.
|
||||
console.log('ci-export-emitted-baseline-env: GITHUB_ENV not set — skipping export (not running under Actions?)');
|
||||
return 0;
|
||||
}
|
||||
|
||||
fs.appendFileSync(githubEnv, `GSD_EMITTED_BASELINE=${CACHE_PATH}\n`, 'utf8');
|
||||
console.log(`ci-export-emitted-baseline-env: exported GSD_EMITTED_BASELINE=${CACHE_PATH}`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
process.exitCode = main();
|
||||
@@ -76,10 +76,10 @@ const {
|
||||
const {
|
||||
BASELINE_ENV,
|
||||
BASELINE_VERSION,
|
||||
DEFAULT_CACHE_PATH,
|
||||
resolveBaseline,
|
||||
} = require('./helpers/emitted-baseline.cjs');
|
||||
|
||||
|
||||
const SHA_A = 'a'.repeat(40);
|
||||
const SHA_B = 'b'.repeat(40);
|
||||
|
||||
@@ -1431,9 +1431,15 @@ test('baseline resolution precedence is explicit and reported', () => {
|
||||
|
||||
// …and an explicitly-pointed-at stale baseline is a hard stop, not a fall-through:
|
||||
// the operator said "use this one".
|
||||
//
|
||||
// #2854: this fixture used to be named '/tmp/from-cache-restore.json', which asserted
|
||||
// the exact conflation that broke CI — a CI cache restore is NOT an operator pin, and
|
||||
// naming it one here documented the defect as intended behavior. The hard stop is a
|
||||
// real guarantee for a HAND-SET path and is preserved; what changed is that CI no
|
||||
// longer routes its restore through this door at all.
|
||||
const envStale = resolveBaseline({
|
||||
expectedSha: SHA_A,
|
||||
env: { [BASELINE_ENV]: '/tmp/x.json' },
|
||||
env: { [BASELINE_ENV]: '/tmp/operator-pinned-baseline.json' },
|
||||
cachePath: 'cache.json',
|
||||
readJson: () => goodBaseline(SHA_B),
|
||||
buildFallback: () => goodBaseline(SHA_A),
|
||||
@@ -1450,6 +1456,182 @@ test('baseline resolution precedence is explicit and reported', () => {
|
||||
assert.equal(viaBuild.via, 'build');
|
||||
});
|
||||
|
||||
// ── #2854: a CI cache restore is not an operator pin ─────────────────────────────
|
||||
//
|
||||
// ADR-2719 §5: "Cache miss falls back to an in-job build at `origin/next`." The PR lane
|
||||
// restores the baseline keyed on the PR's RECORDED base sha (`test.yml:198`) while the
|
||||
// gate resolves the base ref LIVE (`resolveBase()`), so the two drift whenever `next`
|
||||
// advances between a PR's last sync and its run. The restore was published straight to
|
||||
// GSD_EMITTED_BASELINE, where a mismatch is fatal — turning a recoverable cache into a
|
||||
// hard failure on diffs that touched nothing related. The export step is the boundary
|
||||
// that must be conservative in what it sends.
|
||||
|
||||
const CI_CACHE_PATH = '.gsd-cache/emitted-baseline.json';
|
||||
|
||||
/** Resolve exactly as CI does: cache restored to the DEFAULT path, nothing announced. */
|
||||
const resolveAsCI = (doc, expectedSha = SHA_A) => resolveBaseline({
|
||||
expectedSha,
|
||||
env: {}, // no operator pin — this is the whole point
|
||||
cachePath: CI_CACHE_PATH,
|
||||
readJson: typeof doc === 'function' ? doc : () => doc,
|
||||
buildFallback: () => goodBaseline(expectedSha),
|
||||
});
|
||||
|
||||
test('#2854: a drifted cache restore degrades to the in-job build', () => {
|
||||
const r = resolveAsCI(goodBaseline(SHA_B)); // restored under a drifted key
|
||||
assert.ok(r.ok, `must resolve via the in-job build; got: ${(r.errors || []).join('; ')}`);
|
||||
assert.equal(r.via, 'build');
|
||||
assert.equal(r.sha, SHA_A);
|
||||
assert.deepEqual(r.attempted, [`cache:${CI_CACHE_PATH}`, 'build'],
|
||||
'the cache must be tried and rejected before the build, and the trail must say so');
|
||||
});
|
||||
|
||||
test('#2854: a current cache restore is used directly (the fast path survives)', () => {
|
||||
const r = resolveAsCI(goodBaseline(SHA_A));
|
||||
assert.ok(r.ok);
|
||||
assert.equal(r.via, `cache:${CI_CACHE_PATH}`, 'a valid cache must not pay for a rebuild');
|
||||
});
|
||||
|
||||
test('#2854: every recoverable malformation degrades rather than failing the run', () => {
|
||||
// The blocker an isolated reviewer caught: an earlier revision gated only on sha
|
||||
// equality, so a doc with the RIGHT sha but a wrong schema version or broken
|
||||
// manifests was announced as an operator pin and hard-stopped downstream —
|
||||
// reproducing this bug's own class, triggered by malformation instead of staleness.
|
||||
// Routing through the cache path makes every one of these recoverable by construction.
|
||||
const cases = {
|
||||
'stale sha': goodBaseline(SHA_B),
|
||||
'wrong schema version': { version: BASELINE_VERSION + 998, sha: SHA_A, manifests: { c: {} }, sizes: {} },
|
||||
'manifests is an array': { version: BASELINE_VERSION, sha: SHA_A, manifests: [], sizes: {} },
|
||||
'manifests absent': { version: BASELINE_VERSION, sha: SHA_A },
|
||||
'sha absent': { version: BASELINE_VERSION, manifests: { c: {} }, sizes: {} },
|
||||
'sha not 40-hex': { version: BASELINE_VERSION, sha: 'g'.repeat(40), manifests: { c: {} }, sizes: {} },
|
||||
'absent file': null,
|
||||
};
|
||||
|
||||
for (const [name, doc] of Object.entries(cases)) {
|
||||
const r = resolveAsCI(doc);
|
||||
assert.ok(r.ok, `${name}: must degrade to the build, not fail — got ${(r.errors || []).join('; ')}`);
|
||||
assert.equal(r.via, 'build', `${name}: must reach the in-job build`);
|
||||
}
|
||||
});
|
||||
|
||||
test('#2854: sha length boundary — 39, 40, 41 hex', () => {
|
||||
// limit-1 / limit / limit+1 on the 40-hex contract validateBaseline enforces.
|
||||
for (const len of [39, 41]) {
|
||||
const r = resolveAsCI({ version: BASELINE_VERSION, sha: 'a'.repeat(len), manifests: { c: {} }, sizes: {} });
|
||||
assert.equal(r.via, 'build', `${len} hex is not a sha — must not be used as the baseline`);
|
||||
}
|
||||
const exact = resolveAsCI(goodBaseline(SHA_A));
|
||||
assert.equal(exact.via, `cache:${CI_CACHE_PATH}`, '40 hex matching is the contract');
|
||||
});
|
||||
|
||||
test('#2854: valid JSON that is not an object degrades rather than passing vacuously', () => {
|
||||
for (const doc of [0, 'str', [], true]) {
|
||||
const r = resolveAsCI(doc);
|
||||
assert.ok(r.ok, `${JSON.stringify(doc)}: must degrade to the build`);
|
||||
assert.equal(r.via, 'build', `${JSON.stringify(doc)} must never read as a usable baseline`);
|
||||
}
|
||||
});
|
||||
|
||||
test('#2854: an unreadable cache degrades and does not throw', () => {
|
||||
// Deterministic IO failure by injection — never chmod 0o000, which root bypasses.
|
||||
const r = resolveAsCI(() => { throw new Error('EACCES: permission denied'); });
|
||||
assert.ok(r.ok);
|
||||
assert.equal(r.via, 'build');
|
||||
});
|
||||
|
||||
test('#2854: the cache is used exactly when it is valid for the sha under test', () => {
|
||||
const hex40 = fc.string({
|
||||
unit: fc.constantFrom(...'0123456789abcdef'), minLength: 40, maxLength: 40,
|
||||
});
|
||||
fc.assert(fc.property(hex40, hex40, (built, expected) => {
|
||||
const r = resolveAsCI(goodBaseline(built), expected);
|
||||
// Always resolves; the only question is whether it paid for a rebuild.
|
||||
if (!r.ok) return false;
|
||||
return (r.via === `cache:${CI_CACHE_PATH}`) === (built === expected);
|
||||
}), { numRuns: 200 });
|
||||
});
|
||||
|
||||
test('#2854: the gate is pinned to the SAME base the tree was merged with', () => {
|
||||
// The deepest half of this bug. "Rebase check" merges `pull_request.base.sha`,
|
||||
// pinned by #2472 so all 12 matrix jobs agree on one tree. But resolveBase()
|
||||
// otherwise falls through to `origin/next`, which `fetch-depth: 0` leaves at the
|
||||
// LIVE tip. When `next` advanced mid-flight the gate compared a tree built on
|
||||
// base.sha against a baseline at a NEWER commit — so the correctly-keyed cache
|
||||
// was rejected as "stale" and the run died. Worse than dying would be surviving:
|
||||
// a baseline at the wrong commit attributes other people's merges to this PR.
|
||||
//
|
||||
// Two surfaces read one value, which is the generative-divergence shape this repo
|
||||
// has been bitten by before, so the parity is asserted rather than assumed.
|
||||
const yaml = require('js-yaml');
|
||||
const wf = yaml.load(fs.readFileSync(path.join(REPO_ROOT, '.github/workflows/test.yml'), 'utf8'));
|
||||
|
||||
const jobsUnderTest = Object.entries(wf.jobs).filter(([, job]) =>
|
||||
(job.steps || []).some((s) => typeof s.run === 'string' && s.run.includes('ci-rebase-check.cjs')));
|
||||
|
||||
assert.ok(jobsUnderTest.length >= 2,
|
||||
`expected the rebase-pinned jobs to be found, got ${jobsUnderTest.length}`);
|
||||
|
||||
for (const [name, job] of jobsUnderTest) {
|
||||
const rebaseStep = job.steps.find((s) => typeof s.run === 'string' && s.run.includes('ci-rebase-check.cjs'));
|
||||
const mergedBase = (rebaseStep.env || {}).CI_REBASE_BASE_SHA;
|
||||
const gateBase = (job.env || {}).GSD_EMITTED_BASE;
|
||||
|
||||
assert.ok(mergedBase, `job ${name}: rebase step must pin CI_REBASE_BASE_SHA`);
|
||||
assert.equal(gateBase, mergedBase,
|
||||
`job ${name}: the emitted gate's base (GSD_EMITTED_BASE=${JSON.stringify(gateBase)}) must equal ` +
|
||||
`the commit the tree was merged with (CI_REBASE_BASE_SHA=${JSON.stringify(mergedBase)}). ` +
|
||||
'Diverging them makes the differential compare a tree against a baseline from a ' +
|
||||
'different commit, which mis-attributes unrelated merges to this PR.');
|
||||
}
|
||||
});
|
||||
|
||||
test('#2854: an explicit base pin outranks the live branch tip', () => {
|
||||
// The mechanism the workflow pin relies on: GSD_EMITTED_BASE must win over
|
||||
// origin/<base>, or setting it in CI would change nothing.
|
||||
const pinned = 'c'.repeat(40);
|
||||
assert.equal(baseRefCandidates({ GSD_EMITTED_BASE: pinned, GITHUB_BASE_REF: 'next' })[0], pinned,
|
||||
'an explicit pin must be tried before origin/next');
|
||||
});
|
||||
|
||||
test('#2854: an EMPTY base pin is ignored, not treated as a candidate', () => {
|
||||
// The pin is job-level env, so on push/workflow_dispatch — where there is no
|
||||
// pull_request — `${{ github.event.pull_request.base.sha }}` renders as an empty
|
||||
// string rather than being unset. baseRefCandidates' truthy check already excludes
|
||||
// it, but nothing asserted that, so narrowing the check to `!== undefined` would
|
||||
// silently push '' as the first candidate and have `git rev-parse ''` decide the
|
||||
// baseline. Pinned here because the workflow now guarantees this input shape.
|
||||
assert.deepEqual(
|
||||
baseRefCandidates({ GSD_EMITTED_BASE: '', GITHUB_BASE_REF: 'next' }),
|
||||
baseRefCandidates({ GITHUB_BASE_REF: 'next' }),
|
||||
'an empty pin must behave exactly as an absent one',
|
||||
);
|
||||
assert.ok(!baseRefCandidates({ GSD_EMITTED_BASE: '', GITHUB_BASE_REF: 'next' }).includes(''),
|
||||
'the empty string must never become a base-ref candidate');
|
||||
});
|
||||
|
||||
test('#2854: the resolution summary names only sources actually attempted', () => {
|
||||
// The caller's assertion message hardcoded "(tried env, <cache>, and an in-job build)"
|
||||
// on every failure, including early returns that reached none of them.
|
||||
const envOnly = resolveBaseline({
|
||||
expectedSha: SHA_A,
|
||||
env: { [BASELINE_ENV]: '/tmp/operator-pinned-baseline.json' },
|
||||
cachePath: 'cache.json',
|
||||
readJson: () => goodBaseline(SHA_B),
|
||||
buildFallback: () => goodBaseline(SHA_A),
|
||||
});
|
||||
assert.ok(!envOnly.ok);
|
||||
assert.deepEqual(envOnly.attempted, [`env:${BASELINE_ENV}`],
|
||||
'an env hard stop reaches neither the cache nor the build — the summary must say so');
|
||||
|
||||
const allThree = resolveBaseline({
|
||||
expectedSha: SHA_A, env: {}, cachePath: 'cache.json',
|
||||
readJson: () => goodBaseline(SHA_B),
|
||||
buildFallback: () => goodBaseline(SHA_A),
|
||||
});
|
||||
assert.deepEqual(allThree.attempted, ['cache:cache.json', 'build']);
|
||||
});
|
||||
|
||||
test('base-ref candidates are ordered most-specific first and de-duplicated', () => {
|
||||
// The gate went red on its first matrix run because it hard-depended on
|
||||
// `origin/next`, which cannot exist in the gsd-test container (shallow clone +
|
||||
@@ -2200,8 +2382,12 @@ test('differential attribution over the real tree', { timeout: 900_000 }, async
|
||||
});
|
||||
assert.ok(
|
||||
resolvedBaseline.ok,
|
||||
`no usable emitted baseline for ${base}@${baseSha.slice(0, 12)} (tried env, ` +
|
||||
`${DEFAULT_CACHE_PATH}, and an in-job build):\n ${(resolvedBaseline.errors || []).join('\n ')}`,
|
||||
// #2854: report the sources actually reached, not a hardcoded list of all three. An
|
||||
// early return could claim it "tried an in-job build" it never called, which sent
|
||||
// contributors hunting a rebuild that had not run.
|
||||
`no usable emitted baseline for ${base}@${baseSha.slice(0, 12)} (tried ` +
|
||||
`${(resolvedBaseline.attempted || []).join(', ') || 'nothing'}):` +
|
||||
`\n ${(resolvedBaseline.errors || []).join('\n ')}`,
|
||||
);
|
||||
const baseline = resolvedBaseline.baseline;
|
||||
assert.ok(baseline && Object.keys(baseline).length > 0, `resolved baseline via ${resolvedBaseline.via} has no families`);
|
||||
|
||||
@@ -26,7 +26,22 @@
|
||||
* unit-testable without touching the filesystem or spawning 19 installers.
|
||||
*/
|
||||
|
||||
/** Env var a CI cache-restore step points at the recovered baseline artifact. */
|
||||
/**
|
||||
* Operator pin: an explicit "use THIS baseline artifact" override.
|
||||
*
|
||||
* #2854: this was previously described as "the env var a CI cache-restore step points
|
||||
* at", and CI did exactly that. That is what broke — a value arriving here is treated
|
||||
* as intentional, so ANY rejection is a hard stop rather than a fall-through (see
|
||||
* `resolveBaseline` below). Correct for a human who pinned a path on purpose; wrong
|
||||
* for a cache restore, which is recoverable by definition.
|
||||
*
|
||||
* CI no longer publishes anything here. Its restore lands on `DEFAULT_CACHE_PATH`,
|
||||
* which `resolveBaseline` reads anyway, so a stale, malformed, or wrong-version
|
||||
* artifact degrades to the in-job build as ADR-2719 §5 specifies. Announcing the same
|
||||
* file through this door could only ever convert recoverable into fatal.
|
||||
*
|
||||
* Keep it that way: anything that sets this variable is asserting "use THIS, or stop".
|
||||
*/
|
||||
const BASELINE_ENV = 'GSD_EMITTED_BASELINE';
|
||||
|
||||
/** Default on-disk cache location, relative to the repo root. */
|
||||
@@ -94,19 +109,26 @@ function validateBaseline(doc, expectedSha, source) {
|
||||
* Resolve the baseline through the documented precedence, reporting WHICH step supplied
|
||||
* it so a failure message can say where the answer came from.
|
||||
*
|
||||
* 1. `GSD_EMITTED_BASELINE` — explicit path (CI cache restore writes here)
|
||||
* 2. the on-disk cache, validated against the expected sha
|
||||
* 1. `GSD_EMITTED_BASELINE` — an operator pin; a mismatch here is a HARD STOP
|
||||
* 2. the on-disk cache, validated against the expected sha — a mismatch RECOVERS
|
||||
* 3. an in-job build at `origin/next` (slow fallback)
|
||||
* 4. none → explicit failure (NEVER a silent pass)
|
||||
*
|
||||
* #2854: steps 1 and 2 differ only in what a mismatch means, so which door a given
|
||||
* artifact arrives through decides whether the run recovers or dies. CI publishes to
|
||||
* step 1 only after validating, so an unvalidated restore lands on step 2 and degrades.
|
||||
*
|
||||
* @param {object} opts
|
||||
* @param {string} opts.expectedSha `next` sha under test
|
||||
* @param {object} [opts.env] environment (injected)
|
||||
* @param {string} [opts.cachePath]
|
||||
* @param {function} opts.readJson (path) => parsed | null (null when absent)
|
||||
* @param {function} [opts.buildFallback] () => artifact | null (the slow path)
|
||||
* @returns {{ok: true, via: string, baseline: object, sizeBaseline: object|null, sha: string}
|
||||
* |{ok: false, via: string, errors: string[]}}
|
||||
* @returns {{ok: true, via: string, attempted: string[], baseline: object,
|
||||
* sizeBaseline: object|null, sha: string}
|
||||
* |{ok: false, via: string, attempted: string[], errors: string[]}}
|
||||
* `attempted` lists the sources actually REACHED, in order — callers render it
|
||||
* instead of assuming all three ran.
|
||||
*/
|
||||
function resolveBaseline({
|
||||
expectedSha,
|
||||
@@ -116,13 +138,21 @@ function resolveBaseline({
|
||||
buildFallback = null,
|
||||
} = {}) {
|
||||
if (typeof readJson !== 'function') {
|
||||
return { ok: false, via: 'none', errors: ['resolveBaseline: readJson must be supplied'] };
|
||||
return {
|
||||
ok: false, via: 'none', attempted: [],
|
||||
errors: ['resolveBaseline: readJson must be supplied'],
|
||||
};
|
||||
}
|
||||
|
||||
const attempts = [];
|
||||
// #2854: the sources actually REACHED, in order. The caller renders this in its
|
||||
// failure message; hardcoding "tried env, cache, and an in-job build" there claimed
|
||||
// three attempts on every early return, including ones that reached only the first.
|
||||
const attempted = [];
|
||||
|
||||
const envPath = env && env[BASELINE_ENV];
|
||||
if (envPath) {
|
||||
attempted.push(`env:${BASELINE_ENV}`);
|
||||
let doc = null;
|
||||
let readError = null;
|
||||
try {
|
||||
@@ -134,14 +164,19 @@ function resolveBaseline({
|
||||
attempts.push(readError);
|
||||
} else {
|
||||
const v = validateBaseline(doc, expectedSha, `${BASELINE_ENV}=${envPath}`);
|
||||
if (v.ok) return { ok: true, via: `env:${BASELINE_ENV}`, ...v };
|
||||
if (v.ok) return { ok: true, via: `env:${BASELINE_ENV}`, attempted, ...v };
|
||||
attempts.push(...v.errors);
|
||||
// An EXPLICITLY pointed-at baseline that is stale or malformed is a hard stop, not
|
||||
// a reason to quietly fall through to a different one — the operator said "use this".
|
||||
return { ok: false, via: `env:${BASELINE_ENV}`, errors: attempts };
|
||||
//
|
||||
// #2854: this is a guarantee about a HAND-SET path, and it is why CI must not
|
||||
// publish its cache restore here. It used to, so a restore that was merely stale
|
||||
// or malformed died here instead of degrading one branch below.
|
||||
return { ok: false, via: `env:${BASELINE_ENV}`, attempted, errors: attempts };
|
||||
}
|
||||
}
|
||||
|
||||
attempted.push(`cache:${cachePath}`);
|
||||
let cacheDoc = null;
|
||||
let cacheErr = null;
|
||||
try {
|
||||
@@ -153,7 +188,7 @@ function resolveBaseline({
|
||||
attempts.push(cacheErr);
|
||||
} else if (cacheDoc !== null && cacheDoc !== undefined) {
|
||||
const v = validateBaseline(cacheDoc, expectedSha, cachePath);
|
||||
if (v.ok) return { ok: true, via: `cache:${cachePath}`, ...v };
|
||||
if (v.ok) return { ok: true, via: `cache:${cachePath}`, attempted, ...v };
|
||||
// A stale CACHE is recoverable: fall through to the build fallback, but keep the
|
||||
// reason so the final message explains why the slow path ran.
|
||||
attempts.push(...v.errors);
|
||||
@@ -162,17 +197,18 @@ function resolveBaseline({
|
||||
}
|
||||
|
||||
if (typeof buildFallback === 'function') {
|
||||
attempted.push('build');
|
||||
let built = null;
|
||||
try {
|
||||
built = buildFallback();
|
||||
} catch (err) {
|
||||
attempts.push(`in-job build at origin/next failed: ${err.message}`);
|
||||
return { ok: false, via: 'build', errors: attempts };
|
||||
return { ok: false, via: 'build', attempted, errors: attempts };
|
||||
}
|
||||
const v = validateBaseline(built, expectedSha, 'in-job build at origin/next');
|
||||
if (v.ok) return { ok: true, via: 'build', ...v };
|
||||
if (v.ok) return { ok: true, via: 'build', attempted, ...v };
|
||||
attempts.push(...v.errors);
|
||||
return { ok: false, via: 'build', errors: attempts };
|
||||
return { ok: false, via: 'build', attempted, errors: attempts };
|
||||
}
|
||||
|
||||
attempts.push(
|
||||
@@ -180,7 +216,7 @@ function resolveBaseline({
|
||||
'failure on purpose: a skipped propagation gate is worth less than a slow one ' +
|
||||
'(ADR-2719 §6), and in node:test a bare `return` is a PASS, not a skip.',
|
||||
);
|
||||
return { ok: false, via: 'none', errors: attempts };
|
||||
return { ok: false, via: 'none', attempted, errors: attempts };
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
|
||||
Reference in New Issue
Block a user