chore(3686): add release-tarball lifecycle smoke to install-smoke workflow (#3692)
* chore(3686): add release-tarball lifecycle smoke to install-smoke workflow Closes #3686. Adds a non-interactive lifecycle smoke that runs against the installed tarball (not the working tree). Catches the two recent release-time bug classes that the working-tree test suite cannot see: * #3684 — symbol mismatches between init.cjs imports and secrets.cjs exports that landed in v1.42.3 (closed/fixed-pending-release). * #3668 — bare `gsd-sdk` invocations in 75 of 78 workflow files with no `command -v gsd-sdk … elif node "$GSD_TOOLS"` fallback (open). Shape: * `scripts/release-tarball-smoke.cjs` — pure CJS module exporting a frozen `SMOKE` enum and a `runSmoke({ tarballPath, installPrefix, expectedVersion, fixtureDir, lifecycleCommands })` function. CLI `--json` mode prints `JSON.stringify(result)` and exits 0 iff `result.code === SMOKE.OK`. Install is `--prefix <tmpdir>` so it does not pollute global node_modules. * `tests/release-tarball-smoke.test.cjs` — 6 tests covering happy path, version mismatch, lifecycle command file resolution, missing-command detection, sdk binary callability, and structural workflow-body checks. Tests assert on the SMOKE enum directly; no `assert.match` on rendered prose, no try/finally in test bodies, no source-grep theater. Uses `before`/`after` to pack+install once across the test file. * `tests/release-tarball-smoke-workflow.test.cjs` — 7 structural assertions on the parsed install-smoke.yml IR (workflow_call trigger preserved, lifecycle step calls release-tarball-smoke.cjs with --json, jq check enforces result.code === "ok", path filter includes the new files, artifact-on-failure step present). * `.github/workflows/install-smoke.yml` — extended (not duplicated). New "Lifecycle smoke" step after the existing version check, on the same matrix. Artifact upload on failure for debugging. Path filter now triggers on changes to the new script + test. Per CONTRIBUTING.md §"Prohibited: Raw Text Matching on Test Outputs" this PR avoids the same anti-pattern that caused PR #3666 to be reverted (PR #3688) — the script returns frozen enum codes, tests assert on the enum, never on stdout strings. Workflow-body checks in Cycle 3 are INFORMATIONAL (count returned, not enforced) on this PR. After #3668's fix lands and the 75 missing fallbacks are added, the lane can be tightened to enforce zero. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(3686): harden release smoke workflow and query scanner * fix(3686): move tarball-smoke test to install suite to fix Windows ETIMEDOUT + coverage OOM Windows (Node 22/24/26, jobs 76565215694/76565215710/76565215812): release-tarball-smoke.test.cjs had no suite marker so run-tests.cjs classified it as 'unit', running it on Windows PR CI. The before() hook calls execFileSync(npm.cmd install -g ...) with a 55 s timeout; on Windows GHA runners this npm global install consistently hits ETIMEDOUT (~62 s observed), causing all 3 Windows lanes to fail. Coverage (job 76565215085): c8 ran test:coverage:unit (unit suite only) with V8 coverage tracking active across child processes. The tarball-smoke test's before() hook spawned npm install subprocesses while c8 held V8 coverage descriptors open, driving the Node heap to 4 GB+ and triggering an OOM abort during report generation (exit code 134, all 5682 tests had already passed). Fix: rename to tests/release-tarball-smoke.install.test.cjs so run-tests.cjs routes it to the 'install' suite. The install suite is already skipped on PR CI by design (test.yml lines 179-181: only runs on main push). The dedicated install-smoke.yml workflow continues to exercise this test on its own matrix. Also update the install-smoke.yml PR path filter and the structural wiring test assertion to match the new filename. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor(test): route tarball-smoke install test through tests/helpers.cjs Replaces direct fs.mkdtempSync and execFileSync calls in tests/release-tarball-smoke.install.test.cjs with createTempDir() and a new runNpm() helper in tests/helpers.cjs. Cleanup is now automatic via the helper. Addresses CodeRabbit Major refactor at https://github.com/gsd-build/get-shit-done/pull/3692#discussion_r3260433892. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
@@ -230,4 +230,29 @@ function toPosixPath(p) {
|
||||
return p == null ? p : p.split(path.sep).join('/');
|
||||
}
|
||||
|
||||
module.exports = { runGsdTools, createTempDir, createTempProject, createTempGitProject, cleanup, parseFrontmatter, isUsageOutput, captureConsole, toPosixPath, TOOLS_PATH };
|
||||
/**
|
||||
* Run an npm command via execFileSync with cross-platform portability.
|
||||
*
|
||||
* Handles the Windows `npm.cmd` vs POSIX `npm` distinction and the
|
||||
* `shell: true` requirement on Windows so tests do not need to
|
||||
* re-implement platform detection inline.
|
||||
*
|
||||
* @param {string[]} args - npm subcommand and flags (e.g. ['pack', '--pack-destination', dir]).
|
||||
* @param {object} [options] - execFileSync options merged with platform defaults.
|
||||
* `cwd`, `encoding`, `timeout`, and `env` are the commonly overridden keys.
|
||||
* @returns {string} trimmed stdout string (encoding: 'utf-8').
|
||||
* @throws {Error} re-throws the execFileSync error on non-zero exit so callers
|
||||
* get the full stderr in the error message.
|
||||
*/
|
||||
function runNpm(args, options = {}) {
|
||||
const isWindows = process.platform === 'win32';
|
||||
const npmCmd = isWindows ? 'npm.cmd' : 'npm';
|
||||
const defaults = {
|
||||
encoding: 'utf-8',
|
||||
shell: isWindows,
|
||||
timeout: 55000,
|
||||
};
|
||||
return execFileSync(npmCmd, args, { ...defaults, ...options }).trim();
|
||||
}
|
||||
|
||||
module.exports = { runGsdTools, createTempDir, createTempProject, createTempGitProject, cleanup, parseFrontmatter, isUsageOutput, captureConsole, toPosixPath, runNpm, TOOLS_PATH };
|
||||
|
||||
187
tests/release-tarball-smoke-workflow.test.cjs
Normal file
187
tests/release-tarball-smoke-workflow.test.cjs
Normal file
@@ -0,0 +1,187 @@
|
||||
// allow-test-rule: source-text-is-the-product
|
||||
// The GitHub Actions YAML is the deployed runtime contract. These tests assert
|
||||
// on the parsed IR (line-tokenised YAML structure) — not on prose or rendered
|
||||
// output — to lock the wiring without testing GHA execution semantics.
|
||||
|
||||
'use strict';
|
||||
|
||||
const { describe, test } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const WORKFLOW_PATH = path.join(__dirname, '..', '.github', 'workflows', 'install-smoke.yml');
|
||||
|
||||
/**
|
||||
* Minimal structural extractor for this specific YAML shape.
|
||||
* Returns an object with:
|
||||
* { onTriggers: string[], steps: Array<{ name?: string, run?: string }> }
|
||||
*
|
||||
* Strategy:
|
||||
* 1. Collect top-level `on:` keys by scanning lines after `^on:` until the
|
||||
* next top-level key.
|
||||
* 2. Collect step `run:` blocks by scanning all ` - name:` / ` run:`
|
||||
* entries. No full YAML parse needed — we only need substring presence.
|
||||
*/
|
||||
function parseWorkflowStructure(src) {
|
||||
const lines = src.split('\n');
|
||||
|
||||
// --- Extract on: trigger keys ---
|
||||
const onTriggers = [];
|
||||
let inOn = false;
|
||||
for (const line of lines) {
|
||||
if (/^on:/.test(line)) { inOn = true; continue; }
|
||||
if (inOn) {
|
||||
// A new top-level key ends the `on:` block
|
||||
if (/^[a-zA-Z]/.test(line) && !line.startsWith(' ')) { inOn = false; continue; }
|
||||
// Direct children of `on:` are trigger names at 2-space indent
|
||||
const m = line.match(/^ ([a-zA-Z_][a-zA-Z0-9_]*):/);
|
||||
if (m) onTriggers.push(m[1]);
|
||||
}
|
||||
}
|
||||
|
||||
// --- Extract PR path filters ---
|
||||
const prPaths = [];
|
||||
let inPrPaths = false;
|
||||
let inPullRequest = false;
|
||||
for (const line of lines) {
|
||||
if (/^ pull_request:/.test(line)) { inPullRequest = true; continue; }
|
||||
if (inPullRequest) {
|
||||
if (/^ paths:/.test(line)) { inPrPaths = true; continue; }
|
||||
if (inPrPaths) {
|
||||
const m = line.match(/^ - '(.+)'/);
|
||||
if (m) { prPaths.push(m[1]); continue; }
|
||||
// End of paths list
|
||||
if (/^ [a-zA-Z]/.test(line)) inPrPaths = false;
|
||||
}
|
||||
// End of pull_request block
|
||||
if (/^ [a-zA-Z]/.test(line) && !line.startsWith(' ')) inPullRequest = false;
|
||||
}
|
||||
}
|
||||
|
||||
// --- Extract all step names + run blocks ---
|
||||
const steps = [];
|
||||
let currentStep = null;
|
||||
let inRun = false;
|
||||
let runLines = [];
|
||||
|
||||
for (const line of lines) {
|
||||
// Step boundary: 6-space "- name:" or "- uses:"
|
||||
if (/^ - name:/.test(line)) {
|
||||
if (currentStep && runLines.length) {
|
||||
currentStep.run = runLines.join('\n');
|
||||
}
|
||||
if (currentStep) steps.push(currentStep);
|
||||
currentStep = { name: line.replace(/^ - name:\s*/, '').trim() };
|
||||
inRun = false;
|
||||
runLines = [];
|
||||
continue;
|
||||
}
|
||||
if (/^ - uses:/.test(line)) {
|
||||
if (currentStep && runLines.length) {
|
||||
currentStep.run = runLines.join('\n');
|
||||
}
|
||||
if (currentStep) steps.push(currentStep);
|
||||
currentStep = { uses: line.replace(/^ - uses:\s*/, '').trim() };
|
||||
inRun = false;
|
||||
runLines = [];
|
||||
continue;
|
||||
}
|
||||
// uses: field inside a named step (e.g. "- name: Foo\n uses: actions/...")
|
||||
if (currentStep && /^ uses:\s/.test(line)) {
|
||||
currentStep.uses = line.replace(/^ uses:\s*/, '').trim();
|
||||
continue;
|
||||
}
|
||||
// run: block inside a step
|
||||
if (currentStep && /^ run:\s*\|/.test(line)) {
|
||||
inRun = true;
|
||||
runLines = [];
|
||||
continue;
|
||||
}
|
||||
if (currentStep && /^ run:\s*(?!\|)/.test(line)) {
|
||||
// Inline run (no |)
|
||||
currentStep.run = line.replace(/^ run:\s*/, '').trim();
|
||||
inRun = false;
|
||||
continue;
|
||||
}
|
||||
if (inRun) {
|
||||
// Lines deeper than 8 spaces belong to the run block
|
||||
if (/^ /.test(line) || line.trim() === '') {
|
||||
runLines.push(line);
|
||||
} else {
|
||||
inRun = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
// Flush final step
|
||||
if (currentStep) {
|
||||
if (runLines.length) currentStep.run = runLines.join('\n');
|
||||
steps.push(currentStep);
|
||||
}
|
||||
|
||||
return { onTriggers, prPaths, steps };
|
||||
}
|
||||
|
||||
describe('install-smoke.yml structural wiring', () => {
|
||||
const src = fs.readFileSync(WORKFLOW_PATH, 'utf-8');
|
||||
const { onTriggers, prPaths, steps } = parseWorkflowStructure(src);
|
||||
|
||||
test('workflow_call trigger is present (release.yml integration preserved)', () => {
|
||||
assert.ok(
|
||||
onTriggers.includes('workflow_call'),
|
||||
`Expected 'workflow_call' in on: triggers. Found: ${JSON.stringify(onTriggers)}`
|
||||
);
|
||||
});
|
||||
|
||||
test('lifecycle smoke step calls release-tarball-smoke.cjs', () => {
|
||||
const smokeStep = steps.find(s => s.run && s.run.includes('release-tarball-smoke.cjs'));
|
||||
assert.ok(
|
||||
smokeStep,
|
||||
'Expected a step whose run block includes "release-tarball-smoke.cjs". ' +
|
||||
'Steps found: ' + JSON.stringify(steps.map(s => s.name || s.uses))
|
||||
);
|
||||
});
|
||||
|
||||
test('lifecycle smoke step invokes script with --json flag', () => {
|
||||
const smokeStep = steps.find(s => s.run && s.run.includes('release-tarball-smoke.cjs'));
|
||||
assert.ok(smokeStep, 'No lifecycle smoke step found');
|
||||
assert.ok(
|
||||
smokeStep.run.includes('--json'),
|
||||
`Expected --json in lifecycle smoke run block. Got: ${smokeStep.run}`
|
||||
);
|
||||
});
|
||||
|
||||
test('lifecycle smoke result is checked via jq', () => {
|
||||
const smokeStep = steps.find(s => s.run && s.run.includes('release-tarball-smoke.cjs'));
|
||||
assert.ok(smokeStep, 'No lifecycle smoke step found');
|
||||
assert.ok(
|
||||
smokeStep.run.includes('jq'),
|
||||
`Expected jq invocation in lifecycle smoke run block. Got: ${smokeStep.run}`
|
||||
);
|
||||
});
|
||||
|
||||
test('PR path filter includes scripts/release-tarball-smoke.cjs', () => {
|
||||
assert.ok(
|
||||
prPaths.includes('scripts/release-tarball-smoke.cjs'),
|
||||
`Expected 'scripts/release-tarball-smoke.cjs' in PR paths filter. Found: ${JSON.stringify(prPaths)}`
|
||||
);
|
||||
});
|
||||
|
||||
test('PR path filter includes tests/release-tarball-smoke.install.test.cjs', () => {
|
||||
assert.ok(
|
||||
prPaths.includes('tests/release-tarball-smoke.install.test.cjs'),
|
||||
`Expected 'tests/release-tarball-smoke.install.test.cjs' in PR paths filter. Found: ${JSON.stringify(prPaths)}`
|
||||
);
|
||||
});
|
||||
|
||||
test('artifact upload step is present for failure debugging', () => {
|
||||
const uploadStep = steps.find(
|
||||
s => s.uses && s.uses.startsWith('actions/upload-artifact')
|
||||
);
|
||||
assert.ok(
|
||||
uploadStep,
|
||||
'Expected an actions/upload-artifact step for lifecycle smoke failure debugging. ' +
|
||||
'Steps (name + uses): ' + JSON.stringify(steps.map(s => ({ name: s.name, uses: s.uses })))
|
||||
);
|
||||
});
|
||||
});
|
||||
187
tests/release-tarball-smoke.install.test.cjs
Normal file
187
tests/release-tarball-smoke.install.test.cjs
Normal file
@@ -0,0 +1,187 @@
|
||||
// allow-test-rule: integration-test-input
|
||||
// The script under test (scripts/release-tarball-smoke.cjs) is the system
|
||||
// under test. We exercise it via its exported pure function, not by reading
|
||||
// source text. The tarball fixture is produced by npm pack in before().
|
||||
|
||||
'use strict';
|
||||
|
||||
process.env.GSD_TEST_MODE = '1';
|
||||
|
||||
const { describe, test, before, after } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
|
||||
const { cleanup, createTempDir, runNpm } = require('./helpers.cjs');
|
||||
const { SMOKE, runSmoke } = require('../scripts/release-tarball-smoke.cjs');
|
||||
|
||||
const PKG_PATH = path.join(__dirname, '..', 'package.json');
|
||||
const pkg = JSON.parse(fs.readFileSync(PKG_PATH, 'utf-8'));
|
||||
|
||||
describe('release-tarball-smoke', () => {
|
||||
// Shared fixture state: pack the tarball once, install it once, reuse for all tests.
|
||||
let packDir;
|
||||
let installPrefix;
|
||||
let tarballPath;
|
||||
// fixtureDir for lifecycle / init tests; created once in before(), cleaned in after().
|
||||
let fixtureDir;
|
||||
|
||||
before(async () => {
|
||||
// Pack once into a temp dir.
|
||||
packDir = createTempDir('gsd-smoke-pack-');
|
||||
installPrefix = createTempDir('gsd-smoke-prefix-');
|
||||
fixtureDir = createTempDir('gsd-smoke-fixture-');
|
||||
|
||||
const packOutput = runNpm(
|
||||
['pack', '--pack-destination', packDir],
|
||||
{ cwd: path.join(__dirname, '..') },
|
||||
);
|
||||
|
||||
// npm pack prints the filename as the last line of stdout.
|
||||
const lines = packOutput.split(/\r?\n/).filter(Boolean);
|
||||
const tgzName = lines[lines.length - 1];
|
||||
tarballPath = path.join(packDir, tgzName);
|
||||
if (!fs.existsSync(tarballPath)) {
|
||||
const found = fs.readdirSync(packDir).find((f) => f.endsWith('.tgz'));
|
||||
if (!found) throw new Error(`npm pack produced no .tgz in ${packDir}; output: ${packOutput}`);
|
||||
tarballPath = path.join(packDir, found);
|
||||
}
|
||||
|
||||
// Install once into installPrefix. All tests share this install.
|
||||
runNpm(['install', '-g', '--prefix', installPrefix, tarballPath]);
|
||||
});
|
||||
|
||||
after(() => {
|
||||
cleanup(packDir);
|
||||
cleanup(installPrefix);
|
||||
cleanup(fixtureDir);
|
||||
});
|
||||
|
||||
// ── Test A — happy path ────────────────────────────────────────────────────
|
||||
test('A: happy path — installed version matches package.json', () => {
|
||||
const result = runSmoke({
|
||||
tarballPath,
|
||||
installPrefix,
|
||||
expectedVersion: pkg.version,
|
||||
fixtureDir,
|
||||
});
|
||||
|
||||
assert.equal(result.code, SMOKE.OK);
|
||||
assert.equal(result.details.version, pkg.version);
|
||||
});
|
||||
|
||||
// ── Test B — version mismatch detected ────────────────────────────────────
|
||||
test('B: version mismatch detected — returns VERSION_MISMATCH', () => {
|
||||
const result = runSmoke({
|
||||
tarballPath,
|
||||
installPrefix,
|
||||
expectedVersion: '99.99.99',
|
||||
fixtureDir,
|
||||
});
|
||||
|
||||
assert.equal(result.code, SMOKE.VERSION_MISMATCH);
|
||||
});
|
||||
|
||||
// ── Test C — happy lifecycle ───────────────────────────────────────────────
|
||||
// Verifies that the installed package has all expected command .md files and
|
||||
// that each command resolves a workflow .md file that also exists.
|
||||
// Also verifies that `get-shit-done-cc --local --claude` (init) succeeds in
|
||||
// the fixtureDir and creates the expected .claude/ directories.
|
||||
test('C: happy lifecycle — command + workflow files resolve OK', () => {
|
||||
const result = runSmoke({
|
||||
tarballPath,
|
||||
installPrefix,
|
||||
expectedVersion: pkg.version,
|
||||
fixtureDir,
|
||||
lifecycleCommands: ['init', 'discuss-phase', 'plan-phase'],
|
||||
});
|
||||
|
||||
assert.equal(result.code, SMOKE.OK);
|
||||
|
||||
// Each non-init command must be in lifecycleResolved with both paths populated
|
||||
const resolved = result.details.lifecycleResolved;
|
||||
assert.ok(Array.isArray(resolved));
|
||||
|
||||
for (const entry of resolved) {
|
||||
assert.ok(
|
||||
typeof entry.commandPath === 'string' && entry.commandPath.length > 0,
|
||||
`expected commandPath for ${entry.command}`,
|
||||
);
|
||||
assert.ok(
|
||||
fs.existsSync(entry.commandPath) && fs.statSync(entry.commandPath).isFile(),
|
||||
`commandPath must be an existing file: ${entry.commandPath}`,
|
||||
);
|
||||
assert.ok(
|
||||
typeof entry.workflowPath === 'string' && entry.workflowPath.length > 0,
|
||||
`expected workflowPath for ${entry.command}`,
|
||||
);
|
||||
assert.ok(
|
||||
fs.existsSync(entry.workflowPath) && fs.statSync(entry.workflowPath).isFile(),
|
||||
`workflowPath must be an existing file: ${entry.workflowPath}`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
// ── Test D — missing command detected ─────────────────────────────────────
|
||||
// Passes a nonexistent command name; expects the smoke to detect the missing
|
||||
// command .md file and return COMMAND_FILE_MISSING with the right details.
|
||||
test('D: missing command detected — returns COMMAND_FILE_MISSING', () => {
|
||||
const result = runSmoke({
|
||||
tarballPath,
|
||||
installPrefix,
|
||||
expectedVersion: pkg.version,
|
||||
fixtureDir,
|
||||
lifecycleCommands: ['init', 'nonexistent-phase-xyz'],
|
||||
});
|
||||
|
||||
assert.equal(result.code, SMOKE.COMMAND_FILE_MISSING);
|
||||
assert.equal(result.details.command, 'nonexistent-phase-xyz');
|
||||
assert.ok(typeof result.details.path === 'string' && result.details.path.length > 0);
|
||||
});
|
||||
|
||||
// ── Test E — SDK binary callable ──────────────────────────────────────────
|
||||
// Verifies that `gsd-sdk query state.json` exits 0 and returns parseable JSON.
|
||||
// This is the primary guard for SDK_BINARY_NOT_CALLABLE regressions.
|
||||
test('E: sdk binary callable — gsd-sdk query produces parseable JSON', () => {
|
||||
const result = runSmoke({
|
||||
tarballPath,
|
||||
installPrefix,
|
||||
expectedVersion: pkg.version,
|
||||
fixtureDir,
|
||||
lifecycleCommands: [], // skip lifecycle checks; isolate SDK check
|
||||
});
|
||||
|
||||
// If the binary can't be called, code would be SDK_BINARY_NOT_CALLABLE
|
||||
assert.notEqual(result.code, SMOKE.SDK_BINARY_NOT_CALLABLE);
|
||||
assert.equal(result.details.sdkQueryParsed, true);
|
||||
});
|
||||
|
||||
// ── Test F — workflow-body checks run (informational) ─────────────────────
|
||||
// Asserts that the workflow-body scanning machinery ran (structural assertion).
|
||||
// Does NOT assert colonLeakCount or missingFallbackCount are zero — they will
|
||||
// be non-zero against current main per #3668 and the /gsd: leak backlog.
|
||||
// When those issues are fixed, this test continues to pass unchanged.
|
||||
test('F: workflow-body checks run — scan counts are present integers', () => {
|
||||
const result = runSmoke({
|
||||
tarballPath,
|
||||
installPrefix,
|
||||
expectedVersion: pkg.version,
|
||||
fixtureDir,
|
||||
lifecycleCommands: [],
|
||||
});
|
||||
|
||||
// Structural: the scan ran and populated the counters
|
||||
assert.ok(
|
||||
Number.isInteger(result.details.workflowsScanned) && result.details.workflowsScanned >= 1,
|
||||
`expected workflowsScanned >= 1, got ${result.details.workflowsScanned}`,
|
||||
);
|
||||
assert.ok(
|
||||
Number.isInteger(result.details.colonLeakCount),
|
||||
`expected colonLeakCount to be an integer, got ${result.details.colonLeakCount}`,
|
||||
);
|
||||
assert.ok(
|
||||
Number.isInteger(result.details.missingFallbackCount),
|
||||
`expected missingFallbackCount to be an integer, got ${result.details.missingFallbackCount}`,
|
||||
);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user