* test(#4135): regression rows for pristine regen coverage collapse RED skeleton: src/pristine-baseline.cts exports findPristineInGit as a null-returning stub (wired into verifyFile after the #4145 orphan tier, behavior-neutral) so the git-history rows fail behaviorally, not at require time. Failing-first rows: baseline_covered aggregate on a 1-of-13 multi-version fixture, coverageHeadline typed renderer, the opt-in --min-baseline-coverage gate (exit 3, >= threshold semantics, vacuous-pass and malformed-value boundaries), git-history baseline recovery (dropped-line catch + surviving-line verify + older-commit hop), findPristineInGit unit, Step 5a workflow headline contract, and the installer-side describeBaselineCoverage honest N-of-M summary with the collapse disk-state pinned. Negative-space rows pin today: non-git ok_no_baseline posture, no-match-no-adoption, #3657 drift never rescued, canonical precedence, and no git tier without --pristine-dir. * fix(#4135): headline baseline coverage, opt-in strict gate, git-history widening The #3407 promotion rule regenerates gsd-pristine/ baselines from the INCOMING release source and keeps only candidates byte-identical with the OUTGOING recorded hash — correct in isolation, but on a multi-version jump the surviving set is precisely the files upstream did NOT change. The verifier then reports ok_no_baseline (advisory, exit 0) for everything else, and no surface distinguishes a 12-of-13-unverified green run from a fully-verified one: the human summary printed Checked/Failures only, the JSON had no coverage aggregate, and the installer's update output gave per-bucket counts without N-of-M framing. All three issue directions, none exclusive: - Report coverage prominently: --json gains an additive baseline_covered aggregate; the human summary leads with 'Baseline coverage: N of M file(s)...' on every run plus an advisory section naming each skipped file and reason; the installer prints an honest covered-of-modified line via the exported describeBaselineCoverage helper (typed return, exact contract); workflow Step 5a computes and prints the headline before any pass/fail framing. - Fail louder on low coverage: opt-in --min-baseline-coverage <0..1> exits with new documented code 3 when coverage falls below the threshold (>= semantics; empty run vacuously passes; content failure exit 1 outranks it; malformed values are usage errors, exit 2). Default posture unchanged — no_baseline stays advisory per #934. - Widen the promotion rule (its only trustworthy form): when no baseline resolves under gsd-pristine/ and a hash is recorded, the verifier now recovers the baseline from the config dir's own git history — the workflow's documented Option A — anchored by the same authority every tier trusts, exact pristine_hashes sha-256 equality. Read-only (git log/git show, windowsHide per #685), bounded (100 commits/file, 10s/subprocess), null-on-any-failure so ok_no_baseline remains the universal fallback. Tier order: canonical join -> #4145 orphan scan -> git history -> OK_NO_BASELINE; #3657 drift and canonical precedence untouched. Hash validation in saveLocalPatches is NOT relaxed — the collapse is legitimate conservatism; hiding it was the bug. Measured on the issue's shape (13 files, 12 changed upstream, 1.10->1.12): non-git installs report baseline_covered 1/13 with the headline and can gate at exit 3; a git-managed config dir with the outgoing bytes in history verifies 13/13. Review fixes folded in: workflow headline derives the unverified count from checked - baseline_covered (not the drift+no_baseline sum), and the new site-scoped allow-test-rule annotation carries its ADR-456 see-ref on the marker line. Emitted-Drift-Ack-Growth: reapply-patches.md — #4135 — +20 lines / ~1.5 KB, prose and bash only: two additive parse lines (BASELINE_COVERED, CHECKED_COUNT), a Step 5a coverage-headline block printed BEFORE any pass/fail statement (documents the opt-in --min-baseline-coverage exit-3 gate), and one Option B sentence noting the verifier's read-only git-history fallback. No step ordering, gate, tool-invocation, or dispatch shape changed; 5a's fail/drift/advisory handling is unchanged, the headline only precedes it. * chore(#4135): backfill PR number into changeset fragment --------- Co-authored-by: agent-4135 <agent-4135@gsd.local>
This commit is contained in:
Tom Boucher
5
.changeset/clever-badgers-gather.md
Normal file
5
.changeset/clever-badgers-gather.md
Normal file
@@ -0,0 +1,5 @@
|
||||
---
|
||||
type: Fixed
|
||||
pr: 4376
|
||||
---
|
||||
**The reapply verifier now headlines its baseline coverage instead of reading as fully verified when most files were skipped** — after a multi-version update, /gsd-update --reapply reports 'Baseline coverage: N of M file(s)' in the verifier summary, the reapply output, and the installer's update log; on git-managed config dirs the verifier additionally recovers pristine baselines from history by recorded hash, so files upstream heavily changed are diff-verified instead of skipped; an opt-in --min-baseline-coverage <0..1> flag lets cautious operators fail the gate (exit 3) below a coverage threshold. (#4135)
|
||||
@@ -10217,6 +10217,29 @@ function recoverOrphanedPristine(pristineDir, relPath, recordedHash, canonicalSk
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #4135: honest N-of-M accounting for gsd-pristine/ baselines after an
|
||||
* update. The #3407 promotion rule keeps only hash-validated candidates
|
||||
* (byte-identical across the version span), so a multi-version update
|
||||
* legitimately ends with near-zero baselines — the collapse itself is NOT a
|
||||
* bug to hide; hiding it is. This renders the covered-of-total line the
|
||||
* update output prints either way, so "1 of 13" can never present like a
|
||||
* fully-covered run. Pure function (typed return) so tests lock the exact
|
||||
* contract without matching console prose.
|
||||
*/
|
||||
function describeBaselineCoverage(totalModified, covered) {
|
||||
const total = Math.max(0, totalModified);
|
||||
const have = Math.min(Math.max(0, covered), total);
|
||||
const uncovered = total - have;
|
||||
return {
|
||||
complete: uncovered === 0,
|
||||
uncovered,
|
||||
text: uncovered === 0
|
||||
? `gsd-pristine/ baselines cover ${have} of ${total} modified file(s)`
|
||||
: `gsd-pristine/ baselines cover ${have} of ${total} modified file(s) — ${uncovered} will be reported no_baseline by the reapply verifier`,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect user-modified GSD files by comparing against install manifest.
|
||||
* Backs up modified files to gsd-local-patches/ for reapply after update.
|
||||
@@ -10467,6 +10490,17 @@ function saveLocalPatches(configDir, pristineCtx) {
|
||||
if (removed > 0) {
|
||||
console.log(' ' + yellow + 'i' + reset + ' Removed ' + removed + ' stale gsd-pristine/ snapshot(s); regenerated ' + regenerated + ' of those — falls back to over-broad verify heuristic for the rest');
|
||||
}
|
||||
// #4135: the honest N-of-M coverage line. Preserved/rescued/regenerated
|
||||
// are disjoint buckets (see their accounting comments above), so their
|
||||
// sum is exactly the files that ended this update with a hash-valid
|
||||
// baseline. A partial result renders as an info line, not an error:
|
||||
// the collapse is legitimate (#3407), hiding it was the bug.
|
||||
const coverage = describeBaselineCoverage(modified.length, preserved + rescued + regenerated);
|
||||
if (coverage.complete) {
|
||||
console.log(' ' + green + '✓' + reset + ' ' + coverage.text);
|
||||
} else {
|
||||
console.log(' ' + yellow + 'i' + reset + ' ' + coverage.text);
|
||||
}
|
||||
}
|
||||
}
|
||||
return modified;
|
||||
@@ -14209,6 +14243,7 @@ module.exports = {
|
||||
reportLocalPatches,
|
||||
validateHookFields,
|
||||
populatePristineDir,
|
||||
describeBaselineCoverage,
|
||||
_resolveUserArtifactStagingRoot,
|
||||
_tryResolveUserArtifactStagingRoot,
|
||||
finishInstall,
|
||||
|
||||
@@ -20,11 +20,21 @@
|
||||
* [--classify] # pre-merge mode: classify each backed-up
|
||||
* # file as incorporated / needs_merge /
|
||||
* # unknown (#4136); always exits 0
|
||||
* [--min-baseline-coverage <0..1>] # OPT-IN strict gate (#4135): exit 3
|
||||
* # when the fraction of files verified
|
||||
* # against a resolved pristine baseline
|
||||
* # (baseline_covered / checked) falls below
|
||||
* # the threshold. Default: off — a
|
||||
* # low-coverage run stays green per the
|
||||
* # documented #934 advisory posture, but
|
||||
* # its coverage is ALWAYS headline-reported.
|
||||
*
|
||||
* Exit codes (default gate mode):
|
||||
* 0 — every user-added line is present in the merged file (gate passes)
|
||||
* 1 — at least one missing line in at least one file (gate fails)
|
||||
* 1 — at least one missing line in at least one file (gate fails); outranks
|
||||
* a coverage-gate failure when both apply
|
||||
* 2 — usage / structural error (e.g. patches dir missing)
|
||||
* 3 — opt-in coverage gate failed (--min-baseline-coverage not met; #4135)
|
||||
*
|
||||
* Bug #2969: the Step 5 gate previously trusted Claude's free-text "verified:
|
||||
* yes/no" reporting per hunk. The LLM was filling in `yes` even when content
|
||||
@@ -49,13 +59,13 @@ const { ExitError, runMain } = require('./lib/cli-exit.cjs');
|
||||
// path under gsd-pristine/ (e.g. without the gsd-core/ prefix an earlier
|
||||
// release's writer dropped). Same module the installer's preserve-check uses,
|
||||
// so the two readers cannot drift apart again.
|
||||
const { findPristineByHash } = require('./lib/pristine-baseline.cjs');
|
||||
const { findPristineByHash, findPristineInGit } = require('./lib/pristine-baseline.cjs');
|
||||
|
||||
const SIGNIFICANT_MIN_CHARS = 12;
|
||||
const GSD_HOOK_VERSION_LINE_RE = /^(?:\/\/|#)\s*gsd-hook-version:\s*\S+\s*$/i;
|
||||
|
||||
function parseArgs(argv) {
|
||||
const opts = { patchesDir: null, configDir: null, pristineDir: null, json: false, classify: false };
|
||||
const opts = { patchesDir: null, configDir: null, pristineDir: null, json: false, classify: false, minBaselineCoverage: null };
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const arg = argv[i];
|
||||
if (arg === '--patches-dir') opts.patchesDir = argv[++i];
|
||||
@@ -63,9 +73,19 @@ function parseArgs(argv) {
|
||||
else if (arg === '--pristine-dir') opts.pristineDir = argv[++i];
|
||||
else if (arg === '--json') opts.json = true;
|
||||
else if (arg === '--classify') opts.classify = true;
|
||||
else if (arg === '--help' || arg === '-h') {
|
||||
else if (arg === '--min-baseline-coverage') {
|
||||
const raw = argv[++i];
|
||||
if (raw === undefined) {
|
||||
throw new ExitError(2, '--min-baseline-coverage requires a value between 0 and 1');
|
||||
}
|
||||
const value = Number(raw);
|
||||
if (!Number.isFinite(value) || value < 0 || value > 1) {
|
||||
throw new ExitError(2, `--min-baseline-coverage must be a number between 0 and 1, got: ${raw}`);
|
||||
}
|
||||
opts.minBaselineCoverage = value;
|
||||
} else if (arg === '--help' || arg === '-h') {
|
||||
process.stdout.write(
|
||||
'usage: verify-reapply-patches.cjs --patches-dir <path> --config-dir <path> [--pristine-dir <path>] [--json] [--classify]\n',
|
||||
'usage: verify-reapply-patches.cjs --patches-dir <path> --config-dir <path> [--pristine-dir <path>] [--json] [--classify] [--min-baseline-coverage <0..1>]\n',
|
||||
);
|
||||
throw new ExitError(0);
|
||||
} else {
|
||||
@@ -253,7 +273,7 @@ const PRISTINE_RESOLUTION = Object.freeze({
|
||||
* guard. Extracted from verifyFile's inline block so --classify reasons over
|
||||
* the exact same baseline semantics the post-merge gate enforces.
|
||||
*/
|
||||
function resolvePristineBaseline({ relPath, pristineDir, pristineHashes }) {
|
||||
function resolvePristineBaseline({ relPath, configDir, pristineDir, pristineHashes }) {
|
||||
const hashKey = relPath.replace(/\\/g, '/');
|
||||
const recordedHash = pristineHashes && pristineHashes[hashKey];
|
||||
if (pristineDir) {
|
||||
@@ -314,6 +334,27 @@ function resolvePristineBaseline({ relPath, pristineDir, pristineHashes }) {
|
||||
// Present but not a regular file — over-broad mode is the safe side.
|
||||
return { resolution: PRISTINE_RESOLUTION.OVERBROAD, content: null };
|
||||
}
|
||||
|
||||
// Bug #4135: still nothing under gsd-pristine/ — the multi-version
|
||||
// regeneration collapse (the #3407 promotion rule only keeps files
|
||||
// byte-identical across the WHOLE version span, so the surviving set is
|
||||
// precisely the files upstream did not change). When the config dir is
|
||||
// itself a git repository, its history may hold the outgoing bytes: this
|
||||
// is the workflow's documented Option A (reapply-patches.md Step 2),
|
||||
// anchored by the SAME authority every other tier trusts — exact
|
||||
// pristine_hashes equality. Read-only (git log / git show); any failure
|
||||
// (no git, not a repo, no matching blob) degrades to ABSENT_RECORDED.
|
||||
// A recovered baseline is hash-confirmed by construction, so it VALIDATES.
|
||||
if (!pristinePathExists && recordedHash) {
|
||||
try {
|
||||
const fromGit = findPristineInGit(configDir, hashKey, recordedHash);
|
||||
if (fromGit !== null) {
|
||||
return { resolution: PRISTINE_RESOLUTION.VALIDATED, content: fromGit };
|
||||
}
|
||||
} catch {
|
||||
// git unavailable or history walk failed — ABSENT_RECORDED posture
|
||||
}
|
||||
}
|
||||
// Bug #934: recordedHash is present (modern installer) but no
|
||||
// hash-matching pristine exists anywhere under gsd-pristine/ (the stat
|
||||
// missed and the #4145 recovery found nothing).
|
||||
@@ -387,6 +428,47 @@ function resolveSkillsRedirect(configDir) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #4135: baseline-coverage accounting. A file counts as baseline-covered
|
||||
* when its diff was actually computed against a resolved pristine baseline,
|
||||
* i.e. its reason is one of the baseline-diff outcomes (null = lines
|
||||
* verified present, OK_NO_USER_LINES_VS_PRISTINE = no user lines versus
|
||||
* the baseline, FAIL_USER_LINES_MISSING = lines missing). Everything else
|
||||
* is uncovered: the silent skips (no baseline anywhere per #934/#4135,
|
||||
* drift per #3657, no-pristine over-broad with nothing significant) AND
|
||||
* the structural failures (installed missing / not a file / read error),
|
||||
* which never reached a baseline — they are loud blocking failures, but
|
||||
* they were not baseline-verified either, and this aggregate exists to
|
||||
* state exactly how much of the run the gate could reason about. When
|
||||
* --pristine-dir is absent the whole run is over-broad (#2998 fallback):
|
||||
* no file was verified against a baseline, covered is 0 by construction.
|
||||
*/
|
||||
const BASELINE_VERIFIED_REASONS = new Set([
|
||||
REASON.OK_NO_USER_LINES_VS_PRISTINE,
|
||||
REASON.FAIL_USER_LINES_MISSING,
|
||||
]);
|
||||
|
||||
function classifyBaselineCoverage(results, pristineDirProvided) {
|
||||
if (!pristineDirProvided) return 0;
|
||||
let covered = 0;
|
||||
for (const r of results) {
|
||||
if (r.reason === null || BASELINE_VERIFIED_REASONS.has(r.reason)) covered++;
|
||||
}
|
||||
return covered;
|
||||
}
|
||||
|
||||
/**
|
||||
* #4135: the headline string for the human summary. A run with 12 of 13
|
||||
* files unbaselined must not present the same way as a fully-verified one —
|
||||
* the N-of-M form is the issue's own framing, and the unverified tail is
|
||||
* only rendered when it is non-zero.
|
||||
*/
|
||||
function coverageHeadline(covered, total) {
|
||||
const unverified = Math.max(0, total - covered);
|
||||
const base = `Baseline coverage: ${covered} of ${total} file(s) verified against a pristine baseline`;
|
||||
return unverified > 0 ? `${base} (${unverified} unverified)` : base;
|
||||
}
|
||||
|
||||
function verifyFile({ relPath, patchesDir, configDir, pristineDir, pristineHashes, skillsRedirect }) {
|
||||
const backupPath = path.join(patchesDir, relPath);
|
||||
const installedPath = resolveInstalledPath(configDir, relPath, skillsRedirect);
|
||||
@@ -431,7 +513,7 @@ function verifyFile({ relPath, patchesDir, configDir, pristineDir, pristineHashe
|
||||
// hash-first recovery + #934 absent guard) moved into resolvePristineBaseline
|
||||
// so the pre-merge classifier reasons over the exact same semantics.
|
||||
const { resolution, content: pristineContent } =
|
||||
resolvePristineBaseline({ relPath, pristineDir, pristineHashes });
|
||||
resolvePristineBaseline({ relPath, configDir, pristineDir, pristineHashes });
|
||||
|
||||
// Bug #3657: the resolved snapshot hash-mismatches the recorded baseline.
|
||||
// Skip the file with a diagnostic code rather than diffing against the
|
||||
@@ -531,7 +613,7 @@ function classifyFile({ relPath, patchesDir, configDir, pristineDir, pristineHas
|
||||
}
|
||||
|
||||
const { resolution, content: pristineContent } =
|
||||
resolvePristineBaseline({ relPath, pristineDir, pristineHashes });
|
||||
resolvePristineBaseline({ relPath, configDir, pristineDir, pristineHashes });
|
||||
|
||||
if (resolution === PRISTINE_RESOLUTION.DRIFTED) {
|
||||
result.reason = REASON.OK_PRISTINE_DRIFT_DETECTED;
|
||||
@@ -669,14 +751,52 @@ function main() {
|
||||
const no_baseline = noBaselineResults.length;
|
||||
const no_baseline_files = noBaselineResults.map((r) => r.file);
|
||||
|
||||
// Bug #4135: baseline coverage is a first-class aggregate. The collapse
|
||||
// this reports is silent by design in every other surface (no_baseline is
|
||||
// advisory, exit stays 0), which is exactly why it must be counted here:
|
||||
// "A clean verifier exit reads as 'hunks survived'. On this run it meant
|
||||
// '12 of 13 files were not checked at all.'"
|
||||
const baseline_covered = classifyBaselineCoverage(results, Boolean(opts.pristineDir));
|
||||
|
||||
// Bug #4135: the opt-in strict coverage gate. Threshold semantics are >=
|
||||
// (a run exactly at the threshold passes); an empty run is vacuously
|
||||
// covered (nothing checked cannot be under-covered). A content failure
|
||||
// (exit 1) outranks a coverage failure — it is the louder, more specific
|
||||
// signal.
|
||||
let coverageGateFailed = false;
|
||||
if (opts.minBaselineCoverage !== null) {
|
||||
const ratio = results.length === 0 ? 1 : baseline_covered / results.length;
|
||||
coverageGateFailed = ratio < opts.minBaselineCoverage;
|
||||
}
|
||||
|
||||
if (opts.json) {
|
||||
process.stdout.write(
|
||||
JSON.stringify({ checked: results.length, failures: failures.length, drifted, drifted_files, no_baseline, no_baseline_files, results }, null, 2) + '\n',
|
||||
JSON.stringify({ checked: results.length, failures: failures.length, drifted, drifted_files, no_baseline, no_baseline_files, baseline_covered, results }, null, 2) + '\n',
|
||||
);
|
||||
} else {
|
||||
process.stdout.write(`# Hunk Verification Gate (#2969)\n\n`);
|
||||
process.stdout.write(`Checked: ${results.length} file(s)\n`);
|
||||
process.stdout.write(`Failures: ${failures.length}\n\n`);
|
||||
process.stdout.write(`Failures: ${failures.length}\n`);
|
||||
// #4135: the coverage headline is printed on EVERY run, before any
|
||||
// per-file detail, so a near-zero-coverage green run can never render
|
||||
// identically to a fully-verified one.
|
||||
process.stdout.write(`${coverageHeadline(baseline_covered, results.length)}\n\n`);
|
||||
if (baseline_covered < results.length) {
|
||||
if (!opts.pristineDir) {
|
||||
process.stdout.write(`No --pristine-dir: over-broad fallback — nothing was verified against a pristine baseline (#2998).\n\n`);
|
||||
} else {
|
||||
const skipped = results.filter((r) => r.reason !== null && !BASELINE_VERIFIED_REASONS.has(r.reason));
|
||||
process.stdout.write(`## Files not verified against a pristine baseline\n\n`);
|
||||
process.stdout.write(`Advisory (non-blocking): their user customizations may or may not have survived the merge.\n\n`);
|
||||
for (const r of skipped) {
|
||||
process.stdout.write(`- ${r.file} (${r.reason})\n`);
|
||||
}
|
||||
process.stdout.write('\n');
|
||||
}
|
||||
}
|
||||
if (coverageGateFailed) {
|
||||
process.stdout.write(`COVERAGE GATE FAILED: baseline coverage ${(opts.minBaselineCoverage * 100).toFixed(1)}% required, ${results.length === 0 ? 100 : ((baseline_covered / results.length) * 100).toFixed(1)}% achieved (#4135 strict mode).\n\n`);
|
||||
}
|
||||
if (failures.length > 0) {
|
||||
process.stdout.write(`## Files with missing user-added content\n\n`);
|
||||
for (const r of failures) {
|
||||
@@ -692,11 +812,14 @@ function main() {
|
||||
}
|
||||
}
|
||||
|
||||
return failures.length > 0 ? 1 : 0;
|
||||
if (failures.length > 0) return 1;
|
||||
if (coverageGateFailed) return 3;
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
runMain(main);
|
||||
}
|
||||
|
||||
module.exports = { computeUserAddedLines, isSignificantLine, verifyFile, classifyFile, walk, REASON, CLASSIFICATION, PRISTINE_RESOLUTION, resolvePristineBaseline, readPristineHashes, sha256, resolveInstalledPath, resolveSkillsRedirect };
|
||||
module.exports = { computeUserAddedLines, isSignificantLine, verifyFile, classifyFile, walk, REASON, CLASSIFICATION, PRISTINE_RESOLUTION, resolvePristineBaseline, readPristineHashes, sha256, resolveInstalledPath, resolveSkillsRedirect, coverageHeadline, classifyBaselineCoverage };
|
||||
|
||||
|
||||
@@ -169,7 +169,7 @@ Check if a `gsd-pristine/` directory exists alongside `gsd-local-patches/`:
|
||||
```bash
|
||||
PRISTINE_DIR="$CONFIG_DIR/gsd-pristine"
|
||||
```
|
||||
If it exists, the installer saved pristine copies at install time. Use these as the baseline. Both the deterministic verifier and the installer's preserve-check resolve each file's snapshot at its canonical path first and, when that misses, by the SHA-256 recorded in `pristine_hashes` — so a snapshot stored at a legacy path (for example, without the `gsd-core/` prefix an earlier release dropped) is still found and, on the next update, relocated to its canonical path (#4145).
|
||||
If it exists, the installer saved pristine copies at install time. Use these as the baseline. Both the deterministic verifier and the installer's preserve-check resolve each file's snapshot at its canonical path first and, when that misses, by the SHA-256 recorded in `pristine_hashes` — so a snapshot stored at a legacy path (for example, without the `gsd-core/` prefix an earlier release dropped) is still found and, on the next update, relocated to its canonical path (#4145). When no snapshot resolves under `gsd-pristine/`, the deterministic verifier additionally falls back to Option A's git-history walk itself (read-only, same recorded-hash match), which is what keeps Step 5a coverage non-zero on multi-version updates where the hash-validated regeneration had nothing it could promote (#4135).
|
||||
|
||||
### Option C: No baseline available (two-way fallback)
|
||||
If neither git history nor pristine snapshots are available, fall back to two-way comparison — but with **strengthened heuristics** (see Step 3).
|
||||
@@ -362,9 +362,27 @@ VERIFY_STATUS=$?
|
||||
DRIFTED_COUNT="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));process.stdout.write(String(d.drifted||0))")"
|
||||
DRIFTED_FILES="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));(d.drifted_files||[]).forEach(f=>process.stdout.write(f+'\n'))")"
|
||||
NO_BASELINE_COUNT="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));process.stdout.write(String(d.no_baseline||0))")"
|
||||
NO_BASELINE_FILES="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));(d.no_baseline_files||[]).forEach(f=>process.stdout.write(f+'\n'))")"
|
||||
NO_BASELINE_FILES="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));(d.no_baseline_files||[]).forEach(f=>process.stdout.write(f+'\n'))")"
|
||||
BASELINE_COVERED="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));process.stdout.write(String(d.baseline_covered||0))")"
|
||||
CHECKED_COUNT="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));process.stdout.write(String(d.checked||0))")"
|
||||
```
|
||||
|
||||
**Baseline coverage headline (#4135)** — BEFORE any pass/fail statement, print the coverage
|
||||
N-of-M line. A run where 12 of 13 files were not diff-verified must never present the same way
|
||||
as a fully-verified one:
|
||||
|
||||
```text
|
||||
Baseline coverage: {BASELINE_COVERED} of {CHECKED_COUNT} file(s) verified against a pristine
|
||||
baseline ({CHECKED_COUNT - BASELINE_COVERED} unverified)
|
||||
```
|
||||
|
||||
The verifier also reports this on every run (JSON field `baseline_covered`; the human summary
|
||||
leads with the same line). Operators who want the gate itself to fail on low coverage (cautious
|
||||
environments) pass the opt-in strict flag when invoking the verifier:
|
||||
`--min-baseline-coverage <fraction between 0 and 1>` — below the threshold the verifier exits
|
||||
with code 3 (a real content failure still exits 1 and outranks it). Without the flag the
|
||||
default advisory posture is unchanged.
|
||||
|
||||
**If `NO_BASELINE_COUNT` is greater than 0**, emit an advisory warning (non-blocking — the gate still exits 0 for these files). Do NOT halt:
|
||||
|
||||
```text
|
||||
|
||||
@@ -24,6 +24,7 @@
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import crypto from 'node:crypto';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
|
||||
/**
|
||||
* SHA-256 hex digest of a file's raw bytes. Byte-for-byte the same digest
|
||||
@@ -92,3 +93,90 @@ export function findPristineByHash(
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* #4135: recover a pristine baseline from the config dir's OWN git history,
|
||||
* anchored by the recorded pristine_hashes entry.
|
||||
*
|
||||
* The #3407 promotion rule keeps only regeneration candidates byte-identical
|
||||
* across the whole version span, so a multi-version update leaves
|
||||
* gsd-pristine/ holding exactly the files upstream did NOT change — near-zero
|
||||
* coverage precisely where upstream churned the most. On a git-managed config
|
||||
* dir the outgoing bytes often still exist in history (the workflow's
|
||||
* documented Option A), and pristine_hashes is the same authority every other
|
||||
* resolution tier trusts: a blob whose SHA-256 equals the recorded hash cannot
|
||||
* be the wrong baseline. This is read-only recovery (git log / git show only).
|
||||
*
|
||||
* Guarantees:
|
||||
* - Only an EXACT sha-256 match with the recorded hash is ever returned.
|
||||
* - Newest-first commit order (git log default) makes multi-match resolution
|
||||
* deterministic; byte-identical matches are interchangeable anyway.
|
||||
* - Any failure (git absent, not a repository, empty history, unreadable
|
||||
* blob, subprocess timeout) yields null — never a throw — so the caller's
|
||||
* OK_NO_BASELINE posture is the universal fallback.
|
||||
* - The walk is bounded: at most GIT_MAX_COMMITS_PER_FILE commits per file.
|
||||
*/
|
||||
const GIT_MAX_COMMITS_PER_FILE = 100;
|
||||
/** Per-subprocess bound in ms — an unbounded git call is an indefinite hang. */
|
||||
const GIT_SUBPROCESS_TIMEOUT_MS = 10_000;
|
||||
/** git log --format=%H output cap; 100 full shas are ~4 KB, this is headroom. */
|
||||
const GIT_MAX_BUFFER_BYTES = 16 * 1024 * 1024;
|
||||
|
||||
function isCleanRelativePosixPath(relPath: string): boolean {
|
||||
if (!relPath || relPath.startsWith('/') || relPath.includes('\\') || relPath.includes('\0')) {
|
||||
return false;
|
||||
}
|
||||
const segments = relPath.split('/');
|
||||
return segments.every((seg) => seg.length > 0 && seg !== '.' && seg !== '..');
|
||||
}
|
||||
|
||||
function gitExec(gitDir: string, args: string[]): string {
|
||||
return execFileSync('git', args, {
|
||||
cwd: gitDir,
|
||||
encoding: 'utf8',
|
||||
timeout: GIT_SUBPROCESS_TIMEOUT_MS,
|
||||
maxBuffer: GIT_MAX_BUFFER_BYTES,
|
||||
// windowsHide (#685): a console-window flash per git call would spam the
|
||||
// user on Windows for what is a background, read-only history walk.
|
||||
windowsHide: true,
|
||||
// stderr is discarded: "file absent in commit" is an expected walk outcome,
|
||||
// not operator-visible diagnostics.
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
});
|
||||
}
|
||||
|
||||
export function findPristineInGit(
|
||||
gitDir: string,
|
||||
relPath: string,
|
||||
recordedHash: string,
|
||||
): string | null {
|
||||
if (!gitDir || typeof relPath !== 'string' || typeof recordedHash !== 'string'
|
||||
|| recordedHash.length === 0 || !isCleanRelativePosixPath(relPath)) {
|
||||
return null;
|
||||
}
|
||||
let commits: string[];
|
||||
try {
|
||||
const logOutput = gitExec(gitDir, ['log', '--format=%H', '--', relPath]).trim();
|
||||
if (!logOutput) return null;
|
||||
commits = logOutput.split('\n').slice(0, GIT_MAX_COMMITS_PER_FILE);
|
||||
} catch {
|
||||
return null; // git absent, not a repository, or the walk failed
|
||||
}
|
||||
for (const commit of commits) {
|
||||
if (!/^[0-9a-f]{40}$/i.test(commit)) continue;
|
||||
try {
|
||||
const blob = gitExec(gitDir, ['show', `${commit}:${relPath}`]);
|
||||
if (sha256String(blob) === recordedHash) {
|
||||
return blob;
|
||||
}
|
||||
} catch {
|
||||
// blob absent in this commit (rename/add boundary) — keep walking
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** sha256 of a utf8 string, matching how manifest hashes are recorded. */
|
||||
function sha256String(content: string): string {
|
||||
return crypto.createHash('sha256').update(content, 'utf8').digest('hex');
|
||||
}
|
||||
|
||||
@@ -4237,3 +4237,144 @@ describe('Bug #4145: saveLocalPatches rescues hash-matching orphaned pristine sn
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Folded regression block — #4135 (installer side). saveLocalPatches'
|
||||
// hash-validated regeneration (the #3407 promotion rule) keeps only
|
||||
// candidates byte-identical across the whole version span, so a
|
||||
// multi-version update leaves gsd-pristine/ holding near-zero baselines —
|
||||
// and the update output never says so in N-of-M terms. The fix keeps the
|
||||
// hash validation untouched (disk behavior is pinned here) and adds an
|
||||
// honest coverage summary via an exported typed helper.
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
{
|
||||
const { describe: __foldDescribe } = require('node:test');
|
||||
__foldDescribe('folded:bug-4135-saveLocalPatches-coverage-line', () => {
|
||||
'use strict';
|
||||
|
||||
process.env.GSD_TEST_MODE = '1';
|
||||
|
||||
const { test, describe, beforeEach } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const os = require('node:os');
|
||||
const crypto = require('node:crypto');
|
||||
|
||||
const ROOT = path.join(__dirname, '..');
|
||||
const INSTALL = require(path.join(ROOT, 'bin', 'install.js'));
|
||||
const { cleanup } = require('./helpers.cjs');
|
||||
|
||||
const MANIFEST_NAME = 'gsd-file-manifest.json';
|
||||
|
||||
function sha256(content) {
|
||||
return crypto.createHash('sha256').update(content, 'utf8').digest('hex');
|
||||
}
|
||||
|
||||
function countFiles(dir) {
|
||||
let n = 0;
|
||||
if (!fs.existsSync(dir)) return 0;
|
||||
for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
|
||||
if (entry.isDirectory()) n += countFiles(path.join(dir, entry.name));
|
||||
else if (entry.isFile()) n += 1;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
describe('Bug #4135: saveLocalPatches reports honest gsd-pristine coverage on multi-version updates', () => {
|
||||
let tmpDir;
|
||||
let configDir;
|
||||
let newSrcDir;
|
||||
let pristineDir;
|
||||
|
||||
beforeEach((t) => {
|
||||
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-4135-cov-'));
|
||||
configDir = path.join(tmpDir, 'config');
|
||||
newSrcDir = path.join(tmpDir, 'new-release-src');
|
||||
pristineDir = path.join(configDir, 'gsd-pristine');
|
||||
fs.mkdirSync(configDir, { recursive: true });
|
||||
fs.mkdirSync(newSrcDir, { recursive: true });
|
||||
t.after(() => {
|
||||
cleanup(tmpDir);
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* Seeds a multi-version-update fixture: `total` modified files, of which
|
||||
* `identical` are byte-identical across the span (the only regenerable
|
||||
* baselines) and the rest changed upstream in the incoming source.
|
||||
*/
|
||||
function seedMultiVersionFixture(total, identical) {
|
||||
const manifestFiles = {};
|
||||
for (let i = 1; i <= total; i++) {
|
||||
const rel = `gsd-core/workflows/flow-${String(i).padStart(2, '0')}.md`;
|
||||
const pristine =
|
||||
`# Flow ${i}\nStock content of the outgoing release for file ${i}.\n` +
|
||||
`Second outgoing stock line ${i} with plenty of substance.\n`;
|
||||
const user = pristine + `## User customisation ${i}\nCustom section on top of the outgoing release.\n`;
|
||||
const incoming = (i <= identical)
|
||||
? pristine
|
||||
: `# Flow ${i} (rewritten)\nIncoming release rewrote file ${i} across the span.\n`;
|
||||
manifestFiles[rel] = sha256(pristine);
|
||||
fs.mkdirSync(path.dirname(path.join(configDir, rel)), { recursive: true });
|
||||
fs.writeFileSync(path.join(configDir, rel), user);
|
||||
fs.mkdirSync(path.dirname(path.join(newSrcDir, rel)), { recursive: true });
|
||||
fs.writeFileSync(path.join(newSrcDir, rel), incoming);
|
||||
}
|
||||
fs.writeFileSync(
|
||||
path.join(configDir, MANIFEST_NAME),
|
||||
JSON.stringify({ version: '1.10.0', timestamp: '2026-08-01T00:00:00Z', runtime: 'claude', scope: 'global', files: manifestFiles }, null, 2),
|
||||
);
|
||||
return total;
|
||||
}
|
||||
|
||||
/**
|
||||
* Core installer regression: the multi-version collapse itself is pinned
|
||||
* (hash validation untouched — only the byte-identical file survives),
|
||||
* and the honest N-of-M summary is available via the typed helper.
|
||||
*/
|
||||
test('#4135: saveLocalPatches multi-version regen keeps hash validation and reports 1-of-13 coverage', () => {
|
||||
const total = seedMultiVersionFixture(13, 1);
|
||||
|
||||
INSTALL.saveLocalPatches(configDir, {
|
||||
packageSrc: newSrcDir, runtime: 'claude', pathPrefix: '$HOME/.claude/', isGlobal: true,
|
||||
});
|
||||
|
||||
const covered = countFiles(pristineDir);
|
||||
assert.equal(covered, 1,
|
||||
'the collapse is pinned: only the byte-identical file survives hash-validated regeneration');
|
||||
assert.equal(typeof INSTALL.describeBaselineCoverage, 'function',
|
||||
'the coverage summary must be rendered by an exported typed helper');
|
||||
const summary = INSTALL.describeBaselineCoverage(total, covered);
|
||||
assert.equal(summary.complete, false);
|
||||
assert.equal(summary.uncovered, 12);
|
||||
assert.equal(
|
||||
summary.text,
|
||||
'gsd-pristine/ baselines cover 1 of 13 modified file(s) — 12 will be reported no_baseline by the reapply verifier',
|
||||
'partial coverage states the N-of-M collapse and its downstream effect',
|
||||
);
|
||||
});
|
||||
|
||||
/** Positive boundary: every modified file covered renders a complete summary. */
|
||||
test('#4135: describeBaselineCoverage reports complete when every modified file is covered', () => {
|
||||
const total = seedMultiVersionFixture(3, 3);
|
||||
|
||||
INSTALL.saveLocalPatches(configDir, {
|
||||
packageSrc: newSrcDir, runtime: 'claude', pathPrefix: '$HOME/.claude/', isGlobal: true,
|
||||
});
|
||||
|
||||
const covered = countFiles(pristineDir);
|
||||
assert.equal(covered, 3, 'a fully byte-identical span regenerates every baseline');
|
||||
const summary = INSTALL.describeBaselineCoverage(total, covered);
|
||||
assert.equal(summary.complete, true);
|
||||
assert.equal(summary.uncovered, 0);
|
||||
assert.equal(
|
||||
summary.text,
|
||||
'gsd-pristine/ baselines cover 3 of 3 modified file(s)',
|
||||
'complete coverage renders without a collapse tail',
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
@@ -305,7 +305,8 @@ describe('Bug #2969: deterministic Step 5 verification gate', () => {
|
||||
// Bug #3657 (Finding 1): drifted + drifted_files are additive fields added to surface
|
||||
// pristine-drift skips distinctly from failures. Shape-lock updated to include them.
|
||||
// Bug #934: no_baseline + no_baseline_files are additive fields for missing-pristine advisory.
|
||||
assert.deepEqual(Object.keys(report).sort(), ['checked', 'drifted', 'drifted_files', 'failures', 'no_baseline', 'no_baseline_files', 'results']);
|
||||
// Bug #4135: baseline_covered is the additive coverage aggregate (headline reporting).
|
||||
assert.deepEqual(Object.keys(report).sort(), ['baseline_covered', 'checked', 'drifted', 'drifted_files', 'failures', 'no_baseline', 'no_baseline_files', 'results']);
|
||||
const r0 = report.results[0];
|
||||
assert.deepEqual(Object.keys(r0).sort(), ['file', 'missing', 'reason', 'status']);
|
||||
assert.equal(typeof r0.file, 'string');
|
||||
@@ -1985,3 +1986,510 @@ describe('Bug #4136: workflow consumes the classifier (contract rows)', () => {
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
// Folded regression block — #4135 (gsd-pristine/ regeneration yields
|
||||
// near-zero coverage on a multi-version update). The #3407 promotion rule
|
||||
// only keeps regeneration candidates byte-identical across the WHOLE
|
||||
// version span, so the surviving baseline set is precisely the files
|
||||
// upstream did NOT change — and no reporting surface (verifier summary,
|
||||
// --json, workflow Step 5a) distinguishes a 12-of-13 unbaselined green run
|
||||
// from a fully-verified one. The fix reports baseline coverage as a
|
||||
// headline, adds an opt-in strict coverage gate, and widens resolution
|
||||
// with a git-history tier anchored by the same pristine_hashes authority.
|
||||
// ────────────────────────────────────────────────────────────────────────
|
||||
{
|
||||
const { describe: __foldDescribe } = require('node:test');
|
||||
__foldDescribe('folded:bug-4135-pristine-regen-coverage', () => {
|
||||
'use strict';
|
||||
|
||||
process.env.GSD_TEST_MODE = '1';
|
||||
|
||||
const { test, describe, before, after } = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const crypto = require('node:crypto');
|
||||
const { cleanup } = require('./helpers.cjs');
|
||||
const { runNode } = require('./helpers/process-seam.cjs');
|
||||
const { gitOrThrow } = require('./helpers/git-fixture.cjs');
|
||||
|
||||
const ROOT = path.join(__dirname, '..');
|
||||
const SCRIPT = path.join(ROOT, 'gsd-core', 'bin', 'verify-reapply-patches.cjs');
|
||||
const { REASON } = require(SCRIPT);
|
||||
const { findPristineInGit } = require(
|
||||
path.join(ROOT, 'gsd-core', 'bin', 'lib', 'pristine-baseline.cjs'),
|
||||
);
|
||||
const WORKFLOW_PATH = path.join(ROOT, 'gsd-core', 'workflows', 'reapply-patches.md');
|
||||
|
||||
// 30000ms: same class as the folded blocks above — one deterministic
|
||||
// verifier pass (plus, for the git tier rows, the in-process git log/show
|
||||
// walk the verifier performs) over a small mkdtemp fixture tree.
|
||||
const VERIFIER_TIMEOUT_MS = 30_000;
|
||||
|
||||
let tmpRoot;
|
||||
let patchesDir;
|
||||
let configDir;
|
||||
let pristineDir;
|
||||
|
||||
function sha256(content) {
|
||||
return crypto.createHash('sha256').update(content, 'utf8').digest('hex');
|
||||
}
|
||||
|
||||
function writeFile(absPath, content) {
|
||||
fs.mkdirSync(path.dirname(absPath), { recursive: true });
|
||||
fs.writeFileSync(absPath, content);
|
||||
}
|
||||
|
||||
function writeBackupMeta(pristine_hashes) {
|
||||
writeFile(path.join(patchesDir, 'backup-meta.json'), JSON.stringify({ pristine_hashes }, null, 2));
|
||||
}
|
||||
|
||||
function resetFixture() {
|
||||
for (const dir of [patchesDir, configDir, pristineDir]) {
|
||||
cleanup(dir);
|
||||
}
|
||||
fs.mkdirSync(patchesDir);
|
||||
fs.mkdirSync(configDir);
|
||||
fs.mkdirSync(pristineDir);
|
||||
}
|
||||
|
||||
/** Runs the verifier with --json plus any extra argv (strict-gate flags). */
|
||||
function runVerifier(extraArgs = []) {
|
||||
const r = runNode([
|
||||
SCRIPT,
|
||||
'--patches-dir', patchesDir,
|
||||
'--config-dir', configDir,
|
||||
'--pristine-dir', pristineDir,
|
||||
'--json',
|
||||
...extraArgs,
|
||||
], { timeoutMs: VERIFIER_TIMEOUT_MS });
|
||||
return {
|
||||
status: r.exitCode,
|
||||
report: r.stdout && r.stdout.length ? JSON.parse(r.stdout) : null,
|
||||
};
|
||||
}
|
||||
|
||||
/** git fixture plumbing that must abort loudly when setup breaks. */
|
||||
function gitIn(dir, args) {
|
||||
gitOrThrow(args, { cwd: dir });
|
||||
}
|
||||
function gitCommitAll(dir, message) {
|
||||
gitIn(dir, ['add', '-A']);
|
||||
gitIn(dir, ['-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-qm', message]);
|
||||
}
|
||||
|
||||
before(() => {
|
||||
tmpRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-4135-'));
|
||||
patchesDir = path.join(tmpRoot, 'patches');
|
||||
configDir = path.join(tmpRoot, 'installed');
|
||||
pristineDir = path.join(tmpRoot, 'pristine');
|
||||
resetFixture();
|
||||
});
|
||||
|
||||
after(() => {
|
||||
cleanup(tmpRoot);
|
||||
});
|
||||
|
||||
describe('Bug #4135: baseline coverage is reported, gateable, and widened from git history', () => {
|
||||
/**
|
||||
* Core regression (headline): the issue's exact scenario — 13 backed-up
|
||||
* customised files on a multi-version update, 12 changed upstream, 1
|
||||
* byte-identical. gsd-pristine/ holds only the byte-identical survivor.
|
||||
* The green exit must CARRY a countable coverage aggregate instead of
|
||||
* presenting like a fully-verified run.
|
||||
*/
|
||||
test('#4135: report aggregates baseline_covered so a 12-of-13 unbaselined run is countable', () => {
|
||||
resetFixture();
|
||||
const pristineHashes = {};
|
||||
for (let i = 1; i <= 13; i++) {
|
||||
const rel = `gsd-core/workflows/flow-${String(i).padStart(2, '0')}.md`;
|
||||
const pristine =
|
||||
`# Flow ${i}\nStock content of the outgoing release for file ${i}.\n` +
|
||||
`Line two of outgoing stock content ${i} with plenty of substance.\n`;
|
||||
const userLine = `## User customisation ${i}\nA custom section the user added for file ${i}.\n`;
|
||||
const upstream =
|
||||
`# Flow ${i} (rewritten)\nUpstream rewrote file ${i} across the multi-version span.\n` +
|
||||
`New stock structure with several new lines for file ${i}.\n`;
|
||||
pristineHashes[rel] = sha256(pristine);
|
||||
writeFile(path.join(patchesDir, rel), pristine + userLine);
|
||||
writeFile(path.join(configDir, rel), upstream + userLine);
|
||||
if (i === 7) {
|
||||
// The single byte-identical-across-the-span survivor.
|
||||
writeFile(path.join(pristineDir, rel), pristine);
|
||||
}
|
||||
}
|
||||
writeBackupMeta(pristineHashes);
|
||||
|
||||
const { status, report } = runVerifier();
|
||||
|
||||
assert.equal(status, 0, 'no-baseline files stay advisory — the default gate must stay green');
|
||||
assert.equal(report.checked, 13);
|
||||
assert.equal(report.no_baseline, 12);
|
||||
assert.equal(report.failures, 0);
|
||||
assert.equal(report.baseline_covered, 1,
|
||||
`coverage collapse must be countable; got ${JSON.stringify(report.baseline_covered)}`);
|
||||
});
|
||||
|
||||
/** Human-mode headline renderer: exact-contract unit on the typed helper. */
|
||||
test('#4135: human summary headlines baseline coverage N-of-M', () => {
|
||||
const script = require(SCRIPT);
|
||||
assert.equal(typeof script.coverageHeadline, 'function',
|
||||
'the human summary headline must be rendered by an exported typed helper');
|
||||
assert.equal(
|
||||
script.coverageHeadline(1, 13),
|
||||
'Baseline coverage: 1 of 13 file(s) verified against a pristine baseline (12 unverified)',
|
||||
'partial coverage renders N-of-M plus the unverified count');
|
||||
assert.equal(
|
||||
script.coverageHeadline(13, 13),
|
||||
'Baseline coverage: 13 of 13 file(s) verified against a pristine baseline',
|
||||
'full coverage renders without an unverified tail');
|
||||
});
|
||||
|
||||
/**
|
||||
* Core regression (strict gate): the same collapsed fixture under
|
||||
* `--min-baseline-coverage 0.9` must FAIL LOUDLY (new opt-in exit code 3)
|
||||
* while still emitting the parseable JSON report.
|
||||
*/
|
||||
test('#4135: opt-in --min-baseline-coverage exits 3 below threshold', () => {
|
||||
resetFixture();
|
||||
const pristineHashes = {};
|
||||
for (let i = 1; i <= 13; i++) {
|
||||
const rel = `gsd-core/workflows/flow-${String(i).padStart(2, '0')}.md`;
|
||||
const pristine = `# Flow ${i}\nOutgoing stock line with substantial content ${i}.\n`;
|
||||
const userLine = `User customisation line that survived the merge for file ${i}.\n`;
|
||||
const upstream = `# Flow ${i} new\nIncoming release rewrote this file upstream ${i}.\n`;
|
||||
pristineHashes[rel] = sha256(pristine);
|
||||
writeFile(path.join(patchesDir, rel), pristine + userLine);
|
||||
writeFile(path.join(configDir, rel), upstream + userLine);
|
||||
}
|
||||
writeBackupMeta(pristineHashes);
|
||||
|
||||
const { status, report } = runVerifier(['--min-baseline-coverage', '0.9']);
|
||||
|
||||
assert.equal(status, 3, 'a 1-of-13 run under a 0.9 threshold must exit non-zero (coverage gate)');
|
||||
assert.ok(report, 'the JSON report must still be emitted for scripting consumers');
|
||||
assert.equal(report.failures, 0, 'the failure here is coverage, not content');
|
||||
assert.equal(report.baseline_covered, 0);
|
||||
});
|
||||
|
||||
/** Precedence: a real content failure outranks the coverage failure. */
|
||||
test('#4135: strict gate yields to exit 1 when real content failed', () => {
|
||||
resetFixture();
|
||||
const rel = 'gsd-core/workflows/one.md';
|
||||
const pristine = 'outgoing stock line with substantial content here\n';
|
||||
const userLine = 'user customisation line that was dropped by the merge\n';
|
||||
writeBackupMeta({ [rel]: sha256(pristine) });
|
||||
writeFile(path.join(patchesDir, rel), pristine + userLine);
|
||||
writeFile(path.join(configDir, rel), pristine); // user line dropped
|
||||
writeFile(path.join(pristineDir, rel), pristine);
|
||||
|
||||
const { status, report } = runVerifier(['--min-baseline-coverage', '1']);
|
||||
|
||||
assert.equal(status, 1, 'content failure is the louder, more specific signal');
|
||||
assert.equal(report.failures, 1);
|
||||
assert.equal(report.results[0].reason, REASON.FAIL_USER_LINES_MISSING);
|
||||
});
|
||||
|
||||
/** Ratio boundary: at exactly the threshold the gate passes (>= semantics). */
|
||||
test('#4135: strict coverage gate passes at exactly the threshold', () => {
|
||||
resetFixture();
|
||||
seedTwoFilesOneCovered();
|
||||
const { status } = runVerifier(['--min-baseline-coverage', '0.5']);
|
||||
assert.equal(status, 0, '1 of 2 covered satisfies a 0.5 threshold (>= passes)');
|
||||
});
|
||||
|
||||
/** Ratio boundary: just below the threshold the gate fails. */
|
||||
test('#4135: strict coverage gate fails just below the threshold', () => {
|
||||
resetFixture();
|
||||
seedTwoFilesOneCovered();
|
||||
const { status } = runVerifier(['--min-baseline-coverage', '0.51']);
|
||||
assert.equal(status, 3, '1 of 2 covered does not satisfy a 0.51 threshold');
|
||||
});
|
||||
|
||||
/** Vacuous input: nothing checked cannot be under-covered. */
|
||||
test('#4135: strict coverage gate is vacuously satisfied when nothing is checked', () => {
|
||||
resetFixture();
|
||||
const { status, report } = runVerifier(['--min-baseline-coverage', '1']);
|
||||
assert.equal(status, 0);
|
||||
assert.equal(report.checked, 0);
|
||||
});
|
||||
|
||||
/** Arg validation: malformed thresholds are usage errors (exit 2). */
|
||||
test('#4135: malformed --min-baseline-coverage values are usage errors', () => {
|
||||
resetFixture();
|
||||
for (const bad of ['1.5', '-1', 'notanumber']) {
|
||||
const r = runNode([
|
||||
SCRIPT,
|
||||
'--patches-dir', patchesDir,
|
||||
'--config-dir', configDir,
|
||||
'--pristine-dir', pristineDir,
|
||||
'--json',
|
||||
'--min-baseline-coverage', bad,
|
||||
], { timeoutMs: VERIFIER_TIMEOUT_MS });
|
||||
assert.equal(r.exitCode, 2, `threshold "${bad}" must be a usage error, not silently clamped`);
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* Core regression (widening): multi-version collapse — no baseline under
|
||||
* gsd-pristine/, but the config dir is a git repository whose history
|
||||
* holds the outgoing bytes (recorded pristine_hashes match). The
|
||||
* recovered baseline must produce a REAL diff: the dropped user line is
|
||||
* caught as fail_user_lines_missing, not skipped as ok_no_baseline.
|
||||
*/
|
||||
test('#4135: resolves the baseline from git history by recorded hash and catches a dropped user line', () => {
|
||||
resetFixture();
|
||||
const rel = 'gsd-core/workflows/flow-01.md';
|
||||
const pristine =
|
||||
'# Flow 1\nOutgoing release stock line one with substance.\n' +
|
||||
'Outgoing release stock line two also with substance.\n';
|
||||
const userLine = 'user customisation line that the merge dropped for flow one';
|
||||
const backup = pristine + userLine + '\n';
|
||||
const upstreamRewrite =
|
||||
'# Flow 1 (rewritten)\nIncoming release replaced the stock body upstream.\n';
|
||||
|
||||
writeBackupMeta({ [rel]: sha256(pristine) });
|
||||
writeFile(path.join(patchesDir, rel), backup);
|
||||
// Config dir IS a git repo: outgoing pristine state, then the merged state.
|
||||
gitIn(configDir, ['init', '-q', '-b', 'main']);
|
||||
writeFile(path.join(configDir, rel), pristine);
|
||||
gitCommitAll(configDir, 'gsd install of the outgoing release');
|
||||
writeFile(path.join(configDir, rel), upstreamRewrite); // user line dropped
|
||||
gitCommitAll(configDir, 'post-update merged state');
|
||||
|
||||
const { status, report } = runVerifier();
|
||||
|
||||
assert.equal(status, 1, 'the git-recovered baseline must catch the dropped line');
|
||||
assert.equal(report.no_baseline, 0, 'the baseline was recoverable — no skip');
|
||||
assert.equal(report.baseline_covered, 1);
|
||||
const r0 = report.results[0];
|
||||
assert.equal(r0.status, 'fail');
|
||||
assert.equal(r0.reason, REASON.FAIL_USER_LINES_MISSING);
|
||||
assert.deepEqual(r0.missing, [userLine],
|
||||
'the diff ran against the RECOVERED baseline — upstream-removed lines are not "missing"');
|
||||
});
|
||||
|
||||
/**
|
||||
* Net observable: same recovery, user line PRESENT — the file is verified
|
||||
* (exit 0, covered) instead of silently skipped.
|
||||
*/
|
||||
test('#4135: git-recovered baseline verifies surviving user lines instead of skipping', () => {
|
||||
resetFixture();
|
||||
const rel = 'gsd-core/workflows/flow-02.md';
|
||||
const pristine = '# Flow 2\nOutgoing stock line with substantial content.\n';
|
||||
const userLine = 'user customisation line that survived the merge for flow two';
|
||||
const upstreamRewrite = '# Flow 2 (rewritten)\nIncoming release rewrote the stock body.\n';
|
||||
|
||||
writeBackupMeta({ [rel]: sha256(pristine) });
|
||||
writeFile(path.join(patchesDir, rel), pristine + userLine + '\n');
|
||||
gitIn(configDir, ['init', '-q', '-b', 'main']);
|
||||
writeFile(path.join(configDir, rel), pristine);
|
||||
gitCommitAll(configDir, 'gsd install of the outgoing release');
|
||||
writeFile(path.join(configDir, rel), upstreamRewrite + userLine + '\n');
|
||||
gitCommitAll(configDir, 'post-update merged state');
|
||||
|
||||
const { status, report } = runVerifier();
|
||||
|
||||
assert.equal(status, 0);
|
||||
assert.equal(report.no_baseline, 0);
|
||||
assert.equal(report.baseline_covered, 1);
|
||||
assert.equal(report.results[0].status, 'ok');
|
||||
assert.notEqual(report.results[0].reason, REASON.OK_NO_BASELINE);
|
||||
});
|
||||
|
||||
/**
|
||||
* Version-hop boundary (N−1 / multi-commit span): history holds TWO
|
||||
* upstream versions; the recorded hash is the OLDER one, so the walk must
|
||||
* pass the newer (mismatching) commit and match deeper history — the
|
||||
* single-hop shape of the same recovery.
|
||||
*/
|
||||
test('#4135: single-version hop also recovers the baseline from git history', () => {
|
||||
resetFixture();
|
||||
const rel = 'gsd-core/workflows/flow-03.md';
|
||||
const vA = '# Flow 3\nVersion A stock line with substantial content.\n';
|
||||
const vB = '# Flow 3\nVersion B stock line with substantial content.\n';
|
||||
const userLine = 'user customisation line present in the merged output';
|
||||
writeBackupMeta({ [rel]: sha256(vA) }); // outgoing = the OLDER commit's bytes
|
||||
writeFile(path.join(patchesDir, rel), vA + userLine + '\n');
|
||||
gitIn(configDir, ['init', '-q', '-b', 'main']);
|
||||
writeFile(path.join(configDir, rel), vA);
|
||||
gitCommitAll(configDir, 'gsd install vA');
|
||||
writeFile(path.join(configDir, rel), vB);
|
||||
gitCommitAll(configDir, 'gsd update to vB');
|
||||
writeFile(path.join(configDir, rel), vB + userLine + '\n');
|
||||
gitCommitAll(configDir, 'post-update merged state');
|
||||
|
||||
const { status, report } = runVerifier();
|
||||
|
||||
assert.equal(status, 0);
|
||||
assert.equal(report.no_baseline, 0, 'the walk must reach the older vA commit');
|
||||
assert.equal(report.baseline_covered, 1);
|
||||
});
|
||||
|
||||
/** Negative space: no git, no pristine — the #934 advisory posture is unchanged. */
|
||||
test('#4135: non-git config dirs keep the ok_no_baseline advisory posture', () => {
|
||||
resetFixture();
|
||||
const rel = 'gsd-core/workflows/flow-04.md';
|
||||
const pristine = '# Flow 4\nOutgoing stock line with substantial content.\n';
|
||||
writeBackupMeta({ [rel]: sha256(pristine) });
|
||||
writeFile(path.join(patchesDir, rel), pristine + 'user line that survived the merge here\n');
|
||||
writeFile(path.join(configDir, rel), 'incoming rewrite\nuser line that survived the merge here\n');
|
||||
|
||||
const { status, report } = runVerifier();
|
||||
|
||||
assert.equal(status, 0);
|
||||
assert.equal(report.no_baseline, 1);
|
||||
assert.equal(report.results[0].reason, REASON.OK_NO_BASELINE);
|
||||
assert.equal(report.baseline_covered, 0);
|
||||
});
|
||||
|
||||
/** Negative space: git present but no blob matches the recorded hash. */
|
||||
test('#4135: git tier adopts nothing when no blob matches the recorded hash', () => {
|
||||
resetFixture();
|
||||
const rel = 'gsd-core/workflows/flow-05.md';
|
||||
const pristine = '# Flow 5\nOutgoing stock line with substantial content.\n';
|
||||
writeBackupMeta({ [rel]: sha256(pristine) });
|
||||
writeFile(path.join(patchesDir, rel), pristine + 'user line that survived the merge here\n');
|
||||
gitIn(configDir, ['init', '-q', '-b', 'main']);
|
||||
writeFile(path.join(configDir, rel), 'history only ever held user-modified bytes\n');
|
||||
gitCommitAll(configDir, 'only user state was ever committed');
|
||||
|
||||
const { status, report } = runVerifier();
|
||||
|
||||
assert.equal(status, 0);
|
||||
assert.equal(report.no_baseline, 1, 'hash equality is the authority — nothing else is trusted');
|
||||
assert.equal(report.results[0].reason, REASON.OK_NO_BASELINE);
|
||||
});
|
||||
|
||||
/** Drift-path lock: #3657 is never bypassed by the git tier. */
|
||||
test('#4135: canonical drift is never rescued by the git-history tier', () => {
|
||||
resetFixture();
|
||||
const rel = 'gsd-core/workflows/flow-06.md';
|
||||
const pristine = '# Flow 6\nOutgoing stock line with substantial content.\n';
|
||||
const drifted = '# Flow 6\nRefreshed newer-release stock line with substance.\n';
|
||||
writeBackupMeta({ [rel]: sha256(pristine) });
|
||||
writeFile(path.join(patchesDir, rel), pristine + 'user line that survived the merge here\n');
|
||||
writeFile(path.join(configDir, rel), drifted + 'user line that survived the merge here\n');
|
||||
writeFile(path.join(pristineDir, rel), drifted); // canonical present, hash-mismatched
|
||||
gitIn(configDir, ['init', '-q', '-b', 'main']);
|
||||
writeFile(path.join(configDir, rel), pristine);
|
||||
gitCommitAll(configDir, 'history holds the recorded-hash bytes');
|
||||
|
||||
const { status, report } = runVerifier();
|
||||
|
||||
assert.equal(status, 0);
|
||||
assert.equal(report.results[0].reason, REASON.OK_PRISTINE_DRIFT_DETECTED,
|
||||
'a present-but-drifted canonical stays drift — no rescue');
|
||||
assert.equal(report.drifted, 1);
|
||||
});
|
||||
|
||||
/** Precedence lock: a matching canonical pristine beats the git tier. */
|
||||
test('#4135: canonical pristine keeps precedence over the git-history tier', () => {
|
||||
resetFixture();
|
||||
const rel = 'gsd-core/workflows/flow-07.md';
|
||||
const pristine = '# Flow 7\nOutgoing stock line with substantial content.\n';
|
||||
const droppedLine = 'user customisation line that the merge dropped for flow seven';
|
||||
writeBackupMeta({ [rel]: sha256(pristine) });
|
||||
writeFile(path.join(patchesDir, rel), pristine + droppedLine + '\n');
|
||||
writeFile(path.join(configDir, rel), pristine); // dropped — caught via canonical
|
||||
writeFile(path.join(pristineDir, rel), pristine); // canonical, hash-matching
|
||||
gitIn(configDir, ['init', '-q', '-b', 'main']);
|
||||
writeFile(path.join(configDir, rel), pristine);
|
||||
gitCommitAll(configDir, 'history also holds the bytes');
|
||||
|
||||
const { status, report } = runVerifier();
|
||||
|
||||
assert.equal(status, 1);
|
||||
assert.equal(report.results[0].reason, REASON.FAIL_USER_LINES_MISSING);
|
||||
assert.ok(report.results[0].missing.includes(droppedLine));
|
||||
});
|
||||
|
||||
/** Invocation-shape lock: no --pristine-dir → over-broad fallback, no git tier. */
|
||||
test('#4135: no git-history resolution when --pristine-dir is not provided', () => {
|
||||
resetFixture();
|
||||
const rel = 'gsd-core/workflows/flow-08.md';
|
||||
const pristine = '# Flow 8\nOutgoing stock line with substantial content.\n';
|
||||
const userLine = 'user customisation line that survived the merge for flow eight';
|
||||
writeBackupMeta({ [rel]: sha256(pristine) });
|
||||
writeFile(path.join(patchesDir, rel), pristine + userLine + '\n');
|
||||
gitIn(configDir, ['init', '-q', '-b', 'main']);
|
||||
writeFile(path.join(configDir, rel), pristine);
|
||||
gitCommitAll(configDir, 'history holds the recorded-hash bytes');
|
||||
// Merge kept everything — over-broad mode passes on this shape.
|
||||
writeFile(path.join(configDir, rel), pristine + userLine + '\n');
|
||||
|
||||
const r = runNode([
|
||||
SCRIPT, '--patches-dir', patchesDir, '--config-dir', configDir, '--json',
|
||||
], { timeoutMs: VERIFIER_TIMEOUT_MS });
|
||||
const report = r.stdout && r.stdout.length ? JSON.parse(r.stdout) : null;
|
||||
|
||||
assert.equal(r.exitCode, 0, 'all backup lines present — over-broad fallback passes');
|
||||
assert.equal(report.results[0].reason, null,
|
||||
'without --pristine-dir the baseline logic (incl. git tier) must not run');
|
||||
assert.equal(report.baseline_covered, 0,
|
||||
'an over-broad run verifies nothing against a pristine baseline — coverage stays 0');
|
||||
});
|
||||
|
||||
/** Module unit: findPristineInGit contract on a real fixture repo. */
|
||||
test('#4135: findPristineInGit unit — match, no-match, non-repo', () => {
|
||||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-4135-git-unit-'));
|
||||
const plain = fs.mkdtempSync(path.join(os.tmpdir(), 'gsd-4135-plain-'));
|
||||
try {
|
||||
const rel = 'gsd-core/workflows/unit.md';
|
||||
const content = '# Unit\noutgoing pristine bytes for the git walk unit test\n';
|
||||
gitIn(root, ['init', '-q', '-b', 'main']);
|
||||
writeFile(path.join(root, rel), content);
|
||||
gitCommitAll(root, 'seed');
|
||||
writeFile(path.join(root, rel), 'later user-modified bytes committed on top\n');
|
||||
gitCommitAll(root, 'user state');
|
||||
|
||||
assert.equal(findPristineInGit(root, rel, sha256(content)), content,
|
||||
'an exact recorded-hash blob in history is returned verbatim');
|
||||
assert.equal(findPristineInGit(root, rel, sha256('bytes that never existed anywhere\n')), null,
|
||||
'no matching blob resolves to null');
|
||||
assert.equal(findPristineInGit(plain, rel, sha256(content)), null,
|
||||
'a non-repo directory resolves to null without throwing');
|
||||
} finally {
|
||||
cleanup(root);
|
||||
cleanup(plain);
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* Workflow contract row (source-text-is-the-product): Step 5a must make
|
||||
* coverage a headline and document the opt-in strict gate.
|
||||
*/
|
||||
test('#4135: Step 5a headlines baseline coverage and documents the opt-in strict gate', () => {
|
||||
// allow-test-rule: source-text-is-the-product — Step 5a contract text (see #4135)
|
||||
const content = fs.readFileSync(WORKFLOW_PATH, 'utf8');
|
||||
assert.ok(content.includes('Baseline coverage:'),
|
||||
'Step 5a must print a Baseline coverage headline, not bury no_baseline in parseable fields');
|
||||
assert.ok(content.includes('--min-baseline-coverage'),
|
||||
'the opt-in strict coverage gate must be documented for operators');
|
||||
});
|
||||
|
||||
/**
|
||||
* Fixture: 2 files, exactly 1 baseline-covered — the minimal exact-ratio
|
||||
* fixture for the threshold boundary rows.
|
||||
*/
|
||||
function seedTwoFilesOneCovered() {
|
||||
const specs = [
|
||||
{ rel: 'gsd-core/workflows/a.md', covered: true },
|
||||
{ rel: 'gsd-core/workflows/b.md', covered: false },
|
||||
];
|
||||
const pristineHashes = {};
|
||||
for (const { rel, covered } of specs) {
|
||||
const pristine = `outgoing stock line with substantial content for ${rel}\n`;
|
||||
const userLine = `user customisation line that survived the merge for ${rel}\n`;
|
||||
pristineHashes[rel] = sha256(pristine);
|
||||
writeFile(path.join(patchesDir, rel), pristine + userLine);
|
||||
writeFile(path.join(configDir, rel), `incoming rewrite for ${rel}\n` + userLine);
|
||||
if (covered) writeFile(path.join(pristineDir, rel), pristine);
|
||||
}
|
||||
writeBackupMeta(pristineHashes);
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user