* test(#4135): regression rows for pristine regen coverage collapse RED skeleton: src/pristine-baseline.cts exports findPristineInGit as a null-returning stub (wired into verifyFile after the #4145 orphan tier, behavior-neutral) so the git-history rows fail behaviorally, not at require time. Failing-first rows: baseline_covered aggregate on a 1-of-13 multi-version fixture, coverageHeadline typed renderer, the opt-in --min-baseline-coverage gate (exit 3, >= threshold semantics, vacuous-pass and malformed-value boundaries), git-history baseline recovery (dropped-line catch + surviving-line verify + older-commit hop), findPristineInGit unit, Step 5a workflow headline contract, and the installer-side describeBaselineCoverage honest N-of-M summary with the collapse disk-state pinned. Negative-space rows pin today: non-git ok_no_baseline posture, no-match-no-adoption, #3657 drift never rescued, canonical precedence, and no git tier without --pristine-dir. * fix(#4135): headline baseline coverage, opt-in strict gate, git-history widening The #3407 promotion rule regenerates gsd-pristine/ baselines from the INCOMING release source and keeps only candidates byte-identical with the OUTGOING recorded hash — correct in isolation, but on a multi-version jump the surviving set is precisely the files upstream did NOT change. The verifier then reports ok_no_baseline (advisory, exit 0) for everything else, and no surface distinguishes a 12-of-13-unverified green run from a fully-verified one: the human summary printed Checked/Failures only, the JSON had no coverage aggregate, and the installer's update output gave per-bucket counts without N-of-M framing. All three issue directions, none exclusive: - Report coverage prominently: --json gains an additive baseline_covered aggregate; the human summary leads with 'Baseline coverage: N of M file(s)...' on every run plus an advisory section naming each skipped file and reason; the installer prints an honest covered-of-modified line via the exported describeBaselineCoverage helper (typed return, exact contract); workflow Step 5a computes and prints the headline before any pass/fail framing. - Fail louder on low coverage: opt-in --min-baseline-coverage <0..1> exits with new documented code 3 when coverage falls below the threshold (>= semantics; empty run vacuously passes; content failure exit 1 outranks it; malformed values are usage errors, exit 2). Default posture unchanged — no_baseline stays advisory per #934. - Widen the promotion rule (its only trustworthy form): when no baseline resolves under gsd-pristine/ and a hash is recorded, the verifier now recovers the baseline from the config dir's own git history — the workflow's documented Option A — anchored by the same authority every tier trusts, exact pristine_hashes sha-256 equality. Read-only (git log/git show, windowsHide per #685), bounded (100 commits/file, 10s/subprocess), null-on-any-failure so ok_no_baseline remains the universal fallback. Tier order: canonical join -> #4145 orphan scan -> git history -> OK_NO_BASELINE; #3657 drift and canonical precedence untouched. Hash validation in saveLocalPatches is NOT relaxed — the collapse is legitimate conservatism; hiding it was the bug. Measured on the issue's shape (13 files, 12 changed upstream, 1.10->1.12): non-git installs report baseline_covered 1/13 with the headline and can gate at exit 3; a git-managed config dir with the outgoing bytes in history verifies 13/13. Review fixes folded in: workflow headline derives the unverified count from checked - baseline_covered (not the drift+no_baseline sum), and the new site-scoped allow-test-rule annotation carries its ADR-456 see-ref on the marker line. Emitted-Drift-Ack-Growth: reapply-patches.md — #4135 — +20 lines / ~1.5 KB, prose and bash only: two additive parse lines (BASELINE_COVERED, CHECKED_COUNT), a Step 5a coverage-headline block printed BEFORE any pass/fail statement (documents the opt-in --min-baseline-coverage exit-3 gate), and one Option B sentence noting the verifier's read-only git-history fallback. No step ordering, gate, tool-invocation, or dispatch shape changed; 5a's fail/drift/advisory handling is unchanged, the headline only precedes it. * chore(#4135): backfill PR number into changeset fragment --------- Co-authored-by: agent-4135 <agent-4135@gsd.local>
This commit is contained in:
Tom Boucher
@@ -20,11 +20,21 @@
|
||||
* [--classify] # pre-merge mode: classify each backed-up
|
||||
* # file as incorporated / needs_merge /
|
||||
* # unknown (#4136); always exits 0
|
||||
* [--min-baseline-coverage <0..1>] # OPT-IN strict gate (#4135): exit 3
|
||||
* # when the fraction of files verified
|
||||
* # against a resolved pristine baseline
|
||||
* # (baseline_covered / checked) falls below
|
||||
* # the threshold. Default: off — a
|
||||
* # low-coverage run stays green per the
|
||||
* # documented #934 advisory posture, but
|
||||
* # its coverage is ALWAYS headline-reported.
|
||||
*
|
||||
* Exit codes (default gate mode):
|
||||
* 0 — every user-added line is present in the merged file (gate passes)
|
||||
* 1 — at least one missing line in at least one file (gate fails)
|
||||
* 1 — at least one missing line in at least one file (gate fails); outranks
|
||||
* a coverage-gate failure when both apply
|
||||
* 2 — usage / structural error (e.g. patches dir missing)
|
||||
* 3 — opt-in coverage gate failed (--min-baseline-coverage not met; #4135)
|
||||
*
|
||||
* Bug #2969: the Step 5 gate previously trusted Claude's free-text "verified:
|
||||
* yes/no" reporting per hunk. The LLM was filling in `yes` even when content
|
||||
@@ -49,13 +59,13 @@ const { ExitError, runMain } = require('./lib/cli-exit.cjs');
|
||||
// path under gsd-pristine/ (e.g. without the gsd-core/ prefix an earlier
|
||||
// release's writer dropped). Same module the installer's preserve-check uses,
|
||||
// so the two readers cannot drift apart again.
|
||||
const { findPristineByHash } = require('./lib/pristine-baseline.cjs');
|
||||
const { findPristineByHash, findPristineInGit } = require('./lib/pristine-baseline.cjs');
|
||||
|
||||
const SIGNIFICANT_MIN_CHARS = 12;
|
||||
const GSD_HOOK_VERSION_LINE_RE = /^(?:\/\/|#)\s*gsd-hook-version:\s*\S+\s*$/i;
|
||||
|
||||
function parseArgs(argv) {
|
||||
const opts = { patchesDir: null, configDir: null, pristineDir: null, json: false, classify: false };
|
||||
const opts = { patchesDir: null, configDir: null, pristineDir: null, json: false, classify: false, minBaselineCoverage: null };
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const arg = argv[i];
|
||||
if (arg === '--patches-dir') opts.patchesDir = argv[++i];
|
||||
@@ -63,9 +73,19 @@ function parseArgs(argv) {
|
||||
else if (arg === '--pristine-dir') opts.pristineDir = argv[++i];
|
||||
else if (arg === '--json') opts.json = true;
|
||||
else if (arg === '--classify') opts.classify = true;
|
||||
else if (arg === '--help' || arg === '-h') {
|
||||
else if (arg === '--min-baseline-coverage') {
|
||||
const raw = argv[++i];
|
||||
if (raw === undefined) {
|
||||
throw new ExitError(2, '--min-baseline-coverage requires a value between 0 and 1');
|
||||
}
|
||||
const value = Number(raw);
|
||||
if (!Number.isFinite(value) || value < 0 || value > 1) {
|
||||
throw new ExitError(2, `--min-baseline-coverage must be a number between 0 and 1, got: ${raw}`);
|
||||
}
|
||||
opts.minBaselineCoverage = value;
|
||||
} else if (arg === '--help' || arg === '-h') {
|
||||
process.stdout.write(
|
||||
'usage: verify-reapply-patches.cjs --patches-dir <path> --config-dir <path> [--pristine-dir <path>] [--json] [--classify]\n',
|
||||
'usage: verify-reapply-patches.cjs --patches-dir <path> --config-dir <path> [--pristine-dir <path>] [--json] [--classify] [--min-baseline-coverage <0..1>]\n',
|
||||
);
|
||||
throw new ExitError(0);
|
||||
} else {
|
||||
@@ -253,7 +273,7 @@ const PRISTINE_RESOLUTION = Object.freeze({
|
||||
* guard. Extracted from verifyFile's inline block so --classify reasons over
|
||||
* the exact same baseline semantics the post-merge gate enforces.
|
||||
*/
|
||||
function resolvePristineBaseline({ relPath, pristineDir, pristineHashes }) {
|
||||
function resolvePristineBaseline({ relPath, configDir, pristineDir, pristineHashes }) {
|
||||
const hashKey = relPath.replace(/\\/g, '/');
|
||||
const recordedHash = pristineHashes && pristineHashes[hashKey];
|
||||
if (pristineDir) {
|
||||
@@ -314,6 +334,27 @@ function resolvePristineBaseline({ relPath, pristineDir, pristineHashes }) {
|
||||
// Present but not a regular file — over-broad mode is the safe side.
|
||||
return { resolution: PRISTINE_RESOLUTION.OVERBROAD, content: null };
|
||||
}
|
||||
|
||||
// Bug #4135: still nothing under gsd-pristine/ — the multi-version
|
||||
// regeneration collapse (the #3407 promotion rule only keeps files
|
||||
// byte-identical across the WHOLE version span, so the surviving set is
|
||||
// precisely the files upstream did not change). When the config dir is
|
||||
// itself a git repository, its history may hold the outgoing bytes: this
|
||||
// is the workflow's documented Option A (reapply-patches.md Step 2),
|
||||
// anchored by the SAME authority every other tier trusts — exact
|
||||
// pristine_hashes equality. Read-only (git log / git show); any failure
|
||||
// (no git, not a repo, no matching blob) degrades to ABSENT_RECORDED.
|
||||
// A recovered baseline is hash-confirmed by construction, so it VALIDATES.
|
||||
if (!pristinePathExists && recordedHash) {
|
||||
try {
|
||||
const fromGit = findPristineInGit(configDir, hashKey, recordedHash);
|
||||
if (fromGit !== null) {
|
||||
return { resolution: PRISTINE_RESOLUTION.VALIDATED, content: fromGit };
|
||||
}
|
||||
} catch {
|
||||
// git unavailable or history walk failed — ABSENT_RECORDED posture
|
||||
}
|
||||
}
|
||||
// Bug #934: recordedHash is present (modern installer) but no
|
||||
// hash-matching pristine exists anywhere under gsd-pristine/ (the stat
|
||||
// missed and the #4145 recovery found nothing).
|
||||
@@ -387,6 +428,47 @@ function resolveSkillsRedirect(configDir) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* #4135: baseline-coverage accounting. A file counts as baseline-covered
|
||||
* when its diff was actually computed against a resolved pristine baseline,
|
||||
* i.e. its reason is one of the baseline-diff outcomes (null = lines
|
||||
* verified present, OK_NO_USER_LINES_VS_PRISTINE = no user lines versus
|
||||
* the baseline, FAIL_USER_LINES_MISSING = lines missing). Everything else
|
||||
* is uncovered: the silent skips (no baseline anywhere per #934/#4135,
|
||||
* drift per #3657, no-pristine over-broad with nothing significant) AND
|
||||
* the structural failures (installed missing / not a file / read error),
|
||||
* which never reached a baseline — they are loud blocking failures, but
|
||||
* they were not baseline-verified either, and this aggregate exists to
|
||||
* state exactly how much of the run the gate could reason about. When
|
||||
* --pristine-dir is absent the whole run is over-broad (#2998 fallback):
|
||||
* no file was verified against a baseline, covered is 0 by construction.
|
||||
*/
|
||||
const BASELINE_VERIFIED_REASONS = new Set([
|
||||
REASON.OK_NO_USER_LINES_VS_PRISTINE,
|
||||
REASON.FAIL_USER_LINES_MISSING,
|
||||
]);
|
||||
|
||||
function classifyBaselineCoverage(results, pristineDirProvided) {
|
||||
if (!pristineDirProvided) return 0;
|
||||
let covered = 0;
|
||||
for (const r of results) {
|
||||
if (r.reason === null || BASELINE_VERIFIED_REASONS.has(r.reason)) covered++;
|
||||
}
|
||||
return covered;
|
||||
}
|
||||
|
||||
/**
|
||||
* #4135: the headline string for the human summary. A run with 12 of 13
|
||||
* files unbaselined must not present the same way as a fully-verified one —
|
||||
* the N-of-M form is the issue's own framing, and the unverified tail is
|
||||
* only rendered when it is non-zero.
|
||||
*/
|
||||
function coverageHeadline(covered, total) {
|
||||
const unverified = Math.max(0, total - covered);
|
||||
const base = `Baseline coverage: ${covered} of ${total} file(s) verified against a pristine baseline`;
|
||||
return unverified > 0 ? `${base} (${unverified} unverified)` : base;
|
||||
}
|
||||
|
||||
function verifyFile({ relPath, patchesDir, configDir, pristineDir, pristineHashes, skillsRedirect }) {
|
||||
const backupPath = path.join(patchesDir, relPath);
|
||||
const installedPath = resolveInstalledPath(configDir, relPath, skillsRedirect);
|
||||
@@ -431,7 +513,7 @@ function verifyFile({ relPath, patchesDir, configDir, pristineDir, pristineHashe
|
||||
// hash-first recovery + #934 absent guard) moved into resolvePristineBaseline
|
||||
// so the pre-merge classifier reasons over the exact same semantics.
|
||||
const { resolution, content: pristineContent } =
|
||||
resolvePristineBaseline({ relPath, pristineDir, pristineHashes });
|
||||
resolvePristineBaseline({ relPath, configDir, pristineDir, pristineHashes });
|
||||
|
||||
// Bug #3657: the resolved snapshot hash-mismatches the recorded baseline.
|
||||
// Skip the file with a diagnostic code rather than diffing against the
|
||||
@@ -531,7 +613,7 @@ function classifyFile({ relPath, patchesDir, configDir, pristineDir, pristineHas
|
||||
}
|
||||
|
||||
const { resolution, content: pristineContent } =
|
||||
resolvePristineBaseline({ relPath, pristineDir, pristineHashes });
|
||||
resolvePristineBaseline({ relPath, configDir, pristineDir, pristineHashes });
|
||||
|
||||
if (resolution === PRISTINE_RESOLUTION.DRIFTED) {
|
||||
result.reason = REASON.OK_PRISTINE_DRIFT_DETECTED;
|
||||
@@ -669,14 +751,52 @@ function main() {
|
||||
const no_baseline = noBaselineResults.length;
|
||||
const no_baseline_files = noBaselineResults.map((r) => r.file);
|
||||
|
||||
// Bug #4135: baseline coverage is a first-class aggregate. The collapse
|
||||
// this reports is silent by design in every other surface (no_baseline is
|
||||
// advisory, exit stays 0), which is exactly why it must be counted here:
|
||||
// "A clean verifier exit reads as 'hunks survived'. On this run it meant
|
||||
// '12 of 13 files were not checked at all.'"
|
||||
const baseline_covered = classifyBaselineCoverage(results, Boolean(opts.pristineDir));
|
||||
|
||||
// Bug #4135: the opt-in strict coverage gate. Threshold semantics are >=
|
||||
// (a run exactly at the threshold passes); an empty run is vacuously
|
||||
// covered (nothing checked cannot be under-covered). A content failure
|
||||
// (exit 1) outranks a coverage failure — it is the louder, more specific
|
||||
// signal.
|
||||
let coverageGateFailed = false;
|
||||
if (opts.minBaselineCoverage !== null) {
|
||||
const ratio = results.length === 0 ? 1 : baseline_covered / results.length;
|
||||
coverageGateFailed = ratio < opts.minBaselineCoverage;
|
||||
}
|
||||
|
||||
if (opts.json) {
|
||||
process.stdout.write(
|
||||
JSON.stringify({ checked: results.length, failures: failures.length, drifted, drifted_files, no_baseline, no_baseline_files, results }, null, 2) + '\n',
|
||||
JSON.stringify({ checked: results.length, failures: failures.length, drifted, drifted_files, no_baseline, no_baseline_files, baseline_covered, results }, null, 2) + '\n',
|
||||
);
|
||||
} else {
|
||||
process.stdout.write(`# Hunk Verification Gate (#2969)\n\n`);
|
||||
process.stdout.write(`Checked: ${results.length} file(s)\n`);
|
||||
process.stdout.write(`Failures: ${failures.length}\n\n`);
|
||||
process.stdout.write(`Failures: ${failures.length}\n`);
|
||||
// #4135: the coverage headline is printed on EVERY run, before any
|
||||
// per-file detail, so a near-zero-coverage green run can never render
|
||||
// identically to a fully-verified one.
|
||||
process.stdout.write(`${coverageHeadline(baseline_covered, results.length)}\n\n`);
|
||||
if (baseline_covered < results.length) {
|
||||
if (!opts.pristineDir) {
|
||||
process.stdout.write(`No --pristine-dir: over-broad fallback — nothing was verified against a pristine baseline (#2998).\n\n`);
|
||||
} else {
|
||||
const skipped = results.filter((r) => r.reason !== null && !BASELINE_VERIFIED_REASONS.has(r.reason));
|
||||
process.stdout.write(`## Files not verified against a pristine baseline\n\n`);
|
||||
process.stdout.write(`Advisory (non-blocking): their user customizations may or may not have survived the merge.\n\n`);
|
||||
for (const r of skipped) {
|
||||
process.stdout.write(`- ${r.file} (${r.reason})\n`);
|
||||
}
|
||||
process.stdout.write('\n');
|
||||
}
|
||||
}
|
||||
if (coverageGateFailed) {
|
||||
process.stdout.write(`COVERAGE GATE FAILED: baseline coverage ${(opts.minBaselineCoverage * 100).toFixed(1)}% required, ${results.length === 0 ? 100 : ((baseline_covered / results.length) * 100).toFixed(1)}% achieved (#4135 strict mode).\n\n`);
|
||||
}
|
||||
if (failures.length > 0) {
|
||||
process.stdout.write(`## Files with missing user-added content\n\n`);
|
||||
for (const r of failures) {
|
||||
@@ -692,11 +812,14 @@ function main() {
|
||||
}
|
||||
}
|
||||
|
||||
return failures.length > 0 ? 1 : 0;
|
||||
if (failures.length > 0) return 1;
|
||||
if (coverageGateFailed) return 3;
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (require.main === module) {
|
||||
runMain(main);
|
||||
}
|
||||
|
||||
module.exports = { computeUserAddedLines, isSignificantLine, verifyFile, classifyFile, walk, REASON, CLASSIFICATION, PRISTINE_RESOLUTION, resolvePristineBaseline, readPristineHashes, sha256, resolveInstalledPath, resolveSkillsRedirect };
|
||||
module.exports = { computeUserAddedLines, isSignificantLine, verifyFile, classifyFile, walk, REASON, CLASSIFICATION, PRISTINE_RESOLUTION, resolvePristineBaseline, readPristineHashes, sha256, resolveInstalledPath, resolveSkillsRedirect, coverageHeadline, classifyBaselineCoverage };
|
||||
|
||||
|
||||
@@ -169,7 +169,7 @@ Check if a `gsd-pristine/` directory exists alongside `gsd-local-patches/`:
|
||||
```bash
|
||||
PRISTINE_DIR="$CONFIG_DIR/gsd-pristine"
|
||||
```
|
||||
If it exists, the installer saved pristine copies at install time. Use these as the baseline. Both the deterministic verifier and the installer's preserve-check resolve each file's snapshot at its canonical path first and, when that misses, by the SHA-256 recorded in `pristine_hashes` — so a snapshot stored at a legacy path (for example, without the `gsd-core/` prefix an earlier release dropped) is still found and, on the next update, relocated to its canonical path (#4145).
|
||||
If it exists, the installer saved pristine copies at install time. Use these as the baseline. Both the deterministic verifier and the installer's preserve-check resolve each file's snapshot at its canonical path first and, when that misses, by the SHA-256 recorded in `pristine_hashes` — so a snapshot stored at a legacy path (for example, without the `gsd-core/` prefix an earlier release dropped) is still found and, on the next update, relocated to its canonical path (#4145). When no snapshot resolves under `gsd-pristine/`, the deterministic verifier additionally falls back to Option A's git-history walk itself (read-only, same recorded-hash match), which is what keeps Step 5a coverage non-zero on multi-version updates where the hash-validated regeneration had nothing it could promote (#4135).
|
||||
|
||||
### Option C: No baseline available (two-way fallback)
|
||||
If neither git history nor pristine snapshots are available, fall back to two-way comparison — but with **strengthened heuristics** (see Step 3).
|
||||
@@ -362,9 +362,27 @@ VERIFY_STATUS=$?
|
||||
DRIFTED_COUNT="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));process.stdout.write(String(d.drifted||0))")"
|
||||
DRIFTED_FILES="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));(d.drifted_files||[]).forEach(f=>process.stdout.write(f+'\n'))")"
|
||||
NO_BASELINE_COUNT="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));process.stdout.write(String(d.no_baseline||0))")"
|
||||
NO_BASELINE_FILES="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));(d.no_baseline_files||[]).forEach(f=>process.stdout.write(f+'\n'))")"
|
||||
NO_BASELINE_FILES="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));(d.no_baseline_files||[]).forEach(f=>process.stdout.write(f+'\n'))")"
|
||||
BASELINE_COVERED="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));process.stdout.write(String(d.baseline_covered||0))")"
|
||||
CHECKED_COUNT="$(echo "$VERIFY_OUTPUT" | node -e "const d=JSON.parse(require('fs').readFileSync('/dev/stdin','utf8'));process.stdout.write(String(d.checked||0))")"
|
||||
```
|
||||
|
||||
**Baseline coverage headline (#4135)** — BEFORE any pass/fail statement, print the coverage
|
||||
N-of-M line. A run where 12 of 13 files were not diff-verified must never present the same way
|
||||
as a fully-verified one:
|
||||
|
||||
```text
|
||||
Baseline coverage: {BASELINE_COVERED} of {CHECKED_COUNT} file(s) verified against a pristine
|
||||
baseline ({CHECKED_COUNT - BASELINE_COVERED} unverified)
|
||||
```
|
||||
|
||||
The verifier also reports this on every run (JSON field `baseline_covered`; the human summary
|
||||
leads with the same line). Operators who want the gate itself to fail on low coverage (cautious
|
||||
environments) pass the opt-in strict flag when invoking the verifier:
|
||||
`--min-baseline-coverage <fraction between 0 and 1>` — below the threshold the verifier exits
|
||||
with code 3 (a real content failure still exits 1 and outranks it). Without the flag the
|
||||
default advisory posture is unchanged.
|
||||
|
||||
**If `NO_BASELINE_COUNT` is greater than 0**, emit an advisory warning (non-blocking — the gate still exits 0 for these files). Do NOT halt:
|
||||
|
||||
```text
|
||||
|
||||
Reference in New Issue
Block a user