fix(verification): keep passed uat results across verifier re-runs
A phase whose human UAT passed could loop forever between execute-phase and verify-work. A gap-closure plan, or a later phase editing a covered file, made the report stale; verify-work only recorded a passed UAT against human_needed; the re-run verifier could not see the UAT file and re-emitted human_needed; and execute-phase's human_needed branch then rewrote *-UAT.md with every row back to [pending]. - verification.uat-evidence: recorded UAT rows plus the covered implementation files changed since the UAT. The verifier (Step 8b) treats a row that passed on unchanged code as verified and re-lists only affected rows, each with a retest_reason. - verification.seed-uat: merges the report's human items into *-UAT.md. Existing rows are kept byte-for-byte; a pass is reset only with a recorded reason. - verification.canonicalize-uat: the single human_needed -> passed flip, gated on the uat-only row predicate; refuses a stale report. - verification.status reports stale_reason; init.progress lists every executed-but-stale phase in reverify_phases. - verify-work and progress re-run the verifier themselves for a stale report instead of routing to execute-phase; execute-phase records a gap-closure checkpoint's UAT answers before dispatching the verifier. Fail-closed properties are unchanged: a malformed fingerprint and a real change to covered code still read stale. Emitted-Drift-Ack-Growth: execute-phase.md — verify_phase_goal gains the gap-closure uat-record pointer and the merge-not-overwrite human_needed branch; the uat template it replaced moved into verification.seed-uat Emitted-Drift-Ack-Growth: msd-verifier.md — new step 8b pointer to the lazily loaded verifier-uat-evidence reference plus the human_verified frontmatter key Emitted-Drift-Ack-Growth: progress.md — route v.stale now re-verifies every stale executed phase in place instead of naming one command
This commit is contained in:
Jakub Zych
18
src/init.cts
18
src/init.cts
@@ -272,6 +272,12 @@ interface PhaseCompletionProjection {
|
||||
* complete) or ran to completion.
|
||||
*/
|
||||
verification_stale_check_indeterminate: boolean;
|
||||
/**
|
||||
* WHY the report reads `stale` (`readVerificationStatus`'s `stale_reason`),
|
||||
* '' for every other status. Lets progress tell a gap-closure plan the
|
||||
* report never saw from a covered file drifting under a finished phase.
|
||||
*/
|
||||
verification_stale_reason: string;
|
||||
}
|
||||
|
||||
|
||||
@@ -330,6 +336,7 @@ function buildPhaseCompletionProjection(
|
||||
// internal staleness check could not run to completion.
|
||||
verification_stale_check_indeterminate: 'staleCheckIndeterminate' in verificationStatus
|
||||
&& verificationStatus.staleCheckIndeterminate === true,
|
||||
verification_stale_reason: verificationStatus.stale_reason ?? '',
|
||||
};
|
||||
}
|
||||
|
||||
@@ -3872,6 +3879,17 @@ function cmdInitProgress(cwd: string, raw: boolean, options: Record<string, unkn
|
||||
phases,
|
||||
phase_count: phases.length,
|
||||
completed_count: phases.filter((p) => p['status'] === 'complete').length,
|
||||
// Every phase whose plans are all summarized and whose ONLY open item is a
|
||||
// stale report — one verifier re-run each, no plan executes. Progress
|
||||
// offers these as a single "re-verify" action instead of walking the user
|
||||
// back through them one execute-phase route per session.
|
||||
reverify_phases: phases
|
||||
.filter((p) => p['implementation_complete'] === true && p['verification_status'] === 'stale')
|
||||
.map((p) => ({
|
||||
number: p['number'],
|
||||
directory: p['directory'],
|
||||
stale_reason: p['verification_stale_reason'],
|
||||
})),
|
||||
in_progress_count: phases.filter((p) =>
|
||||
['executed', 'in_progress'].includes(p['status'] as string),
|
||||
).length,
|
||||
|
||||
735
src/uat-evidence.cts
Normal file
735
src/uat-evidence.cts
Normal file
@@ -0,0 +1,735 @@
|
||||
/**
|
||||
* UAT Evidence — the seam that lets a recorded human UAT survive a verifier
|
||||
* re-run.
|
||||
*
|
||||
* A phase's `*-VERIFICATION.md` goes `stale` whenever the verifier has to look
|
||||
* again: a gap-closure plan added a PLAN/SUMMARY it never declared, or a
|
||||
* covered source file changed. Re-running the verifier is the right remedy —
|
||||
* but before this module nothing carried the human's UAT results across that
|
||||
* re-run. The verifier could not see `*-UAT.md`, so it re-emitted the same
|
||||
* `human_needed` items, and execute-phase's `human_needed` branch then wrote a
|
||||
* fresh `*-UAT.md` over the completed one, every row back to `[pending]`. The
|
||||
* user was asked for the same UAT again, indefinitely.
|
||||
*
|
||||
* Two verbs close that loop, both under the `verification` family:
|
||||
*
|
||||
* - `verification.uat-evidence <phaseDir> [files…]` (read-only) — the
|
||||
* recorded UAT rows, plus which covered implementation files changed
|
||||
* AFTER the UAT was recorded. The verifier is an LLM, not a clock: this
|
||||
* does the deterministic part so the agent only decides which rows a
|
||||
* changed file actually affects.
|
||||
* - `verification.seed-uat <phaseDir>` — writes the report's
|
||||
* `human_verification` items into `*-UAT.md` by MERGING: existing rows and
|
||||
* their results are kept byte-for-byte, only unseen items are appended as
|
||||
* `[pending]`, and a passing row is reset only when the verifier recorded
|
||||
* a `retest_reason` for it (the reason is written into the row).
|
||||
*
|
||||
* - `verification.canonicalize-uat <phaseDir>` — the ONE place a
|
||||
* `human_needed` report becomes `passed` on the strength of a passed UAT
|
||||
* (#4663). verify-work, execute-phase and progress all call it, so the
|
||||
* rule cannot be re-derived differently per workflow.
|
||||
*
|
||||
* The verifier remains the only emitter of a verdict it reached itself; the
|
||||
* canonicalization records the human's half of a `human_needed` verdict, and
|
||||
* refuses every other status — a `stale` report is never flipped.
|
||||
*
|
||||
* ADR-457 build-at-publish: compiled by tsc to msd-core/bin/lib/uat-evidence.cjs.
|
||||
*/
|
||||
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { findProjectRoot } from './project-root.cjs';
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- io.cjs is an export= CommonJS module
|
||||
import io = require('./io.cjs');
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- phase-id.cjs is an export= CommonJS module
|
||||
import phaseId = require('./phase-id.cjs');
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- frontmatter.cjs is an export= CommonJS module
|
||||
import frontmatterMod = require('./frontmatter.cjs');
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- core-utils.cjs is an export= CommonJS module
|
||||
import coreUtilsMod = require('./core-utils.cjs');
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- verification.cjs is an export= CommonJS module
|
||||
import verification = require('./verification.cjs');
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports -- uat-predicate.cjs is an export= CommonJS module
|
||||
import uatPredicate = require('./uat-predicate.cjs');
|
||||
|
||||
const { output, error } = io;
|
||||
const { evaluateUatPassed } = uatPredicate;
|
||||
const { extractPhaseToken } = phaseId;
|
||||
const { extractFrontmatter, frontmatterListEntries, spliceFrontmatter } = frontmatterMod;
|
||||
const { normalizeLineEndings } = coreUtilsMod;
|
||||
const {
|
||||
resolveUatFile,
|
||||
resolveVerificationFile,
|
||||
readVerificationStatus,
|
||||
defaultPhaseCleanCommitTimesMs,
|
||||
canonicalizeCoveredFiles,
|
||||
parseFingerprintFileArgs,
|
||||
} = verification;
|
||||
|
||||
// ─── Types ────────────────────────────────────────────────────────────────────
|
||||
|
||||
interface UatRow {
|
||||
test: number;
|
||||
name: string;
|
||||
/** Lowercased `result:` value without brackets; `missing` when the row has none. */
|
||||
result: string;
|
||||
passing: boolean;
|
||||
expected: string;
|
||||
/** Present when a previous seed reset this row for re-test. */
|
||||
retest_reason: string;
|
||||
/** Line indexes into the scanned document (internal to the merge). */
|
||||
headingLine: number;
|
||||
resultLine: number;
|
||||
endLine: number;
|
||||
}
|
||||
|
||||
interface HumanItem {
|
||||
test: string;
|
||||
expected: string;
|
||||
why_human: string;
|
||||
/** Set by the verifier ONLY when it re-lists an item whose UAT row already passed. */
|
||||
retest_reason: string;
|
||||
}
|
||||
|
||||
type PublicRow = Pick<UatRow, 'test' | 'name' | 'result' | 'passing' | 'expected' | 'retest_reason'>;
|
||||
|
||||
interface UatEvidence {
|
||||
uat_file: string;
|
||||
uat_status: string;
|
||||
/** ISO time the UAT file last changed (commit time when clean, else mtime); '' when no UAT file. */
|
||||
recorded_at: string;
|
||||
rows: PublicRow[];
|
||||
/** Implementation files compared against `recorded_at` (`.planning/` paths are never compared). */
|
||||
checked_files: string[];
|
||||
changed_since_uat: Array<{ file: string; reason: 'modified' | 'missing' }>;
|
||||
/** True only when a UAT file exists and no checked file changed after it. */
|
||||
code_unchanged_since_uat: boolean;
|
||||
}
|
||||
|
||||
interface SeedUatResult {
|
||||
uat_file: string;
|
||||
created: boolean;
|
||||
changed: boolean;
|
||||
kept: Array<{ test: number; name: string; result: string }>;
|
||||
appended: Array<{ test: number; name: string }>;
|
||||
retested: Array<{ test: number; name: string; previous_result: string; reason: string }>;
|
||||
pending: number;
|
||||
/** Every row passes — verify-work can canonicalize without asking the human anything. */
|
||||
all_passing: boolean;
|
||||
}
|
||||
|
||||
type CleanCommitTimesFn = (dir: string, files: string[]) => Map<string, number>;
|
||||
|
||||
// ─── Row scanning ─────────────────────────────────────────────────────────────
|
||||
|
||||
const PASSING_RESULTS: ReadonlySet<string> = new Set(['pass', 'passed']);
|
||||
const HEADING_RE = /^###\s*(\d+)\.\s*(.+)$/;
|
||||
const SECTION_RE = /^##\s+\S/;
|
||||
const RESULT_RE = /^result:[ \t]*\[?([\w-]+)\]?/i;
|
||||
const FENCE_RE = /^\s{0,3}(`{3,}|~{3,})/;
|
||||
|
||||
/** Index of the first body line — the line after a byte-0 frontmatter block, or 0. */
|
||||
function bodyStart(lines: readonly string[]): number {
|
||||
if (lines[0] !== '---') return 0;
|
||||
for (let i = 1; i < lines.length; i++) {
|
||||
if (/^---[ \t]*$/.test(lines[i])) return i + 1;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Scan a `*-UAT.md` body for `### N. Name` test rows, with the line positions
|
||||
* the merge needs. Headings and `result:` lines inside fenced code or an HTML
|
||||
* comment are not rows — the same contexts `uat-predicate.cts` discards, so a
|
||||
* row this scan would rewrite is always one the completion predicate reads.
|
||||
*/
|
||||
function scanUatRows(lines: readonly string[]): UatRow[] {
|
||||
const rows: UatRow[] = [];
|
||||
let current: UatRow | null = null;
|
||||
let fence: string | null = null;
|
||||
let inComment = false;
|
||||
|
||||
const close = (endLine: number): void => {
|
||||
if (current) {
|
||||
current.endLine = endLine;
|
||||
rows.push(current);
|
||||
current = null;
|
||||
}
|
||||
};
|
||||
|
||||
for (let i = bodyStart(lines); i < lines.length; i++) {
|
||||
const line = lines[i];
|
||||
if (inComment) {
|
||||
if (line.includes('-->')) inComment = false;
|
||||
continue;
|
||||
}
|
||||
const fenceMatch = FENCE_RE.exec(line);
|
||||
if (fence) {
|
||||
if (fenceMatch && fenceMatch[1][0] === fence[0] && fenceMatch[1].length >= fence.length) fence = null;
|
||||
continue;
|
||||
}
|
||||
if (fenceMatch) {
|
||||
fence = fenceMatch[1];
|
||||
continue;
|
||||
}
|
||||
if (line.includes('<!--') && !line.includes('-->', line.indexOf('<!--'))) {
|
||||
inComment = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
const heading = HEADING_RE.exec(line);
|
||||
if (heading) {
|
||||
close(i);
|
||||
current = {
|
||||
test: Number.parseInt(heading[1], 10),
|
||||
name: heading[2].trim(),
|
||||
result: 'missing',
|
||||
passing: false,
|
||||
expected: '',
|
||||
retest_reason: '',
|
||||
headingLine: i,
|
||||
resultLine: -1,
|
||||
endLine: lines.length,
|
||||
};
|
||||
continue;
|
||||
}
|
||||
if (SECTION_RE.test(line)) {
|
||||
close(i);
|
||||
continue;
|
||||
}
|
||||
if (!current) continue;
|
||||
|
||||
// FIRST match wins for every key, matching uat-predicate's contract.
|
||||
const result = RESULT_RE.exec(line);
|
||||
if (result && current.resultLine === -1) {
|
||||
current.result = result[1].toLowerCase();
|
||||
current.passing = PASSING_RESULTS.has(current.result);
|
||||
current.resultLine = i;
|
||||
continue;
|
||||
}
|
||||
const expected = /^expected:[ \t]*(.*)$/.exec(line);
|
||||
if (expected && current.expected === '') {
|
||||
const inline = expected[1].trim();
|
||||
if (inline === '|' || inline === '>' || inline === '') {
|
||||
const block: string[] = [];
|
||||
for (let j = i + 1; j < lines.length && /^\s+\S/.test(lines[j]); j++) block.push(lines[j].trim());
|
||||
current.expected = block.join(' ');
|
||||
} else {
|
||||
current.expected = inline;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const retest = /^retest_reason:[ \t]*(.*)$/.exec(line);
|
||||
if (retest && current.retest_reason === '') current.retest_reason = unquote(retest[1].trim());
|
||||
}
|
||||
close(lines.length);
|
||||
return rows;
|
||||
}
|
||||
|
||||
function unquote(value: string): string {
|
||||
if (value.length >= 2 && value.startsWith('"') && value.endsWith('"')) {
|
||||
return value.slice(1, -1).replace(/\\"/g, '"').replace(/\\\\/g, '\\');
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function quote(value: string): string {
|
||||
return `"${value.replace(/\\/g, '\\\\').replace(/"/g, '\\"')}"`;
|
||||
}
|
||||
|
||||
/** One line, whitespace collapsed — a heading or a quoted reason must never span lines. */
|
||||
function oneLine(value: string): string {
|
||||
return value.replace(/\s+/g, ' ').trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* The key a verifier item and a UAT row are matched on: lowercase, punctuation
|
||||
* stripped, whitespace collapsed — the normalization `verification-overrides`
|
||||
* already uses to match a must-have across verifier runs.
|
||||
*/
|
||||
function matchKey(name: string): string {
|
||||
return name
|
||||
.toLowerCase()
|
||||
.replace(/[^\p{L}\p{N}]+/gu, ' ')
|
||||
.trim();
|
||||
}
|
||||
|
||||
function publicRow(row: UatRow): PublicRow {
|
||||
return {
|
||||
test: row.test,
|
||||
name: row.name,
|
||||
result: row.result,
|
||||
passing: row.passing,
|
||||
expected: row.expected,
|
||||
retest_reason: row.retest_reason,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Phase file resolution ────────────────────────────────────────────────────
|
||||
|
||||
interface PhaseFiles {
|
||||
uatFile: string | null;
|
||||
verificationFile: string | null;
|
||||
}
|
||||
|
||||
function resolvePhaseFiles(phaseDir: string): PhaseFiles {
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = fs.readdirSync(phaseDir);
|
||||
} catch {
|
||||
return { uatFile: null, verificationFile: null };
|
||||
}
|
||||
const phaseDirName = path.basename(phaseDir);
|
||||
const options = { allowBare: true, phaseToken: extractPhaseToken(phaseDirName), phaseDirName };
|
||||
return {
|
||||
uatFile: resolveUatFile(entries, options),
|
||||
verificationFile: resolveVerificationFile(entries, options),
|
||||
};
|
||||
}
|
||||
|
||||
function readNormalized(filePath: string): string | null {
|
||||
try {
|
||||
return normalizeLineEndings(fs.readFileSync(filePath, 'utf-8'));
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function stringField(entry: unknown, key: string): string {
|
||||
if (entry === null || typeof entry !== 'object') return '';
|
||||
const value = (entry as Record<string, unknown>)[key];
|
||||
return typeof value === 'string' ? value.trim() : '';
|
||||
}
|
||||
|
||||
/** The report's `human_verification` items; an item without a `test` is not an item. */
|
||||
function readHumanItems(verificationContent: string): HumanItem[] {
|
||||
const entries = frontmatterListEntries(verificationContent, 'human_verification') ?? [];
|
||||
const items: HumanItem[] = [];
|
||||
for (const entry of entries) {
|
||||
const test = oneLine(stringField(entry, 'test'));
|
||||
if (test.length === 0) continue;
|
||||
items.push({
|
||||
test,
|
||||
expected: stringField(entry, 'expected'),
|
||||
why_human: stringField(entry, 'why_human'),
|
||||
retest_reason: oneLine(stringField(entry, 'retest_reason')),
|
||||
});
|
||||
}
|
||||
return items;
|
||||
}
|
||||
|
||||
// ─── uat-evidence ─────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Read the recorded UAT rows for a phase and report which implementation files
|
||||
* changed after the UAT was recorded.
|
||||
*
|
||||
* `files` defaults to the current report's own `covered_files`. Paths under
|
||||
* `.planning/` are never compared: they are bookkeeping (the SUMMARY of the
|
||||
* very plan whose checkpoint WAS the UAT is written after it), not the code a
|
||||
* human tested.
|
||||
*
|
||||
* Time is commit time for a committed-and-clean file, mtime otherwise — the
|
||||
* same clock the legacy staleness check uses (#2348), so a fresh clone and the
|
||||
* machine that recorded the UAT agree. A file that no longer exists counts as
|
||||
* changed: fail closed, the same direction as the fingerprint.
|
||||
*/
|
||||
function readUatEvidence(
|
||||
phaseDir: string,
|
||||
files?: readonly string[],
|
||||
cleanCommitTimesMs: CleanCommitTimesFn = defaultPhaseCleanCommitTimesMs,
|
||||
): UatEvidence {
|
||||
const empty: UatEvidence = {
|
||||
uat_file: '',
|
||||
uat_status: '',
|
||||
recorded_at: '',
|
||||
rows: [],
|
||||
checked_files: [],
|
||||
changed_since_uat: [],
|
||||
code_unchanged_since_uat: false,
|
||||
};
|
||||
const { uatFile, verificationFile } = resolvePhaseFiles(phaseDir);
|
||||
if (!uatFile) return empty;
|
||||
const uatPath = path.join(phaseDir, uatFile);
|
||||
const content = readNormalized(uatPath);
|
||||
if (content === null) return empty;
|
||||
|
||||
let declared: readonly string[] = files ?? [];
|
||||
if (files === undefined && verificationFile) {
|
||||
const report = readNormalized(path.join(phaseDir, verificationFile));
|
||||
const covered = report === null ? undefined : extractFrontmatter(report)['covered_files'];
|
||||
if (Array.isArray(covered)) declared = covered.filter((f): f is string => typeof f === 'string');
|
||||
}
|
||||
const checked = (canonicalizeCoveredFiles(declared)).filter(
|
||||
(f) => f !== '.planning' && !f.startsWith('.planning/') && !f.startsWith('../') && !path.isAbsolute(f),
|
||||
);
|
||||
|
||||
const projectRoot = findProjectRoot(phaseDir);
|
||||
const uatRel = path.relative(projectRoot, uatPath).replace(/\\/g, '/');
|
||||
const commitTimes = cleanCommitTimesMs(projectRoot, [uatRel, ...checked]);
|
||||
const timeOf = (rel: string): number | null => {
|
||||
const committed = commitTimes.get(rel);
|
||||
if (committed !== undefined) return committed;
|
||||
try {
|
||||
return fs.statSync(path.resolve(projectRoot, rel)).mtimeMs;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
const recordedAt = timeOf(uatRel);
|
||||
const changed: UatEvidence['changed_since_uat'] = [];
|
||||
for (const file of checked) {
|
||||
const changedAt = timeOf(file);
|
||||
if (changedAt === null) changed.push({ file, reason: 'missing' });
|
||||
else if (recordedAt === null || changedAt > recordedAt) changed.push({ file, reason: 'modified' });
|
||||
}
|
||||
|
||||
const status = extractFrontmatter(content, uatPath)['status'];
|
||||
return {
|
||||
uat_file: uatPath,
|
||||
uat_status: typeof status === 'string' ? status.trim() : '',
|
||||
recorded_at: recordedAt === null ? '' : new Date(recordedAt).toISOString(),
|
||||
rows: scanUatRows(content.split('\n')).map(publicRow),
|
||||
checked_files: checked,
|
||||
changed_since_uat: changed,
|
||||
code_unchanged_since_uat: changed.length === 0,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── seed-uat ─────────────────────────────────────────────────────────────────
|
||||
|
||||
function renderRow(test: number, item: HumanItem): string[] {
|
||||
const expected = item.expected.split('\n').map((l) => l.trim()).filter((l) => l.length > 0);
|
||||
const lines = [`### ${test}. ${item.test}`];
|
||||
if (expected.length <= 1) lines.push(`expected: ${expected[0] ?? ''}`);
|
||||
else lines.push('expected: |', ...expected.map((l) => ` ${l}`));
|
||||
lines.push('result: [pending]', '');
|
||||
return lines;
|
||||
}
|
||||
|
||||
function countResults(rows: readonly UatRow[]): Record<string, number> {
|
||||
const count = (pred: (r: UatRow) => boolean): number => rows.filter(pred).length;
|
||||
return {
|
||||
total: rows.length,
|
||||
passed: count((r) => r.passing),
|
||||
issues: count((r) => r.result === 'issue'),
|
||||
pending: count((r) => r.result === 'pending'),
|
||||
skipped: count((r) => r.result === 'skipped'),
|
||||
blocked: count((r) => r.result === 'blocked'),
|
||||
};
|
||||
}
|
||||
|
||||
/** Rewrite the `## Summary` counters that already exist; never invent a section. */
|
||||
function refreshSummary(lines: string[], rows: readonly UatRow[]): void {
|
||||
const counts = countResults(rows);
|
||||
const start = lines.findIndex((l, i) => i >= bodyStart(lines) && /^##\s+Summary\s*$/.test(l));
|
||||
if (start === -1) return;
|
||||
for (let i = start + 1; i < lines.length && !SECTION_RE.test(lines[i]); i++) {
|
||||
const m = /^(total|passed|issues|pending|skipped|blocked):/.exec(lines[i]);
|
||||
if (m) lines[i] = `${m[1]}: ${counts[m[1]]}`;
|
||||
}
|
||||
}
|
||||
|
||||
/** A re-opened session is `testing` again; only the two frontmatter lines that say so change. */
|
||||
function reopenFrontmatter(lines: string[], nowIso: string): void {
|
||||
const end = bodyStart(lines);
|
||||
for (let i = 1; i < end - 1; i++) {
|
||||
if (/^status:/.test(lines[i])) lines[i] = 'status: testing';
|
||||
else if (/^updated:/.test(lines[i])) lines[i] = `updated: ${nowIso}`;
|
||||
}
|
||||
}
|
||||
|
||||
function newUatDocument(phaseLabel: string, sourceFile: string, items: readonly HumanItem[], nowIso: string): string {
|
||||
const first = items[0];
|
||||
const lines = [
|
||||
'---',
|
||||
'status: testing',
|
||||
`phase: ${phaseLabel}`,
|
||||
`source: [${sourceFile}]`,
|
||||
`started: ${nowIso}`,
|
||||
`updated: ${nowIso}`,
|
||||
'---',
|
||||
'',
|
||||
'## Current Test',
|
||||
'',
|
||||
'number: 1',
|
||||
`name: ${first.test}`,
|
||||
'expected: |',
|
||||
...first.expected.split('\n').map((l) => ` ${l.trim()}`),
|
||||
'awaiting: user response',
|
||||
'',
|
||||
'## Tests',
|
||||
'',
|
||||
];
|
||||
items.forEach((item, i) => lines.push(...renderRow(i + 1, item)));
|
||||
lines.push(
|
||||
'## Summary',
|
||||
'',
|
||||
`total: ${items.length}`,
|
||||
'passed: 0',
|
||||
'issues: 0',
|
||||
`pending: ${items.length}`,
|
||||
'skipped: 0',
|
||||
'blocked: 0',
|
||||
'',
|
||||
'## Gaps',
|
||||
'',
|
||||
);
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge the report's `human_verification` items into the phase's `*-UAT.md`.
|
||||
*
|
||||
* - no UAT file → create one, every item `[pending]`.
|
||||
* - item matches no row → appended as a new `[pending]` row.
|
||||
* - item matches a row → the row is KEPT exactly as recorded, unless the
|
||||
* row passes AND the item carries a `retest_reason`: then (and only then)
|
||||
* the row is reset to `[pending]`, with `previous_result` and the reason
|
||||
* written into the row. A pass is never reset without a recorded reason.
|
||||
* - row matches no item → kept. The verifier no longer asking for a check
|
||||
* is not a reason to forget that the human ran it.
|
||||
*
|
||||
* When nothing is appended or reset the file is not written at all.
|
||||
*
|
||||
* Returns `null` when the phase has no verification report to seed from.
|
||||
*/
|
||||
function seedUatFromVerification(phaseDir: string, opts: { now?: Date } = {}): SeedUatResult | null {
|
||||
const { uatFile, verificationFile } = resolvePhaseFiles(phaseDir);
|
||||
if (!verificationFile) return null;
|
||||
const report = readNormalized(path.join(phaseDir, verificationFile));
|
||||
if (report === null) return null;
|
||||
const items = readHumanItems(report);
|
||||
const nowIso = (opts.now ?? new Date()).toISOString();
|
||||
|
||||
if (!uatFile) {
|
||||
const result: SeedUatResult = {
|
||||
uat_file: '',
|
||||
created: false,
|
||||
changed: false,
|
||||
kept: [],
|
||||
appended: [],
|
||||
retested: [],
|
||||
pending: 0,
|
||||
all_passing: false,
|
||||
};
|
||||
if (items.length === 0) return result;
|
||||
const fileName = verificationFile.replace(/VERIFICATION\.md$/, 'UAT.md');
|
||||
const phase = extractFrontmatter(report)['phase'];
|
||||
const phaseLabel = typeof phase === 'string' && phase.trim() ? phase.trim() : path.basename(phaseDir);
|
||||
const uatPath = path.join(phaseDir, fileName);
|
||||
fs.writeFileSync(uatPath, newUatDocument(phaseLabel, verificationFile, items, nowIso), 'utf-8');
|
||||
return {
|
||||
...result,
|
||||
uat_file: uatPath,
|
||||
created: true,
|
||||
changed: true,
|
||||
appended: items.map((item, i) => ({ test: i + 1, name: item.test })),
|
||||
pending: items.length,
|
||||
};
|
||||
}
|
||||
|
||||
const uatPath = path.join(phaseDir, uatFile);
|
||||
const content = readNormalized(uatPath);
|
||||
if (content === null) return null;
|
||||
const lines = content.split('\n');
|
||||
const rows = scanUatRows(lines);
|
||||
const byKey = new Map<string, UatRow>();
|
||||
for (const row of rows) if (!byKey.has(matchKey(row.name))) byKey.set(matchKey(row.name), row);
|
||||
|
||||
const appended: SeedUatResult['appended'] = [];
|
||||
const retested: SeedUatResult['retested'] = [];
|
||||
const resetRows = new Set<UatRow>();
|
||||
const toAppend: HumanItem[] = [];
|
||||
for (const item of items) {
|
||||
const key = matchKey(item.test);
|
||||
const row = byKey.get(key);
|
||||
if (!row) {
|
||||
if (!toAppend.some((queued) => matchKey(queued.test) === key)) toAppend.push(item);
|
||||
} else if (row.passing && item.retest_reason.length > 0 && !resetRows.has(row)) {
|
||||
resetRows.add(row);
|
||||
retested.push({ test: row.test, name: row.name, previous_result: row.result, reason: item.retest_reason });
|
||||
}
|
||||
}
|
||||
|
||||
// Edit bottom-up so earlier line indexes stay valid.
|
||||
const nextTest = rows.reduce((max, r) => Math.max(max, r.test), 0) + 1;
|
||||
if (toAppend.length > 0) {
|
||||
const body = bodyStart(lines);
|
||||
const isSection = (re: RegExp) => (l: string, i: number) => i >= body && re.test(l);
|
||||
let insertAt = lines.findIndex(isSection(/^##\s+Summary\s*$/));
|
||||
if (insertAt === -1) insertAt = lines.findIndex(isSection(/^##\s+Gaps\s*$/));
|
||||
const lastRow = rows[rows.length - 1];
|
||||
if (insertAt === -1 || (lastRow && insertAt < lastRow.headingLine)) insertAt = lines.length;
|
||||
const block: string[] = [];
|
||||
if (insertAt > 0 && lines[insertAt - 1].trim() !== '') block.push('');
|
||||
toAppend.forEach((item, i) => {
|
||||
block.push(...renderRow(nextTest + i, item));
|
||||
appended.push({ test: nextTest + i, name: item.test });
|
||||
});
|
||||
lines.splice(insertAt, 0, ...block);
|
||||
}
|
||||
for (const row of [...resetRows].sort((a, b) => b.resultLine - a.resultLine)) {
|
||||
const reason = retested.find((r) => r.test === row.test && r.name === row.name)?.reason ?? '';
|
||||
lines.splice(
|
||||
row.resultLine,
|
||||
1,
|
||||
'result: [pending]',
|
||||
`previous_result: ${row.result}`,
|
||||
`retest_reason: ${quote(reason)}`,
|
||||
);
|
||||
}
|
||||
|
||||
const changed = appended.length > 0 || retested.length > 0;
|
||||
let finalRows = rows;
|
||||
if (changed) {
|
||||
finalRows = scanUatRows(lines);
|
||||
refreshSummary(lines, finalRows);
|
||||
reopenFrontmatter(lines, nowIso);
|
||||
fs.writeFileSync(uatPath, lines.join('\n'), 'utf-8');
|
||||
}
|
||||
return {
|
||||
uat_file: uatPath,
|
||||
created: false,
|
||||
changed,
|
||||
kept: rows.filter((r) => !resetRows.has(r)).map((r) => ({ test: r.test, name: r.name, result: r.result })),
|
||||
appended,
|
||||
retested,
|
||||
pending: finalRows.filter((r) => r.result === 'pending').length,
|
||||
all_passing: finalRows.length > 0 && finalRows.every((r) => r.passing),
|
||||
};
|
||||
}
|
||||
|
||||
// ─── canonicalize-uat ─────────────────────────────────────────────────────────
|
||||
|
||||
/** Frozen reason enum for `canonicalizeHumanNeeded` — callers branch on these, never on prose. */
|
||||
const CANONICALIZE_REASON = Object.freeze({
|
||||
UAT_PASSED: 'uat_passed',
|
||||
NOT_HUMAN_NEEDED: 'not_human_needed',
|
||||
UAT_NOT_PASSED: 'uat_not_passed',
|
||||
REPORT_UNWRITABLE: 'report_unwritable',
|
||||
} as const);
|
||||
type CanonicalizeReason = (typeof CANONICALIZE_REASON)[keyof typeof CANONICALIZE_REASON];
|
||||
|
||||
interface CanonicalizeResult {
|
||||
canonicalized: boolean;
|
||||
/** The verification status AFTER this call. */
|
||||
status: string;
|
||||
reason: CanonicalizeReason;
|
||||
/** The UAT rows that refused the flip (`uat_not_passed` only). */
|
||||
blockers: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Promote a `human_needed` report to `passed` when — and only when — the
|
||||
* human UAT passed (#4663).
|
||||
*
|
||||
* The status is read through `readVerificationStatus`, so staleness and
|
||||
* `gaps_found` outrank the raw frontmatter exactly as they do everywhere
|
||||
* else: a `stale` report is refused (`not_human_needed`) and must be
|
||||
* re-verified first. The UAT verdict is the SAME row predicate the
|
||||
* phase-close uses, in its `uat-only` form: at least one check passed and no
|
||||
* row is pending, blocked, failed, or skipped without a deferral reason. Zero
|
||||
* issues is not pass evidence on its own.
|
||||
*/
|
||||
function canonicalizeHumanNeeded(phaseDir: string): CanonicalizeResult {
|
||||
const status = (readVerificationStatus(phaseDir) as { status: string }).status;
|
||||
if (status !== 'human_needed') {
|
||||
return { canonicalized: false, status, reason: CANONICALIZE_REASON.NOT_HUMAN_NEEDED, blockers: [] };
|
||||
}
|
||||
const uat = evaluateUatPassed(phaseDir, { policy: { uatOnly: true } });
|
||||
if (!uat.passed) {
|
||||
return {
|
||||
canonicalized: false,
|
||||
status,
|
||||
reason: CANONICALIZE_REASON.UAT_NOT_PASSED,
|
||||
blockers: uat.blockers,
|
||||
};
|
||||
}
|
||||
const { verificationFile } = resolvePhaseFiles(phaseDir);
|
||||
try {
|
||||
const reportPath = path.join(phaseDir, verificationFile as string);
|
||||
const content = fs.readFileSync(reportPath, 'utf-8');
|
||||
const fm = extractFrontmatter(content, reportPath);
|
||||
fm['status'] = 'passed';
|
||||
fs.writeFileSync(reportPath, spliceFrontmatter(content, fm), 'utf-8');
|
||||
} catch {
|
||||
return { canonicalized: false, status, reason: CANONICALIZE_REASON.REPORT_UNWRITABLE, blockers: [] };
|
||||
}
|
||||
return { canonicalized: true, status: 'passed', reason: CANONICALIZE_REASON.UAT_PASSED, blockers: [] };
|
||||
}
|
||||
|
||||
// ─── CLI handlers ─────────────────────────────────────────────────────────────
|
||||
|
||||
function resolvePhaseDirArg(cwd: string, phaseDirArg: string | undefined, verb: string): string | null {
|
||||
if (!phaseDirArg) {
|
||||
error(`phase directory required for ${verb}`);
|
||||
return null;
|
||||
}
|
||||
const phaseDir = path.resolve(cwd, phaseDirArg);
|
||||
let isDir = false;
|
||||
try {
|
||||
isDir = fs.statSync(phaseDir).isDirectory();
|
||||
} catch {
|
||||
// not found → not a directory
|
||||
}
|
||||
if (!isDir) {
|
||||
error(`phase directory not found: ${phaseDirArg}`);
|
||||
return null;
|
||||
}
|
||||
return phaseDir;
|
||||
}
|
||||
|
||||
/**
|
||||
* `verification.uat-evidence <phaseDir> [files… | --files a,b]` — read-only.
|
||||
* With no files, compares the current report's own `covered_files`.
|
||||
*/
|
||||
function cmdVerificationUatEvidence(
|
||||
cwd: string,
|
||||
phaseDirArg: string | undefined,
|
||||
fileArgs: readonly string[],
|
||||
raw: boolean,
|
||||
): void {
|
||||
const phaseDir = resolvePhaseDirArg(cwd, phaseDirArg, 'verification.uat-evidence');
|
||||
if (phaseDir === null) return;
|
||||
const parsed = parseFingerprintFileArgs(fileArgs);
|
||||
if ('error' in parsed) {
|
||||
error(parsed.error.replace(/verification\.fingerprint/g, 'verification.uat-evidence'));
|
||||
return;
|
||||
}
|
||||
output(readUatEvidence(phaseDir, parsed.files.length > 0 ? parsed.files : undefined), raw);
|
||||
}
|
||||
|
||||
/** `verification.seed-uat <phaseDir>` — merge the report's human items into `*-UAT.md`. */
|
||||
function cmdVerificationSeedUat(cwd: string, phaseDirArg: string | undefined, raw: boolean): void {
|
||||
const phaseDir = resolvePhaseDirArg(cwd, phaseDirArg, 'verification.seed-uat');
|
||||
if (phaseDir === null) return;
|
||||
const result = seedUatFromVerification(phaseDir);
|
||||
if (result === null) {
|
||||
error('no readable *-VERIFICATION.md in the phase directory — verification.seed-uat merges that report\'s human_verification items');
|
||||
return;
|
||||
}
|
||||
output(result, raw, result.uat_file);
|
||||
}
|
||||
|
||||
/** `verification.canonicalize-uat <phaseDir>` — record a passed UAT against a `human_needed` report. */
|
||||
function cmdVerificationCanonicalizeUat(cwd: string, phaseDirArg: string | undefined, raw: boolean): void {
|
||||
const phaseDir = resolvePhaseDirArg(cwd, phaseDirArg, 'verification.canonicalize-uat');
|
||||
if (phaseDir === null) return;
|
||||
const result = canonicalizeHumanNeeded(phaseDir);
|
||||
output(result, raw, result.status);
|
||||
}
|
||||
|
||||
export = {
|
||||
CANONICALIZE_REASON,
|
||||
canonicalizeHumanNeeded,
|
||||
cmdVerificationCanonicalizeUat,
|
||||
scanUatRows,
|
||||
readHumanItems,
|
||||
readUatEvidence,
|
||||
seedUatFromVerification,
|
||||
cmdVerificationUatEvidence,
|
||||
cmdVerificationSeedUat,
|
||||
};
|
||||
@@ -21,8 +21,16 @@ interface VerificationModule {
|
||||
cmdVerificationFingerprint(cwd: string, phaseDirArg: string | undefined, files: string[], raw: boolean): void;
|
||||
}
|
||||
|
||||
/** UAT-evidence verbs live in their own module (it imports `verification`, not the reverse). */
|
||||
interface UatEvidenceModule {
|
||||
cmdVerificationUatEvidence(cwd: string, phaseDirArg: string | undefined, files: string[], raw: boolean): void;
|
||||
cmdVerificationSeedUat(cwd: string, phaseDirArg: string | undefined, raw: boolean): void;
|
||||
cmdVerificationCanonicalizeUat(cwd: string, phaseDirArg: string | undefined, raw: boolean): void;
|
||||
}
|
||||
|
||||
interface RouteVerificationCommandOptions {
|
||||
verification: VerificationModule;
|
||||
uatEvidence: UatEvidenceModule;
|
||||
args: string[];
|
||||
cwd: string;
|
||||
raw: boolean;
|
||||
@@ -31,10 +39,11 @@ interface RouteVerificationCommandOptions {
|
||||
|
||||
// ─── Implementation ───────────────────────────────────────────────────────────
|
||||
|
||||
const VERIFICATION_SUBCOMMANDS = ['status', 'resolve-file', 'fingerprint'];
|
||||
const VERIFICATION_SUBCOMMANDS = ['status', 'resolve-file', 'fingerprint', 'uat-evidence', 'seed-uat', 'canonicalize-uat'];
|
||||
|
||||
function routeVerificationCommand({
|
||||
verification,
|
||||
uatEvidence,
|
||||
args,
|
||||
cwd,
|
||||
raw,
|
||||
@@ -51,6 +60,9 @@ function routeVerificationCommand({
|
||||
status: () => verification.cmdVerificationStatus(cwd, args[2], raw),
|
||||
'resolve-file': () => verification.cmdVerificationResolveFile(cwd, args[2], raw),
|
||||
fingerprint: () => verification.cmdVerificationFingerprint(cwd, args[2], args.slice(3), raw),
|
||||
'uat-evidence': () => uatEvidence.cmdVerificationUatEvidence(cwd, args[2], args.slice(3), raw),
|
||||
'seed-uat': () => uatEvidence.cmdVerificationSeedUat(cwd, args[2], raw),
|
||||
'canonicalize-uat': () => uatEvidence.cmdVerificationCanonicalizeUat(cwd, args[2], raw),
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
@@ -118,7 +118,11 @@ const VERIFICATION_ROUTING_TABLE: Record<string, VerificationRoute> = {
|
||||
// here was an advice loop. execute-phase resumes at the verification
|
||||
// gates and re-runs the verifier (its resume tree routes a stale report
|
||||
// to re-verification), which regenerates VERIFICATION.md and its digest.
|
||||
next_action: 'Verification is stale — covered source files changed after the verifier last ran. Re-run execute-phase for this phase: it resumes at the verification gates and re-runs the verifier, regenerating VERIFICATION.md and its digest. verify-work alone cannot refresh a stale report.',
|
||||
// the action is named for what it is — a verifier re-run, not an
|
||||
// execution. No plan re-runs, and recorded UAT results are kept: the
|
||||
// verifier reads them (`verification.uat-evidence`) and the UAT writer
|
||||
// merges instead of overwriting (`verification.seed-uat`).
|
||||
next_action: 'Verification is stale — the report no longer matches the phase. Re-run the verifier: execute-phase runs only plans that have no SUMMARY (none, when every plan is summarized), then resumes at the verification gates and regenerates VERIFICATION.md and its digest. Recorded UAT results are kept, not re-requested.',
|
||||
next_command: 'execute-phase',
|
||||
},
|
||||
// INTERNAL SENTINEL: constructed when no *-VERIFICATION.md file exists or when
|
||||
@@ -865,8 +869,42 @@ interface VerificationStatusResult {
|
||||
* `gaps_found` short-circuit above it, or no verification file at all).
|
||||
*/
|
||||
staleCheckIndeterminate?: boolean;
|
||||
/**
|
||||
* WHY a `stale` result is stale — present only when `status` is
|
||||
* `stale`. Every reason still means "re-run the verifier"; the split lets
|
||||
* a caller say what happened (a gap-closure plan the report never saw is
|
||||
* not the same event as a covered source file drifting under a finished
|
||||
* phase) and lets progress batch the drift case across phases.
|
||||
*/
|
||||
stale_reason?: StaleReason;
|
||||
/**
|
||||
* the report's own frontmatter `status` underneath a `stale` result —
|
||||
* present only when `status` is `stale`. `human_needed` here means a UAT
|
||||
* is still owed (or already recorded) once the verifier has re-run.
|
||||
*/
|
||||
raw_status?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* the closed set of reasons a report reads `stale`.
|
||||
* - `fingerprint_malformed` — the `covered_files`/`covered_digest` pair is
|
||||
* incomplete, wrong-shaped, or names an unknown version (fail closed, #4155).
|
||||
* - `covered_changed` — a declared covered file's bytes changed, or
|
||||
* the file is gone.
|
||||
* - `uncovered_artifacts` — every declared file still matches, but the
|
||||
* phase directory holds a PLAN/SUMMARY the report never declared (the
|
||||
* shape every gap-closure cycle produces).
|
||||
* - `summary_newer` — legacy (pre-#4155) report with a SUMMARY
|
||||
* changed after it.
|
||||
*/
|
||||
type StaleReason = 'fingerprint_malformed' | 'covered_changed' | 'uncovered_artifacts' | 'summary_newer';
|
||||
const STALE_REASONS: ReadonlyArray<StaleReason> = [
|
||||
'fingerprint_malformed',
|
||||
'covered_changed',
|
||||
'uncovered_artifacts',
|
||||
'summary_newer',
|
||||
];
|
||||
|
||||
function findStaleVerificationSummary(
|
||||
phaseDir: string,
|
||||
fsImpl: FsLike = defaultFsImpl,
|
||||
@@ -1096,6 +1134,7 @@ function readVerificationStatus(
|
||||
|
||||
let staleCheckIndeterminate = false;
|
||||
let isStale: boolean;
|
||||
let staleReason: StaleReason = 'summary_newer';
|
||||
if (declaresFingerprint) {
|
||||
// Stated directly rather than relying on `null !== coveredDigestVal`
|
||||
// being true whenever the pair is malformed: `!hasWellFormedFingerprint`
|
||||
@@ -1115,11 +1154,22 @@ function readVerificationStatus(
|
||||
hasWellFormedFingerprint && typeof coveredDigestVal === 'string'
|
||||
? parseFingerprintVersion(coveredDigestVal)
|
||||
: null;
|
||||
isStale =
|
||||
!hasWellFormedFingerprint ||
|
||||
storedVersion === null ||
|
||||
computeCoveredDigest(findProjectRoot(phaseDir), coveredFilesVal, storedVersion, { phaseDir }) !== coveredDigestVal ||
|
||||
!allCurrentArtifactsCovered(phaseDir, coveredFilesVal);
|
||||
// same three checks, same order, same short-circuit — only named, so
|
||||
// the result can say which one fired.
|
||||
if (!hasWellFormedFingerprint || storedVersion === null) {
|
||||
isStale = true;
|
||||
staleReason = 'fingerprint_malformed';
|
||||
} else if (
|
||||
computeCoveredDigest(findProjectRoot(phaseDir), coveredFilesVal, storedVersion, { phaseDir }) !== coveredDigestVal
|
||||
) {
|
||||
isStale = true;
|
||||
staleReason = 'covered_changed';
|
||||
} else if (!allCurrentArtifactsCovered(phaseDir, coveredFilesVal)) {
|
||||
isStale = true;
|
||||
staleReason = 'uncovered_artifacts';
|
||||
} else {
|
||||
isStale = false;
|
||||
}
|
||||
} else {
|
||||
const staleCheck = findStaleVerificationSummary(
|
||||
phaseDir,
|
||||
@@ -1144,6 +1194,8 @@ function readVerificationStatus(
|
||||
// the verifier, regenerating VERIFICATION.md and its digest — the same
|
||||
// routing the `missing` sentinel has used since #2868.
|
||||
next_command: projectNextCommand('execute-phase', runtime, phaseArg),
|
||||
stale_reason: staleReason,
|
||||
raw_status: rawStatus,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1470,6 +1522,8 @@ function cmdVerificationFingerprint(
|
||||
export = {
|
||||
VERIFIER_STATUSES,
|
||||
VERIFICATION_ROUTING_TABLE,
|
||||
STALE_REASONS,
|
||||
canonicalizeCoveredFiles,
|
||||
defaultPhaseCleanCommitTimesMs,
|
||||
resolveVerificationFile,
|
||||
resolveUatFile,
|
||||
|
||||
Reference in New Issue
Block a user