feat(#247): runtime-neutral phase uat-passed predicate from HUMAN-UAT results (#1063)

* feat(#247): runtime-neutral phase uat-passed predicate from HUMAN-UAT results

Wire the already-reserved `phase.uat-passed` alias (subcommand `uat-passed`,
mutation:false) into the phase command router with a new markdown-aware
predicate that evaluates HUMAN-UAT results and reports pass only when every
required check passes. Post-SDK-retirement (ADR-0174/#174) successor to the
SDK-framed #70, with no SDK-specific API surface.

New pure module src/uat-predicate.cts:
- stripFalsePositiveContexts: frontmatter -> HTML-comment -> CommonMark-style
  fenced-block state machine (tracks delimiter char+length) -> blockquote,
  each a small composable step, so a `result: passed` inside frontmatter, a
  fenced/~~~ block (incl. ~~~ nested in a ``` fence), a comment, or a
  blockquote is never counted.
- parseUatResultItems: heading-block parser, column-0-anchored same-line
  result; a heading with no result -> `missing` (fail-closed).
- analyzeMarkdown: unterminated fence/comment detection (malformed -> blocker).
- evaluateUatPassed: allowlist pass/verification semantics; passed = no
  blockers && >=1 check && all passing; no_uat_artifacts discriminator (no
  vacuous pass); optional requireVerification policy hook.

Thin cmdPhaseUatPassed handler in phase.cts; router closure rejects unknown
flags via makeInvalidArgs. Hardened across two Codex adversarial passes
(vacuous pass, dropped failing tests, permissive verification status,
nested-fence escape, cross-line result value, masked unterminated comment) —
all fixed fail-closed. New unit + CLI-integration suites incl. a fast-check
property test; docs, CONTEXT glossary, inventory, and changeset updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(#247): backfill changeset PR number (#1063)

* fix(#247): indexOf paired-scan for unterminated-comment detection

CodeQL js/incomplete-multi-character-sanitization (high) flagged the
`raw.replace(/<!--[\s\S]*?-->/g,'')`-then-`.includes('<!--')` detection in
analyzeMarkdown as incomplete sanitization (a single regex pass can leave a
residual `<!--`). Replace it with a paired left-to-right indexOf scan that
contains no `.replace()` of the comment token — CodeQL-clean and strictly
more correct (a closed earlier comment can never mask a later unterminated
one). Behaviour unchanged; 98 predicate tests + scoped docker run green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tom Boucher
2026-06-11 16:37:17 -04:00
committed by GitHub
parent 8813ee5f95
commit 9223f2f4c8
14 changed files with 2081 additions and 1 deletions

View File

@@ -0,0 +1,5 @@
---
type: Added
pr: 1063
---
**`phase uat-passed` predicate** — new runtime-neutral command evaluates HUMAN-UAT results with markdown-aware parsing (ignores frontmatter, fenced code, blockquotes, and HTML comments) and reports pass only when every required check passes. (#1063)

1
.gitignore vendored
View File

@@ -173,6 +173,7 @@ build/
/gsd-core/bin/lib/verify.cjs
/gsd-core/bin/lib/init.cjs
/gsd-core/bin/lib/uat.cjs
/gsd-core/bin/lib/uat-predicate.cjs
/gsd-core/bin/lib/workstream.cjs
/gsd-core/bin/lib/roadmap.cjs
/gsd-core/bin/lib/audit.cjs

View File

@@ -198,6 +198,9 @@ The GSD-RESEARCH capability behind an L2-hybrid seam: code owns cache + provider
- `GSD-RESEARCH.CONTEXT-DISCIPLINE=less-context levers: subagent isolation + compact provider output + fetches-to-disk + cache-returns-digest; API clear_tool_uses/memory tool are the conceptual model, not a Claude Code harness knob`
- `DEFECT.RESEARCH-PROVIDER-PROSE-DRIFT=provider waterfall duplicated across N researcher agent .md files drifts independently (META.RULE.brief-no-paraphrase); fix-forward=research-provider.cjs single source of truth + generated agents (#657)`
### UAT-Passed Predicate
Runtime-neutral predicate evaluating `*-UAT.md` / `*-VERIFICATION.md` result fields with markdown-aware parsing that ignores false-positive contexts (frontmatter body, fenced code, HTML comments, blockquotes). Returns `passed: true` only when all required checks pass; supports `--require-verification` to demand at least one VERIFICATION.md file alongside UAT results. Output envelope: `{ passed, uat_files[], verification_files[], checks[], blockers[], policy }`. Source: `gsd-core/bin/lib/uat-predicate.cjs` (generated from `src/uat-predicate.cts`). Wired via `phase uat-passed` alias → `phase-command-router` → `cmdPhaseUatPassed`.
### MVP Mode
Phase-level planning mode that frames work as a vertical slice (UI → API → DB) of one user-visible capability instead of horizontal layers. Resolved at workflow init via the precedence chain: `--mvp` CLI flag → ROADMAP.md `**Mode:** mvp` field → `workflow.mvp_mode` config → false. All-or-nothing per phase (PRD #2826 Q1). Surfaced as `MVP_MODE=true|false` to the planner, executor, verifier, and discovery surfaces (progress, stats, graphify). Canonical parser: `roadmap.cjs` `**Mode:**` field; canonical resolution chain documented in `workflows/plan-phase.md`. Concept index: `references/mvp-concepts.md`.

View File

@@ -118,6 +118,10 @@ node gsd-tools.cjs phase remove <phase> [--force]
# Mark phase complete, update state + roadmap
node gsd-tools.cjs phase complete <phase>
# Evaluate HUMAN-UAT results for a phase (markdown-aware; ignores false-positive contexts)
# Returns JSON: { passed, uat_files[], verification_files[], checks[], blockers[], policy }
node gsd-tools.cjs phase uat-passed <phase> [--require-verification]
# Index plans with waves and status
node gsd-tools.cjs phase-plan-index <phase>

View File

@@ -490,6 +490,37 @@ Retroactively audit and fill Nyquist validation gaps.
---
### `phase uat-passed <N> [--require-verification]`
Runtime-neutral predicate that evaluates HUMAN-UAT results for a phase and reports whether all required checks passed. Uses markdown-aware parsing that ignores false-positive contexts (YAML frontmatter, fenced code blocks, HTML comments, and blockquotes), so incomplete checkbox fragments in prose sections never trigger a false pass.
| Argument | Required | Description |
|----------|----------|-------------|
| `N` | **Yes** | Phase number to evaluate |
| `--require-verification` | No | Require at least one `*-VERIFICATION.md` file alongside UAT results; fails if none are found |
**Output fields (JSON):**
| Field | Type | Description |
|-------|------|-------------|
| `passed` | `boolean` | `true` only when at least one check exists AND all checks pass AND no blockers — fail-closed (no vacuous pass) |
| `uat_files` | `string[]` | Filenames of `*-UAT.md` files evaluated |
| `verification_files` | `string[]` | Filenames of `*-VERIFICATION.md` files evaluated |
| `checks[]` | `{ file, test, name, result, passing }[]` | Per-item evaluation results parsed from heading blocks |
| `blockers[]` | `string[]` | Human-readable reasons for failure (frontmatter issues, failing/missing test items, policy violations, malformed markdown) — NOT a subset of `checks[]` |
| `no_uat_artifacts` | `boolean` | `true` when no real UAT test items were parsed (no `*-UAT.md` files, unreadable dir, or files with no test blocks); when `true`, `passed` is always `false` |
| `policy.require_verification` | `boolean` | Whether `--require-verification` was active |
**Programmatic access:** `node gsd-tools.cjs phase uat-passed <N> [--require-verification] [--raw]` — see [CLI Tools Reference](CLI-TOOLS.md)
```bash
node gsd-tools.cjs phase uat-passed 3 # Evaluate UAT for phase 3
node gsd-tools.cjs phase uat-passed 3 --require-verification # Also require VERIFICATION.md
node gsd-tools.cjs phase uat-passed 3 --raw # Machine-readable JSON output
```
---
## Navigation Commands
### `/gsd-progress`

View File

@@ -164,6 +164,7 @@
- [Statusline Context Position](#140-statusline-context-position)
- [Milestone Tag Creation Toggle](#141-milestone-tag-creation-toggle)
- [Structured JSON Error Mode](#142-structured-json-error-mode)
- [UAT-Passed Predicate](#143-uat-passed-predicate)
---
@@ -3046,6 +3047,38 @@ explicit reviewer flags -> --all -> review.default_reviewers -> all detected rev
---
### 143. UAT-Passed Predicate
**CLI:** `node gsd-tools.cjs phase uat-passed <N> [--require-verification]`
**Purpose:** Provide a runtime-neutral, automatable predicate that evaluates HUMAN-UAT results for a phase and returns a structured pass/fail verdict with full diagnostic detail.
**Behavior:** Locates `*-UAT.md` and optionally `*-VERIFICATION.md` files for the given phase, parses UAT test blocks (heading-block parser, column-0 result lines) with a markdown-aware stripper that removes false-positive contexts (YAML frontmatter, fenced code blocks, HTML comments, and blockquotes). Returns `passed: true` only when at least one check exists AND all checks pass AND no blockers — fail-closed, no vacuous pass. The `--require-verification` flag requires at least one `*-VERIFICATION.md` with an allowlisted passing status; the command fails without one.
**Output envelope:** `{ passed, uat_files[], verification_files[], checks[], blockers[], no_uat_artifacts, policy: { require_verification } }`
| Field | Type | Description |
|-------|------|-------------|
| `passed` | `boolean` | `true` only when ≥1 check exists AND all passing AND no blockers |
| `uat_files` | `string[]` | Filenames of `*-UAT.md` files evaluated |
| `verification_files` | `string[]` | Filenames of `*-VERIFICATION.md` files evaluated |
| `checks[]` | `{ file, test, name, result, passing }[]` | Per-item results from heading blocks |
| `blockers[]` | `string[]` | Human-readable failure reasons (frontmatter, failing/missing items, policy, malformed markdown) |
| `no_uat_artifacts` | `boolean` | `true` when no test items were parsed; `passed` is always `false` when `true` |
| `policy.require_verification` | `boolean` | Whether `--require-verification` was active |
**Requirements:**
- REQ-UAT-PRED-01: The predicate MUST ignore result lines inside YAML frontmatter, fenced code blocks, HTML comments, and blockquotes.
- REQ-UAT-PRED-02: `passed: true` MUST require at least one check AND all checks passing AND no blockers (fail-closed, no vacuous pass).
- REQ-UAT-PRED-03: `--require-verification` MUST cause the command to fail when no `*-VERIFICATION.md` file with an allowlisted passing status is found.
- REQ-UAT-PRED-04: `blockers[]` contains all human-readable failure reasons including frontmatter issues, policy violations, and malformed markdown — NOT limited to a subset of `checks[]`.
- REQ-UAT-PRED-05: The module MUST be runtime-neutral (no runtime-specific env checks or exit shortcuts).
- REQ-UAT-PRED-06: A heading block with no column-0 `result:` line emits `result:'missing'` (blocker); test items are never silently dropped.
**Reference:** [Phase Management Commands](COMMANDS.md#phase-uat-passed-n---require-verification)
---
## Related
- [Commands](COMMANDS.md)

View File

@@ -356,6 +356,7 @@
"surface.cjs",
"task-command-router.cjs",
"template.cjs",
"uat-predicate.cjs",
"uat.cjs",
"ui-safety-gate.cjs",
"update-context.cjs",

View File

@@ -371,7 +371,7 @@ The `gsd-planner` agent is decomposed into a core agent plus reference modules t
---
## CLI Modules (105 shipped)
## CLI Modules (106 shipped)
Full listing: `gsd-core/bin/lib/*.cjs`.
@@ -468,6 +468,7 @@ Full listing: `gsd-core/bin/lib/*.cjs`.
| `task-command-router.cjs` | Thin CJS subcommand router adapter for `gsd-tools task` |
| `template.cjs` | Template selection and filling with variable substitution |
| `uat.cjs` | UAT file parsing, verification debt tracking, audit-uat support |
| `uat-predicate.cjs` | UAT-passed predicate — markdown-aware evaluation of HUMAN-UAT results; returns pass only when all required checks pass; ignores false-positive contexts (frontmatter, fenced code, blockquotes, HTML comments) |
| `ui-safety-gate.cjs` | Shell-free word-boundary UI token detector (#3706, #3718); reads phase-section text from stdin, exits 0 (UI found) or 1 (no UI); also deployed to `gsd-core/bin/lib/` so the GSD installer ships it to `$RUNTIME_DIR` (#448) |
| `update-context.cjs` | Pure install-context resolver for `/gsd:update` — runtime/scope/config-dir/version detection (LOCAL/GLOBAL/UNKNOWN) ported from update.md bash; backs `gsd-tools update-context` (#498) |
| `validate-command-router.cjs` | Thin CJS subcommand router adapter for `gsd-tools validate` |

View File

@@ -137,6 +137,7 @@ export default tseslint.config(
'gsd-core/bin/lib/profile-pipeline.cjs',
'gsd-core/bin/lib/template.cjs',
'gsd-core/bin/lib/uat.cjs',
'gsd-core/bin/lib/uat-predicate.cjs',
'gsd-core/bin/lib/workstream.cjs',
'gsd-core/bin/lib/roadmap.cjs',
'gsd-core/bin/lib/audit.cjs',

View File

@@ -32,6 +32,7 @@ interface PhaseHandlers {
cmdPhaseInsert: (cwd: string, pos: string | undefined, desc: string, raw: boolean) => void;
cmdPhaseRemove: (cwd: string, phaseNum: string, opts: { force: boolean }, raw: boolean) => void;
cmdPhaseComplete: (cwd: string, phaseNum: string | undefined, raw: boolean) => void;
cmdPhaseUatPassed: (cwd: string, phaseNum: string | undefined, raw: boolean, opts?: { policy?: { requireVerification?: boolean } }) => void;
}
interface RoutePhaseCommandOptions {
@@ -164,6 +165,23 @@ function routePhaseCommand({ phase, args, cwd, raw, error }: RoutePhaseCommandOp
phase.cmdPhaseComplete(cwd, args[2], raw);
return { ok: true as const, data: null };
},
'uat-passed': (_ctx: Record<string, unknown>): { ok: true; data: null } => {
let requireVerification = false;
const positional: string[] = [];
for (const token of args.slice(2)) {
if (token === '--require-verification') {
requireVerification = true;
} else if (token === '--raw') {
// --raw is handled by the outer CLI layer; accepted here silently
} else if (token.startsWith('--')) {
return makeInvalidArgs(token, `phase uat-passed does not support ${token}`) as never;
} else {
positional.push(token);
}
}
phase.cmdPhaseUatPassed(cwd, positional[0], raw, { policy: { requireVerification } });
return { ok: true as const, data: null };
},
},
};

View File

@@ -29,6 +29,9 @@ import stateMod = require('./state.cjs');
import { platformWriteSync, platformReadSync, platformEnsureDir } from './shell-command-projection.cjs';
import { formatGsdSlash, resolveRuntime } from './runtime-slash.cjs';
import { deriveProgressFromRoadmap, clampPercent } from './phase-lifecycle.cjs';
// eslint-disable-next-line @typescript-eslint/no-require-imports -- uat-predicate.cjs is an export= CommonJS module
import uatPredicate = require('./uat-predicate.cjs');
const { evaluateUatPassed } = uatPredicate;
const {
escapeRegex,
@@ -1712,6 +1715,28 @@ function cmdPhaseComplete(cwd: string, phaseNum: string, raw: boolean): void {
output(result, raw);
}
function cmdPhaseUatPassed(
cwd: string,
phaseNum: string | undefined,
raw: boolean,
opts: { policy?: { requireVerification?: boolean } } = {},
): void {
if (!phaseNum) {
error('phase number required for phase uat-passed');
}
const phaseInfoRaw = findPhaseInternal(cwd, phaseNum!);
if (!phaseInfoRaw) {
error(`Phase ${phaseNum} not found`);
}
const phaseInfo = phaseInfoRaw as unknown as Record<string, unknown>;
const phaseFullDir = path.join(cwd, phaseInfo['directory'] as string);
const report = evaluateUatPassed(phaseFullDir, { policy: opts.policy });
output({ phase: phaseNum, ...report }, raw);
}
export = {
cmdPhasesList,
cmdPhaseNextDecimal,
@@ -1723,5 +1748,6 @@ export = {
cmdPhaseInsert,
cmdPhaseRemove,
cmdPhaseComplete,
cmdPhaseUatPassed,
computeDependencyLevels,
};

394
src/uat-predicate.cts Normal file
View File

@@ -0,0 +1,394 @@
/**
* UAT Predicate — Pure-computation UAT pass/fail evaluation
*
* Evaluates all *-UAT.md and *-VERIFICATION.md files in a phase directory and
* returns a typed report. Used by `phase uat-passed` to harden against the
* naive whole-file regex in cmdPhaseComplete which false-matches `result:` lines
* inside frontmatter, fenced code blocks, blockquotes, and HTML comments.
*
* Issue #247 — phase uat-passed predicate
*
* ADR-457 build-at-publish: compiled by tsc to gsd-core/bin/lib/uat-predicate.cjs.
*/
import fs from 'node:fs';
import path from 'node:path';
// eslint-disable-next-line @typescript-eslint/no-require-imports
import frontmatter = require('./frontmatter.cjs');
const { extractFrontmatter } = frontmatter;
// ─── Types ────────────────────────────────────────────────────────────────────
interface UatCheckItem {
file: string;
test: number;
name: string;
result: string;
passing: boolean;
}
interface UatPassedReport {
passed: boolean;
uat_files: string[];
verification_files: string[];
checks: UatCheckItem[];
blockers: string[];
no_uat_artifacts: boolean;
policy: {
require_verification: boolean;
};
}
// ─── Blocking state sets (documented for maintainability) ─────────────────────
// UAT file frontmatter `status` values that indicate the file is not fully done
const BLOCKING_UAT_FM_STATUSES = new Set([
'partial', 'diagnosed', 'pending', 'blocked', 'in_progress', 'failed',
]);
// UAT file frontmatter `result` values that indicate failure
const BLOCKING_UAT_FM_RESULTS = new Set(['pending', 'blocked', 'failed']);
// VERIFICATION file frontmatter `status` values that indicate passing
const PASSING_VERIFICATION_STATUSES = new Set([
'complete', 'verified', 'passed', 'human_passed',
]);
// VERIFICATION file frontmatter `status` values that explicitly block
const BLOCKING_VERIFICATION_FM_STATUSES = new Set([
'human_needed', 'gaps_found', 'pending', 'blocked', 'partial',
'failed', 'in_progress',
]);
// UAT test-item `result` values that count as passing
const PASSING_RESULTS = new Set(['passed', 'pass']);
// ─── stripFalsePositiveContexts ───────────────────────────────────────────────
/**
* Remove contexts that can contain `result: ...` lines that are NOT real test results:
* (a) leading frontmatter block at byte 0
* (b) HTML comments (unterminated comments swallow to EOF — fail-closed)
* (c) fenced code blocks (backtick and tilde, indented too) via CommonMark state machine
* (d) blockquote lines
*
* Each step is a composable function (Kernighan's Law — independently testable).
* Returns surviving lines joined by '\n'. Robust to CRLF input.
*/
function stripFalsePositiveContexts(content: string): string {
// Step (a): strip leading frontmatter block only at byte 0
let stripped = content.replace(/^---\r?\n[\s\S]*?\r?\n---[ \t]*(\r?\n|$)/, '');
// Step (b): remove HTML comments anywhere; unterminated comment swallows to EOF
stripped = stripped.replace(/<!--[\s\S]*?(?:-->|$)/g, '');
// Step (c): remove fenced code blocks via CommonMark-style state machine (handles CRLF + indented fences)
stripped = _stripFencedBlocks(stripped).text;
// Step (d): remove blockquote lines
stripped = stripped
.split('\n')
.filter(line => !/^\s*>/.test(line))
.join('\n');
return stripped;
}
interface FenceState {
char: '`' | '~';
len: number;
}
interface StripFencedResult {
text: string;
unterminatedFence: boolean;
}
/**
* CommonMark-style fenced-code-block stripper.
* Tracks the opening delimiter char and length so that a ~~~ line inside a
* ``` fence is correctly treated as fence content, not a closing delimiter.
*
* Opening rule: first delimiter line with char+len sets openFence.
* Closing rule: delimiter line with SAME char, run length >= openFence.len,
* and NO trailing non-whitespace text closes the fence.
* All delimiter and content lines are dropped; non-fence lines are kept.
* Returns the kept text plus unterminatedFence:true if EOF inside a fence.
*/
function _stripFencedBlocks(content: string): StripFencedResult {
const lines = content.split('\n');
const kept: string[] = [];
let openFence: FenceState | null = null;
const delimRe = /^(\s*)(`{3,}|~{3,})(.*)$/;
for (const rawLine of lines) {
// Tolerate CRLF: strip trailing \r for matching, but we work on split-by-\n lines
// (the outer caller joined by \n already; we just handle a stray \r in the last char)
const line = rawLine.replace(/\r$/, '');
const m = delimRe.exec(line);
if (m) {
const char = m[2][0] as '`' | '~';
const len = m[2].length;
const trailing = m[3];
if (openFence === null) {
// Opening delimiter — drop this line and record the fence
openFence = { char, len };
} else if (char === openFence.char && len >= openFence.len && /^\s*$/.test(trailing)) {
// Closing delimiter (same char, sufficient length, no trailing text) — drop and close
openFence = null;
}
// else: mismatched delimiter inside fence (e.g. ~~~ inside ```) — drop as content
continue; // delimiter lines are always dropped
}
if (openFence === null) {
kept.push(rawLine);
}
// Lines inside fence are dropped
}
return { text: kept.join('\n'), unterminatedFence: openFence !== null };
}
/**
* Analyse raw markdown for structural anomalies (unterminated fence / comment).
* Exported for unit-testability and used by evaluateUatPassed for per-file malformed detection.
*
* FIX C: properly balanced comments are stripped before checking for a dangling <!--,
* so an earlier closed comment does not mask a later unterminated one.
*/
function analyzeMarkdown(raw: string): { unterminatedFence: boolean; unterminatedComment: boolean } {
// Detect an unterminated HTML comment via a paired scan: every `<!--` must
// have a following `-->`. Using indexOf (not a regex .replace of the comment
// token) avoids the js/incomplete-multi-character-sanitization pattern — and
// is exact: a closed earlier comment never masks a later unterminated one.
let unterminatedComment = false;
for (let i = 0; ; ) {
const open = raw.indexOf('<!--', i);
if (open === -1) break;
const close = raw.indexOf('-->', open + 4);
if (close === -1) { unterminatedComment = true; break; }
i = close + 3;
}
// Fence state machine gives the accurate unterminated-fence signal.
const { unterminatedFence } = _stripFencedBlocks(raw);
return { unterminatedFence, unterminatedComment };
}
// ─── parseUatResultItems ──────────────────────────────────────────────────────
/**
* HEADING-BLOCK parser: scan the CLEANED body (after stripFalsePositiveContexts)
* for UAT test blocks.
*
* For each ### N. Name heading, the block spans until the next ### heading or EOF.
* Within each block, find a column-0 anchored result line (rejects indented YAML
* block-scalar bodies and inline/quoted fakes).
*
* - If a heading block has NO column-0 result line → emit result:'missing' (blocker).
* - Support bracketed [passed] and bare passed (#2273).
* - Returns ALL items (both passing and non-passing).
*/
function parseUatResultItems(cleanContent: string): Array<{ test: number; name: string; result: string }> {
const items: Array<{ test: number; name: string; result: string }> = [];
// Find all ### N. Name headings (line-anchored)
const headingPattern = /^###\s*(\d+)\.\s*(.+)$/gm;
const headings: Array<{ index: number; test: number; name: string }> = [];
let hMatch: RegExpExecArray | null;
while ((hMatch = headingPattern.exec(cleanContent)) !== null) {
headings.push({
index: hMatch.index + hMatch[0].length,
test: parseInt(hMatch[1], 10),
name: hMatch[2].trim(),
});
}
for (let i = 0; i < headings.length; i++) {
const h = headings[i];
const blockStart = h.index;
// More precise: find next heading's position in the original string
// We'll slice from current heading end to the position just before next heading's "###"
const nextHeadingMatch = i + 1 < headings.length
? cleanContent.lastIndexOf('\n###', headings[i + 1].index)
: -1;
const blockContent = nextHeadingMatch >= blockStart
? cleanContent.slice(blockStart, nextHeadingMatch)
: cleanContent.slice(blockStart);
// Column-0 anchored result line: /^result:[ \t]*\[?([\w-]+)\]?/mi
// Uses [ \t]* (not \s*) so the captured value must sit on the SAME line as result:.
// A result: key with the value on a subsequent line yields no match → 'missing' (blocker).
const resultMatch = /^result:[ \t]*\[?([\w-]+)\]?/mi.exec(blockContent);
if (resultMatch) {
items.push({
test: h.test,
name: h.name,
result: resultMatch[1].toLowerCase(),
});
} else {
// No column-0 result line → emit 'missing' (a non-passing state)
items.push({
test: h.test,
name: h.name,
result: 'missing',
});
}
}
return items;
}
// ─── evaluateUatPassed ────────────────────────────────────────────────────────
/**
* Evaluate all UAT/VERIFICATION files in a phase directory.
* Returns a UatPassedReport with the locked, stable shape defined by the interface.
*
* FAIL-CLOSED: any absence/ambiguity/malformed input → NOT passed.
* Pass requires at least one real passing check AND no blockers.
*/
function evaluateUatPassed(
phaseFullDir: string,
opts?: { policy?: { requireVerification?: boolean } },
): UatPassedReport {
const requireVerification = opts?.policy?.requireVerification === true;
const blockers: string[] = [];
const checks: UatCheckItem[] = [];
const uatFiles: string[] = [];
const verificationFiles: string[] = [];
// Read the directory — if unreadable, treat as no files (fail-closed: no artifacts → not passed)
let dirEntries: string[] = [];
try {
dirEntries = fs.readdirSync(phaseFullDir);
} catch {
// Unreadable dir — no_uat_artifacts:true, passed:false
const no_uat_artifacts = true;
if (requireVerification) {
blockers.push('policy: verification required but no passing *-VERIFICATION.md found');
}
return {
passed: false,
uat_files: [],
verification_files: [],
checks: [],
blockers,
no_uat_artifacts,
policy: { require_verification: requireVerification },
};
}
// Filter UAT and VERIFICATION files using the same filter as cmdPhaseComplete
const uatFileNames = dirEntries.filter(f => f.includes('-UAT') && f.endsWith('.md'));
const verFileNames = dirEntries.filter(f => f.includes('-VERIFICATION') && f.endsWith('.md'));
// ── Process UAT files ──────────────────────────────────────────────────────
for (const file of uatFileNames) {
uatFiles.push(file);
let raw = '';
try {
raw = fs.readFileSync(path.join(phaseFullDir, file), 'utf-8');
} catch {
blockers.push(`${file}: could not read file`);
continue;
}
// ── Per-file malformed markdown guard ──────────────────────────────────
// FIX D: use accurate signals from analyzeMarkdown instead of heuristics.
// unterminatedFence: CommonMark state machine detects a genuinely unclosed fence.
// unterminatedComment: strips balanced comments first, then checks for leftover <!--.
const { unterminatedFence, unterminatedComment } = analyzeMarkdown(raw);
if (unterminatedFence || unterminatedComment) {
blockers.push(`${file}: malformed markdown (unterminated fence or comment)`);
}
const fm = extractFrontmatter(raw) as Record<string, unknown>;
// File-level frontmatter status check
if (fm['status'] && BLOCKING_UAT_FM_STATUSES.has(fm['status'] as string)) {
blockers.push(`${file}: frontmatter status=${fm['status'] as string}`);
}
// File-level frontmatter result check
if (fm['result'] && BLOCKING_UAT_FM_RESULTS.has(fm['result'] as string)) {
blockers.push(`${file}: frontmatter result=${fm['result'] as string}`);
}
// Parse test items from the cleaned body (hardened against false positives)
const cleanContent = stripFalsePositiveContexts(raw);
const items = parseUatResultItems(cleanContent);
for (const item of items) {
const passing = PASSING_RESULTS.has(item.result);
checks.push({
file,
test: item.test,
name: item.name,
result: item.result,
passing,
});
if (!passing) {
blockers.push(`${file}: test ${item.test} (${item.result})`);
}
}
}
// ── Process VERIFICATION files ─────────────────────────────────────────────
let hasPassingVerification = false;
for (const file of verFileNames) {
verificationFiles.push(file);
let raw = '';
try {
raw = fs.readFileSync(path.join(phaseFullDir, file), 'utf-8');
} catch {
blockers.push(`${file}: could not read verification file`);
continue;
}
const vfm = extractFrontmatter(raw) as Record<string, unknown>;
const vStatus = vfm['status'] as string | undefined;
if (vStatus && BLOCKING_VERIFICATION_FM_STATUSES.has(vStatus)) {
blockers.push(`${file}: verification status=${vStatus}`);
} else if (vStatus && PASSING_VERIFICATION_STATUSES.has(vStatus)) {
// Allowlist: only explicitly-passing statuses count
hasPassingVerification = true;
}
// Missing or unknown status: does NOT count as passing, does NOT push a blocker
// (handled by the requireVerification policy check below if needed)
}
// ── Policy: requireVerification ───────────────────────────────────────────
if (requireVerification && !hasPassingVerification) {
blockers.push('policy: verification required but no passing *-VERIFICATION.md found');
}
// ── Determine no_uat_artifacts and passed ─────────────────────────────────
// no_uat_artifacts: true when no real UAT test items were parsed from any file
const no_uat_artifacts = checks.length === 0;
// FIX 1: require positive passing evidence; no vacuous pass
// passed = no blockers AND at least one check AND all checks passing
const passed = blockers.length === 0 && checks.length > 0 && checks.every(c => c.passing);
return {
passed,
uat_files: uatFiles,
verification_files: verificationFiles,
checks,
blockers,
no_uat_artifacts,
policy: {
require_verification: requireVerification,
},
};
}
export = {
stripFalsePositiveContexts,
parseUatResultItems,
analyzeMarkdown,
evaluateUatPassed,
};

View File

@@ -0,0 +1,267 @@
'use strict';
/**
* Integration tests for `phase uat-passed <N>` CLI command.
* Issue #247 — phase uat-passed predicate
*
* Tests the full dispatch path: gsd-tools → phase-command-router → phase.cmdPhaseUatPassed
*/
const { test, describe, beforeEach, afterEach } = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const { runGsdTools, createTempProject, cleanup } = require('./helpers.cjs');
// ─── Helpers ──────────────────────────────────────────────────────────────────
/**
* Set up a minimal project with a phase directory and ROADMAP so that
* findPhaseInternal(cwd, phaseNum) can resolve it.
* Returns { tmpDir, phaseDir }.
*/
function setupProject(phaseSlug = '01-feature') {
const tmpDir = createTempProject();
fs.writeFileSync(
path.join(tmpDir, '.planning', 'ROADMAP.md'),
[
'# Roadmap',
'',
'- [ ] Phase 1: Feature',
'',
'### Phase 1: Feature',
'**Goal:** Build feature',
'**Plans:** 1 plans',
'',
].join('\n'),
);
const phaseDir = path.join(tmpDir, '.planning', 'phases', phaseSlug);
fs.mkdirSync(phaseDir, { recursive: true });
return { tmpDir, phaseDir };
}
function writeUatFile(phaseDir, filename, content) {
fs.writeFileSync(path.join(phaseDir, filename), content, 'utf-8');
}
function makePassingUat() {
return [
'---',
'status: passed',
'---',
'',
'# UAT Results',
'',
'### 1. Login works',
'expected: User logs in successfully',
'result: passed',
'',
].join('\n');
}
function makePendingUat() {
return [
'---',
'status: partial',
'---',
'',
'# UAT Results',
'',
'### 1. Login works',
'expected: User logs in successfully',
'result: passed',
'',
'### 2. Logout works',
'expected: User logs out successfully',
'result: pending',
'',
].join('\n');
}
function makeFencedFalsePositiveUat() {
// Only "result: passed" lines are inside a fenced block.
// The real test has result: pending → should evaluate to passed:false.
return [
'---',
'status: partial',
'---',
'',
'# UAT Results',
'',
'## Example (do not run)',
'```',
'### 1. Test',
'expected: Example',
'result: passed',
'```',
'',
'### 1. Real Test',
'expected: The thing works',
'result: pending',
'',
].join('\n');
}
// ─── Basic pass/fail cases ─────────────────────────────────────────────────────
describe('phase uat-passed — basic pass/fail', () => {
let tmpDir;
let phaseDir;
beforeEach(() => {
({ tmpDir, phaseDir } = setupProject('01-feature'));
});
afterEach(() => {
cleanup(tmpDir);
});
test('passing UAT → passed:true with correct JSON shape', () => {
writeUatFile(phaseDir, 'feature-UAT.md', makePassingUat());
const result = runGsdTools('phase uat-passed 1', tmpDir);
assert.ok(result.success, `Command failed: ${result.error}\nOutput: ${result.output}`);
const out = JSON.parse(result.output);
assert.strictEqual(out.passed, true);
assert.strictEqual(out.phase, '1');
assert.ok(Array.isArray(out.uat_files), 'uat_files must be an array');
assert.ok(Array.isArray(out.verification_files), 'verification_files must be an array');
assert.ok(Array.isArray(out.checks), 'checks must be an array');
assert.ok(Array.isArray(out.blockers), 'blockers must be an array');
assert.ok(out.policy && typeof out.policy.require_verification === 'boolean',
'policy.require_verification must be a boolean');
assert.strictEqual(typeof out.no_uat_artifacts, 'boolean', 'no_uat_artifacts must be a boolean');
assert.strictEqual(out.no_uat_artifacts, false, 'no_uat_artifacts must be false when checks exist');
assert.strictEqual(out.blockers.length, 0);
});
test('pending UAT → passed:false', () => {
writeUatFile(phaseDir, 'feature-UAT.md', makePendingUat());
const result = runGsdTools('phase uat-passed 1', tmpDir);
assert.ok(result.success, `Command failed: ${result.error}`);
const out = JSON.parse(result.output);
assert.strictEqual(out.passed, false);
assert.strictEqual(out.phase, '1');
assert.ok(out.blockers.length > 0, 'Should have blockers for pending test');
});
test('false-positive only (fenced block) → passed:false', () => {
writeUatFile(phaseDir, 'feature-UAT.md', makeFencedFalsePositiveUat());
const result = runGsdTools('phase uat-passed 1', tmpDir);
assert.ok(result.success, `Command failed: ${result.error}`);
const out = JSON.parse(result.output);
assert.strictEqual(out.passed, false,
'result:passed inside a fenced block must not flip the predicate to passed');
});
test('no UAT files → passed:false + no_uat_artifacts:true (fail-closed, no vacuous pass)', () => {
// Phase directory exists but has no UAT files — fail-closed: absence is NOT a pass
const result = runGsdTools('phase uat-passed 1', tmpDir);
assert.ok(result.success, `Command failed: ${result.error}`);
const out = JSON.parse(result.output);
assert.strictEqual(out.passed, false,
'Phase with no UAT files must NOT vacuously pass — fail-closed predicate');
assert.strictEqual(out.no_uat_artifacts, true,
'no_uat_artifacts must be true when no UAT items found');
assert.deepStrictEqual(out.uat_files, []);
});
});
// ─── --require-verification flag ──────────────────────────────────────────────
describe('phase uat-passed — --require-verification flag', () => {
let tmpDir;
let phaseDir;
beforeEach(() => {
({ tmpDir, phaseDir } = setupProject('01-feature'));
});
afterEach(() => {
cleanup(tmpDir);
});
test('--require-verification with no verification file → passed:false', () => {
writeUatFile(phaseDir, 'feature-UAT.md', makePassingUat());
const result = runGsdTools('phase uat-passed 1 --require-verification', tmpDir);
assert.ok(result.success, `Command failed: ${result.error}`);
const out = JSON.parse(result.output);
assert.strictEqual(out.passed, false,
'require-verification with no verification file should fail');
assert.strictEqual(out.policy.require_verification, true);
assert.ok(out.blockers.some(b => /verification required/i.test(b)),
`Expected verification-required blocker, got: ${JSON.stringify(out.blockers)}`);
});
test('--require-verification with passing verification → passed:true', () => {
writeUatFile(phaseDir, 'feature-UAT.md', makePassingUat());
writeUatFile(phaseDir, 'feature-VERIFICATION.md', '---\nstatus: passed\n---\n\nVerified OK.');
const result = runGsdTools('phase uat-passed 1 --require-verification', tmpDir);
assert.ok(result.success, `Command failed: ${result.error}`);
const out = JSON.parse(result.output);
assert.strictEqual(out.passed, true);
assert.strictEqual(out.policy.require_verification, true);
});
});
// ─── Error cases ──────────────────────────────────────────────────────────────
describe('phase uat-passed — error cases', () => {
let tmpDir;
beforeEach(() => {
tmpDir = createTempProject();
// Write a minimal ROADMAP so phase 1 exists
fs.writeFileSync(
path.join(tmpDir, '.planning', 'ROADMAP.md'),
[
'# Roadmap',
'',
'### Phase 1: Feature',
'**Goal:** Build feature',
'',
].join('\n'),
);
fs.mkdirSync(path.join(tmpDir, '.planning', 'phases', '01-feature'), { recursive: true });
});
afterEach(() => {
cleanup(tmpDir);
});
test('missing phase number → error message', () => {
const result = runGsdTools('phase uat-passed', tmpDir);
assert.ok(!result.success, 'Should fail with no phase number');
assert.ok(
result.error.includes('phase number required') ||
result.error.includes('Available:'),
`Expected phase-number-required error, got: ${result.error}`,
);
});
test('unknown phase number → error message', () => {
const result = runGsdTools('phase uat-passed 99', tmpDir);
assert.ok(!result.success, 'Should fail for unknown phase');
assert.ok(
result.error.includes('not found') || result.error.includes('99'),
`Expected not-found error, got: ${result.error}`,
);
});
test('unknown flag (typo --require-verifcation) → InvalidArgs error, not silent pass', () => {
const result = runGsdTools('phase uat-passed 1 --require-verifcation', tmpDir);
assert.ok(!result.success,
'Unknown flag must cause an error, not silently pass');
assert.ok(
result.error.includes('--require-verifcation') ||
result.error.includes('does not support') ||
result.error.includes('invalid'),
`Expected unknown-flag error, got: ${result.error}`,
);
});
});

1295
tests/uat-predicate.test.cjs Normal file

File diff suppressed because it is too large Load Diff