fix(#3597): count scenario expectation failures in the QA gate, and fix the workstream scope split it exposed (#3607)
* fix(#3597): count scenario expectation failures in the QA ratchet gate buildReport counts totals.violations as oracle violations PLUS scenario expectFailures, but collectFindings read only step.violations. A scenario whose declared expect failed therefore produced ok:false and violations:1 in the report while the ratchet printed "0 violations" and exited 0. multi-workstream has failed that way on every CI run since 2026-08-10, when #3217 (PR #3318) made computeProgressPercent withhold a percentage whose scope is not COMPLETE. The walk detected the change the day it landed; nothing was listening. - collectFindings returns a third bucket, expectationFailures, carrying no fingerprint so it can never be baselined or acked away - both modes of main() print and gate on it; the summary line reports it - guard runMain(main) behind require.main === module, so the QA suite can require the script to test collectFindings without running a real walk (that import side effect is why the gate logic had no test) - multi-workstream now asserts the true contract: phase_scope unreadable and percent null, per ADR-3180 7.6 rule 4 - the perturbation test asserts scenario ok, closing the test-side half Closes #3597 * fix(#3597): resolve the milestone window against the active workstream listMilestonePhaseDirs defaulted its ws option to null. planningDir treats undefined as "resolve the ambient workstream" and null as "force the project root", so that default suppressed the ambient resolution every other planning-path read uses. All 18 call sites derive phasesDir ambiently via planningPaths(cwd), so the counts came from the workstream while the milestone window came from the root .planning/ROADMAP.md — the exact numerator/denominator scope split ADR-3180 7.6 rule 3 forbids. workstream create migrates that root roadmap away, so the read threw and scope stayed UNREADABLE, and rule 4 then correctly withheld the percentage. Proof: with a workstream tree byte-unchanged, copying its own ROADMAP to the project root flipped --ws alpha progress from phase_scope:unreadable/percent:null to complete/100. This is the defect the loop QA walk was pointing at all along; the scenario expectation is restored to percent:100 rather than bent to match the bug. - pass ws through as undefined so ambient resolution applies - multi-workstream asserts phase_scope complete + percent 100 - regression test in completion-ratio-scope-withholding covers a workstream-only project with no root ROADMAP - replace the vacuous require.main test: runMain defers through a promise, so the in-process timing check passed against the unguarded file too; a child-process spawn now observes the guard for real - tie the oracle-violation test to expectationFailures, and cover the absent-key, multi-scenario and zero-step report shapes in parity - flatten scenario-authored strings before rendering them into the step summary and CI logs (forged markdown / ANSI injection) - widen the scenario contract assertions past perturbation-* so multi-workstream is actually covered test-side Closes #3597 * fix(#3597): flatten scenario-authored strings on the CI-log output path The step-summary path already routed findings through flattenUntrusted; the check-mode NEW-smell and STALE-entry console.error blocks, and the repro line in both printers, still interpolated raw. detail carries a scenario-authored expect[].path verbatim, and reason/scenario/id come from contributor-authored baseline and ack fragments validated only as non-empty strings. A crafted path could print a forged summary line into the CI log directly above the real one, plus ANSI repaint and unbounded length. Exit codes are unaffected — this is log spoofing, not gate bypass. * fix(#3597): refuse to archive on an unreadable milestone window; close review gaps Resolving the milestone window against the active workstream can leave the window UNREADABLE when that workstream has no ROADMAP of its own. getMilestonePhaseFilter throws, the window degrades to a pass-all fallback, and milestone complete would then move every phase dir -- breaking the guarantee stated at the archive site that no out-of-window directory is touched. milestone complete now refuses to archive when the window is UNREADABLE and reports the refusal; --dry-run previews the same refusal from the same shared derivation. The guard is scoped to UNREADABLE, not to every non-COMPLETE scope. A broader condition regressed ordinary root projects: the QA walk caught milestone-rollover leaving 01-parser on disk, which then tripped the #1447 abort in phases clear. UNSCOPED and TRUNCATED are pre-existing classifications and keep their existing behavior. Review fixes: - the workstream regression test asserted complete/100 but its fixture wrote no workstream STATE.md, so it resolved unscoped/null and the test failed; it now asserts a milestone and genuinely fails-first - the parity test hand-supplied totals.violations, hardcoding the very formula under test; at least one case now goes through the real buildReport - drop a vacuous qa-report.json assertion (jsonOut defaults to null, so no report is written by either shape) - buildRepro emitted a repo-relative binary path after cd-ing into a temp project, so every repro died with MODULE_NOT_FOUND; it now resolves an absolute path - flattenUntrusted truncated the repro to 300 chars, handing reviewers a command that looks complete and is not; length capping is now opt-out for repro while newline/control/backtick stripping still applies * chore(#3597): backfill changeset pr number (#3607) --------- Co-authored-by: sim <sim@local>
This commit is contained in:
Tom Boucher
@@ -749,15 +749,23 @@ function cmdMilestoneComplete(cwd: string, version: string, options: MilestoneCo
|
||||
let totalTasks = 0;
|
||||
const accomplishments: string[] = [];
|
||||
|
||||
try {
|
||||
// #3185 (ADR-3180 Decision 1): "which phase directories belong to the
|
||||
// CURRENT milestone" — routed through the canonical owner (with the
|
||||
// explicit `version` this command already resolved) instead of a
|
||||
// hand-rolled readdirSync + isDirInMilestone filter, which also never
|
||||
// excluded sentinels, unlike the owner.
|
||||
const dirs = listMilestonePhaseDirs(phasesDir, { cwd, versionOverride: version }).value;
|
||||
// #3597 (ADR-3180 Decision 2): SINGLE resolution of "which phase
|
||||
// directories belong to the current milestone" AND the SCOPE discriminator
|
||||
// that resolution came from — shared verbatim by the read-only stats loop
|
||||
// immediately below, the --dry-run preview, and the real archive pass, so
|
||||
// none of the three can ever disagree. `listMilestonePhaseDirs` never
|
||||
// throws (its own doc comment), so this is safe to call unguarded ahead of
|
||||
// the try/catch that scopes the stats roll-up below.
|
||||
//
|
||||
// The stats loop's own ENUMERATION behavior is intentionally left
|
||||
// unaffected by a non-COMPLETE scope — it reports what is actually on
|
||||
// disk, same as before #3597. Only the destructive archive pass (and its
|
||||
// --dry-run preview) refuses to act on a non-COMPLETE (non-answer) scope;
|
||||
// see the guard built from `milestonePhaseScope` further down.
|
||||
const { value: milestonePhaseDirs, scope: milestonePhaseScope } = listMilestonePhaseDirs(phasesDir, { cwd, versionOverride: version });
|
||||
|
||||
for (const dir of dirs) {
|
||||
try {
|
||||
for (const dir of milestonePhaseDirs) {
|
||||
phaseCount++;
|
||||
// #3183: canonical plan/summary sets (root+nested, superseded-excluded)
|
||||
// from the single owner, rather than a root-only hand-rolled readdirSync
|
||||
@@ -803,15 +811,41 @@ function cmdMilestoneComplete(cwd: string, version: string, options: MilestoneCo
|
||||
* accomplishments=[] rather than crash `milestone complete`. */
|
||||
}
|
||||
|
||||
// #3597 (ADR-3180 Decision 2): `SCOPE.UNREADABLE` is the ONE classification
|
||||
// where `getMilestonePhaseFilter` throws (no workstream ROADMAP of its
|
||||
// own), leaves `milestonePhaseNums` empty, and the window degrades to a
|
||||
// pass-all fallback — that fallback is what silently WIDENS the archive
|
||||
// set past the single-derivation guarantee this block exists to protect
|
||||
// (confirmed empirically: a workstream with phase dirs but no workstream
|
||||
// ROADMAP.md reports UNREADABLE and used to enumerate every directory on
|
||||
// disk). `SCOPE.TRUNCATED` already refuses the WHOLE command above.
|
||||
// `SCOPE.UNSCOPED` is a DIFFERENT, pre-existing classification — e.g. a
|
||||
// root project with no milestone asserted in STATE.md — whose
|
||||
// `listMilestonePhaseDirs` resolution is a real (non-degraded) answer, and
|
||||
// its rollover archive behavior predates this branch and must not change.
|
||||
// The guard therefore refuses ONLY on UNREADABLE, not on "not COMPLETE" —
|
||||
// widening to every non-COMPLETE scope was itself a regression (a root
|
||||
// project with no active workstream resolves UNSCOPED, and refusing to
|
||||
// archive there broke the ordinary `milestone complete` -> `phases clear`
|
||||
// rollover). Computed once here from the single
|
||||
// `milestonePhaseDirs`/`milestonePhaseScope` resolution above, so the
|
||||
// --dry-run preview below and the real archive pass further down can never
|
||||
// disagree about whether (or what) to archive.
|
||||
const phasesArchiveSkippedForScope = options.archivePhases !== false && milestonePhaseScope === SCOPE.UNREADABLE;
|
||||
const phasesArchiveSkipReason = phasesArchiveSkippedForScope
|
||||
? `milestone window scope is "${milestonePhaseScope}" — refusing to archive phase directories until the window can be resolved (ADR-3180)`
|
||||
: null;
|
||||
|
||||
// #2118: --dry-run preview — compute what WOULD happen without mutating.
|
||||
// The stats above are read-only; all mutations start at the archive section below.
|
||||
if (options.dryRun) {
|
||||
const phaseDirsToArchive: string[] = [];
|
||||
if (options.archivePhases !== false) {
|
||||
// #3185 (ADR-3180 Decision 1): same routed derivation as the stats loop
|
||||
// above — the dry-run preview must list exactly what the real archive
|
||||
// pass below would move.
|
||||
phaseDirsToArchive.push(...listMilestonePhaseDirs(phasesDir, { cwd, versionOverride: version }).value);
|
||||
if (options.archivePhases !== false && !phasesArchiveSkippedForScope) {
|
||||
// #3185 (ADR-3180 Decision 1) / #3597: same single routed derivation as
|
||||
// the stats loop above — the dry-run preview must list exactly what
|
||||
// the real archive pass below would move, including refusing to list
|
||||
// anything when the window scope is not COMPLETE.
|
||||
phaseDirsToArchive.push(...milestonePhaseDirs);
|
||||
}
|
||||
// #2142 MAJOR 5 (review): dry-run preview of quick-task archival —
|
||||
// read-only, routed through the SAME `listQuickTaskDirsForArchive`
|
||||
@@ -838,6 +872,8 @@ function cmdMilestoneComplete(cwd: string, version: string, options: MilestoneCo
|
||||
? { source: path.relative(cwd, path.join(planningBase, `${version}-MILESTONE-AUDIT.md`)).split(path.sep).join('/'), target: path.relative(cwd, path.join(archiveDir, `${version}-MILESTONE-AUDIT.md`)).split(path.sep).join('/') }
|
||||
: null,
|
||||
phases: phaseDirsToArchive,
|
||||
phases_archive_skipped: phasesArchiveSkippedForScope,
|
||||
phases_archive_skip_reason: phasesArchiveSkipReason,
|
||||
quick: quickDirsToArchive,
|
||||
},
|
||||
would_update: {
|
||||
@@ -1051,7 +1087,15 @@ function cmdMilestoneComplete(cwd: string, version: string, options: MilestoneCo
|
||||
// Archive phase directories if requested
|
||||
let phasesArchived = false;
|
||||
// #1871: archive phase dirs by default on milestone complete (opt out via --no-archive-phases).
|
||||
if (options.archivePhases !== false) {
|
||||
// #3597: `phasesArchiveSkippedForScope` (computed once, above, from the SAME
|
||||
// `milestonePhaseDirs`/`milestonePhaseScope` resolution the --dry-run preview
|
||||
// consumed) refuses this destructive rename loop entirely when the window
|
||||
// scope is UNREADABLE — the one scope whose resolution degrades to a
|
||||
// pass-all fallback (ADR-3180). UNSCOPED/TRUNCATED are real, pre-existing
|
||||
// answers and archive exactly as they did before this branch.
|
||||
// `phasesArchived` stays false and the refusal is surfaced on `result` below;
|
||||
// nothing on disk moves, and `phaseArchiveDir` is never even created.
|
||||
if (options.archivePhases !== false && !phasesArchiveSkippedForScope) {
|
||||
// #2245 audit (was ERROR-HIDING): retryRenameSync moves one phase dir at a
|
||||
// time — a mid-loop failure (e.g. the Nth rename) used to leave
|
||||
// `phasesArchived` at its `false` default even though the first N-1 dirs
|
||||
@@ -1064,12 +1108,12 @@ function cmdMilestoneComplete(cwd: string, version: string, options: MilestoneCo
|
||||
const phaseArchiveDir = path.join(archiveDir, `${version}-phases`);
|
||||
platformEnsureDir(phaseArchiveDir);
|
||||
|
||||
// #3185 (ADR-3180 Decision 1): same routed derivation as the stats
|
||||
// loop above — only the CURRENT milestone's phase directories move,
|
||||
// never a sentinel or an out-of-window directory left for a later
|
||||
// milestone.
|
||||
const phaseDirNames = listMilestonePhaseDirs(phasesDir, { cwd, versionOverride: version }).value;
|
||||
for (const dir of phaseDirNames) {
|
||||
// #3185 (ADR-3180 Decision 1) / #3597: same single routed derivation as
|
||||
// the stats loop and the --dry-run preview above — only the CURRENT
|
||||
// milestone's phase directories move, never a sentinel or an
|
||||
// out-of-window directory left for a later milestone, and never a
|
||||
// pass-all degrade from a non-COMPLETE scope (refused above).
|
||||
for (const dir of milestonePhaseDirs) {
|
||||
retryRenameSync(path.join(phasesDir, dir), path.join(phaseArchiveDir, dir));
|
||||
archivedCount++;
|
||||
}
|
||||
@@ -1095,6 +1139,13 @@ function cmdMilestoneComplete(cwd: string, version: string, options: MilestoneCo
|
||||
requirements: fs.existsSync(path.join(archiveDir, `${version}-REQUIREMENTS.md`)),
|
||||
audit: fs.existsSync(path.join(archiveDir, `${version}-MILESTONE-AUDIT.md`)),
|
||||
phases: phasesArchived,
|
||||
// #3597: machine-readable refusal signal — distinguishes "nothing to
|
||||
// archive because the milestone genuinely has no phase directories"
|
||||
// (phases: false, phases_archive_skipped: false) from "refused to
|
||||
// archive because the milestone window scope was not COMPLETE"
|
||||
// (phases: false, phases_archive_skipped: true, with a reason).
|
||||
phases_archive_skipped: phasesArchiveSkippedForScope,
|
||||
phases_archive_skip_reason: phasesArchiveSkipReason,
|
||||
quick: !!quickArchiveResult && quickArchiveResult.archived > 0,
|
||||
},
|
||||
milestones_updated: true,
|
||||
|
||||
@@ -347,6 +347,11 @@ function findPhaseInternal(cwd: string, phase: unknown): PhaseSearchResult | nul
|
||||
* an absent `phasesDir` is a real empty (a new project genuinely has no
|
||||
* phases) and inherits the window's scope, whereas a `phasesDir` that exists
|
||||
* but cannot be read is UNREADABLE.
|
||||
*
|
||||
* `opts.ws` is tri-state, matching `planningDir`'s own contract: `undefined`
|
||||
* (the default — do not pass `ws` at all) resolves the AMBIENT workstream
|
||||
* from `GSD_WORKSTREAM`; `null` FORCES the project root regardless of any
|
||||
* ambient workstream; a string forces that specific workstream.
|
||||
*/
|
||||
function listMilestonePhaseDirs(
|
||||
phasesDir: string,
|
||||
@@ -357,7 +362,17 @@ function listMilestonePhaseDirs(
|
||||
phaseIdConvention?: string | null;
|
||||
} = {},
|
||||
): { value: string[]; scope: Scope } {
|
||||
const { cwd, ws = null, versionOverride = null, phaseIdConvention = null } = opts;
|
||||
// #3597: `ws` must default to `undefined`, NOT `null`. `undefined` means
|
||||
// "resolve the ambient workstream" (mirrors planningDir's own contract,
|
||||
// src/planning-workspace.cts:124); `null` means "force the project root".
|
||||
// Every cwd-bearing caller derives `phasesDir` ambiently (planningPaths(cwd)
|
||||
// / planningDir(cwd) with no explicit ws), so defaulting `ws` to `null` here
|
||||
// forced the milestone WINDOW to the root ROADMAP while the caller's
|
||||
// `phasesDir` stayed workstream-scoped — numerator and denominator drawn
|
||||
// from different scoped sets (ADR-3180 §7.6 rule 3 violation). That is what
|
||||
// made `--ws <name> progress` read `phase_scope: "unreadable"` and withhold
|
||||
// `percent` once `workstream create` migrated the root ROADMAP away.
|
||||
const { cwd, ws, versionOverride = null, phaseIdConvention = null } = opts;
|
||||
|
||||
// Without a cwd there is nothing to scope AGAINST — the caller asked for an
|
||||
// unscoped read, which is a real answer (mirrors extractCurrentMilestoneScoped's
|
||||
|
||||
Reference in New Issue
Block a user