fix(#3707): surface UAT rows audit-uat silently dropped, and never report a clean result for a file it could not read (#3887)
* test(#3707): failing-first coverage for the three parseUatItems false negatives Nine tests that must be red and three controls that must already be green. The controls are the point of the split. `result: pass` staying unsurfaced is what stops the fix inverting the filter so eagerly that every passing test becomes an outstanding item, and the classic single-line shape is the no-churn control for rewriting the adjacency regex. Both were confirmed green against the current build before being written down; a control that is red today would be a second bug, not a control. Each failing fixture was run through the built parser first and returns [] for its stated cause — the issue row matched then filtered, the block-scalar and wrapped rows never matched at all, the all-unparseable file vanishing whole. That evidence is in 50-test-matrix.md rather than asserted. Tests target ../gsd-core/bin/lib/uat.cjs, the built live module, and drive the real CLI through runGsdTools. #3706 lost a full RED/GREEN cycle to tests that imported a different copy of the function under test, so the import target was verified before anything was written. * fix(#3707): stop parseUatItems dropping outstanding UAT rows Three independent false negatives, all in the audit path, plus one the issue did not mention. The matcher no longer requires `expected:` and `result:` to be adjacent single lines. It slices each `### N.` block to the next heading and reads the first `result:` line within it, taking `expected:` from parseExpectedFromTestBlock — the seam that already parsed both the block-scalar and inline forms correctly and was sitting unused two hundred lines away. Two parsers in one module read the same field with different grammars; now there is one. The result filter is inverted from an inclusion list of three to an exclusion of a minimal PASS set. This was the issue's one open design question, which the reporter explicitly declined to answer for the maintainer; it was asked and decided deliberately. The fail-safe direction is what parseGapsItems documents seventy lines below for this same false-negative class (#2286): a token nobody recognised surfaces rather than vanishing. The trade is a visible, correctable false positive if a project invents a novel pass-word, against today's silent and invisible drop. `issue` also needed a category. It is template-sanctioned with its own `issues:` counter, but categorizeItem fell through to `unknown` — surfacing it in the wrong bucket would have been a half-fix. Finally, a file parsing to zero items no longer vanishes with its frontmatter `status:`. One with a non-terminal status is reported with `parse_gap: true`, so the reader gets a cue to look; a `complete` one stays omitted as before. That is what made the first two defects dangerous rather than merely lossy — the audit omitted the phase instead of under-counting it. * fix(#3707): close the review blockers, including a regression I introduced The remote suite was RED on the previous commit and both reviews found real defects. Everything below was verified by execution, not by reading. I introduced a regression against origin/next. The rewritten result matcher was END-anchored where the old one was not, so `result: pending (blocked on staging)`, `result: [skipped] # no device` and `result: blocked - waiting` all returned a row before this branch and returned nothing on it — me reproducing the exact defect class this issue exists to kill, in the fix for it. The anchor is gone and each shape has a regression test; trailing text now falls back to `reason` when the block has none. `parse_gap` was inferred from the wrong signal. It fired for ANY zero-item file whose status was not `complete`, which asserted something false about a perfectly-parsed all-pass file, swept in archived phases left at `testing`, and is what turned the #2286 Gaps tests red — a control this change was supposed to keep green. It now derives from headings SEEN BUT UNYIELDED, reported by a new parseUatItemsWithStats, so an all-pass file and a Gaps-only file are not parse gaps and a file whose blocks carry no `result:` line is. The fix was also invisible end to end, which both reviewers caught independently. parse_gap entries carry no items, and both audit-uat.md and progress.md gate on `total_items === 0` — so the headline symptom, the phase vanishing, still reproduced for a user and only the raw JSON had changed. There is now a `parse_gap_files` counter and both workflows gate and report on it. Also: categorizeItem compared case-sensitively while the new PASS check lowercased, so `result: PENDING` surfaced as `unknown`; blocks are bounded at the next heading of any level, so a trailing `## Gaps` entry no longer bleeds its `reason` onto the preceding test; dead unreachable fallbacks removed; and the all-pass control was strengthened, since asserting only `total_items === 0` let it stay green through the bogus parse_gap entry. * chore(#3707): acknowledge the workflow growth the fix required The emitted-attribution guard went red because audit-uat.md and progress.md grew, and it is right to ask: runtime-loaded workflow prose is the product, so growth there is a real change to what an executing agent reads. The growth is not incidental to this fix, it IS the fix reaching a user. Both reviewers found independently that emitting `parse_gap` in the JSON changed nothing observable, because both workflows gated their output on `total_items === 0` and parse-gap entries carry no items — so a phase whose rows could not be parsed still printed "All Clear" and still vanished from the progress report. The widened gates and the branches that name the unparsed files with their phase and path are what close that. Acks exactly the two paths the guard reported, keyed on the bare filename. The three spent acknowledgments it also listed are inert by its own description — the base already absorbs them — so they are left alone rather than swept up here, where they would just add unrelated churn to this diff. * fix(#3707): close the mixed-file blocker and the second false-clean surface The suite was GREEN and the isolated review still found a blocker, which is the useful part: none of this was covered by a test. A MIXED file dropped its unparseable rows silently. `parse_gap` sat behind an `else if` on `items.length > 0`, so one parseable row was enough to discard `headingsSeen` entirely — a file with one pending row and two unreadable blocks reported one item and no gap. That is the exact class this issue exists to kill, reappearing inside its own fix for the third time. The flag is now set independently of item count and the entry carries `unparsed_blocks`, so the count is quantified rather than merely flagged. A `result:` inside a fenced code block was being read as real, so a PASSING test could be reported as outstanding from a value in a code sample — another regression against origin/next, whose adjacency regex ignored it. Field scans now run against a fence-stripped copy while `expected:` still reads the raw block, since a block scalar may legitimately contain fenced-looking text. The workflow report was still unreachable whenever anything else was outstanding: the unparsed table lived in the all-clear branch, so a project with one pending row in phase 01 and an unreadable phase 02 rendered phase 02 nowhere. It now fires on `parse_gap_files > 0` from the `present` step. planning-inspect was the second surface making a false-clean claim — for exactly the files audit-uat now flags it emitted `scope: 'complete'` with an empty unresolved list, positively asserting completeness over a file it could not read. It consumes the stats now and reports SCOPE.TRUNCATED with a `uat_unreadable` diagnostic, reusing the vocabulary already used two lines above for an unreadable file rather than inventing a token. Also: headings with no name no longer vanish whole; the trailing-text-to-reason synthesis I had added is removed, since it was never required by the blocker and silently changed categorization for `result: [skipped] # no device`; the emitted `result` token is normalized to lower case so it agrees with `category` in a published contract; and an O(n^2) indexOf is gone from the heading loop. * fix(#3707): stop rows stealing each other's fields, on all three surfaces The suite was green when the security review found these. Two are blocker-severity and one of them is a direct hit on my own verification. A `### N.` line indented two spaces inside an `expected: |` scalar is a valid ATX heading, so it became a phantom row that STOLE the real row's result token while the real row vanished. I had probed this shape and declared it fixed — my probe asserted the item COUNT and the result token, both of which the phantom satisfied, so it passed for exactly the reason it should have failed. Block scalar bodies are now masked to blank lines (line count preserved, so offsets still line up) before headings are tokenized, and the tests assert row IDENTITY — number and name — not presence. Feeding parseExpectedFromTestBlock the raw slice let one row publish another row's `expected:` from inside a fence the stripped view had correctly excluded. Blocks handed to it are now clipped at the first fence opener. This was not cosmetic on the render-checkpoint path: a checkpoint banner a HUMAN reads and answers was rendering a different row's expected text. A balanced fence pair straddling a test block made that heading invisible, so an outstanding row disappeared with no item, no gap and no count — the exact false-clean this issue exists to close, and a regression against origin/next. Suppressed `### N.` lines now count toward headingsSeen so the file is flagged. An unterminated fence swallowed the rest of the document including `## Gaps`, producing a whole-file false clean. Such a file is now treated as a parse gap, following what uat-predicate already does. Found and fixed inline while there: parseExpectedFromTestBlock's scalar opener required a bare newline, so a CRLF `expected: |` fell through to the inline arm and published `expected: "|"`, silently discarding the entire value. The same fall-through hit `|-` and `|+`. parseFirstPendingTest had the identical exposure on the render-checkpoint path and now shares the same masking and clipping. Five legitimate fixtures — inline expected, a real block scalar, CRLF, bracketed pending, and a first-pending that is not the first test — are byte-identical before and after. Also from the code review: the admit condition disagreed with the terminal status guard, so a `status: complete` file with an unparseable block was emitted as an empty entry that rendered nowhere but inflated total_files; a control test was vacuous because its fixture filename did not match its phase dir, so #3511 scoping meant the file was never opened — and that vacuity is why the admit regression shipped green; an unterminated fence discarded the flag that would have caught it; `### 1.2.3` parsed as test 1; and planning-inspect did not share the terminal-status rule. * test(#3707): assert what the render-checkpoint fix actually does The suite went red on three of my own tests and the source was right — the assertions were wrong, in a way worth naming. One forbade the rendered checkpoint from containing `### 3. Fake Row`. But in that fixture the string IS row 1's legitimate `expected:` block-scalar value; a heading-shaped line inside a scalar is inert text and rendering it is correct. The test was forbidding correct output. It now asserts row IDENTITY — the checkpoint is for test 1 named Alpha and never test 3 named Fake Row — which is the property that actually distinguishes the fix from the bug. The other expected success where the correct outcome is a clean error: row 1 in that fixture has no `expected:` of its own and only ever appeared to have one by stealing row 2's from inside a fence. Depending on the bug to produce a pass is how a test ends up pinning the defect. The fixture now gives row 1 its own value and asserts the checkpoint carries it and never the fence-hidden text, and the error path gets its own test asserting it fails cleanly without leaking. All three were checked against the real rendered output before the assertion was written, and each was reasoned through for whether it can fail: the identity test breaks if a phantom row is parsed, the clipping test breaks if the raw block is read again, and the error test would pass-not-fail under the old stealing behavior. * fix(#3707): correct the scalar masking frame and cover every YAML block opener Two reviews independently found the same blocker, and it is the sharpest defect on this issue: maskBlockScalarBodies computed line offsets in UTF-16 units but spliced them into Array.from(content), a CODE POINT array. One emoji anywhere earlier in the file shifted every later mask write, so the mask blanked the wrong characters and spilled past line ends. Measured: at two astral characters a result token truncated `pending` to `pendi` and recategorized to unknown; at six the real row vanished; at twelve the FOLLOWING row's `result: blocked` disappeared and the file reported clean. That is the false-clean class this issue exists to close, reintroduced by the mitigation written to prevent it, and defeating both new detectors at once. The mask is rebuilt line by line now, which is frame-agnostic and length-preserving by construction. The opener grammar was also incomplete. YAML block scalar headers take an optional indentation indicator and an optional chomping indicator in either order, so `|2`, `|2-`, `|-2`, `>2`, `>2+` are all valid — and none were matched. An unmasked `expected: |2` body meant a `### N.` line inside the value became a real heading: reproduced, row 1 disappeared and a fabricated row 2 named "Phantom" took its identity. Fixing that exposed a third instance of the same family, found by my own probe rather than by review: the value extractor understood only the `|` openers, so every `>` folded scalar published the LITERAL OPENER as its value — `expected` came back as ">" or ">2+" and the whole scalar was discarded. The extractor now shares the opener grammar and implements real folding, joining paragraph lines with a space and turning a blank line into a newline, rather than pretending `>` means `|`. Also from the reviews: the shortfall counter scanned the masked copy but not a fence-stripped one, so a `### N.`-shaped line inside a properly closed documentation fence — the ordinary way to document the row format inside a UAT file — counted as a suppressed row and flagged the file against nothing; and clipping at the first fence discarded a legitimate `expected:` that appeared after a closed fence, which is silent field loss. Every opener now verified for both row identity and exact extracted value, in LF and CRLF, alongside the emoji fixtures at 1/2/6/12. * refactor(#3707): replace the scalar masking with a column-0 heading rule The fix had grown to five helpers whose only job was undoing one over-permissive rule: tokenizeHeadings treats a heading indented up to three spaces as real, so a `### N.` inside an `expected: |` body was parsed as a row and stole the real row's identity. Every blocker in the last three review rounds came out of that machinery rather than the reported bug — worst of all a UTF-16-versus-code-point frame mismatch that corrupted any document containing an emoji. A UAT test heading is at column 0. The shipped template puts all of them there, no `*UAT*.md` in the repo has an indented one, and the only indented `### N.` lines in the tree are the adversarial fixtures that must not parse. Requiring column 0 makes a scalar-interior heading a non-heading by construction, so maskBlockScalarBodies, indentWidthOf and BLOCK_SCALAR_OPENER_RE are gone along with the mask-invariant test that existed only to guard them. The frame bug is now structurally unreachable: no code-point array or offset splicing remains. The premise was incomplete and the reviewer caught it rather than forcing it through. Masking had been doing double duty — it also hid indented FENCE delimiters from the tokenizer, so removing it let a two-space fence inside a scalar body swallow a later column-0 row. The alternative on offer was to rewrite that test to assert the row is merely counted, which is a behavior regression dressed as a passing suite. Instead there is a small line-based pass that blanks only indented fence delimiters — same "column 0 is structure" rule extended consistently, no YAML knowledge, and line-based by construction so the frame bug cannot come back. It was proven load-bearing by a negative control: reverting the wiring reproduces the regression exactly. Kept, because they fix defects column-0 does not touch: the fence clipping that stops one row reading another's `expected:` from inside a fence, and the folded scalar handling that stopped `expected: >` publishing the literal ">". Also corrects a comment left pointing at a symbol this commit deletes. * fix(#3707): blank neutralized fence bodies, and give both parse paths one grammar Reviewing the simplification found two more, and the first is row theft again — the sixth time this class has surfaced on this issue, and the second time inside a mitigation written to stop it. Neutralizing an indented fence blanked only its DELIMITER lines. If the block's body held a column-0 `### N.`, un-hiding the delimiters made that line a real heading, which then took the preceding row's fields: an `### 1. Alpha` document came back as a single row 9 named Phantom, with Alpha gone. Neutralized blocks are now blanked open-to-close, body included, which is the honest reading of the intent — an indented fence inside a scalar is content, so nothing in it should be able to produce structure. The raw block is still what the expected extractor reads, so a legitimate `expected: |` carrying a fenced code sample keeps its full text. The two parse paths also disagreed about what a test row IS. parseFirstPendingTest filtered on `^\d+\.\s+` while parseUatItemsWithStats used `^\d+\.(?!\d)`, so `### 3.Foo` was a row when audited and not a row when resumed. Both now share one predicate and one extractor. The extractor mattered as much as the filter: the checkpoint path's name-mandatory pattern would have skipped exactly the shapes the widened filter admits, so fixing the filter alone would have moved the divergence down a line rather than closing it. Also from the security pass: parseUatItems had become an export with no callers and no direct test once both consumers moved to the stats form. It stays, since deleting an exported symbol from a shipped module is a contract change and not this issue's business, but it is now documented as the items-only wrapper and has a test. And the PASS check lowercased a value that extraction had already lowercased; normalization now happens once. * fix(#3707): revert the whole-block fence blanking, and pin the rule instead My previous commit over-corrected and the suite caught it. Blanking a neutralized fence block open-to-close destroys content legitimately living between the delimiters, and on an UNTERMINATED opener it blanks to EOF and deletes every later row. No framing makes that correct, and it is what turned two earlier tests red. The "blocker" that prompted it was my own misreading. This change adopted the rule that column 0 is structure and indentation is content. Under that rule an indented fence delimiter is not a fence, so a column-0 `### 9.` sitting between two indented delimiters genuinely IS a heading, and a `result:` after it genuinely belongs to it. That document is malformed and the parser reading it that way is consistent, not stealing. Nothing is silently lost either: the row whose result was taken surfaces as the parse gap. So the blanking is back to delimiters only, and rather than leaving the question open, the behavior is now pinned by a test asserting the rows by identity, with the rule stated at the site — so the next person does not oscillate the way I just did. Kept from the reverted commit: the shared row-heading grammar and extractor across both parse paths, the parseUatItems wrapper documentation and its test, and the single point of lowercasing. * fix(#3707): scope the shortfall scan, and stop neutralized content becoming structure The security review found a HIGH that is the earlier frame-mismatch bug wearing different clothes. The shortfall scan compared a SECTION-scoped raw line count — the `## Tests` body — against a DOCUMENT-wide token count. So a single legal `### N.` row anywhere outside `## Tests` decremented the shortfall and switched the fence-straddle detector off: an identical `## Tests` section went from `headingsSeen 1, parse_gap true` to `headingsSeen 0, no gap` purely because a `## Prior` section existed. A document whose rows render as ordinary blocked rows in any CommonMark renderer audited as totally clean. Both sides of the comparison now come from the same surface, by filtering tokens to the scan span rather than re-tokenizing, so there is no second offset basis to keep in step. Neutralizing a fence could also promote its former CONTENT into structure: a column-0 delimiter run inside an indented pair became an opener once the enclosing delimiters were blanked, hiding every later heading to EOF. Column-0 delimiter-shaped lines inside a neutralized block are now blanked too — and only those, so a column-0 heading between neutralized delimiters is still a heading (the pinned behaviour) and the field lines of a row living between two scalars still survive. The reviewer corrected my repro while fixing it: an even number of inner runs re-pairs and hides nothing, so the live shape needs an odd one, and both are now tests. Two more from that pass. A legal scalar header carrying a trailing comment (`expected: | # sample`) failed the end-anchored grammar, publishing the literal header and raising a false gap. And the indented-row counter walked backwards per row: 3.6 seconds at sixteen thousand rows, now 15ms, via one forward pass — though the reviewer also established my example was not the quadratic shape, which needs an uninterrupted scalar body. Carried in from the previous round: the indented-row counter keys on any block scalar rather than only `expected:`, so a template-sanctioned `reported: |` holding user prose with a heading-shaped line no longer raises a false gap; and `reason:`/`blocked_by:` read block scalars through the same shared extractor instead of publishing the literal `"|"`, which also means a multi-line reason can finally reach categorizeItem — a `reason:` mentioning a server now categorizes as server_blocked, which was impossible while the value was thrown away. * fix(#3707): make the shortfall scan whole-document on both sides Second HIGH in this area, and the diagnosis is the useful part: I closed the first one by making the two sides agree, but I did it by NARROWING the token side to the `## Tests` span while the parse side stayed whole-document. Rows outside that section are still parsed and surfaced when visible, so when a fence straddled one it fell through both sides of the comparison — no item, no gap, file never entered the results at all. A `## Regression Tests` section, or a second `## Tests` (collectSection takes the first), audited as totally clean while origin/next surfaced those rows. Both sides are whole-document now. Symmetry is the property that matters here; every attempt to be clever about which scope to compare has produced one of these, twice at HIGH severity. That reinstates a known over-report, deliberately: a `### N.`-shaped line inside a closed fence in a `## Notes` section — the ordinary way to document the row format — counts as a suppressed row and raises a gap on a file with nothing missing. Noisy, but visible and fail-safe, against two silent false-cleans on the other side of the trade. This issue exists to eliminate false cleans, so the trade goes that way, and the reasoning is written at the site so it does not get optimized back. Three existing tests encoded the retired scoping and are replaced rather than worked around: two now assert the accepted over-report, and one asserting a 4-space row is "not counted" was already contradicted by widening the counter to any indentation — refusing to PARSE a 4-space heading is right, refusing to COUNT it reopened the hole the counter exists to close. Also in this commit, from the same review round: the inner-delimiter sweep tested a column-0-anchored pattern, so an INDENTED delimiter inside a neutralized block was still promoted to structure and lost a row; it is indent-tolerant now. A refinement was identified and deliberately not taken — keying the documentation-sample exemption on the fence info string rather than on section scope. It is content-based and symmetric, so it would not reintroduce the asymmetry, but it belongs in its own change rather than riding this one. * docs(#3707): correct two claims in the over-report justification Both from review, both comment-only, and both matter because they would mislead the next person into "fixing" something correct. The over-report note called the triggering shape "the ordinary way to document the row format". It is narrower than that: the scan requires literal digits, so the conventional placeholder `### N. Name` does not trigger it at all — only a sample written with real numbers does, and no phase UAT file in-tree has one, only the shipped template, which selectPhaseUatFiles never scans. A maintainer who tested the documented placeholder form would find no over-report and could reasonably conclude the pin was stale. That is now stated, and it also makes the trade look better than I claimed: the real-world frequency is lower. The attribution guard is described as structural rather than positional. It is positional in one respect: the walk stops at the nearest column-0 line, so a block scalar nested inside a `## Gaps` bullet is transparent to it and a heading-shaped line in that value gets counted. Same accepted over-report, reached by a path the comment did not mention — recorded so it is not later mistaken for a new defect. * fix(#3707): a complete status no longer switches off the parse-gap detector The security review named this as the last silent-clean path in the change, and its phrase is the right one: a self-declared kill switch over the very detector this issue built. A file whose frontmatter said `status: complete` was omitted unconditionally, so one containing a fence-straddled `result: blocked` computed headingsSeen = 1 — the detector fired — and then emitted no entry at all. The audit reported nothing. The predicate is now status-independent: a file is surfaced when blocks were seen but yielded nothing, whatever it claims about itself. A terminal status is an assertion by the author, and an assertion is exactly what must not be allowed to suppress the signal that would contradict it. What does not change is the thing the status is actually for — a complete file with nothing to parse, and a complete file whose rows all parse and all pass, both stay silent, verified through the real CLI. I replaced a control test of mine, and it is worth saying why that is not a weakening: its name was already false. "A zero-item file with a complete status is still omitted" used a fixture with a `### 1.` block carrying no `result:` line, so headingsSeen was 1 — it was never a zero-item file, it was the kill switch itself, pinned. The intent it claimed is now covered by two stricter tests, one for a file with no blocks and one for a file where every row parses and passes, each asserting both that no entry exists and that no items are counted, where the old test asserted only the former. Everything else is byte-identical: 61 regression cases and the non-complete equivalents of all four shapes produce exactly the same output as before, with the delta confined to the two cells this change is meant to move. * fix(#3707): close the moved kill switch, and keep the archive out of the live gate Both reviewers independently found that closing the kill switch on one surface left it standing on the other. cmdAuditUat dropped the terminal-status guard, but buildUatRows in planning-inspect kept it — and its comment justified that by claiming to mirror a guard cmdAuditUat no longer had. One byte-identical file with `status: complete` and a fence-straddled `result: blocked` reported parse_gap through audit-uat while planning-inspect published `uat.scope: "complete"` with no diagnostic at all. That is the repo's own generative-fix-divergence class, and no test pinned that arm, which is why it survived. The clause and the false comment are gone and the arm now has tests. Removing it exposed a MAJOR the security pass had not reached: archived phase dirs are deliberately not milestone-filtered and archived UAT files are `complete` by definition, so status-independence newly admitted the entire project archive. One live pending row plus four signed-off milestones produced parse_gap_files 4 — and since progress.md gates Verification Debt on that counter, a mature project would have warned on every run, forever, about closed history no user action can clear. Warning fatigue that buries the next real gap is the feature defeating itself. So the counter is split rather than suppressed: `parse_gap_files` counts live phases only and remains the gate, `archived_parse_gap_files` carries the rest, and every archived entry stays in `results` with its parse_gap and its milestone. Nothing became silent; the live signal stayed actionable. Both workflows report the archived bucket as closed history rather than as something to act on. The scope cascade is also decoupled, on the security reviewer's advice that it is load-bearing here rather than a follow-up: `uat.scope` still reports TRUNCATED honestly so no completeness is claimed over an unread row, while the accepted fence-shortfall over-report no longer flips the aggregate fold that withholds a phase's percentage. A genuinely unreadable file degrades as before. Also corrects a frequency claim of mine: "no phase UAT file in-tree triggers this" was true over a sample of zero, since the only UAT file in the tree is the shipped template. The comment now says the shape is uncommon, which is what I can actually support. * fix(#3707): state that the live/archived split does not extend to outstanding_debt Review MINOR: the split's rationale read as though it governed every counter, but `summary.total_items` was never split — so a single archived `result: pending` row re-trips the same Verification Debt warning the split exists to stop. The asymmetry is deliberate: an archived parse gap is a row nobody can read, so the warning can never be cleared, whereas an archived pending row is legible work someone can still pay down by retesting. Debt that can be settled stays counted. The prose now says so at the point of the claim, instead of leaving the next reader to file it as a miss. Also rewrites the changeset, which described only the secondary fixes and omitted all three defects the issue actually reports: `result: issue` dropped, any wrapped or block-scalar `expected:` never matched at all, and the phase vanishing outright. * fix(#3707): count every parse gap, dropping the live/archived split The split had two regressions, both reproduced through the real CLI, and its premise was false. uat.cts carries #2766's rationale ~190 lines above the code I added: 'Outstanding UAT items do not stop mattering when a milestone closes: a deferred human-UAT scenario or a skipped live-stack test is exactly what gets archived still-open.' So 'archived UAT files are complete by definition' was never true, and the split rested on it. Regression 1: archived-ness was inferred from path shape alone. A phase in the CURRENT milestone, status in_progress, filed under .planning/milestones/v1.1-phases/ was classified archived and demoted out of the gate — live work reported as closed history that needs no action. Regression 2: the split was one-sided. total_items has no archived split, so an archived outstanding row that PARSES gates Verification Debt while the identical row that fails to parse was informational. The parse failure was what buried the debt — the exact bug class this issue exists to fix, re-created one surface over. parse_gap_files counts every parse_gap entry again, archived or not, so it agrees with total_items on what archived means. The pre-existing archived_milestone field and archived-phase scanning are untouched. Regression tests added for both cases. * fix(#3707): correct the changeset clause left behind by the split revert The changeset was rewritten before the split was removed, so its final clause still claimed archived parse gaps are counted separately because signed-off history is not work anyone can act on. There is one counter now, and that premise is the one uat.cts refutes and the revert was made over. This text lands in CHANGELOG.md verbatim, so it would have shipped a description of behavior the code does not have. * chore(#3707): backfill changeset pr number --------- Co-authored-by: sim <sim@local>
This commit is contained in:
5
.changeset/rapid-tunas-leap.md
Normal file
5
.changeset/rapid-tunas-leap.md
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
---
|
||||||
|
type: Fixed
|
||||||
|
pr: 3887
|
||||||
|
---
|
||||||
|
**`audit-uat` no longer reports a clean result for UAT files it silently failed to read.** A phase with three outstanding tests reported zero and then vanished from the report entirely, so nothing cued the reader to go and look. Rows using the template's own `result: issue` outcome, or any wrapped or `expected: |` block-scalar description, were dropped — the second kind was never matched at all, so its result was never read whatever it said. A result token the parser does not recognize is now surfaced rather than discarded, uppercase tokens (`PENDING`, `Blocked`) categorize correctly, a result line with trailing text (`result: pending (blocked on staging)`) is matched again, and an interleaved `## Gaps` section no longer bleeds its reason onto the preceding row. A file whose blocks genuinely fail to parse is reported as a parse gap and counted, so `audit-uat` and `progress` stop declaring all-clear over it — including files in archived milestones, which can still hold a deferred scenario someone left open. (#3707)
|
||||||
@@ -14,7 +14,7 @@ AUDIT=$(gsd_run query audit-uat --raw)
|
|||||||
|
|
||||||
Parse JSON for `results` array and `summary` object.
|
Parse JSON for `results` array and `summary` object.
|
||||||
|
|
||||||
If `summary.total_items` is 0:
|
If `summary.total_items` is 0 AND `summary.parse_gap_files` is 0:
|
||||||
```
|
```
|
||||||
## All Clear
|
## All Clear
|
||||||
|
|
||||||
@@ -22,6 +22,8 @@ No outstanding UAT or verification items found across all phases.
|
|||||||
All tests are passing, resolved, or diagnosed with fix plans.
|
All tests are passing, resolved, or diagnosed with fix plans.
|
||||||
```
|
```
|
||||||
Stop here.
|
Stop here.
|
||||||
|
|
||||||
|
If `summary.total_items` is 0 but `summary.parse_gap_files` is greater than 0, this is NOT all-clear — one or more files have test blocks that could not be parsed and may be hiding outstanding work. Continue to the `categorize` and `present` steps below; `present` renders the Unparsed section from these `parse_gap: true` entries even though `items` is empty for a mixed project.
|
||||||
</step>
|
</step>
|
||||||
|
|
||||||
<step name="categorize">
|
<step name="categorize">
|
||||||
@@ -45,6 +47,19 @@ For each item in "Testable Now", use Grep/Read to check if the underlying featur
|
|||||||
</step>
|
</step>
|
||||||
|
|
||||||
<step name="present">
|
<step name="present">
|
||||||
|
If `summary.parse_gap_files` is greater than 0, render this section FIRST (before the `## UAT Audit Report` below, or standalone if `total_items` is also 0). Filter `results` for entries with `parse_gap: true`:
|
||||||
|
```
|
||||||
|
## Unparsed UAT Files ({parse_gap_files} files)
|
||||||
|
|
||||||
|
| Phase | File | Unparsed Blocks |
|
||||||
|
|-------|------|------------------|
|
||||||
|
| {phase} | {file_path} | {unparsed_blocks} |
|
||||||
|
...
|
||||||
|
|
||||||
|
These files have `### N.` test blocks with no readable `result:` line. Fix the file's structure, then re-run the audit.
|
||||||
|
```
|
||||||
|
This fires for ANY project with `parse_gap_files > 0`, including a mixed project where other phases also have real outstanding `items` — the two sections are not mutually exclusive. An outstanding item does not stop mattering because its phase belongs to an already-archived milestone (a deferred human-UAT scenario or a `skipped` live-stack test is exactly what gets archived still-open), so an archived phase's parse gap is listed in this same table, unfiltered by `archived_milestone`. If `total_items` is 0, stop after this section (there is no `## UAT Audit Report` to present). Otherwise continue below.
|
||||||
|
|
||||||
Present the audit report:
|
Present the audit report:
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -252,11 +252,13 @@ Scan ALL phases in the current milestone for outstanding verification debt using
|
|||||||
DEBT=$(gsd_run query audit-uat --raw 2>/dev/null)
|
DEBT=$(gsd_run query audit-uat --raw 2>/dev/null)
|
||||||
```
|
```
|
||||||
|
|
||||||
Parse JSON for `summary.total_items` and `summary.total_files`.
|
Parse JSON for `summary.total_items`, `summary.total_files`, and `summary.parse_gap_files`.
|
||||||
|
|
||||||
Track: `outstanding_debt` — `summary.total_items` from the audit.
|
Track: `outstanding_debt` — `summary.total_items` from the audit. Track `parse_gap_files` — `summary.parse_gap_files` from the audit.
|
||||||
|
|
||||||
**If outstanding_debt > 0:** Add a warning section to the progress report output (in the `report` step), placed between "## What's Next" and the route suggestion:
|
`summary.parse_gap_files` counts EVERY file with `parse_gap: true`, archived or not — the same as `outstanding_debt` (`summary.total_items`), which has no archived split either. An outstanding item does not stop mattering because its phase belongs to an already-archived milestone: a deferred human-UAT scenario or a `skipped` live-stack test is exactly what gets archived still-open, so an archived parse gap is exactly as much unread outstanding work as an archived `result: pending` row.
|
||||||
|
|
||||||
|
**If outstanding_debt > 0 OR parse_gap_files > 0:** Add a warning section to the progress report output (in the `report` step), placed between "## What's Next" and the route suggestion:
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
## Verification Debt ({N} files across prior phases)
|
## Verification Debt ({N} files across prior phases)
|
||||||
@@ -266,12 +268,13 @@ Track: `outstanding_debt` — `summary.total_items` from the audit.
|
|||||||
| {phase} | {filename} | {pending_count} pending, {skipped_count} skipped, {blocked_count} blocked |
|
| {phase} | {filename} | {pending_count} pending, {skipped_count} skipped, {blocked_count} blocked |
|
||||||
| {phase} | {filename} | human_needed — {count} items |
|
| {phase} | {filename} | human_needed — {count} items |
|
||||||
| {phase} | {filename} | {unresolved_count} deferred items |
|
| {phase} | {filename} | {unresolved_count} deferred items |
|
||||||
|
| {phase} | {filename} | unparsed — test blocks with no readable `result:` line |
|
||||||
|
|
||||||
Review: `/gsd:audit-uat ${GSD_WS}` — full cross-phase audit
|
Review: `/gsd:audit-uat ${GSD_WS}` — full cross-phase audit
|
||||||
Resume testing: `/gsd:verify-work {phase} ${GSD_WS}` — retest specific phase
|
Resume testing: `/gsd:verify-work {phase} ${GSD_WS}` — retest specific phase
|
||||||
```
|
```
|
||||||
|
|
||||||
This is a WARNING, not a blocker — routing proceeds normally. The debt is visible so the user can make an informed choice.
|
The unparsed row comes from `results` entries with `parse_gap: true` (`summary.parse_gap_files` counts exactly those, archived or not). This is a WARNING, not a blocker — routing proceeds normally. The debt is visible so the user can make an informed choice.
|
||||||
|
|
||||||
**Step 1.7: Check verification status for the current phase**
|
**Step 1.7: Check verification status for the current phase**
|
||||||
|
|
||||||
|
|||||||
@@ -59,7 +59,7 @@ import gapCheckerMod = require('./gap-checker.cjs');
|
|||||||
const { parseRequirements } = gapCheckerMod;
|
const { parseRequirements } = gapCheckerMod;
|
||||||
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
||||||
import uatMod = require('./uat.cjs');
|
import uatMod = require('./uat.cjs');
|
||||||
const { parseUatItems, selectPhaseUatFiles } = uatMod;
|
const { parseUatItemsWithStats, selectPhaseUatFiles } = uatMod;
|
||||||
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
||||||
import phaseLifecycleMod = require('./phase-lifecycle.cjs');
|
import phaseLifecycleMod = require('./phase-lifecycle.cjs');
|
||||||
const { clampPercent } = phaseLifecycleMod;
|
const { clampPercent } = phaseLifecycleMod;
|
||||||
@@ -808,7 +808,7 @@ function buildUatRows(
|
|||||||
phaseDirName: string,
|
phaseDirName: string,
|
||||||
diagnostics: Diagnostic[],
|
diagnostics: Diagnostic[],
|
||||||
planningRoot: string,
|
planningRoot: string,
|
||||||
): { items: unknown[]; scope: Scope } {
|
): { items: unknown[]; scope: Scope; foldScope: Scope } {
|
||||||
const phaseDir = path.join(phasesDir, phaseDirName);
|
const phaseDir = path.join(phasesDir, phaseDirName);
|
||||||
// GAP 1 (#2790 follow-up security review): same rationale as
|
// GAP 1 (#2790 follow-up security review): same rationale as
|
||||||
// `buildPlanRows` above — `readdirSync` below FOLLOWS a directory symlink,
|
// `buildPlanRows` above — `readdirSync` below FOLLOWS a directory symlink,
|
||||||
@@ -823,7 +823,7 @@ function buildUatRows(
|
|||||||
subject: phaseDirName,
|
subject: phaseDirName,
|
||||||
detail: 'Phase directory could not be listed; UAT presence is unknown.',
|
detail: 'Phase directory could not be listed; UAT presence is unknown.',
|
||||||
});
|
});
|
||||||
return { items: [], scope: SCOPE.UNREADABLE };
|
return { items: [], scope: SCOPE.UNREADABLE, foldScope: SCOPE.UNREADABLE };
|
||||||
}
|
}
|
||||||
let entries: string[];
|
let entries: string[];
|
||||||
try {
|
try {
|
||||||
@@ -834,7 +834,7 @@ function buildUatRows(
|
|||||||
subject: phaseDirName,
|
subject: phaseDirName,
|
||||||
detail: 'Phase directory could not be listed; UAT presence is unknown.',
|
detail: 'Phase directory could not be listed; UAT presence is unknown.',
|
||||||
});
|
});
|
||||||
return { items: [], scope: SCOPE.UNREADABLE };
|
return { items: [], scope: SCOPE.UNREADABLE, foldScope: SCOPE.UNREADABLE };
|
||||||
}
|
}
|
||||||
|
|
||||||
const uatFiles = selectPhaseUatFiles(entries, phaseDirName);
|
const uatFiles = selectPhaseUatFiles(entries, phaseDirName);
|
||||||
@@ -844,11 +844,12 @@ function buildUatRows(
|
|||||||
subject: phaseDirName,
|
subject: phaseDirName,
|
||||||
detail: 'No UAT document for this phase. This does not affect phase acceptance.',
|
detail: 'No UAT document for this phase. This does not affect phase acceptance.',
|
||||||
});
|
});
|
||||||
return { items: [], scope: SCOPE.COMPLETE };
|
return { items: [], scope: SCOPE.COMPLETE, foldScope: SCOPE.COMPLETE };
|
||||||
}
|
}
|
||||||
|
|
||||||
const items: unknown[] = [];
|
const items: unknown[] = [];
|
||||||
let scope: Scope = SCOPE.COMPLETE;
|
let scope: Scope = SCOPE.COMPLETE;
|
||||||
|
let foldScope: Scope = SCOPE.COMPLETE;
|
||||||
for (const file of uatFiles) {
|
for (const file of uatFiles) {
|
||||||
const doc = readDocument(path.join(phaseDir, file), planningRoot);
|
const doc = readDocument(path.join(phaseDir, file), planningRoot);
|
||||||
if (doc.text === null) {
|
if (doc.text === null) {
|
||||||
@@ -858,11 +859,65 @@ function buildUatRows(
|
|||||||
detail: 'UAT document exists but could not be read.',
|
detail: 'UAT document exists but could not be read.',
|
||||||
});
|
});
|
||||||
scope = SCOPE.TRUNCATED;
|
scope = SCOPE.TRUNCATED;
|
||||||
|
foldScope = SCOPE.TRUNCATED;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
items.push(...parseUatItems(doc.text));
|
// #3707-class false-clean, second surface (security review finding 1):
|
||||||
|
// `parseUatItemsWithStats`'s `headingsSeen` counts `### N.` test blocks
|
||||||
|
// that yielded ZERO items (a row missing its `result:` line, or otherwise
|
||||||
|
// unrecognised) — the audit-uat side (`cmdAuditUat`, above) already flags
|
||||||
|
// this as `parse_gap`. Reading the file successfully is not the same as
|
||||||
|
// deriving every row from it, so the gap is ALWAYS REPORTED.
|
||||||
|
//
|
||||||
|
// #3078 round-8 HIGH — NO FRONTMATTER KILL SWITCH. There is deliberately
|
||||||
|
// no `status !== 'complete'` term here. `cmdAuditUat` dropped its own such
|
||||||
|
// guard (see the long rationale at src/uat.cts, above `headingsSeen > 0`):
|
||||||
|
// a terminal status is an ASSERTION BY THE AUTHOR, and an assertion must
|
||||||
|
// not be able to switch off the detector that would contradict it. A guard
|
||||||
|
// here let one word of frontmatter turn a fence-straddled `result: blocked`
|
||||||
|
// into an affirmative `scope: "complete"` with ZERO diagnostics. The
|
||||||
|
// condition is `headingsSeen > 0` alone, matching src/uat.cts's own check
|
||||||
|
// line for line. Do not re-add a status term to "align the surfaces" — the
|
||||||
|
// surfaces are aligned precisely BY its absence.
|
||||||
|
//
|
||||||
|
// #3078 round-8 — REPORTING THE GAP AND WITHHOLDING THE PERCENTAGE ARE TWO
|
||||||
|
// DIFFERENT DECISIONS, so they get two different fields. `scope` is what
|
||||||
|
// this phase's own `uat.scope` reports: it stays honest and goes TRUNCATED
|
||||||
|
// for every gap, because a document that did not yield all its rows must
|
||||||
|
// never carry an affirmative `"complete"`. `foldScope` is what is handed to
|
||||||
|
// `worstScope` in the caller, and it is the one with teeth — a non-COMPLETE
|
||||||
|
// fold raises `phase_scope_degraded` AND, via `phaseScope`/`makeFraction`,
|
||||||
|
// withholds BOTH progress percentages for the WHOLE milestone.
|
||||||
|
//
|
||||||
|
// Those teeth cannot bite on `shortfallBlocks`: that is the subset of
|
||||||
|
// `headingsSeen` produced by the fence-suppression shortfall scan, the ONE
|
||||||
|
// gap class `src/uat.cts` documents as an ACCEPTED OVER-REPORT — a
|
||||||
|
// closed-fence documentation sample written with literal digits is provably
|
||||||
|
// indistinguishable from a genuinely fence-straddled row (fence-closedness
|
||||||
|
// is identical in both), so the scan deliberately over-reports. A COMPLETED
|
||||||
|
// phase is terminal: nobody reopens its UAT file, so a shortfall-only
|
||||||
|
// over-report there would withhold the project's percentages in every
|
||||||
|
// future audit FOREVER over a paragraph of prose. Reported-and-dismissible
|
||||||
|
// is the right shape for it; silent is not, and neither is permanent.
|
||||||
|
//
|
||||||
|
// Every OTHER gap class (a block with no `result:` line, an unattributed
|
||||||
|
// indented row, an unterminated fence) has no such false-positive story and
|
||||||
|
// still degrades the fold, as does the unreadable-FILE case above — which
|
||||||
|
// is what `SCOPE.TRUNCATED` means per src/planning-scope.cts: the scan
|
||||||
|
// could not SEE part of the evidence.
|
||||||
|
const { items: fileItems, headingsSeen, shortfallBlocks } = parseUatItemsWithStats(doc.text);
|
||||||
|
items.push(...fileItems);
|
||||||
|
if (headingsSeen > 0) {
|
||||||
|
diagnostics.push({
|
||||||
|
code: INSPECT_DIAGNOSTIC.UAT_UNREADABLE,
|
||||||
|
subject: `${phaseDirName}/${file}`,
|
||||||
|
detail: `UAT document has ${headingsSeen} test block(s) with no parseable result; unresolved is not a complete answer.`,
|
||||||
|
});
|
||||||
|
scope = SCOPE.TRUNCATED;
|
||||||
|
if (headingsSeen > shortfallBlocks) foldScope = SCOPE.TRUNCATED;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return { items, scope };
|
return { items, scope, foldScope };
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── Progress ─────────────────────────────────────────────────────────────────
|
// ─── Progress ─────────────────────────────────────────────────────────────────
|
||||||
@@ -1171,7 +1226,14 @@ function buildPlanningInspect(cwd: string): Record<string, unknown> {
|
|||||||
const phaseId = token ? token[1] : null;
|
const phaseId = token ? token[1] : null;
|
||||||
const { goal, dependencies } = buildPhaseGoalAndDependencies(cwd, roadmapDoc, phaseId, phase.dir, diagnostics);
|
const { goal, dependencies } = buildPhaseGoalAndDependencies(cwd, roadmapDoc, phaseId, phase.dir, diagnostics);
|
||||||
|
|
||||||
const folded = worstScope(phase.scope, plans.scope, uat.scope, goal.scope, dependencies.scope);
|
// `uat.foldScope`, NOT `uat.scope` (#3078 round-8). The two differ for
|
||||||
|
// exactly one case: a UAT document whose ONLY parse gap is the
|
||||||
|
// fence-suppression shortfall, `src/uat.cts`'s documented ACCEPTED
|
||||||
|
// OVER-REPORT class. That still reports honestly on the row itself
|
||||||
|
// (`uat.scope === "truncated"` plus the `uat_unreadable` diagnostic below),
|
||||||
|
// but it must not raise `phase_scope_degraded` and must not withhold the
|
||||||
|
// milestone's percentages — see `buildUatRows` for the full rationale.
|
||||||
|
const folded = worstScope(phase.scope, plans.scope, uat.foldScope, goal.scope, dependencies.scope);
|
||||||
if (folded !== SCOPE.COMPLETE) {
|
if (folded !== SCOPE.COMPLETE) {
|
||||||
diagnostics.push({
|
diagnostics.push({
|
||||||
code: INSPECT_DIAGNOSTIC.PHASE_SCOPE_DEGRADED,
|
code: INSPECT_DIAGNOSTIC.PHASE_SCOPE_DEGRADED,
|
||||||
|
|||||||
1031
src/uat.cts
1031
src/uat.cts
File diff suppressed because it is too large
Load Diff
12
tests/emitted-drift-acks/3707-parse-gap-reporting.json
Normal file
12
tests/emitted-drift-acks/3707-parse-gap-reporting.json
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
{
|
||||||
|
"$comment": "Growth ack (#2914 fragment). Reason: #3707 fixed `audit-uat` reporting zero outstanding items for a UAT file that had three, then dropping the phase from the report entirely. Fixing the parser alone was not enough — both reviewers found independently that the fix was invisible end to end. A file that parses to zero items is now reported with `parse_gap: true` and counted in a new `summary.parse_gap_files`, but parse-gap entries carry no items, and BOTH of these workflows gated their user-visible output on `summary.total_items === 0`. audit-uat.md printed '## All Clear ... Stop here.' and progress.md suppressed its Verification Debt section, so the headline symptom — the phase vanishing — still reproduced for a reader while only the raw JSON had changed. The growth in each file is the widened gate plus the branch that actually reports the unparsed files, with their phase and path, so the reader gets the cue to go and look. Prose is the product here: these steps ARE what an executing agent reads and acts on, so a smaller form would just move the omission. A later revision on this branch tried splitting `summary.parse_gap_files` into a live-only counter plus a separate `summary.archived_parse_gap_files` for archived phases, on the premise that an archived milestone's UAT files are complete by definition. That premise is false — #2766's own rationale states outstanding UAT items do not stop mattering when a milestone closes, since a deferred human-UAT scenario or a `skipped` live-stack test is exactly what gets archived still-open — and the split produced two executed regressions: a phase belonging to the CURRENT milestone but filed under the `milestones/` archive tree was demoted out of the gate, and an archived outstanding row that failed to parse gave `parse_gap_files: 0` where the identical row, parsed, gave `total_items: 1` — burying exactly the debt class #3707 exists to surface. The split is reverted; `parse_gap_files` is ONE counter again, counting every `parse_gap: true` entry regardless of `archived_milestone`, mirroring `total_items`, which never had a split. Both workflow files consequently shrink back toward (but not fully to) their #3707-only size: they retain the original all-clear-gate widening and unparsed-files reporting, but drop every live/archived-split rule, filter, and informational sub-section added afterward. audit-uat.md origin/next 5582 -> 7124 bytes (+1542); progress.md origin/next 24791 -> 25626 bytes (+835). Both still exceed their origin/next size — the ack remains armed for the #3707 all-clear-gate widening and unparsed-file reporting alone, which is real, permanent growth; only the archived-split narrative and its rules are gone.",
|
||||||
|
"version": 1,
|
||||||
|
"paths": {
|
||||||
|
"audit-uat.md": {
|
||||||
|
"reason": "#3707: the all-clear gate widens from `total_items === 0` to `total_items === 0 && parse_gap_files === 0`, and a new branch reports unparsed UAT files (`parse_gap: true` entries) with phase and path — filtered on `parse_gap: true` alone, never on `archived_milestone`. Without it the command still prints 'All Clear' for a phase whose rows it could not parse, which is the exact symptom this issue reports. A live/archived split was added and then reverted on this branch (see `$comment`): the split's extra rule in `initialize`, the narrowed Unparsed-table filter, and the separate 'Unparsed UAT Files in Archived Milestones' informational section are all removed, so the file settles at origin/next 5582 -> 7124 bytes (+1542, final)."
|
||||||
|
},
|
||||||
|
"progress.md": {
|
||||||
|
"reason": "#3707: the Verification Debt section no longer gates on outstanding items alone (`outstanding_debt > 0 OR parse_gap_files > 0`), and the table gains a row naming the unparsed file, sourced from `results` entries with `parse_gap: true` — with no `archived_milestone` filter. A live/archived split was added and then reverted on this branch (see `$comment`): the split's `archived_parse_gap_files` tracking, the paragraph explaining why it must not fold into the gate, the paragraph stating the split deliberately does not extend to `outstanding_debt`, and the standalone informational FYI line are all removed, so the file settles at origin/next 24791 -> 25626 bytes (+835, final)."
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1122,6 +1122,199 @@ describe('planning inspect — evidence kept separate, never folded', () => {
|
|||||||
assert.deepStrictEqual(sortedKeys(phase), EXPECTED_PHASE_ROW_KEYS);
|
assert.deepStrictEqual(sortedKeys(phase), EXPECTED_PHASE_ROW_KEYS);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('uatParseGapNeverClaimsCompleteScopeWithEmptyUnresolved', (t) => {
|
||||||
|
const tmpDir = createTempProject();
|
||||||
|
t.after(() => cleanup(tmpDir));
|
||||||
|
// Security review finding 1 (#3707 second surface): a `### N.` test block
|
||||||
|
// with no `result:` line is a genuine parse gap — the audit-uat side
|
||||||
|
// (`cmdAuditUat`) already flags this file as `parse_gap: true`. Before
|
||||||
|
// the fix, planning-inspect's `buildUatRows` called `parseUatItems`
|
||||||
|
// (which silently drops a gap-only heading from BOTH items and any
|
||||||
|
// scope signal), so this exact file was reported as `uat: { unresolved:
|
||||||
|
// [], scope: 'complete' }` — an affirmative completeness claim over rows
|
||||||
|
// it never actually derived.
|
||||||
|
const phaseDir = declarePhase(tmpDir, '1', 'Foo');
|
||||||
|
writeVerification(phaseDir, '1', 'passed');
|
||||||
|
writeUatDoc(phaseDir, '1', [
|
||||||
|
'### 1. Check something',
|
||||||
|
'expected: it works',
|
||||||
|
'',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const payload = parseInspect(tmpDir);
|
||||||
|
const phase = payload.phases[0];
|
||||||
|
assert.deepStrictEqual(phase.uat.unresolved, []);
|
||||||
|
assert.notStrictEqual(phase.uat.scope, 'complete');
|
||||||
|
assert.strictEqual(phase.uat.scope, 'truncated');
|
||||||
|
assert.ok(payload.diagnostics.some((d) => d.code === 'uat_unreadable' && d.subject.includes('1-UAT.md')));
|
||||||
|
});
|
||||||
|
|
||||||
|
// ─── #3078 round-8: no frontmatter kill switch over the parse-gap detector ──
|
||||||
|
//
|
||||||
|
// `buildUatRows` used to carry `&& status !== 'complete'` alongside its
|
||||||
|
// `headingsSeen > 0` check, long after `cmdAuditUat` dropped the identical
|
||||||
|
// guard. Nothing pinned this arm: the fixture above writes no frontmatter
|
||||||
|
// `status` at all, so the clause was unreachable from the test suite and one
|
||||||
|
// word of frontmatter silently switched off the detector on the real CLI.
|
||||||
|
// These four rows are that pin. Each asserts the SCOPE VALUE and the
|
||||||
|
// DIAGNOSTIC explicitly — "something came back" is what let this survive.
|
||||||
|
|
||||||
|
/** A UAT document with `status:` frontmatter, for the kill-switch rows below. */
|
||||||
|
function writeUatDocWithStatus(phaseDir, phaseToken, status, bodyLines) {
|
||||||
|
writeUatDoc(phaseDir, phaseToken, ['---', `status: ${status}`, '---', '', ...bodyLines]);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A closed fence that OPENS after test 1 and CLOSES after test 2, hiding
|
||||||
|
* test 2's `result: blocked` from the heading tokenizer entirely
|
||||||
|
* (`src/uat.cts`'s fence-straddle case — the row is absent from the token
|
||||||
|
* stream, not merely unparseable).
|
||||||
|
*/
|
||||||
|
const FENCE_STRADDLED_BLOCKED_BODY = [
|
||||||
|
'### 1. Alpha',
|
||||||
|
'expected: a',
|
||||||
|
'result: pass',
|
||||||
|
'',
|
||||||
|
'```',
|
||||||
|
'### 2. Beta',
|
||||||
|
'expected: b',
|
||||||
|
'result: blocked',
|
||||||
|
'```',
|
||||||
|
'',
|
||||||
|
];
|
||||||
|
|
||||||
|
for (const status of ['complete', 'in_progress']) {
|
||||||
|
test(`reportsAFenceStraddledBlockedRowOnAUatFileMarkedStatus_${status}`, (t) => {
|
||||||
|
const tmpDir = createTempProject();
|
||||||
|
t.after(() => cleanup(tmpDir));
|
||||||
|
const phaseDir = declarePhase(tmpDir, '1', 'Foo');
|
||||||
|
writeVerification(phaseDir, '1', 'passed');
|
||||||
|
writeUatDocWithStatus(phaseDir, '1', status, FENCE_STRADDLED_BLOCKED_BODY);
|
||||||
|
|
||||||
|
const payload = parseInspect(tmpDir);
|
||||||
|
const phase = payload.phases[0];
|
||||||
|
// The hidden row yields no item — it was never in the token stream — so
|
||||||
|
// the DIAGNOSTIC is the only channel that reports it. Under the old
|
||||||
|
// guard, `status: complete` produced `scope: "complete"` with ZERO
|
||||||
|
// diagnostics: an affirmative completeness claim over a `blocked` row
|
||||||
|
// the tool never read.
|
||||||
|
assert.strictEqual(phase.uat.scope, 'truncated');
|
||||||
|
assert.deepStrictEqual(phase.uat.unresolved, []);
|
||||||
|
assert.ok(payload.diagnostics.some((d) => d.code === 'uat_unreadable' && d.subject.includes('1-UAT.md')));
|
||||||
|
});
|
||||||
|
|
||||||
|
test(`claimsCompleteUatScopeWithNoDiagnosticWhenEveryRowParsesAndPassesAtStatus_${status}`, (t) => {
|
||||||
|
const tmpDir = createTempProject();
|
||||||
|
t.after(() => cleanup(tmpDir));
|
||||||
|
const phaseDir = declarePhase(tmpDir, '1', 'Foo');
|
||||||
|
writeVerification(phaseDir, '1', 'passed');
|
||||||
|
writeUatDocWithStatus(phaseDir, '1', status, [
|
||||||
|
'### 1. Alpha',
|
||||||
|
'expected: a',
|
||||||
|
'result: pass',
|
||||||
|
'',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const payload = parseInspect(tmpDir);
|
||||||
|
const phase = payload.phases[0];
|
||||||
|
// The other half of the pin: removing the status guard must NOT make
|
||||||
|
// every file noisy. A file with nothing unread still claims `complete`
|
||||||
|
// and raises nothing — otherwise the detector would be uninformative.
|
||||||
|
assert.strictEqual(phase.uat.scope, 'complete');
|
||||||
|
assert.deepStrictEqual(phase.uat.unresolved, []);
|
||||||
|
assert.strictEqual(payload.diagnostics.filter((d) => d.code === 'uat_unreadable').length, 0);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── #3078 round-8: a shortfall is REPORTED, it does not WITHHOLD ──────────
|
||||||
|
//
|
||||||
|
// `phase.uat.scope` (what this phase's UAT evidence is worth) and
|
||||||
|
// `phase.scope` (the `worstScope` fold, which gates `phase_scope_degraded`
|
||||||
|
// and — via `progress.*` — the milestone's percentages) are separate
|
||||||
|
// decisions. The fence-suppression shortfall is `src/uat.cts`'s documented
|
||||||
|
// ACCEPTED OVER-REPORT class: a closed-fence documentation sample with
|
||||||
|
// literal digits is indistinguishable from a straddled row. A COMPLETED
|
||||||
|
// phase is terminal, so letting that class withhold percentages would make a
|
||||||
|
// paragraph of prose suppress the project's numbers in every future audit
|
||||||
|
// forever.
|
||||||
|
|
||||||
|
test('shortfallAloneReportsTheGapWithoutDegradingThePhaseOrWithholdingThePercentage', (t) => {
|
||||||
|
const tmpDir = createTempProject();
|
||||||
|
t.after(() => cleanup(tmpDir));
|
||||||
|
const phaseDir = declarePhase(tmpDir, '1', 'Foo');
|
||||||
|
writeVerification(phaseDir, '1', 'passed');
|
||||||
|
// A `## Notes` section documenting the row format inside a CLOSED fence —
|
||||||
|
// literal digits, so `TEST_HEADING_LINE_RE` counts it and the shortfall
|
||||||
|
// fires. This is prose, not an outstanding row.
|
||||||
|
writeUatDocWithStatus(phaseDir, '1', 'complete', [
|
||||||
|
'# UAT',
|
||||||
|
'',
|
||||||
|
'## Notes',
|
||||||
|
'',
|
||||||
|
'How to write a row:',
|
||||||
|
'',
|
||||||
|
'```',
|
||||||
|
'### 1. Example Row',
|
||||||
|
'expected: x',
|
||||||
|
'result: pass',
|
||||||
|
'```',
|
||||||
|
'',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const payload = parseInspect(tmpDir);
|
||||||
|
const phase = payload.phases[0];
|
||||||
|
// REPORTED …
|
||||||
|
assert.ok(payload.diagnostics.some((d) => d.code === 'uat_unreadable' && d.subject.includes('1-UAT.md')));
|
||||||
|
assert.strictEqual(phase.uat.scope, 'truncated');
|
||||||
|
// … but NOT degraded, and the numbers still publish.
|
||||||
|
assert.strictEqual(phase.scope, 'complete');
|
||||||
|
assert.strictEqual(payload.diagnostics.filter((d) => d.code === 'phase_scope_degraded').length, 0);
|
||||||
|
assert.strictEqual(payload.diagnostics.filter((d) => d.code === 'percent_withheld').length, 0);
|
||||||
|
assert.strictEqual(payload.progress.accepted_phases.percent, 100);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('aNonShortfallParseGapStillDegradesThePhaseAndWithholdsThePercentage', (t) => {
|
||||||
|
const tmpDir = createTempProject();
|
||||||
|
t.after(() => cleanup(tmpDir));
|
||||||
|
const phaseDir = declarePhase(tmpDir, '1', 'Foo');
|
||||||
|
writeVerification(phaseDir, '1', 'passed');
|
||||||
|
// A column-0 `### N.` block with no `result:` line — visible to the
|
||||||
|
// tokenizer, so it is NOT a shortfall. No accepted-over-report story
|
||||||
|
// exists for it, so the teeth stay on.
|
||||||
|
writeUatDocWithStatus(phaseDir, '1', 'complete', [
|
||||||
|
'### 1. Alpha',
|
||||||
|
'expected: a',
|
||||||
|
'',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const payload = parseInspect(tmpDir);
|
||||||
|
const phase = payload.phases[0];
|
||||||
|
assert.strictEqual(phase.uat.scope, 'truncated');
|
||||||
|
assert.strictEqual(phase.scope, 'truncated');
|
||||||
|
assert.ok(payload.diagnostics.some((d) => d.code === 'phase_scope_degraded'));
|
||||||
|
assert.strictEqual(payload.progress.accepted_phases.percent, null);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('anUnreadableUatFileStillDegradesThePhaseAndWithholdsThePercentage', (t) => {
|
||||||
|
const tmpDir = createTempProject();
|
||||||
|
t.after(() => cleanup(tmpDir));
|
||||||
|
const phaseDir = declarePhase(tmpDir, '1', 'Foo');
|
||||||
|
writeVerification(phaseDir, '1', 'passed');
|
||||||
|
// The pre-existing genuinely-truncated derivation: the UAT path exists and
|
||||||
|
// is selected, but reading it yields nothing. A DIRECTORY at the file path
|
||||||
|
// makes the read fail deterministically on every OS and as root — no mode
|
||||||
|
// bits, which root bypasses (CLAUDE.md, IO-failure injection).
|
||||||
|
fs.mkdirSync(path.join(phaseDir, '1-UAT.md'));
|
||||||
|
|
||||||
|
const payload = parseInspect(tmpDir);
|
||||||
|
const phase = payload.phases[0];
|
||||||
|
assert.strictEqual(phase.uat.scope, 'truncated');
|
||||||
|
assert.strictEqual(phase.scope, 'truncated');
|
||||||
|
assert.ok(payload.diagnostics.some((d) => d.code === 'uat_unreadable' && d.subject.includes('1-UAT.md')));
|
||||||
|
assert.ok(payload.diagnostics.some((d) => d.code === 'phase_scope_degraded'));
|
||||||
|
assert.strictEqual(payload.progress.accepted_phases.percent, null);
|
||||||
|
});
|
||||||
|
|
||||||
test('roadmapAcceptanceIsNeverAuthoritativeOnAnyPhaseRow', (t) => {
|
test('roadmapAcceptanceIsNeverAuthoritativeOnAnyPhaseRow', (t) => {
|
||||||
const tmpDir = createTempProject();
|
const tmpDir = createTempProject();
|
||||||
t.after(() => cleanup(tmpDir));
|
t.after(() => cleanup(tmpDir));
|
||||||
|
|||||||
3689
tests/uat.test.cjs
3689
tests/uat.test.cjs
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user