37 KiB
37 KiB
phase, verified, status, score, covered_files, covered_digest, behavior_unverified, overrides_applied, re_verification, decision_coverage, advisory, human_verification
| phase | verified | status | score | covered_files | covered_digest | behavior_unverified | overrides_applied | re_verification | decision_coverage | advisory | human_verification | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 11.1-summercms-documentation-for-humans-and-ai-agents | 2026-10-01T07:31:53Z | human_needed | 13/13 must-haves verified |
|
v2:sha256:69e8741c6e6d238b3ae5164f594e9dce56016ddf6cda5b738f3fb81f3fec780e | 0 | 0 |
|
|
|
|
Phase 11.1: SummerCMS documentation for humans and AI agents Verification Report
Phase Goal: SummerCMS has a WinterCMS-style documentation set that serves both humans and AI agents. The Markdown source lives in summercms.go/docs/ and a summer CLI command builds it into a static site with sidebar navigation, search, llms.txt/llms-full.txt and a raw .md per page. Every code example compiles and is tested, and a "Coming from WinterCMS" map plus an acme/blog porting walkthrough cover the migration path. It documents the framework as it stands after Phase 11 and never names a consuming application.
Verified: 2026-10-01T07:31:53Z
Status: human_needed
Re-verification: Yes — after gap closure (plan 11.1-07)
Verdict in one paragraph
The three gaps from 2026-09-30 are closed in the code, and the phase gate is green. collectFences walks the goldmark AST the renderer uses. A src= fence inside a callout, blockquote or list item, and any src= fence in a module README, fails Check and is not captioned. Go-lexer aliases need src=. A .go target must be in the default build and reached from a Test or an Example with output. go doc -c is case-sensitive, and commandWord accepts env prefixes, cobra-style flags, go run ./cmd/summer and bin/{app}. bash scripts/check-phase11.1.sh --all printed phase11.1 all passed (exit 0), including a scratch-tree self-test that plants the old holes. Automated must-haves are 13/13. Status stays human_needed because the plans' backstop truths (search, theme, no-JS layout, editorial usefulness) and seven judgment-tier prohibitions still need a person. That is not a code gap.
Goal Achievement
Observable Truths
The scored set is the same 13 truths as the previous report. Plan 11.1-07's ten truths are the re-test of truths 6–8; they are listed under that table and are not a second denominator.
| # | Truth | Status | Evidence |
|---|---|---|---|
| 1 | SC1: docs/ holds Markdown pages with frontmatter in Setup, Architecture, Plugins, Backend, Database, Services, Console, API reference, and every framework module is reachable from the sidebar | ✓ VERIFIED | Regression: docs/site.yaml still lists those eight sections. scripts/check-phase11.1.sh --all ran TestPhase11_1Acceptance under --named (-count=1, skips refused) and printed phase11.1 named passed. docs:build wrote 72 pages. |
| 2 | SC2: a summer command builds a self-contained static site with sidebar, TOC, prev/next, client-side search and dark mode, no Node toolchain; only new dep is approved |
✓ VERIFIED | --deps passed (chroma/v2 only). --docs built the site. Search keystrokes and theme flash stay in human verification; markup is covered by the acceptance test the gate ran. |
| 3 | SC3/DOCS-03: llms.txt, llms-full.txt and a clean .md per page, with a test asserting sync with the page tree |
✓ VERIFIED | --docs requires llms.txt, llms-full.txt and index.md. TestDocsAIOutputsInSync is in the named set that passed. Usefulness of the raw files stays a backstop. |
| 4 | SC4 (current content): every Go example in a docs/ page is compiled and run by go test ./... |
✓ VERIFIED | TestPhase11_1Acceptance passed via --named. The acceptance scanner now marks blockquoted fences nested (scanDocFences / TestAcceptanceFenceScanner) and the real tree has no nested src= fence. |
| 5 | SC5 / DOCS-07: the concept map and the acme/blog walkthrough exist, and the walkthrough's code is verified under SC4 | ✓ VERIFIED | Both pages are still present. --named ran the blog package tests, including the Docker migrate/route/command tests, and refuses a skip. phase11.1 named passed. |
| 6 | DOCS-04 enforcement: a test fails on a missing or drifted snippet; every Go fence must carry src= | ✓ VERIFIED | Closed by plan 07 truths G1–G5 and G10. Self-test on a scratch copy of the real tree refused a callout src= fence (must be a top-level block) and a golang fence with no src=. TestPlantedViolations walks all 81 violation directories, including the new plants, and the self-test stage passed. |
| 7 | D-07 proof of execution: src= must name compiled source, and code no Test or Example runs is a snippet problem | ✓ VERIFIED | Closed by G6–G7. loadTestGraph roots are isTestName plus doc.Examples with Output or EmptyOutput. checkBuiltGo uses build.ImportDir on build.Default and refuses testdata and _ directories. TestSnippetRootsAndBuild and the planted fixtures snippet-example-no-output-helper, snippet-build-ignored and snippet-testdata-go are in the corpus the self-test ran. |
| 8 | DOCS-05: checkers fail on stale identifiers, broken links/anchors, unknown command names and forbidden names | ✓ VERIFIED | Closed by G8–G9 for the two holes. goDoc runs go doc -c. Self-test refused lagoon.Openfromapp, FOO=1 summer no:such and summer --root . no:such. Links, anchors and forbidden names were already refused and the same self-test still plants them. |
| 9 | DOCS-06: "Coming from WinterCMS" maps Winter concepts to SummerCMS, with checker-verified identifiers and explicit Not provided rows | ✓ VERIFIED | Regression: the Not provided table in docs/setup/coming-from-wintercms.md still links docs/services/frontend-and-ajax.md. Identifiers are inside the checkers the gate ran. |
| 10 | DOCS-08: CLAUDE.md records the docs update rule and names the checkers | ✓ VERIFIED | CLAUDE.md ## Documentation still names TestDocsTree and summer docs:build --check. --claude passed. |
| 11 | D-11: no page, snippet or built output names a consuming application | ✓ VERIFIED | grep -rliE over docs/**/*.md found no consuming-application spelling. --forbidden passed. A planted name is still in the self-test. |
| 12 | Phase-end tests: docsite coverage ≥85%, a planted fixture per rule, and acceptance subtests SC1–SC5 | ✓ VERIFIED | This pass: go test -count=1 -cover ./internal/docsite reported coverage: 94.1% of statements. 81 violation directories. TestPlantedViolations and TestPhase11_1Acceptance passed inside the gate. |
| 13 | The gate: check-phase11.1.sh --all is green and --named refuses skipped or zero-match runs |
✓ VERIFIED | This pass ran bash scripts/check-phase11.1.sh --all. Every stage passed and it printed phase11.1 all passed, exit 0. detect_json still exits on skip, "no tests to run" and a missing required name. |
Score: 13/13 truths verified (0 present, behavior-unverified)
Five backstop truths (search rows and keys, no theme flash, contrast, no-JS layout, llms usefulness, editorial and walkthrough followability) stay non-inferable. They are not in the 13. They route to human verification with reason insufficient_spec.
Plan 11.1-07 closure truths
All ten are verified. They are the detailed contract for truths 6–8, not extra score rows.
| # | Truth | Status | Evidence |
|---|---|---|---|
| G1 | Snippet, policy and command checks find every goldmark *ast.FencedCodeBlock |
✓ VERIFIED | collectFences in internal/docsite/fences.go walks the document from newMarkdown. checkSnippets, checkPolicy and checkCommands call it. TestCollectFences expects the callout, blockquote and list fences and rejects a four-space indented fence. |
| G2 | A non-top-level src= fence fails Check, docs:build --check and docs:sync, and sync writes nothing |
✓ VERIFIED | nestedSrcMessage in checkFences returns before Extract. TestSyncParsedFences expects one problem and an unchanged tree. Self-test plant on the real-tree copy was refused. |
| G3 | A module README src= fence is refused and no unchecked fence gets a figcaption |
✓ VERIFIED | readmeSrcMessage when page.Module != "". fenceAnnotator sets data-src only when Module == "" and the parent is the document. TestFenceCaptionsOnlyVerified expects one guide caption and none on the API page. |
| G4 | Any fence whose chroma lexer is Go needs src=, at any depth; go-html-template, text and empty do not |
✓ VERIFIED | goLang uses lexerFor. checkPolicy applies it to every collected fence of a docs page. TestGoLang and TestNestedFenceChecks (callout go, GO, main.go). Self-test refused golang. |
| G5 | Shell fences inside callouts are command-checked; four or more leading spaces are not a fence | ✓ VERIFIED | checkCommands reads f.code from the AST for sh/shell/bash/console. openFence rejects indent above 3. TestCollectFences wants 6 fences, not 7. Plant command-in-callout. |
| G6 | Reachability roots are Test functions plus Examples with output, from test files in the default build | ✓ VERIFIED | loadTestGraph skips Examples with empty Output and no EmptyOutput, and only parses TestGoFiles/XTestGoFiles. TestSnippetRootsAndBuild asserts ExampleNone and onlyFromIgnored are not reachable. |
| G7 | A .go src= target fails when it is under testdata or _, or build.ImportDir does not list it |
✓ VERIFIED | checkBuiltGo. Plants snippet-build-ignored, snippet-build-ignored-test-root, snippet-testdata-go. |
| G8 | go doc -c, so demo.Openfromapp fails |
✓ VERIFIED | check_identifiers.go argv is go, doc, -c. TestIdentifierGoDocCaseSensitive expects Openfromapp false. Self-test refused lagoon.Openfromapp. Plant identifier-wrong-case. |
| G9 | Command word after VAR=value and flags, for summer, go run ./cmd/summer, ./bin/{app} and bin/{app}; summer --help alone is not a command |
✓ VERIFIED | commandWord in check_commands.go. TestCommandWord covers those forms. Self-test refused the env-prefix and flag-first plants. |
| G10 | Each hole has a planted fixture; the real tree passes TestDocsTree, docs:build --check and TestPhase11_1Acceptance; the gate prints phase11.1 all passed |
✓ VERIFIED | 18 new plants are in testdata/violations and TestPlantedViolations runs every directory. Gate exit 0, quoted above. scanDocFences strips > and goFenceLang uses chroma. SC4 cross-checks figcaption counts. |
Required Artifacts
| Artifact | Expected | Status | Details |
|---|---|---|---|
internal/docsite/fences.go |
AST fence discovery | ✓ VERIFIED | func collectFences; wired from snippet, policy and command checks |
internal/docsite/snippet.go |
top-level src=, README refusal, build membership, Example roots | ✓ VERIFIED | build.ImportDir, doc.Examples, nestedSrcMessage, readmeSrcMessage |
internal/docsite/highlight.go |
goLang via the highlighter lookup |
✓ VERIFIED | func goLang; checkPolicy calls goLang( |
internal/docsite/check_commands.go |
commandWord |
✓ VERIFIED | env prefix, flags, go run ./cmd/summer, bin/ |
internal/docsite/check_identifiers.go |
case-sensitive go doc fallback | ✓ VERIFIED | "doc", "-c" |
internal/docsite/render.go |
captions only on verified fences | ✓ VERIFIED | Module == "" and document parent |
internal/docsite/testdata/violations/snippet-callout-drift/want.txt |
CR-01 plant | ✓ VERIFIED | message contains top-level block |
cmd/summer/phase11_1_acceptance_test.go |
independent scanner and caption cross-check | ✓ VERIFIED | TestAcceptanceFenceScanner, scanDocFences |
scripts/check-phase11.1.sh |
gate plants the holes; --all |
✓ VERIFIED | ran this pass |
internal/docsite/docsite.go |
Check calls checkSnippets after parse |
✓ VERIFIED | call at checkContent; not left only in assemble |
docs/setup/coming-from-wintercms.md |
concept map | ✓ VERIFIED | regression, file present, Not provided rows |
docs/setup/porting-a-plugin.md |
walkthrough | ✓ VERIFIED | regression, file present |
docs/examples/blog/* |
in-root plugin | ✓ VERIFIED | named blog tests passed inside the gate |
CLAUDE.md |
documentation rule | ✓ VERIFIED | --claude passed |
Key Link Verification
| From | To | Via | Status | Details |
|---|---|---|---|---|
internal/docsite/render.go |
internal/docsite/snippet.go |
caption set equals the fences checkSnippets compares |
WIRED | Module == "" and KindDocument in fenceAnnotator; TestFenceCaptionsOnlyVerified |
internal/docsite/check_policy.go |
internal/docsite/highlight.go |
goLang reuses lexerFor |
WIRED | goLang( at the policy loop |
internal/docsite/snippet.go |
go/build and go/doc |
build.ImportDir and doc.Examples |
WIRED | buildPackage, exampleHasOutput, loadTestGraph |
internal/docsite/check_identifiers.go |
go doc -c |
goDoc argv |
WIRED | exec.CommandContext(ctx, "go", "doc", "-c", ...) |
scripts/check-phase11.1.sh |
new tests and fixtures | --named list includes TestNestedFenceChecks, TestCommandWord, TestSnippetRootsAndBuild |
WIRED | --named passed |
cmd/summer/docs.go |
internal/docsite |
Build / Check / Sync |
WIRED | regression; gate builds ./cmd/summer and runs docs:build --check |
docs/setup/porting-a-plugin.md |
docs/examples/blog |
src= fences |
WIRED | acceptance SC4/SC5 inside the gate |
Data-Flow Trace (Level 4)
The docsite does not render application data from a database. Page HTML is produced from the Markdown sources.
| Artifact | Data Variable | Source | Produces Real Data | Status |
|---|---|---|---|---|
| Built guide HTML | fenced code and figcaption | docs page body plus Extract of a src= target, caption only when the checker compared it |
Yes | ✓ FLOWING |
llms.txt / per-page .md |
title, description, body | loaded page tree (emit.go) |
Yes | ✓ FLOWING |
| Search index | page titles and text | the same page tree written by the build | Yes | ✓ FLOWING |
| API reference pages | README body | modules/*/README.md via loadModules |
Yes | ✓ FLOWING |
Behavioral Spot-Checks
| Behavior | Command | Result | Status |
|---|---|---|---|
| Full phase gate, including scratch-tree plants of the old holes | bash scripts/check-phase11.1.sh --all |
all stages passed, phase11.1 all passed, exit 0 |
✓ PASS |
| Docsite coverage at least 85% | go test -count=1 -cover ./internal/docsite |
coverage: 94.1% of statements |
✓ PASS |
| Callout src=, golang fence, wrong-case ident, env-prefix and flag-first commands | self-test plants inside the gate (real-tree copy) | each expect_refusal returned; stage printed phase11.1 self-test passed |
✓ PASS |
| Named tests, skips refused | gate --named (go test -json -count=1) |
phase11.1 named passed |
✓ PASS |
| Full module tests | gate --go (go vet ./... then go test ./...) |
phase11.1 go passed (many packages (cached) after --named had already run the docs packages with -count=1) |
✓ PASS |
Probe Execution
| Probe | Command | Result | Status |
|---|---|---|---|
scripts/check-phase11.1.sh |
bash scripts/check-phase11.1.sh --all |
exit 0, phase11.1 all passed |
PASS |
Requirements Coverage
| Requirement | Source Plan | Description | Status | Evidence |
|---|---|---|---|---|
| DOCS-01 | 01, 03, 04, 06 | Markdown pages with strict frontmatter in Winter sections; every module reachable via an API page | ✓ SATISFIED | Truth 1 |
| DOCS-02 | 01, 02, 06 | self-contained site, docs:serve on loopback, no Node, only chroma/v2 added | ✓ SATISFIED | Truth 2. Search and theme interaction remain human UAT, as 11.1-VALIDATION.md already records. |
| DOCS-03 | 01, 06 | llms.txt, llms-full.txt, .md per page, sync test | ✓ SATISFIED | Truth 3 |
| DOCS-04 | 01–07 | every Go fence in docs/ has src=; a test fails on a missing or drifted snippet; Examples carry Output and run | ✓ SATISFIED | Truths 4, 6, 7 and G1–G7, G10. REQUIREMENTS.md still shows this box open and the traceability row as Gaps Found; this dispatch was not allowed to edit that file. The code satisfies the requirement. |
| DOCS-05 | 02, 06, 07 | checkers fail on stale identifiers, broken links and anchors, unknown commands, forbidden names | ✓ SATISFIED | Truth 8 and G8–G9. Same note: the REQUIREMENTS.md checkbox was left for a later edit and is not evidence. |
| DOCS-06 | 03, 04, 06 | concept map with checker-verified identifiers | ✓ SATISFIED | Truth 9 |
| DOCS-07 | 05, 06 | acme/blog walkthrough as a real in-root package | ✓ SATISFIED | Truth 5 |
| DOCS-08 | 02, 06 | CLAUDE.md documentation rule naming the checkers | ✓ SATISFIED | Truth 10 |
No orphaned requirement. REQUIREMENTS.md maps exactly DOCS-01 through DOCS-08 to Phase 11.1, and every ID is claimed by at least one plan.
Decision Coverage
All trackable CONTEXT.md decisions are honored by shipped artifacts. 18 honored, 18 total, none missing. Non-blocking.
Prohibitions (judgment-tier)
unverified-prohibition — human review recommended. These verdicts are a non-authoritative LLM judge. They are not a silent pass. Autonomous completion is complete with 7 flagged prohibitions.
| # | Statement | Verdict | Evidence |
|---|---|---|---|
| 1 | The built site must not reference a third-party origin for assets (plans 01, 02) | holds | Theme templates and assets were not modified by plan 07. The gate rebuilt the site. Prior inspection (no external src=, vendored font url()) still applies to those unchanged files. |
| 2 | The theme must not load third-party fonts, icons or scripts, and has no analytics | holds | Same unchanged theme tree. Licences ship under the theme assets. |
| 3 | The docs must not describe an unprovided WinterCMS feature as available (plans 03, 04) | holds | Not provided table re-read this pass; it links Frontend and AJAX. docs/site.yaml llms_notes still say the framework is headless. |
| 4 | The docs must not present an insecure setting as the default (plan 04) | holds (spot-check carried forward) | Mail, push, search and serve pages were not in the plan 07 diff. A full read is still a human item. |
| 5 | The walkthrough must not show a write without a fill allow-list, or an admin controller without a permission | holds | lagoon.Fill(post, post.Fillable(), ...) in docs/examples/blog/models/post.go; RequiredPermissions in controllers/posts.go. |
| 6 | The gate must not count a skipped, filtered or zero-match run as a pass | holds | detect_json exits on skip, no tests to run, zero passes and a missing required name. --named passed under that detector. |
| 7 | The site must not show a source caption over a code block the snippet checker did not compare (plan 07) | holds | fenceAnnotator matches the checked set. TestFenceCaptionsOnlyVerified and the acceptance figcaption cross-check ran inside the gate. |
Test Quality Audit
| Test File | Linked Req | Active | Skipped | Circular | Assertion Level | Verdict |
|---|---|---|---|---|---|---|
internal/docsite/violations_test.go |
DOCS-04, DOCS-05 | 81 directory cases plus runtime forbidden plants; each expects exactly one problem and a refused Build | 0 (t.Skip absent in internal/docsite/*_test.go) |
No. Expected text is the planted want.txt, not output of the checker |
Value | OK |
internal/docsite/snippet_test.go TestSnippetRootsAndBuild |
DOCS-04 | reachability and Extract error substrings |
0 | No. Oracle is go/doc and go/build, not the checker echoing itself |
Behavioral | OK |
internal/docsite/checks_test.go TestCommandWord, TestIdentifierGoDocCaseSensitive, TestNestedFenceChecks |
DOCS-04, DOCS-05 | table and fixture assertions | 0 | No | Value | OK |
cmd/summer/phase11_1_acceptance_test.go |
DOCS-01..07 | SC1–SC5 plus TestAcceptanceFenceScanner |
0 in the named run (a skip would have failed the gate) | No. Scanner is independent of docsite.collectFences |
Behavioral | OK |
scripts/check-phase11.1.sh self-test |
DOCS-04, DOCS-05 | plants on a copy of the real tree | n/a | No. The plant is a hand-written bad fence, not generated by the checker | Behavioral | OK |
Disabled tests on requirements: 0 Circular patterns detected: 0 Insufficient assertions: 0
Anti-Patterns Found
| File | Line | Pattern | Severity | Impact |
|---|---|---|---|---|
phase Go, the gate script, docs/console/utilities.md, README.md |
— | TBD / FIXME / XXX | — | none |
internal/docsite |
— | t.Skip |
— | none |
Advisory (New Scope, Unevidenced)
New-scope findings from this re-verification with no deterministic evidence. Not blocking. They do not reopen a closed must-have.
| # | Finding | Category | Why Advisory |
|---|---|---|---|
| 1 | Identifier index still parses build-ignored Go files | other | Named in the plan 07 summary as out of scope. Not a prior gap. No failing test was run. |
Human Verification Required
Automated checks passed. These items are why status is human_needed rather than passed.
1. Search interaction
Test: Run summer docs:serve, search for a module name and a CLI command, use arrow keys and Enter.
Expected: Rows show section, title, heading and a marked excerpt; Enter opens the page; the live region announces the count.
Why human: Plan 02 and plan 06 backstop. No JS test runner.
2. Theme, contrast, no flash
Test: Cycle light, dark and system; reload; check a guide page and an API page at 1280px, 1024px and 375px. Expected: No flash; the choice persists; UI-SPEC contrast holds. Why human: Visual and first-paint timing.
3. No-JS layout
Test: Disable JavaScript and load a guide page below 1024px. Expected: Sidebar above the content; search and theme buttons hidden. Why human: Plan 02 backstop.
4. Raw files for an agent
Test: Read llms.txt, one page .md and llms-full.txt.
Expected: Spec shape and URLs that are useful without HTML.
Why human: Plan 01 backstop (insufficient_spec). unverified — held-out test recommended.
5. Editorial and walkthrough
Test: Read Setup, Backend, Database and Services, and follow docs/setup/porting-a-plugin.md.
Expected: WinterCMS tone, accurate to the READMEs, ending at docs/examples/blog.
Why human: Plans 03, 04 and 05 backstop.
6. Judgment-tier prohibitions
Test: Confirm the seven prohibition rows above. Expected: Each holds. Why human: unverified-prohibition — human review recommended.
Gaps Summary
No code gap remains from the previous report. Fence discovery, proof of execution, and the identifier and command forms now fail closed, with planted fixtures and a green scripts/check-phase11.1.sh --all. Nothing in a later milestone phase was needed to defer them. The open work is human UAT already scheduled in 11.1-VALIDATION.md, plus explicit resolution of the judgment-tier prohibitions. Do not open another gap-closure plan for those.
Verified: 2026-10-01T07:31:53Z Verifier: the agent (gsd-verifier)