From 9637100ffad0a700e9e8cf23bf38e9911a094bd5 Mon Sep 17 00:00:00 2001 From: Jakub Zych Date: Wed, 4 Nov 2026 23:42:41 +0100 Subject: [PATCH] docs(33-15): complete the Go plugins gap-closure plan --- .../33-15-SUMMARY.md | 227 ++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 .planning/phases/33-seo-output-audit-and-content/33-15-SUMMARY.md diff --git a/.planning/phases/33-seo-output-audit-and-content/33-15-SUMMARY.md b/.planning/phases/33-seo-output-audit-and-content/33-15-SUMMARY.md new file mode 100644 index 00000000..c91347f3 --- /dev/null +++ b/.planning/phases/33-seo-output-audit-and-content/33-15-SUMMARY.md @@ -0,0 +1,227 @@ +--- +phase: 33-seo-output-audit-and-content +plan: 15 +subsystem: go-plugins +tags: [seo, sitemap, journal, crawler, indexnow, hreflang, route-report, gap-closure, postgres, goldmark] +status: complete + +requires: + - phase: 33-13 + provides: the release decision (after-tags) and the review and verification findings this plan closes + - phase: 33-16 + provides: the layer side of the gap-2 contract (vue-seo-layer sends the route report's noindex list) +provides: + - "sm-seo-plugin: route report key noindex (golem15_seo_noindex_paths, models.NoindexPath, ReportNoindex, NoindexPathCount), the sitemap filter's noindex reason source, crawl.ToolSitemapSettings and Env.SitemapTool" + - "sm-seo-plugin: crawl lifecycle (JobTimeout, FinishTimeout, HeartbeatEvery, StaleAfter, RecoverStale, heartbeat_at and cancel_requested_at, cancel owned by the job, API/MCP cancel_requested, crawls.st_cancelling), TabOpenTimeout, effectivePort and the port-aware allowRequest, classes.PathKey and classes.EscapeKeyPath" + - "sm-sitemap-plugin: contract.Rebuilder (published by the plugin), indexnow_pending_at, the rebuild advisory lock, WithAfterSlotRead, models.ConvertLegacyItems and models.DroppedItem, unpinned HTML sitemap headings" + - "sm-journal-plugin: tall-original skip, markdown-safe ConvertContentH1, tokenizer tails kept, journal:fix-headings via lookupDB, SetMarkdownImageAlt and SetImageAltOutsideCode, descriptors with URLs only for locales with content" + - "summercms.go: the attach/webp decoder note in the lagoon README and docs/database/attachments.md (docs only)" +affects: [33-14, 33-VERIFICATION, v0.6.2 backlog] + +actuals: + tokens: 45521 + tasks: 6 + commits: 20 +plan_head_before: 8d562004cb05d8d40d12253c23e406259de61006 +plan_head_after: fec697b +plan_heads: + summercms.go: { before: 8d56200, after: fec697b, commits: 2, note: "0e95127 and the two 33-16 commits between are 33-16's" } + sm-seo-plugin: { before: 075cf17, after: 2a288a9, commits: 7 } + sm-sitemap-plugin: { before: 1048c01, after: b2b3a1d, commits: 6 } + sm-journal-plugin: { before: 311c28a, after: 0a62090, commits: 5 } + +tech-stack: + added: [] + patterns: + - "A job's terminal writes run on context.WithoutCancel(jobCtx) with their own bound; the job owns the terminal status, a cancel only records the request" + - "Heartbeat column plus stale recovery inside the start lock, so a row whose job is gone never blocks the one-at-a-time guard" + - "Listener marks (indexnow_pending_at) read and cleared by the next rebuild with a <= read-value condition, so a later mark survives" + - "Whole read-modify-write rebuild under pg_advisory_xact_lock(hashtext(...)) in one transaction" + - "Markdown rewrites only outside goldmark AST code ranges; HTML rewrites only when LooksLikeHTML" + +key-files: + created: + - ../sm-seo-plugin/updates/202611040001_create_golem15_seo_noindex_paths.go + - ../sm-seo-plugin/updates/202611040002_add_golem15_seo_crawls_heartbeat.go + - ../sm-seo-plugin/models/noindex_path.go + - ../sm-seo-plugin/crawl_lifecycle_test.go + - ../sm-seo-plugin/classes/crawl/fetch_test.go + - ../sm-sitemap-plugin/updates/202611040001_add_golem15_sitemap_urls_indexnow_pending.go + - ../sm-sitemap-plugin/models/legacy_test.go + - ../sm-sitemap-plugin/rebuild_lock_test.go + - ../sm-sitemap-plugin/type_titles_test.go + modified: + - ../sm-seo-plugin/classes/registry.go + - ../sm-seo-plugin/classes/sitemap_filter.go + - ../sm-seo-plugin/controllers/api/registry.go + - ../sm-seo-plugin/classes/crawl/job.go + - ../sm-seo-plugin/classes/crawl/store.go + - ../sm-seo-plugin/classes/crawl/fetch.go + - ../sm-seo-plugin/classes/crawl/render.go + - ../sm-seo-plugin/classes/crawl/render_chrome.go + - ../sm-seo-plugin/classes/linkgraph.go + - ../sm-sitemap-plugin/contract/contract.go + - ../sm-sitemap-plugin/classes/rebuild.go + - ../sm-sitemap-plugin/classes/lastmod.go + - ../sm-sitemap-plugin/models/legacy.go + - ../sm-sitemap-plugin/updates/202610280002_update_golem15_sitemap_settings.go + - ../sm-sitemap-plugin/extensions_settings.go + - ../sm-journal-plugin/classes/image_variants.go + - ../sm-journal-plugin/classes/headings.go + - ../sm-journal-plugin/classes/image_alt.go + - ../sm-journal-plugin/classes/content_descriptors.go + - ../sm-journal-plugin/classes/menu_item_types.go + - ../sm-journal-plugin/controllers/api/maintenance.go + - ../sm-journal-plugin/console/fix_headings.go + - modules/lagoon/README.md + - docs/database/attachments.md + +key-decisions: + - "P33-58 to P33-66 implemented as planned; noindex entries that are patterns (* or a :param segment) are skipped and counted, and a percent-encoded entry is stored decoded when that is a valid address (the filter compares decoded paths)" + - "The crawl key's decoded path form lives in classes.PathKey (shared by Fetcher.Key and linkgraph.sitePath); a decoded path holding % or ? keeps its encoded form so Absolute can tell the two forms apart" + - "compareRendered takes the fetcher's Key function, so the rendered comparison inherits the port and path rules" + - "NewRenderer keeps its signature and derives the site's port from its site argument; newChromeRenderer takes the port" + - "G2 not fixed: signpost.Activate refuses a menu item type whose DefaultPattern does not start with /, so the change needs the framework (v0.6.2)" + +patterns-established: + - "Gap-closure tests that pin a fix are checked to fail on the old code by swapping the old line back in before committing" + +requirements-completed: [SC-1, SC-2, SC-3, D-14, D-24, D-30, D-31, D-32, D-33, D-34, D-50, D-55, D-57, D-85, D-95, D-10] + +duration: 2h15m +completed: 2026-11-04 +--- + +# Phase 33 Plan 15: Go plugin gap fixes before the release Summary + +**Noindex paths from the route report leave the sitemaps (G1, CMS side), Journal descriptors give only locales with content an address (G3), and every in-scope L1 to L3 review finding is fixed with a test that fails on the old code, in sm-seo-plugin, sm-sitemap-plugin and sm-journal-plugin, plus one framework docs note.** + +## Performance + +- **Duration:** about 2 h 15 min +- **Started:** 2026-11-04T21:37:51Z +- **Tasks:** 6 of 6 +- **Commits:** 20 (seo 7, sitemap 6, journal 5, summercms.go 2), nothing pushed or tagged + +## BASE (recorded before Task 1) + +| Repository | BASE | +|---|---| +| summercms.go | 8d562004cb05d8d40d12253c23e406259de61006 | +| sm-seo-plugin | 075cf1700797722f35ba77daa2ed408790e4c898 | +| sm-sitemap-plugin | 1048c01fd913173866efd678d0e4bfc28b52dade | +| sm-journal-plugin | 311c28af3adecfabde8bc142703777779590fb35 | + +## Commits + +**sm-sitemap-plugin** (main) +- 6b67ab7 feat: contract.Rebuilder asks for a debounced rebuild (Task 1 A) +- f58d555 fix: IndexNow hears of deleted and edited records (Task 4 A) +- 2200e46 fix: rebuilds collect under one lock and keep moved lastmods (Task 4 B) +- b37b226 fix: the definitions migration keeps and logs every url item (Task 4 C) +- da32044 fix: HTML sitemap headings follow each locale until an editor sets one (Task 4 D) +- b2b3a1d test: Journal-shaped records in the hreflang test (Task 6 C) + +**sm-seo-plugin** (main) +- 9de6b2e feat: noindex paths from the route report leave the sitemaps (tracer) (Task 1 B) +- d7597bb fix: a cancelled or timed-out crawl stores what it fetched and ends (Task 2 A) +- cd71823 fix: a crawl cancel waits for its job, and a stale crawl is ended (Task 2 B) +- cfeb3df fix: bound opening a Chrome tab (Task 2 C) +- 3fdb44b fix: the crawler and the Chrome guard keep to the site's port (Task 3 A) +- e2af5b4 fix: one decoded path form for crawl keys and CMS lookups (Task 3 B) +- 2a288a9 test: Journal-shaped records in the hreflang test (Task 6 B) + +**sm-journal-plugin** (main) +- 45d66f6 fix: tall originals no longer stop the image variants job (Task 5 A) +- 0e739c8 fix: H1 conversion leaves markdown and unfinished tags alone (Task 5 B) +- 5f683cb fix: journal:fix-headings runs from the application CLI (Task 5 C) +- 7662d54 fix: image alt texts on markdown posts are written into the markdown (Task 5 D) +- 0a62090 fix: content descriptors list only locales with content (Task 6 A) + +**summercms.go** (master) +- 1f96899 docs(lagoon): attach/webp also decodes WebP (Task 6 D, docs only, no Go file) +- fec697b docs(33-15): release notes, release log and deferred items for the gap fixes (Task 6 E) + +## Finding to test + +| Finding | Fix | Test (fails on the old code) | +|---|---|---| +| G1 (D-24, D-33, D-95, SC-1) | route report `noindex`, stored set, filter reason, `contract.Rebuilder` | TestNoindexPathsTracer, TestRouteReport (extended), TestListing (Rebuilder) | +| G1 (SC-3, D-14) crawler tool | `sitemap_update_settings` / fix: code for a listed noindex address without a record | TestSitemapNoindexTool | +| G3 (D-55, D-57, SC-2) | descriptors give a locale a URL only with content (P33-64) | TestContentDescriptorsTranslatedLocales; TestAlternates and TestHreflangAlternates on Journal-shaped records | +| L1 CR-01 (D-85) | skip a width whose height exceeds 4096 px | TestImageVariantsTallOriginal | +| L1 CR-02 (D-50) | ConvertContentH1 rewrites content only when LooksLikeHTML | TestConvertContentH1, TestConvertTranslationH1, TestHeadingConversion (markdown subtest), TestFixHeadingsCommand (markdown post) | +| L1 WR-01 (D-50) | journal:fix-headings uses lookupDB | TestFixHeadingsCommandCLI | +| L1 WR-02 | docs only (lagoon README, attachments.md) | TestDocsTree, docs:build --check | +| L1 WR-03 | tokenizer tails kept in ConvertH1 and SetImageAlt | TestConvertH1, TestSetImageAltEdges | +| L1 WR-05 (D-85, D-10) | SetMarkdownImageAlt and SetImageAltOutsideCode; 422 changes.N.src otherwise | TestSetMarkdownImageAlt, TestSetImageAlt (markdown part) | +| L2 WR-01 (D-31) | indexnow_pending_at set by RecordDeleted/RecordSaved, read by apply | TestIndexNow (fires the record events), TestIndexNowStaging | +| L2 WR-02 (D-30) | pg_advisory_xact_lock around settings, collect and apply | TestConcurrentRebuilds | +| L2 WR-03 (D-30) | apply writes lastmod only for extras and created rows | TestRebuildKeepsMovedLastmod | +| L2 WR-04 (D-32) | ConvertLegacyItems, per-item warn log | TestConvertLegacyItems, TestMigrateDefinitions (extended) | +| L2 WR-05 (D-34) | the form loads stored headings only | TestTypeTitlesNotPinned | +| L2 IN-01 | nil check before reading ev | TestRecordSavedNilEvent | +| L3 CR-01 (D-14) | detached bounded finish, time-limit failure, LoadLimits fallback, canceled fetches | TestCrawlTimeout, TestCrawlCancelRunning | +| L3 CR-01 stale | heartbeat_at, RecoverStale before Start and DeleteCrawls | TestCrawlStaleRecovery | +| L3 WR-01 | cancel_requested_at; the job writes cancelled | TestCrawlCancelRunning | +| L3 WR-03 | effective-port check in Fetcher.Allowed, Key, allowRequest | TestAllowRequest (new rows), TestFetcherPortAndKey, TestCrawlerNonASCII (8443 link) | +| L3 WR-04 | one decoded path form (classes.PathKey), Absolute escapes | TestFetcherPortAndKey, TestCrawlerNonASCII | +| L3 WR-06 | one deadline for tab open and render | TestRenderTabTimeout | + +TestCrawlerNonASCII (L3 WR-04), TestRebuildKeepsMovedLastmod (L2 WR-03) and TestTypeTitlesNotPinned (L2 WR-05) were also run with the old code swapped back in and failed there; the other new tests assert behaviour or schema the old code did not have (new columns, keys, functions), so they cannot pass on it. + +## Framework Go files changed since v0.6.0 + +22 at BASE, 22 at HEAD: this plan changed no framework Go file (1f96899 changes two markdown files). + +## G2 (not fixed) + +G2 (the site answer lists the journal-tag pattern while `golem15.journal.tag_pages` is false) stays open for v0.6.2: `signpost.Activate` refuses a menu item type whose DefaultPattern does not start with `/` (modules/signpost/signpost.go), so sm-journal-plugin cannot drop the pattern while tag pages are off, and dropping the descriptor would remove the tag form's SEO tab and the sitemap types row. It needs an additive pact field (for example `ContentDescriptor.Routable`) or a signpost rule. Recorded in deferred-items.md ("From 33-12" entry and "From 33-15"). + +## Gate stages + +`PHASE33_REQUIRE_DOCKER=1 bash scripts/check-phase33.sh --self-test --go --plugins --docker --admin --docs --marker --additive --compat`: exit 0, `PASS` for self-test, go, plugins, docker, admin, docs, marker, additive and compat. The compat stage prints `refuse:` lines for sm-feedback-plugin, sm-presentations-plugin and sm-quizzes-plugin and skips them because they fail at v0.6.0 too (they do not resolve the framework through a workspace); every other consumer builds. The layers and consumers stages were not run (33-16's and the orchestrator's `--all`). + +Per task, each repository's `go vet ./...` and `go test -short ./...` stayed green, the plan's Docker suites passed (the full sm-seo-plugin suite in 116 s, the full sm-sitemap-plugin and sm-journal-plugin suites), sm-journal-plugin's vitest passed (257 tests), and `go test ./cmd/summer -run TestDocsTree` and `summer docs:build --check` passed. `go.mod`, `go.sum` and `package.json` of all four repositories are unchanged since BASE (T-33-SC). `go list -deps` of sm-seo-plugin lists only `sm-sitemap-plugin/contract` (D-95). No exported Journal function was removed or changed its signature (`git diff 311c28a..HEAD` check empty, D-106); menu_item_types_test.go and menu_item_types_smoke_test.go are unchanged and pass. + +## For 33-14 Task 4 + +- **New migrations:** sm-seo-plugin `202611040001_create_golem15_seo_noindex_paths`, `202611040002_add_golem15_seo_crawls_heartbeat`; sm-sitemap-plugin `202611040001_add_golem15_sitemap_urls_indexnow_pending`. +- **Edited migration:** sm-sitemap-plugin `202610280002_update_golem15_sitemap_settings` now converts the url items through `models.ConvertLegacyItems`. Untagged, so no production database ran the old one; the local rehearsal databases of 33-12 (demo) and 33-13 (grzyby) ran it and keep its old result. **Recreate them from their source dumps before the 33-14 rehearsal.** +- **Release notes:** the drafts of sm-seo-plugin v0.2.0, sm-sitemap-plugin v0.2.0, sm-journal-plugin v0.4.0 and summercms v0.6.1 carry the fixes (fec697b); the sitemap draft names the `definition url item not copied` log line and the automatic noindex exclusion; the Journal draft states the descriptor rule as a behaviour change. +- **Release log:** 1f96899 is listed under "Framework commits of later Phase 33 plans". +- **Version pins:** sm-seo-plugin now needs `contract.Rebuilder`, so its go.mod must require sm-sitemap-plugin v0.2.0 when tagged (development resolves it through go.work). + +## Deviations from Plan + +### Auto-fixed and adjusted + +**1. [Rule 2] Route report noindex entries that are patterns are refused, and encoded entries stored decoded.** The contract says entries carry no `:` parameter or `*`; ReportNoindex skips such entries and counts them in `skipped`, and stores a percent-encoded path decoded when that is a valid address, because the filter compares decoded paths. Commit 9de6b2e. + +**2. [Rule 3] Decoded path form shared in sm-seo-plugin's classes package.** `Fetcher.Key` and `linkgraph.sitePath` both use `classes.PathKey`, with `classes.EscapeKeyPath` for request URLs; a decoded path holding `%` or `?` keeps its encoded form so the two forms never collide. The acceptance grep for `u.Path` in fetch.go therefore finds it in classes/linkgraph.go instead. Commit e2af5b4. + +**3. [Rule 3] compareRendered takes the fetcher's Key function** instead of a host list, so the rendered comparison inherits the port and path rules. NewRenderer keeps its exported signature and derives the port from its site argument; only the unexported newChromeRenderer and allowRequest take the port. Commit 3fdb44b. + +**4. [Rule 3] Tests added in their own files:** rebuild_lock_test.go and type_titles_test.go (sm-sitemap-plugin, the latter with a fake plugin whose type label has en and pl translations), crawl_lifecycle_test.go (sm-seo-plugin); TestCrawlerNonASCII is in crawler_test.go, TestFixHeadingsCommandCLI in image_variants_cli_test.go. + +**5. [Rule 1] Existing Journal descriptor tests updated for the G3 behaviour change.** `seedContent` now gives the category a pl name and the tag a translated published post; TestContentDescriptorsMatrix expects `pl=` for the untranslated draft and tag, and TestJournalAuthorType translates Ada's post and expects `pl=` for Bob. menu_item_types_test.go and the smoke test are untouched (D-106). Commit 0a62090. + +**6. Task 2 commit split.** Groups A and B were written together and then split: A (d7597bb) carries the detached finish, the time-limit status, the LoadLimits fallback and canceled fetches, with TestCrawlTimeout; B (cd71823) adds the columns, heartbeat, job-owned cancel and RecoverStale with TestCrawlCancelRunning and TestCrawlStaleRecovery. Each commit was green on its own. + +**7. SetImageAltOutsideCode exported** next to SetMarkdownImageAlt (the plan's "helper that runs a rewrite only outside goldmark code ranges"); additive. + +### Not done + +- `requirements.mark-complete` was not run: it would leave REQUIREMENTS.md modified outside the ROADMAP-only tracking commit. + +## Known Stubs + +None. + +## Threat Flags + +None beyond the plan's threat register: T-33-71 (port guard), T-33-72 (noindex entries validated, capped, token as for routes), T-33-73 (crawl lifecycle), T-33-74 (escaped alt labels, code untouched) and T-33-76 (rebuild lock) are mitigated as planned. + +## Self-Check: PASSED + +All created files exist; every commit hash above is in its repository's `git log`.