docs(quick-261104-xae): phase 33 gap-fix follow-ups and the gap-fix re-review

This commit is contained in:
Jakub Zych
2026-11-05 00:32:56 +01:00
parent 9d299a4455
commit 6e99b1886a
3 changed files with 707 additions and 0 deletions

View File

@@ -0,0 +1,197 @@
---
phase: 33-seo-output-audit-and-content
scope: gap-closure re-review (33-15, 33-16)
reviewed: 2026-11-04T23:30:00Z
depth: deep (targeted)
files_reviewed: 52
files_reviewed_list:
- sm-sitemap-plugin (1048c01..HEAD): contract/contract.go, plugin.go, classes/rebuild.go, classes/lastmod.go, classes/listing.go, models/legacy.go, models/url.go, extensions_settings.go, updates/202610280002_update_golem15_sitemap_settings.go, updates/202611040001_add_golem15_sitemap_urls_indexnow_pending.go
- sm-seo-plugin (075cf17..HEAD): classes/crawl/job.go, classes/crawl/store.go, classes/crawl/actions.go, classes/crawl/crawler.go, classes/crawl/fetch.go, classes/crawl/render.go, classes/crawl/render_chrome.go, classes/crawl/compare.go, classes/crawl/checks.go, classes/crawl/fix.go, classes/linkgraph.go, classes/registry.go, classes/sitemap_filter.go, controllers/api/registry.go, controllers/api/crawls.go, controllers/crawls.go, models/crawl.go, models/noindex_path.go, jobs.go, mcp.go, updates/202611040001_create_golem15_seo_noindex_paths.go, updates/202611040002_add_golem15_seo_crawls_heartbeat.go
- sm-journal-plugin (311c28a..HEAD): classes/headings.go, classes/image_alt.go, classes/image_variants.go, classes/content_descriptors.go, classes/menu_item_types.go, console/fix_headings.go, controllers/api/maintenance.go, mcp.go
- vue-seo-layer (b3a0078..HEAD): lib/seo-module.ts, lib/sitemapPages.ts, lib/routes.ts, lib/options.ts, server/lib/routeReport.ts, server/lib/site.ts, server/middleware/seo-root.ts, server/plugins/seo-route-report.ts, server/seo-routes.d.ts, app/components/Seo/SitemapLink.vue, app/components/global/SeoHtmlSitemapPage.vue, app/utils/pageHref.ts, package.json
- vue-journal-layer (e8f307b..HEAD): app/journal-pages/PostPage.vue, app/utils/seo.ts, app/utils/routes.ts, lib/journal-module.ts, scripts/check-git-install.sh, package.json
- vue-menu-layer (a7602c9), vue-faq-layer (5e5950a): package.json, README.md
findings:
critical: 0
warning: 4
info: 0
total: 4
status: issues_found
---
# Phase 33 gap-closure re-review (33-15, 33-16)
Paths are relative to `/media/nvme/dev/golem15/summercms.io/summercms/`.
## Summary
Most fixes do what their findings asked. Three are only partly done:
- **L1 CR-02.** A markdown body that starts with `<` is still rewritten by the tokenizer. I confirmed this by running the code.
- **L1 WR-05.** Bodies stored as content_html only now get a 422 where they used to succeed (a regression), and the `<`-prefixed markdown subset still goes down the HTML path.
- **L4 CR-01.** The guard, the version fields and the install pins are done. The `link:` swap was deliberately left to 33-14 Task 4, so the layers still cannot be tagged from HEAD.
I found no critical defects in the new code. There are four new warnings (below).
Things I checked that hold up:
- **Advisory lock.** It does not deadlock. The pool is unbounded (lagoon sets no `MaxOpenConns`), the lock order is the same on every path (advisory lock, then the settings row), and the lock wait is bounded by the job context (conga's default 300 s).
- **Crawl heartbeat.** The heartbeat goroutine stops cleanly (closed channel, then wait), and it keeps writing during `finish` because `stop` is deferred after it.
- **River rescuer.** It honours the worker's 6 h timeout, so it does not rescue a crawl that is still running.
- **Cancel.** `Cancel` keeps the guard: the row stays `running` until the job ends it, and `ActiveCrawl`, `DeleteCrawls` and `Prune` all see it as active.
- **Migrations.** The new migrations only add a table or nullable columns and have rollbacks. `202610280002` keeps its ID, and no production database has run it (33-13 deployed nothing; the rehearsal databases are to be recreated, as the 33-15 SUMMARY says).
- **Gap-2 contract.** Both directions work. When `noindex` is absent (or `null`), the stored set stays unchanged. The layer keeps the body at or under 4,113,024 bytes and drops `noindex` entries first. The new plugin accepts twice that.
- **Concurrent route reports.** Two at once end up with the union of both sets (`ON CONFLICT DO NOTHING`), not with an error.
- **noindex input.** It is validated:
- same bearer token, compared in constant time;
- at most 2000 entries, checked before the transaction (422 and nothing stored);
- each entry passes `ValidPagePath` and the 2048-byte cap;
- patterns (`*`, `:param`) are refused;
- the body is capped with `MaxBytesReader`.
- **i18n hook.** Registering `pages:resolved` from `ready` puts it after i18n's `modules:done` hook, and the function strips its own routes again on dev regenerations.
- **sm-journal-plugin compatibility.** Exported signatures are unchanged. The behaviour changes (descriptor URL `""` for locales without content, the 422 for markdown images the tool cannot reach) are deliberate and in the v0.4.0 notes.
## Verdicts
| Finding | Verdict | Evidence |
|---|---|---|
| L1 CR-01 tall original | Resolved | `sm-journal-plugin/classes/image_variants.go:195-202` skips a width whose height is over 4096 and continues. `MaxImageWidth` (4096) equals `attach` `maxThumbEdge` (`summercms.go/modules/lagoon/attach/thumb.go:29`) only by coincidence. Other `ThumbKey` errors still abort the batch (`:203-206`); that was outside the plan |
| L1 CR-02 H1 rewrite of markdown | **Partially resolved** | `classes/headings.go:79-82` skips `content` only when `!LooksLikeHTML(content)`, and `LooksLikeHTML` only checks for a leading `<` (`classes/blog_editor.go:129-132`). The API always renders `content` as markdown (`controllers/api/posts_helpers.go:537,593`), so a markdown body that starts with an HTML block still has `<h1>` rewritten inside its code fences. See new WR-01 |
| L1 WR-01 fix-headings CLI | Resolved | `console/fix_headings.go:51-55` uses `lookupDB` |
| L1 WR-03 tokenizer tail | Resolved | `classes/headings.go:26-36`, `classes/image_alt.go:31-40` write `html[consumed:]` on `ErrorToken`; `Raw()` slices are contiguous, so the offset is exact |
| L1 WR-05 alt on markdown | **Partially resolved** | `controllers/api/maintenance.go:172-188`: an inline `![..](src)` image or a raw img outside code is written into the markdown. But a body with empty `content` (a content_html-only translation) now always gets a 422, and markdown that starts with `<` goes down the HTML branch. See new WR-02 |
| L2 WR-01 IndexNow deletes/edits | Resolved | `classes/lastmod.go:135,158-162` set `indexnow_pending_at`; `classes/rebuild.go:516-524,539-546` add the row to Changed/Removed; `:631-637` clear only marks that are not newer than the value read |
| L2 WR-02 concurrent rebuilds | Resolved | `classes/rebuild.go:144-170`: `pg_advisory_xact_lock` is taken before the settings, type settings and `collect` |
| L2 WR-03 lastmod revert | Resolved | `classes/rebuild.go:621-626`: `lastmod` is written only for `ExtraTypeKey` rows |
| L2 WR-04 migration drops items | Resolved (as scoped in P33-66) | `models/legacy.go:202-278` and the migration at `updates/202610280002_…go:84-92`: leading slash added, spaces and unsafe bytes encoded, items parsed one by one, one warn log per dropped item. Absolute URLs with `&` are still dropped, but now logged |
| L2 WR-05 pinned headings | Resolved | `extensions_settings.go:128-135` loads stored titles only; `pickTitle` ignores empty strings (`classes/htmlgroups.go:193-201`) |
| L2 IN-01 nil event | Resolved | `classes/lastmod.go:114-116` |
| L3 CR-01 cancel/timeout loses results | Resolved (one gap, new WR-03) | `classes/crawl/job.go:222-228`: detached `FinishTimeout` context; `:342-356` maps a deadline to failed; `:365-368` falls back to default limits; `crawler.go:250-253` marks cut-short fetches "canceled"; `store.go:199-239` adds `RecoverStale`. The terminal write shares the store's deadline (WR-03) |
| L3 WR-01 cancel clears guard | Resolved | `job.go:541-556` sets `cancel_requested_at` and leaves the status `running`; `finish` writes `cancelled`; `controllers/crawls.go:84` hides the action while cancelling |
| L3 WR-03 any port | Resolved | `fetch.go:150-152` (`Allowed`), `render.go:184` (`allowRequest`), `compare.go:29-33` uses `Fetcher.Key` |
| L3 WR-04 two path forms | Resolved | `classes/linkgraph.go:261-287` (`PathKey`, `EscapeKeyPath`), `fetch.go:176-196`, `sitePath` handles both inputs. Residual edge case: `%2F` decodes to `/` |
| L3 WR-06 tab-open timeout | Resolved | `render_chrome.go:126-160`: one deadline (`TabOpenTimeout+RenderTimeout`), a 5 s open timer, the tab closed on timeout; the result channel is buffered, so the goroutine does not leak |
| G1 CMS side | Resolved | `classes/registry.go:821-925` (`ReportNoindex`, `Rebuilder` called after commit), `classes/sitemap_filter.go:174-176`, `controllers/api/registry.go:19-29,78-97`, `crawl/checks.go:338-354` (sitemap tool for extras) |
| G3 descriptors per locale | Resolved | `classes/content_descriptors.go:177-182,497-500,664`, `classes/menu_item_types.go:1046,1108-1228`; the sitemap skips `URL ""` (`sm-sitemap-plugin/classes/rebuild.go:250-252`), so older rows for those locales drop out as holes |
| L4 CR-01 link: dependency | **Partially resolved (deferred by plan)** | `check-git-install.sh --check-links` and the tag-mode guard are in place, versions are 0.2.0/0.8.0/0.5.0/0.5.0 and the README pins match. All three content layers still have `"vue-seo-layer": "link:../vue-seo-layer"` (checked with `package.json` grep). Do not tag before the 33-14 Task 4 swap |
| L4 WR-01 page-1 self-redirect | Resolved | `app/utils/pageHref.ts:38-41` (`target === fullPath` gives `null`), `SeoHtmlSitemapPage.vue:71-83`. Note: the CMS segment wins over the segment the route was registered with; if they differ, the page is served without a redirect (no loop) |
| L4 WR-02 og:type article | Resolved | `vue-journal-layer/app/journal-pages/PostPage.vue:56-68`, `app/utils/seo.ts:78-83` |
| L4 WR-03 path-form segments | Resolved | `vue-journal-layer/lib/journal-module.ts:58-64`, `app/utils/routes.ts:101-107` read `runtimeConfig.public.seo.pagination`, which the SEO module's `setup` writes (`vue-seo-layer/lib/seo-module.ts:206-221`) |
| L4 WR-04 catch-all shadowing | Resolved | `lib/sitemapPages.ts:362-373`; the `overCatchAll` opt-in is documented in all four READMEs |
| L4 WR-05 i18n copies | Resolved | `lib/seo-module.ts:259-269` (`ready`, then `pages:resolved`), `lib/sitemapPages.ts:348-381` (`<name><sep><locale>`), `SitemapLink.vue:39-46` checks the route |
| L4 WR-08 root files while unknown | Resolved for the reported case (new WR-04) | `server/lib/site.ts:96-99`: FIXED names are proxied in state `unknown`; a failing fetch gives 503. Side effect in WR-04 |
| G1 layer side | Resolved | `lib/routes.ts:161-171`, `lib/seo-module.ts:283-287`, `server/lib/routeReport.ts:428-478` (bounded body, `noindex` dropped first, `[]` always sent) |
Counts: 26 verdict rows. 19 resolved (L3 CR-01 and L4 WR-08 each with a new warning attached), 3 partially resolved (L1 CR-02, L1 WR-05, L4 CR-01), 0 not resolved.
## Narrative Findings (AI reviewer)
## Warnings
### WR-01: Markdown that starts with `<` still gets `<h1>` rewritten inside code samples (CR-02 residual)
**File:** `sm-journal-plugin/classes/headings.go:79-87` (with `classes/blog_editor.go:129-132`, `controllers/api/posts_helpers.go:537,593`, and the call sites in `posts_helpers.go:865`, `console/import_posts.go`, `classes/heading_fix.go`)
**Issue:** `ConvertContentH1` treats `content` as HTML whenever its first non-space byte is `<`. The API, however, always renders `content` as markdown (`FormatHTML`), so a markdown post that opens with an HTML block (`<p class="lead">…</p>`, `<img …>`, `<!-- … -->`, an autolink `<https://…>`) is still run through the `x/net/html` tokenizer. Every `<h1>` token in its fenced or inline code samples becomes `<h2>`, silently and with the default-on setting, on every API or MCP save, on import and on `journal:fix-headings --apply`. It is worse than before: the count now comes from `content`, so the corruption is counted as a conversion. This is the same defect class as CR-02, with a narrower trigger.
**Evidence:** I ran the shipped `headings.go` (copied into the scratchpad, `GOWORK=off go run`):
```
in : "<p class=\"lead\">Intro</p>\n\nExample:\n\n```html\n<h1>Hello</h1>\n```\n"
out: "<p class=\"lead\">Intro</p>\n\nExample:\n\n```html\n<h2>Hello</h2>\n```\n" n=1, content_html unchanged
```
**Fix:** Decide by the render path, not by the first byte. Where the body is rendered as markdown (the API, import, fix-headings of an API post), rewrite only outside goldmark's code ranges. The helper for that already exists (`markdownCodeRanges` in `classes/image_alt.go`):
```go
func ConvertContentH1(content, contentHTML string) (string, string, int) {
contentHTML, nHTML := ConvertH1(contentHTML)
var nContent int
content, nContent = convertH1OutsideCode(content) // split on markdownCodeRanges(content), ConvertH1 on the parts
if nHTML == 0 { nHTML = nContent }
return content, contentHTML, nHTML
}
```
For a real HTML body, `markdownCodeRanges` finds no fences, so the result is the same as today. Add a regression case with a leading `<p>` to `headings_test.go`.
### WR-02: `images/alt` now refuses images in bodies stored as content_html only (regression), and misses markdown that starts with `<`
**File:** `sm-journal-plugin/controllers/api/maintenance.go:172-188` (with `loadImageAltText` at `:263-282`)
**Issue:** The markdown branch runs whenever `!LooksLikeHTML(text.content)`, and that includes `content == ""`. A translation that carries only `content_html` is a supported shape:
- `classes/format_html.go:101-102`;
- `classes/blog_editor.go:292`;
- `controllers/api/posts_helpers.go:802-804`.
For such a body, `SetMarkdownImageAlt("")` and `SetImageAltOutsideCode("")` both return 0, so every change for it is refused with 422 `changes.N.src` ("The image is not an inline image of the post's markdown"). The batch is all or nothing, so one such image fails the whole MCP `set_image_alt` call. Before this change it succeeded and stuck, because no markdown re-derives that `content_html`.
Markdown that starts with `<` takes the HTML branch instead. There, `SetImageAlt` finds no raw `<img>` for a `![..](src)` image, so `m == 0`, yet the change is answered `changed: true` and is reverted on the next save (the original WR-05). A raw `<img>` shown inside a code fence is also rewritten.
**Fix:** Treat an empty `content` as "content_html is the body", and route markdown by the same rule as WR-01:
```go
switch {
case strings.TrimSpace(text.content) == "":
// content_html-only body: nothing to keep in sync
case classes.LooksLikeHTML(text.content) && !hasMarkdownImage(text.content, c.src):
if content, m, _ := classes.SetImageAlt(text.content, c.src, c.alt); m > 0 { text.content = content }
default:
content, m, _ := classes.SetMarkdownImageAlt(text.content, c.src, c.alt)
content, r, _ := classes.SetImageAltOutsideCode(content, c.src, c.alt)
if m+r == 0 { /* 422 */ }
text.content = content
}
```
Add a test with a translation that has only `content_html`.
### WR-03: The crawl's terminal status write shares the store's 10-minute deadline
**File:** `sm-seo-plugin/classes/crawl/job.go:225-228, 383-397`
**Issue:** `finish` runs the store transaction and then the terminal `UPDATE … status/finished_at` on the same `fctx` (`FinishTimeout` = 10 min). A large crawl can run out that budget inside `SavePages`/`SaveIssues`: up to 50 000 pages and 1000 links per page in one transaction (L3 WR-02 left it unbounded). The store then fails with `context deadline exceeded`, and the terminal update fails on the same expired context, so `finish` returns at `:397`. Three things follow:
- the row stays `running`, and the real error is lost;
- the conga record is not ended by `finish`;
- the guard blocks new crawls until `RecoverStale` runs another 10 minutes later (heartbeat plus `StaleAfter`) and records a misleading "no heartbeat since …" error.
**Fix:** Give the final state its own fresh, short bound, separate from the store's:
```go
if err := lagoon.Transaction(ctx, db, store); err != nil { status, errText = models.CrawlFailed, err.Error() }
wctx, wcancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second)
defer wcancel()
wdb := db.WithContext(wctx)
if err := wdb.Model(&models.Crawl{}).Where("id = ?", row.ID).Updates(updates).Error; err != nil { return err }
// FailJob/StopJob/CompleteJob and Prune on wctx too
```
Add a test with a store that blocks past a small `FinishTimeout`, which checks for `failed` with the store error.
### WR-04: The root-file 503 also catches sites whose backend has no SEO plugin, after any outage longer than the cache TTL
**File:** `vue-seo-layer/server/lib/site.ts:47-63, 96-99`
**Issue:** `lastOk` remembers only an `ok` answer. An `absent` answer (a backend without SEO v0.2.0, for which the layer promises v0.1.1 behaviour) is cached for 5 minutes only. Once it expires, any moment the backend is unreachable or answers something other than 200 or 404 is `unknown` with `lastOk == null`. The new branch then proxies `/robots.txt`, `/sitemap.xml` and `/llms.txt` to a backend that never served them:
- if the fetch fails or answers 5xx, the layer serves 503;
- if the backend answers 404, the layer passes that 404 on.
Before 33-16 these requests reached the app. That covers robots.txt or sitemaps produced by a Nuxt server route or module (`@nuxtjs/robots`, `@nuxtjs/sitemap`), or the app's own 404. Files in Nuxt's `public/` are still served first by Nitro's static handler.
A 5xx robots.txt is read by Google as a site-wide disallow for up to about 12 hours. So on an SEO-less site, every short backend outage now stops crawling of the whole site. The same happens to a backend whose `/site` answers 200 with HTML (a catch-all) or 401/403 from an auth gateway: it is `unknown` for good, and its root files get hijacked.
**Fix:** Remember the last definite answer, not only `ok`, and narrow `unknown` to "never answered":
```ts
let lastKnown: SeoSiteState | null = null
...
if (got.state !== 'unknown') lastKnown = got
if (got.state === 'unknown' && lastKnown) return lastKnown
```
`rootFileTargets` then returns null for a remembered `absent`. Add a `rootFiles.test.ts` case: absent, then unknown, should still pass through.
## Notes below the warning bar (not counted)
- **Shutdown during a crawl.** A graceful worker shutdown (`conga.Worker.Stop` falls back to `StopAndCancel` with 5 s, `summercms.go/modules/conga/worker.go:219-226`) cancels the job context. `finalStatus` maps that to `cancelled` with no actor (`job.go:345-346`), so a deploy during a nightly crawl can leave a partial "cancelled" crawl as the current audit. The `Canceled → cancelled` mapping also covers `Cancel` calling `CancelJob` before its own transaction commits (`job.go:557-562`). Moving `CancelJob` to `AfterCommit` would allow a separate "interrupted" status.
- **DeleteCrawls stale rows.** `DeleteCrawls` calls `RecoverStale` but not `endStaleJobs` (`actions.go:81-83`), so the conga records of those crawls are never ended.
- **Open transaction during collect.** `Rebuild` keeps its transaction open (idle) for all of `collect` (`rebuild.go:144-170`). A database with `idle_in_transaction_session_timeout` set would kill long rebuilds.
- **Parameter limit.** `postContentLocales` passes every published post id of a page of tags or authors as one `IN ?` list (`menu_item_types.go:1150-1162`, `786-805`). Past about 65k posts, Postgres's parameter limit fails the descriptor call and with it the rebuild.
- **Repeat IndexNow pings.** `RecordSaved` marks rows that are already excluded (an unpublished record's old rows), and `apply` then sends them as removed again on every edit (`rebuild.go:539-545`).
- **Version pin.** sm-seo-plugin's go.mod still requires sm-sitemap-plugin v0.1.2, which has no `contract` package (`GOWORK=off go build` fails). The v0.2.0 pin is a 33-14 step, as the SUMMARY records.
---
_Reviewed: 2026-11-04_
_Reviewer: Claude (msd-code-reviewer)_
_Depth: deep (targeted re-review of the 33-15 and 33-16 ranges)_