chore: require SummerCMS v0.6.1 and sm-sitemap-plugin v0.2.0

The store keeps FinishTimeout, the final status write and the conga job
end get their own 30 s bound and prune its own store-sized bound, so the
row no longer stays running with the error lost until RecoverStale; adds
the code-only crawl.WithFinishTimeout seam.

quick-261104-xae, review WR-03.

README.md

@@ -1,6 +1,6 @@
 # golem15.seo
 
-The SEO plugin for SummerCMS applications: an SEO tab on every content type, per-URL rules, redirects and the 404 log, robots.txt with AI bot switches, global and per-type SEO settings, change history, and the redirect decision and effective SEO of any URL for the frontend.
+The SEO plugin for SummerCMS applications: an SEO tab on every content type, per-URL rules, redirects and the 404 log, robots.txt with AI bot switches, global and per-type SEO settings, change history, a site audit with a built-in crawler, and the redirect decision and effective SEO of any URL for the frontend.
 
 ```go
 import _ "git.golem15.com/golem15/sm-seo-plugin"
@@ -12,9 +12,9 @@ Module `git.golem15.com/golem15/sm-seo-plugin`, package `seo`, plugin ID `golem1
 
 The plugin owns all SEO data of a site; the framework only carries the generic seams it builds on: form extensions (`pact.AdminFormExtension`), record events (`pact.RecordSaved`, `pact.RecordDeleted`) and content descriptors (`pact.ContentDescriptor` in signpost). Content plugins never import it. A content type takes part by registering a content descriptor on its menu item type; the plugin then adds an **SEO** tab to that type's admin form and stores the values in its own table, keyed by the record's morph type, its id and the locale.
 
-The frontend (a Nuxt layer or app) asks `GET /_seo/api/v1/resolve?url=...` for the page it renders. The answer is first the redirect decision (URL normalisation, redirects, strict URLs, reached in one hop), then the effective SEO. Every value is resolved with one precedence: **URL rule > record > per-type template or global settings > values derived from the content**, and the resolver records the source of each value. The frontend enforces the decision; the plugin registers no routes at the site root except `/robots.txt`.
+The frontend (a Nuxt layer or app) asks `GET /_seo/api/v1/resolve?url=...` for the page it renders. The answer is first the redirect decision (URL normalisation, redirects, strict URLs, reached in one hop), then the effective SEO. Every value is resolved with one precedence: **URL rule > record > per-type template or global settings > values derived from the content**, and the resolver records the source of each value. The frontend enforces the decision; the plugin registers no routes at the site root except `/robots.txt` and `/llms.txt`.
 
-`Requires()` is empty: content types are found through signpost, never by plugin ID.
+`Requires()` names `golem15.sitemap` (`SitemapPluginID`): the plugin publishes the sitemap plugin's `contract.Filter` (`classes.SitemapFilter`) in `Register` and reads the sitemaps only through that plugin's `contract` package (`contract.Listing`). Content types are still found through signpost, never by plugin ID; the sitemap plugin keeps working without this one.
 
 ## Features
 
@@ -34,14 +34,25 @@ The frontend (a Nuxt layer or app) asks `GET /_seo/api/v1/resolve?url=...` for t
 - **Non-production lock.** Outside the `production` environment every resolve answer is `noindex, nofollow` with `staging: true`, and `/robots.txt` is `User-agent: *` / `Disallow: /` with `X-Robots-Tag: noindex, nofollow`, whatever is stored. It depends on the environment only; no setting turns it off.
 - **History** (`golem15_seo_revisions`): every SEO write (the SEO tab, rules, redirects, robots.txt, settings, the 404 log) is recorded with author, time, via (admin, MCP, import, auto for address changes, undo) and the changed fields before and after. Deleted records keep their values in history.
 - **Undo** of one change or of a whole batch, from the History preview (record actions **Undo this change** and **Undo the whole batch**) and through MCP. An undo is refused when a later change touched the same values (`history.undo_stale`, naming who and when), when it would create a redirect loop (`history.undo_loop`) or when the record is gone (`history.undo_gone`); an undo is itself recorded.
-- **Batches** (D-12): a list of changes (SEO by URL, redirects, URL rules) applied all or nothing under one batch id. A dry run answers the diff and the conflicts (loops, chains, noindex on a published record's address, a canonical pointing to a redirected or noindex address) and writes nothing.
+- **Batches** (D-12): a list of changes (SEO by URL, redirects, URL rules) applied all or nothing under one batch id. A dry run answers the diff and the conflicts (loops, chains, noindex on an address a sitemap file lists now, a canonical pointing to a redirected or noindex address) and writes nothing.
 - **CSV import and export** of redirects, URL rules and the SEO of records (by URL), in the **Import and export** screen (element `golem15-seo-transfer`) and through MCP. Files are UTF-8 with a BOM, comma or semicolon; exported cells that start a formula (`=`, `+`, `-`, `@`, tab, CR), or apostrophes followed by one, are prefixed with an apostrophe, and the import drops exactly that one again, so a re-imported export is unchanged. An import is previewed row by row (new, changed, unchanged, refused, with notes) and applied as one batch with via `import`, or not at all while a row is refused. The screen's apply token is an HMAC-SHA256 under the app key over the file and a fingerprint of the data the preview read; it is refused with `transfer.stale` when the data changed. The screen previews files up to 60 KB and downloads exports up to 250 KB; MCP imports take up to 900 KB per call and exports come in pages of 1000 rows.
 - **Record events.** A deleted record loses its entries after commit; a created record gets its address snapshot; an updated record whose address changed gets its 301.
 - **Entry generation.** The `golem15.seo.generate_entries` job (conga) creates missing entries with the current address of every published record and refreshes the effective-value cache; a record whose address changed without a record event gets its address change (with its 301) like an admin save. Editor values are never changed.
-- **Field types** `seotext`, `mlseotext`, `seorobots`, `mlseorobots`, `seoimage` and `robotstxt`, with the custom elements `golem15-seo-text`, `golem15-seo-robots`, `golem15-seo-serp` (the result preview widget), `golem15-seo-image`, `golem15-seo-robotstxt` (the robots.txt editor), `golem15-seo-robotstester` (the tester widget) and `golem15-seo-transfer` (the import and export widget).
+- **Field types** `seotext`, `mlseotext`, `seorobots`, `mlseorobots`, `seoimage`, `robotstxt`, `seoschema` and `seoparent`, with the custom elements `golem15-seo-text`, `golem15-seo-robots`, `golem15-seo-serp` (the result preview widget), `golem15-seo-image`, `golem15-seo-robotstxt` (the robots.txt editor), `golem15-seo-robotstester` (the tester widget), `golem15-seo-transfer` (the import and export widget), `golem15-seo-schema` (structured data), `golem15-seo-graph` (the structured data preview widget), `golem15-seo-parent` (the primary category) and `golem15-seo-links` (the internal links widget).
-- **One permission**, `golem15.seo.manage`, for the SEO tab, rules, redirects, the 404 log, robots.txt, settings and history, in the admin and through MCP.
+- **Structured data.** Every `ok` resolve answer carries one JSON-LD `@graph` built from the content descriptor and the settings: the page node (WebPage or its subtype), the article with its authors, dates, images, word count and section, the person of a profile page, a Service with its provider and offer, the BreadcrumbList, one VideoObject per YouTube video, the WebSite (with a SearchAction when `search_url` is set) and the publisher. Editors pick the schema type per page, fill the Service fields and add custom JSON-LD; the SEO tab previews the graph with Google's required and recommended properties and links the Rich Results Test.
+- **Publisher and local business** settings: an organisation or a person with logo, website, contact and sameAs profiles, optionally a local business with address, coordinates, opening hours and price range.
+- **Breadcrumbs, hreflang and pagination.** The resolve answer carries the breadcrumb trail (through the record's primary category), the hreflang alternates with `x-default`, `html_lang` and `og:locale`, and the page of a paginated address in the site's form (`?page=N` or `/<segment>/N`).
+- **One permission**, `golem15.seo.manage`, for the SEO tab, rules, redirects, the 404 log, pages, auto-links, robots.txt, settings, history and the site audit, in the admin and through MCP.
 - **Management API and MCP.** Every SEO admin operation has a JSON route under `/_seo/api/v1` and an MCP tool bound to it (pier, prefix `seo_`). Tools address pages by URL: a change goes to the record behind the URL, else to an exact URL rule, and the answer says where. A read grant sees only the read tools. `AdminOperations` maps each admin operation to its tool, and a test fails when one has none.
-- **Issue report.** Missing and duplicate titles and descriptions, URL rules overriding a record, redirect chains, canonicals pointing to a noindex or redirected address, and noindex addresses robots.txt blocks, each with the tool that fixes it.
+- **Site audit** (SEO › **Site audit** and **Crawls**): a built-in crawler fetches the site's own pages like a search engine and reports per URL what it finds (statuses, sitemaps, canonicals, titles and descriptions, headings, structured data, hreflang, links, images, content that needs JavaScript, orphan pages), stored with the CMS-data checks: missing and duplicate titles and descriptions, URL rules overriding a record, redirect chains, canonicals pointing to a noindex or redirected address, noindex addresses robots.txt blocks, sitemap addresses the production robots.txt keeps Googlebot from (`sitemap_blocked`, severity `error`, with the deciding line), record images without alt text and H1 headings in content. Every issue names the MCP tool that fixes it, or `fix: code` with the template element to change. See [Site audit and crawler](#site-audit-and-crawler).
+- **Sitemap filter.** The sitemaps leave out every address whose effective SEO is noindex, whose canonical URL points elsewhere or that redirects (normalisation or a redirect rule), with the reason `noindex`, `canonical` or `redirect`. The filter decides on the production values, so staging and production list the same set. The SEO tab's status shows **Not in the sitemaps** with the reason, also when the content type is switched off in the sitemap settings.
+- **Page registry** (`golem15_seo_pages`, SEO › **Pages**): the site's own frontend pages (home, contact, offer pages) with an address and a name per locale, a kind (page, offer, service, landing), a parent, the date their content changed and a note. The registry is the content type `seo-page`, so each page gets the SEO tab, resolves `ok` and is listed in the sitemaps, grouped by kind into `sitemap-pages`, `sitemap-offers`, `sitemap-services` and `sitemap-landing-pages` files. An address another page or another content type owns is refused (`pages.url_taken`, `pages.url_owned`).
+- **Route report.** The frontend server reports its static routes; unknown ones become registry pages, a reported page missing from a newer report is marked **Not on the site**, and nothing is ever deleted or overwritten.
+- **Site answer** (`GET /_seo/api/v1/site`) with the facts the frontend layer needs, and **llms.txt** (off by default) listing the indexable pages from the sitemaps.
+- **Links** settings: links to other sites open in a new tab (on by default), and domains whose links always get `nofollow` or `sponsored`; the resolve answer carries them as `outbound`.
+- **Auto-links** (`golem15_seo_autolinks`, SEO › **Auto-links**): phrases that texts link to a page, per locale or for all, with an active switch, an origin and a note. Every `ok` resolve answer carries the phrases of its locale (never one linking to the page itself) as `autolinks`, with `autolink_max` from the Links tab (3 by default, 0 turns auto-linking off); the frontend layer links each phrase's first occurrence. **Add focus keyphrases** turns the records' focus keyphrases into phrases. See [Auto-links](#auto-links).
+- **Internal links** in the SEO tab (E10): the page's inbound and outbound link counts from the last finished crawl, a warning when no page links to it, and up to 10 pages its text could link to and 10 pages that could link to it (titles and focus keyphrases found in the crawled text), each with **Copy address**; `GET /_seo/api/v1/links` and `seo_get_link_suggestions` answer the same.
+- **Site cache purge.** Every SEO write (admin, MCP, batch, import, undo, automatic) asks the frontend to drop its cached pages after the commit, once per transaction, through the framework's site-cache purger (`site_cache.purge_url`, `site_cache.purge_token`).
 
 ## Usage
 
@@ -49,11 +60,13 @@ Mount the plugin in an application as a git submodule at `plugins/golem15/seo`,
 
 ```yaml
 plugins:
+  - id: golem15.sitemap
+    module: git.golem15.com/golem15/sm-sitemap-plugin
   - id: golem15.seo
     module: git.golem15.com/golem15/sm-seo-plugin
 ```
 
-The plugin's own `go.mod` and `go.work` replace the framework with `../summercms.go` for a sibling checkout.
+The SEO plugin requires the sitemap plugin, so both are listed. The plugin's own `go.mod` replaces the framework with `../summercms.go`, and its `go.work` also the sitemap plugin with `../sm-sitemap-plugin`, for sibling checkouts.
 
 ### Making a content type take part
 
@@ -99,10 +112,17 @@ The type then gets the SEO tab, a tab in the SEO settings and a place in the res
     "og": {"title": "About | Acme", "description": "Who we are", "image": "https://example.com/storage/media/seo/abc-about.png", "url": "https://example.com/pages/about"},
     "twitter": {"card": "summary_large_image", "site": "@acme"},
     "verification": []
-  }
+  },
+  "outbound": {"new_tab": true, "nofollow_domains": ["example.org"], "sponsored_domains": []},
+  "autolinks": [{"phrase": "lwia grzywa", "url": "/grzyby/lwia-grzywa"}],
+  "autolink_max": 3
 }
 ```
 
+`outbound` (present on `ok` answers) carries the settings' Links tab: open links to other sites in a new tab, and the domains (subdomains included) whose links always get `nofollow` or `sponsored`. The layer treats an absent key as empty.
+
+`autolinks` and `autolink_max` (present on `ok` answers) are the auto-link phrases of the page's locale and of every locale, active ones only, without those linking to the page itself, longest phrase first, at most 1000; `autolink_max` is the most the layer links in one text. With `autolink_max` 0 the answer carries no phrases.
+
 The pipeline runs normalisation, then redirects on the normalised path (followed to the final target), then strict URLs, then the SEO. When the URL is not in its final form the answer is a redirect, reached in one hop:
 
 ```json
@@ -113,6 +133,36 @@ The pipeline runs normalisation, then redirects on the normalised path (followed
 
 `status` is 302 when a temporary redirect takes part. A 410 rule answers `"action": "gone"`, `"status": 410`. A redirect cycle met at runtime (stored past the save check) stops after 10 hops with `not_found` and is logged at most once an hour per cycle (keyed by its rules, not by the requested URL). A path that no published record owns and no URL rule covers answers `"action": "not_found"`, `"status": 404` and `"seo": null` (the home page `/` is always known). A `url` that is neither a path nor an http(s) URL is a `422`. Absolute URLs are built from `golem15.seo.site_url`, or from the request URL when it names the site's host or its `www` twin (for the scheme and host rules), never from the request's Host header.
 
+### Structured data, breadcrumbs, hreflang and pagination
+
+An `ok` resolve answer also carries (all additive, absent on other actions):
+
+```json
+{
+  "html_lang": "en",
+  "alternates": [
+    {"hreflang": "en", "url": "https://example.com/blog/first-post"},
+    {"hreflang": "pl", "url": "https://example.com/pl/blog/pierwszy-wpis"},
+    {"hreflang": "x-default", "url": "https://example.com/blog/first-post"}
+  ],
+  "breadcrumbs": [
+    {"name": "Home", "url": "https://example.com/", "path": "/"},
+    {"name": "Recipes", "url": "https://example.com/blog/category/recipes", "path": "/blog/category/recipes"},
+    {"name": "First post", "url": "https://example.com/blog/first-post", "path": "/blog/first-post"}
+  ],
+  "jsonld": {"@context": "https://schema.org", "@graph": ["…"]},
+  "seo": {"og": {"locale": "en_US", "locale_alternates": ["pl_PL"]}}
+}
+```
+
+- **The graph.** Nodes link by `@id`: `<site>/#website`, `<site>/#organization` or `<site>/#person` (the publisher), `<url>#webpage`, `<url>#breadcrumb`, `<url>#article`, `<url>#profile`, `<url>#service`, `<url>#video-<n>`; a collection page's `mainEntity` is `<url>#itemlist`, the node the frontend adds with the listed items. The type follows the descriptor's `Kind`: article → BlogPosting (Article or NewsArticle by choice), collection → CollectionPage, profile → ProfilePage with a Person, page → WebPage (AboutPage, ContactPage or Service by choice; registry pages of kind offer or service default to Service). No e-commerce type is generated.
+- **Custom JSON-LD** (the SEO tab's Structured data field, or `schema` on `seo_set_url_seo`): an object or a list of objects, at most 16 KB and 8 levels deep, without a `<script`; the server decodes it and stores it re-serialised. A node with a generated `@id` is merged into that node with the custom values first (the save answers the `schema.merged` notice); other nodes are added.
+- **Breadcrumbs** start with the home crumb (setting **Home page name in breadcrumbs**, default Home / Strona główna), follow the record's primary category (the SEO tab's Primary category, else the first of the descriptor's `Parents`) and each parent's own primary parent up to 6 levels, leaving out unpublished parents, and end with the page. The BreadcrumbList node has the same items; the home page's trail is the home crumb alone.
+- **hreflang** lists every locale the record has an address in, the page itself included, plus `x-default` on the default locale's address; a record in one locale has none. `og:locale` comes from `golem15.seo.og_locales`, else `pl_PL`, `en_US`, `de_DE`, else the language with its region.
+- **Pagination** (settings tab Pagination): the form is the query (`?page=N`, default) or the path (`/<segment>/N`, the segment per locale, default `strona` for pl and `page` for en). `?page=1` and `/<segment>/1` answer 301 to the unnumbered address, the other form 301s to the configured one, together with the address rules in one hop. A page value that is not a number answers `not_found` with `"pagination": {"invalid": true, …}`, so the frontend answers 404. Page 2 and later answers `ok` with its own canonical, `index, follow`, the title suffix after the page title and the description suffix at the end; `%page%` is filled and `%pages%` is left for the frontend, which knows the page count. Such an answer carries `"pagination": {"form", "param", "segment", "page", "invalid"}`.
+
+`GET /_seo/api/v1/site` also answers `"pagination": {"form", "param", "segments"}` and `"home_crumb": {"name", "url", "names"}` (the name in the default locale and per locale).
+
 ### The 404 report endpoint
 
 `POST /_seo/api/v1/notfound` takes `{"url": "/missing", "referer": "https://..."}` from the frontend server with `Authorization: Bearer <golem15.seo.notfound_token>` (compared in constant time) and answers `204`. A missing or wrong token is `401`; with no token configured the endpoint is off and answers `404`. URLs are capped at 2048 bytes (`422` beyond) and referers at 1024. The log holds at most `golem15.seo.notfound_max_rows` addresses; new ones beyond it are dropped with one warning an hour. Throttle bucket `seo-notfound`, 600 requests a minute per client IP.
@@ -131,7 +181,7 @@ Disallow: /
 Sitemap: https://example.com/sitemap.xml
 ```
 
-The stored rules come first, then one group per switched-off AI bot, then one `Sitemap:` line per entry of `golem15.seo.robots_sitemaps`, absolute on the site URL. Outside production the body is `User-agent: *` / `Disallow: /` with `X-Robots-Tag: noindex, nofollow`. It never answers a 5xx: when the stored file cannot be read it serves the last body it served, else `User-agent: *` / `Disallow:` (allow all), and logs the error.
+The stored rules come first, then one group per switched-off AI bot, then the `Sitemap:` line: the sitemap plugin's index (`contract.ListingSettings.IndexURL`, its site address plus `/sitemap.xml`; the per-type files are never listed, the index names them), or one line per entry of `golem15.seo.robots_sitemaps` when that key is set, absolute on the site URL. Outside production the body is `User-agent: *` / `Disallow: /` with `X-Robots-Tag: noindex, nofollow`. It never answers a 5xx: when the stored file cannot be read it serves the last body it served, else `User-agent: *` / `Disallow:` (allow all), and logs the error.
 
 The **robots.txt** screen (a singleton) has the editor, the tester and **What crawlers get** on its first tab and the AI bot switches on the second. The tester sends the editor's text and the switches as they are on screen, saved or not, and answers allowed or blocked for the chosen crawler with the deciding line, or that a bot switch blocked it.
 
@@ -139,6 +189,80 @@ The **robots.txt** screen (a singleton) has the editor, the tester and **What cr
 
 `GET /_seo/api/v1/verification/{name}` serves a verification file from the settings tab Verification (name `[A-Za-z0-9._-]+\.(html|txt|xml)`) with the content type of its extension, `X-Content-Type-Options: nosniff` and a sandboxing Content-Security-Policy; any other name is `404`. The frontend serves it at the site root; the plugin registers no `/{name}` route, which would capture every one-segment page of a site served at `GET /`. Every `/_seo/api/v1` answer carries `X-Robots-Tag: noindex`.
 
+### The page registry and the route report
+
+**SEO › Pages** (controller `golem15.seo.pages`, after URL rules) lists the registry by address with the kind, origin, content date and the **Not on the site** state; the form takes the address and the name per locale (a locale left empty has no version of the page), the kind, the parent and the content date (the sitemap's lastmod), with the SEO tab next to it. A changed address gets its 301 like any record.
+
+`POST /_seo/api/v1/registry/routes` takes the frontend server's static routes and, optionally, the static paths it serves noindex, with `Authorization: Bearer <token>`: `golem15.seo.route_report_token`, else `golem15.seo.notfound_token` (compared in constant time; prefer a dedicated `route_report_token`):
+
+```json
+{"routes": ["/", "/blog", "/kontakt"], "noindex": ["/konto", "/pl/konto", "/szkic"]}
+```
+
+It answers `200` with the counts and `noindex`, the number of stored noindex paths:
+
+```json
+{"inserted": 1, "updated": 3, "skipped": 0, "reported_at": "2026-11-04T20:00:00Z", "noindex": 3}
+```
+
+A missing or wrong token is `401`, no token configured `404`, more than 2000 entries in either list `422` `{"error": "too_many_routes", "max": 2000}` (nothing is stored).
+
+`noindex` lists every static path the frontend's build serves noindex (each starting with `/`, without a `:` parameter or `*`, at most 2048 bytes), the locale copies as their own prefixed paths. When the key is present (also as `[]`) the stored set (`golem15_seo_noindex_paths`) becomes exactly its valid entries, normalised like the routes; an invalid entry or a pattern is skipped and counted in `skipped`. When the key is absent the set stays as it is, so a frontend that sends no `noindex` keeps working. The sitemap filter leaves every address whose path is in the set out of the XML and HTML sitemaps with the reason `noindex` (a record's address, a registry page or an Additional address of the sitemap settings), and a changed set asks the sitemap for one debounced rebuild through `contract.Rebuilder` (with an older sitemap plugin the next scheduled rebuild picks it up).
+
+For the routes, an unknown path becomes a page with origin `reported`, kind `page` and a name from its last segment; a known page only gets `reported_at`; a path another content type owns, or one that is no path, is skipped. A report never deletes a page or changes a value an editor set; a reported page the newest report did not name shows **Not on the site** and a warning on its form. Throttle bucket `seo-notfound`. A content type whose file group is also `pages` shares the file with the registry's pages.
+
+### Auto-links
+
+**SEO › Auto-links** (controller `golem15.seo.autolinks`, after Pages, icon `wand-sparkles`) lists the phrases (50 per page) with the address they link to, the language (**All languages** or a locale), the origin (**Manual**, **Focus keyphrase**, **MCP**) and the update time; an inactive phrase shows **Off**. Filters: language, origin and **Active only**. The form takes the phrase, the address, the language and the **Active** switch, and a note. A phrase needs 3 characters (`autolinks.phrase_short`); a phrase exists once per language, a phrase of every language once at all (`autolinks.phrase_taken` names the address it already links to); the address is a path starting with one `/` or a full `http(s)` address (`autolinks.url_invalid`: `javascript:`, `data:` and protocol-relative addresses are refused). The toolbar's **Add focus keyphrases** adds one phrase per record whose focus keyphrase in a locale is not yet a phrase there, linking to the record's address in that locale (origin **Focus keyphrase**), and toasts the count; a second run adds nothing new. Every change is recorded in History and purges the site's cached pages.
+
+The SEO tab's **Internal links** (widget `golem15-seo-links`, action `seo_links`, after the structured data preview, on the update form of content types whose records have a body: not for collections, profiles or the page registry) reads the last finished crawl: **Links to this page** (pages linking here) and **Links from this page** (pages it links to on the site), a warning when nothing links here, **Pages you could link to from this text** (pages whose H1 or focus keyphrase the crawled text contains and that it does not link to yet, with the public address and **Copy address**) and **Pages that could link here** (pages whose crawled text contains this page's title, H1 or focus keyphrase and that do not link here yet, with their admin form), 10 each. Before the first crawl it says when the counts appear.
+
+The layer links a phrase at its first occurrence in a text, as whole words and ignoring letter case, never inside headings, links, `code`, `pre`, `figcaption` or buttons, and never on the page it links to, at most `autolink_max` phrases per text (the Links tab's **Auto-links per page**, 0 to 20).
+
+### The site answer and llms.txt
+
+`GET /_seo/api/v1/site` (public, bucket `seo-resolve`, `X-Robots-Tag: noindex`) answers the facts the frontend layer needs, all public:
+
+```json
+{
+  "site_url": "https://example.com",
+  "default_locale": "pl",
+  "locales": ["pl", "en"],
+  "unprefixed_locale": "pl",
+  "links": {"new_tab": true},
+  "html_sitemap": {"enabled": true, "paths": {"pl": "/mapa-strony", "en": "/sitemap"}, "page_size": 150},
+  "routes": [{"type": "journal-post", "pattern": "/blog/:slug"}],
+  "root_files": ["google0123.html", "0123456789abcdef0123456789abcdef.txt"],
+  "llms": false
+}
+```
+
+`routes` lists each content type's layered URL pattern (the registry has none); `root_files` the verification files and the IndexNow key file while IndexNow is on; `html_sitemap` comes from the sitemap plugin's `contract.Listing`.
+
+`GET /llms.txt` (bucket `seo-robots`) is off by default. With **Serve llms.txt** on (robots.txt screen, AI bots tab, or `seo_update_robots` with `llms_enabled`) and in production it answers `text/plain`: `# <site name>`, the introduction (`llms_intro`, else the default meta description) as a quote, then one `## <group>` section per sitemap group with `- [title](url)` lines of the default locale's listed addresses, in the content types' order. Addresses the sitemaps leave out never appear. Off, or outside production, it is `404`.
+
+### Site audit and crawler
+
+A crawl is a conga job (`golem15.seo.crawl`, one attempt) started from **Start crawl** (on Site audit and Crawls), through MCP (`seo_start_crawl`), with `seo:crawl`, or by the nightly schedule. One crawl runs at a time: a start while one is queued or running refuses with `crawls.already_running` (the start locks the settings row, so two starts cannot both pass), and a site without an address (`golem15.seo.site_url`, else `app.url`) with `crawls.no_site_url`.
+
+**What it fetches.** `robots.txt`, the sitemap index at `/sitemap.xml` and its files (or a single urlset), then breadth-first from the home page and the sitemap addresses over same-host links, redirect targets, canonicals and hreflang alternates. Requests go through a `fetchguard` client limited to the site host and its `www` twin: HTTPS only, private and reserved addresses refused at dial time, redirects never followed (each `Location` is a new address checked again), the User-Agent `SummerCMS-SEO-Crawler/1 (+<site>)`. Links to other hosts, credentials in a URL and plain http are never fetched. A staging or local site served over http or on a private address needs `golem15.seo.crawl_trusted: true` in config (or `SUMMER_GOLEM15__SEO__CRAWL_TRUSTED`), which no setting or MCP call can set; trusted mode still fetches only the site host and its twin. robots.txt rules are read for the report, not obeyed.
+
+**Limits.** Two requests at a time with the settings' pause between requests (default 250 ms), a 10 s timeout and a 5 MiB body cap per request, at most 1000 links read per page, and the settings' page limit (default 5000; a crawl that stops there is marked "limit reached"). Cancelling (**Cancel crawl**, `seo_cancel_crawl`) ends a queued crawl at once; a running crawl records the request (`cancel_requested` in the API and MCP answers, **Cancelling** in the list) and keeps running until its job stops between requests, then ends cancelled with what was fetched. Until then no other crawl starts (409 `crawls.already_running`) and the crawl cannot be deleted; a failed crawl keeps the pages and issues found before the failure and records the error. A crawl that reaches the job's time limit (`crawl.JobTimeout`, 6 h) ends failed with "the crawl reached its time limit of 6h" and keeps what it fetched: storing what it found runs on a context the job's cancellation cannot reach, bounded by `crawl.FinishTimeout` (10 min), and the crawl's final status is then written on a 30-second bound of its own, so a crawl whose results take longer to store still ends at once, failed with "storing the crawl results took longer than 10m0s: …". A running crawl's job writes `heartbeat_at` every minute (`crawl.HeartbeatEvery`); a running crawl without a heartbeat for 10 minutes (`crawl.StaleAfter`, its job is gone, for example after a worker restart) is ended by the next start or delete (`crawl.RecoverStale`): failed with "the crawl job stopped without finishing (no heartbeat since …)", or cancelled when its cancel was requested. Every crawled address is keyed by its decoded path with the raw query (`classes.PathKey`, the form of the resolver, the descriptors and the SEO entries; a path that does not decode to valid text, or holds `%` or `?`, keeps its encoded form) and requested percent-encoded (`classes.EscapeKeyPath`), so a non-ASCII address matches its CMS record, its focus keyphrase and its links in the E10 widget, written either way. The crawler writes only its own tables (`golem15_seo_crawls`, `golem15_seo_crawl_pages`, `golem15_seo_crawl_links`, `golem15_seo_issues`); crawls beyond the settings' "Crawls kept" are deleted with their pages, links and issues.
+
+**Checks** (kind, severity): `page_status` (error), `sitemap_error` (error), `sitemap_noindex` (error), `sitemap_blocked` (error), `sitemap_missing` (notice), `broken_link` (error, with the linking pages), `redirected_link` (notice), `canonical_missing` (warning), `canonical_error` (error), `missing_title` and `duplicate_title` (warning), `missing_description` and `duplicate_description` (notice), `h1_missing` and `h1_multiple` (error), `heading_skipped` and `heading_empty` (warning), `h1_title_mismatch` (notice), `jsonld_invalid` (error), `jsonld_multiple` (warning), `jsonld_required` (error) and `jsonld_recommended` (notice) from the same rules as the structured data preview (`schema.Validate`; FAQPage and other types without Google rules find nothing), `hreflang_reciprocal` (warning), `hreflang_error` (error), `generic_anchor` (warning, the settings' generic link texts, whole text, any case), `image_alt` (warning: an `<img>` without `alt`; `alt=""` is decorative), `image_size` and `featured_lazy` (notice), `content_js` (warning) and `orphan_page` (warning: an indexable sitemap page no crawled page links to, with up to 10 pages whose text mentions its title or focus keyphrase). The CMS-data checks run at the end of every crawl (and when the list is opened before any crawl) and are stored with the crawl's issues as source `cms`; a CMS issue the crawl found too is stored once.
+
+**Raw HTML against the CMS.** The server-rendered HTML is checked against the record behind the page (`content_js` with `details.check: cms`): the record's title must appear in an H1, the text must reach half of the record's word count, and the record's primary parent (the breadcrumb link) must be linked. The comparison with the JavaScript-rendered page is `details.check: rendered` (next section); without a Chrome path every crawl records it as skipped (P6 and `seo_get_crawl` say so).
+
+**Rendered comparison.** When `golem15.seo.crawl_chrome_path` names a Chrome or Chromium binary the operator installed on the server (none is installed by default), the crawler renders the first `golem15.seo.crawl_render_max` pages (default 200, in crawl order, so the home page and the top-level pages come first) in headless Chrome through `github.com/chromedp/chromedp`, right after each page's raw fetch, and compares the DOM after JavaScript with the raw HTML: same-host links, an H1, or at least 50 words and 20 % more text that appear only after JavaScript are one `content_js` finding with `details.check: rendered` and `details.what` naming them (up to five links). The browser is held to the crawler's limits: before Chrome starts, the site host and its `www` twin are resolved and every address checked like the crawler's dial guard (`fetchguard.IsPrivateAddr`; a private address refuses unless `golem15.seo.crawl_trusted` is on), and Chrome is pinned to those addresses through `--host-resolver-rules` while every other host resolves to nothing; every request is intercepted and continued only when it is https (http only in trusted mode) to the site host or its twin on the site's own port (the port of `golem15.seo.site_url`, else 443 for https and 80 for http), or a `data:`/`blob:` URL, so page code cannot reach other services on the site host's address, so third-party scripts, fonts, analytics and metadata addresses are never loaded. Each crawl gets a fresh temporary profile, removed when the crawl ends, with no cookies, credentials or extensions and downloads denied; one tab renders at a time, at most 5 s to open the tab (`crawl.TabOpenTimeout`) and 20 s for the page under one deadline, so a hung Chrome cannot stall the crawl, the DOM is capped at 5 MiB, Chrome's own sandbox stays on, and Chrome is closed when the crawl finishes, fails or is cancelled. The path is config only (or `SUMMER_GOLEM15__SEO__CRAWL_CHROME_PATH`), never a setting or MCP value. The crawl records `render_status` `done` with the pages compared, `skipped` without a path, or `failed` with the reason (a relative path or one that is not an executable file, a private address, Chrome not starting, a page that cannot be rendered); a renderer failure never fails the crawl, and P6, `GET /_seo/api/v1/crawls/{id}` and `seo_get_crawl` (`render {status, pages, note, message}`) say which.
+
+**Fix routes.** An issue a CMS value fixes names its tool: `seo_set_url_seo` (titles, descriptions, canonicals, sitemap membership of a record, structured data), `seo_update_robots`, `seo_create_redirect` (a missing page, a page that answers an error), `seo_create_page` (a page missing from the registry), `seo_update_settings` (publisher and site nodes), `sitemap_update_settings` (a listed noindex address without a CMS record, an Additional address of the sitemap: remove it there; `details.extra` is true, and without the sitemap plugin the issue is `fix: code` naming the sitemap's address list), or the content type's own tool from `pact.ContentDescriptor.Tools` when the problem sits in the record's content: H1 findings prefer `Tools[headings]`, image alt text `Tools[image_alt]`, other content findings `Tools[edit]` (the registry pages name `seo_update_page`). A finding of the site's template code, or of content whose type names no fitting tool, is `fix: code` with `details.what` naming the element to change. No issue has neither.
+
+**Ignore and check again.** An issue's fingerprint (kind, address, locale and the `content_js` check) can be ignored (**Ignore**, bulk **Ignore issues**, `seo_ignore_issues`); it stays hidden in later crawls until **Stop ignoring** (`seo_unignore_issues`). **Check this address again** (`seo_recheck_url`) fetches the issue's address (with its canonical and hreflang targets) and replaces that address's issues; findings that depend on other pages (duplicates, orphans, link findings while the page still errors) stay until the next crawl.
+
+**Staging.** On a non-production environment every page carries the staging lock's noindex; the crawl never counts it (no `sitemap_noindex`, the pages stay indexable for the other checks) and P6 says why.
+
+**Schedule.** The schedule entry runs `seo:crawl --scheduled` every night at 02:30 app time; it queues a crawl when the setting **Automatic crawl** is daily, or weekly on a Monday in `app.timezone`, and does nothing while it is off or a crawl runs.
+
 ### Management API and MCP tools
 
 The management routes live under `/_seo/api/v1` (bucket `seo-api`, 300 requests a minute per client IP). Each resolves the administrator from a backend JWT, a personal API token (scope `golem15.seo.*`) or pier's in-process dispatch, so an MCP call runs as the backend account linked to the MCP user; each answers `401` without one and `403` without `golem15.seo.manage`. Field refusals are `422 {"error": "validation_failed", "errors": {...}}`; a redirect that would create a loop is `422 {"error": "redirect_loop", "path": [...], "message": "..."}`, so the tool call fails; a missing row is `404`. History records these writes with via `mcp` when they come through MCP, and redirects created through MCP have origin `mcp`.
@@ -146,24 +270,40 @@ The management routes live under `/_seo/api/v1` (bucket `seo-api`, 300 requests
 | Tool (`seo_` + name) | Route | What it does |
 |------|-------|--------------|
 | `get_url_seo` | `GET /seo?url=` | Effective SEO of a URL with the source of every value, the redirect decision, the owner and the SEO tab's warnings. |
-| `set_url_seo` | `PUT /seo` | Values for a URL: the record behind it in the URL's locale, else an exact URL rule; `force_rule` writes a rule. Answers `written_to` (`record` or `rule`), `type`, `id`, `locale`, `rule_id`. |
+| `set_url_seo` | `PUT /seo` | Values for a URL: the record behind it in the URL's locale, else an exact URL rule; `force_rule` writes a rule. `primary_parent` (`<type>:<id>`) and `schema` (the structured data JSON string) are records only and shared by every locale. Answers `written_to` (`record` or `rule`), `type`, `id`, `locale`, `rule_id`. |
+| `get_graph` | `GET /graph?url=&schema=` | The JSON-LD graph of a URL and every node's missing required and recommended properties (`rules` false for a type Google has no rules for); `schema` previews an unsaved value. Answers `{locale, url, graph, nodes}`. |
+| `get_link_suggestions` | `GET /links?url=` | A page's internal links from the last finished crawl: `{crawledAt, inbound, outbound, orphan, out [{phrase, title, url}], in [{phrase, title, adminUrl}]}`, each list at most 10; `crawledAt`, `inbound` and `outbound` are `null` before the first crawl. |
 | `list_rules`, `show_rule`, `create_rule`, `update_rule`, `delete_rule` | `GET/POST /rules`, `GET/PUT/DELETE /rules/{id}` | URL rules. |
 | `list_redirects`, `show_redirect`, `create_redirect`, `update_redirect`, `delete_redirect` | `GET/POST /redirects`, `GET/PUT/DELETE /redirects/{id}` | Redirects; a chain is saved with a warning. |
 | `list_not_found`, `redirect_not_found`, `gone_not_found`, `delete_not_found` | `GET /notfound`, `POST /notfound/{id}/redirect`, `POST /notfound/{id}/gone`, `DELETE /notfound/{id}` | The 404 log and its actions; an address a rule answers is `409`. |
-| `show_robots`, `update_robots`, `test_robots` | `GET/PUT /robots`, `GET /robots/test?url=&agent=` | robots.txt, the AI bot switches and the tester on the saved file. |
+| `show_robots`, `update_robots`, `test_robots` | `GET/PUT /robots`, `GET /robots/test?url=&agent=` | robots.txt, the AI bot switches, `llms_enabled` and `llms_intro`, and the tester on the saved file. |
-| `show_settings`, `update_settings` | `GET/PUT /settings` | The SEO settings and, under `types`, the per-type settings. |
+| `show_settings`, `update_settings` | `GET/PUT /settings` | The SEO settings (including `links_new_tab`, `nofollow_domains`, `sponsored_domains`, `autolink_max`, `breadcrumb_home`, the `page_*` pagination keys, the `publisher_*` and `local_*` keys) and, under `types`, the per-type settings. |
 | `list_history`, `show_history` | `GET /history`, `GET /history/{id}` | History, filtered by `via`, `kind` and `batch`. |
 | `undo_change`, `undo_batch` | `POST /history/{id}/undo`, `POST /history/batches/{batch}/undo` | Undo; refusals are `409` with `undo_stale`, `undo_loop`, `undo_gone` or `already_undone`. |
 | `apply_batch` | `POST /batch` | Changes (`set_seo`, `create_redirect`, `update_redirect`, `delete_redirect`, `create_rule`, `update_rule`, `delete_rule`) all or nothing; `dry_run` answers the diff and the conflicts. A refused change is `422` (`redirect_loop` with the change and the path, or `batch_refused`). |
 | `export_csv` | `GET /export?dataset=&delimiter=&page=` | One page of 1000 rows as CSV text. |
 | `import_csv` | `POST /import` | A CSV file of a dataset; `dry_run` previews it. A file over 900 KB is `413`; `missing_column`, `unreadable` and `import_refused` are `422`. |
-| `list_issues` | `GET /issues` | The CMS-data issue report with a count per kind. |
+| `list_issues` | `GET /issues` | The site audit's current issues, errors first, with the Phase 32 keys (`kind`, `url`, `locale`, `details`, `tool`) and `id`, `severity`, `source`, `fix`, `found_at`, `ignored`, `crawl_id`, and a count per kind. Filters `severity`, `kind`, `source`, `fix`, `url`, `crawl` and `status` (`open` by default, `ignored` or `all`). |
+| `recheck_url` | `POST /issues/{id}/recheck` | Fetches an issue's address again and replaces its issues; answers `{url, remaining, message}`. |
+| `ignore_issues`, `unignore_issues` | `POST /issues/ignore`, `POST /issues/unignore` | Ignore issues by `ids` (hidden in later crawls) or show them again. |
+| `start_crawl` | `POST /crawls` | Starts a crawl (`202` with the crawl, trigger `mcp`); `409 crawls.already_running`, `422 crawls.no_site_url`. |
+| `list_crawls`, `get_crawl` | `GET /crawls`, `GET /crawls/{id}` | Crawls, newest first, with `render {status, pages, note, message}` (`skipped`, `done` or `failed`); one crawl adds `staging`, `issues`, `severities` and `kinds`. |
+| `cancel_crawl` | `POST /crawls/{id}/cancel` | Cancels a queued or running crawl, keeping what it fetched; another answers `409 crawls.not_running`. |
+| `delete_crawls` | `DELETE /crawls?ids=` | Deletes crawls with their pages and issues; a queued or running one refuses the call with `403 crawls.delete_running`. |
+| `list_autolinks`, `create_autolink`, `update_autolink`, `delete_autolinks` | `GET/POST /autolinks`, `PUT /autolinks/{id}`, `DELETE /autolinks?ids=` | The auto-link phrases: `phrase`, `url`, `locale` (`""` or `all` for every locale), `enabled`, `note`; the list filters by `search`, `locale`, `origin` and `active`. A phrase created through MCP has origin `mcp`. |
+| `autolinks_from_keyphrases` | `POST /autolinks/from-keyphrases` | Adds the records' focus keyphrases as phrases; answers `{added, message}`. |
+| `list_pages`, `create_page`, `update_page`, `delete_page` | `GET/POST /pages`, `PUT/DELETE /pages/{id}` | The page registry: `url` and `title` per locale (a plain string is the default locale's), `kind`, `parent_id`, `content_updated_at`, `note`; the list filters by `id`, `kind`, `origin` and `missing`. A page created through MCP has origin `mcp`. |
 
-Lists take `page`, `per_page` (at most 200) and `search`, and answer `{"data": [...], "page", "per_page", "total"}`. GET tools are read-only; the deletes and `gone_not_found` are destructive.
+Lists take `page`, `per_page` (at most 200) and `search`, and answer `{"data": [...], "page", "per_page", "total"}`. GET tools are read-only; the deletes (`delete_crawls` and `delete_autolinks` included) and `gone_not_found` are destructive.
 
 ### Admin
 
-- Navigation item **SEO** (icon `search`), landing on **Redirects**, with the side menu **Redirects**, **404 log**, **URL rules**, **robots.txt**, **History** and **Import and export**.
+- Navigation item **SEO** (icon `search`), landing on **Redirects**, with the side menu **Redirects**, **404 log**, **URL rules**, **Pages**, **Auto-links** (`wand-sparkles`), **Site audit** (`gauge`), **Crawls** (`globe`), **robots.txt**, **History** and **Import and export**.
+- **Site audit** lists the current issues, errors first, with severity, problem, address, details, found by and fix; filters severity, problem, found by, fix and status (Open or Ignored; ignored rows carry the **Ignored** badge). The preview shows what was found, how to fix it (the tool to ask an MCP client for, or the template element) and the page's address, with **Check this address again**, **Ignore** and **Stop ignoring**. The toolbar has **Start crawl**, the list the bulk **Ignore issues**.
+- **Crawls** lists the runs newest first (started, status, pages, issues, started by, duration). The preview shows one status callout (running, queued, failed, cancelled, limit reached), the staging and rendered-comparison lines, the stats strip and the issues per kind, each count linking to the filtered Site audit; **Cancel crawl** while queued or running. There is no live progress: reload the page.
+- The SEO settings' **Site audit** tab (before Verification): **Automatic crawl** (off, daily, weekly), **Pages per crawl** (10–50 000), **Pause between requests** (100–5000 ms), **Crawls kept** (1–50) and **Generic link texts** (seeded with the English and Polish lists).
+- The SEO settings have a **Pagination** tab (after Addresses), a **Links** tab (**Auto-links per page**, 0 to 20, the new-tab switch and the nofollow and sponsored domain lists, one domain per line; `settings.domain_invalid` names a bad line), a **Publisher** tab with the generated publisher node and its findings, and a **Local business** tab. A person cannot be a local business (`settings.local_person`).
+- The SEO tab adds **Primary category** (article types), **Structured data** and its **preview** after the social image, then **Internal links** (types with a body); its status warns when a stored primary category is no longer assigned, and the effective values show the schema type and the breadcrumb trail.
 - Settings tile **SEO settings** (`seo_settings`), a link to the settings singleton controller.
 - Every SEO screen and the SEO tab require `golem15.seo.manage`. Without it an administrator sees no SEO tab, and a posted SEO value changes nothing.
 
@@ -177,8 +317,35 @@ The element bundle (`assets/dist/seo.js` and `seo.css`) is built from `assets/sr
 | `TabSEO`, `TabGeneral`, `TabRobotsFile`, `TabRobotsBots` | The SEO tab key, the settings' General tab key and the robots.txt screen's tab keys. |
 | `BucketResolve`, `BucketNotFound`, `BucketRobots`, `BucketAPI` | The throttle buckets of resolve (and the verification files), the 404 report, `/robots.txt` and the management API. |
 | `AdminOperations`, `AdminOperation`, `ContentController` | Every SEO admin operation (controller, kind) and its MCP tool; `ContentController` (`*`) stands for the form of every content type. |
-| `OpList`, `OpShow`, `OpCreate`, `OpUpdate`, `OpDelete`, `OpExtensionSave`, `OpUpload`, `OpRecordAction`, `OpBulkAction`, `OpWidgetAction` | The kinds of an admin operation. |
+| `OpList`, `OpShow`, `OpCreate`, `OpUpdate`, `OpDelete`, `OpExtensionSave`, `OpUpload`, `OpRecordAction`, `OpBulkAction`, `OpWidgetAction`, `OpToolbarAction` | The kinds of an admin operation. |
 | `RobotsTestAction` | `robots_test`, the tester's widget action. |
+| `SitemapPluginID` | `golem15.sitemap`, the plugin `Requires()` names. |
+| `classes.SitemapFilter` | The published `contract.Filter`: `Decide` (reasons `noindex` (the effective SEO, or a path the route report names noindex), `canonical`, `redirect` on the production values; registry pages grouped by kind) and `Groups` (the kinds' groups with their titles per locale). |
+| `classes.PageMenuItemType`, `classes.PageTypeKey`, `classes.PagesController`, `classes.PageModel`, `classes.PageGroup` | The registry's menu item type `seo-page` and its content descriptor, and the file group of a kind. |
+| `classes.PageWrite`, `classes.RouteReport`, `classes.ValidPagePath`, `classes.PageMissing` | A registry write, a route report's outcome, the address check and the missing state. Service: `CheckPage`, `SavePage`, `DeletePage`, `ApplyPageTexts`, `ReportRoutes`, `ReportNoindex`, `RouteReportToken`, `PageAt`; `classes.NoindexPathCount` counts the stored noindex paths (`models.NoindexPath`). |
+| `classes.SiteAnswer` | `Service.Site`, the site answer. `Service.LLMSText` builds llms.txt. |
+| `classes.Outbound`, `classes.OutboundOf` | The resolve answer's `outbound` from the settings. |
+| `schema.Build`, `schema.Input`, `schema.Validate`, `schema.Normalize`, `schema.DecodeCustom`, `schema.Merge` | Package `classes/schema`: the JSON-LD graph of a page, Google's required and recommended properties per type, the structured data value and custom JSON-LD. |
+| `classes.Alternate`, `classes.Alternates`, `classes.LangTag` | hreflang alternates and language tags; `Service.OGLocale`. |
+| `classes.Pagination`, `classes.PageConfig` | The resolve answer's pagination block and a site's page form (`Service.PageConfigOf`). |
+| `classes.LinkReport`, `classes.LinkOut`, `classes.LinkIn`, `classes.RecordLinksAction`, `classes.LinksActionName`, `classes.MaxLinkSuggestions`, `classes.NormalText` | The internal links of a page (`Service.LinkSuggestions`, `Service.LinksOfURL`, `Service.LinksOfRecord`) and the SEO tab's `seo_links` action. |
+| `classes.GraphAnswer` | `Service.GraphOfURL` and `Service.GraphOfRecord`, the graph and its findings; `classes.RecordGraphAction` is the SEO tab's preview action. |
+| `classes.PrimaryParent`, `classes.ParentOption` | A record's primary category and its choices (`Service.ParentOptions`); `Service.Trail` builds the breadcrumb trail, `Service.PublisherOf` the publisher. |
+| `classes.SitemapExclusion` | `Service.RecordSitemapExclusions`, why the sitemaps leave out a record's addresses; `Service.SitemapListed` reads the listed addresses; `Service.SitemapLines` the robots.txt `Sitemap:` addresses. |
+| `classes.SiteCacheKind` | `seo`, the kind of the site-cache purge after an SEO write. |
+| `classes.IssueSitemapBlocked`, `classes.SeverityError` | The `sitemap_blocked` issue kind and its severity. |
+| `controllers.PagesControllerID` | The Pages screen. |
+| `controllers.AutolinksControllerID`, `controllers.ActionFromKeyphrases`, `controllers.KeyphrasesMessage` | The Auto-links screen, its toolbar action and the action's toast. |
+| `classes.AutolinkLink`, `classes.AutolinkWrite`, `classes.AutolinkValues`, `classes.NormalizeAutolink`, `classes.ValidAutolinkURL`, `classes.LoadAutolink`, `classes.AutolinkOrigin`, `classes.MaxPageAutolinks` | An auto-link in the resolve answer, a write, its values, the address check and the origin of a write. Service: `AutolinkProblems`, `SaveAutolink`, `DeleteAutolinks`, `RecordAutolink`, `ForPage`, `FromKeyphrases`. |
+| `controllers.IssuesControllerID`, `controllers.CrawlsControllerID`, `controllers.ActionStartCrawl`, `controllers.CrawlActionCancel` | Site audit and Crawls, their toolbar start and the cancel record action. |
+| `crawl.Crawl`, `crawl.Options`, `crawl.Result`, `crawl.Page`, `crawl.Fetcher`, `crawl.NewFetcher`, `crawl.Parse`, `crawl.PageFacts` | Package `classes/crawl`: one crawl's frontier and limits, the guarded fetcher and the facts of an HTML document. |
+| `crawl.Check`, `crawl.Env`, `crawl.Finding`, `crawl.Route`, `crawl.Fingerprint`, `crawl.Kinds`, `crawl.Severity`, `crawl.Record` | The checks, a finding's fix route (tool or code) and identity, and the kinds with their severities. |
+| `crawl.Start`, `crawl.Run`, `crawl.Cancel`, `crawl.Recheck`, `crawl.Ignore`, `crawl.Unignore`, `crawl.DeleteCrawls`, `crawl.Prune`, `crawl.Current`, `crawl.ScheduledDue` | Starting, running and cancelling a crawl, checking an address again, ignores, deletes and the schedule's decision. |
+| `crawl.CrawlArgs`, `crawl.CrawlKind` | The crawl job's arguments and River kind `golem15.seo.crawl`. |
+| `crawl.Renderer`, `crawl.WithRenderer`, `crawl.NewRenderer`, `crawl.RenderMax` | The rendered comparison's seam, its code-only test override, the guarded Chrome renderer built from config and the configured page cap. |
+| `crawl.WithFinishTimeout` | A code-only test override of `crawl.FinishTimeout`, like `crawl.WithRenderer`. |
+| `crawl.ChromePathConfigKey`, `crawl.RenderMaxConfigKey`, `crawl.TrustedConfigKey`, `crawl.RenderTimeout`, `crawl.MaxDOM` | The renderer's and crawler's config keys and the per-page render limits. |
+| `classes.IssueImageAlt`, `classes.IssueH1InContent` | The CMS-data kinds from the content descriptors (`Service.ContentIssues`). |
 | `classes.Service` | The single entry point for SEO reads and writes: `SaveRecord`, `Record` (history), `GenerateEntries`, `EnqueueGenerate`, `RecordDeleted`, `RecordCreated`, `Resolver`, `SiteURL`, `Staging`. Redirects: `SaveRedirect`, `DeleteRedirect`, `CheckRedirect`, `AddressChanges`, `AddressChanged`. 404 log: `ReportNotFound`, `ResolveNotFound`. robots.txt: `ServedFile`, `TestRobots`, `BlockedNoindex`, `SaveRobots`. `VerificationFileByName`. Management: `FindOwner`, `SetURLSEO`, `SaveRule`, `DeleteRule`, `SaveSettings`, `DeleteNotFound`, `Issues`. |
 | `classes.Owner`, `classes.URLSEOWrite`, `classes.URLSEOResult` | The owner of a URL (`classes.WrittenToRecord` or `classes.WrittenToRule`) and a write addressed by URL. |
 | `classes.Issue` | One finding of the issue report (`classes.IssueMissingTitle` and the other kinds). |
@@ -199,7 +366,7 @@ The element bundle (`assets/dist/seo.js` and `seo.css`) is built from `assets/sr
 | `classes.FieldTypes` | The SEO admin field types. |
 | `classes.WithVia`, `classes.ViaOf` | The writer history records (`admin`, `mcp`, `import`, `auto`, `undo`). |
 | `controllers.PermissionManage` | `golem15.seo.manage`. |
-| `models.Entry`, `models.Rule`, `models.Redirect`, `models.NotFound`, `models.Robots`, `models.Settings`, `models.TypeSetting`, `models.Revision` | The plugin's models. |
+| `models.Entry`, `models.Rule`, `models.Redirect`, `models.NotFound`, `models.Robots`, `models.Settings`, `models.TypeSetting`, `models.Revision`, `models.Page`, `models.Autolink`, `models.Crawl`, `models.CrawlPage`, `models.CrawlLink`, `models.Issue`, `models.IssueIgnore` | The plugin's models. |
 
 ## Configuration
 
@@ -209,7 +376,13 @@ The element bundle (`assets/dist/seo.js` and `seo.css`) is built from `assets/sr
 | `golem15.seo.unprefixed_locale` | `""` | The locale whose public URLs carry no locale prefix; empty leaves the choice to each content type. |
 | `golem15.seo.notfound_token` | `""` | The bearer token of `POST /_seo/api/v1/notfound`; empty turns the endpoint off. |
 | `golem15.seo.notfound_max_rows` | `10000` | The most addresses the 404 log holds. |
-| `golem15.seo.robots_sitemaps` | `["/sitemap.xml"]` | The sitemap addresses robots.txt names, one `Sitemap:` line each, absolute on the site URL. |
+| `golem15.seo.robots_sitemaps` | unset | The sitemap addresses robots.txt names, one `Sitemap:` line each, absolute on the site URL. Unset, robots.txt names the sitemap plugin's index. |
+| `golem15.seo.og_locales` | unset | Locale code to `og:locale` value, for example `{pl: pl_PL, en: en_GB}`. Unlisted codes use `pl_PL`, `en_US`, `de_DE`, else the language with its region. |
+| `golem15.seo.search_url` | unset | The site search address with `{search_term_string}` (`/szukaj?q={search_term_string}`); set, the WebSite node carries a SearchAction. |
+| `golem15.seo.route_report_token` | `""` | The bearer token of `POST /_seo/api/v1/registry/routes`; empty uses `notfound_token`, and both empty turn the route report off. |
+| `golem15.seo.crawl_trusted` | `false` | Lets the crawler fetch plain http and private addresses (a staging or local site), still only on the site host and its `www` twin. Config only. |
+| `golem15.seo.crawl_chrome_path` | `""` | The absolute path of an operator-installed Chrome or Chromium binary for the rendered comparison; empty skips it. Config only. |
+| `golem15.seo.crawl_render_max` | `200` | The pages rendered per crawl, the first in crawl order. |
 
 The non-production lock reads the application environment (`SUMMER_ENV`); it has no key of its own. The address rules (host, trailing slash, lowercase) are settings, on the Addresses tab.
 
@@ -218,12 +391,16 @@ The non-production lock reads the application environment (`SUMMER_ENV`); it has
 | Command | Description |
 |---------|-------------|
 | `seo:generate-entries [--type <key>]` | Queues the entry generation job for every content type, or for one. A worker runs it. |
+| `seo:crawl [--scheduled]` | Queues a crawl of the site. `--scheduled` (what the nightly schedule entry at 02:30 runs) follows the **Automatic crawl** setting: daily, weekly on Mondays in `app.timezone`, nothing while off or while a crawl runs. |
 
 ## Dependencies
 
-- SummerCMS v0.5.1 modules: `pact`, `party`, `backpack`, `cabana`, `signpost`, `lagoon`, `conga`, `surf`, `wire`, `bouncer`, `phrasebook`, `bonfire`, `compass`, `diary`.
+- SummerCMS v0.6.1 modules: `pact`, `party`, `backpack`, `cabana`, `signpost`, `lagoon`, `conga`, `cooler` (the site-cache purge), `surf`, `wire`, `bouncer`, `phrasebook`, `bonfire`, `compass`, `diary`.
+- `git.golem15.com/golem15/sm-sitemap-plugin` v0.2.0, its `contract` package only (the Filter the plugin publishes, and the Listing and Rebuilder it reads); the plugin requires `golem15.sitemap`.
 - `gorm.io/gorm`, `github.com/go-gormigrate/gormigrate/v2`, `gocloud.dev/blob` (the media bucket for social images), `github.com/goccy/go-yaml` (the per-type settings tabs and robots form documents).
 - `github.com/jimsmart/grobotstxt` v1.0.3, the robots.txt matcher (decision note `.planning/notes/2026-10-27-robots-txt-grobotstxt.md` in the framework repository).
+- `golang.org/x/net/html`, the crawler's HTML parser (already required by the framework).
+- `github.com/chromedp/chromedp` v0.21.0 (with `github.com/chromedp/cdproto`), the optional rendered comparison's DevTools client; it drives an operator-installed Chrome and needs no cgo or Node (decision note `.planning/notes/2026-10-28-chromedp-rendered-crawl.md` in the framework repository).
 - Element toolchain (development only, exact pins): `typescript`, `vite`, `vitest`, `happy-dom`. The bundle has no runtime dependencies.
 
 ## Testing
@@ -237,4 +414,4 @@ npm ci && npm run typecheck && npm test && bash scripts/check-dist.sh
 
 The element tests (`tests/elements`, vitest on happy-dom) drive the widget elements through a fake of the admin's widget host (`tests/elements/host.ts`), including a host that drops an action.
 
-The smoke tests boot the plugin with fake content plugins (`acme.pages`, and `acme.blog` with a category segment for strict URLs) and exercise the SEO tab, the settings tabs, the resolver precedence, history, generation, the URL rule form, redirects (engine, loops and chains, address changes from the admin and from events), the 404 log, normalisation and strict URLs, robots.txt and verification files through the admin and public APIs, and the management API as an MCP client (`TestMCPSetURLSeoTracer`, `TestManagementAPI`, `TestIssues`), batches, undo and CSV (`TestBatch`, `TestUndo`, `TestCSVTransfer`, and the `mcp` and `import` cases of `TestLoopRefused`). `TestEveryAdminOperationHasTool` and `TestMCPReadToolsAreGET` run without Docker.
+The smoke tests boot the plugin with fake content plugins (`acme.pages`, and `acme.blog` with a category segment for strict URLs) and exercise the SEO tab, the settings tabs, the resolver precedence, history, generation, the URL rule form, redirects (engine, loops and chains, address changes from the admin and from events), the 404 log, normalisation and strict URLs, robots.txt and verification files through the admin and public APIs, and the management API as an MCP client (`TestMCPSetURLSeoTracer`, `TestManagementAPI`, `TestIssues`), batches, undo and CSV (`TestBatch`, `TestUndo`, `TestCSVTransfer`, and the `mcp` and `import` cases of `TestLoopRefused`). The sitemap, the registry and the site-level endpoints are covered by `TestSitemapFilterTracer`, `TestFilterReasons`, `TestNoindexPublishedConflict`, `TestSitemapBlockedIssue`, `TestRegistry`, `TestRouteReport`, `TestNoindexPathsTracer`, `TestSiteEndpoint`, `TestLLMSTxt`, `TestRobotsSitemapLine`, `TestSettingsLinks` and `TestPurgeAfterWrite`; the fixture boots the sitemap plugin next to the SEO plugin. The site audit is covered by `TestCrawlerTracer`, `TestCrawlerChecks` (a planted site with one page per kind and a clean control page, served to the crawler through `fetchguard.WithTransport`), `TestIssueMerge`, `TestStagingCrawl`, `TestRenderSkipped`, `TestCrawlAdmin`, `TestCrawlSchedule` and `TestCrawlTools`, with the fake content plugin `acme.crawl` naming fixing tools. Auto-links are covered by `TestAutolinksTracer` (the phrases of a page's locale in the resolve answer, never the page's own, `autolink_max` 0) and `TestAutolinks` (the screen, its refusals, Add focus keyphrases, the API and MCP); `TestLinkSuggestions` crawls a planted site and checks E10's counts and suggestions through the API and the widget action. `TestChromeRenderer` renders a page in a real headless Chrome (it runs when `SEO_TEST_CHROME` names a binary or `chromium`, `chromium-browser` or `google-chrome` is on `PATH`, and is skipped otherwise) and `TestRendererGuards` checks the refused paths and private addresses. `TestEveryAdminOperationHasTool`, `TestMCPReadToolsAreGET`, `TestIssueKindLabels` and the static parts of `TestCrawlTools` and `TestCrawlSchedule` run without Docker.

admin_navigation.go

@@ -38,6 +38,35 @@ func (*Plugin) Navigation() []pact.NavigationItem {
 				Permissions: []string{controllers.PermissionManage},
 				Controller:  controllers.RulesControllerID,
 			},
+			{
+				Code:        "pages",
+				Label:       "golem15.seo::lang.nav.pages",
+				Icon:        "files",
+				Permissions: []string{controllers.PermissionManage},
+				Controller:  controllers.PagesControllerID,
+			},
+			// Auto-links, controller golem15.seo.autolinks (UI-SPEC C15).
+			{
+				Code:        "autolinks",
+				Label:       "golem15.seo::lang.nav.autolinks",
+				Icon:        "wand-sparkles",
+				Permissions: []string{controllers.PermissionManage},
+				Controller:  controllers.AutolinksControllerID,
+			},
+			{
+				Code:        "audit",
+				Label:       "golem15.seo::lang.nav.audit",
+				Icon:        "gauge",
+				Permissions: []string{controllers.PermissionManage},
+				Controller:  controllers.IssuesControllerID,
+			},
+			{
+				Code:        "crawls",
+				Label:       "golem15.seo::lang.nav.crawls",
+				Icon:        "globe",
+				Permissions: []string{controllers.PermissionManage},
+				Controller:  controllers.CrawlsControllerID,
+			},
 			{
 				Code:        "robots",
 				Label:       "golem15.seo::lang.nav.robots",

assets/src/elements/graph.ts

@@ -0,0 +1,240 @@
+import { ActionQueue } from './actions'
+import { el, fmt, icon, jsonAttr, labelsFor, uiLocaleOf, type Labels } from './labels'
+
+export const GRAPH_TAG = 'golem15-seo-graph'
+
+/** One missing property of a node. */
+export interface GraphFinding {
+  level: 'required' | 'recommended'
+  property: string
+}
+
+/** A node of the graph with its findings. */
+export interface GraphNode {
+  id: string
+  type: string
+  findings: GraphFinding[]
+  rules: boolean
+}
+
+/** The seo_graph action's answer (UI-SPEC C18). */
+export interface GraphData {
+  locale: string
+  url: string | null
+  graph: Record<string, unknown> | null
+  nodes: GraphNode[]
+}
+
+interface Payload {
+  op: 'graph'
+  locale: string
+}
+
+/** The Rich Results Test address of a page. */
+export function richResultsURL(url: string): string {
+  return 'https://search.google.com/test/rich-results?url=' + encodeURIComponent(url)
+}
+
+/** The fragment of an @id (#article), the whole id when it has none. */
+function fragment(id: string): string {
+  const i = id.indexOf('#')
+  return i >= 0 ? id.slice(i) : id
+}
+
+/**
+ * E9 golem15-seo-graph (UI-SPEC C18, D-46): the effective @graph of the
+ * saved page with the unsaved seo_schema value, every node with Google's
+ * required and recommended findings, the JSON and the Rich Results Test
+ * link. It asks through summer-action only ({op: graph, locale}), on
+ * connect, 800 ms after the last fill change and on the ML locale switch.
+ */
+export class SeoGraphElement extends HTMLElement {
+  static observedAttributes = ['fill-values', 'busy', 'state', 'ui-locale', 'disabled', 'label']
+
+  private data: GraphData | null = null
+  private failed = false
+  private pending = false
+  private showJSON = false
+  private shownLocale = ''
+  private lastFill = ''
+  private timer: ReturnType<typeof setTimeout> | undefined
+  private connected = false
+  private readonly actions = new ActionQueue<Payload>(this, () => this.fail())
+
+  private readonly onResult = (ev: Event): void => {
+    const detail = (ev as CustomEvent).detail as { data?: unknown } | undefined
+    this.actions.settle()
+    const data = detail?.data as GraphData | undefined
+    if (data && Array.isArray(data.nodes)) {
+      this.data = data
+      this.failed = false
+    }
+    this.pending = this.actions.current !== null
+    this.render()
+  }
+
+  private readonly onLocale = (ev: Event): void => {
+    const locale = (ev as CustomEvent).detail
+    if (typeof locale === 'string' && locale !== '' && locale !== this.shownLocale) {
+      this.shownLocale = locale
+      this.check()
+    }
+  }
+
+  connectedCallback(): void {
+    this.connected = true
+    this.lastFill = this.getAttribute('fill-values') ?? ''
+    this.addEventListener('summer-result', this.onResult)
+    window.addEventListener('summer:ml-locale', this.onLocale)
+    this.render()
+    queueMicrotask(() => {
+      if (this.connected) {
+        this.check()
+      }
+    })
+  }
+
+  disconnectedCallback(): void {
+    this.connected = false
+    this.removeEventListener('summer-result', this.onResult)
+    window.removeEventListener('summer:ml-locale', this.onLocale)
+    clearTimeout(this.timer)
+    this.actions.reset()
+    this.pending = false
+  }
+
+  attributeChangedCallback(name: string, _old: string | null, value: string | null): void {
+    if (!this.connected) {
+      return
+    }
+    if (name === 'busy' && value === null) {
+      this.actions.flush()
+      return
+    }
+    if (name === 'state' && value === 'error') {
+      if (this.actions.settle()) {
+        this.fail()
+      }
+      return
+    }
+    if (name === 'fill-values') {
+      const fill = value ?? ''
+      if (fill !== this.lastFill) {
+        this.lastFill = fill
+        clearTimeout(this.timer)
+        this.timer = setTimeout(() => this.check(), 800)
+      }
+      return
+    }
+    this.render()
+  }
+
+  private labels(): Labels {
+    return labelsFor(uiLocaleOf(this))
+  }
+
+  /** Asks for the graph of the shown locale (one queued request at most). */
+  check(): void {
+    clearTimeout(this.timer)
+    this.pending = true
+    this.actions.request({ op: 'graph', locale: this.shownLocale })
+    this.render()
+  }
+
+  private fail(): void {
+    this.failed = true
+    this.pending = this.actions.current !== null
+    this.render()
+  }
+
+  private render(): void {
+    if (!this.connected) {
+      return
+    }
+    const labels = this.labels()
+    const sections: HTMLElement[] = []
+    this.setAttribute('aria-busy', String(this.pending))
+    const data = this.data
+
+    const header = el('p', 'seo-graph__meta')
+    if (data) {
+      header.textContent = data.url ? fmt(labels.forLocale, { locale: (data.locale || '').toUpperCase(), url: data.url }) : labels.graphNoUrl
+    }
+    if (this.pending) {
+      const line = el('p', 'seo-graph__meta seo-graph__checking', labels.graphChecking)
+      line.setAttribute('role', 'status')
+      sections.push(line)
+    }
+    if (data) {
+      sections.push(header)
+      const list = el('div', 'seo-graph__nodes')
+      let problems = 0
+      for (const node of data.nodes) {
+        const row = el('div', 'seo-graph__node')
+        const head = el('div', 'seo-graph__node-head')
+        head.append(el('span', 'seo-graph__pill', node.type || '—'), el('code', 'seo-graph__id', fragment(node.id || '')))
+        row.append(head)
+        if (!node.rules) {
+          row.append(el('p', 'seo-graph__meta', fmt(labels.noRules, { type: node.type })))
+        }
+        for (const f of node.findings ?? []) {
+          problems++
+          const required = f.level === 'required'
+          const line = el('p', 'seo-graph__finding seo-graph__finding--' + (required ? 'required' : 'recommended'))
+          line.append(icon(required ? 'circle-alert' : 'triangle-alert'), el('span', '', fmt(required ? labels.required : labels.recommended, { property: f.property, type: node.type })))
+          row.append(line)
+        }
+        list.append(row)
+      }
+      sections.push(list)
+      if (problems === 0 && data.nodes.length > 0) {
+        const ok = el('p', 'seo-graph__ok')
+        ok.append(icon('circle-check'), el('span', '', labels.allGood))
+        sections.push(ok)
+      }
+      if (data.graph) {
+        const toggle = el('button', 'seo-graph__disclosure')
+        toggle.type = 'button'
+        toggle.setAttribute('aria-expanded', String(this.showJSON))
+        toggle.append(el('span', '', this.showJSON ? labels.hideJson : labels.showJson), icon('chevron-down'))
+        toggle.addEventListener('click', () => {
+          this.showJSON = !this.showJSON
+          this.render()
+        })
+        sections.push(toggle)
+        if (this.showJSON) {
+          sections.push(el('pre', 'seo-graph__json', JSON.stringify(data.graph, null, 2)))
+        }
+      }
+      const notes = el('div', 'seo-graph__links')
+      if (data.url) {
+        const link = el('a', 'seo-graph__link')
+        link.href = richResultsURL(data.url)
+        link.target = '_blank'
+        link.rel = 'noopener'
+        link.append(el('span', '', labels.richResults), icon('external-link'))
+        notes.append(link)
+      }
+      notes.append(el('p', 'seo-graph__meta', labels.graphUnsaved), el('p', 'seo-graph__meta', labels.layerNodes))
+      sections.push(notes)
+    }
+    if (this.failed) {
+      const line = el('p', 'seo-graph__error')
+      line.setAttribute('role', 'status')
+      line.append(icon('circle-alert'), el('span', '', labels.graphFailed))
+      sections.push(line)
+    }
+    if (!this.hasAttribute('disabled')) {
+      const button = el('button', 'seo-graph__button', this.getAttribute('label') || labels.graphAction)
+      button.type = 'button'
+      button.addEventListener('click', () => this.check())
+      sections.push(button)
+    }
+    this.replaceChildren(...sections)
+  }
+
+  /** The fill snapshot (seo_schema), for tests. */
+  fill(): Record<string, unknown> {
+    return jsonAttr<Record<string, unknown>>(this, 'fill-values', {})
+  }
+}

assets/src/elements/index.ts

@@ -1,8 +1,12 @@
 import './styles.css'
+import { GRAPH_TAG, SeoGraphElement } from './graph'
 import { IMAGE_TAG, SeoImageElement } from './image'
+import { LINKS_TAG, SeoLinksElement } from './links'
+import { PARENT_TAG, SeoParentElement } from './parent'
 import { ROBOTS_TAG, SeoRobotsElement } from './robots'
 import { ROBOTSTESTER_TAG, SeoRobotsTesterElement } from './robotstester'
 import { ROBOTSTXT_TAG, SeoRobotsTxtElement } from './robotstxt'
+import { SCHEMA_TAG, SeoSchemaElement } from './schema'
 import { SERP_TAG, SeoSerpElement } from './serp'
 import { SeoTextElement, TEXT_TAG } from './text'
 import { SeoTransferElement, TRANSFER_TAG } from './transfer'
@@ -18,6 +22,10 @@ const elements: Array<[string, CustomElementConstructor]> = [
   [ROBOTSTXT_TAG, SeoRobotsTxtElement],
   [ROBOTSTESTER_TAG, SeoRobotsTesterElement],
   [TRANSFER_TAG, SeoTransferElement],
+  ['golem15-seo-schema', SeoSchemaElement],
+  ['golem15-seo-graph', SeoGraphElement],
+  ['golem15-seo-parent', SeoParentElement],
+  ['golem15-seo-links', SeoLinksElement],
 ]
 for (const [tag, ctor] of elements) {
   if (!customElements.get(tag)) {
@@ -26,7 +34,15 @@ for (const [tag, ctor] of elements) {
 }
 
 export {
+  GRAPH_TAG,
   IMAGE_TAG,
+  LINKS_TAG,
+  PARENT_TAG,
+  SCHEMA_TAG,
+  SeoGraphElement,
+  SeoLinksElement,
+  SeoParentElement,
+  SeoSchemaElement,
   ROBOTS_TAG,
   ROBOTSTESTER_TAG,
   ROBOTSTXT_TAG,

assets/src/elements/labels.ts

@@ -1,5 +1,6 @@
 // Element copy of the SEO field elements (UI-SPEC copywriting E1 to E6,
-// plus E7 for the social image), chosen by the ui-locale attribute. Plural
+// E7 for the social image, E8 and E9 for structured data, E10 for internal
+// links), chosen by the ui-locale attribute. Plural
 // texts are CLDR maps; placeholders use :name.
 
 /** A CLDR plural map: one/other in English, one/few/many/other in Polish. */
@@ -134,6 +135,74 @@ export interface Labels {
   readFailed: string
   openHistory: string
   applyCount: Plural
+  // E8 golem15-seo-schema
+  schemaType: string
+  typeDefault: string
+  tBlogPosting: string
+  tArticle: string
+  tNewsArticle: string
+  tWebPage: string
+  tAboutPage: string
+  tContactPage: string
+  tService: string
+  tCollectionPage: string
+  tProfilePage: string
+  noteArticle: string
+  notePage: string
+  noteContact: string
+  noteService: string
+  noteCollection: string
+  noteProfile: string
+  serviceType: string
+  serviceTypePh: string
+  areaServed: string
+  areaServedHint: string
+  price: string
+  priceHint: string
+  currency: string
+  currencyHint: string
+  priceInvalid: string
+  currencyInvalid: string
+  custom: string
+  customSet: string
+  customHint: string
+  jsonValid: Plural
+  jsonError: string
+  jsonNotObject: string
+  jsonTooLarge: string
+  // E9 golem15-seo-graph
+  forLocale: string
+  graphChecking: string
+  allGood: string
+  required: string
+  recommended: string
+  noRules: string
+  showJson: string
+  hideJson: string
+  richResults: string
+  graphNoUrl: string
+  graphUnsaved: string
+  layerNodes: string
+  graphAction: string
+  graphFailed: string
+  // E10 golem15-seo-links
+  linksInbound: string
+  linksOutbound: string
+  linksCrawled: string
+  linksNotCrawled: string
+  linksOrphanTitle: string
+  linksOrphanText: string
+  linksOut: string
+  linksIn: string
+  linksPhrase: string
+  linksCopy: string
+  linksCopied: string
+  linksNone: string
+  linksNoRecord: string
+  linksChecking: string
+  linksFailed: string
+  // golem15-seo-parent
+  primaryDefault: string
 }
 
 const en: Labels = {
@@ -253,6 +322,70 @@ const en: Labels = {
   readFailed: 'The file could not be read.',
   openHistory: 'Open History',
   applyCount: { one: 'Apply :count change', other: 'Apply :count changes' },
+  schemaType: 'Schema type',
+  typeDefault: 'Default (:type)',
+  tBlogPosting: 'BlogPosting (blog post)',
+  tArticle: 'Article (article)',
+  tNewsArticle: 'NewsArticle (news)',
+  tWebPage: 'WebPage (page)',
+  tAboutPage: 'AboutPage (about us)',
+  tContactPage: 'ContactPage (contact)',
+  tService: 'Service (service or offer)',
+  tCollectionPage: 'CollectionPage (listing)',
+  tProfilePage: 'ProfilePage (profile)',
+  noteArticle: 'Author, dates, image, word count and section come from the content.',
+  notePage: 'Name, description and breadcrumbs come from the SEO tab.',
+  noteContact: 'Address and opening hours come from the SEO settings (Publisher, Local business).',
+  noteService: 'The provider is the publisher from the SEO settings.',
+  noteCollection: 'Lists the items shown on the page.',
+  noteProfile: 'Name, image and profiles come from the author.',
+  serviceType: 'Kind of service',
+  serviceTypePh: 'e.g. Mushroom workshops',
+  areaServed: 'Area served',
+  areaServedHint: 'Cities or regions, separated by commas.',
+  price: 'Price',
+  priceHint: 'Leave empty when the price is set individually.',
+  currency: 'Currency',
+  currencyHint: 'ISO code, e.g. PLN.',
+  priceInvalid: 'Enter a price such as 150 or 149.99.',
+  currencyInvalid: 'Use a three-letter currency code, e.g. PLN.',
+  custom: 'Custom JSON-LD (advanced)',
+  customSet: 'Set',
+  customHint: 'An object or a list of objects. A node with the @id of a generated node is merged into it, and your values win; other nodes are added to the graph. Paste only the JSON, without a script tag.',
+  jsonValid: { one: 'Valid JSON: :count node', other: 'Valid JSON: :count nodes' },
+  jsonError: 'Line :line, column :column: :message',
+  jsonNotObject: 'Use an object or a list of objects.',
+  jsonTooLarge: 'Over 16 KB: shorten the custom JSON-LD.',
+  forLocale: 'JSON-LD for :locale — :url',
+  graphChecking: 'Building the graph…',
+  allGood: 'No problems: every node has its required and recommended properties.',
+  required: 'Missing required property “:property” (:type)',
+  recommended: 'Missing recommended property “:property” (:type)',
+  noRules: ':type has no rich-result rules at Google; it is checked as valid schema.org only.',
+  showJson: 'Show JSON-LD',
+  hideJson: 'Hide JSON-LD',
+  richResults: "Check in Google's Rich Results Test",
+  graphNoUrl: 'The Rich Results Test link appears after the first save, when the page has an address.',
+  graphUnsaved: 'Shows the saved page with your unsaved structured data changes.',
+  layerNodes: 'The site may add nodes of its own, such as FAQPage. The site audit checks the rendered result.',
+  graphAction: 'Check structured data',
+  graphFailed: 'The structured data could not be built. Try again.',
+  linksInbound: 'Links to this page',
+  linksOutbound: 'Links from this page',
+  linksCrawled: 'From the crawl of :time.',
+  linksNotCrawled: 'Link counts appear after the first crawl (SEO › Site audit).',
+  linksOrphanTitle: 'No page links here',
+  linksOrphanText: 'Search engines and visitors may not find this page. Link to it from one of the pages below.',
+  linksOut: 'Pages you could link to from this text',
+  linksIn: 'Pages that could link here',
+  linksPhrase: '“:phrase” → :title',
+  linksCopy: 'Copy address',
+  linksCopied: 'Address copied.',
+  linksNone: 'No suggestions. They come from titles and focus keyphrases found in the saved text.',
+  linksNoRecord: 'Suggestions appear after the first save.',
+  linksChecking: 'Loading the internal links…',
+  linksFailed: 'The internal links could not be loaded. Try again.',
+  primaryDefault: 'Default (first category)',
 }
 
 const pl: Labels = {
@@ -382,6 +515,70 @@ const pl: Labels = {
   readFailed: 'Nie udało się odczytać pliku.',
   openHistory: 'Otwórz Historię zmian',
   applyCount: { one: 'Zastosuj :count zmianę', few: 'Zastosuj :count zmiany', many: 'Zastosuj :count zmian', other: 'Zastosuj :count zmiany' },
+  schemaType: 'Typ schema',
+  typeDefault: 'Domyślny (:type)',
+  tBlogPosting: 'BlogPosting (wpis na blogu)',
+  tArticle: 'Article (artykuł)',
+  tNewsArticle: 'NewsArticle (wiadomość)',
+  tWebPage: 'WebPage (strona)',
+  tAboutPage: 'AboutPage (o nas)',
+  tContactPage: 'ContactPage (kontakt)',
+  tService: 'Service (usługa lub oferta)',
+  tCollectionPage: 'CollectionPage (lista)',
+  tProfilePage: 'ProfilePage (profil)',
+  noteArticle: 'Autor, daty, obraz, liczba słów i sekcja pochodzą z treści.',
+  notePage: 'Nazwa, opis i ścieżka nawigacji pochodzą z zakładki SEO.',
+  noteContact: 'Adres i godziny otwarcia pochodzą z ustawień SEO (Wydawca, Firma lokalna).',
+  noteService: 'Usługodawcą jest wydawca z ustawień SEO.',
+  noteCollection: 'Wymienia elementy pokazane na stronie.',
+  noteProfile: 'Imię i nazwisko, zdjęcie i profile pochodzą od autora.',
+  serviceType: 'Rodzaj usługi',
+  serviceTypePh: 'np. warsztaty grzybowe',
+  areaServed: 'Obszar działania',
+  areaServedHint: 'Miasta lub regiony oddzielone przecinkami.',
+  price: 'Cena',
+  priceHint: 'Zostaw puste, gdy cena jest ustalana indywidualnie.',
+  currency: 'Waluta',
+  currencyHint: 'Kod ISO, np. PLN.',
+  priceInvalid: 'Wpisz cenę, np. 150 lub 149,99.',
+  currencyInvalid: 'Użyj trzyliterowego kodu waluty, np. PLN.',
+  custom: 'Własny JSON-LD (zaawansowane)',
+  customSet: 'Ustawiony',
+  customHint: 'Obiekt lub lista obiektów. Węzeł z @id wygenerowanego węzła zostaje z nim scalony, a Twoje wartości mają pierwszeństwo; pozostałe węzły są dodawane do grafu. Wklej sam JSON, bez znacznika script.',
+  jsonValid: { one: 'Poprawny JSON: :count węzeł', few: 'Poprawny JSON: :count węzły', many: 'Poprawny JSON: :count węzłów', other: 'Poprawny JSON: :count węzła' },
+  jsonError: 'Wiersz :line, kolumna :column: :message',
+  jsonNotObject: 'Użyj obiektu lub listy obiektów.',
+  jsonTooLarge: 'Ponad 16 KB: skróć własny JSON-LD.',
+  forLocale: 'JSON-LD dla :locale — :url',
+  graphChecking: 'Budowanie grafu…',
+  allGood: 'Brak problemów: każdy węzeł ma wymagane i zalecane właściwości.',
+  required: 'Brak wymaganej właściwości „:property” (:type)',
+  recommended: 'Brak zalecanej właściwości „:property” (:type)',
+  noRules: ':type nie ma reguł wyników z elementami rozszerzonymi w Google; sprawdzana jest tylko poprawność schema.org.',
+  showJson: 'Pokaż JSON-LD',
+  hideJson: 'Ukryj JSON-LD',
+  richResults: 'Sprawdź w Teście wyników z elementami rozszerzonymi Google',
+  graphNoUrl: 'Link do testu pojawi się po pierwszym zapisie, gdy strona będzie miała adres.',
+  graphUnsaved: 'Pokazuje zapisaną stronę z niezapisanymi zmianami danych strukturalnych.',
+  layerNodes: 'Witryna może dodać własne węzły, np. FAQPage. Audyt witryny sprawdza wynik na stronie.',
+  graphAction: 'Sprawdź dane strukturalne',
+  graphFailed: 'Nie udało się zbudować danych strukturalnych. Spróbuj ponownie.',
+  linksInbound: 'Linki do tej strony',
+  linksOutbound: 'Linki z tej strony',
+  linksCrawled: 'Według skanowania z :time.',
+  linksNotCrawled: 'Liczby linków pojawią się po pierwszym skanowaniu (SEO › Audyt witryny).',
+  linksOrphanTitle: 'Żadna strona tu nie linkuje',
+  linksOrphanText: 'Wyszukiwarki i odwiedzający mogą nie znaleźć tej strony. Dodaj do niej link z jednej ze stron poniżej.',
+  linksOut: 'Strony, do których możesz dodać link w tym tekście',
+  linksIn: 'Strony, które mogą tu linkować',
+  linksPhrase: '„:phrase” → :title',
+  linksCopy: 'Kopiuj adres',
+  linksCopied: 'Skopiowano adres.',
+  linksNone: 'Brak propozycji. Powstają z tytułów i fraz kluczowych znalezionych w zapisanym tekście.',
+  linksNoRecord: 'Propozycje pojawią się po pierwszym zapisie.',
+  linksChecking: 'Wczytywanie linków wewnętrznych…',
+  linksFailed: 'Nie udało się wczytać linków wewnętrznych. Spróbuj ponownie.',
+  primaryDefault: 'Domyślna (pierwsza kategoria)',
 }
 
 const dictionaries: Record<string, Labels> = { en, pl }
@@ -441,6 +638,7 @@ const ICONS: Record<string, string[]> = {
   'circle-check': ['M12 22a10 10 0 1 0 0-20 10 10 0 0 0 0 20z', 'm9 12 2 2 4-4'],
   'circle-alert': ['M12 22a10 10 0 1 0 0-20 10 10 0 0 0 0 20z', 'M12 8v4', 'M12 16h.01'],
   'chevron-down': ['m6 9 6 6 6-6'],
+  'external-link': ['M15 3h6v6', 'M10 14 21 3', 'M18 13v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6'],
 }
 
 /** An aria-hidden inline SVG icon of size px. */

assets/src/elements/links.ts

@@ -0,0 +1,261 @@
+import { ActionQueue } from './actions'
+import { el, fmt, icon, labelsFor, num, uiLocaleOf, type Labels } from './labels'
+
+export const LINKS_TAG = 'golem15-seo-links'
+
+/** A page this text could link to. */
+export interface LinkOut {
+  phrase: string
+  title: string
+  url: string
+}
+
+/** A page that could link here. */
+export interface LinkIn {
+  phrase: string
+  title: string
+  adminUrl: string
+}
+
+/** The seo_links action's answer (UI-SPEC C19). */
+export interface LinksData {
+  crawledAt: string | null
+  inbound: number | null
+  outbound: number | null
+  orphan: boolean
+  out: LinkOut[]
+  in: LinkIn[]
+}
+
+interface Payload {
+  op: 'links'
+  locale: string
+}
+
+/** Whether the page can write to the clipboard (a secure context). */
+function canCopy(): boolean {
+  return typeof navigator !== 'undefined' && !!navigator.clipboard && typeof navigator.clipboard.writeText === 'function'
+}
+
+/** A crawl time in the ui-locale. */
+export function crawlTime(value: string, locale: string): string {
+  const d = new Date(value)
+  if (Number.isNaN(d.getTime())) {
+    return value
+  }
+  try {
+    return new Intl.DateTimeFormat(locale, { dateStyle: 'medium', timeStyle: 'short' }).format(d)
+  } catch {
+    return d.toISOString()
+  }
+}
+
+/**
+ * E10 golem15-seo-links (UI-SPEC C19, D-82): the page's inbound and
+ * outbound link counts from the last crawl, the orphan warning, up to 10
+ * pages this text could link to (with Copy address) and 10 pages that
+ * could link here. It asks through summer-action only ({op: links,
+ * locale}), on connect and on the ML locale switch, and changes nothing
+ * itself.
+ */
+export class SeoLinksElement extends HTMLElement {
+  static observedAttributes = ['busy', 'state', 'ui-locale', 'disabled']
+
+  private data: LinksData | null = null
+  private failed = false
+  private pending = false
+  private shownLocale = ''
+  private copied = ''
+  private copiedTimer: ReturnType<typeof setTimeout> | undefined
+  private connected = false
+  private readonly actions = new ActionQueue<Payload>(this, () => this.fail())
+
+  private readonly onResult = (ev: Event): void => {
+    const detail = (ev as CustomEvent).detail as { data?: unknown } | undefined
+    this.actions.settle()
+    const data = detail?.data as LinksData | undefined
+    if (data && Array.isArray(data.out) && Array.isArray(data.in)) {
+      this.data = data
+      this.failed = false
+    }
+    this.pending = this.actions.current !== null
+    this.render()
+  }
+
+  private readonly onLocale = (ev: Event): void => {
+    const locale = (ev as CustomEvent).detail
+    if (typeof locale === 'string' && locale !== '' && locale !== this.shownLocale) {
+      this.shownLocale = locale
+      this.load()
+    }
+  }
+
+  connectedCallback(): void {
+    this.connected = true
+    this.addEventListener('summer-result', this.onResult)
+    window.addEventListener('summer:ml-locale', this.onLocale)
+    this.render()
+    queueMicrotask(() => {
+      if (this.connected) {
+        this.load()
+      }
+    })
+  }
+
+  disconnectedCallback(): void {
+    this.connected = false
+    this.removeEventListener('summer-result', this.onResult)
+    window.removeEventListener('summer:ml-locale', this.onLocale)
+    clearTimeout(this.copiedTimer)
+    this.actions.reset()
+    this.pending = false
+  }
+
+  attributeChangedCallback(name: string, _old: string | null, value: string | null): void {
+    if (!this.connected) {
+      return
+    }
+    if (name === 'busy' && value === null) {
+      this.actions.flush()
+      return
+    }
+    if (name === 'state' && value === 'error') {
+      if (this.actions.settle()) {
+        this.fail()
+      }
+      return
+    }
+    this.render()
+  }
+
+  private labels(): Labels {
+    return labelsFor(uiLocaleOf(this))
+  }
+
+  /** Asks for the links of the shown locale (one queued request at most). */
+  load(): void {
+    this.pending = true
+    this.actions.request({ op: 'links', locale: this.shownLocale })
+    this.render()
+  }
+
+  private fail(): void {
+    this.failed = true
+    this.pending = this.actions.current !== null
+    this.render()
+  }
+
+  /** Writes url to the clipboard and says so for 2 s. */
+  private copy(url: string): void {
+    if (!canCopy()) {
+      return
+    }
+    void navigator.clipboard.writeText(url).then(() => {
+      this.copied = url
+      clearTimeout(this.copiedTimer)
+      this.copiedTimer = setTimeout(() => {
+        this.copied = ''
+        this.render()
+      }, 2000)
+      this.render()
+    }, () => undefined)
+  }
+
+  /** The suggestion line “:phrase” → :title with the title as a link (or text). */
+  private line(labels: Labels, phrase: string, title: string, href: string): HTMLElement {
+    const line = el('span', 'seo-links__line')
+    const [before, after = ''] = labels.linksPhrase.split(':title')
+    line.append(fmt(before ?? '', { phrase }))
+    if (href) {
+      const a = el('a', 'seo-links__title', title)
+      a.href = href
+      a.target = '_blank'
+      a.rel = 'noopener'
+      line.append(a)
+    } else {
+      line.append(el('span', 'seo-links__title', title))
+    }
+    line.append(fmt(after, { phrase }))
+    return line
+  }
+
+  private render(): void {
+    if (!this.connected) {
+      return
+    }
+    const labels = this.labels()
+    const locale = uiLocaleOf(this)
+    const sections: HTMLElement[] = []
+    this.setAttribute('aria-busy', String(this.pending))
+    if (this.pending) {
+      const line = el('p', 'seo-links__meta seo-links__checking', labels.linksChecking)
+      line.setAttribute('role', 'status')
+      sections.push(line)
+    }
+    const data = this.data
+    if (data) {
+      const counts = el('div', 'seo-links__counts')
+      if (data.crawledAt !== null && data.inbound !== null && data.outbound !== null) {
+        const stats = el('div', 'summer-stats')
+        for (const [value, label] of [
+          [data.inbound, labels.linksInbound],
+          [data.outbound, labels.linksOutbound],
+        ] as Array<[number, string]>) {
+          const stat = el('div', 'summer-stat')
+          stat.append(el('span', 'summer-stat__value', num(value, locale)), el('span', 'summer-stat__label', label))
+          stats.append(stat)
+        }
+        counts.append(stats, el('p', 'seo-links__meta', fmt(labels.linksCrawled, { time: crawlTime(data.crawledAt, locale) })))
+      } else {
+        counts.append(el('p', 'seo-links__meta seo-links__not-crawled', labels.linksNotCrawled))
+      }
+      sections.push(counts)
+      if (data.orphan) {
+        const callout = el('div', 'summer-callout summer-callout--warning seo-links__orphan')
+        callout.append(el('p', 'summer-callout__title', labels.linksOrphanTitle), el('p', 'summer-callout__text', labels.linksOrphanText))
+        sections.push(callout)
+      }
+      const copyable = canCopy()
+      const out = el('div', 'seo-links__list seo-links__list--out')
+      out.append(el('p', 'seo-links__heading', labels.linksOut))
+      for (const s of data.out.slice(0, 10)) {
+        const row = el('div', 'seo-links__row')
+        row.append(this.line(labels, s.phrase, s.title, s.url))
+        if (copyable && !this.hasAttribute('disabled')) {
+          const button = el('button', 'seo-links__copy', labels.linksCopy)
+          button.type = 'button'
+          button.addEventListener('click', () => this.copy(s.url))
+          row.append(button)
+        } else {
+          row.append(el('code', 'seo-links__url', s.url))
+        }
+        out.append(row)
+      }
+      if (data.out.length === 0) {
+        out.append(el('p', 'seo-links__meta seo-links__none', labels.linksNone))
+      }
+      sections.push(out)
+      const inbound = el('div', 'seo-links__list seo-links__list--in')
+      inbound.append(el('p', 'seo-links__heading', labels.linksIn))
+      for (const s of data.in.slice(0, 10)) {
+        const row = el('div', 'seo-links__row')
+        row.append(this.line(labels, s.phrase, s.title, s.adminUrl))
+        inbound.append(row)
+      }
+      if (data.in.length === 0) {
+        inbound.append(el('p', 'seo-links__meta seo-links__none', labels.linksNone))
+      }
+      sections.push(inbound)
+    }
+    if (this.failed) {
+      const line = el('p', 'seo-links__error')
+      line.setAttribute('role', 'status')
+      line.append(icon('circle-alert'), el('span', '', labels.linksFailed))
+      sections.push(line)
+    }
+    const status = el('p', 'seo-links__status', this.copied ? labels.linksCopied : '')
+    status.setAttribute('role', 'status')
+    sections.push(status)
+    this.replaceChildren(...sections)
+  }
+}

assets/src/elements/parent.ts

@@ -0,0 +1,97 @@
+import { el, emit, labelsFor, uid, uiLocaleOf } from './labels'
+
+export const PARENT_TAG = 'golem15-seo-parent'
+
+/** One parent the record can take as its primary category. */
+export interface ParentOption {
+  value: string
+  label: string
+}
+
+/** The loaded value: the stored choice and the record's parents. */
+export interface ParentValue {
+  selected: string
+  options: ParentOption[]
+}
+
+/** Reads the loaded value (an object) or a plain stored choice. */
+export function parseParent(v: unknown): ParentValue {
+  if (typeof v === 'string') {
+    return { selected: v, options: [] }
+  }
+  const out: ParentValue = { selected: '', options: [] }
+  if (v && typeof v === 'object') {
+    const raw = v as { selected?: unknown; options?: unknown }
+    out.selected = typeof raw.selected === 'string' ? raw.selected : ''
+    if (Array.isArray(raw.options)) {
+      for (const o of raw.options) {
+        if (o && typeof o === 'object' && typeof (o as ParentOption).value === 'string') {
+          const opt = o as ParentOption
+          out.options.push({ value: opt.value, label: typeof opt.label === 'string' && opt.label ? opt.label : opt.value })
+        }
+      }
+    }
+  }
+  return out
+}
+
+/**
+ * golem15-seo-parent (P33-23, D-86): the primary category of a record, a
+ * 44px select over the record's parents with the "first category" default.
+ * The value is the chosen <type>:<id>, "" for the default. A stored choice
+ * that is no longer one of the parents falls back to the default (P1 says
+ * so).
+ */
+export class SeoParentElement extends HTMLElement {
+  static observedAttributes = ['disabled', 'invalid', 'ui-locale']
+
+  private current: ParentValue = { selected: '', options: [] }
+
+  get value(): string {
+    return this.current.options.some((o) => o.value === this.current.selected) ? this.current.selected : ''
+  }
+
+  set value(v: unknown) {
+    const next = parseParent(v)
+    if (typeof v === 'string' && this.current.options.length > 0) {
+      next.options = this.current.options
+    }
+    this.current = next
+    this.render()
+  }
+
+  connectedCallback(): void {
+    this.render()
+  }
+
+  attributeChangedCallback(): void {
+    if (this.isConnected) {
+      this.render()
+    }
+  }
+
+  private render(): void {
+    if (!this.isConnected) {
+      return
+    }
+    const labels = labelsFor(uiLocaleOf(this))
+    const select = el('select', 'seo-parent__select')
+    select.id = uid('seo-parent')
+    select.disabled = this.hasAttribute('disabled')
+    const def = el('option', '', labels.primaryDefault)
+    def.value = ''
+    select.append(def)
+    for (const o of this.current.options) {
+      const opt = el('option', '', o.label)
+      opt.value = o.value
+      select.append(opt)
+    }
+    select.value = this.value
+    select.addEventListener('change', () => {
+      this.current.selected = select.value
+      emit(this, 'summer-change', { value: select.value })
+    })
+    this.classList.toggle('seo-parent--invalid', this.hasAttribute('invalid'))
+    this.replaceChildren(select)
+  }
+}

assets/src/elements/schema.ts

@@ -0,0 +1,384 @@
+import { el, emit, fmt, icon, jsonAttr, labelsFor, plural, uid, uiLocaleOf, type Labels } from './labels'
+
+export const SCHEMA_TAG = 'golem15-seo-schema'
+
+/** The C17 cap of custom JSON-LD, in bytes. */
+export const MAX_CUSTOM_BYTES = 16 * 1024
+
+/** The editor's Service fields. */
+export interface ServiceFields {
+  serviceType?: string
+  areaServed?: string
+  price?: string
+  currency?: string
+}
+
+/** The seoschema value, stored as a JSON string. */
+export interface SchemaValue {
+  type: string
+  fields: ServiceFields
+  custom: string
+}
+
+/** The types an editor may choose per kind (UI-SPEC C17), the default first. */
+export function typesFor(kind: string): string[] {
+  switch (kind) {
+    case 'article':
+      return ['BlogPosting', 'Article', 'NewsArticle']
+    case 'collection':
+      return ['CollectionPage']
+    case 'profile':
+      return ['ProfilePage']
+  }
+  return ['WebPage', 'AboutPage', 'ContactPage', 'Service']
+}
+
+/** Reads a stored value; anything else is the empty value. */
+export function parseValue(raw: unknown): SchemaValue {
+  const out: SchemaValue = { type: '', fields: {}, custom: '' }
+  if (typeof raw !== 'string' || raw.trim() === '') {
+    return out
+  }
+  try {
+    const v = JSON.parse(raw) as Partial<SchemaValue>
+    out.type = typeof v.type === 'string' ? v.type : ''
+    out.fields = v.fields && typeof v.fields === 'object' ? { ...v.fields } : {}
+    out.custom = typeof v.custom === 'string' ? v.custom : ''
+  } catch {
+    // An unreadable value starts empty.
+  }
+  return out
+}
+
+/** Writes a value as its JSON string; the empty value is "". */
+export function serializeValue(v: SchemaValue): string {
+  const fields: ServiceFields = {}
+  for (const key of ['serviceType', 'areaServed', 'price', 'currency'] as const) {
+    const text = (v.fields[key] ?? '').trim()
+    if (text !== '') {
+      fields[key] = text
+    }
+  }
+  const custom = v.custom.trim()
+  if (v.type === '' && custom === '' && Object.keys(fields).length === 0) {
+    return ''
+  }
+  return JSON.stringify({ type: v.type, fields, custom })
+}
+
+/** A price such as 150, 149.99 or 149,99. */
+export function validPrice(s: string): boolean {
+  return s.trim() === '' || /^\d+([.,]\d{1,2})?$/.test(s.trim())
+}
+
+/** A three-letter currency code. */
+export function validCurrency(s: string): boolean {
+  return s.trim() === '' || /^[A-Za-z]{3}$/.test(s.trim())
+}
+
+/** The status of custom JSON-LD text. */
+export type JsonStatus =
+  | { kind: 'empty' }
+  | { kind: 'valid'; nodes: number }
+  | { kind: 'error'; line: number; column: number; message: string }
+  | { kind: 'notObject' }
+  | { kind: 'tooLarge' }
+
+/** Line and column of a character offset. */
+function position(text: string, offset: number): { line: number; column: number } {
+  const before = text.slice(0, Math.max(0, Math.min(offset, text.length)))
+  const lines = before.split('\n')
+  return { line: lines.length, column: lines[lines.length - 1].length + 1 }
+}
+
+/** Checks custom JSON-LD the way the server will (UI-SPEC C17). */
+export function checkCustom(text: string): JsonStatus {
+  if (text.trim() === '') {
+    return { kind: 'empty' }
+  }
+  if (new TextEncoder().encode(text.trim()).length > MAX_CUSTOM_BYTES) {
+    return { kind: 'tooLarge' }
+  }
+  let v: unknown
+  try {
+    v = JSON.parse(text)
+  } catch (err) {
+    const message = err instanceof Error ? err.message : String(err)
+    const lc = /line (\d+) column (\d+)/i.exec(message)
+    if (lc) {
+      return { kind: 'error', line: Number(lc[1]), column: Number(lc[2]), message }
+    }
+    const pos = /position (\d+)/i.exec(message)
+    const at = pos ? position(text, Number(pos[1])) : position(text, text.length)
+    return { kind: 'error', line: at.line, column: at.column, message }
+  }
+  const isObject = (x: unknown): boolean => x !== null && typeof x === 'object' && !Array.isArray(x)
+  if (isObject(v)) {
+    return { kind: 'valid', nodes: 1 }
+  }
+  if (Array.isArray(v) && v.length > 0 && v.every(isObject)) {
+    return { kind: 'valid', nodes: v.length }
+  }
+  return { kind: 'notObject' }
+}
+
+const typeLabel = (labels: Labels, type: string): string => (labels as unknown as Record<string, string>)['t' + type] ?? type
+
+const noteFor = (labels: Labels, type: string): string => {
+  switch (type) {
+    case 'BlogPosting':
+    case 'Article':
+    case 'NewsArticle':
+      return labels.noteArticle
+    case 'ContactPage':
+      return labels.noteContact
+    case 'Service':
+      return labels.noteService
+    case 'CollectionPage':
+      return labels.noteCollection
+    case 'ProfilePage':
+      return labels.noteProfile
+  }
+  return labels.notePage
+}
+
+/**
+ * E8 golem15-seo-schema (UI-SPEC C17, D-44): the page's schema type per
+ * content kind (option kind), the Service fields and custom JSON-LD with a
+ * live parse status. Its value is a JSON string, so E9 reads it through
+ * fill; the server checks it again on save.
+ */
+export class SeoSchemaElement extends HTMLElement {
+  static observedAttributes = ['options', 'disabled', 'invalid', 'ui-locale']
+
+  private current: SchemaValue = { type: '', fields: {}, custom: '' }
+  private open = false
+  private status: JsonStatus = { kind: 'empty' }
+  private timer: ReturnType<typeof setTimeout> | undefined
+  private fieldErrors: Record<string, string> = {}
+  private connected = false
+
+  get value(): string {
+    return serializeValue(this.current)
+  }
+
+  set value(v: unknown) {
+    this.current = parseValue(v)
+    this.status = checkCustom(this.current.custom)
+    this.open = this.current.custom.trim() !== ''
+    this.render()
+  }
+
+  connectedCallback(): void {
+    this.connected = true
+    this.render()
+  }
+
+  disconnectedCallback(): void {
+    this.connected = false
+    clearTimeout(this.timer)
+  }
+
+  attributeChangedCallback(): void {
+    if (this.connected) {
+      this.render()
+    }
+  }
+
+  private kind(): string {
+    const options = jsonAttr<Record<string, unknown>>(this, 'options', {})
+    return typeof options.kind === 'string' ? options.kind : ''
+  }
+
+  private labels(): Labels {
+    return labelsFor(uiLocaleOf(this))
+  }
+
+  private changed(): void {
+    emit(this, 'summer-change', { value: this.value })
+  }
+
+  private effectiveType(): string {
+    const types = typesFor(this.kind())
+    return types.includes(this.current.type) ? this.current.type : types[0]
+  }
+
+  private render(): void {
+    if (!this.connected) {
+      return
+    }
+    const labels = this.labels()
+    const disabled = this.hasAttribute('disabled')
+    const types = typesFor(this.kind())
+    const sections: HTMLElement[] = []
+
+    // 1. The type row.
+    const typeRow = el('div', 'seo-schema__section')
+    const selectId = uid('seo-schema-type')
+    const label = el('label', 'seo-schema__label', labels.schemaType)
+    label.htmlFor = selectId
+    if (types.length > 1) {
+      const select = el('select', 'seo-schema__select')
+      select.id = selectId
+      select.disabled = disabled
+      const def = el('option', '', fmt(labels.typeDefault, { type: types[0] }))
+      def.value = ''
+      select.append(def)
+      for (const t of types) {
+        const opt = el('option', '', typeLabel(labels, t))
+        opt.value = t
+        select.append(opt)
+      }
+      select.value = types.includes(this.current.type) ? this.current.type : ''
+      select.addEventListener('change', () => {
+        this.current.type = select.value
+        this.changed()
+        this.render()
+      })
+      typeRow.append(label, select)
+    } else {
+      const fixed = el('div', 'seo-schema__fixed', typeLabel(labels, types[0]))
+      fixed.id = selectId
+      typeRow.append(label, fixed)
+    }
+    typeRow.append(el('p', 'seo-schema__note', noteFor(labels, this.effectiveType())))
+    sections.push(typeRow)
+
+    // 2. The Service fields.
+    if (this.effectiveType() === 'Service') {
+      const grid = el('div', 'seo-schema__grid')
+      const field = (key: keyof ServiceFields, text: string, hint: string, placeholder = '', mode = ''): void => {
+        const wrap = el('div', 'seo-schema__cell')
+        const id = uid('seo-schema-' + key)
+        const l = el('label', 'seo-schema__label', text)
+        l.htmlFor = id
+        const input = el('input', 'seo-schema__input')
+        input.id = id
+        input.type = 'text'
+        input.disabled = disabled
+        input.value = this.current.fields[key] ?? ''
+        if (placeholder) {
+          input.placeholder = placeholder
+        }
+        if (mode) {
+          input.inputMode = mode
+        }
+        if (key === 'currency') {
+          input.maxLength = 3
+        }
+        const error = el('p', 'seo-schema__error', this.fieldErrors[key] ?? '')
+        error.setAttribute('role', 'status')
+        input.addEventListener('input', () => {
+          const v = input.value
+          const ok = key === 'price' ? validPrice(v) : key === 'currency' ? validCurrency(v) : true
+          if (ok) {
+            this.current.fields[key] = v
+            delete this.fieldErrors[key]
+            error.textContent = ''
+            this.changed()
+          } else {
+            this.fieldErrors[key] = key === 'price' ? labels.priceInvalid : labels.currencyInvalid
+            error.textContent = this.fieldErrors[key]
+          }
+        })
+        if (key === 'currency') {
+          input.addEventListener('blur', () => {
+            input.value = input.value.toUpperCase()
+            if (validCurrency(input.value)) {
+              this.current.fields.currency = input.value
+              this.changed()
+            }
+          })
+        }
+        wrap.append(l, input)
+        if (hint) {
+          wrap.append(el('p', 'seo-schema__note', hint))
+        }
+        wrap.append(error)
+        grid.append(wrap)
+      }
+      field('serviceType', labels.serviceType, '', labels.serviceTypePh)
+      field('areaServed', labels.areaServed, labels.areaServedHint)
+      field('price', labels.price, labels.priceHint, '', 'decimal')
+      field('currency', labels.currency, labels.currencyHint)
+      sections.push(grid)
+    }
+
+    // 3. Custom JSON-LD.
+    const customSection = el('div', 'seo-schema__section')
+    const toggle = el('button', 'seo-schema__disclosure')
+    toggle.type = 'button'
+    toggle.setAttribute('aria-expanded', String(this.open))
+    const panelId = uid('seo-schema-custom')
+    toggle.setAttribute('aria-controls', panelId)
+    toggle.append(el('span', '', labels.custom), icon('chevron-down'))
+    toggle.addEventListener('click', () => {
+      this.open = !this.open
+      this.render()
+    })
+    const head = el('div', 'seo-schema__disclosure-row')
+    head.append(toggle)
+    if (this.current.custom.trim() !== '') {
+      head.append(el('span', 'seo-schema__note', labels.customSet))
+    }
+    customSection.append(head)
+    if (this.open) {
+      const panel = el('div', 'seo-schema__panel')
+      panel.id = panelId
+      const areaId = uid('seo-schema-json')
+      const hint = el('p', 'seo-schema__note', labels.customHint)
+      hint.id = areaId + '-hint'
+      const area = el('textarea', 'seo-schema__json')
+      area.id = areaId
+      area.spellcheck = false
+      area.disabled = disabled
+      area.value = this.current.custom
+      area.setAttribute('aria-label', labels.custom)
+      area.setAttribute('aria-describedby', hint.id)
+      const line = el('p', 'seo-schema__status')
+      line.setAttribute('role', 'status')
+      line.setAttribute('aria-live', 'polite')
+      this.statusLine(line, labels)
+      area.addEventListener('input', () => {
+        this.current.custom = area.value
+        this.changed()
+        clearTimeout(this.timer)
+        this.timer = setTimeout(() => {
+          this.status = checkCustom(area.value)
+          this.statusLine(line, labels)
+        }, 300)
+      })
+      panel.append(hint, area, line)
+      customSection.append(panel)
+    }
+    sections.push(customSection)
+
+    this.classList.toggle('seo-schema--invalid', this.hasAttribute('invalid'))
+    this.replaceChildren(...sections)
+  }
+
+  private statusLine(line: HTMLElement, labels: Labels): void {
+    const s = this.status
+    line.className = 'seo-schema__status'
+    switch (s.kind) {
+      case 'empty':
+        line.replaceChildren()
+        return
+      case 'valid':
+        line.classList.add('seo-schema__status--ok')
+        line.replaceChildren(icon('circle-check'), el('span', '', plural(labels.jsonValid, s.nodes, uiLocaleOf(this))))
+        return
+      case 'error':
+        line.classList.add('seo-schema__status--error')
+        line.replaceChildren(icon('circle-alert'), el('span', '', fmt(labels.jsonError, { line: s.line, column: s.column, message: s.message })))
+        return
+      case 'notObject':
+        line.classList.add('seo-schema__status--error')
+        line.replaceChildren(icon('circle-alert'), el('span', '', labels.jsonNotObject))
+        return
+      case 'tooLarge':
+        line.classList.add('seo-schema__status--error')
+        line.replaceChildren(icon('circle-alert'), el('span', '', labels.jsonTooLarge))
+    }
+  }
+}

assets/src/elements/styles.css

@@ -850,3 +850,392 @@ golem15-seo-transfer .seo-transfer__actions {
     transition: none;
   }
 }
+
+/* E8 golem15-seo-schema (UI-SPEC C17) */
+golem15-seo-schema {
+  display: flex;
+  flex-direction: column;
+  gap: 24px;
+}
+
+golem15-seo-schema .seo-schema__section {
+  display: flex;
+  flex-direction: column;
+  gap: 6px;
+}
+
+golem15-seo-schema .seo-schema__label {
+  font-size: 14px;
+  font-weight: 600;
+  line-height: 1.5;
+  color: var(--c-text);
+}
+
+golem15-seo-schema .seo-schema__select,
+golem15-seo-schema .seo-schema__input,
+golem15-seo-parent .seo-parent__select {
+  box-sizing: border-box;
+  height: 44px;
+  padding: 0 14px;
+  border: 1px solid var(--c-border-strong);
+  border-radius: 10px;
+  background: var(--c-surface);
+  color: var(--c-text);
+  font: inherit;
+  font-size: 14px;
+  transition: border-color 150ms;
+}
+
+golem15-seo-schema .seo-schema__select,
+golem15-seo-parent .seo-parent__select {
+  min-width: 280px;
+  max-width: 100%;
+}
+
+golem15-seo-schema .seo-schema__input {
+  width: 100%;
+}
+
+golem15-seo-schema .seo-schema__fixed {
+  font-size: 14px;
+  color: var(--c-text);
+}
+
+golem15-seo-schema .seo-schema__note,
+golem15-seo-graph .seo-graph__meta {
+  margin: 0;
+  font-size: 13px;
+  line-height: 1.5;
+  color: var(--c-muted);
+}
+
+golem15-seo-schema .seo-schema__section > .seo-schema__note {
+  margin-top: 2px;
+}
+
+golem15-seo-schema .seo-schema__grid {
+  display: grid;
+  grid-template-columns: 1fr 1fr;
+  gap: 16px;
+}
+
+@media (max-width: 600px) {
+  golem15-seo-schema .seo-schema__grid {
+    grid-template-columns: 1fr;
+  }
+}
+
+golem15-seo-schema .seo-schema__cell {
+  display: flex;
+  flex-direction: column;
+  gap: 6px;
+}
+
+golem15-seo-schema .seo-schema__error {
+  margin: 0;
+  font-size: 13px;
+  color: var(--c-danger);
+}
+
+golem15-seo-schema .seo-schema__error:empty {
+  display: none;
+}
+
+golem15-seo-schema .seo-schema__disclosure-row {
+  display: flex;
+  align-items: center;
+  gap: 8px;
+}
+
+golem15-seo-schema .seo-schema__disclosure,
+golem15-seo-graph .seo-graph__disclosure {
+  display: inline-flex;
+  align-items: center;
+  gap: 8px;
+  padding: 0;
+  border: 0;
+  background: none;
+  color: var(--c-text);
+  font: inherit;
+  font-size: 14px;
+  font-weight: 600;
+  cursor: pointer;
+  align-self: flex-start;
+}
+
+golem15-seo-schema .seo-schema__disclosure[aria-expanded='true'] .seo-icon,
+golem15-seo-graph .seo-graph__disclosure[aria-expanded='true'] .seo-icon {
+  transform: rotate(180deg);
+}
+
+golem15-seo-schema .seo-schema__panel {
+  display: flex;
+  flex-direction: column;
+  gap: 8px;
+}
+
+golem15-seo-schema .seo-schema__json {
+  box-sizing: border-box;
+  width: 100%;
+  min-height: 160px;
+  resize: vertical;
+  padding: 12px 14px;
+  border: 1px solid var(--c-border-strong);
+  border-radius: 10px;
+  background: var(--c-surface);
+  color: var(--c-text);
+  font-family: 'DM Mono', ui-monospace, SFMono-Regular, Menlo, monospace;
+  font-size: 13px;
+  line-height: 1.6;
+}
+
+golem15-seo-schema .seo-schema__status,
+golem15-seo-graph .seo-graph__finding,
+golem15-seo-graph .seo-graph__ok,
+golem15-seo-graph .seo-graph__error {
+  display: flex;
+  align-items: center;
+  gap: 8px;
+  margin: 0;
+  font-size: 13px;
+  line-height: 1.5;
+}
+
+golem15-seo-schema .seo-schema__status--ok,
+golem15-seo-graph .seo-graph__ok {
+  color: var(--c-ok-text);
+}
+
+golem15-seo-schema .seo-schema__status--error,
+golem15-seo-graph .seo-graph__error,
+golem15-seo-graph .seo-graph__finding--required .seo-icon {
+  color: var(--c-danger);
+}
+
+golem15-seo-graph .seo-graph__finding {
+  font-size: 14px;
+  color: var(--c-text);
+}
+
+golem15-seo-schema .seo-schema__select:focus-visible,
+golem15-seo-schema .seo-schema__input:focus-visible,
+golem15-seo-schema .seo-schema__json:focus-visible,
+golem15-seo-schema .seo-schema__disclosure:focus-visible,
+golem15-seo-graph .seo-graph__disclosure:focus-visible,
+golem15-seo-graph .seo-graph__button:focus-visible,
+golem15-seo-graph .seo-graph__link:focus-visible,
+golem15-seo-parent .seo-parent__select:focus-visible {
+  outline: none;
+  box-shadow: 0 0 0 3px var(--c-ring);
+}
+
+golem15-seo-schema.seo-schema--invalid .seo-schema__json,
+golem15-seo-parent.seo-parent--invalid .seo-parent__select {
+  border-color: var(--c-danger);
+}
+
+/* E9 golem15-seo-graph (UI-SPEC C18) */
+golem15-seo-graph {
+  display: flex;
+  flex-direction: column;
+  gap: 24px;
+}
+
+golem15-seo-graph .seo-graph__nodes {
+  display: flex;
+  flex-direction: column;
+  gap: 16px;
+}
+
+golem15-seo-graph .seo-graph__node {
+  display: flex;
+  flex-direction: column;
+  gap: 8px;
+}
+
+golem15-seo-graph .seo-graph__node-head {
+  display: flex;
+  align-items: center;
+  gap: 8px;
+}
+
+golem15-seo-graph .seo-graph__pill {
+  display: inline-flex;
+  align-items: center;
+  height: 24px;
+  padding: 0 10px;
+  border-radius: 999px;
+  background: var(--c-subtle);
+  color: var(--c-muted);
+  font-size: 12px;
+  font-weight: 600;
+}
+
+golem15-seo-graph .seo-graph__id {
+  font-family: 'DM Mono', ui-monospace, SFMono-Regular, Menlo, monospace;
+  font-size: 13px;
+  color: var(--c-muted);
+}
+
+golem15-seo-graph .seo-graph__json {
+  box-sizing: border-box;
+  max-height: 360px;
+  overflow: auto;
+  margin: 0;
+  padding: 12px 14px;
+  border: 1px solid var(--c-border);
+  border-radius: 10px;
+  background: var(--c-subtle);
+  color: var(--c-text);
+  font-family: 'DM Mono', ui-monospace, SFMono-Regular, Menlo, monospace;
+  font-size: 13px;
+  line-height: 1.6;
+  white-space: pre;
+}
+
+golem15-seo-graph .seo-graph__links {
+  display: flex;
+  flex-direction: column;
+  gap: 8px;
+}
+
+golem15-seo-graph .seo-graph__link {
+  display: inline-flex;
+  align-items: center;
+  gap: 6px;
+  color: var(--c-primary);
+  font-size: 14px;
+  align-self: flex-start;
+}
+
+golem15-seo-graph .seo-graph__button {
+  align-self: flex-start;
+  height: 42px;
+  padding: 0 18px;
+  border: 1px solid var(--c-border-strong);
+  border-radius: 10px;
+  background: var(--c-surface);
+  color: var(--c-text);
+  font: inherit;
+  font-size: 14px;
+  font-weight: 600;
+  cursor: pointer;
+  transition: background-color 150ms;
+}
+
+golem15-seo-graph .seo-graph__button:hover {
+  background: var(--c-subtle);
+}
+
+/* E10 golem15-seo-links (UI-SPEC C19) */
+golem15-seo-links {
+  display: flex;
+  flex-direction: column;
+  gap: 24px;
+}
+
+golem15-seo-links .seo-links__counts,
+golem15-seo-links .seo-links__list {
+  display: flex;
+  flex-direction: column;
+  gap: 8px;
+}
+
+golem15-seo-links .seo-links__counts {
+  gap: 16px;
+}
+
+golem15-seo-links .seo-links__meta {
+  margin: 0;
+  font-size: 13px;
+  line-height: 1.5;
+  color: var(--c-muted);
+}
+
+golem15-seo-links .seo-links__heading {
+  margin: 0;
+  font-size: 14px;
+  font-weight: 600;
+  line-height: 1.5;
+  color: var(--c-text);
+}
+
+golem15-seo-links .seo-links__row {
+  display: flex;
+  align-items: center;
+  justify-content: space-between;
+  gap: 8px;
+  font-size: 14px;
+  line-height: 1.5;
+  color: var(--c-text);
+}
+
+@media (max-width: 600px) {
+  golem15-seo-links .seo-links__row {
+    flex-direction: column;
+    align-items: flex-start;
+  }
+}
+
+golem15-seo-links .seo-links__line {
+  min-width: 0;
+  overflow-wrap: anywhere;
+}
+
+golem15-seo-links .seo-links__title {
+  color: var(--c-text);
+  text-decoration: underline;
+}
+
+golem15-seo-links .seo-links__url {
+  font-family: 'DM Mono', ui-monospace, SFMono-Regular, Menlo, monospace;
+  font-size: 13px;
+  color: var(--c-muted);
+  user-select: all;
+  overflow-wrap: anywhere;
+}
+
+golem15-seo-links .seo-links__copy {
+  flex: none;
+  height: 34px;
+  padding: 0 12px;
+  border: 1px solid var(--c-border-strong);
+  border-radius: 8px;
+  background: var(--c-surface);
+  color: var(--c-text);
+  font: inherit;
+  font-size: 13px;
+  font-weight: 600;
+  cursor: pointer;
+  transition: background-color 150ms;
+}
+
+golem15-seo-links .seo-links__copy:hover {
+  background: var(--c-subtle);
+}
+
+golem15-seo-links .seo-links__copy:focus-visible,
+golem15-seo-links .seo-links__title:focus-visible {
+  outline: none;
+  box-shadow: 0 0 0 3px var(--c-ring);
+}
+
+golem15-seo-links .seo-links__error {
+  display: flex;
+  align-items: center;
+  gap: 8px;
+  margin: 0;
+  font-size: 13px;
+  line-height: 1.5;
+  color: var(--c-danger);
+}
+
+golem15-seo-links .seo-links__status {
+  margin: 0;
+  font-size: 13px;
+  color: var(--c-muted);
+}
+
+golem15-seo-links .seo-links__status:empty {
+  display: none;
+}

batch_test.go

@@ -70,6 +70,8 @@ func TestBatch(t *testing.T) {
 	token := env.admin(t, true)
 	env.createPage(t, token, map[string]any{"title": "Page A", "slug": "a", "excerpt": "About A", "published": true})
 	env.createPage(t, token, map[string]any{"title": "Page B", "slug": "b", "excerpt": "About B", "published": true})
+	// noindex_published needs the address in a sitemap file.
+	env.rebuildSitemap(t)
 	agent := env.principal(t, env.admin(t, false, "golem15.seo.manage"))
 
 	changes := []any{

classes/alternates.go

@@ -0,0 +1,119 @@
+package classes
+
+import (
+	"strings"
+
+	"git.golem15.com/golem15/summercms/modules/pact"
+)
+
+// OGLocalesConfigKey maps locale codes to og:locale values (D-55); unset
+// codes fall back to DefaultOGLocales, else <code>_<CODE>.
+const OGLocalesConfigKey = "golem15.seo.og_locales"
+
+// SearchURLConfigKey is the site search address with the
+// {search_term_string} placeholder (P33-25); empty means no SearchAction.
+const SearchURLConfigKey = "golem15.seo.search_url"
+
+// DefaultOGLocales are the og:locale values of the common locales.
+var DefaultOGLocales = map[string]string{"pl": "pl_PL", "en": "en_US", "de": "de_DE"}
+
+// Alternate is one hreflang alternate of a page (D-55): a language tag or
+// x-default and the absolute URL.
+type Alternate struct {
+	Hreflang string `json:"hreflang"`
+	URL      string `json:"url"`
+}
+
+// LangTag turns a locale code into its language tag: pl stays pl, pt_BR
+// and pt-br become pt-BR (the region is kept when the locale has one).
+func LangTag(locale string) string {
+	locale = strings.TrimSpace(strings.ReplaceAll(locale, "_", "-"))
+	lang, region, ok := strings.Cut(locale, "-")
+	if !ok || region == "" {
+		return strings.ToLower(lang)
+	}
+	return strings.ToLower(lang) + "-" + strings.ToUpper(region)
+}
+
+// OGLocale is the og:locale value of locale: the configured value, else
+// the default table, else the language with its region (pt_BR) or the
+// language doubled (fr_FR).
+func (s *Service) OGLocale(locale string) string {
+	tag := LangTag(locale)
+	if tag == "" {
+		return ""
+	}
+	if cfg := s.Config(); cfg != nil {
+		for _, key := range []string{locale, tag, strings.ToLower(tag)} {
+			if v := strings.TrimSpace(cfg.String(OGLocalesConfigKey + "." + key)); v != "" {
+				return v
+			}
+		}
+	}
+	if v, ok := DefaultOGLocales[tag]; ok {
+		return v
+	}
+	lang, region, ok := strings.Cut(tag, "-")
+	if ok {
+		return lang + "_" + region
+	}
+	return lang + "_" + strings.ToUpper(lang)
+}
+
+// translatedLocales lists the locales of rec with a public address, in the
+// record's order.
+func translatedLocales(rec *pact.ContentRecord) []pact.ContentLocale {
+	if rec == nil {
+		return nil
+	}
+	var out []pact.ContentLocale
+	seen := map[string]bool{}
+	for _, l := range rec.Locales {
+		if strings.TrimSpace(l.URL) == "" || l.Locale == "" || seen[l.Locale] {
+			continue
+		}
+		seen[l.Locale] = true
+		out = append(out, l)
+	}
+	return out
+}
+
+// Alternates lists the hreflang alternates of a record (D-55 to D-57):
+// each translated locale with its absolute URL, the page itself included,
+// then x-default on the default locale's URL. A record translated into one
+// locale only has none; a locale without a translation is absent.
+func Alternates(site string, rec *pact.ContentRecord, defaultLocale string) []Alternate {
+	locales := translatedLocales(rec)
+	if len(locales) < 2 {
+		return []Alternate{}
+	}
+	out := make([]Alternate, 0, len(locales)+1)
+	xDefault := ""
+	for _, l := range locales {
+		u := AbsoluteURL(site, l.URL)
+		out = append(out, Alternate{Hreflang: LangTag(l.Locale), URL: u})
+		if l.Locale == defaultLocale {
+			xDefault = u
+		}
+	}
+	if xDefault != "" {
+		out = append(out, Alternate{Hreflang: "x-default", URL: xDefault})
+	}
+	return out
+}
+
+// OGLocaleAlternates are the og:locale:alternate values of a record: one
+// per other translated locale.
+func (s *Service) OGLocaleAlternates(rec *pact.ContentRecord, locale string) []string {
+	out := []string{}
+	locales := translatedLocales(rec)
+	if len(locales) < 2 {
+		return out
+	}
+	for _, l := range locales {
+		if l.Locale != locale {
+			out = append(out, s.OGLocale(l.Locale))
+		}
+	}
+	return out
+}

classes/alternates_test.go

@@ -0,0 +1,93 @@
+package classes
+
+import (
+	"testing"
+
+	"git.golem15.com/golem15/summercms/modules/pact"
+)
+
+// TestLangTag pins the language tags of locale codes.
+func TestLangTag(t *testing.T) {
+	for in, want := range map[string]string{"pl": "pl", "PL": "pl", "pt_BR": "pt-BR", "pt-br": "pt-BR", " en-gb ": "en-GB", "de-": "de", "": ""} {
+		if got := LangTag(in); got != want {
+			t.Errorf("LangTag(%q) = %q, want %q", in, got, want)
+		}
+	}
+}
+
+// TestAlternates pins D-55 to D-57: each translated locale with its
+// absolute URL (the page itself included), x-default on the default
+// locale's URL, none for one locale, a locale without an address or a
+// repeated locale absent, and no x-default when the default locale is not
+// translated.
+func TestAlternates(t *testing.T) {
+	const site = "https://example.test"
+	rec := &pact.ContentRecord{Locales: []pact.ContentLocale{
+		{Locale: "pl", URL: "/blog/grzyby"},
+		{Locale: "en", URL: "/en/blog/mushrooms"},
+		{Locale: "de", URL: " "},
+		{Locale: "pl", URL: "/blog/again"},
+		{Locale: "pt_BR", URL: "https://example.test/pt/cogumelos"},
+	}}
+	got := Alternates(site, rec, "pl")
+	want := []Alternate{
+		{Hreflang: "pl", URL: site + "/blog/grzyby"},
+		{Hreflang: "en", URL: site + "/en/blog/mushrooms"},
+		{Hreflang: "pt-BR", URL: site + "/pt/cogumelos"},
+		{Hreflang: "x-default", URL: site + "/blog/grzyby"},
+	}
+	if len(got) != len(want) {
+		t.Fatalf("alternates = %+v", got)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Fatalf("alternate %d = %+v, want %+v", i, got[i], want[i])
+		}
+	}
+	if noDefault := Alternates(site, rec, "fr"); len(noDefault) != 3 || noDefault[2].Hreflang == "x-default" {
+		t.Fatalf("without the default locale = %+v", noDefault)
+	}
+	one := &pact.ContentRecord{Locales: []pact.ContentLocale{{Locale: "pl", URL: "/a"}, {Locale: "en", URL: ""}}}
+	if got := Alternates(site, one, "pl"); got == nil || len(got) != 0 {
+		t.Fatalf("one locale = %+v, want []", got)
+	}
+	if got := Alternates(site, nil, "pl"); len(got) != 0 {
+		t.Fatal("no record, no alternates")
+	}
+
+	// G3: a Journal-shaped record, as the Journal descriptors describe one
+	// (P33-64): every enabled locale in sort order, the untranslated de
+	// with URL "" and the default locale's title, description and word
+	// count. de gets no hreflang.
+	words := 120
+	journalShaped := &pact.ContentRecord{Locales: []pact.ContentLocale{
+		{Locale: "pl", URL: "/blog/grzyby", Title: "Grzyby", Description: "O grzybach", Words: &words},
+		{Locale: "en", URL: "/en/blog/mushrooms", Title: "Mushrooms", Description: "About mushrooms", Words: &words},
+		{Locale: "de", URL: "", Title: "Grzyby", Description: "O grzybach", Words: &words},
+	}}
+	got = Alternates(site, journalShaped, "pl")
+	want = []Alternate{
+		{Hreflang: "pl", URL: site + "/blog/grzyby"},
+		{Hreflang: "en", URL: site + "/en/blog/mushrooms"},
+		{Hreflang: "x-default", URL: site + "/blog/grzyby"},
+	}
+	if len(got) != len(want) {
+		t.Fatalf("Journal-shaped alternates = %+v", got)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Fatalf("Journal-shaped alternate %d = %+v, want %+v", i, got[i], want[i])
+		}
+	}
+}
+
+// TestOGLocale pins og:locale without configuration: the default table,
+// the region form and the doubled language.
+func TestOGLocale(t *testing.T) {
+	s := &Service{}
+	for in, want := range map[string]string{"pl": "pl_PL", "en": "en_US", "de": "de_DE", "pt_BR": "pt_BR", "fr": "fr_FR", "": ""} {
+		if got := s.OGLocale(in); got != want {
+			t.Errorf("OGLocale(%q) = %q, want %q", in, got, want)
+		}
+	}
+}

classes/autolinks.go

@@ -0,0 +1,333 @@
+package classes
+
+import (
+	"context"
+	"errors"
+	"net/url"
+	"strings"
+	"time"
+	"unicode"
+	"unicode/utf8"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"gorm.io/gorm"
+)
+
+// MaxPageAutolinks caps the phrases one resolve answer carries.
+const MaxPageAutolinks = 1000
+
+// AutolinkLink is one phrase of a resolve answer's autolinks (D-82): the
+// layer links the phrase's first occurrence in a text to URL.
+type AutolinkLink struct {
+	Phrase string `json:"phrase"`
+	URL    string `json:"url"`
+}
+
+// AutolinkValues are an auto-link's values for history and the API.
+func AutolinkValues(a models.Autolink) map[string]any {
+	return map[string]any{
+		"phrase": a.Phrase, "url": a.URL, "locale": a.Locale,
+		"enabled": a.Enabled, "origin": a.Origin, "note": a.Note,
+	}
+}
+
+// AutolinkWrite is a create or update of an auto-link: only the fields
+// set change.
+type AutolinkWrite struct {
+	Phrase  *string `json:"phrase,omitempty"`
+	URL     *string `json:"url,omitempty"`
+	Locale  *string `json:"locale,omitempty"`
+	Enabled *bool   `json:"enabled,omitempty"`
+	Note    *string `json:"note,omitempty"`
+}
+
+// Apply copies the fields set into a.
+func (w AutolinkWrite) Apply(a *models.Autolink) {
+	if w.Phrase != nil {
+		a.Phrase = *w.Phrase
+	}
+	if w.URL != nil {
+		a.URL = *w.URL
+	}
+	if w.Locale != nil {
+		a.Locale = *w.Locale
+	}
+	if w.Enabled != nil {
+		a.Enabled = *w.Enabled
+	}
+	if w.Note != nil {
+		a.Note = *w.Note
+	}
+}
+
+// NormalizeAutolink trims the phrase (inner whitespace collapsed to one
+// space), the address, the locale ("all" is every locale) and the note.
+func NormalizeAutolink(a *models.Autolink) {
+	a.Phrase = strings.Join(strings.Fields(a.Phrase), " ")
+	a.URL = strings.TrimSpace(a.URL)
+	a.Locale = strings.TrimSpace(a.Locale)
+	if a.Locale == models.AutolinkLocaleAll {
+		a.Locale = ""
+	}
+	a.Note = strings.TrimSpace(a.Note)
+	if a.Origin == "" {
+		a.Origin = models.AutolinkOriginManual
+	}
+}
+
+// ValidAutolinkURL reports whether u is a path on the site (starting with
+// one /) or an absolute http(s) address, without whitespace or control
+// characters (T-33-50: javascript:, data: and protocol-relative addresses
+// are refused).
+func ValidAutolinkURL(u string) bool {
+	if u == "" || len(u) > models.MaxAutolinkURL || !utf8.ValidString(u) {
+		return false
+	}
+	for _, r := range u {
+		if unicode.IsSpace(r) || unicode.IsControl(r) {
+			return false
+		}
+	}
+	if strings.HasPrefix(u, "/") {
+		return !strings.HasPrefix(u, "//") && !strings.HasPrefix(u, `/\`)
+	}
+	parsed, err := url.Parse(u)
+	if err != nil {
+		return false
+	}
+	scheme := strings.ToLower(parsed.Scheme)
+	return (scheme == "http" || scheme == "https") && parsed.Host != "" && parsed.User == nil
+}
+
+// AutolinkProblems refuses a phrase under 3 characters (or over 200), an
+// address that is not a path or an http(s) address, an unknown locale,
+// and a phrase another row already has in the same locale (a phrase of
+// every locale meets the phrases of each locale), naming that row's
+// address (UI-SPEC C22). a is normalised first.
+func (s *Service) AutolinkProblems(ctx context.Context, db *gorm.DB, a *models.Autolink) (map[string][]models.Problem, error) {
+	NormalizeAutolink(a)
+	out := map[string][]models.Problem{}
+	n := utf8.RuneCountInString(a.Phrase)
+	switch {
+	case n < models.MinAutolinkPhrase:
+		out["phrase"] = append(out["phrase"], models.Problem{Key: "golem15.seo::lang.autolinks.phrase_short"})
+	case n > models.MaxAutolinkPhrase:
+		out["phrase"] = append(out["phrase"], models.Problem{Key: "lagoon::validation.max.string", Params: map[string]string{"attribute": "phrase", "max": "200"}})
+	}
+	if !ValidAutolinkURL(a.URL) {
+		out["url"] = append(out["url"], models.Problem{Key: "golem15.seo::lang.autolinks.url_invalid"})
+	}
+	if a.Locale != "" {
+		known := false
+		for _, code := range s.Locales(ctx, db) {
+			if code == a.Locale {
+				known = true
+				break
+			}
+		}
+		if !known {
+			out["locale"] = append(out["locale"], models.Problem{Key: "golem15.seo::lang.api.value_invalid"})
+		}
+	}
+	if len(out["phrase"]) == 0 {
+		q := db.Model(&models.Autolink{}).Where("lower(phrase) = lower(?)", a.Phrase)
+		if a.Locale != "" {
+			q = q.Where("locale = ? OR locale = ''", a.Locale)
+		}
+		if a.ID != 0 {
+			q = q.Where("id <> ?", a.ID)
+		}
+		var rows []models.Autolink
+		if err := q.Order("id ASC").Limit(1).Find(&rows).Error; err != nil {
+			return nil, err
+		}
+		if len(rows) > 0 {
+			out["phrase"] = append(out["phrase"], models.Problem{Key: "golem15.seo::lang.autolinks.phrase_taken", Params: map[string]string{"url": rows[0].URL}})
+		}
+	}
+	return out, nil
+}
+
+// LoadAutolink returns auto-link id; ErrNoRecord when there is none.
+func LoadAutolink(db *gorm.DB, id uint) (models.Autolink, error) {
+	var rows []models.Autolink
+	if err := db.Where("id = ?", id).Limit(1).Find(&rows).Error; err != nil {
+		return models.Autolink{}, err
+	}
+	if len(rows) == 0 {
+		return models.Autolink{}, ErrNoRecord
+	}
+	return rows[0], nil
+}
+
+// AutolinkOrigin is the origin of a phrase created through ctx: mcp
+// through MCP, else manual.
+func AutolinkOrigin(ctx context.Context) string {
+	if ViaOf(ctx) == models.ViaMCP {
+		return models.AutolinkOriginMCP
+	}
+	return models.AutolinkOriginManual
+}
+
+// autolinkWhat is an auto-link's history description.
+func (s *Service) autolinkWhat(a models.Autolink) string {
+	return s.Phrase("golem15.seo::lang.history.what_autolink", map[string]string{"phrase": a.Phrase})
+}
+
+// RecordAutolink records an auto-link's change in history inside tx; the
+// write purges the site's cached pages, since the phrase may appear in any
+// text. before is nil for a created phrase.
+func (s *Service) RecordAutolink(ctx context.Context, tx *gorm.DB, before map[string]any, a models.Autolink, deleted bool) error {
+	c := Change{Kind: models.SubjectAutolink, SubjectID: a.IDText(), What: s.autolinkWhat(a), Before: before}
+	if !deleted {
+		c.After = AutolinkValues(a)
+	}
+	return s.Record(ctx, tx, c)
+}
+
+// SaveAutolink checks and writes an auto-link inside tx and records it in
+// history; a refused phrase is a *ProblemsError.
+func (s *Service) SaveAutolink(ctx context.Context, tx *gorm.DB, a *models.Autolink) error {
+	if a == nil {
+		return errors.New("golem15.seo: no auto-link")
+	}
+	problems, err := s.AutolinkProblems(ctx, tx, a)
+	if err != nil {
+		return err
+	}
+	if len(problems) > 0 {
+		return &ProblemsError{Problems: problems}
+	}
+	now := time.Now().UTC()
+	var before map[string]any
+	if a.ID == 0 {
+		a.CreatedAt, a.UpdatedAt = &now, &now
+		if err := tx.Create(a).Error; err != nil {
+			return err
+		}
+	} else {
+		old, err := LoadAutolink(tx, a.ID)
+		if err != nil {
+			return err
+		}
+		before = AutolinkValues(old)
+		a.CreatedAt, a.UpdatedAt = old.CreatedAt, &now
+		if err := tx.Save(a).Error; err != nil {
+			return err
+		}
+	}
+	return s.RecordAutolink(ctx, tx, before, *a, false)
+}
+
+// DeleteAutolinks deletes the auto-links ids inside tx, records each in
+// history and returns the ids deleted (unknown ids are skipped).
+func (s *Service) DeleteAutolinks(ctx context.Context, tx *gorm.DB, ids []uint) ([]uint, error) {
+	if len(ids) == 0 {
+		return []uint{}, nil
+	}
+	var rows []models.Autolink
+	if err := tx.Where("id IN ?", ids).Order("id ASC").Find(&rows).Error; err != nil {
+		return nil, err
+	}
+	out := make([]uint, 0, len(rows))
+	for _, row := range rows {
+		if err := tx.Delete(&models.Autolink{}, row.ID).Error; err != nil {
+			return nil, err
+		}
+		if err := s.RecordAutolink(ctx, tx, AutolinkValues(row), row, true); err != nil {
+			return nil, err
+		}
+		out = append(out, row.ID)
+	}
+	return out, nil
+}
+
+// autolinkPath is the site path an auto-link's address names: the path of
+// a / address, or of an absolute address on the site's host (its www twin
+// included); "" for another site.
+func (s *Service) autolinkPath(target string) string {
+	if strings.HasPrefix(target, "/") {
+		path, _, _ := strings.Cut(target, "#")
+		path, _, _ = strings.Cut(path, "?")
+		return path
+	}
+	u, err := url.Parse(target)
+	if err != nil {
+		return ""
+	}
+	site := strings.TrimPrefix(strings.ToLower(s.host()), "www.")
+	if site == "" || strings.TrimPrefix(strings.ToLower(u.Host), "www.") != site {
+		return ""
+	}
+	if u.Path == "" {
+		return "/"
+	}
+	return u.Path
+}
+
+// ForPage lists the auto-link phrases of a page in locale (D-82): the
+// active phrases of that locale and of every locale, without those that
+// link to the page itself (selfPath, a trailing slash ignored), longest
+// phrase first so a longer phrase wins over a phrase it contains, one row
+// per phrase, at most MaxPageAutolinks.
+func (s *Service) ForPage(ctx context.Context, db *gorm.DB, locale, selfPath string) ([]AutolinkLink, error) {
+	var rows []models.Autolink
+	err := db.WithContext(ctx).Where("enabled AND (locale = '' OR locale = ?)", locale).
+		Order("char_length(phrase) DESC, lower(phrase) ASC, locale DESC, id ASC").
+		Limit(MaxPageAutolinks).Find(&rows).Error
+	if err != nil {
+		return nil, err
+	}
+	self := trimSlash(selfPath)
+	seen := map[string]bool{}
+	out := make([]AutolinkLink, 0, len(rows))
+	for _, row := range rows {
+		key := strings.ToLower(row.Phrase)
+		if seen[key] {
+			continue
+		}
+		if p := s.autolinkPath(row.URL); p != "" && trimSlash(p) == self {
+			continue
+		}
+		seen[key] = true
+		out = append(out, AutolinkLink{Phrase: row.Phrase, URL: row.URL})
+	}
+	return out, nil
+}
+
+// FromKeyphrases adds one auto-link per record whose focus keyphrase in a
+// locale is not yet a phrase in that locale (UI-SPEC C22): the record's
+// address in that locale, origin keyphrase, active. A keyphrase another
+// row already has, one under 3 characters or one whose address is not a
+// path is skipped. It runs inside tx and returns the rows added; a second
+// run adds nothing.
+func (s *Service) FromKeyphrases(ctx context.Context, tx *gorm.DB) (int, error) {
+	var rows []struct {
+		Keyphrase string
+		Locale    string
+		URL       string
+	}
+	err := tx.Raw(`SELECT keyphrase, locale, url FROM golem15_seo_entries e
+WHERE e.keyphrase <> '' AND e.url <> ''
+AND NOT EXISTS (SELECT 1 FROM golem15_seo_autolinks a WHERE lower(a.phrase) = lower(e.keyphrase) AND (a.locale = e.locale OR a.locale = ''))
+ORDER BY e.locale ASC, e.updated_at DESC, e.id DESC`).Scan(&rows).Error
+	if err != nil {
+		return 0, err
+	}
+	added := 0
+	for _, row := range rows {
+		a := models.Autolink{Phrase: row.Keyphrase, URL: row.URL, Locale: row.Locale, Enabled: true, Origin: models.AutolinkOriginKeyphrase}
+		problems, err := s.AutolinkProblems(ctx, tx, &a)
+		if err != nil {
+			return added, err
+		}
+		if len(problems) > 0 {
+			continue
+		}
+		if err := s.SaveAutolink(ctx, tx, &a); err != nil {
+			return added, err
+		}
+		added++
+	}
+	return added, nil
+}

classes/autolinks_test.go

@@ -0,0 +1,60 @@
+package classes
+
+import (
+	"strings"
+	"testing"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+)
+
+// TestValidAutolinkURL pins T-33-50: a site path from one / or an
+// absolute http(s) address; javascript:, data:, protocol-relative, a
+// backslash path, credentials, whitespace and control characters refused.
+func TestValidAutolinkURL(t *testing.T) {
+	for u, want := range map[string]bool{
+		"/blog/chaga": true, "/": true, "https://example.test/a": true, "HTTP://example.test": true,
+		"": false, "javascript:alert(1)": false, "data:text/html,x": false, "//evil.test/a": false, `/\evil.test`: false,
+		"https://user:pw@example.test/": false, "/a b": false, "/a\tb": false, "/a\x00": false, "mailto:a@b.c": false,
+		"https://": false, "relative/path": false, "/" + strings.Repeat("a", models.MaxAutolinkURL): false,
+	} {
+		if got := ValidAutolinkURL(u); got != want {
+			t.Errorf("ValidAutolinkURL(%q) = %v, want %v", u, got, want)
+		}
+	}
+}
+
+// TestNormalizeAutolink pins the stored form: the phrase's inner
+// whitespace collapsed, the address, locale and note trimmed, "all" as
+// every locale (""), and the manual origin by default.
+func TestNormalizeAutolink(t *testing.T) {
+	a := &models.Autolink{Phrase: "  lwia \n  grzywa ", URL: " /blog/lwia ", Locale: models.AutolinkLocaleAll, Note: " n "}
+	NormalizeAutolink(a)
+	if a.Phrase != "lwia grzywa" || a.URL != "/blog/lwia" || a.Locale != "" || a.Note != "n" || a.Origin != models.AutolinkOriginManual {
+		t.Fatalf("normalised = %+v", a)
+	}
+	b := &models.Autolink{Locale: " pl ", Origin: "keyphrase"}
+	NormalizeAutolink(b)
+	if b.Locale != "pl" || b.Origin != "keyphrase" {
+		t.Fatalf("kept = %+v", b)
+	}
+}
+
+// TestAutolinkWrite pins a partial write: only the fields set change, and
+// the values for history carry every field.
+func TestAutolinkWrite(t *testing.T) {
+	a := &models.Autolink{Phrase: "chaga", URL: "/a", Locale: "pl", Enabled: true, Note: "x"}
+	phrase, off := "inonotus", false
+	AutolinkWrite{Phrase: &phrase, Enabled: &off}.Apply(a)
+	if a.Phrase != "inonotus" || a.Enabled || a.URL != "/a" || a.Locale != "pl" || a.Note != "x" {
+		t.Fatalf("after the write = %+v", a)
+	}
+	v := AutolinkValues(*a)
+	for _, k := range []string{"phrase", "url", "locale", "enabled", "origin", "note"} {
+		if _, ok := v[k]; !ok {
+			t.Errorf("history values lack %s", k)
+		}
+	}
+	if v["phrase"] != "inonotus" || v["enabled"] != false {
+		t.Fatalf("values = %v", v)
+	}
+}

classes/batch.go

@@ -290,8 +290,9 @@ func (s *Service) runBatch(ctx context.Context, opts BatchOptions, build func(ct
 			}
 			res.Changes = append(res.Changes, item)
 		}
+		listed := &listedOnce{svc: s}
 		for i := range res.Changes {
-			late, err := s.batchConflicts(ctx, tx, i, changes[i], res.Changes[i])
+			late, err := s.batchConflicts(ctx, tx, listed, i, changes[i], res.Changes[i])
 			if err != nil {
 				return err
 			}
@@ -452,12 +453,34 @@ func (s *Service) applyChange(ctx context.Context, tx *gorm.DB, index int, ch Ba
 	return errUnknownOp
 }
 
+// listedOnce reads the addresses the sitemaps list once per batch.
+type listedOnce struct {
+	svc    *Service
+	read   bool
+	ok     bool
+	listed map[string]bool
+}
+
+// has reports whether the sitemaps list address; ok is false without the
+// sitemap plugin's Listing.
+func (l *listedOnce) has(ctx context.Context, address string) (listed, ok bool, err error) {
+	if !l.read {
+		l.listed, l.ok, err = l.svc.SitemapListed(context.WithoutCancel(ctx))
+		if err != nil {
+			return false, false, err
+		}
+		l.read = true
+	}
+	return l.listed[address], l.ok, nil
+}
+
 // batchConflicts are the warnings that depend on the whole batch, checked
 // once every change is in place: noindex on the address of a published
-// record (the records the sitemaps list; Phase 33's sitemap filter
+// record that a sitemap file lists now (read through contract.Listing, so
-// refines it) and a canonical pointing to a redirected or noindex
+// an address the sitemaps already leave out, such as one of a content
-// address.
+// type switched off, is no conflict; D-12, D-24), and a canonical pointing
-func (s *Service) batchConflicts(ctx context.Context, tx *gorm.DB, index int, ch BatchChange, item BatchItem) ([]Conflict, error) {
+// to a redirected or noindex address.
+func (s *Service) batchConflicts(ctx context.Context, tx *gorm.DB, listed *listedOnce, index int, ch BatchChange, item BatchItem) ([]Conflict, error) {
 	if item.Error != "" {
 		return nil, nil
 	}
@@ -490,7 +513,15 @@ func (s *Service) batchConflicts(ctx context.Context, tx *gorm.DB, index int, ch
 		if err != nil && !errors.Is(err, ErrBadURL) {
 			return nil, err
 		}
-		if err == nil && owner.WrittenTo == WrittenToRecord && owner.Published {
+		inSitemap := err == nil && owner.WrittenTo == WrittenToRecord && owner.Published
+		if inSitemap {
+			has, ok, lerr := listed.has(ctx, owner.URL)
+			if lerr != nil {
+				return nil, lerr
+			}
+			inSitemap = !ok || has
+		}
+		if inSitemap {
 			out = append(out, Conflict{Change: index, Kind: ConflictNoindexPublished, URL: owner.URL,
 				Message: s.PhraseCtx(ctx, "golem15.seo::lang.api.conflict_noindex_published", map[string]string{"url": owner.URL})})
 		}

classes/breadcrumbs.go

@@ -0,0 +1,198 @@
+package classes
+
+import (
+	"context"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes/schema"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
+)
+
+// The entry data keys of the structured data fields (P33-23): one value per
+// record, written on every locale row as the social image is.
+const (
+	DataPrimaryParent = "primary_parent"
+	DataSchema        = "schema"
+)
+
+// MaxCrumbDepth caps the parent chain of a breadcrumb trail (T-33-28).
+const MaxCrumbDepth = 6
+
+// ParentValue is the stored form of a parent reference: <type>:<id>.
+func ParentValue(ref pact.ContentRef) string { return ref.Type + ":" + ref.ID }
+
+// entryData reads a string value of an entry's data.
+func entryData(e models.Entry, key string) string {
+	if e.Data == nil {
+		return ""
+	}
+	s, _ := e.Data[key].(string)
+	return strings.TrimSpace(s)
+}
+
+// PrimaryParent is a record's primary parent (D-86): the stored choice
+// when the record still has that parent, else its first parent. gone is
+// true when a stored choice is no longer one of the parents; ok is false
+// when the record has no parent.
+func PrimaryParent(rec *pact.ContentRecord, stored string) (ref pact.ContentRef, ok, gone bool) {
+	if rec == nil || len(rec.Parents) == 0 {
+		return pact.ContentRef{}, false, stored != ""
+	}
+	if stored != "" {
+		for _, p := range rec.Parents {
+			if ParentValue(p) == stored {
+				return p, true, false
+			}
+		}
+		gone = true
+	}
+	return rec.Parents[0], true, gone
+}
+
+// localeOf returns the record's description in locale exactly; ok is
+// false when it has none with an address.
+func localeOf(rec *pact.ContentRecord, locale string) (pact.ContentLocale, bool) {
+	if rec == nil {
+		return pact.ContentLocale{}, false
+	}
+	for _, l := range rec.Locales {
+		if l.Locale == locale && strings.TrimSpace(l.URL) != "" {
+			return l, true
+		}
+	}
+	return pact.ContentLocale{}, false
+}
+
+// HomeCrumbName is the home crumb's name in locale: the setting, else
+// the default copy.
+func (s *Service) HomeCrumbName(ctx context.Context, db *gorm.DB, settings models.Settings, locale string) string {
+	name := strings.TrimSpace(s.translatedSetting(ctx, db, &settings, "breadcrumb_home", settings.BreadcrumbHome, locale))
+	if name == "" {
+		name = s.PhraseIn(locale, "golem15.seo::lang.settings.breadcrumb_home_default", nil)
+	}
+	return name
+}
+
+// homePath is the home page of a locale's address: /<locale> when the
+// page's path carries that prefix, else /.
+func homePath(locale, path string) string {
+	if locale != "" {
+		prefix := "/" + strings.ToLower(locale)
+		if lower := strings.ToLower(path); lower == prefix || strings.HasPrefix(lower, prefix+"/") {
+			return prefix
+		}
+	}
+	return "/"
+}
+
+// parentChain lists a record's parents for its trail, the nearest last:
+// the primary parent (the stored choice, else the first parent), then that
+// parent's primary parent, at most MaxCrumbDepth levels. An unpublished
+// parent or one without an address in locale is left out of the trail but
+// its own parent is still followed; a cycle ends the chain.
+func (s *Service) parentChain(ctx context.Context, db *gorm.DB, rec *pact.ContentRecord, stored, locale string) []parentCrumb {
+	reg := s.Registry()
+	if reg == nil || rec == nil {
+		return nil
+	}
+	var out []parentCrumb
+	seen := map[string]bool{}
+	current, choice := rec, stored
+	for depth := 0; depth < MaxCrumbDepth; depth++ {
+		ref, ok, _ := PrimaryParent(current, choice)
+		if !ok || seen[ParentValue(ref)] {
+			break
+		}
+		seen[ParentValue(ref)] = true
+		parent, err := reg.Describe(ctx, ref.Type, ref.ID, s.UnprefixedLocale())
+		if err != nil || parent == nil {
+			break
+		}
+		if l, has := localeOf(parent, locale); has && parent.Published {
+			out = append(out, parentCrumb{ref: ref, locale: l})
+		}
+		choice = ""
+		if typ, ok := reg.ContentType(ref.Type); ok && typ.Content != nil {
+			if e, found, err := LoadEntry(db, typ.Content.Model, ref.ID, locale); err == nil && found {
+				choice = entryData(e, DataPrimaryParent)
+			}
+		}
+		current = parent
+	}
+	for i, j := 0, len(out)-1; i < j; i, j = i+1, j-1 {
+		out[i], out[j] = out[j], out[i]
+	}
+	return out
+}
+
+// parentCrumb is one parent of a trail.
+type parentCrumb struct {
+	ref    pact.ContentRef
+	locale pact.ContentLocale
+}
+
+// Trail is a page's breadcrumb trail (D-86): the home crumb, the record's
+// parent chain and the page itself, last. The home page's trail is the
+// home crumb only. The breadcrumbs of the resolve answer and the
+// BreadcrumbList node are both this list.
+func (s *Service) Trail(ctx context.Context, db *gorm.DB, settings models.Settings, rec *pact.ContentRecord, stored, locale, path, name string) []schema.Crumb {
+	site := s.SiteURL()
+	home := homePath(locale, path)
+	homeURL := site + home
+	if home == "/" {
+		homeURL = site + "/"
+	}
+	out := []schema.Crumb{{Name: s.HomeCrumbName(ctx, db, settings, locale), URL: homeURL, Path: home}}
+	if path == "/" || path == home {
+		return out
+	}
+	for _, p := range s.parentChain(ctx, db, rec, stored, locale) {
+		title := strings.TrimSpace(p.locale.Title)
+		if title == "" {
+			continue
+		}
+		out = append(out, schema.Crumb{Name: title, URL: AbsoluteURL(site, p.locale.URL), Path: p.locale.URL})
+	}
+	if strings.TrimSpace(name) == "" {
+		name = path
+	}
+	return append(out, schema.Crumb{Name: name, URL: site + path, Path: path})
+}
+
+// ParentOption is one choice of the primary category field (P33-23).
+type ParentOption struct {
+	Value string `json:"value"`
+	Label string `json:"label"`
+}
+
+// ParentOptions lists a record's parents as primary category choices, in
+// descriptor order, labelled with their title in locale (the default
+// locale when "").
+func (s *Service) ParentOptions(ctx context.Context, key, id, locale string) []ParentOption {
+	out := []ParentOption{}
+	reg := s.Registry()
+	if reg == nil || key == "" || id == "" {
+		return out
+	}
+	rec, err := reg.Describe(ctx, key, id, s.UnprefixedLocale())
+	if err != nil || rec == nil {
+		return out
+	}
+	if locale == "" {
+		if db, err := s.DB(ctx); err == nil {
+			locale = s.DefaultLocale(ctx, db)
+		}
+	}
+	for _, ref := range rec.Parents {
+		label := ParentValue(ref)
+		if parent, err := reg.Describe(ctx, ref.Type, ref.ID, s.UnprefixedLocale()); err == nil && parent != nil {
+			if l := contentLocale(parent, locale); strings.TrimSpace(l.Title) != "" {
+				label = l.Title
+			}
+		}
+		out = append(out, ParentOption{Value: ParentValue(ref), Label: label})
+	}
+	return out
+}

classes/breadcrumbs_test.go

@@ -0,0 +1,72 @@
+package classes
+
+import (
+	"testing"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/pact"
+)
+
+// TestPrimaryParent pins D-86: the stored choice while the record still
+// has that parent, else the first parent with gone set for a stale
+// choice, and no parent at all (gone only when a choice was stored).
+func TestPrimaryParent(t *testing.T) {
+	a := pact.ContentRef{Type: "cat", ID: "1"}
+	b := pact.ContentRef{Type: "cat", ID: "2"}
+	rec := &pact.ContentRecord{Parents: []pact.ContentRef{a, b}}
+	for _, tc := range []struct {
+		stored   string
+		want     pact.ContentRef
+		ok, gone bool
+	}{
+		{"", a, true, false}, {"cat:2", b, true, false}, {"cat:9", a, true, true}, {"tag:2", a, true, true},
+	} {
+		got, ok, gone := PrimaryParent(rec, tc.stored)
+		if got != tc.want || ok != tc.ok || gone != tc.gone {
+			t.Errorf("PrimaryParent(%q) = %v %v %v, want %v %v %v", tc.stored, got, ok, gone, tc.want, tc.ok, tc.gone)
+		}
+	}
+	if _, ok, gone := PrimaryParent(&pact.ContentRecord{}, ""); ok || gone {
+		t.Fatal("no parents, nothing stored")
+	}
+	if _, ok, gone := PrimaryParent(nil, "cat:1"); ok || !gone {
+		t.Fatal("no record with a stored choice is gone")
+	}
+	if ParentValue(b) != "cat:2" {
+		t.Fatal("ParentValue is <type>:<id>")
+	}
+}
+
+// TestHomePath pins the home crumb's address: /<locale> when the page is
+// under that prefix (any case), else /.
+func TestHomePath(t *testing.T) {
+	for _, tc := range []struct{ locale, path, want string }{
+		{"pl", "/blog/a", "/"}, {"en", "/en/blog/a", "/en"}, {"en", "/EN", "/en"}, {"en", "/english", "/"}, {"", "/en/a", "/"},
+	} {
+		if got := homePath(tc.locale, tc.path); got != tc.want {
+			t.Errorf("homePath(%q, %q) = %q, want %q", tc.locale, tc.path, got, tc.want)
+		}
+	}
+}
+
+// TestLocaleOfAndEntryData pins the exact locale lookup (an empty address
+// does not count) and the entry data reader, and the depth cap.
+func TestLocaleOfAndEntryData(t *testing.T) {
+	rec := &pact.ContentRecord{Locales: []pact.ContentLocale{{Locale: "pl", URL: "/a"}, {Locale: "en", URL: " "}}}
+	if l, ok := localeOf(rec, "pl"); !ok || l.URL != "/a" {
+		t.Fatal("pl is there")
+	}
+	if _, ok := localeOf(rec, "en"); ok {
+		t.Fatal("en has no address")
+	}
+	if _, ok := localeOf(nil, "pl"); ok {
+		t.Fatal("no record")
+	}
+	e := models.Entry{Data: map[string]any{"primary_parent": " cat:1 ", "n": 5}}
+	if entryData(e, "primary_parent") != "cat:1" || entryData(e, "n") != "" || entryData(models.Entry{}, "x") != "" {
+		t.Fatal("entryData is wrong")
+	}
+	if MaxCrumbDepth != 6 {
+		t.Fatalf("MaxCrumbDepth = %d", MaxCrumbDepth)
+	}
+}

classes/crawl/actions.go

@@ -0,0 +1,309 @@
+package crawl
+
+import (
+	"context"
+	"errors"
+	"net/url"
+	"time"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"gorm.io/gorm"
+	"gorm.io/gorm/clause"
+)
+
+// ErrDeleteRunning refuses deleting a queued or running crawl
+// (crawls.delete_running).
+var ErrDeleteRunning = errors.New("golem15.seo: a running crawl cannot be deleted")
+
+// LoadIssue reads one stored issue.
+func LoadIssue(db *gorm.DB, id uint) (models.Issue, error) {
+	var rows []models.Issue
+	if err := db.Where("id = ?", id).Limit(1).Find(&rows).Error; err != nil {
+		return models.Issue{}, err
+	}
+	if len(rows) == 0 {
+		return models.Issue{}, gorm.ErrRecordNotFound
+	}
+	return rows[0], nil
+}
+
+// IsIgnored reports whether an issue's fingerprint is ignored.
+func IsIgnored(db *gorm.DB, issue models.Issue) (bool, error) {
+	set, err := IgnoredSet(db, []string{issue.Fingerprint})
+	return set[issue.Fingerprint], err
+}
+
+// Ignore hides issues in this and later crawls (UI-SPEC C23): their
+// fingerprints are stored with the administrator and the time. It returns
+// the number of fingerprints newly ignored.
+func Ignore(tx *gorm.DB, issues []models.Issue, adminID *uint) (int, error) {
+	n := 0
+	now := time.Now().UTC()
+	seen := map[string]bool{}
+	for _, i := range issues {
+		if seen[i.Fingerprint] {
+			continue
+		}
+		seen[i.Fingerprint] = true
+		row := models.IssueIgnore{Fingerprint: i.Fingerprint, AdminID: adminID, IgnoredAt: &now}
+		res := tx.Clauses(clause.OnConflict{Columns: []clause.Column{{Name: "fingerprint"}}, DoNothing: true}).Create(&row)
+		if res.Error != nil {
+			return n, res.Error
+		}
+		n += int(res.RowsAffected)
+	}
+	return n, nil
+}
+
+// Unignore shows issues in the audit again; it returns the number of
+// fingerprints no longer ignored.
+func Unignore(tx *gorm.DB, issues []models.Issue) (int, error) {
+	var fps []string
+	for _, i := range issues {
+		fps = append(fps, i.Fingerprint)
+	}
+	if len(fps) == 0 {
+		return 0, nil
+	}
+	res := tx.Where("fingerprint IN ?", fps).Delete(&models.IssueIgnore{})
+	return int(res.RowsAffected), res.Error
+}
+
+// DeleteCrawls deletes crawls with their pages, links and issues; a
+// queued or running one (also one whose cancel is requested) refuses the
+// whole call with ErrDeleteRunning. A running crawl without a heartbeat is
+// ended first (RecoverStale), so it can be deleted.
+func DeleteCrawls(tx *gorm.DB, ids []uint) (int, error) {
+	if len(ids) == 0 {
+		return 0, nil
+	}
+	if _, err := RecoverStale(tx, time.Now()); err != nil {
+		return 0, err
+	}
+	var active int64
+	if err := tx.Model(&models.Crawl{}).Where("id IN ? AND status IN ?", ids, []string{models.CrawlQueued, models.CrawlRunning}).Count(&active).Error; err != nil {
+		return 0, err
+	}
+	if active > 0 {
+		return 0, ErrDeleteRunning
+	}
+	res := tx.Where("id IN ?", ids).Delete(&models.Crawl{})
+	return int(res.RowsAffected), res.Error
+}
+
+// Prune deletes the finished, failed and cancelled crawls beyond the
+// newest keep, with their pages, links and issues.
+func Prune(db *gorm.DB, keep int) error {
+	if keep < 1 {
+		keep = 1
+	}
+	return db.Exec(`DELETE FROM golem15_seo_crawls WHERE status NOT IN ('queued', 'running') AND id NOT IN (
+	SELECT id FROM golem15_seo_crawls WHERE status NOT IN ('queued', 'running') ORDER BY id DESC LIMIT ?)`, keep).Error
+}
+
+// crossPageKinds are the crawl kinds that depend on other pages of the
+// crawl; a recheck of one address keeps them.
+var crossPageKinds = map[string]bool{
+	KindDuplicateTitle: true, KindDuplicateDesc: true, KindSitemapMissing: true,
+	KindOrphanPage: true, KindBrokenLink: true, KindRedirectedLink: true, KindSitemapBlocked: true,
+}
+
+// RecheckResult is what a recheck left for the address.
+type RecheckResult struct {
+	URL       string
+	Remaining int
+}
+
+// Recheck fetches one issue's address again through the crawler client
+// (with its canonical and hreflang targets) and replaces the address's
+// issues in the issue's crawl (UI-SPEC C23): the per-page crawl findings
+// are checked again, broken and redirected link findings are dropped once
+// the address answers 200, and the CMS-data issues of the address are
+// checked again. An issue stored before any crawl refreshes the CMS-data
+// issues. Remaining counts the address's issues afterwards.
+func Recheck(ctx context.Context, svc *classes.Service, tx *gorm.DB, issue models.Issue) (RecheckResult, error) {
+	out := RecheckResult{URL: issue.URL}
+	ctx = classes.WithTx(ctx, tx)
+	cms, err := svc.Issues(ctx)
+	if err != nil {
+		return out, err
+	}
+	var mine []classes.Issue
+	for _, c := range cms {
+		if c.URL == issue.URL {
+			mine = append(mine, c)
+		}
+	}
+	if issue.CrawlID == nil {
+		found := map[string]bool{}
+		for _, c := range mine {
+			found[Fingerprint(c.Kind, c.URL, c.Locale, "")] = true
+		}
+		var existing []models.Issue
+		if err := tx.Where("crawl_id IS NULL AND url = ?", issue.URL).Find(&existing).Error; err != nil {
+			return out, err
+		}
+		have := map[string]bool{}
+		for _, e := range existing {
+			if found[e.Fingerprint] {
+				have[e.Fingerprint] = true
+			} else if err := tx.Delete(&models.Issue{}, e.ID).Error; err != nil {
+				return out, err
+			}
+		}
+		var fresh []classes.Issue
+		for _, c := range mine {
+			if !have[Fingerprint(c.Kind, c.URL, c.Locale, "")] {
+				fresh = append(fresh, c)
+			}
+		}
+		if err := SaveIssues(tx, nil, nil, fresh); err != nil {
+			return out, err
+		}
+		return out, countRemaining(tx, nil, &out)
+	}
+	row, err := LoadCrawl(tx, *issue.CrawlID)
+	if err != nil {
+		return out, err
+	}
+	f, err := NewFetcher(row.Site, svc.Config())
+	if err != nil {
+		return out, err
+	}
+	u, err := url.Parse(f.Absolute(issue.URL))
+	if err != nil || f.Key(u) == "" {
+		return out, classes.ErrBadURL
+	}
+	res := &Result{Pages: map[string]*Page{}, Listed: map[string]string{}, Render: RenderState{Status: renderStatusSkipped}, fetcher: f}
+	fetch := func(key string) *Page {
+		p := &Page{Key: key, Resp: f.Get(ctx, key), FetchedAt: time.Now().UTC()}
+		if p.Resp.Err == "" && p.Resp.Status == 200 && p.Resp.HTML() {
+			base, _ := url.Parse(f.Absolute(key))
+			p.Facts = Parse(base, p.Resp.Body)
+		}
+		p.Resp.Body = nil
+		res.Pages[key] = p
+		return p
+	}
+	var stored []models.CrawlPage
+	if err := tx.Where("crawl_id = ? AND url = ?", row.ID, issue.URL).Limit(1).Find(&stored).Error; err != nil {
+		return out, err
+	}
+	main := fetch(issue.URL)
+	res.Order = []string{issue.URL}
+	if len(stored) > 0 && stored[0].InSitemap {
+		main.InSitemap, main.SitemapFile = true, stored[0].SitemapFile
+		res.Listed[issue.URL] = stored[0].SitemapFile
+	}
+	if main.Facts != nil {
+		var extra []string
+		if c, err := url.Parse(main.Facts.Canonical); err == nil && main.Facts.Canonical != "" {
+			extra = append(extra, f.Key(c))
+		}
+		for _, a := range main.Facts.Alternates {
+			extra = append(extra, f.Key(a.URL))
+		}
+		for _, key := range extra {
+			if key != "" && res.Pages[key] == nil {
+				fetch(key)
+			}
+		}
+	}
+	limits, err := LoadLimits(tx)
+	if err != nil {
+		return out, err
+	}
+	env := &Env{Lookup: Lookup(ctx, svc, tx), Staging: svc.Staging(), GenericAnchors: limits.GenericAnchors}
+	rec := env.record(issue.URL)
+	var findings []Finding
+	findings = append(findings, checkStatus(res, main, rec)...)
+	if main.OK() {
+		findings = append(findings, checkHead(res, main, rec)...)
+		findings = append(findings, checkHeadings(main, rec)...)
+		findings = append(findings, checkJSONLD(main, rec)...)
+		findings = append(findings, checkHreflang(res, main, rec)...)
+		findings = append(findings, checkGenericAnchors(res, main, rec, env.GenericAnchors)...)
+		findings = append(findings, checkImages(main, rec)...)
+		findings = append(findings, checkContentCMS(res, main, rec)...)
+	}
+	for _, sf := range checkSitemaps(res, env) {
+		if sf.URL == issue.URL && sf.Kind != KindSitemapMissing && sf.Kind != KindSitemapBlocked {
+			findings = append(findings, sf)
+		}
+	}
+	var keep []string
+	for kind := range crossPageKinds {
+		if main.Resp.Status == 200 && (kind == KindBrokenLink || kind == KindRedirectedLink) {
+			continue
+		}
+		keep = append(keep, kind)
+	}
+	// The address's issues found again keep their row (and id); the
+	// others are deleted, the new ones stored. Cross-page findings stay,
+	// and so does the rendered comparison, which a recheck does not repeat.
+	found := map[string]bool{}
+	for _, f := range findings {
+		found[Fingerprint(f.Kind, f.URL, f.Locale, f.Check)] = true
+	}
+	for _, c := range mine {
+		found[Fingerprint(c.Kind, c.URL, c.Locale, "")] = true
+	}
+	id := row.ID
+	var existing []models.Issue
+	if err := tx.Where("crawl_id = ? AND url = ?", id, issue.URL).Find(&existing).Error; err != nil {
+		return out, err
+	}
+	keepKind := map[string]bool{}
+	for _, k := range keep {
+		keepKind[k] = true
+	}
+	have := map[string]bool{}
+	var gone []uint
+	for _, e := range existing {
+		stays := found[e.Fingerprint] ||
+			(e.Source == models.SourceCrawl && keepKind[e.Kind]) ||
+			(e.Source == models.SourceCrawl && e.Kind == KindContentJS && e.DetailsMap()["check"] == checkRendered)
+		if stays {
+			have[e.Fingerprint] = true
+		} else {
+			gone = append(gone, e.ID)
+		}
+	}
+	if len(gone) > 0 {
+		if err := tx.Where("id IN ?", gone).Delete(&models.Issue{}).Error; err != nil {
+			return out, err
+		}
+	}
+	var fresh []Finding
+	for _, f := range findings {
+		if !have[Fingerprint(f.Kind, f.URL, f.Locale, f.Check)] {
+			fresh = append(fresh, f)
+		}
+	}
+	var freshCMS []classes.Issue
+	for _, c := range mine {
+		if !have[Fingerprint(c.Kind, c.URL, c.Locale, "")] {
+			freshCMS = append(freshCMS, c)
+		}
+	}
+	if err := SaveIssues(tx, &id, fresh, freshCMS); err != nil {
+		return out, err
+	}
+	return out, countRemaining(tx, &id, &out)
+}
+
+func countRemaining(tx *gorm.DB, crawlID *uint, out *RecheckResult) error {
+	q := tx.Model(&models.Issue{}).Where("url = ?", out.URL)
+	if crawlID == nil {
+		q = q.Where("crawl_id IS NULL")
+	} else {
+		q = q.Where("crawl_id = ?", *crawlID)
+	}
+	var n int64
+	if err := q.Count(&n).Error; err != nil {
+		return err
+	}
+	out.Remaining = int(n)
+	return nil
+}

classes/crawl/checks.go

@@ -0,0 +1,739 @@
+package crawl
+
+import (
+	"encoding/json"
+	"net/url"
+	"sort"
+	"strconv"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes/schema"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"github.com/jimsmart/grobotstxt"
+)
+
+// The crawl kinds of UI-SPEC C23.
+const (
+	KindPageStatus        = "page_status"
+	KindSitemapError      = "sitemap_error"
+	KindSitemapNoindex    = "sitemap_noindex"
+	KindSitemapBlocked    = "sitemap_blocked"
+	KindSitemapMissing    = "sitemap_missing"
+	KindBrokenLink        = "broken_link"
+	KindRedirectedLink    = "redirected_link"
+	KindCanonicalMissing  = "canonical_missing"
+	KindCanonicalError    = "canonical_error"
+	KindMissingTitle      = "missing_title"
+	KindDuplicateTitle    = "duplicate_title"
+	KindMissingDesc       = "missing_description"
+	KindDuplicateDesc     = "duplicate_description"
+	KindH1Missing         = "h1_missing"
+	KindH1Multiple        = "h1_multiple"
+	KindHeadingSkipped    = "heading_skipped"
+	KindHeadingEmpty      = "heading_empty"
+	KindH1TitleMismatch   = "h1_title_mismatch"
+	KindJSONLDInvalid     = "jsonld_invalid"
+	KindJSONLDMultiple    = "jsonld_multiple"
+	KindJSONLDRequired    = "jsonld_required"
+	KindJSONLDRecommended = "jsonld_recommended"
+	KindHreflangRecip     = "hreflang_reciprocal"
+	KindHreflangError     = "hreflang_error"
+	KindGenericAnchor     = "generic_anchor"
+	KindImageAlt          = "image_alt"
+	KindImageSize         = "image_size"
+	KindFeaturedLazy      = "featured_lazy"
+	KindContentJS         = "content_js"
+	KindOrphanPage        = "orphan_page"
+	// KindH1InContent is the CMS-data check of a record whose body holds
+	// an H1 although its page renders the title as the H1.
+	KindH1InContent = "h1_in_content"
+)
+
+// severities are the default severities of the kinds table (UI-SPEC C23),
+// the Phase 32 CMS kinds included.
+var severities = map[string]string{
+	KindPageStatus:          models.SeverityError,
+	KindSitemapError:        models.SeverityError,
+	KindSitemapNoindex:      models.SeverityError,
+	KindSitemapBlocked:      models.SeverityError,
+	KindSitemapMissing:      models.SeverityNotice,
+	KindBrokenLink:          models.SeverityError,
+	KindRedirectedLink:      models.SeverityNotice,
+	KindCanonicalMissing:    models.SeverityWarning,
+	KindCanonicalError:      models.SeverityError,
+	"canonical_noindex":     models.SeverityError,
+	"canonical_redirected":  models.SeverityWarning,
+	KindMissingTitle:        models.SeverityWarning,
+	KindDuplicateTitle:      models.SeverityWarning,
+	KindMissingDesc:         models.SeverityNotice,
+	KindDuplicateDesc:       models.SeverityNotice,
+	"rule_overrides_record": models.SeverityNotice,
+	"redirect_chain":        models.SeverityWarning,
+	"noindex_blocked":       models.SeverityError,
+	KindH1Missing:           models.SeverityError,
+	KindH1Multiple:          models.SeverityError,
+	KindH1InContent:         models.SeverityWarning,
+	KindHeadingSkipped:      models.SeverityWarning,
+	KindHeadingEmpty:        models.SeverityWarning,
+	KindH1TitleMismatch:     models.SeverityNotice,
+	KindJSONLDInvalid:       models.SeverityError,
+	KindJSONLDMultiple:      models.SeverityWarning,
+	KindJSONLDRequired:      models.SeverityError,
+	KindJSONLDRecommended:   models.SeverityNotice,
+	KindHreflangRecip:       models.SeverityWarning,
+	KindHreflangError:       models.SeverityError,
+	KindGenericAnchor:       models.SeverityWarning,
+	KindImageAlt:            models.SeverityWarning,
+	KindImageSize:           models.SeverityNotice,
+	KindFeaturedLazy:        models.SeverityNotice,
+	KindContentJS:           models.SeverityWarning,
+	KindOrphanPage:          models.SeverityWarning,
+}
+
+// Severity returns a kind's severity (warning for an unknown kind).
+func Severity(kind string) string {
+	if s, ok := severities[kind]; ok {
+		return s
+	}
+	return models.SeverityWarning
+}
+
+// Kinds lists every kind the audit stores, in the kinds table order.
+func Kinds() []string {
+	return []string{
+		KindPageStatus, KindSitemapError, KindSitemapNoindex, KindSitemapBlocked,
+		KindSitemapMissing, KindBrokenLink, KindRedirectedLink, KindCanonicalMissing,
+		KindCanonicalError, "canonical_noindex", "canonical_redirected", KindMissingTitle,
+		KindDuplicateTitle, KindMissingDesc, KindDuplicateDesc, "rule_overrides_record",
+		"redirect_chain", "noindex_blocked", KindH1Missing, KindH1Multiple, KindH1InContent,
+		KindHeadingSkipped, KindHeadingEmpty, KindH1TitleMismatch, KindJSONLDInvalid,
+		KindJSONLDMultiple, KindJSONLDRequired, KindJSONLDRecommended, KindHreflangRecip,
+		KindHreflangError, KindGenericAnchor, KindImageAlt, KindImageSize, KindFeaturedLazy,
+		KindContentJS, KindOrphanPage,
+	}
+}
+
+// Env is what the checks know besides the crawl: the record behind an
+// address, whether the site runs outside production (the staging lock's
+// noindex is never reported, RESEARCH Pitfall 12) and the generic link
+// texts.
+type Env struct {
+	// Lookup returns the CMS record of an address, nil when none.
+	Lookup func(key string) *Record
+	// Staging is true outside production.
+	Staging bool
+	// GenericAnchors are the generic link texts (any case).
+	GenericAnchors []string
+	// Keyphrases maps an address to its SEO entry's focus keyphrase.
+	Keyphrases map[string]string
+	// SitemapTool is the tool that edits the sitemap's Additional
+	// addresses (ToolSitemapSettings while the sitemap plugin is mounted,
+	// else ""): a listed noindex address without a CMS record is one of
+	// them.
+	SitemapTool string
+}
+
+func (e *Env) record(key string) *Record {
+	if e == nil || e.Lookup == nil {
+		return nil
+	}
+	return e.Lookup(key)
+}
+
+// locale is the issue locale of a page: its record's.
+func locale(rec *Record) string {
+	if rec == nil {
+		return ""
+	}
+	return rec.Locale
+}
+
+// noindex reports whether a page asks not to be indexed (meta robots or
+// X-Robots-Tag). On a non-production environment every page carries the
+// staging lock's noindex, so it is never counted there.
+func (e *Env) noindex(p *Page) bool {
+	if e != nil && e.Staging {
+		return false
+	}
+	if p.Facts != nil && p.Facts.Noindex() {
+		return true
+	}
+	return p.Resp != nil && hasNoindex(p.Resp.XRobots)
+}
+
+// selfCanonical reports whether a page's canonical is missing or names
+// the page itself.
+func selfCanonical(res *Result, p *Page) bool {
+	if p.Facts == nil || p.Facts.Canonical == "" || res.fetcher == nil {
+		return true
+	}
+	u, err := url.Parse(p.Facts.Canonical)
+	if err != nil {
+		return true
+	}
+	return res.fetcher.Key(u) == p.Key
+}
+
+// indexable reports whether a page answers 200 with HTML, may be indexed
+// and is its own canonical.
+func (e *Env) indexable(res *Result, p *Page) bool {
+	return p.OK() && !e.noindex(p) && selfCanonical(res, p)
+}
+
+// Check runs every check over a crawl's pages: the per-page checks of
+// UI-SPEC C23 and the site-wide ones (sitemaps, statuses, duplicates).
+func Check(res *Result, env *Env) []Finding {
+	if env == nil {
+		env = &Env{}
+	}
+	var out []Finding
+	for _, key := range res.Order {
+		p := res.Pages[key]
+		rec := env.record(key)
+		out = append(out, checkStatus(res, p, rec)...)
+		if !p.OK() {
+			continue
+		}
+		out = append(out, checkHead(res, p, rec)...)
+		out = append(out, checkHeadings(p, rec)...)
+		out = append(out, checkJSONLD(p, rec)...)
+		out = append(out, checkHreflang(res, p, rec)...)
+		out = append(out, checkGenericAnchors(res, p, rec, env.GenericAnchors)...)
+		out = append(out, checkImages(p, rec)...)
+		out = append(out, checkContentCMS(res, p, rec)...)
+		out = append(out, checkRenderedPage(res, p, rec)...)
+	}
+	out = append(out, checkLinks(res, env)...)
+	out = append(out, checkOrphans(res, env)...)
+	out = append(out, checkSitemaps(res, env)...)
+	out = append(out, checkDuplicates(res, env)...)
+	return out
+}
+
+// checkStatus reports page_status for an address that answers an error
+// (other than 404 and 410, which are broken links when linked) or cannot
+// be fetched.
+func checkStatus(res *Result, p *Page, rec *Record) []Finding {
+	r := p.Resp
+	if r == nil {
+		return nil
+	}
+	switch {
+	case r.Err != "" && r.Err != "canceled":
+		return []Finding{{Kind: KindPageStatus, URL: p.Key, Locale: locale(rec), Record: rec, Tool: ToolCreateRedirect, Details: map[string]any{"status": 0, "error": r.Err}}}
+	case r.Status >= 400 && r.Status != 404 && r.Status != 410:
+		return []Finding{{Kind: KindPageStatus, URL: p.Key, Locale: locale(rec), Record: rec, Tool: ToolCreateRedirect, Details: map[string]any{"status": r.Status}}}
+	}
+	return nil
+}
+
+// checkHead reports a missing title, description or canonical and a
+// canonical pointing to an error page.
+func checkHead(res *Result, p *Page, rec *Record) []Finding {
+	var out []Finding
+	f := p.Facts
+	loc := locale(rec)
+	if strings.TrimSpace(f.Title) == "" {
+		out = append(out, Finding{Kind: KindMissingTitle, URL: p.Key, Locale: loc, Record: rec, Tool: ToolSetURLSEO})
+	}
+	if strings.TrimSpace(f.Description) == "" {
+		out = append(out, Finding{Kind: KindMissingDesc, URL: p.Key, Locale: loc, Record: rec, Tool: ToolSetURLSEO})
+	}
+	if f.Canonical == "" {
+		out = append(out, Finding{Kind: KindCanonicalMissing, URL: p.Key, Locale: loc, Record: rec})
+	} else if u, err := url.Parse(f.Canonical); err == nil && res.fetcher != nil {
+		if key := res.fetcher.Key(u); key != "" && key != p.Key {
+			if t := res.Page(key); t != nil && t.Resp != nil && (t.Resp.Err != "" || t.Resp.Status >= 400) {
+				out = append(out, Finding{Kind: KindCanonicalError, URL: p.Key, Locale: loc, Record: rec, Tool: ToolSetURLSEO, Details: map[string]any{"url": key, "status": t.Resp.Status}})
+			}
+		}
+	}
+	if h1 := f.H1(); len(h1) > 0 && strings.TrimSpace(f.Title) != "" && strings.TrimSpace(h1[0]) != "" {
+		a, b := normal(h1[0]), normal(f.Title)
+		if !strings.Contains(b, a) && !strings.Contains(a, b) {
+			out = append(out, Finding{Kind: KindH1TitleMismatch, URL: p.Key, Locale: loc, Record: rec, Tool: ToolSetURLSEO, Details: map[string]any{"h1": h1[0], "title": f.Title}})
+		}
+	}
+	return out
+}
+
+// checkHeadings reports h1_missing, h1_multiple, heading_skipped and
+// heading_empty. A heading inside the record's own content is fixed
+// through the record's tool: H1 findings prefer Tools[headings], the
+// others use Tools[edit] (P33-38).
+func checkHeadings(p *Page, rec *Record) []Finding {
+	var out []Finding
+	loc := locale(rec)
+	h1 := p.Facts.H1()
+	switch {
+	case len(h1) == 0:
+		out = append(out, Finding{Kind: KindH1Missing, URL: p.Key, Locale: loc, Record: rec})
+	case len(h1) > 1:
+		f := Finding{Kind: KindH1Multiple, URL: p.Key, Locale: loc, Record: rec, Details: map[string]any{"count": len(h1), "h1": h1}}
+		if rec.hasHeading(1, "") {
+			f.Content = []string{pact.ContentToolHeadings, pact.ContentToolEdit}
+		}
+		out = append(out, f)
+	}
+	var skips []map[string]any
+	var first *Finding
+	prev := 0
+	for _, h := range p.Facts.Headings {
+		if prev > 0 && h.Level > prev+1 {
+			skips = append(skips, map[string]any{"from": prev, "to": h.Level, "text": h.Text})
+			if first == nil {
+				first = &Finding{Kind: KindHeadingSkipped, URL: p.Key, Locale: loc, Record: rec, Details: map[string]any{"from": prev, "to": h.Level, "text": h.Text}}
+				if rec.hasHeading(h.Level, h.Text) {
+					first.Content = []string{pact.ContentToolEdit}
+				}
+			}
+		}
+		prev = h.Level
+	}
+	if first != nil {
+		first.Details["skips"] = skips
+		first.Details["count"] = len(skips)
+		out = append(out, *first)
+	}
+	empty := 0
+	var firstEmpty *Finding
+	for _, h := range p.Facts.Headings {
+		if strings.TrimSpace(h.Text) != "" {
+			continue
+		}
+		empty++
+		if firstEmpty == nil {
+			firstEmpty = &Finding{Kind: KindHeadingEmpty, URL: p.Key, Locale: loc, Record: rec, Details: map[string]any{"level": h.Level}}
+			if hasEmptyHeading(rec, h.Level) {
+				if h.Level == 1 {
+					firstEmpty.Content = []string{pact.ContentToolHeadings, pact.ContentToolEdit}
+				} else {
+					firstEmpty.Content = []string{pact.ContentToolEdit}
+				}
+			}
+		}
+	}
+	if firstEmpty != nil {
+		firstEmpty.Details["count"] = empty
+		out = append(out, *firstEmpty)
+	}
+	return out
+}
+
+// hasEmptyHeading reports whether the record's own headings hold an empty
+// one of level.
+func hasEmptyHeading(rec *Record, level int) bool {
+	if rec == nil {
+		return false
+	}
+	for _, h := range rec.Headings {
+		if h.Level == level && strings.TrimSpace(h.Text) == "" {
+			return true
+		}
+	}
+	return false
+}
+
+// sitemapNoindex is the finding of a listed noindex page. A CMS record's
+// page is fixed with seo_set_url_seo; an address without a record is an
+// Additional address of the sitemap (or another listed address the SEO
+// plugin cannot change), fixed by removing it from the sitemap's address
+// list: env.SitemapTool, else fix code naming that list.
+func sitemapNoindex(key, file string, rec *Record, env *Env) Finding {
+	f := Finding{Kind: KindSitemapNoindex, URL: key, Locale: locale(rec), Record: rec, Details: map[string]any{"file": file}}
+	if rec != nil {
+		f.Tool = ToolSetURLSEO
+		return f
+	}
+	f.Details["extra"] = true
+	if env != nil {
+		f.Tool = env.SitemapTool
+	}
+	return f
+}
+
+// checkSitemaps compares the sitemaps with what the crawl found:
+// sitemap_error for a listed address that does not answer 200,
+// sitemap_noindex for a listed noindex page, sitemap_blocked for a listed
+// address robots.txt keeps Googlebot from, and sitemap_missing for an
+// indexable page found only through links.
+func checkSitemaps(res *Result, env *Env) []Finding {
+	var out []Finding
+	var keys []string
+	for key := range res.Listed {
+		keys = append(keys, key)
+	}
+	sort.Strings(keys)
+	robots := res.Robots
+	for _, key := range keys {
+		file := res.Listed[key]
+		rec := env.record(key)
+		if p := res.Page(key); p != nil && p.Resp != nil {
+			switch {
+			case p.Resp.Err == "canceled":
+			case p.Resp.Err != "" || p.Resp.Status != 200:
+				out = append(out, Finding{Kind: KindSitemapError, URL: key, Locale: locale(rec), Record: rec, Tool: ToolSetURLSEO, Details: map[string]any{"status": p.Resp.Status, "file": file}})
+			case env.noindex(p):
+				out = append(out, sitemapNoindex(key, file, rec, env))
+			}
+		}
+		if robots != "" && res.fetcher != nil {
+			m := grobotstxt.NewRobotsMatcher()
+			if !m.AgentAllowed(robots, "Googlebot", res.fetcher.Absolute(key)) {
+				line := m.MatchingLine()
+				rule := ""
+				if lines := strings.Split(robots, "\n"); line > 0 && line <= len(lines) {
+					rule = strings.TrimSpace(lines[line-1])
+				}
+				out = append(out, Finding{Kind: KindSitemapBlocked, URL: key, Locale: locale(rec), Record: rec, Tool: ToolUpdateRobots, Details: map[string]any{"line": line, "rule": rule, "file": file}})
+			}
+		}
+	}
+	links := inbound(res)
+	for _, key := range res.Order {
+		p := res.Pages[key]
+		if _, listed := res.Listed[key]; listed || !env.indexable(res, p) {
+			continue
+		}
+		rec := env.record(key)
+		f := Finding{Kind: KindSitemapMissing, URL: key, Locale: locale(rec), Record: rec, Tool: ToolCreatePage, Details: map[string]any{"count": len(links[key])}}
+		if rec != nil {
+			f.Tool = ToolSetURLSEO
+		}
+		out = append(out, f)
+	}
+	return out
+}
+
+// inbound maps each address to the pages linking to it (each once, a page
+// linking to itself left out), in crawl order.
+func inbound(res *Result) map[string][]string {
+	out := map[string][]string{}
+	if res.fetcher == nil {
+		return out
+	}
+	for _, from := range res.Order {
+		p := res.Pages[from]
+		if p.Facts == nil {
+			continue
+		}
+		seen := map[string]bool{}
+		for _, l := range p.Facts.Links {
+			to := res.fetcher.Key(l.URL)
+			if to == "" || to == from || seen[to] {
+				continue
+			}
+			seen[to] = true
+			out[to] = append(out[to], from)
+		}
+	}
+	return out
+}
+
+// checkDuplicates reports titles and descriptions shared by indexable
+// pages of the crawl (compared without case and extra spaces).
+func checkDuplicates(res *Result, env *Env) []Finding {
+	titles := map[string][]string{}
+	descs := map[string][]string{}
+	for _, key := range res.Order {
+		p := res.Pages[key]
+		if !env.indexable(res, p) {
+			continue
+		}
+		if t := normal(p.Facts.Title); t != "" {
+			titles[t] = append(titles[t], key)
+		}
+		if d := normal(p.Facts.Description); d != "" {
+			descs[d] = append(descs[d], key)
+		}
+	}
+	var out []Finding
+	emit := func(groups map[string][]string, kind string) {
+		var values []string
+		for v := range groups {
+			values = append(values, v)
+		}
+		sort.Strings(values)
+		for _, v := range values {
+			list := groups[v]
+			if len(list) < 2 {
+				continue
+			}
+			for _, key := range list {
+				others := make([]string, 0, len(list)-1)
+				for _, o := range list {
+					if o != key {
+						others = append(others, o)
+					}
+				}
+				rec := env.record(key)
+				out = append(out, Finding{Kind: kind, URL: key, Locale: locale(rec), Record: rec, Tool: ToolSetURLSEO, Details: map[string]any{"value": v, "count": len(others), "also": others}})
+			}
+		}
+	}
+	emit(titles, KindDuplicateTitle)
+	emit(descs, KindDuplicateDesc)
+	return out
+}
+
+// publisherTypes are the node types the SEO settings generate (publisher,
+// local business, the site): their missing properties are fixed through
+// seo_update_settings.
+var publisherTypes = map[string]bool{
+	"Organization": true, "Person": true, "WebSite": true, "LocalBusiness": true,
+	"ProfessionalService": true, "Store": true, "FoodEstablishment": true,
+	"HealthAndBeautyBusiness": true, "MedicalBusiness": true, "LegalService": true,
+	"HomeAndConstructionBusiness": true, "AutomotiveBusiness": true, "FinancialService": true,
+}
+
+// checkJSONLD reads every JSON-LD script of a page: jsonld_invalid for one
+// that is not JSON, jsonld_multiple for more than one script, and the
+// graph validated with the rules of the admin preview (schema.Validate,
+// D-46): jsonld_required and jsonld_recommended. Types without Google
+// rules (FAQPage and the like) find nothing.
+func checkJSONLD(p *Page, rec *Record) []Finding {
+	scripts := p.Facts.JSONLD
+	if len(scripts) == 0 {
+		return nil
+	}
+	loc := locale(rec)
+	var out []Finding
+	var errs []string
+	var nodes []any
+	for _, raw := range scripts {
+		var v any
+		dec := json.NewDecoder(strings.NewReader(raw))
+		dec.UseNumber()
+		if err := dec.Decode(&v); err != nil {
+			errs = append(errs, err.Error())
+			continue
+		}
+		switch x := v.(type) {
+		case map[string]any:
+			if g, ok := x["@graph"].([]any); ok {
+				nodes = append(nodes, g...)
+			} else {
+				nodes = append(nodes, x)
+			}
+		case []any:
+			nodes = append(nodes, x...)
+		}
+	}
+	if len(errs) > 0 {
+		f := Finding{Kind: KindJSONLDInvalid, URL: p.Key, Locale: loc, Record: rec, Details: map[string]any{"error": errs[0], "count": len(errs)}}
+		if rec != nil && rec.CustomJSONLD {
+			f.Tool = ToolSetURLSEO
+		}
+		out = append(out, f)
+	}
+	if len(scripts) > 1 {
+		out = append(out, Finding{Kind: KindJSONLDMultiple, URL: p.Key, Locale: loc, Record: rec, Details: map[string]any{"count": len(scripts)}})
+	}
+	if len(nodes) == 0 {
+		return out
+	}
+	type miss struct{ typ, prop string }
+	var required, recommended []miss
+	for _, rep := range schema.Validate(map[string]any{"@graph": nodes}) {
+		for _, f := range rep.Findings {
+			m := miss{rep.Type, f.Property}
+			if f.Level == schema.LevelRequired {
+				required = append(required, m)
+			} else {
+				recommended = append(recommended, m)
+			}
+		}
+	}
+	emit := func(kind string, list []miss) {
+		if len(list) == 0 {
+			return
+		}
+		all := make([]map[string]any, 0, len(list))
+		for _, m := range list {
+			all = append(all, map[string]any{"type": m.typ, "property": m.prop})
+		}
+		tool := ToolSetURLSEO
+		if publisherTypes[list[0].typ] {
+			tool = ToolUpdateSettings
+		}
+		out = append(out, Finding{Kind: kind, URL: p.Key, Locale: loc, Record: rec, Tool: tool, Details: map[string]any{
+			"type": list[0].typ, "property": list[0].prop, "count": len(list), "missing": all,
+		}})
+	}
+	emit(KindJSONLDRequired, required)
+	emit(KindJSONLDRecommended, recommended)
+	return out
+}
+
+// checkHreflang follows a page's hreflang alternates: hreflang_error for
+// an alternate that does not answer 200, hreflang_reciprocal for one that
+// answers but does not link back.
+func checkHreflang(res *Result, p *Page, rec *Record) []Finding {
+	if res.fetcher == nil {
+		return nil
+	}
+	loc := locale(rec)
+	var broken, oneWay []map[string]any
+	for _, a := range p.Facts.Alternates {
+		key := res.fetcher.Key(a.URL)
+		if key == "" || key == p.Key || strings.EqualFold(a.Lang, "x-default") {
+			continue
+		}
+		t := res.Page(key)
+		if t == nil || t.Resp == nil || t.Resp.Err == "canceled" {
+			continue
+		}
+		if !t.OK() {
+			broken = append(broken, map[string]any{"locale": a.Lang, "url": key, "status": t.Resp.Status})
+			continue
+		}
+		back := false
+		for _, b := range t.Facts.Alternates {
+			if res.fetcher.Key(b.URL) == p.Key {
+				back = true
+				break
+			}
+		}
+		if !back {
+			oneWay = append(oneWay, map[string]any{"locale": a.Lang, "url": key})
+		}
+	}
+	var out []Finding
+	if len(broken) > 0 {
+		d := map[string]any{"count": len(broken), "alternates": broken}
+		for k, v := range broken[0] {
+			d[k] = v
+		}
+		out = append(out, Finding{Kind: KindHreflangError, URL: p.Key, Locale: loc, Record: rec, Tool: ToolCreateRedirect, Details: d})
+	}
+	if len(oneWay) > 0 {
+		d := map[string]any{"count": len(oneWay), "alternates": oneWay}
+		for k, v := range oneWay[0] {
+			d[k] = v
+		}
+		f := Finding{Kind: KindHreflangRecip, URL: p.Key, Locale: loc, Record: rec, Details: d}
+		if rec != nil {
+			f.Content = []string{pact.ContentToolEdit}
+		}
+		out = append(out, f)
+	}
+	return out
+}
+
+// maxListed caps the images a finding names.
+const maxListed = 10
+
+// checkImages reports image_alt (an <img> without an alt attribute; an
+// empty alt is a decorative image), image_size (no width or height) and
+// featured_lazy (the og:image loads lazily). An image the record lists is
+// fixed through the record's image_alt tool.
+func checkImages(p *Page, rec *Record) []Finding {
+	loc := locale(rec)
+	var noAlt, noSize []string
+	var lazy string
+	recordImage := false
+	for _, img := range p.Facts.Images {
+		if !img.HasAlt {
+			noAlt = append(noAlt, img.Src)
+			if rec != nil {
+				for _, ri := range rec.Images {
+					if ri.URL != "" && (strings.HasSuffix(img.Src, ri.URL) || ri.URL == img.Src) {
+						recordImage = true
+					}
+				}
+			}
+		}
+		if strings.TrimSpace(img.Width) == "" || strings.TrimSpace(img.Height) == "" {
+			noSize = append(noSize, img.Src)
+		}
+		if p.Facts.OGImage != "" && img.Src == p.Facts.OGImage && img.Loading == "lazy" {
+			lazy = img.Src
+		}
+	}
+	cut := func(list []string) []string {
+		if len(list) > maxListed {
+			return list[:maxListed]
+		}
+		return list
+	}
+	var out []Finding
+	if len(noAlt) > 0 {
+		f := Finding{Kind: KindImageAlt, URL: p.Key, Locale: loc, Record: rec, Details: map[string]any{"count": len(noAlt), "images": cut(noAlt)}}
+		if recordImage {
+			f.Content = []string{pact.ContentToolImageAlt, pact.ContentToolEdit}
+		}
+		out = append(out, f)
+	}
+	if len(noSize) > 0 {
+		out = append(out, Finding{Kind: KindImageSize, URL: p.Key, Locale: loc, Record: rec, Details: map[string]any{"count": len(noSize), "images": cut(noSize)}})
+	}
+	if lazy != "" {
+		out = append(out, Finding{Kind: KindFeaturedLazy, URL: p.Key, Locale: loc, Record: rec, Details: map[string]any{"image": lazy}})
+	}
+	return out
+}
+
+// checkContentCMS compares the server-rendered HTML with what the CMS
+// record says (D-70, D-71; check cms): the record title must appear in an
+// H1, the text must reach half of the record's word count, and the
+// record's CMS-known links must be present. A miss is content_js.
+func checkContentCMS(res *Result, p *Page, rec *Record) []Finding {
+	if rec == nil {
+		return nil
+	}
+	var what []string
+	details := map[string]any{}
+	if title := normal(rec.Title); title != "" {
+		found := false
+		for _, h := range p.Facts.H1() {
+			if strings.Contains(normal(h), title) {
+				found = true
+			}
+		}
+		if !found {
+			details["title"] = rec.Title
+			what = append(what, "the title “"+rec.Title+"” in the H1")
+		}
+	}
+	if rec.Words != nil && *rec.Words > 0 && p.Facts.Words*2 < *rec.Words {
+		details["words_raw"] = p.Facts.Words
+		details["words_cms"] = *rec.Words
+		what = append(what, strconv.Itoa(p.Facts.Words)+" of "+strconv.Itoa(*rec.Words)+" words")
+	}
+	if len(rec.Links) > 0 && res.fetcher != nil {
+		have := map[string]bool{}
+		for _, l := range p.Facts.Links {
+			if k := res.fetcher.Key(l.URL); k != "" {
+				have[k] = true
+			}
+		}
+		var missing []string
+		for _, l := range rec.Links {
+			if !have[l] {
+				missing = append(missing, l)
+			}
+		}
+		if len(missing) > 0 {
+			details["links"] = missing
+			what = append(what, "links "+strings.Join(missing, ", "))
+		}
+	}
+	if len(what) == 0 {
+		return nil
+	}
+	return []Finding{{Kind: KindContentJS, URL: p.Key, Locale: locale(rec), Record: rec, Check: checkCMS, Details: details, What: strings.Join(what, "; ")}}
+}
+
+// checkRenderedPage runs the raw versus rendered comparison of a page the
+// renderer rendered.
+func checkRenderedPage(res *Result, p *Page, rec *Record) []Finding {
+	if p.Rendered == nil || res.fetcher == nil {
+		return nil
+	}
+	out := compareRendered(p.Facts, p.Rendered, res.fetcher.Key)
+	for i := range out {
+		out[i].URL, out[i].Locale, out[i].Record = p.Key, locale(rec), rec
+	}
+	return out
+}

classes/crawl/checks_test.go

@@ -0,0 +1,252 @@
+package crawl
+
+import (
+	"testing"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/pact"
+)
+
+func okPage(key string, facts *PageFacts) *Page {
+	return &Page{Key: key, Resp: &Response{Status: 200, ContentType: "text/html"}, Facts: facts}
+}
+
+func kinds(fs []Finding) map[string]Finding {
+	out := map[string]Finding{}
+	for _, f := range fs {
+		out[f.Kind] = f
+	}
+	return out
+}
+
+// TestSeverityAndKinds pins the kinds table: every listed kind has a
+// severity, an unknown kind is a warning, and each kind is listed once.
+func TestSeverityAndKinds(t *testing.T) {
+	seen := map[string]bool{}
+	for _, k := range Kinds() {
+		if seen[k] {
+			t.Fatalf("%s is listed twice", k)
+		}
+		seen[k] = true
+		if _, ok := severities[k]; !ok {
+			t.Errorf("%s has no severity", k)
+		}
+	}
+	if len(seen) != len(severities) {
+		t.Fatalf("%d kinds listed, %d severities", len(seen), len(severities))
+	}
+	if Severity("nope") != models.SeverityWarning || Severity(KindPageStatus) != models.SeverityError || Severity(KindSitemapMissing) != models.SeverityNotice {
+		t.Fatal("Severity is wrong")
+	}
+}
+
+// TestCheckStatus pins page_status: a fetch error (not a cancel) and an
+// error status other than 404 and 410; nothing for 200, 301, 404, 410.
+func TestCheckStatus(t *testing.T) {
+	for _, tc := range []struct {
+		resp *Response
+		want bool
+	}{
+		{&Response{Err: "timeout"}, true}, {&Response{Err: "canceled"}, false}, {&Response{Status: 500}, true}, {&Response{Status: 403}, true},
+		{&Response{Status: 404}, false}, {&Response{Status: 410}, false}, {&Response{Status: 200}, false}, {&Response{Status: 301}, false}, {nil, false},
+	} {
+		got := checkStatus(&Result{}, &Page{Key: "/a", Resp: tc.resp}, nil)
+		if (len(got) == 1) != tc.want || (tc.want && (got[0].Kind != KindPageStatus || got[0].Tool != ToolCreateRedirect)) {
+			t.Errorf("%+v: %+v", tc.resp, got)
+		}
+	}
+}
+
+// TestCheckHead pins missing title, description and canonical (positive)
+// and their absence on a complete head (negative), and the H1/title
+// mismatch only when neither contains the other.
+func TestCheckHead(t *testing.T) {
+	bad := kinds(checkHead(&Result{}, okPage("/a", &PageFacts{Headings: []Heading{{1, "Kanię"}}, Title: "Grzyby"}), nil))
+	for _, k := range []string{KindMissingDesc, KindCanonicalMissing, KindH1TitleMismatch} {
+		if _, ok := bad[k]; !ok {
+			t.Errorf("missing %s in %v", k, bad)
+		}
+	}
+	if bad[KindMissingDesc].Tool != ToolSetURLSEO {
+		t.Fatal("a missing description is fixed with seo_set_url_seo")
+	}
+	if _, ok := kinds(checkHead(&Result{}, okPage("/a", &PageFacts{}), nil))[KindMissingTitle]; !ok {
+		t.Fatal("no missing_title for an empty title")
+	}
+	good := checkHead(&Result{}, okPage("/a", &PageFacts{Title: "Grzyby lecznicze | Acme", Description: "d", Canonical: "https://example.test/a", Headings: []Heading{{1, "Grzyby  LECZNICZE"}}}), nil)
+	if len(good) != 0 {
+		t.Fatalf("a complete head: %+v", good)
+	}
+}
+
+// TestCheckHeadings pins h1_missing, h1_multiple, heading_skipped (first
+// skip with every skip counted) and heading_empty, and the fitting tool:
+// an H1 in the record's content routes to the headings tool, a skipped
+// level the record holds to the edit tool, a template heading to code.
+func TestCheckHeadings(t *testing.T) {
+	rec := &Record{Headings: []pact.ContentHeading{{Level: 1, Text: "In body"}, {Level: 4, Text: "Deep"}}, Tools: map[string]string{pact.ContentToolHeadings: "journal_fix_headings", pact.ContentToolEdit: "journal_update_post"}}
+	got := kinds(checkHeadings(okPage("/a", &PageFacts{Headings: []Heading{{1, "Title"}, {1, "In body"}, {2, "A"}, {4, "Deep"}, {2, "B"}, {5, "Deeper"}, {3, ""}}}), rec))
+	multi, skip, empty := got[KindH1Multiple], got[KindHeadingSkipped], got[KindHeadingEmpty]
+	if multi.Details["count"] != 2 || len(multi.Content) != 2 || multi.Content[0] != pact.ContentToolHeadings {
+		t.Fatalf("h1_multiple = %+v", multi)
+	}
+	if fix, tool, _ := Route(multi); fix != models.FixTool || tool != "journal_fix_headings" {
+		t.Fatalf("h1_multiple routes to %s %s, want the headings tool", fix, tool)
+	}
+	if skip.Details["from"] != 2 || skip.Details["to"] != 4 || skip.Details["count"] != 2 || len(skip.Content) != 1 || skip.Content[0] != pact.ContentToolEdit {
+		t.Fatalf("heading_skipped = %+v", skip)
+	}
+	if fix, tool, _ := Route(skip); fix != models.FixTool || tool != "journal_update_post" {
+		t.Fatalf("heading_skipped routes to %s %s, want the edit tool", fix, tool)
+	}
+	if empty.Details["level"] != 3 || empty.Details["count"] != 1 || len(empty.Content) != 0 {
+		t.Fatalf("heading_empty = %+v", empty)
+	}
+	if fix, _, what := Route(empty); fix != models.FixCode || what == "" {
+		t.Fatalf("a template heading routes to code, got %s %q", fix, what)
+	}
+	// A category's skipped level the record does not hold is fixed in code.
+	cat := &Record{Tools: map[string]string{pact.ContentToolEdit: "journal_update_category"}}
+	skipCat := kinds(checkHeadings(okPage("/c", &PageFacts{Headings: []Heading{{1, "Cat"}, {3, "Post card"}}}), cat))[KindHeadingSkipped]
+	if fix, _, what := Route(skipCat); fix != models.FixCode || what != defaultWhat[KindHeadingSkipped] {
+		t.Fatalf("category skip routes to %s %q, want code", fix, what)
+	}
+	if _, ok := kinds(checkHeadings(okPage("/b", &PageFacts{}), nil))[KindH1Missing]; !ok {
+		t.Fatal("no h1_missing")
+	}
+	if clean := checkHeadings(okPage("/b", &PageFacts{Headings: []Heading{{1, "T"}, {2, "a"}, {3, "b"}, {2, "c"}}}), nil); len(clean) != 0 {
+		t.Fatalf("clean headings: %+v", clean)
+	}
+}
+
+// TestCheckJSONLD pins jsonld_invalid (the custom tool when the record
+// stores custom JSON-LD), jsonld_multiple, jsonld_required and
+// jsonld_recommended (the settings tool for a publisher node), and nothing
+// for a valid single graph.
+func TestCheckJSONLD(t *testing.T) {
+	got := kinds(checkJSONLD(okPage("/a", &PageFacts{JSONLD: []string{`{"@type":`, `{"@graph":[{"@type":"BlogPosting","headline":"h"},{"@type":"Organization"}]}`}}), &Record{CustomJSONLD: true}))
+	if got[KindJSONLDInvalid].Tool != ToolSetURLSEO || got[KindJSONLDMultiple].Details["count"] != 2 {
+		t.Fatalf("invalid/multiple = %+v", got)
+	}
+	if rec := got[KindJSONLDRecommended]; rec.Details["type"] != "BlogPosting" || rec.Tool != ToolSetURLSEO {
+		t.Fatalf("recommended = %+v", rec)
+	}
+	pub := kinds(checkJSONLD(okPage("/a", &PageFacts{JSONLD: []string{`[{"@type":"Person"}]`}}), nil))[KindJSONLDRequired]
+	if pub.Details["property"] != "name" || pub.Tool != ToolUpdateSettings {
+		t.Fatalf("publisher required = %+v", pub)
+	}
+	if none := checkJSONLD(okPage("/a", &PageFacts{JSONLD: []string{`{"@type":"WebSite","name":"n","url":"u"}`}}), nil); len(none) != 0 {
+		t.Fatalf("a valid graph: %+v", none)
+	}
+	if none := checkJSONLD(okPage("/a", &PageFacts{}), nil); none != nil {
+		t.Fatal("no scripts, no findings")
+	}
+}
+
+// TestCheckImages pins image_alt (an empty alt is decorative, not a
+// finding; a record image routes to the image_alt tool), image_size and
+// featured_lazy, and the list cap.
+func TestCheckImages(t *testing.T) {
+	facts := &PageFacts{OGImage: "https://example.test/m/cover.jpg", Images: []Image{
+		{Src: "https://example.test/m/cover.jpg", Alt: "c", HasAlt: true, Width: "10", Height: "10", Loading: "lazy"},
+		{Src: "https://example.test/m/a.jpg", HasAlt: false, Width: "1", Height: "1"},
+		{Src: "https://example.test/m/deco.png", Alt: "", HasAlt: true},
+	}}
+	rec := &Record{Images: []pact.ContentImage{{URL: "/m/a.jpg"}}, Tools: map[string]string{pact.ContentToolImageAlt: "journal_set_image_alt"}}
+	got := kinds(checkImages(okPage("/a", facts), rec))
+	if alt := got[KindImageAlt]; alt.Details["count"] != 1 || len(alt.Content) != 2 {
+		t.Fatalf("image_alt = %+v", alt)
+	}
+	if _, tool, _ := Route(got[KindImageAlt]); tool != "journal_set_image_alt" {
+		t.Fatalf("image_alt routes to %q", tool)
+	}
+	if got[KindImageSize].Details["count"] != 1 || got[KindFeaturedLazy].Details["image"] != "https://example.test/m/cover.jpg" {
+		t.Fatalf("size/lazy = %+v", got)
+	}
+	many := &PageFacts{}
+	for i := 0; i < 15; i++ {
+		many.Images = append(many.Images, Image{Src: "/x.jpg", Width: "1", Height: "1"})
+	}
+	if alt := kinds(checkImages(okPage("/b", many), nil))[KindImageAlt]; alt.Details["count"] != 15 || len(alt.Details["images"].([]string)) != maxListed || len(alt.Content) != 0 {
+		t.Fatalf("capped image_alt = %+v", alt)
+	}
+}
+
+// TestCheckContentCMS pins content_js (check cms): the record title
+// missing from the H1, under half the record's words; nothing when both
+// are met or without a record.
+func TestCheckContentCMS(t *testing.T) {
+	words := 100
+	rec := &Record{Title: "Chaga", Words: &words}
+	got := checkContentCMS(&Result{}, okPage("/a", &PageFacts{Headings: []Heading{{1, "Other"}}, Words: 40}), rec)
+	if len(got) != 1 || got[0].Kind != KindContentJS || got[0].Check != checkCMS || got[0].Details["words_cms"] != 100 || got[0].Details["title"] != "Chaga" {
+		t.Fatalf("content_js = %+v", got)
+	}
+	if ok := checkContentCMS(&Result{}, okPage("/a", &PageFacts{Headings: []Heading{{1, "Chaga — grzyb"}}, Words: 50}), rec); ok != nil {
+		t.Fatalf("met: %+v", ok)
+	}
+	if none := checkContentCMS(&Result{}, okPage("/a", &PageFacts{}), nil); none != nil {
+		t.Fatal("no record, no finding")
+	}
+}
+
+// TestEnvNoindex pins the staging lock: on a non-production environment
+// no page counts as noindex; otherwise meta robots and X-Robots-Tag do.
+func TestEnvNoindex(t *testing.T) {
+	meta := okPage("/a", &PageFacts{Robots: "noindex"})
+	header := &Page{Key: "/b", Resp: &Response{Status: 200, XRobots: "noindex, nofollow"}, Facts: &PageFacts{}}
+	if !(&Env{}).noindex(meta) || !(&Env{}).noindex(header) || (&Env{Staging: true}).noindex(meta) || (&Env{}).noindex(okPage("/c", &PageFacts{})) {
+		t.Fatal("noindex is wrong")
+	}
+	var nilEnv *Env
+	if nilEnv.record("x") != nil || locale(nil) != "" || locale(&Record{Locale: "pl"}) != "pl" {
+		t.Fatal("nil env and record helpers")
+	}
+}
+
+// TestSitemapNoindexTool pins G1 (SC-3, D-14): a listed noindex page with
+// a CMS record names seo_set_url_seo; one without a record (an Additional
+// address of the sitemap) names the sitemap's settings tool while the
+// sitemap plugin is mounted, and fix: code naming the sitemap's address
+// list otherwise.
+func TestSitemapNoindexTool(t *testing.T) {
+	res := &Result{
+		Pages: map[string]*Page{
+			"/cms":   okPage("/cms", &PageFacts{Robots: "noindex"}),
+			"/extra": okPage("/extra", &PageFacts{Robots: "noindex"}),
+		},
+		Listed: map[string]string{"/cms": "/sitemap-pages-1.xml", "/extra": "/sitemap-pages-1.xml"},
+	}
+	rec := &Record{Type: "seo-page", ID: "1"}
+	lookup := func(key string) *Record {
+		if key == "/cms" {
+			return rec
+		}
+		return nil
+	}
+	find := func(env *Env, key string) Finding {
+		t.Helper()
+		for _, f := range checkSitemaps(res, env) {
+			if f.Kind == KindSitemapNoindex && f.URL == key {
+				return f
+			}
+		}
+		t.Fatalf("no sitemap_noindex on %s", key)
+		return Finding{}
+	}
+	mounted := &Env{Lookup: lookup, SitemapTool: ToolSitemapSettings}
+	if fix, tool, _ := Route(find(mounted, "/cms")); fix != models.FixTool || tool != ToolSetURLSEO {
+		t.Fatalf("CMS record: %s %q", fix, tool)
+	}
+	extra := find(mounted, "/extra")
+	if fix, tool, _ := Route(extra); fix != models.FixTool || tool != "sitemap_update_settings" || extra.Details["extra"] != true {
+		t.Fatalf("Additional address with the sitemap mounted: %s %q %v", fix, tool, extra.Details)
+	}
+	if _, ok := find(mounted, "/cms").Details["extra"]; ok {
+		t.Fatal("a CMS record's finding is marked extra")
+	}
+	alone := &Env{Lookup: lookup}
+	if fix, tool, what := Route(find(alone, "/extra")); fix != models.FixCode || tool != "" || what != "the address list of the sitemap that names this page" {
+		t.Fatalf("Additional address without the sitemap: %s %q %q", fix, tool, what)
+	}
+}

classes/crawl/compare.go

@@ -0,0 +1,86 @@
+package crawl
+
+import (
+	"net/url"
+	"strconv"
+	"strings"
+)
+
+// The thresholds of the rendered comparison (P33-55): text counts as
+// added by JavaScript when the rendered page has at least growthWords more
+// words and growthShare more than the raw HTML.
+const (
+	growthWords   = 50
+	growthShare   = 0.2
+	maxJSOnlyURLs = 5
+)
+
+// compareRendered compares a page's raw HTML with its JavaScript-rendered
+// DOM (D-70, P33-55; pure): same-site links, an H1 or a substantial amount
+// of text that appear only after JavaScript are one content_js finding
+// with details.check rendered and details.what naming them. key is the
+// crawl's address of a link (Fetcher.Key: "" off the site, another port
+// included). An identical rendering finds nothing.
+func compareRendered(raw, rendered *PageFacts, key func(*url.URL) string) []Finding {
+	if raw == nil || rendered == nil {
+		return nil
+	}
+	same := func(u *url.URL) (string, bool) {
+		if u == nil || key == nil {
+			return "", false
+		}
+		k := key(u)
+		return k, k != ""
+	}
+	before := map[string]bool{}
+	for _, l := range raw.Links {
+		if k, ok := same(l.URL); ok {
+			before[k] = true
+		}
+	}
+	var added []string
+	seen := map[string]bool{}
+	for _, l := range rendered.Links {
+		k, ok := same(l.URL)
+		if !ok || before[k] || seen[k] {
+			continue
+		}
+		seen[k] = true
+		added = append(added, k)
+	}
+	details := map[string]any{}
+	var what []string
+	if len(added) > 0 {
+		shown := added
+		if len(shown) > maxJSOnlyURLs {
+			shown = shown[:maxJSOnlyURLs]
+		}
+		details["links"] = shown
+		what = append(what, "links "+strings.Join(shown, ", "))
+	}
+	if len(raw.H1()) == 0 {
+		if h1 := rendered.H1(); len(h1) > 0 {
+			details["h1"] = h1[0]
+			what = append(what, "H1 “"+h1[0]+"”")
+		}
+	}
+	if grow := rendered.Words - raw.Words; grow >= growthWords && float64(grow) >= growthShare*float64(raw.Words) {
+		details["words_raw"] = raw.Words
+		details["words_rendered"] = rendered.Words
+		what = append(what, strconv.Itoa(grow)+" words of text")
+	}
+	if len(what) == 0 {
+		return nil
+	}
+	return []Finding{{Kind: KindContentJS, Check: checkRendered, Details: details, What: strings.Join(what, "; ")}}
+}
+
+// The two content_js checks: the raw HTML against the CMS record, and the
+// raw HTML against the JavaScript-rendered DOM.
+const (
+	checkCMS      = "cms"
+	checkRendered = "rendered"
+)
+
+// parseURL parses an absolute or site-relative address.
+func parseURL(raw string) (*url.URL, error) { return url.Parse(strings.TrimSpace(raw)) }

classes/crawl/compare_test.go

@@ -0,0 +1,82 @@
+package crawl
+
+import (
+	"net/url"
+	"strings"
+	"testing"
+)
+
+func linkFacts(words int, h1 string, hrefs ...string) *PageFacts {
+	f := &PageFacts{Words: words}
+	if h1 != "" {
+		f.Headings = []Heading{{Level: 1, Text: h1}}
+	}
+	for _, h := range hrefs {
+		u, _ := url.Parse(h)
+		f.Links = append(f.Links, Link{URL: u, Href: h})
+	}
+	return f
+}
+
+// siteKey is the crawl key of a link on https://example.test (its www
+// twin too, port 443 only).
+func siteKey(u *url.URL) string {
+	f, err := NewFetcher("https://example.test", nil)
+	if err != nil {
+		panic(err)
+	}
+	return f.Key(u)
+}
+
+// TestCompareRendered pins the rendered comparison (P33-55): a same-host
+// link (the www twin too), an H1 and growth of at least 50 words and 20 %
+// present only after JavaScript are one content_js finding with check
+// rendered; other-host links, smaller growth and an identical rendering
+// find nothing.
+func TestCompareRendered(t *testing.T) {
+	raw := linkFacts(100, "", "https://example.test/a")
+	rendered := linkFacts(160, "Grzyby", "https://example.test/a", "https://www.example.test/b?x=1", "https://other.test/c", "https://example.test/b?x=1")
+	got := compareRendered(raw, rendered, siteKey)
+	if len(got) != 1 || got[0].Kind != KindContentJS || got[0].Check != checkRendered {
+		t.Fatalf("findings = %+v", got)
+	}
+	d := got[0].Details
+	if links := d["links"].([]string); len(links) != 1 || links[0] != "/b?x=1" || d["h1"] != "Grzyby" || d["words_raw"] != 100 || d["words_rendered"] != 160 {
+		t.Fatalf("details = %v", d)
+	}
+	if !strings.Contains(got[0].What, "links /b?x=1") || !strings.Contains(got[0].What, "H1 “Grzyby”") || !strings.Contains(got[0].What, "60 words of text") {
+		t.Fatalf("what = %q", got[0].What)
+	}
+
+	for name, pair := range map[string][2]*PageFacts{
+		"identical":           {linkFacts(100, "T", "https://example.test/a"), linkFacts(100, "T", "https://example.test/a")},
+		"other host only":     {linkFacts(100, "T"), linkFacts(100, "T", "https://other.test/x")},
+		"49 words":            {linkFacts(100, "T"), linkFacts(149, "T")},
+		"50 words under 20 %": {linkFacts(300, "T"), linkFacts(350, "T")},
+		"H1 already there":    {linkFacts(10, "Raw"), linkFacts(10, "Rendered")},
+	} {
+		if got := compareRendered(pair[0], pair[1], siteKey); got != nil {
+			t.Errorf("%s: %+v", name, got)
+		}
+	}
+	if got := compareRendered(linkFacts(0, "T"), linkFacts(50, "T"), siteKey); len(got) != 1 {
+		t.Fatalf("50 words from an empty page: %+v", got)
+	}
+	if compareRendered(nil, rendered, siteKey) != nil || compareRendered(raw, nil, siteKey) != nil {
+		t.Fatal("a missing side compares to nothing")
+	}
+}
+
+// TestCompareRenderedCap pins the cap of the links a finding names, and
+// that a path without a slash counts as /.
+func TestCompareRenderedCap(t *testing.T) {
+	var hrefs []string
+	for _, p := range []string{"", "/1", "/2", "/3", "/4", "/5", "/6"} {
+		hrefs = append(hrefs, "https://example.test"+p)
+	}
+	got := compareRendered(linkFacts(10, "T"), linkFacts(10, "T", hrefs...), siteKey)
+	links := got[0].Details["links"].([]string)
+	if len(links) != maxJSOnlyURLs || links[0] != "/" {
+		t.Fatalf("links = %v", links)
+	}
+}

classes/crawl/crawler.go

@@ -0,0 +1,435 @@
+package crawl
+
+import (
+	"bytes"
+	"context"
+	"encoding/xml"
+	"errors"
+	"io"
+	"net/url"
+	"strings"
+	"sync"
+	"time"
+)
+
+// ErrNoSiteURL is a crawl started without a site address
+// (golem15.seo.site_url, else app.url).
+var ErrNoSiteURL = errors.New("golem15.seo: the site address for crawls is not configured")
+
+// The paths the crawler seeds from (RESEARCH Pattern 7).
+const (
+	HomePath    = "/"
+	SitemapPath = "/sitemap.xml"
+	RobotsPath  = "/robots.txt"
+)
+
+// maxSitemapFiles caps the sitemap files read from an index.
+const maxSitemapFiles = 500
+
+// How a page entered the frontier.
+const (
+	viaHome      = "home"
+	viaSitemap   = "sitemap"
+	viaLink      = "link"
+	viaRedirect  = "redirect"
+	viaCanonical = "canonical"
+	viaAlternate = "alternate"
+)
+
+// Page is one fetched address of a crawl: the answer, the facts of its raw
+// HTML, the facts of its JavaScript-rendered DOM when the comparison ran,
+// its depth from the home page and the sitemap file that lists it.
+type Page struct {
+	Key         string
+	Depth       int
+	Via         string
+	Resp        *Response
+	Facts       *PageFacts
+	Rendered    *PageFacts
+	InSitemap   bool
+	SitemapFile string
+	FetchedAt   time.Time
+}
+
+// OK reports whether the page answered 200 with HTML.
+func (p *Page) OK() bool {
+	return p != nil && p.Resp != nil && p.Resp.Err == "" && p.Resp.Status == 200 && p.Facts != nil
+}
+
+// SitemapFile is one sitemap the crawl read: its address, status and the
+// page addresses it lists.
+type SitemapFile struct {
+	Key    string
+	Status int
+	URLs   []string
+}
+
+// RenderState is the outcome of the rendered comparison (P33-55).
+type RenderState struct {
+	Status string
+	Pages  int
+	Note   string
+}
+
+// Result is what one crawl fetched, in frontier order.
+type Result struct {
+	Pages    map[string]*Page
+	Order    []string
+	Sitemaps []SitemapFile
+	// Listed maps a page address to the sitemap file listing it.
+	Listed       map[string]string
+	Robots       string
+	RobotsStatus int
+	LimitHit     bool
+	Canceled     bool
+	Render       RenderState
+	fetcher      *Fetcher
+}
+
+// Page returns the fetched page at key, nil when none.
+func (r *Result) Page(key string) *Page {
+	if r == nil {
+		return nil
+	}
+	return r.Pages[key]
+}
+
+// Options are the limits and seams of one crawl.
+type Options struct {
+	// MaxPages caps the fetched addresses (D-14, default 5000).
+	MaxPages int
+	// Delay is the pause between two requests (default 250 ms).
+	Delay time.Duration
+	// Renderer renders pages for the comparison; nil skips it.
+	Renderer Renderer
+	// RenderMax caps the rendered pages (default 200, crawl order).
+	RenderMax int
+	// RenderErr is why the renderer could not be built: the comparison is
+	// recorded as failed with it, and the raw checks run.
+	RenderErr error
+	// Canceled is asked between requests; true stops the crawl and keeps
+	// what was fetched.
+	Canceled func(ctx context.Context) bool
+	// Progress is told the number of fetched pages.
+	Progress func(ctx context.Context, fetched int)
+}
+
+// item is one frontier entry.
+type item struct {
+	key   string
+	depth int
+	via   string
+}
+
+// pacer spaces requests at least delay apart across the workers.
+type pacer struct {
+	mu    sync.Mutex
+	next  time.Time
+	delay time.Duration
+}
+
+func (p *pacer) wait(ctx context.Context) error {
+	p.mu.Lock()
+	now := time.Now()
+	at := p.next
+	if at.Before(now) {
+		at = now
+	}
+	p.next = at.Add(p.delay)
+	p.mu.Unlock()
+	if d := time.Until(at); d > 0 {
+		t := time.NewTimer(d)
+		defer t.Stop()
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+		case <-t.C:
+		}
+	}
+	return nil
+}
+
+// Crawl fetches the site through f, a fetchguard client limited to the
+// site host and its www twin (fetchguard.AllowHostsMode, HTTPS, no
+// redirects followed; TrustedMode only from Go config): robots.txt, the
+// sitemap index and its files, then breadth-first from the home page and
+// the sitemap addresses over same-host links, redirect targets, canonicals
+// and hreflang alternates, two requests at a time with the delay between
+// requests, up to MaxPages. robots.txt rules are read for the report, not
+// obeyed (P33-41). Cancelling keeps what was fetched. After each HTML page
+// answering 200, a configured Renderer renders it (one tab at a time, up
+// to RenderMax pages) for the raw versus rendered comparison.
+func Crawl(ctx context.Context, f *Fetcher, opt Options) (*Result, error) {
+	if opt.MaxPages <= 0 {
+		opt.MaxPages = 5000
+	}
+	if opt.RenderMax <= 0 {
+		opt.RenderMax = DefaultRenderMax
+	}
+	res := &Result{Pages: map[string]*Page{}, Listed: map[string]string{}, Render: RenderState{Status: renderStatusSkipped}, fetcher: f}
+	switch {
+	case opt.Renderer != nil:
+		res.Render.Status = renderStatusDone
+	case opt.RenderErr != nil:
+		res.Render = RenderState{Status: renderStatusFailed, Note: opt.RenderErr.Error()}
+	}
+	pace := &pacer{delay: opt.Delay}
+	canceled := func() bool {
+		if ctx.Err() != nil {
+			return true
+		}
+		return opt.Canceled != nil && opt.Canceled(ctx)
+	}
+
+	// robots.txt, for the sitemap_blocked check.
+	if err := pace.wait(ctx); err != nil {
+		return res, err
+	}
+	if r := f.Get(ctx, RobotsPath); r.Err == "" {
+		res.RobotsStatus = r.Status
+		if r.Status == 200 {
+			res.Robots = string(r.Body)
+		}
+	}
+
+	// The sitemap index and its files.
+	if err := readSitemaps(ctx, f, pace, res); err != nil {
+		return res, err
+	}
+
+	seen := map[string]bool{}
+	var queue []item
+	push := func(key string, depth int, via string) {
+		if key == "" || seen[key] {
+			return
+		}
+		seen[key] = true
+		queue = append(queue, item{key: key, depth: depth, via: via})
+	}
+	push(HomePath, 0, viaHome)
+	for _, sm := range res.Sitemaps {
+		for _, key := range sm.URLs {
+			push(key, 1, viaSitemap)
+		}
+	}
+
+	var renderMu sync.Mutex
+	rendered := 0
+	renderOff := opt.Renderer == nil
+	for len(queue) > 0 {
+		if canceled() {
+			res.Canceled = true
+			break
+		}
+		if len(res.Order) >= opt.MaxPages {
+			res.LimitHit = true
+			break
+		}
+		n := Workers
+		if room := opt.MaxPages - len(res.Order); n > room {
+			n = room
+		}
+		if n > len(queue) {
+			n = len(queue)
+		}
+		batch := queue[:n]
+		queue = queue[n:]
+		pages := make([]*Page, len(batch))
+		var wg sync.WaitGroup
+		for i, it := range batch {
+			wg.Add(1)
+			go func(i int, it item) {
+				defer wg.Done()
+				p := &Page{Key: it.key, Depth: it.depth, Via: it.via}
+				if err := pace.wait(ctx); err != nil {
+					p.Resp = &Response{Err: "canceled"}
+					pages[i] = p
+					return
+				}
+				p.Resp = f.Get(ctx, it.key)
+				if p.Resp.Err != "" && ctx.Err() != nil {
+					// The fetch was cut short by the cancellation.
+					p.Resp = &Response{Err: "canceled"}
+				}
+				p.FetchedAt = time.Now().UTC()
+				if p.Resp.Err == "" && p.Resp.Status == 200 && p.Resp.HTML() {
+					base, _ := url.Parse(f.Absolute(it.key))
+					p.Facts = Parse(base, p.Resp.Body)
+					renderMu.Lock()
+					if !renderOff && rendered < opt.RenderMax && ctx.Err() == nil {
+						rendered++
+						dom, err := opt.Renderer.Render(ctx, f.Absolute(it.key))
+						if err != nil {
+							renderOff = true
+							rendered--
+							res.Render = RenderState{Status: renderStatusFailed, Note: err.Error()}
+						} else {
+							p.Rendered = Parse(base, []byte(dom))
+						}
+					}
+					renderMu.Unlock()
+				}
+				pages[i] = p
+			}(i, it)
+		}
+		wg.Wait()
+		for _, p := range pages {
+			if file, ok := res.Listed[p.Key]; ok {
+				p.InSitemap, p.SitemapFile = true, file
+			}
+			res.Pages[p.Key] = p
+			res.Order = append(res.Order, p.Key)
+			p.Resp.Body = nil
+			for _, next := range discovered(f, p) {
+				push(next.key, p.Depth+1, next.via)
+			}
+		}
+		if opt.Progress != nil {
+			opt.Progress(ctx, len(res.Order))
+		}
+	}
+	if len(queue) > 0 && !res.Canceled && len(res.Order) >= opt.MaxPages {
+		res.LimitHit = true
+	}
+	res.Render.Pages = rendered
+	return res, nil
+}
+
+// discovered lists the same-host addresses a fetched page leads to: its
+// redirect target, its links, canonical and hreflang alternates.
+func discovered(f *Fetcher, p *Page) []item {
+	var out []item
+	add := func(raw string, via string) {
+		u, err := url.Parse(raw)
+		if err != nil {
+			return
+		}
+		u.Fragment = ""
+		if key := f.Key(u); key != "" {
+			out = append(out, item{key: key, via: via})
+		}
+	}
+	if p.Resp != nil && p.Resp.Location != "" {
+		add(p.Resp.Location, viaRedirect)
+	}
+	if p.Facts == nil {
+		return out
+	}
+	for _, l := range p.Facts.Links {
+		if l.URL != nil {
+			add(l.URL.String(), viaLink)
+		}
+	}
+	if p.Facts.Canonical != "" {
+		add(p.Facts.Canonical, viaCanonical)
+	}
+	for _, a := range p.Facts.Alternates {
+		if a.URL != nil {
+			add(a.URL.String(), viaAlternate)
+		}
+	}
+	return out
+}
+
+// readSitemaps reads /sitemap.xml: an index's files (same host only) or a
+// single urlset.
+func readSitemaps(ctx context.Context, f *Fetcher, pace *pacer, res *Result) error {
+	if err := pace.wait(ctx); err != nil {
+		return err
+	}
+	r := f.Get(ctx, SitemapPath)
+	if r.Err != "" || r.Status != 200 {
+		res.Sitemaps = append(res.Sitemaps, SitemapFile{Key: SitemapPath, Status: r.Status})
+		return nil
+	}
+	index, locs := parseSitemap(r.Body)
+	if !index {
+		res.addSitemap(f, SitemapFile{Key: SitemapPath, Status: 200}, locs)
+		return nil
+	}
+	for i, loc := range locs {
+		if i >= maxSitemapFiles {
+			break
+		}
+		u, err := url.Parse(loc)
+		if err != nil {
+			continue
+		}
+		key := f.Key(u)
+		if key == "" {
+			continue
+		}
+		if err := pace.wait(ctx); err != nil {
+			return err
+		}
+		sr := f.Get(ctx, key)
+		file := SitemapFile{Key: key, Status: sr.Status}
+		if sr.Err != "" || sr.Status != 200 {
+			res.Sitemaps = append(res.Sitemaps, file)
+			continue
+		}
+		_, urls := parseSitemap(sr.Body)
+		res.addSitemap(f, file, urls)
+	}
+	return nil
+}
+
+func (res *Result) addSitemap(f *Fetcher, file SitemapFile, locs []string) {
+	for _, loc := range locs {
+		u, err := url.Parse(loc)
+		if err != nil {
+			continue
+		}
+		key := f.Key(u)
+		if key == "" {
+			continue
+		}
+		if _, dup := res.Listed[key]; !dup {
+			res.Listed[key] = file.Key
+		}
+		file.URLs = append(file.URLs, key)
+	}
+	res.Sitemaps = append(res.Sitemaps, file)
+}
+
+// parseSitemap reads the <loc> values of a sitemap document; index is true
+// for a <sitemapindex>.
+func parseSitemap(body []byte) (index bool, locs []string) {
+	dec := xml.NewDecoder(io.LimitReader(bytes.NewReader(body), MaxBody))
+	depth := 0
+	inLoc := false
+	var parent string
+	var stack []string
+	for {
+		tok, err := dec.Token()
+		if err != nil {
+			break
+		}
+		switch t := tok.(type) {
+		case xml.StartElement:
+			depth++
+			name := strings.ToLower(t.Name.Local)
+			if depth == 1 && name == "sitemapindex" {
+				index = true
+			}
+			stack = append(stack, name)
+			if name == "loc" && len(stack) >= 2 {
+				parent = stack[len(stack)-2]
+				inLoc = parent == "url" || parent == "sitemap"
+			}
+		case xml.EndElement:
+			depth--
+			if len(stack) > 0 {
+				stack = stack[:len(stack)-1]
+			}
+			inLoc = false
+		case xml.CharData:
+			if inLoc {
+				if s := strings.TrimSpace(string(t)); s != "" {
+					locs = append(locs, s)
+				}
+			}
+		}
+	}
+	return index, locs
+}

classes/crawl/fetch.go

@@ -0,0 +1,224 @@
+// Package crawl is the site audit's crawler (D-14): it fetches the
+// rendered site like a search engine, checks every page (UI-SPEC C23),
+// stores the findings with the CMS-data issues and names how each one is
+// fixed (P33-38).
+package crawl
+
+import (
+	"context"
+	"net/http"
+	"net/url"
+	"strings"
+	"time"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/summercms/modules/compass"
+	"git.golem15.com/golem15/summercms/modules/fetchguard"
+)
+
+// The config keys of the crawler.
+const (
+	// TrustedConfigKey lets the crawler fetch plain http and private
+	// addresses (staging, local sites). Only Go config can set it.
+	TrustedConfigKey = "golem15.seo.crawl_trusted"
+)
+
+// The crawler's limits (D-14, RESEARCH Pattern 7).
+const (
+	// MaxBody caps one answer's body.
+	MaxBody = 5 << 20
+	// FetchTimeout is one request's timeout.
+	FetchTimeout = 10 * time.Second
+	// Workers is the number of requests at a time.
+	Workers = 2
+	// MaxLinksPerPage caps the links stored and followed per page.
+	MaxLinksPerPage = 1000
+)
+
+// UserAgent is the crawler's identity (P33-41).
+func UserAgent(site string) string {
+	return "SummerCMS-SEO-Crawler/1 (+" + site + ")"
+}
+
+// Response is one fetched address: its status, Location, content type,
+// X-Robots-Tag and body, or the fetch error.
+type Response struct {
+	Status      int
+	Location    string
+	ContentType string
+	XRobots     string
+	Body        []byte
+	Err         string
+}
+
+// HTML reports whether the answer is an HTML page.
+func (r *Response) HTML() bool {
+	ct := strings.ToLower(r.ContentType)
+	return ct == "" || strings.Contains(ct, "text/html") || strings.Contains(ct, "application/xhtml")
+}
+
+// Fetcher fetches the site's own addresses through a fetchguard client
+// limited to the site host and its www twin: HTTPS only, private
+// addresses refused at dial time and redirects never followed. Trusted
+// mode (golem15.seo.crawl_trusted, Go config only) allows plain http and
+// private addresses, still only on those hosts.
+type Fetcher struct {
+	client  *fetchguard.Client
+	site    *url.URL
+	hosts   []string
+	port    string
+	trusted bool
+	ua      string
+}
+
+// effectivePort is u's port: the explicit one, else 443 for https and 80
+// for http ("" for another scheme without a port).
+func effectivePort(u *url.URL) string {
+	if u == nil {
+		return ""
+	}
+	if p := u.Port(); p != "" {
+		return p
+	}
+	switch strings.ToLower(u.Scheme) {
+	case "https":
+		return "443"
+	case "http":
+		return "80"
+	}
+	return ""
+}
+
+// SiteHosts returns the host of site and its www twin.
+func SiteHosts(host string) []string {
+	host = strings.ToLower(strings.TrimSpace(host))
+	if host == "" {
+		return nil
+	}
+	if bare, ok := strings.CutPrefix(host, "www."); ok {
+		return []string{host, bare}
+	}
+	return []string{host, "www." + host}
+}
+
+// Trusted reports whether config turns trusted mode on.
+func Trusted(cfg *compass.Config) bool {
+	return cfg != nil && cfg.Bool(TrustedConfigKey)
+}
+
+// NewFetcher builds the fetcher of site (an absolute origin).
+func NewFetcher(site string, cfg *compass.Config) (*Fetcher, error) {
+	u, err := url.Parse(strings.TrimRight(site, "/"))
+	if err != nil || u.Host == "" || (u.Scheme != "https" && u.Scheme != "http") {
+		return nil, ErrNoSiteURL
+	}
+	f := &Fetcher{site: u, hosts: SiteHosts(u.Hostname()), port: effectivePort(u), trusted: Trusted(cfg), ua: UserAgent(u.Scheme + "://" + u.Host)}
+	policy := fetchguard.Policy{Mode: fetchguard.AllowHostsMode, AllowHosts: f.hosts, MaxBytes: MaxBody, Timeout: FetchTimeout}
+	if f.trusted {
+		policy.Mode = fetchguard.TrustedMode
+	}
+	f.client, err = fetchguard.NewClient(policy, cfg)
+	if err != nil {
+		return nil, err
+	}
+	return f, nil
+}
+
+// Hosts returns the site host and its www twin.
+func (f *Fetcher) Hosts() []string { return f.hosts }
+
+// IsTrusted reports whether the fetcher runs in trusted mode.
+func (f *Fetcher) IsTrusted() bool { return f.trusted }
+
+// Site returns the site origin.
+func (f *Fetcher) Site() *url.URL { return f.site }
+
+// UA returns the crawler's User-Agent.
+func (f *Fetcher) UA() string { return f.ua }
+
+// Port returns the site's effective port (explicit, else 443 for https
+// and 80 for http).
+func (f *Fetcher) Port() string { return f.port }
+
+// Allowed reports whether u may be fetched: https (http only in trusted
+// mode) on the site host or its www twin and on the site's effective port,
+// without credentials. In trusted mode the host list is checked here,
+// since fetchguard checks none then.
+func (f *Fetcher) Allowed(u *url.URL) bool {
+	if u == nil || u.User != nil {
+		return false
+	}
+	if effectivePort(u) != f.port {
+		return false
+	}
+	switch strings.ToLower(u.Scheme) {
+	case "https":
+	case "http":
+		if !f.trusted {
+			return false
+		}
+	default:
+		return false
+	}
+	host := strings.ToLower(u.Hostname())
+	for _, h := range f.hosts {
+		if host == h {
+			return true
+		}
+	}
+	return false
+}
+
+// Key is the address of u on the site: its path in the decoded form of
+// classes.PathKey with the raw query, or "" when u is not on the site
+// (another host, scheme or port). The crawl's keys, the rendered
+// comparison, the CMS lookups and the link graph use this one form.
+func (f *Fetcher) Key(u *url.URL) string {
+	if !f.Allowed(u) {
+		return ""
+	}
+	p := classes.PathKey(u)
+	if u.RawQuery != "" {
+		p += "?" + u.RawQuery
+	}
+	return p
+}
+
+// Absolute returns the site URL of an address key, its path
+// percent-encoded for the request (classes.EscapeKeyPath).
+func (f *Fetcher) Absolute(key string) string {
+	p, q, hasQuery := strings.Cut(key, "?")
+	out := f.site.Scheme + "://" + f.site.Host + classes.EscapeKeyPath(p)
+	if hasQuery {
+		out += "?" + q
+	}
+	return out
+}
+
+// Get fetches one address key of the site.
+func (f *Fetcher) Get(ctx context.Context, key string) *Response {
+	header := http.Header{}
+	header.Set("User-Agent", f.ua)
+	header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.5")
+	res, err := f.client.Get(ctx, f.Absolute(key), header)
+	if err != nil {
+		reason := "network_error"
+		if fe, ok := err.(*fetchguard.Error); ok {
+			reason = string(fe.Reason)
+		}
+		return &Response{Err: reason}
+	}
+	out := &Response{
+		Status:      res.StatusCode,
+		ContentType: res.Header.Get("Content-Type"),
+		XRobots:     strings.Join(res.Header.Values("X-Robots-Tag"), ", "),
+		Body:        res.Body,
+	}
+	if loc := res.Header.Get("Location"); loc != "" {
+		base, _ := url.Parse(f.Absolute(key))
+		if ref, err := url.Parse(loc); err == nil && base != nil {
+			out.Location = base.ResolveReference(ref).String()
+		}
+	}
+	return out
+}

classes/crawl/fetch_test.go

@@ -0,0 +1,77 @@
+package crawl
+
+import (
+	"net/url"
+	"testing"
+)
+
+// TestFetcherPortAndKey pins L3 WR-03 (and WR-04, P33-61): the fetcher
+// accepts an address only on the site's effective port, so a link to
+// another port is neither fetched nor reported as a page of the site.
+func TestFetcherPortAndKey(t *testing.T) {
+	f, err := NewFetcher("https://site.test", nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if f.Port() != "443" {
+		t.Fatalf("Port = %q", f.Port())
+	}
+	parse := func(raw string) *url.URL {
+		u, err := url.Parse(raw)
+		if err != nil {
+			t.Fatal(err)
+		}
+		return u
+	}
+	for raw, want := range map[string]string{
+		"https://site.test:8443/x":   "",
+		"https://site.test:443/x":    "/x",
+		"https://www.site.test/x":    "/x",
+		"https://site.test:80/x":     "",
+		"http://site.test/x":         "",
+		"https://other.test/x":       "",
+		"https://site.test":          "/",
+		"https://site.test/a?b=1&c=": "/a?b=1&c=",
+	} {
+		u := parse(raw)
+		if got := f.Key(u); got != want {
+			t.Errorf("Key(%s) = %q, want %q", raw, got, want)
+		}
+		if f.Allowed(u) != (want != "") {
+			t.Errorf("Allowed(%s) = %v", raw, f.Allowed(u))
+		}
+	}
+	// One decoded path form (WR-04, P33-61): the key is decoded, the
+	// request URL encoded; a path that does not decode to text keeps its
+	// encoded form, and so does one holding "%" or "?".
+	for raw, want := range map[string]string{
+		"https://site.test/blog/za%C5%BC%C3%B3%C5%82%C4%87?a=1": "/blog/zażółć?a=1",
+		"https://site.test/blog/zażółć":                         "/blog/zażółć",
+		"https://site.test/%FF":                                 "/%FF",
+		"https://site.test/a%00b":                               "/a%00b",
+		"https://site.test/100%25":                              "/100%25",
+		"https://site.test/a%3Fb?c=1":                           "/a%3Fb?c=1",
+	} {
+		if got := f.Key(parse(raw)); got != want {
+			t.Errorf("Key(%s) = %q, want %q", raw, got, want)
+		}
+	}
+	for key, want := range map[string]string{
+		"/blog/zażółć?a=1": "https://site.test/blog/za%C5%BC%C3%B3%C5%82%C4%87?a=1",
+		"/%FF":             "https://site.test/%FF",
+		"/100%25":          "https://site.test/100%25",
+		"/a b":             "https://site.test/a%20b",
+		"/":                "https://site.test/",
+	} {
+		if got := f.Absolute(key); got != want {
+			t.Errorf("Absolute(%q) = %q, want %q", key, got, want)
+		}
+	}
+	local, err := NewFetcher("http://localhost:3000", nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if local.Port() != "3000" || effectivePort(parse("http://x")) != "80" || effectivePort(parse("ftp://x")) != "" || effectivePort(nil) != "" {
+		t.Fatal("effectivePort is wrong")
+	}
+}

classes/crawl/fix.go

@@ -0,0 +1,150 @@
+package crawl
+
+import (
+	"crypto/sha256"
+	"encoding/hex"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/pact"
+)
+
+// The SEO tools an issue may name (pier prefixes them with seo_).
+const (
+	ToolSetURLSEO      = "seo_set_url_seo"
+	ToolUpdateRobots   = "seo_update_robots"
+	ToolCreateRedirect = "seo_create_redirect"
+	ToolUpdatePage     = "seo_update_page"
+	ToolCreatePage     = "seo_create_page"
+	ToolUpdateSettings = "seo_update_settings"
+	// ToolSitemapSettings is the sitemap plugin's settings tool, which
+	// edits its Additional addresses.
+	ToolSitemapSettings = "sitemap_update_settings"
+)
+
+// Record is the CMS record behind a crawled page: its content type, id
+// and locale, the descriptor facts the checks compare the page with
+// (title, word count, headings, images, the highest heading level) and the
+// fixing tools its type names (pact.ContentDescriptor.Tools).
+type Record struct {
+	Type       string
+	ID         string
+	Locale     string
+	Title      string
+	Words      *int
+	Headings   []pact.ContentHeading
+	Images     []pact.ContentImage
+	TopHeading int
+	// Links are the record's CMS-known links: the address of its primary
+	// parent, which the page's breadcrumb trail links to.
+	Links []string
+	Tools map[string]string
+	// CustomJSONLD is true when the record's SEO tab stores custom JSON-LD.
+	CustomJSONLD bool
+}
+
+// tool returns the record's tool for the first operation of ops it
+// names, "" when it names none.
+func (r *Record) tool(ops ...string) string {
+	if r == nil {
+		return ""
+	}
+	for _, op := range ops {
+		if t := strings.TrimSpace(r.Tools[op]); t != "" {
+			return t
+		}
+	}
+	return ""
+}
+
+// hasHeading reports whether the record's own headings hold one of level
+// (any level when level is 0) whose text is text (any text when empty).
+func (r *Record) hasHeading(level int, text string) bool {
+	if r == nil {
+		return false
+	}
+	for _, h := range r.Headings {
+		if level != 0 && h.Level != level {
+			continue
+		}
+		if text == "" || normal(h.Text) == normal(text) {
+			return true
+		}
+	}
+	return false
+}
+
+// Finding is one problem a check found on a page, before it is stored:
+// the kind, the address and locale, the details of the kind, and how it
+// is fixed (P33-38). Tool is a fixed CMS tool (seo_*); Content lists the
+// record operations (pact.ContentTool*) that fix a finding inside the
+// record's own content, tried in order; What names the template element
+// when it is fixed in code.
+type Finding struct {
+	Kind    string
+	URL     string
+	Locale  string
+	Details map[string]any
+	Tool    string
+	Content []string
+	Record  *Record
+	What    string
+	// Check tells the two content_js findings apart (cms, rendered).
+	Check string
+}
+
+// Route applies P33-38: a finding inside the record's own content names
+// the record type's fitting tool (H1 findings prefer Tools[headings],
+// other content findings Tools[edit]); else a fixed CMS tool; else fix
+// code with the template element in details.what. Every finding gets one
+// of the two.
+func Route(f Finding) (fix, tool, what string) {
+	if len(f.Content) > 0 {
+		if t := f.Record.tool(f.Content...); t != "" {
+			return models.FixTool, t, ""
+		}
+	}
+	if f.Tool != "" {
+		return models.FixTool, f.Tool, ""
+	}
+	what = f.What
+	if what == "" {
+		what = defaultWhat[f.Kind]
+	}
+	if what == "" {
+		what = "the page template"
+	}
+	return models.FixCode, "", what
+}
+
+// defaultWhat names the template element of a kind fixed in code.
+var defaultWhat = map[string]string{
+	KindCanonicalMissing: `<link rel="canonical"> in the page head`,
+	KindH1Missing:        "the <h1> of the page template",
+	KindH1Multiple:       "the <h1> headings of the page template",
+	KindHeadingSkipped:   "the heading levels of the page template",
+	KindHeadingEmpty:     "the empty heading of the page template",
+	KindJSONLDInvalid:    `the <script type="application/ld+json"> of the page template`,
+	KindJSONLDMultiple:   `the <script type="application/ld+json"> blocks of the page template`,
+	KindHreflangRecip:    `the <link rel="alternate" hreflang> links of the page template`,
+	KindImageAlt:         "the alt attribute of the template's <img>",
+	KindImageSize:        "the width and height attributes of the template's <img>",
+	KindFeaturedLazy:     `loading="lazy" on the main image`,
+	KindContentJS:        "server-side rendering of the page content",
+	KindGenericAnchor:    "the link text in the page template",
+	KindRedirectedLink:   "the link in the page template",
+	KindOrphanPage:       "a link to this page from the site navigation or another page",
+	KindSitemapNoindex:   "the address list of the sitemap that names this page",
+}
+
+// Fingerprint identifies a finding across crawls: the kind, the address,
+// the locale and the check (content_js), so an ignore survives later
+// crawls (P33-37).
+func Fingerprint(kind, url, locale, check string) string {
+	sum := sha256.Sum256([]byte(kind + "\x00" + url + "\x00" + locale + "\x00" + check))
+	return hex.EncodeToString(sum[:16])
+}
+
+// normal is text compared without case and extra whitespace.
+func normal(s string) string { return classes.NormalText(s) }

classes/crawl/fix_test.go

@@ -0,0 +1,80 @@
+package crawl
+
+import (
+	"testing"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/pact"
+)
+
+// TestRoute pins P33-38: a content finding names the record type's first
+// fitting tool in the order given; without one a fixed CMS tool; else
+// code with the finding's own what, the kind's default, or the page
+// template.
+func TestRoute(t *testing.T) {
+	both := &Record{Tools: map[string]string{pact.ContentToolHeadings: "acme_fix_headings", pact.ContentToolEdit: "acme_update"}}
+	editOnly := &Record{Tools: map[string]string{pact.ContentToolEdit: " acme_update "}}
+	for _, tc := range []struct {
+		name            string
+		f               Finding
+		fix, tool, what string
+	}{
+		{"headings first", Finding{Kind: KindH1Multiple, Content: []string{pact.ContentToolHeadings, pact.ContentToolEdit}, Record: both}, models.FixTool, "acme_fix_headings", ""},
+		{"edit fallback", Finding{Kind: KindH1Multiple, Content: []string{pact.ContentToolHeadings, pact.ContentToolEdit}, Record: editOnly}, models.FixTool, "acme_update", ""},
+		{"no record tool, fixed tool", Finding{Kind: KindJSONLDInvalid, Content: []string{pact.ContentToolEdit}, Record: &Record{}, Tool: ToolSetURLSEO}, models.FixTool, ToolSetURLSEO, ""},
+		{"nil record", Finding{Kind: KindImageAlt, Content: []string{pact.ContentToolImageAlt}}, models.FixCode, "", defaultWhat[KindImageAlt]},
+		{"fixed tool", Finding{Kind: KindMissingTitle, Tool: ToolSetURLSEO}, models.FixTool, ToolSetURLSEO, ""},
+		{"own what", Finding{Kind: KindContentJS, What: "links /a"}, models.FixCode, "", "links /a"},
+		{"default what", Finding{Kind: KindCanonicalMissing}, models.FixCode, "", defaultWhat[KindCanonicalMissing]},
+		{"page template", Finding{Kind: "unknown"}, models.FixCode, "", "the page template"},
+	} {
+		fix, tool, what := Route(tc.f)
+		if fix != tc.fix || tool != tc.tool || what != tc.what {
+			t.Errorf("%s: %s %q %q, want %s %q %q", tc.name, fix, tool, what, tc.fix, tc.tool, tc.what)
+		}
+	}
+}
+
+// TestRouteEveryKind pins that every kind gets a tool or a code fix.
+func TestRouteEveryKind(t *testing.T) {
+	for _, k := range Kinds() {
+		fix, tool, what := Route(Finding{Kind: k})
+		if (fix == models.FixTool && tool == "") || (fix == models.FixCode && what == "") || (fix != models.FixTool && fix != models.FixCode) {
+			t.Errorf("%s: %s %q %q", k, fix, tool, what)
+		}
+	}
+}
+
+// TestFingerprint pins the identity of a finding across crawls: stable,
+// 32 hex characters, and different for each of its four parts.
+func TestFingerprint(t *testing.T) {
+	a := Fingerprint(KindContentJS, "/a", "pl", "cms")
+	if a != Fingerprint(KindContentJS, "/a", "pl", "cms") || len(a) != 32 {
+		t.Fatalf("fingerprint %q", a)
+	}
+	for _, other := range []string{
+		Fingerprint(KindH1Missing, "/a", "pl", "cms"), Fingerprint(KindContentJS, "/b", "pl", "cms"),
+		Fingerprint(KindContentJS, "/a", "en", "cms"), Fingerprint(KindContentJS, "/a", "pl", "rendered"),
+		Fingerprint(KindContentJS, "/a\x00pl", "", "cms"),
+	} {
+		if other == a {
+			t.Fatal("a different finding has the same fingerprint")
+		}
+	}
+}
+
+// TestRecordHeadingHelpers pins hasHeading (any level or text, compared
+// without case and extra space) and hasEmptyHeading on a nil record too.
+func TestRecordHeadingHelpers(t *testing.T) {
+	rec := &Record{Headings: []pact.ContentHeading{{Level: 2, Text: "Lwia  Grzywa"}, {Level: 3, Text: " "}}}
+	if !rec.hasHeading(2, "lwia grzywa") || !rec.hasHeading(0, "") || rec.hasHeading(1, "") || rec.hasHeading(2, "Chaga") {
+		t.Fatal("hasHeading is wrong")
+	}
+	if !hasEmptyHeading(rec, 3) || hasEmptyHeading(rec, 2) || hasEmptyHeading(nil, 1) {
+		t.Fatal("hasEmptyHeading is wrong")
+	}
+	var none *Record
+	if none.hasHeading(0, "") || none.tool(pact.ContentToolEdit) != "" {
+		t.Fatal("a nil record has no headings and no tools")
+	}
+}

classes/crawl/job.go

@@ -0,0 +1,630 @@
+package crawl
+
+import (
+	"context"
+	"errors"
+	"strconv"
+	"strings"
+	"time"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/classes/schema"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/sm-sitemap-plugin/contract"
+	"git.golem15.com/golem15/summercms/modules/conga"
+	"git.golem15.com/golem15/summercms/modules/diary"
+	"git.golem15.com/golem15/summercms/modules/lagoon"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
+)
+
+// SitemapTool is ToolSitemapSettings while the application publishes the
+// sitemap plugin's contract.Listing (the sitemap plugin is mounted), else
+// "".
+func SitemapTool(svc *classes.Service) string {
+	app := svc.App()
+	if app == nil {
+		return ""
+	}
+	if _, ok := app.Lookup[contract.Listing](); ok {
+		return ToolSitemapSettings
+	}
+	return ""
+}
+
+// CrawlKind is the River kind of the crawl job.
+const CrawlKind = "golem15.seo.crawl"
+
+// CrawlArgs are the arguments of the crawl job: the crawl row it runs.
+type CrawlArgs struct {
+	CrawlID uint `json:"crawl_id"`
+}
+
+// Kind is CrawlKind.
+func (CrawlArgs) Kind() string { return CrawlKind }
+
+// The crawl job's time bounds (P33-60). JobTimeout is the job's deadline;
+// FinishTimeout bounds storing what a crawl found, on a context the job's
+// cancellation cannot reach; the final status is then written on a
+// 30-second bound of its own. The job writes heartbeat_at every
+// HeartbeatEvery, and a running crawl without a heartbeat for StaleAfter
+// is ended by the next Start or delete (RecoverStale).
+const (
+	JobTimeout     = 6 * time.Hour
+	FinishTimeout  = 10 * time.Minute
+	HeartbeatEvery = time.Minute
+	StaleAfter     = 10 * time.Minute
+)
+
+// heartbeatWrite bounds one heartbeat update.
+const heartbeatWrite = 10 * time.Second
+
+// finalWriteTimeout bounds a crawl's final status write and the end of its
+// conga record, on a context of their own, so a store that used up its
+// FinishTimeout bound still ends the crawl (33-REVIEW-GAPFIX WR-03).
+const finalWriteTimeout = 30 * time.Second
+
+type finishTimeoutKey struct{}
+
+// WithFinishTimeout returns a context whose crawls bound storing their
+// results by d instead of FinishTimeout. It is the test seam of the store
+// bound, settable only from Go code (like WithRenderer and
+// fetchguard.WithTransport): no config key, setting, request or MCP call
+// reaches it.
+func WithFinishTimeout(ctx context.Context, d time.Duration) context.Context {
+	return context.WithValue(ctx, finishTimeoutKey{}, d)
+}
+
+// finishTimeoutFrom returns the store bound of ctx: its WithFinishTimeout
+// value when that is positive, else FinishTimeout.
+func finishTimeoutFrom(ctx context.Context) time.Duration {
+	if ctx != nil {
+		if d, ok := ctx.Value(finishTimeoutKey{}).(time.Duration); ok && d > 0 {
+			return d
+		}
+	}
+	return FinishTimeout
+}
+
+// The crawl limits used until the Site audit settings say otherwise.
+const (
+	DefaultMaxPages = 5000
+	DefaultDelay    = 250 * time.Millisecond
+)
+
+// AlreadyRunningError refuses a start while a crawl is queued or running
+// (P33-39): one crawl runs at a time.
+type AlreadyRunningError struct {
+	Crawl models.Crawl
+}
+
+func (e *AlreadyRunningError) Error() string { return "golem15.seo: a crawl is already running" }
+
+// Since is the time the running crawl started, else its creation time.
+func (e *AlreadyRunningError) Since() time.Time {
+	if e.Crawl.StartedAt != nil {
+		return *e.Crawl.StartedAt
+	}
+	if e.Crawl.CreatedAt != nil {
+		return *e.Crawl.CreatedAt
+	}
+	return time.Time{}
+}
+
+// Start queues a crawl (D-14, P33-39) inside tx: the settings row is
+// locked so two starts cannot both pass the check that no crawl is queued
+// or running, a running crawl whose job stopped writing its heartbeat is
+// ended (RecoverStale), then the crawl row is written and the conga job
+// dispatched in the same transaction. A missing site address refuses with
+// ErrNoSiteURL, a queued or running crawl (also one whose cancel is
+// requested but whose job has not stopped) with *AlreadyRunningError.
+func Start(ctx context.Context, svc *classes.Service, tx *gorm.DB, trigger, name string) (models.Crawl, error) {
+	site := svc.SiteURL()
+	if site == "" {
+		return models.Crawl{}, ErrNoSiteURL
+	}
+	if _, err := classes.SettingsRow(tx); err != nil {
+		return models.Crawl{}, err
+	}
+	if err := tx.Exec(`SELECT id FROM golem15_seo_settings WHERE id = 1 FOR UPDATE`).Error; err != nil {
+		return models.Crawl{}, err
+	}
+	stale, err := RecoverStale(tx, time.Now().UTC())
+	if err != nil {
+		return models.Crawl{}, err
+	}
+	endStaleJobs(ctx, svc, tx, stale)
+	active, err := ActiveCrawl(tx)
+	if err != nil {
+		return models.Crawl{}, err
+	}
+	if active != nil {
+		return models.Crawl{}, &AlreadyRunningError{Crawl: *active}
+	}
+	limits, err := LoadLimits(tx)
+	if err != nil {
+		return models.Crawl{}, err
+	}
+	now := time.Now().UTC()
+	row := models.Crawl{
+		Status: models.CrawlQueued, Trigger: trigger, TriggerName: name, Site: site,
+		MaxPages: limits.MaxPages, RenderStatus: models.RenderSkipped, CreatedAt: &now,
+	}
+	if err := tx.Create(&row).Error; err != nil {
+		return models.Crawl{}, err
+	}
+	m, err := conga.From(svc.App())
+	if err != nil {
+		return models.Crawl{}, err
+	}
+	id, err := m.Dispatch(ctx, tx, CrawlArgs{CrawlID: row.ID}, conga.DispatchOpts{Label: svc.Phrase("golem15.seo::lang.jobs.crawl", nil)})
+	if err != nil {
+		return models.Crawl{}, err
+	}
+	row.JobID = &id
+	if err := tx.Model(&models.Crawl{}).Where("id = ?", row.ID).Update("job_id", id).Error; err != nil {
+		return models.Crawl{}, err
+	}
+	return row, nil
+}
+
+// Limits are a crawl's politeness limits and the Site audit settings it
+// reads: pages per crawl, the pause between requests, the crawls kept and
+// the generic link texts.
+type Limits struct {
+	MaxPages       int
+	Delay          time.Duration
+	Keep           int
+	GenericAnchors []string
+}
+
+// DefaultKeep is the number of crawls kept.
+const DefaultKeep = 10
+
+// LoadLimits reads the crawl limits from the Site audit settings; a value
+// outside its range (written around the settings screen) falls back to the
+// default.
+func LoadLimits(db *gorm.DB) (Limits, error) {
+	st, err := classes.LoadSettings(db)
+	if err != nil {
+		return Limits{}, err
+	}
+	out := Limits{MaxPages: DefaultMaxPages, Delay: DefaultDelay, Keep: DefaultKeep, GenericAnchors: ParseAnchors(st.GenericAnchors)}
+	if st.CrawlMaxPages >= models.MinCrawlMaxPages && st.CrawlMaxPages <= models.MaxCrawlMaxPages {
+		out.MaxPages = st.CrawlMaxPages
+	}
+	if st.CrawlDelay >= models.MinCrawlDelay && st.CrawlDelay <= models.MaxCrawlDelay {
+		out.Delay = time.Duration(st.CrawlDelay) * time.Millisecond
+	}
+	if st.CrawlKeep >= models.MinCrawlKeep && st.CrawlKeep <= models.MaxCrawlKeep {
+		out.Keep = st.CrawlKeep
+	}
+	return out, nil
+}
+
+// LoadCrawl reads one crawl.
+func LoadCrawl(db *gorm.DB, id uint) (models.Crawl, error) {
+	var rows []models.Crawl
+	if err := db.Where("id = ?", id).Limit(1).Find(&rows).Error; err != nil {
+		return models.Crawl{}, err
+	}
+	if len(rows) == 0 {
+		return models.Crawl{}, gorm.ErrRecordNotFound
+	}
+	return rows[0], nil
+}
+
+// Run runs one queued crawl: it fetches the site, checks every page,
+// stores the pages, links and findings with the CMS-data issues, and marks
+// the crawl finished, cancelled (pages and issues fetched so far are kept)
+// or failed (with the error; what was fetched before the failure is kept).
+// A crawl that is no longer queued is left alone.
+//
+// The job's context ends on a cancel (River cancels a running job's
+// context) or at the job timeout; what was fetched is then stored on a
+// context detached from it with its own FinishTimeout bound, and the final
+// status is written on another detached context bounded by
+// finalWriteTimeout, so a cancelled or timed-out crawl keeps its results
+// and a crawl whose store ran out of time still ends: none stays running.
+// While it runs, the job writes heartbeat_at every HeartbeatEvery.
+func Run(ctx context.Context, svc *classes.Service, id uint) error {
+	gdb, err := svc.GormDB()
+	if err != nil {
+		return err
+	}
+	db := gdb.WithContext(ctx)
+	row, err := LoadCrawl(db, id)
+	if err != nil {
+		return err
+	}
+	if row.Status != models.CrawlQueued {
+		return nil
+	}
+	now := time.Now().UTC()
+	moved := db.Model(&models.Crawl{}).Where("id = ? AND status = ?", id, models.CrawlQueued).
+		Updates(map[string]any{"status": models.CrawlRunning, "started_at": now, "heartbeat_at": now})
+	if moved.Error != nil {
+		return moved.Error
+	}
+	if moved.RowsAffected == 0 {
+		return nil
+	}
+	stop := heartbeat(ctx, gdb, id)
+	defer stop()
+	res, runErr := run(ctx, svc, db, row)
+	return finish(context.WithoutCancel(ctx), svc, gdb, row, res, runErr, ctx.Err(), finishTimeoutFrom(ctx))
+}
+
+// heartbeat writes the crawl's heartbeat_at every HeartbeatEvery until the
+// returned stop is called; each write runs on a context detached from the
+// job's, bounded by heartbeatWrite.
+func heartbeat(ctx context.Context, gdb *gorm.DB, id uint) (stop func()) {
+	done := make(chan struct{})
+	finished := make(chan struct{})
+	go func() {
+		defer close(finished)
+		t := time.NewTicker(HeartbeatEvery)
+		defer t.Stop()
+		for {
+			select {
+			case <-done:
+				return
+			case <-t.C:
+				hctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), heartbeatWrite)
+				_ = gdb.WithContext(hctx).Model(&models.Crawl{}).Where("id = ? AND status = ?", id, models.CrawlRunning).
+					Update("heartbeat_at", time.Now().UTC()).Error
+				cancel()
+			}
+		}
+	}()
+	return func() {
+		close(done)
+		<-finished
+	}
+}
+
+// run fetches and checks the site of one crawl.
+func run(ctx context.Context, svc *classes.Service, db *gorm.DB, row models.Crawl) (*Result, error) {
+	f, err := NewFetcher(row.Site, svc.Config())
+	if err != nil {
+		return nil, err
+	}
+	limits, err := LoadLimits(db)
+	if err != nil {
+		return nil, err
+	}
+	if row.MaxPages > 0 {
+		limits.MaxPages = row.MaxPages
+	}
+	var m *conga.Manager
+	if app := svc.App(); app != nil {
+		m, _ = conga.From(app)
+	}
+	// The rendered comparison: the test seam's renderer, else Chrome from
+	// golem15.seo.crawl_chrome_path (none: skipped). Chrome is closed when
+	// the crawl finishes, fails or is cancelled.
+	renderer, renderErr := rendererFrom(ctx), error(nil)
+	if renderer == nil {
+		renderer, renderErr = NewRenderer(ctx, svc.Config(), f.Site(), f.IsTrusted(), f.UA())
+		if renderer != nil {
+			defer func() { _ = renderer.Close() }()
+		}
+	}
+	opt := Options{
+		MaxPages:  limits.MaxPages,
+		Delay:     limits.Delay,
+		Renderer:  renderer,
+		RenderErr: renderErr,
+		RenderMax: RenderMax(svc.Config()),
+		Canceled: func(ctx context.Context) bool {
+			if cancelRequested(db, row.ID) {
+				return true
+			}
+			if m != nil && row.JobID != nil {
+				if c, err := m.CheckIfCanceled(ctx, *row.JobID); err == nil && c {
+					return true
+				}
+			}
+			return false
+		},
+		Progress: func(ctx context.Context, n int) {
+			_ = db.Model(&models.Crawl{}).Where("id = ?", row.ID).Update("pages", n).Error
+			if m != nil && row.JobID != nil {
+				_ = m.UpdateJobState(ctx, *row.JobID, n, nil)
+			}
+		},
+	}
+	if m != nil && row.JobID != nil {
+		_ = m.StartJob(ctx, *row.JobID, limits.MaxPages)
+	}
+	return Crawl(ctx, f, opt)
+}
+
+// cancelRequested reports whether a cancel of the crawl was requested: its
+// cancel_requested_at is set, or its status is cancelled.
+func cancelRequested(db *gorm.DB, id uint) bool {
+	var rows []struct {
+		Status    string
+		Requested bool
+	}
+	if err := db.Raw(`SELECT status, cancel_requested_at IS NOT NULL AS requested FROM golem15_seo_crawls WHERE id = ?`, id).Scan(&rows).Error; err != nil || len(rows) == 0 {
+		return false
+	}
+	return rows[0].Requested || rows[0].Status == models.CrawlCancelled
+}
+
+// timeLimitError is the error of a crawl that reached JobTimeout.
+func timeLimitError() string {
+	limit := JobTimeout.String()
+	if JobTimeout%time.Hour == 0 {
+		limit = strconv.Itoa(int(JobTimeout/time.Hour)) + "h"
+	}
+	return "the crawl reached its time limit of " + limit
+}
+
+// finalStatus decides a crawl's final status and error: a requested
+// cancel, or a cancelled job context, ends cancelled; the job's deadline
+// fails it with the time-limit error; another run error fails it with that
+// error; else it finished (cancelled when the crawl stopped on a cancel).
+func finalStatus(requested bool, res *Result, runErr, ctxErr error) (string, string) {
+	switch {
+	case requested:
+		return models.CrawlCancelled, ""
+	case errors.Is(ctxErr, context.DeadlineExceeded) || errors.Is(runErr, context.DeadlineExceeded):
+		return models.CrawlFailed, timeLimitError()
+	case errors.Is(ctxErr, context.Canceled) || errors.Is(runErr, context.Canceled):
+		return models.CrawlCancelled, ""
+	case runErr != nil:
+		return models.CrawlFailed, runErr.Error()
+	case res != nil && res.Canceled:
+		return models.CrawlCancelled, ""
+	}
+	return models.CrawlFinished, ""
+}
+
+// finish stores what a crawl found and writes its final state. parent is
+// Run's context detached from the job's cancellation, gdb the undecorated
+// database, ctxErr the job context's error (nil while it was alive). The
+// cancel check, the limits and the store run on a context bounded by
+// storeBound; a store that runs past it fails the crawl with an error that
+// names the bound and the store error. The final status write
+// and the end of the conga record then run on a fresh context bounded by
+// finalWriteTimeout, and Prune on one of its own. A LoadLimits failure
+// falls back to the default limits.
+func finish(parent context.Context, svc *classes.Service, gdb *gorm.DB, row models.Crawl, res *Result, runErr, ctxErr error, storeBound time.Duration) error {
+	storeCtx, cancelStore := context.WithTimeout(parent, storeBound)
+	defer cancelStore()
+	db := gdb.WithContext(storeCtx)
+	status, errText := finalStatus(cancelRequested(db, row.ID), res, runErr, ctxErr)
+	limits, err := LoadLimits(db)
+	if err != nil {
+		pluginLog.Warn("golem15.seo: crawl limits, using the defaults", "crawl", row.ID, "error", err)
+		limits = Limits{MaxPages: DefaultMaxPages, Delay: DefaultDelay, Keep: DefaultKeep}
+	}
+	store := func(ctx context.Context, tx *gorm.DB) error {
+		if res == nil {
+			return nil
+		}
+		if err := SavePages(tx, row.ID, res); err != nil {
+			return err
+		}
+		keyphrases, err := Keyphrases(tx)
+		if err != nil {
+			return err
+		}
+		findings := Check(res, &Env{Lookup: Lookup(ctx, svc, tx), Staging: svc.Staging(), GenericAnchors: limits.GenericAnchors, Keyphrases: keyphrases, SitemapTool: SitemapTool(svc)})
+		cms, err := svc.Issues(classes.WithTx(ctx, tx))
+		if err != nil {
+			return err
+		}
+		id := row.ID
+		return SaveIssues(tx, &id, findings, cms)
+	}
+	if err := lagoon.Transaction(storeCtx, db, store); err != nil {
+		status, errText = models.CrawlFailed, err.Error()
+		if errors.Is(storeCtx.Err(), context.DeadlineExceeded) {
+			errText = "storing the crawl results took longer than " + storeBound.String() + ": " + err.Error()
+		}
+	}
+	ctx, cancelWrite := context.WithTimeout(parent, finalWriteTimeout)
+	defer cancelWrite()
+	updates := map[string]any{"status": status, "error": errText, "finished_at": time.Now().UTC()}
+	if res != nil {
+		updates["pages"] = len(res.Order)
+		updates["limit_hit"] = res.LimitHit
+		updates["render_status"] = res.Render.Status
+		updates["render_pages"] = res.Render.Pages
+		updates["render_note"] = res.Render.Note
+	}
+	if err := gdb.WithContext(ctx).Model(&models.Crawl{}).Where("id = ?", row.ID).Updates(updates).Error; err != nil {
+		return err
+	}
+	if m, err := conga.From(svc.App()); err == nil && row.JobID != nil {
+		meta := map[string]any{"crawl_id": row.ID, "status": status}
+		switch status {
+		case models.CrawlFailed:
+			_ = m.FailJob(ctx, *row.JobID, meta)
+		case models.CrawlCancelled:
+			_ = m.StopJob(ctx, *row.JobID, meta)
+		default:
+			_ = m.CompleteJob(ctx, *row.JobID, meta)
+		}
+	}
+	if status == models.CrawlFailed {
+		pluginLog.Warn("golem15.seo: crawl failed", "crawl", row.ID, "error", errText)
+	}
+	// Prune deletes old crawls with their pages and links, so it keeps a
+	// store-sized bound; it runs after the final state is written, so it
+	// never holds a crawl running.
+	pruneCtx, cancelPrune := context.WithTimeout(parent, storeBound)
+	defer cancelPrune()
+	if err := Prune(gdb.WithContext(pruneCtx), limits.Keep); err != nil {
+		pluginLog.Warn("golem15.seo: prune crawls", "error", err)
+	}
+	return nil
+}
+
+// endStaleJobs fails or stops the conga jobs of crawls RecoverStale ended,
+// once tx commits; best effort.
+func endStaleJobs(ctx context.Context, svc *classes.Service, tx *gorm.DB, rows []models.Crawl) {
+	if len(rows) == 0 {
+		return
+	}
+	lagoon.AfterCommit(ctx, tx, func(ctx context.Context, _ *gorm.DB) {
+		m, err := conga.From(svc.App())
+		if err != nil {
+			return
+		}
+		for _, r := range rows {
+			if r.JobID == nil {
+				continue
+			}
+			meta := map[string]any{"crawl_id": r.ID, "status": r.Status}
+			if r.Status == models.CrawlCancelled {
+				_ = m.StopJob(ctx, *r.JobID, meta)
+			} else {
+				_ = m.FailJob(ctx, *r.JobID, meta)
+			}
+		}
+	})
+}
+
+// Lookup returns the CMS record behind a crawled address (cached per
+// address): the content type whose Match answers the path exactly, its
+// descriptor facts in the matched locale, its primary parent's address and
+// whether its SEO tab stores custom JSON-LD.
+func Lookup(ctx context.Context, svc *classes.Service, db *gorm.DB) func(key string) *Record {
+	cache := map[string]*Record{}
+	reg := svc.Registry()
+	unprefixed := svc.UnprefixedLocale()
+	return func(key string) *Record {
+		if reg == nil {
+			return nil
+		}
+		path, _, _ := strings.Cut(key, "?")
+		if r, ok := cache[path]; ok {
+			return r
+		}
+		cache[path] = nil
+		typeKey, match, err := reg.MatchPath(ctx, path, unprefixed)
+		if err != nil || match == nil || !match.Exact {
+			return nil
+		}
+		typ, ok := reg.ContentType(typeKey)
+		if !ok || typ.Content == nil {
+			return nil
+		}
+		rec, err := reg.Describe(ctx, typeKey, match.ID, unprefixed)
+		if err != nil || rec == nil {
+			return nil
+		}
+		out := &Record{Type: typeKey, ID: match.ID, Locale: match.Locale, TopHeading: rec.TopHeading, Tools: typ.Content.Tools}
+		var loc *pact.ContentLocale
+		for i := range rec.Locales {
+			if rec.Locales[i].Locale == match.Locale {
+				loc = &rec.Locales[i]
+				break
+			}
+		}
+		if loc == nil && len(rec.Locales) > 0 {
+			loc = &rec.Locales[0]
+		}
+		if loc != nil {
+			out.Locale = loc.Locale
+			out.Title, out.Words, out.Headings, out.Images = loc.Title, loc.Words, loc.Headings, loc.Images
+		}
+		if len(rec.Parents) > 0 {
+			if parent, err := reg.Describe(ctx, rec.Parents[0].Type, rec.Parents[0].ID, unprefixed); err == nil && parent != nil {
+				for _, pl := range parent.Locales {
+					if pl.Locale == out.Locale && pl.URL != "" {
+						out.Links = append(out.Links, pl.URL)
+					}
+				}
+			}
+		}
+		if entry, found, err := classes.LoadEntry(db, typ.Content.Model, match.ID, out.Locale); err == nil && found && entry.Data != nil {
+			if raw, _ := entry.Data["schema"].(string); raw != "" {
+				if v, err := schema.ParseValue(raw); err == nil && strings.TrimSpace(v.Custom) != "" {
+					out.CustomJSONLD = true
+				}
+			}
+		}
+		cache[path] = out
+		return out
+	}
+}
+
+// Keyphrases maps each address with an SEO entry to its focus keyphrase.
+func Keyphrases(db *gorm.DB) (map[string]string, error) {
+	var rows []struct {
+		URL       string
+		Keyphrase string
+	}
+	if err := db.Raw(`SELECT url, keyphrase FROM golem15_seo_entries WHERE url <> '' AND keyphrase <> ''`).Scan(&rows).Error; err != nil {
+		return nil, err
+	}
+	out := make(map[string]string, len(rows))
+	for _, r := range rows {
+		out[r.URL] = r.Keyphrase
+	}
+	return out, nil
+}
+
+// errCancelledCrawl refuses cancelling a crawl that is not queued or
+// running.
+var errCancelledCrawl = errors.New("golem15.seo: the crawl is not running")
+
+// Cancel stops a queued or running crawl inside tx. A queued crawl ends
+// cancelled at once (by name, finished_at set). A running crawl gets
+// cancel_requested_at and cancelled_by and stays running: its job stops
+// between requests and writes cancelled with the pages and issues fetched
+// so far, and until then no other crawl starts. The conga job is cancelled
+// in both cases.
+func Cancel(ctx context.Context, svc *classes.Service, tx *gorm.DB, row models.Crawl, name string) error {
+	if !row.Active() {
+		return errCancelledCrawl
+	}
+	now := time.Now().UTC()
+	queued := tx.Model(&models.Crawl{}).Where("id = ? AND status = ?", row.ID, models.CrawlQueued).
+		Updates(map[string]any{"status": models.CrawlCancelled, "cancelled_by": name, "finished_at": now})
+	if queued.Error != nil {
+		return queued.Error
+	}
+	if queued.RowsAffected == 0 {
+		running := tx.Model(&models.Crawl{}).Where("id = ? AND status = ?", row.ID, models.CrawlRunning).
+			Updates(map[string]any{"cancel_requested_at": now, "cancelled_by": name})
+		if running.Error != nil {
+			return running.Error
+		}
+		if running.RowsAffected == 0 {
+			return errCancelledCrawl
+		}
+	}
+	if row.JobID != nil {
+		if m, err := conga.From(svc.App()); err == nil {
+			if err := m.CancelJob(ctx, *row.JobID); err != nil {
+				pluginLog.Warn("golem15.seo: cancel crawl job", "crawl", row.ID, "error", err)
+			}
+		}
+	}
+	return nil
+}
+
+// ErrNotRunning is a cancel of a crawl that is not queued or running.
+var ErrNotRunning = errCancelledCrawl
+
+// pluginLog is the crawler's logger (component seo).
+var pluginLog = diary.For("seo")
+
+// ScheduledDue reports whether the nightly entry starts a crawl on day
+// (in the app's location) for the crawl_schedule setting: daily every
+// night, weekly on Mondays, never when off (P33-40).
+func ScheduledDue(schedule string, day time.Time) bool {
+	switch schedule {
+	case models.CrawlDaily:
+		return true
+	case models.CrawlWeekly:
+		return day.Weekday() == time.Monday
+	}
+	return false
+}

classes/crawl/links.go

@@ -0,0 +1,102 @@
+package crawl
+
+import (
+	"strings"
+
+	"git.golem15.com/golem15/summercms/modules/pact"
+)
+
+// DefaultGenericAnchors are the generic link texts of the en and pl seed
+// lists (UI-SPEC C27, Q-1): the Site audit settings start with them.
+var DefaultGenericAnchors = []string{
+	"click here", "here", "more", "read more", "learn more", "this link", "link",
+	"kliknij tutaj", "tutaj", "tu", "więcej", "czytaj więcej", "czytaj dalej", "ten link",
+}
+
+// ParseAnchors splits the generic anchors setting (one per line).
+func ParseAnchors(text string) []string {
+	var out []string
+	for _, line := range strings.Split(text, "\n") {
+		if v := normal(line); v != "" {
+			out = append(out, v)
+		}
+	}
+	return out
+}
+
+// maxLinkedFrom caps the linking pages a link finding keeps.
+const maxLinkedFrom = 10
+
+// checkLinks reports, per linked address, broken_link (404 or 410) and
+// redirected_link (3xx) with the pages linking to it.
+func checkLinks(res *Result, env *Env) []Finding {
+	var out []Finding
+	links := inbound(res)
+	for _, key := range res.Order {
+		p := res.Pages[key]
+		from := links[key]
+		if len(from) == 0 || p.Resp == nil || p.Resp.Err != "" {
+			continue
+		}
+		kept := from
+		if len(kept) > maxLinkedFrom {
+			kept = kept[:maxLinkedFrom]
+		}
+		rec := env.record(key)
+		switch s := p.Resp.Status; {
+		case s == 404 || s == 410:
+			out = append(out, Finding{Kind: KindBrokenLink, URL: key, Locale: locale(rec), Record: rec, Tool: ToolCreateRedirect, Details: map[string]any{"status": s, "count": len(from), "from": kept}})
+		case s >= 300 && s < 400:
+			target := p.Resp.Location
+			if u, err := parseURL(target); err == nil && res.fetcher != nil {
+				if k := res.fetcher.Key(u); k != "" {
+					target = k
+				}
+			}
+			// The link sits on the linking pages: their record's edit tool
+			// fixes it, else the template code.
+			linking := env.record(from[0])
+			f := Finding{Kind: KindRedirectedLink, URL: key, Locale: locale(rec), Record: linking, Details: map[string]any{"status": s, "target": target, "count": len(from), "from": kept}}
+			if linking != nil {
+				f.Content = []string{pact.ContentToolEdit}
+			}
+			out = append(out, f)
+		}
+	}
+	return out
+}
+
+// checkGenericAnchors reports a page whose links have a generic text
+// (D-81): the whole text, any case, is one of the generic anchors.
+func checkGenericAnchors(res *Result, p *Page, rec *Record, anchors []string) []Finding {
+	if len(anchors) == 0 {
+		return nil
+	}
+	set := map[string]bool{}
+	for _, a := range anchors {
+		set[normal(a)] = true
+	}
+	var found []map[string]any
+	for _, l := range p.Facts.Links {
+		if !set[normal(l.Text)] {
+			continue
+		}
+		target := l.URL.String()
+		if res.fetcher != nil {
+			if k := res.fetcher.Key(l.URL); k != "" {
+				target = k
+			}
+		}
+		found = append(found, map[string]any{"text": l.Text, "target": target})
+	}
+	if len(found) == 0 {
+		return nil
+	}
+	f := Finding{Kind: KindGenericAnchor, URL: p.Key, Locale: locale(rec), Record: rec, Details: map[string]any{
+		"count": len(found), "text": found[0]["text"], "target": found[0]["target"], "anchors": found,
+	}}
+	if rec != nil {
+		f.Content = []string{pact.ContentToolEdit}
+	}
+	return []Finding{f}
+}

classes/crawl/orphans.go

@@ -0,0 +1,79 @@
+package crawl
+
+import (
+	"strings"
+
+	"git.golem15.com/golem15/summercms/modules/pact"
+)
+
+// maxSuggestions caps the pages suggested to link to an orphan.
+const maxSuggestions = 10
+
+// minPhrase is the shortest phrase a suggestion matches (in bytes).
+const minPhrase = 3
+
+// checkOrphans reports orphan_page (D-82): an indexable page a sitemap
+// lists that no crawled page links to (the home page is the crawl's
+// entry and never an orphan). Its suggestions are the pages whose main
+// text mentions the orphan's title or its focus keyphrase; the first
+// suggestion's record names the tool that adds the link, else the fix is
+// a link in the site's code.
+func checkOrphans(res *Result, env *Env) []Finding {
+	links := inbound(res)
+	var out []Finding
+	for _, key := range res.Order {
+		p := res.Pages[key]
+		if key == HomePath || !p.InSitemap || !env.indexable(res, p) || len(links[key]) > 0 {
+			continue
+		}
+		rec := env.record(key)
+		var phrases []string
+		add := func(s string) {
+			if s = normal(s); len(s) >= minPhrase {
+				for _, have := range phrases {
+					if have == s {
+						return
+					}
+				}
+				phrases = append(phrases, s)
+			}
+		}
+		if rec != nil {
+			add(rec.Title)
+		}
+		if h1 := p.Facts.H1(); len(h1) > 0 {
+			add(h1[0])
+		}
+		add(p.Facts.Title)
+		if env.Keyphrases != nil {
+			add(env.Keyphrases[key])
+		}
+		var suggestions []map[string]any
+		var first *Record
+		for _, other := range res.Order {
+			if len(suggestions) >= maxSuggestions {
+				break
+			}
+			op := res.Pages[other]
+			if other == key || !op.OK() {
+				continue
+			}
+			text := normal(op.Facts.Text)
+			for _, phrase := range phrases {
+				if strings.Contains(text, phrase) {
+					suggestions = append(suggestions, map[string]any{"url": other, "phrase": phrase})
+					if first == nil {
+						first = env.record(other)
+					}
+					break
+				}
+			}
+		}
+		f := Finding{Kind: KindOrphanPage, URL: key, Locale: locale(rec), Record: first, Details: map[string]any{"count": len(suggestions), "suggestions": suggestions}}
+		if first != nil {
+			f.Content = []string{pact.ContentToolEdit}
+		}
+		out = append(out, f)
+	}
+	return out
+}

classes/crawl/parse.go

@@ -0,0 +1,293 @@
+package crawl
+
+import (
+	"bytes"
+	"net/url"
+	"strings"
+
+	"golang.org/x/net/html"
+	"golang.org/x/net/html/atom"
+)
+
+// Heading is one heading of a page in document order.
+type Heading struct {
+	Level int    `json:"level"`
+	Text  string `json:"text"`
+}
+
+// Link is one <a href> of a page, resolved against the page's address.
+type Link struct {
+	URL  *url.URL
+	Href string
+	Text string
+	Rel  string
+}
+
+// Image is one <img> of a page.
+type Image struct {
+	Src     string
+	Alt     string
+	HasAlt  bool
+	Width   string
+	Height  string
+	Loading string
+}
+
+// Alternate is one <link rel="alternate" hreflang>.
+type Alternate struct {
+	Lang string
+	URL  *url.URL
+	Href string
+}
+
+// PageFacts is what one HTML document says, parsed with
+// golang.org/x/net/html (no regular expressions over HTML): the language,
+// title, meta description, canonical, meta robots, og:image, headings,
+// links, images, JSON-LD scripts, hreflang alternates and the main text
+// with its word count. The rendered comparison parses the JavaScript
+// rendered DOM into the same facts.
+type PageFacts struct {
+	Lang           string
+	Title          string
+	HasTitle       bool
+	Description    string
+	HasDescription bool
+	Canonical      string
+	Robots         string
+	OGImage        string
+	Headings       []Heading
+	Links          []Link
+	Images         []Image
+	JSONLD         []string
+	Alternates     []Alternate
+	Text           string
+	Words          int
+}
+
+// H1 returns the texts of the page's level-1 headings.
+func (p *PageFacts) H1() []string {
+	var out []string
+	for _, h := range p.Headings {
+		if h.Level == 1 {
+			out = append(out, h.Text)
+		}
+	}
+	return out
+}
+
+// Noindex reports whether the meta robots value says noindex.
+func (p *PageFacts) Noindex() bool { return hasNoindex(p.Robots) }
+
+// hasNoindex reports whether a robots value contains noindex or none.
+func hasNoindex(v string) bool {
+	for _, part := range strings.FieldsFunc(strings.ToLower(v), func(r rune) bool { return r == ',' || r == ' ' || r == ';' }) {
+		if part == "noindex" || part == "none" {
+			return true
+		}
+	}
+	return false
+}
+
+// Parse reads the facts of an HTML document fetched from base.
+func Parse(base *url.URL, body []byte) *PageFacts {
+	out := &PageFacts{}
+	doc, err := html.Parse(bytes.NewReader(body))
+	if err != nil {
+		return out
+	}
+	resolve := func(href string) *url.URL {
+		href = strings.TrimSpace(href)
+		if href == "" {
+			return nil
+		}
+		ref, err := url.Parse(href)
+		if err != nil {
+			return nil
+		}
+		if base == nil {
+			return ref
+		}
+		u := base.ResolveReference(ref)
+		u.Fragment = ""
+		return u
+	}
+	var main *html.Node
+	var bodyNode *html.Node
+	var walk func(n *html.Node)
+	walk = func(n *html.Node) {
+		if n.Type == html.ElementNode {
+			switch n.DataAtom {
+			case atom.Html:
+				out.Lang = strings.TrimSpace(attr(n, "lang"))
+			case atom.Title:
+				if !out.HasTitle {
+					out.HasTitle = true
+					out.Title = collapse(textOf(n))
+				}
+			case atom.Meta:
+				name := strings.ToLower(attr(n, "name"))
+				prop := strings.ToLower(attr(n, "property"))
+				switch {
+				case name == "description" && !out.HasDescription:
+					out.HasDescription = true
+					out.Description = collapse(attr(n, "content"))
+				case name == "robots" || name == "googlebot":
+					if v := strings.TrimSpace(attr(n, "content")); v != "" {
+						if out.Robots != "" {
+							out.Robots += ", "
+						}
+						out.Robots += v
+					}
+				case prop == "og:image" && out.OGImage == "":
+					if u := resolve(attr(n, "content")); u != nil {
+						out.OGImage = u.String()
+					}
+				}
+			case atom.Link:
+				rel := strings.ToLower(attr(n, "rel"))
+				switch {
+				case relHas(rel, "canonical") && out.Canonical == "":
+					if u := resolve(attr(n, "href")); u != nil {
+						out.Canonical = u.String()
+					}
+				case relHas(rel, "alternate") && hasAttr(n, "hreflang"):
+					href := attr(n, "href")
+					out.Alternates = append(out.Alternates, Alternate{Lang: strings.TrimSpace(attr(n, "hreflang")), URL: resolve(href), Href: href})
+				}
+			case atom.H1, atom.H2, atom.H3, atom.H4, atom.H5, atom.H6:
+				out.Headings = append(out.Headings, Heading{Level: int(n.Data[1] - '0'), Text: collapse(textOf(n))})
+			case atom.A:
+				if hasAttr(n, "href") && len(out.Links) < MaxLinksPerPage {
+					href := attr(n, "href")
+					low := strings.ToLower(strings.TrimSpace(href))
+					if !strings.HasPrefix(low, "javascript:") && !strings.HasPrefix(low, "mailto:") && !strings.HasPrefix(low, "tel:") && !strings.HasPrefix(low, "#") {
+						if u := resolve(href); u != nil {
+							out.Links = append(out.Links, Link{URL: u, Href: href, Text: collapse(textOf(n)), Rel: strings.ToLower(attr(n, "rel"))})
+						}
+					}
+				}
+			case atom.Img:
+				alt, hasAlt := attrOK(n, "alt")
+				src := attr(n, "src")
+				if u := resolve(src); u != nil {
+					src = u.String()
+				}
+				out.Images = append(out.Images, Image{Src: src, Alt: alt, HasAlt: hasAlt, Width: attr(n, "width"), Height: attr(n, "height"), Loading: strings.ToLower(attr(n, "loading"))})
+			case atom.Script:
+				if strings.EqualFold(strings.TrimSpace(attr(n, "type")), "application/ld+json") {
+					out.JSONLD = append(out.JSONLD, textOf(n))
+				}
+				return
+			case atom.Main:
+				if main == nil {
+					main = n
+				}
+			case atom.Body:
+				bodyNode = n
+			}
+		}
+		for c := n.FirstChild; c != nil; c = c.NextSibling {
+			walk(c)
+		}
+	}
+	walk(doc)
+	root := main
+	if root == nil {
+		root = bodyNode
+	}
+	if root != nil {
+		var b strings.Builder
+		visibleText(root, &b)
+		out.Text = collapse(b.String())
+		if len(out.Text) > maxText {
+			out.Text = truncateUTF8(out.Text, maxText)
+		}
+		out.Words = len(strings.Fields(out.Text))
+	}
+	return out
+}
+
+// maxText caps the main text kept per page (characters of UTF-8, bytes
+// bound).
+const maxText = 100_000
+
+func truncateUTF8(s string, n int) string {
+	if len(s) <= n {
+		return s
+	}
+	for n > 0 && n < len(s) && (s[n]&0xC0) == 0x80 {
+		n--
+	}
+	return s[:n]
+}
+
+// visibleText writes the text of n without script, style, noscript and
+// template content, one space between block parts.
+func visibleText(n *html.Node, b *strings.Builder) {
+	if n.Type == html.ElementNode {
+		switch n.DataAtom {
+		case atom.Script, atom.Style, atom.Noscript, atom.Template, atom.Svg:
+			return
+		}
+	}
+	if n.Type == html.TextNode {
+		b.WriteString(n.Data)
+		b.WriteByte(' ')
+		return
+	}
+	for c := n.FirstChild; c != nil; c = c.NextSibling {
+		visibleText(c, b)
+	}
+}
+
+func textOf(n *html.Node) string {
+	var b strings.Builder
+	var walk func(*html.Node)
+	walk = func(n *html.Node) {
+		if n.Type == html.TextNode {
+			b.WriteString(n.Data)
+			b.WriteByte(' ')
+		}
+		if n.Type == html.ElementNode && n.DataAtom == atom.Img {
+			if alt := attr(n, "alt"); alt != "" {
+				b.WriteString(alt)
+				b.WriteByte(' ')
+			}
+		}
+		for c := n.FirstChild; c != nil; c = c.NextSibling {
+			walk(c)
+		}
+	}
+	walk(n)
+	return strings.TrimSpace(b.String())
+}
+
+func collapse(s string) string { return strings.Join(strings.Fields(s), " ") }
+
+func attrOK(n *html.Node, key string) (string, bool) {
+	for _, a := range n.Attr {
+		if strings.EqualFold(a.Key, key) {
+			return a.Val, true
+		}
+	}
+	return "", false
+}
+
+func attr(n *html.Node, key string) string {
+	v, _ := attrOK(n, key)
+	return v
+}
+
+func hasAttr(n *html.Node, key string) bool {
+	_, ok := attrOK(n, key)
+	return ok
+}
+
+func relHas(rel, want string) bool {
+	for _, part := range strings.Fields(rel) {
+		if part == want {
+			return true
+		}
+	}
+	return false
+}

classes/crawl/parse_test.go

@@ -0,0 +1,128 @@
+package crawl
+
+import (
+	"net/url"
+	"strings"
+	"testing"
+)
+
+const fixtureHTML = `<!doctype html>
+<html lang="pl-PL"><head>
+<title>  Grzyby   lecznicze </title>
+<title>Second</title>
+<meta name="description" content=" O grzybach ">
+<meta name="robots" content="index, follow">
+<meta name="googlebot" content="noimageindex">
+<meta property="og:image" content="/media/cover.jpg">
+<link rel="canonical" href="/blog/grzyby#top">
+<link rel="canonical" href="/other">
+<link rel="alternate" hreflang="en" href="/en/blog/mushrooms">
+<link rel="alternate" hreflang="x-default" href="https://example.test/blog/grzyby">
+<link rel="alternate" type="application/rss+xml" href="/feed.xml">
+<script type="application/ld+json">{"@type":"WebPage"}</script>
+<script>var x = "<h1>not a heading</h1>";</script>
+</head><body>
+<header><a href="/">Start</a></header>
+<main>
+<h1>Grzyby <img src="/i/logo.png" alt="lecznicze"></h1>
+<h2></h2>
+<h4>Detail</h4>
+<p>Raz dwa trzy.</p>
+<a href="/blog/chaga?x=1#frag" rel="NoFollow">Chaga</a>
+<a href="javascript:void(0)">js</a><a href="mailto:a@b.c">mail</a><a href="tel:1">tel</a><a href="#top">top</a>
+<a href="https://other.test/x">Other</a>
+<img src="/media/a.jpg" alt="" width="10" height="5" loading="LAZY">
+<img src="/media/b.jpg">
+<noscript>hidden text</noscript>
+<style>.x{}</style>
+</main>
+<footer>Stopka</footer>
+</body></html>`
+
+// TestParseFacts pins Parse on an HTML fixture: language, the first title
+// collapsed, description, robots joined from robots and googlebot,
+// og:image and the first canonical resolved without the fragment,
+// hreflang alternates only, JSON-LD scripts only, headings in order (an
+// image alt counts as heading text, script content never), links without
+// javascript:, mailto:, tel: and fragment-only ones, images with and
+// without alt, and the main text without noscript and style.
+func TestParseFacts(t *testing.T) {
+	base, _ := url.Parse("https://example.test/blog/grzyby")
+	f := Parse(base, []byte(fixtureHTML))
+	if f.Lang != "pl-PL" || !f.HasTitle || f.Title != "Grzyby lecznicze" || !f.HasDescription || f.Description != "O grzybach" {
+		t.Fatalf("head = %+v", f)
+	}
+	if f.Robots != "index, follow, noimageindex" || f.Noindex() {
+		t.Fatalf("robots = %q", f.Robots)
+	}
+	if f.OGImage != "https://example.test/media/cover.jpg" || f.Canonical != "https://example.test/blog/grzyby" {
+		t.Fatalf("og:image %q canonical %q", f.OGImage, f.Canonical)
+	}
+	if len(f.Alternates) != 2 || f.Alternates[0].Lang != "en" || f.Alternates[0].URL.String() != "https://example.test/en/blog/mushrooms" || f.Alternates[1].Lang != "x-default" {
+		t.Fatalf("alternates = %+v", f.Alternates)
+	}
+	if len(f.JSONLD) != 1 || f.JSONLD[0] != `{"@type":"WebPage"}` {
+		t.Fatalf("json-ld = %q", f.JSONLD)
+	}
+	if len(f.Headings) != 3 || f.Headings[0] != (Heading{1, "Grzyby lecznicze"}) || f.Headings[1] != (Heading{2, ""}) || f.Headings[2] != (Heading{4, "Detail"}) {
+		t.Fatalf("headings = %+v", f.Headings)
+	}
+	if h1 := f.H1(); len(h1) != 1 || h1[0] != "Grzyby lecznicze" {
+		t.Fatalf("H1 = %v", h1)
+	}
+	var links []string
+	for _, l := range f.Links {
+		links = append(links, l.URL.String())
+	}
+	if strings.Join(links, " ") != "https://example.test/ https://example.test/blog/chaga?x=1 https://other.test/x" || f.Links[1].Rel != "nofollow" || f.Links[1].Text != "Chaga" {
+		t.Fatalf("links = %v (%+v)", links, f.Links)
+	}
+	if len(f.Images) != 3 || !f.Images[1].HasAlt || f.Images[1].Alt != "" || f.Images[1].Loading != "lazy" || f.Images[2].HasAlt || f.Images[2].Src != "https://example.test/media/b.jpg" {
+		t.Fatalf("images = %+v", f.Images)
+	}
+	if strings.Contains(f.Text, "hidden text") || strings.Contains(f.Text, ".x{}") || strings.Contains(f.Text, "Stopka") || !strings.Contains(f.Text, "Raz dwa trzy.") || f.Words != len(strings.Fields(f.Text)) {
+		t.Fatalf("text = %q (%d words)", f.Text, f.Words)
+	}
+}
+
+// TestParseBodyText pins the text root: main when there is one, else the
+// body; and an empty document parses to empty facts.
+func TestParseBodyText(t *testing.T) {
+	f := Parse(nil, []byte(`<html><body><nav>Menu</nav><p>One two</p></body></html>`))
+	if f.Text != "Menu One two" || f.Words != 3 || f.HasTitle {
+		t.Fatalf("body text = %q", f.Text)
+	}
+	if empty := Parse(nil, nil); empty.Words != 0 || len(empty.Links) != 0 {
+		t.Fatalf("empty = %+v", empty)
+	}
+}
+
+// TestNoindexValues pins the robots values that mean noindex.
+func TestNoindexValues(t *testing.T) {
+	for v, want := range map[string]bool{"noindex": true, "NOINDEX, follow": true, "none": true, "index;noindex": true, "index, follow": false, "": false, "noindexing": false} {
+		if got := hasNoindex(v); got != want {
+			t.Errorf("hasNoindex(%q) = %v, want %v", v, got, want)
+		}
+	}
+}
+
+// TestParseCaps pins the link cap and the text cap with a UTF-8 safe cut.
+func TestParseCaps(t *testing.T) {
+	var b strings.Builder
+	b.WriteString("<main>")
+	for i := 0; i < MaxLinksPerPage+10; i++ {
+		b.WriteString(`<a href="/x">x</a>`)
+	}
+	b.WriteString("</main>")
+	if got := len(Parse(nil, []byte(b.String())).Links); got != MaxLinksPerPage {
+		t.Fatalf("%d links, want the %d cap", got, MaxLinksPerPage)
+	}
+	long := "<main>" + strings.Repeat("żółw ", maxText/4) + "</main>"
+	f := Parse(nil, []byte(long))
+	if len(f.Text) > maxText || !strings.HasPrefix(f.Text, "żółw") || strings.ContainsRune(f.Text, '<27>') {
+		t.Fatalf("text of %d bytes", len(f.Text))
+	}
+	if truncateUTF8("żółw", 2) != "ż" || truncateUTF8("ab", 5) != "ab" {
+		t.Fatal("truncateUTF8 is wrong")
+	}
+}

classes/crawl/render.go

@@ -0,0 +1,194 @@
+package crawl
+
+import (
+	"context"
+	"fmt"
+	"net/netip"
+	"net/url"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"time"
+
+	"git.golem15.com/golem15/summercms/modules/compass"
+)
+
+// Renderer renders a page of the site in a browser for the raw versus
+// rendered comparison (D-70, P33-55): Render answers the page's DOM after
+// JavaScript ran, Close releases the browser. A crawl without a Renderer
+// records the comparison as skipped; a Render error records it as failed
+// with the error and never fails the crawl.
+type Renderer interface {
+	Render(ctx context.Context, pageURL string) (string, error)
+	Close() error
+}
+
+// DefaultRenderMax is the number of pages rendered per crawl when
+// golem15.seo.crawl_render_max is unset.
+const DefaultRenderMax = 200
+
+// The render states of a crawl (models.RenderSkipped and the others).
+const (
+	renderStatusSkipped = "skipped"
+	renderStatusDone    = "done"
+	renderStatusFailed  = "failed"
+)
+
+type rendererKey struct{}
+
+// WithRenderer returns a context whose crawls use r instead of the
+// renderer built from config. It is the test seam of the comparison,
+// settable only from Go code (like fetchguard.WithTransport): no config
+// key, setting, request or MCP call reaches it.
+func WithRenderer(ctx context.Context, r Renderer) context.Context {
+	return context.WithValue(ctx, rendererKey{}, r)
+}
+
+// rendererFrom returns the renderer of ctx, nil when none.
+func rendererFrom(ctx context.Context) Renderer {
+	if ctx == nil {
+		return nil
+	}
+	r, _ := ctx.Value(rendererKey{}).(Renderer)
+	return r
+}
+
+// The config keys of the rendered comparison (P33-55). The executable
+// path is Go config only (environment-overridable like every key), never
+// an admin setting or MCP value: an editable path would let any holder of
+// golem15.seo.manage make the server run an arbitrary binary.
+const (
+	// ChromePathConfigKey names an operator-installed Chrome or Chromium
+	// binary; empty skips the comparison.
+	ChromePathConfigKey = "golem15.seo.crawl_chrome_path"
+	// RenderMaxConfigKey caps the pages rendered per crawl.
+	RenderMaxConfigKey = "golem15.seo.crawl_render_max"
+)
+
+// The limits of one rendered page.
+const (
+	RenderTimeout = 20 * time.Second
+	MaxDOM        = 5 << 20
+)
+
+// RenderMax is the configured page cap of the comparison (default 200).
+func RenderMax(cfg *compass.Config) int {
+	if cfg != nil {
+		if n := cfg.Int(RenderMaxConfigKey); n > 0 {
+			return n
+		}
+	}
+	return DefaultRenderMax
+}
+
+// NewRenderer builds the renderer of a crawl of site from config: nil
+// without error when golem15.seo.crawl_chrome_path is empty (the
+// comparison is skipped); an error, recorded as the comparison's failure
+// note, when the path is not absolute, not an executable regular file, the
+// site host resolves to a private address outside trusted mode, or Chrome
+// does not start. ua is the crawler's User-Agent.
+func NewRenderer(ctx context.Context, cfg *compass.Config, site *url.URL, trusted bool, ua string) (Renderer, error) {
+	if cfg == nil || site == nil {
+		return nil, nil
+	}
+	path := strings.TrimSpace(cfg.String(ChromePathConfigKey))
+	if path == "" {
+		return nil, nil
+	}
+	if err := checkExecutable(path); err != nil {
+		return nil, err
+	}
+	hosts, err := resolveHosts(ctx, SiteHosts(site.Hostname()), trusted)
+	if err != nil {
+		return nil, err
+	}
+	r, err := newChromeRenderer(path, hosts, effectivePort(site), trusted, ua)
+	if err != nil {
+		return nil, err
+	}
+	return r, nil
+}
+
+// checkExecutable refuses a Chrome path that is not absolute or not an
+// executable regular file.
+func checkExecutable(path string) error {
+	if !filepath.IsAbs(path) {
+		return fmt.Errorf("the Chrome path %q is not absolute", path)
+	}
+	info, err := os.Stat(path)
+	if err != nil {
+		return fmt.Errorf("the Chrome path %q cannot be read", path)
+	}
+	if !info.Mode().IsRegular() || info.Mode().Perm()&0o111 == 0 {
+		return fmt.Errorf("the Chrome path %q is not an executable file", path)
+	}
+	return nil
+}
+
+// resolverRules is Chrome's --host-resolver-rules value: one MAP rule per
+// allowed host to one of its checked addresses (IPv4 first), then
+// MAP * ~NOTFOUND for every other host. Chrome applies the first rule
+// that matches a host, so the catch-all comes last.
+func resolverRules(hosts map[string][]netip.Addr) string {
+	names := make([]string, 0, len(hosts))
+	for h := range hosts {
+		names = append(names, h)
+	}
+	sort.Strings(names)
+	var rules []string
+	for _, h := range names {
+		addrs := hosts[h]
+		if len(addrs) == 0 {
+			continue
+		}
+		// One address per host: the first IPv4 one, else the first.
+		a := addrs[0]
+		for _, c := range addrs {
+			if c.Is4() {
+				a = c
+				break
+			}
+		}
+		target := a.String()
+		if a.Is6() {
+			target = "[" + target + "]"
+		}
+		rules = append(rules, "MAP "+h+" "+target)
+	}
+	rules = append(rules, "MAP * ~NOTFOUND")
+	return strings.Join(rules, ", ")
+}
+
+// allowRequest reports whether the headless browser may load rawURL: https
+// (http only in trusted mode) to one of hosts on the site's effective port
+// (port; the request's explicit port, else 443 for https and 80 for http),
+// or a data: or blob: URL. Every other request, third-party scripts and
+// other services on the site host's address included, is failed.
+func allowRequest(rawURL string, hosts []string, port string, trusted bool) bool {
+	u, err := url.Parse(rawURL)
+	if err != nil {
+		return false
+	}
+	switch strings.ToLower(u.Scheme) {
+	case "data", "blob":
+		return true
+	case "https":
+	case "http":
+		if !trusted {
+			return false
+		}
+	default:
+		return false
+	}
+	if u.User != nil || effectivePort(u) != port {
+		return false
+	}
+	host := strings.ToLower(u.Hostname())
+	for _, h := range hosts {
+		if host == strings.ToLower(strings.Trim(h, "[]")) {
+			return true
+		}
+	}
+	return false
+}

classes/crawl/render_chrome.go

@@ -0,0 +1,277 @@
+package crawl
+
+import (
+	"context"
+	"fmt"
+	"net"
+	"net/netip"
+	"os"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"git.golem15.com/golem15/summercms/modules/fetchguard"
+	"github.com/chromedp/cdproto/browser"
+	"github.com/chromedp/cdproto/cdp"
+	"github.com/chromedp/cdproto/fetch"
+	"github.com/chromedp/cdproto/network"
+	"github.com/chromedp/chromedp"
+)
+
+// chromeRenderer renders pages in an operator-installed headless Chrome
+// or Chromium through github.com/chromedp/chromedp (D-70, P33-55), held to
+// the crawler's fetchguard limits:
+//
+//   - Chrome resolves only the site host and its www twin, pinned through
+//     --host-resolver-rules to the addresses resolveHosts checked with
+//     fetchguard.IsPrivateAddr; every other host resolves to NOTFOUND;
+//   - every request is paused through the Fetch domain and continued only
+//     when allowRequest passes (https, http only in trusted mode, to an
+//     allowed host on the site's port, or data: and blob:); the others fail with
+//     BlockedByClient, so third-party scripts are never loaded;
+//   - a fresh temporary profile per crawl, removed by Close; no cookies,
+//     credentials or extensions; downloads denied;
+//   - one tab at a time (the crawler serialises Render), at most
+//     TabOpenTimeout (5 s) to open the tab and RenderTimeout (20 s) for the
+//     page, both under one deadline, so a hung Chrome never holds the
+//     crawl's render lock longer; the DOM capped at 5 MiB; Close stops
+//     Chrome.
+//
+// chromedp's default allocator flags are used as they are: nothing is
+// added that weakens Chrome's sandbox.
+type chromeRenderer struct {
+	allowed []string
+	port    string
+	trusted bool
+	profile string
+
+	allocCancel   context.CancelFunc
+	browserCtx    context.Context
+	browserCancel context.CancelFunc
+
+	// newTab and startTab open a tab: chromedp.NewContext and chromedp.Do
+	// unless a test replaces them.
+	newTab   func(parent context.Context) (context.Context, context.CancelFunc)
+	startTab func(tab context.Context) error
+
+	mu      sync.Mutex
+	blocked []string
+	closed  bool
+}
+
+// TabOpenTimeout bounds opening a Chrome tab (P33-60, L3 WR-06); Render
+// bounds opening the tab and rendering the page with one deadline of
+// TabOpenTimeout plus RenderTimeout.
+var TabOpenTimeout = 5 * time.Second
+
+// seconds writes d as whole seconds ("5 s") when it is one, else as
+// time.Duration does.
+func seconds(d time.Duration) string {
+	if d%time.Second == 0 {
+		return strconv.Itoa(int(d/time.Second)) + " s"
+	}
+	return d.String()
+}
+
+// newChromeRenderer starts Chrome at path for the hosts and their checked
+// addresses, on the site's effective port.
+func newChromeRenderer(path string, hosts map[string][]netip.Addr, port string, trusted bool, ua string) (*chromeRenderer, error) {
+	profile, err := os.MkdirTemp("", "summercms-seo-chrome-")
+	if err != nil {
+		return nil, fmt.Errorf("the Chrome profile cannot be created: %w", err)
+	}
+	r := &chromeRenderer{port: port, trusted: trusted, profile: profile}
+	for h := range hosts {
+		r.allowed = append(r.allowed, h)
+	}
+	sort.Strings(r.allowed)
+	opts := append(chromedp.DefaultExecAllocatorOptions[:],
+		chromedp.ExecPath(path),
+		chromedp.UserDataDir(profile),
+		chromedp.Flag("host-resolver-rules", resolverRules(hosts)),
+		chromedp.UserAgent(ua),
+	)
+	allocCtx, allocCancel := chromedp.NewExecAllocator(context.Background(), opts...)
+	r.allocCancel = allocCancel
+	r.browserCtx, r.browserCancel = chromedp.NewContext(allocCtx)
+	// The first Run starts the browser; it runs without a timeout, since a
+	// timeout on this context would stop the whole browser.
+	if err := chromedp.Do(r.browserCtx); err != nil {
+		_ = r.Close()
+		return nil, fmt.Errorf("Chrome did not start: %w", err)
+	}
+	if _, err := chromedp.CallBrowser(r.browserCtx, browser.SetDownloadBehavior, browser.SetDownloadBehaviorParams{Behavior: browser.SetDownloadBehaviorBehaviorDeny}); err != nil {
+		_ = r.Close()
+		return nil, fmt.Errorf("Chrome downloads cannot be denied: %w", err)
+	}
+	return r, nil
+}
+
+// Render loads pageURL in a new tab and answers the DOM after JavaScript
+// ran (the outer HTML of the document, capped at 5 MiB).
+func (r *chromeRenderer) Render(ctx context.Context, pageURL string) (string, error) {
+	r.mu.Lock()
+	closed := r.closed
+	r.mu.Unlock()
+	if closed {
+		return "", fmt.Errorf("Chrome is closed")
+	}
+	if !allowRequest(pageURL, r.allowed, r.port, r.trusted) {
+		return "", fmt.Errorf("the address %s is outside the site", pageURL)
+	}
+	// One deadline bounds opening the tab and rendering the page; the tab
+	// ends with it or with the crawl's context (a cancelled crawl).
+	rctx, rcancel := context.WithTimeout(ctx, TabOpenTimeout+RenderTimeout)
+	defer rcancel()
+	newTab, startTab := r.newTab, r.startTab
+	if newTab == nil {
+		newTab = func(parent context.Context) (context.Context, context.CancelFunc) { return chromedp.NewContext(parent) }
+	}
+	if startTab == nil {
+		startTab = func(tab context.Context) error { return chromedp.Do(tab) }
+	}
+	tab, closeTab := newTab(r.browserCtx)
+	defer closeTab()
+	stop := context.AfterFunc(rctx, closeTab)
+	defer stop()
+	opened := make(chan error, 1)
+	go func() { opened <- startTab(tab) }()
+	openTimer := time.NewTimer(TabOpenTimeout)
+	defer openTimer.Stop()
+	select {
+	case err := <-opened:
+		if err != nil {
+			if ctx.Err() != nil {
+				return "", ctx.Err()
+			}
+			return "", fmt.Errorf("Chrome cannot open a tab: %w", err)
+		}
+	case <-openTimer.C:
+		closeTab()
+		return "", fmt.Errorf("Chrome did not open a tab within %s", seconds(TabOpenTimeout))
+	case <-rctx.Done():
+		closeTab()
+		if ctx.Err() != nil {
+			return "", ctx.Err()
+		}
+		return "", fmt.Errorf("Chrome did not open a tab within %s", seconds(TabOpenTimeout))
+	}
+	page, cancel := context.WithTimeout(tab, RenderTimeout)
+	defer cancel()
+	target := chromedp.FromContext(tab).Target
+	paused := chromedp.Events(page, fetch.RequestPaused)
+	if _, err := chromedp.Call(page, fetch.Enable, fetch.EnableParams{Patterns: []*fetch.RequestPattern{{URLPattern: "*"}}}); err != nil {
+		return "", fmt.Errorf("Chrome cannot intercept requests: %w", err)
+	}
+	var wg sync.WaitGroup
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		for ev, err := range paused {
+			if err != nil {
+				return
+			}
+			r.decide(page, target, ev)
+		}
+	}()
+	html, err := chromedp.Run(page, func(ctx context.Context, t *chromedp.Target) (string, error) {
+		if _, err := chromedp.Navigate(pageURL)(ctx, t); err != nil {
+			return "", err
+		}
+		if _, err := chromedp.WaitReady(chromedp.CSS("body"))(ctx, t); err != nil {
+			return "", err
+		}
+		return chromedp.OuterHTML(chromedp.CSS("html"))(ctx, t)
+	})
+	cancel()
+	closeTab()
+	wg.Wait()
+	if err != nil {
+		if ctx.Err() != nil {
+			return "", ctx.Err()
+		}
+		return "", fmt.Errorf("Chrome could not render %s: %w", pageURL, err)
+	}
+	if len(html) > MaxDOM {
+		html = truncateUTF8(html, MaxDOM)
+	}
+	return html, nil
+}
+
+// decide continues a paused request that allowRequest passes and fails
+// every other one with BlockedByClient, keeping its address.
+func (r *chromeRenderer) decide(ctx context.Context, target cdp.Session, ev fetch.EventRequestPaused) {
+	u := ""
+	if ev.Request != nil {
+		u = ev.Request.URL
+	}
+	if allowRequest(u, r.allowed, r.port, r.trusted) {
+		_, _ = cdp.Call(ctx, target, fetch.ContinueRequest, fetch.ContinueRequestParams{RequestID: ev.RequestID})
+		return
+	}
+	r.mu.Lock()
+	r.blocked = append(r.blocked, u)
+	r.mu.Unlock()
+	_, _ = cdp.Call(ctx, target, fetch.FailRequest, fetch.FailRequestParams{RequestID: ev.RequestID, ErrorReason: network.ErrorReasonBlockedByClient})
+}
+
+// Blocked lists the addresses the browser was kept from loading.
+func (r *chromeRenderer) Blocked() []string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return append([]string(nil), r.blocked...)
+}
+
+// ProfileDir is the temporary profile of this crawl's browser.
+func (r *chromeRenderer) ProfileDir() string { return r.profile }
+
+// Close stops Chrome and removes the temporary profile.
+func (r *chromeRenderer) Close() error {
+	r.mu.Lock()
+	if r.closed {
+		r.mu.Unlock()
+		return nil
+	}
+	r.closed = true
+	r.mu.Unlock()
+	if r.browserCancel != nil {
+		r.browserCancel()
+	}
+	if r.allocCancel != nil {
+		r.allocCancel()
+	}
+	return os.RemoveAll(r.profile)
+}
+
+// resolveHosts resolves the site host and its www twin. A private or
+// reserved address (fetchguard.IsPrivateAddr) refuses outside trusted
+// mode; a twin that does not resolve is left out.
+func resolveHosts(ctx context.Context, hosts []string, trusted bool) (map[string][]netip.Addr, error) {
+	out := map[string][]netip.Addr{}
+	for i, host := range hosts {
+		if addr, err := netip.ParseAddr(strings.Trim(host, "[]")); err == nil {
+			if !trusted && fetchguard.IsPrivateAddr(addr) {
+				return nil, fmt.Errorf("the site host %s is a private address", addr)
+			}
+			out[host] = []netip.Addr{addr.Unmap()}
+			continue
+		}
+		addrs, err := net.DefaultResolver.LookupNetIP(ctx, "ip", host)
+		if err != nil || len(addrs) == 0 {
+			if i == 0 {
+				return nil, fmt.Errorf("the site host %s does not resolve", host)
+			}
+			continue
+		}
+		for _, a := range addrs {
+			a = a.Unmap()
+			if !trusted && fetchguard.IsPrivateAddr(a) {
+				return nil, fmt.Errorf("the site host %s resolves to the private address %s", host, a)
+			}
+			out[host] = append(out[host], a)
+		}
+	}
+	return out, nil
+}

classes/crawl/render_test.go

@@ -0,0 +1,178 @@
+package crawl
+
+import (
+	"context"
+	"net/netip"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"git.golem15.com/golem15/summercms/modules/compass"
+)
+
+// TestAllowRequest pins what the headless browser may load: https on the
+// site host and its www twin on the site's port, data: and blob:; http
+// only in trusted mode, also on the site's port; other hosts and ports,
+// credentials in the URL and every other scheme refused (L3 WR-03).
+func TestAllowRequest(t *testing.T) {
+	hosts := []string{"example.test", "www.example.test"}
+	for _, tc := range []struct {
+		url     string
+		trusted bool
+		want    bool
+	}{
+		{"https://example.test/a.js", false, true},
+		{"https://WWW.example.test/b.css", false, true},
+		{"https://example.test:8443/x", false, false},
+		{"https://example.test:443/x", false, true},
+		{"https://example.test:80/x", false, false},
+		{"data:image/png;base64,AAAA", false, true},
+		{"blob:https://example.test/uuid", false, true},
+		{"http://example.test/a.js", false, false},
+		{"http://example.test/a.js", true, false}, // trusted, but port 80 is not the site's 443
+		{"https://cdn.other.test/lib.js", false, false},
+		{"https://example.test.evil.test/x", false, false},
+		{"https://user:pw@example.test/x", false, false},
+		{"ws://example.test/socket", false, false},
+		{"file:///etc/passwd", false, false},
+		{"javascript:alert(1)", true, false},
+		{"://bad", false, false},
+	} {
+		if got := allowRequest(tc.url, hosts, "443", tc.trusted); got != tc.want {
+			t.Errorf("allowRequest(%q, trusted=%v) = %v, want %v", tc.url, tc.trusted, got, tc.want)
+		}
+	}
+	// A trusted http site on its own port: that port only.
+	if !allowRequest("http://example.test:3000/x", hosts, "3000", true) || allowRequest("http://example.test:3001/x", hosts, "3000", true) ||
+		allowRequest("http://example.test/x", hosts, "3000", true) {
+		t.Error("trusted http keeps to the site's port")
+	}
+	if !allowRequest("data:text/plain,x", hosts, "3000", false) || !allowRequest("blob:https://example.test/uuid", hosts, "3000", false) {
+		t.Error("data: and blob: are allowed on any port")
+	}
+	if !allowRequest("https://[::1]/x", []string{"[::1]"}, "443", false) {
+		t.Fatal("a bracketed IPv6 host must match")
+	}
+}
+
+// TestResolverRules pins Chrome's --host-resolver-rules: one MAP per
+// allowed host, sorted, to its first IPv4 address (else the first one, in
+// brackets for IPv6), a host without addresses left out, and MAP *
+// ~NOTFOUND last, since Chrome takes the first matching rule.
+func TestResolverRules(t *testing.T) {
+	got := resolverRules(map[string][]netip.Addr{
+		"www.example.test": {netip.MustParseAddr("2001:db8::1")},
+		"example.test":     {netip.MustParseAddr("2001:db8::2"), netip.MustParseAddr("192.0.2.10"), netip.MustParseAddr("192.0.2.11")},
+		"empty.test":       nil,
+	})
+	want := "MAP example.test 192.0.2.10, MAP www.example.test [2001:db8::1], MAP * ~NOTFOUND"
+	if got != want {
+		t.Fatalf("rules = %q\nwant    %q", got, want)
+	}
+	if got := resolverRules(nil); got != "MAP * ~NOTFOUND" {
+		t.Fatalf("no hosts = %q", got)
+	}
+	if !strings.HasSuffix(got, "MAP * ~NOTFOUND") || strings.Count(got, "~NOTFOUND") != 1 {
+		t.Fatal("the catch-all must come once and last")
+	}
+}
+
+// TestNewRenderer pins the comparison's start: no renderer and no error
+// without a Chrome path (skipped), an error for a relative path, a
+// missing file, a directory or a file that is not executable (failed).
+func TestNewRenderer(t *testing.T) {
+	site, _ := url.Parse("https://example.test")
+	cfg := func(path string) *compass.Config {
+		c, err := compass.Open(compass.Options{Dir: t.TempDir()})
+		if err != nil {
+			t.Fatal(err)
+		}
+		if path != "" {
+			if err := c.Set(ChromePathConfigKey, path); err != nil {
+				t.Fatal(err)
+			}
+		}
+		return c
+	}
+	if r, err := NewRenderer(context.Background(), cfg(""), site, false, "ua"); r != nil || err != nil {
+		t.Fatalf("no path: %v %v, want skipped", r, err)
+	}
+	if r, err := NewRenderer(context.Background(), nil, site, false, "ua"); r != nil || err != nil {
+		t.Fatal("no config: skipped")
+	}
+	dir := t.TempDir()
+	plain := filepath.Join(dir, "chrome.txt")
+	if err := os.WriteFile(plain, []byte("x"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	for name, path := range map[string]string{"relative": "bin/chrome", "missing": filepath.Join(dir, "nope"), "directory": dir, "not executable": plain} {
+		r, err := NewRenderer(context.Background(), cfg(path), site, false, "ua")
+		if r != nil || err == nil || !strings.Contains(err.Error(), "Chrome path") {
+			t.Errorf("%s: %v %v, want a Chrome path error", name, r, err)
+		}
+	}
+}
+
+// TestRenderMax pins the page cap: the default 200, a positive config
+// value, and the default for 0 or less.
+func TestRenderMax(t *testing.T) {
+	if RenderMax(nil) != DefaultRenderMax || DefaultRenderMax != 200 {
+		t.Fatal("default cap")
+	}
+	for v, want := range map[int]int{50: 50, 0: DefaultRenderMax, -1: DefaultRenderMax} {
+		c, _ := compass.Open(compass.Options{Dir: t.TempDir()})
+		_ = c.Set(RenderMaxConfigKey, v)
+		if got := RenderMax(c); got != want {
+			t.Errorf("RenderMax(%d) = %d, want %d", v, got, want)
+		}
+	}
+	if got := SiteHosts("WWW.Example.test"); len(got) != 2 || got[0] != "www.example.test" || got[1] != "example.test" {
+		t.Fatalf("SiteHosts = %v", got)
+	}
+	if got := SiteHosts("example.test"); got[1] != "www.example.test" || SiteHosts(" ") != nil {
+		t.Fatalf("SiteHosts = %v", got)
+	}
+}
+
+// TestRenderTabTimeout pins L3 WR-06: a tab that does not open answers an
+// error within TabOpenTimeout (lowered for the test) plus a margin, so the
+// crawl's render lock is released; the next Render is not held up by the
+// first one's hung tab.
+func TestRenderTabTimeout(t *testing.T) {
+	saved := TabOpenTimeout
+	TabOpenTimeout = 50 * time.Millisecond
+	t.Cleanup(func() { TabOpenTimeout = saved })
+	hung := make(chan struct{})
+	t.Cleanup(func() { close(hung) })
+	r := &chromeRenderer{
+		allowed:    []string{"example.test"},
+		port:       "443",
+		browserCtx: context.Background(),
+		newTab: func(parent context.Context) (context.Context, context.CancelFunc) {
+			return context.WithCancel(parent)
+		},
+		// A hung Chrome: opening the tab never answers, even when the tab
+		// is closed.
+		startTab: func(context.Context) error {
+			<-hung
+			return nil
+		},
+	}
+	for i := range 2 {
+		start := time.Now()
+		_, err := r.Render(t.Context(), "https://example.test/a")
+		if err == nil || !strings.Contains(err.Error(), "did not open a tab within 50ms") {
+			t.Fatalf("Render %d: %v, want the tab timeout", i, err)
+		}
+		if d := time.Since(start); d > time.Second {
+			t.Fatalf("Render %d returned after %v", i, d)
+		}
+	}
+	TabOpenTimeout = 5 * time.Second
+	if got := seconds(TabOpenTimeout); got != "5 s" {
+		t.Fatalf("seconds = %q", got)
+	}
+}

classes/crawl/store.go

@@ -0,0 +1,331 @@
+package crawl
+
+import (
+	"context"
+	"sort"
+	"time"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/lagoon"
+	"gorm.io/gorm"
+)
+
+// insertBatch is the rows written per INSERT.
+const insertBatch = 200
+
+// SavePages writes the pages and links one crawl fetched. The crawler
+// writes only its own tables (D-14).
+func SavePages(tx *gorm.DB, crawlID uint, res *Result) error {
+	if res == nil {
+		return nil
+	}
+	in := map[string]int{}
+	var links []models.CrawlLink
+	for _, key := range res.Order {
+		p := res.Pages[key]
+		if p.Facts == nil {
+			continue
+		}
+		seen := map[string]bool{}
+		for _, l := range p.Facts.Links {
+			to := ""
+			internal := false
+			if res.fetcher != nil {
+				to = res.fetcher.Key(l.URL)
+				internal = to != ""
+			}
+			if !internal {
+				to = l.URL.String()
+			}
+			links = append(links, models.CrawlLink{CrawlID: crawlID, FromURL: key, ToURL: to, Text: truncateUTF8(l.Text, 500), Rel: l.Rel, Internal: internal})
+			if internal && to != key && !seen[to] {
+				seen[to] = true
+				in[to]++
+			}
+		}
+	}
+	pages := make([]models.CrawlPage, 0, len(res.Order))
+	for _, key := range res.Order {
+		p := res.Pages[key]
+		row := models.CrawlPage{CrawlID: crawlID, URL: key, Depth: p.Depth, InSitemap: p.InSitemap, SitemapFile: p.SitemapFile, LinksIn: in[key]}
+		if !p.FetchedAt.IsZero() {
+			t := p.FetchedAt
+			row.FetchedAt = &t
+		}
+		if r := p.Resp; r != nil {
+			row.Status, row.Location, row.ContentType, row.XRobots = r.Status, r.Location, truncateUTF8(r.ContentType, 200), truncateUTF8(r.XRobots, 500)
+		}
+		if f := p.Facts; f != nil {
+			row.Title, row.Description, row.Canonical, row.Robots = truncateUTF8(f.Title, 1000), truncateUTF8(f.Description, 2000), truncateUTF8(f.Canonical, 2048), truncateUTF8(f.Robots, 500)
+			row.H1 = lagoon.Jsonable[[]string]{Data: f.H1(), Valid: true}
+			hs := make([]models.CrawlHeading, 0, len(f.Headings))
+			for _, h := range f.Headings {
+				hs = append(hs, models.CrawlHeading{Level: h.Level, Text: truncateUTF8(h.Text, 500)})
+			}
+			row.Headings = lagoon.Jsonable[[]models.CrawlHeading]{Data: hs, Valid: true}
+			row.Words, row.Text = f.Words, truncateUTF8(f.Text, models.MaxPageText)
+			row.LinksOut = len(f.Links)
+		}
+		pages = append(pages, row)
+	}
+	if len(pages) > 0 {
+		if err := tx.CreateInBatches(pages, insertBatch).Error; err != nil {
+			return err
+		}
+	}
+	if len(links) > 0 {
+		if err := tx.CreateInBatches(links, insertBatch).Error; err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// issueRow turns a finding into its stored row.
+func issueRow(crawlID *uint, source string, f Finding, now time.Time) models.Issue {
+	fix, tool, what := Route(f)
+	details := map[string]any{}
+	for k, v := range f.Details {
+		details[k] = v
+	}
+	if what != "" {
+		details["what"] = what
+	}
+	if f.Check != "" {
+		details["check"] = f.Check
+	}
+	sev := Severity(f.Kind)
+	t := now
+	return models.Issue{
+		CrawlID:     crawlID,
+		Source:      source,
+		Kind:        f.Kind,
+		Severity:    sev,
+		Rank:        models.SeverityRank(sev),
+		URL:         truncateUTF8(f.URL, 2048),
+		Locale:      f.Locale,
+		Details:     lagoon.Jsonable[map[string]any]{Data: details, Valid: true},
+		Tool:        tool,
+		Fix:         fix,
+		Fingerprint: Fingerprint(f.Kind, f.URL, f.Locale, f.Check),
+		FoundAt:     &t,
+	}
+}
+
+// cmsFinding turns a CMS-data issue of the classes report into a finding:
+// its tool is the one the report names; a CMS issue without one is fixed
+// in the record's admin form.
+func cmsFinding(issue classes.Issue) Finding {
+	f := Finding{Kind: issue.Kind, URL: issue.URL, Locale: issue.Locale, Details: issue.Details, Tool: issue.Tool}
+	if f.Tool == "" {
+		f.What = "the record in its admin form"
+	}
+	return f
+}
+
+// SaveIssues stores a crawl's findings and the CMS-data issues with it
+// (crawlID nil: the CMS issues of a site not crawled yet). A CMS issue
+// with the fingerprint of a crawl finding is the same problem and is not
+// stored twice.
+func SaveIssues(tx *gorm.DB, crawlID *uint, findings []Finding, cms []classes.Issue) error {
+	now := time.Now().UTC()
+	seen := map[string]bool{}
+	var rows []models.Issue
+	for _, f := range findings {
+		row := issueRow(crawlID, models.SourceCrawl, f, now)
+		if seen[row.Fingerprint] {
+			continue
+		}
+		seen[row.Fingerprint] = true
+		rows = append(rows, row)
+	}
+	for _, issue := range cms {
+		row := issueRow(crawlID, models.SourceCMS, cmsFinding(issue), now)
+		if seen[row.Fingerprint] {
+			continue
+		}
+		seen[row.Fingerprint] = true
+		rows = append(rows, row)
+	}
+	sort.SliceStable(rows, func(i, j int) bool {
+		if rows[i].Rank != rows[j].Rank {
+			return rows[i].Rank < rows[j].Rank
+		}
+		if rows[i].Kind != rows[j].Kind {
+			return rows[i].Kind < rows[j].Kind
+		}
+		return rows[i].URL < rows[j].URL
+	})
+	if len(rows) == 0 {
+		return nil
+	}
+	return tx.CreateInBatches(rows, insertBatch).Error
+}
+
+// LatestCrawl returns the newest crawl that is no longer queued or
+// running, nil when there is none: its issues are the audit's current
+// ones.
+func LatestCrawl(db *gorm.DB) (*models.Crawl, error) {
+	var rows []models.Crawl
+	if err := db.Where("status NOT IN ?", []string{models.CrawlQueued, models.CrawlRunning}).Order("id DESC").Limit(1).Find(&rows).Error; err != nil {
+		return nil, err
+	}
+	if len(rows) == 0 {
+		return nil, nil
+	}
+	return &rows[0], nil
+}
+
+// ActiveCrawl returns the queued or running crawl, nil when none.
+func ActiveCrawl(db *gorm.DB) (*models.Crawl, error) {
+	var rows []models.Crawl
+	if err := db.Where("status IN ?", []string{models.CrawlQueued, models.CrawlRunning}).Order("id DESC").Limit(1).Find(&rows).Error; err != nil {
+		return nil, err
+	}
+	if len(rows) == 0 {
+		return nil, nil
+	}
+	return &rows[0], nil
+}
+
+// RecoverStale ends the running crawls whose job stopped writing its
+// heartbeat (P33-60): a running row whose COALESCE(heartbeat_at,
+// started_at, created_at) is older than StaleAfter before now becomes
+// cancelled when its cancel was requested, else failed with the error
+// "the crawl job stopped without finishing (no heartbeat since <time>)",
+// with finished_at now. It answers the rows it ended, with their new
+// status, so the caller can end their conga jobs.
+func RecoverStale(tx *gorm.DB, now time.Time) ([]models.Crawl, error) {
+	now = now.UTC()
+	var rows []models.Crawl
+	if err := tx.Where("status = ? AND COALESCE(heartbeat_at, started_at, created_at) < ?", models.CrawlRunning, now.Add(-StaleAfter)).
+		Order("id ASC").Find(&rows).Error; err != nil {
+		return nil, err
+	}
+	out := rows[:0]
+	for _, r := range rows {
+		last := r.HeartbeatAt
+		if last == nil {
+			last = r.StartedAt
+		}
+		if last == nil {
+			last = r.CreatedAt
+		}
+		updates := map[string]any{"finished_at": now}
+		if r.CancelRequestedAt != nil {
+			updates["status"], updates["error"] = models.CrawlCancelled, ""
+		} else {
+			since := "an unknown time"
+			if last != nil {
+				since = last.UTC().Format(time.RFC3339)
+			}
+			updates["status"], updates["error"] = models.CrawlFailed, "the crawl job stopped without finishing (no heartbeat since "+since+")"
+		}
+		res := tx.Model(&models.Crawl{}).Where("id = ? AND status = ?", r.ID, models.CrawlRunning).Updates(updates)
+		if res.Error != nil {
+			return nil, res.Error
+		}
+		if res.RowsAffected == 0 {
+			continue
+		}
+		r.Status = updates["status"].(string)
+		r.Error = updates["error"].(string)
+		r.FinishedAt = &now
+		pluginLog.Warn("golem15.seo: ended a stale crawl", "crawl", r.ID, "status", r.Status)
+		out = append(out, r)
+	}
+	return out, nil
+}
+
+// CurrentScope narrows an issues query to the audit's current issues: the
+// newest finished, failed or cancelled crawl's, or before any crawl the
+// stored CMS-data issues.
+func CurrentScope(db *gorm.DB, latest *models.Crawl) *gorm.DB {
+	if latest == nil {
+		return db.Where("golem15_seo_issues.crawl_id IS NULL")
+	}
+	return db.Where("golem15_seo_issues.crawl_id = ?", latest.ID)
+}
+
+// IgnoredSet returns which of the fingerprints are ignored.
+func IgnoredSet(db *gorm.DB, fingerprints []string) (map[string]bool, error) {
+	out := map[string]bool{}
+	if len(fingerprints) == 0 {
+		return out, nil
+	}
+	var rows []models.IssueIgnore
+	for start := 0; start < len(fingerprints); start += 1000 {
+		end := min(start+1000, len(fingerprints))
+		var part []models.IssueIgnore
+		if err := db.Where("fingerprint IN ?", fingerprints[start:end]).Find(&part).Error; err != nil {
+			return nil, err
+		}
+		rows = append(rows, part...)
+	}
+	for _, r := range rows {
+		out[r.Fingerprint] = true
+	}
+	return out, nil
+}
+
+// Current returns the crawl whose issues are the audit's current ones (nil
+// before any crawl). Before any crawl, the CMS-data checks run now and
+// their issues replace the stored ones, so the list shows the CMS data as
+// it is (UI-SPEC C23: when the list is opened with no crawl yet).
+func Current(ctx context.Context, svc *classes.Service, db *gorm.DB) (*models.Crawl, error) {
+	latest, err := LatestCrawl(db)
+	if err != nil || latest != nil {
+		return latest, err
+	}
+	return nil, RefreshCMS(ctx, svc, db)
+}
+
+// RefreshCMS brings the stored CMS-data issues of a site not crawled yet
+// up to date with the report, in one transaction: an issue found again
+// keeps its row (and id), the others are deleted and new ones stored.
+func RefreshCMS(ctx context.Context, svc *classes.Service, db *gorm.DB) error {
+	return lagoon.Transaction(ctx, db, func(ctx context.Context, tx *gorm.DB) error {
+		cms, err := svc.Issues(classes.WithTx(ctx, tx))
+		if err != nil {
+			return err
+		}
+		found := map[string]classes.Issue{}
+		for _, c := range cms {
+			found[Fingerprint(c.Kind, c.URL, c.Locale, "")] = c
+		}
+		var existing []models.Issue
+		if err := tx.Where("crawl_id IS NULL").Find(&existing).Error; err != nil {
+			return err
+		}
+		have := map[string]bool{}
+		var gone []uint
+		now := time.Now().UTC()
+		for _, e := range existing {
+			c, ok := found[e.Fingerprint]
+			if !ok || have[e.Fingerprint] {
+				gone = append(gone, e.ID)
+				continue
+			}
+			have[e.Fingerprint] = true
+			row := issueRow(nil, models.SourceCMS, cmsFinding(c), now)
+			if err := tx.Model(&models.Issue{}).Where("id = ?", e.ID).Updates(map[string]any{
+				"details": row.Details, "tool": row.Tool, "fix": row.Fix, "severity": row.Severity, "rank": row.Rank,
+			}).Error; err != nil {
+				return err
+			}
+		}
+		if len(gone) > 0 {
+			if err := tx.Where("id IN ?", gone).Delete(&models.Issue{}).Error; err != nil {
+				return err
+			}
+		}
+		var fresh []classes.Issue
+		for _, c := range cms {
+			if !have[Fingerprint(c.Kind, c.URL, c.Locale, "")] {
+				fresh = append(fresh, c)
+			}
+		}
+		return SaveIssues(tx, nil, nil, fresh)
+	})
+}

classes/entries.go

@@ -22,6 +22,30 @@ var EntryColumns = []string{
 // every locale row (Q-5): the social image.
 const SharedColumn = "og_image"
 
+// DataColumns are the SEO tab values stored in an entry's data (P33-23):
+// the primary parent and the structured data value. Like the social image
+// they are one per record, written on every locale row.
+var DataColumns = []string{DataPrimaryParent, DataSchema}
+
+// sharedColumns are every value shared by the locale rows of a record.
+var sharedColumns = append([]string{SharedColumn}, DataColumns...)
+
+// SharedData returns a record's shared data value: the first non-empty
+// value in locale order.
+func SharedData(entries map[string]models.Entry, key string) string {
+	locales := make([]string, 0, len(entries))
+	for locale := range entries {
+		locales = append(locales, locale)
+	}
+	sort.Strings(locales)
+	for _, locale := range locales {
+		if v := entryData(entries[locale], key); v != "" {
+			return v
+		}
+	}
+	return ""
+}
+
 // SharedValue returns a record's shared social image: the first non-empty
 // value in locale order.
 func SharedValue(entries map[string]models.Entry) string {
@@ -61,6 +85,8 @@ func entryValue(e models.Entry, column string) string {
 		return e.OGDescription
 	case "og_image":
 		return e.OGImage
+	case DataPrimaryParent, DataSchema:
+		return entryData(e, column)
 	}
 	return ""
 }
@@ -85,6 +111,17 @@ func setEntryValue(e *models.Entry, column, value string) {
 		e.OGDescription = value
 	case "og_image":
 		e.OGImage = value
+	case DataPrimaryParent, DataSchema:
+		data := models.JSONMap{}
+		for k, v := range e.Data {
+			data[k] = v
+		}
+		if value == "" {
+			delete(data, column)
+		} else {
+			data[column] = value
+		}
+		e.Data = data
 	}
 }
 
@@ -152,7 +189,11 @@ func (s *Service) SaveRecord(ctx context.Context, db *gorm.DB, in RecordWrite) (
 	}
 	described := s.recordLocales(ctx, in.Type.Key, in.ID)
 	values := in.Values
-	if shared, ok := values[SharedColumn]; ok {
+	for _, column := range sharedColumns {
+		shared, ok := values[column]
+		if !ok {
+			continue
+		}
 		// One value for every locale of the record (Q-5).
 		v := ""
 		for _, text := range shared {
@@ -166,10 +207,10 @@ func (s *Service) SaveRecord(ctx context.Context, db *gorm.DB, in RecordWrite) (
 			all[locale] = v
 		}
 		copied := RecordValues{}
-		for column, perLocale := range values {
+		for c, perLocale := range values {
-			copied[column] = perLocale
+			copied[c] = perLocale
 		}
-		copied[SharedColumn] = all
+		copied[column] = all
 		values = copied
 	}
 	touched := map[string]models.Entry{}

classes/field_types.go

@@ -2,10 +2,13 @@ package classes
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"net/url"
 	"strings"
 
+	"git.golem15.com/golem15/sm-seo-plugin/classes/schema"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
 	"git.golem15.com/golem15/summercms/modules/backpack"
 	"git.golem15.com/golem15/summercms/modules/pact"
 )
@@ -18,6 +21,9 @@ const (
 	ElementImage        = "golem15-seo-image"
 	ElementRobotsTxt    = "golem15-seo-robotstxt"
 	ElementRobotsTester = "golem15-seo-robotstester"
+	ElementSchema       = "golem15-seo-schema"
+	ElementGraph        = "golem15-seo-graph"
+	ElementParent       = "golem15-seo-parent"
 )
 
 // The element bundle every SEO field type loads.
@@ -73,6 +79,131 @@ func FieldTypes(app func() *backpack.App, permission string) []pact.FormFieldTyp
 			Scripts: elementScripts, Styles: elementStyles,
 			Decode: decodeRobotsTxt,
 		},
+		{
+			Name: "seoschema", Element: ElementSchema,
+			Scripts: elementScripts, Styles: elementStyles,
+			Options: []string{"kind", "contentType"}, CheckOptions: checkKind,
+			Decode: decodeSchema(app),
+		},
+		{
+			Name: "seoparent", Element: ElementParent,
+			Scripts: elementScripts, Styles: elementStyles,
+			Options: []string{"contentType"},
+			Decode:  decodeParent(app),
+		},
+	}
+}
+
+// checkKind accepts the content kinds of pact.ContentDescriptor.
+func checkKind(options map[string]any) error {
+	if raw, ok := options["kind"]; ok {
+		switch s, _ := raw.(string); s {
+		case "", schema.KindArticle, schema.KindCollection, schema.KindProfile, schema.KindPage:
+		default:
+			return fmt.Errorf("kind must be article, collection, profile or page, got %v", raw)
+		}
+	}
+	return nil
+}
+
+// optionString reads a string option.
+func optionString(options map[string]any, key string) string {
+	s, _ := options[key].(string)
+	return s
+}
+
+// SchemaRefusal renders a refused seo_schema value with the schema.* copy
+// in the request locale (UI-SPEC C17).
+func (s *Service) SchemaRefusal(ctx context.Context, kind string, err error) string {
+	var se *schema.Error
+	if !errors.As(err, &se) {
+		return s.PhraseCtx(ctx, "golem15.seo::lang.schema.invalid_json", map[string]string{"error": err.Error()})
+	}
+	switch se.Code {
+	case schema.CodeInvalidJSON:
+		return s.PhraseCtx(ctx, "golem15.seo::lang.schema.invalid_json", map[string]string{"error": se.Detail})
+	case schema.CodeTypeUnknown:
+		return s.PhraseCtx(ctx, "golem15.seo::lang.schema.type_unknown", map[string]string{"type": kind})
+	case schema.CodeFieldInvalid:
+		label, message := "golem15.seo::lang.schema.field_price", "golem15.seo::lang.schema.price_invalid"
+		if se.Field == "currency" {
+			label, message = "golem15.seo::lang.schema.field_currency", "golem15.seo::lang.schema.currency_invalid"
+		}
+		return s.PhraseCtx(ctx, "golem15.seo::lang.schema.field_invalid", map[string]string{
+			"field":   s.PhraseCtx(ctx, label, nil),
+			"message": s.PhraseCtx(ctx, message, nil),
+		})
+	}
+	keys := map[string]string{
+		schema.CodeNotObject: "golem15.seo::lang.schema.not_object",
+		schema.CodeTooLarge:  "golem15.seo::lang.schema.too_large",
+		schema.CodeTooDeep:   "golem15.seo::lang.schema.too_deep",
+		schema.CodeScriptTag: "golem15.seo::lang.schema.script_tag",
+	}
+	return s.PhraseCtx(ctx, keys[se.Code], nil)
+}
+
+// SchemaVariant is the registry kind of a record of contentType, "" for
+// other types.
+func SchemaVariant(record any) string {
+	if p, ok := record.(*models.Page); ok && p != nil {
+		return p.Kind
+	}
+	return ""
+}
+
+// decodeSchema checks and normalises the seoschema value (E8): a JSON
+// string or "" (UI-SPEC C17); every refusal answers the schema.* copy.
+func decodeSchema(app func() *backpack.App) func(context.Context, pact.FormFieldValue) (pact.FormFieldDecoded, error) {
+	return func(ctx context.Context, in pact.FormFieldValue) (pact.FormFieldDecoded, error) {
+		raw, ok := stringValue(in.Value)
+		if !ok {
+			return pact.FormFieldDecoded{}, &pact.FormFieldError{Messages: []string{"golem15.seo::lang.schema.not_object"}}
+		}
+		kind := optionString(in.Options, "kind")
+		stored, _, err := schema.Normalize(kind, SchemaVariant(in.Record), raw)
+		if err != nil {
+			svc, serr := NewService(app())
+			if serr != nil {
+				return pact.FormFieldDecoded{}, &pact.FormFieldError{Messages: []string{err.Error()}}
+			}
+			return pact.FormFieldDecoded{}, &pact.FormFieldError{Messages: []string{svc.SchemaRefusal(ctx, kind, err)}}
+		}
+		return pact.FormFieldDecoded{Value: stored}, nil
+	}
+}
+
+// decodeParent accepts "" (the first parent) or one of the record's
+// parents as <type>:<id>.
+func decodeParent(app func() *backpack.App) func(context.Context, pact.FormFieldValue) (pact.FormFieldDecoded, error) {
+	return func(ctx context.Context, in pact.FormFieldValue) (pact.FormFieldDecoded, error) {
+		value := in.Value
+		if m, isMap := value.(map[string]any); isMap {
+			// The loaded {selected, options} sent back unchanged.
+			value = m["selected"]
+		}
+		raw, ok := stringValue(value)
+		raw = strings.TrimSpace(raw)
+		if !ok {
+			return pact.FormFieldDecoded{}, &pact.FormFieldError{Messages: []string{"golem15.seo::lang.api.value_invalid"}}
+		}
+		if raw == "" {
+			return pact.FormFieldDecoded{Value: ""}, nil
+		}
+		svc, err := NewService(app())
+		if err != nil {
+			return pact.FormFieldDecoded{}, err
+		}
+		id := ""
+		if in.Record != nil && in.DB != nil {
+			id, _ = RecordKey(in.DB, in.Record)
+		}
+		for _, opt := range svc.ParentOptions(ctx, optionString(in.Options, "contentType"), id, "") {
+			if opt.Value == raw {
+				return pact.FormFieldDecoded{Value: raw}, nil
+			}
+		}
+		return pact.FormFieldDecoded{}, &pact.FormFieldError{Messages: []string{"golem15.seo::lang.api.value_invalid"}}
 	}
 }
 
@@ -165,3 +296,29 @@ func ValidURLRef(v string) bool {
 	u, err := url.Parse(v)
 	return err == nil && (u.Scheme == "http" || u.Scheme == "https") && u.Host != ""
 }
+
+// SchemaProblem is a refused seo_schema value as an API problem.
+func SchemaProblem(kind string, err error) models.Problem {
+	var se *schema.Error
+	if !errors.As(err, &se) {
+		return models.Problem{Key: "golem15.seo::lang.schema.invalid_json", Params: map[string]string{"error": err.Error()}}
+	}
+	switch se.Code {
+	case schema.CodeInvalidJSON:
+		return models.Problem{Key: "golem15.seo::lang.schema.invalid_json", Params: map[string]string{"error": se.Detail}}
+	case schema.CodeTypeUnknown:
+		return models.Problem{Key: "golem15.seo::lang.schema.type_unknown", Params: map[string]string{"type": kind}}
+	case schema.CodeFieldInvalid:
+		if se.Field == "currency" {
+			return models.Problem{Key: "golem15.seo::lang.schema.currency_invalid"}
+		}
+		return models.Problem{Key: "golem15.seo::lang.schema.price_invalid"}
+	case schema.CodeNotObject:
+		return models.Problem{Key: "golem15.seo::lang.schema.not_object"}
+	case schema.CodeTooLarge:
+		return models.Problem{Key: "golem15.seo::lang.schema.too_large"}
+	case schema.CodeTooDeep:
+		return models.Problem{Key: "golem15.seo::lang.schema.too_deep"}
+	}
+	return models.Problem{Key: "golem15.seo::lang.schema.script_tag"}
+}

classes/generate.go

@@ -7,6 +7,7 @@ import (
 
 	"git.golem15.com/golem15/sm-seo-plugin/models"
 	"git.golem15.com/golem15/summercms/modules/conga"
+	"git.golem15.com/golem15/summercms/modules/lagoon"
 	"git.golem15.com/golem15/summercms/modules/pact"
 	"gorm.io/gorm"
 )
@@ -193,7 +194,9 @@ func (s *Service) RecordDeleted(ctx context.Context, ev *pact.RecordDeleted) err
 		return err
 	}
 	ctx = WithVia(ctx, models.ViaAuto)
-	return db.Transaction(func(tx *gorm.DB) error {
+	// A lagoon transaction, so the history it records purges the site
+	// cache after the commit.
+	return lagoon.Transaction(ctx, db, func(ctx context.Context, tx *gorm.DB) error {
 		entries, err := LoadEntries(tx, typ.Content.Model, ev.ID)
 		if err != nil || len(entries) == 0 {
 			return err

classes/history.go

@@ -7,10 +7,13 @@ import (
 	"fmt"
 	"reflect"
 	"strings"
+	"sync"
 	"time"
 
 	"git.golem15.com/golem15/sm-seo-plugin/models"
 	"git.golem15.com/golem15/summercms/modules/bouncer"
+	"git.golem15.com/golem15/summercms/modules/cooler"
+	"git.golem15.com/golem15/summercms/modules/lagoon"
 	"git.golem15.com/golem15/summercms/modules/pact"
 	"git.golem15.com/golem15/summercms/modules/phrasebook"
 	"gorm.io/gorm"
@@ -232,10 +235,83 @@ func (s *Service) Record(ctx context.Context, db *gorm.DB, changes ...Change) er
 		if err := db.Create(&row).Error; err != nil {
 			return err
 		}
+		s.purgeAfterCommit(ctx, db, purgeSlug(c))
 	}
 	return nil
 }
 
+// SiteCacheKind is the purge kind of SEO writes (P33-19).
+const SiteCacheKind = "seo"
+
+// purgeAfterCommit queues the rendered site's cache purge once the write
+// db belongs to commits (P33-19, D-97): every SEO write that records
+// history (admin, MCP, batch, import, undo, automatic) refreshes the
+// layer's cached resolve answers and pages. A transaction purges once,
+// however many rows it records; a rolled-back write never purges. Without
+// a configured purger (site_cache.purge_url) it does nothing.
+func (s *Service) purgeAfterCommit(ctx context.Context, db *gorm.DB, slug string) {
+	if s == nil || s.app == nil {
+		return
+	}
+	p := cooler.SiteCacheFromApp(s.app)
+	if p == nil {
+		return
+	}
+	var pool gorm.ConnPool
+	if db != nil && db.Statement != nil {
+		if _, inTx := db.Statement.ConnPool.(gorm.TxCommitter); inTx {
+			pool = db.Statement.ConnPool
+		}
+	}
+	if pool != nil {
+		purgeMu.Lock()
+		if purging == nil || len(purging) > maxPendingPurges {
+			// Rolled-back transactions leave their key behind; the map is
+			// reset rather than grow.
+			purging = map[gorm.ConnPool]bool{}
+		}
+		if purging[pool] {
+			purgeMu.Unlock()
+			return
+		}
+		purging[pool] = true
+		purgeMu.Unlock()
+		lagoon.AfterCommit(ctx, db, func(context.Context, *gorm.DB) {
+			purgeMu.Lock()
+			delete(purging, pool)
+			purgeMu.Unlock()
+		})
+	}
+	cooler.PurgeSiteCacheAfterCommit(ctx, db, p, SiteCacheKind, slug)
+}
+
+// maxPendingPurges caps the transactions purgeAfterCommit remembers.
+const maxPendingPurges = 1024
+
+// purging holds the open transactions whose site-cache purge is queued, so
+// each purges once whichever Service instance records in it.
+var (
+	purgeMu sync.Mutex
+	purging map[gorm.ConnPool]bool
+)
+
+// purgeSlug is the purge's slug: "*" for the settings, robots.txt and an
+// auto-link phrase (it may appear in any text), else the changed object's
+// address.
+func purgeSlug(c Change) string {
+	if c.Kind == models.SubjectSettings || c.Kind == models.SubjectRobots || c.Kind == models.SubjectAutolink {
+		return "*"
+	}
+	for _, values := range []map[string]any{c.After, c.Before} {
+		for _, key := range []string{"url", "source"} {
+			if v, ok := values[key].(string); ok && strings.TrimSpace(v) != "" {
+				return v
+			}
+		}
+	}
+	return "*"
+}
+
 // translator returns the application translator, nil when none.
 func (s *Service) translator() *phrasebook.Translator {
 	if s == nil || s.app == nil {
@@ -267,6 +343,11 @@ func entryValues(e models.Entry) map[string]any {
 	for _, column := range EntryColumns {
 		out[column] = entryValue(e, column)
 	}
+	for _, column := range DataColumns {
+		if v := entryData(e, column); v != "" {
+			out[column] = v
+		}
+	}
 	return out
 }
 

classes/issues.go

@@ -7,6 +7,8 @@ import (
 	"strings"
 
 	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
 )
 
 // The kinds of the CMS-data issue report (D-13, D-14 CMS part). The
@@ -21,6 +23,22 @@ const (
 	IssueCanonicalNoindex    = "canonical_noindex"
 	IssueCanonicalRedirected = "canonical_redirected"
 	IssueNoindexBlocked      = "noindex_blocked"
+	// IssueSitemapBlocked is a sitemap address the production robots.txt
+	// keeps Googlebot from (D-24): an error, since the sitemap asks for a
+	// page the crawler may not fetch.
+	IssueSitemapBlocked = "sitemap_blocked"
+	// IssueImageAlt is a record image without alternative text that is
+	// not marked decorative (D-85), from the content descriptors.
+	IssueImageAlt = "image_alt"
+	// IssueH1InContent is an H1 in the body of a record whose page renders
+	// the title as the only H1 (descriptor TopHeading 2, D-50; Journal's
+	// H1 setting on).
+	IssueH1InContent = "h1_in_content"
+)
+
+// The severities an issue may name.
+const (
+	SeverityError = "error"
 )
 
 // Issue is one finding of the issue report: the address, the kind, the
@@ -31,6 +49,9 @@ type Issue struct {
 	Locale  string         `json:"locale,omitempty"`
 	Details map[string]any `json:"details,omitempty"`
 	Tool    string         `json:"tool"`
+	// Severity is set for kinds that are errors (sitemap_blocked); "" for
+	// the others.
+	Severity string `json:"severity,omitempty"`
 }
 
 // The tools the issues point to (pier prefixes them with seo_).
@@ -71,7 +92,100 @@ func (s *Service) Issues(ctx context.Context) ([]Issue, error) {
 	if data.noindex, err = noindexAddresses(db); err != nil {
 		return nil, err
 	}
-	return s.issuesOf(ctx, data, s.canonicalProblem)
+	listed, _, err := s.SitemapListed(ctx)
+	if err != nil {
+		return nil, err
+	}
+	for address := range listed {
+		data.listed = append(data.listed, address)
+	}
+	sort.Strings(data.listed)
+	out, err := s.issuesOf(ctx, data, s.canonicalProblem)
+	if err != nil {
+		return nil, err
+	}
+	content, err := s.ContentIssues(ctx)
+	if err != nil {
+		return nil, err
+	}
+	return append(out, content...), nil
+}
+
+// contentIssuesPage is the page size of the content walk.
+const contentIssuesPage = 200
+
+// ContentIssues are the CMS-data checks on the content descriptors of
+// every published record (D-85, D-50): image_alt for images with an empty
+// alt that are not decorative, and h1_in_content for an H1 in the body of
+// a record whose descriptor sets TopHeading 2 (off when the type sets no
+// rule, as Journal does with its H1 setting off). Each names the record
+// type's fitting tool (ContentDescriptor.Tools: image_alt or headings,
+// else edit); "" when the type names none.
+func (s *Service) ContentIssues(ctx context.Context) ([]Issue, error) {
+	reg := s.Registry()
+	if reg == nil {
+		return nil, nil
+	}
+	var out []Issue
+	for _, typ := range reg.ContentTypes() {
+		if typ.Content == nil || typ.Content.Records == nil {
+			continue
+		}
+		tool := func(ops ...string) string {
+			for _, op := range ops {
+				if t := strings.TrimSpace(typ.Content.Tools[op]); t != "" {
+					return t
+				}
+			}
+			return ""
+		}
+		after := ""
+		for {
+			page, err := reg.Records(ctx, typ.Key, after, contentIssuesPage, s.UnprefixedLocale())
+			if err != nil {
+				return nil, err
+			}
+			for _, rec := range page.Records {
+				if !rec.Published {
+					continue
+				}
+				for _, l := range rec.Locales {
+					if strings.TrimSpace(l.URL) == "" {
+						continue
+					}
+					var missing []string
+					for _, img := range l.Images {
+						if !img.Decorative && strings.TrimSpace(img.Alt) == "" && img.URL != "" {
+							missing = append(missing, img.URL)
+						}
+					}
+					if len(missing) > 0 {
+						out = append(out, Issue{Kind: IssueImageAlt, URL: l.URL, Locale: l.Locale, Tool: tool(pact.ContentToolImageAlt, pact.ContentToolEdit), Details: map[string]any{
+							"count": len(missing), "images": missing, "type": typ.Key, "id": rec.ID,
+						}})
+					}
+					if rec.TopHeading == 2 {
+						var h1 []string
+						for _, h := range l.Headings {
+							if h.Level == 1 {
+								h1 = append(h1, h.Text)
+							}
+						}
+						if len(h1) > 0 {
+							out = append(out, Issue{Kind: IssueH1InContent, URL: l.URL, Locale: l.Locale, Tool: tool(pact.ContentToolHeadings, pact.ContentToolEdit), Details: map[string]any{
+								"count": len(h1), "h1": h1, "type": typ.Key, "id": rec.ID,
+							}})
+						}
+					}
+				}
+			}
+			if page.Next == "" || page.Next == after || len(page.Records) == 0 {
+				break
+			}
+			after = page.Next
+		}
+	}
+	return out, nil
 }
 
 // issueData is what the issue report reads from the database: the SEO
@@ -85,6 +199,8 @@ type issueData struct {
 	redirects []models.Redirect
 	robots    models.Robots
 	noindex   []string
+	// listed are the addresses the sitemaps list (path with query).
+	listed []string
 }
 
 // issuesOf builds the report from data; canonical classifies a canonical
@@ -228,10 +344,22 @@ func (s *Service) issuesOf(ctx context.Context, data issueData, canonical func(c
 	}
 
 	// Noindex addresses robots.txt blocks.
+	file := ComposeRobots(data.robots.Content, data.robots.Bots, s.SitemapLines(ctx))
 	for _, u := range s.blockedNoindex(data.noindex, data.robots.Content, data.robots.Bots) {
 		out = append(out, Issue{Kind: IssueNoindexBlocked, URL: u, Tool: toolUpdateRobots})
 	}
 
+	// Sitemap addresses the production robots.txt blocks for Googlebot.
+	for _, u := range data.listed {
+		test := s.testRobotsFile(file, data.robots.Content, "Googlebot", u)
+		if test.Allowed {
+			continue
+		}
+		out = append(out, Issue{Kind: IssueSitemapBlocked, URL: u, Tool: toolUpdateRobots, Severity: SeverityError, Details: map[string]any{
+			"line": test.Line, "rule": test.Rule,
+		}})
+	}
+
 	sort.SliceStable(out, func(i, j int) bool {
 		if out[i].Kind != out[j].Kind {
 			return out[i].Kind < out[j].Kind
@@ -275,3 +403,11 @@ func (s *Service) canonicalProblem(ctx context.Context, canonical string) (kind,
 	}
 	return IssueCanonicalNoindex, "", nil
 }
+
+// Actor names who runs an audit operation (a crawl start, a cancel, an
+// ignore): the backend administrator's id and name, and whether the call
+// came through MCP.
+func (s *Service) Actor(ctx context.Context, db *gorm.DB) (id *uint, name string, mcp bool) {
+	id, name = s.author(ctx, db, models.ViaAdmin)
+	return id, name, ViaOf(ctx) == models.ViaMCP
+}

classes/linkgraph.go

@@ -0,0 +1,390 @@
+package classes
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"net/url"
+	"strconv"
+	"strings"
+	"time"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/cabana"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
+)
+
+// MaxLinkSuggestions caps each suggestion list of the internal links
+// widget (UI-SPEC C19).
+const MaxLinkSuggestions = 10
+
+// minLinkPhrase is the shortest phrase a link suggestion matches, in
+// bytes (the orphan suggestions' limit).
+const minLinkPhrase = 3
+
+// LinksActionName is the SEO tab's internal links widget action (E10).
+const LinksActionName = "seo_links"
+
+// LinkOut is a page this text could link to: the phrase found in this
+// page's crawled text, the target's title and its public address.
+type LinkOut struct {
+	Phrase string `json:"phrase"`
+	Title  string `json:"title"`
+	URL    string `json:"url"`
+}
+
+// LinkIn is a page that could link here: the phrase of this page found in
+// its crawled text, its title and its admin form ("" when no record of
+// the admin is behind it).
+type LinkIn struct {
+	Phrase   string `json:"phrase"`
+	Title    string `json:"title"`
+	AdminURL string `json:"adminUrl"`
+}
+
+// LinkReport is the internal links of a page from the last finished crawl
+// (D-82, UI-SPEC C19): the time of that crawl, the pages linking here and
+// the pages linked from here (both nil before the first crawl), whether
+// nothing links to the page, and up to 10 pages this text could link to
+// and 10 that could link here.
+type LinkReport struct {
+	CrawledAt *time.Time `json:"crawledAt"`
+	Inbound   *int       `json:"inbound"`
+	Outbound  *int       `json:"outbound"`
+	Orphan    bool       `json:"orphan"`
+	Out       []LinkOut  `json:"out"`
+	In        []LinkIn   `json:"in"`
+}
+
+// NormalText is text in lower case with its whitespace collapsed to one
+// space: the form the crawler's suggestions compare phrases in.
+func NormalText(s string) string { return strings.ToLower(strings.Join(strings.Fields(s), " ")) }
+
+// errEnough stops a batched scan once a list is full.
+var errEnough = errors.New("golem15.seo: enough suggestions")
+
+// lastFinishedCrawl is the newest finished crawl, nil before the first.
+func lastFinishedCrawl(db *gorm.DB) (*models.Crawl, error) {
+	var rows []models.Crawl
+	if err := db.Where("status = ?", models.CrawlFinished).Order("finished_at DESC NULLS LAST, id DESC").Limit(1).Find(&rows).Error; err != nil {
+		return nil, err
+	}
+	if len(rows) == 0 {
+		return nil, nil
+	}
+	return &rows[0], nil
+}
+
+// pageHeading is a crawled page's first H1, else its title.
+func pageHeading(p models.CrawlPage) (h1, title string) {
+	if list := p.H1.Get(); len(list) > 0 {
+		h1 = strings.TrimSpace(list[0])
+	}
+	title = h1
+	if title == "" {
+		title = strings.TrimSpace(p.Title)
+	}
+	return h1, title
+}
+
+// entryKeyphrases maps each address with an SEO entry to its focus
+// keyphrase.
+func entryKeyphrases(db *gorm.DB) (map[string]string, error) {
+	var rows []struct {
+		URL       string
+		Keyphrase string
+	}
+	if err := db.Raw(`SELECT url, keyphrase FROM golem15_seo_entries WHERE url <> '' AND keyphrase <> ''`).Scan(&rows).Error; err != nil {
+		return nil, err
+	}
+	out := make(map[string]string, len(rows))
+	for _, r := range rows {
+		out[r.URL] = r.Keyphrase
+	}
+	return out, nil
+}
+
+// phraseList keeps the phrases of at least 3 bytes, once each (by their
+// normal form), in order.
+func phraseList(phrases ...string) []string {
+	var out []string
+	seen := map[string]bool{}
+	for _, p := range phrases {
+		p = strings.Join(strings.Fields(p), " ")
+		n := NormalText(p)
+		if len(n) < minLinkPhrase || seen[n] {
+			continue
+		}
+		seen[n] = true
+		out = append(out, p)
+	}
+	return out
+}
+
+// mentioned returns the first phrase whose normal form normal (a page's
+// NormalText) contains, "" when none.
+func mentioned(normal string, phrases []string) string {
+	for _, p := range phrases {
+		if strings.Contains(normal, NormalText(p)) {
+			return p
+		}
+	}
+	return ""
+}
+
+// adminURLOf is the admin form of the record behind a site path, "" when
+// no content type with an admin controller owns it.
+func (s *Service) adminURLOf(ctx context.Context, path string) string {
+	reg := s.Registry()
+	if reg == nil {
+		return ""
+	}
+	p, _, _ := strings.Cut(path, "?")
+	key, match, err := reg.MatchPath(ctx, p, s.UnprefixedLocale())
+	if err != nil || match == nil || !match.Exact {
+		return ""
+	}
+	typ, ok := reg.ContentType(key)
+	if !ok || typ.Content == nil || strings.TrimSpace(typ.Content.Controller) == "" {
+		return ""
+	}
+	return s.adminFormURL(typ.Content.Controller, match.ID)
+}
+
+// LinkSuggestions builds the internal links of the page at path from the
+// last finished crawl (D-82, UI-SPEC C19), like the crawl's orphan
+// suggestions: the pages whose crawled text contains one of phrases (the
+// page's title and focus keyphrase; its crawled H1 is added) and do not
+// link here yet, and the pages whose H1 or focus keyphrase this page's
+// crawled text contains and that it does not link to yet. Before the
+// first crawl the counts are nil and the lists empty.
+func (s *Service) LinkSuggestions(ctx context.Context, db *gorm.DB, path string, phrases []string) (LinkReport, error) {
+	out := LinkReport{Out: []LinkOut{}, In: []LinkIn{}}
+	db = db.WithContext(ctx)
+	c, err := lastFinishedCrawl(db)
+	if err != nil || c == nil {
+		return out, err
+	}
+	out.CrawledAt = c.FinishedAt
+	var links []models.CrawlLink
+	if err := db.Select("from_url", "to_url").Where("crawl_id = ? AND internal AND from_url <> to_url AND (from_url = ? OR to_url = ?)", c.ID, path, path).Find(&links).Error; err != nil {
+		return out, err
+	}
+	linkedFrom, linksTo := map[string]bool{}, map[string]bool{}
+	for _, l := range links {
+		if l.FromURL == path {
+			linksTo[l.ToURL] = true
+		} else {
+			linkedFrom[l.FromURL] = true
+		}
+	}
+	inbound, outbound := len(linkedFrom), len(linksTo)
+	out.Inbound, out.Outbound = &inbound, &outbound
+	var self []models.CrawlPage
+	if err := db.Where("crawl_id = ? AND url = ?", c.ID, path).Limit(1).Find(&self).Error; err != nil {
+		return out, err
+	}
+	keyphrases, err := entryKeyphrases(db)
+	if err != nil {
+		return out, err
+	}
+	site := s.SiteURL()
+	if len(self) > 0 {
+		out.Orphan = inbound == 0 && path != "/"
+		h1, _ := pageHeading(self[0])
+		phrases = append(phrases, h1)
+		text := NormalText(self[0].Text)
+		var others []models.CrawlPage
+		if err := db.Select("id", "url", "title", "h1").Where("crawl_id = ? AND url <> ? AND status = 200", c.ID, path).Order("url ASC").Find(&others).Error; err != nil {
+			return out, err
+		}
+		for _, o := range others {
+			if len(out.Out) >= MaxLinkSuggestions {
+				break
+			}
+			if linksTo[o.URL] {
+				continue
+			}
+			oh1, title := pageHeading(o)
+			if p := mentioned(text, phraseList(oh1, keyphrases[o.URL])); p != "" {
+				out.Out = append(out.Out, LinkOut{Phrase: p, Title: title, URL: site + o.URL})
+			}
+		}
+	}
+	own := phraseList(append(phrases, keyphrases[path])...)
+	if len(own) == 0 {
+		return out, nil
+	}
+	var batch []models.CrawlPage
+	err = db.Select("id", "url", "title", "h1", "text").Where("crawl_id = ? AND url <> ? AND status = 200", c.ID, path).
+		FindInBatches(&batch, 100, func(*gorm.DB, int) error {
+			for _, p := range batch {
+				if len(out.In) >= MaxLinkSuggestions {
+					return errEnough
+				}
+				if linkedFrom[p.URL] {
+					continue
+				}
+				if phrase := mentioned(NormalText(p.Text), own); phrase != "" {
+					_, title := pageHeading(p)
+					out.In = append(out.In, LinkIn{Phrase: phrase, Title: title, AdminURL: s.adminURLOf(ctx, p.URL)})
+				}
+			}
+			return nil
+		}).Error
+	if err != nil && !errors.Is(err, errEnough) {
+		return out, err
+	}
+	return out, nil
+}
+
+// sitePath is the crawl key of a path or an absolute URL (PathKey with
+// the raw query); ErrBadURL otherwise. A path and the absolute URL of the
+// same page give the same answer, written decoded or percent-encoded.
+func sitePath(raw string) (string, error) {
+	raw = strings.TrimSpace(raw)
+	if strings.HasPrefix(raw, "/") && !strings.HasPrefix(raw, "//") {
+		u, err := url.Parse(raw)
+		if err != nil {
+			return "", ErrBadURL
+		}
+		return joinQuery(PathKey(u), u.RawQuery), nil
+	}
+	u, err := url.Parse(raw)
+	if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+		return "", ErrBadURL
+	}
+	return joinQuery(PathKey(u), u.RawQuery), nil
+}
+
+// PathKey is the path form of the crawl's addresses (P33-61), the form
+// the resolver, the descriptors and the SEO entries use: u's decoded path
+// ("/" when empty). A path whose decoded form is not valid UTF-8 without
+// NUL, or holds "%" or "?" (which would read as an escape or a query),
+// keeps its percent-encoded form.
+func PathKey(u *url.URL) string {
+	if u == nil {
+		return "/"
+	}
+	p := u.Path
+	if !safeText(p) || strings.ContainsAny(p, "%?") {
+		p = u.EscapedPath()
+	}
+	if p == "" {
+		p = "/"
+	}
+	return p
+}
+
+// EscapeKeyPath is the request form of a PathKey path: a decoded path is
+// percent-encoded, an encoded one (it holds "%") is kept.
+func EscapeKeyPath(p string) string {
+	if strings.Contains(p, "%") {
+		return p
+	}
+	return (&url.URL{Path: p}).EscapedPath()
+}
+
+// LinksOfURL is the internal links of a page addressed by URL (a path or
+// an absolute URL of the site): the record behind it gives its title in
+// the matched locale.
+func (s *Service) LinksOfURL(ctx context.Context, db *gorm.DB, raw string) (LinkReport, error) {
+	path, err := sitePath(raw)
+	if err != nil {
+		return LinkReport{}, err
+	}
+	var phrases []string
+	if reg := s.Registry(); reg != nil {
+		p, _, _ := strings.Cut(path, "?")
+		if key, match, err := reg.MatchPath(ctx, p, s.UnprefixedLocale()); err == nil && match != nil && match.Exact {
+			if rec, err := reg.Describe(ctx, key, match.ID, s.UnprefixedLocale()); err == nil && rec != nil {
+				for _, l := range rec.Locales {
+					if l.Locale == match.Locale {
+						phrases = append(phrases, l.Title)
+					}
+				}
+			}
+		}
+	}
+	return s.LinkSuggestions(ctx, db, path, phrases)
+}
+
+// LinksOfRecord is the internal links of record id of content type key in
+// locale: its address and title in that locale and its focus keyphrase. A
+// record without an address in the locale answers the crawl time only.
+func (s *Service) LinksOfRecord(ctx context.Context, db *gorm.DB, key, id, locale string) (LinkReport, error) {
+	reg := s.Registry()
+	if reg == nil {
+		return LinkReport{Out: []LinkOut{}, In: []LinkIn{}}, nil
+	}
+	rec, err := reg.Describe(ctx, key, id, s.UnprefixedLocale())
+	if err != nil {
+		return LinkReport{}, err
+	}
+	var loc *pact.ContentLocale
+	if rec != nil {
+		for i := range rec.Locales {
+			if rec.Locales[i].Locale == locale {
+				loc = &rec.Locales[i]
+			}
+		}
+	}
+	if loc == nil || strings.TrimSpace(loc.URL) == "" {
+		out := LinkReport{Out: []LinkOut{}, In: []LinkIn{}}
+		c, err := lastFinishedCrawl(db.WithContext(ctx))
+		if err == nil && c != nil {
+			out.CrawledAt = c.FinishedAt
+		}
+		return out, err
+	}
+	phrases := []string{loc.Title}
+	if typ, ok := reg.ContentType(key); ok && typ.Content != nil {
+		if entry, found, err := LoadEntry(db, typ.Content.Model, id, locale); err == nil && found {
+			phrases = append(phrases, entry.Keyphrase)
+		}
+	}
+	return s.LinkSuggestions(ctx, db, loc.URL, phrases)
+}
+
+// linksPayload is the seo_links action's payload.
+type linksPayload struct {
+	Op     string `json:"op"`
+	Locale string `json:"locale"`
+}
+
+// RecordLinksAction is the SEO tab's internal links widget action (E10,
+// UI-SPEC C19) for content type key: {op: links, locale} answers the
+// record's LinkReport in that locale (the default when empty).
+func RecordLinksAction(svc func() *Service, key, label, permission string) pact.AdminAction {
+	return pact.AdminAction{
+		Name:        LinksActionName,
+		Label:       label,
+		Permissions: []string{permission},
+		Run: func(ctx context.Context, in pact.AdminActionInput) (pact.AdminActionResult, error) {
+			s := svc()
+			if s == nil {
+				return pact.AdminActionResult{}, ErrNotBooted
+			}
+			var p linksPayload
+			if len(in.Payload) > 0 {
+				if err := json.Unmarshal(in.Payload, &p); err != nil || (p.Op != "" && p.Op != "links") {
+					return pact.AdminActionResult{}, &cabana.ValidationError{Details: map[string]any{in.Field: []string{"golem15.seo::lang.api.value_invalid"}}}
+				}
+			}
+			db, err := s.DB(ctx)
+			if err != nil {
+				return pact.AdminActionResult{}, err
+			}
+			if in.RecordID == nil {
+				return pact.AdminActionResult{Data: LinkReport{Out: []LinkOut{}, In: []LinkIn{}}}, nil
+			}
+			locale := p.Locale
+			if locale == "" {
+				locale = s.DefaultLocale(ctx, db)
+			}
+			report, err := s.LinksOfRecord(ctx, db, key, strconv.FormatUint(*in.RecordID, 10), locale)
+			return pact.AdminActionResult{Data: report}, err
+		},
+	}
+}

classes/llms.go

@@ -0,0 +1,177 @@
+package classes
+
+import (
+	"context"
+	"sort"
+	"strings"
+
+	"git.golem15.com/golem15/sm-sitemap-plugin/contract"
+)
+
+// MaxLLMSEntries caps the links llms.txt lists.
+const MaxLLMSEntries = 10000
+
+// llmsGroup is one section of llms.txt.
+type llmsGroup struct {
+	title   string
+	order   int
+	key     string
+	entries []contract.Entry
+}
+
+// LLMSText builds /llms.txt (D-25, UI-SPEC C25): "# <site name>", the
+// introduction (llms_intro, else the default meta description) as a quote,
+// then one "## <group title>" section per sitemap group with listed
+// addresses in the default locale, each a "- [title](absolute url)" line.
+// It reads the sitemaps through contract.Listing, so an address the
+// sitemaps leave out (noindex, unpublished, canonicalised) never appears
+// (T-33-20). The groups follow the content types' order, as in the HTML
+// sitemap. ok is false while llms.txt is off and outside production.
+func (s *Service) LLMSText(ctx context.Context) (string, bool, error) {
+	if s.Staging() {
+		return "", false, nil
+	}
+	db, err := s.DB(ctx)
+	if err != nil {
+		return "", false, err
+	}
+	robots, err := LoadRobots(db)
+	if err != nil {
+		return "", false, err
+	}
+	if !robots.LlmsEnabled {
+		return "", false, nil
+	}
+	settings, err := LoadSettings(db)
+	if err != nil {
+		return "", false, err
+	}
+	locale := s.DefaultLocale(ctx, db)
+	var b strings.Builder
+	name := s.SiteName(ctx, settings, locale)
+	if name == "" {
+		name = s.SiteURL()
+	}
+	b.WriteString("# " + llmsLine(name) + "\n")
+	intro := strings.TrimSpace(robots.LlmsIntro)
+	if intro == "" {
+		intro = strings.TrimSpace(s.translatedSetting(ctx, db, &settings, "default_description", settings.DefaultDescription, locale))
+	}
+	if intro != "" {
+		b.WriteString("\n> " + llmsLine(intro) + "\n")
+	}
+	groups, err := s.llmsGroups(ctx, locale)
+	if err != nil {
+		return "", false, err
+	}
+	for _, g := range groups {
+		b.WriteString("\n## " + llmsLine(g.title) + "\n\n")
+		for _, e := range g.entries {
+			title := strings.TrimSpace(e.Title)
+			if title == "" {
+				title = e.URL
+			}
+			b.WriteString("- [" + llmsText(title) + "](" + llmsURL(e.URL) + ")\n")
+		}
+	}
+	return b.String(), true, nil
+}
+
+// llmsGroups reads the listed addresses of every sitemap group in locale,
+// ordered by the content type of their first address (registration order),
+// then by key.
+func (s *Service) llmsGroups(ctx context.Context, locale string) ([]llmsGroup, error) {
+	l := s.listing()
+	if l == nil {
+		return nil, nil
+	}
+	st, err := l.Settings(ctx)
+	if err != nil {
+		return nil, err
+	}
+	order := map[string]int{}
+	if reg := s.Registry(); reg != nil {
+		for i, typ := range reg.ContentTypes() {
+			order[typ.Key] = i
+		}
+	}
+	total := 0
+	var out []llmsGroup
+	for _, g := range st.Groups {
+		if g.Count == 0 || total >= MaxLLMSEntries {
+			continue
+		}
+		grp := llmsGroup{key: g.Key, order: len(order) + 1, title: groupTitle(g, locale, st.DefaultLocale)}
+		after := ""
+		for total < MaxLLMSEntries {
+			page, err := l.Entries(ctx, contract.EntriesQuery{Group: g.Key, Locale: locale, After: after, Limit: 1000})
+			if err != nil {
+				return nil, err
+			}
+			for _, e := range page.Entries {
+				if total >= MaxLLMSEntries {
+					break
+				}
+				if len(grp.entries) == 0 {
+					if i, ok := order[e.TypeKey]; ok {
+						grp.order = i
+					}
+				}
+				grp.entries = append(grp.entries, e)
+				total++
+			}
+			if page.Next == "" || page.Next == after {
+				break
+			}
+			after = page.Next
+		}
+		if len(grp.entries) > 0 {
+			out = append(out, grp)
+		}
+	}
+	sort.SliceStable(out, func(i, j int) bool {
+		if out[i].order != out[j].order {
+			return out[i].order < out[j].order
+		}
+		return out[i].key < out[j].key
+	})
+	return out, nil
+}
+
+// groupTitle is a group's heading in locale: its title there, else in the
+// sitemap's default locale, else any, else its key.
+func groupTitle(g contract.GroupInfo, locale, def string) string {
+	for _, l := range []string{locale, def} {
+		if t := strings.TrimSpace(g.Titles[l]); t != "" {
+			return t
+		}
+	}
+	keys := make([]string, 0, len(g.Titles))
+	for k := range g.Titles {
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+	for _, k := range keys {
+		if t := strings.TrimSpace(g.Titles[k]); t != "" {
+			return t
+		}
+	}
+	return g.Key
+}
+
+// llmsLine collapses text to one line.
+func llmsLine(text string) string {
+	return strings.Join(strings.Fields(text), " ")
+}
+
+// llmsText is a link text: one line with the Markdown brackets and
+// backslashes escaped.
+func llmsText(text string) string {
+	return strings.NewReplacer(`\`, `\\`, `[`, `\[`, `]`, `\]`).Replace(llmsLine(text))
+}
+
+// llmsURL is a link target: spaces and parentheses percent-encoded, so
+// the Markdown link stays whole.
+func llmsURL(u string) string {
+	return strings.NewReplacer(" ", "%20", "(", "%28", ")", "%29", "<", "%3C", ">", "%3E").Replace(strings.TrimSpace(u))
+}

classes/owner.go

@@ -5,6 +5,7 @@ import (
 	"net/url"
 	"strings"
 
+	"git.golem15.com/golem15/sm-seo-plugin/classes/schema"
 	"git.golem15.com/golem15/sm-seo-plugin/models"
 	"git.golem15.com/golem15/summercms/modules/pact"
 	"gorm.io/gorm"
@@ -126,6 +127,9 @@ func (s *Service) SetURLSEO(ctx context.Context, tx *gorm.DB, in URLSEOWrite) (U
 	for _, column := range EntryColumns {
 		known[column] = true
 	}
+	for _, column := range DataColumns {
+		known[column] = true
+	}
 	for column, raw := range in.Values {
 		v := strings.TrimSpace(raw)
 		switch {
@@ -148,6 +152,27 @@ func (s *Service) SetURLSEO(ctx context.Context, tx *gorm.DB, in URLSEOWrite) (U
 		case column == "og_image" && !ValidImageURL(v):
 			problems[column] = append(problems[column], models.Problem{Key: errImageURL})
 			continue
+		case (column == DataPrimaryParent || column == DataSchema) && owner.WrittenTo != WrittenToRecord:
+			if v != "" {
+				problems[column] = append(problems[column], models.Problem{Key: "golem15.seo::lang.api.record_only"})
+			}
+			continue
+		case column == DataPrimaryParent && v != "":
+			found := false
+			for _, opt := range s.ParentOptions(ctx, typ.Key, owner.ID, "") {
+				found = found || opt.Value == v
+			}
+			if !found {
+				problems[column] = append(problems[column], models.Problem{Key: "golem15.seo::lang.api.value_invalid"})
+				continue
+			}
+		case column == DataSchema:
+			stored, _, err := schema.Normalize(typ.Content.Kind, pageVariant(tx, typ.Key, owner.ID), v)
+			if err != nil {
+				problems[column] = append(problems[column], SchemaProblem(typ.Content.Kind, err))
+				continue
+			}
+			v = stored
 		}
 		values[column] = v
 	}
@@ -181,6 +206,9 @@ func (s *Service) SetURLSEO(ctx context.Context, tx *gorm.DB, in URLSEOWrite) (U
 		for _, column := range EntryColumns {
 			out[column] = EntryValue(e, column)
 		}
+		for _, column := range DataColumns {
+			out[column] = EntryValue(e, column)
+		}
 		return URLSEOResult{Owner: owner, Values: out}, nil
 	}
 	rule := models.Rule{URL: owner.URL, Match: models.MatchExact}

classes/pagination.go

@@ -0,0 +1,257 @@
+package classes
+
+import (
+	"context"
+	"net/url"
+	"regexp"
+	"strconv"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"gorm.io/gorm"
+)
+
+// PageParam is the query parameter of the query form (?page=N).
+const PageParam = "page"
+
+// The placeholders of the page suffixes (D-41). %page% is filled by the
+// resolver; %pages% is left for the layer, which knows the page count
+// (P33-24).
+const (
+	PlaceholderPage  = "%page%"
+	PlaceholderPages = "%pages%"
+)
+
+// DefaultPageSegments are the path segments of the path form when the
+// settings name none for a locale; other locales use "page".
+var DefaultPageSegments = map[string]string{"pl": "strona", "en": "page", "de": "seite"}
+
+// segmentPattern is a valid path segment (lowercase letters, digits,
+// hyphens).
+var segmentPattern = regexp.MustCompile(`^[a-z0-9-]+$`)
+
+// ValidPageSegment reports whether s may be a page segment.
+func ValidPageSegment(s string) bool { return segmentPattern.MatchString(s) }
+
+// Pagination is the resolve answer's pagination block (P33-21), present
+// only when the request carries a page: the site's form, the parameter of
+// the query form, the request locale's segment of the path form, the page
+// number (0 when the value is not a number) and whether the value is
+// invalid (the layer answers 404).
+type Pagination struct {
+	Form    string `json:"form"`
+	Param   string `json:"param"`
+	Segment string `json:"segment"`
+	Page    int    `json:"page"`
+	Invalid bool   `json:"invalid"`
+}
+
+// PageConfig is a site's pagination form (D-40): query or path, and the
+// path segment per locale.
+type PageConfig struct {
+	Form     string
+	Segments map[string]string
+}
+
+// Segment is the path segment of locale.
+func (c PageConfig) Segment(locale string) string {
+	if s := c.Segments[locale]; s != "" {
+		return s
+	}
+	if s := DefaultPageSegments[locale]; s != "" {
+		return s
+	}
+	return "page"
+}
+
+// PageConfigOf reads the pagination form of the settings: the segment of
+// each site locale is its translated setting when valid, else the default.
+func (s *Service) PageConfigOf(ctx context.Context, db *gorm.DB, settings models.Settings) PageConfig {
+	cfg := PageConfig{Form: models.PageFormQuery, Segments: map[string]string{}}
+	if settings.PageForm == models.PageFormPath {
+		cfg.Form = models.PageFormPath
+	}
+	for _, locale := range s.Locales(ctx, db) {
+		seg := strings.TrimSpace(s.translatedSetting(ctx, db, &settings, "page_segment", settings.PageSegment, locale))
+		if !ValidPageSegment(seg) {
+			seg = DefaultPageSegments[locale]
+			if seg == "" {
+				seg = "page"
+			}
+		}
+		cfg.Segments[locale] = seg
+	}
+	return cfg
+}
+
+// pageRequest is what a URL says about its page.
+type pageRequest struct {
+	// carried is true when the URL carries a page value.
+	carried bool
+	invalid bool
+	page    int
+	// form and segment are the form (and path segment) the URL used.
+	form    string
+	segment string
+	// base is the path without the path-form suffix; rest the query
+	// without the page parameter.
+	base string
+	rest url.Values
+	// exact is true when the number is written canonically (no leading
+	// zero, one value).
+	exact bool
+}
+
+// pageNumber reads a page value: digits only, at least 1.
+func pageNumber(v string) (int, bool) {
+	if v == "" || len(v) > 9 {
+		return 0, false
+	}
+	for _, r := range v {
+		if r < '0' || r > '9' {
+			return 0, false
+		}
+	}
+	n, err := strconv.Atoi(v)
+	if err != nil || n < 1 {
+		return 0, false
+	}
+	return n, true
+}
+
+// parsePage reads the page of a normalised path and query in either form:
+// ?page=N, or a trailing /<segment>/N with any locale's segment. A path
+// suffix whose value is not a number counts only in the path form.
+func parsePage(path string, query url.Values, cfg PageConfig) pageRequest {
+	pr := pageRequest{base: path, rest: url.Values{}, exact: true}
+	for k, vs := range query {
+		if k != PageParam {
+			pr.rest[k] = vs
+		}
+	}
+	if vs, ok := query[PageParam]; ok {
+		pr.carried, pr.form = true, models.PageFormQuery
+		v := ""
+		if len(vs) > 0 {
+			v = vs[0]
+		}
+		n, ok := pageNumber(v)
+		if !ok {
+			pr.invalid = true
+			return pr
+		}
+		pr.page, pr.exact = n, len(vs) == 1 && v == strconv.Itoa(n)
+	}
+	trimmed := strings.TrimSuffix(path, "/")
+	i := strings.LastIndex(trimmed, "/")
+	if i <= 0 {
+		return pr
+	}
+	head, value := trimmed[:i], trimmed[i+1:]
+	j := strings.LastIndex(head, "/")
+	if j < 0 {
+		return pr
+	}
+	prefix, segment := head[:j], head[j+1:]
+	known := false
+	for _, s := range cfg.Segments {
+		if s == segment {
+			known = true
+		}
+	}
+	if !known {
+		return pr
+	}
+	n, ok := pageNumber(value)
+	if !ok {
+		if cfg.Form == models.PageFormPath {
+			pr.carried, pr.invalid, pr.form, pr.segment = true, true, models.PageFormPath, segment
+		}
+		return pr
+	}
+	if prefix == "" {
+		prefix = "/"
+	}
+	exact := value == strconv.Itoa(n) && !pr.carried
+	pr.carried, pr.invalid, pr.form, pr.segment = true, false, models.PageFormPath, segment
+	pr.page, pr.base, pr.exact = n, prefix, exact
+	return pr
+}
+
+// pagedPath is base with the path-form suffix of page.
+func pagedPath(base, segment string, page int) string {
+	b := strings.TrimSuffix(base, "/")
+	return b + "/" + segment + "/" + strconv.Itoa(page)
+}
+
+// pageLocation is the address of page of base in the site's form, with
+// the other query parameters; page 1 (or none) is the address without a
+// number (D-41, D-42). It is escaped for a Location.
+func pageLocation(base string, rest url.Values, page int, cfg PageConfig, locale string, opts NormalizeOptions) string {
+	path := base
+	query := url.Values{}
+	for k, vs := range rest {
+		query[k] = vs
+	}
+	if page >= 2 {
+		if cfg.Form == models.PageFormPath {
+			path = pagedPath(base, cfg.Segment(locale), page)
+			if n, _ := Normalize(&url.URL{Path: path}, opts); n != nil {
+				path = n.Path
+			}
+		} else {
+			query.Set(PageParam, strconv.Itoa(page))
+		}
+	}
+	return joinQuery(escapePath(path), encodeQuery(query))
+}
+
+// pageAt is the page of a resolved URL (page 2 and later): its form, the
+// segment of the path form and the number.
+type pageAt struct {
+	form    string
+	segment string
+	page    int
+}
+
+// fillPage fills %page% of a suffix and keeps %pages%.
+func fillPage(suffix string, page int) string {
+	return strings.ReplaceAll(suffix, PlaceholderPage, strconv.Itoa(page))
+}
+
+// pageSuffix is the title or description suffix in locale: the translated
+// setting, else the default copy.
+func (s *Service) pageSuffix(ctx context.Context, db *gorm.DB, settings models.Settings, locale string, title bool) string {
+	field, own, key := "page_description_suffix", settings.PageDescriptionSuffix, "golem15.seo::lang.settings.page_description_suffix_default"
+	if title {
+		field, own, key = "page_title_suffix", settings.PageTitleSuffix, "golem15.seo::lang.settings.page_title_suffix_default"
+	}
+	v := strings.TrimSpace(s.translatedSetting(ctx, db, &settings, field, own, locale))
+	if v == "" {
+		v = s.PhraseIn(locale, key, nil)
+	}
+	return v
+}
+
+// localeOfPath guesses the locale of a path without a record: the site
+// locale whose /<code> prefix it carries, else the default locale.
+func (s *Service) localeOfPath(ctx context.Context, db *gorm.DB, path string) string {
+	lower := strings.ToLower(path)
+	for _, l := range s.Locales(ctx, db) {
+		prefix := "/" + strings.ToLower(l)
+		if lower == prefix || strings.HasPrefix(lower, prefix+"/") {
+			return l
+		}
+	}
+	return s.DefaultLocale(ctx, db)
+}
+
+// localeOfTag is the site locale whose language tag is tag, else tag.
+func (s *Service) localeOfTag(ctx context.Context, db *gorm.DB, tag string) string {
+	for _, l := range s.Locales(ctx, db) {
+		if LangTag(l) == tag {
+			return l
+		}
+	}
+	return tag
+}

classes/pagination_test.go

@@ -0,0 +1,108 @@
+package classes
+
+import (
+	"net/url"
+	"testing"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+)
+
+var pathForm = PageConfig{Form: models.PageFormPath, Segments: map[string]string{"pl": "strona", "en": "page"}}
+var queryForm = PageConfig{Form: models.PageFormQuery, Segments: map[string]string{"pl": "strona", "en": "page"}}
+
+// TestPageNumber pins the page value: digits only, at least 1, at most
+// nine digits.
+func TestPageNumber(t *testing.T) {
+	for v, want := range map[string]int{"1": 1, "2": 2, "007": 7, "999999999": 999999999} {
+		if n, ok := pageNumber(v); !ok || n != want {
+			t.Errorf("pageNumber(%q) = %d %v", v, n, ok)
+		}
+	}
+	for _, v := range []string{"", "0", "-1", "1.5", "a", "1e3", "1234567890", " 2"} {
+		if _, ok := pageNumber(v); ok {
+			t.Errorf("pageNumber(%q) accepted", v)
+		}
+	}
+}
+
+// TestParsePage pins both forms: ?page=N (other parameters kept, a
+// non-canonical number not exact, an invalid value), and a trailing
+// /<segment>/N with any locale's segment, an unknown segment ignored, and
+// a non-number suffix invalid only in the path form.
+func TestParsePage(t *testing.T) {
+	q := url.Values{"page": {"3"}, "sort": {"new"}}
+	pr := parsePage("/blog", q, queryForm)
+	if !pr.carried || pr.invalid || pr.page != 3 || pr.form != models.PageFormQuery || !pr.exact || pr.rest.Get("sort") != "new" || pr.rest.Has("page") || pr.base != "/blog" {
+		t.Fatalf("query form = %+v", pr)
+	}
+	if pr := parsePage("/blog", url.Values{"page": {"03"}}, queryForm); pr.exact || pr.page != 3 {
+		t.Fatalf("leading zero = %+v, want not exact", pr)
+	}
+	if pr := parsePage("/blog", url.Values{"page": {"2", "3"}}, queryForm); pr.exact {
+		t.Fatal("two values are not exact")
+	}
+	if pr := parsePage("/blog", url.Values{"page": {"x"}}, queryForm); !pr.carried || !pr.invalid {
+		t.Fatalf("invalid query value = %+v", pr)
+	}
+	if pr := parsePage("/blog", url.Values{"page": {""}}, queryForm); !pr.invalid {
+		t.Fatal("an empty page value is invalid")
+	}
+
+	pr = parsePage("/blog/strona/2", nil, pathForm)
+	if !pr.carried || pr.invalid || pr.page != 2 || pr.form != models.PageFormPath || pr.segment != "strona" || pr.base != "/blog" || !pr.exact {
+		t.Fatalf("path form = %+v", pr)
+	}
+	if pr := parsePage("/page/4/", nil, pathForm); pr.page != 4 || pr.base != "/" || pr.segment != "page" {
+		t.Fatalf("root path form = %+v", pr)
+	}
+	if pr := parsePage("/blog/seite/2", nil, pathForm); pr.carried {
+		t.Fatalf("an unknown segment = %+v", pr)
+	}
+	if pr := parsePage("/blog/strona/x", nil, pathForm); !pr.carried || !pr.invalid {
+		t.Fatalf("a non-number suffix in the path form = %+v", pr)
+	}
+	if pr := parsePage("/blog/strona/x", nil, queryForm); pr.carried {
+		t.Fatalf("a non-number suffix in the query form = %+v", pr)
+	}
+	if pr := parsePage("/blog/strona/02", nil, pathForm); pr.exact || pr.page != 2 {
+		t.Fatalf("a leading zero in the path = %+v", pr)
+	}
+	if pr := parsePage("/blog", nil, pathForm); pr.carried {
+		t.Fatal("no page")
+	}
+}
+
+// TestPageLocation pins the 301 targets: page 1 is the address without a
+// number, page 2 and later in the site's form with the other parameters.
+func TestPageLocation(t *testing.T) {
+	rest := url.Values{"sort": {"new"}}
+	if got := pageLocation("/blog", rest, 1, queryForm, "pl", NormalizeOptions{}); got != "/blog?sort=new" {
+		t.Fatalf("page 1 query = %q", got)
+	}
+	if got := pageLocation("/blog", rest, 3, queryForm, "pl", NormalizeOptions{}); got != "/blog?page=3&sort=new" {
+		t.Fatalf("page 3 query = %q", got)
+	}
+	if got := pageLocation("/blog/", nil, 2, pathForm, "pl", NormalizeOptions{}); got != "/blog/strona/2" {
+		t.Fatalf("page 2 path pl = %q", got)
+	}
+	if got := pageLocation("/blog", nil, 2, pathForm, "de", NormalizeOptions{}); got != "/blog/seite/2" {
+		t.Fatalf("page 2 path de = %q", got)
+	}
+	if got := pagedPath("/", "page", 5); got != "/page/5" {
+		t.Fatalf("pagedPath = %q", got)
+	}
+}
+
+// TestPageSegments pins the segment per locale and the placeholders.
+func TestPageSegments(t *testing.T) {
+	cfg := PageConfig{Segments: map[string]string{"pl": "str"}}
+	if cfg.Segment("pl") != "str" || cfg.Segment("de") != "seite" || cfg.Segment("en") != "page" || cfg.Segment("fr") != "page" {
+		t.Fatal("Segment is wrong")
+	}
+	if !ValidPageSegment("strona-2") || ValidPageSegment("Strona") || ValidPageSegment("") || ValidPageSegment("a/b") {
+		t.Fatal("ValidPageSegment is wrong")
+	}
+	if fillPage(" – strona %page% z %pages%", 2) != " – strona 2 z %pages%" {
+		t.Fatal("fillPage must fill the page number and keep the pages placeholder")
+	}
+}

classes/partials.go

@@ -5,9 +5,12 @@ import (
 	"fmt"
 	"net/url"
 	"reflect"
+	"strconv"
 	"strings"
 
+	"git.golem15.com/golem15/sm-seo-plugin/classes/schema"
 	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/sm-sitemap-plugin/contract"
 	"gorm.io/gorm"
 )
 
@@ -15,10 +18,12 @@ import (
 // the request locale), never a GORM model; html/template escapes every
 // value.
 
-// Callout is one P1 warning.
+// Callout is one P1 callout: a warning, or a plain note when Plain is
+// set (the sitemap status).
 type Callout struct {
 	Title string
 	Text  string
+	Plain bool
 }
 
 // StatusView is the P1 seo_status view model.
@@ -101,7 +106,7 @@ func (s *Service) EffectiveRows(ctx context.Context, res Result) []EffectiveRow
 		"title": res.SEO.Title, "description": res.SEO.Description, "canonical": res.SEO.Canonical,
 		"robots": res.SEO.Robots, "og_title": res.SEO.OG.Title, "og_description": res.SEO.OG.Description, "og_image": res.SEO.OG.Image,
 	}
-	out := make([]EffectiveRow, 0, len(effectiveProperties))
+	out := make([]EffectiveRow, 0, len(effectiveProperties)+2)
 	for _, p := range effectiveProperties {
 		out = append(out, EffectiveRow{
 			Property: s.PhraseCtx(ctx, p.label, nil),
@@ -110,9 +115,47 @@ func (s *Service) EffectiveRows(ctx context.Context, res Result) []EffectiveRow
 			Mono:     p.mono,
 		})
 	}
+	// Structured data (UI-SPEC C16): the page's schema type and its
+	// breadcrumb trail.
+	if res.JSONLD != nil {
+		typ, source := effectiveSchemaType(res), SourceDefault
+		if res.Sources["schema"] != "" {
+			source = res.Sources["schema"]
+		}
+		out = append(out, EffectiveRow{
+			Property: s.PhraseCtx(ctx, "golem15.seo::lang.effective.schema", nil),
+			Value:    typ,
+			Source:   s.PhraseCtx(ctx, "golem15.seo::lang.effective.source_"+source, nil),
+		})
+		names := make([]string, 0, len(res.Breadcrumbs))
+		for _, c := range res.Breadcrumbs {
+			names = append(names, c.Name)
+		}
+		out = append(out, EffectiveRow{
+			Property: s.PhraseCtx(ctx, "golem15.seo::lang.effective.trail", nil),
+			Value:    strings.Join(names, " › "),
+			Source:   s.PhraseCtx(ctx, "golem15.seo::lang.effective.source_"+SourceContent, nil),
+		})
+	}
 	return out
 }
 
+// effectiveSchemaType is the page's main schema type: its article or
+// service node, else its page node.
+func effectiveSchemaType(res Result) string {
+	page := ""
+	for _, n := range schema.Nodes(res.JSONLD) {
+		id, _ := n["@id"].(string)
+		switch id {
+		case res.URL + "#article", res.URL + "#service":
+			return schema.NodeType(n)
+		case res.URL + "#webpage":
+			page = schema.NodeType(n)
+		}
+	}
+	return page
+}
+
 // newEffectiveView starts a P2 view with its labels and the staging note.
 func (s *Service) newEffectiveView(ctx context.Context) EffectiveView {
 	return EffectiveView{
@@ -202,9 +245,36 @@ func (s *Service) RecordStatus(ctx context.Context, key, id string) (StatusView,
 			}
 		}
 	}
+	if c, ok := s.primaryGoneCallout(ctx, db, key, id); ok {
+		view.Callouts = append(view.Callouts, c)
+	}
+	exclusions, err := s.RecordSitemapExclusions(ctx, key, id)
+	if err != nil {
+		return view, err
+	}
+	for _, x := range exclusions {
+		key, ok := excludedTexts[x.Reason]
+		if !ok {
+			continue
+		}
+		view.Callouts = append(view.Callouts, Callout{
+			Title: s.PhraseCtx(ctx, "golem15.seo::lang.status.excluded_title", nil),
+			Text:  s.PhraseCtx(ctx, key, map[string]string{"url": x.Target}),
+			Plain: true,
+		})
+	}
 	return view, nil
 }
 
+// excludedTexts are the P1 texts of the sitemap exclusion reasons (UI-SPEC
+// C16).
+var excludedTexts = map[string]string{
+	contract.ReasonNoindex:   "golem15.seo::lang.status.excluded_noindex",
+	contract.ReasonCanonical: "golem15.seo::lang.status.excluded_canonical",
+	contract.ReasonRedirect:  "golem15.seo::lang.status.excluded_redirect",
+	contract.ReasonType:      "golem15.seo::lang.status.excluded_type",
+}
+
 // ruleCallout is the override callout of the first rule that sets a value.
 func (s *Service) ruleCallout(ctx context.Context, rules []models.Rule) (Callout, bool) {
 	for _, r := range rules {
@@ -319,3 +389,149 @@ func RecordKey(db *gorm.DB, record any) (string, error) {
 func reflectValue(record any) reflect.Value {
 	return reflect.Indirect(reflect.ValueOf(record))
 }
+
+// primaryGoneCallout warns that a record's stored primary category is no
+// longer one of its parents, so the first one is used (UI-SPEC C16).
+func (s *Service) primaryGoneCallout(ctx context.Context, db *gorm.DB, key, id string) (Callout, bool) {
+	reg := s.Registry()
+	if reg == nil {
+		return Callout{}, false
+	}
+	typ, ok := reg.ContentType(key)
+	if !ok || typ.Content == nil {
+		return Callout{}, false
+	}
+	entries, err := LoadEntries(db, typ.Content.Model, id)
+	if err != nil {
+		return Callout{}, false
+	}
+	stored := SharedData(entries, DataPrimaryParent)
+	if stored == "" {
+		return Callout{}, false
+	}
+	rec, err := reg.Describe(ctx, key, id, s.UnprefixedLocale())
+	if err != nil || rec == nil {
+		return Callout{}, false
+	}
+	if _, _, gone := PrimaryParent(rec, stored); !gone {
+		return Callout{}, false
+	}
+	name := stored
+	if pt, pid, ok := strings.Cut(stored, ":"); ok {
+		if parent, err := reg.Describe(ctx, pt, pid, s.UnprefixedLocale()); err == nil && parent != nil {
+			if l := contentLocale(parent, s.DefaultLocale(ctx, db)); strings.TrimSpace(l.Title) != "" {
+				name = l.Title
+			}
+		}
+	}
+	return Callout{
+		Title: s.PhraseCtx(ctx, "golem15.seo::lang.status.primary_gone", map[string]string{"name": name}),
+	}, true
+}
+
+// IssueParams turns an issue's details into the :name parameters of its
+// issues.sum_<kind> phrase: scalars as text, lists by their length or
+// joined (fields, also), with the aliases the Phase 32 kinds need (count
+// from also or hops, url from canonical).
+func IssueParams(details map[string]any) map[string]string {
+	out := map[string]string{}
+	for k, v := range details {
+		switch x := v.(type) {
+		case string:
+			out[k] = x
+		case float64:
+			out[k] = strconv.FormatFloat(x, 'f', -1, 64)
+		case int:
+			out[k] = strconv.Itoa(x)
+		case bool:
+			out[k] = strconv.FormatBool(x)
+		case []any:
+			parts := make([]string, 0, len(x))
+			for _, e := range x {
+				if s, ok := e.(string); ok {
+					parts = append(parts, s)
+				}
+			}
+			out[k] = strings.Join(parts, ", ")
+		case []string:
+			out[k] = strings.Join(x, ", ")
+		}
+	}
+	if _, ok := out["count"]; !ok {
+		if also, ok := details["also"].([]any); ok {
+			out["count"] = strconv.Itoa(len(also))
+		} else if hops, ok := out["hops"]; ok {
+			out["count"] = hops
+		}
+	}
+	if _, ok := out["url"]; !ok {
+		if c, ok := out["canonical"]; ok {
+			out["url"] = c
+		}
+	}
+	return out
+}
+
+// IssueSummary is the one-line Details cell of an issue (issues.sum_<kind>).
+func (s *Service) IssueSummary(ctx context.Context, kind string, details map[string]any) string {
+	return s.PhraseCtx(ctx, "golem15.seo::lang.issues.sum_"+kind, IssueParams(details))
+}
+
+// IssueRow is one row of P7's "What was found" table.
+type IssueRow struct {
+	Label string
+	Value string
+	Mono  bool
+}
+
+// IssueLink is a page named in P7 (a linking page or a suggestion): its
+// address and, when the page has an admin form, the admin path.
+type IssueLink struct {
+	URL    string
+	Admin  string
+	Phrase string
+}
+
+// IssueDetailsView is the P7 issue_details view model (UI-SPEC C23).
+type IssueDetailsView struct {
+	Rows         []IssueRow
+	LinkedFrom   []IssueLink
+	More         string
+	Orphan       bool
+	Suggestions  []IssueLink
+	FixText      string
+	PageURL      string
+	RichResults  bool
+	RichURL      string
+	IgnoredNotes string
+}
+
+// CrawlStat is one item of P6's stats strip.
+type CrawlStat struct {
+	Label string
+	Value string
+}
+
+// CrawlKindRow is one row of P6's per-kind table.
+type CrawlKindRow struct {
+	Kind     string
+	Label    string
+	Severity string
+	Count    int
+	Link     string
+}
+
+// CrawlSummaryView is the P6 crawl_summary view model (UI-SPEC C24): one
+// status callout (Tone "", warning or danger), the staging and rendered
+// comparison lines, the stats strip and the per-kind table.
+type CrawlSummaryView struct {
+	Tone       string
+	Title      string
+	Text       string
+	Staging    string
+	Render     string
+	Stats      []CrawlStat
+	Kinds      []CrawlKindRow
+	NoIssues   bool
+	HasCallout bool
+}

classes/registry.go

@@ -0,0 +1,979 @@
+package classes
+
+import (
+	"context"
+	"errors"
+	"net/url"
+	"sort"
+	"strconv"
+	"strings"
+	"time"
+	"unicode"
+	"unicode/utf8"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/sm-sitemap-plugin/contract"
+	"git.golem15.com/golem15/summercms/modules/cabana"
+	"git.golem15.com/golem15/summercms/modules/lagoon"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
+	"gorm.io/gorm/clause"
+)
+
+// The page registry (D-33): the site's own frontend pages (home, contact,
+// offer pages) as a signpost content type of the SEO plugin, so each page
+// gets the SEO tab, resolves ok and is listed in the sitemaps.
+const (
+	// PageTypeKey is the registry's menu item type.
+	PageTypeKey = "seo-page"
+	// PagesController is the Pages screen's admin controller.
+	PagesController = "golem15.seo.pages"
+	// PagePattern is the registry type's pattern. A registry page stores
+	// its own address per locale, so the pattern only satisfies signpost.
+	PagePattern = "/:page"
+	// PageLabel is the registry type's label (a content type's label names
+	// its tab in the SEO settings, so it is not the plain "Page" another
+	// content type may use).
+	PageLabel = "golem15.seo::lang.pages.type"
+)
+
+// PageModel is the registry page's morph type, the model its SEO entries
+// and record events name.
+var PageModel = models.Page{}.MorphName()
+
+// The route report's limits (P33-20).
+const (
+	MaxReportedRoutes = 2000
+	MaxRouteBytes     = 2048
+)
+
+// RouteReportTokenConfigKey is the bearer token the frontend server sends
+// with its route report to POST /_seo/api/v1/registry/routes (P33-20).
+// Empty uses golem15.seo.notfound_token; both empty turn the report off.
+const RouteReportTokenConfigKey = "golem15.seo.route_report_token"
+
+// RouteReportToken is the route report's token: route_report_token, else
+// notfound_token, "" when neither is set.
+func (s *Service) RouteReportToken() string {
+	if cfg := s.Config(); cfg != nil {
+		if t := strings.TrimSpace(cfg.String(RouteReportTokenConfigKey)); t != "" {
+			return t
+		}
+	}
+	return s.NotFoundToken()
+}
+
+// PageOrigin is the origin of a page created through the management API:
+// mcp through MCP, else manual.
+func PageOrigin(ctx context.Context) string {
+	if ViaOf(ctx) == models.ViaMCP {
+		return models.PageOriginMCP
+	}
+	return models.PageOriginManual
+}
+
+// ErrTooManyRoutes refuses a route report of more than MaxReportedRoutes
+// paths.
+var ErrTooManyRoutes = errors.New("golem15.seo: a route report holds at most 2000 paths")
+
+// The phrase keys of the registry refusals (UI-SPEC C21).
+const (
+	errPageURLRequired = "golem15.seo::lang.pages.url_required"
+	errPageURLInvalid  = "golem15.seo::lang.pages.url_invalid"
+	errPageURLTaken    = "golem15.seo::lang.pages.url_taken"
+	errPageURLOwned    = "golem15.seo::lang.pages.url_owned"
+	errPageParentLoop  = "golem15.seo::lang.pages.parent_loop"
+	errValueInvalid    = "golem15.seo::lang.api.value_invalid"
+)
+
+// pageGroups are the sitemap file groups of the page kinds (P33-18).
+var pageGroups = map[string]string{
+	models.PageKindPage:    "pages",
+	models.PageKindOffer:   "offers",
+	models.PageKindService: "services",
+	models.PageKindLanding: "landing-pages",
+}
+
+// PageKindLabels are the phrase keys of the page kinds.
+var PageKindLabels = map[string]string{
+	models.PageKindPage:    "golem15.seo::lang.pages.kind_page",
+	models.PageKindOffer:   "golem15.seo::lang.pages.kind_offer",
+	models.PageKindService: "golem15.seo::lang.pages.kind_service",
+	models.PageKindLanding: "golem15.seo::lang.pages.kind_landing",
+}
+
+// PageGroup is the sitemap file group of a page kind; an unknown kind is
+// listed with the pages.
+func PageGroup(kind string) string {
+	if g, ok := pageGroups[kind]; ok {
+		return g
+	}
+	return pageGroups[models.PageKindPage]
+}
+
+// LoadPage returns registry page id; ErrNoRecord when there is none.
+func LoadPage(db *gorm.DB, id uint) (models.Page, error) {
+	var rows []models.Page
+	if err := db.Where("id = ?", id).Limit(1).Find(&rows).Error; err != nil {
+		return models.Page{}, err
+	}
+	if len(rows) == 0 {
+		return models.Page{}, ErrNoRecord
+	}
+	return rows[0], nil
+}
+
+// pageByID reads a registry page by its id as text, nil when none.
+func pageByID(db *gorm.DB, id string) (*models.Page, error) {
+	n, err := strconv.ParseUint(strings.TrimSpace(id), 10, 64)
+	if err != nil || n == 0 {
+		return nil, nil
+	}
+	p, err := LoadPage(db, uint(n))
+	if errors.Is(err, ErrNoRecord) {
+		return nil, nil
+	}
+	if err != nil {
+		return nil, err
+	}
+	return &p, nil
+}
+
+// PageAt returns the registry page whose address in any locale is path,
+// with that locale; nil when none.
+func (s *Service) PageAt(ctx context.Context, db *gorm.DB, path string) (*models.Page, string, error) {
+	var rows []models.Page
+	if err := db.Where("url = ?", path).Limit(1).Find(&rows).Error; err != nil {
+		return nil, "", err
+	}
+	if len(rows) > 0 {
+		return &rows[0], s.DefaultLocale(ctx, db), nil
+	}
+	var hit []struct {
+		ID     uint   `gorm:"column:id"`
+		Locale string `gorm:"column:locale"`
+	}
+	if err := db.Raw(`SELECT p.id, e.key AS locale FROM golem15_seo_pages p, jsonb_each_text(p.urls) e WHERE e.value = ? ORDER BY p.id LIMIT 1`, path).Scan(&hit).Error; err != nil {
+		return nil, "", err
+	}
+	if len(hit) == 0 {
+		return nil, "", nil
+	}
+	p, err := LoadPage(db, hit[0].ID)
+	if err != nil {
+		return nil, "", err
+	}
+	if def := s.DefaultLocale(ctx, db); hit[0].Locale == def {
+		return nil, "", nil
+	}
+	return &p, hit[0].Locale, nil
+}
+
+// PageRecord describes a registry page as a signpost content record: its
+// address and name per locale (the default locale first, then the other
+// enabled locales with an address), its content date as lastmod and its
+// parent page.
+func (s *Service) PageRecord(ctx context.Context, db *gorm.DB, p models.Page) pact.ContentRecord {
+	def := s.DefaultLocale(ctx, db)
+	urls := p.LocaleURLs(def)
+	rec := pact.ContentRecord{ID: p.IDText(), Published: true, LastMod: p.UpdatedAt}
+	if p.ContentUpdatedAt != nil && !p.ContentUpdatedAt.IsZero() {
+		day := p.ContentUpdatedAt.Time(time.UTC)
+		rec.LastMod = &day
+	}
+	for _, locale := range s.pageLocales(ctx, db, urls) {
+		rec.Locales = append(rec.Locales, pact.ContentLocale{Locale: locale, URL: urls[locale], Title: p.TitleIn(locale)})
+	}
+	if p.ParentID != nil && *p.ParentID != 0 {
+		rec.Parents = []pact.ContentRef{{Type: PageTypeKey, ID: strconv.FormatUint(uint64(*p.ParentID), 10)}}
+	}
+	return rec
+}
+
+// pageLocales orders the locales of urls: the site's locales first (the
+// default one leading), then any other stored locale by code.
+func (s *Service) pageLocales(ctx context.Context, db *gorm.DB, urls map[string]string) []string {
+	var out []string
+	seen := map[string]bool{}
+	for _, l := range s.Locales(ctx, db) {
+		if urls[l] != "" && !seen[l] {
+			out = append(out, l)
+			seen[l] = true
+		}
+	}
+	var rest []string
+	for l := range urls {
+		if !seen[l] {
+			rest = append(rest, l)
+		}
+	}
+	sort.Strings(rest)
+	return append(out, rest...)
+}
+
+// PageMenuItemType is the registry's menu item type seo-page: a menu item
+// may point at a registry page, and its content descriptor lists, describes
+// and matches the pages for signpost's consumers (the SEO tab, the
+// resolver, the sitemaps). service is asked on every call.
+func PageMenuItemType(service func() *Service) pact.MenuItemType {
+	svc := func() (*Service, error) {
+		if service == nil {
+			return nil, ErrNotBooted
+		}
+		s := service()
+		if s == nil {
+			return nil, ErrNotBooted
+		}
+		return s, nil
+	}
+	return pact.MenuItemType{
+		Key:            PageTypeKey,
+		Label:          PageLabel,
+		DefaultPattern: PagePattern,
+		References: func(ctx context.Context) ([]pact.MenuItemReference, error) {
+			s, err := svc()
+			if err != nil {
+				return nil, err
+			}
+			return s.pageReferences(ctx)
+		},
+		Resolve: func(ctx context.Context, in pact.MenuItemResolve) (*pact.ResolvedMenuItem, error) {
+			s, err := svc()
+			if err != nil {
+				return nil, err
+			}
+			return s.resolvePageItem(ctx, in.Reference)
+		},
+		Content: &pact.ContentDescriptor{
+			Controller: PagesController,
+			Model:      PageModel,
+			Kind:       pact.ContentKindPage,
+			// A registry page's fields are changed through seo_update_page,
+			// the tool the site audit names for its content (P33-38).
+			Tools: map[string]string{pact.ContentToolEdit: "seo_update_page"},
+			Records: func(ctx context.Context, in pact.ContentList) (pact.ContentPage, error) {
+				s, err := svc()
+				if err != nil {
+					return pact.ContentPage{}, err
+				}
+				return s.pageRecords(ctx, in)
+			},
+			Describe: func(ctx context.Context, in pact.ContentDescribe) (*pact.ContentRecord, error) {
+				s, err := svc()
+				if err != nil {
+					return nil, err
+				}
+				db, err := s.DB(ctx)
+				if err != nil {
+					return nil, err
+				}
+				p, err := pageByID(db, in.ID)
+				if err != nil || p == nil {
+					return nil, err
+				}
+				rec := s.PageRecord(ctx, db, *p)
+				return &rec, nil
+			},
+			Match: func(ctx context.Context, in pact.ContentMatch) (*pact.ContentMatchResult, error) {
+				s, err := svc()
+				if err != nil {
+					return nil, err
+				}
+				db, err := s.DB(ctx)
+				if err != nil {
+					return nil, err
+				}
+				p, locale, err := s.PageAt(ctx, db, in.Path)
+				if err != nil || p == nil {
+					return nil, err
+				}
+				return &pact.ContentMatchResult{ID: p.IDText(), Locale: locale, Exact: true}, nil
+			},
+		},
+	}
+}
+
+// pageRecords is one page of registry records by id.
+func (s *Service) pageRecords(ctx context.Context, in pact.ContentList) (pact.ContentPage, error) {
+	db, err := s.DB(ctx)
+	if err != nil {
+		return pact.ContentPage{}, err
+	}
+	limit := in.Limit
+	if limit <= 0 || limit > 1000 {
+		limit = 1000
+	}
+	after, _ := strconv.ParseUint(strings.TrimSpace(in.After), 10, 64)
+	var rows []models.Page
+	if err := db.Where("id > ? AND url <> ''", after).Order("id ASC").Limit(limit).Find(&rows).Error; err != nil {
+		return pact.ContentPage{}, err
+	}
+	out := pact.ContentPage{}
+	for _, p := range rows {
+		out.Records = append(out.Records, s.PageRecord(ctx, db, p))
+	}
+	if len(rows) == limit {
+		out.Next = rows[len(rows)-1].IDText()
+	}
+	return out, nil
+}
+
+// pageReferences lists the registry pages a menu item can point at, as a
+// tree by parent, each level by address.
+func (s *Service) pageReferences(ctx context.Context) ([]pact.MenuItemReference, error) {
+	db, err := s.DB(ctx)
+	if err != nil {
+		return nil, err
+	}
+	var rows []models.Page
+	if err := db.Where("url <> ''").Order("url ASC").Limit(5000).Find(&rows).Error; err != nil {
+		return nil, err
+	}
+	children := map[uint][]models.Page{}
+	known := map[uint]bool{}
+	for _, p := range rows {
+		known[p.ID] = true
+	}
+	for _, p := range rows {
+		parent := p.Parent()
+		if !known[parent] {
+			parent = 0
+		}
+		children[parent] = append(children[parent], p)
+	}
+	visited := map[uint]bool{}
+	var build func(parent uint, depth int) []pact.MenuItemReference
+	build = func(parent uint, depth int) []pact.MenuItemReference {
+		var out []pact.MenuItemReference
+		for _, p := range children[parent] {
+			if visited[p.ID] || depth > 50 {
+				continue
+			}
+			visited[p.ID] = true
+			label := p.Title
+			if strings.TrimSpace(label) == "" {
+				label = p.URL
+			}
+			out = append(out, pact.MenuItemReference{Value: p.IDText(), Label: label, Items: build(p.ID, depth+1)})
+		}
+		return out
+	}
+	return build(0, 0), nil
+}
+
+// resolvePageItem resolves a menu item pointing at registry page id: its
+// default-locale address and name, and one alternate per locale.
+func (s *Service) resolvePageItem(ctx context.Context, id string) (*pact.ResolvedMenuItem, error) {
+	db, err := s.DB(ctx)
+	if err != nil {
+		return nil, err
+	}
+	p, err := pageByID(db, id)
+	if err != nil || p == nil {
+		return nil, err
+	}
+	rec := s.PageRecord(ctx, db, *p)
+	item := &pact.ResolvedMenuItem{Title: p.Title, URL: p.URL, MTime: rec.LastMod}
+	for _, l := range rec.Locales {
+		item.Alternates = append(item.Alternates, pact.MenuItemAlternate{Locale: l.Locale, URL: l.URL, Title: l.Title})
+	}
+	return item, nil
+}
+
+// PageWrite is one change of a registry page through the API or MCP: a
+// nil field is left as it is. URL and Title map locales to text; a locale
+// sent empty removes that locale's address or name, and the "" key stands
+// for the default locale. ParentSet with a nil ParentID moves the page to
+// the top level.
+type PageWrite struct {
+	URL              map[string]string
+	Title            map[string]string
+	Kind             *string
+	ParentSet        bool
+	ParentID         *uint
+	ContentUpdatedAt *string
+	Note             *string
+}
+
+// ApplyPageTexts merges submitted addresses and names into p: def is the
+// default locale, whose values go to the url and title columns. Every
+// address is trimmed and normalised with the site's address rules, so it
+// is stored in the form the resolver matches.
+func (s *Service) ApplyPageTexts(ctx context.Context, db *gorm.DB, p *models.Page, urls, titles map[string]string) error {
+	def := s.DefaultLocale(ctx, db)
+	settings, err := LoadSettings(db)
+	if err != nil {
+		return err
+	}
+	opts := NormalizeOptionsOf(settings)
+	if urls != nil {
+		others := map[string]string{}
+		for l, v := range p.URLs.Get() {
+			others[l] = v
+		}
+		for locale, path := range urls {
+			if locale == "" {
+				locale = def
+			}
+			path = normalPagePath(path, opts)
+			if locale == def {
+				p.URL = path
+				continue
+			}
+			if path == "" {
+				delete(others, locale)
+			} else {
+				others[locale] = path
+			}
+		}
+		p.URLs = lagoon.Jsonable[map[string]string]{Data: others, Valid: true}
+	}
+	if titles != nil {
+		others := map[string]string{}
+		for l, v := range p.Titles.Get() {
+			others[l] = v
+		}
+		for locale, title := range titles {
+			title = strings.TrimSpace(title)
+			if locale == "" {
+				locale = def
+			}
+			if locale == def {
+				p.Title = title
+				continue
+			}
+			if title == "" {
+				delete(others, locale)
+			} else {
+				others[locale] = title
+			}
+		}
+		p.Titles = lagoon.Jsonable[map[string]string]{Data: others, Valid: true}
+	}
+	if !p.URLs.Valid {
+		p.URLs = lagoon.Jsonable[map[string]string]{Data: map[string]string{}, Valid: true}
+	}
+	if !p.Titles.Valid {
+		p.Titles = lagoon.Jsonable[map[string]string]{Data: map[string]string{}, Valid: true}
+	}
+	return nil
+}
+
+// normalPagePath trims an address and, when it is a valid path, applies
+// the site's address rules (lowercase, trailing slash) to it.
+func normalPagePath(path string, opts NormalizeOptions) string {
+	path = strings.TrimSpace(path)
+	if !ValidPagePath(path) {
+		return path
+	}
+	norm, _ := Normalize(&url.URL{Path: path}, opts)
+	if norm.Path == "" {
+		return path
+	}
+	return norm.Path
+}
+
+// ValidPagePath reports whether path is a registry address: a path
+// starting with a single "/", without a domain, query or fragment, of at
+// most MaxRouteBytes bytes of valid UTF-8 without control characters.
+func ValidPagePath(path string) bool {
+	if path == "" || len(path) > MaxRouteBytes || !strings.HasPrefix(path, "/") || strings.HasPrefix(path, "//") {
+		return false
+	}
+	if !utf8.ValidString(path) || strings.ContainsAny(path, "?#\\ ") {
+		return false
+	}
+	for _, r := range path {
+		if unicode.IsControl(r) {
+			return false
+		}
+	}
+	u, err := url.Parse(path)
+	return err == nil && !u.IsAbs() && u.Host == ""
+}
+
+// ApplyPageFields sets the scalar fields of w on p; an invalid kind or
+// date is a problem.
+func ApplyPageFields(p *models.Page, w PageWrite) map[string][]models.Problem {
+	problems := map[string][]models.Problem{}
+	if w.Kind != nil {
+		p.Kind = strings.TrimSpace(*w.Kind)
+	}
+	if w.ParentSet {
+		p.ParentID = nil
+		if w.ParentID != nil && *w.ParentID != 0 {
+			ref := models.PageRef(*w.ParentID)
+			p.ParentID = &ref
+		}
+	}
+	if w.ContentUpdatedAt != nil {
+		v := strings.TrimSpace(*w.ContentUpdatedAt)
+		if v == "" {
+			p.ContentUpdatedAt = nil
+		} else if d, err := lagoon.ParseDate(v); err == nil {
+			p.ContentUpdatedAt = &d
+		} else {
+			problems["content_updated_at"] = append(problems["content_updated_at"], models.Problem{Key: errValueInvalid})
+		}
+	}
+	if w.Note != nil {
+		p.Note = strings.TrimSpace(*w.Note)
+	}
+	return problems
+}
+
+// CheckPage lists the refusals of a registry page before a save (UI-SPEC
+// C21), keyed by field (url, kind, parent_id): a missing default-locale
+// address, an invalid address, an address another registry page uses, an
+// address another content type owns (naming the type and the record), an
+// unknown kind, and a parent that is unknown or would make a loop.
+func (s *Service) CheckPage(ctx context.Context, db *gorm.DB, p models.Page) (map[string][]models.Problem, error) {
+	problems := map[string][]models.Problem{}
+	add := func(field string, pr models.Problem) {
+		for _, have := range problems[field] {
+			if have.Key == pr.Key {
+				return
+			}
+		}
+		problems[field] = append(problems[field], pr)
+	}
+	if !models.KnownPageKind(p.Kind) {
+		add("kind", models.Problem{Key: errValueInvalid})
+	}
+	def := s.DefaultLocale(ctx, db)
+	if strings.TrimSpace(p.URL) == "" {
+		add("url", models.Problem{Key: errPageURLRequired})
+	}
+	urls := p.LocaleURLs(def)
+	var paths []string
+	for _, locale := range s.pageLocales(ctx, db, urls) {
+		path := urls[locale]
+		if !ValidPagePath(path) {
+			add("url", models.Problem{Key: errPageURLInvalid})
+			continue
+		}
+		paths = append(paths, path)
+	}
+	if len(paths) > 0 {
+		var taken int64
+		q := db.Model(&models.Page{}).Where(`(url IN ? OR EXISTS (SELECT 1 FROM jsonb_each_text(golem15_seo_pages.urls) e WHERE e.value IN ?))`, paths, paths)
+		if p.ID != 0 {
+			q = q.Where("id <> ?", p.ID)
+		}
+		if err := q.Count(&taken).Error; err != nil {
+			return nil, err
+		}
+		if taken > 0 {
+			add("url", models.Problem{Key: errPageURLTaken})
+		}
+		for _, path := range paths {
+			owner, err := s.ownerOutsideRegistry(ctx, path)
+			if err != nil {
+				return nil, err
+			}
+			if owner != nil {
+				add("url", *owner)
+				break
+			}
+		}
+	}
+	if parent := p.Parent(); parent != 0 {
+		loop, known, err := parentLoop(db, p.ID, parent)
+		if err != nil {
+			return nil, err
+		}
+		switch {
+		case !known:
+			add("parent_id", models.Problem{Key: errValueInvalid})
+		case loop:
+			add("parent_id", models.Problem{Key: errPageParentLoop})
+		}
+	}
+	return problems, nil
+}
+
+// parentLoop walks up from parent: loop is true when it reaches id (or
+// goes round), known false when parent does not exist.
+func parentLoop(db *gorm.DB, id, parent uint) (loop, known bool, err error) {
+	seen := map[uint]bool{}
+	cur := parent
+	for depth := 0; cur != 0; depth++ {
+		if (id != 0 && cur == id) || seen[cur] || depth > 100 {
+			return true, true, nil
+		}
+		seen[cur] = true
+		p, err := LoadPage(db, cur)
+		if errors.Is(err, ErrNoRecord) {
+			return false, depth > 0, nil
+		}
+		if err != nil {
+			return false, false, err
+		}
+		if p.Parent() == 0 {
+			return false, true, nil
+		}
+		cur = p.Parent()
+	}
+	return false, true, nil
+}
+
+// ownerOutsideRegistry returns the url_owned refusal when a content type
+// other than the registry owns path exactly: the type's label and the
+// record's title as params. nil when none does.
+func (s *Service) ownerOutsideRegistry(ctx context.Context, path string) (*models.Problem, error) {
+	reg := s.Registry()
+	if reg == nil {
+		return nil, nil
+	}
+	unprefixed := s.UnprefixedLocale()
+	for _, typ := range reg.ContentTypes() {
+		if typ.Key == PageTypeKey || typ.Content == nil || typ.Content.Match == nil {
+			continue
+		}
+		got, err := typ.Content.Match(ctx, pact.ContentMatch{Path: trimSlash(path), Pattern: reg.Pattern(typ.Key, ""), UnprefixedLocale: unprefixed})
+		if err != nil {
+			return nil, err
+		}
+		if got == nil || !got.Exact {
+			continue
+		}
+		title := path
+		if rec, err := reg.Describe(ctx, typ.Key, got.ID, unprefixed); err == nil && rec != nil {
+			if l := contentLocale(rec, got.Locale); strings.TrimSpace(l.Title) != "" {
+				title = l.Title
+			}
+		}
+		label := strings.TrimSpace(typ.Label)
+		switch {
+		case label == "":
+			label = typ.Key
+		case strings.Contains(label, "::"):
+			label = s.Phrase(label, nil)
+		}
+		return &models.Problem{Key: errPageURLOwned, Params: map[string]string{"type": label, "title": title}}, nil
+	}
+	return nil, nil
+}
+
+// PageValues are a registry page's values for history.
+func PageValues(p models.Page) map[string]any {
+	parent := any(nil)
+	if p.Parent() != 0 {
+		parent = p.Parent()
+	}
+	date := ""
+	if p.ContentUpdatedAt != nil {
+		date = p.ContentUpdatedAt.String()
+	}
+	return map[string]any{
+		"url": p.URL, "urls": plainValue(p.URLs.Get()), "title": p.Title, "titles": plainValue(p.Titles.Get()),
+		"kind": p.Kind, "origin": p.Origin, "parent_id": parent, "content_updated_at": date, "note": p.Note,
+	}
+}
+
+// pageWhat is a registry page's history description.
+func (s *Service) pageWhat(p models.Page) string {
+	return s.Phrase("golem15.seo::lang.history.what_page", map[string]string{"url": p.URL})
+}
+
+// RecordPage records a registry page's change in history.
+func (s *Service) RecordPage(ctx context.Context, tx *gorm.DB, before map[string]any, p models.Page, deleted bool) error {
+	c := Change{Kind: models.SubjectPage, SubjectID: p.IDText(), What: s.pageWhat(p), Before: before}
+	if !deleted {
+		c.After = PageValues(p)
+	}
+	return s.Record(ctx, tx, c)
+}
+
+// SavePage checks and writes a registry page inside tx, records it in
+// history, and fires the record event after commit, so the SEO tab's
+// address tracking and the sitemap follow as they do for an admin save.
+func (s *Service) SavePage(ctx context.Context, tx *gorm.DB, p *models.Page, before map[string]any) error {
+	problems, err := s.CheckPage(ctx, tx, *p)
+	if err != nil {
+		return err
+	}
+	if len(problems) > 0 {
+		return &ProblemsError{Problems: problems}
+	}
+	now := time.Now().UTC()
+	created := p.ID == 0
+	if created {
+		p.CreatedAt = &now
+	}
+	p.UpdatedAt = &now
+	if err := tx.Save(p).Error; err != nil {
+		return err
+	}
+	if err := s.RecordPage(ctx, tx, before, *p, false); err != nil {
+		return err
+	}
+	cabana.EmitRecordSaved(ctx, s.app, tx, pact.RecordSaved{Controller: PagesController, Model: PageModel, ID: p.IDText(), Created: created})
+	return nil
+}
+
+// DeletePage deletes a registry page inside tx, records it in history and
+// fires the record event after commit: the SEO plugin then deletes the
+// page's SEO entries and the sitemap drops its addresses.
+func (s *Service) DeletePage(ctx context.Context, tx *gorm.DB, p models.Page) error {
+	if err := tx.Delete(&models.Page{}, p.ID).Error; err != nil {
+		return err
+	}
+	if err := s.RecordPage(ctx, tx, PageValues(p), p, true); err != nil {
+		return err
+	}
+	cabana.EmitRecordDeleted(ctx, s.app, tx, pact.RecordDeleted{Controller: PagesController, Model: PageModel, ID: p.IDText()})
+	return nil
+}
+
+// RouteReport is the outcome of a route report: the paths inserted as new
+// registry pages, the known pages whose reported_at moved, and the paths
+// skipped (invalid, or owned by another content type).
+type RouteReport struct {
+	Inserted   int       `json:"inserted"`
+	Updated    int       `json:"updated"`
+	Skipped    int       `json:"skipped"`
+	ReportedAt time.Time `json:"reported_at"`
+}
+
+// ReportRoutes takes the frontend server's static routes (D-33, P33-20)
+// inside tx: a path no registry page has is inserted with origin reported,
+// kind page and a name from its last segment; a known page gets
+// reported_at; a path another content type owns, or that is no valid
+// address, is skipped. Nothing is ever deleted, and no value an editor set
+// is overwritten. The settings' routes_reported_at becomes the report's
+// time, so a reported page missing from it shows as missing. More than
+// MaxReportedRoutes paths is ErrTooManyRoutes.
+func (s *Service) ReportRoutes(ctx context.Context, tx *gorm.DB, paths []string) (RouteReport, error) {
+	now := time.Now().UTC()
+	out := RouteReport{ReportedAt: now}
+	if len(paths) > MaxReportedRoutes {
+		return out, ErrTooManyRoutes
+	}
+	settings, err := SettingsRow(tx)
+	if err != nil {
+		return out, err
+	}
+	opts := NormalizeOptionsOf(settings)
+	ctx = WithVia(ctx, models.ViaAuto)
+	seen := map[string]bool{}
+	for _, raw := range paths {
+		path := strings.TrimSpace(raw)
+		if len(path) > MaxRouteBytes || !ValidPagePath(path) {
+			out.Skipped++
+			continue
+		}
+		path = normalPagePath(path, opts)
+		if seen[path] {
+			continue
+		}
+		seen[path] = true
+		page, _, err := s.PageAt(ctx, tx, path)
+		if err != nil {
+			return out, err
+		}
+		if page != nil {
+			if err := tx.Model(&models.Page{}).Where("id = ?", page.ID).UpdateColumn("reported_at", now).Error; err != nil {
+				return out, err
+			}
+			out.Updated++
+			continue
+		}
+		owned, err := s.ownerOutsideRegistry(ctx, path)
+		if err != nil {
+			return out, err
+		}
+		if owned != nil {
+			out.Skipped++
+			continue
+		}
+		p := models.Page{
+			URL:        path,
+			URLs:       lagoon.Jsonable[map[string]string]{Data: map[string]string{}, Valid: true},
+			Title:      s.routeTitle(path),
+			Titles:     lagoon.Jsonable[map[string]string]{Data: map[string]string{}, Valid: true},
+			Kind:       models.PageKindPage,
+			Origin:     models.PageOriginReported,
+			ReportedAt: &now,
+			CreatedAt:  &now,
+			UpdatedAt:  &now,
+		}
+		res := tx.Clauses(clause.OnConflict{DoNothing: true}).Create(&p)
+		if res.Error != nil {
+			return out, res.Error
+		}
+		if res.RowsAffected == 0 || p.ID == 0 {
+			// A concurrent report inserted it first.
+			continue
+		}
+		out.Inserted++
+		if err := s.RecordPage(ctx, tx, nil, p, false); err != nil {
+			return out, err
+		}
+		cabana.EmitRecordSaved(ctx, s.app, tx, pact.RecordSaved{Controller: PagesController, Model: PageModel, ID: p.IDText(), Created: true})
+	}
+	if err := tx.Model(&models.Settings{}).Where("id = ?", 1).UpdateColumn("routes_reported_at", now).Error; err != nil {
+		return out, err
+	}
+	return out, nil
+}
+
+// ReportNoindex replaces the stored noindex paths (P33-58, the route
+// report's noindex key) inside tx with the valid entries of paths, each
+// normalised like a reported route (normalPagePath with the site's address
+// options; a percent-encoded path is stored decoded when its decoded form
+// is a valid address, the form the sitemap filter compares). An entry that
+// is no valid address, or that is a pattern (a "*" or a ":" parameter
+// segment), is skipped and counted. changed reports whether the stored set
+// differs from the one before; when it does, the sitemap is asked for one
+// debounced rebuild through contract.Rebuilder once tx commits (nothing
+// when no Rebuilder is published). More than MaxReportedRoutes paths is
+// ErrTooManyRoutes and stores nothing.
+func (s *Service) ReportNoindex(ctx context.Context, tx *gorm.DB, paths []string) (stored int, skipped int, changed bool, err error) {
+	if len(paths) > MaxReportedRoutes {
+		return 0, 0, false, ErrTooManyRoutes
+	}
+	settings, err := SettingsRow(tx)
+	if err != nil {
+		return 0, 0, false, err
+	}
+	opts := NormalizeOptionsOf(settings)
+	want := map[string]bool{}
+	for _, raw := range paths {
+		path, ok := noindexPath(raw, opts)
+		if !ok {
+			skipped++
+			continue
+		}
+		want[path] = true
+	}
+	var have []string
+	if err := tx.Model(&models.NoindexPath{}).Pluck("path", &have).Error; err != nil {
+		return 0, skipped, false, err
+	}
+	changed = len(have) != len(want)
+	for _, p := range have {
+		if !want[p] {
+			changed = true
+			break
+		}
+	}
+	if !changed {
+		return len(want), skipped, false, nil
+	}
+	if err := tx.Exec(`DELETE FROM golem15_seo_noindex_paths`).Error; err != nil {
+		return 0, skipped, false, err
+	}
+	now := time.Now().UTC()
+	rows := make([]models.NoindexPath, 0, len(want))
+	for p := range want {
+		rows = append(rows, models.NoindexPath{Path: p, ReportedAt: now})
+	}
+	sort.Slice(rows, func(i, j int) bool { return rows[i].Path < rows[j].Path })
+	if len(rows) > 0 {
+		if err := tx.Clauses(clause.OnConflict{DoNothing: true}).CreateInBatches(rows, 500).Error; err != nil {
+			return 0, skipped, false, err
+		}
+	}
+	lagoon.AfterCommit(ctx, tx, func(ctx context.Context, _ *gorm.DB) {
+		s.requestSitemapRebuild(ctx)
+	})
+	return len(rows), skipped, true, nil
+}
+
+// NoindexPathCount is the number of stored noindex paths.
+func NoindexPathCount(db *gorm.DB) (int, error) {
+	var n int64
+	err := db.Model(&models.NoindexPath{}).Count(&n).Error
+	return int(n), err
+}
+
+// noindexPath is a reported noindex entry as it is stored: trimmed,
+// decoded when its decoded form is a valid address, and normalised; ok is
+// false for an invalid address or a pattern.
+func noindexPath(raw string, opts NormalizeOptions) (string, bool) {
+	path := strings.TrimSpace(raw)
+	if len(path) > MaxRouteBytes || !ValidPagePath(path) || strings.Contains(path, "*") {
+		return "", false
+	}
+	for _, seg := range strings.Split(path, "/") {
+		if strings.HasPrefix(seg, ":") {
+			return "", false
+		}
+	}
+	if strings.Contains(path, "%") {
+		if dec, err := url.PathUnescape(path); err == nil && ValidPagePath(dec) {
+			path = dec
+		}
+	}
+	return normalPagePath(path, opts), true
+}
+
+// requestSitemapRebuild asks the sitemap plugin for one debounced rebuild
+// through contract.Rebuilder; nothing when none is published, a warning
+// when the request fails.
+func (s *Service) requestSitemapRebuild(ctx context.Context) {
+	if s == nil || s.app == nil {
+		return
+	}
+	rb, ok := s.app.Lookup[contract.Rebuilder]()
+	if !ok || rb == nil {
+		return
+	}
+	if err := rb.RequestRebuild(ctx); err != nil {
+		log.Warn("golem15.seo: request a sitemap rebuild", "error", err)
+	}
+}
+
+// routeTitle is a reported page's name: its last path segment with
+// hyphens and underscores as spaces and the first letter capitalised; the
+// home page is named pages.home_title.
+func (s *Service) routeTitle(path string) string {
+	seg := strings.Trim(path, "/")
+	if i := strings.LastIndex(seg, "/"); i >= 0 {
+		seg = seg[i+1:]
+	}
+	if decoded, err := url.PathUnescape(seg); err == nil {
+		seg = decoded
+	}
+	seg = strings.TrimSpace(strings.NewReplacer("-", " ", "_", " ").Replace(seg))
+	if seg == "" {
+		return s.Phrase("golem15.seo::lang.pages.home_title", nil)
+	}
+	r, size := utf8.DecodeRuneInString(seg)
+	return string(unicode.ToUpper(r)) + seg[size:]
+}
+
+// PageMissing reports whether p is a reported page that the newest route
+// report (at reported) did not name.
+func PageMissing(p models.Page, reported *time.Time) bool {
+	if p.Origin != models.PageOriginReported || reported == nil {
+		return false
+	}
+	return p.ReportedAt == nil || p.ReportedAt.Before(*reported)
+}
+
+// pageKinds reads the kind of each registry page id (as text).
+func pageKinds(db *gorm.DB, ids []string) (map[string]string, error) {
+	var nums []uint64
+	for _, id := range ids {
+		if n, err := strconv.ParseUint(id, 10, 64); err == nil {
+			nums = append(nums, n)
+		}
+	}
+	out := map[string]string{}
+	if len(nums) == 0 {
+		return out, nil
+	}
+	var rows []struct {
+		ID   uint64 `gorm:"column:id"`
+		Kind string `gorm:"column:kind"`
+	}
+	if err := db.Raw(`SELECT id, kind FROM golem15_seo_pages WHERE id IN ?`, nums).Scan(&rows).Error; err != nil {
+		return nil, err
+	}
+	for _, r := range rows {
+		out[strconv.FormatUint(r.ID, 10)] = r.Kind
+	}
+	return out, nil
+}

classes/registry_test.go

@@ -0,0 +1,110 @@
+package classes
+
+import (
+	"strings"
+	"testing"
+	"time"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+)
+
+// TestValidPagePath pins the registry address rule: a path from one /,
+// no domain, query, fragment, backslash or space, valid UTF-8 without
+// control characters, at most MaxRouteBytes.
+func TestValidPagePath(t *testing.T) {
+	for p, want := range map[string]bool{
+		"/": true, "/kontakt": true, "/trwają-prace": true, "/a/b/c": true, "/" + strings.Repeat("a", MaxRouteBytes-1): true,
+		"": false, "kontakt": false, "//evil.test": false, "/a?b": false, "/a#b": false, `/a\b`: false, "/a b": false,
+		"/a\x01": false, "/\xff": false, "https://example.test/a": false, "/" + strings.Repeat("a", MaxRouteBytes): false,
+	} {
+		if got := ValidPagePath(p); got != want {
+			t.Errorf("ValidPagePath(%q) = %v, want %v", p, got, want)
+		}
+	}
+}
+
+// TestNormalPagePath pins the site's address rules on a registry path: the
+// path trimmed, lower-cased and slashed as the settings say, an invalid
+// path returned trimmed.
+func TestNormalPagePath(t *testing.T) {
+	if got := normalPagePath(" /Kontakt/ ", NormalizeOptions{Lowercase: true, Slash: "remove"}); got != "/kontakt" {
+		t.Fatalf("lowercase, no slash = %q", got)
+	}
+	if got := normalPagePath("/Kontakt", NormalizeOptions{}); got != "/Kontakt" {
+		t.Fatalf("no rules = %q", got)
+	}
+	if got := normalPagePath(" kontakt ", NormalizeOptions{Lowercase: true}); got != "kontakt" {
+		t.Fatalf("invalid path = %q", got)
+	}
+}
+
+// TestApplyPageFields pins the scalar fields of a page write: kind and
+// note trimmed, the parent set and cleared, the content date parsed,
+// cleared by "" and refused when invalid; unset fields stay.
+func TestApplyPageFields(t *testing.T) {
+	kind, note, date, parent := " offer ", " n ", "2026-11-04", uint(7)
+	p := &models.Page{Kind: models.PageKindPage}
+	if problems := ApplyPageFields(p, PageWrite{Kind: &kind, Note: &note, ContentUpdatedAt: &date, ParentSet: true, ParentID: &parent}); len(problems) != 0 {
+		t.Fatalf("problems = %v", problems)
+	}
+	if p.Kind != "offer" || p.Note != "n" || p.ContentUpdatedAt == nil || p.ContentUpdatedAt.String() != "2026-11-04" || p.ParentID == nil || uint(*p.ParentID) != 7 {
+		t.Fatalf("page = %+v", p)
+	}
+	empty, bad, zero := "", "04.11.2026", uint(0)
+	if problems := ApplyPageFields(p, PageWrite{ContentUpdatedAt: &empty, ParentSet: true, ParentID: &zero}); len(problems) != 0 || p.ContentUpdatedAt != nil || p.ParentID != nil || p.Kind != "offer" {
+		t.Fatalf("cleared = %+v %v", p, problems)
+	}
+	if problems := ApplyPageFields(p, PageWrite{ContentUpdatedAt: &bad}); len(problems["content_updated_at"]) != 1 {
+		t.Fatalf("an invalid date: %v", problems)
+	}
+}
+
+// TestPageGroupAndMissing pins the sitemap group per kind (unknown kinds
+// with the pages) and the Not on the site flag: a reported page the
+// newest report did not name; never a manual page or before any report.
+func TestPageGroupAndMissing(t *testing.T) {
+	for kind, want := range map[string]string{models.PageKindPage: "pages", models.PageKindOffer: "offers", models.PageKindService: "services", models.PageKindLanding: "landing-pages", "other": "pages"} {
+		if got := PageGroup(kind); got != want {
+			t.Errorf("PageGroup(%q) = %q, want %q", kind, got, want)
+		}
+	}
+	now := time.Now().UTC()
+	older := now.Add(-time.Hour)
+	for _, tc := range []struct {
+		p        models.Page
+		reported *time.Time
+		want     bool
+	}{
+		{models.Page{Origin: models.PageOriginReported, ReportedAt: &older}, &now, true},
+		{models.Page{Origin: models.PageOriginReported}, &now, true},
+		{models.Page{Origin: models.PageOriginReported, ReportedAt: &now}, &now, false},
+		{models.Page{Origin: models.PageOriginManual, ReportedAt: &older}, &now, false},
+		{models.Page{Origin: models.PageOriginReported, ReportedAt: &older}, nil, false},
+	} {
+		if got := PageMissing(tc.p, tc.reported); got != tc.want {
+			t.Errorf("PageMissing(%+v) = %v, want %v", tc.p, got, tc.want)
+		}
+	}
+}
+
+// TestPageBeforeSaveUTC pins the registry page's timestamps in UTC
+// whatever the process's zone: a new row gets both, a saved row keeps its
+// created_at.
+func TestPageBeforeSaveUTC(t *testing.T) {
+	local := time.Local
+	time.Local = time.FixedZone("UTC+3", 3*3600)
+	defer func() { time.Local = local }()
+	p := &models.Page{}
+	if err := p.BeforeSave(nil); err != nil {
+		t.Fatal(err)
+	}
+	if p.CreatedAt == nil || p.UpdatedAt == nil || p.CreatedAt.Location() != time.UTC || p.UpdatedAt.Location() != time.UTC {
+		t.Fatalf("stamps = %v %v", p.CreatedAt, p.UpdatedAt)
+	}
+	created := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
+	p.CreatedAt = &created
+	_ = p.BeforeSave(nil)
+	if !p.CreatedAt.Equal(created) || !p.UpdatedAt.After(created) {
+		t.Fatalf("a saved row = %v %v", p.CreatedAt, p.UpdatedAt)
+	}
+}

classes/resolver.go

@@ -5,9 +5,11 @@ import (
 	"errors"
 	"net/url"
 	"sort"
+	"strconv"
 	"strings"
 	"unicode/utf8"
 
+	"git.golem15.com/golem15/sm-seo-plugin/classes/schema"
 	"git.golem15.com/golem15/sm-seo-plugin/models"
 	"git.golem15.com/golem15/summercms/modules/cabana"
 	"git.golem15.com/golem15/summercms/modules/pact"
@@ -62,10 +64,33 @@ type Result struct {
 	Locale   string `json:"locale"`
 	Staging  bool   `json:"staging"`
 	SEO      *SEO   `json:"seo"`
+	// Outbound is how the layer renders links to other sites (P33-16),
+	// present on ok answers.
+	Outbound *Outbound `json:"outbound,omitempty"`
+	// Autolinks are the auto-link phrases of the page's locale without
+	// those linking to the page itself, longest first, and AutolinkMax the
+	// most the layer links in one text (D-82); present on ok answers, no
+	// phrases when AutolinkMax is 0.
+	Autolinks   []AutolinkLink `json:"autolinks,omitempty"`
+	AutolinkMax *int           `json:"autolink_max,omitempty"`
+	// HTMLLang is the page's <html lang> (P33-21); Alternates its hreflang
+	// list, the page itself and x-default included (D-55 to D-57); JSONLD
+	// its one @graph (D-43); Breadcrumbs its trail, the same items as the
+	// BreadcrumbList node (D-86). All are set on ok answers only.
+	HTMLLang    string         `json:"html_lang,omitempty"`
+	Alternates  []Alternate    `json:"alternates,omitempty"`
+	JSONLD      map[string]any `json:"jsonld,omitempty"`
+	Breadcrumbs []schema.Crumb `json:"breadcrumbs,omitempty"`
+	// Pagination is present when the request carries a page (D-40 to
+	// D-42): a not_found with invalid true for a page value that is not a
+	// page number.
+	Pagination *Pagination       `json:"pagination,omitempty"`
 	Sources    map[string]string `json:"-"`
 	// TypeKey and RecordID name the record behind the URL, "" when none.
 	TypeKey  string `json:"-"`
 	RecordID string `json:"-"`
+	// Merged names the generated values custom JSON-LD replaced.
+	Merged []schema.Merged `json:"-"`
 }
 
 // SEO is the head metadata of a resolved page.
@@ -80,12 +105,45 @@ type SEO struct {
 	Verification []MetaTag  `json:"verification"`
 }
 
+// Outbound holds the settings' Links rules for links to other sites
+// (D-83): open them in a new tab, and the domains (subdomains included)
+// whose links always get nofollow or sponsored.
+type Outbound struct {
+	NewTab           bool     `json:"new_tab"`
+	NofollowDomains  []string `json:"nofollow_domains"`
+	SponsoredDomains []string `json:"sponsored_domains"`
+}
+
+// OutboundOf is the Links rules of the settings.
+func OutboundOf(s models.Settings) *Outbound {
+	return &Outbound{
+		NewTab:           s.LinksNewTab,
+		NofollowDomains:  validDomains(s.NofollowDomains),
+		SponsoredDomains: validDomains(s.SponsoredDomains),
+	}
+}
+
+// validDomains lists the valid domains of a stored list.
+func validDomains(list string) []string {
+	out := []string{}
+	for _, d := range models.DomainLines(list) {
+		if models.ValidDomain(d) {
+			out = append(out, d)
+		}
+	}
+	return out
+}
+
 // OpenGraph holds the og: values.
 type OpenGraph struct {
 	Title       string `json:"title"`
 	Description string `json:"description"`
 	Image       string `json:"image"`
 	URL         string `json:"url"`
+	// Locale is og:locale (pl_PL) and LocaleAlternates the og:locale of the
+	// record's other translated locales (D-55), set on ok answers.
+	Locale           string   `json:"locale"`
+	LocaleAlternates []string `json:"locale_alternates"`
 }
 
 // TwitterTag holds the twitter: values.
@@ -103,6 +161,13 @@ type MetaTag struct {
 // Resolver answers the effective SEO of a URL.
 type Resolver struct {
 	Service *Service
+	// production computes the values as production serves them: the
+	// non-production lock (D-22) is not applied. The sitemap filter decides
+	// on these, so staging lists the same addresses as production.
+	production bool
+	// schema replaces the stored seo_schema value of the page when set
+	// (the graph preview of unsaved changes and the MCP graph tool).
+	schema *string
 }
 
 // Resolver returns the service's resolver.
@@ -393,6 +458,24 @@ func (r *Resolver) Resolve(ctx context.Context, rawURL string) (Result, error) {
 	}
 	npath, nquery := norm.Path, norm.RawQuery
 
+	// Pagination (D-40 to D-42): the page of the URL in either form. A
+	// path-form suffix on a path a record owns is not a page.
+	pages := svc.PageConfigOf(ctx, db, settings)
+	nvalues, _ := url.ParseQuery(nquery)
+	pr := parsePage(npath, nvalues, pages)
+	if pr.carried && pr.form == models.PageFormPath {
+		if owner, err := r.Strict(ctx, npath, opts); err != nil {
+			return Result{}, err
+		} else if owner.Key != "" {
+			pr = pageRequest{base: npath}
+		}
+	}
+	if pr.invalid {
+		res := Result{Action: ActionNotFound, Status: 404, URL: self, Staging: svc.Staging(), Sources: map[string]string{}}
+		res.Pagination = &Pagination{Form: pages.Form, Param: PageParam, Segment: pr.segment, Invalid: true}
+		return res, nil
+	}
+
 	// 2. Redirects on the normalised path.
 	set, err := LoadRedirectSet(db)
 	if err != nil {
@@ -423,23 +506,67 @@ func (r *Resolver) Resolve(ctx context.Context, rawURL string) (Result, error) {
 		return res, nil
 	}
 
-	// 3. Strict URLs.
+	// 3. Strict URLs, on the path without its page.
-	st, err := r.Strict(ctx, npath, opts)
+	base := npath
+	if pr.carried {
+		base = pr.base
+	}
+	st, err := r.Strict(ctx, base, opts)
 	if err != nil {
 		return Result{}, err
 	}
+	move := changed
 	if st.Key != "" && st.Mismatch {
 		if st.Canonical == "" {
 			return Result{Action: ActionNotFound, Status: 404, URL: self, Staging: svc.Staging(), Sources: map[string]string{}}, nil
 		}
-		return r.moved(self, origin+joinQuery(escapePath(st.Canonical), nquery)), nil
+		base, move = st.Canonical, true
 	}
-	if changed {
+	if !pr.carried {
-		return r.moved(self, origin+joinQuery(escapePath(npath), nquery)), nil
+		if move {
+			return r.moved(self, origin+joinQuery(escapePath(base), nquery)), nil
+		}
+		// 4. The effective SEO.
+		return r.resolveAt(ctx, db, settings, npath, query, nquery, st.Key, st.Match, st.Record, false, nil)
 	}
 
-	// 4. The effective SEO.
+	// A page: page 1 and the other form 301 to the configured address,
-	return r.resolveAt(ctx, db, settings, npath, query, nquery, st.Key, st.Match, st.Record, false)
+	// together with normalisation and strict URLs in one hop.
+	locale := ""
+	if st.Key != "" && st.Match != nil {
+		locale = st.Match.Locale
+	} else {
+		locale = svc.localeOfPath(ctx, db, base)
+	}
+	segment := pages.Segment(locale)
+	canonical := pr.page >= 2 && pr.exact && pr.form == pages.Form && (pr.form == models.PageFormQuery || pr.segment == segment)
+	if move || !canonical {
+		return r.moved(self, origin+pageLocation(base, pr.rest, pr.page, pages, locale, opts)), nil
+	}
+	pg := &pageAt{form: pages.Form, segment: segment, page: pr.page}
+	res, err := r.resolveAt(ctx, db, settings, base, query, nquery, st.Key, st.Match, st.Record, false, pg)
+	if err != nil {
+		return Result{}, err
+	}
+	// The hreflang alternates of a page point at the same page of each
+	// translation.
+	def := svc.DefaultLocale(ctx, db)
+	for i, a := range res.Alternates {
+		u, err := url.Parse(a.URL)
+		if err != nil || !strings.HasPrefix(a.URL, site) {
+			continue
+		}
+		code := def
+		if a.Hreflang != "x-default" {
+			code = svc.localeOfTag(ctx, db, a.Hreflang)
+		}
+		res.Alternates[i].URL = site + pageLocation(u.Path, url.Values{}, pr.page, pages, code, opts)
+	}
+	res.Pagination = &Pagination{Form: pages.Form, Param: PageParam, Segment: segment, Page: pr.page}
+	if pages.Form == models.PageFormQuery {
+		res.Pagination.Segment = ""
+	}
+	return res, nil
 }
 
 // moved is a 301 answer to location.
@@ -486,17 +613,29 @@ func (r *Resolver) ResolveRecord(ctx context.Context, key, id, locale string) (R
 	if err != nil {
 		return Result{}, false, err
 	}
-	res, err := r.resolveAt(ctx, db, settings, target, query, rawQueryOf(path), key, &pact.ContentMatchResult{ID: id, Locale: locale, Exact: true}, rec, true)
+	res, err := r.resolveAt(ctx, db, settings, target, query, rawQueryOf(path), key, &pact.ContentMatchResult{ID: id, Locale: locale, Exact: true}, rec, true, nil)
 	return res, err == nil, err
 }
 
 // resolveAt computes the answer for a path whose owner lookup is done:
-// key, match and rec name the record behind it (key "" when none).
+// key, match and rec name the record behind it (key "" when none). pg is
-func (r *Resolver) resolveAt(ctx context.Context, db *gorm.DB, settings models.Settings, path string, query url.Values, rawQuery, key string, match *pact.ContentMatchResult, rec *pact.ContentRecord, force bool) (Result, error) {
+// the page of a paginated address (page 2 and later), nil otherwise: the
+// page stays in the URL and the self-referencing canonical, and the title
+// and description get the page suffixes (D-21, D-41).
+func (r *Resolver) resolveAt(ctx context.Context, db *gorm.DB, settings models.Settings, path string, query url.Values, rawQuery, key string, match *pact.ContentMatchResult, rec *pact.ContentRecord, force bool, pg *pageAt) (Result, error) {
 	svc := r.Service
 	site := svc.SiteURL()
 	kept, filtered := splitQuery(settings.Params(), path, query)
-	self := site + path
+	selfPath := path
+	if pg != nil {
+		if pg.form == models.PageFormPath {
+			selfPath = pagedPath(path, pg.segment, pg.page)
+			kept.Del(PageParam)
+		} else {
+			kept.Set(PageParam, strconv.Itoa(pg.page))
+		}
+	}
+	self := site + selfPath
 	if q := encodeQuery(kept); q != "" {
 		self += "?" + q
 	}
@@ -541,15 +680,18 @@ func (r *Resolver) resolveAt(ctx context.Context, db *gorm.DB, settings models.S
 	// the content title > content title > site name. The home page without
 	// a record uses the home template.
 	var titleTemplate candidate
+	titleTpl := ""
 	switch {
 	case key != "":
-		tpl := typeSetting.TitleTemplate
+		titleTpl = typeSetting.TitleTemplate
-		if strings.TrimSpace(tpl) == "" {
+		if strings.TrimSpace(titleTpl) == "" {
-			tpl = settings.TitleTemplate
+			titleTpl = settings.TitleTemplate
 		}
-		titleTemplate = candidate{RenderTemplate(tpl, vars), SourceTemplate}
 	case home:
-		titleTemplate = candidate{RenderTemplate(settings.HomeTemplate, vars), SourceTemplate}
+		titleTpl = settings.HomeTemplate
+	}
+	if titleTpl != "" {
+		titleTemplate = candidate{RenderTemplate(titleTpl, vars), SourceTemplate}
 	}
 	seo.Title, src["title"] = pick(
 		candidate{ruleValue(rules, "title"), SourceRule},
@@ -572,12 +714,31 @@ func (r *Resolver) resolveAt(ctx context.Context, db *gorm.DB, settings models.S
 		candidate{defaultDescription, SourceGlobal},
 	)
 
-	// Canonical: rule > record > self.
+	// Page 2 and later (D-41): the title suffix goes after the page title,
+	// inside the template when the title comes from one, and the
+	// description suffix at the end; %page% is filled, %pages% is left for
+	// the layer (P33-24).
+	if pg != nil {
+		suffix := fillPage(svc.pageSuffix(ctx, db, settings, locale, true), pg.page)
+		if src["title"] == SourceTemplate && strings.Contains(titleTpl, PlaceholderTitle) {
+			paged := vars
+			paged.Title = strings.TrimSpace(paged.Title + " " + suffix)
+			seo.Title = RenderTemplate(titleTpl, paged)
+		} else {
+			seo.Title = strings.TrimSpace(seo.Title + " " + suffix)
+		}
+		if desc := fillPage(svc.pageSuffix(ctx, db, settings, locale, false), pg.page); desc != "" {
+			seo.Description = strings.TrimSpace(seo.Description + " " + desc)
+		}
+	}
+
+	// Canonical: rule > record > self; page 2 and later is its own
+	// canonical (D-21).
 	canonical, canonicalSource := pick(
 		candidate{ruleValue(rules, "canonical"), SourceRule},
 		candidate{entry.Canonical, SourceRecord},
 	)
-	if canonical == "" {
+	if canonical == "" || pg != nil {
 		canonical, canonicalSource = self, SourceDefault
 	} else {
 		canonical = AbsoluteURL(site, canonical)
@@ -600,7 +761,7 @@ func (r *Resolver) resolveAt(ctx context.Context, db *gorm.DB, settings models.S
 		layers = append(layers, robotsLayer{typeSetting.Robots, SourceTemplate})
 	}
 	robots, robotsSource := resolveRobots(layers)
-	if res.Staging {
+	if res.Staging && !r.production {
 		robots, robotsSource = Robots{Index: "noindex", Follow: "nofollow"}, SourceDefault
 	}
 	seo.Robots, src["robots"] = robots.Header(), robotsSource
@@ -642,6 +803,25 @@ func (r *Resolver) resolveAt(ctx context.Context, db *gorm.DB, settings models.S
 	}
 
 	res.Action, res.Status, res.Locale, res.SEO = ActionOK, 200, locale, seo
+	res.Outbound = OutboundOf(settings)
+	// The sitemap filter (production) decides on robots and canonical
+	// only; it skips the auto-links and the structured data.
+	if !r.production {
+		most := min(models.MaxAutolinkMax, max(models.MinAutolinkMax, settings.AutolinkMax))
+		res.AutolinkMax = &most
+		if most > 0 {
+			if res.Autolinks, err = svc.ForPage(ctx, db, locale, path); err != nil {
+				return Result{}, err
+			}
+		}
+		if err := svc.structuredData(ctx, db, settings, &res, structuredIn{
+			key: key, id: res.RecordID, rec: rec, entry: entry, content: content,
+			locale: locale, path: path, url: self, home: home, siteName: svc.SiteName(ctx, settings, locale), seo: seo,
+			schemaValue: r.schema, sources: src,
+		}); err != nil {
+			return Result{}, err
+		}
+	}
 	return res, nil
 }
 

classes/robots.go

@@ -11,8 +11,8 @@ import (
 )
 
 // RobotsSitemapsConfigKey lists the sitemap addresses robots.txt names,
-// one Sitemap line each, absolute on the site URL (P-23); default
+// one Sitemap line each, absolute on the site URL (P-23). Unset, robots.txt
-// ["/sitemap.xml"].
+// names the sitemap plugin's index (D-22, SitemapLines).
 const RobotsSitemapsConfigKey = "golem15.seo.robots_sitemaps"
 
 // MaxRobotsBytes is the largest robots.txt the editor stores: Google
@@ -29,8 +29,49 @@ const (
 	FallbackRobots = "User-agent: *\nDisallow:\n"
 )
 
+// SitemapLines are the addresses of robots.txt's Sitemap lines (D-22):
+// golem15.seo.robots_sitemaps when it is set, else the sitemap index of
+// the sitemap plugin's contract.Listing (its IndexURL), else /sitemap.xml
+// on the site URL. The per-type files are never listed: the index names
+// them.
+func (s *Service) SitemapLines(ctx context.Context) []string {
+	if s.robotsSitemapsSet() {
+		return s.RobotsSitemaps()
+	}
+	if l := s.listing(); l != nil {
+		if st, err := l.Settings(ctx); err == nil && strings.TrimSpace(st.IndexURL) != "" {
+			return []string{strings.TrimSpace(st.IndexURL)}
+		} else if err != nil {
+			log.Warn("golem15.seo: sitemap index address", "error", err)
+		}
+	}
+	return s.RobotsSitemaps()
+}
+
+// robotsSitemapsSet reports whether golem15.seo.robots_sitemaps is set to
+// a non-empty list.
+func (s *Service) robotsSitemapsSet() bool {
+	cfg := s.Config()
+	if cfg == nil {
+		return false
+	}
+	raw, ok := cfg.Lookup(RobotsSitemapsConfigKey)
+	if !ok || raw == nil {
+		return false
+	}
+	switch v := raw.(type) {
+	case []any:
+		return len(v) > 0
+	case []string:
+		return len(v) > 0
+	case string:
+		return strings.TrimSpace(v) != ""
+	}
+	return false
+}
+
 // RobotsSitemaps returns the configured sitemap addresses, absolute on
-// the site URL.
+// the site URL; /sitemap.xml when none is configured.
 func (s *Service) RobotsSitemaps() []string {
 	var list []string
 	if cfg := s.Config(); cfg != nil {
@@ -128,7 +169,7 @@ func (s *Service) ServedFile(ctx context.Context, production bool) (string, erro
 	if err != nil {
 		return "", err
 	}
-	body := ComposeRobots(row.Content, row.Bots, s.RobotsSitemaps())
+	body := ComposeRobots(row.Content, row.Bots, s.SitemapLines(ctx))
 	if s != nil {
 		s.robots.Store(&body)
 	}
@@ -164,7 +205,12 @@ type RobotsTest struct {
 // grobotstxt, Google's matcher (D-22, D-99): longest match, Allow wins a
 // tie, * and $.
 func (s *Service) TestRobots(content string, bots models.BotSwitches, agent, path string) RobotsTest {
-	file := ComposeRobots(content, bots, s.RobotsSitemaps())
+	return s.testRobotsFile(ComposeRobots(content, bots, s.SitemapLines(context.Background())), content, agent, path)
+}
+
+// testRobotsFile is TestRobots on a composed file; content is the editor's
+// part of it (the generated bot rules follow it).
+func (s *Service) testRobotsFile(file, content, agent, path string) RobotsTest {
 	uri := AbsoluteURL(s.SiteURL(), path)
 	if s.SiteURL() == "" {
 		uri = "http://localhost" + path
@@ -228,8 +274,12 @@ func (s *Service) BlockedNoindex(db *gorm.DB, content string, bots models.BotSwi
 // the production file composed from content and bots.
 func (s *Service) blockedNoindex(urls []string, content string, bots models.BotSwitches) []string {
 	var out []string
+	if len(urls) == 0 {
+		return out
+	}
+	file := ComposeRobots(content, bots, s.SitemapLines(context.Background()))
 	for _, u := range urls {
-		if !s.TestRobots(content, bots, "Googlebot", u).Allowed {
+		if !s.testRobotsFile(file, content, "Googlebot", u).Allowed {
 			out = append(out, u)
 		}
 	}

classes/robots_write.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"errors"
 	"strconv"
+	"strings"
 	"time"
 
 	"git.golem15.com/golem15/sm-seo-plugin/models"
@@ -42,7 +43,7 @@ func RobotsValues(r models.Robots) map[string]any {
 	for _, b := range Bots {
 		bots[b.Token] = r.Bots.Allowed(b.Token)
 	}
-	return map[string]any{"content": r.Content, "bots": bots}
+	return map[string]any{"content": r.Content, "bots": bots, "llms_enabled": r.LlmsEnabled, "llms_intro": r.LlmsIntro}
 }
 
 // RobotsWrite is one change of robots.txt: the editor's text when Content
@@ -50,6 +51,8 @@ func RobotsValues(r models.Robots) map[string]any {
 type RobotsWrite struct {
 	Content     *string
 	Bots        map[string]bool
+	LlmsEnabled *bool
+	LlmsIntro   *string
 }
 
 // SaveRobots writes robots.txt inside tx and records it in history
@@ -87,10 +90,16 @@ func (s *Service) SaveRobots(ctx context.Context, tx *gorm.DB, in RobotsWrite) (
 		bots[b.Token] = on
 	}
 	row.Bots = bots
+	if in.LlmsEnabled != nil {
+		row.LlmsEnabled = *in.LlmsEnabled
+	}
+	if in.LlmsIntro != nil {
+		row.LlmsIntro = strings.TrimSpace(*in.LlmsIntro)
+	}
 	now := time.Now().UTC()
 	row.UpdatedAt = &now
 	if err := tx.Model(&models.Robots{}).Where("id = ?", row.ID).Updates(map[string]any{
-		"content": row.Content, "bots": row.Bots, "updated_at": now,
+		"content": row.Content, "bots": row.Bots, "llms_enabled": row.LlmsEnabled, "llms_intro": row.LlmsIntro, "updated_at": now,
 	}).Error; err != nil {
 		return models.Robots{}, nil, err
 	}

classes/schema/custom.go

@@ -0,0 +1,285 @@
+package schema
+
+import (
+	"bytes"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"regexp"
+	"sort"
+	"strings"
+)
+
+// The caps of custom JSON-LD (UI-SPEC C17, T-33-24).
+const (
+	MaxCustomBytes = 16 * 1024
+	MaxCustomDepth = 8
+)
+
+// The codes of a refused value; each is a phrase key under schema.
+const (
+	CodeInvalidJSON  = "invalid_json"
+	CodeNotObject    = "not_object"
+	CodeTooLarge     = "too_large"
+	CodeTooDeep      = "too_deep"
+	CodeScriptTag    = "script_tag"
+	CodeTypeUnknown  = "type_unknown"
+	CodeFieldInvalid = "field_invalid"
+)
+
+// Error is a refused structured data value: a code, and for invalid JSON
+// the parser's message with its line and column; for a field the field
+// name.
+type Error struct {
+	Code   string
+	Detail string
+	Field  string
+}
+
+func (e *Error) Error() string {
+	if e.Detail != "" {
+		return "schema: " + e.Code + ": " + e.Detail
+	}
+	return "schema: " + e.Code
+}
+
+// Value is the stored seo_schema value (UI-SPEC C17): the editor's type
+// ("" for the kind's default), the Service fields and the custom JSON-LD.
+type Value struct {
+	Type   string        `json:"type"`
+	Fields ServiceFields `json:"fields"`
+	Custom string        `json:"custom"`
+}
+
+// ParseValue reads a stored or submitted value; "" is the zero value.
+func ParseValue(raw string) (Value, error) {
+	var v Value
+	if strings.TrimSpace(raw) == "" {
+		return v, nil
+	}
+	if err := json.Unmarshal([]byte(raw), &v); err != nil {
+		return Value{}, &Error{Code: CodeInvalidJSON, Detail: err.Error()}
+	}
+	return v, nil
+}
+
+// scriptTag finds a literal <script in any letter case.
+var scriptTag = regexp.MustCompile(`(?i)<\s*/?\s*script`)
+
+// currencyPattern is an ISO 4217 code.
+var currencyPattern = regexp.MustCompile(`^[A-Z]{3}$`)
+
+// Normalize checks a submitted value for kind and returns it in its
+// stored form: the type checked against the kind's list, Service fields
+// kept only for Service (price and currency checked), the custom JSON-LD
+// decoded with the C17 caps and re-serialised, never echoed as typed. The
+// empty value stays "".
+func Normalize(kind, variant, raw string) (string, Value, error) {
+	v, err := ParseValue(raw)
+	if err != nil {
+		return "", Value{}, err
+	}
+	v.Type = strings.TrimSpace(v.Type)
+	if !Allowed(kind, v.Type) {
+		return "", Value{}, &Error{Code: CodeTypeUnknown}
+	}
+	if EffectiveType(kind, variant, v.Type) == TypeService {
+		f := v.Fields
+		f.ServiceType, f.AreaServed = strings.TrimSpace(f.ServiceType), strings.TrimSpace(f.AreaServed)
+		f.Price, f.Currency = strings.TrimSpace(f.Price), strings.ToUpper(strings.TrimSpace(f.Currency))
+		if f.Price != "" {
+			if NormalizePrice(f.Price) == "" {
+				return "", Value{}, &Error{Code: CodeFieldInvalid, Field: "price"}
+			}
+			f.Price = NormalizePrice(f.Price)
+		}
+		if f.Currency != "" && !currencyPattern.MatchString(f.Currency) {
+			return "", Value{}, &Error{Code: CodeFieldInvalid, Field: "currency"}
+		}
+		v.Fields = f
+	} else {
+		v.Fields = ServiceFields{}
+	}
+	v.Custom = strings.TrimSpace(v.Custom)
+	if v.Custom != "" {
+		nodes, err := DecodeCustom(v.Custom)
+		if err != nil {
+			return "", Value{}, err
+		}
+		var out any = nodes
+		if len(nodes) == 1 {
+			out = nodes[0]
+		}
+		pretty, err := json.MarshalIndent(out, "", "  ")
+		if err != nil {
+			return "", Value{}, &Error{Code: CodeInvalidJSON, Detail: err.Error()}
+		}
+		v.Custom = string(pretty)
+	}
+	if v.Type == "" && v.Custom == "" && v.Fields == (ServiceFields{}) {
+		return "", v, nil
+	}
+	stored, err := json.Marshal(v)
+	if err != nil {
+		return "", Value{}, err
+	}
+	return string(stored), v, nil
+}
+
+// position turns a byte offset of text into its line and column.
+func position(text string, offset int64) (int, int) {
+	if offset > int64(len(text)) {
+		offset = int64(len(text))
+	}
+	line, col := 1, 1
+	for _, r := range text[:offset] {
+		if r == '\n' {
+			line, col = line+1, 1
+		} else {
+			col++
+		}
+	}
+	return line, col
+}
+
+// depth is the nesting depth of a decoded JSON value (an object or a list
+// counts one level).
+func depth(v any) int {
+	switch x := v.(type) {
+	case map[string]any:
+		d := 0
+		for _, c := range x {
+			d = max(d, depth(c))
+		}
+		return d + 1
+	case []any:
+		d := 0
+		for _, c := range x {
+			d = max(d, depth(c))
+		}
+		return d + 1
+	}
+	return 0
+}
+
+// DecodeCustom decodes custom JSON-LD (UI-SPEC C17): an object or a list
+// of objects, at most 16 KB and 8 levels deep, without a literal <script.
+// Invalid JSON is refused with the parser's line and column.
+func DecodeCustom(text string) ([]map[string]any, error) {
+	text = strings.TrimSpace(text)
+	if text == "" {
+		return nil, nil
+	}
+	if len(text) > MaxCustomBytes {
+		return nil, &Error{Code: CodeTooLarge}
+	}
+	if scriptTag.MatchString(text) {
+		return nil, &Error{Code: CodeScriptTag}
+	}
+	dec := json.NewDecoder(strings.NewReader(text))
+	dec.UseNumber()
+	var v any
+	if err := dec.Decode(&v); err != nil {
+		var syn *json.SyntaxError
+		offset := dec.InputOffset()
+		if errors.As(err, &syn) {
+			offset = syn.Offset
+		}
+		line, col := position(text, offset)
+		msg := err.Error()
+		if errors.Is(err, io.ErrUnexpectedEOF) || errors.Is(err, io.EOF) {
+			msg = "unexpected end of JSON input"
+		}
+		return nil, &Error{Code: CodeInvalidJSON, Detail: fmt.Sprintf("line %d, column %d: %s", line, col, msg)}
+	}
+	if dec.More() {
+		line, col := position(text, dec.InputOffset())
+		return nil, &Error{Code: CodeInvalidJSON, Detail: fmt.Sprintf("line %d, column %d: unexpected data after the JSON value", line, col)}
+	}
+	if depth(v) > MaxCustomDepth {
+		return nil, &Error{Code: CodeTooDeep}
+	}
+	var out []map[string]any
+	switch x := v.(type) {
+	case map[string]any:
+		out = []map[string]any{x}
+	case []any:
+		if len(x) == 0 {
+			return nil, &Error{Code: CodeNotObject}
+		}
+		for _, item := range x {
+			m, ok := item.(map[string]any)
+			if !ok {
+				return nil, &Error{Code: CodeNotObject}
+			}
+			out = append(out, m)
+		}
+	default:
+		return nil, &Error{Code: CodeNotObject}
+	}
+	for _, n := range out {
+		delete(n, "@context")
+	}
+	return out, nil
+}
+
+// Merged is one generated node a custom node was merged into, with the
+// generated properties the custom values replace.
+type Merged struct {
+	ID    string
+	Type  string
+	Props []string
+}
+
+// Merge adds custom nodes to a graph (UI-SPEC C17): a node whose @id is a
+// generated node's is merged into it, custom values first; any other node
+// is appended. It answers the merges that replaced generated values.
+func Merge(graph map[string]any, custom []map[string]any) (map[string]any, []Merged) {
+	nodes := Nodes(graph)
+	byID := map[string]map[string]any{}
+	for _, n := range nodes {
+		if id, _ := n["@id"].(string); id != "" {
+			byID[id] = n
+		}
+	}
+	var merged []Merged
+	for _, c := range custom {
+		id, _ := c["@id"].(string)
+		target, ok := byID[id]
+		if id == "" || !ok {
+			nodes = append(nodes, c)
+			if id != "" {
+				byID[id] = c
+			}
+			continue
+		}
+		var props []string
+		for k, v := range c {
+			if k == "@id" {
+				continue
+			}
+			if old, had := target[k]; had && !sameJSON(old, v) {
+				props = append(props, k)
+			}
+			target[k] = v
+		}
+		if len(props) > 0 {
+			sort.Strings(props)
+			merged = append(merged, Merged{ID: id, Type: NodeType(target), Props: props})
+		}
+	}
+	list := make([]any, len(nodes))
+	for i, n := range nodes {
+		list[i] = n
+	}
+	out := map[string]any{"@context": Context, "@graph": list}
+	return out, merged
+}
+
+// sameJSON reports whether two values encode alike.
+func sameJSON(a, b any) bool {
+	x, err1 := json.Marshal(a)
+	y, err2 := json.Marshal(b)
+	return err1 == nil && err2 == nil && bytes.Equal(x, y)
+}

classes/schema/custom_test.go

@@ -0,0 +1,148 @@
+package schema
+
+import (
+	"encoding/json"
+	"errors"
+	"strings"
+	"testing"
+)
+
+func codeOf(err error) string {
+	var e *Error
+	if errors.As(err, &e) {
+		return e.Code
+	}
+	return ""
+}
+
+// TestDecodeCustom pins the C17 caps: an object or a non-empty list of
+// objects, 16 KB, 8 levels, no <script in any case or spacing, the
+// parser's line and column for invalid JSON, nothing after the value, and
+// @context dropped.
+func TestDecodeCustom(t *testing.T) {
+	nodes, err := DecodeCustom(` {"@context":"https://schema.org","@type":"FAQPage","n":1.50} `)
+	if err != nil || len(nodes) != 1 || nodes[0]["@context"] != nil || nodes[0]["n"].(json.Number).String() != "1.50" {
+		t.Fatalf("object = %v err=%v", nodes, err)
+	}
+	if nodes, err := DecodeCustom(`[{"@type":"A"},{"@type":"B"}]`); err != nil || len(nodes) != 2 {
+		t.Fatalf("list = %v err=%v", nodes, err)
+	}
+	if nodes, err := DecodeCustom("  "); err != nil || nodes != nil {
+		t.Fatalf("blank = %v err=%v", nodes, err)
+	}
+	deep := strings.Repeat(`{"a":`, 8) + "1" + strings.Repeat("}", 8)
+	if _, err := DecodeCustom(deep); err != nil {
+		t.Fatalf("8 levels must pass: %v", err)
+	}
+	for name, tc := range map[string]struct{ text, code string }{
+		"too large":   {`{"a":"` + strings.Repeat("x", MaxCustomBytes) + `"}`, CodeTooLarge},
+		"too deep":    {strings.Repeat(`{"a":`, 9) + "1" + strings.Repeat("}", 9), CodeTooDeep},
+		"script":      {`{"a":"</script><script>x"}`, CodeScriptTag},
+		"script case": {`{"a":"< SCRIPT src=x>"}`, CodeScriptTag},
+		"string":      {`"x"`, CodeNotObject},
+		"number":      {`5`, CodeNotObject},
+		"empty list":  {`[]`, CodeNotObject},
+		"mixed list":  {`[{"a":1},2]`, CodeNotObject},
+		"syntax":      {"{\n  \"a\": 1,\n  \"b\" 2\n}", CodeInvalidJSON},
+		"truncated":   {`{"a":`, CodeInvalidJSON},
+		"trailing":    {`{"a":1} {"b":2}`, CodeInvalidJSON},
+	} {
+		if _, err := DecodeCustom(tc.text); codeOf(err) != tc.code {
+			t.Errorf("%s: err %v, want code %s", name, err, tc.code)
+		}
+	}
+	_, err = DecodeCustom("{\n  \"a\": 1,\n  \"b\" 2\n}")
+	if e := err.(*Error); !strings.HasPrefix(e.Detail, "line 3, column ") {
+		t.Fatalf("syntax detail = %q, want line 3", e.Detail)
+	}
+	if _, err := DecodeCustom(`{"a":`); !strings.Contains(err.Error(), "unexpected end of JSON input") {
+		t.Fatalf("truncated = %v", err)
+	}
+}
+
+// TestNormalize pins the stored form of the seo_schema value: the type
+// checked against the kind, Service fields only for Service (price and
+// currency normalised and checked), custom JSON-LD re-serialised, and the
+// empty value stored as "".
+func TestNormalize(t *testing.T) {
+	stored, v, err := Normalize(KindPage, VariantOffer, `{"type":"","fields":{"price":" 149,99 ","currency":"pln","areaServed":" Kraków "},"custom":"{\"@type\":\"FAQPage\"}"}`)
+	if err != nil || v.Fields.Price != "149.99" || v.Fields.Currency != "PLN" || v.Fields.AreaServed != "Kraków" || !strings.Contains(v.Custom, "\n  \"@type\": \"FAQPage\"\n") {
+		t.Fatalf("service value = %+v err=%v", v, err)
+	}
+	var back Value
+	if json.Unmarshal([]byte(stored), &back) != nil || back != v {
+		t.Fatalf("stored = %s, want the normalised value", stored)
+	}
+	if _, v, _ := Normalize(KindArticle, "", `{"type":"Article","fields":{"price":"5"}}`); v.Fields != (ServiceFields{}) || v.Type != TypeArticle {
+		t.Fatalf("an article keeps no Service fields: %+v", v)
+	}
+	if stored, _, err := Normalize(KindPage, "", `{"type":" ","fields":{"price":"5"}}`); err != nil || stored != "" {
+		t.Fatalf("empty value = %q err=%v", stored, err)
+	}
+	if stored, _, err := Normalize(KindPage, "", ""); err != nil || stored != "" {
+		t.Fatalf("blank = %q err=%v", stored, err)
+	}
+	if s, _, _ := Normalize(KindPage, "", `{"custom":"[{\"@type\":\"A\"},{\"@type\":\"B\"}]"}`); !strings.Contains(s, `[\n  {`) {
+		t.Fatalf("a list stays a list: %s", s)
+	}
+	for name, tc := range map[string]struct{ kind, variant, raw, code, field string }{
+		"type":     {KindPage, "", `{"type":"BlogPosting"}`, CodeTypeUnknown, ""},
+		"price":    {KindPage, VariantService, `{"fields":{"price":"abc"}}`, CodeFieldInvalid, "price"},
+		"currency": {KindPage, VariantService, `{"fields":{"currency":"zloty"}}`, CodeFieldInvalid, "currency"},
+		"json":     {KindPage, "", `{"type":`, CodeInvalidJSON, ""},
+		"custom":   {KindPage, "", `{"custom":"<script>"}`, CodeScriptTag, ""},
+	} {
+		_, _, err := Normalize(tc.kind, tc.variant, tc.raw)
+		var e *Error
+		if !errors.As(err, &e) || e.Code != tc.code || e.Field != tc.field {
+			t.Errorf("%s: err %v, want %s %s", name, err, tc.code, tc.field)
+		}
+	}
+}
+
+// TestMerge pins the merge of custom nodes: a node with a generated @id is
+// merged into it (custom values win, the replaced properties reported
+// sorted), a node without @id or with a new @id is appended, and an equal
+// value is not reported.
+func TestMerge(t *testing.T) {
+	g := map[string]any{"@context": Context, "@graph": []any{
+		map[string]any{"@type": TypeWebPage, "@id": "p#webpage", "name": "Old", "url": "u"},
+		map[string]any{"@type": TypeWebSite, "@id": "s#website", "name": "Site"},
+	}}
+	out, merged := Merge(g, []map[string]any{
+		{"@id": "p#webpage", "name": "New", "url": "u", "about": "x", "description": "d"},
+		{"@type": "FAQPage"},
+		{"@id": "p#faq", "@type": "FAQPage"},
+		{"@id": "s#website", "name": "Site"},
+	})
+	nodes := Nodes(out)
+	if len(nodes) != 4 || nodes[0]["name"] != "New" || nodes[0]["description"] != "d" || nodes[2]["@type"] != "FAQPage" || nodes[3]["@id"] != "p#faq" {
+		t.Fatalf("merged graph = %v", nodes)
+	}
+	if len(merged) != 1 || merged[0].ID != "p#webpage" || merged[0].Type != TypeWebPage || strings.Join(merged[0].Props, ",") != "name" {
+		t.Fatalf("merges = %+v", merged)
+	}
+	if out["@context"] != Context {
+		t.Fatal("the merged graph keeps its context")
+	}
+	_, none := Merge(g, nil)
+	if none != nil {
+		t.Fatalf("no custom nodes reported %v", none)
+	}
+}
+
+// TestParseValue pins the stored value's reading.
+func TestParseValue(t *testing.T) {
+	if v, err := ParseValue(""); err != nil || v != (Value{}) {
+		t.Fatal("blank is the zero value")
+	}
+	if _, err := ParseValue("{"); codeOf(err) != CodeInvalidJSON {
+		t.Fatalf("invalid = %v", err)
+	}
+	if v, _ := ParseValue(`{"type":"Article","custom":"{}"}`); v.Type != "Article" || v.Custom != "{}" {
+		t.Fatalf("value = %+v", v)
+	}
+	if (&Error{Code: "x"}).Error() != "schema: x" || (&Error{Code: "x", Detail: "d"}).Error() != "schema: x: d" {
+		t.Fatal("Error() is wrong")
+	}
+}

classes/schema/graph.go

@@ -0,0 +1,300 @@
+// Package schema builds the JSON-LD @graph of a page (D-43): one
+// {"@context": "https://schema.org", "@graph": [...]} object whose nodes
+// link to each other by @id (P33-22). It works on plain data only, so the
+// resolver, the admin preview and the MCP tool share one builder.
+package schema
+
+import (
+	"context"
+	"strings"
+	"time"
+)
+
+// Context is the @context of every graph.
+const Context = "https://schema.org"
+
+// The schema.org types the builder generates or an editor may choose. No
+// e-commerce type is offered (out of scope).
+const (
+	TypeWebPage        = "WebPage"
+	TypeAboutPage      = "AboutPage"
+	TypeContactPage    = "ContactPage"
+	TypeCollectionPage = "CollectionPage"
+	TypeProfilePage    = "ProfilePage"
+	TypeBlogPosting    = "BlogPosting"
+	TypeArticle        = "Article"
+	TypeNewsArticle    = "NewsArticle"
+	TypeService        = "Service"
+	TypeWebSite        = "WebSite"
+	TypeOrganization   = "Organization"
+	TypePerson         = "Person"
+	TypeBreadcrumbList = "BreadcrumbList"
+	TypeVideoObject    = "VideoObject"
+	TypeLocalBusiness  = "LocalBusiness"
+)
+
+// The content kinds the builder knows (pact.ContentKind* values).
+const (
+	KindArticle    = "article"
+	KindCollection = "collection"
+	KindProfile    = "profile"
+	KindPage       = "page"
+)
+
+// The registry page kinds whose default type is Service.
+const (
+	VariantOffer   = "offer"
+	VariantService = "service"
+)
+
+// The publisher types of the settings.
+const (
+	PublisherOrganization = "organization"
+	PublisherPerson       = "person"
+)
+
+// Crumb is one breadcrumb: its name, absolute URL and site path.
+type Crumb struct {
+	Name string `json:"name"`
+	URL  string `json:"url"`
+	Path string `json:"path"`
+}
+
+// Person is an author: a name and, when the author has a page, its
+// absolute URL.
+type Person struct {
+	Name string
+	URL  string
+}
+
+// Video is an embedded video: its host and its id there.
+type Video struct {
+	Provider string
+	ID       string
+}
+
+// Hours is one opening-hours row: a day (mon … sun) and 24-hour times.
+type Hours struct {
+	Day    string `json:"day"`
+	Opens  string `json:"opens"`
+	Closes string `json:"closes"`
+}
+
+// Local is the publisher's local business data (D-45).
+type Local struct {
+	Type       string
+	PriceRange string
+	Street     string
+	PostalCode string
+	City       string
+	Region     string
+	Country    string
+	Latitude   *float64
+	Longitude  *float64
+	Hours      []Hours
+}
+
+// Publisher is who publishes the site (D-03, D-45): an organisation or a
+// person, optionally a local business.
+type Publisher struct {
+	Type   string
+	Name   string
+	Logo   string
+	URL    string
+	Email  string
+	Phone  string
+	SameAs []string
+	Local  *Local
+}
+
+// ServiceFields are the editor's Service fields (UI-SPEC C17).
+type ServiceFields struct {
+	ServiceType string `json:"serviceType,omitempty"`
+	AreaServed  string `json:"areaServed,omitempty"`
+	Price       string `json:"price,omitempty"`
+	Currency    string `json:"currency,omitempty"`
+}
+
+// Input is what Build composes a page's graph from.
+type Input struct {
+	// Site is the site's origin without a trailing slash.
+	Site string
+	// URL is the page's absolute URL (its canonical address).
+	URL string
+	// Lang is the page's language tag (pl, pl-PL).
+	Lang string
+	// Home is true for the site's home page.
+	Home bool
+	// Kind is the content kind (article, collection, profile, page), ""
+	// when the page has no record or the type names none.
+	Kind string
+	// Variant is the registry page kind (page, offer, service, landing).
+	Variant string
+	// Type is the editor's schema type, "" for the kind's default.
+	Type string
+	// Name and Description are the page's name and description.
+	Name        string
+	Description string
+	// Headline is the article's headline, Name when empty.
+	Headline string
+	// Images are absolute image URLs, the featured one first.
+	Images      []string
+	PublishedAt *time.Time
+	ModifiedAt  *time.Time
+	Words       *int
+	// Section is the article's section: its primary parent's name.
+	Section string
+	Authors []Person
+	Videos  []Video
+	// Crumbs is the breadcrumb trail, home first, the page last.
+	Crumbs    []Crumb
+	SiteName  string
+	SearchURL string
+	Publisher Publisher
+	Service   ServiceFields
+	// Profile is the person of a profile page; Name when empty.
+	Profile *Person
+}
+
+// DefaultType is the type a page of kind (and registry variant) gets when
+// the editor chose none.
+func DefaultType(kind, variant string) string {
+	switch kind {
+	case KindArticle:
+		return TypeBlogPosting
+	case KindCollection:
+		return TypeCollectionPage
+	case KindProfile:
+		return TypeProfilePage
+	}
+	if variant == VariantOffer || variant == VariantService {
+		return TypeService
+	}
+	return TypeWebPage
+}
+
+// AllowedTypes lists the types an editor may choose for kind (UI-SPEC
+// C17): one fixed type for collections and profiles.
+func AllowedTypes(kind string) []string {
+	switch kind {
+	case KindArticle:
+		return []string{TypeBlogPosting, TypeArticle, TypeNewsArticle}
+	case KindCollection:
+		return []string{TypeCollectionPage}
+	case KindProfile:
+		return []string{TypeProfilePage}
+	}
+	return []string{TypeWebPage, TypeAboutPage, TypeContactPage, TypeService}
+}
+
+// Allowed reports whether typ may be chosen for kind; "" always may.
+func Allowed(kind, typ string) bool {
+	if typ == "" {
+		return true
+	}
+	for _, t := range AllowedTypes(kind) {
+		if t == typ {
+			return true
+		}
+	}
+	return false
+}
+
+// EffectiveType is the page's schema type: the editor's choice when it is
+// allowed for the kind, else the default.
+func EffectiveType(kind, variant, typ string) string {
+	if typ != "" && Allowed(kind, typ) {
+		return typ
+	}
+	return DefaultType(kind, variant)
+}
+
+// isArticle reports whether typ is an article type.
+func isArticle(typ string) bool {
+	return typ == TypeBlogPosting || typ == TypeArticle || typ == TypeNewsArticle
+}
+
+// IDs are the @id values of a page (P33-22).
+type IDs struct {
+	WebSite    string
+	Publisher  string
+	Logo       string
+	WebPage    string
+	Breadcrumb string
+	Article    string
+	ItemList   string
+	Profile    string
+	Service    string
+}
+
+// PageIDs returns the @id values of a page at url on site.
+func PageIDs(site, url, publisherType string) IDs {
+	pub := site + "/#organization"
+	if publisherType == PublisherPerson {
+		pub = site + "/#person"
+	}
+	return IDs{
+		WebSite:    site + "/#website",
+		Publisher:  pub,
+		Logo:       site + "/#logo",
+		WebPage:    url + "#webpage",
+		Breadcrumb: url + "#breadcrumb",
+		Article:    url + "#article",
+		ItemList:   url + "#itemlist",
+		Profile:    url + "#profile",
+		Service:    url + "#service",
+	}
+}
+
+// Build composes the page's graph: the page node (WebPage or its subtype),
+// its BreadcrumbList when the trail has two items or more, the WebSite and
+// the publisher, and per kind the article with its videos, the profile's
+// person or the service.
+func Build(_ context.Context, in Input) (map[string]any, error) {
+	typ := EffectiveType(in.Kind, in.Variant, in.Type)
+	ids := PageIDs(in.Site, in.URL, in.Publisher.Type)
+	var graph []map[string]any
+
+	page := webPageNode(in, typ, ids)
+	graph = append(graph, page)
+	if isArticle(typ) {
+		graph = append(graph, articleNode(in, typ, ids))
+	}
+	switch typ {
+	case TypeProfilePage:
+		graph = append(graph, profileNode(in, ids))
+	case TypeService:
+		graph = append(graph, serviceNode(in, ids))
+	}
+	if crumbs := breadcrumbNode(in.Crumbs, ids); crumbs != nil {
+		graph = append(graph, crumbs)
+	}
+	graph = append(graph, videoNodes(in, ids)...)
+	graph = append(graph, webSiteNode(in, ids))
+	if pub := PublisherNode(in.Site, in.Publisher, in.SiteName); pub != nil {
+		graph = append(graph, pub)
+	}
+	nodes := make([]any, len(graph))
+	for i, n := range graph {
+		nodes[i] = n
+	}
+	return map[string]any{"@context": Context, "@graph": nodes}, nil
+}
+
+// put sets key to a non-empty string value.
+func put(n map[string]any, key, value string) {
+	if v := strings.TrimSpace(value); v != "" {
+		n[key] = v
+	}
+}
+
+// ref is an @id reference.
+func ref(id string) map[string]any { return map[string]any{"@id": id} }
+
+// date formats a time for schema.org (RFC 3339, UTC).
+func date(t *time.Time) string {
+	if t == nil || t.IsZero() {
+		return ""
+	}
+	return t.UTC().Format(time.RFC3339)
+}

classes/schema/graph_test.go

@@ -0,0 +1,214 @@
+package schema
+
+import (
+	"context"
+	"encoding/json"
+	"strings"
+	"testing"
+	"time"
+)
+
+const site = "https://example.test"
+
+func nodesByType(t *testing.T, g map[string]any) map[string]map[string]any {
+	t.Helper()
+	if g["@context"] != Context {
+		t.Fatalf("@context = %v", g["@context"])
+	}
+	out := map[string]map[string]any{}
+	for _, n := range Nodes(g) {
+		out[NodeType(n)] = n
+	}
+	return out
+}
+
+func refID(v any) string {
+	m, _ := v.(map[string]any)
+	s, _ := m["@id"].(string)
+	return s
+}
+
+// TestDefaultAndAllowedTypes pins the type table (UI-SPEC C17): the
+// default per kind and registry variant, the choices per kind, and an
+// editor's choice outside them falling back to the default.
+func TestDefaultAndAllowedTypes(t *testing.T) {
+	for _, tc := range []struct{ kind, variant, want string }{
+		{KindArticle, "", TypeBlogPosting}, {KindCollection, "", TypeCollectionPage}, {KindProfile, "", TypeProfilePage},
+		{KindPage, "", TypeWebPage}, {"", "", TypeWebPage}, {KindPage, VariantOffer, TypeService}, {KindPage, VariantService, TypeService},
+		{KindPage, "landing", TypeWebPage}, {KindArticle, VariantOffer, TypeBlogPosting},
+	} {
+		if got := DefaultType(tc.kind, tc.variant); got != tc.want {
+			t.Errorf("DefaultType(%q, %q) = %q, want %q", tc.kind, tc.variant, got, tc.want)
+		}
+	}
+	for _, tc := range []struct {
+		kind, typ string
+		want      bool
+	}{
+		{KindArticle, TypeNewsArticle, true}, {KindArticle, TypeService, false}, {KindCollection, TypeCollectionPage, true},
+		{KindCollection, TypeWebPage, false}, {KindProfile, TypeProfilePage, true}, {KindPage, TypeContactPage, true},
+		{KindPage, TypeBlogPosting, false}, {KindPage, "Product", false}, {KindArticle, "", true},
+	} {
+		if got := Allowed(tc.kind, tc.typ); got != tc.want {
+			t.Errorf("Allowed(%q, %q) = %v, want %v", tc.kind, tc.typ, got, tc.want)
+		}
+	}
+	if EffectiveType(KindPage, "", TypeBlogPosting) != TypeWebPage || EffectiveType(KindPage, VariantOffer, "") != TypeService || EffectiveType(KindArticle, "", TypeArticle) != TypeArticle {
+		t.Fatal("EffectiveType is wrong")
+	}
+}
+
+// TestPageIDs pins P33-22's @id values: site-level ids on the origin, the
+// person publisher's own fragment, page-level ids on the page URL.
+func TestPageIDs(t *testing.T) {
+	ids := PageIDs(site, site+"/blog/a", PublisherOrganization)
+	if ids.WebSite != site+"/#website" || ids.Publisher != site+"/#organization" || ids.Logo != site+"/#logo" ||
+		ids.WebPage != site+"/blog/a#webpage" || ids.Breadcrumb != site+"/blog/a#breadcrumb" || ids.Article != site+"/blog/a#article" ||
+		ids.ItemList != site+"/blog/a#itemlist" || ids.Profile != site+"/blog/a#profile" || ids.Service != site+"/blog/a#service" {
+		t.Fatalf("ids = %+v", ids)
+	}
+	if PageIDs(site, site+"/", PublisherPerson).Publisher != site+"/#person" {
+		t.Fatal("a person publisher is /#person")
+	}
+}
+
+// TestBuildArticle pins an article page's graph: WebPage, BlogPosting,
+// BreadcrumbList, VideoObject, WebSite and the publisher, linked by @id;
+// headline, authors with profile ids, images, dates (dateModified falling
+// back to datePublished), word count and section.
+func TestBuildArticle(t *testing.T) {
+	at := time.Date(2026, 11, 1, 10, 0, 0, 0, time.FixedZone("CET", 3600))
+	words := 420
+	g, err := Build(context.Background(), Input{
+		Site: site, URL: site + "/blog/a", Lang: "pl", Kind: KindArticle, Name: "Grzyby", Description: "O grzybach",
+		Images: []string{site + "/a.jpg", site + "/b.jpg"}, PublishedAt: &at, Words: &words, Section: "Lecznicze",
+		Authors:  []Person{{Name: "Ada", URL: site + "/autor/ada"}, {Name: " "}, {Name: "Gość"}},
+		Videos:   []Video{{Provider: "YouTube", ID: "abc 1"}, {Provider: "vimeo", ID: "9"}, {Provider: "youtube", ID: ""}},
+		Crumbs:   []Crumb{{Name: "Start", URL: site + "/"}, {Name: "Grzyby", URL: site + "/blog/a"}},
+		SiteName: "Acme", SearchURL: "/szukaj?q={search_term_string}",
+		Publisher: Publisher{Type: PublisherOrganization, Logo: site + "/logo.png"},
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	by := nodesByType(t, g)
+	for _, typ := range []string{TypeWebPage, TypeBlogPosting, TypeBreadcrumbList, TypeVideoObject, TypeWebSite, TypeOrganization} {
+		if by[typ] == nil {
+			t.Fatalf("graph lacks %s: %v", typ, g)
+		}
+	}
+	if len(Nodes(g)) != 6 {
+		t.Fatalf("%d nodes, want 6 (one video)", len(Nodes(g)))
+	}
+	page, art := by[TypeWebPage], by[TypeBlogPosting]
+	if refID(page["isPartOf"]) != site+"/#website" || refID(page["breadcrumb"]) != site+"/blog/a#breadcrumb" || page["inLanguage"] != "pl" {
+		t.Fatalf("page = %v", page)
+	}
+	if art["headline"] != "Grzyby" || refID(art["mainEntityOfPage"]) != site+"/blog/a#webpage" || refID(art["publisher"]) != site+"/#organization" ||
+		art["datePublished"] != "2026-11-01T09:00:00Z" || art["dateModified"] != "2026-11-01T09:00:00Z" || art["wordCount"] != 420 || art["articleSection"] != "Lecznicze" {
+		t.Fatalf("article = %v", art)
+	}
+	authors := art["author"].([]any)
+	if len(authors) != 2 || authors[0].(map[string]any)["@id"] != site+"/autor/ada#profile" || authors[1].(map[string]any)["url"] != nil {
+		t.Fatalf("authors = %v", authors)
+	}
+	video := by[TypeVideoObject]
+	if video["embedUrl"] != "https://www.youtube.com/embed/abc%201" || video["@id"] != site+"/blog/a#video-1" || video["uploadDate"] != "2026-11-01T09:00:00Z" || video["description"] != "O grzybach" {
+		t.Fatalf("video = %v", video)
+	}
+	ws := by[TypeWebSite]
+	action := ws["potentialAction"].(map[string]any)
+	if ws["url"] != site+"/" || action["target"].(map[string]any)["urlTemplate"] != site+"/szukaj?q={search_term_string}" {
+		t.Fatalf("website = %v", ws)
+	}
+	org := by[TypeOrganization]
+	if org["name"] != "Acme" || refID(org["image"]) != site+"/#logo" {
+		t.Fatalf("publisher = %v", org)
+	}
+	if _, err := json.Marshal(g); err != nil {
+		t.Fatalf("graph does not marshal: %v", err)
+	}
+}
+
+// TestBuildPerKind pins the page node and main entity per kind: a
+// collection's ItemList, a profile's Person, a service's offer with the
+// normalised price, a contact page about the publisher, the home page
+// about the publisher, and no BreadcrumbList for a single crumb.
+func TestBuildPerKind(t *testing.T) {
+	build := func(in Input) map[string]map[string]any {
+		in.Site, in.SiteName = site, "Acme"
+		if in.URL == "" {
+			in.URL = site + "/x"
+		}
+		g, err := Build(context.Background(), in)
+		if err != nil {
+			t.Fatal(err)
+		}
+		return nodesByType(t, g)
+	}
+	col := build(Input{Kind: KindCollection, Name: "Lecznicze"})
+	if refID(col[TypeCollectionPage]["mainEntity"]) != site+"/x#itemlist" {
+		t.Fatalf("collection = %v", col)
+	}
+	prof := build(Input{Kind: KindProfile, Name: "Ada page", Profile: &Person{Name: "Ada"}, Images: []string{site + "/ada.jpg"}})
+	if refID(prof[TypeProfilePage]["mainEntity"]) != site+"/x#profile" || prof[TypePerson]["name"] != "Ada" || prof[TypePerson]["image"] != site+"/ada.jpg" {
+		t.Fatalf("profile = %v", prof)
+	}
+	svc := build(Input{Kind: KindPage, Variant: VariantOffer, Name: "Audyt", Service: ServiceFields{ServiceType: "SEO", AreaServed: "Kraków, Warszawa", Price: "149,99", Currency: "pln"}})
+	offer := svc[TypeService]["offers"].(map[string]any)
+	if refID(svc[TypeWebPage]["mainEntity"]) != site+"/x#service" || offer["price"] != "149.99" || offer["priceCurrency"] != "PLN" || len(svc[TypeService]["areaServed"].([]any)) != 2 {
+		t.Fatalf("service = %v", svc)
+	}
+	one := build(Input{Kind: KindPage, Variant: VariantService, Service: ServiceFields{AreaServed: " Kraków ", Price: "abc"}})
+	if one[TypeService]["areaServed"] != "Kraków" || one[TypeService]["offers"] != nil {
+		t.Fatalf("service with one area and no price = %v", one[TypeService])
+	}
+	contact := build(Input{Type: TypeContactPage, Crumbs: []Crumb{{Name: "Start", URL: site + "/"}}})
+	if refID(contact[TypeContactPage]["mainEntity"]) != site+"/#organization" || refID(contact[TypeContactPage]["about"]) != site+"/#organization" || contact[TypeBreadcrumbList] != nil {
+		t.Fatalf("contact = %v", contact)
+	}
+	home := build(Input{URL: site + "/", Home: true})
+	if refID(home[TypeWebPage]["about"]) != site+"/#organization" {
+		t.Fatalf("home = %v", home)
+	}
+}
+
+// TestPublisherNode pins the publisher: named after the site when unnamed,
+// nil without any name, a person's image instead of a logo, and a local
+// business with its address, geo, hours (bad rows dropped) and price
+// range.
+func TestPublisherNode(t *testing.T) {
+	if PublisherNode(site, Publisher{}, " ") != nil {
+		t.Fatal("no name must give no node")
+	}
+	person := PublisherNode(site, Publisher{Type: PublisherPerson, Name: "Ada", Logo: site + "/ada.jpg", SameAs: []string{"https://x.test/ada"}}, "Acme")
+	if person["@type"] != TypePerson || person["@id"] != site+"/#person" || person["logo"] != nil || person["image"].(map[string]any)["url"] != site+"/ada.jpg" || len(person["sameAs"].([]any)) != 1 {
+		t.Fatalf("person = %v", person)
+	}
+	lat, lng := 50.06, 19.94
+	local := PublisherNode(site, Publisher{Name: "Sklep", Phone: "+48 1", Local: &Local{
+		Type: "Store", PriceRange: "$$", Street: "Rynek 1", City: "Kraków", Country: "pl", Latitude: &lat, Longitude: &lng,
+		Hours: []Hours{{Day: "mon", Opens: "09:00", Closes: "17:00"}, {Day: "xyz", Opens: "1", Closes: "2"}, {Day: "tue", Opens: "09:00"}},
+	}}, "")
+	addr := local["address"].(map[string]any)
+	if local["@type"] != "Store" || addr["addressCountry"] != "PL" || addr["streetAddress"] != "Rynek 1" || local["geo"] == nil ||
+		len(local["openingHoursSpecification"].([]any)) != 1 || local["priceRange"] != "$$" || local["telephone"] != "+48 1" || local["url"] != site+"/" {
+		t.Fatalf("local business = %v", local)
+	}
+	bare := PublisherNode(site, Publisher{Name: "Sklep", Local: &Local{}}, "")
+	if bare["@type"] != TypeLocalBusiness || bare["address"] != nil {
+		t.Fatalf("local business without data = %v", bare)
+	}
+}
+
+// TestNormalizePrice pins the price forms an editor may type.
+func TestNormalizePrice(t *testing.T) {
+	for in, want := range map[string]string{"150": "150", "149,99": "149.99", " 149.99 ": "149.99", "0": "0", "": "", "abc": "", "-5": "", "1e3": "", "+5": "", "12,5,0": ""} {
+		if got := NormalizePrice(in); got != want {
+			t.Errorf("NormalizePrice(%q) = %q, want %q", in, got, want)
+		}
+	}
+	if strings.Join(splitList(" a, ,b ,"), "|") != "a|b" {
+		t.Fatal("splitList is wrong")
+	}
+}

classes/schema/nodes.go

@@ -0,0 +1,340 @@
+package schema
+
+import (
+	"fmt"
+	"net/url"
+	"strconv"
+	"strings"
+)
+
+// webPageNode is the page node: WebPage or the subtype the effective type
+// asks for, part of the WebSite, pointing at its breadcrumb trail and, per
+// type, at its main entity.
+func webPageNode(in Input, typ string, ids IDs) map[string]any {
+	pageType := TypeWebPage
+	switch typ {
+	case TypeAboutPage, TypeContactPage, TypeCollectionPage, TypeProfilePage:
+		pageType = typ
+	}
+	n := map[string]any{
+		"@type":    pageType,
+		"@id":      ids.WebPage,
+		"url":      in.URL,
+		"isPartOf": ref(ids.WebSite),
+	}
+	put(n, "name", in.Name)
+	put(n, "description", in.Description)
+	put(n, "inLanguage", in.Lang)
+	if len(in.Crumbs) >= 2 {
+		n["breadcrumb"] = ref(ids.Breadcrumb)
+	}
+	if len(in.Images) > 0 {
+		n["primaryImageOfPage"] = map[string]any{"@type": "ImageObject", "url": in.Images[0]}
+	}
+	if d := date(in.PublishedAt); d != "" {
+		n["datePublished"] = d
+	}
+	if d := date(in.ModifiedAt); d != "" {
+		n["dateModified"] = d
+	}
+	switch typ {
+	case TypeCollectionPage:
+		n["mainEntity"] = ref(ids.ItemList)
+	case TypeProfilePage:
+		n["mainEntity"] = ref(ids.Profile)
+	case TypeService:
+		n["mainEntity"] = ref(ids.Service)
+	case TypeContactPage, TypeAboutPage:
+		n["about"] = ref(ids.Publisher)
+		if typ == TypeContactPage {
+			n["mainEntity"] = ref(ids.Publisher)
+		}
+	}
+	if in.Home {
+		n["about"] = ref(ids.Publisher)
+	}
+	return n
+}
+
+// articleNode is the article of an article page: headline, authors,
+// publisher, images, dates, word count and section.
+func articleNode(in Input, typ string, ids IDs) map[string]any {
+	n := map[string]any{
+		"@type":            typ,
+		"@id":              ids.Article,
+		"isPartOf":         ref(ids.WebPage),
+		"mainEntityOfPage": ref(ids.WebPage),
+		"publisher":        ref(ids.Publisher),
+	}
+	headline := in.Headline
+	if strings.TrimSpace(headline) == "" {
+		headline = in.Name
+	}
+	put(n, "headline", headline)
+	put(n, "description", in.Description)
+	put(n, "inLanguage", in.Lang)
+	put(n, "articleSection", in.Section)
+	if d := date(in.PublishedAt); d != "" {
+		n["datePublished"] = d
+	}
+	if d := date(in.ModifiedAt); d != "" {
+		n["dateModified"] = d
+	} else if d := date(in.PublishedAt); d != "" {
+		n["dateModified"] = d
+	}
+	if in.Words != nil && *in.Words > 0 {
+		n["wordCount"] = *in.Words
+	}
+	if len(in.Images) > 0 {
+		imgs := make([]any, len(in.Images))
+		for i, u := range in.Images {
+			imgs[i] = u
+		}
+		n["image"] = imgs
+	}
+	var authors []any
+	for _, a := range in.Authors {
+		if strings.TrimSpace(a.Name) == "" {
+			continue
+		}
+		p := map[string]any{"@type": TypePerson, "name": strings.TrimSpace(a.Name)}
+		if a.URL != "" {
+			p["url"] = a.URL
+			p["@id"] = a.URL + "#profile"
+		}
+		authors = append(authors, p)
+	}
+	if len(authors) > 0 {
+		n["author"] = authors
+	}
+	return n
+}
+
+// profileNode is the person of a profile page.
+func profileNode(in Input, ids IDs) map[string]any {
+	n := map[string]any{"@type": TypePerson, "@id": ids.Profile, "url": in.URL, "mainEntityOfPage": ref(ids.WebPage)}
+	name := in.Name
+	if in.Profile != nil && strings.TrimSpace(in.Profile.Name) != "" {
+		name = in.Profile.Name
+	}
+	put(n, "name", name)
+	put(n, "description", in.Description)
+	if len(in.Images) > 0 {
+		n["image"] = in.Images[0]
+	}
+	return n
+}
+
+// serviceNode is the service or offer of a page: provided by the
+// publisher, with the area served and an optional offer (price and
+// currency).
+func serviceNode(in Input, ids IDs) map[string]any {
+	n := map[string]any{
+		"@type":            TypeService,
+		"@id":              ids.Service,
+		"url":              in.URL,
+		"provider":         ref(ids.Publisher),
+		"mainEntityOfPage": ref(ids.WebPage),
+	}
+	put(n, "name", in.Name)
+	put(n, "description", in.Description)
+	put(n, "serviceType", in.Service.ServiceType)
+	if areas := splitList(in.Service.AreaServed); len(areas) > 0 {
+		if len(areas) == 1 {
+			n["areaServed"] = areas[0]
+		} else {
+			list := make([]any, len(areas))
+			for i, a := range areas {
+				list[i] = a
+			}
+			n["areaServed"] = list
+		}
+	}
+	if len(in.Images) > 0 {
+		n["image"] = in.Images[0]
+	}
+	if price := NormalizePrice(in.Service.Price); price != "" {
+		offer := map[string]any{"@type": "Offer", "price": price, "url": in.URL}
+		put(offer, "priceCurrency", strings.ToUpper(in.Service.Currency))
+		n["offers"] = offer
+	}
+	return n
+}
+
+// NormalizePrice turns an editor's price (150, 149,99, 149.99) into its
+// schema.org form, "" when it is not a price.
+func NormalizePrice(s string) string {
+	s = strings.ReplaceAll(strings.TrimSpace(s), ",", ".")
+	if s == "" {
+		return ""
+	}
+	f, err := strconv.ParseFloat(s, 64)
+	if err != nil || f < 0 || strings.ContainsAny(s, "eE+-") {
+		return ""
+	}
+	return s
+}
+
+// splitList splits a comma-separated list into its trimmed items.
+func splitList(s string) []string {
+	var out []string
+	for _, part := range strings.Split(s, ",") {
+		if p := strings.TrimSpace(part); p != "" {
+			out = append(out, p)
+		}
+	}
+	return out
+}
+
+// breadcrumbNode is the BreadcrumbList of a trail of two items or more
+// (Google's minimum); nil otherwise.
+func breadcrumbNode(crumbs []Crumb, ids IDs) map[string]any {
+	if len(crumbs) < 2 {
+		return nil
+	}
+	items := make([]any, len(crumbs))
+	for i, c := range crumbs {
+		items[i] = map[string]any{"@type": "ListItem", "position": i + 1, "name": c.Name, "item": c.URL}
+	}
+	return map[string]any{"@type": TypeBreadcrumbList, "@id": ids.Breadcrumb, "itemListElement": items}
+}
+
+// videoNodes is one VideoObject per YouTube video (D-43).
+func videoNodes(in Input, ids IDs) []map[string]any {
+	var out []map[string]any
+	n := 0
+	for _, v := range in.Videos {
+		id := strings.TrimSpace(v.ID)
+		if !strings.EqualFold(v.Provider, "youtube") || id == "" {
+			continue
+		}
+		n++
+		esc := url.PathEscape(id)
+		node := map[string]any{
+			"@type":        TypeVideoObject,
+			"@id":          fmt.Sprintf("%s#video-%d", in.URL, n),
+			"embedUrl":     "https://www.youtube.com/embed/" + esc,
+			"url":          "https://www.youtube.com/watch?v=" + url.QueryEscape(id),
+			"thumbnailUrl": []any{"https://i.ytimg.com/vi/" + esc + "/hqdefault.jpg"},
+			"isPartOf":     ref(ids.WebPage),
+		}
+		name := in.Headline
+		if strings.TrimSpace(name) == "" {
+			name = in.Name
+		}
+		put(node, "name", name)
+		desc := in.Description
+		if strings.TrimSpace(desc) == "" {
+			desc = name
+		}
+		put(node, "description", desc)
+		if d := date(in.PublishedAt); d != "" {
+			node["uploadDate"] = d
+		}
+		out = append(out, node)
+	}
+	return out
+}
+
+// webSiteNode is the WebSite, published by the publisher, with a
+// SearchAction when the site has a search address.
+func webSiteNode(in Input, ids IDs) map[string]any {
+	n := map[string]any{
+		"@type":     TypeWebSite,
+		"@id":       ids.WebSite,
+		"url":       in.Site + "/",
+		"publisher": ref(ids.Publisher),
+	}
+	put(n, "name", in.SiteName)
+	put(n, "inLanguage", in.Lang)
+	if search := strings.TrimSpace(in.SearchURL); search != "" && strings.Contains(search, "{search_term_string}") {
+		target := search
+		if strings.HasPrefix(search, "/") {
+			target = in.Site + search
+		}
+		n["potentialAction"] = map[string]any{
+			"@type":       "SearchAction",
+			"target":      map[string]any{"@type": "EntryPoint", "urlTemplate": target},
+			"query-input": "required name=search_term_string",
+		}
+	}
+	return n
+}
+
+// dayNames maps the settings' day codes to schema.org day names.
+var dayNames = map[string]string{
+	"mon": "Monday", "tue": "Tuesday", "wed": "Wednesday", "thu": "Thursday",
+	"fri": "Friday", "sat": "Saturday", "sun": "Sunday",
+}
+
+// PublisherNode is the publisher's node: an Organization (or the local
+// business type) or a Person, named after the site when the settings name
+// none. It is nil when there is no name at all.
+func PublisherNode(site string, p Publisher, siteName string) map[string]any {
+	name := strings.TrimSpace(p.Name)
+	if name == "" {
+		name = strings.TrimSpace(siteName)
+	}
+	if name == "" {
+		return nil
+	}
+	ids := PageIDs(site, site+"/", p.Type)
+	home := strings.TrimSpace(p.URL)
+	if home == "" {
+		home = site + "/"
+	}
+	n := map[string]any{"@id": ids.Publisher, "name": name, "url": home}
+	if p.Type == PublisherPerson {
+		n["@type"] = TypePerson
+		if logo := strings.TrimSpace(p.Logo); logo != "" {
+			n["image"] = map[string]any{"@type": "ImageObject", "url": logo}
+		}
+	} else {
+		n["@type"] = TypeOrganization
+		if logo := strings.TrimSpace(p.Logo); logo != "" {
+			n["logo"] = map[string]any{"@type": "ImageObject", "@id": ids.Logo, "url": logo, "contentUrl": logo}
+			n["image"] = ref(ids.Logo)
+		}
+	}
+	put(n, "email", p.Email)
+	put(n, "telephone", p.Phone)
+	if len(p.SameAs) > 0 {
+		list := make([]any, len(p.SameAs))
+		for i, s := range p.SameAs {
+			list[i] = s
+		}
+		n["sameAs"] = list
+	}
+	if l := p.Local; l != nil && p.Type != PublisherPerson {
+		typ := strings.TrimSpace(l.Type)
+		if typ == "" {
+			typ = TypeLocalBusiness
+		}
+		n["@type"] = typ
+		addr := map[string]any{"@type": "PostalAddress"}
+		put(addr, "streetAddress", l.Street)
+		put(addr, "postalCode", l.PostalCode)
+		put(addr, "addressLocality", l.City)
+		put(addr, "addressRegion", l.Region)
+		put(addr, "addressCountry", strings.ToUpper(l.Country))
+		if len(addr) > 1 {
+			n["address"] = addr
+		}
+		if l.Latitude != nil && l.Longitude != nil {
+			n["geo"] = map[string]any{"@type": "GeoCoordinates", "latitude": *l.Latitude, "longitude": *l.Longitude}
+		}
+		var hours []any
+		for _, h := range l.Hours {
+			day, ok := dayNames[h.Day]
+			if !ok || h.Opens == "" || h.Closes == "" {
+				continue
+			}
+			hours = append(hours, map[string]any{"@type": "OpeningHoursSpecification", "dayOfWeek": day, "opens": h.Opens, "closes": h.Closes})
+		}
+		if len(hours) > 0 {
+			n["openingHoursSpecification"] = hours
+		}
+		put(n, "priceRange", l.PriceRange)
+	}
+	return n
+}

classes/schema/validate.go

@@ -0,0 +1,153 @@
+package schema
+
+import "strings"
+
+// The levels of a finding.
+const (
+	LevelRequired    = "required"
+	LevelRecommended = "recommended"
+)
+
+// Finding is one missing property of a node.
+type Finding struct {
+	Level    string `json:"level"`
+	Property string `json:"property"`
+}
+
+// Report is a node with its findings. Rules is false for a type Google has
+// no rich-result rules for (FAQPage, CollectionPage, WebPage, Service …;
+// RESEARCH Pitfall 11): it is checked as valid schema.org only.
+type Report struct {
+	ID       string    `json:"id"`
+	Type     string    `json:"type"`
+	Findings []Finding `json:"findings"`
+	Rules    bool      `json:"rules"`
+}
+
+// rule is the required and recommended properties of a type (Google's
+// structured data documentation, cited in 33-RESEARCH.md).
+type rule struct {
+	required    []string
+	recommended []string
+}
+
+var articleRule = rule{recommended: []string{"author", "datePublished", "dateModified", "headline", "image"}}
+
+var localRule = rule{required: []string{"name", "address"}, recommended: []string{"telephone", "url", "geo", "openingHoursSpecification", "priceRange", "image"}}
+
+// rules lists the types with Google rules; every other type has none.
+var rules = map[string]rule{
+	TypeArticle:        articleRule,
+	TypeBlogPosting:    articleRule,
+	TypeNewsArticle:    articleRule,
+	TypeBreadcrumbList: {required: []string{"itemListElement"}},
+	TypeVideoObject:    {required: []string{"name", "thumbnailUrl", "uploadDate"}, recommended: []string{"description", "embedUrl"}},
+	TypeOrganization:   {recommended: []string{"name", "url", "logo"}},
+	TypePerson:         {required: []string{"name"}, recommended: []string{"url", "image"}},
+	TypeWebSite:        {required: []string{"name", "url"}},
+	TypeProfilePage:    {required: []string{"mainEntity"}},
+}
+
+// localTypes are the LocalBusiness types (UI-SPEC C20), checked with the
+// LocalBusiness rules.
+var localTypes = map[string]bool{
+	"LocalBusiness": true, "ProfessionalService": true, "Store": true,
+	"FoodEstablishment": true, "HealthAndBeautyBusiness": true,
+	"MedicalBusiness": true, "LegalService": true,
+	"HomeAndConstructionBusiness": true, "AutomotiveBusiness": true,
+	"FinancialService": true,
+}
+
+// RulesFor returns the rule of typ and whether Google has rules for it.
+func RulesFor(typ string) (required, recommended []string, ok bool) {
+	if localTypes[typ] {
+		return localRule.required, localRule.recommended, true
+	}
+	r, ok := rules[typ]
+	return r.required, r.recommended, ok
+}
+
+// NodeType is a node's first @type, "" when it has none.
+func NodeType(n map[string]any) string {
+	switch t := n["@type"].(type) {
+	case string:
+		return t
+	case []any:
+		for _, v := range t {
+			if s, ok := v.(string); ok {
+				return s
+			}
+		}
+	case []string:
+		if len(t) > 0 {
+			return t[0]
+		}
+	}
+	return ""
+}
+
+// present reports whether a node property carries a value.
+func present(v any) bool {
+	switch x := v.(type) {
+	case nil:
+		return false
+	case string:
+		return strings.TrimSpace(x) != ""
+	case []any:
+		return len(x) > 0
+	case map[string]any:
+		return len(x) > 0
+	}
+	return true
+}
+
+// ValidateNode checks one node against its type's rules.
+func ValidateNode(n map[string]any) Report {
+	typ := NodeType(n)
+	id, _ := n["@id"].(string)
+	rep := Report{ID: id, Type: typ, Findings: []Finding{}}
+	required, recommended, ok := RulesFor(typ)
+	rep.Rules = ok
+	for _, p := range required {
+		if !present(n[p]) {
+			rep.Findings = append(rep.Findings, Finding{Level: LevelRequired, Property: p})
+		}
+	}
+	if typ == TypeBreadcrumbList {
+		if items, _ := n["itemListElement"].([]any); len(items) > 0 && len(items) < 2 {
+			rep.Findings = append(rep.Findings, Finding{Level: LevelRequired, Property: "itemListElement"})
+		}
+	}
+	for _, p := range recommended {
+		if !present(n[p]) {
+			rep.Findings = append(rep.Findings, Finding{Level: LevelRecommended, Property: p})
+		}
+	}
+	return rep
+}
+
+// Nodes returns the node objects of a graph.
+func Nodes(graph map[string]any) []map[string]any {
+	var out []map[string]any
+	switch list := graph["@graph"].(type) {
+	case []any:
+		for _, raw := range list {
+			if n, ok := raw.(map[string]any); ok {
+				out = append(out, n)
+			}
+		}
+	case []map[string]any:
+		out = append(out, list...)
+	}
+	return out
+}
+
+// Validate checks every node of a graph (D-46): the admin preview, the
+// MCP tool and the publisher preview share it.
+func Validate(graph map[string]any) []Report {
+	out := []Report{}
+	for _, n := range Nodes(graph) {
+		out = append(out, ValidateNode(n))
+	}
+	return out
+}

classes/schema/validate_test.go

@@ -0,0 +1,86 @@
+package schema
+
+import (
+	"strings"
+	"testing"
+)
+
+func findings(r Report) string {
+	var out []string
+	for _, f := range r.Findings {
+		out = append(out, f.Level+":"+f.Property)
+	}
+	return strings.Join(out, " ")
+}
+
+// TestValidateNodeRules pins every rule of the table: the required and
+// recommended properties per type, the LocalBusiness subtypes checked as
+// LocalBusiness, and a type without Google rules reported as Rules false
+// with no findings (RESEARCH Pitfall 11).
+func TestValidateNodeRules(t *testing.T) {
+	for _, tc := range []struct {
+		node  map[string]any
+		rules bool
+		want  string
+	}{
+		{map[string]any{"@type": TypeBlogPosting}, true, "recommended:author recommended:datePublished recommended:dateModified recommended:headline recommended:image"},
+		{map[string]any{"@type": TypeArticle, "headline": "x", "author": []any{map[string]any{"name": "a"}}, "datePublished": "d", "dateModified": "d", "image": []any{"i"}}, true, ""},
+		{map[string]any{"@type": TypeNewsArticle, "headline": " "}, true, "recommended:author recommended:datePublished recommended:dateModified recommended:headline recommended:image"},
+		{map[string]any{"@type": TypeBreadcrumbList, "itemListElement": []any{}}, true, "required:itemListElement"},
+		{map[string]any{"@type": TypeVideoObject, "name": "v"}, true, "required:thumbnailUrl required:uploadDate recommended:description recommended:embedUrl"},
+		{map[string]any{"@type": TypeOrganization, "name": "o", "url": "u", "logo": map[string]any{"url": "l"}}, true, ""},
+		{map[string]any{"@type": TypePerson}, true, "required:name recommended:url recommended:image"},
+		{map[string]any{"@type": TypeWebSite, "url": "u"}, true, "required:name"},
+		{map[string]any{"@type": TypeProfilePage, "mainEntity": map[string]any{}}, true, "required:mainEntity"},
+		{map[string]any{"@type": "Store", "name": "s"}, true, "required:address recommended:telephone recommended:url recommended:geo recommended:openingHoursSpecification recommended:priceRange recommended:image"},
+		{map[string]any{"@type": "FinancialService", "name": "s", "address": map[string]any{"a": 1}, "telephone": "1", "url": "u", "geo": map[string]any{"x": 1}, "openingHoursSpecification": []any{1}, "priceRange": "$", "image": "i"}, true, ""},
+		{map[string]any{"@type": TypeWebPage}, false, ""},
+		{map[string]any{"@type": TypeService}, false, ""},
+		{map[string]any{"@type": "FAQPage"}, false, ""},
+		{map[string]any{"@type": TypeCollectionPage}, false, ""},
+	} {
+		r := ValidateNode(tc.node)
+		if r.Rules != tc.rules || findings(r) != tc.want || r.Findings == nil {
+			t.Errorf("%v: rules=%v findings=%q, want rules=%v %q", tc.node["@type"], r.Rules, findings(r), tc.rules, tc.want)
+		}
+	}
+}
+
+// TestNodeType pins the first @type of a node in each JSON form.
+func TestNodeType(t *testing.T) {
+	for _, tc := range []struct {
+		node map[string]any
+		want string
+	}{
+		{map[string]any{"@type": "Article"}, "Article"},
+		{map[string]any{"@type": []any{1, "Store", "LocalBusiness"}}, "Store"},
+		{map[string]any{"@type": []string{"Person"}}, "Person"},
+		{map[string]any{"@type": []string{}}, ""},
+		{map[string]any{}, ""},
+		{map[string]any{"@type": 5}, ""},
+	} {
+		if got := NodeType(tc.node); got != tc.want {
+			t.Errorf("NodeType(%v) = %q, want %q", tc.node, got, tc.want)
+		}
+	}
+}
+
+// TestValidateGraph pins Validate over a graph: one report per node with
+// its @id, in order, and Nodes reading both graph forms and skipping
+// non-objects.
+func TestValidateGraph(t *testing.T) {
+	g := map[string]any{"@graph": []any{map[string]any{"@type": TypeWebSite, "@id": "w", "name": "n", "url": "u"}, "junk", map[string]any{"@type": TypePerson, "@id": "p"}}}
+	reps := Validate(g)
+	if len(reps) != 2 || reps[0].ID != "w" || len(reps[0].Findings) != 0 || reps[1].ID != "p" || findings(reps[1]) != "required:name recommended:url recommended:image" {
+		t.Fatalf("reports = %+v", reps)
+	}
+	if len(Nodes(map[string]any{"@graph": []map[string]any{{"@id": "a"}}})) != 1 || Nodes(map[string]any{}) != nil {
+		t.Fatal("Nodes is wrong")
+	}
+	if got := Validate(map[string]any{}); got == nil || len(got) != 0 {
+		t.Fatalf("empty graph = %v, want []", got)
+	}
+	if r, rec, ok := RulesFor("Unknown"); ok || r != nil || rec != nil {
+		t.Fatal("an unknown type has no rules")
+	}
+}

classes/settings_write.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"strconv"
 	"strings"
 	"time"
 
@@ -37,8 +38,72 @@ var SettingsColumns = []string{
 	"site_name", "twitter_site", "title_template", "home_template",
 	"default_description", "default_og_image", "query_params",
 	"noindex_paths", "search_paths", "url_host", "url_slash",
-	"url_lowercase", "verify_google", "verify_bing", "verify_yandex",
+	"url_lowercase", "links_new_tab", "nofollow_domains",
-	"verify_other", "verify_files", "csv_delimiter",
+	"sponsored_domains", "verify_google", "verify_bing", "verify_yandex",
+	"verify_other", "verify_files", "csv_delimiter", "breadcrumb_home",
+	"publisher_type", "publisher_name", "publisher_logo", "publisher_url",
+	"publisher_email", "publisher_phone", "publisher_same_as",
+	"local_enabled", "local_type", "local_price_range", "local_street",
+	"local_postal_code", "local_city", "local_region", "local_country",
+	"local_latitude", "local_longitude", "local_hours", "page_form",
+	"page_segment", "page_title_suffix", "page_description_suffix",
+	"crawl_schedule", "crawl_max_pages", "crawl_delay", "crawl_keep",
+	"generic_anchors", "autolink_max",
+}
+
+// settingsText maps the plain text settings columns to their fields.
+func settingsText(st *models.Settings) map[string]*string {
+	return map[string]*string{
+		"site_name":               &st.SiteName,
+		"twitter_site":            &st.TwitterSite,
+		"title_template":          &st.TitleTemplate,
+		"home_template":           &st.HomeTemplate,
+		"default_description":     &st.DefaultDescription,
+		"default_og_image":        &st.DefaultOGImage,
+		"noindex_paths":           &st.NoindexPaths,
+		"search_paths":            &st.SearchPaths,
+		"url_host":                &st.URLHost,
+		"url_slash":               &st.URLSlash,
+		"verify_google":           &st.VerifyGoogle,
+		"verify_bing":             &st.VerifyBing,
+		"verify_yandex":           &st.VerifyYandex,
+		"csv_delimiter":           &st.CSVDelimiter,
+		"nofollow_domains":        &st.NofollowDomains,
+		"sponsored_domains":       &st.SponsoredDomains,
+		"breadcrumb_home":         &st.BreadcrumbHome,
+		"publisher_type":          &st.PublisherType,
+		"publisher_name":          &st.PublisherName,
+		"publisher_logo":          &st.PublisherLogo,
+		"publisher_url":           &st.PublisherURL,
+		"publisher_email":         &st.PublisherEmail,
+		"publisher_phone":         &st.PublisherPhone,
+		"publisher_same_as":       &st.PublisherSameAs,
+		"local_type":              &st.LocalType,
+		"local_price_range":       &st.LocalPriceRange,
+		"local_street":            &st.LocalStreet,
+		"local_postal_code":       &st.LocalPostalCode,
+		"local_city":              &st.LocalCity,
+		"local_region":            &st.LocalRegion,
+		"local_country":           &st.LocalCountry,
+		"local_latitude":          &st.LocalLatitude,
+		"local_longitude":         &st.LocalLongitude,
+		"page_form":               &st.PageForm,
+		"page_segment":            &st.PageSegment,
+		"page_title_suffix":       &st.PageTitleSuffix,
+		"page_description_suffix": &st.PageDescriptionSuffix,
+		"crawl_schedule":          &st.CrawlSchedule,
+		"generic_anchors":         &st.GenericAnchors,
+	}
+}
+
+// settingsInts maps the number settings columns to their fields.
+func settingsInts(st *models.Settings) map[string]*int {
+	return map[string]*int{
+		"crawl_max_pages": &st.CrawlMaxPages,
+		"crawl_delay":     &st.CrawlDelay,
+		"crawl_keep":      &st.CrawlKeep,
+		"autolink_max":    &st.AutolinkMax,
+	}
 }
 
 // SettingsValues are the settings row's values keyed by column, for
@@ -57,12 +122,44 @@ func SettingsValues(st models.Settings) map[string]any {
 		"url_host":                st.URLHost,
 		"url_slash":               st.URLSlash,
 		"url_lowercase":           st.URLLowercase,
+		"links_new_tab":           st.LinksNewTab,
+		"nofollow_domains":        st.NofollowDomains,
+		"sponsored_domains":       st.SponsoredDomains,
 		"verify_google":           st.VerifyGoogle,
 		"verify_bing":             st.VerifyBing,
 		"verify_yandex":           st.VerifyYandex,
 		"verify_other":            plainValue(st.OtherMeta()),
 		"verify_files":            plainValue(st.Files()),
 		"csv_delimiter":           st.CSVDelimiter,
+		"breadcrumb_home":         st.BreadcrumbHome,
+		"publisher_type":          st.PublisherType,
+		"publisher_name":          st.PublisherName,
+		"publisher_logo":          st.PublisherLogo,
+		"publisher_url":           st.PublisherURL,
+		"publisher_email":         st.PublisherEmail,
+		"publisher_phone":         st.PublisherPhone,
+		"publisher_same_as":       st.PublisherSameAs,
+		"local_enabled":           st.LocalEnabled,
+		"local_type":              st.LocalType,
+		"local_price_range":       st.LocalPriceRange,
+		"local_street":            st.LocalStreet,
+		"local_postal_code":       st.LocalPostalCode,
+		"local_city":              st.LocalCity,
+		"local_region":            st.LocalRegion,
+		"local_country":           st.LocalCountry,
+		"local_latitude":          st.LocalLatitude,
+		"local_longitude":         st.LocalLongitude,
+		"local_hours":             plainValue(st.Hours()),
+		"page_form":               st.PageForm,
+		"page_segment":            st.PageSegment,
+		"page_title_suffix":       st.PageTitleSuffix,
+		"page_description_suffix": st.PageDescriptionSuffix,
+		"crawl_schedule":          st.CrawlSchedule,
+		"crawl_max_pages":         st.CrawlMaxPages,
+		"crawl_delay":             st.CrawlDelay,
+		"crawl_keep":              st.CrawlKeep,
+		"generic_anchors":         st.GenericAnchors,
+		"autolink_max":            st.AutolinkMax,
 	}
 }
 
@@ -89,42 +186,54 @@ func SetSettingsValue(st *models.Settings, column string, v any) error {
 		}
 		return json.Unmarshal(raw, dst)
 	}
+	if field, ok := settingsText(st)[column]; ok {
+		var err error
+		*field, err = text()
+		return err
+	}
+	if field, ok := settingsInts(st)[column]; ok {
+		switch n := v.(type) {
+		case float64:
+			if n != float64(int(n)) {
+				return fmt.Errorf("golem15.seo: settings %s is not a whole number", column)
+			}
+			*field = int(n)
+		case int:
+			*field = n
+		case string:
+			i, err := strconv.Atoi(strings.TrimSpace(n))
+			if err != nil {
+				return fmt.Errorf("golem15.seo: settings %s is not a number", column)
+			}
+			*field = i
+		default:
+			return fmt.Errorf("golem15.seo: settings %s is not a number", column)
+		}
+		return nil
+	}
 	var err error
 	switch column {
-	case "site_name":
-		st.SiteName, err = text()
-	case "twitter_site":
-		st.TwitterSite, err = text()
-	case "title_template":
-		st.TitleTemplate, err = text()
-	case "home_template":
-		st.HomeTemplate, err = text()
-	case "default_description":
-		st.DefaultDescription, err = text()
-	case "default_og_image":
-		st.DefaultOGImage, err = text()
-	case "noindex_paths":
-		st.NoindexPaths, err = text()
-	case "search_paths":
-		st.SearchPaths, err = text()
-	case "url_host":
-		st.URLHost, err = text()
-	case "url_slash":
-		st.URLSlash, err = text()
-	case "verify_google":
-		st.VerifyGoogle, err = text()
-	case "verify_bing":
-		st.VerifyBing, err = text()
-	case "verify_yandex":
-		st.VerifyYandex, err = text()
-	case "csv_delimiter":
-		st.CSVDelimiter, err = text()
 	case "url_lowercase":
 		b, ok := v.(bool)
 		if !ok {
 			return fmt.Errorf("golem15.seo: settings %s is not a bool", column)
 		}
 		st.URLLowercase = b
+	case "links_new_tab", "local_enabled":
+		b, ok := v.(bool)
+		if !ok {
+			return fmt.Errorf("golem15.seo: settings %s is not a bool", column)
+		}
+		if column == "links_new_tab" {
+			st.LinksNewTab = b
+		} else {
+			st.LocalEnabled = b
+		}
+	case "local_hours":
+		var out []models.OpeningHours
+		if err = list(&out); err == nil {
+			st.LocalHours = lagoon.Jsonable[[]models.OpeningHours]{Data: out, Valid: true}
+		}
 	case "query_params":
 		var out []models.QueryParam
 		if err = list(&out); err == nil {
@@ -202,12 +311,23 @@ func (s *Service) SaveSettings(ctx context.Context, tx *gorm.DB, in SettingsWrit
 	oneOf("url_host", row.URLHost, models.HostKeep, models.HostWWW, models.HostBare)
 	oneOf("url_slash", row.URLSlash, models.SlashKeep, models.SlashAdd, models.SlashDrop)
 	oneOf("csv_delimiter", row.CSVDelimiter, "", ",", ";")
-	if _, sent := in.Values["default_og_image"]; sent && !ValidImageURL(row.DefaultOGImage) {
+	for _, column := range []string{"default_og_image", "publisher_logo"} {
-		problems["default_og_image"] = append(problems["default_og_image"], models.Problem{Key: errImageURL})
+		value := row.DefaultOGImage
+		if column == "publisher_logo" {
+			value = row.PublisherLogo
+		}
+		if _, sent := in.Values[column]; sent && !ValidImageURL(value) {
+			problems[column] = append(problems[column], models.Problem{Key: errImageURL})
+		}
 	}
 	for field, list := range row.Problems() {
 		problems[field] = append(problems[field], list...)
 	}
+	for field, list := range row.AuditProblems() {
+		if _, sent := in.Values[field]; sent {
+			problems[field] = append(problems[field], list...)
+		}
+	}
 	types, err := LoadTypeSettings(tx)
 	if err != nil {
 		return models.Settings{}, err
@@ -256,6 +376,8 @@ func (s *Service) SaveSettings(ctx context.Context, tx *gorm.DB, in SettingsWrit
 	if len(problems) > 0 {
 		return models.Settings{}, &ProblemsError{Problems: problems}
 	}
+	row.NofollowDomains = models.NormalizeDomains(row.NofollowDomains)
+	row.SponsoredDomains = models.NormalizeDomains(row.SponsoredDomains)
 	now := time.Now().UTC()
 	row.UpdatedAt = &now
 	if err := tx.Save(&row).Error; err != nil {

classes/site.go

@@ -0,0 +1,166 @@
+package classes
+
+import (
+	"context"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/sm-sitemap-plugin/contract"
+)
+
+// SiteAnswer is the answer of GET /_seo/api/v1/site (P33-17): the facts
+// about the whole site the frontend layer needs, all public. SiteURL,
+// DefaultLocale, Locales and UnprefixedLocale are the resolver's; Links
+// the settings' new-tab switch; HTMLSitemap the sitemap plugin's HTML
+// sitemap settings; Routes each content type's layered URL pattern for
+// the layer's drift check; RootFiles the file names the layer serves at
+// the site root (verification files, the IndexNow key file); LLMS whether
+// /llms.txt is served; Pagination the site's page address form and
+// HomeCrumb the home crumb (33-06, D-40, D-86).
+type SiteAnswer struct {
+	SiteURL          string          `json:"site_url"`
+	DefaultLocale    string          `json:"default_locale"`
+	Locales          []string        `json:"locales"`
+	UnprefixedLocale string          `json:"unprefixed_locale"`
+	Links            SiteLinks       `json:"links"`
+	HTMLSitemap      SiteHTMLSitemap `json:"html_sitemap"`
+	Routes           []SiteRoute     `json:"routes"`
+	RootFiles        []string        `json:"root_files"`
+	LLMS             bool            `json:"llms"`
+	Pagination       SitePagination  `json:"pagination"`
+	HomeCrumb        SiteHomeCrumb   `json:"home_crumb"`
+}
+
+// SitePagination is the site answer's pagination block: the form (query
+// or path), the query parameter and the path segment per locale, so the
+// layer builds numbered links and checks drift.
+type SitePagination struct {
+	Form     string            `json:"form"`
+	Param    string            `json:"param"`
+	Segments map[string]string `json:"segments"`
+}
+
+// SiteHomeCrumb is the home crumb: its name in the default locale, the
+// home URL and the name per locale.
+type SiteHomeCrumb struct {
+	Name  string            `json:"name"`
+	URL   string            `json:"url"`
+	Names map[string]string `json:"names"`
+}
+
+// SiteLinks is the site answer's links block.
+type SiteLinks struct {
+	NewTab bool `json:"new_tab"`
+}
+
+// SiteHTMLSitemap is the site answer's HTML sitemap block: whether the
+// sitemap plugin serves it, its path per locale and its page size.
+type SiteHTMLSitemap struct {
+	Enabled  bool              `json:"enabled"`
+	Paths    map[string]string `json:"paths"`
+	PageSize int               `json:"page_size"`
+}
+
+// SiteRoute is one content type's URL pattern.
+type SiteRoute struct {
+	Type    string `json:"type"`
+	Pattern string `json:"pattern"`
+}
+
+// listing returns the sitemap plugin's published contract.Listing, nil
+// when none is published.
+func (s *Service) listing() contract.Listing {
+	if s == nil || s.app == nil {
+		return nil
+	}
+	l, ok := s.app.Lookup[contract.Listing]()
+	if !ok {
+		return nil
+	}
+	return l
+}
+
+// Site builds the site answer. It names only public facts: no token, no
+// unpublished data (T-33-19).
+func (s *Service) Site(ctx context.Context) (SiteAnswer, error) {
+	out := SiteAnswer{
+		SiteURL:          s.SiteURL(),
+		UnprefixedLocale: s.UnprefixedLocale(),
+		HTMLSitemap:      SiteHTMLSitemap{Paths: map[string]string{}},
+		Routes:           []SiteRoute{},
+		RootFiles:        []string{},
+	}
+	db, err := s.DB(ctx)
+	if err != nil {
+		return out, err
+	}
+	settings, err := LoadSettings(db)
+	if err != nil {
+		return out, err
+	}
+	robots, err := LoadRobots(db)
+	if err != nil {
+		return out, err
+	}
+	out.DefaultLocale = s.DefaultLocale(ctx, db)
+	out.Locales = s.Locales(ctx, db)
+	out.Links.NewTab = settings.LinksNewTab
+	out.LLMS = robots.LlmsEnabled
+	pages := s.PageConfigOf(ctx, db, settings)
+	out.Pagination = SitePagination{Form: pages.Form, Param: PageParam, Segments: pages.Segments}
+	out.HomeCrumb = SiteHomeCrumb{URL: out.SiteURL + "/", Names: map[string]string{}}
+	for _, locale := range out.Locales {
+		out.HomeCrumb.Names[locale] = s.HomeCrumbName(ctx, db, settings, locale)
+	}
+	out.HomeCrumb.Name = out.HomeCrumb.Names[out.DefaultLocale]
+	if reg := s.Registry(); reg != nil {
+		for _, typ := range reg.ContentTypes() {
+			if typ.Key == PageTypeKey {
+				continue
+			}
+			out.Routes = append(out.Routes, SiteRoute{Type: typ.Key, Pattern: reg.Pattern(typ.Key, "")})
+		}
+	}
+	seen := map[string]bool{}
+	addFile := func(name string) {
+		name = strings.TrimSpace(name)
+		if name != "" && verificationName.MatchString(name) && !seen[name] {
+			seen[name] = true
+			out.RootFiles = append(out.RootFiles, name)
+		}
+	}
+	for _, f := range settings.Files() {
+		addFile(f.Filename)
+	}
+	if l := s.listing(); l != nil {
+		st, err := l.Settings(ctx)
+		if err != nil {
+			return out, err
+		}
+		out.HTMLSitemap.Enabled = st.HTMLEnabled
+		for locale, path := range st.HTMLPaths {
+			if strings.TrimSpace(path) != "" {
+				out.HTMLSitemap.Paths[locale] = path
+			}
+		}
+		out.HTMLSitemap.PageSize = st.HTMLPageSize
+		addFile(st.KeyFile)
+	}
+	return out, nil
+}
+
+// SiteName is the site's name in locale: the settings' site name, else
+// the application name.
+func (s *Service) SiteName(ctx context.Context, settings models.Settings, locale string) string {
+	db, err := s.DB(ctx)
+	name := settings.SiteName
+	if err == nil {
+		name = s.translatedSetting(ctx, db, &settings, "site_name", settings.SiteName, locale)
+	}
+	if strings.TrimSpace(name) == "" {
+		if cfg := s.Config(); cfg != nil {
+			name = cfg.String("app.name")
+		}
+	}
+	return strings.TrimSpace(name)
+}

classes/sitemap_filter.go

@@ -0,0 +1,332 @@
+package classes
+
+import (
+	"context"
+	"net/url"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/sm-sitemap-plugin/contract"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
+)
+
+// SitemapFilter is the SEO plugin's contract.Filter for the sitemap plugin
+// (D-95, D-24): an address the frontend reports noindex (the route
+// report's noindex paths), whose effective SEO is noindex, whose canonical
+// URL points elsewhere, or that redirects leaves the XML and HTML sitemaps
+// with the reason noindex, canonical or redirect. It decides on the
+// production values: the non-production lock (every page noindex outside
+// production, D-22) never excludes an address, so staging and production
+// list the same set. Service is asked on every call, because the filter is
+// published in Register, before the service exists.
+type SitemapFilter struct {
+	Service func() *Service
+}
+
+var _ contract.Filter = SitemapFilter{}
+
+// filterBatch is what one Decide call reads once for all its candidates.
+type filterBatch struct {
+	svc      *Service
+	resolver *Resolver
+	db       *gorm.DB
+	settings models.Settings
+	opts     NormalizeOptions
+	set      *RedirectSet
+	site     string
+	noindex  map[string]bool
+}
+
+// Decide answers one Decision per candidate, in order. Without a booted
+// service every address stays listed.
+func (f SitemapFilter) Decide(ctx context.Context, batch []contract.Candidate) ([]contract.Decision, error) {
+	out := make([]contract.Decision, len(batch))
+	var svc *Service
+	if f.Service != nil {
+		svc = f.Service()
+	}
+	if svc == nil || len(batch) == 0 {
+		return out, nil
+	}
+	b, err := svc.newFilterBatch(ctx)
+	if err != nil {
+		return nil, err
+	}
+	var pageIDs []string
+	for _, c := range batch {
+		if c.TypeKey == PageTypeKey {
+			pageIDs = append(pageIDs, c.RecordID)
+		}
+	}
+	kinds, err := pageKinds(b.db, pageIDs)
+	if err != nil {
+		return nil, err
+	}
+	for i, c := range batch {
+		d, err := b.decide(ctx, c)
+		if err != nil {
+			return nil, err
+		}
+		if c.TypeKey == PageTypeKey && !d.Exclude {
+			d.Group = PageGroup(kinds[c.RecordID])
+		}
+		out[i] = d
+	}
+	return out, nil
+}
+
+// Groups lists the file groups the decisions name: the registry's page
+// kinds (pages, offers, services, landing-pages), titled in every locale
+// of the site with the kind's name (P33-18).
+func (f SitemapFilter) Groups(ctx context.Context) ([]contract.Group, error) {
+	var svc *Service
+	if f.Service != nil {
+		svc = f.Service()
+	}
+	if svc == nil {
+		return nil, nil
+	}
+	db, err := svc.DB(ctx)
+	if err != nil {
+		return nil, err
+	}
+	locales := svc.Locales(ctx, db)
+	tr := svc.translator()
+	out := make([]contract.Group, 0, len(models.PageKinds))
+	for _, kind := range models.PageKinds {
+		titles := map[string]string{}
+		key := PageKindLabels[kind]
+		for _, locale := range locales {
+			if tr == nil {
+				titles[locale] = key
+				continue
+			}
+			titles[locale] = tr.GetIn(locale, key, nil)
+		}
+		out = append(out, contract.Group{Key: PageGroup(kind), Titles: titles})
+	}
+	return out, nil
+}
+
+// newFilterBatch reads the settings, the redirects and the noindex paths
+// of the route report once for a batch.
+func (s *Service) newFilterBatch(ctx context.Context) (*filterBatch, error) {
+	db, err := s.DB(ctx)
+	if err != nil {
+		return nil, err
+	}
+	settings, err := LoadSettings(db)
+	if err != nil {
+		return nil, err
+	}
+	set, err := LoadRedirectSet(db)
+	if err != nil {
+		return nil, err
+	}
+	var paths []string
+	if err := db.Model(&models.NoindexPath{}).Pluck("path", &paths).Error; err != nil {
+		return nil, err
+	}
+	noindex := make(map[string]bool, len(paths))
+	for _, p := range paths {
+		noindex[p] = true
+	}
+	return &filterBatch{
+		svc:      s,
+		resolver: &Resolver{Service: s, production: true},
+		db:       db,
+		settings: settings,
+		opts:     NormalizeOptionsOf(settings),
+		set:      set,
+		site:     s.SiteURL(),
+		noindex:  noindex,
+	}, nil
+}
+
+// decide is the decision of one candidate.
+func (b *filterBatch) decide(ctx context.Context, c contract.Candidate) (contract.Decision, error) {
+	d, _, err := b.decideTarget(ctx, c)
+	return d, err
+}
+
+// decideTarget is the decision of one candidate and, for a canonical or a
+// redirect, the address the sitemaps would list instead: a redirected
+// address first (it is no page), then a path the frontend's route report
+// names noindex (P33-58), then the effective SEO's noindex, then a
+// canonical URL elsewhere.
+func (b *filterBatch) decideTarget(ctx context.Context, c contract.Candidate) (contract.Decision, string, error) {
+	path, query, rawQuery, ok := b.candidatePath(c.URL)
+	if !ok {
+		return contract.Decision{}, "", nil
+	}
+	norm, changed := Normalize(&url.URL{Path: path, RawQuery: rawQuery}, b.opts)
+	if changed {
+		return contract.Decision{Exclude: true, Reason: contract.ReasonRedirect}, joinQuery(norm.Path, norm.RawQuery), nil
+	}
+	chain, err := b.set.Follow(b.site, norm.Path, norm.RawQuery, MaxRedirectHops)
+	if err != nil {
+		return contract.Decision{}, "", err
+	}
+	if len(chain.Hops) > 0 {
+		return contract.Decision{Exclude: true, Reason: contract.ReasonRedirect}, displayURL(b.site, chain.Final), nil
+	}
+	if b.noindex[norm.Path] {
+		return contract.Decision{Exclude: true, Reason: contract.ReasonNoindex}, "", nil
+	}
+	key := ""
+	var match *pact.ContentMatchResult
+	if reg := b.svc.Registry(); reg != nil && c.RecordID != "" {
+		if typ, ok := reg.ContentType(c.TypeKey); ok && typ.Content != nil {
+			key = c.TypeKey
+			match = &pact.ContentMatchResult{ID: c.RecordID, Locale: c.Locale, Exact: true}
+		}
+	}
+	res, err := b.resolver.resolveAt(ctx, b.db, b.settings, path, query, rawQuery, key, match, nil, true, nil)
+	if err != nil {
+		return contract.Decision{}, "", err
+	}
+	if res.SEO == nil {
+		return contract.Decision{}, "", nil
+	}
+	if r, err := ParseRobots(res.SEO.Robots); err == nil && r.Index == "noindex" {
+		return contract.Decision{Exclude: true, Reason: contract.ReasonNoindex}, "", nil
+	}
+	if !sameAddress(res.SEO.Canonical, res.URL) {
+		return contract.Decision{Exclude: true, Reason: contract.ReasonCanonical}, displayURL(b.site, res.SEO.Canonical), nil
+	}
+	return contract.Decision{}, "", nil
+}
+
+// SitemapExclusion is why a record's address is left out of the sitemaps:
+// a contract reason and, for canonical and redirect, the address listed
+// instead.
+type SitemapExclusion struct {
+	Locale string
+	Reason string
+	Target string
+}
+
+// RecordSitemapExclusions lists why the sitemaps leave out the addresses
+// of record id of content type key, one per reason (the first locale that
+// has it): the type switched off in the sitemap settings (read through
+// contract.Listing), else the SEO plugin's own decision on each address
+// (noindex, canonical elsewhere, redirect). Nothing for an unpublished
+// record: the sitemaps never list it.
+func (s *Service) RecordSitemapExclusions(ctx context.Context, key, id string) ([]SitemapExclusion, error) {
+	reg := s.Registry()
+	if reg == nil {
+		return nil, nil
+	}
+	rec, err := reg.Describe(ctx, key, id, s.UnprefixedLocale())
+	if err != nil || rec == nil || !rec.Published {
+		return nil, err
+	}
+	if l := s.listing(); l != nil {
+		st, err := l.Settings(ctx)
+		if err != nil {
+			return nil, err
+		}
+		for _, off := range st.TypesOff {
+			if off == key {
+				return []SitemapExclusion{{Reason: contract.ReasonType}}, nil
+			}
+		}
+	}
+	b, err := s.newFilterBatch(ctx)
+	if err != nil {
+		return nil, err
+	}
+	var out []SitemapExclusion
+	seen := map[string]bool{}
+	for _, l := range rec.Locales {
+		if strings.TrimSpace(l.URL) == "" {
+			continue
+		}
+		d, target, err := b.decideTarget(ctx, contract.Candidate{TypeKey: key, RecordID: id, Locale: l.Locale, URL: l.URL})
+		if err != nil {
+			return nil, err
+		}
+		if d.Exclude && !seen[d.Reason] {
+			seen[d.Reason] = true
+			out = append(out, SitemapExclusion{Locale: l.Locale, Reason: d.Reason, Target: target})
+		}
+	}
+	return out, nil
+}
+
+// SitemapListed is the set of addresses the sitemaps list, read through
+// contract.Listing: their paths with the query, as the owner of a URL
+// names them. ok is false without a published Listing. At most
+// MaxListedAddresses are read.
+func (s *Service) SitemapListed(ctx context.Context) (map[string]bool, bool, error) {
+	l := s.listing()
+	if l == nil {
+		return nil, false, nil
+	}
+	st, err := l.Settings(ctx)
+	if err != nil {
+		return nil, false, err
+	}
+	out := map[string]bool{}
+	read := 0
+	for _, g := range st.Groups {
+		after := ""
+		for read < MaxListedAddresses {
+			page, err := l.Entries(ctx, contract.EntriesQuery{Group: g.Key, After: after, Limit: 1000})
+			if err != nil {
+				return nil, false, err
+			}
+			for _, e := range page.Entries {
+				read++
+				if u, err := url.Parse(e.URL); err == nil {
+					path := u.Path
+					if path == "" {
+						path = "/"
+					}
+					out[joinQuery(path, u.RawQuery)] = true
+				}
+			}
+			if page.Next == "" || page.Next == after {
+				break
+			}
+			after = page.Next
+		}
+	}
+	return out, true, nil
+}
+
+// MaxListedAddresses caps the sitemap addresses SitemapListed reads.
+const MaxListedAddresses = 100000
+
+// candidatePath turns a candidate's address (a path, or an absolute URL on
+// the site or its www twin) into the decoded path and its query; ok is
+// false for an address the plugin cannot judge (another host, malformed).
+func (b *filterBatch) candidatePath(raw string) (string, url.Values, string, bool) {
+	raw = strings.TrimSpace(raw)
+	if raw == "" {
+		return "", nil, "", false
+	}
+	if u, err := url.Parse(raw); err == nil && u.IsAbs() && !sameSite(b.site, u.Host) {
+		return "", nil, "", false
+	}
+	path, query, err := parseTarget(raw)
+	if err != nil {
+		return "", nil, "", false
+	}
+	return path, query, rawQueryOf(raw), true
+}
+
+// sameAddress reports whether two absolute URLs name the same page: the
+// host without letter case, the path and the query as written.
+func sameAddress(a, b string) bool {
+	if a == b {
+		return true
+	}
+	ua, errA := url.Parse(strings.TrimSpace(a))
+	ub, errB := url.Parse(strings.TrimSpace(b))
+	if errA != nil || errB != nil {
+		return false
+	}
+	return strings.EqualFold(ua.Host, ub.Host) && ua.EscapedPath() == ub.EscapedPath() && ua.RawQuery == ub.RawQuery
+}

classes/sitemap_filter_test.go

@@ -0,0 +1,64 @@
+package classes
+
+import (
+	"context"
+	"testing"
+
+	"git.golem15.com/golem15/sm-sitemap-plugin/contract"
+)
+
+// TestCandidatePath pins the addresses the filter judges: a path and an
+// absolute URL on the site or its www twin (path and query split), and
+// not another host, a non-http scheme or a blank address. The reasons per
+// record are TestFilterReasons' on the database.
+func TestCandidatePath(t *testing.T) {
+	b := &filterBatch{site: "https://example.test"}
+	for _, tc := range []struct {
+		raw, path, query string
+		ok               bool
+	}{
+		{"/blog/a", "/blog/a", "", true},
+		{"/blog?page=2", "/blog", "page=2", true},
+		{"https://example.test/kontakt", "/kontakt", "", true},
+		{"https://www.example.test/kontakt?x=1", "/kontakt", "x=1", true},
+		{"https://other.test/kontakt", "", "", false},
+		{"ftp://example.test/a", "", "", false},
+		{"  ", "", "", false},
+	} {
+		path, _, raw, ok := b.candidatePath(tc.raw)
+		if ok != tc.ok || (ok && (path != tc.path || raw != tc.query)) {
+			t.Errorf("candidatePath(%q) = %q %q %v, want %q %q %v", tc.raw, path, raw, ok, tc.path, tc.query, tc.ok)
+		}
+	}
+}
+
+// TestSameAddress pins the comparison of a canonical with the page's
+// address: the host without case, the path and the query as written.
+func TestSameAddress(t *testing.T) {
+	for _, tc := range []struct {
+		a, b string
+		want bool
+	}{
+		{"https://example.test/a", "https://example.test/a", true},
+		{"https://EXAMPLE.test/a", "https://example.test/a", true},
+		{"https://example.test/a", "https://example.test/A", false},
+		{"https://example.test/a?x=1", "https://example.test/a?x=2", false},
+		{"https://example.test/a", "https://www.example.test/a", false},
+		{"https://example.test/a%20b", "https://example.test/a b", true},
+		{"%zz", "https://example.test/", false},
+	} {
+		if got := sameAddress(tc.a, tc.b); got != tc.want {
+			t.Errorf("sameAddress(%q, %q) = %v, want %v", tc.a, tc.b, got, tc.want)
+		}
+	}
+}
+
+// TestFilterWithoutService pins the fail-open filter: without a booted
+// service every candidate stays listed, one decision each, in order.
+func TestFilterWithoutService(t *testing.T) {
+	f := SitemapFilter{Service: func() *Service { return nil }}
+	got, err := f.Decide(context.Background(), []contract.Candidate{{URL: "/a"}, {URL: "/b"}})
+	if err != nil || len(got) != 2 || got[0].Exclude || got[1].Exclude {
+		t.Fatalf("decisions = %+v err=%v", got, err)
+	}
+}

classes/structured.go

@@ -0,0 +1,442 @@
+package classes
+
+import (
+	"context"
+	"encoding/json"
+	"strconv"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes/schema"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/cabana"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
+)
+
+// PhraseIn renders a phrase key in locale; the key itself when the
+// application has no translator.
+func (s *Service) PhraseIn(locale, key string, params map[string]string) string {
+	tr := s.translator()
+	if tr == nil {
+		return key
+	}
+	return tr.GetIn(locale, key, params)
+}
+
+// SearchURL is the configured site search address, "" when unset.
+func (s *Service) SearchURL() string {
+	if cfg := s.Config(); cfg != nil {
+		return strings.TrimSpace(cfg.String(SearchURLConfigKey))
+	}
+	return ""
+}
+
+// SameAsLines splits a sameAs list into its trimmed, non-empty lines.
+func SameAsLines(list string) []string {
+	var out []string
+	for _, line := range strings.Split(strings.ReplaceAll(list, "\r\n", "\n"), "\n") {
+		if v := strings.TrimSpace(line); v != "" {
+			out = append(out, v)
+		}
+	}
+	return out
+}
+
+// PublisherOf is the settings' publisher in locale (D-45): its name is the
+// translated publisher name (empty: the site name, filled by the builder),
+// URLs absolute on the site, and the local business data only for an
+// organisation with the switch on.
+func (s *Service) PublisherOf(ctx context.Context, db *gorm.DB, settings models.Settings, locale string) schema.Publisher {
+	site := s.SiteURL()
+	p := schema.Publisher{
+		Type:  schema.PublisherOrganization,
+		Name:  strings.TrimSpace(s.translatedSetting(ctx, db, &settings, "publisher_name", settings.PublisherName, locale)),
+		Email: strings.TrimSpace(settings.PublisherEmail),
+		Phone: strings.TrimSpace(settings.PublisherPhone),
+	}
+	if settings.PublisherType == models.PublisherPerson {
+		p.Type = schema.PublisherPerson
+	}
+	if logo := strings.TrimSpace(settings.PublisherLogo); logo != "" {
+		p.Logo = AbsoluteURL(site, logo)
+	}
+	if u := strings.TrimSpace(settings.PublisherURL); u != "" {
+		p.URL = AbsoluteURL(site, u)
+	}
+	p.SameAs = SameAsLines(settings.PublisherSameAs)
+	if settings.LocalEnabled && p.Type == schema.PublisherOrganization {
+		l := &schema.Local{
+			Type:       strings.TrimSpace(settings.LocalType),
+			PriceRange: strings.TrimSpace(settings.LocalPriceRange),
+			Street:     strings.TrimSpace(settings.LocalStreet),
+			PostalCode: strings.TrimSpace(settings.LocalPostalCode),
+			City:       strings.TrimSpace(settings.LocalCity),
+			Region:     strings.TrimSpace(settings.LocalRegion),
+			Country:    strings.TrimSpace(settings.LocalCountry),
+		}
+		if lat, err := strconv.ParseFloat(strings.TrimSpace(settings.LocalLatitude), 64); err == nil {
+			if lng, err := strconv.ParseFloat(strings.TrimSpace(settings.LocalLongitude), 64); err == nil {
+				l.Latitude, l.Longitude = &lat, &lng
+			}
+		}
+		for _, h := range settings.Hours() {
+			l.Hours = append(l.Hours, schema.Hours{Day: h.Day, Opens: h.Opens, Closes: h.Closes})
+		}
+		p.Local = l
+	}
+	return p
+}
+
+// pageVariant is the registry kind of a registry page, "" for other types.
+func pageVariant(db *gorm.DB, key, id string) string {
+	if key != PageTypeKey || id == "" {
+		return ""
+	}
+	kinds, err := pageKinds(db, []string{id})
+	if err != nil {
+		return ""
+	}
+	return kinds[id]
+}
+
+// structuredIn holds what resolveAt hands to the structured data.
+type structuredIn struct {
+	key      string
+	id       string
+	rec      *pact.ContentRecord
+	entry    models.Entry
+	content  pact.ContentLocale
+	locale   string
+	path     string
+	url      string
+	home     bool
+	siteName string
+	seo      *SEO
+	// schemaValue replaces the stored seo_schema value when set (the
+	// admin preview of unsaved changes, the MCP graph tool).
+	schemaValue *string
+	// sources is the answer's value sources (the schema type's is set).
+	sources map[string]string
+}
+
+// schemaRaw is the seo_schema value a graph is built with.
+func (in structuredIn) schemaRaw() string {
+	if in.schemaValue != nil {
+		return *in.schemaValue
+	}
+	return entryData(in.entry, DataSchema)
+}
+
+// graphInput composes the builder's input of a page.
+func (s *Service) graphInput(ctx context.Context, db *gorm.DB, settings models.Settings, in structuredIn, crumbs []schema.Crumb) schema.Input {
+	site := s.SiteURL()
+	name := strings.TrimSpace(in.content.Title)
+	if name == "" && in.home {
+		name = in.siteName
+	}
+	if name == "" && in.seo != nil {
+		name = in.seo.Title
+	}
+	gi := schema.Input{
+		Site:      site,
+		URL:       in.url,
+		Lang:      LangTag(in.locale),
+		Home:      in.home,
+		Name:      name,
+		Headline:  strings.TrimSpace(in.content.Title),
+		Crumbs:    crumbs,
+		SiteName:  in.siteName,
+		SearchURL: s.SearchURL(),
+		Publisher: s.PublisherOf(ctx, db, settings, in.locale),
+		Words:     in.content.Words,
+	}
+	if in.seo != nil {
+		gi.Description = in.seo.Description
+	}
+	if value, err := schema.ParseValue(in.schemaRaw()); err == nil {
+		gi.Type, gi.Service = value.Type, value.Fields
+		if value.Type != "" && in.sources != nil {
+			in.sources["schema"] = SourceRecord
+		}
+	}
+	if in.key != "" {
+		if typ, ok := s.Registry().ContentType(in.key); ok && typ.Content != nil {
+			gi.Kind = typ.Content.Kind
+		}
+		gi.Variant = pageVariant(db, in.key, in.id)
+	}
+	seen := map[string]bool{}
+	addImage := func(u string) {
+		if u = strings.TrimSpace(u); u == "" {
+			return
+		}
+		u = AbsoluteURL(site, u)
+		if !seen[u] {
+			seen[u] = true
+			gi.Images = append(gi.Images, u)
+		}
+	}
+	for _, img := range in.content.Images {
+		if !img.Decorative {
+			addImage(img.URL)
+		}
+	}
+	addImage(in.content.Image)
+	if len(gi.Images) == 0 && in.seo != nil {
+		addImage(in.seo.OG.Image)
+	}
+	if rec := in.rec; rec != nil {
+		gi.PublishedAt, gi.ModifiedAt = rec.PublishedAt, rec.LastMod
+		for _, a := range rec.Authors {
+			p := schema.Person{Name: a.Name}
+			if a.Ref.Type != "" {
+				if l, ok := s.describeLocale(ctx, a.Ref, in.locale); ok {
+					p.URL = AbsoluteURL(site, l.URL)
+				}
+			}
+			gi.Authors = append(gi.Authors, p)
+		}
+		for _, v := range in.content.Videos {
+			gi.Videos = append(gi.Videos, schema.Video{Provider: v.Provider, ID: v.ID})
+		}
+	}
+	if len(crumbs) >= 3 {
+		gi.Section = crumbs[len(crumbs)-2].Name
+	}
+	return gi
+}
+
+// describeLocale describes a referenced record in locale.
+func (s *Service) describeLocale(ctx context.Context, ref pact.ContentRef, locale string) (pact.ContentLocale, bool) {
+	reg := s.Registry()
+	if reg == nil {
+		return pact.ContentLocale{}, false
+	}
+	rec, err := reg.Describe(ctx, ref.Type, ref.ID, s.UnprefixedLocale())
+	if err != nil || rec == nil || !rec.Published {
+		return pact.ContentLocale{}, false
+	}
+	return localeOf(rec, locale)
+}
+
+// structuredData fills the structured keys of an ok answer: html_lang,
+// og:locale and its alternates, the hreflang alternates, the breadcrumb
+// trail and the JSON-LD graph (D-43, D-55 to D-57, D-86).
+func (s *Service) structuredData(ctx context.Context, db *gorm.DB, settings models.Settings, res *Result, in structuredIn) error {
+	res.HTMLLang = LangTag(in.locale)
+	if res.SEO != nil {
+		res.SEO.OG.Locale = s.OGLocale(in.locale)
+		res.SEO.OG.LocaleAlternates = s.OGLocaleAlternates(in.rec, in.locale)
+	}
+	res.Alternates = Alternates(s.SiteURL(), in.rec, s.DefaultLocale(ctx, db))
+	name := strings.TrimSpace(in.content.Title)
+	if name == "" && res.SEO != nil {
+		name = res.SEO.Title
+	}
+	res.Breadcrumbs = s.Trail(ctx, db, settings, in.rec, entryData(in.entry, DataPrimaryParent), in.locale, in.path, name)
+	graph, merged, err := s.buildGraph(ctx, db, settings, in, res.Breadcrumbs)
+	if err != nil {
+		return err
+	}
+	res.JSONLD, res.Merged = graph, merged
+	return nil
+}
+
+// buildGraph builds a page's graph with its custom JSON-LD merged in
+// (D-44); merged names the generated values the custom nodes replace. A
+// stored custom value that no longer decodes is left out.
+func (s *Service) buildGraph(ctx context.Context, db *gorm.DB, settings models.Settings, in structuredIn, crumbs []schema.Crumb) (map[string]any, []schema.Merged, error) {
+	graph, err := schema.Build(ctx, s.graphInput(ctx, db, settings, in, crumbs))
+	if err != nil {
+		return nil, nil, err
+	}
+	value, err := schema.ParseValue(in.schemaRaw())
+	if err != nil || value.Custom == "" {
+		return graph, nil, nil
+	}
+	custom, err := schema.DecodeCustom(value.Custom)
+	if err != nil || len(custom) == 0 {
+		return graph, nil, nil
+	}
+	graph, merged := schema.Merge(graph, custom)
+	return graph, merged, nil
+}
+
+// PublisherPreview is the view model of the settings' P8 partial: the
+// generated publisher node as indented JSON and its findings as lines, or
+// the all-good line.
+type PublisherPreview struct {
+	JSON     string
+	Findings []string
+	AllGood  string
+}
+
+// FindingText renders a finding with the graph copy of the request locale.
+func (s *Service) FindingText(ctx context.Context, typ string, f schema.Finding) string {
+	key := "golem15.seo::lang.graph.recommended"
+	if f.Level == schema.LevelRequired {
+		key = "golem15.seo::lang.graph.required"
+	}
+	return s.PhraseCtx(ctx, key, map[string]string{"property": f.Property, "type": typ})
+}
+
+// PublisherPreview builds P8 from the stored settings in the default
+// locale (UI-SPEC C20).
+func (s *Service) PublisherPreview(ctx context.Context) (PublisherPreview, error) {
+	db, err := s.DB(ctx)
+	if err != nil {
+		return PublisherPreview{}, err
+	}
+	settings, err := LoadSettings(db)
+	if err != nil {
+		return PublisherPreview{}, err
+	}
+	locale := s.DefaultLocale(ctx, db)
+	node := schema.PublisherNode(s.SiteURL(), s.PublisherOf(ctx, db, settings, locale), s.SiteName(ctx, settings, locale))
+	out := PublisherPreview{AllGood: s.PhraseCtx(ctx, "golem15.seo::lang.graph.all_good", nil)}
+	if node == nil {
+		return out, nil
+	}
+	raw, err := json.MarshalIndent(node, "", "  ")
+	if err != nil {
+		return PublisherPreview{}, err
+	}
+	out.JSON = string(raw)
+	rep := schema.ValidateNode(node)
+	for _, f := range rep.Findings {
+		out.Findings = append(out.Findings, s.FindingText(ctx, rep.Type, f))
+	}
+	return out, nil
+}
+
+// GraphAnswer is the structured data of one page (E9, the MCP graph tool):
+// the locale, the page's absolute URL (nil when it has none yet), the
+// graph and every node with its findings (D-46).
+type GraphAnswer struct {
+	Locale string          `json:"locale"`
+	URL    *string         `json:"url"`
+	Graph  map[string]any  `json:"graph"`
+	Nodes  []schema.Report `json:"nodes"`
+}
+
+// graphAnswer turns a resolve answer into a graph answer.
+func graphAnswer(res Result, locale string) GraphAnswer {
+	out := GraphAnswer{Locale: locale, Nodes: []schema.Report{}}
+	if res.Action != ActionOK || res.JSONLD == nil {
+		return out
+	}
+	u := res.URL
+	out.URL, out.Graph, out.Locale = &u, res.JSONLD, res.Locale
+	out.Nodes = schema.Validate(res.JSONLD)
+	return out
+}
+
+// GraphOfURL builds the graph of a public URL; schemaValue (nil: the
+// stored one) replaces the page's seo_schema value. A URL that is not an
+// ok page answers no graph.
+func (s *Service) GraphOfURL(ctx context.Context, rawURL string, schemaValue *string) (GraphAnswer, Result, error) {
+	r := &Resolver{Service: s, schema: schemaValue}
+	res, err := r.Resolve(ctx, rawURL)
+	if err != nil {
+		return GraphAnswer{}, Result{}, err
+	}
+	return graphAnswer(res, res.Locale), res, nil
+}
+
+// GraphOfRecord builds the graph of a record in locale at its own
+// address, published or not; schemaValue (nil: the stored one) replaces
+// its seo_schema value. A record without an address in locale answers no
+// URL and no graph.
+func (s *Service) GraphOfRecord(ctx context.Context, key, id, locale string, schemaValue *string) (GraphAnswer, Result, error) {
+	r := &Resolver{Service: s, schema: schemaValue}
+	res, ok, err := r.ResolveRecord(ctx, key, id, locale)
+	if err != nil {
+		return GraphAnswer{}, Result{}, err
+	}
+	if !ok {
+		return GraphAnswer{Locale: locale, Nodes: []schema.Report{}}, res, nil
+	}
+	return graphAnswer(res, locale), res, nil
+}
+
+// GraphActionName is the SEO tab's graph preview widget action (E9).
+const GraphActionName = "seo_graph"
+
+// graphPayload is the E9 action payload.
+type graphPayload struct {
+	Op     string `json:"op"`
+	Locale string `json:"locale"`
+}
+
+// RecordGraphAction is the E9 widget action of content type key (UI-SPEC
+// C18): op graph answers the graph of the saved record in locale with the
+// unsaved seo_schema value of the form, and its findings.
+func RecordGraphAction(svc func() *Service, key, label, permission string) pact.AdminAction {
+	return pact.AdminAction{
+		Name:        GraphActionName,
+		Label:       label,
+		Permissions: []string{permission},
+		Run: func(ctx context.Context, in pact.AdminActionInput) (pact.AdminActionResult, error) {
+			s := svc()
+			if s == nil {
+				return pact.AdminActionResult{}, ErrNotBooted
+			}
+			var p graphPayload
+			if len(in.Payload) > 0 {
+				if err := json.Unmarshal(in.Payload, &p); err != nil || (p.Op != "" && p.Op != "graph") {
+					return pact.AdminActionResult{}, &cabana.ValidationError{Details: map[string]any{in.Field: []string{"golem15.seo::lang.api.value_invalid"}}}
+				}
+			}
+			db, err := s.DB(ctx)
+			if err != nil {
+				return pact.AdminActionResult{}, err
+			}
+			locale := p.Locale
+			if locale == "" {
+				locale = s.DefaultLocale(ctx, db)
+			}
+			var override *string
+			if raw, ok := in.Values[FieldPrefix+DataSchema]; ok {
+				text, _ := raw.(string)
+				override = &text
+			}
+			if in.RecordID == nil {
+				return pact.AdminActionResult{Data: GraphAnswer{Locale: locale, Nodes: []schema.Report{}}}, nil
+			}
+			answer, _, err := s.GraphOfRecord(ctx, key, strconv.FormatUint(*in.RecordID, 10), locale, override)
+			return pact.AdminActionResult{Data: answer}, err
+		},
+	}
+}
+
+// SchemaMerges are the schema.merged save notices of a record (UI-SPEC
+// C17): one per generated node whose values its custom JSON-LD replaces,
+// in the default locale.
+func (s *Service) SchemaMerges(ctx context.Context, key, id string) ([]pact.FormFieldNotice, error) {
+	db, err := s.DB(ctx)
+	if err != nil {
+		return nil, err
+	}
+	_, res, err := s.GraphOfRecord(ctx, key, id, s.DefaultLocale(ctx, db), nil)
+	if err != nil {
+		return nil, err
+	}
+	var out []pact.FormFieldNotice
+	for _, m := range res.Merged {
+		out = append(out, pact.FormFieldNotice{Message: "golem15.seo::lang.schema.merged", Params: map[string]string{"props": strings.Join(m.Props, ", "), "node": m.Type}})
+	}
+	return out, nil
+}
+
+// CheckSchema refuses a structured data value the SEO tab of record id of
+// content type key would refuse, as a *ProblemsError on schema.
+func (s *Service) CheckSchema(ctx context.Context, kind, key, id, raw string) error {
+	db, err := s.DB(ctx)
+	if err != nil {
+		return err
+	}
+	if _, _, err := schema.Normalize(kind, pageVariant(db, key, id), raw); err != nil {
+		return &ProblemsError{Problems: map[string][]models.Problem{DataSchema: {SchemaProblem(kind, err)}}}
+	}
+	return nil
+}

classes/undo.go

@@ -471,6 +471,13 @@ func (s *Service) restore(ctx context.Context, tx *gorm.DB, rev models.Revision,
 				w.Bots[token] = b
 			}
 		}
+		if v, ok := rev.Before["llms_enabled"].(bool); ok {
+			w.LlmsEnabled = &v
+		}
+		if v, ok := rev.Before["llms_intro"]; ok {
+			intro := text(v)
+			w.LlmsIntro = &intro
+		}
 		_, _, err := s.SaveRobots(ctx, tx, w)
 		return err
 	case models.SubjectSettings:

commands.go

@@ -2,19 +2,24 @@ package seo
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"strings"
+	"time"
 
 	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/classes/crawl"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/backpack"
 	"git.golem15.com/golem15/summercms/modules/bonfire"
 	"git.golem15.com/golem15/summercms/modules/lagoon"
 	"git.golem15.com/golem15/summercms/modules/pact"
 	"gorm.io/gorm"
 )
 
-// Commands registers seo:generate-entries.
+// Commands registers seo:generate-entries and seo:crawl.
 func (p *Plugin) Commands() []bonfire.Command {
-	return []bonfire.Command{{
+	return []bonfire.Command{p.crawlCommand(), {
 		Name:        "seo:generate-entries",
 		Description: "Queue the job that creates missing SEO entries",
 		Long: `Queues the entry generation job on the background queue. The job walks the
@@ -59,6 +64,94 @@ changed. A worker (summer serve or queue:work) runs the job.`,
 	}}
 }
 
+// crawlCommand is seo:crawl [--scheduled]: it queues a crawl of the site
+// (D-14) for the worker; --scheduled is the nightly entry (P33-40), which
+// queues one only when the Site audit setting crawl_schedule is daily, or
+// weekly on a Monday in app.timezone, and skips while a crawl runs.
+func (p *Plugin) crawlCommand() bonfire.Command {
+	return bonfire.Command{
+		Name:        "seo:crawl",
+		Description: "Queue a crawl of the site for the site audit",
+		Long: `Queues a crawl job on the background queue. The crawler fetches the site's own
+pages over HTTP like a search engine (from the home page, the sitemap index
+and its files, and same-host links; two requests at a time, the Site audit
+settings' pause and page limit) and stores what it finds in SEO › Site audit.
+One crawl runs at a time. With --scheduled, as the nightly schedule runs it,
+the command queues a crawl only when the Site audit setting "Automatic crawl"
+is daily, or weekly and it is Monday in app.timezone, and does nothing while
+a crawl is queued or running. A worker (summer serve or queue:work) runs the
+job.`,
+		Flags: []bonfire.Flag{
+			{Name: "scheduled", Description: "Run as the nightly schedule: follow the Automatic crawl setting", Bare: true},
+		},
+		Examples: []bonfire.Example{
+			{Command: "seo:crawl", Description: "Queue a crawl now"},
+			{Command: "seo:crawl --scheduled", Description: "What the nightly schedule runs"},
+		},
+		Run: func(ctx context.Context, in bonfire.Input, out bonfire.Output) error {
+			svc := p.Service()
+			if svc == nil {
+				return classes.ErrNotBooted
+			}
+			db, done, err := p.commandDB(ctx)
+			if err != nil {
+				return err
+			}
+			defer done()
+			scheduled := bonfire.FlagBool(in, "scheduled")
+			trigger, name := models.TriggerAdmin, "console"
+			if scheduled {
+				st, err := classes.LoadSettings(db)
+				if err != nil {
+					return err
+				}
+				if !crawl.ScheduledDue(st.CrawlSchedule, time.Now().In(appLocation(p.app))) {
+					out.Info("No crawl is due (Automatic crawl: " + st.CrawlSchedule + ").")
+					return nil
+				}
+				trigger, name = models.TriggerSchedule, ""
+			}
+			var row models.Crawl
+			err = lagoon.Transaction(ctx, db, func(ctx context.Context, tx *gorm.DB) error {
+				var err error
+				row, err = crawl.Start(ctx, svc, tx, trigger, name)
+				return err
+			})
+			var running *crawl.AlreadyRunningError
+			switch {
+			case errors.As(err, &running) && scheduled:
+				out.Info(fmt.Sprintf("Crawl #%d is still running; no new crawl was queued.", running.Crawl.ID))
+				return nil
+			case errors.As(err, &running):
+				return fmt.Errorf("Crawl #%d is already running. Wait for it, or cancel it in SEO › Crawls.", running.Crawl.ID)
+			case errors.Is(err, crawl.ErrNoSiteURL):
+				return fmt.Errorf("The site address for crawls is not configured (golem15.seo.site_url).")
+			case err != nil:
+				return err
+			}
+			job := uint(0)
+			if row.JobID != nil {
+				job = *row.JobID
+			}
+			out.Success(fmt.Sprintf("Queued crawl #%d of %s (job #%d).", row.ID, row.Site, job))
+			return nil
+		},
+	}
+}
+
+// appLocation is the app.timezone location, UTC when unset or unknown.
+func appLocation(app *backpack.App) *time.Location {
+	if app == nil || app.Config == nil {
+		return time.UTC
+	}
+	if name := strings.TrimSpace(app.Config.String("app.timezone")); name != "" {
+		if loc, err := time.LoadLocation(name); err == nil {
+			return loc
+		}
+	}
+	return time.UTC
+}
+
 // commandDB returns the app database, opening database.dsn when the binary
 // published none for console commands. done closes a pool it opened.
 func (p *Plugin) commandDB(ctx context.Context) (*gorm.DB, func(), error) {

config/config.yaml

@@ -21,5 +21,47 @@ notfound_token: ""
 notfound_max_rows: 10000
 #
 # robots_sitemaps lists the sitemap addresses robots.txt names, one
-# "Sitemap:" line each, made absolute on the site URL.
+# "Sitemap:" line each, made absolute on the site URL. Unset, robots.txt
-robots_sitemaps: ["/sitemap.xml"]
+# names the sitemap plugin's index (its site address plus /sitemap.xml);
+# the per-type files are never listed, the index names them. Set it only
+# to name other sitemaps:
+# robots_sitemaps: ["/sitemap.xml"]
+#
+# route_report_token is the shared secret the frontend server sends as
+# "Authorization: Bearer <token>" with its route report to
+# POST /_seo/api/v1/registry/routes (the page registry). Empty uses
+# notfound_token; both empty turn the report off (it answers 404).
+route_report_token: ""
+#
+# og_locales maps locale codes to their og:locale values (the resolve
+# answer's seo.og.locale and locale_alternates). Codes not listed use
+# pl_PL, en_US and de_DE for pl, en and de, else the language with its
+# region (pt-BR: pt_BR) or the language doubled (fr: fr_FR):
+# og_locales: {pl: pl_PL, en: en_GB}
+#
+# search_url is the site search address with the {search_term_string}
+# placeholder, for example /szukaj?q={search_term_string}. When set, the
+# WebSite node of the JSON-LD graph carries a SearchAction. Empty means no
+# SearchAction:
+# search_url: "/szukaj?q={search_term_string}"
+#
+# crawl_trusted lets the site audit's crawler fetch plain http addresses
+# and addresses inside the network (a staging or local site). Without it
+# the crawler fetches only https addresses of the site host and its www
+# twin, and refuses private addresses. Only config (or the environment,
+# SUMMER_GOLEM15__SEO__CRAWL_TRUSTED) sets it, never a setting or MCP.
+crawl_trusted: false
+#
+# The site audit can compare each page's raw HTML with the page after
+# JavaScript ran, in a headless Chrome or Chromium that the operator
+# installs on the server (none is installed by default). Set the absolute
+# path of its executable here or through the environment
+# (SUMMER_GOLEM15__SEO__CRAWL_CHROME_PATH); empty skips the comparison and
+# every crawl says so. The path is config only, never a setting or MCP
+# value. Chrome loads only the site host and its www twin (pinned to
+# addresses checked like the crawler's), so third-party scripts are not
+# loaded; every crawl gets a fresh profile, removed afterwards.
+crawl_chrome_path: ""
+#
+# The number of pages rendered per crawl, the first in crawl order.
+crawl_render_max: 200

controllers/admin_registry.go

@@ -14,7 +14,8 @@ import (
 )
 
 // PermissionManage is the one SEO permission (D-15): the SEO tab, URL
-// rules, redirects, the 404 log, robots.txt, settings and history.
+// rules, redirects, the 404 log, pages, auto-links, the site audit,
+// robots.txt, settings and history.
 const PermissionManage = "golem15.seo.manage"
 
 // The admin controller IDs.
@@ -32,6 +33,10 @@ func AdminControllers(app func() *backpack.App) []pact.AdminController {
 		redirectsAdminController{app: app},
 		notFoundAdminController{app: app},
 		rulesAdminController{app: app},
+		pagesAdminController{app: app},
+		autolinksAdminController{app: app},
+		issuesAdminController{app: app},
+		crawlsAdminController{app: app},
 		robotsAdminController{app: app},
 		historyAdminController{app: app},
 		transferAdminController{app: app},

controllers/api/autolinks.go

@@ -0,0 +1,197 @@
+package api
+
+import (
+	"context"
+	"net/http"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"gorm.io/gorm"
+)
+
+// ListAutolinksInput is the query of GET /_seo/api/v1/autolinks.
+type ListAutolinksInput struct {
+	Page    int    `json:"page,omitempty" query:"page" jsonschema:"the page number, from 1"`
+	PerPage int    `json:"per_page,omitempty" query:"per_page" jsonschema:"rows per page, at most 200 (50 by default)"`
+	Search  string `json:"search,omitempty" query:"search" jsonschema:"text the phrase or the address contains"`
+	Locale  string `json:"locale,omitempty" query:"locale" jsonschema:"a content locale code, or all for the phrases of every locale"`
+	Origin  string `json:"origin,omitempty" query:"origin" jsonschema:"manual, keyphrase or mcp"`
+	Active  string `json:"active,omitempty" query:"active" jsonschema:"true: active phrases only; false: inactive ones only"`
+}
+
+// CreateAutolinkInput is the body of POST /_seo/api/v1/autolinks.
+type CreateAutolinkInput struct {
+	Phrase  string  `json:"phrase" jsonschema:"the phrase texts link at its first occurrence (whole words, any letter case), at least 3 characters"`
+	URL     string  `json:"url" jsonschema:"the address it links to: a path such as /grzyby/lwia-grzywa or a full http(s) address"`
+	Locale  *string `json:"locale,omitempty" jsonschema:"the content locale it applies in; empty or all for every locale (the default)"`
+	Enabled *bool   `json:"enabled,omitempty" jsonschema:"whether texts link it (default true)"`
+	Note    *string `json:"note,omitempty" jsonschema:"an internal note"`
+}
+
+// UpdateAutolinkInput is PUT /_seo/api/v1/autolinks/{id}: only the fields
+// sent change.
+type UpdateAutolinkInput struct {
+	ID      uint    `json:"id" path:"id" jsonschema:"the auto-link id"`
+	Phrase  *string `json:"phrase,omitempty" jsonschema:"the phrase, at least 3 characters"`
+	URL     *string `json:"url,omitempty" jsonschema:"a path or a full http(s) address"`
+	Locale  *string `json:"locale,omitempty" jsonschema:"a content locale code; empty or all for every locale"`
+	Enabled *bool   `json:"enabled,omitempty" jsonschema:"whether texts link it"`
+	Note    *string `json:"note,omitempty" jsonschema:"an internal note"`
+}
+
+// DeleteAutolinksInput is the query of DELETE /_seo/api/v1/autolinks.
+type DeleteAutolinksInput struct {
+	IDs string `json:"ids" query:"ids" jsonschema:"the auto-link ids to delete, comma-separated, e.g. 3,4"`
+}
+
+// AutolinksFromKeyphrasesInput is POST /_seo/api/v1/autolinks/from-keyphrases
+// (no fields).
+type AutolinksFromKeyphrasesInput struct{}
+
+// AutolinkJSON is an auto-link in an answer; locale is "" for every
+// locale.
+func AutolinkJSON(a models.Autolink) map[string]any {
+	out := classes.AutolinkValues(a)
+	out["id"] = a.ID
+	out["created_at"] = a.CreatedAt
+	out["updated_at"] = a.UpdatedAt
+	return out
+}
+
+// ListAutolinks serves GET /_seo/api/v1/autolinks: by phrase, filtered by
+// search text, locale, origin and the active state.
+func ListAutolinks(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		db, err := read(ctx, svc)
+		if err != nil {
+			return 0, nil, err
+		}
+		in := listParams(r)
+		q := db.Model(&models.Autolink{}).Order("lower(phrase) ASC, locale ASC, id ASC")
+		if in.Search != "" {
+			p := like(in.Search)
+			q = q.Where("lower(phrase) LIKE ? OR lower(url) LIKE ?", p, p)
+		}
+		query := r.URL.Query()
+		if l := strings.TrimSpace(query.Get("locale")); l != "" {
+			if l == models.AutolinkLocaleAll {
+				l = ""
+			}
+			q = q.Where("locale = ?", l)
+		}
+		if o := strings.TrimSpace(query.Get("origin")); o != "" {
+			q = q.Where("origin = ?", o)
+		}
+		switch strings.TrimSpace(query.Get("active")) {
+		case "true", "1":
+			q = q.Where("enabled")
+		case "false", "0":
+			q = q.Where("NOT enabled")
+		}
+		var rows []models.Autolink
+		total, err := paginate(q, in, &rows)
+		if err != nil {
+			return 0, nil, err
+		}
+		out := make([]map[string]any, 0, len(rows))
+		for _, row := range rows {
+			out = append(out, AutolinkJSON(row))
+		}
+		return http.StatusOK, listBody(out, in, total), nil
+	})
+}
+
+// CreateAutolink serves POST /_seo/api/v1/autolinks: 201 with the phrase,
+// 422 validation_failed for a short phrase, an invalid address or a
+// phrase the locale has (UI-SPEC C22). The origin is mcp through MCP.
+func CreateAutolink(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		var body CreateAutolinkInput
+		if err := decode(r, maxBody, &body); err != nil {
+			return 0, nil, err
+		}
+		row := models.NewAutolink()
+		row.Origin = classes.AutolinkOrigin(ctx)
+		classes.AutolinkWrite{Phrase: &body.Phrase, URL: &body.URL, Locale: body.Locale, Enabled: body.Enabled, Note: body.Note}.Apply(&row)
+		err := write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			return svc.SaveAutolink(ctx, tx, &row)
+		})
+		if err != nil {
+			return 0, nil, err
+		}
+		return http.StatusCreated, map[string]any{"data": AutolinkJSON(row)}, nil
+	})
+}
+
+// UpdateAutolink serves PUT /_seo/api/v1/autolinks/{id}: the fields sent
+// change, checked like a create.
+func UpdateAutolink(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		id, err := pathID(r, "id")
+		if err != nil {
+			return 0, nil, err
+		}
+		var body classes.AutolinkWrite
+		if err := decode(r, maxBody, &body); err != nil {
+			return 0, nil, err
+		}
+		var row models.Autolink
+		err = write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			var err error
+			if row, err = classes.LoadAutolink(tx, id); err != nil {
+				return err
+			}
+			body.Apply(&row)
+			return svc.SaveAutolink(ctx, tx, &row)
+		})
+		if err != nil {
+			return 0, nil, err
+		}
+		return http.StatusOK, map[string]any{"data": AutolinkJSON(row)}, nil
+	})
+}
+
+// DeleteAutolinks serves DELETE /_seo/api/v1/autolinks?ids=: the phrases
+// named, each recorded in history; unknown ids are skipped.
+func DeleteAutolinks(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		ids, err := idList(r.URL.Query().Get("ids"))
+		if err != nil {
+			return 0, nil, err
+		}
+		var deleted []uint
+		err = write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			var err error
+			deleted, err = svc.DeleteAutolinks(ctx, tx, ids)
+			return err
+		})
+		if err != nil {
+			return 0, nil, err
+		}
+		return http.StatusOK, map[string]any{"deleted": deleted}, nil
+	})
+}
+
+// AutolinksFromKeyphrases serves POST
+// /_seo/api/v1/autolinks/from-keyphrases: one phrase per record whose
+// focus keyphrase in a locale is not yet a phrase there (origin
+// keyphrase), with the count added and the toast text.
+func AutolinksFromKeyphrases(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		added := 0
+		err := write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			var err error
+			added, err = svc.FromKeyphrases(ctx, tx)
+			return err
+		})
+		if err != nil {
+			return 0, nil, err
+		}
+		message := svc.PhraseCtx(ctx, "golem15.seo::lang.autolinks.keyphrases_none", nil)
+		if added > 0 {
+			message = svc.ChoiceCtx(ctx, "golem15.seo::lang.autolinks.keyphrases_added", added, nil)
+		}
+		return http.StatusOK, map[string]any{"added": added, "message": message}, nil
+	})
+}

controllers/api/crawls.go

@@ -0,0 +1,312 @@
+package api
+
+import (
+	"context"
+	"errors"
+	"net/http"
+	"strconv"
+	"strings"
+	"time"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/classes/crawl"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"gorm.io/gorm"
+)
+
+// StartCrawlInput is the body of POST /_seo/api/v1/crawls: nothing.
+type StartCrawlInput struct{}
+
+// CrawlView is a crawl in an answer.
+type CrawlView struct {
+	ID          uint       `json:"id"`
+	Status      string     `json:"status"`
+	Trigger     string     `json:"trigger"`
+	TriggerName string     `json:"trigger_name,omitempty"`
+	Site        string     `json:"site"`
+	MaxPages    int        `json:"max_pages"`
+	Pages       int        `json:"pages"`
+	LimitHit    bool       `json:"limit_hit"`
+	Error       string     `json:"error,omitempty"`
+	StartedAt   *time.Time `json:"started_at"`
+	FinishedAt  *time.Time `json:"finished_at"`
+	CreatedAt   *time.Time `json:"created_at"`
+}
+
+// crawlView is the answer form of a crawl.
+func crawlView(c models.Crawl) CrawlView {
+	return CrawlView{
+		ID: c.ID, Status: c.Status, Trigger: c.Trigger, TriggerName: c.TriggerName, Site: c.Site,
+		MaxPages: c.MaxPages, Pages: c.Pages, LimitHit: c.LimitHit, Error: c.Error,
+		StartedAt: c.StartedAt, FinishedAt: c.FinishedAt, CreatedAt: c.CreatedAt,
+	}
+}
+
+// StartCrawl serves POST /_seo/api/v1/crawls (D-14, P33-39): it queues a
+// crawl started by the administrator (trigger admin, or mcp through MCP)
+// and answers 202 with it. A crawl already queued or running refuses with
+// 409 crawls.already_running, a missing site address with 422
+// crawls.no_site_url.
+func StartCrawl(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		var row models.Crawl
+		err := write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			_, name, viaMCP := svc.Actor(ctx, tx)
+			trigger := models.TriggerAdmin
+			if viaMCP {
+				trigger = models.TriggerMCP
+			}
+			var err error
+			row, err = crawl.Start(ctx, svc, tx, trigger, name)
+			return err
+		})
+		if err != nil {
+			return 0, nil, StartRefusal(ctx, svc, err)
+		}
+		return http.StatusAccepted, map[string]any{"data": crawlMap(ctx, svc, row), "message": svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.started", nil)}, nil
+	})
+}
+
+// StartRefusal maps a refused start to its answer.
+func StartRefusal(ctx context.Context, svc *classes.Service, err error) error {
+	var running *crawl.AlreadyRunningError
+	switch {
+	case errors.As(err, &running):
+		return &apiError{Status: http.StatusConflict, Body: map[string]any{
+			"error":    "crawls.already_running",
+			"crawl_id": running.Crawl.ID,
+			"message":  RunningMessage(ctx, svc, running),
+		}}
+	case errors.Is(err, crawl.ErrNoSiteURL):
+		return &apiError{Status: http.StatusUnprocessableEntity, Body: map[string]any{
+			"error":   "crawls.no_site_url",
+			"message": svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.no_site_url", nil),
+		}}
+	}
+	return err
+}
+
+// RunningMessage is crawls.already_running with the running crawl's start.
+func RunningMessage(ctx context.Context, svc *classes.Service, running *crawl.AlreadyRunningError) string {
+	return svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.already_running", map[string]string{"time": running.Since().UTC().Format("2006-01-02 15:04 UTC")})
+}
+
+// RenderView is a crawl's rendered comparison in an answer (P33-55):
+// skipped (no golem15.seo.crawl_chrome_path), done with the pages
+// compared, or failed with the note; message says it in the request
+// locale.
+type RenderView struct {
+	Status  string `json:"status"`
+	Pages   int    `json:"pages"`
+	Note    string `json:"note,omitempty"`
+	Message string `json:"message"`
+}
+
+// renderView is the answer form of a crawl's comparison.
+func renderView(ctx context.Context, svc *classes.Service, c models.Crawl) RenderView {
+	out := RenderView{Status: c.RenderStatus, Pages: c.RenderPages, Note: c.RenderNote}
+	switch c.RenderStatus {
+	case models.RenderDone:
+		out.Message = svc.ChoiceCtx(ctx, "golem15.seo::lang.crawls.render_done", c.RenderPages, nil)
+	case models.RenderFailed:
+		out.Message = svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.render_failed", map[string]string{"error": c.RenderNote})
+	default:
+		out.Status = models.RenderSkipped
+		out.Message = svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.render_skipped", nil)
+	}
+	return out
+}
+
+// ListCrawlsInput is the query of GET /_seo/api/v1/crawls.
+type ListCrawlsInput struct {
+	Page    int    `json:"page,omitempty" query:"page" jsonschema:"the page number, from 1"`
+	PerPage int    `json:"per_page,omitempty" query:"per_page" jsonschema:"rows per page, at most 200 (50 by default)"`
+	Status  string `json:"status,omitempty" query:"status" jsonschema:"queued, running, finished, failed or cancelled"`
+}
+
+// CrawlIDInput names one crawl.
+type CrawlIDInput struct {
+	ID uint `json:"id" path:"id" jsonschema:"the crawl id"`
+}
+
+// DeleteCrawlsInput is the query of DELETE /_seo/api/v1/crawls.
+type DeleteCrawlsInput struct {
+	IDs string `json:"ids" query:"ids" jsonschema:"the crawl ids to delete, comma-separated, e.g. 3,4; a queued or running crawl refuses the whole call"`
+}
+
+// ListCrawls serves GET /_seo/api/v1/crawls: the crawls, newest first,
+// with their issue counts and rendered comparison.
+func ListCrawls(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		db, err := read(ctx, svc)
+		if err != nil {
+			return 0, nil, err
+		}
+		in := listParams(r)
+		q := db.Model(&models.Crawl{}).Order("id DESC")
+		if s := strings.TrimSpace(r.URL.Query().Get("status")); s != "" {
+			q = q.Where("status = ?", s)
+		}
+		var rows []models.Crawl
+		total, err := paginate(q, in, &rows)
+		if err != nil {
+			return 0, nil, err
+		}
+		ids := make([]uint, 0, len(rows))
+		for _, c := range rows {
+			ids = append(ids, c.ID)
+		}
+		counts := map[uint]int{}
+		if len(ids) > 0 {
+			var cs []struct {
+				CrawlID uint
+				N       int
+			}
+			if err := db.Raw(`SELECT crawl_id, COUNT(*) AS n FROM golem15_seo_issues WHERE crawl_id IN ? GROUP BY crawl_id`, ids).Scan(&cs).Error; err != nil {
+				return 0, nil, err
+			}
+			for _, c := range cs {
+				counts[c.CrawlID] = c.N
+			}
+		}
+		out := make([]map[string]any, 0, len(rows))
+		for _, c := range rows {
+			v := crawlMap(ctx, svc, c)
+			v["issues"] = counts[c.ID]
+			out = append(out, v)
+		}
+		return http.StatusOK, listBody(out, in, total), nil
+	})
+}
+
+// crawlMap is a crawl with its rendered comparison.
+func crawlMap(ctx context.Context, svc *classes.Service, c models.Crawl) map[string]any {
+	v := crawlView(c)
+	return map[string]any{
+		"id": v.ID, "status": v.Status, "trigger": v.Trigger, "trigger_name": v.TriggerName, "site": v.Site,
+		"max_pages": v.MaxPages, "pages": v.Pages, "limit_hit": v.LimitHit, "error": v.Error,
+		"started_at": v.StartedAt, "finished_at": v.FinishedAt, "created_at": v.CreatedAt,
+		"cancelled_by": c.CancelledBy, "cancel_requested": c.CancelRequestedAt != nil, "render": renderView(ctx, svc, c),
+	}
+}
+
+// ShowCrawl serves GET /_seo/api/v1/crawls/{id}: one crawl with its
+// rendered comparison, whether the staging lock's noindex was left out,
+// and its issues per kind and severity.
+func ShowCrawl(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		id, err := pathID(r, "id")
+		if err != nil {
+			return 0, nil, err
+		}
+		db, err := read(ctx, svc)
+		if err != nil {
+			return 0, nil, err
+		}
+		c, err := crawl.LoadCrawl(db, id)
+		if err != nil {
+			return 0, nil, err
+		}
+		var kinds []struct {
+			Kind     string `json:"kind"`
+			Severity string `json:"severity"`
+			Count    int    `json:"count"`
+		}
+		if err := db.Raw(`SELECT kind, severity, COUNT(*) AS count FROM golem15_seo_issues WHERE crawl_id = ? GROUP BY kind, severity ORDER BY MIN(rank), COUNT(*) DESC, kind`, id).Scan(&kinds).Error; err != nil {
+			return 0, nil, err
+		}
+		out := crawlMap(ctx, svc, c)
+		out["staging"] = svc.Staging()
+		bySeverity := map[string]int{}
+		total := 0
+		for _, k := range kinds {
+			bySeverity[k.Severity] += k.Count
+			total += k.Count
+		}
+		out["issues"] = total
+		out["severities"] = bySeverity
+		out["kinds"] = kinds
+		return http.StatusOK, map[string]any{"data": out}, nil
+	})
+}
+
+// CancelCrawl serves POST /_seo/api/v1/crawls/{id}/cancel: a queued crawl
+// ends cancelled at once; a running one answers with cancel_requested true
+// and stays running until its job stops, then ends cancelled and keeps
+// what it fetched; another answers 409 crawls.not_running.
+func CancelCrawl(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		id, err := pathID(r, "id")
+		if err != nil {
+			return 0, nil, err
+		}
+		var row models.Crawl
+		err = write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			c, err := crawl.LoadCrawl(tx, id)
+			if err != nil {
+				return err
+			}
+			_, name, _ := svc.Actor(ctx, tx)
+			if err := crawl.Cancel(ctx, svc, tx, c, name); err != nil {
+				return err
+			}
+			row, err = crawl.LoadCrawl(tx, id)
+			return err
+		})
+		if errors.Is(err, crawl.ErrNotRunning) {
+			return 0, nil, Fail(http.StatusConflict, "crawls.not_running")
+		}
+		if err != nil {
+			return 0, nil, err
+		}
+		return http.StatusOK, map[string]any{"data": crawlMap(ctx, svc, row), "message": svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.cancelled", nil)}, nil
+	})
+}
+
+// DeleteCrawls serves DELETE /_seo/api/v1/crawls?ids=…: the crawls go with
+// their pages and issues; a queued or running one refuses the whole call
+// with 403 crawls.delete_running.
+func DeleteCrawls(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		ids, err := idList(r.URL.Query().Get("ids"))
+		if err != nil {
+			return 0, nil, err
+		}
+		n := 0
+		err = write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			var err error
+			n, err = crawl.DeleteCrawls(tx, ids)
+			return err
+		})
+		if errors.Is(err, crawl.ErrDeleteRunning) {
+			return 0, nil, &apiError{Status: http.StatusForbidden, Body: map[string]any{
+				"error":   "crawls.delete_running",
+				"message": svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.delete_running", nil),
+			}}
+		}
+		if err != nil {
+			return 0, nil, err
+		}
+		return http.StatusOK, map[string]any{"deleted": n}, nil
+	})
+}
+
+// idList reads a comma-separated id list of at most 500 ids.
+func idList(raw string) ([]uint, error) {
+	var out []uint
+	for _, part := range strings.Split(raw, ",") {
+		part = strings.TrimSpace(part)
+		if part == "" {
+			continue
+		}
+		n, err := strconv.ParseUint(part, 10, 64)
+		if err != nil || n == 0 {
+			return nil, Fail(http.StatusUnprocessableEntity, "ids_invalid")
+		}
+		out = append(out, uint(n))
+	}
+	if len(out) == 0 || len(out) > 500 {
+		return nil, Fail(http.StatusUnprocessableEntity, "ids_invalid")
+	}
+	return out, nil
+}

controllers/api/graph.go

@@ -0,0 +1,48 @@
+package api
+
+import (
+	"context"
+	"errors"
+	"net/http"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+)
+
+// GetGraphInput is the query of GET /_seo/api/v1/graph.
+type GetGraphInput struct {
+	URL    string  `json:"url" query:"url" jsonschema:"the page: a path such as /blog/post or an absolute URL of the site"`
+	Schema *string `json:"schema,omitempty" query:"schema" jsonschema:"an unsaved structured data value (the seo_set_url_seo schema string) to preview instead of the stored one; omitted: the stored value"`
+}
+
+// Graph serves GET /_seo/api/v1/graph?url=&schema= (D-46, D-10): the
+// JSON-LD @graph the resolve answer of the URL carries, built with the
+// given unsaved schema value when sent, and every node with its missing
+// required and recommended properties (rules false for a type Google has
+// no rules for). It answers {locale, url, graph, nodes}; url and graph are
+// null for an address that is not an ok page. 422 for a URL that is not a
+// path or http(s) URL, or for a schema value the SEO tab would refuse.
+func Graph(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		q := r.URL.Query()
+		var override *string
+		if q.Has("schema") {
+			v := q.Get("schema")
+			override = &v
+		}
+		answer, res, err := svc.GraphOfURL(ctx, q.Get("url"), override)
+		if errors.Is(err, classes.ErrBadURL) {
+			return 0, nil, Fail(http.StatusUnprocessableEntity, err.Error())
+		}
+		if err != nil {
+			return 0, nil, err
+		}
+		if override != nil && res.TypeKey != "" {
+			if typ, ok := svc.Registry().ContentType(res.TypeKey); ok && typ.Content != nil {
+				if err := svc.CheckSchema(ctx, typ.Content.Kind, res.TypeKey, res.RecordID, *override); err != nil {
+					return 0, nil, err
+				}
+			}
+		}
+		return http.StatusOK, answer, nil
+	})
+}

controllers/api/issues.go

@@ -2,34 +2,231 @@ package api
 
 import (
 	"context"
+	"errors"
 	"net/http"
+	"strconv"
+	"strings"
+	"time"
 
 	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/classes/crawl"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"gorm.io/gorm"
 )
 
 // ListIssuesInput is the query of GET /_seo/api/v1/issues.
 type ListIssuesInput struct {
-	Kind string `json:"kind,omitempty" query:"kind" jsonschema:"one kind only: missing_title, missing_description, duplicate_title, duplicate_description, rule_overrides_record, redirect_chain, canonical_noindex, canonical_redirected or noindex_blocked"`
+	Kind     string `json:"kind,omitempty" query:"kind" jsonschema:"one kind only, e.g. h1_multiple, broken_link, missing_title, content_js, orphan_page, redirect_chain or sitemap_blocked"`
+	Severity string `json:"severity,omitempty" query:"severity" jsonschema:"error, warning or notice"`
+	Source   string `json:"source,omitempty" query:"source" jsonschema:"crawl (found by the crawler) or cms (CMS data checks)"`
+	Fix      string `json:"fix,omitempty" query:"fix" jsonschema:"tool (fixable through the MCP tool the issue names) or code (needs a change in the site's templates)"`
+	Status   string `json:"status,omitempty" query:"status" jsonschema:"open (the default), ignored or all"`
+	URL      string `json:"url,omitempty" query:"url" jsonschema:"only issues of this address (a path such as /blog/post)"`
+	Crawl    uint   `json:"crawl,omitempty" query:"crawl" jsonschema:"the issues of this crawl id instead of the current ones"`
 }
 
-// ListIssues serves GET /_seo/api/v1/issues (D-13, D-14 CMS part): the
+// IssueIDInput names one issue.
-// CMS-data issue report, each issue with its URL, kind, details and the
+type IssueIDInput struct {
-// tool that fixes it, and a count per kind.
+	ID uint `json:"id" path:"id" jsonschema:"the issue id"`
+}
+
+// IssueIDsInput is the body of the ignore and unignore calls.
+type IssueIDsInput struct {
+	IDs []uint `json:"ids" jsonschema:"the issue ids"`
+}
+
+// IssueView is a stored issue in an answer: the Phase 32 keys (kind, url,
+// locale, details, tool) and the audit's id, severity, source, fix,
+// found_at, ignored and crawl_id. tool is the fixing MCP tool when fix is
+// tool; for fix code, details.what names the template element.
+type IssueView struct {
+	ID       uint           `json:"id"`
+	Kind     string         `json:"kind"`
+	URL      string         `json:"url"`
+	Locale   string         `json:"locale,omitempty"`
+	Details  map[string]any `json:"details,omitempty"`
+	Tool     string         `json:"tool"`
+	Severity string         `json:"severity"`
+	Source   string         `json:"source"`
+	Fix      string         `json:"fix"`
+	FoundAt  *time.Time     `json:"found_at"`
+	Ignored  bool           `json:"ignored"`
+	CrawlID  *uint          `json:"crawl_id"`
+}
+
+// issueViews turns stored issues into their answer form.
+func issueViews(db *gorm.DB, rows []models.Issue) ([]IssueView, error) {
+	fps := make([]string, 0, len(rows))
+	for _, r := range rows {
+		fps = append(fps, r.Fingerprint)
+	}
+	ignored, err := crawl.IgnoredSet(db, fps)
+	if err != nil {
+		return nil, err
+	}
+	out := make([]IssueView, 0, len(rows))
+	for _, r := range rows {
+		out = append(out, IssueView{
+			ID: r.ID, Kind: r.Kind, URL: r.URL, Locale: r.Locale, Details: r.DetailsMap(), Tool: r.Tool,
+			Severity: r.Severity, Source: r.Source, Fix: r.Fix, FoundAt: r.FoundAt,
+			Ignored: ignored[r.Fingerprint], CrawlID: r.CrawlID,
+		})
+	}
+	return out, nil
+}
+
+// ListIssues serves GET /_seo/api/v1/issues (D-13, D-14): the audit's
+// current issues, errors first: the newest crawl's findings with the
+// CMS-data issues stored with it, or before any crawl the CMS-data issues
+// as they are now (crawl selects another crawl's). Each issue names the
+// tool that fixes it, or fix code. The filters are severity, kind,
+// source, fix, url and status (open by default: ignored issues are left
+// out; ignored or all). counts holds the number per kind before the kind
+// filter.
 func ListIssues(service func() *classes.Service) http.HandlerFunc {
 	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
-		issues, err := svc.Issues(ctx)
+		db, err := read(ctx, svc)
 		if err != nil {
 			return 0, nil, err
 		}
-		kind := r.URL.Query().Get("kind")
+		latest, err := crawl.Current(ctx, svc, db)
+		if err != nil {
+			return 0, nil, err
+		}
+		q := r.URL.Query()
+		base := crawl.CurrentScope(db.Model(&models.Issue{}), latest)
+		if id, err := strconv.ParseUint(strings.TrimSpace(q.Get("crawl")), 10, 64); err == nil && id > 0 {
+			base = db.Model(&models.Issue{}).Where("golem15_seo_issues.crawl_id = ?", id)
+		}
+		var all []models.Issue
+		if err := base.Order("rank ASC, kind ASC, url ASC, id ASC").Find(&all).Error; err != nil {
+			return 0, nil, err
+		}
+		views, err := issueViews(db, all)
+		if err != nil {
+			return 0, nil, err
+		}
+		status := strings.TrimSpace(q.Get("status"))
+		if status == "" {
+			status = models.IssueOpen
+		}
+		match := map[string]string{
+			"kind": strings.TrimSpace(q.Get("kind")), "severity": strings.TrimSpace(q.Get("severity")),
+			"source": strings.TrimSpace(q.Get("source")), "fix": strings.TrimSpace(q.Get("fix")),
+			"url": strings.TrimSpace(q.Get("url")),
+		}
 		counts := map[string]int{}
-		out := make([]classes.Issue, 0, len(issues))
+		out := make([]IssueView, 0, len(views))
-		for _, issue := range issues {
+		for _, v := range views {
-			counts[issue.Kind]++
+			if status == models.IssueOpen && v.Ignored || status == models.IssueIgnored && !v.Ignored {
-			if kind == "" || issue.Kind == kind {
+				continue
-				out = append(out, issue)
+			}
+			if !matches(match["severity"], v.Severity) || !matches(match["source"], v.Source) || !matches(match["fix"], v.Fix) || !matches(match["url"], v.URL) {
+				continue
+			}
+			counts[v.Kind]++
+			if matches(match["kind"], v.Kind) {
+				out = append(out, v)
 			}
 		}
 		return http.StatusOK, map[string]any{"data": out, "counts": counts, "total": len(out)}, nil
 	})
 }
+
+// matches reports whether a filter value (empty: any) matches.
+func matches(filter, value string) bool { return filter == "" || filter == value }
+
+// RecheckIssue serves POST /_seo/api/v1/issues/{id}/recheck: the issue's
+// address is fetched again through the crawler client and its issues are
+// replaced; the answer has the address, the issues left and the message.
+func RecheckIssue(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		id, err := pathID(r, "id")
+		if err != nil {
+			return 0, nil, err
+		}
+		var res crawl.RecheckResult
+		err = write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			issue, err := crawl.LoadIssue(tx, id)
+			if err != nil {
+				return err
+			}
+			res, err = crawl.Recheck(ctx, svc, tx, issue)
+			return err
+		})
+		if errors.Is(err, crawl.ErrNoSiteURL) {
+			return 0, nil, StartRefusal(ctx, svc, err)
+		}
+		if err != nil {
+			return 0, nil, err
+		}
+		msg := svc.PhraseCtx(ctx, "golem15.seo::lang.issues.recheck_fixed", nil)
+		if res.Remaining > 0 {
+			msg = svc.ChoiceCtx(ctx, "golem15.seo::lang.issues.rechecked", res.Remaining, nil)
+		}
+		return http.StatusOK, map[string]any{"url": res.URL, "remaining": res.Remaining, "message": msg}, nil
+	})
+}
+
+// IgnoreIssues serves POST /_seo/api/v1/issues/ignore: the issues are
+// hidden in this and later crawls until they are no longer ignored.
+func IgnoreIssues(service func() *classes.Service) http.HandlerFunc {
+	return ignoreHandler(service, true)
+}
+
+// UnignoreIssues serves POST /_seo/api/v1/issues/unignore: the issues are
+// shown in the audit again.
+func UnignoreIssues(service func() *classes.Service) http.HandlerFunc {
+	return ignoreHandler(service, false)
+}
+
+func ignoreHandler(service func() *classes.Service, ignore bool) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		var in IssueIDsInput
+		if err := decode(r, maxBody, &in); err != nil {
+			return 0, nil, err
+		}
+		if len(in.IDs) == 0 || len(in.IDs) > 1000 {
+			return 0, nil, Fail(http.StatusUnprocessableEntity, "ids_invalid")
+		}
+		n := 0
+		err := write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			var rows []models.Issue
+			if err := tx.Where("id IN ?", in.IDs).Find(&rows).Error; err != nil {
+				return err
+			}
+			if len(rows) != len(uniqueIDs(in.IDs)) {
+				return gorm.ErrRecordNotFound
+			}
+			var err error
+			if ignore {
+				adminID, _, _ := svc.Actor(ctx, tx)
+				n, err = crawl.Ignore(tx, rows, adminID)
+			} else {
+				n, err = crawl.Unignore(tx, rows)
+			}
+			return err
+		})
+		if err != nil {
+			return 0, nil, err
+		}
+		key, field := "golem15.seo::lang.issues.unignored", "unignored"
+		if ignore {
+			key, field = "golem15.seo::lang.issues.ignored", "ignored"
+		}
+		return http.StatusOK, map[string]any{field: n, "message": svc.ChoiceCtx(ctx, key, n, nil)}, nil
+	})
+}
+
+// uniqueIDs drops repeated ids.
+func uniqueIDs(ids []uint) []uint {
+	seen := map[uint]bool{}
+	var out []uint
+	for _, id := range ids {
+		if !seen[id] {
+			seen[id] = true
+			out = append(out, id)
+		}
+	}
+	return out
+}

controllers/api/links.go

@@ -0,0 +1,33 @@
+package api
+
+import (
+	"context"
+	"net/http"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+)
+
+// GetLinksInput is the query of GET /_seo/api/v1/links.
+type GetLinksInput struct {
+	URL string `json:"url" query:"url" jsonschema:"the page: a path such as /blog/post or an absolute URL of the site"`
+}
+
+// Links serves GET /_seo/api/v1/links?url= (D-82, UI-SPEC C19): the
+// page's internal links from the last finished crawl, {crawledAt,
+// inbound, outbound, orphan, out [{phrase, title, url}], in [{phrase,
+// title, adminUrl}]}; before the first crawl crawledAt, inbound and
+// outbound are null and the lists empty. A url that is not a path or an
+// http(s) URL is 422.
+func Links(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		db, err := read(ctx, svc)
+		if err != nil {
+			return 0, nil, err
+		}
+		report, err := svc.LinksOfURL(ctx, db, r.URL.Query().Get("url"))
+		if err != nil {
+			return 0, nil, err
+		}
+		return http.StatusOK, report, nil
+	})
+}

controllers/api/llms.go

@@ -0,0 +1,34 @@
+package api
+
+import (
+	"net/http"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+)
+
+// LLMS serves GET /llms.txt at the site root (D-25, UI-SPEC C25): while
+// llms.txt is on (the robots.txt screen's AI bots tab) and in production,
+// the site's indexable pages from the sitemaps as text/plain; otherwise
+// 404.
+func LLMS(service func() *classes.Service) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		svc := service()
+		if svc == nil {
+			http.Error(w, "SEO is not available", http.StatusServiceUnavailable)
+			return
+		}
+		body, ok, err := svc.LLMSText(r.Context())
+		if err != nil {
+			log.Error("golem15.seo: llms.txt", "error", err)
+			http.Error(w, "Internal Server Error", http.StatusInternalServerError)
+			return
+		}
+		if !ok {
+			http.NotFound(w, r)
+			return
+		}
+		w.Header().Set("Content-Type", "text/plain; charset=utf-8")
+		w.Header().Set("X-Content-Type-Options", "nosniff")
+		_, _ = w.Write([]byte(body))
+	}
+}

controllers/api/registry.go

@@ -0,0 +1,393 @@
+package api
+
+import (
+	"bytes"
+	"context"
+	"crypto/subtle"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"strconv"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/wire"
+	"gorm.io/gorm"
+)
+
+// maxRouteReportBody caps the route report body: two lists (routes and
+// noindex) of 2000 paths of 2048 bytes with their JSON quoting.
+const maxRouteReportBody = 2*classes.MaxReportedRoutes*(classes.MaxRouteBytes+8) + 1024
+
+// routeReport is the route report body. Noindex is nil when the key is
+// absent and non-nil (also empty) when it is present.
+type routeReport struct {
+	Routes  []string  `json:"routes"`
+	Noindex *[]string `json:"noindex"`
+}
+
+// routeReportAnswer is the route report answer: the counts of the routes
+// and the number of stored noindex paths.
+type routeReportAnswer struct {
+	classes.RouteReport
+	Noindex int `json:"noindex"`
+}
+
+// RouteReport serves POST /_seo/api/v1/registry/routes (D-33, P33-20): the
+// frontend server reports its static routes with "Authorization: Bearer
+// <token>", golem15.seo.route_report_token (else notfound_token), compared
+// in constant time. Unknown paths become registry pages (origin reported),
+// known ones get reported_at; nothing is deleted or overwritten.
+//
+// The optional noindex key (P33-58) lists the static paths the frontend
+// serves noindex: when it is present (also as []), the stored set becomes
+// exactly its valid entries, normalised like the routes, in the same
+// transaction (an invalid entry is skipped and counted in skipped); when it
+// is absent the stored set is unchanged. The sitemap filter leaves those
+// paths out of the sitemaps, and a changed set asks the sitemap for one
+// debounced rebuild.
+//
+// It answers 200 with the counts and noindex, the number of stored noindex
+// paths; 401 for a missing or wrong token, 404 when no token is configured
+// (the report is off), and 422 for a malformed body or more than 2000
+// paths in either list, storing nothing.
+func RouteReport(service func() *classes.Service) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		svc := service()
+		if svc == nil {
+			wire.WriteJSON(w, http.StatusServiceUnavailable, map[string]string{"error": "SEO is not available"})
+			return
+		}
+		want := svc.RouteReportToken()
+		if want == "" {
+			wire.WriteJSON(w, http.StatusNotFound, map[string]string{"error": errNotFound})
+			return
+		}
+		got, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
+		if !ok || subtle.ConstantTimeCompare([]byte(strings.TrimSpace(got)), []byte(want)) != 1 {
+			wire.WriteJSON(w, http.StatusUnauthorized, map[string]string{"error": "Unauthorized"})
+			return
+		}
+		var in routeReport
+		dec := json.NewDecoder(http.MaxBytesReader(w, r.Body, maxRouteReportBody))
+		if err := dec.Decode(&in); err != nil {
+			wire.WriteJSON(w, http.StatusUnprocessableEntity, map[string]string{"error": errInvalidBody})
+			return
+		}
+		if len(in.Routes) > classes.MaxReportedRoutes || (in.Noindex != nil && len(*in.Noindex) > classes.MaxReportedRoutes) {
+			wire.WriteJSON(w, http.StatusUnprocessableEntity, map[string]any{"error": "too_many_routes", "max": classes.MaxReportedRoutes})
+			return
+		}
+		var out routeReportAnswer
+		err := write(r.Context(), svc, func(ctx context.Context, tx *gorm.DB) error {
+			var err error
+			out.RouteReport, err = svc.ReportRoutes(ctx, tx, in.Routes)
+			if err != nil {
+				return err
+			}
+			if in.Noindex != nil {
+				var skipped int
+				out.Noindex, skipped, _, err = svc.ReportNoindex(ctx, tx, *in.Noindex)
+				out.Skipped += skipped
+				return err
+			}
+			out.Noindex, err = classes.NoindexPathCount(tx)
+			return err
+		})
+		if err != nil {
+			if errors.Is(err, classes.ErrTooManyRoutes) {
+				wire.WriteJSON(w, http.StatusUnprocessableEntity, map[string]any{"error": "too_many_routes", "max": classes.MaxReportedRoutes})
+				return
+			}
+			log.Error("golem15.seo: route report", "error", err)
+			wire.WriteJSON(w, http.StatusInternalServerError, map[string]string{"error": errInternal})
+			return
+		}
+		wire.WriteJSON(w, http.StatusOK, out)
+	}
+}
+
+// ListPagesInput is the query of GET /_seo/api/v1/pages.
+type ListPagesInput struct {
+	Page    int    `json:"page,omitempty" query:"page" jsonschema:"the page number, from 1"`
+	PerPage int    `json:"per_page,omitempty" query:"per_page" jsonschema:"rows per page, at most 200 (50 by default)"`
+	Search  string `json:"search,omitempty" query:"search" jsonschema:"text the address, another locale's address or the name contains"`
+	ID      uint   `json:"id,omitempty" query:"id" jsonschema:"one page by id"`
+	Kind    string `json:"kind,omitempty" query:"kind" jsonschema:"page, offer, service or landing"`
+	Origin  string `json:"origin,omitempty" query:"origin" jsonschema:"manual, reported or mcp"`
+	Missing string `json:"missing,omitempty" query:"missing" jsonschema:"only: reported pages the site's newest route report did not include; hide: all others"`
+}
+
+// PageIDInput names one registry page.
+type PageIDInput struct {
+	ID uint `json:"id" path:"id" jsonschema:"the registry page id"`
+}
+
+// CreatePageInput is the body of POST /_seo/api/v1/pages.
+type CreatePageInput struct {
+	URL              map[string]string `json:"url" jsonschema:"the page's path per locale code, e.g. {\"pl\": \"/kontakt\", \"en\": \"/contact\"}; the default locale is required, a locale left out or empty has no version of the page; a plain string is the default locale's path"`
+	Title            map[string]string `json:"title,omitempty" jsonschema:"the page's name per locale code (default SEO title, HTML sitemap, breadcrumbs); a plain string is the default locale's"`
+	Kind             *string           `json:"kind,omitempty" jsonschema:"page (the default), offer, service or landing: decides the sitemap file and the default structured data"`
+	ParentID         *uint             `json:"parent_id,omitempty" jsonschema:"the parent registry page's id (breadcrumbs); null or 0 for the top level"`
+	ContentUpdatedAt *string           `json:"content_updated_at,omitempty" jsonschema:"the date the page's content last changed, YYYY-MM-DD (sitemap lastmod); today by default"`
+	Note             *string           `json:"note,omitempty" jsonschema:"an internal note"`
+}
+
+// UpdatePageInput is PUT /_seo/api/v1/pages/{id}: only the fields sent are
+// changed; in url and title only the locales sent.
+type UpdatePageInput struct {
+	ID               uint              `json:"id" path:"id" jsonschema:"the registry page id"`
+	URL              map[string]string `json:"url,omitempty" jsonschema:"paths per locale code to change; an empty path removes that locale's version (the default locale's cannot be removed)"`
+	Title            map[string]string `json:"title,omitempty" jsonschema:"names per locale code to change"`
+	Kind             *string           `json:"kind,omitempty" jsonschema:"page, offer, service or landing"`
+	ParentID         *uint             `json:"parent_id,omitempty" jsonschema:"the parent registry page's id; null or 0 for the top level"`
+	ContentUpdatedAt *string           `json:"content_updated_at,omitempty" jsonschema:"YYYY-MM-DD; empty clears it"`
+	Note             *string           `json:"note,omitempty" jsonschema:"an internal note"`
+}
+
+// pageBody is a decoded create or update body; a translatable value is a
+// locale map or a plain string (the default locale's).
+type pageBody struct {
+	write classes.PageWrite
+}
+
+func (b *pageBody) UnmarshalJSON(raw []byte) error {
+	var fields map[string]json.RawMessage
+	if err := json.Unmarshal(raw, &fields); err != nil {
+		return err
+	}
+	texts := func(name string) (map[string]string, error) {
+		v, ok := fields[name]
+		if !ok || bytes.Equal(bytes.TrimSpace(v), []byte("null")) {
+			return nil, nil
+		}
+		var s string
+		if json.Unmarshal(v, &s) == nil {
+			return map[string]string{"": s}, nil
+		}
+		var m map[string]string
+		if err := json.Unmarshal(v, &m); err != nil {
+			return nil, err
+		}
+		if m == nil {
+			m = map[string]string{}
+		}
+		return m, nil
+	}
+	var err error
+	if b.write.URL, err = texts("url"); err != nil {
+		return err
+	}
+	if b.write.Title, err = texts("title"); err != nil {
+		return err
+	}
+	str := func(name string) (*string, error) {
+		v, ok := fields[name]
+		if !ok || bytes.Equal(bytes.TrimSpace(v), []byte("null")) {
+			return nil, nil
+		}
+		var s string
+		if err := json.Unmarshal(v, &s); err != nil {
+			return nil, err
+		}
+		return &s, nil
+	}
+	if b.write.Kind, err = str("kind"); err != nil {
+		return err
+	}
+	if b.write.ContentUpdatedAt, err = str("content_updated_at"); err != nil {
+		return err
+	}
+	if b.write.Note, err = str("note"); err != nil {
+		return err
+	}
+	if v, ok := fields["parent_id"]; ok {
+		b.write.ParentSet = true
+		if !bytes.Equal(bytes.TrimSpace(v), []byte("null")) {
+			var n uint
+			if err := json.Unmarshal(v, &n); err != nil {
+				return err
+			}
+			if n != 0 {
+				b.write.ParentID = &n
+			}
+		}
+	}
+	return nil
+}
+
+// PageJSON is a registry page in an answer: its default-locale address and
+// name, every locale's address and name, and whether the newest route
+// report missed it.
+func PageJSON(p models.Page, def string, reported *bool) map[string]any {
+	urls := p.LocaleURLs(def)
+	titles := map[string]string{def: p.Title}
+	for l, t := range p.Titles.Get() {
+		if l != def && t != "" {
+			titles[l] = t
+		}
+	}
+	var parent any
+	if p.Parent() != 0 {
+		parent = p.Parent()
+	}
+	var date any
+	if p.ContentUpdatedAt != nil && !p.ContentUpdatedAt.IsZero() {
+		date = p.ContentUpdatedAt.String()
+	}
+	out := map[string]any{
+		"id": p.ID, "url": p.URL, "urls": urls, "title": p.Title, "titles": titles,
+		"kind": p.Kind, "origin": p.Origin, "parent_id": parent, "content_updated_at": date,
+		"reported_at": p.ReportedAt, "note": p.Note, "created_at": p.CreatedAt, "updated_at": p.UpdatedAt,
+	}
+	if reported != nil {
+		out["missing"] = *reported
+	}
+	return out
+}
+
+// pageAnswer is a page in an answer with its missing state.
+func pageAnswer(ctx context.Context, svc *classes.Service, db *gorm.DB, p models.Page) (map[string]any, error) {
+	settings, err := classes.LoadSettings(db)
+	if err != nil {
+		return nil, err
+	}
+	missing := classes.PageMissing(p, settings.RoutesReportedAt)
+	return PageJSON(p, svc.DefaultLocale(ctx, db), &missing), nil
+}
+
+// ListPages serves GET /_seo/api/v1/pages: the registry by address,
+// filtered by search text, id, kind, origin and the missing state.
+func ListPages(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		db, err := read(ctx, svc)
+		if err != nil {
+			return 0, nil, err
+		}
+		in := listParams(r)
+		query := r.URL.Query()
+		q := db.Model(&models.Page{}).Where("url <> ''").Order("url ASC, id ASC")
+		if in.Search != "" {
+			p := like(in.Search)
+			q = q.Where(`lower(url) LIKE ? OR lower(title) LIKE ? OR lower(urls::text) LIKE ? OR lower(titles::text) LIKE ?`, p, p, p, p)
+		}
+		if id, err := strconv.ParseUint(query.Get("id"), 10, 64); err == nil && id > 0 {
+			q = q.Where("id = ?", id)
+		}
+		for param, scope := range map[string]string{"kind": models.FilterPageKind, "origin": models.FilterPageOrigin, "missing": models.FilterPageMissing} {
+			if v := strings.TrimSpace(query.Get(param)); v != "" {
+				q = models.Page{}.FilterScope(scope, q, v)
+			}
+		}
+		var rows []models.Page
+		total, err := paginate(q, in, &rows)
+		if err != nil {
+			return 0, nil, err
+		}
+		settings, err := classes.LoadSettings(db)
+		if err != nil {
+			return 0, nil, err
+		}
+		def := svc.DefaultLocale(ctx, db)
+		out := make([]map[string]any, 0, len(rows))
+		for _, row := range rows {
+			missing := classes.PageMissing(row, settings.RoutesReportedAt)
+			out = append(out, PageJSON(row, def, &missing))
+		}
+		return http.StatusOK, listBody(out, in, total), nil
+	})
+}
+
+// savePage applies a body to p and saves it in tx.
+func savePage(ctx context.Context, svc *classes.Service, tx *gorm.DB, p *models.Page, w classes.PageWrite, before map[string]any) error {
+	problems := classes.ApplyPageFields(p, w)
+	if len(problems) > 0 {
+		return &classes.ProblemsError{Problems: problems}
+	}
+	if err := svc.ApplyPageTexts(ctx, tx, p, w.URL, w.Title); err != nil {
+		return err
+	}
+	return svc.SavePage(ctx, tx, p, before)
+}
+
+// CreatePage serves POST /_seo/api/v1/pages: 201 with the page, 422
+// validation_failed with the C21 refusals (url_required, url_invalid,
+// url_taken, url_owned, parent_loop). The origin is mcp through MCP.
+func CreatePage(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		var body pageBody
+		if err := decode(r, maxBody, &body); err != nil {
+			return 0, nil, err
+		}
+		row := models.NewPage()
+		row.Origin = classes.PageOrigin(ctx)
+		var answer map[string]any
+		err := write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			if err := savePage(ctx, svc, tx, &row, body.write, nil); err != nil {
+				return err
+			}
+			var err error
+			answer, err = pageAnswer(ctx, svc, tx, row)
+			return err
+		})
+		if err != nil {
+			return 0, nil, err
+		}
+		return http.StatusCreated, map[string]any{"data": answer}, nil
+	})
+}
+
+// UpdatePage serves PUT /_seo/api/v1/pages/{id}: the fields sent change
+// and are checked like a create.
+func UpdatePage(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		id, err := pathID(r, "id")
+		if err != nil {
+			return 0, nil, err
+		}
+		var body pageBody
+		if err := decode(r, maxBody, &body); err != nil {
+			return 0, nil, err
+		}
+		var answer map[string]any
+		err = write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			row, err := classes.LoadPage(tx, id)
+			if err != nil {
+				return err
+			}
+			if err := savePage(ctx, svc, tx, &row, body.write, classes.PageValues(row)); err != nil {
+				return err
+			}
+			answer, err = pageAnswer(ctx, svc, tx, row)
+			return err
+		})
+		if err != nil {
+			return 0, nil, err
+		}
+		return http.StatusOK, map[string]any{"data": answer}, nil
+	})
+}
+
+// DeletePage serves DELETE /_seo/api/v1/pages/{id}: the page and its SEO
+// entries go (history keeps them); a page the site reports comes back with
+// its next route report.
+func DeletePage(service func() *classes.Service) http.HandlerFunc {
+	return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) {
+		id, err := pathID(r, "id")
+		if err != nil {
+			return 0, nil, err
+		}
+		err = write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
+			row, err := classes.LoadPage(tx, id)
+			if err != nil {
+				return err
+			}
+			return svc.DeletePage(ctx, tx, row)
+		})
+		if err != nil {
+			return 0, nil, err
+		}
+		return http.StatusOK, map[string]any{"deleted": id}, nil
+	})
+}

controllers/api/robots.go

@@ -50,6 +50,9 @@ func Robots(service func() *classes.Service) http.HandlerFunc {
 type UpdateRobotsInput struct {
 	Content *string         `json:"content,omitempty" jsonschema:"the robots.txt rules; Sitemap lines and the AI bot rules are added when the file is served"`
 	Bots    map[string]bool `json:"bots,omitempty" jsonschema:"AI bot switches by user-agent token, for example {\"GPTBot\": false}; a bot switched off is disallowed"`
+	// LlmsEnabled and LlmsIntro are the llms.txt switch and introduction.
+	LlmsEnabled *bool   `json:"llms_enabled,omitempty" jsonschema:"serve /llms.txt: the site's indexable pages from the sitemaps, for AI tools (off by default)"`
+	LlmsIntro   *string `json:"llms_intro,omitempty" jsonschema:"a short description of the site at the top of llms.txt; empty uses the default meta description"`
 }
 
 // TestRobotsInput is the query of GET /_seo/api/v1/robots/test.
@@ -66,10 +69,12 @@ func robotsJSON(ctx context.Context, svc *classes.Service, row models.Robots) (m
 	for _, b := range classes.Bots {
 		bots = append(bots, map[string]any{"token": b.Token, "operator": b.Operator, "kind": b.Kind, "allowed": row.Bots.Allowed(b.Token)})
 	}
-	production := classes.ComposeRobots(row.Content, row.Bots, svc.RobotsSitemaps())
+	production := classes.ComposeRobots(row.Content, row.Bots, svc.SitemapLines(ctx))
 	return map[string]any{
 		"content":      row.Content,
 		"bots":         bots,
+		"llms_enabled": row.LlmsEnabled,
+		"llms_intro":   row.LlmsIntro,
 		"production":   production,
 		"staging":      svc.Staging(),
 	}, nil
@@ -107,7 +112,7 @@ func UpdateRobots(service func() *classes.Service) http.HandlerFunc {
 		var blocked []string
 		err := write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error {
 			var err error
-			row, blocked, err = svc.SaveRobots(ctx, tx, classes.RobotsWrite{Content: in.Content, Bots: in.Bots})
+			row, blocked, err = svc.SaveRobots(ctx, tx, classes.RobotsWrite{Content: in.Content, Bots: in.Bots, LlmsEnabled: in.LlmsEnabled, LlmsIntro: in.LlmsIntro})
 			return err
 		})
 		if err != nil {

controllers/api/seo.go

@@ -20,6 +20,8 @@ type SEOValues struct {
 	OGTitle       *string `json:"og_title,omitempty" jsonschema:"the social title"`
 	OGDescription *string `json:"og_description,omitempty" jsonschema:"the social description"`
 	OGImage       *string `json:"og_image,omitempty" jsonschema:"the social image: a path or an absolute URL"`
+	PrimaryParent *string `json:"primary_parent,omitempty" jsonschema:"records only: the primary category of the breadcrumb trail and the article section, one of the record's parents as <type>:<id> (seo_get_url_seo and the graph tool name them); empty for the first parent. Shared by every locale"`
+	Schema        *string `json:"schema,omitempty" jsonschema:"records only: the structured data as a JSON string {\"type\": schema type or empty for the default, \"fields\": {serviceType, areaServed, price, currency} for Service, \"custom\": custom JSON-LD text (an object or a list, at most 16 KB, no script tag; a node with a generated @id is merged into it)}; empty resets it. Shared by every locale"`
 }
 
 // Map returns the values that are set, keyed by column.
@@ -29,6 +31,7 @@ func (v SEOValues) Map() map[string]string {
 		"title": v.Title, "description": v.Description, "keyphrase": v.Keyphrase,
 		"canonical": v.Canonical, "robots": v.Robots, "og_title": v.OGTitle,
 		"og_description": v.OGDescription, "og_image": v.OGImage,
+		"primary_parent": v.PrimaryParent, "schema": v.Schema,
 	} {
 		if p != nil {
 			out[column] = *p

controllers/api/settings.go

@@ -25,12 +25,44 @@ type UpdateSettingsInput struct {
 	URLHost               *string                             `json:"url_host,omitempty" jsonschema:"keep, www or bare"`
 	URLSlash              *string                             `json:"url_slash,omitempty" jsonschema:"keep, add or remove a trailing slash"`
 	URLLowercase          *bool                               `json:"url_lowercase,omitempty" jsonschema:"lowercase the ASCII letters of addresses"`
+	LinksNewTab           *bool                               `json:"links_new_tab,omitempty" jsonschema:"open links to other sites in a new tab (rel=noopener, announced to screen readers); default true"`
+	NofollowDomains       *string                             `json:"nofollow_domains,omitempty" jsonschema:"domains whose links always get nofollow, one per line, e.g. example.com (subdomains included; no https:// and no path)"`
+	SponsoredDomains      *string                             `json:"sponsored_domains,omitempty" jsonschema:"domains whose links always get sponsored, one per line (subdomains included)"`
+	AutolinkMax           *int                                `json:"autolink_max,omitempty" jsonschema:"the most phrases of the auto-link table linked in one text, 0 to 20 (default 3); 0 turns auto-linking off"`
 	VerifyGoogle          *string                             `json:"verify_google,omitempty" jsonschema:"the Google Search Console verification code"`
 	VerifyBing            *string                             `json:"verify_bing,omitempty" jsonschema:"the Bing verification code"`
 	VerifyYandex          *string                             `json:"verify_yandex,omitempty" jsonschema:"the Yandex verification code"`
 	VerifyOther           *[]models.VerifyMeta                `json:"verify_other,omitempty" jsonschema:"other verification meta tags (name, content)"`
 	VerifyFiles           *[]models.VerifyFile                `json:"verify_files,omitempty" jsonschema:"verification files served from the site root (filename, body)"`
 	CSVDelimiter          *string                             `json:"csv_delimiter,omitempty" jsonschema:"the import and export delimiter: , or ;"`
+	BreadcrumbHome        *string                             `json:"breadcrumb_home,omitempty" jsonschema:"the home page name in breadcrumb trails and BreadcrumbList (default locale); empty: Home / Strona główna"`
+	PublisherType         *string                             `json:"publisher_type,omitempty" jsonschema:"who publishes the site in the JSON-LD of every page: organization or person"`
+	PublisherName         *string                             `json:"publisher_name,omitempty" jsonschema:"the publisher name (default locale); empty: the site name"`
+	PublisherLogo         *string                             `json:"publisher_logo,omitempty" jsonschema:"the organisation logo (at least 112x112 px) or the person's photo: a URL or a site path"`
+	PublisherURL          *string                             `json:"publisher_url,omitempty" jsonschema:"the publisher website, an https URL; empty: the site address"`
+	PublisherEmail        *string                             `json:"publisher_email,omitempty" jsonschema:"the publisher contact email"`
+	PublisherPhone        *string                             `json:"publisher_phone,omitempty" jsonschema:"the publisher phone with the country code, e.g. +48 600 000 000"`
+	PublisherSameAs       *string                             `json:"publisher_same_as,omitempty" jsonschema:"profiles elsewhere (sameAs), one https URL per line"`
+	LocalEnabled          *bool                               `json:"local_enabled,omitempty" jsonschema:"publish the organisation as a local business (not for a person)"`
+	LocalType             *string                             `json:"local_type,omitempty" jsonschema:"the local business type: LocalBusiness, ProfessionalService, Store, FoodEstablishment, HealthAndBeautyBusiness, MedicalBusiness, LegalService, HomeAndConstructionBusiness, AutomotiveBusiness or FinancialService"`
+	LocalPriceRange       *string                             `json:"local_price_range,omitempty" jsonschema:"the price range, e.g. 50–200 zł or $$"`
+	LocalStreet           *string                             `json:"local_street,omitempty" jsonschema:"street and number (required for a local business)"`
+	LocalPostalCode       *string                             `json:"local_postal_code,omitempty" jsonschema:"postal code (required for a local business)"`
+	LocalCity             *string                             `json:"local_city,omitempty" jsonschema:"city (required for a local business)"`
+	LocalRegion           *string                             `json:"local_region,omitempty" jsonschema:"region"`
+	LocalCountry          *string                             `json:"local_country,omitempty" jsonschema:"two-letter country code, e.g. PL"`
+	LocalLatitude         *string                             `json:"local_latitude,omitempty" jsonschema:"latitude in decimal degrees (-90 to 90)"`
+	LocalLongitude        *string                             `json:"local_longitude,omitempty" jsonschema:"longitude in decimal degrees (-180 to 180)"`
+	PageForm              *string                             `json:"page_form,omitempty" jsonschema:"the paginated address form: query (/blog?page=2) or path (/blog/<segment>/2); the other form, ?page=1 and /<segment>/1 answer 301"`
+	PageSegment           *string                             `json:"page_segment,omitempty" jsonschema:"the path form's segment in the default locale, lowercase letters, digits and hyphens (default strona for pl, page for en)"`
+	PageTitleSuffix       *string                             `json:"page_title_suffix,omitempty" jsonschema:"added after the page title on page 2 and later (default locale); placeholders %page% and %pages%"`
+	PageDescriptionSuffix *string                             `json:"page_description_suffix,omitempty" jsonschema:"added at the end of the meta description on page 2 and later (default locale); placeholders %page% and %pages%"`
+	CrawlSchedule         *string                             `json:"crawl_schedule,omitempty" jsonschema:"the automatic crawl at night: off, daily or weekly (Mondays)"`
+	CrawlMaxPages         *int                                `json:"crawl_max_pages,omitempty" jsonschema:"pages per crawl, 10 to 50000 (default 5000)"`
+	CrawlDelay            *int                                `json:"crawl_delay,omitempty" jsonschema:"the pause between two crawler requests in milliseconds, 100 to 5000 (default 250); at most two requests run at a time"`
+	CrawlKeep             *int                                `json:"crawl_keep,omitempty" jsonschema:"crawls kept, 1 to 50 (default 10); older crawls are deleted with their pages and issues"`
+	GenericAnchors        *string                             `json:"generic_anchors,omitempty" jsonschema:"generic link texts, one per line; the crawl reports links whose whole text is one of them, ignoring letter case"`
+	LocalHours            *[]models.OpeningHours              `json:"local_hours,omitempty" jsonschema:"opening hours rows: day (mon … sun), opens and closes in 24-hour HH:MM; two rows split a day, a day without rows is closed"`
 	Types                 map[string]classes.TypeSettingWrite `json:"types,omitempty" jsonschema:"per content type key: inject (show the SEO tab), title_template, description_template, robots"`
 }
 

controllers/api/site.go

@@ -0,0 +1,29 @@
+package api
+
+import (
+	"net/http"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/summercms/modules/wire"
+)
+
+// Site serves GET /_seo/api/v1/site (P33-17): the site facts the frontend
+// layer needs (site URL, locales, the new-tab switch, the HTML sitemap
+// settings, the content types' URL patterns, the root file names and
+// whether llms.txt is served). Every value is public.
+func Site(service func() *classes.Service) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		svc := service()
+		if svc == nil {
+			wire.WriteJSON(w, http.StatusServiceUnavailable, map[string]string{"error": "SEO is not available"})
+			return
+		}
+		out, err := svc.Site(r.Context())
+		if err != nil {
+			log.Error("golem15.seo: site answer", "error", err)
+			wire.WriteJSON(w, http.StatusInternalServerError, map[string]string{"error": errInternal})
+			return
+		}
+		wire.WriteJSON(w, http.StatusOK, out)
+	}
+}

controllers/autolinks.go

@@ -0,0 +1,236 @@
+package controllers
+
+import (
+	"context"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/backpack"
+	"git.golem15.com/golem15/summercms/modules/lagoon"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
+)
+
+// AutolinksControllerID is the Auto-links screen (UI-SPEC C22).
+const AutolinksControllerID = "golem15.seo.autolinks"
+
+// ActionFromKeyphrases is the Auto-links toolbar action that adds the
+// records' focus keyphrases (UI-SPEC C22).
+const ActionFromKeyphrases = "from_keyphrases"
+
+// autolinksAdminController lists and edits the auto-link phrases (D-82).
+// History of a save comes from the plugin's own field-less form extension
+// on this controller, which runs inside the save transaction once the row
+// is written.
+type autolinksAdminController struct {
+	app func() *backpack.App
+}
+
+func (autolinksAdminController) ID() string        { return AutolinksControllerID }
+func (autolinksAdminController) ModelName() string { return `Golem15\SEO\Models\Autolink` }
+func (autolinksAdminController) ConfigDir() string { return "controllers/autolinks" }
+func (autolinksAdminController) RequiredPermissions() []string {
+	return []string{PermissionManage}
+}
+
+// NewRecord is a phrase with the form's defaults: every locale, active,
+// entered by hand.
+func (autolinksAdminController) NewRecord() any {
+	a := models.NewAutolink()
+	return &a
+}
+
+// FormBeforeCreate refuses a short phrase, an invalid address and a
+// phrase the locale already has (UI-SPEC C22); a create is manual.
+func (c autolinksAdminController) FormBeforeCreate(ctx context.Context, model any) error {
+	if a, ok := model.(*models.Autolink); ok && a != nil {
+		a.Origin = models.AutolinkOriginManual
+	}
+	return c.check(ctx, model)
+}
+
+// FormBeforeUpdate checks an update like a create.
+func (c autolinksAdminController) FormBeforeUpdate(ctx context.Context, model any) error {
+	return c.check(ctx, model)
+}
+
+func (c autolinksAdminController) check(ctx context.Context, model any) error {
+	a, ok := model.(*models.Autolink)
+	if !ok || a == nil {
+		return nil
+	}
+	svc, err := classes.NewService(appOf(c.app))
+	if err != nil {
+		return err
+	}
+	db, err := database(ctx, c.app)
+	if err != nil {
+		return err
+	}
+	problems, err := svc.AutolinkProblems(ctx, db, a)
+	if err != nil {
+		return err
+	}
+	return problemsError(ctx, translator(c.app), problems)
+}
+
+// FormAfterDelete records the deleted phrase in history inside the
+// delete's transaction.
+func (c autolinksAdminController) FormAfterDelete(ctx context.Context, model any) error {
+	a, ok := model.(*models.Autolink)
+	if !ok || a == nil {
+		return nil
+	}
+	svc, err := classes.NewService(appOf(c.app))
+	if err != nil {
+		return err
+	}
+	db, err := database(ctx, c.app)
+	if err != nil {
+		return err
+	}
+	return svc.RecordAutolink(ctx, db, classes.AutolinkValues(*a), *a, true)
+}
+
+// ListRowStates marks an inactive phrase disabled (autolinks.state_off).
+func (autolinksAdminController) ListRowStates(_ context.Context, _ *gorm.DB, records []any) ([][]pact.RowState, error) {
+	out := make([][]pact.RowState, len(records))
+	for i, record := range records {
+		if a, ok := record.(*models.Autolink); ok && a != nil && !a.Enabled {
+			out[i] = []pact.RowState{pact.RowStateDisabled}
+		}
+	}
+	return out, nil
+}
+
+// autolinkOriginLabels are the phrase keys of the origins.
+var autolinkOriginLabels = map[string]string{
+	models.AutolinkOriginManual:    "golem15.seo::lang.autolinks.origin_manual",
+	models.AutolinkOriginKeyphrase: "golem15.seo::lang.autolinks.origin_keyphrase",
+	models.AutolinkOriginMCP:       "golem15.seo::lang.autolinks.origin_mcp",
+}
+
+// locales returns the site's content locales, the default first.
+func (c autolinksAdminController) locales(ctx context.Context) []string {
+	svc, err := classes.NewService(appOf(c.app))
+	if err != nil {
+		return nil
+	}
+	db, err := database(ctx, c.app)
+	if err != nil {
+		return nil
+	}
+	return svc.Locales(ctx, db)
+}
+
+// FilterOptions serves the choices of the locale and origin filters.
+func (c autolinksAdminController) FilterOptions(scope string) []pact.Option {
+	switch scope {
+	case models.FilterAutolinkLocale:
+		out := []pact.Option{{Value: models.AutolinkLocaleAll, Label: "golem15.seo::lang.autolinks.locale_all"}}
+		for _, code := range c.locales(context.Background()) {
+			out = append(out, pact.Option{Value: code, Label: strings.ToUpper(code)})
+		}
+		return out
+	case models.FilterAutolinkOrigin:
+		out := make([]pact.Option, 0, len(models.AutolinkOrigins))
+		for _, o := range models.AutolinkOrigins {
+			out = append(out, pact.Option{Value: o, Label: autolinkOriginLabels[o]})
+		}
+		return out
+	}
+	return nil
+}
+
+// DropdownOptions serves the locale dropdown: each content locale (the
+// empty option is every locale).
+func (c autolinksAdminController) DropdownOptions(field string) []pact.Option {
+	if field != "locale" {
+		return nil
+	}
+	codes := c.locales(context.Background())
+	out := make([]pact.Option, 0, len(codes))
+	for _, code := range codes {
+		out = append(out, pact.Option{Value: code, Label: strings.ToUpper(code)})
+	}
+	return out
+}
+
+// ListExtendRecords shows the locale (all languages when empty) and the
+// origin of the list's rows as their labels in the request locale. The
+// rows are list values only and never saved.
+func (c autolinksAdminController) ListExtendRecords(ctx context.Context, _ *gorm.DB, records []any) error {
+	tr := translator(c.app)
+	for _, record := range records {
+		a, ok := record.(*models.Autolink)
+		if !ok || a == nil {
+			continue
+		}
+		if a.Locale == "" {
+			a.Locale = Phrase(ctx, tr, "golem15.seo::lang.autolinks.locale_all", nil)
+		} else {
+			a.Locale = strings.ToUpper(a.Locale)
+		}
+		if key, ok := autolinkOriginLabels[a.Origin]; ok {
+			a.Origin = Phrase(ctx, tr, key, nil)
+		}
+	}
+	return nil
+}
+
+// AdminActions is the toolbar's Add focus keyphrases.
+func (c autolinksAdminController) AdminActions() []pact.AdminAction {
+	return []pact.AdminAction{{
+		Name:        ActionFromKeyphrases,
+		Label:       "golem15.seo::lang.autolinks.from_keyphrases",
+		Permissions: []string{PermissionManage},
+		Run: func(ctx context.Context, _ pact.AdminActionInput) (pact.AdminActionResult, error) {
+			svc, err := classes.NewService(appOf(c.app))
+			if err != nil {
+				return pact.AdminActionResult{}, err
+			}
+			db, err := database(ctx, c.app)
+			if err != nil {
+				return pact.AdminActionResult{}, err
+			}
+			added := 0
+			err = lagoon.Transaction(ctx, db, func(ctx context.Context, tx *gorm.DB) error {
+				var err error
+				added, err = svc.FromKeyphrases(classes.WithTx(ctx, tx), tx)
+				return err
+			})
+			if err != nil {
+				return pact.AdminActionResult{}, err
+			}
+			return pact.AdminActionResult{Message: KeyphrasesMessage(ctx, c.app, added)}, nil
+		},
+	}}
+}
+
+// KeyphrasesMessage is the Add focus keyphrases toast: the count added,
+// or that there was nothing to add.
+func KeyphrasesMessage(ctx context.Context, app func() *backpack.App, added int) string {
+	tr := translator(app)
+	if added == 0 {
+		return Phrase(ctx, tr, "golem15.seo::lang.autolinks.keyphrases_none", nil)
+	}
+	if tr == nil {
+		return "golem15.seo::lang.autolinks.keyphrases_added"
+	}
+	return tr.Choice(ctx, "golem15.seo::lang.autolinks.keyphrases_added", added, nil)
+}
+
+var (
+	_ pact.AdminController         = autolinksAdminController{}
+	_ pact.AdminPermissioned       = autolinksAdminController{}
+	_ pact.AdminRecordSource       = autolinksAdminController{}
+	_ pact.FormBeforeCreate        = autolinksAdminController{}
+	_ pact.FormBeforeUpdate        = autolinksAdminController{}
+	_ pact.FormAfterDelete         = autolinksAdminController{}
+	_ pact.ListRowStates           = autolinksAdminController{}
+	_ pact.FilterOptions           = autolinksAdminController{}
+	_ pact.ListExtendRecords       = autolinksAdminController{}
+	_ pact.DropdownOptionsProvider = autolinksAdminController{}
+	_ pact.HasAdminActions         = autolinksAdminController{}
+)

controllers/autolinks/config_filter.yaml

@@ -0,0 +1,16 @@
+# Auto-links filters (UI-SPEC C22): locale, origin and active only.
+scopes:
+    locale:
+        label: golem15.seo::lang.autolinks.locale
+        modelClass: Golem15\SEO\Models\Autolink
+        nameFrom: locale
+        scope: FilterAutolinkLocale
+    origin:
+        label: golem15.seo::lang.autolinks.origin
+        modelClass: Golem15\SEO\Models\Autolink
+        nameFrom: origin
+        scope: FilterAutolinkOrigin
+    active:
+        label: golem15.seo::lang.autolinks.filter_active
+        type: switch
+        column: enabled

controllers/autolinks/config_form.yaml

@@ -0,0 +1,18 @@
+# Auto-link form (UI-SPEC C22).
+name: golem15.seo::lang.autolinks.name
+form: ~/plugins/golem15/seo/models/autolink/fields.yaml
+modelClass: Golem15\SEO\Models\Autolink
+defaultRedirect: golem15/seo/autolinks
+
+create:
+    redirect: golem15/seo/autolinks/update/:id
+    redirectClose: golem15/seo/autolinks
+
+update:
+    redirect: golem15/seo/autolinks
+    redirectClose: golem15/seo/autolinks
+
+messages:
+    create: golem15.seo::lang.autolinks.new
+    update: golem15.seo::lang.autolinks.name
+    deleteConfirm: golem15.seo::lang.autolinks.delete_confirm

controllers/autolinks/config_list.yaml

@@ -0,0 +1,26 @@
+# Auto-links list (UI-SPEC C22): the phrases by phrase; the toolbar
+# creates, deletes and adds the records' focus keyphrases.
+list: ~/plugins/golem15/seo/models/autolink/columns.yaml
+modelClass: Golem15\SEO\Models\Autolink
+title: golem15.seo::lang.autolinks.list_title
+recordUrl: golem15/seo/autolinks/update/:id
+noRecordsMessage: golem15.seo::lang.autolinks.list_empty
+recordsPerPage: 50
+showCheckboxes: true
+showSetup: true
+filter: config_filter.yaml
+
+defaultSort:
+    column: phrase
+    direction: asc
+
+toolbar:
+    buttons: [create, delete, from_keyphrases]
+    search:
+        prompt: backend::lang.list.search_prompt
+
+messages:
+    create: golem15.seo::lang.autolinks.new
+    empty: golem15.seo::lang.autolinks.list_empty
+    deleteConfirm: golem15.seo::lang.autolinks.delete_selected_confirm
+    rowStateDisabled: golem15.seo::lang.autolinks.state_off

controllers/crawls.go

@@ -0,0 +1,352 @@
+package controllers
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net/url"
+	"sort"
+	"strconv"
+	"time"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/classes/crawl"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/backpack"
+	"git.golem15.com/golem15/summercms/modules/cabana"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
+)
+
+// CrawlsControllerID is the Crawls screen (UI-SPEC C24).
+const CrawlsControllerID = "golem15.seo.crawls"
+
+// CrawlActionCancel is the preview's Cancel crawl.
+const CrawlActionCancel = "cancel"
+
+// errCrawlReadOnly refuses creating or editing crawls.
+var errCrawlReadOnly = errors.New("golem15.seo: crawls are read-only")
+
+// crawlsAdminController lists the crawls, newest first, and previews one
+// with P6 and Cancel while it is queued or running; the toolbar starts a
+// crawl and deletes the selected ones (a running crawl refuses).
+type crawlsAdminController struct {
+	app func() *backpack.App
+}
+
+func (crawlsAdminController) ID() string        { return CrawlsControllerID }
+func (crawlsAdminController) ModelName() string { return `Golem15\SEO\Models\Crawl` }
+func (crawlsAdminController) ConfigDir() string { return "controllers/crawls" }
+func (crawlsAdminController) NewRecord() any    { return &models.Crawl{} }
+func (crawlsAdminController) RequiredPermissions() []string {
+	return []string{PermissionManage}
+}
+
+// FormBeforeCreate refuses: crawls are started with Start crawl.
+func (crawlsAdminController) FormBeforeCreate(context.Context, any) error { return errCrawlReadOnly }
+
+// FormBeforeUpdate refuses: crawls are never edited.
+func (crawlsAdminController) FormBeforeUpdate(context.Context, any) error { return errCrawlReadOnly }
+
+// FormBeforeDelete refuses a queued or running crawl with
+// crawls.delete_running (403).
+func (c crawlsAdminController) FormBeforeDelete(ctx context.Context, model any) error {
+	row, ok := model.(*models.Crawl)
+	if ok && row != nil && row.Active() {
+		return &cabana.ForbiddenError{Message: Phrase(ctx, translator(c.app), "golem15.seo::lang.crawls.delete_running", nil)}
+	}
+	return nil
+}
+
+func (c crawlsAdminController) service() *classes.Service {
+	svc, err := classes.NewService(appOf(c.app))
+	if err != nil {
+		return nil
+	}
+	return svc
+}
+
+// AdminActions is the toolbar's Start crawl.
+func (c crawlsAdminController) AdminActions() []pact.AdminAction {
+	return []pact.AdminAction{startCrawl(c.app)}
+}
+
+// AdminRecordActions is Cancel crawl while queued or running; the pages
+// and issues fetched so far are kept.
+func (c crawlsAdminController) AdminRecordActions() []pact.AdminRecordAction {
+	return []pact.AdminRecordAction{{
+		Name:        CrawlActionCancel,
+		Label:       "golem15.seo::lang.crawls.cancel",
+		Confirm:     "golem15.seo::lang.crawls.cancel_confirm",
+		Permissions: []string{PermissionManage},
+		Applies: func(_ context.Context, record any) (bool, error) {
+			row, ok := record.(*models.Crawl)
+			return ok && row != nil && row.Active() && !row.Cancelling(), nil
+		},
+		Run: func(ctx context.Context, in pact.AdminRecordActionInput) (pact.AdminRecordActionResult, error) {
+			row, ok := in.Record.(*models.Crawl)
+			svc := c.service()
+			if !ok || row == nil || svc == nil {
+				return pact.AdminRecordActionResult{}, classes.ErrNotBooted
+			}
+			db, err := database(ctx, c.app)
+			if err != nil {
+				return pact.AdminRecordActionResult{}, err
+			}
+			_, name, _ := svc.Actor(ctx, db)
+			if err := crawl.Cancel(ctx, svc, db, *row, name); err != nil {
+				return pact.AdminRecordActionResult{}, err
+			}
+			return pact.AdminRecordActionResult{Message: "golem15.seo::lang.crawls.cancelled"}, nil
+		},
+	}}
+}
+
+// statusKeys are the phrase keys of the crawl states.
+var statusKeys = map[string]string{
+	models.CrawlQueued:    "golem15.seo::lang.crawls.st_queued",
+	models.CrawlRunning:   "golem15.seo::lang.crawls.st_running",
+	models.CrawlFinished:  "golem15.seo::lang.crawls.st_finished",
+	models.CrawlFailed:    "golem15.seo::lang.crawls.st_failed",
+	models.CrawlCancelled: "golem15.seo::lang.crawls.st_cancelled",
+}
+
+// TriggerLabel is the Started by cell: Manual: name, Schedule or MCP:
+// client.
+func TriggerLabel(ctx context.Context, svc *classes.Service, row models.Crawl) string {
+	switch row.Trigger {
+	case models.TriggerSchedule:
+		return svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.by_schedule", nil)
+	case models.TriggerMCP:
+		return svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.by_mcp", map[string]string{"client": row.TriggerName})
+	}
+	return svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.by_admin", map[string]string{"name": row.TriggerName})
+}
+
+// Duration is a crawl's duration, e.g. "4 min 12 s"; "—" before it ends.
+func Duration(row models.Crawl) string {
+	if row.StartedAt == nil {
+		return "—"
+	}
+	end := time.Now().UTC()
+	if row.FinishedAt != nil {
+		end = *row.FinishedAt
+	}
+	d := end.Sub(*row.StartedAt).Round(time.Second)
+	if d < 0 {
+		d = 0
+	}
+	h, m, s := int(d.Hours()), int(d.Minutes())%60, int(d.Seconds())%60
+	switch {
+	case h > 0:
+		return fmt.Sprintf("%d h %d min", h, m)
+	case m > 0:
+		return fmt.Sprintf("%d min %d s", m, s)
+	}
+	return fmt.Sprintf("%d s", s)
+}
+
+// issueCounts counts the issues of crawls.
+func issueCounts(db *gorm.DB, ids []uint) (map[uint]int, error) {
+	out := map[uint]int{}
+	if len(ids) == 0 {
+		return out, nil
+	}
+	var rows []struct {
+		CrawlID uint
+		N       int
+	}
+	if err := db.Raw(`SELECT crawl_id, COUNT(*) AS n FROM golem15_seo_issues WHERE crawl_id IN ? GROUP BY crawl_id`, ids).Scan(&rows).Error; err != nil {
+		return nil, err
+	}
+	for _, r := range rows {
+		out[r.CrawlID] = r.N
+	}
+	return out, nil
+}
+
+// ListExtendRecords shows the state, pages (with the limit note), issue
+// count, trigger and duration of each crawl in the request locale.
+func (c crawlsAdminController) ListExtendRecords(ctx context.Context, db *gorm.DB, records []any) error {
+	svc := c.service()
+	if svc == nil {
+		return nil
+	}
+	var ids []uint
+	for _, r := range records {
+		if row, ok := r.(*models.Crawl); ok && row != nil {
+			ids = append(ids, row.ID)
+		}
+	}
+	counts, err := issueCounts(db.WithContext(ctx), ids)
+	if err != nil {
+		return err
+	}
+	for _, r := range records {
+		row, ok := r.(*models.Crawl)
+		if !ok || row == nil {
+			continue
+		}
+		row.PagesCell = strconv.Itoa(row.Pages)
+		if row.LimitHit {
+			row.PagesCell = svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.pages_limit", map[string]string{"count": strconv.Itoa(row.Pages)})
+		}
+		row.IssuesCell = strconv.Itoa(counts[row.ID])
+		row.DurationCell = Duration(*row)
+		row.TriggerCell = TriggerLabel(ctx, svc, *row)
+		if row.Cancelling() {
+			row.Status = svc.PhraseCtx(ctx, "golem15.seo::lang.crawls.st_cancelling", nil)
+		} else if key, ok := statusKeys[row.Status]; ok {
+			row.Status = svc.PhraseCtx(ctx, key, nil)
+		}
+	}
+	return nil
+}
+
+// DropdownOptions labels the preview's read-only status and trigger.
+func (crawlsAdminController) DropdownOptions(field string) []pact.Option {
+	switch field {
+	case "status":
+		out := make([]pact.Option, 0, len(models.CrawlStatuses))
+		for _, s := range models.CrawlStatuses {
+			out = append(out, pact.Option{Value: s, Label: statusKeys[s]})
+		}
+		return out
+	case "trigger":
+		return []pact.Option{
+			{Value: models.TriggerAdmin, Label: "golem15.seo::lang.crawls.trigger_admin"},
+			{Value: models.TriggerSchedule, Label: "golem15.seo::lang.crawls.trigger_schedule"},
+			{Value: models.TriggerMCP, Label: "golem15.seo::lang.crawls.trigger_mcp"},
+		}
+	}
+	return nil
+}
+
+// FilterOptions serves the status filter.
+func (c crawlsAdminController) FilterOptions(scope string) []pact.Option {
+	if scope != models.FilterCrawlStatus {
+		return nil
+	}
+	return c.DropdownOptions("status")
+}
+
+// PartialData serves P6 crawl_summary (UI-SPEC C24).
+func (c crawlsAdminController) PartialData(ctx context.Context, name string, record any) (any, error) {
+	row, ok := record.(*models.Crawl)
+	svc := c.service()
+	if name != "crawl_summary" || !ok || row == nil || svc == nil {
+		return classes.CrawlSummaryView{}, nil
+	}
+	db, err := database(ctx, c.app)
+	if err != nil {
+		return nil, err
+	}
+	return CrawlSummary(ctx, svc, db, c.app, *row)
+}
+
+// CrawlSummary builds P6: one status callout, the staging and rendered
+// comparison lines, the stats strip and the per-kind table whose counts
+// link to the Site audit list filtered by the kind.
+func CrawlSummary(ctx context.Context, svc *classes.Service, db *gorm.DB, app func() *backpack.App, row models.Crawl) (classes.CrawlSummaryView, error) {
+	view := classes.CrawlSummaryView{}
+	p := func(key string, params map[string]string) string {
+		return svc.PhraseCtx(ctx, "golem15.seo::lang.crawls."+key, params)
+	}
+	stamp := func(t *time.Time) string {
+		if t == nil {
+			return "—"
+		}
+		return t.UTC().Format("2006-01-02 15:04 UTC")
+	}
+	switch row.Status {
+	case models.CrawlRunning:
+		view.Title = p("running_title", nil)
+		view.Text = p("running_text", map[string]string{"done": strconv.Itoa(row.Pages), "limit": strconv.Itoa(row.MaxPages)})
+	case models.CrawlQueued:
+		view.Text = p("queued_text", nil)
+	case models.CrawlFailed:
+		view.Tone = "danger"
+		view.Title = p("failed_title", nil)
+		view.Text = p("failed_text", map[string]string{"error": row.Error})
+	case models.CrawlCancelled:
+		view.Text = p("cancelled_text", map[string]string{"name": row.CancelledBy, "time": stamp(row.FinishedAt)})
+	case models.CrawlFinished:
+		if row.LimitHit {
+			view.Tone = "warning"
+			view.Text = p("limit_text", map[string]string{"limit": strconv.Itoa(row.MaxPages)})
+		}
+	}
+	view.HasCallout = view.Text != ""
+	if svc.Staging() {
+		view.Staging = p("staging_text", nil)
+	}
+	switch row.RenderStatus {
+	case models.RenderDone:
+		view.Render = svc.ChoiceCtx(ctx, "golem15.seo::lang.crawls.render_done", row.RenderPages, nil)
+	case models.RenderFailed:
+		view.Render = p("render_failed", map[string]string{"error": row.RenderNote})
+	default:
+		view.Render = p("render_skipped", nil)
+	}
+	var rows []struct {
+		Kind     string
+		Severity string
+		N        int
+	}
+	if err := db.Raw(`SELECT kind, severity, COUNT(*) AS n FROM golem15_seo_issues WHERE crawl_id = ? GROUP BY kind, severity`, row.ID).Scan(&rows).Error; err != nil {
+		return view, err
+	}
+	sevCount := map[string]int{}
+	prefix, err := cabana.AdminPrefix(appOf(app))
+	if err != nil {
+		prefix = cabana.DefaultAdminPrefix
+	}
+	for _, r := range rows {
+		sevCount[r.Severity] += r.N
+		view.Kinds = append(view.Kinds, classes.CrawlKindRow{
+			Kind:     r.Kind,
+			Label:    svc.PhraseCtx(ctx, "golem15.seo::lang.issues.kind_"+r.Kind, nil),
+			Severity: svc.PhraseCtx(ctx, labelKeys[r.Severity], nil),
+			Count:    r.N,
+			Link:     prefix + "/golem15/seo/issues?" + url.Values{"filter[kind]": {r.Kind}}.Encode(),
+		})
+	}
+	rank := map[string]int{}
+	for _, r := range rows {
+		rank[r.Kind] = models.SeverityRank(r.Severity)
+	}
+	sort.SliceStable(view.Kinds, func(i, j int) bool {
+		a, b := view.Kinds[i], view.Kinds[j]
+		if rank[a.Kind] != rank[b.Kind] {
+			return rank[a.Kind] < rank[b.Kind]
+		}
+		if a.Count != b.Count {
+			return a.Count > b.Count
+		}
+		return a.Kind < b.Kind
+	})
+	view.NoIssues = len(view.Kinds) == 0 && !row.Active()
+	view.Stats = []classes.CrawlStat{
+		{Label: p("stat_pages", nil), Value: strconv.Itoa(row.Pages)},
+		{Label: p("stat_errors", nil), Value: strconv.Itoa(sevCount[models.SeverityError])},
+		{Label: p("stat_warnings", nil), Value: strconv.Itoa(sevCount[models.SeverityWarning])},
+		{Label: p("stat_notices", nil), Value: strconv.Itoa(sevCount[models.SeverityNotice])},
+		{Label: p("stat_duration", nil), Value: Duration(row)},
+	}
+	return view, nil
+}
+
+var (
+	_ pact.AdminController         = crawlsAdminController{}
+	_ pact.AdminPermissioned       = crawlsAdminController{}
+	_ pact.AdminRecordSource       = crawlsAdminController{}
+	_ pact.FormBeforeCreate        = crawlsAdminController{}
+	_ pact.FormBeforeUpdate        = crawlsAdminController{}
+	_ pact.FormBeforeDelete        = crawlsAdminController{}
+	_ pact.HasAdminActions         = crawlsAdminController{}
+	_ pact.HasAdminRecordActions   = crawlsAdminController{}
+	_ pact.ListExtendRecords       = crawlsAdminController{}
+	_ pact.DropdownOptionsProvider = crawlsAdminController{}
+	_ pact.FilterOptions           = crawlsAdminController{}
+	_ pact.AdminPartialData        = crawlsAdminController{}
+)

controllers/crawls/_crawl_summary.htm

@@ -0,0 +1,31 @@
+{{- with .Data -}}
+{{- if .HasCallout -}}
+<div class="summer-callout{{ if .Tone }} summer-callout--{{ .Tone }}{{ end }}" role="status">
+{{- if .Title -}}<p class="summer-callout__title">{{ .Title }}</p>{{- end -}}
+<p class="summer-callout__text">{{ .Text }}</p>
+</div>
+{{- end -}}
+{{- if .Staging -}}<p>{{ .Staging }}</p>{{- end -}}
+{{- if .Render -}}<p>{{ .Render }}</p>{{- end -}}
+{{- if .Stats -}}
+<dl class="summer-stats">
+{{- range .Stats -}}
+<div class="summer-stat"><dt class="summer-stat__label">{{ .Label }}</dt><dd class="summer-stat__value">{{ .Value }}</dd></div>
+{{- end -}}
+</dl>
+{{- end -}}
+{{- if .Kinds -}}
+<div class="summer-table-wrap">
+<table class="summer-table">
+<thead><tr><th>{{ trans "golem15.seo::lang.crawls.kinds_problem" }}</th><th>{{ trans "golem15.seo::lang.crawls.kinds_severity" }}</th><th class="summer-table__num">{{ trans "golem15.seo::lang.crawls.kinds_count" }}</th></tr></thead>
+<tbody>
+{{- range .Kinds -}}
+<tr><td>{{ .Label }}</td><td>{{ .Severity }}</td><td class="summer-table__num"><a href="{{ .Link }}">{{ .Count }}</a></td></tr>
+{{- end -}}
+</tbody>
+</table>
+</div>
+{{- else if .NoIssues -}}
+<p>{{ trans "golem15.seo::lang.crawls.no_issues" }}</p>
+{{- end -}}
+{{- end -}}

controllers/crawls/config_filter.yaml

@@ -0,0 +1,11 @@
+# Crawls filters (UI-SPEC C24): status and start date.
+scopes:
+    status:
+        label: golem15.seo::lang.crawls.status
+        modelClass: Golem15\SEO\Models\Crawl
+        nameFrom: status
+        scope: FilterCrawlStatus
+    started_at:
+        label: golem15.seo::lang.crawls.started_at
+        type: daterange
+        column: started_at

controllers/crawls/config_form.yaml

@@ -0,0 +1,14 @@
+# Crawl preview (UI-SPEC C24): read-only, P6 above the fields and Cancel
+# while the crawl is queued or running.
+name: golem15.seo::lang.crawls.name
+form: ~/plugins/golem15/seo/models/crawl/fields.yaml
+modelClass: Golem15\SEO\Models\Crawl
+defaultRedirect: golem15/seo/crawls
+
+preview:
+    headerPartial: crawl_summary
+
+recordActions: [cancel]
+
+messages:
+    preview: golem15.seo::lang.crawls.preview_title

controllers/crawls/config_list.yaml

@@ -0,0 +1,22 @@
+# Crawls (UI-SPEC C24): newest first; the toolbar starts a crawl and
+# deletes the selected ones (a running crawl refuses).
+list: ~/plugins/golem15/seo/models/crawl/columns.yaml
+modelClass: Golem15\SEO\Models\Crawl
+title: golem15.seo::lang.crawls.list_title
+recordUrl: golem15/seo/crawls/preview/:id
+noRecordsMessage: golem15.seo::lang.crawls.list_empty
+recordsPerPage: 20
+showCheckboxes: true
+showSetup: true
+filter: config_filter.yaml
+
+defaultSort:
+    column: started_at
+    direction: desc
+
+toolbar:
+    buttons: [start_crawl, delete]
+
+messages:
+    empty: golem15.seo::lang.crawls.list_empty
+    deleteConfirm: golem15.seo::lang.crawls.delete_selected_confirm

controllers/issues.go

@@ -0,0 +1,505 @@
+package controllers
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strings"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/classes/crawl"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/backpack"
+	"git.golem15.com/golem15/summercms/modules/cabana"
+	"git.golem15.com/golem15/summercms/modules/lagoon"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
+)
+
+// IssuesControllerID is the Site audit (UI-SPEC C23).
+const IssuesControllerID = "golem15.seo.issues"
+
+// The Site audit's actions: the toolbar's start, the record actions and
+// the bulk ignore.
+const (
+	ActionStartCrawl  = "start_crawl"
+	IssueActionCheck  = "recheck"
+	IssueActionIgnore = "ignore"
+	IssueActionShow   = "unignore"
+)
+
+// errIssueReadOnly refuses creating or editing issues: the audit writes
+// them.
+var errIssueReadOnly = errors.New("golem15.seo: issues are read-only")
+
+// issuesAdminController lists the audit's current issues, errors first,
+// and previews one with P7 and its record actions (check again, ignore,
+// stop ignoring); the toolbar starts a crawl and the bulk action ignores.
+type issuesAdminController struct {
+	app func() *backpack.App
+}
+
+func (issuesAdminController) ID() string        { return IssuesControllerID }
+func (issuesAdminController) ModelName() string { return `Golem15\SEO\Models\Issue` }
+func (issuesAdminController) ConfigDir() string { return "controllers/issues" }
+func (issuesAdminController) NewRecord() any    { return &models.Issue{} }
+func (issuesAdminController) RequiredPermissions() []string {
+	return []string{PermissionManage}
+}
+
+// FormBeforeCreate refuses: the audit writes issues.
+func (issuesAdminController) FormBeforeCreate(context.Context, any) error { return errIssueReadOnly }
+
+// FormBeforeUpdate refuses: issues are never edited.
+func (issuesAdminController) FormBeforeUpdate(context.Context, any) error { return errIssueReadOnly }
+
+func (c issuesAdminController) service() *classes.Service {
+	svc, err := classes.NewService(appOf(c.app))
+	if err != nil {
+		return nil
+	}
+	return svc
+}
+
+// ListExtendQuery narrows the list to the current issues (the newest
+// crawl's; before any crawl the CMS-data issues, checked now) and puts
+// errors first, then warnings, then notices.
+func (c issuesAdminController) ListExtendQuery(ctx context.Context, q *gorm.DB) *gorm.DB {
+	svc := c.service()
+	db, err := database(ctx, c.app)
+	if svc == nil || err != nil {
+		return q.Where("1 = 0")
+	}
+	latest, err := crawl.Current(ctx, svc, db)
+	if err != nil {
+		log.Warn("golem15.seo: site audit list", "error", err)
+		return q.Where("1 = 0")
+	}
+	return crawl.CurrentScope(q, latest).Order("golem15_seo_issues.rank ASC")
+}
+
+// startCrawl is the toolbar's Start crawl (also on Crawls): it refuses with
+// crawls.already_running or crawls.no_site_url as an action error and
+// toasts crawls.started.
+func startCrawl(app func() *backpack.App) pact.AdminAction {
+	return pact.AdminAction{
+		Name:        ActionStartCrawl,
+		Label:       "golem15.seo::lang.crawls.start",
+		Permissions: []string{PermissionManage},
+		Run: func(ctx context.Context, _ pact.AdminActionInput) (pact.AdminActionResult, error) {
+			svc, err := classes.NewService(appOf(app))
+			if err != nil {
+				return pact.AdminActionResult{}, err
+			}
+			db, err := database(ctx, app)
+			if err != nil {
+				return pact.AdminActionResult{}, err
+			}
+			err = lagoon.Transaction(ctx, db, func(ctx context.Context, tx *gorm.DB) error {
+				_, name, _ := svc.Actor(ctx, tx)
+				_, err := crawl.Start(ctx, svc, tx, models.TriggerAdmin, name)
+				return err
+			})
+			if err != nil {
+				return pact.AdminActionResult{}, startRefusal(ctx, app, err)
+			}
+			return pact.AdminActionResult{Message: "golem15.seo::lang.crawls.started"}, nil
+		},
+	}
+}
+
+// startRefusal turns a refused start into the admin's action error.
+func startRefusal(ctx context.Context, app func() *backpack.App, err error) error {
+	tr := translator(app)
+	var running *crawl.AlreadyRunningError
+	switch {
+	case errors.As(err, &running):
+		return &cabana.ForbiddenError{Message: Phrase(ctx, tr, "golem15.seo::lang.crawls.already_running", map[string]string{"time": running.Since().UTC().Format("2006-01-02 15:04 UTC")})}
+	case errors.Is(err, crawl.ErrNoSiteURL):
+		return &cabana.ForbiddenError{Message: Phrase(ctx, tr, "golem15.seo::lang.crawls.no_site_url", nil)}
+	}
+	return err
+}
+
+// AdminActions is the toolbar's Start crawl.
+func (c issuesAdminController) AdminActions() []pact.AdminAction {
+	return []pact.AdminAction{startCrawl(c.app)}
+}
+
+// ignored reports whether a record is an ignored issue.
+func (c issuesAdminController) ignored(ctx context.Context, record any) (bool, error) {
+	issue, ok := record.(*models.Issue)
+	if !ok || issue == nil {
+		return false, nil
+	}
+	db, err := database(ctx, c.app)
+	if err != nil {
+		return false, err
+	}
+	return crawl.IsIgnored(db, *issue)
+}
+
+// AdminRecordActions are the preview's Check this address again and Ignore
+// (while open) and Stop ignoring (while ignored).
+func (c issuesAdminController) AdminRecordActions() []pact.AdminRecordAction {
+	open := func(ctx context.Context, record any) (bool, error) {
+		ig, err := c.ignored(ctx, record)
+		return !ig, err
+	}
+	return []pact.AdminRecordAction{
+		{
+			Name:        IssueActionCheck,
+			Label:       "golem15.seo::lang.issues.recheck",
+			Confirm:     "golem15.seo::lang.issues.recheck_confirm",
+			Permissions: []string{PermissionManage},
+			Applies:     open,
+			Run: func(ctx context.Context, in pact.AdminRecordActionInput) (pact.AdminRecordActionResult, error) {
+				issue, ok := in.Record.(*models.Issue)
+				svc := c.service()
+				if !ok || issue == nil || svc == nil {
+					return pact.AdminRecordActionResult{}, classes.ErrNotBooted
+				}
+				db, err := database(ctx, c.app)
+				if err != nil {
+					return pact.AdminRecordActionResult{}, err
+				}
+				res, err := crawl.Recheck(ctx, svc, db, *issue)
+				if err != nil {
+					if errors.Is(err, crawl.ErrNoSiteURL) {
+						return pact.AdminRecordActionResult{}, &cabana.ForbiddenError{Message: Phrase(ctx, translator(c.app), "golem15.seo::lang.crawls.no_site_url", nil)}
+					}
+					return pact.AdminRecordActionResult{}, err
+				}
+				return pact.AdminRecordActionResult{Message: RecheckMessage(ctx, svc, res)}, nil
+			},
+		},
+		{
+			Name:        IssueActionIgnore,
+			Label:       "golem15.seo::lang.issues.ignore",
+			Confirm:     "golem15.seo::lang.issues.ignore_confirm",
+			Permissions: []string{PermissionManage},
+			Applies:     open,
+			Run: func(ctx context.Context, in pact.AdminRecordActionInput) (pact.AdminRecordActionResult, error) {
+				return c.ignore(ctx, []any{in.Record})
+			},
+		},
+		{
+			Name:        IssueActionShow,
+			Label:       "golem15.seo::lang.issues.unignore",
+			Confirm:     "golem15.seo::lang.issues.unignore_confirm",
+			Permissions: []string{PermissionManage},
+			Applies:     c.ignored,
+			Run: func(ctx context.Context, in pact.AdminRecordActionInput) (pact.AdminRecordActionResult, error) {
+				issue, ok := in.Record.(*models.Issue)
+				if !ok || issue == nil {
+					return pact.AdminRecordActionResult{}, classes.ErrNotBooted
+				}
+				db, err := database(ctx, c.app)
+				if err != nil {
+					return pact.AdminRecordActionResult{}, err
+				}
+				n, err := crawl.Unignore(db, []models.Issue{*issue})
+				if err != nil {
+					return pact.AdminRecordActionResult{}, err
+				}
+				svc := c.service()
+				return pact.AdminRecordActionResult{Message: svc.ChoiceCtx(ctx, "golem15.seo::lang.issues.unignored", n, nil)}, nil
+			},
+		},
+	}
+}
+
+// RecheckMessage is the toast of a recheck: issues.rechecked with the
+// remaining count, or issues.recheck_fixed.
+func RecheckMessage(ctx context.Context, svc *classes.Service, res crawl.RecheckResult) string {
+	if res.Remaining == 0 {
+		return svc.PhraseCtx(ctx, "golem15.seo::lang.issues.recheck_fixed", nil)
+	}
+	return svc.ChoiceCtx(ctx, "golem15.seo::lang.issues.rechecked", res.Remaining, nil)
+}
+
+// ignore stores the fingerprints of records as ignored.
+func (c issuesAdminController) ignore(ctx context.Context, records []any) (pact.AdminRecordActionResult, error) {
+	svc := c.service()
+	db, err := database(ctx, c.app)
+	if err != nil || svc == nil {
+		return pact.AdminRecordActionResult{}, errors.Join(err, classes.ErrNotBooted)
+	}
+	var list []models.Issue
+	for _, r := range records {
+		if issue, ok := r.(*models.Issue); ok && issue != nil {
+			list = append(list, *issue)
+		}
+	}
+	adminID, _, _ := svc.Actor(ctx, db)
+	n, err := crawl.Ignore(db, list, adminID)
+	if err != nil {
+		return pact.AdminRecordActionResult{}, err
+	}
+	return pact.AdminRecordActionResult{Message: svc.ChoiceCtx(ctx, "golem15.seo::lang.issues.ignored", n, nil)}, nil
+}
+
+// AdminBulkActions is the list's bulk Ignore issues.
+func (c issuesAdminController) AdminBulkActions() []pact.AdminBulkAction {
+	return []pact.AdminBulkAction{{
+		Name:        IssueActionIgnore,
+		Label:       "golem15.seo::lang.issues.ignore_selected",
+		Confirm:     "golem15.seo::lang.issues.ignore_selected_confirm",
+		Permissions: []string{PermissionManage},
+		Run: func(ctx context.Context, in pact.AdminBulkActionInput) (pact.AdminBulkActionResult, error) {
+			res, err := c.ignore(ctx, in.Records)
+			if err != nil {
+				return pact.AdminBulkActionResult{}, err
+			}
+			return pact.AdminBulkActionResult{Message: res.Message, Affected: len(in.Records)}, nil
+		},
+	}}
+}
+
+// ListRowStates marks ignored issues disabled (issues.state_ignored).
+func (issuesAdminController) ListRowStates(ctx context.Context, db *gorm.DB, records []any) ([][]pact.RowState, error) {
+	var fps []string
+	for _, r := range records {
+		if issue, ok := r.(*models.Issue); ok && issue != nil {
+			fps = append(fps, issue.Fingerprint)
+		}
+	}
+	set, err := crawl.IgnoredSet(db.WithContext(ctx), fps)
+	if err != nil {
+		return nil, err
+	}
+	out := make([][]pact.RowState, len(records))
+	for i, r := range records {
+		if issue, ok := r.(*models.Issue); ok && issue != nil && set[issue.Fingerprint] {
+			out[i] = []pact.RowState{pact.RowStateDisabled}
+		}
+	}
+	return out, nil
+}
+
+// labelKeys are the phrase keys of the stored severity and source values.
+var labelKeys = map[string]string{
+	models.SeverityError:   "golem15.seo::lang.issues.sev_error",
+	models.SeverityWarning: "golem15.seo::lang.issues.sev_warning",
+	models.SeverityNotice:  "golem15.seo::lang.issues.sev_notice",
+	models.SourceCrawl:     "golem15.seo::lang.issues.source_crawl",
+	models.SourceCMS:       "golem15.seo::lang.issues.source_cms",
+}
+
+// FixCell is the Fix column: issues.fix_tool with the tool, or
+// issues.fix_code.
+func FixCell(ctx context.Context, svc *classes.Service, issue models.Issue) string {
+	if issue.Fix == models.FixTool {
+		return svc.PhraseCtx(ctx, "golem15.seo::lang.issues.fix_tool", map[string]string{"tool": issue.Tool})
+	}
+	return svc.PhraseCtx(ctx, "golem15.seo::lang.issues.fix_code", nil)
+}
+
+// ListExtendRecords shows the severity, kind and source labels, the
+// one-line summary and the fix route in the request locale.
+func (c issuesAdminController) ListExtendRecords(ctx context.Context, _ *gorm.DB, records []any) error {
+	svc := c.service()
+	if svc == nil {
+		return nil
+	}
+	for _, r := range records {
+		issue, ok := r.(*models.Issue)
+		if !ok || issue == nil {
+			continue
+		}
+		issue.Summary = svc.IssueSummary(ctx, issue.Kind, issue.DetailsMap())
+		issue.FixCell = FixCell(ctx, svc, *issue)
+		issue.KindCell = svc.PhraseCtx(ctx, "golem15.seo::lang.issues.kind_"+issue.Kind, nil)
+		if key, ok := labelKeys[issue.Severity]; ok {
+			issue.Severity = svc.PhraseCtx(ctx, key, nil)
+		}
+		if key, ok := labelKeys[issue.Source]; ok {
+			issue.Source = svc.PhraseCtx(ctx, key, nil)
+		}
+	}
+	return nil
+}
+
+// DropdownOptions labels the preview's read-only kind, severity and
+// source fields.
+func (issuesAdminController) DropdownOptions(field string) []pact.Option {
+	switch field {
+	case "kind":
+		out := make([]pact.Option, 0, len(crawl.Kinds()))
+		for _, k := range crawl.Kinds() {
+			out = append(out, pact.Option{Value: k, Label: "golem15.seo::lang.issues.kind_" + k})
+		}
+		return out
+	case "severity":
+		return []pact.Option{
+			{Value: models.SeverityError, Label: labelKeys[models.SeverityError]},
+			{Value: models.SeverityWarning, Label: labelKeys[models.SeverityWarning]},
+			{Value: models.SeverityNotice, Label: labelKeys[models.SeverityNotice]},
+		}
+	case "source":
+		return []pact.Option{
+			{Value: models.SourceCrawl, Label: labelKeys[models.SourceCrawl]},
+			{Value: models.SourceCMS, Label: labelKeys[models.SourceCMS]},
+		}
+	}
+	return nil
+}
+
+// FilterOptions serves the severity, kind, source, fix and status filters.
+func (c issuesAdminController) FilterOptions(scope string) []pact.Option {
+	switch scope {
+	case models.FilterIssueSeverity:
+		return c.DropdownOptions("severity")
+	case models.FilterIssueKind:
+		return c.DropdownOptions("kind")
+	case models.FilterIssueSource:
+		return c.DropdownOptions("source")
+	case models.FilterIssueFix:
+		return []pact.Option{
+			{Value: models.FixTool, Label: "golem15.seo::lang.issues.filter_fix_tool"},
+			{Value: models.FixCode, Label: "golem15.seo::lang.issues.filter_fix_code"},
+		}
+	case models.FilterIssueStatus:
+		return []pact.Option{
+			{Value: models.IssueOpen, Label: "golem15.seo::lang.issues.status_open"},
+			{Value: models.IssueIgnored, Label: "golem15.seo::lang.issues.status_ignored"},
+		}
+	}
+	return nil
+}
+
+// maxLinkedShown caps P7's linking pages before issues.and_more.
+const maxLinkedShown = 10
+
+// PartialData serves P7 issue_details (UI-SPEC C23).
+func (c issuesAdminController) PartialData(ctx context.Context, name string, record any) (any, error) {
+	issue, ok := record.(*models.Issue)
+	svc := c.service()
+	if name != "issue_details" || !ok || issue == nil || svc == nil {
+		return classes.IssueDetailsView{}, nil
+	}
+	return IssueDetails(ctx, svc, c.app, *issue), nil
+}
+
+// IssueDetails builds P7: the kind's found rows, the linking pages, the
+// orphan suggestions, how to fix it and the page address.
+func IssueDetails(ctx context.Context, svc *classes.Service, app func() *backpack.App, issue models.Issue) classes.IssueDetailsView {
+	d := issue.DetailsMap()
+	view := classes.IssueDetailsView{}
+	label := func(key string) string { return svc.PhraseCtx(ctx, "golem15.seo::lang.issues."+key, nil) }
+	view.Rows = append(view.Rows, classes.IssueRow{Label: label("summary"), Value: svc.IssueSummary(ctx, issue.Kind, d)})
+	add := func(labelKey string, value any, mono bool) {
+		if value == nil {
+			return
+		}
+		text := fmt.Sprint(value)
+		if list, ok := value.([]any); ok {
+			parts := make([]string, 0, len(list))
+			for _, e := range list {
+				switch x := e.(type) {
+				case map[string]any:
+					var kv []string
+					for _, k := range []string{"type", "property", "text", "target", "locale", "url", "status", "from", "to"} {
+						if v, ok := x[k]; ok {
+							kv = append(kv, fmt.Sprint(v))
+						}
+					}
+					parts = append(parts, strings.Join(kv, " · "))
+				default:
+					parts = append(parts, fmt.Sprint(x))
+				}
+			}
+			text = strings.Join(parts, "; ")
+		}
+		if strings.TrimSpace(text) == "" {
+			return
+		}
+		view.Rows = append(view.Rows, classes.IssueRow{Label: label(labelKey), Value: text, Mono: mono})
+	}
+	add("detail_h1", d["h1"], false)
+	add("detail_skips", d["skips"], false)
+	add("detail_missing", d["missing"], false)
+	add("detail_error", d["error"], true)
+	add("detail_anchors", d["anchors"], false)
+	add("detail_alternates", d["alternates"], true)
+	add("detail_images", d["images"], true)
+	add("detail_target", d["target"], true)
+	add("detail_what", d["what"], false)
+	if from, ok := d["from"].([]any); ok && issue.Kind != classes.IssueImageAlt {
+		if _, isLevel := d["to"]; !isLevel {
+			for i, f := range from {
+				if i >= maxLinkedShown {
+					break
+				}
+				view.LinkedFrom = append(view.LinkedFrom, classes.IssueLink{URL: fmt.Sprint(f)})
+			}
+			total := len(from)
+			if n, ok := d["count"].(float64); ok && int(n) > total {
+				total = int(n)
+			}
+			if total > len(view.LinkedFrom) {
+				view.More = svc.PhraseCtx(ctx, "golem15.seo::lang.issues.and_more", map[string]string{"count": fmt.Sprint(total - len(view.LinkedFrom))})
+			}
+		}
+	}
+	if issue.Kind == crawl.KindOrphanPage {
+		view.Orphan = true
+		if list, ok := d["suggestions"].([]any); ok {
+			for _, e := range list {
+				s, _ := e.(map[string]any)
+				u := fmt.Sprint(s["url"])
+				view.Suggestions = append(view.Suggestions, classes.IssueLink{URL: u, Phrase: fmt.Sprint(s["phrase"])})
+			}
+		}
+	}
+	if issue.Fix == models.FixTool {
+		view.FixText = svc.PhraseCtx(ctx, "golem15.seo::lang.issues.fix_tool_text", map[string]string{"tool": issue.Tool, "where": fixWhere(ctx, svc, issue)})
+	} else {
+		what, _ := d["what"].(string)
+		view.FixText = svc.PhraseCtx(ctx, "golem15.seo::lang.issues.fix_code_text", map[string]string{"what": what})
+	}
+	site := svc.SiteURL()
+	if strings.HasPrefix(issue.URL, "/") && site != "" {
+		view.PageURL = site + issue.URL
+	}
+	if strings.HasPrefix(issue.Kind, "jsonld_") && view.PageURL != "" {
+		view.RichResults = true
+		view.RichURL = "https://search.google.com/test/rich-results?url=" + view.PageURL
+	}
+	return view
+}
+
+// fixWhere names the admin screen where a CMS tool's value is changed.
+func fixWhere(ctx context.Context, svc *classes.Service, issue models.Issue) string {
+	key := "golem15.seo::lang.issues.where_record"
+	switch issue.Tool {
+	case crawl.ToolUpdateRobots:
+		key = "golem15.seo::lang.issues.where_robots"
+	case crawl.ToolUpdateSettings:
+		key = "golem15.seo::lang.issues.where_settings"
+	case crawl.ToolCreateRedirect, "seo_update_redirect":
+		key = "golem15.seo::lang.issues.where_redirects"
+	case crawl.ToolCreatePage, crawl.ToolUpdatePage:
+		key = "golem15.seo::lang.issues.where_pages"
+	case "seo_update_rule":
+		key = "golem15.seo::lang.issues.where_rules"
+	}
+	return svc.PhraseCtx(ctx, key, nil)
+}
+
+var (
+	_ pact.AdminController         = issuesAdminController{}
+	_ pact.AdminPermissioned       = issuesAdminController{}
+	_ pact.AdminRecordSource       = issuesAdminController{}
+	_ pact.FormBeforeCreate        = issuesAdminController{}
+	_ pact.FormBeforeUpdate        = issuesAdminController{}
+	_ pact.ListExtendQuery         = issuesAdminController{}
+	_ pact.HasAdminActions         = issuesAdminController{}
+	_ pact.HasAdminRecordActions   = issuesAdminController{}
+	_ pact.HasAdminBulkActions     = issuesAdminController{}
+	_ pact.ListRowStates           = issuesAdminController{}
+	_ pact.ListExtendRecords       = issuesAdminController{}
+	_ pact.DropdownOptionsProvider = issuesAdminController{}
+	_ pact.FilterOptions           = issuesAdminController{}
+	_ pact.AdminPartialData        = issuesAdminController{}
+)

controllers/issues/_issue_details.htm

@@ -0,0 +1,48 @@
+{{- with .Data -}}
+{{- if .Rows -}}
+<p><strong>{{ trans "golem15.seo::lang.issues.found_heading" }}</strong></p>
+<div class="summer-table-wrap">
+<table class="summer-table">
+<tbody>
+{{- range .Rows -}}
+<tr><th scope="row">{{ .Label }}</th><td>{{ if .Mono }}<code>{{ .Value }}</code>{{ else }}{{ .Value }}{{ end }}</td></tr>
+{{- end -}}
+</tbody>
+</table>
+</div>
+{{- end -}}
+{{- if .LinkedFrom -}}
+<p>{{ trans "golem15.seo::lang.issues.linked_from" }}</p>
+<ul>
+{{- range .LinkedFrom -}}<li><code>{{ .URL }}</code></li>{{- end -}}
+</ul>
+{{- if .More -}}<p class="summer-table__sub">{{ .More }}</p>{{- end -}}
+{{- end -}}
+{{- if .Orphan -}}
+<p>{{ trans "golem15.seo::lang.issues.orphan_suggestions" }}</p>
+{{- if .Suggestions -}}
+<div class="summer-table-wrap">
+<table class="summer-table">
+<thead><tr><th>{{ trans "golem15.seo::lang.issues.orphan_suggestions_page" }}</th><th>{{ trans "golem15.seo::lang.issues.orphan_suggestions_phrase" }}</th></tr></thead>
+<tbody>
+{{- range .Suggestions -}}
+<tr><td><code>{{ .URL }}</code></td><td>{{ .Phrase }}</td></tr>
+{{- end -}}
+</tbody>
+</table>
+</div>
+{{- else -}}
+<p class="summer-table__sub">{{ trans "golem15.seo::lang.issues.orphan_none" }}</p>
+{{- end -}}
+{{- end -}}
+<div class="summer-callout" role="status">
+<p class="summer-callout__title">{{ trans "golem15.seo::lang.issues.fix_heading" }}</p>
+<p class="summer-callout__text">{{ .FixText }}</p>
+</div>
+{{- if .PageURL -}}
+<p>{{ trans "golem15.seo::lang.issues.open_page" }}: <code>{{ .PageURL }}</code></p>
+{{- end -}}
+{{- if .RichResults -}}
+<p>{{ trans "golem15.seo::lang.issues.rich_results" }}: <code>{{ .RichURL }}</code></p>
+{{- end -}}
+{{- end -}}

controllers/issues/config_filter.yaml

@@ -0,0 +1,28 @@
+# Site audit filters (UI-SPEC C23): severity, kind, source, fix and status
+# (Open or Ignored).
+scopes:
+    severity:
+        label: golem15.seo::lang.issues.filter_severity
+        modelClass: Golem15\SEO\Models\Issue
+        nameFrom: severity
+        scope: FilterIssueSeverity
+    kind:
+        label: golem15.seo::lang.issues.filter_kind
+        modelClass: Golem15\SEO\Models\Issue
+        nameFrom: kind
+        scope: FilterIssueKind
+    source:
+        label: golem15.seo::lang.issues.filter_source
+        modelClass: Golem15\SEO\Models\Issue
+        nameFrom: source
+        scope: FilterIssueSource
+    fix:
+        label: golem15.seo::lang.issues.filter_fix
+        modelClass: Golem15\SEO\Models\Issue
+        nameFrom: fix
+        scope: FilterIssueFix
+    status:
+        label: golem15.seo::lang.issues.filter_status
+        modelClass: Golem15\SEO\Models\Issue
+        nameFrom: url
+        scope: FilterIssueStatus

controllers/issues/config_form.yaml

@@ -0,0 +1,14 @@
+# Issue preview (UI-SPEC C23): read-only, P7 above the fields and the
+# record actions Check this address again, Ignore and Stop ignoring.
+name: golem15.seo::lang.issues.name
+form: ~/plugins/golem15/seo/models/issue/fields.yaml
+modelClass: Golem15\SEO\Models\Issue
+defaultRedirect: golem15/seo/issues
+
+preview:
+    headerPartial: issue_details
+
+recordActions: [recheck, ignore, unignore]
+
+messages:
+    preview: golem15.seo::lang.issues.preview

controllers/issues/config_list.yaml

@@ -0,0 +1,25 @@
+# Site audit (UI-SPEC C23): the current issues, errors first (the
+# controller's ListExtendQuery), then by kind; ignored issues badged.
+list: ~/plugins/golem15/seo/models/issue/columns.yaml
+modelClass: Golem15\SEO\Models\Issue
+title: golem15.seo::lang.issues.list_title
+recordUrl: golem15/seo/issues/preview/:id
+noRecordsMessage: golem15.seo::lang.issues.list_empty
+recordsPerPage: 50
+showCheckboxes: true
+showSetup: true
+filter: config_filter.yaml
+bulkActions: [ignore]
+
+defaultSort:
+    column: kind
+    direction: asc
+
+toolbar:
+    buttons: [start_crawl]
+    search:
+        prompt: backend::lang.list.search_prompt
+
+messages:
+    empty: golem15.seo::lang.issues.list_empty
+    rowStateDisabled: golem15.seo::lang.issues.state_ignored

controllers/pages.go

@@ -0,0 +1,287 @@
+package controllers
+
+import (
+	"context"
+	"time"
+
+	"git.golem15.com/golem15/sm-seo-plugin/classes"
+	"git.golem15.com/golem15/sm-seo-plugin/models"
+	"git.golem15.com/golem15/summercms/modules/backpack"
+	"git.golem15.com/golem15/summercms/modules/lagoon"
+	"git.golem15.com/golem15/summercms/modules/pact"
+	"gorm.io/gorm"
+)
+
+// PagesControllerID is the Pages screen, the page registry (UI-SPEC C21).
+const PagesControllerID = classes.PagesController
+
+// pagesAdminController lists and edits the page registry (D-33). The
+// address and the name per locale, their refusals and history come from
+// the plugin's own form extension on this controller, which runs inside
+// the save transaction once the row is written; the SEO tab is added
+// because the registry is the content type seo-page.
+type pagesAdminController struct {
+	app func() *backpack.App
+}
+
+func (pagesAdminController) ID() string        { return PagesControllerID }
+func (pagesAdminController) ModelName() string { return `Golem15\SEO\Models\Page` }
+func (pagesAdminController) ConfigDir() string { return "controllers/pages" }
+func (pagesAdminController) RequiredPermissions() []string {
+	return []string{PermissionManage}
+}
+
+// NewRecord is a page with the form's defaults: kind page, entered by
+// hand, its content changed today.
+func (pagesAdminController) NewRecord() any {
+	p := models.NewPage()
+	return &p
+}
+
+// FormBeforeCreate dates a page without a content date today and refuses
+// an unknown kind or a parent that is unknown or makes a loop.
+func (c pagesAdminController) FormBeforeCreate(ctx context.Context, model any) error {
+	p, ok := model.(*models.Page)
+	if !ok || p == nil {
+		return nil
+	}
+	if p.ContentUpdatedAt == nil || p.ContentUpdatedAt.IsZero() {
+		today := lagoon.DateOf(time.Now().UTC())
+		p.ContentUpdatedAt = &today
+	}
+	if p.Origin == "" {
+		p.Origin = models.PageOriginManual
+	}
+	return c.check(ctx, p)
+}
+
+// FormBeforeUpdate checks an update's kind and parent.
+func (c pagesAdminController) FormBeforeUpdate(ctx context.Context, model any) error {
+	p, ok := model.(*models.Page)
+	if !ok || p == nil {
+		return nil
+	}
+	return c.check(ctx, p)
+}
+
+// check refuses the owner fields' problems (kind, parent_id); the address
+// is checked by the form extension, which receives it.
+func (c pagesAdminController) check(ctx context.Context, p *models.Page) error {
+	if p.ParentID != nil && *p.ParentID == 0 {
+		p.ParentID = nil
+	}
+	svc, err := classes.NewService(appOf(c.app))
+	if err != nil {
+		return err
+	}
+	db, err := database(ctx, c.app)
+	if err != nil {
+		return err
+	}
+	problems, err := svc.CheckPage(ctx, db, *p)
+	if err != nil {
+		return err
+	}
+	owner := map[string][]models.Problem{}
+	for _, field := range []string{"kind", "parent_id"} {
+		if list := problems[field]; len(list) > 0 {
+			owner[field] = list
+		}
+	}
+	return problemsError(ctx, translator(c.app), owner)
+}
+
+// FormAfterDelete records the deleted page in history inside the delete's
+// transaction; the record event then deletes its SEO entries.
+func (c pagesAdminController) FormAfterDelete(ctx context.Context, model any) error {
+	p, ok := model.(*models.Page)
+	if !ok || p == nil {
+		return nil
+	}
+	svc, err := classes.NewService(appOf(c.app))
+	if err != nil {
+		return err
+	}
+	db, err := database(ctx, c.app)
+	if err != nil {
+		return err
+	}
+	return svc.RecordPage(ctx, db, classes.PageValues(*p), *p, true)
+}
+
+// reportedAt is the settings' newest route report time, nil when none.
+func reportedAt(db *gorm.DB) (*time.Time, error) {
+	var rows []models.Settings
+	if err := db.Select("id", "routes_reported_at").Where("id = ?", 1).Limit(1).Find(&rows).Error; err != nil {
+		return nil, err
+	}
+	if len(rows) == 0 {
+		return nil, nil
+	}
+	return rows[0].RoutesReportedAt, nil
+}
+
+// ListRowStates marks a reported page that the newest route report did not
+// name (UI-SPEC C21): negative, labelled pages.state_missing. The state is
+// read from the table, since ListExtendRecords shows the origin as its
+// label.
+func (pagesAdminController) ListRowStates(ctx context.Context, db *gorm.DB, records []any) ([][]pact.RowState, error) {
+	var ids []uint
+	for _, record := range records {
+		if p, ok := record.(*models.Page); ok && p != nil {
+			ids = append(ids, p.ID)
+		}
+	}
+	missing := map[uint]bool{}
+	if len(ids) > 0 {
+		var rows []uint
+		if err := db.WithContext(ctx).Model(&models.Page{}).Where("id IN ?", ids).Where(models.MissingSQL).Pluck("id", &rows).Error; err != nil {
+			return nil, err
+		}
+		for _, id := range rows {
+			missing[id] = true
+		}
+	}
+	out := make([][]pact.RowState, len(records))
+	for i, record := range records {
+		if p, ok := record.(*models.Page); ok && p != nil && missing[p.ID] {
+			out[i] = []pact.RowState{pact.RowStateNegative}
+		}
+	}
+	return out, nil
+}
+
+// pageKindLabels and pageOriginLabels are the phrase keys of the kinds and
+// origins.
+var (
+	pageKindLabels   = classes.PageKindLabels
+	pageOriginLabels = map[string]string{
+		models.PageOriginManual:   "golem15.seo::lang.pages.origin_manual",
+		models.PageOriginReported: "golem15.seo::lang.pages.origin_reported",
+		models.PageOriginMCP:      "golem15.seo::lang.pages.origin_mcp",
+	}
+)
+
+// FilterOptions serves the choices of the kind, origin and missing
+// filters.
+func (pagesAdminController) FilterOptions(scope string) []pact.Option {
+	switch scope {
+	case models.FilterPageKind:
+		out := make([]pact.Option, 0, len(models.PageKinds))
+		for _, k := range models.PageKinds {
+			out = append(out, pact.Option{Value: k, Label: pageKindLabels[k]})
+		}
+		return out
+	case models.FilterPageOrigin:
+		out := make([]pact.Option, 0, len(models.PageOrigins))
+		for _, o := range models.PageOrigins {
+			out = append(out, pact.Option{Value: o, Label: pageOriginLabels[o]})
+		}
+		return out
+	case models.FilterPageMissing:
+		return []pact.Option{
+			{Value: models.MissingOnly, Label: "golem15.seo::lang.pages.filter_missing_only"},
+			{Value: models.MissingHide, Label: "golem15.seo::lang.pages.filter_missing_hide"},
+		}
+	}
+	return nil
+}
+
+// ListExtendRecords shows the kind and the origin of the list's rows as
+// their labels in the request locale. The rows are list values only and
+// never saved; the columns still sort by the stored values.
+func (c pagesAdminController) ListExtendRecords(ctx context.Context, _ *gorm.DB, records []any) error {
+	tr := translator(c.app)
+	for _, record := range records {
+		p, ok := record.(*models.Page)
+		if !ok || p == nil {
+			continue
+		}
+		if key, ok := pageKindLabels[p.Kind]; ok {
+			p.Kind = Phrase(ctx, tr, key, nil)
+		}
+		if key, ok := pageOriginLabels[p.Origin]; ok {
+			p.Origin = Phrase(ctx, tr, key, nil)
+		}
+	}
+	return nil
+}
+
+// DropdownOptions serves the parent dropdown: the registry pages by name
+// (their address when they have none).
+func (c pagesAdminController) DropdownOptions(field string) []pact.Option {
+	if field != "parent_id" {
+		return nil
+	}
+	db, err := database(context.Background(), c.app)
+	if err != nil {
+		return nil
+	}
+	var rows []models.Page
+	if err := db.Where("url <> ''").Order("title ASC, url ASC").Limit(1000).Find(&rows).Error; err != nil {
+		log.Warn("golem15.seo: parent options", "error", err)
+		return nil
+	}
+	out := make([]pact.Option, 0, len(rows))
+	for _, p := range rows {
+		label := p.Title
+		if label == "" {
+			label = p.URL
+		} else {
+			label += " (" + p.URL + ")"
+		}
+		out = append(out, pact.Option{Value: p.IDText(), Label: label})
+	}
+	return out
+}
+
+// pageStatusView is the view model of P9 page_status: the reported or
+// missing line, nothing for a page entered by hand.
+type pageStatusView struct {
+	Text    string
+	Warning bool
+}
+
+// PartialData serves P9 page_status (UI-SPEC C21).
+func (c pagesAdminController) PartialData(ctx context.Context, name string, record any) (any, error) {
+	if name != "page_status" {
+		return nil, nil
+	}
+	p, ok := record.(*models.Page)
+	if !ok || p == nil || p.Origin != models.PageOriginReported {
+		return pageStatusView{}, nil
+	}
+	db, err := database(ctx, c.app)
+	if err != nil {
+		return nil, err
+	}
+	reported, err := reportedAt(db)
+	if err != nil {
+		return nil, err
+	}
+	tr := translator(c.app)
+	stamp := func(t *time.Time) string {
+		if t == nil {
+			return "—"
+		}
+		return t.UTC().Format("2006-01-02 15:04")
+	}
+	if classes.PageMissing(*p, reported) {
+		return pageStatusView{Text: Phrase(ctx, tr, "golem15.seo::lang.pages.status_missing", map[string]string{"time": stamp(reported)}), Warning: true}, nil
+	}
+	return pageStatusView{Text: Phrase(ctx, tr, "golem15.seo::lang.pages.status_reported", map[string]string{"time": stamp(p.ReportedAt)})}, nil
+}
+
+var (
+	_ pact.AdminController         = pagesAdminController{}
+	_ pact.AdminPermissioned       = pagesAdminController{}
+	_ pact.AdminRecordSource       = pagesAdminController{}
+	_ pact.FormBeforeCreate        = pagesAdminController{}
+	_ pact.FormBeforeUpdate        = pagesAdminController{}
+	_ pact.FormAfterDelete         = pagesAdminController{}
+	_ pact.ListRowStates           = pagesAdminController{}
+	_ pact.FilterOptions           = pagesAdminController{}
+	_ pact.ListExtendRecords       = pagesAdminController{}
+	_ pact.DropdownOptionsProvider = pagesAdminController{}
+	_ pact.AdminPartialData        = pagesAdminController{}
+)

controllers/pages/_page_status.htm

@@ -0,0 +1,7 @@
+{{- with .Data -}}
+{{- if .Text -}}
+<div class="summer-callout{{ if .Warning }} summer-callout--warning{{ end }}" role="status">
+<p class="summer-callout__text">{{ .Text }}</p>
+</div>
+{{- end -}}
+{{- end -}}

controllers/pages/config_filter.yaml

@@ -0,0 +1,18 @@
+# Pages filters (UI-SPEC C21): kind, origin and missing from the newest
+# route report.
+scopes:
+    kind:
+        label: golem15.seo::lang.pages.kind
+        modelClass: Golem15\SEO\Models\Page
+        nameFrom: kind
+        scope: FilterPageKind
+    origin:
+        label: golem15.seo::lang.pages.origin
+        modelClass: Golem15\SEO\Models\Page
+        nameFrom: origin
+        scope: FilterPageOrigin
+    missing:
+        label: golem15.seo::lang.pages.filter_missing
+        modelClass: Golem15\SEO\Models\Page
+        nameFrom: url
+        scope: FilterPageMissing

controllers/pages/config_form.yaml

@@ -0,0 +1,20 @@
+# Page form (UI-SPEC C21). The address and the name per locale come from
+# the plugin's own form extension on this controller; the SEO tab is added
+# because the registry is a content type (seo-page).
+name: golem15.seo::lang.pages.name
+form: ~/plugins/golem15/seo/models/page/fields.yaml
+modelClass: Golem15\SEO\Models\Page
+defaultRedirect: golem15/seo/pages
+
+create:
+    redirect: golem15/seo/pages/update/:id
+    redirectClose: golem15/seo/pages
+
+update:
+    redirect: golem15/seo/pages
+    redirectClose: golem15/seo/pages
+
+messages:
+    create: golem15.seo::lang.pages.new
+    update: golem15.seo::lang.pages.name
+    deleteConfirm: golem15.seo::lang.pages.delete_confirm

controllers/pages/config_list.yaml

@@ -0,0 +1,26 @@
+# Pages list (UI-SPEC C21): the page registry by address; reported pages
+# missing from the newest route report are marked.
+list: ~/plugins/golem15/seo/models/page/columns.yaml
+modelClass: Golem15\SEO\Models\Page
+title: golem15.seo::lang.pages.list_title
+recordUrl: golem15/seo/pages/update/:id
+noRecordsMessage: golem15.seo::lang.pages.list_empty
+recordsPerPage: 20
+showCheckboxes: true
+showSetup: true
+filter: config_filter.yaml
+
+defaultSort:
+    column: url
+    direction: asc
+
+toolbar:
+    buttons: [create, delete]
+    search:
+        prompt: backend::lang.list.search_prompt
+
+messages:
+    create: golem15.seo::lang.pages.new
+    empty: golem15.seo::lang.pages.list_empty
+    deleteConfirm: golem15.seo::lang.pages.delete_selected_confirm
+    rowStateNegative: golem15.seo::lang.pages.state_missing