sm-seo-plugin/README.md

# golem15.seo

The SEO plugin for SummerCMS applications: an SEO tab on every content type, per-URL rules, redirects and the 404 log, robots.txt with AI bot switches, global and per-type SEO settings, change history, a site audit with a built-in crawler, and the redirect decision and effective SEO of any URL for the frontend.

```go
import _ "git.golem15.com/golem15/sm-seo-plugin"
```

Module `git.golem15.com/golem15/sm-seo-plugin`, package `seo`, plugin ID `golem15.seo`.

## Overview

The plugin owns all SEO data of a site; the framework only carries the generic seams it builds on: form extensions (`pact.AdminFormExtension`), record events (`pact.RecordSaved`, `pact.RecordDeleted`) and content descriptors (`pact.ContentDescriptor` in signpost). Content plugins never import it. A content type takes part by registering a content descriptor on its menu item type; the plugin then adds an **SEO** tab to that type's admin form and stores the values in its own table, keyed by the record's morph type, its id and the locale.

The frontend (a Nuxt layer or app) asks `GET /_seo/api/v1/resolve?url=...` for the page it renders. The answer is first the redirect decision (URL normalisation, redirects, strict URLs, reached in one hop), then the effective SEO. Every value is resolved with one precedence: **URL rule > record > per-type template or global settings > values derived from the content**, and the resolver records the source of each value. The frontend enforces the decision; the plugin registers no routes at the site root except `/robots.txt` and `/llms.txt`.

`Requires()` names `golem15.sitemap` (`SitemapPluginID`): the plugin publishes the sitemap plugin's `contract.Filter` (`classes.SitemapFilter`) in `Register` and reads the sitemaps only through that plugin's `contract` package (`contract.Listing`). Content types are still found through signpost, never by plugin ID; the sitemap plugin keeps working without this one.

## Features

- **SEO tab** on every content type whose descriptor names an admin controller (`Content.Controller`), placed after the owner's last tab: result preview with duplicate and keyphrase checks, SEO title and meta description with pixel counters, focus keyphrase, canonical URL, robots directives (index/follow plus the advanced directives), social title, description and image, and the effective values with their sources. Values live in `golem15_seo_entries`, one row per record and locale; the owner's tables are untouched.
- **URL rules** (`golem15_seo_rules`): SEO for an exact address or a pattern (`/blog?tag=*`), with or without a record behind it. A rule wins over the record's tab.
- **Settings** (`golem15_seo_settings`, one row): site name, title and home templates with `%title%`, `%description%` and `%site%`, default description and social image, X account, the query-parameter list (content, filter and tracking parameters, optionally per path), never-indexed and search paths, address options and verification codes and files. One tab per content type (`golem15_seo_type_settings`) holds the inject switch, the title and description templates and the default robots (tag types default to `noindex,follow`).
- **Resolver.** Canonical URLs are self-referencing and absolute from the configured site URL, keep content parameters and drop filter and tracking parameters; overrides may be relative or absolute. Robots resolve per directive and are also answered as the `X-Robots-Tag` value. Open Graph and Twitter values inherit from the effective title, description and image. Verification meta tags come with the home page.
- **Redirects** (`golem15_seo_redirects`): 301, 302 and 410 rules with exact, pattern (`/old/*` → `/new/*`) and regex (`^/p/(\d+)$` → `/post/$1`) sources, evaluated exact, then patterns (longest literal prefix first), then regex. Options: ignore letter case, match with and without a trailing slash, ignore or pass the query string; a source with `?` matches its query exactly and wins over the same path without a query. Regex sources use the standard library (RE2, linear time) and are capped at 512 characters. A capture never turns a path target into another host, and an absolute target keeps its own host.
- **Loops and chains.** Every redirect write runs a simulation with the rule in place (a probe URL for patterns and regex, at most 10 hops): a loop is refused, from the admin with `redirects.loop` on the target and nothing saved; a chain is saved with `redirects.chain_notice` naming the final target and is marked in the list. Redirect writes take a transaction-level advisory lock before the simulation, so two concurrent writes cannot together store a loop.
- **Address changes.** When a published record's address changes, a 301 from the old to the new address is created, earlier redirects to the old address are pointed straight at the new one, and an exact redirect that matches the new address (with its case and trailing-slash options) is removed (a slug changed back and forth ends with one redirect per old address). The change is simulated first: when it would loop through a pattern or regex rule, or an address is not a path on the site, no redirect is written and a warning is logged. In the admin this happens inside the save transaction and the editor gets `redirects.auto_created` on the slug field; writes outside the admin (Journal's API, MCP, imports) are handled after commit through `pact.RecordSaved`. Deleting or unpublishing a record creates nothing.
- **404 log** (`golem15_seo_notfound`): the frontend reports each 404 to `POST /_seo/api/v1/notfound`; the log keeps the address, hits, first seen, last hit and last referer. An entry is resolved with one click: **Create redirect…** (a prompted 301), **Mark as gone** (410), or a bulk 410.
- **URL normalisation** (settings tab Addresses): https, `www` or bare host, trailing slash added or removed, optional lowercasing of ASCII letters, double slashes and `/index.html` collapsed. The defaults change nothing but scheme, slashes and index files.
- **Strict URLs.** A path that names a record without being its address (a wrong category, a locale prefix with another locale's slug) 301s to the record's address; a wrong URL never answers `ok`.
- **robots.txt** (`golem15_seo_robots`): an editor with live lint and a size limit of 500 KB, a tester on Google's matcher (`github.com/jimsmart/grobotstxt`: longest match, `Allow` wins a tie, `*` and `$`, the deciding line), and `GET /robots.txt` with the stored rules, the AI bot groups and the `Sitemap:` lines. Saving rules that block a noindex address warns, because Googlebot cannot see a noindex it may not fetch.
- **AI bots.** Switches for OAI-SearchBot, Claude-SearchBot, PerplexityBot (search), ChatGPT-User, Claude-User (user fetch), GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent (training), all allowed by default. A bot switched off gets `User-agent: <token>` / `Disallow: /`.
- **Verification files** stored in the settings are served by `GET /_seo/api/v1/verification/{name}` for the frontend to map at the site root.
- **Non-production lock.** Outside the `production` environment every resolve answer is `noindex, nofollow` with `staging: true`, and `/robots.txt` is `User-agent: *` / `Disallow: /` with `X-Robots-Tag: noindex, nofollow`, whatever is stored. It depends on the environment only; no setting turns it off.
- **History** (`golem15_seo_revisions`): every SEO write (the SEO tab, rules, redirects, robots.txt, settings, the 404 log) is recorded with author, time, via (admin, MCP, import, auto for address changes, undo) and the changed fields before and after. Deleted records keep their values in history.
- **Undo** of one change or of a whole batch, from the History preview (record actions **Undo this change** and **Undo the whole batch**) and through MCP. An undo is refused when a later change touched the same values (`history.undo_stale`, naming who and when), when it would create a redirect loop (`history.undo_loop`) or when the record is gone (`history.undo_gone`); an undo is itself recorded.
- **Batches** (D-12): a list of changes (SEO by URL, redirects, URL rules) applied all or nothing under one batch id. A dry run answers the diff and the conflicts (loops, chains, noindex on an address a sitemap file lists now, a canonical pointing to a redirected or noindex address) and writes nothing.
- **CSV import and export** of redirects, URL rules and the SEO of records (by URL), in the **Import and export** screen (element `golem15-seo-transfer`) and through MCP. Files are UTF-8 with a BOM, comma or semicolon; exported cells that start a formula (`=`, `+`, `-`, `@`, tab, CR), or apostrophes followed by one, are prefixed with an apostrophe, and the import drops exactly that one again, so a re-imported export is unchanged. An import is previewed row by row (new, changed, unchanged, refused, with notes) and applied as one batch with via `import`, or not at all while a row is refused. The screen's apply token is an HMAC-SHA256 under the app key over the file and a fingerprint of the data the preview read; it is refused with `transfer.stale` when the data changed. The screen previews files up to 60 KB and downloads exports up to 250 KB; MCP imports take up to 900 KB per call and exports come in pages of 1000 rows.
- **Record events.** A deleted record loses its entries after commit; a created record gets its address snapshot; an updated record whose address changed gets its 301.
- **Entry generation.** The `golem15.seo.generate_entries` job (conga) creates missing entries with the current address of every published record and refreshes the effective-value cache; a record whose address changed without a record event gets its address change (with its 301) like an admin save. Editor values are never changed.
- **Field types** `seotext`, `mlseotext`, `seorobots`, `mlseorobots`, `seoimage`, `robotstxt`, `seoschema` and `seoparent`, with the custom elements `golem15-seo-text`, `golem15-seo-robots`, `golem15-seo-serp` (the result preview widget), `golem15-seo-image`, `golem15-seo-robotstxt` (the robots.txt editor), `golem15-seo-robotstester` (the tester widget), `golem15-seo-transfer` (the import and export widget), `golem15-seo-schema` (structured data), `golem15-seo-graph` (the structured data preview widget), `golem15-seo-parent` (the primary category) and `golem15-seo-links` (the internal links widget).
- **Structured data.** Every `ok` resolve answer carries one JSON-LD `@graph` built from the content descriptor and the settings: the page node (WebPage or its subtype), the article with its authors, dates, images, word count and section, the person of a profile page, a Service with its provider and offer, the BreadcrumbList, one VideoObject per YouTube video, the WebSite (with a SearchAction when `search_url` is set) and the publisher. Editors pick the schema type per page, fill the Service fields and add custom JSON-LD; the SEO tab previews the graph with Google's required and recommended properties and links the Rich Results Test.
- **Publisher and local business** settings: an organisation or a person with logo, website, contact and sameAs profiles, optionally a local business with address, coordinates, opening hours and price range.
- **Breadcrumbs, hreflang and pagination.** The resolve answer carries the breadcrumb trail (through the record's primary category), the hreflang alternates with `x-default`, `html_lang` and `og:locale`, and the page of a paginated address in the site's form (`?page=N` or `/<segment>/N`).
- **One permission**, `golem15.seo.manage`, for the SEO tab, rules, redirects, the 404 log, pages, auto-links, robots.txt, settings, history and the site audit, in the admin and through MCP.
- **Management API and MCP.** Every SEO admin operation has a JSON route under `/_seo/api/v1` and an MCP tool bound to it (pier, prefix `seo_`). Tools address pages by URL: a change goes to the record behind the URL, else to an exact URL rule, and the answer says where. A read grant sees only the read tools. `AdminOperations` maps each admin operation to its tool, and a test fails when one has none.
- **Site audit** (SEO › **Site audit** and **Crawls**): a built-in crawler fetches the site's own pages like a search engine and reports per URL what it finds (statuses, sitemaps, canonicals, titles and descriptions, headings, structured data, hreflang, links, images, content that needs JavaScript, orphan pages), stored with the CMS-data checks: missing and duplicate titles and descriptions, URL rules overriding a record, redirect chains, canonicals pointing to a noindex or redirected address, noindex addresses robots.txt blocks, sitemap addresses the production robots.txt keeps Googlebot from (`sitemap_blocked`, severity `error`, with the deciding line), record images without alt text and H1 headings in content. Every issue names the MCP tool that fixes it, or `fix: code` with the template element to change. See [Site audit and crawler](#site-audit-and-crawler).
- **Sitemap filter.** The sitemaps leave out every address whose effective SEO is noindex, whose canonical URL points elsewhere or that redirects (normalisation or a redirect rule), with the reason `noindex`, `canonical` or `redirect`. The filter decides on the production values, so staging and production list the same set. The SEO tab's status shows **Not in the sitemaps** with the reason, also when the content type is switched off in the sitemap settings.
- **Page registry** (`golem15_seo_pages`, SEO › **Pages**): the site's own frontend pages (home, contact, offer pages) with an address and a name per locale, a kind (page, offer, service, landing), a parent, the date their content changed and a note. The registry is the content type `seo-page`, so each page gets the SEO tab, resolves `ok` and is listed in the sitemaps, grouped by kind into `sitemap-pages`, `sitemap-offers`, `sitemap-services` and `sitemap-landing-pages` files. An address another page or another content type owns is refused (`pages.url_taken`, `pages.url_owned`).
- **Route report.** The frontend server reports its static routes; unknown ones become registry pages, a reported page missing from a newer report is marked **Not on the site**, and nothing is ever deleted or overwritten.
- **Site answer** (`GET /_seo/api/v1/site`) with the facts the frontend layer needs, and **llms.txt** (off by default) listing the indexable pages from the sitemaps.
- **Links** settings: links to other sites open in a new tab (on by default), and domains whose links always get `nofollow` or `sponsored`; the resolve answer carries them as `outbound`.
- **Auto-links** (`golem15_seo_autolinks`, SEO › **Auto-links**): phrases that texts link to a page, per locale or for all, with an active switch, an origin and a note. Every `ok` resolve answer carries the phrases of its locale (never one linking to the page itself) as `autolinks`, with `autolink_max` from the Links tab (3 by default, 0 turns auto-linking off); the frontend layer links each phrase's first occurrence. **Add focus keyphrases** turns the records' focus keyphrases into phrases. See [Auto-links](#auto-links).
- **Internal links** in the SEO tab (E10): the page's inbound and outbound link counts from the last finished crawl, a warning when no page links to it, and up to 10 pages its text could link to and 10 pages that could link to it (titles and focus keyphrases found in the crawled text), each with **Copy address**; `GET /_seo/api/v1/links` and `seo_get_link_suggestions` answer the same.
- **Site cache purge.** Every SEO write (admin, MCP, batch, import, undo, automatic) asks the frontend to drop its cached pages after the commit, once per transaction, through the framework's site-cache purger (`site_cache.purge_url`, `site_cache.purge_token`).

## Usage

Mount the plugin in an application as a git submodule at `plugins/golem15/seo`, add it to the application's `go.work` and `go.mod`, and list it in `summer.yaml`:

```yaml
plugins:
  - id: golem15.sitemap
    module: git.golem15.com/golem15/sm-sitemap-plugin
  - id: golem15.seo
    module: git.golem15.com/golem15/sm-seo-plugin
```

The SEO plugin requires the sitemap plugin, so both are listed. The plugin's own `go.mod` and `go.work` replace the framework with `../summercms.go` and the sitemap plugin with `../sm-sitemap-plugin` for sibling checkouts.

### Making a content type take part

A content plugin registers a content descriptor on its menu item type. `Controller` names the admin controller whose form gets the SEO tab and `Model` the records' morph type; `Describe` answers each record's URL, title, description and image per locale, and `Match` maps a public path to a record:

```go
func (p *Plugin) MenuItemTypes() []pact.MenuItemType {
	return []pact.MenuItemType{{
		Key:            "acme-page",
		Label:          "acme.pages::lang.page",
		DefaultPattern: "/pages/:slug",
		Resolve:        p.resolvePage,
		Content: &pact.ContentDescriptor{
			Controller: "acme.pages.pages",
			Model:      "acme_pages_pages",
			Describe:   p.describePage,
			Records:    p.pageRecords,
			Match:      p.matchPage,
		},
	}}
}
```

The type then gets the SEO tab, a tab in the SEO settings and a place in the resolver. A descriptor without `Controller` takes part in the settings and the resolver only.

### The resolve endpoint

`GET /_seo/api/v1/resolve?url=<path or absolute URL>` is public (no backend middleware) and throttled per client IP (bucket `seo-resolve`, 600 requests a minute). It answers `200` with:

```json
{
  "action": "ok",
  "status": 200,
  "url": "https://example.com/pages/about",
  "locale": "en",
  "staging": false,
  "seo": {
    "title": "About | Acme",
    "description": "Who we are",
    "canonical": "https://example.com/pages/about",
    "robots": "index, follow",
    "x_robots_tag": "index, follow",
    "og": {"title": "About | Acme", "description": "Who we are", "image": "https://example.com/storage/media/seo/abc-about.png", "url": "https://example.com/pages/about"},
    "twitter": {"card": "summary_large_image", "site": "@acme"},
    "verification": []
  },
  "outbound": {"new_tab": true, "nofollow_domains": ["example.org"], "sponsored_domains": []},
  "autolinks": [{"phrase": "lwia grzywa", "url": "/grzyby/lwia-grzywa"}],
  "autolink_max": 3
}
```

`outbound` (present on `ok` answers) carries the settings' Links tab: open links to other sites in a new tab, and the domains (subdomains included) whose links always get `nofollow` or `sponsored`. The layer treats an absent key as empty.

`autolinks` and `autolink_max` (present on `ok` answers) are the auto-link phrases of the page's locale and of every locale, active ones only, without those linking to the page itself, longest phrase first, at most 1000; `autolink_max` is the most the layer links in one text. With `autolink_max` 0 the answer carries no phrases.

The pipeline runs normalisation, then redirects on the normalised path (followed to the final target), then strict URLs, then the SEO. When the URL is not in its final form the answer is a redirect, reached in one hop:

```json
{"action": "redirect", "status": 301, "location": "https://example.com/pages/b", "url": "https://example.com/pages/a", "locale": "", "staging": false, "seo": null}
```

**Behind the frontend server.** The resolve calls and the 404 reports come from the frontend (SSR) server, not from the visitor, so the throttle buckets see that server's address unless it is listed in `http.trusted_proxies` and forwards the visitor's address in `X-Forwarded-For` (the rightmost hop outside the trusted list counts). Without that, `seo-resolve` and `seo-notfound` are caps for the whole site, 600 requests a minute each (about 10 rendered pages a second), and resolve answers `429` beyond them. Put the frontend server in `http.trusted_proxies` and have it forward `X-Forwarded-For`.

`status` is 302 when a temporary redirect takes part. A 410 rule answers `"action": "gone"`, `"status": 410`. A redirect cycle met at runtime (stored past the save check) stops after 10 hops with `not_found` and is logged at most once an hour per cycle (keyed by its rules, not by the requested URL). A path that no published record owns and no URL rule covers answers `"action": "not_found"`, `"status": 404` and `"seo": null` (the home page `/` is always known). A `url` that is neither a path nor an http(s) URL is a `422`. Absolute URLs are built from `golem15.seo.site_url`, or from the request URL when it names the site's host or its `www` twin (for the scheme and host rules), never from the request's Host header.

### Structured data, breadcrumbs, hreflang and pagination

An `ok` resolve answer also carries (all additive, absent on other actions):

```json
{
  "html_lang": "en",
  "alternates": [
    {"hreflang": "en", "url": "https://example.com/blog/first-post"},
    {"hreflang": "pl", "url": "https://example.com/pl/blog/pierwszy-wpis"},
    {"hreflang": "x-default", "url": "https://example.com/blog/first-post"}
  ],
  "breadcrumbs": [
    {"name": "Home", "url": "https://example.com/", "path": "/"},
    {"name": "Recipes", "url": "https://example.com/blog/category/recipes", "path": "/blog/category/recipes"},
    {"name": "First post", "url": "https://example.com/blog/first-post", "path": "/blog/first-post"}
  ],
  "jsonld": {"@context": "https://schema.org", "@graph": ["…"]},
  "seo": {"og": {"locale": "en_US", "locale_alternates": ["pl_PL"]}}
}
```

- **The graph.** Nodes link by `@id`: `<site>/#website`, `<site>/#organization` or `<site>/#person` (the publisher), `<url>#webpage`, `<url>#breadcrumb`, `<url>#article`, `<url>#profile`, `<url>#service`, `<url>#video-<n>`; a collection page's `mainEntity` is `<url>#itemlist`, the node the frontend adds with the listed items. The type follows the descriptor's `Kind`: article → BlogPosting (Article or NewsArticle by choice), collection → CollectionPage, profile → ProfilePage with a Person, page → WebPage (AboutPage, ContactPage or Service by choice; registry pages of kind offer or service default to Service). No e-commerce type is generated.
- **Custom JSON-LD** (the SEO tab's Structured data field, or `schema` on `seo_set_url_seo`): an object or a list of objects, at most 16 KB and 8 levels deep, without a `<script`; the server decodes it and stores it re-serialised. A node with a generated `@id` is merged into that node with the custom values first (the save answers the `schema.merged` notice); other nodes are added.
- **Breadcrumbs** start with the home crumb (setting **Home page name in breadcrumbs**, default Home / Strona główna), follow the record's primary category (the SEO tab's Primary category, else the first of the descriptor's `Parents`) and each parent's own primary parent up to 6 levels, leaving out unpublished parents, and end with the page. The BreadcrumbList node has the same items; the home page's trail is the home crumb alone.
- **hreflang** lists every locale the record has an address in, the page itself included, plus `x-default` on the default locale's address; a record in one locale has none. `og:locale` comes from `golem15.seo.og_locales`, else `pl_PL`, `en_US`, `de_DE`, else the language with its region.
- **Pagination** (settings tab Pagination): the form is the query (`?page=N`, default) or the path (`/<segment>/N`, the segment per locale, default `strona` for pl and `page` for en). `?page=1` and `/<segment>/1` answer 301 to the unnumbered address, the other form 301s to the configured one, together with the address rules in one hop. A page value that is not a number answers `not_found` with `"pagination": {"invalid": true, …}`, so the frontend answers 404. Page 2 and later answers `ok` with its own canonical, `index, follow`, the title suffix after the page title and the description suffix at the end; `%page%` is filled and `%pages%` is left for the frontend, which knows the page count. Such an answer carries `"pagination": {"form", "param", "segment", "page", "invalid"}`.

`GET /_seo/api/v1/site` also answers `"pagination": {"form", "param", "segments"}` and `"home_crumb": {"name", "url", "names"}` (the name in the default locale and per locale).

### The 404 report endpoint

`POST /_seo/api/v1/notfound` takes `{"url": "/missing", "referer": "https://..."}` from the frontend server with `Authorization: Bearer <golem15.seo.notfound_token>` (compared in constant time) and answers `204`. A missing or wrong token is `401`; with no token configured the endpoint is off and answers `404`. URLs are capped at 2048 bytes (`422` beyond) and referers at 1024. The log holds at most `golem15.seo.notfound_max_rows` addresses; new ones beyond it are dropped with one warning an hour. Throttle bucket `seo-notfound`, 600 requests a minute per client IP.

### robots.txt

`GET /robots.txt` (bucket `seo-robots`, 120 requests a minute) serves `text/plain`:

```text
User-agent: *
Disallow: /admin

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
```

The stored rules come first, then one group per switched-off AI bot, then the `Sitemap:` line: the sitemap plugin's index (`contract.ListingSettings.IndexURL`, its site address plus `/sitemap.xml`; the per-type files are never listed, the index names them), or one line per entry of `golem15.seo.robots_sitemaps` when that key is set, absolute on the site URL. Outside production the body is `User-agent: *` / `Disallow: /` with `X-Robots-Tag: noindex, nofollow`. It never answers a 5xx: when the stored file cannot be read it serves the last body it served, else `User-agent: *` / `Disallow:` (allow all), and logs the error.

The **robots.txt** screen (a singleton) has the editor, the tester and **What crawlers get** on its first tab and the AI bot switches on the second. The tester sends the editor's text and the switches as they are on screen, saved or not, and answers allowed or blocked for the chosen crawler with the deciding line, or that a bot switch blocked it.

### Verification files

`GET /_seo/api/v1/verification/{name}` serves a verification file from the settings tab Verification (name `[A-Za-z0-9._-]+\.(html|txt|xml)`) with the content type of its extension, `X-Content-Type-Options: nosniff` and a sandboxing Content-Security-Policy; any other name is `404`. The frontend serves it at the site root; the plugin registers no `/{name}` route, which would capture every one-segment page of a site served at `GET /`. Every `/_seo/api/v1` answer carries `X-Robots-Tag: noindex`.

### The page registry and the route report

**SEO › Pages** (controller `golem15.seo.pages`, after URL rules) lists the registry by address with the kind, origin, content date and the **Not on the site** state; the form takes the address and the name per locale (a locale left empty has no version of the page), the kind, the parent and the content date (the sitemap's lastmod), with the SEO tab next to it. A changed address gets its 301 like any record.

`POST /_seo/api/v1/registry/routes` takes the frontend server's static routes and, optionally, the static paths it serves noindex, with `Authorization: Bearer <token>`: `golem15.seo.route_report_token`, else `golem15.seo.notfound_token` (compared in constant time; prefer a dedicated `route_report_token`):

```json
{"routes": ["/", "/blog", "/kontakt"], "noindex": ["/konto", "/pl/konto", "/szkic"]}
```

It answers `200` with the counts and `noindex`, the number of stored noindex paths:

```json
{"inserted": 1, "updated": 3, "skipped": 0, "reported_at": "2026-11-04T20:00:00Z", "noindex": 3}
```

A missing or wrong token is `401`, no token configured `404`, more than 2000 entries in either list `422` `{"error": "too_many_routes", "max": 2000}` (nothing is stored).

`noindex` lists every static path the frontend's build serves noindex (each starting with `/`, without a `:` parameter or `*`, at most 2048 bytes), the locale copies as their own prefixed paths. When the key is present (also as `[]`) the stored set (`golem15_seo_noindex_paths`) becomes exactly its valid entries, normalised like the routes; an invalid entry or a pattern is skipped and counted in `skipped`. When the key is absent the set stays as it is, so a frontend that sends no `noindex` keeps working. The sitemap filter leaves every address whose path is in the set out of the XML and HTML sitemaps with the reason `noindex` (a record's address, a registry page or an Additional address of the sitemap settings), and a changed set asks the sitemap for one debounced rebuild through `contract.Rebuilder` (with an older sitemap plugin the next scheduled rebuild picks it up).

For the routes, an unknown path becomes a page with origin `reported`, kind `page` and a name from its last segment; a known page only gets `reported_at`; a path another content type owns, or one that is no path, is skipped. A report never deletes a page or changes a value an editor set; a reported page the newest report did not name shows **Not on the site** and a warning on its form. Throttle bucket `seo-notfound`. A content type whose file group is also `pages` shares the file with the registry's pages.

### Auto-links

**SEO › Auto-links** (controller `golem15.seo.autolinks`, after Pages, icon `wand-sparkles`) lists the phrases (50 per page) with the address they link to, the language (**All languages** or a locale), the origin (**Manual**, **Focus keyphrase**, **MCP**) and the update time; an inactive phrase shows **Off**. Filters: language, origin and **Active only**. The form takes the phrase, the address, the language and the **Active** switch, and a note. A phrase needs 3 characters (`autolinks.phrase_short`); a phrase exists once per language, a phrase of every language once at all (`autolinks.phrase_taken` names the address it already links to); the address is a path starting with one `/` or a full `http(s)` address (`autolinks.url_invalid`: `javascript:`, `data:` and protocol-relative addresses are refused). The toolbar's **Add focus keyphrases** adds one phrase per record whose focus keyphrase in a locale is not yet a phrase there, linking to the record's address in that locale (origin **Focus keyphrase**), and toasts the count; a second run adds nothing new. Every change is recorded in History and purges the site's cached pages.

The SEO tab's **Internal links** (widget `golem15-seo-links`, action `seo_links`, after the structured data preview, on the update form of content types whose records have a body: not for collections, profiles or the page registry) reads the last finished crawl: **Links to this page** (pages linking here) and **Links from this page** (pages it links to on the site), a warning when nothing links here, **Pages you could link to from this text** (pages whose H1 or focus keyphrase the crawled text contains and that it does not link to yet, with the public address and **Copy address**) and **Pages that could link here** (pages whose crawled text contains this page's title, H1 or focus keyphrase and that do not link here yet, with their admin form), 10 each. Before the first crawl it says when the counts appear.

The layer links a phrase at its first occurrence in a text, as whole words and ignoring letter case, never inside headings, links, `code`, `pre`, `figcaption` or buttons, and never on the page it links to, at most `autolink_max` phrases per text (the Links tab's **Auto-links per page**, 0 to 20).

### The site answer and llms.txt

`GET /_seo/api/v1/site` (public, bucket `seo-resolve`, `X-Robots-Tag: noindex`) answers the facts the frontend layer needs, all public:

```json
{
  "site_url": "https://example.com",
  "default_locale": "pl",
  "locales": ["pl", "en"],
  "unprefixed_locale": "pl",
  "links": {"new_tab": true},
  "html_sitemap": {"enabled": true, "paths": {"pl": "/mapa-strony", "en": "/sitemap"}, "page_size": 150},
  "routes": [{"type": "journal-post", "pattern": "/blog/:slug"}],
  "root_files": ["google0123.html", "0123456789abcdef0123456789abcdef.txt"],
  "llms": false
}
```

`routes` lists each content type's layered URL pattern (the registry has none); `root_files` the verification files and the IndexNow key file while IndexNow is on; `html_sitemap` comes from the sitemap plugin's `contract.Listing`.

`GET /llms.txt` (bucket `seo-robots`) is off by default. With **Serve llms.txt** on (robots.txt screen, AI bots tab, or `seo_update_robots` with `llms_enabled`) and in production it answers `text/plain`: `# <site name>`, the introduction (`llms_intro`, else the default meta description) as a quote, then one `## <group>` section per sitemap group with `- [title](url)` lines of the default locale's listed addresses, in the content types' order. Addresses the sitemaps leave out never appear. Off, or outside production, it is `404`.

### Site audit and crawler

A crawl is a conga job (`golem15.seo.crawl`, one attempt) started from **Start crawl** (on Site audit and Crawls), through MCP (`seo_start_crawl`), with `seo:crawl`, or by the nightly schedule. One crawl runs at a time: a start while one is queued or running refuses with `crawls.already_running` (the start locks the settings row, so two starts cannot both pass), and a site without an address (`golem15.seo.site_url`, else `app.url`) with `crawls.no_site_url`.

**What it fetches.** `robots.txt`, the sitemap index at `/sitemap.xml` and its files (or a single urlset), then breadth-first from the home page and the sitemap addresses over same-host links, redirect targets, canonicals and hreflang alternates. Requests go through a `fetchguard` client limited to the site host and its `www` twin: HTTPS only, private and reserved addresses refused at dial time, redirects never followed (each `Location` is a new address checked again), the User-Agent `SummerCMS-SEO-Crawler/1 (+<site>)`. Links to other hosts, credentials in a URL and plain http are never fetched. A staging or local site served over http or on a private address needs `golem15.seo.crawl_trusted: true` in config (or `SUMMER_GOLEM15__SEO__CRAWL_TRUSTED`), which no setting or MCP call can set; trusted mode still fetches only the site host and its twin. robots.txt rules are read for the report, not obeyed.

**Limits.** Two requests at a time with the settings' pause between requests (default 250 ms), a 10 s timeout and a 5 MiB body cap per request, at most 1000 links read per page, and the settings' page limit (default 5000; a crawl that stops there is marked "limit reached"). Cancelling (**Cancel crawl**, `seo_cancel_crawl`) ends a queued crawl at once; a running crawl records the request (`cancel_requested` in the API and MCP answers, **Cancelling** in the list) and keeps running until its job stops between requests, then ends cancelled with what was fetched. Until then no other crawl starts (409 `crawls.already_running`) and the crawl cannot be deleted; a failed crawl keeps the pages and issues found before the failure and records the error. A crawl that reaches the job's time limit (`crawl.JobTimeout`, 6 h) ends failed with "the crawl reached its time limit of 6h" and keeps what it fetched: storing what it found runs on a context the job's cancellation cannot reach, bounded by `crawl.FinishTimeout` (10 min), and the crawl's final status is then written on a 30-second bound of its own, so a crawl whose results take longer to store still ends at once, failed with "storing the crawl results took longer than 10m0s: …". A running crawl's job writes `heartbeat_at` every minute (`crawl.HeartbeatEvery`); a running crawl without a heartbeat for 10 minutes (`crawl.StaleAfter`, its job is gone, for example after a worker restart) is ended by the next start or delete (`crawl.RecoverStale`): failed with "the crawl job stopped without finishing (no heartbeat since …)", or cancelled when its cancel was requested. Every crawled address is keyed by its decoded path with the raw query (`classes.PathKey`, the form of the resolver, the descriptors and the SEO entries; a path that does not decode to valid text, or holds `%` or `?`, keeps its encoded form) and requested percent-encoded (`classes.EscapeKeyPath`), so a non-ASCII address matches its CMS record, its focus keyphrase and its links in the E10 widget, written either way. The crawler writes only its own tables (`golem15_seo_crawls`, `golem15_seo_crawl_pages`, `golem15_seo_crawl_links`, `golem15_seo_issues`); crawls beyond the settings' "Crawls kept" are deleted with their pages, links and issues.

**Checks** (kind, severity): `page_status` (error), `sitemap_error` (error), `sitemap_noindex` (error), `sitemap_blocked` (error), `sitemap_missing` (notice), `broken_link` (error, with the linking pages), `redirected_link` (notice), `canonical_missing` (warning), `canonical_error` (error), `missing_title` and `duplicate_title` (warning), `missing_description` and `duplicate_description` (notice), `h1_missing` and `h1_multiple` (error), `heading_skipped` and `heading_empty` (warning), `h1_title_mismatch` (notice), `jsonld_invalid` (error), `jsonld_multiple` (warning), `jsonld_required` (error) and `jsonld_recommended` (notice) from the same rules as the structured data preview (`schema.Validate`; FAQPage and other types without Google rules find nothing), `hreflang_reciprocal` (warning), `hreflang_error` (error), `generic_anchor` (warning, the settings' generic link texts, whole text, any case), `image_alt` (warning: an `<img>` without `alt`; `alt=""` is decorative), `image_size` and `featured_lazy` (notice), `content_js` (warning) and `orphan_page` (warning: an indexable sitemap page no crawled page links to, with up to 10 pages whose text mentions its title or focus keyphrase). The CMS-data checks run at the end of every crawl (and when the list is opened before any crawl) and are stored with the crawl's issues as source `cms`; a CMS issue the crawl found too is stored once.

**Raw HTML against the CMS.** The server-rendered HTML is checked against the record behind the page (`content_js` with `details.check: cms`): the record's title must appear in an H1, the text must reach half of the record's word count, and the record's primary parent (the breadcrumb link) must be linked. The comparison with the JavaScript-rendered page is `details.check: rendered` (next section); without a Chrome path every crawl records it as skipped (P6 and `seo_get_crawl` say so).

**Rendered comparison.** When `golem15.seo.crawl_chrome_path` names a Chrome or Chromium binary the operator installed on the server (none is installed by default), the crawler renders the first `golem15.seo.crawl_render_max` pages (default 200, in crawl order, so the home page and the top-level pages come first) in headless Chrome through `github.com/chromedp/chromedp`, right after each page's raw fetch, and compares the DOM after JavaScript with the raw HTML: same-host links, an H1, or at least 50 words and 20 % more text that appear only after JavaScript are one `content_js` finding with `details.check: rendered` and `details.what` naming them (up to five links). The browser is held to the crawler's limits: before Chrome starts, the site host and its `www` twin are resolved and every address checked like the crawler's dial guard (`fetchguard.IsPrivateAddr`; a private address refuses unless `golem15.seo.crawl_trusted` is on), and Chrome is pinned to those addresses through `--host-resolver-rules` while every other host resolves to nothing; every request is intercepted and continued only when it is https (http only in trusted mode) to the site host or its twin on the site's own port (the port of `golem15.seo.site_url`, else 443 for https and 80 for http), or a `data:`/`blob:` URL, so page code cannot reach other services on the site host's address, so third-party scripts, fonts, analytics and metadata addresses are never loaded. Each crawl gets a fresh temporary profile, removed when the crawl ends, with no cookies, credentials or extensions and downloads denied; one tab renders at a time, at most 5 s to open the tab (`crawl.TabOpenTimeout`) and 20 s for the page under one deadline, so a hung Chrome cannot stall the crawl, the DOM is capped at 5 MiB, Chrome's own sandbox stays on, and Chrome is closed when the crawl finishes, fails or is cancelled. The path is config only (or `SUMMER_GOLEM15__SEO__CRAWL_CHROME_PATH`), never a setting or MCP value. The crawl records `render_status` `done` with the pages compared, `skipped` without a path, or `failed` with the reason (a relative path or one that is not an executable file, a private address, Chrome not starting, a page that cannot be rendered); a renderer failure never fails the crawl, and P6, `GET /_seo/api/v1/crawls/{id}` and `seo_get_crawl` (`render {status, pages, note, message}`) say which.

**Fix routes.** An issue a CMS value fixes names its tool: `seo_set_url_seo` (titles, descriptions, canonicals, sitemap membership of a record, structured data), `seo_update_robots`, `seo_create_redirect` (a missing page, a page that answers an error), `seo_create_page` (a page missing from the registry), `seo_update_settings` (publisher and site nodes), `sitemap_update_settings` (a listed noindex address without a CMS record, an Additional address of the sitemap: remove it there; `details.extra` is true, and without the sitemap plugin the issue is `fix: code` naming the sitemap's address list), or the content type's own tool from `pact.ContentDescriptor.Tools` when the problem sits in the record's content: H1 findings prefer `Tools[headings]`, image alt text `Tools[image_alt]`, other content findings `Tools[edit]` (the registry pages name `seo_update_page`). A finding of the site's template code, or of content whose type names no fitting tool, is `fix: code` with `details.what` naming the element to change. No issue has neither.

**Ignore and check again.** An issue's fingerprint (kind, address, locale and the `content_js` check) can be ignored (**Ignore**, bulk **Ignore issues**, `seo_ignore_issues`); it stays hidden in later crawls until **Stop ignoring** (`seo_unignore_issues`). **Check this address again** (`seo_recheck_url`) fetches the issue's address (with its canonical and hreflang targets) and replaces that address's issues; findings that depend on other pages (duplicates, orphans, link findings while the page still errors) stay until the next crawl.

**Staging.** On a non-production environment every page carries the staging lock's noindex; the crawl never counts it (no `sitemap_noindex`, the pages stay indexable for the other checks) and P6 says why.

**Schedule.** The schedule entry runs `seo:crawl --scheduled` every night at 02:30 app time; it queues a crawl when the setting **Automatic crawl** is daily, or weekly on a Monday in `app.timezone`, and does nothing while it is off or a crawl runs.

### Management API and MCP tools

The management routes live under `/_seo/api/v1` (bucket `seo-api`, 300 requests a minute per client IP). Each resolves the administrator from a backend JWT, a personal API token (scope `golem15.seo.*`) or pier's in-process dispatch, so an MCP call runs as the backend account linked to the MCP user; each answers `401` without one and `403` without `golem15.seo.manage`. Field refusals are `422 {"error": "validation_failed", "errors": {...}}`; a redirect that would create a loop is `422 {"error": "redirect_loop", "path": [...], "message": "..."}`, so the tool call fails; a missing row is `404`. History records these writes with via `mcp` when they come through MCP, and redirects created through MCP have origin `mcp`.

| Tool (`seo_` + name) | Route | What it does |
|------|-------|--------------|
| `get_url_seo` | `GET /seo?url=` | Effective SEO of a URL with the source of every value, the redirect decision, the owner and the SEO tab's warnings. |
| `set_url_seo` | `PUT /seo` | Values for a URL: the record behind it in the URL's locale, else an exact URL rule; `force_rule` writes a rule. `primary_parent` (`<type>:<id>`) and `schema` (the structured data JSON string) are records only and shared by every locale. Answers `written_to` (`record` or `rule`), `type`, `id`, `locale`, `rule_id`. |
| `get_graph` | `GET /graph?url=&schema=` | The JSON-LD graph of a URL and every node's missing required and recommended properties (`rules` false for a type Google has no rules for); `schema` previews an unsaved value. Answers `{locale, url, graph, nodes}`. |
| `get_link_suggestions` | `GET /links?url=` | A page's internal links from the last finished crawl: `{crawledAt, inbound, outbound, orphan, out [{phrase, title, url}], in [{phrase, title, adminUrl}]}`, each list at most 10; `crawledAt`, `inbound` and `outbound` are `null` before the first crawl. |
| `list_rules`, `show_rule`, `create_rule`, `update_rule`, `delete_rule` | `GET/POST /rules`, `GET/PUT/DELETE /rules/{id}` | URL rules. |
| `list_redirects`, `show_redirect`, `create_redirect`, `update_redirect`, `delete_redirect` | `GET/POST /redirects`, `GET/PUT/DELETE /redirects/{id}` | Redirects; a chain is saved with a warning. |
| `list_not_found`, `redirect_not_found`, `gone_not_found`, `delete_not_found` | `GET /notfound`, `POST /notfound/{id}/redirect`, `POST /notfound/{id}/gone`, `DELETE /notfound/{id}` | The 404 log and its actions; an address a rule answers is `409`. |
| `show_robots`, `update_robots`, `test_robots` | `GET/PUT /robots`, `GET /robots/test?url=&agent=` | robots.txt, the AI bot switches, `llms_enabled` and `llms_intro`, and the tester on the saved file. |
| `show_settings`, `update_settings` | `GET/PUT /settings` | The SEO settings (including `links_new_tab`, `nofollow_domains`, `sponsored_domains`, `autolink_max`, `breadcrumb_home`, the `page_*` pagination keys, the `publisher_*` and `local_*` keys) and, under `types`, the per-type settings. |
| `list_history`, `show_history` | `GET /history`, `GET /history/{id}` | History, filtered by `via`, `kind` and `batch`. |
| `undo_change`, `undo_batch` | `POST /history/{id}/undo`, `POST /history/batches/{batch}/undo` | Undo; refusals are `409` with `undo_stale`, `undo_loop`, `undo_gone` or `already_undone`. |
| `apply_batch` | `POST /batch` | Changes (`set_seo`, `create_redirect`, `update_redirect`, `delete_redirect`, `create_rule`, `update_rule`, `delete_rule`) all or nothing; `dry_run` answers the diff and the conflicts. A refused change is `422` (`redirect_loop` with the change and the path, or `batch_refused`). |
| `export_csv` | `GET /export?dataset=&delimiter=&page=` | One page of 1000 rows as CSV text. |
| `import_csv` | `POST /import` | A CSV file of a dataset; `dry_run` previews it. A file over 900 KB is `413`; `missing_column`, `unreadable` and `import_refused` are `422`. |
| `list_issues` | `GET /issues` | The site audit's current issues, errors first, with the Phase 32 keys (`kind`, `url`, `locale`, `details`, `tool`) and `id`, `severity`, `source`, `fix`, `found_at`, `ignored`, `crawl_id`, and a count per kind. Filters `severity`, `kind`, `source`, `fix`, `url`, `crawl` and `status` (`open` by default, `ignored` or `all`). |
| `recheck_url` | `POST /issues/{id}/recheck` | Fetches an issue's address again and replaces its issues; answers `{url, remaining, message}`. |
| `ignore_issues`, `unignore_issues` | `POST /issues/ignore`, `POST /issues/unignore` | Ignore issues by `ids` (hidden in later crawls) or show them again. |
| `start_crawl` | `POST /crawls` | Starts a crawl (`202` with the crawl, trigger `mcp`); `409 crawls.already_running`, `422 crawls.no_site_url`. |
| `list_crawls`, `get_crawl` | `GET /crawls`, `GET /crawls/{id}` | Crawls, newest first, with `render {status, pages, note, message}` (`skipped`, `done` or `failed`); one crawl adds `staging`, `issues`, `severities` and `kinds`. |
| `cancel_crawl` | `POST /crawls/{id}/cancel` | Cancels a queued or running crawl, keeping what it fetched; another answers `409 crawls.not_running`. |
| `delete_crawls` | `DELETE /crawls?ids=` | Deletes crawls with their pages and issues; a queued or running one refuses the call with `403 crawls.delete_running`. |
| `list_autolinks`, `create_autolink`, `update_autolink`, `delete_autolinks` | `GET/POST /autolinks`, `PUT /autolinks/{id}`, `DELETE /autolinks?ids=` | The auto-link phrases: `phrase`, `url`, `locale` (`""` or `all` for every locale), `enabled`, `note`; the list filters by `search`, `locale`, `origin` and `active`. A phrase created through MCP has origin `mcp`. |
| `autolinks_from_keyphrases` | `POST /autolinks/from-keyphrases` | Adds the records' focus keyphrases as phrases; answers `{added, message}`. |
| `list_pages`, `create_page`, `update_page`, `delete_page` | `GET/POST /pages`, `PUT/DELETE /pages/{id}` | The page registry: `url` and `title` per locale (a plain string is the default locale's), `kind`, `parent_id`, `content_updated_at`, `note`; the list filters by `id`, `kind`, `origin` and `missing`. A page created through MCP has origin `mcp`. |

Lists take `page`, `per_page` (at most 200) and `search`, and answer `{"data": [...], "page", "per_page", "total"}`. GET tools are read-only; the deletes (`delete_crawls` and `delete_autolinks` included) and `gone_not_found` are destructive.

### Admin

- Navigation item **SEO** (icon `search`), landing on **Redirects**, with the side menu **Redirects**, **404 log**, **URL rules**, **Pages**, **Auto-links** (`wand-sparkles`), **Site audit** (`gauge`), **Crawls** (`globe`), **robots.txt**, **History** and **Import and export**.
- **Site audit** lists the current issues, errors first, with severity, problem, address, details, found by and fix; filters severity, problem, found by, fix and status (Open or Ignored; ignored rows carry the **Ignored** badge). The preview shows what was found, how to fix it (the tool to ask an MCP client for, or the template element) and the page's address, with **Check this address again**, **Ignore** and **Stop ignoring**. The toolbar has **Start crawl**, the list the bulk **Ignore issues**.
- **Crawls** lists the runs newest first (started, status, pages, issues, started by, duration). The preview shows one status callout (running, queued, failed, cancelled, limit reached), the staging and rendered-comparison lines, the stats strip and the issues per kind, each count linking to the filtered Site audit; **Cancel crawl** while queued or running. There is no live progress: reload the page.
- The SEO settings' **Site audit** tab (before Verification): **Automatic crawl** (off, daily, weekly), **Pages per crawl** (10–50 000), **Pause between requests** (100–5000 ms), **Crawls kept** (1–50) and **Generic link texts** (seeded with the English and Polish lists).
- The SEO settings have a **Pagination** tab (after Addresses), a **Links** tab (**Auto-links per page**, 0 to 20, the new-tab switch and the nofollow and sponsored domain lists, one domain per line; `settings.domain_invalid` names a bad line), a **Publisher** tab with the generated publisher node and its findings, and a **Local business** tab. A person cannot be a local business (`settings.local_person`).
- The SEO tab adds **Primary category** (article types), **Structured data** and its **preview** after the social image, then **Internal links** (types with a body); its status warns when a stored primary category is no longer assigned, and the effective values show the schema type and the breadcrumb trail.
- Settings tile **SEO settings** (`seo_settings`), a link to the settings singleton controller.
- Every SEO screen and the SEO tab require `golem15.seo.manage`. Without it an administrator sees no SEO tab, and a posted SEO value changes nothing.

The element bundle (`assets/dist/seo.js` and `seo.css`) is built from `assets/src/elements` and committed; the SEO field types load it on every admin page. After changing `assets/src`, run `npm ci`, `npm run build` and commit `assets/dist`; `scripts/check-dist.sh` fails when the committed bundle is stale.

## API reference

| Identifier | Description |
|------------|-------------|
| `Plugin` | The `golem15.seo` plugin; `Plugin.Service` returns the booted `classes.Service`. |
| `TabSEO`, `TabGeneral`, `TabRobotsFile`, `TabRobotsBots` | The SEO tab key, the settings' General tab key and the robots.txt screen's tab keys. |
| `BucketResolve`, `BucketNotFound`, `BucketRobots`, `BucketAPI` | The throttle buckets of resolve (and the verification files), the 404 report, `/robots.txt` and the management API. |
| `AdminOperations`, `AdminOperation`, `ContentController` | Every SEO admin operation (controller, kind) and its MCP tool; `ContentController` (`*`) stands for the form of every content type. |
| `OpList`, `OpShow`, `OpCreate`, `OpUpdate`, `OpDelete`, `OpExtensionSave`, `OpUpload`, `OpRecordAction`, `OpBulkAction`, `OpWidgetAction`, `OpToolbarAction` | The kinds of an admin operation. |
| `RobotsTestAction` | `robots_test`, the tester's widget action. |
| `SitemapPluginID` | `golem15.sitemap`, the plugin `Requires()` names. |
| `classes.SitemapFilter` | The published `contract.Filter`: `Decide` (reasons `noindex` (the effective SEO, or a path the route report names noindex), `canonical`, `redirect` on the production values; registry pages grouped by kind) and `Groups` (the kinds' groups with their titles per locale). |
| `classes.PageMenuItemType`, `classes.PageTypeKey`, `classes.PagesController`, `classes.PageModel`, `classes.PageGroup` | The registry's menu item type `seo-page` and its content descriptor, and the file group of a kind. |
| `classes.PageWrite`, `classes.RouteReport`, `classes.ValidPagePath`, `classes.PageMissing` | A registry write, a route report's outcome, the address check and the missing state. Service: `CheckPage`, `SavePage`, `DeletePage`, `ApplyPageTexts`, `ReportRoutes`, `ReportNoindex`, `RouteReportToken`, `PageAt`; `classes.NoindexPathCount` counts the stored noindex paths (`models.NoindexPath`). |
| `classes.SiteAnswer` | `Service.Site`, the site answer. `Service.LLMSText` builds llms.txt. |
| `classes.Outbound`, `classes.OutboundOf` | The resolve answer's `outbound` from the settings. |
| `schema.Build`, `schema.Input`, `schema.Validate`, `schema.Normalize`, `schema.DecodeCustom`, `schema.Merge` | Package `classes/schema`: the JSON-LD graph of a page, Google's required and recommended properties per type, the structured data value and custom JSON-LD. |
| `classes.Alternate`, `classes.Alternates`, `classes.LangTag` | hreflang alternates and language tags; `Service.OGLocale`. |
| `classes.Pagination`, `classes.PageConfig` | The resolve answer's pagination block and a site's page form (`Service.PageConfigOf`). |
| `classes.LinkReport`, `classes.LinkOut`, `classes.LinkIn`, `classes.RecordLinksAction`, `classes.LinksActionName`, `classes.MaxLinkSuggestions`, `classes.NormalText` | The internal links of a page (`Service.LinkSuggestions`, `Service.LinksOfURL`, `Service.LinksOfRecord`) and the SEO tab's `seo_links` action. |
| `classes.GraphAnswer` | `Service.GraphOfURL` and `Service.GraphOfRecord`, the graph and its findings; `classes.RecordGraphAction` is the SEO tab's preview action. |
| `classes.PrimaryParent`, `classes.ParentOption` | A record's primary category and its choices (`Service.ParentOptions`); `Service.Trail` builds the breadcrumb trail, `Service.PublisherOf` the publisher. |
| `classes.SitemapExclusion` | `Service.RecordSitemapExclusions`, why the sitemaps leave out a record's addresses; `Service.SitemapListed` reads the listed addresses; `Service.SitemapLines` the robots.txt `Sitemap:` addresses. |
| `classes.SiteCacheKind` | `seo`, the kind of the site-cache purge after an SEO write. |
| `classes.IssueSitemapBlocked`, `classes.SeverityError` | The `sitemap_blocked` issue kind and its severity. |
| `controllers.PagesControllerID` | The Pages screen. |
| `controllers.AutolinksControllerID`, `controllers.ActionFromKeyphrases`, `controllers.KeyphrasesMessage` | The Auto-links screen, its toolbar action and the action's toast. |
| `classes.AutolinkLink`, `classes.AutolinkWrite`, `classes.AutolinkValues`, `classes.NormalizeAutolink`, `classes.ValidAutolinkURL`, `classes.LoadAutolink`, `classes.AutolinkOrigin`, `classes.MaxPageAutolinks` | An auto-link in the resolve answer, a write, its values, the address check and the origin of a write. Service: `AutolinkProblems`, `SaveAutolink`, `DeleteAutolinks`, `RecordAutolink`, `ForPage`, `FromKeyphrases`. |
| `controllers.IssuesControllerID`, `controllers.CrawlsControllerID`, `controllers.ActionStartCrawl`, `controllers.CrawlActionCancel` | Site audit and Crawls, their toolbar start and the cancel record action. |
| `crawl.Crawl`, `crawl.Options`, `crawl.Result`, `crawl.Page`, `crawl.Fetcher`, `crawl.NewFetcher`, `crawl.Parse`, `crawl.PageFacts` | Package `classes/crawl`: one crawl's frontier and limits, the guarded fetcher and the facts of an HTML document. |
| `crawl.Check`, `crawl.Env`, `crawl.Finding`, `crawl.Route`, `crawl.Fingerprint`, `crawl.Kinds`, `crawl.Severity`, `crawl.Record` | The checks, a finding's fix route (tool or code) and identity, and the kinds with their severities. |
| `crawl.Start`, `crawl.Run`, `crawl.Cancel`, `crawl.Recheck`, `crawl.Ignore`, `crawl.Unignore`, `crawl.DeleteCrawls`, `crawl.Prune`, `crawl.Current`, `crawl.ScheduledDue` | Starting, running and cancelling a crawl, checking an address again, ignores, deletes and the schedule's decision. |
| `crawl.CrawlArgs`, `crawl.CrawlKind` | The crawl job's arguments and River kind `golem15.seo.crawl`. |
| `crawl.Renderer`, `crawl.WithRenderer`, `crawl.NewRenderer`, `crawl.RenderMax` | The rendered comparison's seam, its code-only test override, the guarded Chrome renderer built from config and the configured page cap. |
| `crawl.WithFinishTimeout` | A code-only test override of `crawl.FinishTimeout`, like `crawl.WithRenderer`. |
| `crawl.ChromePathConfigKey`, `crawl.RenderMaxConfigKey`, `crawl.TrustedConfigKey`, `crawl.RenderTimeout`, `crawl.MaxDOM` | The renderer's and crawler's config keys and the per-page render limits. |
| `classes.IssueImageAlt`, `classes.IssueH1InContent` | The CMS-data kinds from the content descriptors (`Service.ContentIssues`). |
| `classes.Service` | The single entry point for SEO reads and writes: `SaveRecord`, `Record` (history), `GenerateEntries`, `EnqueueGenerate`, `RecordDeleted`, `RecordCreated`, `Resolver`, `SiteURL`, `Staging`. Redirects: `SaveRedirect`, `DeleteRedirect`, `CheckRedirect`, `AddressChanges`, `AddressChanged`. 404 log: `ReportNotFound`, `ResolveNotFound`. robots.txt: `ServedFile`, `TestRobots`, `BlockedNoindex`, `SaveRobots`. `VerificationFileByName`. Management: `FindOwner`, `SetURLSEO`, `SaveRule`, `DeleteRule`, `SaveSettings`, `DeleteNotFound`, `Issues`. |
| `classes.Owner`, `classes.URLSEOWrite`, `classes.URLSEOResult` | The owner of a URL (`classes.WrittenToRecord` or `classes.WrittenToRule`) and a write addressed by URL. |
| `classes.Issue` | One finding of the issue report (`classes.IssueMissingTitle` and the other kinds). |
| `classes.WithTx` | Puts a transaction on the context so every read of the service sees it. |
| `classes.BatchChange`, `classes.BatchOptions`, `classes.BatchResult`, `classes.BatchRefusedError` | A batch: `Service.RunBatch` runs it as a dry run or all or nothing (`classes.WithBatch` marks the history rows). |
| `classes.UndoRefusal`, `classes.Undoable` | `Service.Undo` and `Service.UndoBatch` and their refusals. |
| `classes.ReadCSV`, `classes.WriteCSV`, `classes.Export`, `classes.ImportRequest`, `classes.ImportResult` | CSV files and `Service.Export` and `Service.Import`; `Service.TransferToken` and `Service.ReadTransferToken` sign and verify the screen's apply token. |
| `controllers.TransferControllerID`, `controllers.TransferAction`, `controllers.HistoryActionUndo`, `controllers.HistoryActionUndoBatch` | The import and export screen, its widget action and the History record actions. |
| `classes.Resolver` | `Resolve(ctx, url)` answers a `classes.Result` (actions `ok`, `not_found`, `redirect`, `gone`) with a source per value in `Result.Sources`; `ResolveRecord` resolves a record at its own address; `Strict` checks a path against the record's address. |
| `classes.RedirectSet`, `classes.NewRedirectSet`, `classes.LoadRedirectSet` | A compiled redirect rule set: `Match` and `Follow` (`classes.Chain`, at most `classes.MaxRedirectHops`). |
| `classes.RedirectCheck`, `classes.LoopError`, `classes.ProblemsError` | The save-time simulation's answer and the refusals of `SaveRedirect`. |
| `classes.AddressChange`, `classes.AddressResult` | An address change and what `AddressChanged` did. |
| `classes.Normalize`, `classes.NormalizeOptions` | The address rules as a pure function. |
| `classes.Bots`, `classes.Bot`, `classes.ComposeRobots` | The AI bot list and the composition of the production robots.txt. |
| `classes.ParseRobots`, `classes.Robots` | The robots value grammar: parse, `String` (stored form), `Header` (meta and header form), `Inherit` (per-directive precedence). |
| `classes.RenderTemplate` | Fills a title or description template. |
| `classes.GenerateEntriesArgs` | The arguments of the generation job (`Type` limits it to one content type). |
| `classes.FieldTypes` | The SEO admin field types. |
| `classes.WithVia`, `classes.ViaOf` | The writer history records (`admin`, `mcp`, `import`, `auto`, `undo`). |
| `controllers.PermissionManage` | `golem15.seo.manage`. |
| `models.Entry`, `models.Rule`, `models.Redirect`, `models.NotFound`, `models.Robots`, `models.Settings`, `models.TypeSetting`, `models.Revision`, `models.Page`, `models.Autolink`, `models.Crawl`, `models.CrawlPage`, `models.CrawlLink`, `models.Issue`, `models.IssueIgnore` | The plugin's models. |

## Configuration

| Key | Default | Meaning |
|-----|---------|---------|
| `golem15.seo.site_url` | `""` | The public origin of the site (`https://example.com`); absolute URLs start with it. Empty uses `app.url`. |
| `golem15.seo.unprefixed_locale` | `""` | The locale whose public URLs carry no locale prefix; empty leaves the choice to each content type. |
| `golem15.seo.notfound_token` | `""` | The bearer token of `POST /_seo/api/v1/notfound`; empty turns the endpoint off. |
| `golem15.seo.notfound_max_rows` | `10000` | The most addresses the 404 log holds. |
| `golem15.seo.robots_sitemaps` | unset | The sitemap addresses robots.txt names, one `Sitemap:` line each, absolute on the site URL. Unset, robots.txt names the sitemap plugin's index. |
| `golem15.seo.og_locales` | unset | Locale code to `og:locale` value, for example `{pl: pl_PL, en: en_GB}`. Unlisted codes use `pl_PL`, `en_US`, `de_DE`, else the language with its region. |
| `golem15.seo.search_url` | unset | The site search address with `{search_term_string}` (`/szukaj?q={search_term_string}`); set, the WebSite node carries a SearchAction. |
| `golem15.seo.route_report_token` | `""` | The bearer token of `POST /_seo/api/v1/registry/routes`; empty uses `notfound_token`, and both empty turn the route report off. |
| `golem15.seo.crawl_trusted` | `false` | Lets the crawler fetch plain http and private addresses (a staging or local site), still only on the site host and its `www` twin. Config only. |
| `golem15.seo.crawl_chrome_path` | `""` | The absolute path of an operator-installed Chrome or Chromium binary for the rendered comparison; empty skips it. Config only. |
| `golem15.seo.crawl_render_max` | `200` | The pages rendered per crawl, the first in crawl order. |

The non-production lock reads the application environment (`SUMMER_ENV`); it has no key of its own. The address rules (host, trailing slash, lowercase) are settings, on the Addresses tab.

## CLI commands

| Command | Description |
|---------|-------------|
| `seo:generate-entries [--type <key>]` | Queues the entry generation job for every content type, or for one. A worker runs it. |
| `seo:crawl [--scheduled]` | Queues a crawl of the site. `--scheduled` (what the nightly schedule entry at 02:30 runs) follows the **Automatic crawl** setting: daily, weekly on Mondays in `app.timezone`, nothing while off or while a crawl runs. |

## Dependencies

- SummerCMS v0.6.0 modules: `pact`, `party`, `backpack`, `cabana`, `signpost`, `lagoon`, `conga`, `cooler` (the site-cache purge), `surf`, `wire`, `bouncer`, `phrasebook`, `bonfire`, `compass`, `diary`.
- `git.golem15.com/golem15/sm-sitemap-plugin`, its `contract` package only (the Filter the plugin publishes, and the Listing and Rebuilder it reads); the plugin requires `golem15.sitemap`.
- `gorm.io/gorm`, `github.com/go-gormigrate/gormigrate/v2`, `gocloud.dev/blob` (the media bucket for social images), `github.com/goccy/go-yaml` (the per-type settings tabs and robots form documents).
- `github.com/jimsmart/grobotstxt` v1.0.3, the robots.txt matcher (decision note `.planning/notes/2026-10-27-robots-txt-grobotstxt.md` in the framework repository).
- `golang.org/x/net/html`, the crawler's HTML parser (already required by the framework).
- `github.com/chromedp/chromedp` v0.21.0 (with `github.com/chromedp/cdproto`), the optional rendered comparison's DevTools client; it drives an operator-installed Chrome and needs no cgo or Node (decision note `.planning/notes/2026-10-28-chromedp-rendered-crawl.md` in the framework repository).
- Element toolchain (development only, exact pins): `typescript`, `vite`, `vitest`, `happy-dom`. The bundle has no runtime dependencies.

## Testing

```sh
go vet ./...
go test -short ./...          # without Docker
go test ./...                 # with Docker: the Postgres smoke tests
npm ci && npm run typecheck && npm test && bash scripts/check-dist.sh
```

The element tests (`tests/elements`, vitest on happy-dom) drive the widget elements through a fake of the admin's widget host (`tests/elements/host.ts`), including a host that drops an action.

The smoke tests boot the plugin with fake content plugins (`acme.pages`, and `acme.blog` with a category segment for strict URLs) and exercise the SEO tab, the settings tabs, the resolver precedence, history, generation, the URL rule form, redirects (engine, loops and chains, address changes from the admin and from events), the 404 log, normalisation and strict URLs, robots.txt and verification files through the admin and public APIs, and the management API as an MCP client (`TestMCPSetURLSeoTracer`, `TestManagementAPI`, `TestIssues`), batches, undo and CSV (`TestBatch`, `TestUndo`, `TestCSVTransfer`, and the `mcp` and `import` cases of `TestLoopRefused`). The sitemap, the registry and the site-level endpoints are covered by `TestSitemapFilterTracer`, `TestFilterReasons`, `TestNoindexPublishedConflict`, `TestSitemapBlockedIssue`, `TestRegistry`, `TestRouteReport`, `TestNoindexPathsTracer`, `TestSiteEndpoint`, `TestLLMSTxt`, `TestRobotsSitemapLine`, `TestSettingsLinks` and `TestPurgeAfterWrite`; the fixture boots the sitemap plugin next to the SEO plugin. The site audit is covered by `TestCrawlerTracer`, `TestCrawlerChecks` (a planted site with one page per kind and a clean control page, served to the crawler through `fetchguard.WithTransport`), `TestIssueMerge`, `TestStagingCrawl`, `TestRenderSkipped`, `TestCrawlAdmin`, `TestCrawlSchedule` and `TestCrawlTools`, with the fake content plugin `acme.crawl` naming fixing tools. Auto-links are covered by `TestAutolinksTracer` (the phrases of a page's locale in the resolve answer, never the page's own, `autolink_max` 0) and `TestAutolinks` (the screen, its refusals, Add focus keyphrases, the API and MCP); `TestLinkSuggestions` crawls a planted site and checks E10's counts and suggestions through the API and the widget action. `TestChromeRenderer` renders a page in a real headless Chrome (it runs when `SEO_TEST_CHROME` names a binary or `chromium`, `chromium-browser` or `google-chrome` is on `PATH`, and is skipped otherwise) and `TestRendererGuards` checks the refused paths and private addresses. `TestEveryAdminOperationHasTool`, `TestMCPReadToolsAreGET`, `TestIssueKindLabels` and the static parts of `TestCrawlTools` and `TestCrawlSchedule` run without Docker.