sm-seo-plugin/README.md

golem15.seo

The SEO plugin for SummerCMS applications: an SEO tab on every content type, per-URL rules, redirects and the 404 log, robots.txt with AI bot switches, global and per-type SEO settings, change history, and the redirect decision and effective SEO of any URL for the frontend.

import _ "git.golem15.com/golem15/sm-seo-plugin"

Module git.golem15.com/golem15/sm-seo-plugin, package seo, plugin ID golem15.seo.

Overview

The plugin owns all SEO data of a site; the framework only carries the generic seams it builds on: form extensions (pact.AdminFormExtension), record events (pact.RecordSaved, pact.RecordDeleted) and content descriptors (pact.ContentDescriptor in signpost). Content plugins never import it. A content type takes part by registering a content descriptor on its menu item type; the plugin then adds an SEO tab to that type's admin form and stores the values in its own table, keyed by the record's morph type, its id and the locale.

The frontend (a Nuxt layer or app) asks GET /_seo/api/v1/resolve?url=... for the page it renders. The answer is first the redirect decision (URL normalisation, redirects, strict URLs, reached in one hop), then the effective SEO. Every value is resolved with one precedence: URL rule > record > per-type template or global settings > values derived from the content, and the resolver records the source of each value. The frontend enforces the decision; the plugin registers no routes at the site root except /robots.txt.

Requires() is empty: content types are found through signpost, never by plugin ID.

Features

• SEO tab on every content type whose descriptor names an admin controller (Content.Controller), placed after the owner's last tab: result preview with duplicate and keyphrase checks, SEO title and meta description with pixel counters, focus keyphrase, canonical URL, robots directives (index/follow plus the advanced directives), social title, description and image, and the effective values with their sources. Values live in golem15_seo_entries, one row per record and locale; the owner's tables are untouched.
• URL rules (golem15_seo_rules): SEO for an exact address or a pattern (/blog?tag=*), with or without a record behind it. A rule wins over the record's tab.
• Settings (golem15_seo_settings, one row): site name, title and home templates with %title%, %description% and %site%, default description and social image, X account, the query-parameter list (content, filter and tracking parameters, optionally per path), never-indexed and search paths, address options and verification codes and files. One tab per content type (golem15_seo_type_settings) holds the inject switch, the title and description templates and the default robots (tag types default to noindex,follow).
• Resolver. Canonical URLs are self-referencing and absolute from the configured site URL, keep content parameters and drop filter and tracking parameters; overrides may be relative or absolute. Robots resolve per directive and are also answered as the X-Robots-Tag value. Open Graph and Twitter values inherit from the effective title, description and image. Verification meta tags come with the home page.
• Redirects (golem15_seo_redirects): 301, 302 and 410 rules with exact, pattern (/old/* → /new/*) and regex (^/p/(\d+)$ → /post/$1) sources, evaluated exact, then patterns (longest literal prefix first), then regex. Options: ignore letter case, match with and without a trailing slash, ignore or pass the query string; a source with ? matches its query exactly. Regex sources use the standard library (RE2, linear time) and are capped at 512 characters. A capture never turns a path target into another host, and an absolute target keeps its own host.
• Loops and chains. Every redirect write runs a simulation with the rule in place (a probe URL for patterns and regex, at most 10 hops): a loop is refused, from the admin with redirects.loop on the target and nothing saved; a chain is saved with redirects.chain_notice naming the final target and is marked in the list.
• Address changes. When a published record's address changes, a 301 from the old to the new address is created, earlier redirects to the old address are pointed straight at the new one, and a redirect whose source is the new address is removed (a slug changed back and forth ends with one redirect per old address). In the admin this happens inside the save transaction and the editor gets redirects.auto_created on the slug field; writes outside the admin (Journal's API, MCP, imports) are handled after commit through pact.RecordSaved. Deleting or unpublishing a record creates nothing.
• 404 log (golem15_seo_notfound): the frontend reports each 404 to POST /_seo/api/v1/notfound; the log keeps the address, hits, first seen, last hit and last referer. An entry is resolved with one click: Create redirect… (a prompted 301), Mark as gone (410), or a bulk 410.
• URL normalisation (settings tab Addresses): https, www or bare host, trailing slash added or removed, optional lowercasing of ASCII letters, double slashes and /index.html collapsed. The defaults change nothing but scheme, slashes and index files.
• Strict URLs. A path that names a record without being its address (a wrong category, a locale prefix with another locale's slug) 301s to the record's address; a wrong URL never answers ok.
• robots.txt (golem15_seo_robots): an editor with live lint and a size limit of 500 KB, a tester on Google's matcher (github.com/jimsmart/grobotstxt: longest match, Allow wins a tie, * and $, the deciding line), and GET /robots.txt with the stored rules, the AI bot groups and the Sitemap: lines. Saving rules that block a noindex address warns, because Googlebot cannot see a noindex it may not fetch.
• AI bots. Switches for OAI-SearchBot, Claude-SearchBot, PerplexityBot (search), ChatGPT-User, Claude-User (user fetch), GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent (training), all allowed by default. A bot switched off gets User-agent: <token> / Disallow: /.
• Verification files stored in the settings are served by GET /_seo/api/v1/verification/{name} for the frontend to map at the site root.
• Non-production lock. Outside the production environment every resolve answer is noindex, nofollow with staging: true, and /robots.txt is User-agent: * / Disallow: / with X-Robots-Tag: noindex, nofollow, whatever is stored. It depends on the environment only; no setting turns it off.
• History (golem15_seo_revisions): every SEO write (the SEO tab, rules, redirects, robots.txt, settings) is recorded with author, time, via (admin, MCP, or auto for address changes) and the changed fields before and after. Deleted records keep their values in history.
• Record events. A deleted record loses its entries after commit; a created record gets its address snapshot; an updated record whose address changed gets its 301.
• Entry generation. The golem15.seo.generate_entries job (conga) creates missing entries with the current address of every published record and refreshes the effective-value cache; editor values are never changed.
• Field types seotext, mlseotext, seorobots, mlseorobots, seoimage and robotstxt, with the custom elements golem15-seo-text, golem15-seo-robots, golem15-seo-serp (the result preview widget), golem15-seo-image, golem15-seo-robotstxt (the robots.txt editor) and golem15-seo-robotstester (the tester widget).
• One permission, golem15.seo.manage, for the SEO tab, rules, redirects, the 404 log, robots.txt, settings and history.

Usage

Mount the plugin in an application as a git submodule at plugins/golem15/seo, add it to the application's go.work and go.mod, and list it in summer.yaml:

plugins:
  - id: golem15.seo
    module: git.golem15.com/golem15/sm-seo-plugin

The plugin's own go.mod and go.work replace the framework with ../summercms.go for a sibling checkout.

Making a content type take part

A content plugin registers a content descriptor on its menu item type. Controller names the admin controller whose form gets the SEO tab and Model the records' morph type; Describe answers each record's URL, title, description and image per locale, and Match maps a public path to a record:

func (p *Plugin) MenuItemTypes() []pact.MenuItemType {
	return []pact.MenuItemType{{
		Key:            "acme-page",
		Label:          "acme.pages::lang.page",
		DefaultPattern: "/pages/:slug",
		Resolve:        p.resolvePage,
		Content: &pact.ContentDescriptor{
			Controller: "acme.pages.pages",
			Model:      "acme_pages_pages",
			Describe:   p.describePage,
			Records:    p.pageRecords,
			Match:      p.matchPage,
		},
	}}
}

The type then gets the SEO tab, a tab in the SEO settings and a place in the resolver. A descriptor without Controller takes part in the settings and the resolver only.

The resolve endpoint

GET /_seo/api/v1/resolve?url=<path or absolute URL> is public (no backend middleware) and throttled per client IP (bucket seo-resolve, 600 requests a minute). It answers 200 with:

{
  "action": "ok",
  "status": 200,
  "url": "https://example.com/pages/about",
  "locale": "en",
  "staging": false,
  "seo": {
    "title": "About | Acme",
    "description": "Who we are",
    "canonical": "https://example.com/pages/about",
    "robots": "index, follow",
    "x_robots_tag": "index, follow",
    "og": {"title": "About | Acme", "description": "Who we are", "image": "https://example.com/storage/media/seo/abc-about.png", "url": "https://example.com/pages/about"},
    "twitter": {"card": "summary_large_image", "site": "@acme"},
    "verification": []
  }
}

The pipeline runs normalisation, then redirects on the normalised path (followed to the final target), then strict URLs, then the SEO. When the URL is not in its final form the answer is a redirect, reached in one hop:

{"action": "redirect", "status": 301, "location": "https://example.com/pages/b", "url": "https://example.com/pages/a", "locale": "", "staging": false, "seo": null}

status is 302 when a temporary redirect takes part. A 410 rule answers "action": "gone", "status": 410. A redirect cycle met at runtime (stored past the save check) stops after 10 hops with not_found and is logged once per URL. A path that no published record owns and no URL rule covers answers "action": "not_found", "status": 404 and "seo": null (the home page / is always known). A url that is neither a path nor an http(s) URL is a 422. Absolute URLs are built from golem15.seo.site_url, or from the request URL when it names the site's host or its www twin (for the scheme and host rules), never from the request's Host header.

The 404 report endpoint

POST /_seo/api/v1/notfound takes {"url": "/missing", "referer": "https://..."} from the frontend server with Authorization: Bearer <golem15.seo.notfound_token> (compared in constant time) and answers 204. A missing or wrong token is 401; with no token configured the endpoint is off and answers 404. URLs are capped at 2048 bytes (422 beyond) and referers at 1024. The log holds at most golem15.seo.notfound_max_rows addresses; new ones beyond it are dropped with one warning an hour. Throttle bucket seo-notfound, 600 requests a minute per client IP.

robots.txt

GET /robots.txt (bucket seo-robots, 120 requests a minute) serves text/plain:

User-agent: *
Disallow: /admin

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

The stored rules come first, then one group per switched-off AI bot, then one Sitemap: line per entry of golem15.seo.robots_sitemaps, absolute on the site URL. Outside production the body is User-agent: * / Disallow: / with X-Robots-Tag: noindex, nofollow. It never answers a 5xx: when the stored file cannot be read it serves the last body it served, else User-agent: * / Disallow: (allow all), and logs the error.

The robots.txt screen (a singleton) has the editor, the tester and What crawlers get on its first tab and the AI bot switches on the second. The tester sends the editor's text and the switches as they are on screen, saved or not, and answers allowed or blocked for the chosen crawler with the deciding line, or that a bot switch blocked it.

Verification files

GET /_seo/api/v1/verification/{name} serves a verification file from the settings tab Verification (name [A-Za-z0-9._-]+\.(html|txt|xml)) with the content type of its extension, X-Content-Type-Options: nosniff and a sandboxing Content-Security-Policy; any other name is 404. The frontend serves it at the site root; the plugin registers no /{name} route, which would capture every one-segment page of a site served at GET /. Every /_seo/api/v1 answer carries X-Robots-Tag: noindex.

Admin

• Navigation item SEO (icon search), landing on Redirects, with the side menu Redirects, 404 log, URL rules, robots.txt and History.
• Settings tile SEO settings (seo_settings), a link to the settings singleton controller.
• Every SEO screen and the SEO tab require golem15.seo.manage. Without it an administrator sees no SEO tab, and a posted SEO value changes nothing.

The element bundle (assets/dist/seo.js and seo.css) is built from assets/src/elements and committed; the SEO field types load it on every admin page. After changing assets/src, run npm ci, npm run build and commit assets/dist; scripts/check-dist.sh fails when the committed bundle is stale.

API reference

Identifier	Description
Plugin	The golem15.seo plugin; Plugin.Service returns the booted classes.Service.
TabSEO, TabGeneral, TabRobotsFile, TabRobotsBots	The SEO tab key, the settings' General tab key and the robots.txt screen's tab keys.
BucketResolve, BucketNotFound, BucketRobots	The throttle buckets of resolve (and the verification files), the 404 report and /robots.txt.
RobotsTestAction	robots_test, the tester's widget action.
classes.Service	The single entry point for SEO reads and writes: SaveRecord, Record (history), GenerateEntries, EnqueueGenerate, RecordDeleted, RecordCreated, Resolver, SiteURL, Staging. Redirects: SaveRedirect, DeleteRedirect, CheckRedirect, AddressChanges, AddressChanged. 404 log: ReportNotFound, ResolveNotFound. robots.txt: ServedFile, TestRobots, BlockedNoindex. VerificationFileByName.
classes.Resolver	Resolve(ctx, url) answers a classes.Result (actions ok, not_found, redirect, gone) with a source per value in Result.Sources; ResolveRecord resolves a record at its own address; Strict checks a path against the record's address.
classes.RedirectSet, classes.NewRedirectSet, classes.LoadRedirectSet	A compiled redirect rule set: Match and Follow (classes.Chain, at most classes.MaxRedirectHops).
classes.RedirectCheck, classes.LoopError, classes.ProblemsError	The save-time simulation's answer and the refusals of SaveRedirect.
classes.AddressChange, classes.AddressResult	An address change and what AddressChanged did.
classes.Normalize, classes.NormalizeOptions	The address rules as a pure function.
classes.Bots, classes.Bot, classes.ComposeRobots	The AI bot list and the composition of the production robots.txt.
classes.ParseRobots, classes.Robots	The robots value grammar: parse, String (stored form), Header (meta and header form), Inherit (per-directive precedence).
classes.RenderTemplate	Fills a title or description template.
classes.GenerateEntriesArgs	The arguments of the generation job (Type limits it to one content type).
classes.FieldTypes	The SEO admin field types.
classes.WithVia, classes.ViaOf	The writer history records (admin, mcp, import, auto, undo).
controllers.PermissionManage	golem15.seo.manage.
models.Entry, models.Rule, models.Redirect, models.NotFound, models.Robots, models.Settings, models.TypeSetting, models.Revision	The plugin's models.

Configuration

Key	Default	Meaning
golem15.seo.site_url	""	The public origin of the site (https://example.com); absolute URLs start with it. Empty uses app.url.
golem15.seo.unprefixed_locale	""	The locale whose public URLs carry no locale prefix; empty leaves the choice to each content type.
golem15.seo.notfound_token	""	The bearer token of POST /_seo/api/v1/notfound; empty turns the endpoint off.
golem15.seo.notfound_max_rows	10000	The most addresses the 404 log holds.
golem15.seo.robots_sitemaps	["/sitemap.xml"]	The sitemap addresses robots.txt names, one Sitemap: line each, absolute on the site URL.

The non-production lock reads the application environment (SUMMER_ENV); it has no key of its own. The address rules (host, trailing slash, lowercase) are settings, on the Addresses tab.

CLI commands

Command	Description
seo:generate-entries [--type <key>]	Queues the entry generation job for every content type, or for one. A worker runs it.

Dependencies

• Framework modules: pact, party, backpack, cabana, signpost, lagoon, conga, surf, wire, bouncer, phrasebook, bonfire, compass, diary.
• gorm.io/gorm, github.com/go-gormigrate/gormigrate/v2, gocloud.dev/blob (the media bucket for social images), github.com/goccy/go-yaml (the per-type settings tabs and robots form documents).
• github.com/jimsmart/grobotstxt v1.0.3, the robots.txt matcher (decision note .planning/notes/2026-10-27-robots-txt-grobotstxt.md in the framework repository).
• Element toolchain (development only, exact pins): typescript, vite, vitest, happy-dom. The bundle has no runtime dependencies.

Testing

go vet ./...
go test -short ./...          # without Docker
go test ./...                 # with Docker: the Postgres smoke tests
npm ci && npm run typecheck && bash scripts/check-dist.sh

The smoke tests boot the plugin with fake content plugins (acme.pages, and acme.blog with a category segment for strict URLs) and exercise the SEO tab, the settings tabs, the resolver precedence, history, generation, the URL rule form, redirects (engine, loops and chains, address changes from the admin and from events), the 404 log, normalisation and strict URLs, robots.txt and verification files through the admin and public APIs.