sm-sitemap-plugin/README.md

# golem15.sitemap

The compiled sitemap plugin for SummerCMS applications: it serves `/sitemap.xml` and its stylesheet from one sitemap definition editors manage in the admin, byte for byte the way the WinterCMS Golem15.Sitemap plugin did.

```go
import _ "git.golem15.com/golem15/sm-sitemap-plugin"
```

Module `git.golem15.com/golem15/sm-sitemap-plugin`, package `sitemap`, plugin ID `golem15.sitemap`.

## Overview

The plugin is a compiled SummerCMS plugin. Its `init` registers it with the framework, so an application loads it by listing it in `summer.yaml`; `summer build` then emits the blank import above into the application's `plugins.gen.go`.

It is a port of the WinterCMS plugin for headless sites. An application has one sitemap definition (`models.Definition`, theme key `default`), a list of items: explicit URLs and items of the menu item types other plugins register with the framework's registry (`signpost`). `Requires()` is empty: content plugins such as the Journal register their types and the sitemap reads them through the registry.

The output keeps the PHP plugin's bytes: element order, escaping, whitespace, date format, headers and the `frontend_url` rewrite are not changed. The goldens in `classes/testdata` are generated from the PHP code and pin them.

## Features

- **Routes.** Anonymous `GET /sitemap.xml` and `GET /sitemap.xsl` at the application root, without the backend middleware. `/sitemap.xml` is throttled per client IP and its output is cached for a short time.
- **One definition.** Table `golem15_sitemap_definitions` keeps the PHP columns (`id`, `theme` nullable and indexed, `data` text, timestamps). The `theme` column holds the fixed key `default` (`models.DefinitionTheme`); `/sitemap.xml` serves that row. `models.Definition.Fillable` is exactly `theme` and `data`, the PHP `$fillable` with `$guarded = ['*']`.
- **Items.** An item is a URL (`type: url`) or a registry menu item type. Winter.Pages static pages, all static pages and the CMS page field are not ported. Stored item JSON in the PHP shape decodes tolerantly: numbers, strings and booleans are coerced, and unknown keys (`cmsPage`, `code`, `exists`, `fillable` and any other) are ignored. A save writes the PHP `toArray` key set (`changefreq`, `priority`, `nesting`, `type`, `url`, `reference`, `cmsPage` as null, `urlPattern`).
- **Settings.** A singleton row (`golem15_sitemap_settings`, id 1) holds the frontend URL, the frontend default locale, the updates manifest path and the priority tiers, with the PHP defaults.
- **Priority tiers.** Rules in config (`golem15.sitemap.priority_tiers`) map URL paths to five priority settings; the shipped defaults are the PHP plugin's rules.
- **Hooks.** Typed festival events `classes.BeforeAddItem` and `classes.AddItem` replace the PHP events `golem15.sitemap.beforeAddItem` and `golem15.sitemap.addItem`.
- **Admin.** The definition opens from the Settings page as a singleton form with a repeater of item rows; a second settings entry holds the settings form. Permission `golem15.sitemap.access_definitions`.
- **Commands.** `sitemap:import` (the PHP JSON import) and `sitemap:import-winter` (a copy of a WinterCMS site's definition and settings).
- **Phrasebooks.** `lang/en/lang.yaml` and `lang/pl/lang.yaml` keep the PHP key names.

## Usage

Mount the plugin in an application as a git submodule at `plugins/golem15/sitemap`, add it to the application's `go.work` and `go.mod`, and list it in `summer.yaml`:

```yaml
plugins:
  - id: golem15.sitemap
    module: git.golem15.com/golem15/sm-sitemap-plugin
```

The plugin's own `go.mod` and `go.work` replace the framework with `../summercms.go` for a sibling checkout.

### /sitemap.xml and /sitemap.xsl

`GET /sitemap.xml` answers:

- `404` when there is no definition with the theme `default`;
- `200` with an empty body when the definition has no items;
- otherwise the sitemap, `Content-Type: application/xml`, `Cache-Control: no-cache, private`.

Generating the sitemap resolves every registry item (all published posts, all categories, their translations), so the route is guarded:

- **Throttle.** The surf bucket `sitemap-xml` (`sitemap.BucketSitemap`) allows 60 requests a minute per client IP, read behind the host's trusted proxies. The 429 body is surf's `{"message":"Too Many Attempts."}`. `/sitemap.xsl` is not throttled.
- **Cache.** The generated document is kept in memory for `golem15.sitemap.cache_seconds` seconds (`sitemap.CacheConfigKey`, default `60`; `0` turns it off, a negative value fails the boot). The cache key covers the root URL and the stored definition and settings, so a save of either is served at once; a change of registry content (a new or unpublished post) shows once the entry expires. One generation runs at a time; concurrent requests for a missing entry wait for it.

The document is the PHP `DOMDocument::saveXML()` format: the XML declaration and the `xml-stylesheet` processing instruction each followed by a newline, everything else on one line ending with `</urlset>` and a newline. Items that all resolve to nothing produce a self-closing `<urlset .../>`. Text is escaped like libxml (`&`, `<`, `>` and carriage return; attributes also `"`, tab and newline); single quotes and UTF-8 stay literal.

Each URL is written in the PHP order (`classes.Generator`):

1. The 50000th URL is written, the 50001st is not (`classes.MaxURLs`).
2. A URL item's address goes through the port of Winter's `Url::to` (`classes.URLTo`): root plus the path trimmed of slashes, `/` keeps its slash, every path segment re-encoded like `rawurlencode`, valid absolute, `mailto:`, `tel:` and fragment URLs kept.
3. `lastmod` is the record's modification time, else the updates manifest's date for the URL path, else omitted. It is written as `2006-01-02T15:04:05+00:00` in UTC.
4. `classes.BeforeAddItem` may drop the URL.
5. A URL that does not come out as an `http` or `https` URL (for example a `javascript:` URL an import or a registry type let through) is dropped, and so is such an alternate. The PHP plugin wrote any URL `Url::to` kept; the stylesheet renders `<loc>` as a link on the site's origin.
6. The priority is the first matching tier's setting when tiers are on, else the item's priority, written as stored text (`1.0` stays `1.0`).
7. `loc`, `lastmod`, `changefreq` and `priority` are written, then the alternates as `xhtml:link` elements, only when there are more than one.
8. `classes.AddItem` may append XML before `</url>`.

A registry item is resolved with its URL pattern, its nesting flag and the settings' frontend default locale as the unprefixed locale. A resolved item with a URL is written, then its sub-items once each with their own URL. A list type (a resolved item without a URL, such as all posts) writes each sub-item once per alternate, with that alternate's URL as `loc` and every alternate as `hreflang`, which is what the PHP list types returned.

When the settings hold a frontend URL, the backend URL (`app.url` when set, else the request's scheme and `Host`) is replaced literally inside every `<loc>`, every `xhtml:link href` and the stylesheet href, as the PHP `str_replace` did. The rewrite runs only for a frontend URL `models.AllowedFrontendURL` accepts, and a `<loc>` or `href` that would not stay an `http` or `https` URL keeps its value. The scheme is `https` for a TLS connection, or when the immediate peer is in `http.trusted_proxies` and its `X-Forwarded-Proto` says `https`.

`GET /sitemap.xsl` serves the PHP plugin's stylesheet byte for byte, `Content-Type: text/xsl; charset=UTF-8`.

A frontend served from another host routes both paths to the application, before its catch-all:

```nginx
location = /sitemap.xml {
    proxy_pass http://127.0.0.1:8096;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}
location = /sitemap.xsl {
    proxy_pass http://127.0.0.1:8096;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
}
```

### Item types

| Type | Item fields used |
|------|------------------|
| `url` | `url`, `changefreq`, `priority` |
| A registry type (for example `journal-post`) | `reference` (when the type takes one), `urlPattern` (overrides the host and default patterns), `nesting`, `changefreq`, `priority` |

The admin's type dropdown offers `url` and every registry type in registry order with its label. The reference dropdown depends on the row's type and lists the type's references (`signpost.ReferenceOptions`); `url` and types without references take none, and a reference the type does not offer is refused on save.

A save also refuses, per row, a missing `reference` for a registry type that takes one (such a row would resolve to nothing and be left out of the sitemap), a `url` that is not a path (relative or starting with a single `/`) or an absolute `http`/`https` URL (`classes.AllowedItemURL`: no other scheme, no protocol-relative `//host`, no bare fragment, no surrounding whitespace or control characters), and a `urlPattern` that is not a path starting with a single `/` (`signpost.ValidItemPattern`). The errors are keyed `data.<row>.reference`, `data.<row>.url` and `data.<row>.urlPattern`.

### Settings

| Setting | Default | Meaning |
|---------|---------|---------|
| `frontend_url` | empty | The public frontend URL that replaces the backend URL in the XML: empty, or an absolute `http` or `https` URL with a host and without whitespace, control characters, a double quote, a less-than sign, a greater-than sign or an ampersand (`models.AllowedFrontendURL`). A stored value outside this form is not applied and the sitemap keeps the backend URL. |
| `frontend_default_locale` | empty | The locale whose URLs carry no prefix, passed to registry types; empty leaves it to the type. |
| `updates_manifest_path` | empty | A JSON file mapping a URL path to an ISO 8601 date, used as `lastmod` for items without their own date. |
| `enable_priority_tiers` | on | Tier priorities replace item priorities. |
| `priority_homepage`, `priority_services`, `priority_blog`, `priority_categories`, `priority_legal` | `1.0`, `0.8`, `0.6`, `0.4`, `0.2` | The tier values, stored as text. |

A save refuses a `frontend_url` outside the accepted form with a 422 keyed `frontend_url` (phrase `golem15.sitemap::lang.errors.frontend_url`), and `sitemap:import-winter` stops with an error. The PHP plugin stored any text and wrote it unescaped into `<loc>`, the hreflang links and the stylesheet instruction.

The updates manifest is read only when it is a regular file of at most 5 MiB (`classes.ManifestMaxBytes`) holding a JSON object; paths match with the leading slash added and the trailing slash removed (`/` stays). Any error omits `lastmod` and is logged without the file's content. The manifest is a frontend build output, for example a script that writes the git commit date of each static page's source:

```json
{
  "/": "2026-06-10T09:12:44+00:00",
  "/about": "2026-05-02T14:31:00+00:00"
}
```

### Priority tiers

With `enable_priority_tiers` on, the URL's path without trailing slashes (`""` for the homepage) is matched against the rules in `golem15.sitemap.priority_tiers` in order; the first match's setting value is the priority, and a URL no rule matches keeps its item's priority. A rule names a `setting` and either `exact` (a list of paths) or `pattern` (a regular expression in RE2 syntax). The rules are compiled at boot: an unknown setting, a rule with neither or both of `exact` and `pattern`, or a pattern that does not compile fails the boot (`classes.CompileTiers`).

The plugin's `config/config.yaml` ships the rules the PHP plugin had in code, so a site configured the same way gets the same priorities. A host replaces the whole list in its own config, for example `config/golem15.yaml`:

```yaml
sitemap:
  priority_tiers:
    - setting: priority_homepage
      exact: ["", "/en"]
    - setting: priority_blog
      pattern: '^/(en/)?articles/[^/]+$'
```

### Hooks

`classes.BeforeAddItem` fires through `festival.Bus.UntilHandled` before a URL is counted; a listener that sets `Skip` drops it (the PHP listener returned `false`). `classes.AddItem` fires through `festival.Bus.Fire` after a URL's elements are built; a listener may append escaped child XML to `Extra`. Both carry the definition item and a `classes.ItemInfo` (URL, modification time, written `lastmod`, alternates).

```go
func (p *Plugin) Boot(app *backpack.App) error {
	app.Events.Listen("acme.seo", func(ctx context.Context, e *classes.BeforeAddItem) error {
		if strings.HasSuffix(e.Info.URL, "/draft") {
			e.Skip = true
		}
		return nil
	})
	app.Events.Listen("acme.seo", func(ctx context.Context, e *classes.AddItem) error {
		e.Extra = append(e.Extra, "<mobile>"+classes.EscapeText("yes")+"</mobile>")
		return nil
	})
	return nil
}
```

### Admin

Two entries on the Settings page, category CMS, both requiring `golem15.sitemap.access_definitions` (developer role):

- **Sitemap** links to the definitions controller (`golem15.sitemap.definitions`), a singleton form: the server opens the definition with the theme `default`, creating it when absent, and the form has no list, create or delete. Its one field is a repeater on `data` whose rows hold type, reference, URL, URL pattern, priority, change frequency and the nesting checkbox.
- **Sitemap settings** is the settings form.

The WinterCMS forms are adapted to the admin's YAML:

- The definition form's tab is flattened, and the `SitemapItems` form widget is replaced by a `type: repeater` field on `data`.
- `span: none` and `cssClass` become `left` and `right`.
- The settings form's nginx hint and Priority Tiers section are field comments and this README; the `trigger` conditions on the tier fields are dropped, so those fields always show and their comments say when they apply.
- The tier comments no longer name one site's paths.
- The CMS page field and its messages are not ported; the editor's empty-definition action (`empty_confirm`) is not ported.
- The form's "View sitemap.xml" subtitle link (`definition.view`) waits for a form schema key in the framework; the phrase exists.

## API reference

| Identifier | Description |
|------------|-------------|
| `sitemap.Plugin` | Compiled plugin; `ID` is `golem15.sitemap`. |
| `sitemap.Plugin.Routes` | `GET /sitemap.xml` (throttled, cached) and `GET /sitemap.xsl`. |
| `sitemap.Plugin.Buckets` | Registers the `sitemap-xml` throttle bucket: 60 requests a minute per client IP. |
| `sitemap.BucketSitemap` | `sitemap-xml`, the throttle bucket of `/sitemap.xml`. |
| `sitemap.CacheConfigKey` | `golem15.sitemap.cache_seconds`, the seconds a generated sitemap is cached. |
| `sitemap.Plugin.Boot` | Compiles the tier rules and starts an empty sitemap cache; a bad rule or a negative `cache_seconds` fails the boot. |
| `sitemap.Plugin.ConfigFS` | The default `config/config.yaml` (cache time and tier rules). |
| `sitemap.Plugin.Commands` | Registers `sitemap:import` and `sitemap:import-winter`. |
| `sitemap.Plugin.AdminControllers` | The definitions controller. |
| `sitemap.Plugin.Settings` | The Sitemap link and the Sitemap settings form. |
| `sitemap.Plugin.Permissions` | `golem15.sitemap.access_definitions`. |
| `sitemap.Plugin.LangFS`, `sitemap.Plugin.AdminFS` | Embedded phrasebooks and admin YAML. |
| `sitemap.Plugin.Migrations`, `sitemap.Plugin.Models` | The two migrations and the two models. |
| `models.Definition` | The definition row; `Items` and `SetItems` read and write its item list. |
| `models.DefinitionTheme` | The fixed theme key `default`. |
| `models.DefinitionItem`, `models.DefinitionItems` | One item and the stored list, with the tolerant PHP-shaped decoding. |
| `models.ItemTypeURL` | The built-in item type `url`. |
| `models.Text`, `models.DecodeText` | A text field's value; the tolerant decoding of one stored text value. |
| `models.Settings`, `models.DefaultSettings` | The settings row and its PHP defaults; `Settings.Priority` reads a tier value, `Settings.BeforeValidate` refuses a `frontend_url` outside the accepted form on a save. |
| `models.AllowedFrontendURL` | Whether a value may be the `frontend_url` setting: empty, or an absolute `http`/`https` URL with a host and without whitespace, control characters, `"`, `<`, `>` or `&`. |
| `models.PriorityHomepage`, `models.PriorityServices`, `models.PriorityBlog`, `models.PriorityCategories`, `models.PriorityLegal`, `models.PriorityDefaults` | The tier setting keys and their defaults. |
| `controllers.DefinitionsControllerID`, `controllers.PermissionAccessDefinitions` | `golem15.sitemap.definitions` and `golem15.sitemap.access_definitions`. |
| `classes.Generator` | Writes a definition's sitemap; `Generate`, `Stylesheet`. |
| `classes.Resolver` | What the generator resolves registry items through (`*signpost.Registry`). |
| `classes.Serialize`, `classes.URLEntry`, `classes.Alternate` | The libxml-format writer and its entries. |
| `classes.EscapeText`, `classes.EscapeAttr` | libxml's text and attribute escaping. |
| `classes.URLTo`, `classes.BuildURL`, `classes.URLPath` | Ports of Winter `Url::to`, `Url::buildUrl` and `parse_url(..., PHP_URL_PATH)`. |
| `classes.AllowedItemURL` | Whether a url item's address may be stored: a path or an absolute `http`/`https` URL. |
| `classes.Rewrite`, `classes.BackendURL` | The `frontend_url` rewrite, applied only for a value `models.AllowedFrontendURL` accepts, and the backend URL it replaces. |
| `classes.TiersConfigKey`, `classes.TierConfig`, `classes.TierRule`, `classes.Tiers`, `classes.CompileTiers`, `classes.LoadTiers` | The tier rules. |
| `classes.Manifest`, `classes.LoadManifest`, `classes.ManifestMaxBytes` | The updates manifest. |
| `classes.BeforeAddItem`, `classes.AddItem`, `classes.ItemInfo` | The hooks. |
| `classes.MaxURLs` | 50000. |
| `console.Import`, `console.ImportFile` | `sitemap:import`. |
| `console.ImportWinter`, `console.ImportWinterFiles`, `console.WinterOptions`, `console.WinterReport` | `sitemap:import-winter`. |

## CLI commands

| Command | Arguments and flags | Effect |
|---------|---------------------|--------|
| `sitemap:import` | `FILE`, `--theme`, `--append` | The PHP `ImportCommand`: reads a JSON array of `{url, changefreq, priority}` entries (`monthly` and `0.5` by default). Without `--append` the definition keeps its non-URL items first and the imported URLs replace its URL items; with it the URLs follow every existing item. The definition is always `default`; a `--theme` other than `default` is reported. An entry whose URL `classes.AllowedItemURL` refuses stops the import. |
| `sitemap:import-winter` | `--definitions FILE`, `--settings FILE`, `--theme`, `--map old=new` (repeatable), `--page-url reference=/path` (repeatable), `--force`, `--dry-run` | Copies a WinterCMS export into the definition and the settings row. URL items are kept; static and CMS page items are dropped unless `--page-url` names the URL of their reference (for example `--page-url blog=/blog` copies the CMS page item `blog` as a url item `/blog`); `--map` renames types first (for example `--map blog-post=journal-post`); an item of an unknown type, or whose reference is not among its type's current references, is dropped; a url item whose URL `classes.AllowedItemURL` refuses is dropped, and so is a page item whose `--page-url` it refuses; a `urlPattern` that is not a path is cleared. Every dropped, remapped or cleared item and every dropped `cmsPage` is printed. A definition that already has items is refused without `--force`; `--dry-run` writes nothing. Settings missing from the export take the defaults; a `frontend_url` which `models.AllowedFrontendURL` refuses stops the import before anything is written, also with `--dry-run`. `--theme` picks a definition when the export holds several. |

Both commands open the database themselves when the application has not published one.

The export runs on the WinterCMS database with the read-only scripts in `scripts/`:

```sh
mysql -N -B --raw <database> < scripts/winter-export-definitions.sql > sitemap-definitions.jsonl
mysql -N -B --raw <database> < scripts/winter-export-settings.sql > sitemap-settings.json
./bin/acme sitemap:import-winter --definitions sitemap-definitions.jsonl --settings sitemap-settings.json --dry-run
./bin/acme sitemap:import-winter --definitions sitemap-definitions.jsonl --settings sitemap-settings.json
```

## Dependencies

- SummerCMS modules: backpack, bonfire, cabana, compass, festival, lagoon, pact, party, signpost, surf; bouncer in tests. cabana is a regular dependency: `models` and `controllers` return its validation error.
- Third-party: `gorm.io/gorm`, `github.com/go-gormigrate/gormigrate/v2`. Test-only: `github.com/testcontainers/testcontainers-go`, `github.com/jackc/pgx/v5`.

## Testing

```sh
go test -short ./...   # goldens and config checks, no Docker
go test ./...          # also the Postgres smoke tests
```

`TestSitemapGolden` (`classes`) feeds `testdata/tracer.json`, `urlto.json` and `full.json` through the Go port and compares bytes with the goldens; it also checks the 50000 URL cap. `TestSitemapTiersConfig` checks the tier rule refusals, a host's replacement of the rule list and the manifest's fail-safe reads. `TestRewriteUnsafeFrontendURL` (`classes`), `TestAllowedFrontendURL` and `TestSettingsBeforeValidate` (`models`) and `TestWinterSettingsFrontendURL` (`console`) are the Docker-free checks of the `frontend_url` rules: the accepted form, the rewrite that is not applied for any other value, the save hook and the import refusal. The Postgres smokes (`TestSitemapTracer`, `TestSitemapImportSmoke`, `TestSitemapAdminSmoke`, `TestImportWinter`, `TestSitemapCacheAndThrottle`) boot the plugin with a test content plugin and drive the routes, the import commands and the admin API; they cover the 422 of a refused `frontend_url` save and a value stored straight in the row, which the sitemap does not apply.

The goldens are generated from the PHP plugin's generation code and the vendored Winter URL generator of any WinterCMS application, and committed, so CI needs no PHP:

```sh
WINTER_VENDOR=/path/to/winter-app/vendor php tests/php/gen-goldens.php classes/testdata/full.json classes/testdata/full.golden
```