golem15/sm-sitemap-plugin
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: README for the per-type sitemap format

Browse Source 
- routes, site address, lastmod rules, rebuilds, the contract package
  with a Filter example, the Listing, the settings screen, config keys
- the breaking change from v0.1 and the definitions migration
- plugin.yaml regenerated from the new summary
This commit is contained in:
Jakub Zych
2026-11-04 11:23:19 +01:00
parent ded1095c39
commit 64453d2a91
2 changed files with 145 additions and 165 deletions
Download Patch File Download Diff File Expand all files Collapse all files

308
README.md
View File

@@ -1,6 +1,6 @@
# golem15.sitemap
The compiled sitemap plugin for SummerCMS applications: it serves `/sitemap.xml` and its stylesheet from one sitemap definition editors manage in the admin, byte for byte the way the WinterCMS Golem15.Sitemap plugin did.
The compiled sitemap plugin for SummerCMS applications: it serves a sitemap index at `/sitemap.xml` whose per-type files of a configurable size keep their numbers across rebuilds and carry real lastmod and hreflang alternates, built from every content type's published records.
```go
import _ "git.golem15.com/golem15/sm-sitemap-plugin"
@@ -12,21 +12,30 @@ Module `git.golem15.com/golem15/sm-sitemap-plugin`, package `sitemap`, plugin ID
The plugin is a compiled SummerCMS plugin. Its `init` registers it with the framework, so an application loads it by listing it in `summer.yaml`; `summer build` then emits the blank import above into the application's `plugins.gen.go`.
It is a port of the WinterCMS plugin for headless sites. An application has one sitemap definition (`models.Definition`, theme key `default`), a list of items: explicit URLs and items of the menu item types other plugins register with the framework's registry (`signpost`). `Requires()` is empty: content plugins such as the Journal register their types and the sitemap reads them through the registry.
Content plugins describe their records to the framework's registry (`signpost` content types). The sitemap reads them through `signpost.Registry.Records`, page by page, and never imports a content plugin: `Requires()` is empty. Every listed address is a row of the slot table `golem15_sitemap_urls`, which places it in a file `sitemap-<slug>-<n>.xml` of its group. A rebuild keeps every address in its file, appends new ones to the last file with room and leaves a hole where an address left; only a changed file size renumbers the files. The routes serve from the table and never generate per request.
The output keeps the PHP plugin's bytes: element order, escaping, whitespace, date format, headers and the `frontend_url` rewrite are not changed. The goldens in `classes/testdata` are generated from the PHP code and pin them.
Other plugins narrow the sitemap through the `contract` package: the SEO plugin publishes a `contract.Filter` that leaves noindex, canonicalised and redirected addresses out, and reads the sitemap through the `contract.Listing` the plugin publishes. Without a Filter every published address is listed.
### Breaking change from v0.1 (D-32)
v0.1 served one `/sitemap.xml` in the WinterCMS byte format from a definition of items, with priority, change frequency, priority tiers, an update manifest and the `/sitemap.xsl` stylesheet. The per-type format replaces it:
- `/sitemap.xml` is now a sitemap index; its URL is unchanged, so robots.txt `Sitemap:` lines and Search Console submissions keep working.
- The files are `/sitemap-<slug>-<n>.xml` at the site root. Go serves them at `/_sitemap/files/{file}`; the site's SEO layer (vue-seo-layer) or its proxy maps the root names there.
- `/sitemap.xsl` answers 404, and no `<priority>` or `<changefreq>` is written.
- The definitions screen and the old settings tile are one settings singleton. The url items of the definition move to the additional addresses, `frontend_url` to the site address and `frontend_default_locale` to the default locale (migration `202610280002`). Reference items are dropped: the content types list their records. The `golem15_sitemap_definitions` table and the Phase 17 settings columns stay in the database, unread, so a rollback loses nothing.
- The config keys `golem15.sitemap.cache_seconds` and `golem15.sitemap.priority_tiers` and the festival events `BeforeAddItem` and `AddItem` are gone.
## Features
- **Routes.** Anonymous `GET /sitemap.xml` and `GET /sitemap.xsl` at the application root, without the backend middleware. `/sitemap.xml` is throttled per client IP and its output is cached for a short time.
- **One definition.** Table `golem15_sitemap_definitions` keeps the PHP columns (`id`, `theme` nullable and indexed, `data` text, timestamps). The `theme` column holds the fixed key `default` (`models.DefinitionTheme`); `/sitemap.xml` serves that row. `models.Definition.Fillable` is exactly `theme` and `data`, the PHP `$fillable` with `$guarded = ['*']`.
- **Items.** An item is a URL (`type: url`) or a registry menu item type. Winter.Pages static pages, all static pages and the CMS page field are not ported. Stored item JSON in the PHP shape decodes tolerantly: numbers, strings and booleans are coerced, and unknown keys (`cmsPage`, `code`, `exists`, `fillable` and any other) are ignored. A save writes the PHP `toArray` key set (`changefreq`, `priority`, `nesting`, `type`, `url`, `reference`, `cmsPage` as null, `urlPattern`).
- **Settings.** A singleton row (`golem15_sitemap_settings`, id 1) holds the frontend URL, the frontend default locale, the updates manifest path and the priority tiers, with the PHP defaults.
- **Priority tiers.** Rules in config (`golem15.sitemap.priority_tiers`) map URL paths to five priority settings; the shipped defaults are the PHP plugin's rules.
- **Hooks.** Typed festival events `classes.BeforeAddItem` and `classes.AddItem` replace the PHP events `golem15.sitemap.beforeAddItem` and `golem15.sitemap.addItem`.
- **Admin.** The definition opens from the Settings page as a singleton form with a repeater of item rows; a second settings entry holds the settings form. Permission `golem15.sitemap.access_definitions`.
- **Commands.** `sitemap:import` (the PHP JSON import) and `sitemap:import-winter` (a copy of a WinterCMS site's definition and settings).
- **Phrasebooks.** `lang/en/lang.yaml` and `lang/pl/lang.yaml` keep the PHP key names.
- **Index and files.** `GET /sitemap.xml` lists one `<sitemap>` per file with an address, with the newest lastmod of its addresses. `GET /_sitemap/files/{file}` serves one file: `<loc>`, `<lastmod>` and `xhtml:link` alternates per address, `X-Robots-Tag: noindex`. Both are `application/xml`, anonymous and throttled per client IP.
- **Stable numbering.** A file holds at most the file size setting's addresses (default 1000, 1 to 50 000). Addresses keep their file number; new ones are appended oldest first; a deleted, unpublished or excluded address leaves a hole. A changed file size renumbers every file in the next rebuild.
- **Real lastmod.** A row's lastmod starts at the record's last change and then moves only when a `pact.RecordSaved` reports a change outside `golem15.sitemap.lastmod_ignore`: a no-op save, a counter update or a rebuild leaves it as it was.
- **hreflang.** Every address of a record with more than one listed locale carries an alternate per locale, itself included, plus `x-default` on the default locale's address. A locale without an address gets none, a single-locale record none.
- **Groups.** A content type's group is the last segment of its key in plural form (`journal-post`: `posts`, `journal-category`: `categories`, `seo-page`: `pages`), or the file name set in the settings. Additional addresses go to `pages`. A Filter may move an address to another group (`offers`).
- **Rebuilds.** A conga job (`golem15.sitemap.rebuild`): queued after record changes, debounced to one per `golem15.sitemap.rebuild_delay_seconds`; dispatched on every settings save; run every night at 03:00 by the scheduled `sitemap:rebuild`; and by hand.
- **Settings singleton.** One tile in the CMS settings category: Files (site address, file size, the files table), Content types (per type: include, file name, HTML sitemap heading) and Additional addresses.
- **Contract.** `contract.Filter` for other plugins to implement and `contract.Listing` for other plugins to read (D-95).
## Usage
@@ -40,215 +49,186 @@ plugins:
The plugin's own `go.mod` and `go.work` replace the framework with `../summercms.go` for a sibling checkout.
### /sitemap.xml and /sitemap.xsl
The application needs a conga worker (`summer queue:work`) for the rebuild job, and the scheduler for the nightly rebuild. Without a worker, run `sitemap:rebuild` yourself.
`GET /sitemap.xml` answers:
### Routes
- `404` when there is no definition with the theme `default`;
- `200` with an empty body when the definition has no items;
- otherwise the sitemap, `Content-Type: application/xml`, `Cache-Control: no-cache, private`.
| Route | Answer |
|-------|--------|
| `GET /sitemap.xml` | The index. `404` while no site address is configured (a `<loc>` is never built from the request's Host). Before the first rebuild it is an empty `<sitemapindex>`. |
| `GET /_sitemap/files/{file}` | One file, `{file}` matching `sitemap-[a-z0-9-]+-[0-9]+\.xml`. `404` for an unknown file or one whose addresses all left. |
| `GET /sitemap.xsl` | Gone: `404`. |
Generating the sitemap resolves every registry item (all published posts, all categories, their translations), so the route is guarded:
Both routes use the surf throttle bucket `sitemap-xml` (`sitemap.BucketSitemap`): 60 requests a minute per client IP, read behind the host's trusted proxies. Every value is written through `encoding/xml`'s escaping.
- **Throttle.** The surf bucket `sitemap-xml` (`sitemap.BucketSitemap`) allows 60 requests a minute per client IP, read behind the host's trusted proxies. The 429 body is surf's `{"message":"Too Many Attempts."}`. `/sitemap.xsl` is not throttled.
- **Cache.** The generated document is kept in memory for `golem15.sitemap.cache_seconds` seconds (`sitemap.CacheConfigKey`, default `60`; `0` turns it off, a negative value fails the boot). The cache key covers the root URL and the stored definition and settings, so a save of either is served at once; a change of registry content (a new or unpublished post) shows once the entry expires. One generation runs at a time; concurrent requests for a missing entry wait for it.
The index lists the files at the site root, for example:
The document is the PHP `DOMDocument::saveXML()` format: the XML declaration and the `xml-stylesheet` processing instruction each followed by a newline, everything else on one line ending with `</urlset>` and a newline. Items that all resolve to nothing produce a self-closing `<urlset .../>`. Text is escaped like libxml (`&`, `<`, `>` and carriage return; attributes also `"`, tab and newline); single quotes and UTF-8 stay literal.
```xml
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://www.example.com/sitemap-posts-1.xml</loc><lastmod>2026-10-28T09:00:00Z</lastmod></sitemap>
</sitemapindex>
```
Each URL is written in the PHP order (`classes.Generator`):
1. The 50000th URL is written, the 50001st is not (`classes.MaxURLs`).
2. A URL item's address goes through the port of Winter's `Url::to` (`classes.URLTo`): root plus the path trimmed of slashes, `/` keeps its slash, every path segment re-encoded like `rawurlencode`, valid absolute, `mailto:`, `tel:` and fragment URLs kept.
3. `lastmod` is the record's modification time, else the updates manifest's date for the URL path, else omitted. It is written as `2006-01-02T15:04:05+00:00` in UTC.
4. `classes.BeforeAddItem` may drop the URL.
5. A URL that does not come out as an `http` or `https` URL (for example a `javascript:` URL an import or a registry type let through) is dropped, and so is such an alternate. The PHP plugin wrote any URL `Url::to` kept; the stylesheet renders `<loc>` as a link on the site's origin.
6. The priority is the first matching tier's setting when tiers are on, else the item's priority, written as stored text (`1.0` stays `1.0`).
7. `loc`, `lastmod`, `changefreq` and `priority` are written, then the alternates as `xhtml:link` elements, only when there are more than one.
8. `classes.AddItem` may append XML before `</url>`.
A registry item is resolved with its URL pattern, its nesting flag and the settings' frontend default locale as the unprefixed locale. A resolved item with a URL is written, then its sub-items once each with their own URL. A list type (a resolved item without a URL, such as all posts) writes each sub-item once per alternate, with that alternate's URL as `loc` and every alternate as `hreflang`, which is what the PHP list types returned.
When the settings hold a frontend URL, the backend URL (`app.url` when set, else the request's scheme and `Host`) is replaced literally inside every `<loc>`, every `xhtml:link href` and the stylesheet href, as the PHP `str_replace` did. The rewrite runs only for a frontend URL `models.AllowedFrontendURL` accepts, and a `<loc>` or `href` that would not stay an `http` or `https` URL keeps its value. The scheme is `https` for a TLS connection, or when the immediate peer is in `http.trusted_proxies` and its `X-Forwarded-Proto` says `https`.
`GET /sitemap.xsl` serves the PHP plugin's stylesheet byte for byte, `Content-Type: text/xsl; charset=UTF-8`.
A frontend served from another host routes both paths to the application, before its catch-all:
A site whose frontend proxies the backend maps `/sitemap-<slug>-<n>.xml` to `/_sitemap/files/sitemap-<slug>-<n>.xml`. vue-seo-layer does it from v0.2.0; with nginx:
```nginx
location = /sitemap.xml {
proxy_pass http://127.0.0.1:8096;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location = /sitemap.xsl {
proxy_pass http://127.0.0.1:8096;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
location ~ ^/(sitemap-[a-z0-9-]+-[0-9]+\.xml)$ {
proxy_pass http://backend/_sitemap/files/$1;
}
```
### Item types
### Site address
| Type | Item fields used |
|------|------------------|
| `url` | `url`, `changefreq`, `priority` |
| A registry type (for example `journal-post`) | `reference` (when the type takes one), `urlPattern` (overrides the host and default patterns), `nesting`, `changefreq`, `priority` |
Every `<loc>` starts with the Site address setting; when it is empty, with the config key `golem15.sitemap.site_url`; else with `app.url`. Each must be an absolute `http`/`https` URL without whitespace, quotes, `<`, `>` or `&` (`models.AllowedFrontendURL`); a value outside that form is skipped.
The admin's type dropdown offers `url` and every registry type in registry order with its label. The reference dropdown depends on the row's type and lists the type's references (`signpost.ReferenceOptions`); `url` and types without references take none, and a reference the type does not offer is refused on save.
### Lastmod
A save also refuses, per row, a missing `reference` for a registry type that takes one (such a row would resolve to nothing and be left out of the sitemap), a `url` that is not a path (relative or starting with a single `/`) or an absolute `http`/`https` URL (`classes.AllowedItemURL`: no other scheme, no protocol-relative `//host`, no bare fragment, no surrounding whitespace or control characters), and a `urlPattern` that is not a path starting with a single `/` (`signpost.ValidItemPattern`). The errors are keyed `data.<row>.reference`, `data.<row>.url` and `data.<row>.urlPattern`.
`classes.RecordSaved` handles every `pact.RecordSaved` of a model a content type declares:
### Settings
- a change of an attribute on the record's own row moves the lastmod of all its addresses to now;
- a translated change (`RecordChange.Locale` set) moves only that locale's address;
- changes of attributes in `golem15.sitemap.lastmod_ignore` (default `updated_at`, `created_at`, `views`, `sort_order`) and a save without changes move nothing.
| Setting | Default | Meaning |
|---------|---------|---------|
| `frontend_url` | empty | The public frontend URL that replaces the backend URL in the XML: empty, or an absolute `http` or `https` URL with a host and without whitespace, control characters, a double quote, a less-than sign, a greater-than sign or an ampersand (`models.AllowedFrontendURL`). A stored value outside this form is not applied and the sitemap keeps the backend URL. |
| `frontend_default_locale` | empty | The locale whose URLs carry no prefix, passed to registry types; empty leaves it to the type. |
| `updates_manifest_path` | empty | A JSON file mapping a URL path to an ISO 8601 date, used as `lastmod` for items without their own date. |
| `enable_priority_tiers` | on | Tier priorities replace item priorities. |
| `priority_homepage`, `priority_services`, `priority_blog`, `priority_categories`, `priority_legal` | `1.0`, `0.8`, `0.6`, `0.4`, `0.2` | The tier values, stored as text. |
A `pact.RecordDeleted` takes the record's addresses out of the files at once. Both queue a debounced rebuild. A rebuild sets the lastmod of a new address from the record's `LastMod` (or the Filter's `Decision.LastMod`) and never moves an existing one; an additional address carries the lastmod the editor gives it.
A save refuses a `frontend_url` outside the accepted form with a 422 keyed `frontend_url` (phrase `golem15.sitemap::lang.errors.frontend_url`), and `sitemap:import-winter` stops with an error. The PHP plugin stored any text and wrote it unescaped into `<loc>`, the hreflang links and the stylesheet instruction.
### Rebuilds
The updates manifest is read only when it is a regular file of at most 5 MiB (`classes.ManifestMaxBytes`) holding a JSON object; paths match with the leading slash added and the trailing slash removed (`/` stays). Any error omits `lastmod` and is logged without the file's content. The manifest is a frontend build output, for example a script that writes the git commit date of each static page's source:
`classes.Rebuild` walks `signpost.Registry.ContentTypes` in registration order, pages `Records` 500 at a time, keeps the published records' locales with an address, asks the Filter per page, adds the additional addresses and writes everything in one transaction that holds the settings row (two rebuilds never interleave). It returns the added, changed and removed absolute URLs.
```json
{
"/": "2026-06-10T09:12:44+00:00",
"/about": "2026-05-02T14:31:00+00:00"
}
```
Record events set `rebuild_pending_until` with a conditional update and enqueue one rebuild with that delay only when no pending time lies in the future; the job clears it when it starts. A settings save dispatches a rebuild at once (a `summer_jobs` row labelled "Rebuild the sitemaps").
### Priority tiers
### Filter
With `enable_priority_tiers` on, the URL's path without trailing slashes (`""` for the homepage) is matched against the rules in `golem15.sitemap.priority_tiers` in order; the first match's setting value is the priority, and a URL no rule matches keeps its item's priority. A rule names a `setting` and either `exact` (a list of paths) or `pattern` (a regular expression in RE2 syntax). The rules are compiled at boot: an unknown setting, a rule with neither or both of `exact` and `pattern`, or a pattern that does not compile fails the boot (`classes.CompileTiers`).
The plugin's `config/config.yaml` ships the rules the PHP plugin had in code, so a site configured the same way gets the same priorities. A host replaces the whole list in its own config, for example `config/golem15.yaml`:
```yaml
sitemap:
priority_tiers:
- setting: priority_homepage
exact: ["", "/en"]
- setting: priority_blog
pattern: '^/(en/)?articles/[^/]+$'
```
### Hooks
`classes.BeforeAddItem` fires through `festival.Bus.UntilHandled` before a URL is counted; a listener that sets `Skip` drops it (the PHP listener returned `false`). `classes.AddItem` fires through `festival.Bus.Fire` after a URL's elements are built; a listener may append escaped child XML to `Extra`. Both carry the definition item and a `classes.ItemInfo` (URL, modification time, written `lastmod`, alternates).
A plugin that decides which addresses are indexable implements `contract.Filter` and publishes it in its `Register`. It imports the `contract` package only:
```go
func (p *Plugin) Boot(app *backpack.App) error {
app.Events.Listen("acme.seo", func(ctx context.Context, e *classes.BeforeAddItem) error {
if strings.HasSuffix(e.Info.URL, "/draft") {
e.Skip = true
import "git.golem15.com/golem15/sm-sitemap-plugin/contract"
type noindexFilter struct{}
func (noindexFilter) Decide(ctx context.Context, batch []contract.Candidate) ([]contract.Decision, error) {
out := make([]contract.Decision, len(batch))
for i, c := range batch {
if strings.HasPrefix(c.URL, "/drafts/") {
out[i] = contract.Decision{Exclude: true, Reason: contract.ReasonNoindex}
}
return nil
})
app.Events.Listen("acme.seo", func(ctx context.Context, e *classes.AddItem) error {
e.Extra = append(e.Extra, "<mobile>"+classes.EscapeText("yes")+"</mobile>")
return nil
})
return nil
}
return out, nil
}
func (noindexFilter) Groups(context.Context) ([]contract.Group, error) { return nil, nil }
func (p *Plugin) Register(app *backpack.App) error {
return app.Publish[contract.Filter](noindexFilter{})
}
```
`Decide` gets one page of records of one content type (or the additional addresses, type key `extra`) and answers one decision per candidate in order. An error or a wrong number of decisions fails the rebuild, and the files stay as they were. Excluded addresses are counted by reason in the settings screen (noindex, canonical, redirect; a type switched off counts as type).
### Listing
The plugin publishes `classes.Listing` as `contract.Listing`:
- `Settings` answers the site address, the index address, the default locale and the groups with their listed addresses, their slug and their titles (the content type's HTML sitemap heading, else the Filter's group titles);
- `Entries` pages a group's listed addresses (`EntriesQuery{Group, Locale, After, Limit}`, 100 by default, at most 1000) with absolute URLs, titles and lastmod, `Next` and `Total`.
The HTML sitemap and IndexNow fields of `contract.ListingSettings` stay zero until those settings exist.
### Admin
Two entries on the Settings page, category CMS, both requiring `golem15.sitemap.access_definitions` (developer role):
The Settings page has one Sitemaps tile (icon `network`, category CMS, permission `golem15.sitemap.access_definitions`). It opens the singleton controller `golem15.sitemap.settings`:
- **Sitemap** links to the definitions controller (`golem15.sitemap.definitions`), a singleton form: the server opens the definition with the theme `default`, creating it when absent, and the form has no list, create or delete. Its one field is a repeater on `data` whose rows hold type, reference, URL, URL pattern, priority, change frequency and the nesting checkbox.
- **Sitemap settings** is the settings form.
- **Files:** Site address, Addresses per file and the files table: the index and every file with its address count and newest lastmod, the addresses left out by reason, and a note naming the public index address. Before the first build it says there are no files yet; after a save and until the rebuild ran, that it is rebuilding.
- **Content types:** per signpost content type, in registration order, a switch labelled with the type's label, the file name (`[a-z0-9-]`, unique, `images` and `news` reserved) and the HTML sitemap heading per locale. Stored in `golem15_sitemap_types`.
- **Additional addresses:** a repeater of paths (or absolute URLs) with an optional lastmod as `YYYY-MM-DD`. They go into the `pages` group.
The WinterCMS forms are adapted to the admin's YAML:
- The definition form's tab is flattened, and the `SitemapItems` form widget is replaced by a `type: repeater` field on `data`.
- `span: none` and `cssClass` become `left` and `right`.
- The settings form's nginx hint and Priority Tiers section are field comments and this README; the `trigger` conditions on the tier fields are dropped, so those fields always show and their comments say when they apply.
- The tier comments no longer name one site's paths.
- The CMS page field and its messages are not ported; the editor's empty-definition action (`empty_confirm`) is not ported.
- The form's "View sitemap.xml" subtitle link (`definition.view`) waits for a form schema key in the framework; the phrase exists.
A save refuses an invalid site address, a file size outside 1 to 50 000, an address that is not a path or an `http`/`https` URL, a lastmod that is not a date and an invalid or taken file name, and dispatches a rebuild.
## API reference
| Identifier | Description |
|------------|-------------|
| `sitemap.Plugin` | Compiled plugin; `ID` is `golem15.sitemap`. |
| `sitemap.Plugin.Routes` | `GET /sitemap.xml` (throttled, cached) and `GET /sitemap.xsl`. |
| `sitemap.Plugin.Buckets` | Registers the `sitemap-xml` throttle bucket: 60 requests a minute per client IP. |
| `sitemap.BucketSitemap` | `sitemap-xml`, the throttle bucket of `/sitemap.xml`. |
| `sitemap.CacheConfigKey` | `golem15.sitemap.cache_seconds`, the seconds a generated sitemap is cached. |
| `sitemap.Plugin.Boot` | Compiles the tier rules and starts an empty sitemap cache; a bad rule or a negative `cache_seconds` fails the boot. |
| `sitemap.Plugin.ConfigFS` | The default `config/config.yaml` (cache time and tier rules). |
| `sitemap.Plugin.Commands` | Registers `sitemap:import` and `sitemap:import-winter`. |
| `sitemap.Plugin.AdminControllers` | The definitions controller. |
| `sitemap.Plugin.Settings` | The Sitemap link and the Sitemap settings form. |
| `sitemap.Plugin.Permissions` | `golem15.sitemap.access_definitions`. |
| `sitemap.Plugin.LangFS`, `sitemap.Plugin.AdminFS` | Embedded phrasebooks and admin YAML. |
| `sitemap.Plugin.Migrations`, `sitemap.Plugin.Models` | The two migrations and the two models. |
| `models.Definition` | The definition row; `Items` and `SetItems` read and write its item list. |
| `models.DefinitionTheme` | The fixed theme key `default`. |
| `models.DefinitionItem`, `models.DefinitionItems` | One item and the stored list, with the tolerant PHP-shaped decoding. |
| `models.ItemTypeURL` | The built-in item type `url`. |
| `models.Text`, `models.DecodeText` | A text field's value; the tolerant decoding of one stored text value. |
| `models.Settings`, `models.DefaultSettings` | The settings row and its PHP defaults; `Settings.Priority` reads a tier value, `Settings.BeforeValidate` refuses a `frontend_url` outside the accepted form on a save. |
| `models.AllowedFrontendURL` | Whether a value may be the `frontend_url` setting: empty, or an absolute `http`/`https` URL with a host and without whitespace, control characters, `"`, `<`, `>` or `&`. |
| `models.PriorityHomepage`, `models.PriorityServices`, `models.PriorityBlog`, `models.PriorityCategories`, `models.PriorityLegal`, `models.PriorityDefaults` | The tier setting keys and their defaults. |
| `controllers.DefinitionsControllerID`, `controllers.PermissionAccessDefinitions` | `golem15.sitemap.definitions` and `golem15.sitemap.access_definitions`. |
| `classes.Generator` | Writes a definition's sitemap; `Generate`, `Stylesheet`. |
| `classes.Resolver` | What the generator resolves registry items through (`*signpost.Registry`). |
| `classes.Serialize`, `classes.URLEntry`, `classes.Alternate` | The libxml-format writer and its entries. |
| `classes.EscapeText`, `classes.EscapeAttr` | libxml's text and attribute escaping. |
| `classes.URLTo`, `classes.BuildURL`, `classes.URLPath` | Ports of Winter `Url::to`, `Url::buildUrl` and `parse_url(..., PHP_URL_PATH)`. |
| `classes.AllowedItemURL` | Whether a url item's address may be stored: a path or an absolute `http`/`https` URL. |
| `classes.Rewrite`, `classes.BackendURL` | The `frontend_url` rewrite, applied only for a value `models.AllowedFrontendURL` accepts, and the backend URL it replaces. |
| `classes.TiersConfigKey`, `classes.TierConfig`, `classes.TierRule`, `classes.Tiers`, `classes.CompileTiers`, `classes.LoadTiers` | The tier rules. |
| `classes.Manifest`, `classes.LoadManifest`, `classes.ManifestMaxBytes` | The updates manifest. |
| `classes.BeforeAddItem`, `classes.AddItem`, `classes.ItemInfo` | The hooks. |
| `classes.MaxURLs` | 50000. |
| `console.Import`, `console.ImportFile` | `sitemap:import`. |
| `console.ImportWinter`, `console.ImportWinterFiles`, `console.WinterOptions`, `console.WinterReport` | `sitemap:import-winter`. |
| `sitemap.Plugin` | Compiled plugin; `ID` is `golem15.sitemap`, `Requires` is empty. |
| `sitemap.Plugin.Register` | Publishes `classes.Listing` as `contract.Listing`. |
| `sitemap.Plugin.Boot` | Subscribes to `pact.RecordSaved` and `pact.RecordDeleted`. |
| `sitemap.Plugin.Routes` | `GET /sitemap.xml` and `GET /_sitemap/files/{file}`. |
| `sitemap.Plugin.Buckets`, `sitemap.BucketSitemap` | The `sitemap-xml` throttle bucket: 60 requests a minute per client IP. |
| `sitemap.Plugin.Jobs` | The rebuild job (`classes.RebuildKind`), three attempts. |
| `sitemap.Plugin.Schedule` | `sitemap:rebuild` daily at 03:00. |
| `sitemap.Plugin.Commands` | `sitemap:rebuild`, `sitemap:import`, `sitemap:import-winter`. |
| `sitemap.Plugin.ConfigFS` | The default `config/config.yaml`. |
| `sitemap.Plugin.AdminControllers`, `sitemap.Plugin.Settings`, `sitemap.Plugin.Permissions` | The settings singleton, its tile and `golem15.sitemap.access_definitions`. |
| `sitemap.Plugin.AdminActionExtensions` | The Content types tab on the settings singleton. |
| `sitemap.TabFiles`, `sitemap.TabTypes` | The tab keys of the settings form. |
| `sitemap.Plugin.LangFS`, `sitemap.Plugin.AdminFS` | Embedded phrasebooks, admin YAML and partials. |
| `sitemap.Plugin.Migrations`, `sitemap.Plugin.Models` | The migrations and models. |
| `contract.Filter`, `contract.Candidate`, `contract.Decision`, `contract.Group` | The filter other plugins implement. |
| `contract.ReasonNoindex`, `contract.ReasonCanonical`, `contract.ReasonRedirect`, `contract.ReasonType` | The left-out reasons. |
| `contract.Listing`, `contract.ListingSettings`, `contract.GroupInfo`, `contract.EntriesQuery`, `contract.Entry`, `contract.EntriesPage` | What the sitemap publishes for other plugins. |
| `classes.Rebuild`, `classes.RebuildResult` | One rebuild of the slot table and what it changed. |
| `classes.RebuildArgs`, `classes.RebuildKind` | The rebuild job's arguments and kind `golem15.sitemap.rebuild`. |
| `classes.RequestRebuild`, `classes.DispatchRebuild`, `classes.RequestDebouncedRebuild`, `classes.ClearPending`, `classes.Building` | Dispatching rebuilds and the debounce window. |
| `classes.RecordSaved`, `classes.RecordDeleted`, `classes.RealChanges`, `classes.Changed` | The record event handling. |
| `classes.LastmodIgnore`, `classes.RebuildDelay`, `classes.DefaultLastmodIgnore`, `classes.LastmodIgnoreConfigKey`, `classes.RebuildDelayConfigKey` | The lastmod and debounce config. |
| `classes.AssignSlots`, `classes.Slot` | Stable file numbering. |
| `classes.WriteIndex`, `classes.WriteURLSet`, `classes.IndexFile`, `classes.FileEntry`, `classes.FileAlternate`, `classes.NSSitemap`, `classes.NSXHTML` | The XML writers. |
| `classes.FileName`, `classes.ParseFileName`, `classes.ValidSlug`, `classes.DefaultSlug`, `classes.ReservedSlugs` | File and group names. |
| `classes.SiteURL`, `classes.SiteURLConfigKey`, `classes.Absolute`, `classes.LoadSettings` | The site address and the settings row. |
| `classes.TypeConfig`, `classes.EffectiveType`, `classes.LoadTypeSettings` | The per-type settings. |
| `classes.Listing` | The `contract.Listing` implementation. |
| `classes.ExtraRecordID` | The record id of an additional address. |
| `models.URL`, `models.Alternate` | A row of the slot table and an hreflang alternate. |
| `models.ExtraTypeKey`, `models.ExtraGroup`, `models.ReasonDeleted`, `models.ReasonUnpublished` | Additional addresses and the holes' reasons. |
| `models.Settings`, `models.DefaultSettings`, `models.ExtraURL`, `models.DefaultFileSize`, `models.MaxFileSize` | The settings row; `Validate` and `Problems` are its save checks. |
| `models.TypeSetting` | A row of `golem15_sitemap_types`. |
| `models.AllowedFrontendURL`, `models.AllowedAddress`, `models.ParseDate`, `models.ValidDate` | The value checks. |
| `models.LegacyItem`, `models.ParseLegacyItems`, `models.ParseLegacyItem`, `models.LegacyEntries`, `models.LegacyJSON`, `models.DecodeText`, `models.Text` | Reading Phase 17 and WinterCMS definition data. |
| `controllers.SettingsControllerID`, `controllers.PermissionAccessDefinitions` | `golem15.sitemap.settings` and `golem15.sitemap.access_definitions`. |
| `console.Rebuild`, `console.Import`, `console.ImportFile`, `console.ImportWinter`, `console.ImportWinterFiles`, `console.WinterOptions`, `console.WinterReport` | The commands. |
## Configuration
`config/config.yaml`, read as `golem15.sitemap.<key>`:
| Key | Default | Meaning |
|-----|---------|---------|
| `site_url` | unset | The site address when the setting is empty (before `app.url`). |
| `lastmod_ignore` | `[updated_at, created_at, views, sort_order]` | Attributes whose change does not move lastmod. A host replaces the whole list. |
| `rebuild_delay_seconds` | `60` | The delay of the rebuild a record event queues; events within it share one rebuild. |
## CLI commands
| Command | Arguments and flags | Effect |
|---------|---------------------|--------|
| `sitemap:import` | `FILE`, `--theme`, `--append` | The PHP `ImportCommand`: reads a JSON array of `{url, changefreq, priority}` entries (`monthly` and `0.5` by default). Without `--append` the definition keeps its non-URL items first and the imported URLs replace its URL items; with it the URLs follow every existing item. The definition is always `default`; a `--theme` other than `default` is reported. An entry whose URL `classes.AllowedItemURL` refuses stops the import. Ends with `Imported N URLs into sitemap definition for theme 'default'.` |
| `sitemap:import-winter` | `--definitions FILE`, `--settings FILE`, `--theme`, `--map old=new` (repeatable), `--page-url reference=/path` (repeatable), `--force`, `--dry-run` | Copies a WinterCMS export into the definition and the settings row. URL items are kept; static and CMS page items are dropped unless `--page-url` names the URL of their reference (for example `--page-url blog=/blog` copies the CMS page item `blog` as a url item `/blog`); `--map` renames types first (for example `--map blog-post=journal-post`); an item of an unknown type, or whose reference is not among its type's current references, is dropped; a url item whose URL `classes.AllowedItemURL` refuses is dropped, and so is a page item whose `--page-url` it refuses; a `urlPattern` that is not a path is cleared. Every dropped, remapped or cleared item and every dropped `cmsPage` is printed. A definition that already has items is refused without `--force`; `--dry-run` writes nothing. Settings missing from the export take the defaults; a `frontend_url` which `models.AllowedFrontendURL` refuses stops the import before anything is written, also with `--dry-run`. `--theme` picks a definition when the export holds several. Ends with `Definition copied: …` and `Settings copied.` for the parts it wrote. |
| `sitemap:rebuild` | `--queue` | Rebuilds the files in the process and prints `Rebuilt the sitemaps: N addresses in N files (N added, N changed, N removed).`; with `--queue` dispatches the rebuild job instead. Scheduled daily at 03:00. |
| `sitemap:import` | `FILE`, `--append`, `--theme` | Reads a JSON array of `{url, lastmod}` entries into the additional addresses (replacing them; `--append` adds and skips listed ones). `changefreq` and `priority` are ignored, `--theme` is accepted and does nothing. One bad entry stops the import. |
| `sitemap:import-winter` | `--definitions FILE`, `--settings FILE`, `--theme`, `--map old=new` (repeatable), `--page-url reference=/path` (repeatable), `--force`, `--dry-run` | Copies a WinterCMS export: url items become additional addresses (`--page-url` turns a static or CMS page item into one, `--map` renames types first), every other item is dropped and listed; `frontend_url` becomes the site address and `frontend_default_locale` the default locale, the PHP-parity keys are reported as ignored. Stored addresses are replaced only with `--force`; `--dry-run` writes nothing; an unsafe `frontend_url` stops the import. |
Both commands open the database themselves when the application has not published one. `./bin/acme help <command>` shows each command's long help and examples. The plugin logs through the component `sitemap` (`log.components.sitemap` in `config/log.yaml`).
The commands open the database themselves when the application has not published one. `./bin/acme help <command>` shows each command's long help and examples. The plugin logs through the component `sitemap` (`log.components.sitemap` in `config/log.yaml`).
```sh
./bin/acme sitemap:import urls.json --append
```
The export runs on the WinterCMS database with the read-only scripts in `scripts/`:
The WinterCMS export runs on the WinterCMS database with the read-only scripts in `scripts/`:
```sh
mysql -N -B --raw <database> < scripts/winter-export-definitions.sql > sitemap-definitions.jsonl
mysql -N -B --raw <database> < scripts/winter-export-settings.sql > sitemap-settings.json
./bin/acme sitemap:import-winter --definitions sitemap-definitions.jsonl --settings sitemap-settings.json --dry-run
./bin/acme sitemap:import-winter --definitions sitemap-definitions.jsonl --settings sitemap-settings.json
./bin/acme sitemap:rebuild
```
## Dependencies
- SummerCMS v0.4.0 modules: backpack, bonfire, cabana, compass, diary, festival, lagoon, pact, party, signpost, surf; bouncer in tests. cabana is a regular dependency: `models` and `controllers` return its validation error.
- Third-party: `gorm.io/gorm`, `github.com/go-gormigrate/gormigrate/v2`. Test-only: `github.com/testcontainers/testcontainers-go`, `github.com/jackc/pgx/v5`.
- SummerCMS v0.6.1 modules: backpack, bonfire, cabana, compass, conga, diary, lagoon, pact, party, phrasebook, signpost, surf; bouncer and kiosk in tests. Until v0.6.1 is tagged, `go.mod` requires v0.6.0 and develops against `../summercms.go`.
- Third-party: `gorm.io/gorm`, `github.com/go-gormigrate/gormigrate/v2`, `github.com/goccy/go-yaml`. Test-only: `github.com/testcontainers/testcontainers-go`, `github.com/jackc/pgx/v5`.
## Testing
```sh
go test -short ./... # goldens and config checks, no Docker
go test ./... # also the Postgres smoke tests
go test -short ./... # unit tests, no Docker
go test ./... # also the Postgres tests
```
`TestSitemapGolden` (`classes`) feeds `testdata/tracer.json`, `urlto.json` and `full.json` through the Go port and compares bytes with the goldens; it also checks the 50000 URL cap. `TestSitemapTiersConfig` checks the tier rule refusals, a host's replacement of the rule list and the manifest's fail-safe reads. `TestRewriteUnsafeFrontendURL` (`classes`), `TestAllowedFrontendURL` and `TestSettingsBeforeValidate` (`models`) and `TestWinterSettingsFrontendURL` (`console`) are the Docker-free checks of the `frontend_url` rules: the accepted form, the rewrite that is not applied for any other value, the save hook and the import refusal. The Postgres smokes (`TestSitemapTracer`, `TestSitemapImportSmoke`, `TestSitemapAdminSmoke`, `TestImportWinter`, `TestSitemapCacheAndThrottle`) boot the plugin with a test content plugin and drive the routes, the import commands and the admin API; they cover the 422 of a refused `frontend_url` save and a value stored straight in the row, which the sitemap does not apply. `TestCommandsDocumented` (root package, Docker-free) logs, without failing, each command whose help lacks a description, long help, examples or an option description (`bonfire.CheckDocs`).
Docker-free: `TestAssignSlots` and `TestRealChanges` (`classes`), `TestAllowedFrontendURL` and `TestSettingsValidate` (`models`), `TestWinterSettingsFrontendURL` (`console`), `TestSiteURLPrecedence`, `TestFileNames`, `TestScheduleAndJobs`, `TestCommandsDocumented` and `TestPluginManifest` (root).
The goldens are generated from the PHP plugin's generation code and the vendored Winter URL generator of any WinterCMS application, and committed, so CI needs no PHP:
```sh
WINTER_VENDOR=/path/to/winter-app/vendor php tests/php/gen-goldens.php classes/testdata/full.json classes/testdata/full.golden
```
The Postgres tests boot the plugin with a fake content plugin (two content types over an in-memory store) and, for the Filter, a fake plugin that publishes one: `TestSitemapIndex` (the index and files after a rebuild job, drafts absent, escaping, 404s), `TestStableNumbering`, `TestLastmod`, `TestDebouncedRebuild`, `TestHreflangAlternates`, `TestFilterExcludes`, `TestNoFilter`, `TestListing`, `TestSettingsSingleton` (the admin API, its refusals and the files partial), `TestMigrateDefinitions`, `TestImportWinter`, `TestImportFile` and `TestRebuildCommand`.