Jakub Zych da32044d5b fix: HTML sitemap headings follow each locale until an editor sets one
Settings > Sitemaps no longer prefills a content type's heading with its
label in the administrator's language, so a save that did not touch the
heading stores none and each locale's HTML sitemap keeps the label
translated into it. The field's comment says so.
2026-11-04 23:08:38 +01:00
2026-11-04 12:26:47 +01:00

golem15.sitemap

The compiled sitemap plugin for SummerCMS applications: it serves a sitemap index at /sitemap.xml whose per-type files of a configurable size keep their numbers across rebuilds and carry real lastmod and hreflang alternates, built from every content type's published records, with image and Google News sitemaps, IndexNow notifications and the data of the site's HTML sitemap.

import _ "git.golem15.com/golem15/sm-sitemap-plugin"

Module git.golem15.com/golem15/sm-sitemap-plugin, package sitemap, plugin ID golem15.sitemap.

Overview

The plugin is a compiled SummerCMS plugin. Its init registers it with the framework, so an application loads it by listing it in summer.yaml; summer build then emits the blank import above into the application's plugins.gen.go.

Content plugins describe their records to the framework's registry (signpost content types). The sitemap reads them through signpost.Registry.Records, page by page, and never imports a content plugin: Requires() is empty. Every listed address is a row of the slot table golem15_sitemap_urls, which places it in a file sitemap-<slug>-<n>.xml of its group. A rebuild keeps every address in its file, appends new ones to the last file with room and leaves a hole where an address left; only a changed file size renumbers the files. The routes serve from the table and never generate per request.

Other plugins narrow the sitemap through the contract package: the SEO plugin publishes a contract.Filter that leaves noindex, canonicalised and redirected addresses out, and reads the sitemap through the contract.Listing the plugin publishes. Without a Filter every published address is listed.

The same table feeds the image sitemap (each page's featured and body images), the Google News sitemap (articles of the last two days), IndexNow (each rebuild's added, changed and removed addresses, posted to api.indexnow.org in production) and the HTML sitemap API that the site's HTML sitemap page renders. Every setting can be read and changed through a management API that is also an MCP tool set.

Breaking change from v0.1 (D-32)

v0.1 served one /sitemap.xml in the WinterCMS byte format from a definition of items, with priority, change frequency, priority tiers, an update manifest and the /sitemap.xsl stylesheet. The per-type format replaces it:

  • /sitemap.xml is now a sitemap index; its URL is unchanged, so robots.txt Sitemap: lines and Search Console submissions keep working.
  • The files are /sitemap-<slug>-<n>.xml at the site root. Go serves them at /_sitemap/files/{file}; the site's SEO layer (vue-seo-layer) or its proxy maps the root names there.
  • /sitemap.xsl answers 404, and no <priority> or <changefreq> is written.
  • The definitions screen and the old settings tile are one settings singleton. The url items of the definition move to the additional addresses (models.ConvertLegacyItems): a relative path such as o-nas gets its leading slash, as v0.1 resolved it against the site root, spaces and the characters ", <, > and \ are percent-encoded, and a duplicate is copied once. An item that still cannot be an additional address (an absolute URL with &, another scheme, a malformed item) is not copied and logged at startup as golem15.sitemap: definition url item not copied with the item, its value and the reason; the other items are copied. frontend_url moves to the site address and frontend_default_locale to the default locale (migration 202610280002). Reference items are dropped: the content types list their records. The golem15_sitemap_definitions table and the Phase 17 settings columns stay in the database, unread, so a rollback loses nothing.
  • The config keys golem15.sitemap.cache_seconds and golem15.sitemap.priority_tiers and the festival events BeforeAddItem and AddItem are gone.

Features

  • Index and files. GET /sitemap.xml lists one <sitemap> per file with an address, with the newest lastmod of its addresses. GET /_sitemap/files/{file} serves one file: <loc>, <lastmod> and xhtml:link alternates per address, X-Robots-Tag: noindex. Both are application/xml, anonymous and throttled per client IP.
  • Stable numbering. A file holds at most the file size setting's addresses (default 1000, 1 to 50 000). Addresses keep their file number; new ones are appended oldest first; a deleted, unpublished or excluded address leaves a hole. A changed file size renumbers every file in the next rebuild.
  • Real lastmod. A row's lastmod starts at the record's last change and then moves only when a pact.RecordSaved reports a change outside golem15.sitemap.lastmod_ignore: a no-op save, a counter update or a rebuild leaves it as it was.
  • hreflang. Every address of a record with more than one listed locale carries an alternate per locale, itself included, plus x-default on the default locale's address. A locale without an address gets none, a single-locale record none.
  • Groups. A content type's group is the last segment of its key in plural form (journal-post: posts, journal-category: categories, seo-page: pages), or the file name set in the settings. Additional addresses go to pages. A Filter may move an address to another group (offers).
  • Rebuilds. A conga job (golem15.sitemap.rebuild): queued after record changes, debounced to one per golem15.sitemap.rebuild_delay_seconds; dispatched on every settings save; run every night at 03:00 by the scheduled sitemap:rebuild; and by hand.
  • Image sitemap. sitemap-images-<n>.xml lists every listed page with images: its <loc> and one image:image/image:loc per image (the featured image first, then the body's; at most 1000 per page), numbered stably like the per-type files. On by default.
  • Google News sitemap. sitemap-news.xml lists at most 1000 articles (content kind article) first published in the last two days, newest first, with the publication name, the language, the publication date and the title; narrowed to chosen article types and categories. Off by default.
  • IndexNow. A key generated on save and served as <key>.txt; after each rebuild in production the added, changed and removed addresses go to api.indexnow.org in a queued job, and the last answer shows in the settings. Off by default.
  • HTML sitemap API. GET /_sitemap/api/v1/html and /_sitemap/api/v1/html/{group} answer the HTML sitemap's groups and pages from the same indexable set as the XML sitemaps (D-34); the site's layer renders them.
  • Settings singleton. One tile in the CMS settings category: Files (site address, file size, the files table), Content types (per type: include, file name, HTML sitemap heading), Additional addresses, Images and News, IndexNow and HTML sitemap.
  • Management API and MCP. GET/PUT /_sitemap/api/v1/settings and POST /_sitemap/api/v1/rebuild, and the MCP tools sitemap_get_settings, sitemap_update_settings and sitemap_rebuild (D-10).
  • Contract. contract.Filter for other plugins to implement, contract.Listing for other plugins to read and contract.Rebuilder for other plugins to ask for a rebuild (D-95).

Usage

Mount the plugin in an application as a git submodule at plugins/golem15/sitemap, add it to the application's go.work and go.mod, and list it in summer.yaml:

plugins:
  - id: golem15.sitemap
    module: git.golem15.com/golem15/sm-sitemap-plugin

The plugin's own go.mod and go.work replace the framework with ../summercms.go for a sibling checkout.

The application needs a conga worker (summer queue:work) for the rebuild and IndexNow jobs, and the scheduler for the nightly rebuild. Without a worker, run sitemap:rebuild yourself (IndexNow then waits for a worker).

Routes

Route Answer
GET /sitemap.xml The index: the per-type files, then the image files (while the image sitemap is on), then sitemap-news.xml (while the News sitemap is on and lists an article). 404 while no site address is configured (a <loc> is never built from the request's Host). Before the first rebuild it is an empty <sitemapindex>.
GET /_sitemap/files/{file} One file, {file} matching sitemap-(news|[a-z0-9-]+-[0-9]+)\.xml: a per-type file, an image file sitemap-images-<n>.xml or the News file. 404 for an unknown file, one whose addresses all left, and the image or News file while it is off.
GET /_sitemap/indexnow/{file} The IndexNow key file <key>.txt as text/plain while IndexNow is on; 404 for any other name, an old key or IndexNow off.
GET /_sitemap/api/v1/html?locale= The HTML sitemap overview (JSON, below). 404 while the HTML sitemap is off or for a malformed locale.
GET /_sitemap/api/v1/html/{group}?locale=&page= One page of a group. 404 for an unknown group, a page past the last or a malformed page, and while the HTML sitemap is off.
GET /_sitemap/api/v1/settings, PUT /_sitemap/api/v1/settings, POST /_sitemap/api/v1/rebuild The management API (below).
GET /sitemap.xsl Gone: 404.

The sitemap files and the key file use the surf throttle bucket sitemap-xml (sitemap.BucketSitemap): 60 requests a minute per client IP, read behind the host's trusted proxies. The HTML sitemap API uses sitemap-html (sitemap.BucketHTML, 300 a minute: the site's server calls it for every HTML sitemap page it renders, so it must be a trusted proxy forwarding X-Forwarded-For), the management API sitemap-api (sitemap.BucketAPI, 300 a minute). Every value is written through encoding/xml's escaping; every answer but the index carries X-Robots-Tag: noindex.

The index lists the files at the site root, for example:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://www.example.com/sitemap-posts-1.xml</loc><lastmod>2026-10-28T09:00:00Z</lastmod></sitemap>
</sitemapindex>

A site whose frontend proxies the backend maps /sitemap-<slug>-<n>.xml and /sitemap-news.xml to /_sitemap/files/<name>, and the IndexNow key file /<key>.txt to /_sitemap/indexnow/<key>.txt. vue-seo-layer does it from v0.2.0 (the key file name comes from contract.ListingSettings.KeyFile through the SEO plugin's site answer); with nginx:

location ~ ^/(sitemap-[a-z0-9-]+\.xml)$ {
    proxy_pass http://backend/_sitemap/files/$1;
}
location ~ ^/([A-Za-z0-9-]{8,128}\.txt)$ {
    proxy_pass http://backend/_sitemap/indexnow/$1;
}

Site address

Every <loc> starts with the Site address setting; when it is empty, with the config key golem15.sitemap.site_url; else with app.url. Each must be an absolute http/https URL without whitespace, quotes, <, > or & (models.AllowedFrontendURL); a value outside that form is skipped.

Lastmod

classes.RecordSaved handles every pact.RecordSaved of a model a content type declares:

  • a change of an attribute on the record's own row moves the lastmod of all its addresses to now;
  • a translated change (RecordChange.Locale set) moves only that locale's address;
  • changes of attributes in golem15.sitemap.lastmod_ignore (default updated_at, created_at, views, sort_order) and a save without changes move nothing.

A pact.RecordDeleted takes the record's addresses out of the files at once. Both queue a debounced rebuild. A rebuild sets the lastmod of a new address from the record's LastMod (or the Filter's Decision.LastMod) and never moves an existing one; an additional address carries the lastmod the editor gives it.

Rebuilds

classes.Rebuild walks signpost.Registry.ContentTypes in registration order, pages Records 500 at a time, keeps the published records' locales with an address, asks the Filter per page, adds the additional addresses and writes everything. The whole rebuild runs in one transaction that first takes the advisory lock golem15.sitemap.rebuild and then reads the settings, collects and applies holding the settings row, so a second rebuild collects only after the first has committed and an older snapshot never overwrites a newer one. A rebuild never writes the lastmod of a content row it did not create (only an additional address takes its lastmod from the settings), so a lastmod a record save moved meanwhile stays. It returns the added, changed and removed absolute URLs.

Record events set rebuild_pending_until with a conditional update and enqueue one rebuild with that delay only when no pending time lies in the future; the job clears it when it starts. A settings save dispatches a rebuild at once (a summer_jobs row labelled "Rebuild the sitemaps").

A rebuild also stores each address's page images, the type's kind (pact.ContentDescriptor.Kind), the record's first publication and its parents, which the image and News sitemaps and the HTML sitemap read.

Image sitemap

classes.PageImages takes a locale's featured Image, then its Images in document order: a path on the site or an absolute http/https URL (a //host/... address is read as https), each once, at most classes.MaxImagesPerURL (1000). Other values (data:, javascript:, relative paths) are dropped. The rows with images are numbered into sitemap-images-<n>.xml (image_file_no) with the per-type files' rules and file size: a page that loses its images leaves a hole, a changed file size renumbers. A file has only image:image and image:loc (classes.NSImage), the tags Google still reads. The image sitemap is on by default (images_enabled).

News sitemap

classes.NewsEntries lists the listed addresses of content types whose kind is article, first published (ContentRecord.PublishedAt) within the last 48 hours, newest first, at most 1000 (classes.MaxNewsEntries), as Google requires of a News sitemap. Each entry has the publication name (the news_publication setting, else app.name), the language (the locale's primary subtag, classes.NewsLanguage), the publication date and the title (classes.NSNews).

The News sources setting (news_sources) narrows the list. It takes the values type:<type key> (every article of a type) and category:<type key>:<id> (the articles under that parent, ContentRecord.Parents); empty means every article type. The Images and News tab lists the values the site has (classes.NewsSources: the article types and the parents their articles name), and sitemap_get_settings answers them as news_source_options. Turn the News sitemap on only for a site accepted in Google News.

IndexNow

IndexNow tells Bing, Yandex and the other participating engines about changed addresses at once; Google does not use it.

  • Key. Saving the settings with IndexNow on generates a key of 32 hex characters from crypto/rand (classes.GenerateKey, classes.EnsureKey). The "Generate a new key on save" checkbox (indexnow_new_key, no column) replaces it, and the old key file answers 404 at once. The key file <key>.txt is served at /_sitemap/indexnow/<key>.txt and belongs at the site root, which covers every address of the host, so no keyLocation is sent.
  • Notifications. After a rebuild commits, in the production environment only, with IndexNow on, a key and a site address, classes.NotifyIndexNow queues the conga job golem15.sitemap.indexnow (classes.IndexNowArgs) with the added, changed and removed addresses on the site's host, at most 10 000 per job. A record deleted in the admin (pact.RecordDeleted) is sent as removed and a content edit that keeps the address (pact.RecordSaved moving its lastmod) as changed: the listeners mark the rows (indexnow_pending_at) and the next rebuild sends and clears them; outside production the marks are cleared without a notification. The job (classes.SendIndexNow) posts {host, key, urlList} to https://api.indexnow.org/indexnow through a fetchguard client that may reach only that host (HTTPS, no redirects, 10 s), never an address of another host.
  • Answers. 200 and 202 are success. Every answer is recorded (indexnow_last_at, indexnow_last_count, indexnow_last_status). 400, 403 and 422 are not retried; 429 and 5xx answers and network errors fail the job, which retries up to five times.
  • Status. The IndexNow tab's status partial shows the key and its public address (or that a key is generated on save), the last notification (or that none was sent), and a callout after a 403 (the engines could not read the key file), a 422 (the addresses do not match the site address) or a 429 (with the next attempt's time while a retry is queued).

HTML sitemap API

The site's HTML sitemap page (vue-seo-layer) reads two public JSON routes; both list exactly the listed addresses of the XML sitemaps, never an excluded one (D-24), of the locale asked for and those without a locale (locale empty: the default locale setting).

GET /_sitemap/api/v1/html?locale=pl answers the overview (classes.HTMLOverview):

{"locale": "pl", "page_size": 150, "groups": [
  {"key": "posts", "slug": "posts", "title": "Wpisy", "total": 160, "more": true,
   "entries": [{"title": "Lwia grzywa", "url": "https://www.example.com/blog/lwia-grzywa"}]}
]}
  • Groups come in the order pages, offers, services, categories, posts, tags, authors, then the other content types' groups in registration order, then any other group alphabetically. Empty groups are left out.
  • A group's title in the locale is the content type's HTML sitemap heading, else the Filter's group title, else the type's label.
  • Entries are alphabetical by title, except a group of articles (newest first) and a group of collections with parents, such as categories, which nests children as a tree.
  • Each group lists its first html_page_size entries; more is true when it has more.

GET /_sitemap/api/v1/html/{group}?locale=pl&page=2 answers one page of a group by its slug (classes.HTMLGroupPage): group, key, title, locale, page, pages, total and entries. The settings' HTML sitemap tab sets whether the routes answer (html_enabled, default on), the site's HTML sitemap address per locale (html_paths, default /mapa-strony for pl and /sitemap for en; the layer serves the page there) and the links per page (html_page_size, 50 to 500, default 150).

Management API and MCP

Each management route resolves the backend administrator from a backend JWT, a personal API token covering golem15.sitemap.*, or the MCP dispatch, and requires golem15.sitemap.access_definitions (401 without an administrator, 403 without the permission):

Route MCP tool Effect
GET /_sitemap/api/v1/settings sitemap_get_settings The settings, the content types (label, include, slug, titles), the index address and files with their counts, the addresses left out by reason, the rebuild state, the IndexNow state (key, key URL, last notification, next attempt) and news_source_options.
PUT /_sitemap/api/v1/settings sitemap_update_settings Changes the fields sent (api.UpdateSettingsInput; types by content type key) with the settings screen's checks: a refusal is 422 {"error":"validation_failed","errors":{field:[message]}} and stores nothing. html_paths merges per locale, turning IndexNow on generates its key, indexnow_new_key replaces it, and a rebuild is queued. Answers like GET.
POST /_sitemap/api/v1/rebuild sitemap_rebuild Queues a rebuild now; 202 {"data":{"queued":true}}.

The tools run as the backend account linked to the MCP user and are listed only when it has the permission (sitemap.Plugin.MCP).

Filter

A plugin that decides which addresses are indexable implements contract.Filter and publishes it in its Register. It imports the contract package only:

import "git.golem15.com/golem15/sm-sitemap-plugin/contract"

type noindexFilter struct{}

func (noindexFilter) Decide(ctx context.Context, batch []contract.Candidate) ([]contract.Decision, error) {
	out := make([]contract.Decision, len(batch))
	for i, c := range batch {
		if strings.HasPrefix(c.URL, "/drafts/") {
			out[i] = contract.Decision{Exclude: true, Reason: contract.ReasonNoindex}
		}
	}
	return out, nil
}

func (noindexFilter) Groups(context.Context) ([]contract.Group, error) { return nil, nil }

func (p *Plugin) Register(app *backpack.App) error {
	return app.Publish[contract.Filter](noindexFilter{})
}

Decide gets one page of records of one content type (or the additional addresses, type key extra) and answers one decision per candidate in order. An error or a wrong number of decisions fails the rebuild, and the files stay as they were. Excluded addresses are counted by reason in the settings screen (noindex, canonical, redirect; a type switched off counts as type). An additional address that a content record already lists (the same absolute URL) is left out as duplicate, so the files carry it once, with the record's lastmod and alternates; it is not sent to Decide.

Listing

The plugin publishes classes.Listing as contract.Listing:

  • Settings answers the site address, the index address, the default locale and the groups with their listed addresses, their slug and their titles (the content type's HTML sitemap heading, else the Filter's group titles);
  • Entries pages a group's listed addresses (EntriesQuery{Group, Locale, After, Limit}, 100 by default, at most 1000) with absolute URLs, titles and lastmod, Next and Total.

Settings also answers the HTML sitemap settings (HTMLEnabled, HTMLPaths by locale, HTMLPageSize) and the IndexNow key file name (KeyFile, <key>.txt, empty while IndexNow is off) and the content types switched off (TypesOff, whose addresses are left out with ReasonType), so the SEO plugin and the layer read them without importing the sitemap's internals.

Rebuilder

The plugin also publishes classes.Listing as contract.Rebuilder. RequestRebuild queues one debounced rebuild, the same one a record event queues (a request inside the debounce window joins it), so a plugin whose Filter reads data of its own asks for a rebuild when that data changes; the SEO plugin does it when the frontend reports a changed set of noindex paths. A plugin that finds no contract.Rebuilder (an older sitemap plugin) leaves the change to the next scheduled rebuild.

Admin

The Settings page has one Sitemaps tile (icon network, category CMS, permission golem15.sitemap.access_definitions). It opens the singleton controller golem15.sitemap.settings:

  • Files: Site address, Addresses per file and the files table: the index and every file with its address count and newest lastmod, the addresses left out by reason, and a note naming the public index address. Before the first build it says there are no files yet; after a save and until the rebuild ran, that it is rebuilding.
  • Content types: per signpost content type, in registration order, a switch labelled with the type's label, the file name ([a-z0-9-], unique, images and news reserved) and the HTML sitemap heading per locale. The heading loads empty until an editor sets one, so a save that does not touch it stores none and each language's HTML sitemap shows the type's label translated into that language. Stored in golem15_sitemap_types.
  • Additional addresses: a repeater of paths (or absolute URLs) with an optional lastmod as YYYY-MM-DD. They go into the pages group.
  • Images and News: the image sitemap switch, the News sitemap switch, and while it is on the publication name, the News sources (a tag list of type: and category: values) and the table of the values the site has.
  • IndexNow: the switch, the status partial and, while it is on, "Generate a new key on save".
  • HTML sitemap: the switch, the address per locale (stored as html_paths by the plugin's form extension) and the links per page.

The files table lists the image and News files too. A save refuses an invalid site address, a file size outside 1 to 50 000, an address that is not a path or an http/https URL, a lastmod that is not a date, an invalid or taken file name, a publication name over 255 characters, a News source of another form, an HTML sitemap address that is not a plain path (models.ValidHTMLPath) and links per page outside 50 to 500; it generates the IndexNow key when needed and dispatches a rebuild.

API reference

Identifier Description
sitemap.Plugin Compiled plugin; ID is golem15.sitemap, Requires is empty.
sitemap.Plugin.Register Publishes classes.Listing as contract.Listing.
sitemap.Plugin.Boot Subscribes to pact.RecordSaved and pact.RecordDeleted.
sitemap.Plugin.Routes The sitemap, key file, HTML sitemap and management routes.
sitemap.Plugin.Buckets, sitemap.BucketSitemap, sitemap.BucketHTML, sitemap.BucketAPI The throttle buckets sitemap-xml (60 a minute), sitemap-html and sitemap-api (300 a minute) per client IP.
sitemap.Plugin.Jobs The rebuild job (classes.RebuildKind, three attempts) and the IndexNow job (classes.IndexNowKind, five).
sitemap.Plugin.MCP The tools get_settings, update_settings and rebuild (exposed with the prefix sitemap_).
sitemap.Plugin.Schedule sitemap:rebuild daily at 03:00.
sitemap.Plugin.Commands sitemap:rebuild, sitemap:import, sitemap:import-winter.
sitemap.Plugin.ConfigFS The default config/config.yaml.
sitemap.Plugin.AdminControllers, sitemap.Plugin.Settings, sitemap.Plugin.Permissions The settings singleton, its tile and golem15.sitemap.access_definitions.
sitemap.Plugin.AdminActionExtensions The Content types tab and the HTML sitemap address on the settings singleton.
sitemap.TabFiles, sitemap.TabTypes, sitemap.TabHTML Tab keys of the settings form.
sitemap.Plugin.LangFS, sitemap.Plugin.AdminFS Embedded phrasebooks, admin YAML and partials.
sitemap.Plugin.Migrations, sitemap.Plugin.Models The migrations and models.
contract.Filter, contract.Candidate, contract.Decision, contract.Group The filter other plugins implement.
contract.ReasonNoindex, contract.ReasonCanonical, contract.ReasonRedirect, contract.ReasonType The left-out reasons.
contract.Listing, contract.ListingSettings, contract.GroupInfo, contract.EntriesQuery, contract.Entry, contract.EntriesPage What the sitemap publishes for other plugins.
classes.Rebuild, classes.RebuildResult, classes.WithAfterSlotRead One rebuild of the slot table and what it changed; the test seam called after the rebuild read the slot rows.
classes.RebuildArgs, classes.RebuildKind The rebuild job's arguments and kind golem15.sitemap.rebuild.
classes.RequestRebuild, classes.DispatchRebuild, classes.RequestDebouncedRebuild, classes.ClearPending, classes.Building Dispatching rebuilds and the debounce window.
classes.RecordSaved, classes.RecordDeleted, classes.RealChanges, classes.Changed The record event handling.
classes.LastmodIgnore, classes.RebuildDelay, classes.DefaultLastmodIgnore, classes.LastmodIgnoreConfigKey, classes.RebuildDelayConfigKey The lastmod and debounce config.
classes.AssignSlots, classes.Slot Stable file numbering.
classes.WriteIndex, classes.WriteURLSet, classes.IndexFile, classes.FileEntry, classes.FileAlternate, classes.NSSitemap, classes.NSXHTML The XML writers.
classes.WriteImageURLSet, classes.ImageEntry, classes.NSImage, classes.PageImages, classes.MaxImagesPerURL, classes.ImagesGroup The image sitemap.
classes.WriteNewsURLSet, classes.NewsEntry, classes.NewsEntries, classes.NSNews, classes.NewsFile, classes.NewsWindow, classes.MaxNewsEntries, classes.NewsLanguage, classes.NewsPublication, classes.NewsSources, classes.NewsSource, classes.NewsSourceType, classes.NewsSourceCategory The News sitemap and its sources.
classes.GenerateKey, classes.EnsureKey, classes.ValidKey, classes.KeyFileName, classes.ActiveKey, classes.Production, classes.SiteURLs The IndexNow key and the host filter.
classes.NotifyIndexNow, classes.SendIndexNow, classes.IndexNowArgs, classes.IndexNowKind, classes.IndexNowHost, classes.IndexNowEndpoint, classes.MaxIndexNowURLs, classes.IndexNowAttempts The IndexNow job.
classes.IndexNowStatus, classes.LoadIndexNowStatus, classes.IndexNowRetryAt The IndexNow state of the settings screen and the API.
classes.HTMLSitemap, classes.HTMLOverview, classes.HTMLGroup, classes.HTMLGroupPage, classes.HTMLEntry The HTML sitemap API's groups, order, tree and paging.
classes.MergeHTMLPaths, classes.StoreHTMLPaths The HTML sitemap addresses per locale.
classes.SitemapFiles, classes.SitemapFile, classes.LeftOut The files of the index and the left-out counts.
classes.SaveTypeSettings, classes.TypeFieldBase Writing the content type settings (the form and the API).
classes.FileName, classes.ParseFileName, classes.ValidSlug, classes.DefaultSlug, classes.ReservedSlugs File and group names.
classes.SiteURL, classes.SiteURLConfigKey, classes.Absolute, classes.LoadSettings The site address and the settings row.
classes.TypeConfig, classes.EffectiveType, classes.LoadTypeSettings The per-type settings.
classes.Listing The contract.Listing implementation.
classes.ExtraRecordID The record id of an additional address.
models.URL, models.Alternate, models.Ref A row of the slot table, an hreflang alternate and a parent reference.
models.ExtraTypeKey, models.ExtraGroup, models.ReasonDeleted, models.ReasonUnpublished, models.ReasonDuplicate Additional addresses, the holes' reasons and the reason of an additional address a content record already lists.
models.Settings, models.DefaultSettings, models.ExtraURL, models.DefaultFileSize, models.MaxFileSize The settings row; Validate and Problems are its save checks.
models.DefaultHTMLPageSize, models.MinHTMLPageSize, models.MaxHTMLPageSize, models.DefaultHTMLPaths, models.ValidHTMLPath, models.HTMLPathProblem The HTML sitemap settings and their checks.
models.ValidNewsSource, models.NewsSourceType, models.NewsSourceCategory, models.MaxNewsPublication The News settings' checks.
models.TypeSetting A row of golem15_sitemap_types.
models.AllowedFrontendURL, models.AllowedAddress, models.ParseDate, models.ValidDate The value checks.
models.LegacyItem, models.ParseLegacyItems, models.ParseLegacyItem, models.LegacyEntries, models.LegacyJSON, models.DecodeText, models.Text Reading Phase 17 and WinterCMS definition data.
models.ConvertLegacyItems, models.DroppedItem The url items of a Phase 17 definition as additional addresses, and the items that could not be copied with their reasons.
controllers.SettingsControllerID, controllers.PermissionAccessDefinitions golem15.sitemap.settings and golem15.sitemap.access_definitions.
console.Rebuild, console.Import, console.ImportFile, console.ImportWinter, console.ImportWinterFiles, console.WinterOptions, console.WinterReport The commands.
api.KeyFile, api.KeyFilePattern The IndexNow key file handler (package controllers/api).
api.HTMLOverview, api.HTMLGroup, api.GroupPattern The HTML sitemap API handlers.
api.ShowSettings, api.UpdateSettings, api.Rebuild, api.UpdateSettingsInput, api.TypeInput, api.PermissionManage, api.ProblemsError The management API.

Configuration

config/config.yaml, read as golem15.sitemap.<key>:

Key Default Meaning
site_url unset The site address when the setting is empty (before app.url).
lastmod_ignore [updated_at, created_at, views, sort_order] Attributes whose change does not move lastmod. A host replaces the whole list.
rebuild_delay_seconds 60 The delay of the rebuild a record event queues; events within it share one rebuild.

IndexNow has no config key: the endpoint is fixed (https://api.indexnow.org/indexnow, the only host the plugin's outbound client may reach), the key lives in the settings, and requests are sent only in the production environment. The outbound client's limits fall back to fetchguard's http.fetch.* keys, with a 10 s timeout set by the plugin.

CLI commands

Command Arguments and flags Effect
sitemap:rebuild --queue Rebuilds the files in the process and prints Rebuilt the sitemaps: N addresses in N files (N added, N changed, N removed).; with --queue dispatches the rebuild job instead. Scheduled daily at 03:00.
sitemap:import FILE, --append, --theme Reads a JSON array of {url, lastmod} entries into the additional addresses (replacing them; --append adds and skips listed ones). changefreq and priority are ignored, --theme is accepted and does nothing. One bad entry stops the import.
sitemap:import-winter --definitions FILE, --settings FILE, --theme, --map old=new (repeatable), --page-url reference=/path (repeatable), --force, --dry-run Copies a WinterCMS export: url items become additional addresses (--page-url turns a static or CMS page item into one, --map renames types first), every other item is dropped and listed; frontend_url becomes the site address and frontend_default_locale the default locale, the PHP-parity keys are reported as ignored. Stored addresses are replaced only with --force; --dry-run writes nothing; an unsafe frontend_url stops the import.

The commands open the database themselves when the application has not published one. ./bin/acme help <command> shows each command's long help and examples. The plugin logs through the component sitemap (log.components.sitemap in config/log.yaml).

The WinterCMS export runs on the WinterCMS database with the read-only scripts in scripts/:

mysql -N -B --raw <database> < scripts/winter-export-definitions.sql > sitemap-definitions.jsonl
mysql -N -B --raw <database> < scripts/winter-export-settings.sql > sitemap-settings.json
./bin/acme sitemap:import-winter --definitions sitemap-definitions.jsonl --settings sitemap-settings.json --dry-run
./bin/acme sitemap:import-winter --definitions sitemap-definitions.jsonl --settings sitemap-settings.json
./bin/acme sitemap:rebuild

Dependencies

  • SummerCMS v0.6.1 modules: backpack, bonfire, cabana, compass, conga, diary, fetchguard, lagoon, pact, party, phrasebook, signpost, surf, towel, wire; bouncer, kiosk and pier in tests. Until v0.6.1 is tagged, go.mod requires v0.6.0 and develops against ../summercms.go.
  • Third-party: gorm.io/gorm, github.com/go-gormigrate/gormigrate/v2, github.com/goccy/go-yaml. Test-only: github.com/testcontainers/testcontainers-go, github.com/jackc/pgx/v5.

Testing

go test -short ./...   # unit tests, no Docker
go test ./...          # also the Postgres tests

Docker-free: TestAssignSlots and TestRealChanges (classes), TestAllowedFrontendURL and TestSettingsValidate (models), TestWinterSettingsFrontendURL (console), TestSiteURLPrecedence, TestFileNames, TestScheduleAndJobs, TestCommandsDocumented and TestPluginManifest (root).

The Postgres tests boot the plugin with a fake content plugin (two content types over an in-memory store), a fake blog plugin (an article type and a collection type) and, for the Filter, a fake plugin that publishes one: TestSitemapIndex (the index and files after a rebuild job, drafts absent, escaping, 404s), TestStableNumbering, TestLastmod, TestDebouncedRebuild, TestHreflangAlternates, TestFilterExcludes, TestNoFilter, TestListing, TestSettingsSingleton (the admin API, its refusals and the files partial), TestMigrateDefinitions, TestImportWinter, TestImportFile, TestRebuildCommand, TestImageSitemap, TestNewsSitemap, TestIndexNow (in production, against a fake engine through fetchguard.WithTransport), TestIndexNowStaging, TestHTMLGroups and TestSitemapTools (the management API, an MCP dispatch and pier compiling the tools).

Description
No description provided
Readme 632 KiB
Languages
Go 99.3%
HTML 0.7%