diff --git a/README.md b/README.md index 112b06c..7cf2e61 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # golem15.sitemap -The compiled sitemap plugin for SummerCMS applications: it serves a sitemap index at `/sitemap.xml` whose per-type files of a configurable size keep their numbers across rebuilds and carry real lastmod and hreflang alternates, built from every content type's published records. +The compiled sitemap plugin for SummerCMS applications: it serves a sitemap index at `/sitemap.xml` whose per-type files of a configurable size keep their numbers across rebuilds and carry real lastmod and hreflang alternates, built from every content type's published records, with image and Google News sitemaps, IndexNow notifications and the data of the site's HTML sitemap. ```go import _ "git.golem15.com/golem15/sm-sitemap-plugin" @@ -16,6 +16,8 @@ Content plugins describe their records to the framework's registry (`signpost` c Other plugins narrow the sitemap through the `contract` package: the SEO plugin publishes a `contract.Filter` that leaves noindex, canonicalised and redirected addresses out, and reads the sitemap through the `contract.Listing` the plugin publishes. Without a Filter every published address is listed. +The same table feeds the image sitemap (each page's featured and body images), the Google News sitemap (articles of the last two days), IndexNow (each rebuild's added, changed and removed addresses, posted to `api.indexnow.org` in production) and the HTML sitemap API that the site's HTML sitemap page renders. Every setting can be read and changed through a management API that is also an MCP tool set. + ### Breaking change from v0.1 (D-32) v0.1 served one `/sitemap.xml` in the WinterCMS byte format from a definition of items, with priority, change frequency, priority tiers, an update manifest and the `/sitemap.xsl` stylesheet. The per-type format replaces it: @@ -34,7 +36,12 @@ v0.1 served one `/sitemap.xml` in the WinterCMS byte format from a definition of - **hreflang.** Every address of a record with more than one listed locale carries an alternate per locale, itself included, plus `x-default` on the default locale's address. A locale without an address gets none, a single-locale record none. - **Groups.** A content type's group is the last segment of its key in plural form (`journal-post`: `posts`, `journal-category`: `categories`, `seo-page`: `pages`), or the file name set in the settings. Additional addresses go to `pages`. A Filter may move an address to another group (`offers`). - **Rebuilds.** A conga job (`golem15.sitemap.rebuild`): queued after record changes, debounced to one per `golem15.sitemap.rebuild_delay_seconds`; dispatched on every settings save; run every night at 03:00 by the scheduled `sitemap:rebuild`; and by hand. -- **Settings singleton.** One tile in the CMS settings category: Files (site address, file size, the files table), Content types (per type: include, file name, HTML sitemap heading) and Additional addresses. +- **Image sitemap.** `sitemap-images-.xml` lists every listed page with images: its `` and one `image:image`/`image:loc` per image (the featured image first, then the body's; at most 1000 per page), numbered stably like the per-type files. On by default. +- **Google News sitemap.** `sitemap-news.xml` lists at most 1000 articles (content kind `article`) first published in the last two days, newest first, with the publication name, the language, the publication date and the title; narrowed to chosen article types and categories. Off by default. +- **IndexNow.** A key generated on save and served as `.txt`; after each rebuild in production the added, changed and removed addresses go to `api.indexnow.org` in a queued job, and the last answer shows in the settings. Off by default. +- **HTML sitemap API.** `GET /_sitemap/api/v1/html` and `/_sitemap/api/v1/html/{group}` answer the HTML sitemap's groups and pages from the same indexable set as the XML sitemaps (D-34); the site's layer renders them. +- **Settings singleton.** One tile in the CMS settings category: Files (site address, file size, the files table), Content types (per type: include, file name, HTML sitemap heading), Additional addresses, Images and News, IndexNow and HTML sitemap. +- **Management API and MCP.** `GET`/`PUT /_sitemap/api/v1/settings` and `POST /_sitemap/api/v1/rebuild`, and the MCP tools `sitemap_get_settings`, `sitemap_update_settings` and `sitemap_rebuild` (D-10). - **Contract.** `contract.Filter` for other plugins to implement and `contract.Listing` for other plugins to read (D-95). ## Usage @@ -49,17 +56,21 @@ plugins: The plugin's own `go.mod` and `go.work` replace the framework with `../summercms.go` for a sibling checkout. -The application needs a conga worker (`summer queue:work`) for the rebuild job, and the scheduler for the nightly rebuild. Without a worker, run `sitemap:rebuild` yourself. +The application needs a conga worker (`summer queue:work`) for the rebuild and IndexNow jobs, and the scheduler for the nightly rebuild. Without a worker, run `sitemap:rebuild` yourself (IndexNow then waits for a worker). ### Routes | Route | Answer | |-------|--------| -| `GET /sitemap.xml` | The index. `404` while no site address is configured (a `` is never built from the request's Host). Before the first rebuild it is an empty ``. | -| `GET /_sitemap/files/{file}` | One file, `{file}` matching `sitemap-[a-z0-9-]+-[0-9]+\.xml`. `404` for an unknown file or one whose addresses all left. | +| `GET /sitemap.xml` | The index: the per-type files, then the image files (while the image sitemap is on), then `sitemap-news.xml` (while the News sitemap is on and lists an article). `404` while no site address is configured (a `` is never built from the request's Host). Before the first rebuild it is an empty ``. | +| `GET /_sitemap/files/{file}` | One file, `{file}` matching `sitemap-(news\|[a-z0-9-]+-[0-9]+)\.xml`: a per-type file, an image file `sitemap-images-.xml` or the News file. `404` for an unknown file, one whose addresses all left, and the image or News file while it is off. | +| `GET /_sitemap/indexnow/{file}` | The IndexNow key file `.txt` as `text/plain` while IndexNow is on; `404` for any other name, an old key or IndexNow off. | +| `GET /_sitemap/api/v1/html?locale=` | The HTML sitemap overview (JSON, below). `404` while the HTML sitemap is off or for a malformed locale. | +| `GET /_sitemap/api/v1/html/{group}?locale=&page=` | One page of a group. `404` for an unknown group, a page past the last or a malformed page, and while the HTML sitemap is off. | +| `GET /_sitemap/api/v1/settings`, `PUT /_sitemap/api/v1/settings`, `POST /_sitemap/api/v1/rebuild` | The management API (below). | | `GET /sitemap.xsl` | Gone: `404`. | -Both routes use the surf throttle bucket `sitemap-xml` (`sitemap.BucketSitemap`): 60 requests a minute per client IP, read behind the host's trusted proxies. Every value is written through `encoding/xml`'s escaping. +The sitemap files and the key file use the surf throttle bucket `sitemap-xml` (`sitemap.BucketSitemap`): 60 requests a minute per client IP, read behind the host's trusted proxies. The HTML sitemap API uses `sitemap-html` (`sitemap.BucketHTML`, 300 a minute: the site's server calls it for every HTML sitemap page it renders, so it must be a trusted proxy forwarding `X-Forwarded-For`), the management API `sitemap-api` (`sitemap.BucketAPI`, 300 a minute). Every value is written through `encoding/xml`'s escaping; every answer but the index carries `X-Robots-Tag: noindex`. The index lists the files at the site root, for example: @@ -70,12 +81,15 @@ The index lists the files at the site root, for example: ``` -A site whose frontend proxies the backend maps `/sitemap--.xml` to `/_sitemap/files/sitemap--.xml`. vue-seo-layer does it from v0.2.0; with nginx: +A site whose frontend proxies the backend maps `/sitemap--.xml` and `/sitemap-news.xml` to `/_sitemap/files/`, and the IndexNow key file `/.txt` to `/_sitemap/indexnow/.txt`. vue-seo-layer does it from v0.2.0 (the key file name comes from `contract.ListingSettings.KeyFile` through the SEO plugin's site answer); with nginx: ```nginx -location ~ ^/(sitemap-[a-z0-9-]+-[0-9]+\.xml)$ { +location ~ ^/(sitemap-[a-z0-9-]+\.xml)$ { proxy_pass http://backend/_sitemap/files/$1; } +location ~ ^/([A-Za-z0-9-]{8,128}\.txt)$ { + proxy_pass http://backend/_sitemap/indexnow/$1; +} ``` ### Site address @@ -98,6 +112,59 @@ A `pact.RecordDeleted` takes the record's addresses out of the files at once. Bo Record events set `rebuild_pending_until` with a conditional update and enqueue one rebuild with that delay only when no pending time lies in the future; the job clears it when it starts. A settings save dispatches a rebuild at once (a `summer_jobs` row labelled "Rebuild the sitemaps"). +A rebuild also stores each address's page images, the type's kind (`pact.ContentDescriptor.Kind`), the record's first publication and its parents, which the image and News sitemaps and the HTML sitemap read. + +### Image sitemap + +`classes.PageImages` takes a locale's featured `Image`, then its `Images` in document order: a path on the site or an absolute `http`/`https` URL (a `//host/...` address is read as https), each once, at most `classes.MaxImagesPerURL` (1000). Other values (`data:`, `javascript:`, relative paths) are dropped. The rows with images are numbered into `sitemap-images-.xml` (`image_file_no`) with the per-type files' rules and file size: a page that loses its images leaves a hole, a changed file size renumbers. A file has only `image:image` and `image:loc` (`classes.NSImage`), the tags Google still reads. The image sitemap is on by default (`images_enabled`). + +### News sitemap + +`classes.NewsEntries` lists the listed addresses of content types whose kind is `article`, first published (`ContentRecord.PublishedAt`) within the last 48 hours, newest first, at most 1000 (`classes.MaxNewsEntries`), as Google requires of a News sitemap. Each entry has the publication name (the `news_publication` setting, else `app.name`), the language (the locale's primary subtag, `classes.NewsLanguage`), the publication date and the title (`classes.NSNews`). + +The News sources setting (`news_sources`) narrows the list. It takes the values `type:` (every article of a type) and `category::` (the articles under that parent, `ContentRecord.Parents`); empty means every article type. The Images and News tab lists the values the site has (`classes.NewsSources`: the article types and the parents their articles name), and `sitemap_get_settings` answers them as `news_source_options`. Turn the News sitemap on only for a site accepted in Google News. + +### IndexNow + +[IndexNow](https://www.indexnow.org/documentation) tells Bing, Yandex and the other participating engines about changed addresses at once; Google does not use it. + +- **Key.** Saving the settings with IndexNow on generates a key of 32 hex characters from `crypto/rand` (`classes.GenerateKey`, `classes.EnsureKey`). The "Generate a new key on save" checkbox (`indexnow_new_key`, no column) replaces it, and the old key file answers 404 at once. The key file `.txt` is served at `/_sitemap/indexnow/.txt` and belongs at the site root, which covers every address of the host, so no `keyLocation` is sent. +- **Notifications.** After a rebuild commits, in the `production` environment only, with IndexNow on, a key and a site address, `classes.NotifyIndexNow` queues the conga job `golem15.sitemap.indexnow` (`classes.IndexNowArgs`) with the added, changed and removed addresses on the site's host, at most 10 000 per job. The job (`classes.SendIndexNow`) posts `{host, key, urlList}` to `https://api.indexnow.org/indexnow` through a `fetchguard` client that may reach only that host (HTTPS, no redirects, 10 s), never an address of another host. +- **Answers.** 200 and 202 are success. Every answer is recorded (`indexnow_last_at`, `indexnow_last_count`, `indexnow_last_status`). 400, 403 and 422 are not retried; 429 and 5xx answers and network errors fail the job, which retries up to five times. +- **Status.** The IndexNow tab's status partial shows the key and its public address (or that a key is generated on save), the last notification (or that none was sent), and a callout after a 403 (the engines could not read the key file), a 422 (the addresses do not match the site address) or a 429 (with the next attempt's time while a retry is queued). + +### HTML sitemap API + +The site's HTML sitemap page (vue-seo-layer) reads two public JSON routes; both list exactly the listed addresses of the XML sitemaps, never an excluded one (D-24), of the locale asked for and those without a locale (`locale` empty: the default locale setting). + +`GET /_sitemap/api/v1/html?locale=pl` answers the overview (`classes.HTMLOverview`): + +```json +{"locale": "pl", "page_size": 150, "groups": [ + {"key": "posts", "slug": "posts", "title": "Wpisy", "total": 160, "more": true, + "entries": [{"title": "Lwia grzywa", "url": "https://www.example.com/blog/lwia-grzywa"}]} +]} +``` + +- Groups come in the order `pages`, `offers`, `services`, `categories`, `posts`, `tags`, `authors`, then the other content types' groups in registration order, then any other group alphabetically. Empty groups are left out. +- A group's title in the locale is the content type's HTML sitemap heading, else the Filter's group title, else the type's label. +- Entries are alphabetical by title, except a group of articles (newest first) and a group of collections with parents, such as categories, which nests `children` as a tree. +- Each group lists its first `html_page_size` entries; `more` is true when it has more. + +`GET /_sitemap/api/v1/html/{group}?locale=pl&page=2` answers one page of a group by its slug (`classes.HTMLGroupPage`): `group`, `key`, `title`, `locale`, `page`, `pages`, `total` and `entries`. The settings' HTML sitemap tab sets whether the routes answer (`html_enabled`, default on), the site's HTML sitemap address per locale (`html_paths`, default `/mapa-strony` for pl and `/sitemap` for en; the layer serves the page there) and the links per page (`html_page_size`, 50 to 500, default 150). + +### Management API and MCP + +Each management route resolves the backend administrator from a backend JWT, a personal API token covering `golem15.sitemap.*`, or the MCP dispatch, and requires `golem15.sitemap.access_definitions` (401 without an administrator, 403 without the permission): + +| Route | MCP tool | Effect | +|-------|----------|--------| +| `GET /_sitemap/api/v1/settings` | `sitemap_get_settings` | The settings, the content types (label, include, slug, titles), the index address and files with their counts, the addresses left out by reason, the rebuild state, the IndexNow state (key, key URL, last notification, next attempt) and `news_source_options`. | +| `PUT /_sitemap/api/v1/settings` | `sitemap_update_settings` | Changes the fields sent (`api.UpdateSettingsInput`; `types` by content type key) with the settings screen's checks: a refusal is 422 `{"error":"validation_failed","errors":{field:[message]}}` and stores nothing. `html_paths` merges per locale, turning IndexNow on generates its key, `indexnow_new_key` replaces it, and a rebuild is queued. Answers like GET. | +| `POST /_sitemap/api/v1/rebuild` | `sitemap_rebuild` | Queues a rebuild now; 202 `{"data":{"queued":true}}`. | + +The tools run as the backend account linked to the MCP user and are listed only when it has the permission (`sitemap.Plugin.MCP`). + ### Filter A plugin that decides which addresses are indexable implements `contract.Filter` and publishes it in its `Register`. It imports the `contract` package only: @@ -133,7 +200,7 @@ The plugin publishes `classes.Listing` as `contract.Listing`: - `Settings` answers the site address, the index address, the default locale and the groups with their listed addresses, their slug and their titles (the content type's HTML sitemap heading, else the Filter's group titles); - `Entries` pages a group's listed addresses (`EntriesQuery{Group, Locale, After, Limit}`, 100 by default, at most 1000) with absolute URLs, titles and lastmod, `Next` and `Total`. -The HTML sitemap and IndexNow fields of `contract.ListingSettings` stay zero until those settings exist. +`Settings` also answers the HTML sitemap settings (`HTMLEnabled`, `HTMLPaths` by locale, `HTMLPageSize`) and the IndexNow key file name (`KeyFile`, `.txt`, empty while IndexNow is off), so the SEO plugin and the layer read them without importing the sitemap's internals. ### Admin @@ -142,8 +209,11 @@ The Settings page has one Sitemaps tile (icon `network`, category CMS, permissio - **Files:** Site address, Addresses per file and the files table: the index and every file with its address count and newest lastmod, the addresses left out by reason, and a note naming the public index address. Before the first build it says there are no files yet; after a save and until the rebuild ran, that it is rebuilding. - **Content types:** per signpost content type, in registration order, a switch labelled with the type's label, the file name (`[a-z0-9-]`, unique, `images` and `news` reserved) and the HTML sitemap heading per locale. Stored in `golem15_sitemap_types`. - **Additional addresses:** a repeater of paths (or absolute URLs) with an optional lastmod as `YYYY-MM-DD`. They go into the `pages` group. +- **Images and News:** the image sitemap switch, the News sitemap switch, and while it is on the publication name, the News sources (a tag list of `type:` and `category:` values) and the table of the values the site has. +- **IndexNow:** the switch, the status partial and, while it is on, "Generate a new key on save". +- **HTML sitemap:** the switch, the address per locale (stored as `html_paths` by the plugin's form extension) and the links per page. -A save refuses an invalid site address, a file size outside 1 to 50 000, an address that is not a path or an `http`/`https` URL, a lastmod that is not a date and an invalid or taken file name, and dispatches a rebuild. +The files table lists the image and News files too. A save refuses an invalid site address, a file size outside 1 to 50 000, an address that is not a path or an `http`/`https` URL, a lastmod that is not a date, an invalid or taken file name, a publication name over 255 characters, a News source of another form, an HTML sitemap address that is not a plain path (`models.ValidHTMLPath`) and links per page outside 50 to 500; it generates the IndexNow key when needed and dispatches a rebuild. ## API reference @@ -152,15 +222,16 @@ A save refuses an invalid site address, a file size outside 1 to 50 000, an addr | `sitemap.Plugin` | Compiled plugin; `ID` is `golem15.sitemap`, `Requires` is empty. | | `sitemap.Plugin.Register` | Publishes `classes.Listing` as `contract.Listing`. | | `sitemap.Plugin.Boot` | Subscribes to `pact.RecordSaved` and `pact.RecordDeleted`. | -| `sitemap.Plugin.Routes` | `GET /sitemap.xml` and `GET /_sitemap/files/{file}`. | -| `sitemap.Plugin.Buckets`, `sitemap.BucketSitemap` | The `sitemap-xml` throttle bucket: 60 requests a minute per client IP. | -| `sitemap.Plugin.Jobs` | The rebuild job (`classes.RebuildKind`), three attempts. | +| `sitemap.Plugin.Routes` | The sitemap, key file, HTML sitemap and management routes. | +| `sitemap.Plugin.Buckets`, `sitemap.BucketSitemap`, `sitemap.BucketHTML`, `sitemap.BucketAPI` | The throttle buckets `sitemap-xml` (60 a minute), `sitemap-html` and `sitemap-api` (300 a minute) per client IP. | +| `sitemap.Plugin.Jobs` | The rebuild job (`classes.RebuildKind`, three attempts) and the IndexNow job (`classes.IndexNowKind`, five). | +| `sitemap.Plugin.MCP` | The tools `get_settings`, `update_settings` and `rebuild` (exposed with the prefix `sitemap_`). | | `sitemap.Plugin.Schedule` | `sitemap:rebuild` daily at 03:00. | | `sitemap.Plugin.Commands` | `sitemap:rebuild`, `sitemap:import`, `sitemap:import-winter`. | | `sitemap.Plugin.ConfigFS` | The default `config/config.yaml`. | | `sitemap.Plugin.AdminControllers`, `sitemap.Plugin.Settings`, `sitemap.Plugin.Permissions` | The settings singleton, its tile and `golem15.sitemap.access_definitions`. | -| `sitemap.Plugin.AdminActionExtensions` | The Content types tab on the settings singleton. | -| `sitemap.TabFiles`, `sitemap.TabTypes` | The tab keys of the settings form. | +| `sitemap.Plugin.AdminActionExtensions` | The Content types tab and the HTML sitemap address on the settings singleton. | +| `sitemap.TabFiles`, `sitemap.TabTypes`, `sitemap.TabHTML` | Tab keys of the settings form. | | `sitemap.Plugin.LangFS`, `sitemap.Plugin.AdminFS` | Embedded phrasebooks, admin YAML and partials. | | `sitemap.Plugin.Migrations`, `sitemap.Plugin.Models` | The migrations and models. | | `contract.Filter`, `contract.Candidate`, `contract.Decision`, `contract.Group` | The filter other plugins implement. | @@ -173,19 +244,33 @@ A save refuses an invalid site address, a file size outside 1 to 50 000, an addr | `classes.LastmodIgnore`, `classes.RebuildDelay`, `classes.DefaultLastmodIgnore`, `classes.LastmodIgnoreConfigKey`, `classes.RebuildDelayConfigKey` | The lastmod and debounce config. | | `classes.AssignSlots`, `classes.Slot` | Stable file numbering. | | `classes.WriteIndex`, `classes.WriteURLSet`, `classes.IndexFile`, `classes.FileEntry`, `classes.FileAlternate`, `classes.NSSitemap`, `classes.NSXHTML` | The XML writers. | +| `classes.WriteImageURLSet`, `classes.ImageEntry`, `classes.NSImage`, `classes.PageImages`, `classes.MaxImagesPerURL`, `classes.ImagesGroup` | The image sitemap. | +| `classes.WriteNewsURLSet`, `classes.NewsEntry`, `classes.NewsEntries`, `classes.NSNews`, `classes.NewsFile`, `classes.NewsWindow`, `classes.MaxNewsEntries`, `classes.NewsLanguage`, `classes.NewsPublication`, `classes.NewsSources`, `classes.NewsSource`, `classes.NewsSourceType`, `classes.NewsSourceCategory` | The News sitemap and its sources. | +| `classes.GenerateKey`, `classes.EnsureKey`, `classes.ValidKey`, `classes.KeyFileName`, `classes.ActiveKey`, `classes.Production`, `classes.SiteURLs` | The IndexNow key and the host filter. | +| `classes.NotifyIndexNow`, `classes.SendIndexNow`, `classes.IndexNowArgs`, `classes.IndexNowKind`, `classes.IndexNowHost`, `classes.IndexNowEndpoint`, `classes.MaxIndexNowURLs`, `classes.IndexNowAttempts` | The IndexNow job. | +| `classes.IndexNowStatus`, `classes.LoadIndexNowStatus`, `classes.IndexNowRetryAt` | The IndexNow state of the settings screen and the API. | +| `classes.HTMLSitemap`, `classes.HTMLOverview`, `classes.HTMLGroup`, `classes.HTMLGroupPage`, `classes.HTMLEntry` | The HTML sitemap API's groups, order, tree and paging. | +| `classes.MergeHTMLPaths`, `classes.StoreHTMLPaths` | The HTML sitemap addresses per locale. | +| `classes.SitemapFiles`, `classes.SitemapFile`, `classes.LeftOut` | The files of the index and the left-out counts. | +| `classes.SaveTypeSettings`, `classes.TypeFieldBase` | Writing the content type settings (the form and the API). | | `classes.FileName`, `classes.ParseFileName`, `classes.ValidSlug`, `classes.DefaultSlug`, `classes.ReservedSlugs` | File and group names. | | `classes.SiteURL`, `classes.SiteURLConfigKey`, `classes.Absolute`, `classes.LoadSettings` | The site address and the settings row. | | `classes.TypeConfig`, `classes.EffectiveType`, `classes.LoadTypeSettings` | The per-type settings. | | `classes.Listing` | The `contract.Listing` implementation. | | `classes.ExtraRecordID` | The record id of an additional address. | -| `models.URL`, `models.Alternate` | A row of the slot table and an hreflang alternate. | +| `models.URL`, `models.Alternate`, `models.Ref` | A row of the slot table, an hreflang alternate and a parent reference. | | `models.ExtraTypeKey`, `models.ExtraGroup`, `models.ReasonDeleted`, `models.ReasonUnpublished` | Additional addresses and the holes' reasons. | | `models.Settings`, `models.DefaultSettings`, `models.ExtraURL`, `models.DefaultFileSize`, `models.MaxFileSize` | The settings row; `Validate` and `Problems` are its save checks. | +| `models.DefaultHTMLPageSize`, `models.MinHTMLPageSize`, `models.MaxHTMLPageSize`, `models.DefaultHTMLPaths`, `models.ValidHTMLPath`, `models.HTMLPathProblem` | The HTML sitemap settings and their checks. | +| `models.ValidNewsSource`, `models.NewsSourceType`, `models.NewsSourceCategory`, `models.MaxNewsPublication` | The News settings' checks. | | `models.TypeSetting` | A row of `golem15_sitemap_types`. | | `models.AllowedFrontendURL`, `models.AllowedAddress`, `models.ParseDate`, `models.ValidDate` | The value checks. | | `models.LegacyItem`, `models.ParseLegacyItems`, `models.ParseLegacyItem`, `models.LegacyEntries`, `models.LegacyJSON`, `models.DecodeText`, `models.Text` | Reading Phase 17 and WinterCMS definition data. | | `controllers.SettingsControllerID`, `controllers.PermissionAccessDefinitions` | `golem15.sitemap.settings` and `golem15.sitemap.access_definitions`. | | `console.Rebuild`, `console.Import`, `console.ImportFile`, `console.ImportWinter`, `console.ImportWinterFiles`, `console.WinterOptions`, `console.WinterReport` | The commands. | +| `api.KeyFile`, `api.KeyFilePattern` | The IndexNow key file handler (package `controllers/api`). | +| `api.HTMLOverview`, `api.HTMLGroup`, `api.GroupPattern` | The HTML sitemap API handlers. | +| `api.ShowSettings`, `api.UpdateSettings`, `api.Rebuild`, `api.UpdateSettingsInput`, `api.TypeInput`, `api.PermissionManage`, `api.ProblemsError` | The management API. | ## Configuration @@ -197,6 +282,8 @@ A save refuses an invalid site address, a file size outside 1 to 50 000, an addr | `lastmod_ignore` | `[updated_at, created_at, views, sort_order]` | Attributes whose change does not move lastmod. A host replaces the whole list. | | `rebuild_delay_seconds` | `60` | The delay of the rebuild a record event queues; events within it share one rebuild. | +IndexNow has no config key: the endpoint is fixed (`https://api.indexnow.org/indexnow`, the only host the plugin's outbound client may reach), the key lives in the settings, and requests are sent only in the `production` environment. The outbound client's limits fall back to `fetchguard`'s `http.fetch.*` keys, with a 10 s timeout set by the plugin. + ## CLI commands | Command | Arguments and flags | Effect | @@ -219,7 +306,7 @@ mysql -N -B --raw < scripts/winter-export-settings.sql > sitemap-sett ## Dependencies -- SummerCMS v0.6.1 modules: backpack, bonfire, cabana, compass, conga, diary, lagoon, pact, party, phrasebook, signpost, surf; bouncer and kiosk in tests. Until v0.6.1 is tagged, `go.mod` requires v0.6.0 and develops against `../summercms.go`. +- SummerCMS v0.6.1 modules: backpack, bonfire, cabana, compass, conga, diary, fetchguard, lagoon, pact, party, phrasebook, signpost, surf, towel, wire; bouncer, kiosk and pier in tests. Until v0.6.1 is tagged, `go.mod` requires v0.6.0 and develops against `../summercms.go`. - Third-party: `gorm.io/gorm`, `github.com/go-gormigrate/gormigrate/v2`, `github.com/goccy/go-yaml`. Test-only: `github.com/testcontainers/testcontainers-go`, `github.com/jackc/pgx/v5`. ## Testing @@ -231,4 +318,4 @@ go test ./... # also the Postgres tests Docker-free: `TestAssignSlots` and `TestRealChanges` (`classes`), `TestAllowedFrontendURL` and `TestSettingsValidate` (`models`), `TestWinterSettingsFrontendURL` (`console`), `TestSiteURLPrecedence`, `TestFileNames`, `TestScheduleAndJobs`, `TestCommandsDocumented` and `TestPluginManifest` (root). -The Postgres tests boot the plugin with a fake content plugin (two content types over an in-memory store) and, for the Filter, a fake plugin that publishes one: `TestSitemapIndex` (the index and files after a rebuild job, drafts absent, escaping, 404s), `TestStableNumbering`, `TestLastmod`, `TestDebouncedRebuild`, `TestHreflangAlternates`, `TestFilterExcludes`, `TestNoFilter`, `TestListing`, `TestSettingsSingleton` (the admin API, its refusals and the files partial), `TestMigrateDefinitions`, `TestImportWinter`, `TestImportFile` and `TestRebuildCommand`. +The Postgres tests boot the plugin with a fake content plugin (two content types over an in-memory store), a fake blog plugin (an article type and a collection type) and, for the Filter, a fake plugin that publishes one: `TestSitemapIndex` (the index and files after a rebuild job, drafts absent, escaping, 404s), `TestStableNumbering`, `TestLastmod`, `TestDebouncedRebuild`, `TestHreflangAlternates`, `TestFilterExcludes`, `TestNoFilter`, `TestListing`, `TestSettingsSingleton` (the admin API, its refusals and the files partial), `TestMigrateDefinitions`, `TestImportWinter`, `TestImportFile`, `TestRebuildCommand`, `TestImageSitemap`, `TestNewsSitemap`, `TestIndexNow` (in production, against a fake engine through `fetchguard.WithTransport`), `TestIndexNowStaging`, `TestHTMLGroups` and `TestSitemapTools` (the management API, an MCP dispatch and pier compiling the tools). diff --git a/plugin.yaml b/plugin.yaml index 7c77f06..fc6ede1 100644 --- a/plugin.yaml +++ b/plugin.yaml @@ -4,7 +4,7 @@ id: golem15.sitemap module: git.golem15.com/golem15/sm-sitemap-plugin package: git.golem15.com/golem15/sm-sitemap-plugin name: Sitemap -description: "The compiled sitemap plugin for SummerCMS applications: it serves a sitemap index at /sitemap.xml whose per-type files of a configurable size keep their numbers across rebuilds and carry real lastmod and hreflang alternates, built from every content type's published records." +description: "The compiled sitemap plugin for SummerCMS applications: it serves a sitemap index at /sitemap.xml whose per-type files of a configurable size keep their numbers across rebuilds and carry real lastmod and hreflang alternates, built from every content type's published records, with image and Google News sitemaps, IndexNow notifications and the data of the site's HTML sitemap." framework: v0.6.0 overrides: name: Sitemap