- noindex_published fires only when a sitemap file lists the address now
(read once per batch through contract.Listing)
- CMS issue sitemap_blocked (severity error, tool seo_update_robots,
details line and rule): a listed address the production robots.txt
keeps Googlebot from
- the SEO tab's P1 shows status.excluded_title with the reason (noindex,
canonical elsewhere, redirect, or the type switched off in the sitemap
settings via ListingSettings.TypesOff) as a plain callout
- robots.txt is composed once per check of many addresses
- GET /_seo/api/v1/site (public, noindex): site_url, default_locale,
locales, unprefixed_locale, links.new_tab, html_sitemap (from the
sitemap plugin's Listing), routes of every content type, root_files
(verification files and the IndexNow key file) and llms
- GET /llms.txt: with llms_enabled (robots.txt screen, AI bots tab, off by
default) and in production, the indexable pages from the sitemaps per
group under the site name and llms_intro (else the default
description); otherwise 404; seo_update_robots takes both fields
- robots.txt names the sitemap plugin's index unless
golem15.seo.robots_sitemaps is set; the config default is now a comment
- migration 202610280003: settings links_new_tab (default on),
nofollow_domains and sponsored_domains; robots llms_enabled and
llms_intro (used by the llms.txt commit)
- SEO settings Links tab: the new-tab switch and the two domain lists,
one domain per line, lowercased, refused with settings.domain_invalid
naming the line; writable through seo_update_settings
- the resolve answer of an ok page carries outbound {new_tab,
nofollow_domains, sponsored_domains}; no existing key changes
- POST /_seo/api/v1/registry/routes (public, bucket seo-notfound): the
frontend server reports its static routes with the bearer token
golem15.seo.route_report_token, else notfound_token (constant-time;
both empty answer 404); at most 2000 paths
- unknown paths become registry pages (origin reported, kind page, a name
from the last segment), known ones get reported_at; paths another
content type owns and invalid ones are skipped; nothing is deleted or
overwritten, and the report's time marks missing pages
- golem15_seo_pages (migration 202610280002): the site's own frontend pages
with an address and a name per locale, kind, origin, parent, content date
and note; settings routes_reported_at
- SEO > Pages (golem15.seo.pages) after URL rules: list with kind, origin
and missing filters and the missing row state, form with the P9 partial;
the plugin's form extension saves the address and name per locale and
refuses url_required, url_invalid, url_taken and url_owned
- menu item type seo-page with a content descriptor, so each page gets
the SEO tab, resolves ok and is listed in the sitemaps grouped by kind
(pages, offers, services, landing-pages, titled per locale)
- management API GET/POST /_seo/api/v1/pages, PUT/DELETE /pages/{id} and
the MCP tools seo_list_pages, seo_create_page, seo_update_page,
seo_delete_page (in this commit because the operations parity test maps
every Pages operation to its tool)
coveredSQL and chainSQL compared the source literally, so /missing/ or
/missing?utm=x stayed open after /missing answered it, and /a -> /b/ was
no chain next to /b. Both now use one SQL form of the runtime's exact
match (prefiltered on the indexed source_key). Pattern and regex rules
remain with the per-row badge, which the list_not_found schema text now
says.
A locale given with a URL write overrode the matched one unchecked, so
"xx" created an entry nothing ever reads while the answer reported
success. SetURLSEO (API, MCP, batch set_seo, records import) now requires
one of the record's locales and refuses others with api.locale_unknown.
The API, MCP, batches and the CSV import copied any value and Normalize
replaced it with the default, answering 201 for a 307 saved as a 301.
RedirectFields.Problems refuses values outside {301, 302, 410}, {exact,
pattern, regex} and {ignore, pass} with value_invalid; the import marks
such rows refused.
(page-1)*perPage overflowed for a large page and sliced out of range.
Export compares the page with the page count before multiplying, and the
list endpoints cap the page number.
- POST /_seo/api/v1/batch runs changes in one transaction: dry_run
answers the diff and the conflicts (loops, chains, noindex on a
published record, canonicals to redirected or noindex addresses) and
rolls back; a refused change refuses the batch (D-12)
- undo of a change or a whole batch from History and through MCP, with
the undo_stale, undo_loop and undo_gone refusals (D-13)
- CSV export and import of redirects, rules and records by URL: BOM,
comma or semicolon, formula cells escaped, a dry-run preview, all or
nothing with via import; the screen's apply token is HMAC-SHA256 over
the file and a fingerprint of the data (transfer.stale)
- Import and export singleton with the seo_transfer widget action,
History record actions undo and undo_batch, navigation entry
- MCP tools undo_change, undo_batch, apply_batch, export_csv, import_csv
- TestBatch, TestUndo, TestCSVTransfer, TestLoopRefused mcp and import
- /_seo/api/v1 routes for the effective SEO of a URL with sources and
owner, URL rules, redirects (loops refused with redirect_loop), the
404 log actions, robots.txt and its tester, settings and history
- the CMS-data issue report: missing and duplicate titles and
descriptions, rules over records, redirect chains, canonicals to
noindex or redirected addresses, noindex addresses robots.txt blocks
- 24 MCP tools on those routes (backend guard, golem15.seo.manage)
- AdminOperations maps every admin operation to its tool;
TestEveryAdminOperationHasTool and TestMCPReadToolsAreGET guard drift
- robots.txt writes go through classes.SaveRobots from the admin too
- PUT /_seo/api/v1/seo writes to the record behind a URL in its locale,
else to an exact URL rule (force_rule writes a rule); the answer says
where (D-11)
- management routes resolve the backend administrator from a token or
the MCP dispatch and require golem15.seo.manage (D-10, D-15)
- MCP tool set_url_seo; history records via mcp
- classes.WithTx, FindOwner, SetURLSEO, SaveRule and DeleteRule
- GET /_seo/api/v1/verification/{name} serves a verification file stored
in the SEO settings with its content type, nosniff and a sandboxing CSP
(D-05, D-103); the name pattern is enforced in the route and the
lookup, and no route is registered at the site root
- github.com/jimsmart/grobotstxt v1.0.3 (D-99, decision note in the
framework repo)
- golem15_seo_robots singleton: rules and AI bot switches; eleven bots
incl. Claude-User, all allowed by default
- GET /robots.txt: rules, a Disallow group per switched-off bot and the
golem15.seo.robots_sitemaps lines; outside production the lock with
X-Robots-Tag; never a 5xx (last good body, else allow-all)
- TestRobots on Google's matcher with the deciding line, and the
noindex-blocked check (D-24)
- Normalize: https, www or bare host, double slashes and index files,
trailing slash add/remove, optional ASCII lowercasing (defaults change
nothing but scheme, slashes and index files)
- strict URLs through signpost MatchPath: a path that names a record
without being its address 301s to the record's address
- resolve runs normalisation, redirects, strict URLs, then SEO; the
location is the final address, so a visitor makes one hop
- P1 warns when a record's canonical URL redirects
- POST /_seo/api/v1/notfound with the golem15.seo.notfound_token bearer
(constant-time, off when unset), URL and referer caps, a row cap with
an hourly warning, throttle bucket seo-notfound
- golem15.seo.notfound list by hits with Resolved rows, a preview with the
resolved hint, a prompted 301 and a 410 record action, a bulk 410 that
refuses a selection with a covered address
- every /_seo/api/v1 answer carries X-Robots-Tag: noindex
