- golem15_seo_crawls, crawl pages, links, issues and ignores (P33-37)
- classes/crawl: fetchguard client on the site host and its www twin,
sitemap index and robots.txt seeds, two workers with a delay, limits
- h1_missing and h1_multiple with the P33-38 fix route
- POST /_seo/api/v1/crawls queues the golem15.seo.crawl job; GET /issues
answers the stored issues with id, severity, source, fix, found_at,
ignored and crawl_id
- classes/schema builds one @graph per page (WebPage, article, profile,
service, BreadcrumbList, VideoObject, WebSite, publisher) linked by @id
- breadcrumb trail through the primary parent chain, hreflang alternates
with x-default, html_lang and og:locale in the resolve answer
- migration 202610280004 adds the publisher, local business, home crumb
and pagination settings columns
- migration 202610280003: settings links_new_tab (default on),
nofollow_domains and sponsored_domains; robots llms_enabled and
llms_intro (used by the llms.txt commit)
- SEO settings Links tab: the new-tab switch and the two domain lists,
one domain per line, lowercased, refused with settings.domain_invalid
naming the line; writable through seo_update_settings
- the resolve answer of an ok page carries outbound {new_tab,
nofollow_domains, sponsored_domains}; no existing key changes
- golem15_seo_pages (migration 202610280002): the site's own frontend pages
with an address and a name per locale, kind, origin, parent, content date
and note; settings routes_reported_at
- SEO > Pages (golem15.seo.pages) after URL rules: list with kind, origin
and missing filters and the missing row state, form with the P9 partial;
the plugin's form extension saves the address and name per locale and
refuses url_required, url_invalid, url_taken and url_owned
- menu item type seo-page with a content descriptor, so each page gets
the SEO tab, resolves ok and is listed in the sitemaps grouped by kind
(pages, offers, services, landing-pages, titled per locale)
- management API GET/POST /_seo/api/v1/pages, PUT/DELETE /pages/{id} and
the MCP tools seo_list_pages, seo_create_page, seo_update_page,
seo_delete_page (in this commit because the operations parity test maps
every Pages operation to its tool)
The resolver fetched exact rules with lower(source) IN keys, while the keys
drop one trailing slash and keep the query as written, so a source stored
as /old/ or /list?Page=2 was never read. A new migration adds the stored
generated column source_key (the SQL form of exactKey) with an index for
exact rules, and the fetch compares it.
- github.com/jimsmart/grobotstxt v1.0.3 (D-99, decision note in the
framework repo)
- golem15_seo_robots singleton: rules and AI bot switches; eleven bots
incl. Claude-User, all allowed by default
- GET /robots.txt: rules, a Disallow group per switched-off bot and the
golem15.seo.robots_sitemaps lines; outside production the lock with
X-Robots-Tag; never a 5xx (last good body, else allow-all)
- TestRobots on Google's matcher with the deciding line, and the
noindex-blocked check (D-24)
- POST /_seo/api/v1/notfound with the golem15.seo.notfound_token bearer
(constant-time, off when unset), URL and referer caps, a row cap with
an hourly warning, throttle bucket seo-notfound
- golem15.seo.notfound list by hits with Resolved rows, a preview with the
resolved hint, a prompted 301 and a 410 record action, a bulk 410 that
refuses a selection with a covered address
- every /_seo/api/v1 answer carries X-Robots-Tag: noindex
- rule set: exact sources by key, patterns by literal prefix, regex by id
(stdlib RE2, compiled once per source); ignore case, trailing slash,
query ignore/pass, a source with ? matches its query exactly
- targets: captures escaped, a relative target never leaves the site, an
absolute one keeps its host
- SaveRedirect validates, simulates the rule in place (exact, pattern and
regex probes, 10 hops) and refuses a loop; chains answer the notice
- resolve follows the chain to its final target, 410 answers gone, a
runtime cycle answers not_found and is logged once
- golem15_seo_redirects table and Redirect model
- AddressChanged: old -> new 301 in the save transaction, earlier
redirects pointed at the new address, self-loops dropped
- the SEO tab save compares descriptor URLs with the entry URLs and
answers the redirects.auto_created notice
- resolve answers redirect with the absolute target for an exact source
