- github.com/chromedp/chromedp v0.21.0 (decision note
2026-10-28-chromedp-rendered-crawl.md): with golem15.seo.crawl_chrome_path
the crawl renders up to golem15.seo.crawl_render_max pages (200) and
compares them with the raw HTML; without a path it records skipped
- guard: site host and www twin resolved and checked with
fetchguard.IsPrivateAddr, pinned through --host-resolver-rules with a
final MAP * ~NOTFOUND; Fetch-domain interception fails every request
outside the site (third-party scripts never load); fresh temporary
profile removed on Close; downloads denied; one tab, 20 s, 5 MiB DOM
- a relative or non-executable path, a private address or a Chrome that
does not start records render_status failed; the crawl still finishes
- config keys documented; README Rendered comparison section
- TestChromeRenderer (runs with a local Chromium) and TestRendererGuards
- golem15_seo_crawls, crawl pages, links, issues and ignores (P33-37)
- classes/crawl: fetchguard client on the site host and its www twin,
sitemap index and robots.txt seeds, two workers with a delay, limits
- h1_missing and h1_multiple with the P33-38 fix route
- POST /_seo/api/v1/crawls queues the golem15.seo.crawl job; GET /issues
answers the stored issues with id, severity, source, fix, found_at,
ignored and crawl_id
- classes/schema builds one @graph per page (WebPage, article, profile,
service, BreadcrumbList, VideoObject, WebSite, publisher) linked by @id
- breadcrumb trail through the primary parent chain, hreflang alternates
with x-default, html_lang and og:locale in the resolve answer
- migration 202610280004 adds the publisher, local business, home crumb
and pagination settings columns
- GET /_seo/api/v1/site (public, noindex): site_url, default_locale,
locales, unprefixed_locale, links.new_tab, html_sitemap (from the
sitemap plugin's Listing), routes of every content type, root_files
(verification files and the IndexNow key file) and llms
- GET /llms.txt: with llms_enabled (robots.txt screen, AI bots tab, off by
default) and in production, the indexable pages from the sitemaps per
group under the site name and llms_intro (else the default
description); otherwise 404; seo_update_robots takes both fields
- robots.txt names the sitemap plugin's index unless
golem15.seo.robots_sitemaps is set; the config default is now a comment
- POST /_seo/api/v1/registry/routes (public, bucket seo-notfound): the
frontend server reports its static routes with the bearer token
golem15.seo.route_report_token, else notfound_token (constant-time;
both empty answer 404); at most 2000 paths
- unknown paths become registry pages (origin reported, kind page, a name
from the last segment), known ones get reported_at; paths another
content type owns and invalid ones are skipped; nothing is deleted or
overwritten, and the report's time marks missing pages
- github.com/jimsmart/grobotstxt v1.0.3 (D-99, decision note in the
framework repo)
- golem15_seo_robots singleton: rules and AI bot switches; eleven bots
incl. Claude-User, all allowed by default
- GET /robots.txt: rules, a Disallow group per switched-off bot and the
golem15.seo.robots_sitemaps lines; outside production the lock with
X-Robots-Tag; never a 5xx (last good body, else allow-all)
- TestRobots on Google's matcher with the deciding line, and the
noindex-blocked check (D-24)
- POST /_seo/api/v1/notfound with the golem15.seo.notfound_token bearer
(constant-time, off when unset), URL and referer caps, a row cap with
an hourly warning, throttle bucket seo-notfound
- golem15.seo.notfound list by hits with Resolved rows, a preview with the
resolved hint, a prompted 301 and a 410 record action, a bulk 410 that
refuses a selection with a covered address
- every /_seo/api/v1 answer carries X-Robots-Tag: noindex
