- github.com/jimsmart/grobotstxt v1.0.3 (D-99, decision note in the
framework repo)
- golem15_seo_robots singleton: rules and AI bot switches; eleven bots
incl. Claude-User, all allowed by default
- GET /robots.txt: rules, a Disallow group per switched-off bot and the
golem15.seo.robots_sitemaps lines; outside production the lock with
X-Robots-Tag; never a 5xx (last good body, else allow-all)
- TestRobots on Google's matcher with the deciding line, and the
noindex-blocked check (D-24)
- POST /_seo/api/v1/notfound with the golem15.seo.notfound_token bearer
(constant-time, off when unset), URL and referer caps, a row cap with
an hourly warning, throttle bucket seo-notfound
- golem15.seo.notfound list by hits with Resolved rows, a preview with the
resolved hint, a prompted 301 and a 410 record action, a bulk 410 that
refuses a selection with a covered address
- every /_seo/api/v1 answer carries X-Robots-Tag: noindex
- golem15.seo.redirects list and form (UI-SPEC C8): chain rows marked,
status, match, origin and chain filters, the navigation item lands here
- a field-less extension on the controller refuses a loop with
redirects.loop on target, answers the chain notice and records history
- the status dropdown posts text, read through RedirectStatus
- rule set: exact sources by key, patterns by literal prefix, regex by id
(stdlib RE2, compiled once per source); ignore case, trailing slash,
query ignore/pass, a source with ? matches its query exactly
- targets: captures escaped, a relative target never leaves the site, an
absolute one keeps its host
- SaveRedirect validates, simulates the rule in place (exact, pattern and
regex probes, 10 hops) and refuses a loop; chains answer the notice
- resolve follows the chain to its final target, 410 answers gone, a
runtime cycle answers not_found and is logged once
- golem15_seo_redirects table and Redirect model
- AddressChanged: old -> new 301 in the save transaction, earlier
redirects pointed at the new address, self-loops dropped
- the SEO tab save compares descriptor URLs with the entry URLs and
answers the redirects.auto_created notice
- resolve answers redirect with the absolute target for an exact source
