Issues loads the entries, rules, redirects, robots row and noindex
addresses, then issuesOf builds the report and blockedNoindex checks the
addresses against robots.txt, so the report's rules can be unit-tested
without a database. No behaviour change.
- github.com/jimsmart/grobotstxt v1.0.3 (D-99, decision note in the
framework repo)
- golem15_seo_robots singleton: rules and AI bot switches; eleven bots
incl. Claude-User, all allowed by default
- GET /robots.txt: rules, a Disallow group per switched-off bot and the
golem15.seo.robots_sitemaps lines; outside production the lock with
X-Robots-Tag; never a 5xx (last good body, else allow-all)
- TestRobots on Google's matcher with the deciding line, and the
noindex-blocked check (D-24)
