feat: robots.txt served file, AI bots and the non-production lock
- github.com/jimsmart/grobotstxt v1.0.3 (D-99, decision note in the framework repo) - golem15_seo_robots singleton: rules and AI bot switches; eleven bots incl. Claude-User, all allowed by default - GET /robots.txt: rules, a Disallow group per switched-off bot and the golem15.seo.robots_sitemaps lines; outside production the lock with X-Robots-Tag; never a 5xx (last good body, else allow-all) - TestRobots on Google's matcher with the deciding line, and the noindex-blocked check (D-24)
This commit is contained in:
@@ -19,3 +19,7 @@ notfound_token: ""
|
||||
# notfound_max_rows caps the 404 log; new addresses beyond it are dropped,
|
||||
# with one warning an hour.
|
||||
notfound_max_rows: 10000
|
||||
#
|
||||
# robots_sitemaps lists the sitemap addresses robots.txt names, one
|
||||
# "Sitemap:" line each, made absolute on the site URL.
|
||||
robots_sitemaps: ["/sitemap.xml"]
|
||||
|
||||
Reference in New Issue
Block a user