Jakub Zych
3dd936202f
- github.com/jimsmart/grobotstxt v1.0.3 (D-99, decision note in the framework repo) - golem15_seo_robots singleton: rules and AI bot switches; eleven bots incl. Claude-User, all allowed by default - GET /robots.txt: rules, a Disallow group per switched-off bot and the golem15.seo.robots_sitemaps lines; outside production the lock with X-Robots-Tag; never a 5xx (last good body, else allow-all) - TestRobots on Google's matcher with the deciding line, and the noindex-blocked check (D-24)
26 lines
1.0 KiB
YAML
26 lines
1.0 KiB
YAML
# golem15.seo defaults, read as golem15.seo.<key>.
|
|
#
|
|
# site_url is the public origin of the site (for example
|
|
# https://example.com). Canonical, og:url and every other absolute URL the
|
|
# resolver answers start with it; when it is empty, app.url is used. The
|
|
# request's Host header is never used.
|
|
site_url: ""
|
|
#
|
|
# unprefixed_locale names the locale whose public URLs carry no locale
|
|
# prefix (for example pl on a site whose English pages start with /en).
|
|
# Empty leaves the choice to each content type.
|
|
unprefixed_locale: ""
|
|
#
|
|
# notfound_token is the shared secret the frontend server sends as
|
|
# "Authorization: Bearer <token>" with each 404 report to
|
|
# POST /_seo/api/v1/notfound. Empty turns the endpoint off (it answers 404).
|
|
notfound_token: ""
|
|
#
|
|
# notfound_max_rows caps the 404 log; new addresses beyond it are dropped,
|
|
# with one warning an hour.
|
|
notfound_max_rows: 10000
|
|
#
|
|
# robots_sitemaps lists the sitemap addresses robots.txt names, one
|
|
# "Sitemap:" line each, made absolute on the site URL.
|
|
robots_sitemaps: ["/sitemap.xml"]
|