golem15/sm-seo-plugin
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: robots.txt served file, AI bots and the non-production lock

Browse Source 
- github.com/jimsmart/grobotstxt v1.0.3 (D-99, decision note in the
  framework repo)
- golem15_seo_robots singleton: rules and AI bot switches; eleven bots
  incl. Claude-User, all allowed by default
- GET /robots.txt: rules, a Disallow group per switched-off bot and the
  golem15.seo.robots_sitemaps lines; outside production the lock with
  X-Robots-Tag; never a 5xx (last good body, else allow-all)
- TestRobots on Google's matcher with the deciding line, and the
  noindex-blocked check (D-24)
This commit is contained in:
Jakub Zych
2026-10-27 11:45:44 +01:00
parent 4cb0fbfebb
commit 3dd936202f
11 changed files with 643 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
go.mod
View File

@@ -7,6 +7,7 @@ require (
github.com/go-gormigrate/gormigrate/v2 v2.1.7
github.com/goccy/go-yaml v1.19.2
github.com/jackc/pgx/v5 v5.10.0
github.com/jimsmart/grobotstxt v1.0.3
github.com/testcontainers/testcontainers-go v0.44.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.44.0
gocloud.dev v0.46.0