feat: crawler finds a page with two H1 headings and stores the issue (tracer)

- golem15_seo_crawls, crawl pages, links, issues and ignores (P33-37)
- classes/crawl: fetchguard client on the site host and its www twin,
  sitemap index and robots.txt seeds, two workers with a delay, limits
- h1_missing and h1_multiple with the P33-38 fix route
- POST /_seo/api/v1/crawls queues the golem15.seo.crawl job; GET /issues
  answers the stored issues with id, severity, source, fix, found_at,
  ignored and crawl_id
This commit is contained in:
Jakub Zych
2026-11-04 15:05:45 +01:00
parent d38049a956
commit c2b61bf8e9
23 changed files with 2967 additions and 13 deletions

2
go.mod
View File

@@ -12,6 +12,7 @@ require (
github.com/testcontainers/testcontainers-go v0.44.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.44.0
gocloud.dev v0.46.0
golang.org/x/net v0.58.0
gorm.io/driver/postgres v1.6.3
gorm.io/gorm v1.31.2
)
@@ -162,7 +163,6 @@ require (
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/image v0.46.0 // indirect
golang.org/x/net v0.58.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sync v0.23.0 // indirect
golang.org/x/sys v0.48.0 // indirect