feat: site audit settings and the nightly crawl

- Site audit tab (UI-SPEC C20): crawl_schedule, crawl_max_pages
  (10-50 000), crawl_delay (100-5000 ms), crawl_keep (1-50) and
  generic_anchors seeded with the en and pl lists; ranges refuse with
  lagoon::validation.between on the screen and through the API
- the crawler reads its limits and generic anchors from the settings;
  crawls beyond crawl_keep are deleted with their pages and issues
- seo:crawl [--scheduled] and the nightly schedule entry at 02:30
  (P33-40): off does nothing, weekly runs on Mondays in app.timezone
- TestCrawlSchedule
This commit is contained in:
Jakub Zych
2026-11-04 15:41:54 +01:00
parent c4f0b4f94c
commit fc53d93b9d
13 changed files with 513 additions and 20 deletions

View File

@@ -411,6 +411,50 @@ fields:
type: text
span: right
crawl_schedule:
label: golem15.seo::lang.settings.crawl_schedule
comment: golem15.seo::lang.settings.crawl_schedule_comment
type: dropdown
span: left
default: "off"
options:
"off": golem15.seo::lang.settings.crawl_schedule_off
daily: golem15.seo::lang.settings.crawl_schedule_daily
weekly: golem15.seo::lang.settings.crawl_schedule_weekly
tab: golem15.seo::lang.settings.tab_audit
crawl_max_pages:
label: golem15.seo::lang.settings.crawl_max_pages
comment: golem15.seo::lang.settings.crawl_max_pages_comment
type: number
span: right
default: 5000
tab: golem15.seo::lang.settings.tab_audit
crawl_delay:
label: golem15.seo::lang.settings.crawl_delay
comment: golem15.seo::lang.settings.crawl_delay_comment
type: number
span: left
default: 250
tab: golem15.seo::lang.settings.tab_audit
crawl_keep:
label: golem15.seo::lang.settings.crawl_keep
comment: golem15.seo::lang.settings.crawl_keep_comment
type: number
span: right
default: 10
tab: golem15.seo::lang.settings.tab_audit
generic_anchors:
label: golem15.seo::lang.settings.generic_anchors
comment: golem15.seo::lang.settings.generic_anchors_comment
type: textarea
size: small
span: full
tab: golem15.seo::lang.settings.tab_audit
verify_google:
label: golem15.seo::lang.settings.verify_google
comment: golem15.seo::lang.settings.verify_code_comment