feat: site audit settings and the nightly crawl

- Site audit tab (UI-SPEC C20): crawl_schedule, crawl_max_pages
  (10-50 000), crawl_delay (100-5000 ms), crawl_keep (1-50) and
  generic_anchors seeded with the en and pl lists; ranges refuse with
  lagoon::validation.between on the screen and through the API
- the crawler reads its limits and generic anchors from the settings;
  crawls beyond crawl_keep are deleted with their pages and issues
- seo:crawl [--scheduled] and the nightly schedule entry at 02:30
  (P33-40): off does nothing, weekly runs on Mondays in app.timezone
- TestCrawlSchedule
This commit is contained in:
Jakub Zych
2026-11-04 15:41:54 +01:00
parent c4f0b4f94c
commit fc53d93b9d
13 changed files with 513 additions and 20 deletions

View File

@@ -83,6 +83,20 @@ status:
staging: On this environment every page is served with noindex and robots.txt blocks everything (non-production lock).
primary_gone: "The primary category “:name” is no longer assigned, so the first category is used. Choose another one below."
settings:
tab_audit: "Site audit"
crawl_schedule: "Automatic crawl"
crawl_schedule_off: "Off"
crawl_schedule_daily: "Daily"
crawl_schedule_weekly: "Weekly"
crawl_schedule_comment: "Runs at night. You can always start a crawl by hand in SEO › Crawls."
crawl_max_pages: "Pages per crawl"
crawl_max_pages_comment: "The crawl stops after this many pages. Default 5000."
crawl_delay: "Pause between requests (ms)"
crawl_delay_comment: "At most two requests run at a time. A longer pause spares a small server."
crawl_keep: "Crawls kept"
crawl_keep_comment: "Older crawls are deleted with their pages and issues."
generic_anchors: "Generic link texts"
generic_anchors_comment: "One per line. The crawl reports links whose whole text is one of these, ignoring letter case."
placeholders: Placeholders
placeholders_code: Placeholder
placeholders_meaning: Replaced with

View File

@@ -85,6 +85,20 @@ status:
staging: W tym środowisku każda strona ma noindex, a robots.txt blokuje wszystko (blokada środowiska nieprodukcyjnego).
primary_gone: "Kategoria główna „:name” nie jest już przypisana, więc używana jest pierwsza kategoria. Wybierz inną poniżej."
settings:
tab_audit: "Audyt witryny"
crawl_schedule: "Automatyczne skanowanie"
crawl_schedule_off: "Wyłączone"
crawl_schedule_daily: "Codziennie"
crawl_schedule_weekly: "Co tydzień"
crawl_schedule_comment: "Uruchamiane w nocy. Skanowanie zawsze możesz uruchomić ręcznie w SEO › Skanowania."
crawl_max_pages: "Stron na skanowanie"
crawl_max_pages_comment: "Skanowanie kończy się po tylu stronach. Domyślnie 5000."
crawl_delay: "Przerwa między żądaniami (ms)"
crawl_delay_comment: "Jednocześnie trwają najwyżej dwa żądania. Dłuższa przerwa oszczędza mały serwer."
crawl_keep: "Przechowywane skanowania"
crawl_keep_comment: "Starsze skanowania są usuwane razem ze stronami i problemami."
generic_anchors: "Ogólnikowe teksty linków"
generic_anchors_comment: "Jeden w wierszu. Skanowanie zgłasza linki, których cały tekst jest jednym z nich, bez względu na wielkość liter."
placeholders: Znaczniki
placeholders_code: Znacznik
placeholders_meaning: Zastępowany przez