Cancelling a running crawl records cancel_requested_at and leaves the
row running until its job stops and writes cancelled, so no second
crawl starts and the row cannot be deleted or pruned meanwhile. The job
writes heartbeat_at every minute; a running crawl without a heartbeat
for 10 minutes is ended by the next start or delete.
The crawl job's context ends on a cancel or at the job timeout. Storing
the pages, links and issues now runs on a context detached from it with
its own 10-minute bound, a timeout ends the crawl failed with a
time-limit error, a LoadLimits failure falls back to the defaults, and a
fetch cut short by the cancellation is recorded as canceled.
The route report takes an optional noindex list of static paths. When
the key is present the stored set (golem15_seo_noindex_paths) is
replaced, and the sitemap filter leaves those paths out with the reason
noindex; a changed set asks the sitemap for a debounced rebuild through
contract.Rebuilder. A listed noindex address without a CMS record now
names sitemap_update_settings, or fix: code without the sitemap plugin.
- github.com/chromedp/chromedp v0.21.0 (decision note
2026-10-28-chromedp-rendered-crawl.md): with golem15.seo.crawl_chrome_path
the crawl renders up to golem15.seo.crawl_render_max pages (200) and
compares them with the raw HTML; without a path it records skipped
- guard: site host and www twin resolved and checked with
fetchguard.IsPrivateAddr, pinned through --host-resolver-rules with a
final MAP * ~NOTFOUND; Fetch-domain interception fails every request
outside the site (third-party scripts never load); fresh temporary
profile removed on Close; downloads denied; one tab, 20 s, 5 MiB DOM
- a relative or non-executable path, a private address or a Chrome that
does not start records render_status failed; the crawl still finishes
- config keys documented; README Rendered comparison section
- TestChromeRenderer (runs with a local Chromium) and TestRendererGuards
- Site audit tab (UI-SPEC C20): crawl_schedule, crawl_max_pages
(10-50 000), crawl_delay (100-5000 ms), crawl_keep (1-50) and
generic_anchors seeded with the en and pl lists; ranges refuse with
lagoon::validation.between on the screen and through the API
- the crawler reads its limits and generic anchors from the settings;
crawls beyond crawl_keep are deleted with their pages and issues
- seo:crawl [--scheduled] and the nightly schedule entry at 02:30
(P33-40): off does nothing, weekly runs on Mondays in app.timezone
- TestCrawlSchedule
- golem15.seo.issues (Site audit): current issues errors first, labels,
summary and fix columns, filters, P7 issue_details, Check this address
again, Ignore, Stop ignoring, bulk Ignore, ignored rows badged
- golem15.seo.crawls (Crawls): newest first, P6 crawl_summary with one
status callout, staging and rendered-comparison lines, stats strip and
per-kind table; Cancel; a running crawl refuses delete
- Start crawl toolbar action on both, refusing already_running and
no_site_url; navigation Site audit (gauge) and Crawls (globe)
- routes and MCP tools start/cancel/list/get/delete crawls, list_issues
with filters (open by default), recheck_url, ignore and unignore
- AdminOperations maps both controllers; toolbar actions get their own
operation kind
- TestCrawlAdmin, TestCrawlTools, TestIssueKindLabels
- orphan_page for listed indexable pages no crawled page links to, with
pages mentioning the title or focus keyphrase as suggestions (D-82)
- CMS-data image_alt and h1_in_content from the content descriptors
(TopHeading 2 only), stored with the crawl as source cms (D-85, D-50)
- seo-page names Tools edit seo_update_page; every stored issue carries
a tool or fix code with details.what (P33-38)
- TestIssueMerge (ignored fingerprints survive the next crawl, fixed
pages lose their issues) and TestStagingCrawl (Pitfall 12)
- jsonld_invalid, jsonld_multiple, and jsonld_required/recommended from
schema.Validate (the E9 rules); publisher nodes route to settings
- hreflang_error and hreflang_reciprocal over the fetched alternates
- broken_link and redirected_link with the linking pages, generic_anchor
against the en and pl seed lists
- image_alt, image_size and featured_lazy
- content_js check cms: the record title in the H1, half its words and
its primary parent's link in the raw HTML
- compareRendered (pure): links, an H1 or text only after JavaScript as
content_js check rendered; the Renderer seam and render bookkeeping
- TestRenderSkipped: skipped without a renderer, failed with its note
- page_status, sitemap_error, sitemap_noindex, sitemap_blocked and
sitemap_missing against the fetched sitemap files and robots.txt
- canonical_missing and canonical_error, missing and duplicate titles and
descriptions over the crawl
- h1_missing, h1_multiple, heading_skipped, heading_empty and
h1_title_mismatch; content headings route to the record's tool
- the staging lock's noindex is never counted (Pitfall 12)
- TestCrawlerChecks on the planted site with a clean control page
- golem15_seo_crawls, crawl pages, links, issues and ignores (P33-37)
- classes/crawl: fetchguard client on the site host and its www twin,
sitemap index and robots.txt seeds, two workers with a delay, limits
- h1_missing and h1_multiple with the P33-38 fix route
- POST /_seo/api/v1/crawls queues the golem15.seo.crawl job; GET /issues
answers the stored issues with id, severity, source, fix, found_at,
ignored and crawl_id
