The crawl job's context ends on a cancel or at the job timeout. Storing
the pages, links and issues now runs on a context detached from it with
its own 10-minute bound, a timeout ends the crawl failed with a
time-limit error, a LoadLimits failure falls back to the defaults, and a
fetch cut short by the cancellation is recorded as canceled.
The route report takes an optional noindex list of static paths. When
the key is present the stored set (golem15_seo_noindex_paths) is
replaced, and the sitemap filter leaves those paths out with the reason
noindex; a changed set asks the sitemap for a debounced rebuild through
contract.Rebuilder. A listed noindex address without a CMS record now
names sitemap_update_settings, or fix: code without the sitemap plugin.
- schema: types per kind, @id links, every node kind, the publisher and
local business, every validator rule, custom JSON-LD caps and merge
- crawl: HTML parse, each check's positive and negative case, fix routes
with the fitting tool, the rendered comparison, allowRequest,
resolverRules and NewRenderer
- pagination, alternates and x-default, primary parents, the filter's
addresses, registry paths and fields, auto-link addresses
- E10 golem15-seo-links in the SEO tab after the graph preview (types
with a body): inbound and outbound counts from the last finished crawl,
the orphan warning, up to 10 pages to link to (Copy address) and 10
that could link here
- classes/linkgraph.go builds the report from the crawl's pages and
links; the crawler's orphan suggestions share NormalText
- GET /_seo/api/v1/links and seo_get_link_suggestions; widget action
seo_links mapped in AdminOperations
- element bundle rebuilt
- github.com/chromedp/chromedp v0.21.0 (decision note
2026-10-28-chromedp-rendered-crawl.md): with golem15.seo.crawl_chrome_path
the crawl renders up to golem15.seo.crawl_render_max pages (200) and
compares them with the raw HTML; without a path it records skipped
- guard: site host and www twin resolved and checked with
fetchguard.IsPrivateAddr, pinned through --host-resolver-rules with a
final MAP * ~NOTFOUND; Fetch-domain interception fails every request
outside the site (third-party scripts never load); fresh temporary
profile removed on Close; downloads denied; one tab, 20 s, 5 MiB DOM
- a relative or non-executable path, a private address or a Chrome that
does not start records render_status failed; the crawl still finishes
- config keys documented; README Rendered comparison section
- TestChromeRenderer (runs with a local Chromium) and TestRendererGuards
- Site audit tab (UI-SPEC C20): crawl_schedule, crawl_max_pages
(10-50 000), crawl_delay (100-5000 ms), crawl_keep (1-50) and
generic_anchors seeded with the en and pl lists; ranges refuse with
lagoon::validation.between on the screen and through the API
- the crawler reads its limits and generic anchors from the settings;
crawls beyond crawl_keep are deleted with their pages and issues
- seo:crawl [--scheduled] and the nightly schedule entry at 02:30
(P33-40): off does nothing, weekly runs on Mondays in app.timezone
- TestCrawlSchedule
- golem15.seo.issues (Site audit): current issues errors first, labels,
summary and fix columns, filters, P7 issue_details, Check this address
again, Ignore, Stop ignoring, bulk Ignore, ignored rows badged
- golem15.seo.crawls (Crawls): newest first, P6 crawl_summary with one
status callout, staging and rendered-comparison lines, stats strip and
per-kind table; Cancel; a running crawl refuses delete
- Start crawl toolbar action on both, refusing already_running and
no_site_url; navigation Site audit (gauge) and Crawls (globe)
- routes and MCP tools start/cancel/list/get/delete crawls, list_issues
with filters (open by default), recheck_url, ignore and unignore
- AdminOperations maps both controllers; toolbar actions get their own
operation kind
- TestCrawlAdmin, TestCrawlTools, TestIssueKindLabels
- orphan_page for listed indexable pages no crawled page links to, with
pages mentioning the title or focus keyphrase as suggestions (D-82)
- CMS-data image_alt and h1_in_content from the content descriptors
(TopHeading 2 only), stored with the crawl as source cms (D-85, D-50)
- seo-page names Tools edit seo_update_page; every stored issue carries
a tool or fix code with details.what (P33-38)
- TestIssueMerge (ignored fingerprints survive the next crawl, fixed
pages lose their issues) and TestStagingCrawl (Pitfall 12)
- jsonld_invalid, jsonld_multiple, and jsonld_required/recommended from
schema.Validate (the E9 rules); publisher nodes route to settings
- hreflang_error and hreflang_reciprocal over the fetched alternates
- broken_link and redirected_link with the linking pages, generic_anchor
against the en and pl seed lists
- image_alt, image_size and featured_lazy
- content_js check cms: the record title in the H1, half its words and
its primary parent's link in the raw HTML
- compareRendered (pure): links, an H1 or text only after JavaScript as
content_js check rendered; the Renderer seam and render bookkeeping
- TestRenderSkipped: skipped without a renderer, failed with its note
- page_status, sitemap_error, sitemap_noindex, sitemap_blocked and
sitemap_missing against the fetched sitemap files and robots.txt
- canonical_missing and canonical_error, missing and duplicate titles and
descriptions over the crawl
- h1_missing, h1_multiple, heading_skipped, heading_empty and
h1_title_mismatch; content headings route to the record's tool
- the staging lock's noindex is never counted (Pitfall 12)
- TestCrawlerChecks on the planted site with a clean control page
- golem15_seo_crawls, crawl pages, links, issues and ignores (P33-37)
- classes/crawl: fetchguard client on the site host and its www twin,
sitemap index and robots.txt seeds, two workers with a delay, limits
- h1_missing and h1_multiple with the P33-38 fix route
- POST /_seo/api/v1/crawls queues the golem15.seo.crawl job; GET /issues
answers the stored issues with id, severity, source, fix, found_at,
ignored and crawl_id
