Cancelling a running crawl records cancel_requested_at and leaves the
row running until its job stops and writes cancelled, so no second
crawl starts and the row cannot be deleted or pruned meanwhile. The job
writes heartbeat_at every minute; a running crawl without a heartbeat
for 10 minutes is ended by the next start or delete.
- E10 golem15-seo-links in the SEO tab after the graph preview (types
with a body): inbound and outbound counts from the last finished crawl,
the orphan warning, up to 10 pages to link to (Copy address) and 10
that could link here
- classes/linkgraph.go builds the report from the crawl's pages and
links; the crawler's orphan suggestions share NormalText
- GET /_seo/api/v1/links and seo_get_link_suggestions; widget action
seo_links mapped in AdminOperations
- element bundle rebuilt
- SEO › Auto-links (golem15.seo.autolinks, after Pages): list by phrase
with locale, origin and active filters, the disabled row state, the
form with its refusals, history through a field-less extension
- toolbar Add focus keyphrases (from_keyphrases) and
POST /autolinks/from-keyphrases
- GET/POST /autolinks, PUT /autolinks/{id}, DELETE /autolinks and the
tools seo_list_autolinks, seo_create_autolink, seo_update_autolink,
seo_delete_autolinks, seo_autolinks_from_keyphrases
- the settings tile description and the permission comment name
auto-links and the site audit
- golem15_seo_autolinks (migration 202610280007) and the settings column
autolink_max (Links tab, 0 to 20, default 3)
- classes/autolinks.go: checked save and delete with history, ForPage
- ok resolve answers carry autolinks for the page's locale without
disabled phrases and links to the page itself, and autolink_max
- Site audit tab (UI-SPEC C20): crawl_schedule, crawl_max_pages
(10-50 000), crawl_delay (100-5000 ms), crawl_keep (1-50) and
generic_anchors seeded with the en and pl lists; ranges refuse with
lagoon::validation.between on the screen and through the API
- the crawler reads its limits and generic anchors from the settings;
crawls beyond crawl_keep are deleted with their pages and issues
- seo:crawl [--scheduled] and the nightly schedule entry at 02:30
(P33-40): off does nothing, weekly runs on Mondays in app.timezone
- TestCrawlSchedule
- golem15.seo.issues (Site audit): current issues errors first, labels,
summary and fix columns, filters, P7 issue_details, Check this address
again, Ignore, Stop ignoring, bulk Ignore, ignored rows badged
- golem15.seo.crawls (Crawls): newest first, P6 crawl_summary with one
status callout, staging and rendered-comparison lines, stats strip and
per-kind table; Cancel; a running crawl refuses delete
- Start crawl toolbar action on both, refusing already_running and
no_site_url; navigation Site audit (gauge) and Crawls (globe)
- routes and MCP tools start/cancel/list/get/delete crawls, list_issues
with filters (open by default), recheck_url, ignore and unignore
- AdminOperations maps both controllers; toolbar actions get their own
operation kind
- TestCrawlAdmin, TestCrawlTools, TestIssueKindLabels
- orphan_page for listed indexable pages no crawled page links to, with
pages mentioning the title or focus keyphrase as suggestions (D-82)
- CMS-data image_alt and h1_in_content from the content descriptors
(TopHeading 2 only), stored with the crawl as source cms (D-85, D-50)
- seo-page names Tools edit seo_update_page; every stored issue carries
a tool or fix code with details.what (P33-38)
- TestIssueMerge (ignored fingerprints survive the next crawl, fixed
pages lose their issues) and TestStagingCrawl (Pitfall 12)
- jsonld_invalid, jsonld_multiple, and jsonld_required/recommended from
schema.Validate (the E9 rules); publisher nodes route to settings
- hreflang_error and hreflang_reciprocal over the fetched alternates
- broken_link and redirected_link with the linking pages, generic_anchor
against the en and pl seed lists
- image_alt, image_size and featured_lazy
- content_js check cms: the record title in the H1, half its words and
its primary parent's link in the raw HTML
- compareRendered (pure): links, an H1 or text only after JavaScript as
content_js check rendered; the Renderer seam and render bookkeeping
- TestRenderSkipped: skipped without a renderer, failed with its note
- page_status, sitemap_error, sitemap_noindex, sitemap_blocked and
sitemap_missing against the fetched sitemap files and robots.txt
- canonical_missing and canonical_error, missing and duplicate titles and
descriptions over the crawl
- h1_missing, h1_multiple, heading_skipped, heading_empty and
h1_title_mismatch; content headings route to the record's tool
- the staging lock's noindex is never counted (Pitfall 12)
- TestCrawlerChecks on the planted site with a clean control page
- golem15_seo_crawls, crawl pages, links, issues and ignores (P33-37)
- classes/crawl: fetchguard client on the site host and its www twin,
sitemap index and robots.txt seeds, two workers with a delay, limits
- h1_missing and h1_multiple with the P33-38 fix route
- POST /_seo/api/v1/crawls queues the golem15.seo.crawl job; GET /issues
answers the stored issues with id, severity, source, fix, found_at,
ignored and crawl_id
- seo_schema (seoschema, E8) with the C17 server rules: invalid JSON with
line and column, non-objects, 16 KB, depth 8, a literal <script and a
type outside the kind's list refused; the value re-serialised
- a custom node with a generated @id merges into it (schema.merged save
notice), other nodes are appended to the graph
- seo_primary_parent (seoparent) for article types; both values stored
in the entry data on every locale row
- the seo_graph widget action (E9 data), P1 primary_gone and the P2
schema type and breadcrumb trail rows
- GET /_seo/api/v1/graph and MCP seo_get_graph; seo_set_url_seo and
apply_batch accept primary_parent and schema
The graph tool lands with the validation: the operations parity test
binds the seo_graph widget action to its tool.
- Pagination tab: query or path form, per-locale segment, title and
description suffixes with %page% and %pages%
- ?page=1, /<segment>/1 and the other form answer 301 to the configured
address in one hop with normalisation and strict URLs; a page value
that is not a number answers not_found with pagination.invalid
- page 2 and later: self canonical with the page, index,follow, the
suffixes (%pages% left for the layer), paged hreflang alternates
- GET /site answers pagination {form, param, segments} and home_crumb
- Publisher and Local business tabs with the C20 refusals (sameAs lines,
person as local business, address, times, country, coordinates)
- the publisher node from the settings, as an Organization, a Person or
the chosen local business type with address, geo and opening hours
- P8 publisher preview with Google's required and recommended findings
(classes/schema Validate)
- the home crumb name on the General tab; every new key in the settings
API and seo_update_settings
- classes/schema builds one @graph per page (WebPage, article, profile,
service, BreadcrumbList, VideoObject, WebSite, publisher) linked by @id
- breadcrumb trail through the primary parent chain, hreflang alternates
with x-default, html_lang and og:locale in the resolve answer
- migration 202610280004 adds the publisher, local business, home crumb
and pagination settings columns
- noindex_published fires only when a sitemap file lists the address now
(read once per batch through contract.Listing)
- CMS issue sitemap_blocked (severity error, tool seo_update_robots,
details line and rule): a listed address the production robots.txt
keeps Googlebot from
- the SEO tab's P1 shows status.excluded_title with the reason (noindex,
canonical elsewhere, redirect, or the type switched off in the sitemap
settings via ListingSettings.TypesOff) as a plain callout
- robots.txt is composed once per check of many addresses
- GET /_seo/api/v1/site (public, noindex): site_url, default_locale,
locales, unprefixed_locale, links.new_tab, html_sitemap (from the
sitemap plugin's Listing), routes of every content type, root_files
(verification files and the IndexNow key file) and llms
- GET /llms.txt: with llms_enabled (robots.txt screen, AI bots tab, off by
default) and in production, the indexable pages from the sitemaps per
group under the site name and llms_intro (else the default
description); otherwise 404; seo_update_robots takes both fields
- robots.txt names the sitemap plugin's index unless
golem15.seo.robots_sitemaps is set; the config default is now a comment
- migration 202610280003: settings links_new_tab (default on),
nofollow_domains and sponsored_domains; robots llms_enabled and
llms_intro (used by the llms.txt commit)
- SEO settings Links tab: the new-tab switch and the two domain lists,
one domain per line, lowercased, refused with settings.domain_invalid
naming the line; writable through seo_update_settings
- the resolve answer of an ok page carries outbound {new_tab,
nofollow_domains, sponsored_domains}; no existing key changes
- POST /_seo/api/v1/registry/routes (public, bucket seo-notfound): the
frontend server reports its static routes with the bearer token
golem15.seo.route_report_token, else notfound_token (constant-time;
both empty answer 404); at most 2000 paths
- unknown paths become registry pages (origin reported, kind page, a name
from the last segment), known ones get reported_at; paths another
content type owns and invalid ones are skipped; nothing is deleted or
overwritten, and the report's time marks missing pages
- golem15_seo_pages (migration 202610280002): the site's own frontend pages
with an address and a name per locale, kind, origin, parent, content date
and note; settings routes_reported_at
- SEO > Pages (golem15.seo.pages) after URL rules: list with kind, origin
and missing filters and the missing row state, form with the P9 partial;
the plugin's form extension saves the address and name per locale and
refuses url_required, url_invalid, url_taken and url_owned
- menu item type seo-page with a content descriptor, so each page gets
the SEO tab, resolves ok and is listed in the sitemaps grouped by kind
(pages, offers, services, landing-pages, titled per locale)
- management API GET/POST /_seo/api/v1/pages, PUT/DELETE /pages/{id} and
the MCP tools seo_list_pages, seo_create_page, seo_update_page,
seo_delete_page (in this commit because the operations parity test maps
every Pages operation to its tool)
A locale given with a URL write overrode the matched one unchecked, so
"xx" created an entry nothing ever reads while the answer reported
success. SetURLSEO (API, MCP, batch set_seo, records import) now requires
one of the record's locales and refuses others with api.locale_unknown.
- POST /_seo/api/v1/batch runs changes in one transaction: dry_run
answers the diff and the conflicts (loops, chains, noindex on a
published record, canonicals to redirected or noindex addresses) and
rolls back; a refused change refuses the batch (D-12)
- undo of a change or a whole batch from History and through MCP, with
the undo_stale, undo_loop and undo_gone refusals (D-13)
- CSV export and import of redirects, rules and records by URL: BOM,
comma or semicolon, formula cells escaped, a dry-run preview, all or
nothing with via import; the screen's apply token is HMAC-SHA256 over
the file and a fingerprint of the data (transfer.stale)
- Import and export singleton with the seo_transfer widget action,
History record actions undo and undo_batch, navigation entry
- MCP tools undo_change, undo_batch, apply_batch, export_csv, import_csv
- TestBatch, TestUndo, TestCSVTransfer, TestLoopRefused mcp and import
- /_seo/api/v1 routes for the effective SEO of a URL with sources and
owner, URL rules, redirects (loops refused with redirect_loop), the
404 log actions, robots.txt and its tester, settings and history
- the CMS-data issue report: missing and duplicate titles and
descriptions, rules over records, redirect chains, canonicals to
noindex or redirected addresses, noindex addresses robots.txt blocks
- 24 MCP tools on those routes (backend guard, golem15.seo.manage)
- AdminOperations maps every admin operation to its tool;
TestEveryAdminOperationHasTool and TestMCPReadToolsAreGET guard drift
- robots.txt writes go through classes.SaveRobots from the admin too
- PUT /_seo/api/v1/seo writes to the record behind a URL in its locale,
else to an exact URL rule (force_rule writes a rule); the answer says
where (D-11)
- management routes resolve the backend administrator from a token or
the MCP dispatch and require golem15.seo.manage (D-10, D-15)
- MCP tool set_url_seo; history records via mcp
- classes.WithTx, FindOwner, SetURLSEO, SaveRule and DeleteRule
- golem15.seo.robots singleton (UI-SPEC C10): staging callout and the
served file (P4); the plugin's own extension adds the editor, the
tester and the AI bots tab, saved to the robots row with history
- field type robotstxt (E4, live lint, size meter; Decode refuses over
500 KB) and widget golem15-seo-robotstester (E5) on robots_test, which
tests the unsaved text and switches through grobotstxt
- saving rules that block a noindex address answers robots.blocks_noindex
- POST /_seo/api/v1/notfound with the golem15.seo.notfound_token bearer
(constant-time, off when unset), URL and referer caps, a row cap with
an hourly warning, throttle bucket seo-notfound
- golem15.seo.notfound list by hits with Resolved rows, a preview with the
resolved hint, a prompted 301 and a 410 record action, a bulk 410 that
refuses a selection with a covered address
- every /_seo/api/v1 answer carries X-Robots-Tag: noindex
- golem15.seo.redirects list and form (UI-SPEC C8): chain rows marked,
status, match, origin and chain filters, the navigation item lands here
- a field-less extension on the controller refuses a loop with
redirects.loop on target, answers the chain notice and records history
- the status dropdown posts text, read through RedirectStatus
- rule set: exact sources by key, patterns by literal prefix, regex by id
(stdlib RE2, compiled once per source); ignore case, trailing slash,
query ignore/pass, a source with ? matches its query exactly
- targets: captures escaped, a relative target never leaves the site, an
absolute one keeps its host
- SaveRedirect validates, simulates the rule in place (exact, pattern and
regex probes, 10 hops) and refuses a loop; chains answer the notice
- resolve follows the chain to its final target, 410 answers gone, a
runtime cycle answers not_found and is logged once
- golem15_seo_redirects table and Redirect model
- AddressChanged: old -> new 301 in the save transaction, earlier
redirects pointed at the new address, self-loops dropped
- the SEO tab save compares descriptor URLs with the entry URLs and
answers the redirects.auto_created notice
- resolve answers redirect with the absolute target for an exact source
