Cancelling a running crawl records cancel_requested_at and leaves the
row running until its job stops and writes cancelled, so no second
crawl starts and the row cannot be deleted or pruned meanwhile. The job
writes heartbeat_at every minute; a running crawl without a heartbeat
for 10 minutes is ended by the next start or delete.
The route report takes an optional noindex list of static paths. When
the key is present the stored set (golem15_seo_noindex_paths) is
replaced, and the sitemap filter leaves those paths out with the reason
noindex; a changed set asks the sitemap for a debounced rebuild through
contract.Rebuilder. A listed noindex address without a CMS record now
names sitemap_update_settings, or fix: code without the sitemap plugin.
- E10 golem15-seo-links in the SEO tab after the graph preview (types
with a body): inbound and outbound counts from the last finished crawl,
the orphan warning, up to 10 pages to link to (Copy address) and 10
that could link here
- classes/linkgraph.go builds the report from the crawl's pages and
links; the crawler's orphan suggestions share NormalText
- GET /_seo/api/v1/links and seo_get_link_suggestions; widget action
seo_links mapped in AdminOperations
- element bundle rebuilt
- SEO › Auto-links (golem15.seo.autolinks, after Pages): list by phrase
with locale, origin and active filters, the disabled row state, the
form with its refusals, history through a field-less extension
- toolbar Add focus keyphrases (from_keyphrases) and
POST /autolinks/from-keyphrases
- GET/POST /autolinks, PUT /autolinks/{id}, DELETE /autolinks and the
tools seo_list_autolinks, seo_create_autolink, seo_update_autolink,
seo_delete_autolinks, seo_autolinks_from_keyphrases
- the settings tile description and the permission comment name
auto-links and the site audit
- golem15_seo_autolinks (migration 202610280007) and the settings column
autolink_max (Links tab, 0 to 20, default 3)
- classes/autolinks.go: checked save and delete with history, ForPage
- ok resolve answers carry autolinks for the page's locale without
disabled phrases and links to the page itself, and autolink_max
- Site audit tab (UI-SPEC C20): crawl_schedule, crawl_max_pages
(10-50 000), crawl_delay (100-5000 ms), crawl_keep (1-50) and
generic_anchors seeded with the en and pl lists; ranges refuse with
lagoon::validation.between on the screen and through the API
- the crawler reads its limits and generic anchors from the settings;
crawls beyond crawl_keep are deleted with their pages and issues
- seo:crawl [--scheduled] and the nightly schedule entry at 02:30
(P33-40): off does nothing, weekly runs on Mondays in app.timezone
- TestCrawlSchedule
- golem15.seo.issues (Site audit): current issues errors first, labels,
summary and fix columns, filters, P7 issue_details, Check this address
again, Ignore, Stop ignoring, bulk Ignore, ignored rows badged
- golem15.seo.crawls (Crawls): newest first, P6 crawl_summary with one
status callout, staging and rendered-comparison lines, stats strip and
per-kind table; Cancel; a running crawl refuses delete
- Start crawl toolbar action on both, refusing already_running and
no_site_url; navigation Site audit (gauge) and Crawls (globe)
- routes and MCP tools start/cancel/list/get/delete crawls, list_issues
with filters (open by default), recheck_url, ignore and unignore
- AdminOperations maps both controllers; toolbar actions get their own
operation kind
- TestCrawlAdmin, TestCrawlTools, TestIssueKindLabels
- golem15_seo_crawls, crawl pages, links, issues and ignores (P33-37)
- classes/crawl: fetchguard client on the site host and its www twin,
sitemap index and robots.txt seeds, two workers with a delay, limits
- h1_missing and h1_multiple with the P33-38 fix route
- POST /_seo/api/v1/crawls queues the golem15.seo.crawl job; GET /issues
answers the stored issues with id, severity, source, fix, found_at,
ignored and crawl_id
- seo_schema (seoschema, E8) with the C17 server rules: invalid JSON with
line and column, non-objects, 16 KB, depth 8, a literal <script and a
type outside the kind's list refused; the value re-serialised
- a custom node with a generated @id merges into it (schema.merged save
notice), other nodes are appended to the graph
- seo_primary_parent (seoparent) for article types; both values stored
in the entry data on every locale row
- the seo_graph widget action (E9 data), P1 primary_gone and the P2
schema type and breadcrumb trail rows
- GET /_seo/api/v1/graph and MCP seo_get_graph; seo_set_url_seo and
apply_batch accept primary_parent and schema
The graph tool lands with the validation: the operations parity test
binds the seo_graph widget action to its tool.
- Pagination tab: query or path form, per-locale segment, title and
description suffixes with %page% and %pages%
- ?page=1, /<segment>/1 and the other form answer 301 to the configured
address in one hop with normalisation and strict URLs; a page value
that is not a number answers not_found with pagination.invalid
- page 2 and later: self canonical with the page, index,follow, the
suffixes (%pages% left for the layer), paged hreflang alternates
- GET /site answers pagination {form, param, segments} and home_crumb
- Publisher and Local business tabs with the C20 refusals (sameAs lines,
person as local business, address, times, country, coordinates)
- the publisher node from the settings, as an Organization, a Person or
the chosen local business type with address, geo and opening hours
- P8 publisher preview with Google's required and recommended findings
(classes/schema Validate)
- the home crumb name on the General tab; every new key in the settings
API and seo_update_settings
- noindex_published fires only when a sitemap file lists the address now
(read once per batch through contract.Listing)
- CMS issue sitemap_blocked (severity error, tool seo_update_robots,
details line and rule): a listed address the production robots.txt
keeps Googlebot from
- the SEO tab's P1 shows status.excluded_title with the reason (noindex,
canonical elsewhere, redirect, or the type switched off in the sitemap
settings via ListingSettings.TypesOff) as a plain callout
- robots.txt is composed once per check of many addresses
- GET /_seo/api/v1/site (public, noindex): site_url, default_locale,
locales, unprefixed_locale, links.new_tab, html_sitemap (from the
sitemap plugin's Listing), routes of every content type, root_files
(verification files and the IndexNow key file) and llms
- GET /llms.txt: with llms_enabled (robots.txt screen, AI bots tab, off by
default) and in production, the indexable pages from the sitemaps per
group under the site name and llms_intro (else the default
description); otherwise 404; seo_update_robots takes both fields
- robots.txt names the sitemap plugin's index unless
golem15.seo.robots_sitemaps is set; the config default is now a comment
- migration 202610280003: settings links_new_tab (default on),
nofollow_domains and sponsored_domains; robots llms_enabled and
llms_intro (used by the llms.txt commit)
- SEO settings Links tab: the new-tab switch and the two domain lists,
one domain per line, lowercased, refused with settings.domain_invalid
naming the line; writable through seo_update_settings
- the resolve answer of an ok page carries outbound {new_tab,
nofollow_domains, sponsored_domains}; no existing key changes
- POST /_seo/api/v1/registry/routes (public, bucket seo-notfound): the
frontend server reports its static routes with the bearer token
golem15.seo.route_report_token, else notfound_token (constant-time;
both empty answer 404); at most 2000 paths
- unknown paths become registry pages (origin reported, kind page, a name
from the last segment), known ones get reported_at; paths another
content type owns and invalid ones are skipped; nothing is deleted or
overwritten, and the report's time marks missing pages
- golem15_seo_pages (migration 202610280002): the site's own frontend pages
with an address and a name per locale, kind, origin, parent, content date
and note; settings routes_reported_at
- SEO > Pages (golem15.seo.pages) after URL rules: list with kind, origin
and missing filters and the missing row state, form with the P9 partial;
the plugin's form extension saves the address and name per locale and
refuses url_required, url_invalid, url_taken and url_owned
- menu item type seo-page with a content descriptor, so each page gets
the SEO tab, resolves ok and is listed in the sitemaps grouped by kind
(pages, offers, services, landing-pages, titled per locale)
- management API GET/POST /_seo/api/v1/pages, PUT/DELETE /pages/{id} and
the MCP tools seo_list_pages, seo_create_page, seo_update_page,
seo_delete_page (in this commit because the operations parity test maps
every Pages operation to its tool)
coveredSQL and chainSQL compared the source literally, so /missing/ or
/missing?utm=x stayed open after /missing answered it, and /a -> /b/ was
no chain next to /b. Both now use one SQL form of the runtime's exact
match (prefiltered on the indexed source_key). Pattern and regex rules
remain with the per-row badge, which the list_not_found schema text now
says.
A locale given with a URL write overrode the matched one unchecked, so
"xx" created an entry nothing ever reads while the answer reported
success. SetURLSEO (API, MCP, batch set_seo, records import) now requires
one of the record's locales and refuses others with api.locale_unknown.
The API, MCP, batches and the CSV import copied any value and Normalize
replaced it with the default, answering 201 for a 307 saved as a 301.
RedirectFields.Problems refuses values outside {301, 302, 410}, {exact,
pattern, regex} and {ignore, pass} with value_invalid; the import marks
such rows refused.
(page-1)*perPage overflowed for a large page and sliced out of range.
Export compares the page with the page count before multiplying, and the
list endpoints cap the page number.
- POST /_seo/api/v1/batch runs changes in one transaction: dry_run
answers the diff and the conflicts (loops, chains, noindex on a
published record, canonicals to redirected or noindex addresses) and
rolls back; a refused change refuses the batch (D-12)
- undo of a change or a whole batch from History and through MCP, with
the undo_stale, undo_loop and undo_gone refusals (D-13)
- CSV export and import of redirects, rules and records by URL: BOM,
comma or semicolon, formula cells escaped, a dry-run preview, all or
nothing with via import; the screen's apply token is HMAC-SHA256 over
the file and a fingerprint of the data (transfer.stale)
- Import and export singleton with the seo_transfer widget action,
History record actions undo and undo_batch, navigation entry
- MCP tools undo_change, undo_batch, apply_batch, export_csv, import_csv
- TestBatch, TestUndo, TestCSVTransfer, TestLoopRefused mcp and import
- /_seo/api/v1 routes for the effective SEO of a URL with sources and
owner, URL rules, redirects (loops refused with redirect_loop), the
404 log actions, robots.txt and its tester, settings and history
- the CMS-data issue report: missing and duplicate titles and
descriptions, rules over records, redirect chains, canonicals to
noindex or redirected addresses, noindex addresses robots.txt blocks
- 24 MCP tools on those routes (backend guard, golem15.seo.manage)
- AdminOperations maps every admin operation to its tool;
TestEveryAdminOperationHasTool and TestMCPReadToolsAreGET guard drift
- robots.txt writes go through classes.SaveRobots from the admin too
- PUT /_seo/api/v1/seo writes to the record behind a URL in its locale,
else to an exact URL rule (force_rule writes a rule); the answer says
where (D-11)
- management routes resolve the backend administrator from a token or
the MCP dispatch and require golem15.seo.manage (D-10, D-15)
- MCP tool set_url_seo; history records via mcp
- classes.WithTx, FindOwner, SetURLSEO, SaveRule and DeleteRule
- GET /_seo/api/v1/verification/{name} serves a verification file stored
in the SEO settings with its content type, nosniff and a sandboxing CSP
(D-05, D-103); the name pattern is enforced in the route and the
lookup, and no route is registered at the site root
- github.com/jimsmart/grobotstxt v1.0.3 (D-99, decision note in the
framework repo)
- golem15_seo_robots singleton: rules and AI bot switches; eleven bots
incl. Claude-User, all allowed by default
- GET /robots.txt: rules, a Disallow group per switched-off bot and the
golem15.seo.robots_sitemaps lines; outside production the lock with
X-Robots-Tag; never a 5xx (last good body, else allow-all)
- TestRobots on Google's matcher with the deciding line, and the
noindex-blocked check (D-24)
- Normalize: https, www or bare host, double slashes and index files,
trailing slash add/remove, optional ASCII lowercasing (defaults change
nothing but scheme, slashes and index files)
- strict URLs through signpost MatchPath: a path that names a record
without being its address 301s to the record's address
- resolve runs normalisation, redirects, strict URLs, then SEO; the
location is the final address, so a visitor makes one hop
- P1 warns when a record's canonical URL redirects
- POST /_seo/api/v1/notfound with the golem15.seo.notfound_token bearer
(constant-time, off when unset), URL and referer caps, a row cap with
an hourly warning, throttle bucket seo-notfound
- golem15.seo.notfound list by hits with Resolved rows, a preview with the
resolved hint, a prompted 301 and a 410 record action, a bulk 410 that
refuses a selection with a covered address
- every /_seo/api/v1 answer carries X-Robots-Tag: noindex
