- Site audit tab (UI-SPEC C20): crawl_schedule, crawl_max_pages
(10-50 000), crawl_delay (100-5000 ms), crawl_keep (1-50) and
generic_anchors seeded with the en and pl lists; ranges refuse with
lagoon::validation.between on the screen and through the API
- the crawler reads its limits and generic anchors from the settings;
crawls beyond crawl_keep are deleted with their pages and issues
- seo:crawl [--scheduled] and the nightly schedule entry at 02:30
(P33-40): off does nothing, weekly runs on Mondays in app.timezone
- TestCrawlSchedule
- golem15.seo.issues (Site audit): current issues errors first, labels,
summary and fix columns, filters, P7 issue_details, Check this address
again, Ignore, Stop ignoring, bulk Ignore, ignored rows badged
- golem15.seo.crawls (Crawls): newest first, P6 crawl_summary with one
status callout, staging and rendered-comparison lines, stats strip and
per-kind table; Cancel; a running crawl refuses delete
- Start crawl toolbar action on both, refusing already_running and
no_site_url; navigation Site audit (gauge) and Crawls (globe)
- routes and MCP tools start/cancel/list/get/delete crawls, list_issues
with filters (open by default), recheck_url, ignore and unignore
- AdminOperations maps both controllers; toolbar actions get their own
operation kind
- TestCrawlAdmin, TestCrawlTools, TestIssueKindLabels
- golem15_seo_crawls, crawl pages, links, issues and ignores (P33-37)
- classes/crawl: fetchguard client on the site host and its www twin,
sitemap index and robots.txt seeds, two workers with a delay, limits
- h1_missing and h1_multiple with the P33-38 fix route
- POST /_seo/api/v1/crawls queues the golem15.seo.crawl job; GET /issues
answers the stored issues with id, severity, source, fix, found_at,
ignored and crawl_id
- Pagination tab: query or path form, per-locale segment, title and
description suffixes with %page% and %pages%
- ?page=1, /<segment>/1 and the other form answer 301 to the configured
address in one hop with normalisation and strict URLs; a page value
that is not a number answers not_found with pagination.invalid
- page 2 and later: self canonical with the page, index,follow, the
suffixes (%pages% left for the layer), paged hreflang alternates
- GET /site answers pagination {form, param, segments} and home_crumb
- Publisher and Local business tabs with the C20 refusals (sameAs lines,
person as local business, address, times, country, coordinates)
- the publisher node from the settings, as an Organization, a Person or
the chosen local business type with address, geo and opening hours
- P8 publisher preview with Google's required and recommended findings
(classes/schema Validate)
- the home crumb name on the General tab; every new key in the settings
API and seo_update_settings
- classes/schema builds one @graph per page (WebPage, article, profile,
service, BreadcrumbList, VideoObject, WebSite, publisher) linked by @id
- breadcrumb trail through the primary parent chain, hreflang alternates
with x-default, html_lang and og:locale in the resolve answer
- migration 202610280004 adds the publisher, local business, home crumb
and pagination settings columns
- GET /_seo/api/v1/site (public, noindex): site_url, default_locale,
locales, unprefixed_locale, links.new_tab, html_sitemap (from the
sitemap plugin's Listing), routes of every content type, root_files
(verification files and the IndexNow key file) and llms
- GET /llms.txt: with llms_enabled (robots.txt screen, AI bots tab, off by
default) and in production, the indexable pages from the sitemaps per
group under the site name and llms_intro (else the default
description); otherwise 404; seo_update_robots takes both fields
- robots.txt names the sitemap plugin's index unless
golem15.seo.robots_sitemaps is set; the config default is now a comment
- migration 202610280003: settings links_new_tab (default on),
nofollow_domains and sponsored_domains; robots llms_enabled and
llms_intro (used by the llms.txt commit)
- SEO settings Links tab: the new-tab switch and the two domain lists,
one domain per line, lowercased, refused with settings.domain_invalid
naming the line; writable through seo_update_settings
- the resolve answer of an ok page carries outbound {new_tab,
nofollow_domains, sponsored_domains}; no existing key changes
- golem15_seo_pages (migration 202610280002): the site's own frontend pages
with an address and a name per locale, kind, origin, parent, content date
and note; settings routes_reported_at
- SEO > Pages (golem15.seo.pages) after URL rules: list with kind, origin
and missing filters and the missing row state, form with the P9 partial;
the plugin's form extension saves the address and name per locale and
refuses url_required, url_invalid, url_taken and url_owned
- menu item type seo-page with a content descriptor, so each page gets
the SEO tab, resolves ok and is listed in the sitemaps grouped by kind
(pages, offers, services, landing-pages, titled per locale)
- management API GET/POST /_seo/api/v1/pages, PUT/DELETE /pages/{id} and
the MCP tools seo_list_pages, seo_create_page, seo_update_page,
seo_delete_page (in this commit because the operations parity test maps
every Pages operation to its tool)
Rule.Problems had no length check, so a long address reached the
(url, match) UNIQUE btree index and failed with a 500. It now refuses an
address over models.MaxRuleURL with rules.url_invalid, like redirect
sources.
coveredSQL and chainSQL compared the source literally, so /missing/ or
/missing?utm=x stayed open after /missing answered it, and /a -> /b/ was
no chain next to /b. Both now use one SQL form of the runtime's exact
match (prefiltered on the indexed source_key). Pattern and regex rules
remain with the per-row badge, which the list_not_found schema text now
says.
- POST /_seo/api/v1/batch runs changes in one transaction: dry_run
answers the diff and the conflicts (loops, chains, noindex on a
published record, canonicals to redirected or noindex addresses) and
rolls back; a refused change refuses the batch (D-12)
- undo of a change or a whole batch from History and through MCP, with
the undo_stale, undo_loop and undo_gone refusals (D-13)
- CSV export and import of redirects, rules and records by URL: BOM,
comma or semicolon, formula cells escaped, a dry-run preview, all or
nothing with via import; the screen's apply token is HMAC-SHA256 over
the file and a fingerprint of the data (transfer.stale)
- Import and export singleton with the seo_transfer widget action,
History record actions undo and undo_batch, navigation entry
- MCP tools undo_change, undo_batch, apply_batch, export_csv, import_csv
- TestBatch, TestUndo, TestCSVTransfer, TestLoopRefused mcp and import
- golem15.seo.robots singleton (UI-SPEC C10): staging callout and the
served file (P4); the plugin's own extension adds the editor, the
tester and the AI bots tab, saved to the robots row with history
- field type robotstxt (E4, live lint, size meter; Decode refuses over
500 KB) and widget golem15-seo-robotstester (E5) on robots_test, which
tests the unsaved text and switches through grobotstxt
- saving rules that block a noindex address answers robots.blocks_noindex
- github.com/jimsmart/grobotstxt v1.0.3 (D-99, decision note in the
framework repo)
- golem15_seo_robots singleton: rules and AI bot switches; eleven bots
incl. Claude-User, all allowed by default
- GET /robots.txt: rules, a Disallow group per switched-off bot and the
golem15.seo.robots_sitemaps lines; outside production the lock with
X-Robots-Tag; never a 5xx (last good body, else allow-all)
- TestRobots on Google's matcher with the deciding line, and the
noindex-blocked check (D-24)
- POST /_seo/api/v1/notfound with the golem15.seo.notfound_token bearer
(constant-time, off when unset), URL and referer caps, a row cap with
an hourly warning, throttle bucket seo-notfound
- golem15.seo.notfound list by hits with Resolved rows, a preview with the
resolved hint, a prompted 301 and a 410 record action, a bulk 410 that
refuses a selection with a covered address
- every /_seo/api/v1 answer carries X-Robots-Tag: noindex
- golem15.seo.redirects list and form (UI-SPEC C8): chain rows marked,
status, match, origin and chain filters, the navigation item lands here
- a field-less extension on the controller refuses a loop with
redirects.loop on target, answers the chain notice and records history
- the status dropdown posts text, read through RedirectStatus
- rule set: exact sources by key, patterns by literal prefix, regex by id
(stdlib RE2, compiled once per source); ignore case, trailing slash,
query ignore/pass, a source with ? matches its query exactly
- targets: captures escaped, a relative target never leaves the site, an
absolute one keeps its host
- SaveRedirect validates, simulates the rule in place (exact, pattern and
regex probes, 10 hops) and refuses a loop; chains answer the notice
- resolve follows the chain to its final target, 410 answers gone, a
runtime cycle answers not_found and is logged once
- golem15_seo_redirects table and Redirect model
- AddressChanged: old -> new 301 in the save transaction, earlier
redirects pointed at the new address, self-loops dropped
- the SEO tab save compares descriptor URLs with the entry URLs and
answers the redirects.auto_created notice
- resolve answers redirect with the absolute target for an exact source
