feat: auto-links screen, keyphrase feed and MCP tools

- SEO › Auto-links (golem15.seo.autolinks, after Pages): list by phrase
  with locale, origin and active filters, the disabled row state, the
  form with its refusals, history through a field-less extension
- toolbar Add focus keyphrases (from_keyphrases) and
  POST /autolinks/from-keyphrases
- GET/POST /autolinks, PUT /autolinks/{id}, DELETE /autolinks and the
  tools seo_list_autolinks, seo_create_autolink, seo_update_autolink,
  seo_delete_autolinks, seo_autolinks_from_keyphrases
- the settings tile description and the permission comment name
  auto-links and the site audit
This commit is contained in:
Jakub Zych
2026-11-04 16:30:56 +01:00
parent 6203aec2bc
commit ab4e83416d
19 changed files with 852 additions and 13 deletions

View File

@@ -42,7 +42,7 @@ The frontend (a Nuxt layer or app) asks `GET /_seo/api/v1/resolve?url=...` for t
- **Structured data.** Every `ok` resolve answer carries one JSON-LD `@graph` built from the content descriptor and the settings: the page node (WebPage or its subtype), the article with its authors, dates, images, word count and section, the person of a profile page, a Service with its provider and offer, the BreadcrumbList, one VideoObject per YouTube video, the WebSite (with a SearchAction when `search_url` is set) and the publisher. Editors pick the schema type per page, fill the Service fields and add custom JSON-LD; the SEO tab previews the graph with Google's required and recommended properties and links the Rich Results Test.
- **Publisher and local business** settings: an organisation or a person with logo, website, contact and sameAs profiles, optionally a local business with address, coordinates, opening hours and price range.
- **Breadcrumbs, hreflang and pagination.** The resolve answer carries the breadcrumb trail (through the record's primary category), the hreflang alternates with `x-default`, `html_lang` and `og:locale`, and the page of a paginated address in the site's form (`?page=N` or `/<segment>/N`).
- **One permission**, `golem15.seo.manage`, for the SEO tab, rules, redirects, the 404 log, robots.txt, settings, history and the site audit, in the admin and through MCP.
- **One permission**, `golem15.seo.manage`, for the SEO tab, rules, redirects, the 404 log, pages, auto-links, robots.txt, settings, history and the site audit, in the admin and through MCP.
- **Management API and MCP.** Every SEO admin operation has a JSON route under `/_seo/api/v1` and an MCP tool bound to it (pier, prefix `seo_`). Tools address pages by URL: a change goes to the record behind the URL, else to an exact URL rule, and the answer says where. A read grant sees only the read tools. `AdminOperations` maps each admin operation to its tool, and a test fails when one has none.
- **Site audit** (SEO › **Site audit** and **Crawls**): a built-in crawler fetches the site's own pages like a search engine and reports per URL what it finds (statuses, sitemaps, canonicals, titles and descriptions, headings, structured data, hreflang, links, images, content that needs JavaScript, orphan pages), stored with the CMS-data checks: missing and duplicate titles and descriptions, URL rules overriding a record, redirect chains, canonicals pointing to a noindex or redirected address, noindex addresses robots.txt blocks, sitemap addresses the production robots.txt keeps Googlebot from (`sitemap_blocked`, severity `error`, with the deciding line), record images without alt text and H1 headings in content. Every issue names the MCP tool that fixes it, or `fix: code` with the template element to change. See [Site audit and crawler](#site-audit-and-crawler).
- **Sitemap filter.** The sitemaps leave out every address whose effective SEO is noindex, whose canonical URL points elsewhere or that redirects (normalisation or a redirect rule), with the reason `noindex`, `canonical` or `redirect`. The filter decides on the production values, so staging and production list the same set. The SEO tab's status shows **Not in the sitemaps** with the reason, also when the content type is switched off in the sitemap settings.
@@ -50,6 +50,7 @@ The frontend (a Nuxt layer or app) asks `GET /_seo/api/v1/resolve?url=...` for t
- **Route report.** The frontend server reports its static routes; unknown ones become registry pages, a reported page missing from a newer report is marked **Not on the site**, and nothing is ever deleted or overwritten.
- **Site answer** (`GET /_seo/api/v1/site`) with the facts the frontend layer needs, and **llms.txt** (off by default) listing the indexable pages from the sitemaps.
- **Links** settings: links to other sites open in a new tab (on by default), and domains whose links always get `nofollow` or `sponsored`; the resolve answer carries them as `outbound`.
- **Auto-links** (`golem15_seo_autolinks`, SEO › **Auto-links**): phrases that texts link to a page, per locale or for all, with an active switch, an origin and a note. Every `ok` resolve answer carries the phrases of its locale (never one linking to the page itself) as `autolinks`, with `autolink_max` from the Links tab (3 by default, 0 turns auto-linking off); the frontend layer links each phrase's first occurrence. **Add focus keyphrases** turns the records' focus keyphrases into phrases. See [Auto-links](#auto-links).
- **Site cache purge.** Every SEO write (admin, MCP, batch, import, undo, automatic) asks the frontend to drop its cached pages after the commit, once per transaction, through the framework's site-cache purger (`site_cache.purge_url`, `site_cache.purge_token`).
## Usage
@@ -111,12 +112,16 @@ The type then gets the SEO tab, a tab in the SEO settings and a place in the res
"twitter": {"card": "summary_large_image", "site": "@acme"},
"verification": []
},
"outbound": {"new_tab": true, "nofollow_domains": ["example.org"], "sponsored_domains": []}
"outbound": {"new_tab": true, "nofollow_domains": ["example.org"], "sponsored_domains": []},
"autolinks": [{"phrase": "lwia grzywa", "url": "/grzyby/lwia-grzywa"}],
"autolink_max": 3
}
```
`outbound` (present on `ok` answers) carries the settings' Links tab: open links to other sites in a new tab, and the domains (subdomains included) whose links always get `nofollow` or `sponsored`. The layer treats an absent key as empty.
`autolinks` and `autolink_max` (present on `ok` answers) are the auto-link phrases of the page's locale and of every locale, active ones only, without those linking to the page itself, longest phrase first, at most 1000; `autolink_max` is the most the layer links in one text. With `autolink_max` 0 the answer carries no phrases.
The pipeline runs normalisation, then redirects on the normalised path (followed to the final target), then strict URLs, then the SEO. When the URL is not in its final form the answer is a redirect, reached in one hop:
```json
@@ -189,6 +194,12 @@ The **robots.txt** screen (a singleton) has the editor, the tester and **What cr
`POST /_seo/api/v1/registry/routes` takes `{"routes": ["/", "/kontakt", "/o-nas"]}` from the frontend server with `Authorization: Bearer <token>`: `golem15.seo.route_report_token`, else `golem15.seo.notfound_token` (compared in constant time). It answers `200` with `{"inserted", "updated", "skipped", "reported_at"}`; a missing or wrong token is `401`, no token configured `404`, more than 2000 paths `422`. An unknown path becomes a page with origin `reported`, kind `page` and a name from its last segment; a known page only gets `reported_at`; a path another content type owns, or one that is no path, is skipped. A report never deletes a page or changes a value an editor set; a reported page the newest report did not name shows **Not on the site** and a warning on its form. Throttle bucket `seo-notfound`. A content type whose file group is also `pages` shares the file with the registry's pages.
### Auto-links
**SEO › Auto-links** (controller `golem15.seo.autolinks`, after Pages, icon `wand-sparkles`) lists the phrases (50 per page) with the address they link to, the language (**All languages** or a locale), the origin (**Manual**, **Focus keyphrase**, **MCP**) and the update time; an inactive phrase shows **Off**. Filters: language, origin and **Active only**. The form takes the phrase, the address, the language and the **Active** switch, and a note. A phrase needs 3 characters (`autolinks.phrase_short`); a phrase exists once per language, a phrase of every language once at all (`autolinks.phrase_taken` names the address it already links to); the address is a path starting with one `/` or a full `http(s)` address (`autolinks.url_invalid`: `javascript:`, `data:` and protocol-relative addresses are refused). The toolbar's **Add focus keyphrases** adds one phrase per record whose focus keyphrase in a locale is not yet a phrase there, linking to the record's address in that locale (origin **Focus keyphrase**), and toasts the count; a second run adds nothing new. Every change is recorded in History and purges the site's cached pages.
The layer links a phrase at its first occurrence in a text, as whole words and ignoring letter case, never inside headings, links, `code`, `pre`, `figcaption` or buttons, and never on the page it links to, at most `autolink_max` phrases per text (the Links tab's **Auto-links per page**, 0 to 20).
### The site answer and llms.txt
`GET /_seo/api/v1/site` (public, bucket `seo-resolve`, `X-Robots-Tag: noindex`) answers the facts the frontend layer needs, all public:
@@ -246,7 +257,7 @@ The management routes live under `/_seo/api/v1` (bucket `seo-api`, 300 requests
| `list_redirects`, `show_redirect`, `create_redirect`, `update_redirect`, `delete_redirect` | `GET/POST /redirects`, `GET/PUT/DELETE /redirects/{id}` | Redirects; a chain is saved with a warning. |
| `list_not_found`, `redirect_not_found`, `gone_not_found`, `delete_not_found` | `GET /notfound`, `POST /notfound/{id}/redirect`, `POST /notfound/{id}/gone`, `DELETE /notfound/{id}` | The 404 log and its actions; an address a rule answers is `409`. |
| `show_robots`, `update_robots`, `test_robots` | `GET/PUT /robots`, `GET /robots/test?url=&agent=` | robots.txt, the AI bot switches, `llms_enabled` and `llms_intro`, and the tester on the saved file. |
| `show_settings`, `update_settings` | `GET/PUT /settings` | The SEO settings (including `links_new_tab`, `nofollow_domains`, `sponsored_domains`, `breadcrumb_home`, the `page_*` pagination keys, the `publisher_*` and `local_*` keys) and, under `types`, the per-type settings. |
| `show_settings`, `update_settings` | `GET/PUT /settings` | The SEO settings (including `links_new_tab`, `nofollow_domains`, `sponsored_domains`, `autolink_max`, `breadcrumb_home`, the `page_*` pagination keys, the `publisher_*` and `local_*` keys) and, under `types`, the per-type settings. |
| `list_history`, `show_history` | `GET /history`, `GET /history/{id}` | History, filtered by `via`, `kind` and `batch`. |
| `undo_change`, `undo_batch` | `POST /history/{id}/undo`, `POST /history/batches/{batch}/undo` | Undo; refusals are `409` with `undo_stale`, `undo_loop`, `undo_gone` or `already_undone`. |
| `apply_batch` | `POST /batch` | Changes (`set_seo`, `create_redirect`, `update_redirect`, `delete_redirect`, `create_rule`, `update_rule`, `delete_rule`) all or nothing; `dry_run` answers the diff and the conflicts. A refused change is `422` (`redirect_loop` with the change and the path, or `batch_refused`). |
@@ -259,17 +270,19 @@ The management routes live under `/_seo/api/v1` (bucket `seo-api`, 300 requests
| `list_crawls`, `get_crawl` | `GET /crawls`, `GET /crawls/{id}` | Crawls, newest first, with `render {status, pages, note, message}` (`skipped`, `done` or `failed`); one crawl adds `staging`, `issues`, `severities` and `kinds`. |
| `cancel_crawl` | `POST /crawls/{id}/cancel` | Cancels a queued or running crawl, keeping what it fetched; another answers `409 crawls.not_running`. |
| `delete_crawls` | `DELETE /crawls?ids=` | Deletes crawls with their pages and issues; a queued or running one refuses the call with `403 crawls.delete_running`. |
| `list_autolinks`, `create_autolink`, `update_autolink`, `delete_autolinks` | `GET/POST /autolinks`, `PUT /autolinks/{id}`, `DELETE /autolinks?ids=` | The auto-link phrases: `phrase`, `url`, `locale` (`""` or `all` for every locale), `enabled`, `note`; the list filters by `search`, `locale`, `origin` and `active`. A phrase created through MCP has origin `mcp`. |
| `autolinks_from_keyphrases` | `POST /autolinks/from-keyphrases` | Adds the records' focus keyphrases as phrases; answers `{added, message}`. |
| `list_pages`, `create_page`, `update_page`, `delete_page` | `GET/POST /pages`, `PUT/DELETE /pages/{id}` | The page registry: `url` and `title` per locale (a plain string is the default locale's), `kind`, `parent_id`, `content_updated_at`, `note`; the list filters by `id`, `kind`, `origin` and `missing`. A page created through MCP has origin `mcp`. |
Lists take `page`, `per_page` (at most 200) and `search`, and answer `{"data": [...], "page", "per_page", "total"}`. GET tools are read-only; the deletes (`delete_crawls` included) and `gone_not_found` are destructive.
Lists take `page`, `per_page` (at most 200) and `search`, and answer `{"data": [...], "page", "per_page", "total"}`. GET tools are read-only; the deletes (`delete_crawls` and `delete_autolinks` included) and `gone_not_found` are destructive.
### Admin
- Navigation item **SEO** (icon `search`), landing on **Redirects**, with the side menu **Redirects**, **404 log**, **URL rules**, **Pages**, **Site audit** (`gauge`), **Crawls** (`globe`), **robots.txt**, **History** and **Import and export**.
- Navigation item **SEO** (icon `search`), landing on **Redirects**, with the side menu **Redirects**, **404 log**, **URL rules**, **Pages**, **Auto-links** (`wand-sparkles`), **Site audit** (`gauge`), **Crawls** (`globe`), **robots.txt**, **History** and **Import and export**.
- **Site audit** lists the current issues, errors first, with severity, problem, address, details, found by and fix; filters severity, problem, found by, fix and status (Open or Ignored; ignored rows carry the **Ignored** badge). The preview shows what was found, how to fix it (the tool to ask an MCP client for, or the template element) and the page's address, with **Check this address again**, **Ignore** and **Stop ignoring**. The toolbar has **Start crawl**, the list the bulk **Ignore issues**.
- **Crawls** lists the runs newest first (started, status, pages, issues, started by, duration). The preview shows one status callout (running, queued, failed, cancelled, limit reached), the staging and rendered-comparison lines, the stats strip and the issues per kind, each count linking to the filtered Site audit; **Cancel crawl** while queued or running. There is no live progress: reload the page.
- The SEO settings' **Site audit** tab (before Verification): **Automatic crawl** (off, daily, weekly), **Pages per crawl** (10–50 000), **Pause between requests** (100–5000 ms), **Crawls kept** (1–50) and **Generic link texts** (seeded with the English and Polish lists).
- The SEO settings have a **Pagination** tab (after Addresses), a **Links** tab (the new-tab switch and the nofollow and sponsored domain lists, one domain per line; `settings.domain_invalid` names a bad line), a **Publisher** tab with the generated publisher node and its findings, and a **Local business** tab. A person cannot be a local business (`settings.local_person`).
- The SEO settings have a **Pagination** tab (after Addresses), a **Links** tab (**Auto-links per page**, 0 to 20, the new-tab switch and the nofollow and sponsored domain lists, one domain per line; `settings.domain_invalid` names a bad line), a **Publisher** tab with the generated publisher node and its findings, and a **Local business** tab. A person cannot be a local business (`settings.local_person`).
- The SEO tab adds **Primary category** (article types), **Structured data** and its **preview** after the social image; its status warns when a stored primary category is no longer assigned, and the effective values show the schema type and the breadcrumb trail.
- Settings tile **SEO settings** (`seo_settings`), a link to the settings singleton controller.
- Every SEO screen and the SEO tab require `golem15.seo.manage`. Without it an administrator sees no SEO tab, and a posted SEO value changes nothing.
@@ -301,6 +314,8 @@ The element bundle (`assets/dist/seo.js` and `seo.css`) is built from `assets/sr
| `classes.SiteCacheKind` | `seo`, the kind of the site-cache purge after an SEO write. |
| `classes.IssueSitemapBlocked`, `classes.SeverityError` | The `sitemap_blocked` issue kind and its severity. |
| `controllers.PagesControllerID` | The Pages screen. |
| `controllers.AutolinksControllerID`, `controllers.ActionFromKeyphrases`, `controllers.KeyphrasesMessage` | The Auto-links screen, its toolbar action and the action's toast. |
| `classes.AutolinkLink`, `classes.AutolinkWrite`, `classes.AutolinkValues`, `classes.NormalizeAutolink`, `classes.ValidAutolinkURL`, `classes.LoadAutolink`, `classes.AutolinkOrigin`, `classes.MaxPageAutolinks` | An auto-link in the resolve answer, a write, its values, the address check and the origin of a write. Service: `AutolinkProblems`, `SaveAutolink`, `DeleteAutolinks`, `RecordAutolink`, `ForPage`, `FromKeyphrases`. |
| `controllers.IssuesControllerID`, `controllers.CrawlsControllerID`, `controllers.ActionStartCrawl`, `controllers.CrawlActionCancel` | Site audit and Crawls, their toolbar start and the cancel record action. |
| `crawl.Crawl`, `crawl.Options`, `crawl.Result`, `crawl.Page`, `crawl.Fetcher`, `crawl.NewFetcher`, `crawl.Parse`, `crawl.PageFacts` | Package `classes/crawl`: one crawl's frontier and limits, the guarded fetcher and the facts of an HTML document. |
| `crawl.Check`, `crawl.Env`, `crawl.Finding`, `crawl.Route`, `crawl.Fingerprint`, `crawl.Kinds`, `crawl.Severity`, `crawl.Record` | The checks, a finding's fix route (tool or code) and identity, and the kinds with their severities. |
@@ -329,7 +344,7 @@ The element bundle (`assets/dist/seo.js` and `seo.css`) is built from `assets/sr
| `classes.FieldTypes` | The SEO admin field types. |
| `classes.WithVia`, `classes.ViaOf` | The writer history records (`admin`, `mcp`, `import`, `auto`, `undo`). |
| `controllers.PermissionManage` | `golem15.seo.manage`. |
| `models.Entry`, `models.Rule`, `models.Redirect`, `models.NotFound`, `models.Robots`, `models.Settings`, `models.TypeSetting`, `models.Revision`, `models.Page`, `models.Crawl`, `models.CrawlPage`, `models.CrawlLink`, `models.Issue`, `models.IssueIgnore` | The plugin's models. |
| `models.Entry`, `models.Rule`, `models.Redirect`, `models.NotFound`, `models.Robots`, `models.Settings`, `models.TypeSetting`, `models.Revision`, `models.Page`, `models.Autolink`, `models.Crawl`, `models.CrawlPage`, `models.CrawlLink`, `models.Issue`, `models.IssueIgnore` | The plugin's models. |
## Configuration
@@ -377,4 +392,4 @@ npm ci && npm run typecheck && npm test && bash scripts/check-dist.sh
The element tests (`tests/elements`, vitest on happy-dom) drive the widget elements through a fake of the admin's widget host (`tests/elements/host.ts`), including a host that drops an action.
The smoke tests boot the plugin with fake content plugins (`acme.pages`, and `acme.blog` with a category segment for strict URLs) and exercise the SEO tab, the settings tabs, the resolver precedence, history, generation, the URL rule form, redirects (engine, loops and chains, address changes from the admin and from events), the 404 log, normalisation and strict URLs, robots.txt and verification files through the admin and public APIs, and the management API as an MCP client (`TestMCPSetURLSeoTracer`, `TestManagementAPI`, `TestIssues`), batches, undo and CSV (`TestBatch`, `TestUndo`, `TestCSVTransfer`, and the `mcp` and `import` cases of `TestLoopRefused`). The sitemap, the registry and the site-level endpoints are covered by `TestSitemapFilterTracer`, `TestFilterReasons`, `TestNoindexPublishedConflict`, `TestSitemapBlockedIssue`, `TestRegistry`, `TestRouteReport`, `TestSiteEndpoint`, `TestLLMSTxt`, `TestRobotsSitemapLine`, `TestSettingsLinks` and `TestPurgeAfterWrite`; the fixture boots the sitemap plugin next to the SEO plugin. The site audit is covered by `TestCrawlerTracer`, `TestCrawlerChecks` (a planted site with one page per kind and a clean control page, served to the crawler through `fetchguard.WithTransport`), `TestIssueMerge`, `TestStagingCrawl`, `TestRenderSkipped`, `TestCrawlAdmin`, `TestCrawlSchedule` and `TestCrawlTools`, with the fake content plugin `acme.crawl` naming fixing tools. `TestChromeRenderer` renders a page in a real headless Chrome (it runs when `SEO_TEST_CHROME` names a binary or `chromium`, `chromium-browser` or `google-chrome` is on `PATH`, and is skipped otherwise) and `TestRendererGuards` checks the refused paths and private addresses. `TestEveryAdminOperationHasTool`, `TestMCPReadToolsAreGET`, `TestIssueKindLabels` and the static parts of `TestCrawlTools` and `TestCrawlSchedule` run without Docker.
The smoke tests boot the plugin with fake content plugins (`acme.pages`, and `acme.blog` with a category segment for strict URLs) and exercise the SEO tab, the settings tabs, the resolver precedence, history, generation, the URL rule form, redirects (engine, loops and chains, address changes from the admin and from events), the 404 log, normalisation and strict URLs, robots.txt and verification files through the admin and public APIs, and the management API as an MCP client (`TestMCPSetURLSeoTracer`, `TestManagementAPI`, `TestIssues`), batches, undo and CSV (`TestBatch`, `TestUndo`, `TestCSVTransfer`, and the `mcp` and `import` cases of `TestLoopRefused`). The sitemap, the registry and the site-level endpoints are covered by `TestSitemapFilterTracer`, `TestFilterReasons`, `TestNoindexPublishedConflict`, `TestSitemapBlockedIssue`, `TestRegistry`, `TestRouteReport`, `TestSiteEndpoint`, `TestLLMSTxt`, `TestRobotsSitemapLine`, `TestSettingsLinks` and `TestPurgeAfterWrite`; the fixture boots the sitemap plugin next to the SEO plugin. The site audit is covered by `TestCrawlerTracer`, `TestCrawlerChecks` (a planted site with one page per kind and a clean control page, served to the crawler through `fetchguard.WithTransport`), `TestIssueMerge`, `TestStagingCrawl`, `TestRenderSkipped`, `TestCrawlAdmin`, `TestCrawlSchedule` and `TestCrawlTools`, with the fake content plugin `acme.crawl` naming fixing tools. Auto-links are covered by `TestAutolinksTracer` (the phrases of a page's locale in the resolve answer, never the page's own, `autolink_max` 0) and `TestAutolinks` (the screen, its refusals, Add focus keyphrases, the API and MCP). `TestChromeRenderer` renders a page in a real headless Chrome (it runs when `SEO_TEST_CHROME` names a binary or `chromium`, `chromium-browser` or `google-chrome` is on `PATH`, and is skipped otherwise) and `TestRendererGuards` checks the refused paths and private addresses. `TestEveryAdminOperationHasTool`, `TestMCPReadToolsAreGET`, `TestIssueKindLabels` and the static parts of `TestCrawlTools` and `TestCrawlSchedule` run without Docker.