diff --git a/.planning/WINDOWS.md b/.planning/WINDOWS.md index 2f1cac7f..e335715d 100644 --- a/.planning/WINDOWS.md +++ b/.planning/WINDOWS.md @@ -1,10 +1,10 @@ --- schema_version: 1 -open_count: 23 +open_count: 25 waived_count: 0 fixed_count: 8 -total_count: 31 -last_updated: 2026-11-04T11:42:13.158Z +total_count: 33 +last_updated: 2026-11-04T15:03:28.619Z --- # Broken Windows Ledger @@ -46,6 +46,8 @@ last_updated: 2026-11-04T11:42:13.158Z | 29 | 33 | stub | contract/contract.go | | sm-sitemap-plugin: contract.ListingSettings HTMLEnabled, HTMLPaths, HTMLPageSize and KeyFile stay zero and golem15_sitemap_urls.images stays [] until 33-04 adds the HTML sitemap, IndexNow and image settings | fixed | | 2026-11-04T10:26:56.754Z | 2026-11-04T11:41:58.774Z | | 30 | 33 | deviation | models/settings/fields.yaml | | sm-sitemap-plugin: the Additional addresses lastmod is a text field read as YYYY-MM-DD (server-checked, errors.lastmod) because the framework repeater accepts no datepicker column; swap to a datepicker once cabana supports it in repeaters | open | | 2026-11-04T10:26:56.940Z | | | 31 | 33 | deviation | models/settings/fields.yaml | | sm-sitemap-plugin: News sources is a taglist with custom tags and no options (values listed by the news_sources partial, server-checked) because cabana taglist options are a fixed YAML list and form extensions take no taglist; switch to options from the article types and categories once the framework offers per-request taglist options | open | | 2026-11-04T11:42:13.158Z | | +| 32 | 33 | deviation | controllers/issues/config_filter.yaml | | sm-seo-plugin: the Site audit status filter cannot default to Open (cabana list filters have no default value), so the list shows ignored issues too, badged Ignored; GET /_seo/api/v1/issues and seo_list_issues default to status open; set the default once cabana filters accept one | open | | 2026-11-04T15:03:28.372Z | | +| 33 | 33 | deviation | controllers/issues/_issue_details.htm | | sm-seo-plugin: P7 shows Open the page and the Rich Results Test as labelled addresses, not links, because the partial allowlist keeps only same-origin hrefs; make them links once partials may carry an external link (new tab) | open | | 2026-11-04T15:03:28.619Z | | ````json [ @@ -451,6 +453,32 @@ last_updated: 2026-11-04T11:42:13.158Z "recorded_at": "2026-11-04T11:42:13.158Z", "resolved_at": null, "milestone": "v1.0" + }, + { + "id": 32, + "kind": "deviation", + "phase": "33", + "file": "controllers/issues/config_filter.yaml", + "line": null, + "description": "sm-seo-plugin: the Site audit status filter cannot default to Open (cabana list filters have no default value), so the list shows ignored issues too, badged Ignored; GET /_seo/api/v1/issues and seo_list_issues default to status open; set the default once cabana filters accept one", + "status": "open", + "reason": "", + "recorded_at": "2026-11-04T15:03:28.372Z", + "resolved_at": null, + "milestone": "v1.0" + }, + { + "id": 33, + "kind": "deviation", + "phase": "33", + "file": "controllers/issues/_issue_details.htm", + "line": null, + "description": "sm-seo-plugin: P7 shows Open the page and the Rich Results Test as labelled addresses, not links, because the partial allowlist keeps only same-origin hrefs; make them links once partials may carry an external link (new tab)", + "status": "open", + "reason": "", + "recorded_at": "2026-11-04T15:03:28.619Z", + "resolved_at": null, + "milestone": "v1.0" } ] ```` diff --git a/.planning/phases/33-seo-output-audit-and-content/33-10-SUMMARY.md b/.planning/phases/33-seo-output-audit-and-content/33-10-SUMMARY.md new file mode 100644 index 00000000..a582d96d --- /dev/null +++ b/.planning/phases/33-seo-output-audit-and-content/33-10-SUMMARY.md @@ -0,0 +1,365 @@ +--- +phase: 33-seo-output-audit-and-content +plan: 10 +subsystem: seo +status: complete +tags: [sm-seo-plugin, site-audit, crawler, fetchguard, chromedp, conga, mcp, issues, crawls] + +requires: + - phase: 33-seo-output-audit-and-content + provides: "33-01 descriptor facts (Tools, TopHeading, Images, Headings, Words, Parents); 33-05 sitemap filter, registry and the CMS issue report; 33-06 schema.Validate and the chromedp approval (v0.21.0)" +provides: + - "classes/crawl: the crawler (fetchguard client on the site host and its www twin, sitemap index and robots.txt seeds, two workers, settings delay and page limit, cancel), the C23 checks, fix routes, fingerprints, storage and recheck" + - "golem15_seo_crawls, golem15_seo_crawl_pages, golem15_seo_crawl_links, golem15_seo_issues, golem15_seo_issue_ignores (migration 202610280005)" + - "CMS-data kinds image_alt and h1_in_content from the content descriptors (classes.Service.ContentIssues), stored with each crawl as source cms" + - "Site audit (golem15.seo.issues) and Crawls (golem15.seo.crawls) screens with P7 issue_details and P6 crawl_summary" + - "Site audit settings tab (crawl_schedule, crawl_max_pages, crawl_delay, crawl_keep, generic_anchors; migration 202610280006), seo:crawl [--scheduled] and the 02:30 schedule entry" + - "MCP tools seo_start_crawl, seo_cancel_crawl, seo_list_crawls, seo_get_crawl, seo_delete_crawls, seo_list_issues (filters), seo_recheck_url, seo_ignore_issues, seo_unignore_issues" + - "Optional raw-versus-rendered comparison through github.com/chromedp/chromedp v0.21.0 (golem15.seo.crawl_chrome_path, crawl_render_max), guarded by resolver pinning and request interception" +affects: [33-11, 33-12, 33-13, 33-14] + +actuals: + tokens: 81000 + tasks: 4 + commits: 2 +plan_head_before: e245b54ad994554b4b7cf20ee8cc1ad53dd54f3d +plan_head_after: 835bf3f +sm_seo_plugin_head_before: d38049a9561d892b77f541266f9a7ef6d0086e81 +sm_seo_plugin_head_after: b553fc2 +sm_seo_plugin_commits: 8 + +tech-stack: + added: + - "github.com/chromedp/chromedp v0.21.0 (sm-seo-plugin only; with github.com/chromedp/cdproto v0.157.8)" + patterns: + - "A crawl finding names its fix route once (crawl.Route): the record type's fitting tool for content, else a fixed seo_ tool, else fix code with details.what" + - "Issues are keyed by a fingerprint (kind, url, locale, check): ignores survive crawls, a recheck and the pre-crawl CMS refresh keep the row (and id) of an issue found again" + - "The renderer is a seam (crawl.Renderer); tests inject one through a context (crawl.WithRenderer), like fetchguard.WithTransport for the crawler's requests" + +key-files: + created: + - ../sm-seo-plugin/classes/crawl/crawler.go + - ../sm-seo-plugin/classes/crawl/fetch.go + - ../sm-seo-plugin/classes/crawl/parse.go + - ../sm-seo-plugin/classes/crawl/checks.go + - ../sm-seo-plugin/classes/crawl/links.go + - ../sm-seo-plugin/classes/crawl/orphans.go + - ../sm-seo-plugin/classes/crawl/store.go + - ../sm-seo-plugin/classes/crawl/fix.go + - ../sm-seo-plugin/classes/crawl/job.go + - ../sm-seo-plugin/classes/crawl/actions.go + - ../sm-seo-plugin/classes/crawl/render.go + - ../sm-seo-plugin/classes/crawl/render_chrome.go + - ../sm-seo-plugin/classes/crawl/compare.go + - ../sm-seo-plugin/models/crawl.go + - ../sm-seo-plugin/models/crawl_page.go + - ../sm-seo-plugin/models/issue.go + - ../sm-seo-plugin/models/crawl/columns.yaml + - ../sm-seo-plugin/models/crawl/fields.yaml + - ../sm-seo-plugin/models/issue/columns.yaml + - ../sm-seo-plugin/models/issue/fields.yaml + - ../sm-seo-plugin/updates/202610280005_create_golem15_seo_crawls.go + - ../sm-seo-plugin/updates/202610280006_add_golem15_seo_audit_settings.go + - ../sm-seo-plugin/controllers/issues.go + - ../sm-seo-plugin/controllers/crawls.go + - ../sm-seo-plugin/controllers/issues/config_list.yaml + - ../sm-seo-plugin/controllers/issues/config_form.yaml + - ../sm-seo-plugin/controllers/issues/config_filter.yaml + - ../sm-seo-plugin/controllers/issues/_issue_details.htm + - ../sm-seo-plugin/controllers/crawls/config_list.yaml + - ../sm-seo-plugin/controllers/crawls/config_form.yaml + - ../sm-seo-plugin/controllers/crawls/config_filter.yaml + - ../sm-seo-plugin/controllers/crawls/_crawl_summary.htm + - ../sm-seo-plugin/controllers/api/crawls.go + - ../sm-seo-plugin/schedule.go + - ../sm-seo-plugin/crawler_test.go + - ../sm-seo-plugin/crawl_site_test.go + - ../sm-seo-plugin/crawl_admin_test.go + - ../sm-seo-plugin/crawl_render_test.go + - .planning/notes/2026-10-28-chromedp-rendered-crawl.md + modified: + - ../sm-seo-plugin/classes/issues.go + - ../sm-seo-plugin/classes/partials.go + - ../sm-seo-plugin/classes/registry.go + - ../sm-seo-plugin/classes/settings_write.go + - ../sm-seo-plugin/controllers/api/issues.go + - ../sm-seo-plugin/controllers/api/settings.go + - ../sm-seo-plugin/controllers/admin_registry.go + - ../sm-seo-plugin/controllers/settings.go + - ../sm-seo-plugin/models/settings.go + - ../sm-seo-plugin/models/settings/fields.yaml + - ../sm-seo-plugin/admin_navigation.go + - ../sm-seo-plugin/jobs.go + - ../sm-seo-plugin/commands.go + - ../sm-seo-plugin/menu_item_types.go + - ../sm-seo-plugin/routes.go + - ../sm-seo-plugin/mcp.go + - ../sm-seo-plugin/operations.go + - ../sm-seo-plugin/operations_test.go + - ../sm-seo-plugin/seo_tab_test.go + - ../sm-seo-plugin/settings_test.go + - ../sm-seo-plugin/config/config.yaml + - ../sm-seo-plugin/lang/en/lang.yaml + - ../sm-seo-plugin/lang/pl/lang.yaml + - ../sm-seo-plugin/README.md + - ../sm-seo-plugin/plugin.yaml + - ../sm-seo-plugin/go.mod + - ../sm-seo-plugin/go.sum + - .planning/research/STACK.md + +key-decisions: + - "chromedp pinned exactly at v0.21.0 (Jakub 2026-11-04, recorded in 33-06); the decision note was committed before the go.mod change" + - "Chrome's --host-resolver-rules list the site hosts first and MAP * ~NOTFOUND last: Chrome applies the first matching rule, and a leading catch-all made the site host fail with ERR_NAME_NOT_RESOLVED in Chromium (P33-55 had it leading)" + - "One address per host in the resolver rules, IPv4 first (localhost resolving to ::1 first refused the IPv4 test server)" + - "page_status covers errors other than 404/410 (those are broken_link when linked, sitemap_error when listed), so each planted page has exactly one kind" + - "A record's CMS-known links are its primary parent's address (the breadcrumb link): descriptors carry no body links" + - "broken_link, page_status and hreflang_error name seo_create_redirect; sitemap_missing names seo_create_page (seo_set_url_seo for a record page); publisher and site nodes name seo_update_settings" + - "GET /_seo/api/v1/issues and seo_list_issues default to status open (ignored issues left out; ignored or all on request); without ignores the Phase 32 answer is unchanged" + - "Before any crawl the issues list refreshes the stored CMS-data issues on every read; after the first crawl it shows the newest finished, failed or cancelled crawl's issues" + - "Errors-first ordering comes from the Site audit's ListExtendQuery (rank column); the severity column itself is not sortable" + - "List toolbar actions get their own admin operation kind (toolbar_action) in AdminOperations and its parity test" + +patterns-established: + - "Stored issue fingerprints: kind + url + locale + check, sha256 truncated, unique per crawl" + +requirements-completed: [SC-3, D-14, D-46, D-51, D-70, D-71, D-80, D-81, D-82, D-85, D-24, D-50, D-10, D-94] + +coverage: + - id: C1 + description: "Tracer: an administrator starts a crawl through the API, the job fetches the site with the crawler's User-Agent, GET /issues lists h1_multiple as an error, fix code for the template and the record type's headings tool for an H1 in the record's content" + requirement: "D-14" + verification: + - kind: integration + ref: "../sm-seo-plugin/crawler_test.go#TestCrawlerTracer" + status: pass + human_judgment: false + - id: C2 + description: "The planted site: every C23 crawl kind found exactly once at its page with its severity and details, nothing on /clean, no request to another host, a private address or plain http" + requirement: "D-14" + verification: + - kind: integration + ref: "../sm-seo-plugin/crawler_test.go#TestCrawlerChecks" + status: pass + human_judgment: false + - id: C3 + description: "CMS-data issues stored with the crawl as source cms, image_alt and h1_in_content with the descriptor tools, pillar heading_skipped as fix code, every issue routed, ignores surviving the next crawl, fixed pages losing their issues" + requirement: "SC-3" + verification: + - kind: integration + ref: "../sm-seo-plugin/crawler_test.go#TestIssueMerge" + status: pass + human_judgment: false + - id: C4 + description: "Staging: the lock's noindex is never reported and pages stay indexable for the other checks" + requirement: "D-24" + verification: + - kind: integration + ref: "../sm-seo-plugin/crawler_test.go#TestStagingCrawl" + status: pass + human_judgment: false + - id: C5 + description: "Rendered comparison skipped without a renderer, failed with its note when Render fails, the crawl still finishing with its raw checks" + requirement: "D-70" + verification: + - kind: integration + ref: "../sm-seo-plugin/crawler_test.go#TestRenderSkipped" + status: pass + human_judgment: false + - id: C6 + description: "Site audit and Crawls: start and its refusals, errors first, labels and fix column, P7, ignore, row badge, status filter, unignore, bulk ignore, recheck (admin and API), P6 with callouts, the rendered line and kind links, GET crawl render answer, cancel, refused delete of a running crawl" + requirement: "D-14" + verification: + - kind: integration + ref: "../sm-seo-plugin/crawl_admin_test.go#TestCrawlAdmin" + status: pass + human_judgment: false + - id: C7 + description: "Site audit settings ranges, seeded generic anchors driving generic_anchor, seo:crawl --scheduled off/daily/weekly (ScheduledDue), crawl_keep pruning with pages and issues" + requirement: "D-14" + verification: + - kind: integration + ref: "../sm-seo-plugin/crawl_admin_test.go#TestCrawlSchedule" + status: pass + human_judgment: false + - id: C8 + description: "The nine MCP tools bound to their routes, delete destructive, MCP start (trigger mcp), list, ignore and unignore; every Site audit and Crawls operation mapped" + requirement: "D-10" + verification: + - kind: integration + ref: "../sm-seo-plugin/crawl_admin_test.go#TestCrawlTools" + status: pass + - kind: unit + ref: "../sm-seo-plugin/operations_test.go#TestEveryAdminOperationHasTool" + status: pass + human_judgment: false + - id: C9 + description: "Headless Chromium renders a page with JavaScript-added content; the third-party script and the link-local request are blocked; the profile is removed on Close; the crawl reports content_js check rendered naming /js-only with render done" + requirement: "D-70" + verification: + - kind: integration + ref: "../sm-seo-plugin/crawl_render_test.go#TestChromeRenderer" + status: pass + - kind: unit + ref: "../sm-seo-plugin/crawl_render_test.go#TestRendererGuards" + status: pass + human_judgment: false + - id: C10 + description: "Every stored kind has issues.kind_ and issues.sum_ copy in en and pl" + requirement: "SC-3" + verification: + - kind: unit + ref: "../sm-seo-plugin/crawl_admin_test.go#TestIssueKindLabels" + status: pass + human_judgment: false + +duration: 75min +completed: 2026-11-04 +--- + +# Phase 33 Plan 10: Site audit crawler Summary + +**A built-in crawler (a conga job on a fetchguard client limited to the site host) audits the rendered site for every C23 kind, stores the findings with the CMS-data checks under ignorable fingerprints, routes each issue to an MCP tool or to template code, and compares raw with JavaScript-rendered pages through an optional, guarded chromedp v0.21.0 renderer; Site audit and Crawls screens, a Site audit settings tab, a nightly schedule and nine MCP tools sit on top.** + +## Read first: consumers, versions, what is not done + +- **sm-seo-plugin is additive for its consumers.** No exported identifier, route, tool, lang key or table of the plugin was removed or renamed. `GET /_seo/api/v1/issues` keeps its Phase 32 keys (`data[].kind/url/locale/details/tool`, `counts`, `total`) and gains `id`, `severity`, `source`, `fix`, `found_at`, `ignored`, `crawl_id`; it now reads stored issues (the CMS checks are refreshed on each read until the first crawl, then the newest crawl's), and it leaves ignored issues out unless `status=ignored|all`. `details.what` is added to fix-code issues. The settings answer gains five `crawl_*`/`generic_anchors` keys. Two migrations (202610280005, 202610280006) add tables and settings columns. +- **New Go dependency in sm-seo-plugin only:** `github.com/chromedp/chromedp v0.21.0` (+ `github.com/chromedp/cdproto v0.157.8`, `github.com/go-json-experiment/json`). It is compiled into every app that mounts the plugin; Chrome itself is optional and operator-installed. `golang.org/x/net` moved from indirect to direct. The framework go.mod is untouched (D-90). +- **plugin.yaml** (kiosk manifest): the description follows the README summary, and `requires.golem15.sitemap` is now `v0.1.2` — kiosk derives that once chromedp is in go.mod (it derived `""` before); `TestPluginManifest` passes. +- **Apps pin sm-seo-plugin as a submodule**, so sm-demo-app and sm-grzybyfunkcjonalne-app still build the old plugin. A scratch go.work that points sm-demo-app at the sibling sm-seo-plugin and sm-sitemap-plugin builds (`go build ./...`, exit 0). With only sm-seo-plugin swapped in, the demo's pinned sitemap lacks the `contract` package — pre-existing since 33-05, resolved by the 33-14 pointer bumps. +- **Not done here (deferred-items.md, WINDOWS #32, #33):** the Site audit status filter cannot default to Open (cabana filters have no default); P7's "Open the page" and Rich Results lines are addresses, not links (partials allow same-origin links only); the settings tile and permission texts wait for 33-11. +- sm-seo-plugin is 23 commits ahead of v0.1.1 (15 before this plan + 8), nothing pushed or tagged. + +## Performance + +- **Duration:** about 75 min +- **Started:** 2026-11-04T13:50:37Z +- **Completed:** 2026-11-04T15:05Z +- **Tasks:** 4 (all) +- **Files:** 65 changed in sm-seo-plugin (8512 insertions), 2 planning files in summercms.go plus this summary + +## Accomplishments + +- **Crawler** (`classes/crawl`): robots.txt, the sitemap index and its files, then breadth-first from `/` and the sitemap addresses over same-host links, redirect targets, canonicals and hreflang alternates; `fetchguard.AllowHostsMode` on the site host and its `www` twin (HTTPS, private addresses refused, redirects never followed), `TrustedMode` only from `golem15.seo.crawl_trusted`; two requests at a time with the settings' pause, 10 s, 5 MiB, 1000 links per page, the settings' page limit; cancel keeps what was fetched; one crawl at a time (settings row locked in the start transaction); P33-41 User-Agent. +- **Checks**: all C23 crawl kinds with the kinds-table severities, `schema.Validate` for JSON-LD, the raw HTML against the CMS record (`content_js` check cms: title in an H1, half the word count, the primary parent's link), the rendered comparison (`compareRendered`, check rendered), orphan pages with suggestions from titles and focus keyphrases; CMS kinds `image_alt` and `h1_in_content` from the descriptors; staging noindex never counted. +- **Fix routes** (`crawl.Route`): content findings go to the record type's `Tools` (headings for H1, image_alt, edit), data findings to `seo_set_url_seo`, `seo_update_robots`, `seo_create_redirect`, `seo_create_page`, `seo_update_settings`; everything else is `fix: code` with `details.what`. The registry pages name `seo_update_page` as their edit tool. No stored issue has neither. +- **Storage**: crawls, pages (with main text up to 100 000 characters for 33-11's E10), links, issues with fingerprints, ignores; crawls beyond `crawl_keep` pruned with their rows. +- **Admin**: Site audit (errors first, labels, summary and fix columns, five filters, Ignored badge, P7, Check this address again, Ignore, Stop ignoring, bulk Ignore, Start crawl) and Crawls (newest first, P6 with one status callout, staging and render lines, stats strip and per-kind table with filter links, Cancel, refused delete of a running crawl); navigation Site audit (`gauge`) and Crawls (`globe`) after Pages. +- **Settings**: Site audit tab before Verification with the C20 ranges (`lagoon::validation.between.numeric` on the screen and the API) and the en+pl generic anchors seed. +- **Schedule**: `seo:crawl [--scheduled]`, entry `DailyAt(2, 30)`; weekly runs on Mondays in `app.timezone`. +- **MCP**: nine tools with `AdminOperations` rows; reads are GET, delete is destructive. +- **Rendered comparison**: `NewRenderer` from `golem15.seo.crawl_chrome_path` and `crawl_render_max`; `chromeRenderer` with resolver pinning (`fetchguard.IsPrivateAddr`), Fetch-domain interception (`allowRequest`), fresh profile removed on Close, downloads denied, one tab, 20 s, 5 MiB DOM, default allocator flags (no `--no-sandbox`). Verified against the local Chromium. + +## Task Commits + +sm-seo-plugin (`/media/nvme/dev/golem15/summercms.io/summercms/sm-seo-plugin`, branch main, not pushed): + +1. **Task 1 (tracer)** `c2b61bf` feat: crawler finds a page with two H1 headings and stores the issue (tracer) +2. **Task 2A** `f2c93ae` feat: crawler checks for status, sitemaps, canonical, titles and headings +3. **Task 2B** `8c63233` feat: crawler checks for JSON-LD, hreflang, links, images, raw HTML and the rendered-comparison seam +4. **Task 2C** `edc03ca` feat: orphan pages and merged CMS issues with fix routes +5. **Task 3A+C** `c4f0b4f` feat: Site audit and Crawls screens with their MCP tools +6. **Task 3B** `fc53d93` feat: site audit settings and the nightly crawl +7. **Task 3D** `96963c3` docs: crawler and site audit in README +8. **Task 4B** `b553fc2` feat: rendered comparison through headless Chrome + +summercms.go (master): + +- **Task 4A** `e9504b1` docs(33-10): chromedp decision note and STACK row (before the go.mod change) +- `835bf3f` docs(33-10): resolver rule order in the chromedp note + +## Decisions Made + +See `key-decisions` above. The rule-order and IPv4 decisions were checked against Chromium: with `MAP * ~NOTFOUND` first, `TestChromeRenderer` failed with `ERR_NAME_NOT_RESOLVED`; with it last, it passes. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Resolver rule order** +- **Found during:** Task 4 +- **Issue:** P33-55 (and the first version of the decision note) put `MAP * ~NOTFOUND` first; Chrome applies the first matching rule, so the site host itself did not resolve. +- **Fix:** site host rules first, catch-all last; the note corrected in `835bf3f`. +- **Files modified:** `classes/crawl/render.go`, `.planning/notes/2026-10-28-chromedp-rendered-crawl.md` +- **Commit:** `b553fc2`, `835bf3f` + +**2. [Rule 1 - Bug] IPv6-first resolution refused the site** +- **Found during:** Task 4 (`TestChromeRenderer`: `ERR_CONNECTION_REFUSED`) +- **Issue:** `localhost` resolved to `::1` first while the server listened on IPv4. +- **Fix:** the resolver rule takes the first IPv4 address, else the first. +- **Commit:** `b553fc2` + +**3. [Rule 3 - Blocking] Admin screens and MCP tools in one commit** +- **Found during:** Task 3 +- **Issue:** `TestEveryAdminOperationHasTool` fails for any admin operation without its tool, so commit A (screens) cannot be green without commit C (tools). +- **Fix:** groups A and C landed together as `c4f0b4f`; B and D followed. Commit order: A+C, B, D. + +**4. [Rule 3 - Blocking] Renderer seam in the tracer commit** +- The tracer's `crawler.go` already calls the `Renderer` interface, so `render.go` (interface, statuses, `WithRenderer`) landed in `c2b61bf`; `compareRendered` and the wiring of findings landed in 2B as planned. + +**5. [Rule 3 - Blocking] Exported renderer test seam** +- The crawler tests live in the external `seo_test` package, so the fake renderer is injected through `crawl.WithRenderer(ctx, r)`, a context value settable only from Go code (like `fetchguard.WithTransport`), not an unexported option. + +**6. [Rule 3 - Blocking] plugin.yaml followed the README summary** +- `96963c3` changed the README's first sentence, which kiosk's manifest check compares; the README commit was amended (local, unpushed) with the regenerated description so it passes on its own. Task 4 then set `requires.golem15.sitemap: v0.1.2`, what kiosk derives with chromedp in go.mod. + +**7. [Rule 3 - Blocking] Tests following new screens** +- `TestLangKeysComplete` skips value-built `kind_`/`sum_` keys like it skips `via_`/`source_` (`TestIssueKindLabels` checks every kind instead); `TestSettingsTypeTabs` expects the new Site audit tab before Verification. + +**8. [Rule 2 - Missing] `cancelled_by` column** +- P6's cancelled text names who cancelled; P33-37's table had no column for it. + +**9. [Rule 2 - Missing] Recheck keeps the issue's row** +- The record action reloads the preview of the same issue, so a recheck (and the pre-crawl CMS refresh) keeps the rows of issues found again and only deletes or adds the difference. + +### Plan assumptions adapted + +- **Filter default (C23):** cabana filters take no default; the list shows ignored issues badged, the API defaults to open (deferred item, WINDOWS #32). +- **External links in P7:** the partial allowlist drops cross-origin `href`; the page address and the Rich Results URL are shown in DM Mono (deferred item, WINDOWS #33). +- **P6 kind links** filter the Site audit by kind only; the list shows the current crawl and has no crawl filter. +- **Severity sorting:** errors first comes from `ListExtendQuery`'s `ORDER BY rank`; `defaultSort` is `kind` and the severity column is not sortable. +- **CMS links:** descriptors have no body links, so content_js check cms checks the primary parent's address (the breadcrumb link). +- **`compareRendered(raw, rendered *PageFacts, hosts []string)`** takes the site hosts to tell same-host links apart. +- **The seo-page Tools** sit in `classes/registry.go`, where its descriptor is (menu_item_types.go only registers it). +- **Acceptance greps that assume flat lang keys** (`grep -c "issues.kind_"`) count 0 in the nested YAML; each file has 36 `kind_` and 36 `sum_` keys under `issues:` (`TestIssueKindLabels`). +- **Note-before-go.mod check:** the note's first commit (`e9504b1`, ct 1793803438) precedes the first chromedp go.mod commit (`b553fc2`, ct 1793804316); `git log -1` on the note now prints the later correction commit `835bf3f`. + +## Issues Encountered + +- A shell `cp` alias (`cp -i`) left a background experiment waiting for input after it restored `render.go`; the file was restored with Python, the waiting process killed, and the build re-checked. + +## Known Stubs + +None. + +## Threat Flags + +None beyond the plan's threat model: the new routes are management routes behind `golem15.seo.manage`; outbound HTTP goes through fetchguard; Chrome is held to the same hosts and addresses. + +## Verification run + +In `/media/nvme/dev/golem15/summercms.io/summercms/sm-seo-plugin`, after every commit: `go vet ./...` and `go test -short ./...` green; the full Docker suite (`go test ./...`) green before each code commit (about 100 s at the end). + +- `go test ./... -run 'TestCrawlerTracer|TestCrawlerChecks|TestIssueMerge|TestStagingCrawl|TestRenderSkipped|TestIssues' -v`: all PASS. +- `go test ./... -run 'TestCrawlAdmin|TestCrawlSchedule|TestCrawlTools|TestEveryAdminOperationHasTool|TestMCPReadToolsAreGET|TestCrawlerChecks' -v`: all PASS. +- `go test ./... -run 'TestChromeRenderer|TestRenderSkipped|TestCrawlerChecks|TestRendererGuards' -v`: all PASS; `TestChromeRenderer` ran against `/usr/bin/chromium` (not skipped). +- Acceptance greps: `fetchguard.NewClient`/`AllowHostsMode` in fetch.go; `golem15.seo.crawl` in jobs.go and job.go; `crawl_trusted`, `crawl_chrome_path`, `crawl_render_max` once each in config.yaml; `schema.Validate` in checks.go; `func compareRendered(` and `type Renderer interface` once each; eight tool names in mcp.go; `seo:crawl` in schedule.go and commands.go; the two controller IDs in admin_navigation.go; `render_skipped|render_done|render_failed` in both phrasebooks; `host-resolver-rules` and `fetchguard.IsPrivateAddr` in render_chrome.go; no `no-sandbox`; no `crawl_chrome_path` under models/ or settings_write.go; `github.com/chromedp/chromedp` in go.mod, STACK.md and the note. + +`scripts/check-phase33.sh` in summercms.go (`--layers` skipped, known broken since 33-08): + +- `--plugins`: passed (vet and short tests of sm-sitemap-plugin, sm-seo-plugin, sm-journal-plugin; vitest 87/87 in sm-seo-plugin, 244/244 in sm-journal-plugin). +- `--consumers`: passed (the four apps build with their pinned submodules; see "Read first" for the scratch build against this plugin). +- `--self-test --go --admin --docs --marker --docker --additive --compat`: every stage PASS (`docker` ran the three plugins' Postgres suites; `additive`: 4 packages against v0.6.0). `compat` printed refusals for sm-feedback-plugin, sm-presentations-plugin and sm-quizzes-plugin (their own replace layout resolves the framework to nothing, at the base tag too) and passed. + +## User Setup Required + +None for the crawler. For the rendered comparison an operator installs Chrome or Chromium on the server and sets `golem15.seo.crawl_chrome_path` (absolute path); a staging or local site over http needs `golem15.seo.crawl_trusted: true`. + +## Next Phase Readiness + +- 33-11 can read `golem15_seo_crawl_pages.text` (main text, 100 000 characters) for E10 suggestions, add Auto-links between Pages and Site audit in the navigation, and replace the settings tile and permission texts. +- 33-14: bump the sm-seo-plugin pointer in the apps together with sm-sitemap-plugin (the demo's pinned sitemap lacks `contract`), tag the plugin, and note the chromedp dependency and the two migrations in the release notes. + +## Self-Check: PASSED + +- Files: every created file listed above exists in sm-seo-plugin or summercms.go. +- Commits: `c2b61bf`, `f2c93ae`, `8c63233`, `edc03ca`, `c4f0b4f`, `fc53d93`, `96963c3`, `b553fc2` in sm-seo-plugin; `e9504b1`, `835bf3f` in summercms.go. diff --git a/.planning/phases/33-seo-output-audit-and-content/deferred-items.md b/.planning/phases/33-seo-output-audit-and-content/deferred-items.md index 18b420ba..6ffe870a 100644 --- a/.planning/phases/33-seo-output-audit-and-content/deferred-items.md +++ b/.planning/phases/33-seo-output-audit-and-content/deferred-items.md @@ -59,3 +59,18 @@ Out-of-scope discoveries logged during execution; not fixed in the plan that fou status: open - **Possible fix:** allow `trigger` (hide/disable) on repeater fields in cabana and the SPA, then add it to `local_hours` in `sm-seo-plugin/models/settings/fields.yaml`. - **Why deferred:** 33-06 commits only to sm-seo-plugin; the gap is generic, additive framework work. + +## From 33-10 (sm-seo-plugin) + +- **A list filter cannot have a default value (framework).** UI-SPEC C23 asks the Site audit's status filter to default to Open so ignored issues stay hidden until chosen. cabana's `config_filter.yaml` scopes take no default and `ListExtendQuery` does not see the filter values, so the list shows every current issue and marks ignored ones with the **Ignored** row badge; choosing **Open** hides them. GET `/_seo/api/v1/issues` and `seo_list_issues` do default to `status=open`. WINDOWS.md #32. + status: open + - **Possible fix:** a `default` key on filter scopes in cabana and the SPA, then `default: open` on `status` in `sm-seo-plugin/controllers/issues/config_filter.yaml`. + - **Why deferred:** 33-10 commits only to sm-seo-plugin; the gap is generic, additive framework work. +- **A partial cannot link to another site (framework).** P7 should link **Open the page** (new tab) and **Check in Google's Rich Results Test**. The partial allowlist keeps `a[href]` only for same-origin paths, and the site is on another origin than the admin, so P7 shows both as labelled addresses in DM Mono. WINDOWS.md #33. + status: open + - **Possible fix:** let partials carry an absolute http(s) link that the SPA opens in a new tab with `rel="noopener noreferrer"` (the account contributions already accept such URLs), then turn the two lines into links. + - **Why deferred:** a change to cabana's sanitizer and the SPA, outside 33-10's repository. +- **The settings tile description and the permission comment still have the Phase 32 text (sm-seo-plugin).** UI-SPEC's replacement texts name auto-links, which arrive in 33-11, together with the site audit. + status: open + - **Possible fix:** 33-11 replaces `settings.description` and `permissions.manage_comment` in both phrasebooks with the UI-SPEC texts once Auto-links exists. + - **Why deferred:** the new texts describe a screen that does not exist yet.