golem15/summercms
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(32-04): complete redirects, 404 log and robots.txt plan

Browse Source
This commit is contained in:
Jakub Zych
2026-10-27 11:56:20 +01:00
parent 99889f0bcf
commit 8800137d16
1 changed files with 402 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

402
.planning/phases/32-seo-foundation/32-04-SUMMARY.md Normal file
View File

@@ -0,0 +1,402 @@
---
phase: 32-seo-foundation
plan: 04
subsystem: sm-seo-plugin
tags: [seo, sm-seo-plugin, redirects, 404, robots.txt, grobotstxt, normalisation, strict-urls, custom-elements]
requires:
- phase: 32-01
provides: pact.AdminFormExtension (yaml-document form, Save notices, AdminFormErrors, field-less extensions, extension widget fills)
- phase: 32-02
provides: pact.RecordSaved with Changes, cabana.EmitRecordSaved, signpost Describe/MatchPath/ContentTypeForModel
- phase: 32-03
provides: sm-seo-plugin core (classes.Service, Resolver, entries with URL snapshots, history, listeners, routes, settings address and verification columns, element bundle)
provides:
- "golem15_seo_redirects with the redirect engine (301/302/410; exact, pattern, regex; ignore case, trailing slash, query ignore/pass), save-time loop refusal and chain notices, runtime 10-hop cap"
- "AddressChanged: the automatic 301 on an address change with chain flattening, in the SEO tab save transaction and after commit for writes outside the admin"
- "golem15.seo.redirects and golem15.seo.notfound screens; POST /_seo/api/v1/notfound with bearer token"
- "Normalize, strict URLs and the one-hop redirect decision in GET /_seo/api/v1/resolve (actions redirect and gone)"
- "golem15.seo.robots singleton with the robotstxt field type (E4) and the robots_test tester widget (E5) on grobotstxt; GET /robots.txt with AI bot groups, Sitemap lines and the non-production lock"
- "GET /_seo/api/v1/verification/{name}"
- "Decision note .planning/notes/2026-10-27-robots-txt-grobotstxt.md and a STACK.md row"
affects: [32-05, 32-06, 33, vue-seo-layer, sm-sitemap-plugin]
actuals:
tokens: 60000
tasks: 4
commits: 1
plan_head_before: 930d91a3c1df3da828c5108e984ffae700708635
plan_head_after: 99889f0fe82c5cbdda0bd1b9e4185c49db8dd9fd
tech-stack:
added:
- "github.com/jimsmart/grobotstxt v1.0.3 (sm-seo-plugin only; approved at the 32-03 package gate; decision note and STACK row)"
patterns:
- "One service path for redirect writes: SaveRedirect, AddressChanged, DeleteRedirect, ResolveNotFound run validation, the loop simulation and history inside the caller's transaction"
- "Field-less self-extension on the plugin's own controller to refuse with AdminFormErrors and answer save notices from inside the save transaction (redirects)"
- "A form extension on the plugin's own singleton to carry fields an owner-form widget may not fill (robots.txt editor and bot switches for the tester)"
- "Resolve as a pipeline: normalisation, redirects followed to the final target, strict URLs, then SEO; one hop"
key-files:
created:
- ../sm-seo-plugin/classes/redirects.go
- ../sm-seo-plugin/classes/redirect_graph.go
- ../sm-seo-plugin/classes/addresses.go
- ../sm-seo-plugin/classes/notfound.go
- ../sm-seo-plugin/classes/normalize.go
- ../sm-seo-plugin/classes/strict.go
- ../sm-seo-plugin/classes/robots.go
- ../sm-seo-plugin/classes/bots.go
- ../sm-seo-plugin/classes/verification.go
- ../sm-seo-plugin/controllers/redirects.go
- ../sm-seo-plugin/controllers/notfound.go
- ../sm-seo-plugin/controllers/robots.go
- ../sm-seo-plugin/controllers/api/notfound.go
- ../sm-seo-plugin/controllers/api/robots.go
- ../sm-seo-plugin/controllers/api/verification.go
- ../sm-seo-plugin/extensions_robots.go
- ../sm-seo-plugin/models/redirect.go
- ../sm-seo-plugin/models/notfound.go
- ../sm-seo-plugin/models/robots.go
- ../sm-seo-plugin/updates/202610270006_create_golem15_seo_redirects.go
- ../sm-seo-plugin/updates/202610270007_create_golem15_seo_notfound.go
- ../sm-seo-plugin/updates/202610270008_create_golem15_seo_robots.go
- ../sm-seo-plugin/assets/src/elements/robotstxt.ts
- ../sm-seo-plugin/assets/src/elements/robotstester.ts
- ../sm-seo-plugin/redirects_test.go
- ../sm-seo-plugin/addresses_test.go
- ../sm-seo-plugin/robots_test.go
- .planning/notes/2026-10-27-robots-txt-grobotstxt.md
modified:
- ../sm-seo-plugin/classes/resolver.go
- ../sm-seo-plugin/classes/partials.go
- ../sm-seo-plugin/classes/field_types.go
- ../sm-seo-plugin/classes/service.go
- ../sm-seo-plugin/extensions.go
- ../sm-seo-plugin/listeners.go
- ../sm-seo-plugin/routes.go
- ../sm-seo-plugin/admin_navigation.go
- ../sm-seo-plugin/controllers/admin_registry.go
- ../sm-seo-plugin/config/config.yaml
- ../sm-seo-plugin/lang/en/lang.yaml
- ../sm-seo-plugin/lang/pl/lang.yaml
- ../sm-seo-plugin/README.md
- ../sm-seo-plugin/go.mod
- ../sm-seo-plugin/assets/dist/seo.js
- ../sm-seo-plugin/assets/dist/seo.css
- .planning/research/STACK.md
key-decisions:
- "Redirect statuses are models.RedirectStatus (an int with UnmarshalText): the admin dropdown posts \"301\" as text and lagoon.Fill cannot fill an int from a string"
- "Loop detection refuses a repeated URL, a rule that fires twice, or reaching 10 hops; pattern probes replace * with seoprobe, regex probes are built from regexp/syntax literals"
- "Exact redirects are read per request by lower(source) (functional index added to the unreleased redirects migration); pattern and regex rules are loaded each request, their expressions compiled once per source"
- "A capture is percent-encoded (?, #, \\, %, non-ASCII) before substitution; a relative target collapses leading slashes; an absolute target must keep its literal's scheme and host, else the rule does not match"
- "A redirect chain that ends on a 410 answers gone for every source in it; a 302 anywhere in a chain makes the answer 302"
- "Strict URLs compare the normalised request path with the record's normalised Describe URL in the matched locale; MatchPath gets the path without a trailing slash, so records match under url_slash add"
- "Normalisation uses the request URL's scheme and host only when the host is the site URL's host or its www twin; any other host is ignored"
- "The robots.txt editor, tester and AI bot switches are fields of the plugin's own form extension on golem15.seo.robots (an owner-form widget may fill built-in scalar fields only); the owner form keeps the staging callout and P4"
- "Bot switch labels are \"<token> (<operator>)\" and their comments carry no :operator, because fields.yaml copy takes no parameters"
- "Record action Confirm texts take no parameters and the SPA shows a bulk Confirm as is, so notfound.redirect_confirm, gone_confirm and gone_selected_confirm name \"this address\" / \"the selected addresses\" instead of :url / :count"
- "The last production robots.txt body is cached on the service and served when the store fails; without one the body is allow-all; always 200"
- "Verification files are served with nosniff and Content-Security-Policy default-src 'none'; sandbox"
patterns-established:
- "Service methods return typed refusals (ProblemsError, LoopError, NotFoundHasRuleError) that each caller maps: 422 field lines on a form, 403 toast messages on record and bulk actions"
requirements-completed: [SC-3, D-16, D-17, D-18, D-19, D-20, D-20a, D-20b, D-20c, D-22, D-24, D-25, D-05, D-08, D-06, D-99, D-103, D-105, D-92, D-102]
coverage:
- id: D1
description: "Tracer: an admin slug change creates the 301 in the save transaction, flattens an earlier redirect to the old address, answers redirects.auto_created on slug, and resolve redirects the old address to the absolute new one"
requirement: "SC-3"
verification:
- kind: integration
ref: "../sm-seo-plugin/redirects_test.go#TestSlugChange301"
status: pass
human_judgment: false
- id: D2
description: "Redirect engine: exact, pattern and regex sources with ignore case, trailing slash and query ignore/pass, order exact > pattern > regex, 410 gone, field refusals, loop refusal (exact, pattern and regex probes), chain checks, slug back and forth ending with A→C and B→C, open-redirect safety"
requirement: "D-17"
verification:
- kind: integration
ref: "../sm-seo-plugin/redirects_test.go#TestRedirectEngine"
status: pass
human_judgment: false
- id: D3
description: "Redirects screen: a loop is refused with redirects.loop on target and nothing saved (create and update); a chain is saved with redirects.chain_notice and marked negative in the list; target_required and regex_invalid answer 422; history rows"
requirement: "D-19"
verification:
- kind: integration
ref: "../sm-seo-plugin/redirects_test.go#TestLoopRefused"
status: pass
human_judgment: false
- id: D4
description: "404 log: the bearer-token report endpoint (401, 404 when off, 422 over 2048 bytes), hits, first seen, last hit and referer; the prompted redirect and 410 record actions, Resolved rows, resolved hint, bulk 410 refused for a selection with a covered address, open filter"
requirement: "D-20"
verification:
- kind: integration
ref: "../sm-seo-plugin/redirects_test.go#TestNotFoundLog"
status: pass
human_judgment: false
- id: D5
description: "Writes outside the admin: RecordSaved without a controller creates the 301 after commit (via auto); no change and unpublished records create nothing; an admin save that already applied it changes nothing more"
requirement: "D-102"
verification:
- kind: integration
ref: "../sm-seo-plugin/redirects_test.go#TestAddressChangeFromEvent"
status: pass
human_judgment: false
- id: D6
description: "URL normalisation as a pure function: https, www/bare, double slashes, index files, trailing slash add/remove (root and files kept), ASCII-only lowercasing with Polish letters and percent-encoding kept"
requirement: "D-20b"
verification:
- kind: unit
ref: "../sm-seo-plugin/addresses_test.go#TestNormalize"
status: pass
human_judgment: false
- id: D7
description: "Strict URLs: a wrong category or another locale's slug 301s to the record's address, queries kept; only the record's address answers ok"
requirement: "D-20c"
verification:
- kind: integration
ref: "../sm-seo-plugin/addresses_test.go#TestStrictURL"
status: pass
human_judgment: false
- id: D8
description: "One-hop redirect decision: a non-normalised URL whose normal form is a redirect source (and a chain) answers one 301 to the final normalised target; a runtime cycle answers not_found; P1 shows status.canonical_redirected"
requirement: "D-20a"
verification:
- kind: integration
ref: "../sm-seo-plugin/addresses_test.go#TestOneHop"
status: pass
human_judgment: false
- id: D9
description: "GET /robots.txt: stored rules, AI bot groups and the Sitemap line in production; the lock with X-Robots-Tag outside production whatever is stored; 200 with the last good body or allow-all when the store fails"
requirement: "D-22"
verification:
- kind: integration
ref: "../sm-seo-plugin/robots_test.go#TestRobotsServed"
status: pass
human_judgment: false
- id: D10
description: "Tester on grobotstxt against Google's documented examples (longest match, Allow wins a tie, $ anchor, group choice) with the deciding line; a switched-off bot is reported as blocked by its switch"
requirement: "D-99"
verification:
- kind: integration
ref: "../sm-seo-plugin/robots_test.go#TestRobotsTester"
status: pass
human_judgment: false
- id: D11
description: "robots.txt screen through the admin API: field order, save of rules and switches with history, robots.blocks_noindex notice, 500 KB refusal, tester action on unsaved values incl. Claude-User, served-file partial"
requirement: "D-24"
verification:
- kind: integration
ref: "../sm-seo-plugin/robots_test.go#TestRobotsScreen"
status: pass
- kind: other
ref: "npm --prefix ../sm-seo-plugin run typecheck && bash ../sm-seo-plugin/scripts/check-dist.sh"
status: pass
human_judgment: false
- id: D12
description: "Verification files through /_seo/api/v1/verification/{name} with content type, nosniff, sandbox CSP and noindex; unknown, malformed and root paths 404"
requirement: "D-05"
verification:
- kind: integration
ref: "../sm-seo-plugin/robots_test.go#TestVerificationFile"
status: pass
human_judgment: false
- id: D13
description: "E4 editor and E5 tester, the redirects and 404 log lists (badges, filters, prompted action, danger confirm) and the robots.txt screen look and behave as UI-SPEC C8-C10 in the real admin SPA"
verification: []
human_judgment: true
rationale: "Visual and interaction behaviour inside the SPA; only a throwaway happy-dom smoke of E4/E5 ran here, element unit tests are plan 32-06"
duration: 43min
completed: 2026-10-27
status: complete
---
# Phase 32 Plan 04: Redirects, 404 log, URL rules and robots.txt Summary
**Redirect engine with save-time loop refusal and chain notices, the automatic 301 on an address change (inside the admin save and after commit for API writes), the 404 log with a token-protected report endpoint, URL normalisation and strict URLs folded into a one-hop redirect decision in resolve, and robots.txt with a grobotstxt tester, eleven AI bot switches, a non-production lock that never answers 5xx, and verification files served through the API.**
## Performance
- **Duration:** 43 min
- **Started:** 2026-10-27T10:11:34Z
- **Completed:** 2026-10-27T10:54:32Z
- **Tasks:** 4 (Task 1 tracer, Task 2 in four commit groups, Task 3, Task 4 in five commit groups)
- **Files:** 64 files changed in ../sm-seo-plugin, 2 in this repository
## Accomplishments
- **Redirects (D-16, D-17, D-19).**
- Matching: exact, pattern and regex sources, ignore case, trailing slash, query ignore/pass, and a source with `?` that matches its query exactly (Q-9).
- Safety: stdlib RE2 regexes capped at 512 characters, and captures that cannot leave the site.
- Save-time simulation: a loop is refused and a chain is saved with a notice.
- Runtime: chains are followed to the final target, and a cycle stops after 10 hops.
- **Address changes (D-18, D-92, D-102, P-20).** `AddressChanged` runs inside the SEO tab save with the `redirects.auto_created` notice on the slug field. A `RecordSaved` listener does the same after commit for writers outside the admin. Unpublished and deleted records get no redirect.
- **Redirects screen (C8).** Six columns with the Chain badge, the status/match/origin/chain filters, and the navigation item landing here.
- **404 log (D-20, P-22, C9).**
- Report endpoint `POST /_seo/api/v1/notfound`: bearer token compared in constant time, off when unset, URL and referer caps, a row cap with an hourly warning.
- List by hits, with Resolved rows and the resolved hint.
- Actions: a prompted 301, a danger 410, and a bulk 410 that refuses the whole selection when one row is covered.
- **Redirect decision (D-20a, D-20b, D-20c, D-103, P-21).** Resolve now runs normalisation, then redirects, then strict URLs, then SEO. It answers `redirect` with an absolute `location` (one hop) or `gone`. P1 warns when a canonical URL redirects (D-06).
- **robots.txt (D-22, D-24, D-25, D-99, D-105, P-23, C10).**
- Field type `robotstxt` with element E4 (live lint, size meter).
- Tester E5 on `robots_test`. It works on unsaved text and switches.
- AI bots tab with eleven switches, including `Claude-User`.
- A `robots.blocks_noindex` save notice.
- `GET /robots.txt`: rules, bot groups and Sitemap lines. Outside production it serves the lock with `X-Robots-Tag`. It never answers 5xx.
- **Verification files (D-05, D-103).** `GET /_seo/api/v1/verification/{name}`. There is no root route.
- **Planning docs.** Decision note `2026-10-27-robots-txt-grobotstxt.md` and a STACK.md row.
## Task Commits
../sm-seo-plugin, branch main, not pushed:
1. **Task 1 (tracer):** `e31ac7c` feat: address-change 301 (tracer)
2. **Task 2:**
- A `34a65b8` feat: redirect engine with loop refusal and chain notices
- B `163b0aa` feat: redirects screen
- C `6eb6cb7` feat: 404 log and report endpoint
- D `327925c` feat: address-change redirects for writes outside the admin
3. **Task 3:** `4cb0fbf` feat: URL normalisation, strict URLs and the one-hop redirect decision
4. **Task 4:**
- G1 (this repository) `99889f0` docs(32-04): grobotstxt decision note
- G2 `3dd9362` feat: robots.txt served file, AI bots and the non-production lock
- G3 `51413fb` feat: robots.txt editor and tester screen
- G4 `ec74062` feat: verification files API
- G5 `b398161` docs: robots.txt, AI bots and verification files in README
Commit counts:
- ../sm-seo-plugin: `git rev-list --count b38eb4b..HEAD` = 10.
- This repository before this SUMMARY's commit: `git rev-list --count 930d91a..HEAD` = 1 (G1), which is the frontmatter's `commits`.
Tracer feedback gate: the run was interactive, in end-of-phase mode, with an automated-only verify. TestSlugChange301, vet and the `-short` tests re-ran green before Task 2.
## TDD Gate Compliance
Tasks 2 and 3 are marked `tdd="true"`, and `workflow.tdd_mode` is false.
- **Commits:** the plan defines one commit per resume unit, with the tests included, so there are no separate `test(...)` RED commits.
- **RED evidence:** none. The tests were written next to the implementation, not before it. Only TestAddressChangeFromEvent failed first, and that was on the test's own transaction: `EmitRecordSaved` needs `lagoon.Transaction`.
## Files Created/Modified
See `key-files` in the frontmatter.
- Classes: `classes/redirects.go`, `redirect_graph.go` and `addresses.go` hold the engine, the simulation and the address change. `normalize.go` and `strict.go` hold the redirect-decision stages, and `robots.go`, `bots.go` and `verification.go` the robots.txt side.
- Extension: `extensions_robots.go` builds the robots form extension. Its document is generated from `classes.Bots`.
## Decisions Made
See `key-decisions` in the frontmatter.
## Deviations from Plan
### Auto-fixed Issues
**1. [Rule 3 - Blocking] Redirect status type**
- **Found during:** Task 2B
- **Issue:** the admin dropdown posts `"301"` as text, and `lagoon.Fill` cannot fill an `int` from a string.
- **Fix:** `models.RedirectStatus` (an int with `UnmarshalText`). The SPA compares option values as strings, so `show` matches.
- **Commit:** 163b0aa
**2. [Rule 3 - Blocking, framework constraint] The tester needs unsaved editor text**
- **Found during:** Task 4 G3
- **Issue:** an owner-form widget may fill built-in scalar fields only, so the tester could not fill a `robotstxt` field from the owner form.
- **Fix:** the editor, the tester and the bot switches are fields of the plugin's own form extension on `golem15.seo.robots`. An extension widget may fill its own value fields, plugin types included.
- Load and Save read and write `golem15_seo_robots` (`content`, `bots` jsonb), so the plan's table shape is kept.
- The owner `fields.yaml` holds the staging callout and P4.
- **Files outside the plan's list:** `extensions_robots.go` (new), `extensions.go`.
- **Commit:** 51413fb
**3. [Framework constraint] Copy without parameters**
- **Record action confirms:** `Confirm` is translated without parameters, so `notfound.redirect_confirm` and `notfound.gone_confirm` say "this address" instead of “:url”.
- **Bulk confirm:** the SPA shows a bulk `Confirm` as is, with no plural and no `:count`, so `notfound.gone_selected_confirm` is one plain sentence.
- **Bot switches:** `fields.yaml` comments take no parameters, so the labels are "<token> (<operator>)" and `bots.*_comment` has no `:operator`.
- **Commits:** 6eb6cb7, 51413fb
**4. [Plan detail] Lowercased-source index added to the redirects migration**
- `202610270006` (committed in Task 1, unreleased, not pushed) gained `CREATE INDEX ... (lower(source))` in group A. The resolver reads exact rules by that key.
- **Commit:** 34a65b8
**5. [Plan detail] resolver.go changed outside Task 3's commit**
- `classes/resolver.go` changed in the Task 1 and Task 2 A/B commits, so each commit compiles with the engine of that commit.
- **Commits:** e31ac7c, 34a65b8, 163b0aa
**6. [Plan detail] Extra files and keys**
- Filters: `controllers/redirects/config_filter.yaml` and `controllers/notfound/config_filter.yaml`.
- Classes:
- `classes/partials.go` gained the canonical-redirected callout. `extension/_seo_status.htm` needed no change, because callouts render generically.
- `classes/service.go` caches the last good robots.txt.
- Test: `TestRobotsScreen`, an admin-path smoke test.
- Lang keys not in the UI-SPEC:
- `redirects.target_invalid`: a target that is neither a path nor an http(s) URL;
- `robots.test_invalid` and `robots.served_production` (P4's "Production" heading);
- `robots.staging_text` reuses the UI-SPEC copy.
**7. [Plan detail] Strict URLs and trailing slashes**
- The strict check gives `MatchPath` the path without a trailing slash, so records still match when the site adds slashes.
- With the default `keep`, `/pages/a/` therefore 301s to the record's address `/pages/a` (before this plan it was `not_found`).
**8. [Plan detail] Filters versus badges**
- The 404 log's open/resolved filter runs in SQL over exact rules only. The Resolved badge and the actions use the full rule set, patterns and regex included.
- The redirects Chain badge and the chain filter compare the target with another rule's source literally.
**9. [Plan detail] README timing**
- The plan's G5 grouping moved the README updates for redirects, the 404 log and normalisation into the docs commit. They did not land in each feature commit.
---
**Total deviations:** 9. Two were blocking, and one is a framework copy constraint; the other six are plan details.
**Impact on plan:** behaviour is as planned, and copy changes are limited to the confirm texts and the bot comments.
**Framework candidates:**
- parameters in record-action and bulk confirms;
- parameters in fields.yaml comments;
- owner-form widget fill of plugin field types.
## Issues Encountered
None blocking:
- File permissions in summercms.go were fixed before this run, so this SUMMARY and the planning commit were written in place.
- summercms.go `go vet ./...` and `go test -short ./...` are green.
## Known Stubs
None.
- MCP tools, batch writes, undo and CSV import of redirects are plan 32-05; they reuse `SaveRedirect` and `AddressChanged`.
- Element unit tests are plan 32-06.
- The Nuxt-side enforcement of the redirect decision and of verification files is Phase 33 (D-103).
## User Setup Required
None. To feed the 404 log, a site sets `golem15.seo.notfound_token` and lets its frontend server report 404s; without a token the endpoint stays off.
## Next Phase Readiness
- Plan 32-05 builds on `SaveRedirect`, `CheckRedirect`, `DeleteRedirect`, `AddressChanged`, `ResolveNotFound` and the typed refusals. History is already recorded for redirect and robots writes.
- Phase 33 takes the resolve `redirect`/`gone` actions, the verification API and `robots_sitemaps`. The sitemap rewrite replaces the default.
- Nothing pushed: ../sm-seo-plugin `main` has no upstream.
## Self-Check: PASSED
- **Commits present:**
- ../sm-seo-plugin: e31ac7c, 34a65b8, 163b0aa, 6eb6cb7, 327925c, 4cb0fbf, 3dd9362, 51413fb, ec74062, b398161;
- this repository: 99889f0.
- **Key files:** all exist.
- **../sm-seo-plugin:**
- `go vet ./...`, `go test -short ./...` and `go test -count=1 ./...` (Docker) are green. The Docker run includes TestSlugChange301, TestRedirectEngine, TestLoopRefused, TestNotFoundLog, TestAddressChangeFromEvent, TestNormalize, TestStrictURL, TestOneHop, TestRobotsServed, TestRobotsTester, TestRobotsScreen, TestVerificationFile and TestLangKeysComplete.
- `npm run typecheck` and `scripts/check-dist.sh` are green.
- **summercms.go:** `go vet ./...` and `go test -short ./...` are green.
- **Acceptance greps:**
- `AddressChanged(` appears in `extensions.go` and `listeners.go`; `*pact.RecordSaved` appears once in `listeners.go`.
- `subtle.ConstantTimeCompare` appears in `controllers/api/notfound.go`.
- There is no `regexp.MustCompile(.*source`, dlclark or regexp2.
- grobotstxt v1.0.3 is in the plugin's go.mod and not in the framework's.
- `Claude-User` appears once in `classes/bots.go`.
- `routes.go` has no `Get("/{`.
---
*Phase: 32-seo-foundation*
*Completed: 2026-10-27*