Compare commits

..

10 Commits

Author SHA1 Message Date
Jakub Zych
3198bbe1aa docs(quick-261108-pe4): beachcomber highlights and field projection 2026-11-08 18:45:54 +01:00
Jakub Zych
7819c6746c feat(beachcomber): search highlights and field projection (quick-261108-pe4)
- Additive: Query.IncludeFields, ExcludeFields and Highlight; left empty,
  the Typesense request is the same as before.
- SearchResult.Hits holds one Hit per id, in step with IDs; SearchPage
  always fills it.
- Snippets are plain-text Segment runs split at private-use highlight tags,
  never HTML; MaxSnippetsPerArray is applied client-side.
- Optional HighlightSearcher capability, checked with SupportsHighlights
  (true for Typesense, false for null and third-party engines).
- README and docs: Highlights sections with the rule that a caller drops
  the hit of every id failing the SQL re-check before showing a snippet.

Targets framework v0.6.4 (additive, no breaking change).
2026-11-08 18:45:29 +01:00
Jakub Zych
6f9b77033f docs(35): create phase plan (4 plans, 4 waves) 2026-11-06 19:43:03 +01:00
Jakub Zych
a0242808a0 docs(35): pattern map and planning-time decisions D-22..D-25 2026-11-06 19:10:23 +01:00
Jakub Zych
9232dc1048 docs(35): research and validation strategy for the mail templates editor 2026-11-06 19:05:34 +01:00
Jakub Zych
a917966d31 docs(quick-261106-kt2): in-process deferred:purge, migrate and queue:clear reuse the published database 2026-11-06 15:38:37 +01:00
Jakub Zych
99abe88b6a fix(conga): queue:clear reuses the app's published database (quick-261106-kt2)
Run in-process on an app that already published its pool, queue:clear failed
with "backpack: duplicate provider for *sql.DB". It now goes through
withAppDB; the CLI path is unchanged.
2026-11-06 15:28:08 +01:00
Jakub Zych
16a20088a3 fix(lagoon): run deferred:purge and the migrate commands on the app's published database (quick-261106-kt2)
The nightly framework entry deferred:purge failed under serve's worker and
schedule:run with "backpack: duplicate provider for *sql.DB": withDB always
opened and published a second pool. It now reuses the published *gorm.DB and
leaves that pool open; the CLI path (nothing published) is unchanged.
2026-11-06 15:23:28 +01:00
Jakub Zych
8ebfedf64c docs(quick-261106-kt1): MCP 401 challenge names only resource_metadata, T-19.7-10 updated 2026-11-06 15:15:03 +01:00
Jakub Zych
609caa4f3f fix(pier): MCP 401 challenge names only resource_metadata
- drop the response-writer wrapper that appended scope="read" to every 401
- the SDK handler now gets the original writer; RequireBearerToken still sets no Scopes
- TestPierTracerSmoke pins the exact challenge for a bad and a missing bearer
- README and docs/services/mcp.md describe the header without a scope parameter
2026-11-06 15:09:10 +01:00
36 changed files with 5033 additions and 80 deletions

View File

@@ -1870,7 +1870,17 @@ Plans:
**Requirements**: TBD
**Depends on:** Phase 34
**Context:** postcard already loads plugin templates and layouts from `views/mail/` (`pact.HasMailTemplates`, `postcard.BootPlugin`); this phase adds the editable database layer, admin screens and image handling on top. To confirm in discussion: partials (WinterCMS mail partials), preview and test send, per-locale variants, and whether the screens live in cabana or in a plugin.
**Plans:** 0 plans
**Plans:** 4 plans
Plans:
- [ ] TBD (run /msd-plan-phase 35 to break down)
**Wave 1**
- [ ] 35-01-PLAN.md — pact `HasMailPartials`, postcard-owned mail tables and migrations, per-send override lookup with file fallback, partials, sample vars, Branding CSS, index sync, Editor API, docs
**Wave 2** *(blocked on Wave 1 completion)*
- [ ] 35-02-PLAN.md — cabana Settings → Mail screens, two permission codes, reset/delete rules, drift notice and diff, server field types, preview and own-address test send, OpenAPI, en/pl copy
**Wave 3** *(blocked on Wave 2 completion)*
- [ ] 35-03-PLAN.md — admin SPA: CodeMirror decision note and checkpoint, codeeditor/colour/image fields, image picker, mail preview and test send, rebuilt dist
**Wave 4** *(blocked on Wave 3 completion)*
- [ ] 35-04-PLAN.md — unit tests for the D-21 list and `scripts/check-phase35.sh` (incl. user and paymentgateway mail tests)

View File

@@ -32,7 +32,7 @@ See: .planning/PROJECT.md (updated 2026-10-22)
Phase: 19.7 (Generic MCP module (INSERTED)) — EXECUTING
Plan: 5 of 5
Status: Phase complete — ready for verification
Last activity: 2026-10-23 — Phase 19.7 execution started
Last activity: 2026-11-08 - Completed quick task 261108-pe4: beachcomber highlights and field projection
Progress: [████████░░] 82%
@@ -1071,6 +1071,9 @@ Recent decisions affecting current work:
| 261106-dv0 | shared plugins log boot details at Debug: paymentgateway operators registered, paymentgatewaysubscriptions registries collected through diary.For (todo 2026-10-28-plugins-log-boot-details-at-debug.md) | 2026-11-06 | 8a88b86, 1df6a26 | [261106-dv0-plugin-boot-logs-at-debug](./quick/261106-dv0-plugin-boot-logs-at-debug/) |
| 261106-e5v | purge token rename leftovers: grzybyfunkcjonalne check-deploy refuses NUXT_SITE_* keys, journal/menu/faq READMEs name NUXT_CACHE_PURGE_TOKEN (todo 2026-10-20-rename-site-cache-purge-env-in-consumers.md) | 2026-11-06 | 4e2cf0c, f61c90a, 128a33d, a53511b | [261106-e5v-purge-token-rename-leftovers](./quick/261106-e5v-purge-token-rename-leftovers/) |
| 261106-jhk | wristband MCP authorize drops OIDC identity scopes (openid, profile, email) so the ChatGPT connector can connect | 2026-11-06 | aba678b | [261106-jhk-wristband-mcp-authorize-ignores-oidc-ide](./quick/261106-jhk-wristband-mcp-authorize-ignores-oidc-ide/) |
| 261106-kt1 | pier MCP 401 challenge names only resource_metadata, so clients request the advertised scopes (write included) | 2026-11-06 | 609caa4 | [261106-kt1-pier-mcp-401-challenge-drops-the-fixed-r](./quick/261106-kt1-pier-mcp-401-challenge-drops-the-fixed-r/) |
| 261106-kt2 | deferred:purge, migrate* and queue:clear reuse the app's published database, so the in-server scheduler no longer fails with duplicate provider for *sql.DB | 2026-11-06 | 99abe88 | [261106-kt2-lagoon-deferred-purge-fails-in-the-sched](./quick/261106-kt2-lagoon-deferred-purge-fails-in-the-sched/) |
| 261108-pe4 | beachcomber highlights and field projection: Query.IncludeFields/ExcludeFields/Highlight, SearchResult.Hits (plain-text segments), HighlightSearcher + SupportsHighlights; Typesense parses v1 highlights | 2026-11-08 | 7819c67 | [261108-pe4-beachcomber-highlights-and-field-project](./quick/261108-pe4-beachcomber-highlights-and-field-project/) |
## Deferred Items

View File

@@ -45,7 +45,7 @@ A third fix found by the tests is functional, not a security gap: c29d228 `fix(p
| T-19.7-07 | `payload` is accepted only for an action with a Prompt, only as a JSON string, trimmed, non-empty and capped in runes; every other record action still refuses a payload; a plain action runs with an empty Value. | F `cabana/actions.go:217`, `:256` | `TestRecordActionPromptSmoke`, `TestRecordActionPrompt`, `TestRecordActionPromptRun` (F) | verified |
| T-19.7-08 | Contributed labels, statuses, messages and URLs render as text and attributes (no `v-html`); a URL opens with `noopener` and the visible link carries `rel="noopener noreferrer"`. | F `admin/src/components/account/AccountContributions.vue` | `admin/tests/account/AccountContributions.test.ts` ("Text, never HTML", URL attributes) (F) | corrected: fixed in 44835e4, 88559d5 (F); was verified (see Corrections) |
| T-19.7-09 | wristband's authorize allow-list is exactly `Options.ScopesSupported`; the default server refuses `backend`; `openid` and unlisted scopes stay refused on the MCP flavour. | F `wristband/authorize.go:31`, `:231` | `TestMCPScopesSmoke`, `TestMCPScopes` (F) | verified |
| T-19.7-10 | Every request re-verifies its bearer (`auth.RequireBearerToken` with pier's resolver); a session never replaces it; a failure is 401 with the RFC 9728 challenge and `scope="read"`. | F `pier/pier.go:235-260`; F `pier/identity.go:144` | `TestPierTracerSmoke` (bad bearer), `TestEndpointGuards` (stream ends after revocation), `TestMCPEndToEnd` "revoked token" (F, D) | verified |
| T-19.7-10 | Every request re-verifies its bearer (`auth.RequireBearerToken` with pier's resolver); a session never replaces it; a failure is 401 with the RFC 9728 challenge naming only `resource_metadata` (no scope parameter since quick 261106-kt1, so clients request the PRM `scopes_supported`). | F `pier/pier.go:256-274`; F `pier/identity.go:144` | `TestPierTracerSmoke` (bad bearer, missing bearer), `TestEndpointGuards` (stream ends after revocation), `TestMCPEndToEnd` "revoked token" (F, D) | verified |
| T-19.7-11 | The session record stores only the binding (`oauth:<client>:<user>` or the personal token id), compared on every POST, GET and DELETE (403 `session_binding`); ids from `rand.Text()`; sliding idle TTL; per-binding cap. | F `pier/session.go:143`, `:211`, `:242`, `:139` | `TestPierSessionsSmoke`, `TestSessions` (Redis and memory) (F) | verified |
| T-19.7-12 | The inner request is built fresh (`httptest.NewRequestWithContext`) with no Authorization or Cookie; identity travels only in the marker; client address, X-Forwarded-For, X-Real-IP and Accept-Language are copied. | F `pier/dispatch.go:159-180` | `TestPierTracerSmoke` (echo route `auth: false`), `TestDispatch` (outer Cookie and Authorization absent) (F) | verified |
| T-19.7-13 | A backend subject exists only with `backend` in the grant AND a link the resolver confirms per request (5 s cache), or an admin `g15_` token behind `mcp.allow_personal_tokens`; the read/write ceiling applies on top; list and call use the same `allowed`. | F `pier/identity.go:152`, `:187-198`; F `pier/filter.go:16`, `:37` | `TestToolsList` (no link, no backend tool; union with a link), `TestPierTracerSmoke` "personal tokens behind the flag", `TestMCPEndToEnd` (no backend tool without the scope) (F, D) | verified |

View File

@@ -0,0 +1,516 @@
---
phase: 35-mail-templates-and-layouts-editor
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- modules/pact/capabilities.go
- modules/pact/README.md
- modules/postcard/store.go
- modules/postcard/mailer.go
- modules/postcard/templates.go
- modules/postcard/partials.go
- modules/postcard/samples.go
- modules/postcard/editor.go
- modules/postcard/brand.go
- modules/postcard/assets/system.htm
- modules/postcard/assets/minimal.htm
- modules/postcard/assets/sample.htm
- modules/postcard/assets/brand.css.tmpl
- modules/postcard/assets/partials/button.htm
- modules/postcard/assets/partials/panel.htm
- modules/postcard/assets/partials/table.htm
- modules/postcard/assets/partials/subcopy.htm
- modules/postcard/override_smoke_test.go
- modules/postcard/partials_smoke_test.go
- modules/postcard/editor_smoke_test.go
- modules/postcard/brand_smoke_test.go
- modules/postcard/README.md
- modules/lagoon/migrations.go
- modules/lagoon/pending_test.go
- modules/lagoon/README.md
- modules/lagoon/attach/absolute.go
- docs/services/mail.md
- docs/services/storage.md
- docs/plugins/registration.md
- docs/database/migrations.md
autonomous: false
requirements: [D-01, D-04, D-05, D-06, D-07, D-08, D-10, D-11, D-12, D-13, D-14, D-15, D-16, D-17, D-19, D-22, D-23, D-24, D-25]
estimate:
tokens: 170000
raw_tokens: 170000
tasks: 5
confidence: low
must_haves:
truths:
- "A customized row in system_mail_templates changes the subject, body and layout of the mail postcard sends in any process that published a *gorm.DB (serve, queue:work, console), with no cabana activation; postcard reads it once per send through NewDBSource, never from a boot-time snapshot (D-01 as amended by D-22, Claude's Discretion: per-send lookup)"
- "Without a customized row the sent mail is byte-identical to the file version, and an app with no published database or with the mail tables missing still sends the file version and logs a rate-limited warning (D-14, research Pitfall 7)"
- "An override that fails to render or fails the safety checks at send time is logged with component postcard and the file version is sent instead; an override whose layout code resolves to nothing renders with postcard's default layout and a warning (D-14)"
- "A caller's non-empty Message.Subject still wins over an override subject, and Mailer, Message, Catalog.Register, BootPlugin and pact.HasMailTemplates keep their exact signatures (D-04, D-12)"
- "Plugins declare partials through the additive pact.HasMailPartials capability; names must be <plugin>::mail.partials.<name> and map to views/mail/partials/<name>.htm; a foreign namespace, a missing file or a duplicate fails BootPlugin with the dotted name; postcard ships the built-in partials button, panel, table and subcopy and the file layouts default (unchanged), system and minimal (D-08, D-12)"
- "`{{ partial \"button\" \"url\" .Link \"body\" \"Go\" }}` in a Markdown body survives goldmark through per-render nonce tokens swapped after conversion, renders the partial's HTML section in the HTML part and its text section in the text part, and works the same in layouts (D-12)"
- "Template headers may declare `sample.<path> = value` keys (case kept after the prefix); files without them parse exactly as before, and previews and save validation fill every other variable with a visible `[path]` placeholder, including nested fields, if/with blocks and range bodies (D-15, research Pitfall 3)"
- "postcard's exported Editor validates a template, layout or partial draft (parse, test render with samples, the existing script/iframe/object/embed, event-handler and javascript:/vbscript:/data: checks, subject CR/LF, unknown layout or partial, a layout without {{ .Content }}) and reports the failing field and rule; Preview and SendDraft render a draft through the same path as Send (D-13, D-16, D-17)"
- "SyncIndex upserts one index row per file template, layout and partial (code unique), mirrors file content into rows that are not customized, always refreshes file_hash, source and description, never deletes rows and never touches a customized row's override columns; ResetOverride clears the override columns of one row and re-mirrors the file (D-06 as amended by D-23, D-05 locale column)"
- "Every file entry carries a sha256 hash of its CRLF-normalized bytes, so a saved override's base_hash can be compared with the current file_hash (D-07)"
- "With no row in system_mail_brand_settings the layout's brandCss slot holds config mail.brandCss; once the row exists, CompileBrandCSS of its colours replaces it, mail.css is still applied, and layouts get `{{ .logoUrl }}` as an absolute URL built at send time from storage.uploads.public_url or app.url (D-10, D-11, D-19, D-24)"
- "postcard registers no built-in template (D-25)"
prohibitions:
- statement: "pact.HasMailTemplates, postcard.Mailer, postcard.Message, postcard.Catalog.Register and postcard.BootPlugin keep their exact signatures; changes are additive only."
status: unresolved
verification: judgment
- statement: "Existing plugin mail renders byte-identically when no override exists (the default layout file and its output are unchanged)."
status: unresolved
verification: judgment
- statement: "goldmark is not switched to unsafe HTML rendering for mail bodies."
status: unresolved
verification: judgment
- statement: "postcard does not import cabana or lagoon, and no override is cached across sends."
status: unresolved
verification: judgment
- statement: "Framework code, READMEs and docs name no consuming application (neutral names such as acme or blog)."
status: unresolved
verification: judgment
artifacts:
- path: "modules/postcard/store.go"
provides: "Migrations, table constants, OverrideSource, NewDBSource, SyncIndex, ResetOverride"
contains: "func NewDBSource("
- path: "modules/postcard/partials.go"
provides: "partial template func, token swap"
contains: "MAILPARTIAL"
- path: "modules/postcard/samples.go"
provides: "sample.* keys and the placeholder walker"
contains: "text/template/parse"
- path: "modules/postcard/editor.go"
provides: "Editor, EditorFrom, Draft, DraftError, Preview, SendDraft"
contains: "func EditorFrom("
- path: "modules/postcard/brand.go"
provides: "Brand, DefaultBrand, CompileBrandCSS"
contains: "func CompileBrandCSS("
- path: "modules/lagoon/attach/absolute.go"
provides: "config-based absolute public URL"
contains: "func AbsolutePublicURL("
- path: "modules/pact/capabilities.go"
provides: "HasMailPartials, RowStateCustomized, RowStateOutdated"
contains: "type HasMailPartials interface"
- path: "modules/postcard/override_smoke_test.go"
provides: "TestMailOverrideSmoke (tracer)"
contains: "func TestMailOverrideSmoke("
key_links:
- from: "modules/postcard/mailer.go"
to: "modules/postcard/store.go"
via: "Activate installs NewDBSource(app); Send calls source.MailOverrides once per message"
pattern: "NewDBSource\\(app\\)"
- from: "modules/lagoon/migrations.go"
to: "modules/postcard/store.go"
via: "framework migration set summercms.postcard"
pattern: "postcard\\.Migrations"
- from: "modules/postcard/brand.go"
to: "modules/lagoon/attach/absolute.go"
via: "logo URL resolved at send time from config"
pattern: "attach\\.AbsolutePublicURL"
- from: "modules/postcard/mailer.go"
to: "modules/pact/capabilities.go"
via: "BootPlugin type-asserts pact.HasMailPartials next to pact.HasMailTemplates"
pattern: "pact\\.HasMailPartials"
---
## Phase Goal
ROADMAP Goal (not in user-story format; no story is invented): administrators edit every mail the application sends from the admin panel, as in WinterCMS's Mail templates screen. postcard renders the override when one exists, otherwise the plugin's file, so mail that already works keeps working. This plan builds the mail-domain half that every other plan consumes: postcard's database layer, partials, sample values, the draft editor API and Branding, with no admin code (D-01, D-22).
Spec-less probe fallback skipped: Phase 35 has no requirement IDs and no SPEC.md, so this plan set derives must_haves from the ROADMAP goal and CONTEXT decisions D-01..D-25 (a visible choice, not an omission). Prohibitions recalled from CONTEXT and CLAUDE.md are authored descriptor-less above.
<objective>
Give postcard an editable database layer that works in every process that sends mail:
- postcard owns the mail tables and their migrations (a framework set in lagoon.migrationSets) and a per-send override lookup with fallback to the file version (D-22, D-14);
- an additive pact.HasMailPartials capability, built-in partials and layouts, and partial calls that survive Markdown (D-12, D-08);
- `sample.<path>` header keys plus a placeholder walker, file hashes for drift, index-row sync and reset (D-15, D-07, D-23);
- an exported Editor that validates, previews and test-sends drafts through the real render path (D-13, D-16, D-17);
- Branding: compiled brand CSS replacing mail.brandCss once saved, and a logo URL built at send time from config (D-10, D-11, D-24).
Purpose: 35-02 (cabana screens) and 35-03 (admin SPA) are thin over this API; putting the read path in postcard keeps overrides effective in queue:work, where cabana never activates (research Pattern 1).
Output: commits in this repository only, code and its docs together per commit group below. Nothing is pushed or tagged.
**Planner decisions recorded here (confirmed or adjusted at Task 1):**
- **P35-01 schema** (Claude's Discretion; D-07 costly). One gormigrate migration `202611060001_create_system_mail_tables` in `postcard.Migrations`, raw SQL like attach.Migrations, history ID `summercms.postcard`, placed after `summercms.attach` in migrationSets:
- `system_mail_layouts`: id SERIAL PK; code TEXT NOT NULL UNIQUE (file layout: its full name containing `::`; database-only layout: `^[a-z0-9-]+$`); source TEXT NULL (owning plugin ID; NULL marks a database-only layout); name TEXT NOT NULL; content_html, content_text, content_css TEXT NOT NULL DEFAULT ''; is_custom BOOLEAN NOT NULL DEFAULT FALSE (file row: override active; database-only row: always TRUE); file_hash TEXT NULL; base_hash TEXT NULL; customized_at TIMESTAMPTZ NULL; created_at, updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW().
- `system_mail_templates`: id; code UNIQUE; source TEXT NOT NULL; locale TEXT NULL; description TEXT NOT NULL DEFAULT ''; subject, body TEXT NOT NULL DEFAULT ''; layout_id INTEGER NULL REFERENCES system_mail_layouts(id) ON DELETE RESTRICT (index); is_custom, file_hash, base_hash, customized_at, timestamps as above.
- `system_mail_partials`: id; code UNIQUE; source TEXT NOT NULL; name TEXT NOT NULL DEFAULT ''; content_html, content_text; is_custom, file_hash, base_hash, customized_at, timestamps.
- `system_mail_brand_settings`: id INTEGER PRIMARY KEY (the settings singleton, row 1); logo_file_id INTEGER NULL; the 13 colour columns of the UI-SPEC Branding table (body_bg, content_bg, header_color, footer_color, heading_color, text_color, link_color, button_text_color, button_primary_bg, button_positive_bg, button_negative_bg, panel_bg, table_border_color) TEXT NOT NULL DEFAULT ''; timestamps.
- `system_mail_image_library`: id SERIAL PK; timestamps (the attachMany host of the image library, never the Branding row, research anti-pattern).
- "Outdated" means `is_custom AND source IS NOT NULL AND base_hash IS DISTINCT FROM file_hash`. Hash: sha256 hex of the raw file bytes with CRLF normalized to LF.
- **P35-02 partial call syntax** (D-12 costly; the syntax becomes text in plugin files and stored overrides): `{{ partial "button" "url" .Link "body" "Activate" "type" "positive" }}`, a template func `partial(name string, kv ...any)` taking key/value pairs. A bare name resolves to `postcard::mail.partials.<name>`; plugin partials use the full name. Unknown name or an odd kv count is a render error (rule `partial`). Inside a partial, `markdown` renders a string through goldmark with the table extension; `partial` is not available inside a partial. Built-in partials: button (url, body, type primary|positive|negative), panel (body), table (body, a Markdown table), subcopy (body). Built-in layouts: default (byte-identical), system (Winter's "System layout" with the automatic-message footer), minimal; plugin templates reach `system` and `minimal` through a fallback after the plugin's own aliases.
- **P35-03 sample keys** (D-15): dotted INI keys in the existing header, `sample.Name = "Ada"`, `sample.order.code = "ZAM-1"`; the part after `sample.` keeps its case; values are strings. A `[sample]` section is not used because released postcard versions reject section lines.
- **P35-04 layout data**: layout CSS (content_css of an override) is prepended to config mail.css in the existing `css` slot, so a layout that renders `{{ .css }}` picks it up and file layouts (no CSS section) are unchanged; layouts get a new `logoUrl` string key next to Content, Subject, css and brandCss.
- **P35-05 names used by 35-02 and 35-03**: permission codes `system.manage_mail_templates` and `system.manage_mail_layouts` (D-03); controller IDs `summercms.cabana.mailtemplates`, `summercms.cabana.maillayouts`, `summercms.cabana.mailpartials`, `summercms.cabana.mailimages`; row states `customized` and `outdated`; admin routes `POST /mail/preview`, `POST /mail/test-send`, `GET /mail/images`, `GET /mail/images/{id}`, `POST /mail/images`; test send limited to 5 per admin per 10 minutes (research A7); the image picker lists for holders of either mail permission and uploads only for `system.manage_mail_layouts` (D-03 planner's call, UI-SPEC C4).
- **Reset semantics** (D-23 amends D-06): reset clears the override columns of the index row and re-mirrors the file; custom layouts are deleted, never reset.
</objective>
<execution_context>
@~/.claude/msd-core/workflows/execute-plan.md
@~/.claude/msd-core/templates/summary.md
</execution_context>
<context>
@.planning/phases/35-mail-templates-and-layouts-editor/35-CONTEXT.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-PATTERNS.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md
@CLAUDE.md
<interfaces>
modules/postcard/mailer.go: Message{Template, To, Cc, Bcc, ReplyTo, Vars, Subject}; Mailer interface{ Send(ctx, Message) error } (implemented by outside code: keep it); Options{From, CSS, BrandCSS}; unexported mailer{catalog, driver, opts}; NewMailer(cat, driver, opts) Mailer; Activate[P](app, []P) publishes Mailer; driverFromApp reads mail.driver, mail.from, mail.css, mail.brandCss; BootPlugin[P](app, p) type-asserts pact.HasMailTemplates and the published *mailer, then Catalog.Register; moduleLog = diary.For("postcard").
modules/postcard/templates.go: NewCatalog registers postcard::mail.layouts.default from assets/default.htm; Register(pluginID, fsys, templates, layouts) with assertOwner, assetPath (name -> views/mail/<dots as slashes>.htm), duplicate/missing errors; resolveLayoutLocked (plugin aliases only; "default" reserved); parseTemplateFile (subject, description, layout), parseLayout (header name, text, html); splitEQ normalizes CRLF; parseINI lower-cases keys; render(msg, renderOptions{css, brandCSS}) = execHTML subject (CR/LF refused) -> execHTML body -> goldmark -> validateHTML -> layout text (Content = pre-Markdown body) and HTML (Content, Subject, css, brandCss) -> validateHTML; execHTML/execText use missingkey=zero.
modules/postcard/drivers.go: MemoryDriver{Messages()}, validateRendered(msg, requireFrom).
modules/pact/capabilities.go: HasMailTemplates (~1271, unchanged); RowState consts (~1055); ListRowStates doc says values outside the three constants are dropped.
modules/lagoon/migrations.go: migrationSets (~224) lists {"summercms.attach", "system_files", attach.Migrations} first; modules/lagoon/pending_test.go:349 TestMigrateStatusJSON wantSets.
modules/lagoon/attach: PublicURL(key), PublicURLBase(), PublicPathPrefix() (process globals set by OpenBucket, absent in queue:work), BlobKey(diskName), parsePublicURL(raw), defaultPublicPathPrefix "/storage/uploads", File{ID, DiskName, IsPublic, AttachmentType, Field}.
modules/backpack: app.Publish[T](v), app.Lookup[T]() (T, bool); publishing the same T twice is an error.
Import graph: postcard imports backpack, compass, diary, pact; lagoon will import postcard (party already does); postcard must not import lagoon. Internal postcard tests therefore open Postgres with gorm.io/driver/postgres directly (pgx stdlib, as modules/cabana/auth_test.go adminGorm does) and run postcard.Migrations with gormigrate; they never import lagoon.
</interfaces>
</context>
## Execution units and resume points
- Each task ends with the commit(s) listed below, staged by explicit path (never `-A` or `.`). Other sessions may hold uncommitted `.planning/STATE.md`, `.planning/state.json`, `.planning/G15OFFICE.md`, `examples/hello/plugins.gen.go` or cabana edits; leave them alone. Before editing `modules/postcard`, `modules/pact` or `modules/lagoon`, run `git status --short modules/postcard modules/pact modules/lagoon docs` and stop if another session has uncommitted edits there.
- Resume check: `git log --format=%s -F --grep='<subject>'`; present subjects mean the task is done: re-run its `<verify>` and continue with the next one.
- Docker-backed smoke subtests skip under `-short`; run them without `-short` (Docker is available on this host; if the socket is denied, run through `sudo -u golem` per CLAUDE.md).
| Task | Commit subject | Resume point |
|------|----------------|--------------|
| 1 | none (answer recorded in the SUMMARY) | answer recorded |
| 2 | `feat(postcard): database overrides for mail templates with file fallback` | TestMailOverrideSmoke passes |
| 3 | `feat(postcard): mail partials, built-in layouts and layout and partial overrides` | TestMailPartialsSmoke passes |
| 4 | `feat(postcard): sample values, file hashes, index sync and the draft editor` | TestMailEditorSmoke passes |
| 5 | `feat(attach): absolute public URL from config`, then `feat(postcard): mail branding and the logo URL` | TestMailBrandSmoke passes |
<tasks>
<task type="checkpoint:decision" gate="blocking">
<name>Task 1: Confirm the mail schema, the partial call syntax, the sample keys and the names before any code depends on them</name>
<reversibility rating="one-way">After a release, installs hold rows in system_mail_* and admins and plugins write `{{ partial ... }}` calls and `sample.*` keys into stored overrides and plugin files; renaming a table, column or the call syntax then needs a data migration that rewrites stored bodies.</reversibility>
<decision>Adopt P35-01 (schema), P35-02 (partial syntax and built-ins), P35-03 (sample keys), P35-04 (layout CSS slot and logoUrl) and P35-05 (names, limits, picker upload rule) as written in the objective, or adjust them.</decision>
<context>
CONTEXT leaves table names and schema, the partial call syntax, the built-in set, the D-15 mechanism, the extra predefined layouts and the picker upload permission to the planner. D-22..D-25 (2026-11-06) fixed the ownership (postcard), the index-row model, the logo URL timing and the empty template list, but not these names. 35-02 and 35-03 encode every name fixed here (controller IDs, permission codes, routes, row states, column names in fields.yaml), so this is the only point before all of them. D-03, D-07, D-09 and D-12 are already rated costly by the user and are carried through, not re-asked.
</context>
<options>
<option id="as-planned">
<name>Adopt P35-01..P35-05 as written</name>
<pros>Matches the research recommendation and Winter's table names; the partial syntax is a plain Go template call; sample keys stay readable by released postcard versions</pros>
<cons>Freezes the column and call names now</cons>
</option>
<option id="adjust">
<name>Adopt with the changes Jakub states (table or column names, call syntax, built-in set, limits)</name>
<pros>Names fixed before they become stored data</pros>
<cons>The executor records the adjusted names in the SUMMARY; plans 35-02..35-04 use the SUMMARY's names wherever they differ from their text</cons>
</option>
<option id="sample-capability">
<name>Adopt, but declare sample values through an additive pact capability instead of `sample.*` header keys</name>
<pros>Typed sample values (numbers, lists) instead of strings</pros>
<cons>A second new pact capability to support forever; samples live in Go instead of next to the template text</cons>
</option>
</options>
<resume-signal>Reply "as-planned", "adjust: <changes>" or "sample-capability"</resume-signal>
</task>
<task type="tracer">
<name>Task 2: End-to-end "a customized template row changes the mail a process without cabana sends" — templates only, with the file fallback</name>
<reversibility rating="costly">D-07: the hash columns are part of the override table schema; a later change is a new migration.</reversibility>
<files>modules/postcard/store.go, modules/postcard/mailer.go, modules/postcard/templates.go, modules/postcard/override_smoke_test.go, modules/lagoon/migrations.go, modules/lagoon/pending_test.go, modules/lagoon/README.md, modules/postcard/README.md, docs/services/mail.md, docs/database/migrations.md</files>
<read_first>
- modules/postcard/mailer.go and modules/postcard/templates.go (whole files: Send, Activate, BootPlugin, render)
- modules/postcard/mailer_test.go and modules/postcard/templates_test.go (what byte-identity the existing tests pin)
- modules/lagoon/attach/migrations.go (raw-SQL gormigrate migration to copy)
- modules/lagoon/migrations.go (~200-245 migrationSets) and modules/lagoon/pending_test.go (~330-355)
- modules/cabana/auth_test.go (~321-370 adminGorm: testcontainers postgres:16-alpine, pgx, skip under testing.Short)
- modules/postcard/README.md and docs/services/mail.md (sections to extend)
- .planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md (Pattern 1, Pattern 3, Pitfall 1, Pitfall 7)
</read_first>
<action>
Per D-22, D-14 and D-01 (postcard defines the interface and ships the database-backed implementation), using the Task 1 answer for every name:
- `store.go`: exported constants `TemplatesTable`, `LayoutsTable`, `PartialsTable`, `BrandTable`, `ImageLibraryTable` with the P35-01 names; `Migrations []*gormigrate.Migration` with the single P35-01 migration (create all five tables now, so later tasks add no migration; Rollback drops them in reverse dependency order); `OverrideSource` interface with `MailOverrides(ctx context.Context, template string) (Overrides, error)` where an empty template asks for everything except a template override (the editor uses that in Task 4); `Overrides{Template *TemplateOverride; Layouts map[string]LayoutOverride; Partials map[string]PartialOverride}` (Task 5 adds the `Brand *Brand` field; Task 3 fills Layouts and Partials); `TemplateOverride{Subject, Body, Layout string}` where Layout is a layout code; `LayoutOverride{Name, HTML, Text, CSS string}`; `PartialOverride{HTML, Text string}`; `NewDBSource(app *backpack.App) OverrideSource` that looks up `*gorm.DB` through `app.Lookup` on every call (none published: zero Overrides, nil error) and in this task reads only the template row `WHERE code = ? AND is_custom` plus the code of its layout_id row, inside one read-only query or transaction.
- `mailer.go`: add `Overrides OverrideSource` to Options (additive field) and a `source` field to mailer; NewMailer keeps its signature and takes the source from opts; Activate installs `NewDBSource(app)`. In Send, after the existing checks, ask the source once; a lookup error is logged through moduleLog at warning level at most once per minute per distinct error text and the file version is rendered. Render with the override applied; when that render or validation fails and an override was applied, log at error level with keys template and error and the message "postcard: override failed, sending the file version", then render the file version (D-14). A template override whose layout code resolves to no catalog layout renders with postcard's default layout and logs a warning.
- `templates.go`: refactor render so the file version and an override go through one path (subject source, body, effective layout parts) and the file-only output stays byte-identical; the caller's non-empty Message.Subject still wins over both.
- `modules/lagoon/migrations.go`: add `{"summercms.postcard", "mail", postcard.Migrations}` directly after the attach set; update the set list in `modules/lagoon/pending_test.go` TestMigrateStatusJSON to match.
- `override_smoke_test.go` (package postcard, so it may type-assert the published *mailer and read its MemoryDriver): `TestMailOverrideSmoke`, skipped under testing.Short, starts postgres:16-alpine with testcontainers (pattern of cabana adminGorm), opens GORM with gorm.io/driver/postgres over the pgx stdlib connection, runs Migrations with gormigrate, publishes the *gorm.DB on a backpack.App, calls Activate and BootPlugin for a fixture plugin (fstest.MapFS with `acme.blog::mail.welcome` and a layout alias), and has subtests `override` (an is_custom row's subject and body are sent), `file` (no row: output equals a NewMailer without a source), `fallback` (an override body that fails at runtime, such as a call on a missing nested key, sends the file version and logs the error message), `caller_subject`, `missing_table` (table dropped: file version, warning logged) and `no_db` (app without a published DB: file version).
- Docs in the same commit: postcard README (Overview sentence, Features bullet "database overrides", Usage "Overrides from the admin" with the precedence caller Subject > override > file and the fallback rule, API reference rows for every new identifier, Dependencies: gorm and gormigrate), docs/services/mail.md new section "Overrides from the admin", lagoon README framework-set sentence naming `postcard.Migrations` under the `summercms.postcard` history, and docs/database/migrations.md where it lists the framework sets. Use neutral names (acme) only.
Commit as listed in Execution units.
</action>
<verify>
<automated>go test ./modules/postcard -run 'TestMailOverrideSmoke' -count=1 -v 2>&1 | grep -E '^\s*(--- |FAIL|ok|PASS)' | tail -20</automated>
<fails_when>a "FAIL" line, a "--- SKIP" line, "no tests to run", or no "--- PASS: TestMailOverrideSmoke/fallback" line</fails_when>
<automated>go test ./modules/lagoon -run 'TestMigrateStatusJSON' -count=1 2>&1 | tail -3</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>go vet ./... && go test -short ./...</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree' -count=1 2>&1 | tail -3 && go run ./cmd/summer docs:build --check 2>&1 | tail -5</automated>
<fails_when>non-zero exit, or a "FAIL" or "unknown identifier" line in the output</fails_when>
</verify>
<acceptance_criteria>
- `grep -n "func NewDBSource(app \*backpack.App) OverrideSource" modules/postcard/store.go` prints one line
- `grep -n "MailOverrides(ctx context.Context, template string) (Overrides, error)" modules/postcard/store.go` prints one line
- `grep -n '"summercms.postcard", "mail", postcard.Migrations' modules/lagoon/migrations.go` prints one line
- `go doc ./modules/postcard Options` lists From, CSS, BrandCSS and Overrides
- `go doc ./modules/postcard Mailer` still shows exactly `Send(ctx context.Context, msg Message) error`
- `go list -deps ./modules/postcard | grep -E 'summercms/modules/(lagoon|cabana)$'` prints nothing
- `grep -n "Overrides from the admin" docs/services/mail.md modules/postcard/README.md` prints at least two lines
</acceptance_criteria>
<done>A customized template row reaches real sends in every process with a database, and anything wrong with it degrades to the file version.</done>
</task>
<task type="auto" tdd="true">
<name>Task 3: Partials and built-in layouts: plugins declare partials, bodies and layouts call them, layout and partial overrides apply at send</name>
<reversibility rating="costly">D-12: HasMailPartials, the <plugin>::mail.partials.<name> naming and the partial call syntax are a published contract plugins implement.</reversibility>
<files>modules/pact/capabilities.go, modules/pact/README.md, modules/postcard/partials.go, modules/postcard/templates.go, modules/postcard/mailer.go, modules/postcard/store.go, modules/postcard/assets/system.htm, modules/postcard/assets/minimal.htm, modules/postcard/assets/partials/button.htm, modules/postcard/assets/partials/panel.htm, modules/postcard/assets/partials/table.htm, modules/postcard/assets/partials/subcopy.htm, modules/postcard/partials_smoke_test.go, modules/postcard/README.md, docs/services/mail.md, docs/plugins/registration.md</files>
<read_first>
- modules/postcard/templates.go and modules/postcard/store.go (Task 2 output)
- modules/pact/capabilities.go (~1260-1280 HasMailTemplates; keep it byte-identical)
- modules/postcard/assets/default.htm (layout file format and the `<style>` slot)
- ../examples/golem15-wintercms-starter/modules/system/views/mail/layout-system.php and partial-button.php, partial-panel.php, partial-table.php, partial-subcopy.php (content to port to Go templates) — read only
- docs/plugins/registration.md (capability table)
- .planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md (Pattern 4, Pitfall 2)
</read_first>
<behavior>
- A plugin implementing pact.HasMailPartials with `acme.blog::mail.partials.note` boots; `acme.other::mail.partials.note` from acme.blog fails with "not owned"; a name without the `mail.partials.` segment, a missing file and a duplicate fail with the dotted name
- A plugin that implements only HasMailPartials (no templates) boots and its partial is callable from another plugin's template
- Body `Hi\n\n{{ partial "button" "url" .Link "body" "Go" }}\n\nBye` renders the button table with href = .Link in the HTML part (no "raw HTML omitted" comment) and `Go <link>` in the text part
- A variable value that contains a forged token (MAILPARTIAL followed by digits) is rendered as text, not replaced
- An odd kv count or an unknown partial is a render error naming the partial
- A template with `layout = system` from a plugin without a `system` alias renders with the built-in system layout; a plugin alias named system still wins
- A fake OverrideSource returning a customized file layout, a database-only layout (template override Layout = its code) and a partial override changes the output accordingly; layout CSS appears before mail.css in the style block
- Existing templates_test.go and mailer_test.go still pass unchanged
</behavior>
<action>
Per D-12 and D-08, with the Task 1 syntax:
- `modules/pact/capabilities.go`: add `HasMailPartials{ MailPartialsFS() fs.FS; MailPartials() []string }` next to HasMailTemplates, with a doc comment naming the file path and namespace rule; HasMailTemplates stays byte-identical. Add a row to modules/pact/README.md.
- `templates.go`: `parsedPartial{name, pluginID, headerName, textWrap, htmlWrap}` (layout file format: header, `==`, text, `==`, HTML), a partials map on Catalog, `(*Catalog) RegisterPartials(pluginID string, fsys fs.FS, partials []string) error` reusing assertOwner, assetPath and the missing/duplicate errors and requiring the `mail.partials.` segment; NewCatalog also registers the built-in layouts `postcard::mail.layouts.system` and `postcard::mail.layouts.minimal` and the partials button, panel, table and subcopy from embedded assets; resolveLayoutLocked tries the plugin's aliases, then the built-in aliases default, system and minimal.
- `mailer.go` BootPlugin: check pact.HasMailTemplates and pact.HasMailPartials independently (each optional) and register partials before templates.
- `partials.go`: the `partial` func per P35-02. In the body pass it renders the partial's text and HTML sections with the kv map as data and returns an alphanumeric token `MAILPARTIAL<nonce><n>` (nonce from crypto/rand, fresh per render); after goldmark the HTML part replaces `<p>TOKEN</p>` and then bare TOKEN with the partial HTML, and the text part (the pre-Markdown body) replaces TOKEN with the partial text. In layouts it returns template.HTML (HTML wrapper) or the text (text wrapper) directly. Partial templates get `markdown` (goldmark plus extension.Table, output validated with validateHTML) and no `partial`. Partial outputs pass validateHTML. Never enable goldmark's unsafe HTML option.
- Assets: port Winter's system layout and the four partials to Go templates with the class names the brand CSS will style (`button button-primary|positive|negative`, `panel`, `table`, `subcopy`); `minimal.htm` is a bare HTML wrapper with the style slot and `{{ .Content }}`. default.htm is not touched.
- Overrides at send: NewDBSource now also reads customized file layouts and every database-only layout (`WHERE is_custom`) into Overrides.Layouts by code, and customized partials into Overrides.Partials by code. A template's effective layout is its override's Layout code or its file layout; a customized file layout or a database-only layout replaces the file wrappers; a partial override replaces that partial's sections. P35-04: a non-empty layout CSS is placed before mail.css in the css slot.
- Write the behavior subtests in `partials_smoke_test.go` (`TestMailPartialsSmoke`, runs under -short: fstest.MapFS fixtures, NewMailer with a fake OverrideSource and a MemoryDriver) first, then implement.
- Docs in the same commit: postcard README (partials, built-in layouts and partials, the call syntax with an acme example, the token rule, API rows for RegisterPartials, LayoutOverride/PartialOverride use), docs/services/mail.md sections "Partials" and "Built-in layouts", docs/plugins/registration.md capability row for pact.HasMailPartials.
Commit as listed in Execution units.
</action>
<verify>
<automated>go test ./modules/postcard -run 'TestMailPartialsSmoke|TestMailOverrideSmoke' -count=1 -v 2>&1 | grep -E '^\s*(--- |FAIL|ok|PASS)' | tail -30</automated>
<fails_when>a "FAIL" line, "no tests to run", or no "--- PASS: TestMailPartialsSmoke" line</fails_when>
<automated>go doc ./modules/pact HasMailTemplates | grep -cE '^\s+(MailTemplatesFS\(\) fs\.FS|MailTemplates\(\) \[\]string|MailLayouts\(\) map\[string\]string)$'</automated>
<fails_when>non-zero exit or a count other than 3 (HasMailTemplates changed)</fails_when>
<automated>go vet ./... && go test -short ./...</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree' -count=1 2>&1 | tail -3 && go run ./cmd/summer docs:build --check 2>&1 | tail -5</automated>
<fails_when>non-zero exit, or a "FAIL" or "unknown identifier" line in the output</fails_when>
</verify>
<acceptance_criteria>
- `grep -n "type HasMailPartials interface" modules/pact/capabilities.go` prints one line
- `grep -n "pact.HasMailPartials" modules/postcard/mailer.go` prints at least one line
- `grep -n "func (c \*Catalog) RegisterPartials(pluginID string, fsys fs.FS, partials \[\]string) error" modules/postcard/templates.go` prints one line
- `ls modules/postcard/assets/partials/` lists button.htm, panel.htm, subcopy.htm and table.htm
- `git diff a024280 --stat -- modules/postcard/assets/default.htm` prints nothing
- `grep -n "pact.HasMailPartials" docs/plugins/registration.md` prints at least one line
</acceptance_criteria>
<done>Plugins and admins can use reusable mail blocks in bodies and layouts, and customized layouts and partials reach real sends.</done>
</task>
<task type="auto" tdd="true">
<name>Task 4: Sample values, file hashes, index rows and the draft editor: previews and save checks render like real sends</name>
<files>modules/postcard/samples.go, modules/postcard/editor.go, modules/postcard/templates.go, modules/postcard/store.go, modules/postcard/assets/sample.htm, modules/pact/capabilities.go, modules/pact/README.md, modules/postcard/editor_smoke_test.go, modules/postcard/README.md, docs/services/mail.md</files>
<read_first>
- modules/postcard/templates.go, modules/postcard/partials.go, modules/postcard/store.go (Tasks 2-3 output)
- modules/pact/capabilities.go (~1050-1075 RowState and ListRowStates doc)
- ../sm-paymentgateway-plugin/views/mail/order/paid.htm and ../sm-user-plugin/views/mail/activate.htm (real nested, range and if shapes the walker must survive) — read only
- .planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md (Pattern 2, Pattern 5, "Placeholder walker" code example, Pitfalls 3, 8, 10)
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md ("Save validation (D-13)" table: the rules and fields)
</read_first>
<behavior>
- `sample.Name = "Ada"` and `sample.order.code = "ZAM-1"` parse to Samples{"Name": "Ada", "order.code": "ZAM-1"}; a file without sample keys parses exactly as before (subject, description, layout unchanged)
- Placeholder vars for `{{ .Name }} {{ .order.code }} {{ if .employee.first_name }}x{{ end }} {{ range .order.items }}{{ .description }}{{ end }} {{ range .lines }}{{ . }}{{ end }} {{ with .user }}{{ .email }}{{ end }}` render without error as `Ada ZAM-1 x [order.items.description] [lines] [user.email]`
- Validate of a template draft: a parse error gives Rule parse on Field body; `<script>` gives Rule unsafe with the matched snippet in Message; a subject with a newline gives Rule subject_line on subject; an unknown layout code gives Rule layout on layout; an unknown partial gives Rule partial with Name set
- Validate of a layout draft without `{{ .Content }}` in the HTML (or text) wrapper gives Rule no_content on content_html (or content_text); CSS containing a closing style tag gives Rule unsafe on content_css
- Preview of a template draft returns Subject, HTML and Text rendered through the same render path as Send, with samples and placeholders; Preview of a layout draft wraps assets/sample.htm; Preview of a partial draft renders a sample message calling it with placeholder arguments
- SendDraft delivers only to the given address through the mailer's driver with mail.from
- Every TemplateFile, LayoutFile and PartialFile has a 64-hex Hash that is equal for LF and CRLF copies of the same file
- TemplateLocale("acme.blog::mail.welcome-en", ["en","pl"]) is "en"; "acme.blog::mail.cancel-notice" with the same locales is ""
- SyncIndex run twice leaves the same rows and IDs; a customized row keeps its subject, body, layout_id and base_hash while file_hash is refreshed; a database-only layout is untouched; two concurrent SyncIndex calls both succeed; ResetOverride on a customized template restores the file subject, body and layout_id and clears is_custom, base_hash and customized_at
</behavior>
<action>
Per D-15, D-13, D-16, D-17, D-07, D-05 and D-23 (amending D-06):
- `samples.go`: read `sample.<path>` header keys keeping the case after the prefix (parseINI keeps lower-casing other keys); a parse-tree walker over the html/template Tree (text/template/parse) that builds nested maps with `[a.b]` leaves or the sample value, one-element slices for range targets (a map of the fields used in the body, or a string when the body prints the dot), handles if/with/else, with re-rooting, chains and variables, and parses with the partial and markdown funcs present.
- Field names in DraftError.Field are the admin column names 35-02 uses: subject, layout, body for templates; content_html, content_text, content_css for layouts and partials.
- Hashes and the catalog read API in templates.go: sha256 hex of CRLF-normalized bytes on every parsed entry; exported read structs `TemplateFile{Name, PluginID, Subject, Description, Layout, Body, Samples, Source, Hash}`, `LayoutFile{Name, PluginID, Title, Text, HTML, Source, Hash}`, `PartialFile{Name, PluginID, Title, Text, HTML, Source, Hash}` (Source is the normalized raw file text; Layout is the resolved full layout name); methods `Templates()`, `Layouts()`, `Partials()` (sorted by name) and `Template(name)`, `Layout(name)`, `Partial(name)` returning (value, bool); serializers `FormatTemplate(subject, layout, body string) string`, `FormatLayout(name, text, html, css string) string`, `FormatPartial(name, text, html string) string` producing the header plus `==` sections, used by 35-02's diff on both sides; `TemplateLocale(name string, locales []string) string` (suffix after the last dash of the last dotted segment, only when it is one of locales).
- `editor.go`: type `Kind` with `KindTemplate`, `KindLayout`, `KindPartial`, `KindBrand`; `Draft{Kind; Name; Subject, Body, Layout; HTML, Text, CSS string; Brand *Brand}` (Name is the template full name, layout code or partial full name); `DraftError{Field, Rule, Message, Name string}` implementing error, Rule one of parse, render, unsafe, subject_line, layout, partial, no_content (35-02 maps them to `backend::lang.mail.invalid_<rule>`) and colour (Task 5; 35-02 maps it to `backend::lang.colorpicker.invalid`); Message carries only parser, renderer or safety-check text (never a database error or a file path beyond the name), and for the unsafe rule the matched snippet (research Pitfall 8); `Preview{Subject, HTML, Text string}`; `Editor` with `EditorFrom(app *backpack.App) (*Editor, bool)` (from the published *mailer, as BootPlugin finds it), `Catalog() *Catalog`, `Validate(ctx, Draft) *DraftError`, `Preview(ctx, Draft) (Preview, error)` (a *DraftError through errors.As for draft problems), `SendDraft(ctx, Draft, to string) error`. Preview and Validate read the other pieces (layouts, partials, Brand) through the mailer's source with an empty template name, so a preview matches a real send (D-16). Leave the KindBrand preview to Task 5.
- `assets/sample.htm`: the sample Markdown message used to preview layouts (a heading, a paragraph, a button partial call, a panel).
- `store.go`: `SyncIndex(ctx context.Context, db *gorm.DB, cat *Catalog, locales []string) error`: layouts first, then templates (layout_id from the layout row of the resolved layout), then partials; `INSERT ... ON CONFLICT (code) DO UPDATE` that always sets source, description, locale, name (header name for layouts and partials) and file_hash and sets the mirror columns only `WHERE NOT is_custom`; never deletes. `ResetOverride(ctx context.Context, db *gorm.DB, cat *Catalog, kind Kind, code string) error` clears is_custom, base_hash and customized_at on one file-backed row and re-mirrors the file content; it refuses a database-only layout.
- pact: add `RowStateCustomized RowState = "customized"` and `RowStateOutdated RowState = "outdated"` and update the ListRowStates doc comment (five constants); pact README row.
- Write the behavior subtests first in `editor_smoke_test.go` (`TestMailEditorSmoke`: subtests `samples`, `walker`, `validate`, `preview`, `send_draft`, `hash`, `locale` run under -short; `sync` and `reset` use the Postgres helper from override_smoke_test.go and skip under -short), then implement.
- Docs in the same commit: postcard README (Editor API, sample keys, SyncIndex, ResetOverride, Kind, the read structs, TemplateLocale), docs/services/mail.md "Sample values for previews" with an acme example and the placeholder rule; pact README row-state sentence.
Commit as listed in Execution units.
</action>
<verify>
<automated>go test ./modules/postcard -run 'TestMailEditorSmoke' -count=1 -v 2>&1 | grep -E '^\s*(--- |FAIL|ok|PASS)' | tail -30</automated>
<fails_when>a "FAIL" line, a "--- SKIP" line, "no tests to run", or no "--- PASS: TestMailEditorSmoke/sync" or "--- PASS: TestMailEditorSmoke/walker" line</fails_when>
<automated>go vet ./... && go test -short ./...</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree' -count=1 2>&1 | tail -3 && go run ./cmd/summer docs:build --check 2>&1 | tail -5</automated>
<fails_when>non-zero exit, or a "FAIL" or "unknown identifier" line in the output</fails_when>
</verify>
<acceptance_criteria>
- `grep -n "func EditorFrom(app \*backpack.App) (\*Editor, bool)" modules/postcard/editor.go` prints one line
- `grep -n "func SyncIndex(ctx context.Context, db \*gorm.DB, cat \*Catalog, locales \[\]string) error" modules/postcard/store.go` prints one line
- `grep -n "func ResetOverride(" modules/postcard/store.go` prints one line
- `grep -n 'RowStateCustomized RowState = "customized"' modules/pact/capabilities.go` and `grep -n 'RowStateOutdated RowState = "outdated"' modules/pact/capabilities.go` each print one line
- `go doc ./modules/postcard DraftError` lists Field, Rule, Message and Name
- `grep -n "sample\." docs/services/mail.md` prints at least one line
</acceptance_criteria>
<done>The admin layer can list file mail as rows, reset an override, and validate, preview and test-send a draft exactly as postcard would send it.</done>
</task>
<task type="auto" tdd="true">
<name>Task 5: Branding: saved colours replace mail.brandCss and layouts show the logo by an absolute URL built at send time</name>
<files>modules/lagoon/attach/absolute.go, modules/lagoon/README.md, docs/services/storage.md, modules/postcard/brand.go, modules/postcard/assets/brand.css.tmpl, modules/postcard/store.go, modules/postcard/mailer.go, modules/postcard/templates.go, modules/postcard/editor.go, modules/postcard/brand_smoke_test.go, modules/postcard/README.md, docs/services/mail.md</files>
<read_first>
- modules/lagoon/attach/bucket.go (~35-80 PublicURLBase, PublicPathPrefix, parsePublicURL; ~125-160 OpenBucket) and modules/lagoon/attach/thumb.go (~60-100 BlobKey, PublicURL)
- modules/postcard/store.go, modules/postcard/editor.go (Tasks 2-4 output)
- ../examples/golem15-wintercms-starter/modules/system/models/mailbrandsetting/custom.less and ../examples/golem15-wintercms-starter/modules/system/models/MailBrandSetting.php (colour set and defaults) — read only
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md ("Branding page (C12, D-10, D-11)" colour table with defaults)
- .planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md (Pattern 6, Pattern 8, Pitfall 4)
</read_first>
<behavior>
- AbsolutePublicURL with storage.uploads.public_url `https://cdn.example.com/u` returns `https://cdn.example.com/u/<key>`; without it and with app.url `https://acme.test` and the default prefix it returns `https://acme.test/storage/uploads/<key>`; with neither it returns ErrNoAbsoluteURL; it never reads the process globals set by OpenBucket
- CompileBrandCSS(DefaultBrand()) succeeds and contains every default colour; a colour `#ABC`, `red` or one containing a closing style tag returns an error naming the field; an empty colour uses that field's default; upper-case hex is accepted and emitted lower-case
- No brand row: the sent HTML's style block contains config mail.brandCss; a brand row exists: it contains the compiled CSS, not mail.brandCss, and mail.css still follows it
- A brand row with logo_file_id pointing at a public system_files row: a layout using `{{ .logoUrl }}` gets the absolute URL; a missing or non-public file gives an empty logoUrl and a warning, and the mail is still sent
- Editor Preview of a KindBrand draft renders the sample message in the default layout with the draft colours and logo
</behavior>
<action>
Per D-10, D-11, D-19 and D-24:
- `modules/lagoon/attach/absolute.go` (own commit first): `ErrNoAbsoluteURL` and `AbsolutePublicURL(cfg *compass.Config, key string) (string, error)`: storage.uploads.public_url (validated with parsePublicURL) joined with the key by one slash; else app.url (absolute http or https) joined with storage.uploads.public_path_prefix (default `/storage/uploads`) and the key; else ErrNoAbsoluteURL. It reads only cfg, so it gives the same URL in serve and in queue:work (research Pitfall 4). Document it in modules/lagoon/README.md (attach bullet and API row) and docs/services/storage.md in the same commit.
- `brand.go`: `Brand{BodyBg, ContentBg, HeaderColor, FooterColor, HeadingColor, TextColor, LinkColor, ButtonTextColor, ButtonPrimaryBg, ButtonPositiveBg, ButtonNegativeBg, PanelBg, TableBorderColor, LogoURL string}`, `DefaultBrand() Brand` with the UI-SPEC Winter defaults, `CompileBrandCSS(b Brand) (string, error)` validating each colour against `^#[0-9a-f]{6}$` after lower-casing (empty means default) and executing `assets/brand.css.tmpl` (a text/template port of Winter's custom.less styling body, a, h1-h3, p, table and the built-in partial classes button-primary/positive/negative, panel, table and subcopy, so Branding also shows in plugin layouts that only carry wrapper classes).
- Add `Brand *Brand` to Overrides; NewDBSource reads row 1 of the brand table into Overrides.Brand (no row: nil) and, when logo_file_id is set, the system_files row by id with is_public true, then `attach.AbsolutePublicURL(app.Config, attach.BlobKey(disk_name))` into Brand.LogoURL; a missing or non-public file or ErrNoAbsoluteURL gives an empty LogoURL and a rate-limited warning.
- Send and render: a non-nil Brand replaces the brandCss slot with CompileBrandCSS (D-11; a compile error logs and keeps config mail.brandCss); mail.css is unchanged; layouts get `logoUrl` (empty when no brand or no logo). Editor gains the KindBrand preview (sample message, default layout, draft Brand) and Validate for KindBrand reports an invalid colour with Rule colour and the column name (for example button_primary_bg) in Field.
- Write the behavior subtests first in `brand_smoke_test.go` (`TestMailBrandSmoke`: `compile`, `defaults`, `preview` under -short; `precedence` and `logo` with the Postgres helper, skipped under -short) plus an attach case for AbsolutePublicURL inside the same test (no new attach test file), then implement.
- Docs in the same postcard commit: postcard README (Branding, the D-11 precedence in Configuration next to mail.brandCss and mail.css, logoUrl, API rows), docs/services/mail.md section "Branding".
Commit as listed in Execution units.
</action>
<verify>
<automated>go test ./modules/postcard -run 'TestMailBrandSmoke' -count=1 -v 2>&1 | grep -E '^\s*(--- |FAIL|ok|PASS)' | tail -20</automated>
<fails_when>a "FAIL" line, a "--- SKIP" line, "no tests to run", or no "--- PASS: TestMailBrandSmoke/precedence" or "--- PASS: TestMailBrandSmoke/logo" line</fails_when>
<automated>go test ./modules/postcard/... ./modules/lagoon/... ./modules/pact/... -count=1 2>&1 | tail -8</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>go vet ./... && go test -short ./...</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree' -count=1 2>&1 | tail -3 && go run ./cmd/summer docs:build --check 2>&1 | tail -5</automated>
<fails_when>non-zero exit, or a "FAIL" or "unknown identifier" line in the output</fails_when>
</verify>
<acceptance_criteria>
- `grep -n "func AbsolutePublicURL(cfg \*compass.Config, key string) (string, error)" modules/lagoon/attach/absolute.go` prints one line
- `grep -n "attach.AbsolutePublicURL" modules/postcard/store.go` prints at least one line
- `grep -n "func CompileBrandCSS(b Brand) (string, error)" modules/postcard/brand.go` prints one line
- `grep -n '"logoUrl"' modules/postcard/templates.go` prints at least one line
- `grep -n "AbsolutePublicURL" modules/lagoon/README.md docs/services/storage.md` prints at least two lines
- `git log --format=%s -F --grep='feat(attach): absolute public URL from config' --grep='feat(postcard): mail branding and the logo URL'` prints two lines, the postcard one first (newer)
</acceptance_criteria>
<done>Saving Branding restyles every mail and shows the logo, while apps that never open the page keep their configured brand CSS.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| admin-edited database rows → postcard render | subject, body, layout and partial text written by administrators, rendered into mail sent to members |
| template variables (member data) → rendered mail | plugin-supplied values that may contain markup or a forged partial token |
| database → send path | a missing table, a stale row or an outage during a send |
| config → logo URL | storage.uploads.public_url and app.url become links in sent mail |
## STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|-----------|----------|-----------|----------|-------------|-----------------|
| T-35-01 | Tampering | override rendering in Send | high | mitigate | validateHTML (script/iframe/object/embed, event handlers, javascript:/vbscript:/data:) runs on the body, every partial output and the final HTML at send, exactly as for files; TestMailOverrideSmoke/fallback proves a failing override never ships |
| T-35-02 | Tampering | partial tokens in bodies | high | mitigate | per-render crypto/rand nonce in an alphanumeric token, replaced only after goldmark; a forged token in a variable stays text (Task 3 behavior) |
| T-35-03 | Tampering | subject override | medium | mitigate | existing CR/LF refusal at send and Rule subject_line in Validate |
| T-35-04 | Tampering | brand colours into the style block | high | mitigate | strict `^#[0-9a-f]{6}$` before CompileBrandCSS; layout CSS with a closing style tag refused at Validate; validateHTML backstop |
| T-35-05 | Denial of service | per-send lookup | medium | mitigate | any lookup error or missing table renders the file version with a rate-limited warning (TestMailOverrideSmoke/missing_table, no_db) |
| T-35-06 | Information disclosure | DraftError.Message | medium | mitigate | Message carries parser, renderer and safety text only, never database errors or filesystem paths |
| T-35-07 | Spoofing | logo URL host | low | mitigate | URL built from operator config only (public_url validated by parsePublicURL, app.url absolute http/https); never from request headers |
| T-35-08 | Tampering | stale overrides across instances | medium | mitigate | no cache: every send reads the rows (Claude's Discretion: per-send lookup) |
| T-35-SC | Tampering | npm/pip/cargo installs | high | accept | this plan installs no package; every Go dependency used (gorm, gormigrate, goldmark, testcontainers) is already in go.mod |
</threat_model>
<verification>
- `go vet ./... && go test -short ./...` green after every commit; `go test ./modules/postcard/... ./modules/lagoon/... ./modules/pact/...` (Docker) green after Task 5.
- `go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree'` and `go run ./cmd/summer docs:build --check` pass.
- `(cd ../sm-user-plugin && go test -run 'TestMail|TestPolishMailSubjects' ./...)` and `(cd ../sm-paymentgateway-plugin && go test -run 'TestMailCatalog|TestMailTracer|TestMailersMatrixSmoke' ./...)` pass against this checkout (their go.mod replaces the framework with ../summercms.go); nothing in those repositories is changed.
</verification>
<success_criteria>
- postcard owns the mail tables, reads overrides per send and falls back to the file version on any failure.
- Partials, built-in layouts, sample values, hashes, index sync, reset, the Editor and Branding exist with smoke tests and docs.
- pact gained HasMailPartials and two row states additively; HasMailTemplates is unchanged.
- No push, no tag; no file in a sibling repository changed.
</success_criteria>
<output>
Create `.planning/phases/35-mail-templates-and-layouts-editor/35-01-SUMMARY.md` when done; record the Task 1 answer verbatim and every name that differs from this plan's text.
</output>
## Artifacts this phase produces
- Tables (migration `202611060001_create_system_mail_tables`, history `summercms.postcard`): `system_mail_layouts`, `system_mail_templates`, `system_mail_partials`, `system_mail_brand_settings`, `system_mail_image_library`.
- pact: `pact.HasMailPartials`, `pact.RowStateCustomized`, `pact.RowStateOutdated`.
- postcard types: `postcard.OverrideSource`, `postcard.Overrides`, `postcard.TemplateOverride`, `postcard.LayoutOverride`, `postcard.PartialOverride`, `postcard.Brand`, `postcard.Kind`, `postcard.Draft`, `postcard.DraftError`, `postcard.Preview`, `postcard.Editor`, `postcard.TemplateFile`, `postcard.LayoutFile`, `postcard.PartialFile`; field `postcard.Options.Overrides`.
- postcard functions and methods: `postcard.NewDBSource`, `postcard.SyncIndex`, `postcard.ResetOverride`, `postcard.EditorFrom`, `postcard.DefaultBrand`, `postcard.CompileBrandCSS`, `postcard.FormatTemplate`, `postcard.FormatLayout`, `postcard.FormatPartial`, `postcard.TemplateLocale`, `(*Catalog).RegisterPartials`, `(*Catalog).Templates|Layouts|Partials|Template|Layout|Partial`, `(*Editor).Catalog|Validate|Preview|SendDraft`.
- postcard constants and variables: `postcard.Migrations`, `postcard.TemplatesTable`, `postcard.LayoutsTable`, `postcard.PartialsTable`, `postcard.BrandTable`, `postcard.ImageLibraryTable`, `postcard.KindTemplate`, `postcard.KindLayout`, `postcard.KindPartial`, `postcard.KindBrand`.
- attach: `attach.AbsolutePublicURL`, `attach.ErrNoAbsoluteURL`.
- Built-in mail names: `postcard::mail.layouts.system`, `postcard::mail.layouts.minimal`, `postcard::mail.partials.button|panel|table|subcopy`; layout aliases `system`, `minimal` (fallback after plugin aliases).
- Template language: `{{ partial "<name>" key value ... }}`, `markdown` inside partials, layout key `logoUrl`, header keys `sample.<path>`.
- Files: assets `system.htm`, `minimal.htm`, `sample.htm`, `brand.css.tmpl`, `partials/{button,panel,table,subcopy}.htm`; smoke tests `TestMailOverrideSmoke`, `TestMailPartialsSmoke`, `TestMailEditorSmoke`, `TestMailBrandSmoke`.
- Config keys: none new (reads existing `mail.brandCss`, `mail.css`, `storage.uploads.public_url`, `storage.uploads.public_path_prefix`, `app.url`).
## Phase-wide source coverage audit
| Source | ID | Item | Plan | Status |
|--------|----|------|------|--------|
| GOAL | — | Edit every mail from the admin; list, override, reset; editable layouts with predefined ones; images in layouts and templates; override else file | 01-04 | COVERED |
| REQ | — | none mapped (Requirements: TBD); I18N-03 kept satisfied by the consumer-repo checks | 01, 04 | n/a |
| CONTEXT | D-01 | screens in cabana pseudo-plugin; postcard free of admin code | 01 (interface), 02 (screens) | COVERED |
| CONTEXT | D-02 | Settings → Mail entries | 02 | COVERED |
| CONTEXT | D-03 | two permission codes, no default grants, picker for template editors | 02, 03 | COVERED |
| CONTEXT | D-04 | override = subject + body + layout, description read-only | 01, 02 | COVERED |
| CONTEXT | D-05 | locale column and filter | 01 (TemplateLocale, sync), 02 | COVERED |
| CONTEXT | D-06 | reset to file version (as amended by D-23) | 01 (ResetOverride), 02 | COVERED |
| CONTEXT | D-07 | drift hash, notice, file version view/diff | 01 (hashes), 02 | COVERED |
| CONTEXT | D-08 | file + custom layouts, in-use delete refusal, predefined layouts | 01 (system, minimal), 02 | COVERED |
| CONTEXT | D-09 | codeeditor field type | 02 (server), 03 (SPA) | COVERED |
| CONTEXT | D-10 | Branding page with colours and logo | 01 (compile), 02 (page), 03 (fields) | COVERED |
| CONTEXT | D-11 | Branding replaces mail.brandCss once saved | 01 | COVERED |
| CONTEXT | D-12 | partials capability, built-ins, overrides | 01, 02 | COVERED |
| CONTEXT | D-13 | save validation | 01 (Validate), 02 (hooks) | COVERED |
| CONTEXT | D-14 | fallback at send | 01 | COVERED |
| CONTEXT | D-15 | sample variables, [name] placeholders | 01 | COVERED |
| CONTEXT | D-16 | live preview of the draft, sandboxed iframe | 01 (Preview), 02 (route), 03 (field) | COVERED |
| CONTEXT | D-17 | test send to own address, rate-limited | 01 (SendDraft), 02 (route), 03 (button) | COVERED |
| CONTEXT | D-18 | shared image library via attach | 02, 03 | COVERED |
| CONTEXT | D-19 | absolute public URLs | 01 (helper), 02 (library URLs) | COVERED |
| CONTEXT | D-20 | insert button in editors | 03 | COVERED |
| CONTEXT | D-21 | unit tests last, READMEs/docs, OpenAPI and dist gates | 01-03 (docs), 04 (tests, gate) | COVERED |
| CONTEXT | D-22 | postcard owns tables, migrations, per-send lookup | 01 | COVERED |
| CONTEXT | D-23 | index rows synced at serve start; reset clears columns | 01 (SyncIndex), 02 (serve-start call) | COVERED |
| CONTEXT | D-24 | logo URL built at send time from config | 01 | COVERED |
| CONTEXT | D-25 | no built-in template; empty-state text | 01, 02 | COVERED |
| RESEARCH | Pattern 1-3 | store split, index rows, per-send lookup | 01 | COVERED |
| RESEARCH | Pattern 4-6 | partials, samples, Branding | 01 | COVERED |
| RESEARCH | Pattern 7-9 | routes, permissions, rate limit, images, admin wiring | 02, 03 | COVERED |
| RESEARCH | Pitfall 5 | CodeMirror Mod-Enter | 03 | COVERED |
| RESEARCH | Pitfall 6, 9 | validate in FormAfterUpdate; parameterized messages | 02 | COVERED |
| RESEARCH | Package audit | @codemirror/view 6.43.15 [SUS too-new] → human checkpoint | 03 | COVERED |
| CONTEXT | Deferred | hand-written text parts, WYSIWYG body, CID images, arbitrary test recipient, SMTP screen, MailBlocker | — | excluded (deferred) |

View File

@@ -0,0 +1,472 @@
---
phase: 35-mail-templates-and-layouts-editor
plan: 02
type: execute
wave: 2
depends_on: ["35-01"]
files_modified:
- modules/cabana/backend_screens.go
- modules/cabana/mail_admin.go
- modules/cabana/mail_diff.go
- modules/cabana/mail_routes.go
- modules/cabana/field_mail.go
- modules/cabana/http.go
- modules/cabana/form_schema.go
- modules/cabana/crud.go
- modules/cabana/settings.go
- modules/cabana/query.go
- modules/cabana/messages.go
- modules/cabana/relation_field.go
- modules/cabana/admin_openapi.go
- modules/cabana/mail_screens_smoke_test.go
- modules/cabana/phase10_coverage_test.go
- modules/cabana/openapi_conformance_test.go
- modules/cabana/security_coverage_test.go
- modules/cabana/token_routes_test.go
- modules/cabana/backendscreens/controllers/mailtemplates/config_form.yaml
- modules/cabana/backendscreens/controllers/mailtemplates/config_list.yaml
- modules/cabana/backendscreens/controllers/mailtemplates/config_filter.yaml
- modules/cabana/backendscreens/controllers/mailtemplates/_info.htm
- modules/cabana/backendscreens/controllers/mailtemplates/_file_version.htm
- modules/cabana/backendscreens/controllers/maillayouts/config_form.yaml
- modules/cabana/backendscreens/controllers/maillayouts/config_list.yaml
- modules/cabana/backendscreens/controllers/maillayouts/config_filter.yaml
- modules/cabana/backendscreens/controllers/maillayouts/_info.htm
- modules/cabana/backendscreens/controllers/maillayouts/_file_version.htm
- modules/cabana/backendscreens/controllers/mailpartials/config_form.yaml
- modules/cabana/backendscreens/controllers/mailpartials/config_list.yaml
- modules/cabana/backendscreens/controllers/mailpartials/config_filter.yaml
- modules/cabana/backendscreens/controllers/mailpartials/_info.htm
- modules/cabana/backendscreens/controllers/mailpartials/_file_version.htm
- modules/cabana/backendscreens/controllers/mailimages/config_form.yaml
- modules/cabana/backendscreens/models/mailtemplate/fields.yaml
- modules/cabana/backendscreens/models/mailtemplate/columns.yaml
- modules/cabana/backendscreens/models/maillayout/fields.yaml
- modules/cabana/backendscreens/models/maillayout/columns.yaml
- modules/cabana/backendscreens/models/mailpartial/fields.yaml
- modules/cabana/backendscreens/models/mailpartial/columns.yaml
- modules/cabana/backendscreens/models/mailbrandsetting/fields.yaml
- modules/cabana/backendscreens/models/mailimagelibrary/fields.yaml
- modules/phrasebook/backend/lang/en/lang.yaml
- modules/phrasebook/backend/lang/pl/lang.yaml
- modules/surf/serve.go
- modules/surf/README.md
- admin/openapi/admin.json
- admin/src/api/schema.d.ts
- modules/cabana/README.md
- docs/backend/forms.md
- docs/backend/settings.md
- docs/backend/users-and-permissions.md
- docs/backend/lists-and-filters.md
- docs/services/mail.md
- docs/services/rate-limiting.md
autonomous: true
requirements: [D-01, D-02, D-03, D-04, D-05, D-06, D-07, D-08, D-09, D-10, D-13, D-16, D-17, D-18, D-19, D-23, D-25]
estimate:
tokens: 210000
raw_tokens: 210000
tasks: 4
confidence: low
must_haves:
truths:
- "Every app whose postcard Mailer is published gets five Settings → Mail entries (Mail templates, Mail layouts, Mail partials, Mail branding, Mail images) from cabana's internal pseudo-plugin, with no top-level navigation item and no app or plugin setup (D-01, D-02)"
- "cabana registers `system.manage_mail_templates` (templates list/form, template preview and test send) and `system.manage_mail_layouts` (layouts, partials, Branding, image library) with no role grants; a holder of only the templates code sees one tile and gets 403 on layouts, partials, Branding, image upload and layout or partial preview; the image list and picker answer either code (D-03)"
- "Every file template, layout and partial appears as a row after serve start (postcard.SyncIndex through lagoon.OnDatabase, retried on the first list request when the tables were missing); rows absent from this process's catalog are hidden, database-only layouts are always listed (D-23, D-08)"
- "The templates list shows code, effective subject, description, locale (suffix or Default) and layout, with a locale filter and a Customized / Not customized / File changed status filter; the empty view shows the UI-SPEC empty-state text (D-05, D-25)"
- "Saving a template stores subject, Markdown body and layout (a dropdown over file and custom layouts) as the override, stamps base_hash from the current file hash, and runs postcard's draft validation inside the save transaction; a failure is a 422 field line with the translated rule text and :message, and nothing is saved (D-04, D-13)"
- "Reset to file version, a record action on customized template, file layout and partial rows, clears the override columns and re-mirrors the file; custom layouts are created with a lowercase-dash code and deleted, a delete of one still used by customized templates is a 403 naming the layout and the count, and file rows can never be deleted (D-06 as amended by D-23, D-08)"
- "A customized row whose file changed carries the `customized` and `outdated` row states, the File changed filter finds it, the form shows the drift callout and a File version tab with a line diff of the serialized override against the file, and saving again clears the drift (D-07)"
- "Layouts and partials are edited in the new built-in `codeeditor` field type (`language: html | css | text`, optional size and mailImages), registered in cabana's field registry and the form schema; the layout form has HTML, Plain text, CSS and File version tabs (D-09)"
- "The Branding page is a cabana settings form on system_mail_brand_settings with the 13 colours as `colorpicker` fields and the logo as a `mailimage` field; an invalid colour is a 422 field line, and saving it is what switches mail from mail.brandCss to the compiled colours (D-10, D-11)"
- "Mail images is a singleton screen whose fileupload stores public images through lagoon/attach; GET/POST /mail/images return absolute public URLs from config and refuse with a clear message when neither storage.uploads.public_url nor app.url is set (D-18, D-19, D-24)"
- "POST /mail/preview renders the unsaved draft (template, layout, partial or branding mode) through postcard's Editor with sample values and returns subject, HTML and text; POST /mail/test-send sends the draft only to the signed-in administrator's own email, refuses any recipient in the body, needs the record kind's permission, is interactive-session only and answers 429 with Retry-After after 5 sends per 10 minutes per administrator (D-16, D-17)"
- "admin/openapi/admin.json and admin/src/api/schema.d.ts are regenerated from the annotations, the route inventories list the new routes, and the cabana README and docs pages describe every new screen, permission, route and field type in the same commits (D-21)"
prohibitions:
- statement: "No role is granted system.manage_mail_templates or system.manage_mail_layouts in code or migrations."
status: unresolved
verification: judgment
- statement: "The test-send route accepts no recipient: the address is always the signed-in administrator's own email."
status: unresolved
verification: judgment
- statement: "cabana's mail models carry no postcard rendering logic of their own: validation, preview and sends go through postcard.Editor so previews match real sends."
status: unresolved
verification: judgment
- statement: "Existing controllers see no behaviour change: meta.deletable is omitted unless a mail controller sets it, and the three existing row states keep their order and rendering."
status: unresolved
verification: judgment
- statement: "Framework code, READMEs and docs name no consuming application."
status: unresolved
verification: judgment
artifacts:
- path: "modules/cabana/mail_admin.go"
provides: "mail models, controllers, hooks, row states, reset action, delete refusal"
contains: "PermissionManageMailTemplates"
- path: "modules/cabana/mail_routes.go"
provides: "preview, test-send and image library routes"
contains: "func (s *service) mailTestSend("
- path: "modules/cabana/mail_diff.go"
provides: "bounded LCS line diff for the File version tab"
contains: "func lineDiff("
- path: "modules/cabana/field_mail.go"
provides: "codeeditor, colorpicker, mailimage, mailpreview schema checks"
contains: "codeeditor"
- path: "modules/cabana/mail_screens_smoke_test.go"
provides: "TestMailScreensSmoke (tracer and expansion subtests)"
contains: "func TestMailScreensSmoke("
- path: "modules/cabana/backendscreens/controllers/mailtemplates/config_list.yaml"
provides: "templates list"
contains: "mail.template.list_title"
key_links:
- from: "modules/cabana/mail_admin.go"
to: "modules/postcard/editor.go"
via: "FormAfterUpdate validates through postcard.Editor.Validate"
pattern: "\\.Validate\\(ctx"
- from: "modules/cabana/http.go"
to: "modules/postcard/store.go"
via: "serve-start index sync"
pattern: "postcard\\.SyncIndex"
- from: "modules/cabana/mail_routes.go"
to: "modules/cooler/cooler.go"
via: "per-admin test-send limit through cooler.Attempter, published by surf"
pattern: "cooler\\.Attempter"
- from: "modules/cabana/backend_screens.go"
to: "modules/cabana/mail_admin.go"
via: "pseudo-plugin AdminControllers, Permissions and Settings include the mail screens when postcard.EditorFrom succeeds"
pattern: "postcard\\.EditorFrom"
---
## Phase Goal
ROADMAP Goal (not in user-story format; no story is invented): administrators edit every mail the application sends from the admin panel, as in WinterCMS's Mail templates screen. This plan delivers the server side of the screens on top of 35-01's postcard API: the Settings → Mail entries, two permissions, the templates, layouts and partials screens with reset and drift, Branding, the image library, and the preview and test-send routes the SPA (35-03) calls.
Spec-less probe fallback skipped: no SPEC.md and no requirement IDs; must_haves derive from the ROADMAP goal and CONTEXT D-01..D-25. Names come from 35-01's Task 1 answer (read 35-01-SUMMARY.md; where it differs from this text, the SUMMARY wins).
<objective>
Add the mail screens to cabana's internal pseudo-plugin (D-01, the 19.6.1 pattern):
- templates, layouts, partials and image library controllers over postcard's tables, Branding as a settings model (D-02, D-08, D-10, D-18);
- the two permission codes with no default grants and the picker split (D-03);
- override save with validation, reset, custom layouts with in-use delete refusal, drift states, filters and the File version diff (D-04..D-08, D-13, D-23);
- the built-in field types codeeditor, colorpicker, mailimage and mailpreview on the server (D-09, UI-SPEC C1, C2, C10, C11);
- preview, test-send and image library routes with OpenAPI annotations (D-16, D-17, D-19);
- all UI-SPEC copy in en and pl, the cabana README and docs.
Purpose: 35-03 renders these screens with the generic SPA views plus four field controls; every rule that protects mail (permissions, validation, recipient, rate limit) lives here on the server.
Output: commits in this repository only. Nothing is pushed or tagged.
**Planner decisions recorded here:**
- The mail screens, permissions and settings entries are added only when `postcard.EditorFrom(app)` succeeds (always true in an application, since party.Activate runs postcard.Activate; false in cabana tests that never activate postcard, which therefore see no change).
- Image picker: list and pick for holders of either mail code; upload only for `system.manage_mail_layouts` (D-03 planner's call, UI-SPEC C4 recommendation).
- Test-send limit: 5 per administrator per 10 minutes, key `mail_test:<adminID>`, through `cooler.Attempter` when surf publishes one (Redis store), otherwise an in-process fixed window in cabana (research Pattern 7, A6, A7).
- The custom-layout create form is prefilled from postcard's default layout, injected into the compiled form schema at Activate, so the YAML carries no copy of it (UI-SPEC C5).
- New copy key beyond the UI-SPEC: `mail.images.no_public_url` (EN "Mail images need an absolute address. Set storage.uploads.public_url or app.url in the configuration." PL "Obrazy w wiadomościach wymagają pełnego adresu. Ustaw storage.uploads.public_url lub app.url w konfiguracji.").
</objective>
<execution_context>
@~/.claude/msd-core/workflows/execute-plan.md
@~/.claude/msd-core/templates/summary.md
</execution_context>
<context>
@.planning/phases/35-mail-templates-and-layouts-editor/35-CONTEXT.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-PATTERNS.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-01-SUMMARY.md
@.planning/phases/19.6.1-backend-administrators-screen/19.6.1-CONTEXT.md
@CLAUDE.md
<interfaces>
From 35-01 (postcard): Migrations and table constants TemplatesTable, LayoutsTable, PartialsTable, BrandTable, ImageLibraryTable; SyncIndex(ctx, db, cat, locales); ResetOverride(ctx, db, cat, kind, code); EditorFrom(app) (*Editor, bool); Editor.Catalog() / Validate(ctx, Draft) *DraftError / Preview(ctx, Draft) (Preview, error) / SendDraft(ctx, Draft, to); Draft{Kind, Name, Subject, Body, Layout, HTML, Text, CSS, Brand}; Kind consts KindTemplate|KindLayout|KindPartial|KindBrand; DraftError{Field, Rule, Message, Name}; Catalog.Template|Layout|Partial(name) and Templates|Layouts|Partials(); TemplateFile/LayoutFile/PartialFile{..., Source, Hash}; FormatTemplate/FormatLayout/FormatPartial; TemplateLocale(name, locales); Brand, DefaultBrand(), CompileBrandCSS(Brand); attach.AbsolutePublicURL(cfg, key), attach.ErrNoAbsoluteURL; pact.RowStateCustomized, pact.RowStateOutdated.
modules/cabana/backend_screens.go: backendScreens pseudo-plugin (ID summercms.cabana; AdminFS from //go:embed all:backendscreens; AdminControllers, Permissions, Settings); permission/controller constants next to PermissionManageUsers; withBackendScreens.
modules/cabana/backend_users_admin.go: the controller to mirror (interface assertions block, ID/ModelName/ConfigDir/RequiredPermissions/NewRecord, db()/requestDB(ctx)/translator(), AdminFieldRelations for a belongsTo, ListRowStates, FormBeforeDelete, AdminRecordActions, PartialData, FormRules); recordDataExtender (crud.go ~699) and schemaFieldHider (http.go ~894) are unexported seams.
modules/cabana/http.go: Activate (~81) appends the pseudo-plugin; route group (~305-445): `g.Post("/markdown/preview", requireAjax(s.markdownPreview))`, interactiveOnly wrapper for account routes.
modules/cabana/field_markdown.go (~50-80): AdminMarkdownPreviewRequest/Result, the bouncer.User + principal.Backend + tokenRouteGate gate, WriteError with fixed msg constants.
modules/cabana/contracts.go: Allows(principal, required) any-of with superuser bypass (~327); BackendUser{ID, Email, ...}.
modules/cabana/form_schema.go (~24-51): formFieldTypes, formFieldKeys; crud.go (~1518) scalarFormField; settings.go (~96-102) settings refusal list and BeforeValidate hook use (~261).
modules/cabana/query.go (~182) rowStateOrder; messages.go (~42, 61, 163) row-state message keys; relation_field.go (~104) RecordMeta.
modules/cabana/administrators_harness_test.go (~154) newScreensEnv: compass config, lagoon.Publish, phrasebook.Activate, surf.Assemble (does not activate postcard).
modules/surf/serve.go (~230) installCooler publishes Store only for the Redis rate-limit store; modules/cooler/cooler.go (~68) Attempter has exactly surf.Store's method set.
Route inventories that list every admin route: modules/cabana/phase10_coverage_test.go, openapi_conformance_test.go, security_coverage_test.go, token_routes_test.go (grep `/markdown/preview` in each).
</interfaces>
</context>
## Execution units and resume points
- Phase 33 edits cabana concurrently. Before each task run `git status --short modules/cabana modules/phrasebook admin/openapi admin/src/api modules/surf` and stop if another session holds uncommitted edits there; re-check the line numbers named in interfaces before editing.
- Each task ends with the commit listed below, staged by explicit path. Resume check: `git log --format=%s -F --grep='<subject>'`.
- Docker-backed subtests run without `-short` (through `sudo -u golem` if the Docker socket is denied).
| Task | Commit subject | Resume point |
|------|----------------|--------------|
| 1 | `feat(cabana): mail templates screen over postcard overrides` | TestMailScreensSmoke/templates passes |
| 2 | `feat(cabana): mail layouts and partials screens with reset, drift and custom layouts` | TestMailScreensSmoke/layouts passes |
| 3 | `feat(cabana): code editor, colour and mail image fields, mail branding and the image library` | TestMailScreensSmoke/branding passes |
| 4 | `feat(cabana): mail preview and test send to yourself` | TestMailScreensSmoke/test_send passes |
<tasks>
<task type="tracer">
<name>Task 1: End-to-end "an administrator with system.manage_mail_templates opens Settings → Mail templates, saves a subject override and the next mail uses it" — templates only</name>
<reversibility rating="costly">D-03: the permission codes end up in backend_user_roles.permissions and backend_users.permissions JSON on every install that grants them.</reversibility>
<precondition>35-01 is complete: `git log --format=%s -F --grep='feat(postcard): mail branding and the logo URL'` prints one line and `grep -n "func EditorFrom" modules/postcard/editor.go` prints one line</precondition>
<files>modules/cabana/backend_screens.go, modules/cabana/mail_admin.go, modules/cabana/http.go, modules/cabana/backendscreens/controllers/mailtemplates/config_form.yaml, modules/cabana/backendscreens/controllers/mailtemplates/config_list.yaml, modules/cabana/backendscreens/models/mailtemplate/fields.yaml, modules/cabana/backendscreens/models/mailtemplate/columns.yaml, modules/cabana/mail_screens_smoke_test.go, modules/phrasebook/backend/lang/en/lang.yaml, modules/phrasebook/backend/lang/pl/lang.yaml, modules/cabana/README.md, docs/backend/users-and-permissions.md, docs/services/mail.md</files>
<read_first>
- modules/cabana/backend_screens.go (whole file)
- modules/cabana/backend_users_admin.go (~1-200: assertions, identity, helpers, relations, row states, FormBeforeDelete) and backend_roles_admin.go (a smaller controller)
- modules/cabana/backendscreens/controllers/users/config_form.yaml, config_list.yaml and modules/cabana/backendscreens/models/user/fields.yaml, columns.yaml
- modules/cabana/http.go (~80-150 Activate)
- modules/lagoon/ondatabase.go (OnDatabase: queued errors are returned by Publish, so the hook must not return the sync error)
- modules/cabana/administrators_harness_test.go (~142-360 newScreensEnv, seedAdmin, seedRole, login, request)
- modules/postcard/editor.go and modules/postcard/store.go (35-01 output) and .planning/phases/35-mail-templates-and-layouts-editor/35-01-SUMMARY.md
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md (C3b, C14, Copywriting "Settings entries and permission catalog", "Lists", "Template form")
</read_first>
<action>
Per D-01, D-02, D-03, D-04, D-05, D-13, D-23 and D-25:
- `backend_screens.go`: exported constants `PermissionManageMailTemplates = "system.manage_mail_templates"`, `PermissionManageMailLayouts = "system.manage_mail_layouts"`, `MailTemplatesControllerID = "summercms.cabana.mailtemplates"`, `MailLayoutsControllerID`, `MailPartialsControllerID`, `MailImagesControllerID` (P35-05 names). The pseudo-plugin holds the *postcard.Editor found by `postcard.EditorFrom(app)` when it is built; only then do AdminControllers, Permissions and Settings include the mail parts. In this task: both permission codes (Tab `backend::lang.permissions.tab_system`, Labels `backend::lang.permissions.manage_mail_templates` / `manage_mail_layouts`, no Roles), the templates controller and the settings link `mail_templates` (category `backend::lang.settings.category_mail`, icon `mail`, order 200, Permissions templates code, Controller MailTemplatesControllerID).
- `mail_admin.go`: model `MailTemplate` (TableName postcard.TemplatesTable; columns per P35-01; belongsTo `layout` through layout_id with a minimal `MailLayout` model, TableName postcard.LayoutsTable, enough for the relation and its name column; Fillable subject, body and the layout foreign key) and controller `mailTemplatesAdmin` (ID, ModelName, ConfigDir `controllers/mailtemplates`, RequiredPermissions templates code, NewRecord) with: AdminFieldRelations for `layout` (belongsTo, LabelColumn name) plus pact.RelationExtendOptionsQuery limiting options to layouts present in this process's catalog or database-only; ListExtendQuery and FormExtendQuery hiding template rows whose code is not in the catalog; FormRecordTitle = code; FormBeforeCreate and FormBeforeDelete refusing with ForbiddenError (templates come from files); FormBeforeUpdate stamping is_custom true, base_hash = the catalog file Hash and customized_at = now; FormAfterUpdate building a postcard Draft (KindTemplate, Name code, Subject, Body, Layout = the code of the assigned layout row) and calling Editor.Validate; a *DraftError becomes a ValidationError on its Field with the text of `backend::lang.mail.invalid_<rule>` translated with :message and :name (build it with the translator as account_tokens.go backendMessageParams does, research Pitfall 9).
- Index sync per D-23: in Activate (when the editor exists) register `lagoon.OnDatabase` with a func that calls postcard.SyncIndex with the app's phrasebook translator locales (`Locales("")`, empty when none is published), logs any error through cabana's logger and returns nil; until a sync has succeeded, the first mail list request in the process retries it.
- YAML: controllers/mailtemplates/config_form.yaml (name `backend::lang.mail.template.name`, form `models/mailtemplate/fields.yaml` in the users-controller path style, no create, update redirect to the update form, Save and close to the list, messages per UI-SPEC C3b); config_list.yaml (title `backend::lang.mail.template.list_title`, recordUrl to update, `toolbar.buttons: []`, no checkboxes, recordsPerPage 20, noRecordsMessage `backend::lang.mail.template.list_empty`, default sort code ascending); models/mailtemplate/columns.yaml (code searchable+sortable, subject searchable, description searchable, locale sortable, layout through the relation's name, customized_at invisible sortable); models/mailtemplate/fields.yaml (subject text left required with comment, layout relation right required without an empty option, body markdown size giant, full) — the `_info`, `_preview` and `_file_version` fields and `mailImages` arrive in Tasks 2-4.
- Copy en and pl (UI-SPEC tables, Polish as the design language): `settings.category_mail`, `permissions.manage_mail_templates[_comment]`, `permissions.manage_mail_layouts[_comment]`, the `mail.template.*` keys used here, `mail.code`, `mail.template.locale_default`.
- `mail_screens_smoke_test.go` (package cabana_test): helper `newMailScreensEnv(t)` that follows newScreensEnv but also calls postcard.Activate and postcard.BootPlugin for a fixture plugin `acme.blog` (fstest.MapFS: `mail.welcome`, `mail.welcome-en`, one layout alias) before surf.Assemble, and runs the index sync; `TestMailScreensSmoke/templates`: a superuser lists templates and sees both rows with locale `en` on the suffixed one; GET schema of the form; PUT a new subject on `acme.blog::mail.welcome` gives 200 and is_custom true; the published postcard Mailer then sends that subject; a PUT with a `<script>` body gives 422 with Details.body and leaves the row unchanged; a role holding only system.manage_mail_layouts gets 403 on the templates list; a role holding only the templates code gets 200; no role in the database gained either code; GET /settings shows exactly one Mail tile for the templates-only role.
- Docs in the same commit: cabana README (Mail screens overview, the two permission codes, the controller ID constants, the serve-start sync), docs/backend/users-and-permissions.md (the two codes, no default grant, what each opens), docs/services/mail.md section "Editing mail in the admin" (which screens exist, what is stored, that the override wins until reset).
Commit as listed in Execution units.
</action>
<verify>
<automated>go test ./modules/cabana -run 'TestMailScreensSmoke' -count=1 -v 2>&1 | grep -E '^\s*(--- |FAIL|ok|PASS)' | tail -20</automated>
<fails_when>a "FAIL" line, a "--- SKIP" line, "no tests to run", or no "--- PASS: TestMailScreensSmoke/templates" line</fails_when>
<automated>go vet ./... && go test -short ./...</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>go test ./modules/cabana -count=1 2>&1 | tail -5</automated>
<fails_when>non-zero exit or a "FAIL" line (existing cabana Postgres tests must stay green)</fails_when>
<automated>go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree' -count=1 2>&1 | tail -3 && go run ./cmd/summer docs:build --check 2>&1 | tail -5</automated>
<fails_when>non-zero exit, or a "FAIL" or "unknown identifier" line in the output</fails_when>
</verify>
<acceptance_criteria>
- `grep -n 'PermissionManageMailTemplates = "system.manage_mail_templates"' modules/cabana/backend_screens.go` and `grep -n 'PermissionManageMailLayouts = "system.manage_mail_layouts"' modules/cabana/backend_screens.go` each print one line
- `grep -n "postcard.EditorFrom" modules/cabana/backend_screens.go modules/cabana/http.go` prints at least one line
- `grep -n "postcard.SyncIndex" modules/cabana/*.go | grep -v _test` prints at least one line
- `grep -n "category_mail" modules/phrasebook/backend/lang/en/lang.yaml modules/phrasebook/backend/lang/pl/lang.yaml` prints two lines
- `grep -rn "manage_mail" modules/cabana/*.go | grep -v _test | grep -n "Roles:"` prints nothing
- `grep -n "system.manage_mail_templates" docs/backend/users-and-permissions.md modules/cabana/README.md` prints at least two lines
</acceptance_criteria>
<done>An administrator can find every file template in Settings → Mail templates, save an override that real sends use, and is refused unsafe content and the wrong permission.</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Layouts and partials screens, reset to the file version, custom layouts with the in-use delete refusal, drift states and the File version diff</name>
<reversibility rating="costly">D-07: the drift states read base_hash and file_hash columns; D-08: the custom-layout code is what template overrides store.</reversibility>
<files>modules/cabana/mail_admin.go, modules/cabana/mail_diff.go, modules/cabana/backend_screens.go, modules/cabana/query.go, modules/cabana/messages.go, modules/cabana/relation_field.go, modules/cabana/crud.go, modules/cabana/backendscreens/controllers/mailtemplates/config_form.yaml, modules/cabana/backendscreens/controllers/mailtemplates/config_list.yaml, modules/cabana/backendscreens/controllers/mailtemplates/config_filter.yaml, modules/cabana/backendscreens/controllers/mailtemplates/_info.htm, modules/cabana/backendscreens/controllers/mailtemplates/_file_version.htm, modules/cabana/backendscreens/controllers/maillayouts/config_form.yaml, modules/cabana/backendscreens/controllers/maillayouts/config_list.yaml, modules/cabana/backendscreens/controllers/maillayouts/config_filter.yaml, modules/cabana/backendscreens/controllers/maillayouts/_info.htm, modules/cabana/backendscreens/controllers/maillayouts/_file_version.htm, modules/cabana/backendscreens/controllers/mailpartials/config_form.yaml, modules/cabana/backendscreens/controllers/mailpartials/config_list.yaml, modules/cabana/backendscreens/controllers/mailpartials/config_filter.yaml, modules/cabana/backendscreens/controllers/mailpartials/_info.htm, modules/cabana/backendscreens/controllers/mailpartials/_file_version.htm, modules/cabana/backendscreens/models/mailtemplate/fields.yaml, modules/cabana/backendscreens/models/maillayout/fields.yaml, modules/cabana/backendscreens/models/maillayout/columns.yaml, modules/cabana/backendscreens/models/mailpartial/fields.yaml, modules/cabana/backendscreens/models/mailpartial/columns.yaml, modules/cabana/mail_screens_smoke_test.go, modules/phrasebook/backend/lang/en/lang.yaml, modules/phrasebook/backend/lang/pl/lang.yaml, modules/cabana/README.md, docs/backend/lists-and-filters.md, docs/services/mail.md</files>
<read_first>
- modules/cabana/mail_admin.go and mail_screens_smoke_test.go (Task 1 output)
- modules/cabana/backend_users_admin.go (~178-200 ListRowStates, ~199 FormBeforeDelete, ~325 AdminRecordActions, ~422 PartialData) and modules/cabana/backendscreens/controllers/users/config_filter.yaml, _hint.htm
- modules/cabana/partial_render.go (~105-118 allowlist) and modules/cabana/account_tokens.go (~200-230 backendMessageParams)
- modules/cabana/query.go (~160-200), modules/cabana/messages.go (~35-70, ~150-170), modules/cabana/relation_field.go (~95-120), modules/cabana/crud.go (show response that fills RecordMeta.Actions)
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md (C5, C6, C7, C8 items 1-2, C9, C15, Copywriting "Layout form", "Partial form", "Info partial and drift notice", "Record actions and destructive confirmations")
</read_first>
<behavior>
- Layouts list shows postcard's default first, then by name, including system, minimal, the fixture plugin layout and database-only layouts (source shown as Custom); partials list shows the four built-ins and the fixture plugin partial
- Creating a layout with code `promo-2026` and the default layout content succeeds (is_custom true, source NULL); code `Promo 2026` gives 422 mail.layout.code_format on code; a taken code gives mail.layout.code_taken
- Saving a file layout's HTML without {{ .Content }} gives 422 on content_html with the no_content text; saving a valid change marks it customized and the next send uses it
- A template override pointing at the custom layout, then DELETE of that layout: 403 whose message names the layout and the count 1 (CLDR one form); after resetting the template, the delete succeeds; DELETE of a file layout, a template or a partial is 403
- Record action reset on a customized template restores the file subject, body and layout, clears is_custom, and the record's meta.actions no longer offers reset; reset is not offered on a not-customized row nor on a custom layout
- meta.deletable is false for templates, partials and file layouts, true for custom layouts, and absent for every existing controller
- After changing the fixture file's bytes and re-syncing, the customized row carries row states [customized, outdated], the status filter `outdated` returns it, `_info` contains the drift callout and `_file_version` contains summer-diff__add and summer-diff__del lines; saving the row again clears outdated
- lineDiff with 3 lines of context returns the expected add/del/ctx sequence for a small pair and returns no diff (content block only) above the line cap
</behavior>
<action>
Per D-06 (as amended by D-23), D-07, D-08 and D-12 (admins override and reset partials):
- Models `MailLayout` (full columns) and `MailPartial`; controllers `mailLayoutsAdmin` (ConfigDir controllers/maillayouts, RequiredPermissions layouts code) and `mailPartialsAdmin` (controllers/mailpartials, layouts code), with the same catalog-membership query scoping (database-only layouts always included), stamping in FormBeforeUpdate and postcard validation in FormAfterUpdate (KindLayout with Name = code for file layouts and the custom code otherwise; KindPartial) as Task 1. Layouts: create allowed only for custom layouts (FormBeforeCreate checks the code against `^[a-z0-9-]+$` and uniqueness with mail.layout.code_format / code_taken, sets source NULL and is_custom true); code is create-only (hidden and not writable on update through schemaFieldHider and the update rules); default sort default layout first, then name. Partials: no create, no delete.
- Record action `reset` on all three controllers (Label `backend::lang.mail.reset`, Confirm the kind's `mail.<kind>.reset_confirm`, Danger true, Permissions the controller's code, Applies when the row is file-backed and is_custom) whose Run calls postcard.ResetOverride on the action transaction and returns Message `backend::lang.mail.reset_done`. The templates, layouts and partials config_form.yaml declare `recordActions: [reset]`.
- Delete: FormBeforeDelete refuses file-backed rows with ForbiddenError, and a custom layout still referenced by template rows with a ForbiddenError whose Message is `backend::lang.mail.layout.delete_in_use` translated with :name and the CLDR plural for :count (UI-SPEC C15 note: never a ValidationError). The FK ON DELETE RESTRICT stays as the backstop.
- `meta.deletable`: add `Deletable *bool` with json `deletable,omitempty` to RecordMeta and fill it in the show response only for controllers implementing a new unexported cabana interface (mail controllers): false for templates, partials and file layouts, true for custom layouts.
- Row states: rowStateOrder becomes deleted, negative, disabled, customized, outdated; messages.go gains RowStateCustomized / RowStateOutdated (yaml `rowStateCustomized`, `rowStateOutdated`, defaults `backend::lang.messages.list.row_state_customized` / `row_state_outdated`, json keys in the list messages schema); the mail controllers' ListRowStates give customized for is_custom file-backed rows and also outdated when base_hash differs from file_hash; the three config_list.yaml files set `rowStateCustomized: backend::lang.mail.state_customized` and `rowStateOutdated: backend::lang.mail.state_outdated`.
- Filters (config_filter.yaml, scope type with options as the users `trashed` filter): templates `locale` (default plus one option per locale value found, through pact.FilterOptions) and `status` (customized, not_customized, outdated); layouts and partials `status`. Outdated SQL per P35-01.
- Partials `_info.htm` and `_file_version.htm` for each controller (allowlisted tags only; UI-SPEC C7 content and order; the partial usage line shows the exact P35-02 call, a bare name for postcard built-ins and the full name otherwise; variables line lists the file's sample names), fed by PartialData with curated view models (never a GORM model). `mail_diff.go`: `lineDiff(a, b []string, context int) []diffLine` (stdlib LCS, cap 2000 lines per side, above which only the file content block is shown); both sides serialized with postcard.FormatTemplate / FormatLayout / FormatPartial; the content block shows the catalog Source.
- Template fields.yaml gains `_info` (partial, first, full) and `_file_version` (partial, tab `backend::lang.mail.tab_file_version`, context update); layouts and partials fields.yaml per UI-SPEC C5/C6, using `type: textarea` for content_html, content_text and content_css in this task (Task 3 switches them to codeeditor).
- Copy en and pl for every key used: `mail.layout.*`, `mail.partial.*`, `mail.name_field`, `mail.source`, `mail.source_custom`, `mail.customized_at`, the filter and state keys, `mail.info_*`, `mail.drift_*`, `mail.file_*`, `mail.reset`, `mail.reset_done`, the reset confirms, `mail.layout.delete_in_use` (CLDR maps), `mail.invalid_*`, `mail.layout.code_*`, `messages.list.row_state_customized` / `row_state_outdated`.
- Write the behavior subtests first in mail_screens_smoke_test.go (`layouts`, `partials`, `reset`, `delete`, `drift`, `deletable`, `diff`), then implement.
- Docs in the same commit: cabana README (layouts and partials screens, reset, custom layouts, RecordMeta.Deletable, the two new row states), docs/backend/lists-and-filters.md "Row state" (customized and outdated), docs/services/mail.md (layouts, partials, reset and the drift notice in the admin).
Commit as listed in Execution units.
</action>
<verify>
<automated>go test ./modules/cabana -run 'TestMailScreensSmoke' -count=1 -v 2>&1 | grep -E '^\s*(--- |FAIL|ok|PASS)' | tail -30</automated>
<fails_when>a "FAIL" line, a "--- SKIP" line, "no tests to run", or no "--- PASS: TestMailScreensSmoke/drift" or "--- PASS: TestMailScreensSmoke/delete" line</fails_when>
<automated>go vet ./... && go test -short ./... && go test ./modules/cabana -count=1 2>&1 | tail -5</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree' -count=1 2>&1 | tail -3 && go run ./cmd/summer docs:build --check 2>&1 | tail -5</automated>
<fails_when>non-zero exit, or a "FAIL" or "unknown identifier" line in the output</fails_when>
</verify>
<acceptance_criteria>
- `grep -n "pact.RowStateCustomized, pact.RowStateOutdated" modules/cabana/query.go` prints one line
- `grep -n 'json:"deletable,omitempty"' modules/cabana/relation_field.go` prints one line
- `grep -n "postcard.ResetOverride" modules/cabana/mail_admin.go` prints at least one line
- `grep -n "func lineDiff(" modules/cabana/mail_diff.go` prints one line
- `grep -c "recordActions: \[reset\]" modules/cabana/backendscreens/controllers/mail*/config_form.yaml | grep -c ':1'` prints 3
- `grep -n "delete_in_use" modules/phrasebook/backend/lang/pl/lang.yaml -A 5 | grep -E "few|many"` prints at least two lines
</acceptance_criteria>
<done>Admins can customize, reset and review drift on templates, layouts and partials, create and safely delete their own layouts, and every list shows which records differ from the files.</done>
</task>
<task type="auto" tdd="true">
<name>Task 3: New field types on the server, the Branding page and the mail image library with absolute URLs</name>
<reversibility rating="costly">D-09: `codeeditor` (and colorpicker, mailimage, mailpreview) become published field type names in fields.yaml that plugins may start using.</reversibility>
<files>modules/cabana/field_mail.go, modules/cabana/form_schema.go, modules/cabana/crud.go, modules/cabana/settings.go, modules/cabana/mail_admin.go, modules/cabana/mail_routes.go, modules/cabana/http.go, modules/cabana/backend_screens.go, modules/cabana/admin_openapi.go, modules/cabana/backendscreens/models/mailtemplate/fields.yaml, modules/cabana/backendscreens/models/maillayout/fields.yaml, modules/cabana/backendscreens/models/mailpartial/fields.yaml, modules/cabana/backendscreens/models/mailbrandsetting/fields.yaml, modules/cabana/backendscreens/models/mailimagelibrary/fields.yaml, modules/cabana/backendscreens/controllers/mailimages/config_form.yaml, modules/cabana/mail_screens_smoke_test.go, modules/cabana/phase10_coverage_test.go, modules/cabana/openapi_conformance_test.go, modules/cabana/security_coverage_test.go, modules/cabana/token_routes_test.go, admin/openapi/admin.json, admin/src/api/schema.d.ts, modules/phrasebook/backend/lang/en/lang.yaml, modules/phrasebook/backend/lang/pl/lang.yaml, modules/cabana/README.md, docs/backend/forms.md, docs/backend/settings.md</files>
<read_first>
- modules/cabana/form_schema.go (~20-60 and the per-type key checks), modules/cabana/crud.go (~1510-1540 scalarFormField, valueFormField), modules/cabana/settings.go (~70-140, ~200-280)
- modules/cabana/field_file.go and modules/cabana/field_type_upload.go (how fileupload attaches to a record: attachment_type, field, is_public, limits), modules/cabana/singleton.go (~41-80)
- modules/cabana/example_settings_test.go (settings registration example) and modules/cabana/admin_openapi.go (swag block style, ~460-480)
- modules/lagoon/attach/store.go (~60-125 Upload, Limits, Store) and modules/lagoon/attach/absolute.go (35-01)
- scripts/check-admin-openapi.sh
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md (C1, C2, C4 permission, C11, C12, C13, C14, C15, Copywriting "Image picker and image fields", "Code editor and colour picker", "Branding page")
</read_first>
<behavior>
- A fields.yaml with `type: codeeditor` and `language: html` compiles; `language: php` or a missing language is a boot error naming controller, field and value; `mailImages: true` is accepted on markdown and codeeditor and is a boot error elsewhere; colorpicker with `availableColors` is a boot error (unknown key)
- The form schema JSON carries `language` and `mailImages` for those fields; codeeditor, colorpicker and mailimage values bind to their columns and round-trip through PUT
- GET /settings/mail_brand on a fresh database returns exists false with the UI-SPEC Winter defaults; PUT with `button_primary_bg: "#D66829"` stores `#d66829` and creates row 1; PUT with `red` gives 422 colorpicker.invalid on that field; PUT with a logo_file_id that is not a library image gives 422 mailimage.missing; before the first PUT a send still uses mail.brandCss and after it the compiled CSS
- GET /mail/images as a templates-only holder lists library images with absolute URLs and meta.canUpload false; as a layouts holder canUpload is true; POST /mail/images as a templates-only holder is 403; as a layouts holder with a PNG it is 201 with an absolute URL and the image is attached to the library row; a non-image upload is 422; with neither storage.uploads.public_url nor app.url configured both routes answer with the mail.images.no_public_url message; GET /mail/images/{id} of an image outside the library is 404
- The Mail images singleton form opens for a layouts holder and shows the images fileupload; it is 403 for a templates-only holder
</behavior>
<action>
Per D-09, D-10, D-11, D-18, D-19, D-20 (server half) and D-24:
- `form_schema.go`: add `codeeditor`, `colorpicker`, `mailimage`, `mailpreview` to formFieldTypes and `language`, `mailImages` to formFieldKeys (mode and size exist); `field_mail.go` holds the checks: codeeditor language in html|css|text (required), mailImages boolean only on markdown and codeeditor, mailpreview mode in template|layout|partial|branding (required), and serializes `language` and `mailImages` into the field schema JSON. `crud.go` scalarFormField also returns true for codeeditor, colorpicker and mailimage; mailpreview is valueless (never bound, never accepted in a save body). `settings.go`: colorpicker, mailimage and mailpreview are allowed on settings forms (not in the refusal list); mailpreview is skipped from Writable.
- Switch the layout and partial content fields to codeeditor per UI-SPEC C5/C6 (html giant with mailImages, text huge, css huge with mailImages; partial html huge with mailImages, text large) and the template body to `mailImages: true`. Inject the create defaults of the layouts form (content_html, content_text, content_css) from the editor's catalog default layout into the compiled form schema at Activate.
- Branding: model `MailBrandSetting` (TableName postcard.BrandTable, the 13 colour columns and logo_file_id, Fillable all of them, Rules all nullable) with a BeforeValidate that lower-cases colours, refuses anything but `^#[0-9a-f]{6}$` or empty with a ValidationError on that field (`backend::lang.colorpicker.invalid`) and checks logo_file_id is null or an image in the library (`backend::lang.mailimage.missing`); settings item `mail_brand` (Label `backend::lang.mail.brand.menu_label`, Description `..._description`, category mail, icon palette, order 230, Permissions layouts code, Form `models/mailbrandsetting/fields.yaml`, NewModel); fields.yaml per UI-SPEC C12 (logo mailimage full, colours colorpicker in the row order with the Winter defaults as `default:`; the `_preview` mailpreview arrives in Task 4).
- Image library: model `MailImageLibrary` (TableName postcard.ImageLibraryTable) with an attachMany `images` relation, public; controller `mailImagesAdmin` implementing pact.AdminSingleton (first-or-create, as singleton.go), ConfigDir controllers/mailimages, RequiredPermissions layouts code, form fields.yaml with `images` fileupload mode image, useCaption true, fileTypes jpg,jpeg,png,gif,webp, maxFilesize 5 MB (planner's value; within http.body_limits.upload_bytes); settings link `mail_images` (icon image, order 240, layouts code, Controller MailImagesControllerID). Also add the settings links `mail_layouts` (icon layout-grid, order 210) and `mail_partials` (icon puzzle, order 220), both layouts code.
- `mail_routes.go` image routes (backend group, auth gate as markdownPreview): `GET /mail/images` (query q, page; 40 per page; filename or title match; newest first; allowed for either code via Allows; item `AdminMailImage{ID, FileName, Title, URL, ThumbURL}` with URL from attach.AbsolutePublicURL on the blob key; meta `canUpload` = Allows(layouts), total, page, perPage), `GET /mail/images/{id}` (404 outside the library), `POST /mail/images` (requireAjax, layouts code only, multipart `file`, attach.Store with Public true and image limits, then attached to the library row as the fileupload field does; 201 with the item). When attach.ErrNoAbsoluteURL, answer 409 with the localized `mail.images.no_public_url` message. swag annotations for all three in admin_openapi.go; run `scripts/check-admin-openapi.sh` to regenerate admin/openapi/admin.json and admin/src/api/schema.d.ts; add the routes and handlers to the four route inventories.
- Copy en and pl: `mail.brand.*`, `mail.images.*` (including no_public_url), `codeeditor.*`, `colorpicker.*`, `mailimage.*`, `mail.picker.*`.
- Write the behavior subtests first (`fields`, `branding`, `images`, `images_permissions`) plus field-schema boot cases, then implement.
- Docs in the same commit: cabana README (field types table rows, Branding, image library routes), docs/backend/forms.md "Field types" and "Field options" rows for codeeditor (language, size, mailImages), colorpicker, mailimage and mailpreview (mode), docs/backend/settings.md (the Mail section entries; colorpicker, mailimage and mailpreview on settings forms; Branding precedence over mail.brandCss).
Commit as listed in Execution units.
</action>
<verify>
<automated>go test ./modules/cabana -run 'TestMailScreensSmoke' -count=1 -v 2>&1 | grep -E '^\s*(--- |FAIL|ok|PASS)' | tail -30</automated>
<fails_when>a "FAIL" line, a "--- SKIP" line, "no tests to run", or no "--- PASS: TestMailScreensSmoke/branding" or "--- PASS: TestMailScreensSmoke/images_permissions" line</fails_when>
<automated>scripts/check-admin-openapi.sh --check</automated>
<fails_when>non-zero exit (admin.json or schema.d.ts differs from a fresh generation)</fails_when>
<automated>go vet ./... && go test -short ./... && go test ./modules/cabana -count=1 2>&1 | tail -5</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree' -count=1 2>&1 | tail -3 && go run ./cmd/summer docs:build --check 2>&1 | tail -5</automated>
<fails_when>non-zero exit, or a "FAIL" or "unknown identifier" line in the output</fails_when>
</verify>
<acceptance_criteria>
- `grep -n '"codeeditor": {}' modules/cabana/form_schema.go` and `grep -n '"mailImages": {}' modules/cabana/form_schema.go` each print one line
- `grep -n '/mail/images' admin/openapi/admin.json` prints at least two lines
- `grep -n 'mail/images' modules/cabana/security_coverage_test.go modules/cabana/openapi_conformance_test.go` prints at least two lines
- `grep -n "type: codeeditor" modules/cabana/backendscreens/models/maillayout/fields.yaml` prints three lines
- `grep -n "type: colorpicker" modules/cabana/backendscreens/models/mailbrandsetting/fields.yaml | wc -l` prints 13
- `grep -n "codeeditor" docs/backend/forms.md` prints at least one line
</acceptance_criteria>
<done>Layouts and partials have a real code-editor field on the server, Branding is a settings page that switches mail to its colours once saved, and the image library hands out absolute URLs under the agreed permission split.</done>
</task>
<task type="auto" tdd="true">
<name>Task 4: Preview and test send: the draft renders through postcard and goes only to the signed-in administrator, with a per-admin limit</name>
<files>modules/cabana/mail_routes.go, modules/cabana/http.go, modules/cabana/admin_openapi.go, modules/cabana/backendscreens/models/mailtemplate/fields.yaml, modules/cabana/backendscreens/models/maillayout/fields.yaml, modules/cabana/backendscreens/models/mailpartial/fields.yaml, modules/cabana/backendscreens/models/mailbrandsetting/fields.yaml, modules/cabana/mail_screens_smoke_test.go, modules/cabana/phase10_coverage_test.go, modules/cabana/openapi_conformance_test.go, modules/cabana/security_coverage_test.go, modules/cabana/token_routes_test.go, modules/surf/serve.go, modules/surf/README.md, admin/openapi/admin.json, admin/src/api/schema.d.ts, modules/phrasebook/backend/lang/en/lang.yaml, modules/phrasebook/backend/lang/pl/lang.yaml, modules/cabana/README.md, docs/services/mail.md, docs/services/rate-limiting.md</files>
<read_first>
- modules/cabana/field_markdown.go (~50-80 the route to mirror) and modules/cabana/http.go (route group, interactiveOnly)
- modules/cabana/mail_routes.go (Task 3 output)
- modules/surf/serve.go (~224-245 installCooler) and modules/cooler/cooler.go (~60-75 Attempter)
- modules/surf/limiter.go (how surf answers 429 with Retry-After)
- modules/postcard/editor.go (Preview, SendDraft, DraftError)
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md (C10, C15, Copywriting "Preview and test send")
</read_first>
<behavior>
- POST /mail/preview mode template with id and values {subject, body, layout} returns subject, html and text with sample placeholders; the html contains no script and is what Editor.Preview returned; a draft with an unknown partial returns 422 with details field, rule partial and message
- Mode layout with id 0 renders a new custom layout draft; mode partial renders the sample call; mode branding renders the sample message with the draft colours
- A templates-only holder gets 403 on modes layout, partial and branding and 200 on template; a layouts-only holder gets 403 on template
- POST /mail/test-send as an administrator with email ada@example.test sends one message whose only recipient is ada@example.test (Cc and Bcc empty) and returns that email; a body with a `to`, `email` or `recipient` key is refused before sending; a personal API token request is refused (interactive only)
- The sixth test send within 10 minutes for the same administrator returns 429 with a Retry-After header; another administrator is not limited; the limit uses a published cooler.Attempter when present
- A driver failure returns 5xx with the fixed message and no driver error text
</behavior>
<action>
Per D-16 and D-17:
- `mail_routes.go`: `AdminMailPreviewRequest{Mode string; ID uint64; Values map[string]any}` and `AdminMailPreviewResult{Subject, HTML, Text string}`; handler `mailPreview` (`POST /mail/preview`, requireAjax) with the markdownPreview auth gate, the mode's permission through Allows (template: templates code; layout, partial, branding: layouts code), the record loaded by ID through the controller's FormExtendQuery scope when ID > 0 (its code is the Draft Name; a custom layout create uses ID 0), the values mapped to a postcard Draft (template: subject, body and the `layout` relation value, the selected layout row id, resolved to that row's code; layout: content_html, content_text, content_css, name; partial: content_html, content_text; branding: the colour columns and logo_file_id → absolute URL), then Editor.Preview; a *DraftError answers 422 in the existing error envelope with details `field`, `rule` and `message` and the translated `mail.invalid_<rule>` text as the message; any other error is 500 with a fixed message. `AdminMailTestSendResult{Email string}`; handler `mailTestSend` (`POST /mail/test-send`, interactiveOnly and requireAjax) taking the same request decoded with unknown fields refused, modes template, layout and partial only, the same permission as the mode, the recipient read from the BackendUser row of principal.ID (never from the body), the limit checked before rendering (5 per 10 minutes, key `mail_test:<adminID>`; 429 with Retry-After in seconds as surf does), Editor.SendDraft to that address; a driver error answers 502 with the fixed message `backend::lang.mail.preview.test_failed`.
- Rate-limit store: cabana looks up `cooler.Attempter` from the app per request and falls back to a small in-process fixed-window limiter it owns. `modules/surf/serve.go` installCooler also publishes the Redis-backed store as `cooler.Attempter` (one Publish next to the Store publish), documented in modules/surf/README.md and docs/services/rate-limiting.md.
- YAML: add the mailpreview fields per UI-SPEC: template `_preview` (mode template, label `backend::lang.mail.preview.label`) after body; layouts `_preview_html`, `_preview_text`, `_preview_css` (mode layout) at the end of each tab; partials `_preview_html`, `_preview_text` (mode partial); Branding `_preview` (mode branding, full, comment `backend::lang.mail.brand.preview_comment`).
- swag annotations for both routes in admin_openapi.go (200, 401, 403, 422, 429 for test-send, 502); run `scripts/check-admin-openapi.sh`; add both routes to the four route inventories.
- Copy en and pl: the remaining `mail.preview.*` keys.
- Write the behavior subtests first (`preview`, `preview_permissions`, `test_send`, `test_send_limit`, `test_send_recipient`), then implement.
- Docs in the same commit: cabana README (both routes, the recipient rule, the limit), docs/services/mail.md (preview and test send from the admin; sample values; the limit).
Commit as listed in Execution units.
</action>
<verify>
<automated>go test ./modules/cabana -run 'TestMailScreensSmoke' -count=1 -v 2>&1 | grep -E '^\s*(--- |FAIL|ok|PASS)' | tail -30</automated>
<fails_when>a "FAIL" line, a "--- SKIP" line, "no tests to run", or no "--- PASS: TestMailScreensSmoke/test_send_limit" or "--- PASS: TestMailScreensSmoke/test_send_recipient" line</fails_when>
<automated>scripts/check-admin-openapi.sh --check</automated>
<fails_when>non-zero exit</fails_when>
<automated>go vet ./... && go test -short ./... && go test ./modules/cabana ./modules/surf -count=1 2>&1 | tail -6</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree' -count=1 2>&1 | tail -3 && go run ./cmd/summer docs:build --check 2>&1 | tail -5</automated>
<fails_when>non-zero exit, or a "FAIL" or "unknown identifier" line in the output</fails_when>
</verify>
<acceptance_criteria>
- `grep -n 'g.Post("/mail/test-send", interactiveOnly(requireAjax(' modules/cabana/http.go` prints one line
- `grep -n "type AdminMailPreviewRequest struct" modules/cabana/mail_routes.go` prints one line and that struct has no field named To, Email or Recipient (`go doc ./modules/cabana AdminMailPreviewRequest` lists only Mode, ID and Values)
- `grep -n "cooler.Attempter" modules/surf/serve.go modules/cabana/mail_routes.go` prints at least two lines
- `grep -n '/mail/preview\|/mail/test-send' admin/openapi/admin.json` prints at least two lines
- `grep -c "type: mailpreview" modules/cabana/backendscreens/models/maillayout/fields.yaml` prints 3
</acceptance_criteria>
<done>Every mail form can preview its unsaved draft through postcard and send a test only to the editing administrator, within a per-admin limit.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| admin browser → cabana mail routes | authenticated administrators with varying permission grants post drafts, uploads and test sends |
| personal API tokens → admin API | token principals reach the generic admin routes |
| cabana → postcard Editor → mail driver | drafts become real outbound mail on test send |
| uploads → public disk | administrator files become public URLs linked from mail |
## STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|-----------|----------|-----------|----------|-------------|-----------------|
| T-35-09 | Elevation of privilege | mail controllers and routes | high | mitigate | RequiredPermissions per controller; Allows per preview mode, test send and image upload; codes declared with no Roles (TestMailScreensSmoke/templates, preview_permissions, images_permissions) |
| T-35-10 | Elevation of privilege | picker upload | medium | mitigate | upload needs system.manage_mail_layouts; list allows either code; canUpload computed on the server |
| T-35-11 | Abuse (spam) | POST /mail/test-send | high | mitigate | recipient read from the administrator's own BackendUser row, unknown body keys refused, interactive sessions only, 5 per 10 minutes per admin through cooler.Attempter (TestMailScreensSmoke/test_send_recipient, test_send_limit) |
| T-35-12 | Tampering | override saves | high | mitigate | FormAfterUpdate runs postcard Editor.Validate inside the save transaction; a ValidationError rolls back (research Pitfall 6) |
| T-35-13 | Tampering | brand colours | high | mitigate | BeforeValidate refuses anything but #rrggbb before the row is written; postcard re-validates at compile |
| T-35-14 | Information disclosure | 422 and 5xx answers | medium | mitigate | 422 carries postcard's DraftError text only; driver and database errors map to fixed messages |
| T-35-15 | Tampering | uploads to the public disk | medium | mitigate | attach.Store with Image true, the jpg/jpeg/png/gif/webp allow-list and a 5 MB limit; public flag set explicitly |
| T-35-16 | Repudiation | override changes | low | accept | customized_at and updated_at record when; per-change author history is out of scope (no audit log exists for other admin records either) |
| T-35-17 | Denial of service | data loss through delete | medium | mitigate | file rows never deletable; in-use custom layouts refused with a count; FK ON DELETE RESTRICT backstop |
| T-35-18 | Spoofing | CSRF on new POST routes | high | mitigate | requireAjax on every POST and the existing admin CSRF/cookie guard; token principals refused on test send |
| T-35-SC | Tampering | npm/pip/cargo installs | high | accept | this plan installs no package |
</threat_model>
<verification>
- `go vet ./... && go test -short ./...` and `go test ./modules/cabana ./modules/surf ./modules/postcard/...` green after every commit.
- `scripts/check-admin-openapi.sh --check` passes after Tasks 3 and 4.
- `go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree'` and `go run ./cmd/summer docs:build --check` pass.
</verification>
<success_criteria>
- Five Settings → Mail screens exist behind two permissions with no default grants.
- Override, reset, custom layouts, drift and the File version diff work over postcard's tables.
- Branding, the image library and the four field types exist on the server; preview and test send are permission-checked, recipient-fixed and rate-limited.
- OpenAPI and schema.d.ts regenerated; README and docs updated in the same commits; nothing pushed or tagged.
</success_criteria>
<output>
Create `.planning/phases/35-mail-templates-and-layouts-editor/35-02-SUMMARY.md` when done; list every route, request/response type and field schema key exactly as shipped (35-03 builds against them).
</output>
## Artifacts this phase produces
- Constants: `cabana.PermissionManageMailTemplates`, `cabana.PermissionManageMailLayouts`, `cabana.MailTemplatesControllerID`, `cabana.MailLayoutsControllerID`, `cabana.MailPartialsControllerID`, `cabana.MailImagesControllerID`.
- Types: `cabana.MailTemplate`, `cabana.MailLayout`, `cabana.MailPartial`, `cabana.MailBrandSetting`, `cabana.MailImageLibrary`, `cabana.AdminMailPreviewRequest`, `cabana.AdminMailPreviewResult`, `cabana.AdminMailTestSendResult`, `cabana.AdminMailImage`; field `cabana.RecordMeta.Deletable`.
- Routes: `POST /mail/preview`, `POST /mail/test-send`, `GET /mail/images`, `GET /mail/images/{id}`, `POST /mail/images` (admin API prefix).
- Controllers: `summercms.cabana.mailtemplates`, `summercms.cabana.maillayouts`, `summercms.cabana.mailpartials`, `summercms.cabana.mailimages`; record action `reset`; settings entries `mail_templates`, `mail_layouts`, `mail_partials`, `mail_brand`, `mail_images` in category `backend::lang.settings.category_mail`.
- Field types: `codeeditor`, `colorpicker`, `mailimage`, `mailpreview`; field keys `language`, `mailImages` (and `mode` on mailpreview).
- List messages: `rowStateCustomized`, `rowStateOutdated`; row states `customized`, `outdated` now honoured by cabana.
- Services: surf publishes `cooler.Attempter` (Redis rate-limit store).
- Copy: `backend::lang.mail.*`, `codeeditor.*`, `colorpicker.*`, `mailimage.*`, `settings.category_mail`, `permissions.manage_mail_*`, `messages.list.row_state_customized|outdated` (en, pl).
- Generated: `admin/openapi/admin.json`, `admin/src/api/schema.d.ts`; test `TestMailScreensSmoke`.
- Config keys: none new.

View File

@@ -0,0 +1,440 @@
---
phase: 35-mail-templates-and-layouts-editor
plan: 03
type: execute
wave: 3
depends_on: ["35-02"]
files_modified:
- .planning/notes/2026-11-06-admin-code-editor-codemirror.md
- .planning/research/STACK.md
- admin/package.json
- admin/package-lock.json
- admin/src/components/form/registry.ts
- admin/src/components/form/fields/MailPreviewField.vue
- admin/src/components/form/fields/CodeEditorField.vue
- admin/src/components/form/codeeditor/editor.ts
- admin/src/components/form/fields/ColorPickerField.vue
- admin/src/components/form/fields/MailImageField.vue
- admin/src/components/mail/MailImagePicker.vue
- admin/src/components/mail/insertImage.ts
- admin/src/components/form/fields/MarkdownField.vue
- admin/src/components/form/fields/PartialField.vue
- admin/src/views/FormView.vue
- admin/src/views/SettingsFormView.vue
- admin/src/views/ListView.vue
- admin/src/components/list/RowStateBadges.vue
- admin/src/components/list/DataTable.vue
- admin/src/styles/main.css
- admin/src/api/types.ts
- admin/tests/form/mail-smoke.test.ts
- admin/tests/browser/codeeditor-smoke.test.ts
- modules/boardwalk/dist
- docs/backend/admin-spa.md
- docs/backend/forms.md
autonomous: false
requirements: [D-03, D-07, D-08, D-09, D-10, D-11, D-16, D-17, D-18, D-19, D-20, D-21]
estimate:
tokens: 190000
raw_tokens: 190000
tasks: 6
confidence: low
must_haves:
truths:
- "On a template, layout, partial or Branding form, the `mailpreview` field renders the unsaved draft from FORM_VALUES through POST /mail/preview 500 ms after the last change, in an iframe with an empty sandbox attribute and srcdoc bound only to the server's html, with a Plain text tab, the subject row, a sequence guard and the UI-SPEC error states (D-16)"
- "The Send test to me button posts the draft to POST /mail/test-send with no recipient, shows the signed-in administrator's email in its hint and the success toast, and on 429 shows the limit line and stays disabled for Retry-After seconds (D-17)"
- "Layouts and partials are edited in `codeeditor`, a CodeMirror 6 field loaded lazily in its own chunk with html, css or plain-text highlighting, Go template action highlighting, line numbers, bracket matching and history; Tab moves focus out, Mod+S and Mod+Enter still save the form, Esc does not cancel; a load failure falls back to a monospace textarea that still saves (D-09)"
- "The CodeMirror packages are added only after a decision note and a STACK.md row are committed and Jakub has confirmed the versions at a blocking checkpoint, with exact pins and the committed lockfile (CLAUDE.md dependency rule, research Package Legitimacy Audit)"
- "The Insert image button on the template body (markdown with mailImages) and on codeeditor fields with mailImages opens MailImagePicker, which lists the library from the server, uploads only when the server says canUpload, and inserts `![alt](url)`, `<img src=\"url\" alt=\"alt\">`, `url(\"url\")` or the bare URL with the server's absolute URL, escaped per UI-SPEC C4; MarkdownField hides its own preview toggle when mailImages is set (D-18, D-19, D-20, D-03)"
- "The Branding page renders 13 `colorpicker` fields (native swatch plus #rrggbb text) and the `mailimage` logo field (thumbnail, choose, change, remove, missing-image line), and SettingsFormView provides FORM_VALUES and FORM_LOCALE so its preview reads the draft (D-10, D-11)"
- "FormView shows the record's meta.actions (Reset to file version) in the update footer with the danger confirm, hides Delete when meta.deletable is false, and reloads partial fields after every save and record action, so the drift callout and File version diff are never stale; controllers without these keys behave as before (D-06, D-07, D-08)"
- "Lists show the `customized` (neutral outline) and `outdated` (sel tint) badges after the existing three, in that order, with labels from messages.rowStateCustomized and rowStateOutdated; main.css has the `.summer-diff` kit (D-07)"
- "modules/boardwalk/dist is rebuilt from the committed lockfile and passes scripts/check-admin-dist.sh, and the CodeMirror code is in a separate chunk, not in the entry bundle (D-21)"
prohibitions:
- statement: "The SPA never builds or modifies mail HTML: the preview iframe's srcdoc is only the server's rendered html, the iframe has no src and its sandbox attribute is empty."
status: unresolved
verification: judgment
- statement: "The SPA never builds image URLs; it inserts the absolute URL the server returned."
status: unresolved
verification: judgment
- statement: "The `codemirror` meta package (basicSetup) and indentWithTab are not used; no colour token, font, radius or shadow is added to the design system."
status: unresolved
verification: judgment
- statement: "No npm package is installed before the decision note, the STACK.md row and the human checkpoint."
status: unresolved
verification: judgment
- statement: "There is no free-text recipient input anywhere in the mail screens."
status: unresolved
verification: judgment
artifacts:
- path: "admin/src/components/form/fields/MailPreviewField.vue"
provides: "mailpreview control (tracer)"
contains: "sandbox=\"\""
- path: "admin/src/components/form/codeeditor/editor.ts"
provides: "CodeMirror setup, lazy import target"
contains: "Mod-Enter"
- path: "admin/src/components/form/fields/CodeEditorField.vue"
provides: "codeeditor control"
contains: "codeeditor.load_failed"
- path: "admin/src/components/mail/insertImage.ts"
provides: "pure snippet builder"
contains: "export function insertImage"
- path: "admin/src/components/mail/MailImagePicker.vue"
provides: "library picker dialog"
contains: "radiogroup"
- path: ".planning/notes/2026-11-06-admin-code-editor-codemirror.md"
provides: "dependency decision note"
contains: "@codemirror/view"
key_links:
- from: "admin/src/components/form/fields/MailPreviewField.vue"
to: "modules/cabana/mail_routes.go"
via: "api.POST('/mail/preview') and api.POST('/mail/test-send') typed by schema.d.ts"
pattern: "/mail/preview"
- from: "admin/src/components/form/fields/CodeEditorField.vue"
to: "admin/src/components/form/codeeditor/editor.ts"
via: "dynamic import, so CodeMirror stays out of the entry chunk"
pattern: "import\\(.*codeeditor/editor"
- from: "admin/src/components/mail/MailImagePicker.vue"
to: "modules/cabana/mail_routes.go"
via: "GET/POST /mail/images, canUpload from meta"
pattern: "/mail/images"
- from: "admin/src/components/form/registry.ts"
to: "admin/src/components/form/fields/MailPreviewField.vue"
via: "mailpreview registered, valueless and group-labelled"
pattern: "'mailpreview'"
---
## Phase Goal
ROADMAP Goal (not in user-story format; no story is invented): administrators edit every mail the application sends from the admin panel. This plan gives the Vue admin SPA what the generic views cannot render on their own: the live preview and test send, the code editor, the colour and image fields, the image picker, and the small FormView, SettingsFormView and list additions of UI-SPEC C8 and C9, then rebuilds the embedded admin.
Spec-less probe fallback skipped: no SPEC.md and no requirement IDs; must_haves derive from the ROADMAP goal, CONTEXT D-01..D-25 and the approved 35-UI-SPEC.md. Route, type and key names come from 35-02-SUMMARY.md where they differ from this text.
<objective>
Implement UI-SPEC C1-C4, C8-C11 in admin/:
- `mailpreview` with sandboxed srcdoc preview and test send (D-16, D-17);
- `codeeditor` on CodeMirror 6, lazily loaded, after the decision note, the STACK.md row and a blocking version check (D-09; research Package Legitimacy Audit flags @codemirror/view 6.43.15 as too-new);
- `colorpicker`, `mailimage`, MailImagePicker and insertImage, the Insert image button on markdown and codeeditor (D-10, D-18..D-20);
- FormView footer record actions, meta.deletable, partial reload; SettingsFormView FORM_VALUES/FORM_LOCALE; row-state badges; the `.summer-diff` CSS kit (D-06..D-08);
- the rebuilt modules/boardwalk/dist (D-21).
Purpose: with 35-02's server screens, this completes the editor administrators use; nothing here decides permissions, recipients or URLs, which stay on the server.
Output: commits in this repository only; planning docs (decision note, STACK.md) in their own commit, the dist rebuild in its own commit. Nothing is pushed or tagged.
</objective>
<execution_context>
@~/.claude/msd-core/workflows/execute-plan.md
@~/.claude/msd-core/templates/summary.md
</execution_context>
<context>
@.planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-PATTERNS.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-02-SUMMARY.md
@.planning/notes/2026-10-12-admin-display-sanitizer-dompurify.md
@.planning/notes/2026-10-07-embedded-admin-release-gate.md
@CLAUDE.md
<interfaces>
admin/src/components/form/registry.ts: `renderers` Map (type → component), `valueless` Set (RELATION_MANAGER, widget, partial, fileupload), `groupLabelledTypes` Set, `selfLabelled`, `recordBound`; field components import only from `../control` (never from registry.ts) to avoid an import cycle; header comment carries a per-phase history paragraph.
admin/src/components/form/fields/MarkdownField.vue: PREVIEW_DEBOUNCE_MS, the `sequence` guard, `api.POST('/markdown/preview', ...)`, `refusal(error)` details extraction, controlAttributes/controlClass from ../control.
admin/src/components/form/fields/RichEditorField.vue: lazy import of the TipTap editor with a `bg-skel` loading box (`data-*-loading`, aria-busy) and a failure fallback; toolbar class strings.
admin/src/components/form/formContext.ts: FORM_VALUES, FORM_PATCH, FORM_SESSION, FORM_LOCALE injection keys.
admin/src/views/FormView.vue: provides at ~354-417; `singleton`, `backToPreview` (schema.preview) at ~205-214; footer at ~680-727 with Delete (`mode === 'update' && !singleton`) and onDelete (~546).
admin/src/views/SettingsFormView.vue: provides SETTINGS_SECRETS only (~54).
admin/src/components/form/RecordActions.vue (used by PreviewView; `busy` emit, danger confirm); admin/src/components/relation/RelationPickerModal.vue (dialog shell, search input, chosen tile classes); admin/src/components/form/fields/FileuploadField.vue (tile grid, captions).
admin/src/components/list/RowStateBadges.vue:19 `const ORDER = ['deleted', 'negative', 'disabled']`; DataTable.vue:160 `const ROW_STATES = [...]`; ListView.vue:~108 row-state labels from messages.
From 35-02: POST /mail/preview {mode, id, values} → {subject, html, text}, 422 details {field, rule, message}; POST /mail/test-send same body → {email}, 429 + Retry-After, 502; GET /mail/images?q=&page= → {data: [{id, fileName, title, url, thumbUrl}], meta: {canUpload, total, page, perPage}}, 409 with message when no absolute URL is configured; GET /mail/images/{id}; POST /mail/images multipart `file`; form field schema keys `language`, `mailImages`, `mode`; RecordMeta.deletable; list messages rowStateCustomized / rowStateOutdated.
Admin CSP (modules/boardwalk/boardwalk.go ~61-66) has frame-src 'none'; its comment states a sandboxed srcdoc preview still renders, so no CSP change.
</interfaces>
</context>
## Execution units and resume points
- Before each task run `git status --short admin modules/boardwalk/dist` and stop if another session holds uncommitted edits there.
- Each task ends with the commit(s) listed below, staged by explicit path. Resume check: `git log --format=%s -F --grep='<subject>'`.
- Per task: `npm --prefix admin run test` and `npm --prefix admin run typecheck`; the browser suite needs the Playwright Chromium build already in ~/.cache/ms-playwright.
| Task | Commit subject | Resume point |
|------|----------------|--------------|
| 1 | `feat(admin): mail preview field with sandboxed HTML and test send` | mail-smoke preview cases pass |
| 2 | `docs(35): CodeMirror decision note and STACK.md row for the admin code editor` | note and row committed |
| 3 | none (answer recorded in the SUMMARY) | answer recorded |
| 4 | `feat(admin): codeeditor field on CodeMirror 6, loaded lazily` | codeeditor-smoke passes |
| 5 | `feat(admin): colour picker, mail image field and the image picker with insert` | mail-smoke picker cases pass |
| 6 | `feat(admin): footer record actions, deletable flag, mail row states and diff styles`, then `build(admin): rebuild the embedded admin for the mail editor` | check-admin-dist.sh passes |
<tasks>
<task type="tracer">
<name>Task 1: End-to-end "an administrator edits a template body and sees the rendered mail, then sends a test to their own address" — the mailpreview field over the 35-02 routes</name>
<precondition>35-02 is complete: `grep -n '/mail/preview' admin/src/api/schema.d.ts` prints at least one line</precondition>
<files>admin/src/components/form/fields/MailPreviewField.vue, admin/src/components/form/registry.ts, admin/src/views/SettingsFormView.vue, admin/src/api/types.ts, admin/tests/form/mail-smoke.test.ts</files>
<read_first>
- admin/src/components/form/fields/MarkdownField.vue (whole file: debounce, sequence guard, refusal extraction)
- admin/src/components/form/registry.ts (whole file) and admin/src/components/form/formContext.ts
- admin/src/views/SettingsFormView.vue (whole file) and admin/src/views/FormView.vue (~340-420 provides)
- admin/src/components/form/FormTabs.vue (tab pill class strings) and admin/src/state/useToasts.ts
- admin/tests/form/MarkdownField.test.ts and admin/tests/helpers.ts (how fields are mounted with provides and api mocks)
- .planning/phases/35-mail-templates-and-layouts-editor/35-02-SUMMARY.md (exact request and response shapes)
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md (C10, C15, Copywriting "Preview and test send")
</read_first>
<action>
Per D-16 and D-17 and UI-SPEC C10:
- `MailPreviewField.vue`: reads the whole draft from FORM_VALUES and the record id (0 on create) and posts `{mode: field.mode, id, values}` to `/mail/preview` on mount and 500 ms after the last change, with the MarkdownField sequence guard; while its tab is inactive it sends nothing and renders when the tab becomes active (use the same visibility signal other lazy fields use, or an IntersectionObserver when none exists). Markup and classes exactly per C10: toolbar with the HTML / Plain text switch (FormTabs class strings, role tablist) and the role status line (`mail.preview.rendering`); subject row hidden in branding mode; `<iframe sandbox="" referrerpolicy="no-referrer">` with srcdoc bound only to the html of the latest 2xx answer and the `mail.preview.frame_title` title, white canvas, 640px; the plain-text `pre`; notes (`mail.preview.sample_note` in template mode, `sample_content_note` otherwise, plus the field comment); the error line (`failed` keeping the last render, `failed_first`, `unavailable` on network or 5xx) with :message from the 422 details; the empty-draft state when the mode's content field is blank (no request).
- Test send (not in branding mode): outline small Button with the Send icon and `mail.preview.test_send` / `test_sending`, the hint `mail.preview.test_hint` with :email from the signed-in administrator's session; POST the same payload to `/mail/test-send` (never a recipient); 2xx → toast `mail.preview.test_sent` with the returned email; 429 → inline line `test_limited` with :seconds from Retry-After and the button disabled until then; 422 → inline `test_invalid` with :message; 5xx → danger toast `test_failed`; 403 → danger toast with the server message. No confirm dialog.
- `registry.ts`: register `mailpreview`, add it to `valueless` and `groupLabelledTypes`; add a Phase 35 paragraph to the header history comment. `types.ts`: export any helper types the field needs from schema.d.ts (no hand-written API shapes).
- `SettingsFormView.vue`: `provide(FORM_VALUES, computed(() => values.value))` and `provide(FORM_LOCALE, ...)` as FormView does; no FORM_PATCH or session (UI-SPEC C8 item 4).
- `admin/tests/form/mail-smoke.test.ts` (smoke, happy-dom): mounts MailPreviewField with provided values and a mocked api; asserts the iframe's sandbox attribute is present and empty, there is no src attribute, srcdoc equals the mocked html, a stale answer is dropped by the sequence guard, a 422 after a good render keeps the previous srcdoc and shows the failed line, and a 429 test send disables the button.
Commit as listed in Execution units.
</action>
<verify>
<automated>npm --prefix admin run test -- mail-smoke 2>&1 | tail -8</automated>
<fails_when>non-zero exit, a "FAIL" line, or "No test files found"</fails_when>
<automated>npm --prefix admin run typecheck 2>&1 | tail -3</automated>
<fails_when>non-zero exit or an "error TS" line</fails_when>
<automated>npm --prefix admin run test 2>&1 | tail -6</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
</verify>
<acceptance_criteria>
- `grep -n 'sandbox=""' admin/src/components/form/fields/MailPreviewField.vue` prints one line
- `grep -n "'/mail/test-send'" admin/src/components/form/fields/MailPreviewField.vue` prints one line
- `grep -n "'mailpreview'" admin/src/components/form/registry.ts` prints at least two lines (renderer and valueless or group set)
- `grep -n "provide(FORM_VALUES" admin/src/views/SettingsFormView.vue` prints one line
</acceptance_criteria>
<done>Every mail form shows its unsaved draft exactly as postcard renders it and can send a test to the editing administrator.</done>
</task>
<task type="auto">
<name>Task 2: Write the CodeMirror decision note and the STACK.md row before anything is installed</name>
<files>.planning/notes/2026-11-06-admin-code-editor-codemirror.md, .planning/research/STACK.md</files>
<read_first>
- .planning/notes/2026-10-12-admin-display-sanitizer-dompurify.md (structure to mirror: Decision table, Why, Scope, Legitimacy with integrity values)
- .planning/research/STACK.md (~74-84 "Admin SPA (npm) additions")
- .planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md ("Admin SPA (new npm runtime dependencies)", "Package Legitimacy Audit", "CodeMirror setup", Pitfall 5)
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md (Design System "Why CodeMirror 6", Registry Safety)
</read_first>
<action>
Per D-09 and the CLAUDE.md dependency rule ("A new dependency needs a research doc or a phase/decision note naming it"):
- Create `.planning/notes/2026-11-06-admin-code-editor-codemirror.md`: Decision table with the seven direct runtime packages and exact pins (@codemirror/state 6.7.6, @codemirror/view 6.43.15, @codemirror/language 6.13.1, @codemirror/commands 6.11.1, @codemirror/lang-html 6.4.12, @codemirror/lang-css 6.3.1, @lezer/highlight 1.2.5), what each is used for, that they load lazily from `admin/src/components/form/codeeditor/editor.ts` in their own chunk, that the meta package with basicSetup is not used, and the keymap rule (defaultKeymap without Mod-Enter, no indentWithTab, research Pitfall 5); Why (MIT, modular, no workers so the CSP is unchanged, accessible contenteditable; Monaco, Ace, textarea overlay and TipTap rejected per UI-SPEC); the transitive packages (the @lezer grammars, @codemirror/autocomplete and @codemirror/lang-javascript via lang-html, style-mod, w3c-keyname, crelt, @marijn/find-cluster-break); Legitimacy: the research audit table (maintainer marijn, no postinstall scripts, weekly downloads, source repositories), the [SUS] too-new flag on @codemirror/view 6.43.15 (published 2026-10-19, past the 7-day cooldown) and the `dist.integrity` value of each pinned version read with `npm view <pkg>@<version> dist.integrity` (read-only registry queries; no install).
- `.planning/research/STACK.md` "Admin SPA (npm) additions": one row per direct package (or one grouped CodeMirror row listing all seven pins) with purpose "codeeditor field (Phase 35)" and a link to the note.
Commit both as one planning-docs commit, separate from code (CLAUDE.md), as listed in Execution units.
</action>
<verify>
<automated>grep -c "@codemirror/" .planning/notes/2026-11-06-admin-code-editor-codemirror.md && grep -n "codemirror" .planning/research/STACK.md | head -5</automated>
<fails_when>non-zero exit, a count below 6, or no STACK.md line</fails_when>
<automated>git diff --quiet HEAD -- admin/package.json admin/package-lock.json && echo unchanged</automated>
<fails_when>no "unchanged" line (package files must not change in this task)</fails_when>
</verify>
<acceptance_criteria>
- `grep -n "sha512-" .planning/notes/2026-11-06-admin-code-editor-codemirror.md | wc -l` prints at least 7
- `grep -n "6.43.15" .planning/notes/2026-11-06-admin-code-editor-codemirror.md` prints at least one line mentioning the too-new flag
- `git log --format=%H%x09%s -F --grep='docs(35): CodeMirror decision note and STACK.md row for the admin code editor'` prints one line, and `git show --stat --format= <that sha>` lists only the two planning files
</acceptance_criteria>
<done>The dependency is documented and justified before it exists in the repository.</done>
</task>
<task type="checkpoint:human-verify" gate="blocking-human">
<name>Task 3: Jakub confirms the CodeMirror package versions before npm install (legitimacy gate; never auto-approved)</name>
<what-built>The decision note `.planning/notes/2026-11-06-admin-code-editor-codemirror.md` lists the seven pins and their dist.integrity values. Research flagged `@codemirror/view` 6.43.15 [SUS] as too-new (18 days old on 2026-11-06, past the project's 7-day cooldown, same maintainer as every other package).</what-built>
<how-to-verify>
1. Open https://www.npmjs.com/package/@codemirror/view and check that 6.43.15 is published by marijn, has no install scripts and matches the integrity in the note (or decide to pin 6.43.14 or older).
2. Spot-check https://www.npmjs.com/package/@codemirror/state, /@codemirror/language, /@codemirror/commands, /@codemirror/lang-html, /@codemirror/lang-css and /@lezer/highlight against the note's versions.
3. Confirm that installing exactly these pins into admin/ (runtime dependencies, exact versions, committed lockfile) is approved.
</how-to-verify>
<resume-signal>Reply "approved", or "pin @codemirror/view <version>" (the executor updates the note and STACK.md in a follow-up planning commit before installing), or "stop"</resume-signal>
</task>
<task type="auto" tdd="true">
<name>Task 4: The codeeditor field: CodeMirror 6 loaded lazily, Go template actions highlighted, keyboard rules that keep the form's hotkeys and Tab free</name>
<reversibility rating="costly">D-09: the `codeeditor` type, its `language` and `size` keys and its value shape are a published field type plugins may start using.</reversibility>
<precondition>Task 3 answered "approved" or named a pin, recorded in the plan SUMMARY draft</precondition>
<files>admin/package.json, admin/package-lock.json, admin/src/components/form/codeeditor/editor.ts, admin/src/components/form/fields/CodeEditorField.vue, admin/src/components/form/registry.ts, admin/tests/browser/codeeditor-smoke.test.ts, admin/tests/form/mail-smoke.test.ts</files>
<read_first>
- admin/src/components/form/fields/RichEditorField.vue (lazy import, loading skeleton, failure fallback, toolbar classes)
- admin/src/components/form/control.ts (controlAttributes, controlClass, FieldControlProps)
- admin/vitest.browser.config.ts and admin/tests/browser/richeditor-display.test.ts (browser-mode test shape)
- admin/src/views/FormView.vue (the form hotkey handler and how it ignores plain keys inside contenteditable)
- .planning/notes/2026-11-06-admin-code-editor-codemirror.md (Task 2, as confirmed at Task 3)
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md (C1, Color "Code editor syntax colours", Spacing "Code editor by size")
- .planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md ("CodeMirror setup", Pitfall 5)
</read_first>
<behavior>
- The control shows the `bg-skel` loading box (aria-busy true) at the size's height until the chunk loads; size tiny/small/large/huge/giant gives 128/192/320/480/640 px, default large
- A failed import shows the `codeeditor.load_failed` note and a monospace textarea bound to the value; editing it emits update:modelValue
- Typing in the editor emits update:modelValue with the full document; the content element has the control id, aria-labelledby, aria-describedby, aria-invalid and aria-multiline true
- Browser: Tab from inside the editor moves focus to the next control; Mod+S and Mod+Enter inside the editor trigger the form save; Esc inside the editor does not trigger Cancel
- language html and css use their grammars, text uses none and wraps lines; `{{ ... }}` spans carry the cm-summer-action class
- Read-only rendering is a `pre` with the text (or list.empty_value) and loads no CodeMirror
</behavior>
<action>
Per D-09 and UI-SPEC C1:
- Install exactly the pins confirmed at Task 3 with `npm --prefix admin install --save-exact` (runtime dependencies), then `npm --prefix admin ci` to prove the lockfile.
- `codeeditor/editor.ts` (the only module that imports CodeMirror): a `createCodeEditor(parent, options)` factory taking the initial doc, language (html | css | text), height, ids/aria attributes and an onChange callback, returning an object with `view`, `insert(text)` (replaces the main selection, or appends at the end when the editor never had focus, then focuses with the caret after the text) and `destroy()`. Extensions: lineNumbers, bracketMatching, history, the html() or css() grammar or none, a MatchDecorator plus ViewPlugin marking Go template actions with class `cm-summer-action`, `EditorView.lineWrapping` only for text, contentAttributes for the aria wiring, a HighlightStyle and EditorView.theme using only the existing `var(--c-*)` variables per the UI-SPEC syntax table, and `keymap.of([...historyKeymap, ...defaultKeymap without the Mod-Enter binding])`; indentWithTab is not added.
- `CodeEditorField.vue`: wrapper and toolbar markup per C1 (the toolbar renders only when mailImages is set; its Insert image button is wired in Task 5 through the editor's insert), dynamic `import('../codeeditor/editor')` so CodeMirror lands in its own chunk, the loading box and failure fallback copied from RichEditorField, read-only `pre` rendering, destroy on unmount, external value changes (record reload after reset) replace the document without emitting.
- `registry.ts`: register `codeeditor`.
- Write first: `admin/tests/browser/codeeditor-smoke.test.ts` (browser mode: Tab leaves, Mod+S and Mod+Enter reach a form submit handler, Esc does not reach Cancel, typing emits the value) and happy-dom cases in mail-smoke.test.ts for the loading box, the failure fallback (mock the dynamic import to reject) and read-only rendering; then implement.
Commit as listed in Execution units.
</action>
<verify>
<automated>npm --prefix admin run test:browser -- codeeditor-smoke 2>&1 | tail -8</automated>
<fails_when>non-zero exit, a "FAIL" line, or "No test files found"</fails_when>
<automated>npm --prefix admin run test -- mail-smoke 2>&1 | tail -6 && npm --prefix admin run typecheck 2>&1 | tail -3</automated>
<fails_when>non-zero exit, a "FAIL" line or an "error TS" line</fails_when>
<automated>npm --prefix admin ci 2>&1 | tail -3</automated>
<fails_when>non-zero exit or an "npm error" line</fails_when>
</verify>
<acceptance_criteria>
- `node -e "const p=require('./admin/package.json');for(const n of ['@codemirror/state','@codemirror/view','@codemirror/language','@codemirror/commands','@codemirror/lang-html','@codemirror/lang-css','@lezer/highlight'])if(!/^\d+\.\d+\.\d+$/.test(p.dependencies[n]||''))process.exit(1)"` exits 0
- `grep -n "import('../codeeditor/editor')" admin/src/components/form/fields/CodeEditorField.vue` prints one line
- `grep -rln "@codemirror/" admin/src | grep -v codeeditor/editor.ts` prints nothing (type-only imports included: editor.ts is the only module that names a CodeMirror package)
<!-- planner-discipline-allow: @codemirror/ -->
- `grep -n "'codeeditor'" admin/src/components/form/registry.ts` prints at least one line
</acceptance_criteria>
<done>Layout and partial HTML, text and CSS are edited in a real code editor that respects the form's keyboard model.</done>
</task>
<task type="auto" tdd="true">
<name>Task 5: Colour picker, mail image field and the image picker that inserts library images into bodies and code editors</name>
<files>admin/src/components/form/fields/ColorPickerField.vue, admin/src/components/form/fields/MailImageField.vue, admin/src/components/mail/MailImagePicker.vue, admin/src/components/mail/insertImage.ts, admin/src/components/form/fields/MarkdownField.vue, admin/src/components/form/fields/CodeEditorField.vue, admin/src/components/form/registry.ts, admin/tests/form/mail-smoke.test.ts</files>
<read_first>
- admin/src/components/relation/RelationPickerModal.vue (dialog shell, search, chosen tile classes) and admin/src/components/form/fields/FileuploadField.vue (thumbnails, captions, upload error display)
- admin/src/components/form/fields/TextField.vue and MarkdownField.vue (textarea selection handling, preview toggle)
- admin/src/components/form/fields/CodeEditorField.vue (Task 4) and admin/src/components/ui/Button.vue
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md (C2, C3, C4, C11, C12, Copywriting "Image picker and image fields", "Code editor and colour picker")
- .planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md (Pitfall 4: percent-encode URLs with spaces for Markdown)
</read_first>
<behavior>
- insertImage('markdown', 'https://cdn.example.com/a b.png', 'Logo [x]') returns `![Logo \[x\]](https://cdn.example.com/a%20b.png)`; ('html', url, 'a "b" <c>') entity-escapes &, ", < and > in alt and url; ('css', url, alt) returns `url("<url>")`; ('text', url, alt) returns the url
- The picker lists images from GET /mail/images with a 300 ms debounced server search, loads the next page of 40 on scroll, is a radiogroup with arrow-key movement, keeps the primary disabled until a tile is chosen, prefills alt from the caption title in insert mode, and shows the upload row only when meta.canUpload is true; an upload adds the new image first and selected; a 422 upload shows the server message under the row; empty, no-result and load-failed states use the UI-SPEC copy, and a 409 shows the server's message
- The markdown field with mailImages shows Insert image and no Preview toggle; without mailImages it is unchanged; choosing an image inserts the markdown snippet at the textarea selection and the caret lands after it
- A codeeditor with mailImages inserts the html or css snippet through the editor's insert
- ColorPicker: typing `#D66829` updates the swatch and emits the value; a partial value leaves the swatch at the last valid colour; an empty value shows the not-set placeholder and a white swatch
- MailImage: shows the thumbnail and filename for a stored id (GET /mail/images/{id}), Choose opens the picker in choose mode, Remove emits null, a 404 for the stored id shows the mailimage.missing line and the empty icon
</behavior>
<action>
Per D-18, D-19, D-20, D-10 and D-03 (UI-SPEC C2, C3, C4, C11):
- `insertImage.ts`: pure `insertImage(language: 'markdown' | 'html' | 'css' | 'text', url: string, alt: string): string` per the C4 table, percent-encoding spaces and parentheses in Markdown URLs (research Pitfall 4).
- `MailImagePicker.vue`: Reka Dialog with the RelationPickerModal shell, insert and choose modes, search, upload row from `meta.canUpload` only (POST /mail/images, hidden file input with the four image types), the tile grid radiogroup, alt input in insert mode, footer Cancel + primary; never builds a URL, always uses the server's `url`.
- `MarkdownField.vue`: with `mailImages` the Insert image text button replaces the Preview toggle; insertion through insertImage('markdown', ...) at the textarea selection, focus back with the caret after the text, update:modelValue emitted.
- `CodeEditorField.vue`: the toolbar button opens the picker and inserts insertImage(language, ...) through the editor's insert (the fallback textarea uses the selection path).
- `ColorPickerField.vue` per C2 (native swatch with aria-label `colorpicker.pick`, mono hex input with maxlength 7 and the not-set placeholder; no client-side validation beyond the native control). `MailImageField.vue` per C11 (value is the library image id or null, group-labelled).
- `registry.ts`: register `colorpicker` and `mailimage`; add `mailimage` to groupLabelledTypes.
- Write the behavior cases first in mail-smoke.test.ts (insertImage table, picker canUpload both ways, markdown insert and hidden preview toggle, colour sync, mailimage missing), then implement.
Commit as listed in Execution units.
</action>
<verify>
<automated>npm --prefix admin run test -- mail-smoke 2>&1 | tail -8</automated>
<fails_when>non-zero exit, a "FAIL" line, or "No test files found"</fails_when>
<automated>npm --prefix admin run typecheck 2>&1 | tail -3 && npm --prefix admin run test 2>&1 | tail -6</automated>
<fails_when>non-zero exit, an "error TS" line or a "FAIL" line</fails_when>
</verify>
<acceptance_criteria>
- `grep -n "export function insertImage" admin/src/components/mail/insertImage.ts` prints one line
- `grep -n 'role="radiogroup"' admin/src/components/mail/MailImagePicker.vue` prints one line
- `grep -n "canUpload" admin/src/components/mail/MailImagePicker.vue` prints at least one line
- `grep -n "'colorpicker'\|'mailimage'" admin/src/components/form/registry.ts` prints at least two lines
- `grep -n "mailImages" admin/src/components/form/fields/MarkdownField.vue admin/src/components/form/fields/CodeEditorField.vue` prints at least two lines
</acceptance_criteria>
<done>Administrators pick brand colours and the logo, and place library images into template bodies, layouts and partials without typing a URL.</done>
</task>
<task type="auto" tdd="true">
<name>Task 6: FormView footer record actions, the deletable flag, partial reloads, mail row-state badges and the diff kit, then rebuild the embedded admin</name>
<files>admin/src/views/FormView.vue, admin/src/components/form/fields/PartialField.vue, admin/src/views/ListView.vue, admin/src/components/list/RowStateBadges.vue, admin/src/components/list/DataTable.vue, admin/src/styles/main.css, admin/tests/form/mail-smoke.test.ts, modules/boardwalk/dist, docs/backend/admin-spa.md, docs/backend/forms.md</files>
<read_first>
- admin/src/views/FormView.vue (whole file: footer, onDelete, save path, reload, snapshot reset)
- admin/src/views/PreviewView.vue (RecordActions use and the hintKey reload) and admin/src/components/form/RecordActions.vue
- admin/src/components/form/fields/PartialField.vue and the PartialHost it renders
- admin/src/components/list/RowStateBadges.vue, DataTable.vue (~155-195), ListView.vue (~100-115)
- admin/src/styles/main.css (`@layer components`, `.summer-callout` kit)
- scripts/check-admin-dist.sh and .planning/notes/2026-10-07-embedded-admin-release-gate.md
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md (C7 CSS kit, C8 items 1-3, C9)
</read_first>
<behavior>
- An update form whose record has meta.actions [reset] and no preview screen shows an outline Reset to file version button after Delete in the footer; its confirm is the danger variant; while it runs every footer button is disabled; after a 2xx the mail.reset_done toast shows, values are replaced, the form is clean and the partial field's reloadKey increments; 409 shows the stale toast and reloads; 403 shows a danger toast
- meta.deletable false hides Delete; true or absent shows it (absent = unchanged behaviour for every existing controller)
- A successful save also increments the partial reloadKey
- RowStateBadges renders deleted, negative, disabled, customized, outdated in that order with the customized outline and outdated sel styles; DataTable keeps text-text (no strike-through) for rows with only the new states; ListView passes the rowStateCustomized and rowStateOutdated labels
</behavior>
<action>
Per D-06 (as amended by D-23), D-07 and D-08 (UI-SPEC C7, C8, C9):
- `FormView.vue`: footer record actions from the show response's meta.actions through the existing RecordActions component in the left group after Delete, only on update and only when the controller has no preview screen (the generic rule in C8 item 1); busy disables every footer button; on success toast, reload the record in place and reset the dirty snapshot; Delete hidden when meta.deletable is false; a `reloadKey` provided to partial fields and bumped after every successful save and record action.
- `PartialField.vue`: re-fetch its partial when the provided reloadKey changes (the PreviewView hintKey mechanism).
- Lists: RowStateBadges ORDER and styles, DataTable ROW_STATES (new states do not change the title style), ListView labels.
- `main.css`: the `.summer-diff`, `.summer-diff__add`, `__del`, `__ctx` rules from UI-SPEC C7 in `@layer components`, existing variables only.
- Write the behavior cases first in mail-smoke.test.ts, then implement. Run `npm --prefix admin run test`, `npm --prefix admin run test:browser`, `npm --prefix admin run typecheck` and `scripts/check-admin-openapi.sh --check`, and commit the source change.
- Docs in the same source commit: docs/backend/admin-spa.md (the code editor's keyboard behaviour next to Keyboard shortcuts; the preview and test-send behaviour) and docs/backend/forms.md (how codeeditor, colorpicker, mailimage and mailpreview look and behave in the admin, footer record actions on forms without a preview screen, meta.deletable).
- Then `npm --prefix admin run build`, stage `modules/boardwalk/dist` by path (it is a directory of generated files; `git add modules/boardwalk/dist` is the explicit path), run `scripts/check-admin-dist.sh`, and commit the rebuild separately as listed in Execution units.
</action>
<verify>
<automated>npm --prefix admin run test 2>&1 | tail -6 && npm --prefix admin run test:browser 2>&1 | tail -6 && npm --prefix admin run typecheck 2>&1 | tail -3</automated>
<fails_when>non-zero exit, a "FAIL" line or an "error TS" line</fails_when>
<automated>scripts/check-admin-openapi.sh --check && scripts/check-admin-dist.sh</automated>
<fails_when>non-zero exit (generated types or the committed dist differ from a fresh build)</fails_when>
<automated>go vet ./... && go test -short ./... && go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree' -count=1 2>&1 | tail -3 && go run ./cmd/summer docs:build --check 2>&1 | tail -5</automated>
<fails_when>non-zero exit, or a "FAIL" or "unknown identifier" line</fails_when>
</verify>
<acceptance_criteria>
- `grep -n "'customized', 'outdated'" admin/src/components/list/RowStateBadges.vue` prints one line
- `grep -n "deletable" admin/src/views/FormView.vue` prints at least one line
- `grep -n "\.summer-diff__add" admin/src/styles/main.css` prints one line
- `grep -l "cm-summer-action" modules/boardwalk/dist/assets/*.js` prints exactly one path and it is not an `index-*.js` entry chunk (CodeMirror lives in its own lazy chunk next to the richeditor's)
- `git log --format=%s -F --grep='build(admin): rebuild the embedded admin for the mail editor'` prints one line, and `git show --stat --format= <that commit>` lists only paths under modules/boardwalk/dist
</acceptance_criteria>
<done>The admin shows reset, delete and drift exactly where the UI contract puts them, and the binary embeds the finished mail editor.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| server-rendered mail HTML → admin browser | a draft may contain hostile markup that rendered on the server |
| image library entries → editors | file names and captions typed by administrators |
| npm registry → admin build | new third-party code in the admin bundle |
## STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|-----------|----------|-----------|----------|-------------|-----------------|
| T-35-19 | Elevation of privilege (stored XSS) | MailPreviewField | high | mitigate | iframe with an empty sandbox (no scripts, no same-origin, no forms, no top navigation), srcdoc only from a 2xx answer, no src, never v-html; server already ran validateHTML; CSP frame-src unchanged (mail-smoke sandbox case) |
| T-35-20 | Tampering | insertImage snippets | medium | mitigate | alt and url escaped per target (markdown brackets, HTML entities), URLs only from the server (mail-smoke insertImage table) |
| T-35-21 | Tampering (supply chain) | CodeMirror packages | high | mitigate | decision note and STACK row first, blocking human checkpoint (never auto-approved) with integrity values, exact pins, committed lockfile, npm ci; lazy chunk limits the surface to the code editor |
| T-35-22 | Abuse | test-send button | medium | mitigate | no recipient input; 429 disables the button for Retry-After; the server enforces the rule regardless |
| T-35-23 | Information disclosure | picker and preview errors | low | mitigate | only server-provided messages and fixed copy are shown; no stack traces |
| T-35-24 | Denial of service (keyboard trap) | codeeditor | low | mitigate | Tab unbound in the editor (WCAG 2.1.2), browser smoke test |
| T-35-SC | Tampering | npm/pip/cargo installs | high | mitigate | package-legitimacy gate (research audit, [SUS] @codemirror/view) + blocking human checkpoint Task 3 before the install in Task 4 |
</threat_model>
<verification>
- `npm --prefix admin run test`, `npm --prefix admin run test:browser` and `npm --prefix admin run typecheck` green after every task that touches admin/.
- `scripts/check-admin-openapi.sh --check` and `scripts/check-admin-dist.sh` pass after Task 6.
- `go vet ./... && go test -short ./...` and the docs checker pass.
</verification>
<success_criteria>
- The four field controls, the picker, the footer actions, the badges and the diff kit match UI-SPEC C1-C4 and C7-C11.
- CodeMirror is documented, human-approved, exactly pinned, lazily loaded.
- modules/boardwalk/dist is rebuilt and passes the dist gate; nothing pushed or tagged.
</success_criteria>
<output>
Create `.planning/phases/35-mail-templates-and-layouts-editor/35-03-SUMMARY.md` when done; record the Task 3 answer verbatim and the installed versions.
</output>
## Artifacts this phase produces
- SPA components: `MailPreviewField.vue`, `CodeEditorField.vue`, `ColorPickerField.vue`, `MailImageField.vue`, `MailImagePicker.vue`; modules `codeeditor/editor.ts` (`createCodeEditor`), `mail/insertImage.ts` (`insertImage`).
- Registry entries: `mailpreview` (valueless, group-labelled), `codeeditor`, `colorpicker`, `mailimage` (group-labelled).
- FormView: footer record actions on forms without a preview screen, `meta.deletable`, partial `reloadKey`; SettingsFormView: `FORM_VALUES` and `FORM_LOCALE` provides.
- Lists: `customized` and `outdated` badges; CSS kit `.summer-diff`, `.summer-diff__add`, `.summer-diff__del`, `.summer-diff__ctx`; editor class `cm-summer-action`.
- npm runtime dependencies (exact pins): `@codemirror/state`, `@codemirror/view`, `@codemirror/language`, `@codemirror/commands`, `@codemirror/lang-html`, `@codemirror/lang-css`, `@lezer/highlight`.
- Planning docs: `.planning/notes/2026-11-06-admin-code-editor-codemirror.md`, STACK.md row.
- Generated: rebuilt `modules/boardwalk/dist`; smoke tests `admin/tests/form/mail-smoke.test.ts`, `admin/tests/browser/codeeditor-smoke.test.ts`.
- Config keys: none.

View File

@@ -0,0 +1,339 @@
---
phase: 35-mail-templates-and-layouts-editor
plan: 04
type: execute
wave: 4
depends_on: ["35-01", "35-02", "35-03"]
files_modified:
- scripts/check-phase35.sh
- modules/postcard/override_test.go
- modules/postcard/partials_test.go
- modules/postcard/samples_test.go
- modules/postcard/editor_test.go
- modules/postcard/brand_test.go
- modules/postcard/store_postgres_test.go
- modules/pact/capabilities_test.go
- modules/lagoon/attach/absolute_test.go
- modules/cabana/mail_permissions_postgres_test.go
- modules/cabana/mail_lifecycle_postgres_test.go
- modules/cabana/mail_test_send_postgres_test.go
- modules/cabana/mail_unit_test.go
- modules/cabana/field_mail_test.go
- modules/surf/store_lookup_test.go
- admin/tests/form/CodeEditorField.test.ts
- admin/tests/form/MailPreviewField.test.ts
- admin/tests/form/insertImage.test.ts
- admin/tests/form/MailImagePicker.test.ts
- admin/tests/form/ColorPickerField.test.ts
- admin/tests/form/MailImageField.test.ts
- admin/tests/form/FormViewMail.test.ts
- admin/tests/list/RowStateBadges.test.ts
- admin/tests/views/SettingsFormView.test.ts
- admin/tests/browser/codeeditor.test.ts
- .planning/phases/35-mail-templates-and-layouts-editor/35-VALIDATION.md
autonomous: true
requirements: [D-21, D-03, D-06, D-07, D-11, D-12, D-13, D-14, D-15, D-17, D-23]
estimate:
tokens: 170000
raw_tokens: 170000
tasks: 4
confidence: low
must_haves:
truths:
- "scripts/check-phase35.sh runs every Phase 35 check by stage (self-test, go, docker, consumers, admin, docs, marker, additive), fails closed on a missing repository, a skipped or missing required test, a zero-match run or a planted failure, and its consumers stage runs the sm-user-plugin and sm-paymentgateway-plugin mail tests against this checkout (D-21, Wave 0 of 35-VALIDATION.md)"
- "Unit tests cover override precedence and reset: an override beats the file, the caller's subject beats the override, no row means the file, and ResetOverride restores the file (D-21, D-14, D-23)"
- "Unit tests cover drift detection: a changed file hash marks a customized row outdated in postcard's sync and in cabana's row states and filter, and saving again clears it (D-21, D-07)"
- "Unit tests cover fallback at send: a render failure, a safety-check failure, a lookup error, a missing table and no database each send the file version, and an unresolved layout code uses the default layout (D-21, D-14)"
- "Unit tests cover save validation: every DraftError rule (parse, render, unsafe, subject_line, layout, partial, no_content, colour) in postcard and its 422 field line with translated text in cabana, including the 55 paymentgateway-shaped and 8 user-shaped real template shapes passing validation with no samples (D-21, D-13, D-15)"
- "Unit tests cover Branding precedence over config: no row uses mail.brandCss, a saved row's compiled CSS replaces it, mail.css still applies, and an invalid colour is refused (D-21, D-11)"
- "Unit tests cover the partial capability: boot errors for a foreign namespace, a missing file, a duplicate and a name outside mail.partials; token forgery; layouts and text parts; pact.HasMailTemplates' method set is pinned unchanged (D-21, D-12)"
- "Unit tests cover the permission split: each controller, preview mode, test-send mode, picker list and upload against templates-only, layouts-only, both and superuser principals, and no role holds either code after migrations (D-21, D-03)"
- "Unit tests cover the test-send recipient restriction and rate limit: only the admin's own email receives mail, a recipient key in the body is refused, token principals are refused, and the sixth send in 10 minutes is a 429 with Retry-After through a fake cooler.Attempter (D-21, D-17)"
- "SPA unit and browser tests cover the codeeditor keyboard rules, insertImage escaping, the picker's canUpload states, the preview sandbox and sequence guard, the footer record actions, meta.deletable and the new row-state badges (D-21)"
- "35-VALIDATION.md's per-decision map is filled with the final commands and statuses, and every Wave 0 item is checked"
prohibitions:
- statement: "No production code changes in this plan except fixes for defects the new tests expose, each recorded as a deviation in the SUMMARY with its own fix commit."
status: unresolved
verification: judgment
- statement: "Tests never write into a sibling repository and never reach the network beyond localhost, httptest servers and testcontainers."
status: unresolved
verification: judgment
- statement: "Nothing is pushed or tagged by this plan (tag only at phase completion, on the default branch, after Jakub's go-ahead)."
status: unresolved
verification: judgment
artifacts:
- path: "scripts/check-phase35.sh"
provides: "phase gate (tracer)"
contains: "--self-test"
- path: "modules/postcard/override_test.go"
provides: "precedence and fallback unit tests"
contains: "func TestOverride"
- path: "modules/cabana/mail_permissions_postgres_test.go"
provides: "permission split tests"
contains: "func TestMailPermissions"
- path: "modules/cabana/mail_test_send_postgres_test.go"
provides: "recipient and rate-limit tests"
contains: "func TestMailTestSend"
- path: "admin/tests/browser/codeeditor.test.ts"
provides: "keyboard tests in a real engine"
contains: "Mod"
key_links:
- from: "scripts/check-phase35.sh"
to: "../sm-paymentgateway-plugin"
via: "consumers stage runs TestMailCatalog, TestMailTracer and TestMailersMatrixSmoke against this checkout"
pattern: "TestMailersMatrixSmoke"
- from: "scripts/check-phase35.sh"
to: "scripts/apisurface/main.go"
via: "additive stage compares exported API with PHASE35_BASE_TAG"
pattern: "apisurface"
---
## Phase Goal
ROADMAP Goal (not in user-story format; no story is invented): administrators edit every mail the application sends from the admin panel, and mail that already works keeps working. Project rule 2 puts unit tests in the last plan of the phase; this plan adds them for the D-21 list and the phase gate that also proves the two core consumer plugins still send their mail unchanged.
Spec-less probe fallback skipped: no SPEC.md and no requirement IDs; must_haves derive from CONTEXT D-21 and the other decisions it lists.
<objective>
- `scripts/check-phase35.sh`: one fail-closed gate over the framework, the admin SPA, the docs and the sibling sm-user-plugin and sm-paymentgateway-plugin mail tests.
- Unit tests for postcard, pact and attach (precedence, reset, drift hashes, fallback, validation, samples, partials, Branding).
- Unit tests for cabana and surf (permission split, save validation, drift states and filter, reset, delete refusal, test-send recipient and limit, locale, diff, field schema checks, the published Attempter).
- SPA unit and browser tests; the filled 35-VALIDATION.md.
Purpose: D-21 and the Nyquist map in 35-VALIDATION.md; the gate is what `/msd-verify-work` runs.
Output: test commits in this repository; one planning-docs commit for 35-VALIDATION.md. No sibling repository is changed; nothing is pushed or tagged.
</objective>
<execution_context>
@~/.claude/msd-core/workflows/execute-plan.md
@~/.claude/msd-core/templates/summary.md
</execution_context>
<context>
@.planning/phases/35-mail-templates-and-layouts-editor/35-VALIDATION.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-RESEARCH.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-01-SUMMARY.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-02-SUMMARY.md
@.planning/phases/35-mail-templates-and-layouts-editor/35-03-SUMMARY.md
@CLAUDE.md
<interfaces>
scripts/check-phase34.sh and scripts/check-phase33.sh: the stage engine to copy (usage, detect_go on go test -json, go_named / go_pkg_named for required names, require_dir "missing <path>", marker_hits, npm_vitest, run_self_test planting, additive_case over scripts/apisurface, docker NOT RUN rule, PHASE*_ALLOW_NO_DOCKER).
Smoke tests left by 35-01..35-03: TestMailOverrideSmoke, TestMailPartialsSmoke, TestMailEditorSmoke, TestMailBrandSmoke (modules/postcard, the Postgres helper lives in override_smoke_test.go), TestMailScreensSmoke and newMailScreensEnv (modules/cabana/mail_screens_smoke_test.go), admin/tests/form/mail-smoke.test.ts, admin/tests/browser/codeeditor-smoke.test.ts.
Consumer tests (run against this checkout through their go.mod `replace git.golem15.com/golem15/summercms => ../summercms.go`): ../sm-user-plugin TestMail (mail_test.go), TestPolishMailSubjects (mail_subjects_test.go); ../sm-paymentgateway-plugin TestMailCatalog, TestMailTracer, TestMailersMatrixSmoke (mail_smoke_test.go; that repo also has a go.work).
Base tag for marker and additive: v0.6.3 (newest tag before the phase), overridable with PHASE35_BASE_TAG.
</interfaces>
</context>
## Execution units and resume points
- Each task ends with the commit listed below, staged by explicit path. Resume check: `git log --format=%s -F --grep='<subject>'`.
- A test that exposes a production defect: fix it in its own `fix(<module>): ...` commit, record it as a deviation in the SUMMARY, then continue.
- Docker-backed tests run without `-short` (through `sudo -u golem` if the Docker socket is denied).
| Task | Commit subject | Resume point |
|------|----------------|--------------|
| 1 | `test(35): phase gate script check-phase35.sh` | `--self-test` passes |
| 2 | `test(postcard): unit tests for overrides, fallback, partials, samples, validation, sync and branding` | postcard tests pass |
| 3 | `test(cabana): unit tests for mail permissions, save validation, drift, reset, layout delete and test send` | cabana mail tests pass |
| 4 | `test(admin): unit and browser tests for the mail editor fields`, then `docs(35): fill the validation map` | `scripts/check-phase35.sh --all` passes |
<tasks>
<task type="tracer">
<name>Task 1: End-to-end "one command runs every Phase 35 check, including the core plugins' mail" — the gate script over the smoke tests plans 01-03 left behind</name>
<files>scripts/check-phase35.sh</files>
<read_first>
- scripts/check-phase34.sh (whole file: stage engine, self-test planting, marker word list, additive comparison)
- scripts/apisurface/main.go
- .planning/phases/35-mail-templates-and-layouts-editor/35-VALIDATION.md and the three plan SUMMARYs (test names and commands that worked)
</read_first>
<action>
Create `scripts/check-phase35.sh` from check-phase34.sh's engine, labelled Phase 35 (mail templates and layouts editor), with stages (each a flag; `--all` runs them in this order; no argument prints usage listing every flag and exits 2):
- `--self-test`: plants a failing go test name, a missing required name, a skipped required test, a marker hit in a scratch repository, an additive removal and an interface method added in planted packages, and a missing repository, and requires each to be detected.
- `--go`: `go vet ./...`, `go test -short ./...`, the Docker-free tests by required name in GO_TESTS (TestMailPartialsSmoke, the -short subtests of TestMailEditorSmoke and TestMailBrandSmoke; Tasks 2 and 3 extend the list), and `cd examples/hello && go test ./...`.
- `--docker`: Docker host only (NOT RUN otherwise, which fails --all unless PHASE35_ALLOW_NO_DOCKER=1): `go test ./...` plus DOCKER_TESTS by required name (TestMailOverrideSmoke, TestMailEditorSmoke, TestMailBrandSmoke, TestMailScreensSmoke, TestMigrateStatusJSON; later tasks extend it).
- `--consumers`: for ../sm-user-plugin and ../sm-paymentgateway-plugin: check that the framework module resolves to this checkout (`go list -m -f '{{.Dir}}' git.golem15.com/golem15/summercms` equals this checkout), `go vet ./...`, and the named mail tests (TestMail and TestPolishMailSubjects; TestMailCatalog, TestMailTracer and TestMailersMatrixSmoke) without -short when Docker answers, each required to run and pass; read-only (no file in those repositories is written).
- `--admin`: `npm --prefix admin run test` with required files (ADMIN_FILES: mail-smoke.test.ts now; Task 4 extends it), `npm --prefix admin run test:browser` with codeeditor-smoke required, `npm --prefix admin run typecheck`, `scripts/check-admin-openapi.sh --check`, `scripts/check-admin-dist.sh`.
- `--docs`: TestDocsTree, TestCommandsDocumented and `go run ./cmd/summer docs:build --check`.
- `--marker`: the lines added since PHASE35_BASE_TAG (default v0.6.3) to non-test Go files, READMEs and docs under modules/, cmd/, internal/ and docs/ name no consuming application (the check-phase34 word list).
- `--additive`: scripts/apisurface of every package under modules/ changed since the base tag keeps every exported line and adds no method to an interface that exists at the tag (pact, postcard, cabana, lagoon/attach and surf only gain elements; this proves pact.HasMailTemplates, postcard.Mailer and Catalog.Register are unchanged).
A stage whose repository or directory is missing fails with `missing <path>`. Commit as listed in Execution units.
</action>
<verify>
<automated>bash scripts/check-phase35.sh --self-test</automated>
<fails_when>non-zero exit, or a planted case reported as undetected</fails_when>
<automated>bash scripts/check-phase35.sh --go --docs --marker --additive</automated>
<fails_when>non-zero exit, or a "FAIL", "missing " or marker hit line</fails_when>
<automated>bash scripts/check-phase35.sh --consumers --admin</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
</verify>
<acceptance_criteria>
- `bash scripts/check-phase35.sh; echo $?` prints a usage naming --self-test, --go, --docker, --consumers, --admin, --docs, --marker, --additive, --all and exits 2
- `grep -n "TestMailersMatrixSmoke" scripts/check-phase35.sh` and `grep -n "TestPolishMailSubjects" scripts/check-phase35.sh` each print at least one line
- `grep -c "missing " scripts/check-phase35.sh` is at least 1
- `git -C ../sm-user-plugin status --short` and `git -C ../sm-paymentgateway-plugin status --short` print the same output before and after a `--consumers` run
</acceptance_criteria>
<done>The phase has one gate that proves the mail editor and the unchanged mail of the core plugins together.</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Unit tests for postcard, pact and attach: precedence, reset, drift hashes, fallback, validation, samples, partials and Branding</name>
<files>modules/postcard/override_test.go, modules/postcard/partials_test.go, modules/postcard/samples_test.go, modules/postcard/editor_test.go, modules/postcard/brand_test.go, modules/postcard/store_postgres_test.go, modules/pact/capabilities_test.go, modules/lagoon/attach/absolute_test.go, scripts/check-phase35.sh</files>
<read_first>
- modules/postcard/*.go and the four *_smoke_test.go files (35-01 output; reuse the Postgres helper and fixtures, do not duplicate the smoke assertions)
- modules/pact/capabilities_test.go (~250-265 TestRowStateValues, ~410-420 the set check)
- ../sm-paymentgateway-plugin/views/mail and ../sm-user-plugin/views/mail (read only: copy the template shapes into testdata of the test, never import those repositories)
- .planning/phases/35-mail-templates-and-layouts-editor/35-VALIDATION.md (test names per decision)
</read_first>
<behavior>
- TestOverride: table over (no row, customized template, customized layout, database-only layout, partial override, caller Subject) with a fake OverrideSource; each asserts the exact subject, HTML fragment and text part
- TestFallback: render failure, unsafe override, lookup error, unresolved layout code (default layout plus warning), DB source with no published DB; each sends exactly one message whose body is the file version; log records carry component postcard
- TestPartial: HasMailPartials boot errors (foreign namespace, missing file, duplicate, name without mail.partials), bare built-in name, full plugin name, odd kv count, unknown name, forged token in a variable, layout use, text part, nonce differs between two renders
- TestSample: case kept after the prefix; old headers unchanged; walker over nested, if/else, with re-rooting, range of maps, range of strings, chains and variables; every real consumer template shape copied into testdata validates with no samples
- TestValidateDraft: every Rule with Field and a Message that contains no filesystem path or SQL text
- TestBrand: compile defaults, refuse bad colours with the field name, empty colour uses default, lower-casing, precedence of a brand row over mail.brandCss with mail.css kept, logoUrl absolute or empty
- TestStore and TestSync (Postgres): NewDBSource reads only is_custom rows; SyncIndex idempotent with stable IDs, concurrent calls, customized row keeps its override columns while file_hash refreshes (drift: base_hash differs), database-only layouts untouched, ResetOverride restores the file and refuses a database-only layout
- TestRowStateValues pins five values; a compile-time and reflect check pins pact.HasMailTemplates to exactly MailTemplatesFS, MailTemplates and MailLayouts and HasMailPartials to MailPartialsFS and MailPartials
- TestAbsolutePublicURL: public_url, app.url with default and custom prefix, neither (ErrNoAbsoluteURL), trailing slashes, a public_url with a query refused
</behavior>
<action>
Per D-21 (and D-11, D-12, D-13, D-14, D-15, D-07, D-23 it covers): write the tests above with the names VALIDATION.md lists (TestOverride*, TestFallback*, TestPartial*, TestSample*, TestValidateDraft*, TestBrand*, TestStore*, TestSync*), table-driven, `-short` for everything that needs no database and testing.Short skips for the Postgres ones (reuse the override_smoke_test.go helper; no TestMain scanning os.Args). Consumer template shapes go into the test as string fixtures with neutral plugin IDs (acme.*), never naming a consuming application. Add the new names to GO_TESTS and DOCKER_TESTS in scripts/check-phase35.sh. Fix production defects only as described in Execution units. Commit as listed.
</action>
<verify>
<automated>go test -short ./modules/postcard/... ./modules/pact/... ./modules/lagoon/attach/... -run 'TestOverride|TestFallback|TestPartial|TestSample|TestValidateDraft|TestBrand|TestRowStateValues|TestAbsolutePublicURL' -count=1 -v 2>&1 | grep -E '^(--- |FAIL|ok)' | tail -30</automated>
<fails_when>a "FAIL" line, "no tests to run", or no "--- PASS: TestFallback" line</fails_when>
<automated>go test ./modules/postcard -run 'TestStore|TestSync' -count=1 -v 2>&1 | grep -E '^(--- |FAIL|ok)' | tail -10</automated>
<fails_when>a "FAIL" line, a "--- SKIP" line, or no "--- PASS: TestSync" line</fails_when>
<automated>bash scripts/check-phase35.sh --go --docker</automated>
<fails_when>non-zero exit or a "FAIL" or "missing " line</fails_when>
</verify>
<acceptance_criteria>
- `grep -n "func TestFallback" modules/postcard/override_test.go` and `grep -n "func TestSync" modules/postcard/store_postgres_test.go` each print at least one line
- `grep -n '"customized"' modules/pact/capabilities_test.go` and `grep -n '"outdated"' modules/pact/capabilities_test.go` each print at least one line
- `git show --stat --format= <the Task 2 test commit sha>` lists only `_test.go` files and scripts/check-phase35.sh; any production change is in a separate `fix(...)` commit recorded as a deviation in the SUMMARY
</acceptance_criteria>
<done>postcard's override, fallback, partial, sample, validation, sync and Branding rules are pinned by unit tests.</done>
</task>
<task type="auto" tdd="true">
<name>Task 3: Unit tests for cabana and surf: permission split, save validation, drift, reset, layout delete, test send, locale, diff and field schema</name>
<files>modules/cabana/mail_permissions_postgres_test.go, modules/cabana/mail_lifecycle_postgres_test.go, modules/cabana/mail_test_send_postgres_test.go, modules/cabana/mail_unit_test.go, modules/cabana/field_mail_test.go, modules/surf/store_lookup_test.go, scripts/check-phase35.sh</files>
<read_first>
- modules/cabana/mail_screens_smoke_test.go (newMailScreensEnv) and modules/cabana/administrators_harness_test.go (seedAdmin, seedRole, login, request)
- modules/cabana/mail_admin.go, mail_routes.go, mail_diff.go, field_mail.go (35-02 output)
- modules/surf/store_lookup_test.go and modules/surf/serve.go (installCooler)
- .planning/phases/35-mail-templates-and-layouts-editor/35-VALIDATION.md
</read_first>
<behavior>
- TestMailPermissions: matrix of principals (templates-only, layouts-only, both, superuser, none) × (templates list, form PUT, layouts list, partials list, Branding GET and PUT, image list, image upload, Mail images singleton, preview per mode, test send per mode, settings tiles); expected status per UI-SPEC C14 and D-03; after migrations no role or user permission JSON contains either code
- TestMailSave: each postcard rule becomes a 422 on its field with the translated text and :message filled (en and pl), the row is unchanged afterwards; Branding colour 422 colorpicker.invalid; logo id outside the library 422 mailimage.missing; custom layout code format and taken
- TestMailDrift: base_hash stamped on save; file change plus sync gives [customized, outdated]; the status filter outdated; re-save clears; _info shows the callout only when outdated
- TestMailReset and TestMailLayoutDelete: reset per kind restores the file and removes the action from meta.actions; reset not offered on custom layouts; delete of an in-use custom layout is 403 with name and count in en (one and other) and pl (one, few, many); file rows never deletable; meta.deletable values
- TestMailTestSend: recipient is the admin's email only (Cc and Bcc empty), body with to/email/recipient refused, token principal refused, a fake cooler.Attempter published on the app counts per admin and the sixth call returns 429 with Retry-After, a second admin is independent, the in-process fallback limits without a published Attempter, a failing driver gives 502 with the fixed message
- TestMailLocale (-short): TemplateLocale with enabled and unknown suffixes as cabana computes locales; TestMailDiff (-short): lineDiff context, cap, identical inputs; TestMailFieldSchema (-short): codeeditor language, mailImages placement, mailpreview mode, settings-form acceptance of colorpicker, mailimage and mailpreview
- surf TestInstallCoolerPublishesAttempter: installCooler with the Redis rate-limit store publishes a cooler.Attempter that is the same store as the published Store (miniredis); with the memory store it publishes neither
</behavior>
<action>
Per D-21 (and D-03, D-06, D-07, D-08, D-13, D-17, D-05 it covers): write the tests above with the names VALIDATION.md lists (TestMailPermissions, TestMailSave, TestMailDrift, TestMailReset, TestMailLayoutDelete, TestMailBrand, TestMailTestSend, TestMailLocale) plus TestMailDiff and TestMailFieldSchema, reusing newMailScreensEnv and the 19.6.1 harness helpers; Postgres tests skip under testing.Short. Add the names to GO_TESTS / DOCKER_TESTS in scripts/check-phase35.sh. Fix production defects only as described in Execution units. Commit as listed.
</action>
<verify>
<automated>go test ./modules/cabana -run 'TestMailPermissions|TestMailSave|TestMailDrift|TestMailReset|TestMailLayoutDelete|TestMailBrand|TestMailTestSend' -count=1 -v 2>&1 | grep -E '^(--- |FAIL|ok)' | tail -30</automated>
<fails_when>a "FAIL" line, a "--- SKIP" line, "no tests to run", or no "--- PASS: TestMailTestSend" or "--- PASS: TestMailPermissions" line</fails_when>
<automated>go test -short ./modules/cabana ./modules/surf -run 'TestMailLocale|TestMailDiff|TestMailFieldSchema|TestInstallCoolerPublishesAttempter' -count=1 -v 2>&1 | grep -E '^(--- |FAIL|ok)' | tail -10</automated>
<fails_when>a "FAIL" line, "no tests to run", or no "--- PASS: TestInstallCoolerPublishesAttempter" or "--- PASS: TestMailDiff" line</fails_when>
<automated>bash scripts/check-phase35.sh --go --docker</automated>
<fails_when>non-zero exit or a "FAIL" or "missing " line</fails_when>
</verify>
<acceptance_criteria>
- `grep -n "func TestMailPermissions" modules/cabana/mail_permissions_postgres_test.go` and `grep -n "func TestMailTestSend" modules/cabana/mail_test_send_postgres_test.go` each print one line
- `grep -n "Retry-After" modules/cabana/mail_test_send_postgres_test.go` prints at least one line
- `grep -n "cooler.Attempter" modules/surf/store_lookup_test.go` prints at least one line
- `grep -n "TestMailLayoutDelete" scripts/check-phase35.sh` prints at least one line
</acceptance_criteria>
<done>The permission split, the recipient rule, the limit and every admin-side refusal are pinned by unit tests.</done>
</task>
<task type="auto" tdd="true">
<name>Task 4: SPA unit and browser tests for the mail editor, the filled validation map and the full gate</name>
<files>admin/tests/form/CodeEditorField.test.ts, admin/tests/form/MailPreviewField.test.ts, admin/tests/form/insertImage.test.ts, admin/tests/form/MailImagePicker.test.ts, admin/tests/form/ColorPickerField.test.ts, admin/tests/form/MailImageField.test.ts, admin/tests/form/FormViewMail.test.ts, admin/tests/list/RowStateBadges.test.ts, admin/tests/views/SettingsFormView.test.ts, admin/tests/browser/codeeditor.test.ts, scripts/check-phase35.sh, .planning/phases/35-mail-templates-and-layouts-editor/35-VALIDATION.md</files>
<read_first>
- admin/tests/form/mail-smoke.test.ts and admin/tests/browser/codeeditor-smoke.test.ts (35-03 output; extend coverage, do not duplicate)
- admin/tests/form/RichEditorField.test.ts, admin/tests/form/FormView.test.ts, admin/tests/list/RowStateBadges.test.ts, admin/tests/views/SettingsFormView.test.ts (mount patterns, api mocks)
- .planning/phases/35-mail-templates-and-layouts-editor/35-UI-SPEC.md (UI Considerations rows marked backstop: footer reset, meta.deletable, codeeditor keyboard, insertImage, mailpreview failure and sandbox)
- .planning/phases/35-mail-templates-and-layouts-editor/35-VALIDATION.md
</read_first>
<behavior>
- codeeditor (browser): Tab and Shift+Tab leave the editor; Mod+S and Mod+Enter save; Esc does not cancel; undo and redo through history; the Go action decoration on `{{ .Name }}`; text wraps, html does not
- CodeEditorField (happy-dom): heights per size, loading box, failure fallback textarea saves, read-only pre, external value replacement without an emit
- insertImage: the full C4 table plus spaces and parentheses in URLs and empty alt
- MailPreviewField: sandbox empty and no src, srcdoc only from 2xx, sequence guard, failed and failed_first and unavailable, empty draft sends nothing, inactive tab sends nothing, test send 2xx toast with email, 429 seconds and disabled button, 422 inline, 5xx toast, branding mode without subject row and test send
- MailImagePicker: canUpload true and false, empty-library copy per mode, no results, load failed, 409 server message, upload success selects the new tile, 422 upload line, primary disabled until chosen, radiogroup keys, alt prefill in insert mode only
- ColorPickerField and MailImageField: per UI-SPEC C2 and C11
- FormViewMail: footer reset with danger confirm, busy disables the footer, success toast plus reload plus clean form plus reloadKey bump, 409 and 403 paths, meta.deletable false/true/absent, no footer actions on a controller with a preview screen
- RowStateBadges: five states in order with the two new styles; SettingsFormView: FORM_VALUES and FORM_LOCALE provided
</behavior>
<action>
Per D-21: write the SPA tests above (happy-dom unit tests under admin/tests/form, list and views; the browser test under admin/tests/browser), add every new file to ADMIN_FILES and codeeditor to the required browser files in scripts/check-phase35.sh, and commit as listed. Then fill 35-VALIDATION.md: every per-decision row gets the final command and status, the Wave 0 items are checked (or name the plan that delivered them), `wave_0_complete: true` and `nyquist_compliant: true` in its front matter when every row is green, and commit it as a separate planning-docs commit. Finally run `bash scripts/check-phase35.sh --all` (with PHASE35_ALLOW_NO_DOCKER=1 only if Docker is unavailable, recorded in the SUMMARY).
</action>
<verify>
<automated>npm --prefix admin run test 2>&1 | tail -6 && npm --prefix admin run test:browser 2>&1 | tail -6</automated>
<fails_when>non-zero exit or a "FAIL" line</fails_when>
<automated>bash scripts/check-phase35.sh --all</automated>
<fails_when>non-zero exit, a "FAIL", "missing ", "NOT RUN" (without the allow variable) or marker hit line</fails_when>
</verify>
<acceptance_criteria>
- `ls admin/tests/form/insertImage.test.ts admin/tests/form/MailPreviewField.test.ts admin/tests/browser/codeeditor.test.ts` lists all three
- `grep -n "nyquist_compliant: true" .planning/phases/35-mail-templates-and-layouts-editor/35-VALIDATION.md` prints one line
- `grep -c "⬜ pending" .planning/phases/35-mail-templates-and-layouts-editor/35-VALIDATION.md` prints 0
- `git log --format=%s -F --grep='docs(35): fill the validation map'` prints one line, and `git show --stat --format= <that commit>` lists only 35-VALIDATION.md
</acceptance_criteria>
<done>The mail editor's UI rules are pinned by tests, the validation map is complete and the phase gate is green.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| gate script → sibling repositories | the consumers stage runs go commands inside ../sm-user-plugin and ../sm-paymentgateway-plugin |
| tests → Docker | testcontainers start Postgres containers on the host |
## STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|-----------|----------|-----------|----------|-------------|-----------------|
| T-35-25 | Tampering | consumers stage | medium | mitigate | read-only commands (go list, go vet, go test); acceptance compares `git status --short` of both repositories before and after a run |
| T-35-26 | Repudiation | a gate that passes on zero tests | high | mitigate | go test -json detection fails on skips, zero matches, "no tests to run" and missing required names; self-test plants each case |
| T-35-27 | Elevation of privilege | security regressions after the phase | high | mitigate | TestMailPermissions, TestMailTestSend and the preview sandbox tests pin T-35-09, T-35-11 and T-35-19 so a regression fails the gate |
| T-35-28 | Information disclosure | test fixtures | low | mitigate | fixtures use neutral plugin IDs and example.test addresses; the marker stage refuses consuming-application names |
| T-35-SC | Tampering | npm/pip/cargo installs | high | accept | this plan installs no package |
</threat_model>
<verification>
- `bash scripts/check-phase35.sh --all` green (Docker host), including the consumers stage.
- `go vet ./... && go test ./...`, `npm --prefix admin run test`, `npm --prefix admin run test:browser`, `npm --prefix admin run typecheck`, `scripts/check-admin-openapi.sh --check`, `scripts/check-admin-dist.sh`, `summer docs:build --check` green.
</verification>
<success_criteria>
- Every D-21 item has named unit tests; SPA backstop rows of the UI-SPEC have tests.
- check-phase35.sh exists, fails closed, and proves the two core plugins' mail unchanged.
- 35-VALIDATION.md is complete; nothing pushed or tagged; no sibling repository changed.
</success_criteria>
<output>
Create `.planning/phases/35-mail-templates-and-layouts-editor/35-04-SUMMARY.md` when done; list every deviation fix commit and the final `--all` result.
</output>
## Artifacts this phase produces
- Script: `scripts/check-phase35.sh` with `--self-test`, `--go`, `--docker`, `--consumers`, `--admin`, `--docs`, `--marker`, `--additive`, `--all`; environment `PHASE35_BASE_TAG`, `PHASE35_ALLOW_NO_DOCKER`.
- Go tests: `TestOverride*`, `TestFallback*`, `TestPartial*`, `TestSample*`, `TestValidateDraft*`, `TestBrand*`, `TestStore*`, `TestSync*`, `TestAbsolutePublicURL`, updated `TestRowStateValues`, `TestMailPermissions`, `TestMailSave`, `TestMailDrift`, `TestMailReset`, `TestMailLayoutDelete`, `TestMailBrand`, `TestMailTestSend`, `TestMailLocale`, `TestMailDiff`, `TestMailFieldSchema`, `TestInstallCoolerPublishesAttempter`.
- SPA tests: `CodeEditorField.test.ts`, `MailPreviewField.test.ts`, `insertImage.test.ts`, `MailImagePicker.test.ts`, `ColorPickerField.test.ts`, `MailImageField.test.ts`, `FormViewMail.test.ts`, updated `RowStateBadges.test.ts` and `SettingsFormView.test.ts`, browser `codeeditor.test.ts`.
- Planning: filled `35-VALIDATION.md`.
- No new exported identifiers, config keys or dependencies.

View File

@@ -91,6 +91,12 @@ Not in this phase: mail driver/SMTP settings screens, MailBlocker, newsletters/b
The postcard and cabana READMEs and the affected `docs/` pages are updated in the same commits as API/config/permission changes. The `admin/` changes regenerate `admin/openapi/admin.json` + `schema.d.ts` (`scripts/check-admin-openapi.sh`) and rebuild the committed `modules/boardwalk/dist` (`scripts/check-admin-dist.sh`).
### Planning-time amendments (2026-11-06, confirmed by the user after research)
- **D-22:** postcard, not cabana, owns the mail tables, their migrations (a framework set in `lagoon.migrationSets`, like `attach.Migrations`) and the per-send override lookup, because `queue:work` sends mail in a process where cabana never activates. cabana still owns the screens, permissions and controllers (D-01 otherwise unchanged).
- **D-23:** Every file template, layout and partial gets an index row (Winter `syncAll` style), synced from the file catalog at serve start. Reset (D-06) clears the override columns on that row instead of deleting it; custom layouts are still deleted.
- **D-24:** The Branding logo's absolute URL is built at send time from config (`storage.uploads.public_url` / `app.url`) through a config-based attach helper, not stored at save.
- **D-25:** postcard ships no built-in template; the templates list keeps the UI-SPEC empty-state text.
### Claude's Discretion
- Table names and schema (one overrides table per kind, or a shared one), as long as it is Postgres/GORM with gormigrate migrations owned by the cabana pseudo-plugin, or by postcard if the planner justifies it.
- Exact partial call syntax and the built-in partial set beyond button/panel/table.

View File

@@ -0,0 +1,282 @@
# Phase 35: Mail templates and layouts editor - Pattern Map
**Mapped:** 2026-11-06
**Files analyzed:** 27 (new + modified, from 35-RESEARCH.md "Recommended Project Structure" and Wave 0 gaps)
**Analogs found:** 24 / 27
Line numbers are from master at `9232dc1`. Phase 33 is editing cabana at the same time, so re-check `form_schema.go`, `http.go` and `crud.go` line numbers before executing.
## File Classification
| New/Modified File | Role | Data Flow | Closest Analog | Match Quality |
|---|---|---|---|---|
| `modules/postcard/store.go` (new: Migrations, tables, OverrideSource, index sync) | migration + service | CRUD / batch sync | `modules/lagoon/attach/migrations.go` + `modules/lagoon/ondatabase.go` | exact (migrations), role-match (sync) |
| `modules/postcard/mailer.go` (mod: Activate wires DB source, Send lookup + fallback) | service | request-response | itself, lines 92-109 | exact |
| `modules/postcard/templates.go` (mod: partials, samples, hashes, alias fallback) | service | transform | itself | exact |
| `modules/postcard/partials.go` (new) | utility | transform | `modules/postcard/templates.go` (execHTML + goldmark, lines ~343-372) | role-match |
| `modules/postcard/samples.go` (new) | utility | transform | `modules/postcard/templates.go` parseINI / parseTemplateFile | role-match |
| `modules/postcard/brand.go` (new) | utility | transform | `modules/postcard/templates.go` layout data (lines 358-372) | partial |
| `modules/postcard/editor.go` (new: exported read + Validate/Preview/SendDraft) | service | request-response | `modules/postcard/mailer.go` Send | role-match |
| `modules/postcard/assets/{system,minimal}.htm, partials/*.htm, brand.css.tmpl, sample.htm` | config/asset | file-I/O | `modules/postcard/assets/default.htm` | exact |
| `modules/pact/capabilities.go` (mod: HasMailPartials, RowStateCustomized/Outdated) | config (interface) | n/a | itself, lines 1055-1062 and ~1271 | exact |
| `modules/lagoon/migrations.go` (mod: add postcard set) | migration registry | batch | itself, lines 225-230 | exact |
| `modules/cabana/backend_screens.go` (mod: mail controllers, permissions, settings items) | provider (pseudo-plugin) | n/a | itself, lines 76-117 | exact |
| `modules/cabana/mail_admin.go` (new: models, controllers, hooks, record actions) | controller | CRUD | `modules/cabana/backend_users_admin.go` | exact |
| `modules/cabana/mail_routes.go` (new: preview, test send, image library) | controller (route) | request-response | `modules/cabana/field_markdown.go` (markdownPreview) + `http.go:342` | exact |
| `modules/cabana/field_codeeditor.go` etc. (codeeditor, colorpicker, mailimage, mailpreview) | config (field registry) | n/a | `modules/cabana/form_schema.go:24-51` | exact |
| Branding settings model (in mail_admin.go or settings registration) | model | CRUD singleton | `modules/cabana/settings.go` (lines 96-102, 186, 236-277) | exact |
| Image library host (AdminSingleton) | controller | CRUD singleton | `modules/cabana/singleton.go:41-80` | exact |
| `modules/cabana/backendscreens/controllers/mail{templates,layouts,partials,images}/config_{form,list,filter}.yaml` + `_*.htm` | config | n/a | `modules/cabana/backendscreens/controllers/users/*` | exact |
| `modules/cabana/backendscreens/models/mail*/{fields,columns}.yaml` | config | n/a | `modules/cabana/backendscreens/models/user/*` | exact |
| `admin/src/components/form/fields/CodeEditorField.vue` | component | event-driven | `admin/src/components/form/fields/RichEditorField.vue` (lazy editor) + `MarkdownField.vue` | role-match |
| `admin/src/components/form/fields/MailPreviewField.vue` | component | request-response | `admin/src/components/form/fields/MarkdownField.vue` | exact |
| `admin/src/components/form/fields/ColorPickerField.vue` | component | event-driven | `admin/src/components/form/fields/TextField.vue` | role-match |
| `admin/src/components/form/fields/MailImageField.vue`, `admin/src/components/mail/MailImagePicker.vue` | component | request-response | `admin/src/components/form/fields/FileuploadField.vue` | role-match |
| `admin/src/components/form/registry.ts` (mod) | config | n/a | itself, lines 36-90 | exact |
| `admin/src/components/form/codeeditor/editor.ts` | utility | event-driven | none (CodeMirror is new) | no analog |
| `admin/src/components/mail/insertImage.ts` | utility | transform | none | no analog |
| Go tests (`modules/postcard/*_test.go`, `modules/cabana/mail_*_test.go`) | test | n/a | `modules/cabana/administrators_harness_test.go:154`, `lagoon/postgres_test.go` | exact |
| `admin/tests/form/{codeeditor,mailpreview,insertImage}*.test.ts`, `admin/tests/browser/codeeditor.test.ts` | test | n/a | existing `admin/tests/form/*` markdown tests | role-match |
## Pattern Assignments
### `modules/postcard/store.go` (migration + DB-backed source)
**Analog:** `modules/lagoon/attach/migrations.go` (lines 1-40). Raw SQL statements inside a gormigrate migration, ID prefixed with a date stamp:
```go
var Migrations = []*gormigrate.Migration{
{
ID: "202609180001_create_system_files",
Migrate: func(tx *gorm.DB) error {
stmts := []string{
`CREATE TABLE system_files (
id SERIAL PRIMARY KEY,
...
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
)`,
`CREATE INDEX system_files_field_index ON system_files (field)`,
}
for _, stmt := range stmts {
if err := tx.Exec(stmt).Error; err != nil {
return err
```
Use the schema in 35-RESEARCH.md "Recommended schema" (Winter-style table names). Wire the index sync through `lagoon.OnDatabase` (`modules/lagoon/ondatabase.go:19-25`):
```go
func OnDatabase(app *backpack.App, fn func(sqlDB *sql.DB, gdb *gorm.DB) error) error
```
It runs immediately if the DB is already published, otherwise when Publish runs. Because lagoon imports postcard (for the migration set), postcard must not import lagoon. Look the `*gorm.DB` up through `backpack` (`app.Lookup`) at send time, or take the source through an interface that postcard defines (D-01).
### `modules/lagoon/migrations.go` (modified)
**Analog:** itself, `migrationSets` (lines 220-230):
```go
sets := []migrationSet{
{"summercms.attach", "system_files", attach.Migrations},
{"summercms.cabana", "backend admin", BackendAdminMigrations},
{DeferredHistoryID, "deferred bindings", DeferredBindingMigrations},
{QueueHistoryID, "queue", QueueMigrations(sqlDB)},
}
```
Add `{"summercms.postcard", "mail", postcard.Migrations},` and update the set list in `modules/lagoon/pending_test.go`.
### `modules/postcard/mailer.go` (modified)
**Analog:** itself, `Activate` lines 92-109:
```go
func Activate[P interface{ ID() string }](app *backpack.App, _ []P) error {
...
m := &mailer{catalog: NewCatalog(), driver: driver, opts: opts}
if err := app.Publish[Mailer](m); err != nil {
return fmt.Errorf("postcard: %w", err)
}
return nil
}
```
Do not change the `Mailer` interface: `../sm-paymentgateway-plugin/mail_smoke_test.go:56` implements it. Add the override lookup inside `(*mailer).Send`. Treat any lookup error as "no overrides" and log a rate-limited warning (Pitfall 7).
### `modules/pact/capabilities.go` (modified)
**Analog:** itself. Row states at lines 1055-1062:
```go
const (
RowStateDeleted RowState = "deleted"
RowStateNegative RowState = "negative"
RowStateDisabled RowState = "disabled"
)
```
Append `RowStateCustomized` and `RowStateOutdated` in the same style, and update `TestRowStateValues` and cabana's `rowStateOrder`. `HasMailTemplates` (lines ~1270-1275) stays as it is. Put `HasMailPartials` next to it, in the same three-method `fs.FS` + names shape.
### `modules/cabana/backend_screens.go` (modified)
**Analog:** itself, lines 76-117. Add exported permission and controller ID constants next to `PermissionManageUsers`/`AdministratorsControllerID` (lines 14-28). Extend:
```go
func (s *backendScreens) AdminControllers() []pact.AdminController {
return []pact.AdminController{usersAdmin{screens: s}, rolesAdmin{screens: s}}
}
func (*backendScreens) Permissions() []pact.Permission {
return []pact.Permission{
{Code: PermissionManageUsers, Tab: "backend::lang.permissions.tab_system", Label: "backend::lang.permissions.manage_users"},
```
Settings items follow the `pact.SettingsItem{Code, Label, Description, Category: "backend::lang.settings.category_system", Icon, Order, Permissions, Controller}` shape (lines 96-115). No role grants in code (D-03 / 19.6.1 D-04). The YAML is embedded through `//go:embed all:backendscreens` (line 38), so the new `controllers/mail*/` and `models/mail*/` directories are picked up with no extra wiring.
### `modules/cabana/mail_admin.go` (controller, CRUD)
**Analog:** `modules/cabana/backend_users_admin.go`.
**Struct + interface assertions** (lines 24-53):
```go
type usersAdmin struct {
screens *backendScreens
}
var (
_ pact.AdminController = usersAdmin{}
_ pact.AdminPermissioned = usersAdmin{}
_ pact.ListRowStates = usersAdmin{}
_ pact.FormBeforeUpdate = usersAdmin{}
_ pact.FormBeforeDelete = usersAdmin{}
_ pact.HasAdminRecordActions = usersAdmin{}
_ pact.AdminPartialData = usersAdmin{}
...
)
```
**Identity methods** (lines 55-63): `ID()`, `ModelName()`, `ConfigDir() "controllers/users"`, `RequiredPermissions()`, `NewRecord()`.
**Helpers to copy:** `db()` (109), `requestDB(ctx)` (122), `translator()` (130), `asBackendUser` type assertion (140).
**Row states** (178-193): Customized when `is_custom`, Outdated when `is_custom && base_hash != file_hash`.
```go
func (usersAdmin) ListRowStates(_ context.Context, _ *gorm.DB, records []any) ([][]pact.RowState, error) {
out := make([][]pact.RowState, len(records))
for i, record := range records {
user, err := asBackendUser(record)
if err != nil { return nil, err }
if user.DeletedAt.Valid { out[i] = append(out[i], pact.RowStateDeleted) }
...
```
**Hooks:** FormBeforeDelete (199) is the model for refusing deletion of an in-use layout (use `ForbiddenError`, Pitfall 9). FormRules (475) handles save validation. Record actions (325) cover Reset to default and Test send. PartialData (422) feeds the server-rendered `_info.htm` / drift partials. Parameterized messages are built with `backendMessageParams(ctx, tr, key, fallback, params)` (`account_tokens.go:206,227`), not with `localizeDetails`.
### `modules/cabana/mail_routes.go` (route, request-response)
**Analog:** `modules/cabana/field_markdown.go` lines 55-80 plus mounting at `modules/cabana/http.go:342`:
```go
g.Post("/markdown/preview", requireAjax(s.markdownPreview))
```
```go
type AdminMarkdownPreviewRequest struct { Markdown string `json:"markdown"` }
type AdminMarkdownPreviewResult struct { HTML string `json:"html"` }
func (s *service) markdownPreview(w http.ResponseWriter, r *http.Request) {
principal, ok := bouncer.User(r.Context())
if !ok || principal == nil || !principal.Backend {
WriteError(w, http.StatusUnauthorized, "unauthenticated", msgUnauthenticated)
return
}
if !tokenRouteGate(principal, w) { return }
var body AdminMarkdownPreviewRequest
```
Add an exported `AdminMail*Request/Result` pair per route, swag annotations, and a per-`mode` permission check. The 422 responses use fixed message constants (`msgMarkdownPreviewRefused` style) and never echo the renderer's error. Rate-limit test send with `cooler.Attempter` (`modules/cooler/cooler.go:68-73`). Afterwards run `scripts/check-admin-openapi.sh`.
### New field types (`form_schema.go` + `field_codeeditor.go`)
**Analog:** `modules/cabana/form_schema.go` lines 24-51:
```go
formFieldTypes = map[string]struct{}{
"text": {}, ... "richeditor": {}, "repeater": {}, "money": {}, "taglist": {},
}
formFieldKeys = map[string]struct{}{ ... "mode": {}, ... "separator": {}, "customTags": {}, }
```
Add `codeeditor`, `colorpicker`, `mailimage`, `mailpreview` to the types, and `language` and `mailImages` to the keys. If the Branding settings form uses colorpicker, add it to the settings refusal check at `settings.go:102` only for types that need to be refused.
### Branding settings model
**Analog:** `modules/cabana/settings.go`. Fields are decoded with `decodeFields(raw, types.refusing())` (line 96). The refused built-ins are `widget/partial/fileupload/permissioneditor` (line 102), so a Branding logo upload cannot use `fileupload` there and needs a `mailimage` URL field instead. The singleton row sits at PK 1 (line 186), and the first Put creates it (lines 236-277). Example registration: `modules/cabana/example_settings_test.go`.
### Image library host
**Analog:** `modules/cabana/singleton.go` lines 41-80. Implement `pact.AdminSingleton.SingletonRecord(ctx, db) (any, error)` with first-or-create. cabana already retries on a unique-index race (`uniqueViolation(err)` → `singletonRace{}`). Absolute URLs come from `attach.PublicURL` (`lagoon/attach/thumb.go:80-96`) and are prefixed with `app.url` when no `public_url` is set.
### YAML controllers and models
**Analog:** `modules/cabana/backendscreens/controllers/users/config_form.yaml`:
```yaml
name: backend::lang.user.name
form: ~/plugins/summercms/cabana/models/user/fields.yaml
modelClass: Backend\Models\User
defaultRedirect: summercms/cabana/users
create:
redirect: summercms/cabana/users/preview/:id
redirectClose: summercms/cabana/users
update:
redirect: summercms/cabana/users/update/:id
redirectClose: summercms/cabana/users/preview/:id
preview:
headerPartial: hint
recordActions: [activate, deactivate, restore]
messages: { create: ..., update: ..., saved: ..., deleteConfirm: ..., deleted: ..., edit: ..., preview: ... }
```
The siblings `config_list.yaml`, `config_filter.yaml` and `_hint.htm` cover lists, the Customized/Outdated filters and server partials. The partial allowlist is in `partial_render.go:105-118`.
### `admin/src/components/form/registry.ts` (modified)
**Analog:** itself. Add an import block (lines 38-62) and an entry in the `renderers` map (lines 79-90+):
```ts
const renderers = new Map<string, Component>([
['text', TextField],
...
['fileupload', FileuploadField],
['datepicker', DatepickerField],
```
Add `mailpreview` to `valueless` (`new Set<string>([RELATION_MANAGER, 'widget', 'partial', 'fileupload'])`) and the group-labelled types to `groupLabelledTypes`. Extend the phase-history header comment (lines 1-35) with a Phase 35 paragraph.
### `MailPreviewField.vue` / `CodeEditorField.vue`
**Analog:** `admin/src/components/form/fields/MarkdownField.vue` lines 1-70:
```ts
import { computed, onBeforeUnmount, ref, watch } from 'vue'
import { api } from '../../../api/client'
import { controlAttributes, controlClass, type FieldControlProps } from '../control'
const props = defineProps<FieldControlProps>()
const emit = defineEmits<{ 'update:modelValue': [value: string] }>()
const PREVIEW_DEBOUNCE_MS = 300
let sequence = 0 // drops stale answers: if (current !== sequence) return
...
const { data, error } = await api.POST('/markdown/preview', { body: { markdown: text.value } })
if (data) { sanitizedHtml.value = data.data.html } else { previewError.value = refusal(error) }
```
For the preview, copy the debounce, the sequence guard and the `refusal()` details extraction. Render into a `sandbox` iframe `srcdoc`, which the CSP comment allows. Never `v-html` the mail. For CodeEditorField, copy the lazy-import pattern of `RichEditorField.vue` (TipTap loaded on demand) to load `codeeditor/editor.ts`. Field components import only from `../control`, never from `registry.ts` (registry.ts lines 64-67, to avoid an import cycle). Keep Mod+Enter reaching the form (Pitfall 5).
### Go tests
**Analog:** `modules/cabana/administrators_harness_test.go:154` `newScreensEnv(t, extra ...party.Plugin)`. It builds compass config with the admin secret, `lagoon.Publish(app, adminSQL, gdb)`, phrasebook and `surf.Assemble`. Reuse it for `TestMailPermissions`, routes and lifecycle. For postcard Postgres tests, copy the TestMain from `modules/lagoon/postgres_test.go`, which uses `flag.Parse()` and `testing.Short()` per CLAUDE.md. Smoke-test naming follows `backend_screens_smoke_test.go`.
## Shared Patterns
### Backend auth gate on custom routes
**Source:** `modules/cabana/field_markdown.go:72-79` (`bouncer.User` + `principal.Backend` + `tokenRouteGate`). Mount with `requireAjax` for POSTs (`http.go:342`).
**Apply to:** all mail_routes.go handlers, plus a per-permission check against `PermissionManageMail*`.
### Error responses
**Source:** `WriteError(w, status, code, msgConst)` with fixed `msg*` constants (`field_markdown.go:53,74`). Use `ForbiddenError` for refusals and translated params via `backendMessageParams`.
**Apply to:** routes and hooks.
### HTML safety gate
**Source:** `modules/postcard/templates.go:25-26` and `modules/cabana/field_markdown.go:16-18,37-48` (identical regexes). Reuse them unchanged for draft validation, but report their false positives to the user (Pitfall 8).
### Embedded pseudo-plugin assets
**Source:** `backend_screens.go:35-74` (`//go:embed all:backendscreens`, `fs.Sub`).
### Docs/README and generated artifacts
Update READMEs for postcard and cabana in the same commit. Then run `scripts/check-admin-openapi.sh`, `scripts/check-admin-dist.sh` and `summer docs:build --check`.
## No Analog Found
| File | Role | Data Flow | Reason |
|---|---|---|---|
| `admin/src/components/form/codeeditor/editor.ts` | utility | event-driven | CodeMirror 6 is new. Use 35-RESEARCH.md "CodeMirror setup" (verified export names). Needs a STACK.md row and a decision note before install. |
| `admin/src/components/mail/insertImage.ts` | utility | transform | No Markdown-insertion helper exists. Use the RESEARCH Pitfall 4 rules (absolute URL, percent-encoding). |
| `modules/postcard/samples.go` placeholder walker | utility | transform | No `text/template/parse` walker exists in the repo. Use the RESEARCH "Placeholder walker" prototype. |
## Metadata
**Analog search scope:** modules/postcard, modules/cabana (+ backendscreens), modules/lagoon (+ attach), modules/pact, admin/src/components/form
**Files scanned:** ~20
**Pattern extraction date:** 2026-11-06

View File

@@ -0,0 +1,705 @@
# Phase 35: Mail templates and layouts editor - Research
**Researched:** 2026-11-06
**Domain:** postcard mail rendering with a database override layer, cabana YAML admin screens, a CodeMirror 6 field in the Vue admin SPA
**Confidence:** HIGH for the codebase facts and integration points, MEDIUM for the recommended designs (they rest on verified facts but are proposals)
<user_constraints>
## User Constraints (from CONTEXT.md)
### Locked Decisions
#### Placement and permissions
- **D-01:** The screens live **inside cabana**, as an internal pseudo-plugin. This is the same pattern as Administrators/Roles in 19.6.1 (`modules/cabana/backend_screens.go`, `modules/cabana/backendscreens/`). Every app gets them with no setup. postcard stays free of admin and cabana code; cabana (or a thin glue layer) supplies postcard with the DB-backed layer through an interface postcard defines.
- **D-02:** **Navigation: a "Mail" section in Settings**, with entries for Mail templates, Mail layouts, Mail partials, Mail branding and Mail images. Links use `pact.HasSettings`, as in 19.6.1 D-02. There is no top-level nav item.
- **D-03:** **Two permission codes**, registered by the cabana pseudo-plugin:
- `system.manage_mail_templates` covers the templates list/form, template preview and test send.
- `system.manage_mail_layouts` covers layouts, partials, Branding and the image library, because these change every mail.
Neither code is granted to any role by default (as in 19.6.1 D-04). Only superusers reach the screens until someone grants a code. The image library and its insert picker must also be usable from the template form by holders of `system.manage_mail_templates`, either read/pick only or with upload too (planner's call, stated in the plan). — **Reversibility:** costly — the codes end up stored in `backend_user_roles.permissions` / `backend_users.permissions` JSON on every install that grants them.
#### Editing model: templates
- **D-04:** **A template override stores subject + Markdown body + layout.**
- The body stays Markdown with Go `html/template` variables (`{{ .name }}`), the same format as the file.
- The plain-text part is still derived from the body; there is no hand-written text part.
- The layout is a dropdown over all layouts (file and custom).
- Description stays file-owned and read-only.
- **D-05:** **Locale variants stay separate templates.** `activate.htm` and `activate-en.htm` are their own rows. The list has a locale column, parsed from the suffix convention postcard already uses, plus a locale filter. There is no grouping and no new locale model.
- **D-06:** **Reset to the file version** deletes the override row, on templates, file layouts and file partials alike.
- **D-07:** **Drift notice.** When an override is saved, it stores a hash of the file content it was based on. If the plugin file later changes (for example after a plugin upgrade), the override keeps winning, but the list and the form flag "file changed since override". The form offers a way to view the current file version (a diff if cheap) and the reset button. — **Reversibility:** costly — the hash column is part of the override table schema (migration).
#### Editing model: layouts, Branding, partials
- **D-08:** **Layouts use the same override model as templates.**
- Every file layout is listed: postcard's built-in default plus plugin layouts such as sm-user-plugin `layouts/user.htm` and paymentgateway `layouts/paymentgateway.htm`.
- Each file layout can get a DB override with reset.
- Admins can also **create custom layouts** that exist only in the DB. Custom layouts can be deleted (not reset); deleting one still assigned to template overrides is refused, with a count of those templates.
- "Seeded predefined layouts" means a couple of framework-shipped file layouts, for example the existing default plus one or two more such as system/minimal (the planner picks the set). They are not rows copied into the DB at boot.
- **D-09:** **Layouts are edited in a new `codeeditor` form field type**, Winter-style, with syntax highlighting for HTML/CSS/text. It is added to cabana's field registry and to the admin SPA. The layout form has tabs for the HTML wrapper, the text wrapper and the CSS/header. The field type is reusable by later phases. — **Reversibility:** costly — a published field type name in `fields.yaml` that plugins may start using.
- **D-10:** **Branding page**, as in Winter's mail brand settings.
- It is a settings page (cabana settings model) with Winter's brand colour set: body/content background, text, link, the button primary/positive/negative colours, and header/footer colours.
- It also has a **logo** upload, exposed to layouts as `{{ .logoUrl }}`.
- The colours compile into the brand CSS slot that postcard already inlines.
- **D-11:** **Precedence: Branding replaces config brandCss.** While no Branding record has been saved, `mail.brandCss` from config is used, so existing apps are unchanged. Once Branding is saved, its compiled CSS replaces `mail.brandCss`. `mail.css` is unaffected and still applied.
- **D-12:** **Mail partials are in scope, with file + DB override like templates.**
- Files live at `views/mail/partials/<name>.htm` and are named `<plugin>::mail.partials.<name>`.
- Plugins declare them through a **new additive optional pact capability**. `pact.HasMailTemplates` is not changed: core plugins must not break.
- Partials are used from template bodies and layouts through a Go template call (the planner picks the exact syntax or helper).
- A few built-in partials ship with postcard (button, panel, table).
- Admins override and reset partials; editing them needs `system.manage_mail_layouts`.
— **Reversibility:** costly — a published pact capability and naming convention that plugins implement.
#### Validation and send-time safety
- **D-13:** **Saves are validated.** Saving a template, layout or partial override parses it, test-renders it with the sample vars (D-15) and runs postcard's existing safety checks (script/iframe/object/embed, inline handlers, `javascript:`/`vbscript:`/`data:` URLs, CR/LF in the subject). Failures are field validation errors.
- **D-14:** **Fallback at send.** If an override still fails to render at send time (for example a variable missing at runtime), postcard logs the failure and sends the file version, so mail delivery doesn't stop. A template with no file (impossible for templates; custom layouts only) falls back to the default layout.
#### Preview and test send
- **D-15:** **Sample variables**: plugins may add optional sample values to a template file. One option is a sample section in the INI header; another is an additive capability (planner's call). Variables without a sample render as a visible placeholder such as `[name]`. Existing template files work unchanged.
- **D-16:** **Live preview of the unsaved draft.** The template, layout and partial forms render the current draft through postcard with sample vars. The HTML part is shown in a **sandboxed iframe**, and a tab shows the plain-text part. Rendering happens on the server through postcard, so preview matches real sends.
- **D-17:** **Test send to my own address only.** A button sends the current draft (with sample vars) to the signed-in admin's email through the configured driver. It needs the same permission as editing that record and is rate-limited per admin. There is no free-text recipient field.
#### Images
- **D-18:** **A shared mail image library.** A "Mail images" list stores images through `lagoon/attach` on the public disk. Any template, layout or partial can use them, and the Branding logo also uses this storage.
- **D-19:** **Delivery via absolute public URLs.** `<img src>` uses the attachment's absolute public URL, built from `storage.uploads.public_url` / `app.url`. No CID inline parts; postcard's `RenderedMessage` and drivers are unchanged.
- **D-20:** **An insert button in the editors.** A picker next to the markdown editor (template bodies) and the codeeditor (layouts/partials) inserts `![alt](url)` or `<img src="url" alt="">` for a chosen library image.
#### Quality gates
- **D-21:** Unit tests are the last plan (project rule). They cover:
- override precedence and reset;
- drift detection;
- fallback at send;
- save validation;
- Branding precedence over config;
- the partial capability (boot errors, namespace rule);
- the permission split;
- the test-send recipient restriction and rate limit.
The postcard and cabana READMEs and the affected `docs/` pages are updated in the same commits as API/config/permission changes. The `admin/` changes regenerate `admin/openapi/admin.json` + `schema.d.ts` (`scripts/check-admin-openapi.sh`) and rebuild the committed `modules/boardwalk/dist` (`scripts/check-admin-dist.sh`).
### Claude's Discretion
- Table names and schema (one overrides table per kind, or a shared one), as long as it is Postgres/GORM with gormigrate migrations owned by the cabana pseudo-plugin, or by postcard if the planner justifies it.
- Exact partial call syntax and the built-in partial set beyond button/panel/table.
- Which extra predefined file layouts ship (D-08).
- Whether the image picker on the template form needs `manage_mail_layouts` for uploads (D-03 note).
- How postcard learns about overrides: a lookup interface checked per send, or a cache invalidated on save. It must be correct with several app instances, so per-send lookup or explicit invalidation, never a stale boot-time snapshot.
- How the SPA codeeditor is built (the library choice needs a STACK.md/decision note per the dependency rule).
- The plan split (lean planning: present the count before writing PLAN.md files).
### Deferred Ideas (OUT OF SCOPE)
- Hand-written plain-text template parts (Winter's `content_text`), not needed now.
- A WYSIWYG/rich-editor mail body.
- CID-embedded inline images (only absolute URLs in v1).
- Test send to an arbitrary address.
- Mail driver/SMTP configuration screen in the admin.
#### Reviewed Todos (not folded)
- **Port MailBlocker and the block-mail checkbox** (`.planning/todos/pending/mailblocker-with-mailing.md`): kept separate. It is a send-path feature owned by sm-user-plugin (MailBlocker table, check before send, `block_mail` checkbox) and stays a todo for later mail work.
</user_constraints>
<phase_requirements>
## Phase Requirements
No REQUIREMENTS.md IDs are mapped to this phase (ROADMAP: "Requirements: TBD"). The related shipped requirement is **I18N-03** ("Plugins register mail templates and layouts by dotted name with the per-locale suffix convention, rendered with html/template and sent through a driver interface (SMTP via go-mail)"), which this phase must keep satisfied. The table maps the locked decisions to the research findings that enable them.
| ID | Decision | Research Support |
|----|----------|------------------|
| D-01 | Screens inside cabana; postcard defines the interface | Pattern 1 (store split), Pattern 2 (index rows); cabana pseudo-plugin facts |
| D-02 | Settings → Mail entries | `pact.SettingsItem.Controller` link entries; UI-SPEC C14 |
| D-03 | Two permission codes, no default grants | `pact.Permission` with empty `Roles`; `cabana.Allows` any-of semantics for the picker |
| D-04 | Override = subject + body + layout | Schema; FormBeforeUpdate/FormAfterUpdate ordering (Pitfall 6) |
| D-05 | Locale column/filter | Locale parsing rule (Pattern 2) |
| D-06 | Reset | Record action `reset` that clears the override columns (Open Question 1) |
| D-07 | Drift hash | `file_hash` vs `base_hash`; stdlib LCS diff (Don't Hand-Roll note) |
| D-08 | File layouts + custom layouts + predefined layouts | Index rows; built-in alias fallback; FK RESTRICT for in-use refusal |
| D-09 | `codeeditor` field type | Built-in type list changes; CodeMirror 6 packages, keymap pitfall |
| D-10/D-11 | Branding page and precedence | Settings model facts; brand CSS compiled by postcard; per-send brand lookup |
| D-12 | Partials capability | `pact.HasMailPartials` design; placeholder substitution (verified goldmark behaviour) |
| D-13 | Save validation | postcard draft-validation API; parameterized messages (Pitfall 9) |
| D-14 | Fallback at send | Per-send lookup + render retry without overrides |
| D-15 | Sample vars | INI `sample.*` keys (backward compatible) + verified placeholder walker |
| D-16 | Live preview | Preview route; srcdoc sandbox is allowed by the admin CSP |
| D-17 | Test send to self, rate-limited | Principal has no email; limiter reachability (Pattern 7) |
| D-18/D-19/D-20 | Image library, absolute URLs, insert button | attach URL facts; worker-process URL pitfall (Pitfall 4) |
| D-21 | Tests and gates | Validation Architecture |
</phase_requirements>
## Project Constraints (from CLAUDE.md)
- Standard library first. A new dependency needs a research doc or a phase/decision note naming it (`.planning/research/STACK.md`). The CodeMirror packages need a STACK.md "Admin SPA (npm) additions" row and a decision note written **before** the plan task that installs them.
- YAML is parsed with `goccy/go-yaml`, never `gopkg.in/yaml.v3`.
- Postgres only; GORM plus per-plugin gormigrate migrations. AutoMigrate is not the schema source.
- No runtime plugin loading.
- A change to a module's exported API, config keys, CLI commands or dependencies updates that module's `README.md` and the affected `docs/` pages **in the same commit**. Every identifier in a README or docs page must exist in the package (docs checker). Config keys are reviewed by hand.
- Framework code never names a consuming application (READMEs, docs and examples use neutral names such as `blog` or `acme`).
- A new module needs a beach name, a README in the standard structure and a root README row. (This phase should not need a new module; see Pattern 1.)
- Before tagging a release that changes `admin/`: rebuild, commit `modules/boardwalk/dist`, pass `scripts/check-admin-dist.sh`.
- `TestMain` functions that start containers call `flag.Parse()` and check `testing.Short()`.
- Keep `go vet ./...` and `go test ./...` green at every commit.
- Lean planning: few large plans; present the plan count with a one-line scope each and wait for confirmation. **Unit tests are the last plan.**
- Update STATE.md only through `msd_run query state.*` handlers. g15office sync on phase advance.
- No co-author tags. One logical change per commit; planning docs and code in separate commits. Stage files by path; several sessions work in this repo (Phase 33 is executing and recently touched cabana: `4cd35c9 feat(cabana): per-request options for plugin field types`).
- No push, tag or deploy without the user's go-ahead.
- Core plugins (user, blog, pages, payment) must not get breaking changes: `pact.HasMailTemplates`, `postcard.Catalog.Register` and `postcard.Mailer` keep their exact signatures.
## Summary
postcard today is an in-memory catalog: `Catalog.Register` reads each plugin's `views/mail/*.htm` at boot, and `Mailer.Send` renders a template (Markdown body through `html/template`, then goldmark) inside a layout, puts `mail.brandCss` + `mail.css` into the layout's `<style>` block and hands the result to a driver. There is no database layer, no partial support, no exported way to read the catalog or render a draft, and the published `Mailer` is an unexported `*mailer`. The phase adds all of these.
Three verified facts drive the design. First, the admin is GORM-row-based: list search/sort/filter are SQL over model columns (`query.go`), record IDs are numeric (`pathID` → `uint`), and dropdown/relation options come from rows. File templates therefore need **index rows** (Winter's `MailTemplate::syncAll()` approach) to be listed and opened at all. Second, `cabana.Activate` only runs when `serve` assembles the router, while mail is also sent from `queue:work` (a separate process when `queue.work_in_serve` is false). An override reader supplied by cabana would be absent there, so **postcard should own the read path and the schema**, wired in `postcard.Activate`, with the migrations declared in `lagoon.migrationSets` exactly like `attach.Migrations`. cabana writes the same tables through its admin models. Third, goldmark (default config, as postcard uses it) replaces raw HTML with `<!-- raw HTML omitted -->`, so a partial that emits HTML into a Markdown body would disappear. Partials in bodies need **placeholder substitution after Markdown conversion**.
The rest is mostly wiring through existing seams: Settings link entries and permissions as in 19.6.1, record actions for reset, `ForbiddenError` for the in-use delete refusal, settings model for Branding, `AdminSingleton` + `fileupload` for the image library, and new built-in field types. The UI-SPEC already fixes the SPA contract. On the SPA side the main risk is CodeMirror's `defaultKeymap` binding `Mod-Enter`, which conflicts with the form's save hotkey.
**Primary recommendation:** postcard owns the mail tables (lagoon-declared migration set `summercms.postcard`), a per-send `OverrideSource` lookup with full-file fallback, the partial capability with post-Markdown placeholder substitution, INI `sample.*` keys plus a parse-tree placeholder walker, brand CSS compilation, and an exported editor API (catalog read, draft validate/render/send). cabana adds the screens over those tables, syncing index rows from the catalog at serve start. The SPA adds CodeMirror 6 (six exact-pinned packages plus `@lezer/highlight`), lazily loaded.
## Architectural Responsibility Map
The tiers here are: the postcard module (mail domain), lagoon (migration registry), the cabana module (admin API), the Vue admin SPA (browser), and Postgres.
| Capability | Primary Tier | Secondary Tier | Rationale |
|------------|-------------|----------------|-----------|
| File catalog (templates, layouts, partials, samples, hashes) | postcard | — | Already parses files at boot; one format for files and overrides |
| Override tables and migrations | postcard (DDL), declared in lagoon `migrationSets` | Postgres | Read path must exist in every process; `attach.Migrations` precedent |
| Send-time override lookup + fallback (D-11, D-14) | postcard | Postgres | Runs in serve, `queue:work` and console commands; cabana is absent from the latter two |
| Brand CSS compilation | postcard (pure function) | cabana (calls it for preview) | Send path and preview must compile identically |
| Draft validation / preview render / test send | postcard (exported editor API) | cabana (HTTP routes, permission, rate limit) | Preview must match real sends (D-16) |
| Index row sync (file → rows) | postcard (function) | cabana (calls it at serve start) | Only the admin needs rows; the catalog is immutable after boot |
| Mail screens, permissions, settings entries, record actions | cabana pseudo-plugin | — | D-01, 19.6.1 pattern |
| Image library storage and absolute URLs | lagoon/attach | cabana (library endpoints), postcard (logo URL) | Public disk; worker processes have no bucket open (Pitfall 4) |
| Code editor, colour picker, preview pane, picker | Admin SPA | cabana (schema + routes) | UI-SPEC C1-C15 |
| Partial capability contract | pact | postcard (`BootPlugin`) | Additive interface; `HasMailTemplates` unchanged |
## Standard Stack
### Core (all already in the module graph; no new Go dependency)
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| `html/template`, `text/template`, `text/template/parse` | Go 1.27.2 stdlib | Rendering; parse-tree walk for sample placeholders | Already postcard's renderer [VERIFIED: modules/postcard/templates.go:7,12] |
| `crypto/sha256` | stdlib | D-07 file hash | Stdlib first |
| `github.com/yuin/goldmark` | v1.8.6 | Markdown body → HTML | Already pinned [VERIFIED: go.mod `github.com/yuin/goldmark v1.8.6`] |
| `gorm.io/gorm`, `github.com/go-gormigrate/gormigrate/v2` | as pinned | Override store, migrations | Already imported by `pact` [VERIFIED: `go list` imports of modules/pact] |
| `lagoon/attach` | in-repo | Image library on the public disk | D-18; imports only backpack/compass/diary, so postcard can import it without a cycle [VERIFIED: `go list` of modules/lagoon/attach] |
| `cooler` | in-repo | Shared fixed-window counter type (`cooler.Attempter`) for the test-send limit | Same semantics as surf's limiter, Redis-backed when configured [VERIFIED: modules/cooler/cooler.go:68-73] |
### Admin SPA (new npm runtime dependencies)
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| `@codemirror/state` | 6.7.6 (2026-09-22) | Editor state | `codeeditor` field |
| `@codemirror/view` | 6.43.15 (2026-10-19) | EditorView, keymap, lineNumbers, MatchDecorator, ViewPlugin | `codeeditor` field |
| `@codemirror/language` | 6.13.1 (2026-10-07) | bracketMatching, HighlightStyle, syntaxHighlighting | `codeeditor` field |
| `@codemirror/commands` | 6.11.1 (2026-09-15) | history, historyKeymap, defaultKeymap | `codeeditor` field |
| `@codemirror/lang-html` | 6.4.12 (2026-08-04) | `html()` grammar | `language: html` |
| `@codemirror/lang-css` | 6.3.1 (2024-11-26) | `css()` grammar | `language: css` |
| `@lezer/highlight` | 1.2.5 | `tags` for `HighlightStyle.define` | Imported directly by the theme, so pin it as a direct dependency (UI-SPEC lists it only as transitive) |
Versions, licenses and dates: [VERIFIED: npm registry, `npm view <pkg> version license repository.url` and `npm view <pkg> time`, 2026-11-06]. Exports `MatchDecorator`, `ViewPlugin`, `lineNumbers`, `lineWrapping`, `contentAttributes`, `keymap` (view); `bracketMatching`, `HighlightStyle`, `syntaxHighlighting` (language); `defaultKeymap`, `historyKeymap`, `history`, `indentWithTab`, `standardKeymap` (commands); `html` / `css`; `tags` (@lezer/highlight). [VERIFIED: `npm pack` of each pinned version, grep of `package/dist/index.d.ts`]
### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| CodeMirror 6 modular packages | `codemirror` meta package (`basicSetup`) | Pulls search, lint and autocomplete UI; UI-SPEC forbids it |
| CodeMirror 6 | Monaco / Ace / textarea overlay | Rejected in UI-SPEC (size and workers / accessibility / caret sync) |
| Stdlib LCS line diff (D-07) | `github.com/aymanbagabas/go-udiff` v0.4.1 | Only a test-time transitive dependency today (via charm `x/exp/golden`) [VERIFIED: `go mod why -m`]; using it directly is a new dependency needing a note. Inputs are small, so a bounded stdlib LCS is enough |
| postcard-owned read path | cabana-supplied `OverrideSource` | Missing in `queue:work` processes (Pattern 1) |
**Installation (SPA, after the decision note and STACK.md row are committed):**
```bash
npm --prefix admin install --save-exact @codemirror/state@6.7.6 @codemirror/view@6.43.15 @codemirror/language@6.13.1 @codemirror/commands@6.11.1 @codemirror/lang-html@6.4.12 @codemirror/lang-css@6.3.1 @lezer/highlight@1.2.5
```
## Package Legitimacy Audit
Run with `msd_run query package-legitimacy check --ecosystem npm …` on 2026-11-06. The package names come from the official CodeMirror documentation (codemirror.net examples import `@codemirror/view`, `@codemirror/commands`, `@codemirror/lang-*`) and the registry metadata. No package has a `postinstall` script (`npm view <pkg>@<ver> scripts.postinstall` empty for all).
| Package | Registry | Age | Downloads | Source Repo | Verdict | Disposition |
|---------|----------|-----|-----------|-------------|---------|-------------|
| @codemirror/state 6.7.6 | npm | 45 days | 22.6M/wk | code.haverbeke.berlin/codemirror/state | [OK] | Approved |
| @codemirror/view 6.43.15 | npm | 18 days | 23.1M/wk | code.haverbeke.berlin/codemirror/view | [SUS] (reason: `too-new`) | Flagged: planner adds a checkpoint |
| @codemirror/language 6.13.1 | npm | 30 days | 22.2M/wk | code.haverbeke.berlin/codemirror/language | [OK] | Approved |
| @codemirror/commands 6.11.1 | npm | 52 days | 21.2M/wk | code.haverbeke.berlin/codemirror/commands | [OK] | Approved |
| @codemirror/lang-html 6.4.12 | npm | 94 days | 12.0M/wk | code.haverbeke.berlin/codemirror/lang-html | [OK] | Approved |
| @codemirror/lang-css 6.3.1 | npm | ~2 yrs | 12.0M/wk | github.com/codemirror/lang-css | [OK] | Approved |
| @lezer/highlight 1.2.5 | npm | — | — | (maintainer marijn) | [OK] | Approved |
| @marijn/find-cluster-break (transitive of state) | npm | 77 days | 20.9M/wk | code.haverbeke.berlin/marijn/find-cluster-break | [OK] | Transitive only; not listed in the UI-SPEC's transitive list, so add it to the decision note |
Maintainer of every package: `marijn <marijn@haverbeke.berlin>` (lang-css also `adrianheine`). [VERIFIED: `npm view … maintainers`]
**Packages removed due to [SLOP] verdict:** none
**Packages flagged as suspicious [SUS]:** `@codemirror/view` 6.43.15, flagged only as "too-new" by the seam's threshold. It is past the project's 7-day cooldown (published 2026-10-19). The planner inserts `checkpoint:human-verify` before the install task: confirm the version (or pin 6.43.14 or older), and record the lockfile `dist.integrity` values in the decision note as the DOMPurify note did.
## Architecture Patterns
### System Architecture Diagram
```
┌──────────────── every process (serve, queue:work, console) ───────────────┐
plugin files ──boot──▶ postcard.Catalog (templates, layouts, partials, samples, sha256 per file) │
views/mail/**.htm │ │ │
+ postcard built-ins │ ▼ │
│ Mailer.Send(msg) ──▶ OverrideSource.MailOverrides(ctx, code) ──▶ Postgres │
│ │ (per send; DB from app.Lookup[*gorm.DB]) system_mail_*
│ ▼ │
│ render(file ⊕ overrides, brand) ──ok──▶ validate ──▶ Driver.Send │
│ │ error │
│ └──▶ log + render(file only, brand) ──▶ Driver.Send (D-14) │
└────────────────────────────────────────────────────────────────────────────┘
serve only:
serve start ──▶ lagoon.OnDatabase ──▶ postcard sync(catalog → index rows, ON CONFLICT upsert)
Admin SPA ──HTTP──▶ cabana generic CRUD (lists, forms, settings, singleton, record actions)
│ │ hooks: FormBeforeUpdate (stamp) / FormAfterUpdate (validate via postcard)
│ ▼
│ system_mail_templates / _layouts / _partials / _brand_settings / image library
├──POST /mail/preview ─────▶ cabana (permission per mode) ──▶ postcard editor: render draft ⊕ DB
├──POST /mail/test-send ───▶ cabana (permission, rate limit, admin email from backend_users)
│ └──▶ postcard editor: render draft ──▶ Driver.Send(to = self)
└──GET/POST /mail/images ──▶ cabana (either permission to list; layouts permission to upload)
└──▶ attach.Store (public) ──▶ absolute URL
```
### Recommended Project Structure
```
modules/postcard/
├── templates.go # existing: catalog, parser (+ partials, samples, hashes, built-in alias fallback)
├── mailer.go # existing: Send gains the override lookup and fallback; Activate wires the DB source
├── partials.go # new: partial func, placeholder substitution
├── samples.go # new: sample.* keys + parse-tree placeholder walker
├── brand.go # new: Brand struct, colour validation, CompileBrandCSS
├── store.go # new: Migrations, table constants, DB-backed OverrideSource, index sync
├── editor.go # new: exported catalog read + draft Validate/Preview/SendDraft
└── assets/ # default.htm (unchanged) + system.htm, minimal.htm, partials/{button,panel,table,subcopy}.htm, brand.css.tmpl, sample.htm
modules/pact/capabilities.go # + HasMailPartials, + RowStateCustomized/RowStateOutdated
modules/lagoon/migrations.go # + {"summercms.postcard", "mail", postcard.Migrations}
modules/cabana/
├── backend_screens.go # + mail controllers, permissions, settings items
├── mail_admin.go # new: models over postcard tables, controllers, hooks, record actions
├── mail_routes.go # new: preview, test send, image library endpoints (swag-annotated)
├── field_codeeditor.go … # new built-in types: codeeditor, colorpicker, mailimage, mailpreview
└── backendscreens/controllers/mail{templates,layouts,partials,images}/, models/mail*/
admin/src/components/form/fields/{CodeEditor,ColorPicker,MailImage,MailPreview}Field.vue
admin/src/components/form/codeeditor/editor.ts, admin/src/components/mail/{MailImagePicker.vue,insertImage.ts}
```
### Pattern 1: postcard owns the schema and the read path; cabana owns the screens
**What:** postcard defines `OverrideSource` (D-01's "interface postcard defines") **and** ships the default DB-backed implementation, installed by `postcard.Activate`. It reads `*gorm.DB` lazily per send through `app.Lookup[*gorm.DB]()`. The migrations are `postcard.Migrations`, appended to lagoon's framework sets. cabana defines its own GORM admin models over the same tables (by exported table-name constants), so postcard carries no admin code (no Fillable/Rules/YAML).
**Why this deviates from "cabana supplies the layer":** the evidence says a cabana-supplied source would not exist in every process that sends mail.
- `cabana.Activate` is called only from surf's router assembly: `admin, err := cabana.Activate(app, plugins)` [VERIFIED: modules/surf/router.go:680]. The pseudo-plugin is appended there, not in `party.Activate`: `plugins, screens := withBackendScreens(app, plugins)` [VERIFIED: modules/cabana/http.go:91].
- `queue:work` opens and publishes the DB itself (`sqlDB, gdb, err := lagoon.OpenFromApp(ctx, app)` … `if err := lagoon.Publish(app, sqlDB, gdb); err != nil {`) [VERIFIED: modules/conga/commands.go:402-412], and serve runs that worker in-process "unless queue.work_in_serve is false" [VERIFIED: modules/conga/commands.go:47-48]. Plugins send mail from River jobs.
- `postcard.Activate` runs in every process from `party.Activate`: `if err := postcard.Activate(app, ordered); err != nil {` [VERIFIED: modules/party/registry.go:132].
- The 19.6.1 pseudo-plugin "declares no migrations" and the backend tables live in lagoon: `{"summercms.cabana", "backend admin", BackendAdminMigrations},` [VERIFIED: modules/cabana/backend_screens.go:30-33; modules/lagoon/migrations.go:227].
- Module-owned framework set precedent: `{"summercms.attach", "system_files", attach.Migrations},` [VERIFIED: modules/lagoon/migrations.go:226]. lagoon importing postcard is cycle-free (postcard imports backpack, compass, diary, pact; lagoon already reaches postcard through party) [VERIFIED: `go list` imports].
**Rejected alternative:** register a cabana source from `cabana.RuntimeCommands(app)` (the only cabana call the generated `main.go` makes in every process) or from a package `init()` registry. Both rely on side effects of an unrelated call or of an import.
### Pattern 2: index rows synced from the catalog (Winter's `syncAll`)
**What:** every catalog entry has one row (`code` UNIQUE). For a non-customized row, the override columns mirror the file so that the generic list (search/sort), form (shows the file version) and relation dropdown work unchanged. postcard's send path reads only rows flagged customized; the "File version" tab and drift compare against the in-memory catalog, never the mirror.
**Why rows are needed:** the generic list query is `q := db.WithContext(ctx).Model(model)` with SQL search over model columns [VERIFIED: modules/cabana/query.go:112, 459-499], and record IDs must be integers (`n, err := strconv.ParseUint(raw, 10, 64)` … `"The id field must be an integer."`) [VERIFIED: modules/cabana/crud.go:1278-1284]. Winter does the same: `MailTemplates::index()` calls `MailTemplate::syncAll();` and non-custom rows are filled from the file in `afterFetch` (`if (!$this->is_custom) { $this->fillFromView($this->code); }`) [VERIFIED: examples/golem15-wintercms-starter/modules/system/controllers/MailTemplates.php:54; modules/system/models/MailTemplate.php:130-135].
**When to sync:** the catalog is immutable after boot, so once per serve start is enough. Register it through `lagoon.OnDatabase` from `cabana.Activate` ("runs fn … as soon as the database is available") [VERIFIED: modules/lagoon/ondatabase.go:19-25]. Use `INSERT … ON CONFLICT (code) DO UPDATE … WHERE NOT is_custom` for the mirror columns, and always refresh `file_hash`, `source` and `description`. If the tables are missing (migrations pending), log a warning and retry on the first list request. Never delete rows (a rolling deploy would churn IDs); hide entries absent from this process's catalog with `ListExtendQuery` (`code IN ?` OR database-only layouts).
**Locale column:** parse the last dotted segment's `-<suffix>`, and count it as a locale only when the suffix is one of the app's enabled locales (`activate-en` → `en`; `two_factor_code-en` → `en`; `canceledNotActive` → default). This avoids reading arbitrary dashed names as locales.
### Pattern 3: per-send override lookup with a full-file fallback (D-11, D-14)
**What:** `Send` asks the source once per message for everything the render needs, then renders. On any render or validation error with overrides applied, it logs (component `postcard`) and renders the pure file version, still applying Branding. If the DB is unavailable or a table is missing, it logs a warning (rate-limited) and renders the files. This is correct across instances by construction: there is no cache to invalidate. Mail volume is low, and the cost is three or four indexed reads in one read-only transaction.
```go
// Recommended shape (design, not existing code).
type OverrideSource interface {
// MailOverrides returns what one send of template needs: the template's
// override, its effective layout's override or database-only content,
// every partial override, and the saved Branding. A nil field means the
// file (or, for Brand, config mail.brandCss) wins.
MailOverrides(ctx context.Context, template string) (Overrides, error)
}
type Overrides struct {
Template *TemplateOverride // Subject, Body, Layout (full name or database-only code)
Layout *LayoutOverride // HTML, Text, CSS; set for a customized file layout or a database-only layout
Partials map[string]PartialOverride // full name -> HTML, Text
Brand *Brand // nil until the Branding page has been saved (D-11)
}
```
Keep `postcard.Mailer` unchanged, because outside code implements it: `func (r *mailRecorder) Send(ctx context.Context, msg postcard.Message) error` [VERIFIED: ../sm-paymentgateway-plugin/mail_smoke_test.go:56], `Send(ctx context.Context, msg postcard.Message) error` in an interface [VERIFIED: ../sm-paymentgateway-plugin/events/mailers.go:116], plus fakes in sm-user-plugin and fonoteka tests. Expose the new capability through additive API instead: an `Options.Overrides OverrideSource` field for `NewMailer` (tests), and a separately published editor service from `Activate`. `BootPlugin` already type-asserts the unexported mailer (`m, ok := got.(*mailer)`) [VERIFIED: modules/postcard/mailer.go:180]; the editor service can be reached the same way or published as its own concrete type. `backpack` refuses a second provider for the same type ("Publishing the same T twice is an error.") [VERIFIED: modules/backpack/services.go:20-21,35-36].
Caller subject precedence stays as it is: `if strings.TrimSpace(msg.Subject) != "" { subjectSrc = msg.Subject }` [VERIFIED: modules/postcard/templates.go:332-335]. No production caller sets `Message.Subject` [VERIFIED: grep of sm-*-plugin and fonoteka.go `postcard.Message{` literals], so an admin's subject override is effective everywhere today. Document the precedence in the README.
### Pattern 4: partials (D-12)
**Capability (additive):**
```go
// HasMailPartials is implemented by plugins that ship mail partials under
// views/mail/partials/<name>.htm, named <plugin>::mail.partials.<name>.
type HasMailPartials interface {
MailPartialsFS() fs.FS
MailPartials() []string
}
```
`BootPlugin` checks it next to `pact.HasMailTemplates`, whose definition stays byte-identical:
```go
type HasMailTemplates interface {
MailTemplatesFS() fs.FS
MailTemplates() []string
MailLayouts() map[string]string
}
```
[VERIFIED: modules/pact/capabilities.go:1271-1275]. The existing name→path mapping already gives the right file: `return assetRoot + strings.ReplaceAll(after, ".", "/") + ".htm", nil` with `assetRoot = "views/mail/"` [VERIFIED: modules/postcard/templates.go:21,216], so `x::mail.partials.button` → `views/mail/partials/button.htm`. Reuse `assertOwner` and the duplicate/missing errors for the same boot failures. The partial file format is the layout format (header `name =`, `==`, text section, `==`, HTML section), which is also Winter's (`partial-button.php`: `name = "Button"` / `==` / text / `==` / HTML) [VERIFIED: examples/golem15-wintercms-starter/modules/system/views/mail/partial-button.php].
**Call syntax (recommended):** `{{ partial "button" "url" .Link "body" "Activate" "type" "positive" }}`, a func `partial(name string, kv ...any)` taking key/value pairs. A bare name resolves to postcard's built-ins (`postcard::mail.partials.button`); plugin partials use the full name. Unknown name or odd kv count is a render error (D-13 maps it to `mail.invalid_partial`).
**Why placeholders:** postcard renders bodies as `execHTML` then `markdown.Convert` with `markdown = goldmark.New()` [VERIFIED: modules/postcard/templates.go:29,343-347]. Under goldmark v1.8.6 defaults, raw HTML is dropped: `"Hi\n\n<table>…</table>\n\nBye"` → `"<p>Hi</p>\n<!-- raw HTML omitted -->\n<p>Bye</p>\n"`, and a `template.HTML` returned by a template func inside a body comes out as `<!-- raw HTML omitted -->button<!-- raw HTML omitted -->` [VERIFIED: scratch program against goldmark v1.8.6 from the module cache, 2026-11-06]. Turning on `html.WithUnsafe` would change how every existing body renders, so don't.
**Flow:** in the body pass, `partial` renders the partial's text and HTML sections (html/template for HTML, text/template for text, with the kv map as data) and returns a token `MAILPARTIAL<nonce><n>`, using only `[A-Z0-9]` so neither html/template escaping nor goldmark emphasis touches it. Alphanumeric tokens survive goldmark (`"<p>SUMMERPARTIAL0X9F</p>"`) [VERIFIED: same scratch run]. After conversion, replace `<p>TOKEN</p>` and then bare `TOKEN` with the partial HTML in the HTML part, and the token with the partial text in the text part (the text part is the pre-Markdown body). Use a fresh random nonce per render so variable values cannot forge tokens. In layouts (pure html/template, no Markdown) `partial` returns `template.HTML` directly in the HTML wrapper and text in the text wrapper.
**Built-in set (recommended):** `button` (url, body, type primary/positive/negative), `panel` (body), `table` (body, a Markdown table rendered with goldmark plus `extension.Table` inside the partial helper only), `subcopy` (body). These are the colours the Branding table covers (`button_*`, `panel_bg`, `table_border_color`).
**Predefined layouts (D-08, recommended):** keep `default.htm` byte-identical and add `system` (Winter's "System layout" with its "automatic message" footer) and `minimal`. Make them reachable from plugin templates by a built-in alias fallback in `resolveLayoutLocked`: plugin aliases first, then built-in aliases. This only turns today's `unknown layout` boot errors into successes.
### Pattern 5: sample variables (D-15)
**Recommended mechanism: dotted INI keys in the existing header**, for example `sample.Name = "Ada"` and `sample.order.code = "ZAM-1"`. Old framework versions ignore them. Today's `parseINI` accepts any `key = value` line and `parseTemplateFile` reads only `subject`, `description` and `layout` [VERIFIED: modules/postcard/templates.go:228-235, 291-315]. A `[sample]` INI section would instead fail in every released postcard (`return nil, fmt.Errorf("invalid header line %q", line)`) [VERIFIED: templates.go:298-300].
**Case:** `parseINI` lower-cases keys (`key = strings.ToLower(strings.TrimSpace(key))`) [VERIFIED: templates.go:302], but template variables are case-sensitive (`{{ .Name }}`, `.bankDetailsLines` in consumer files). Keep the original case after the `sample.` prefix.
**Placeholders:** `missingkey=zero` renders a missing top-level key as empty (`[{{ .missing }}]` → `"[]"`), and a missing nested key fails: `executing "d" at <.order.code>: nil pointer evaluating interface {}.code` [VERIFIED: scratch run, Go 1.27.2]. Consumer templates use nested fields, `if .employee.first_name`, `range .order.items` with element fields and `range .banking_details.bankDetailsLines` printing `{{ . }}` [VERIFIED: grep of ../sm-paymentgateway-plugin/views/mail]. So preview and save validation need a parse-tree walker over `html/template`'s `Tree`. It turns every root field chain into nested maps with `"[a.b]"` leaves (or the sample), and every `range` target into a one-element slice built from the fields used inside the range body (a string placeholder when the body prints `{{ . }}`). The prototype in Code Examples renders the real paymentgateway shapes without error.
### Pattern 6: Branding (D-10, D-11)
- Settings forms are per-model singleton tables at primary key 1 (`Where(clause.Eq{… Value: 1}).Take(model)`), and the first Put creates the row (`if !exists { … tx.Create(model) }`) [VERIFIED: modules/cabana/settings.go:186, 236-277]. D-11's "saved" therefore means "row 1 exists". The image-library host must not be this row, or opening Mail images would silently switch apps off `mail.brandCss`.
- Settings forms refuse plugin field types (`decodeFields(raw, types.refusing())`) and these built-ins: `if field.Type == "widget" || field.Type == "partial" || field.Type == "fileupload" || field.Type == permissionFieldType {` [VERIFIED: modules/cabana/settings.go:96,102]. `colorpicker`, `mailimage` and `mailpreview` must be **built-in** cabana types, and `colorpicker`/`mailimage` must be added to `scalarFormField` so they bind to columns. That list is currently `"text", "textarea", "number", "checkbox", "switch", "dropdown", "datepicker", "markdown", "mltext", "mltextarea", "mlmarkdown", "richeditor", moneyType` [VERIFIED: modules/cabana/crud.go:1518-1525].
- The brand CSS is compiled by postcard from a CSS text/template (a port of Winter's `custom.less`, 304 lines [VERIFIED: wc of mailbrandsetting/custom.less]) after strict `^#[0-9a-f]{6}$` validation of each colour. Postcard puts CSS in the layout's `<style>` block (`<style type="text/css">{{ .brandCss }}{{ .css }}</style>`), it does not inline it into attributes [VERIFIED: modules/postcard/assets/default.htm:9]. The sm-user-plugin and paymentgateway layouts only carry `.layout-user` / `.layout-paymentgateway` wrappers [VERIFIED: their `views/mail/layouts/*.htm`]. So the compiled CSS must style generic elements (`body`, `a`, `h1`–`h3`, `p`, `table`) plus the built-in partial classes, otherwise Branding seems to do nothing in plugin layouts.
- Layout data today is exactly `"Content"`, `"Subject"`, `"css"`, `"brandCss"` [VERIFIED: modules/postcard/templates.go:358-372]. Add `"logoUrl"` (string; html/template sanitizes it in `src` attributes).
### Pattern 7: preview, test send, permissions, rate limit (D-16, D-17, D-03)
- Mount the routes like `/markdown/preview` (`g.Post("/markdown/preview", requireAjax(s.markdownPreview))`) [VERIFIED: modules/cabana/http.go:342], with swag annotations. Check permission per `mode`: template → `system.manage_mail_templates`; layout/partial/branding → `system.manage_mail_layouts`. Use `cabana.Allows`, which is any-of with a superuser bypass (`if principal.IsSuperuser || len(required) == 0 { return true }`) [VERIFIED: modules/cabana/contracts.go:327-343]. The image-library list calls `Allows(p, []string{templates, layouts})`; upload calls `Allows(p, []string{layouts})` (UI-SPEC recommendation: `canUpload` only for the layouts code).
- **Test-send recipient:** `bouncer.Principal` has `ID`, `IsSuperuser`, `PermissionGrants`, `Scope` but no email [VERIFIED: modules/bouncer/context.go:26-35]. Load `BackendUser.Email` by `principal.ID` [VERIFIED: modules/cabana/contracts.go:60-65] and never accept a recipient in the body. Restrict to interactive sessions (an `interactiveOnly` wrapper exists for account routes [VERIFIED: modules/cabana/http.go:333]).
- **Rate limit:** cabana cannot import surf, because surf imports cabana [VERIFIED: `go list` imports of modules/surf]. surf publishes its Redis-backed limiter only as `app.Publish[Store](…)` [VERIFIED: modules/surf/serve.go:237]. `cooler.Attempter` has "exactly the method set of surf.Store" [VERIFIED: modules/cooler/cooler.go:68-73]. Recommended: surf also publishes the resolved store as `cooler.Attempter` (one line where it publishes `Store`), and cabana looks it up with a small in-process fixed-window fallback (tests, memory store). Key `mail_test:<adminID>`; answer 429 with `Retry-After` as surf does.
- **Preview iframe:** the admin CSP is `"frame-ancestors 'none'; base-uri 'none'; object-src 'none'; script-src 'self'; form-action 'self'; frame-src 'none'"`, and its comment states that "a sandboxed srcdoc preview is not fetched, so it still renders" [VERIFIED: modules/boardwalk/boardwalk.go:61-66]. No CSP change is needed.
### Pattern 8: image library and absolute URLs (D-18-D-20)
- `attach.PublicURL` returns `strings.TrimRight(base, "/") + "/" + key` when `storage.uploads.public_url` is set, otherwise a root-relative path from `PublicPathPrefix()` [VERIFIED: modules/lagoon/attach/thumb.go:80-96]. Mail needs absolute URLs, so prefix `app.url` when the result starts with `/`. When neither is configured, refuse the upload/listing with a clear message.
- `PublicURLBase()` is a process global set by `OpenBucket` [VERIFIED: modules/lagoon/attach/bucket.go:42-55, 154-157]. `OpenBucket` is called only from `surf/serve.go:213` and `lagoon/commands.go:479` [VERIFIED: grep], not from `queue:work`. Any URL computed at send time in a separate worker would come out relative. Recommendation: the picker inserts the absolute URL into the content at edit time (D-20, which is text), and the logo URL is resolved by a config-based helper (add a pure `attach` function that takes `*compass.Config` instead of the globals) or stored absolute on save.
- Library host: an `AdminSingleton` controller (`SingletonRecord(ctx, db) (any, error)`, first-or-create with a unique-index race retry) [VERIFIED: modules/pact/capabilities.go:1041-1043; modules/cabana/singleton.go:41-80] on a one-row host table; `fileupload` attachMany `images`, public, `useCaption: true`. The picker's immediate upload goes through `attach.Store(ctx, db, bucket, attach.Upload{FileName, Body, Public: true}, attach.Limits{…})` [VERIFIED: modules/lagoon/attach/store.go:72-76,123] attached to the host row. That bypasses deferred binding on purpose (UI-SPEC C4).
### Pattern 9: admin wiring facts the plans touch
- **Built-in field types:** add `codeeditor`, `colorpicker`, `mailimage`, `mailpreview` to `formFieldTypes` (currently `"text" … "taglist"`) [VERIFIED: modules/cabana/form_schema.go:24-31]. Add `language` and `mailImages` to `formFieldKeys` (`mode` and `size` already exist) [VERIFIED: form_schema.go:35-51]. In the SPA, register the renderers in `registry.ts`, add `mailpreview` to `valueless` (`new Set<string>([RELATION_MANAGER, 'widget', 'partial', 'fileupload'])`) and the group-labelled types to `groupLabelledTypes` [VERIFIED: admin/src/components/form/registry.ts].
- **Row states:** pact's set is fixed: `RowStateDeleted RowState = "deleted"`, `RowStateNegative RowState = "negative"`, `RowStateDisabled RowState = "disabled"` [VERIFIED: modules/pact/capabilities.go:1055-1062]. cabana's order is `var rowStateOrder = []pact.RowState{pact.RowStateDeleted, pact.RowStateNegative, pact.RowStateDisabled}` [VERIFIED: modules/cabana/query.go:182]; messages are in `cabana/messages.go:42,61,163`; the SPA has `const ORDER = ['deleted', 'negative', 'disabled']` (RowStateBadges.vue:19) and `const ROW_STATES = ['deleted', 'negative', 'disabled']` (DataTable.vue:160); `TestRowStateValues` pins `len(set) != 3` [VERIFIED: modules/pact/capabilities_test.go:414-415]. All of these change for `customized`/`outdated`, and the OpenAPI message schema gains two keys.
- **Save hooks ordering:** the update path runs Fill → rules → `formBeforeUpdate` → relation checks → `assignBelongsTo` → `tx.Unscoped().Save(target)` → `formAfterUpdate`, with no dirty-check skip [VERIFIED: modules/cabana/crud.go:876-952]. Stamp `is_custom`, `base_hash` and `customized_at` in `FormBeforeUpdate`; run postcard validation in `FormAfterUpdate`, where a belongsTo `layout_id` is already assigned. A `*ValidationError` or `*ForbiddenError` passes through `lifecycleFailure` [VERIFIED: crud.go:1090-1106] and rolls the transaction back.
- **Partials (server-rendered `_info.htm`, `_file_version.htm`):** the allowlist has `div, span, p, strong, … code, pre` and global attributes `class, title, lang, dir, role` [VERIFIED: modules/cabana/partial_render.go:105-118]. That is enough for the drift callout and `.summer-diff`. Data comes from `AdminPartialData.PartialData` as a curated view model, never a GORM model [VERIFIED: modules/pact/capabilities.go:655-663].
- **Consumer layouts to keep rendering byte-identically:** `golem15.user::mail.layouts.user` (text wrapper `-- {{ .Content }} --`) and `golem15.paymentgateway::mail.layouts.paymentgateway` [VERIFIED: ../sm-user-plugin/plugin.go:309-313, views/mail/layouts/user.htm; ../sm-paymentgateway-plugin/mail.go:83-86].
### Recommended schema (design; table names follow the framework's Winter-name convention: `system_files`, `backend_users`)
```sql
CREATE TABLE system_mail_layouts (
id SERIAL PRIMARY KEY,
code TEXT NOT NULL UNIQUE, -- file: full name (contains '::'); database-only: ^[a-z0-9-]+$
source TEXT, -- owning plugin ID; NULL = database-only (deletable, never reset)
name TEXT NOT NULL,
content_html TEXT NOT NULL DEFAULT '', content_text TEXT NOT NULL DEFAULT '', content_css TEXT NOT NULL DEFAULT '',
is_custom BOOLEAN NOT NULL DEFAULT FALSE, -- file row: override active; database-only row: TRUE
file_hash TEXT, base_hash TEXT, -- D-07: current file hash (sync) / hash the override was based on (save)
customized_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE system_mail_templates (
id SERIAL PRIMARY KEY, code TEXT NOT NULL UNIQUE, source TEXT NOT NULL, locale TEXT, description TEXT,
subject TEXT NOT NULL DEFAULT '', body TEXT NOT NULL DEFAULT '',
layout_id INTEGER REFERENCES system_mail_layouts(id) ON DELETE RESTRICT, -- D-08 in-use refusal backstop
is_custom BOOLEAN NOT NULL DEFAULT FALSE, file_hash TEXT, base_hash TEXT, customized_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE system_mail_partials ( /* code, source, name, content_html, content_text, is_custom, file_hash, base_hash, customized_at, timestamps */ );
CREATE TABLE system_mail_brand_settings ( id INTEGER PRIMARY KEY, logo_file_id INTEGER, /* 14 colour TEXT columns */ created_at …, updated_at … );
CREATE TABLE system_mail_image_library ( id SERIAL PRIMARY KEY, created_at …, updated_at … ); -- attachMany host only
```
"Outdated" = `is_custom AND source IS NOT NULL AND base_hash IS DISTINCT FROM file_hash`, which the list filter can express in SQL. The hash is sha256 over the raw file bytes with CRLF normalized (postcard already normalizes in `splitEQ`, `strings.ReplaceAll(src, "\r\n", "\n")`) [VERIFIED: modules/postcard/templates.go:273].
### Anti-Patterns to Avoid
- **Boot-time snapshot of overrides in the mailer.** It is stale across instances (CONTEXT forbids it).
- **Adding methods to `postcard.Mailer` or changing `Catalog.Register`/`HasMailTemplates`.** Outside code implements or calls them directly (sm-user-plugin `mail_test.go:81`, paymentgateway `mail_smoke_test.go:49`).
- **`html.WithUnsafe` on the body renderer to "make partials work".** It changes every existing mail.
- **Deleting index rows on sync.** IDs churn during rolling deploys; filter by catalog membership instead.
- **Building image URLs in the SPA.** The server returns absolute URLs (UI-SPEC C4).
- **Sharing the Branding row as the image-library host.** It flips D-11 precedence.
## Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| Code editor | textarea + highlight overlay | CodeMirror 6 modular packages | Caret/scroll sync, a11y, bracket matching |
| Upload, storage, thumbnails, image guard | custom file handling | `attach.Store` + existing `fileupload` field | MIME/image guard, partitioned keys, public URL logic |
| Rate limiting across instances | per-process map | `cooler.Attempter` (Redis when configured) | Shared counters; same semantics as `throttle` |
| Template parsing / variable discovery | regex over `{{ }}` | `html/template` `Tree` + `text/template/parse` walk | Handles `if`/`range`/`with`/pipelines correctly |
| Markdown | own converter | goldmark (existing pin) | Same output as real sends |
| Settings singleton, record actions, confirm/danger, 409/403 handling | custom endpoints | cabana SettingsService, `pact.AdminRecordAction`, `ForbiddenError` | 19.6.1 already proved these paths |
**Key insight:** the one place a small hand-written piece is justified is the D-07 line diff. Inputs are a few hundred lines, the stdlib-first rule applies, and the UI-SPEC allows omitting the diff. A bounded LCS (cap of about 2,000 lines, beyond which only the file content is shown) is less risk than a new dependency.
## Common Pitfalls
### Pitfall 1: overrides ignored in a separate job worker
**What goes wrong:** mail sent from River jobs uses file versions while serve uses overrides.
**Why:** cabana only activates in serve (Pattern 1).
**How to avoid:** postcard's own DB source, wired in `Activate`, reading `app.Lookup[*gorm.DB]()` per send.
**Warning signs:** a test that sends through a mailer from `party.Activate` with a published DB and no `cabana.Activate` ignores an override.
### Pitfall 2: partial HTML silently vanishes from bodies
**What goes wrong:** `<!-- raw HTML omitted -->` appears where the button should be (verified behaviour).
**How to avoid:** placeholder substitution after goldmark (Pattern 4), a per-render nonce, and `[A-Z0-9]` tokens.
### Pitfall 3: save validation rejects working templates
**What goes wrong:** with an empty sample map, `{{ .order.code }}` fails with "nil pointer evaluating interface {}.code", so every paymentgateway template override would be refused under D-13.
**How to avoid:** validate and preview with the walker-built placeholder map (Pattern 5). Test against all 55 paymentgateway files and the 8 user files.
### Pitfall 4: relative image/logo URLs in mail
**What goes wrong:** `/storage/app/uploads/public/…` in a mail client, or a different URL from a worker process.
**How to avoid:** absolute URLs inserted at edit time, a config-based logo URL helper, and refusal when neither `storage.uploads.public_url` nor `app.url` is set. Note that a URL with a space breaks Markdown image syntax (`![Logo](https://cdn.example.com/a b.png)` stays literal text) [VERIFIED: scratch run]. Percent-encode in `insertImage.ts`; attach disk names are generated, so this mostly matters for `public_url` paths.
### Pitfall 5: CodeMirror swallows the form's Mod+Enter
**What goes wrong:** `defaultKeymap` contains `{ key: "Mod-Enter", run: insertBlankLine }` (and `Escape: simplifySelection`) [VERIFIED: @codemirror/commands 6.11.1 dist/index.js:1796-1816], so Mod+Enter inserts a line instead of saving (UI-SPEC C1).
**How to avoid:** `keymap.of([...historyKeymap, ...defaultKeymap.filter(b => b.key !== 'Mod-Enter')])`. Do not add `indentWithTab`; CodeMirror leaves Tab unbound by default to avoid a keyboard trap [CITED: codemirror.net/examples/tab]. Cover it in the browser test.
### Pitfall 6: validating a template against the old layout
**What goes wrong:** `FormBeforeUpdate` runs before `assignBelongsTo`, so a changed `layout_id` is not on the model yet [VERIFIED: crud.go:895-928].
**How to avoid:** validate in `FormAfterUpdate` (same transaction; a `ValidationError` rolls back).
### Pitfall 7: tables missing after a binary upgrade
**What goes wrong:** the new binary starts before `migrate` runs. Sync fails and per-send lookups hit `undefined_table`.
**How to avoid:** sends treat any lookup error as "no overrides" with a rate-limited warning; sync retries on the first list request. `serve` already reports pending migrations at startup [VERIFIED: modules/surf/serve.go:111]. Update `lagoon/pending_test.go:349` (`wantSets := []string{"summercms.attach", "summercms.cabana", DeferredHistoryID, QueueHistoryID, "demo.alpha", "demo.beta"}`) [VERIFIED].
### Pitfall 8: safety-regex false positives
**What goes wrong:** the existing checks `(?i)(?:javascript|vbscript|data):` and `(?i)\son[a-z]+\s*=` [VERIFIED: modules/postcard/templates.go:25-26] match ordinary text such as "metadata:" or " online = ", and now show up as save errors.
**How to avoid:** keep the behaviour (it matches sends) but pass the matched snippet in `:message` so admins can see why.
### Pitfall 9: parameterized validation lines come out raw
**What goes wrong:** `localizeDetails` localizes plain phrase keys only [VERIFIED: modules/cabana/crud.go:104-150], so `mail.invalid_render` with `:message`, or `mail.layout.delete_in_use` with `:name`/`:count`, shows unsubstituted.
**How to avoid:** build the final text in the hook with the translator, as `backendMessageParams(ctx, tr, key, fallback, params)` does [VERIFIED: modules/cabana/account_tokens.go:206,227]. Use a `ForbiddenError` (not a `ValidationError`) for the delete refusal (UI-SPEC C15 note).
### Pitfall 10: the templates list can be empty
**What goes wrong:** the UI-SPEC says the unfiltered list is "never empty, because postcard registers built-in mail". postcard registers **no** templates today, only the default layout [VERIFIED: modules/postcard/templates.go:62-74]. An app without mail-sending plugins shows an empty list.
**How to avoid:** accept it (the empty copy exists) or ship one built-in template. Decide in the plan.
## Code Examples
### Placeholder walker (prototype, renders the real paymentgateway shapes)
```go
// Source: scratch prototype run 2026-11-06 against Go 1.27.2 html/template.
// Output for the paymentgateway "paid" body: "Cześć [user.first_name], … **ZAM-1** …
// - [order.items.description] × [order.items.count]: … [banking_details.bankDetailsLines]"
func fill(root map[string]any, n parse.Node, prefix []string, samples map[string]string) {
switch x := n.(type) {
case *parse.ListNode:
if x == nil { return }
for _, c := range x.Nodes { fill(root, c, prefix, samples) }
case *parse.ActionNode:
fill(root, x.Pipe, prefix, samples)
case *parse.PipeNode:
if x == nil { return }
for _, c := range x.Cmds { fill(root, c, prefix, samples) }
case *parse.CommandNode:
for _, a := range x.Args { fill(root, a, prefix, samples) }
case *parse.FieldNode:
set(root, x.Ident, prefix, samples) // nested maps, "[a.b]" or the sample
case *parse.IfNode:
fill(root, x.Pipe, prefix, samples); fill(root, x.List, prefix, samples); fill(root, x.ElseList, prefix, samples)
case *parse.WithNode:
fill(root, x.Pipe, prefix, samples); fill(root, x.List, prefix, samples); fill(root, x.ElseList, prefix, samples)
case *parse.RangeNode:
// one-element slice: a map of the fields used in the body, or a string when the body prints {{ . }}
}
}
// t := template.Must(template.New("t").Option("missingkey=zero").Parse(src)); fill(vars, t.Tree.Root, nil, samples)
```
The production version must also handle `ChainNode`/`VariableNode` and `with` re-rooting (a `with` body's fields are relative to the pipeline value), and must apply the `partial` func map before parsing.
### CodeMirror setup (verified export names; `editor.ts` lazy target)
```ts
// Sources: codemirror.net/examples/tab, /examples/decoration; exports verified in the pinned dist .d.ts files.
import { EditorState } from '@codemirror/state'
import { EditorView, keymap, lineNumbers, Decoration, MatchDecorator, ViewPlugin } from '@codemirror/view'
import { defaultKeymap, history, historyKeymap } from '@codemirror/commands'
import { bracketMatching, HighlightStyle, syntaxHighlighting } from '@codemirror/language'
import { html } from '@codemirror/lang-html'
import { css } from '@codemirror/lang-css'
import { tags } from '@lezer/highlight'
const actionMatcher = new MatchDecorator({
regexp: /\{\{[\s\S]*?\}\}/g,
decoration: () => Decoration.mark({ class: 'cm-summer-action' }),
})
const goActions = ViewPlugin.fromClass(class {
decorations
constructor(view: EditorView) { this.decorations = actionMatcher.createDeco(view) }
update(u: import('@codemirror/view').ViewUpdate) { this.decorations = actionMatcher.updateDeco(u, this.decorations) }
}, { decorations: (v) => v.decorations })
const keys = keymap.of([...historyKeymap, ...defaultKeymap.filter((b) => b.key !== 'Mod-Enter')]) // Pitfall 5
// No indentWithTab: Tab moves focus (WCAG 2.1.2). Line wrapping only for language: text (EditorView.lineWrapping).
```
### Lagoon framework set (where the migration goes)
```go
// modules/lagoon/migrations.go, today [VERIFIED: lines 224-230]:
sets := []migrationSet{
{"summercms.attach", "system_files", attach.Migrations},
{"summercms.cabana", "backend admin", BackendAdminMigrations},
{DeferredHistoryID, "deferred bindings", DeferredBindingMigrations},
{QueueHistoryID, "queue", QueueMigrations(sqlDB)},
}
// recommended addition (after attach, because brand logo_file_id may reference system_files):
// {"summercms.postcard", "mail", postcard.Migrations},
```
## State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|--------------|------------------|--------------|--------|
| Winter: Twig mail + DB rows synced on list open, CSS inlined by CssToInlineStyles | postcard: Go templates, `<style>` block (no inliner) | this project | Branding CSS must work as a `<style>` block; an inliner would be a separate dependency decision (not in scope) |
| CodeMirror 5 / Ace (Winter) | CodeMirror 6 modular, Tab unbound by default | CM6 | Lazy chunk, a11y-safe default |
**Deprecated/outdated:** the `codemirror` meta package with `basicSetup` (heavier than needed; UI-SPEC excludes it).
## Assumptions Log
| # | Claim | Section | Risk if Wrong |
|---|-------|---------|---------------|
| A1 | D-01's intent is met when postcard defines the interface **and** ships the DB implementation, with cabana owning only the screens | Pattern 1 | User may insist on cabana-owned storage; then the worker-process gap needs another wiring (CONTEXT allows "by postcard if the planner justifies it") |
| A2 | D-06 "reset deletes the override row" is satisfied by clearing the override columns of the index row (re-mirroring the file), because rows must exist for the generic admin | Pattern 2 / Open Q1 | If the user wants a literal separate override table, the generic form/list need joins (much more cabana work) |
| A3 | D-08 "not rows copied into the DB at boot" refers to predefined layouts not becoming custom records; index rows with a file mirror (synced at serve start) are acceptable | Pattern 2 | Conflicts with a literal reading; confirm |
| A4 | Winter table names (`system_mail_*`) are fine because ported apps run on a fresh Postgres (PHP side is MySQL; Winter data arrives through explicit import commands) | Schema | If an app's Postgres ever holds Winter's `system_mail_*` tables, `CREATE TABLE` collides; Winter's Twig content would then be read as Go templates |
| A5 | INI `sample.*` dotted keys are the better D-15 mechanism (vs a capability) | Pattern 5 | Planner's call per D-15; low risk |
| A6 | surf publishing its store as `cooler.Attempter` is acceptable as a small surf change | Pattern 7 | Alternative: per-instance limit only |
| A7 | Test-send limit of about 5 per 10 minutes per admin | Pattern 7 | Tuning only |
| A8 | Built-in partial set button/panel/table/subcopy and layouts default/system/minimal | Pattern 4 | Planner's discretion |
| A9 | `@lezer/highlight` 1.2.5 should be a direct pinned dependency because the theme imports `tags` | Standard Stack | If the theme avoids `tags`, it is not needed |
## Open Questions
1. **Reset semantics vs D-06 wording ("deletes the override row").**
- What we know: the generic admin needs a row per template (numeric IDs, SQL search).
- What's unclear: whether the user accepts "reset clears the override in the index row".
- Recommendation: state it in the plan's opening summary for confirmation. Behaviourally identical: the file wins, the customized state is gone.
2. **Logo URL: resolve at render (needs a config-based attach helper) or store absolute at save?**
- Recommendation: add a pure `attach` URL function taking `*compass.Config` (additive, documented in the attach README) and resolve at render. A removed library image then yields an empty `logoUrl` instead of a broken link.
3. **Picker upload permission (D-03 note).** Recommendation (as UI-SPEC): `canUpload` only for `system.manage_mail_layouts`; template editors pick only.
4. **Built-in template so the list is never empty (Pitfall 10).** Recommendation: none; keep the filtered/empty copy.
5. **`layout` field type on the template form:** `relation` (belongsTo, `nameFrom: name`, renders as a select) is the natural fit with `layout_id`. Confirm that a relation dropdown can omit the empty option, or use a required rule.
## Environment Availability
| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| Go toolchain | all Go work | ✓ | go1.27.2 | — |
| Node / npm | admin SPA build/tests | ✓ | v24.22.0 / 12.2.1 (`engines.node >=22.6`) | — |
| admin/node_modules | `npm test`, gates | ✓ | present | `npm --prefix admin ci` |
| Docker | testcontainers Postgres/Mailpit tests | ✓ (socket access ok) | client 29.9.1 | `-short` skips; `sudo -u golem` per CLAUDE.md if denied |
| Playwright Chromium | `test:browser` (codeeditor keyboard test) | ✓ | chromium-1169, -1194 in ~/.cache/ms-playwright | — |
| swag | `check-admin-openapi.sh` | ✓ (script runs `go run …swag@v1.16.6`) | v1.16.6 | — |
| `summer` CLI | `summer docs:build --check` | ✓ | installed in GOPATH/bin | `go test ./cmd/summer -run TestDocsTree` |
| Context7 MCP | research only | ✗ | — | WebFetch of codemirror.net + `npm pack` d.ts inspection |
**Missing dependencies with no fallback:** none.
## Validation Architecture
### Test Framework
| Property | Value |
|----------|-------|
| Framework | Go `testing` (+ testcontainers-go for Postgres/Mailpit); Vitest 3.2.7 (happy-dom) and Vitest browser mode with Playwright 1.60.0 |
| Config file | `admin/vitest.config.*`, `admin/vitest.browser.config.ts`; Go: none |
| Quick run command | `go test -short ./modules/postcard/... ./modules/pact/... ./modules/cabana/... ./modules/lagoon/...` |
| Full suite command | `go vet ./... && go test ./... && npm --prefix admin run test && npm --prefix admin run test:browser && npm --prefix admin run typecheck && scripts/check-admin-openapi.sh --check && scripts/check-admin-dist.sh && summer docs:build --check` |
### Phase Decisions → Test Map
| Decision | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| D-11/D-14, Pattern 3 | Override wins; file wins without one; render failure falls back to file; DB error falls back; caller subject still wins | unit (fake `OverrideSource`, MemoryDriver) | `go test -short ./modules/postcard -run 'TestOverride\|TestFallback'` | ❌ Wave 0 |
| Pattern 1 | DB source reads real tables; works with no cabana activation | integration (Postgres) | `go test ./modules/postcard -run TestStore` | ❌ |
| D-12 | `HasMailPartials` boot errors (missing file, duplicate, foreign namespace); body partial survives Markdown; nonce forgery fails; layout partial; text part | unit | `go test -short ./modules/postcard -run TestPartial` | ❌ |
| D-15 | `sample.*` keys parsed case-preserving; old keys ignored; walker fills nested/range/dot | unit | `go test -short ./modules/postcard -run TestSample` | ❌ |
| D-13 | Draft validation errors (parse, render, unsafe, subject CR/LF, unknown layout/partial, missing `{{ .Content }}`) mapped to fields | unit + Postgres | `go test -short ./modules/postcard -run TestValidateDraft`; `go test ./modules/cabana -run TestMailSave` | ❌ |
| D-07 | Drift: `base_hash` ≠ `file_hash` → outdated state and filter; re-save clears | Postgres | `go test ./modules/cabana -run TestMailDrift` | ❌ |
| D-06/D-08 | Reset clears override; custom layout delete refused with count (403); file layout never deletable | Postgres | `go test ./modules/cabana -run 'TestMailReset\|TestMailLayoutDelete'` | ❌ |
| D-10/D-11 | No brand row → `mail.brandCss`; saved → compiled CSS replaces it; `mail.css` still applied; bad colour 422 | unit + Postgres | `go test -short ./modules/postcard -run TestBrand`; `go test ./modules/cabana -run TestMailBrand` | ❌ |
| D-03 | Permission split per controller, preview mode, picker list vs upload; no default role grants | Postgres (19.6.1 `newScreensEnv` harness) | `go test ./modules/cabana -run TestMailPermissions` | ❌ (harness exists: `administrators_harness_test.go`) |
| D-17 | Recipient is always the admin's own email; body recipient refused; 429 + Retry-After after N | Postgres | `go test ./modules/cabana -run TestMailTestSend` | ❌ |
| D-05 | Locale parsing only for enabled locales; locale filter | unit | `go test -short ./modules/cabana -run TestMailLocale` | ❌ |
| Pattern 2 | Sync idempotent; concurrent sync safe; custom rows untouched; missing tables tolerated | Postgres | `go test ./modules/postcard -run TestSync` | ❌ |
| Consumers | user + paymentgateway mail renders unchanged | unit (their repos, `replace ../summercms.go`) | `(cd ../sm-user-plugin && go test -run 'TestMail\|TestPolishMailSubjects' ./...)`; `(cd ../sm-paymentgateway-plugin && go test -run 'TestMailCatalog\|TestMailTracer\|TestMailersMatrixSmoke' ./...)` | ✅ |
| Byte-identity | Existing postcard tests still pass; default layout output unchanged | unit | `go test -short ./modules/postcard/...` | ✅ |
| UI C1/C4/C8/C10 | codeeditor keys (Tab leaves, Mod+S/Mod+Enter save, Esc ≠ Cancel); insertImage escaping; footer record actions, `meta.deletable`; preview sandbox/sequence guard | SPA unit + browser | `npm --prefix admin run test -- mail codeeditor`; `npm --prefix admin run test:browser -- codeeditor` | ❌ |
| Gates | OpenAPI/schema in sync; dist matches; docs identifiers exist | scripts | `scripts/check-admin-openapi.sh --check`; `scripts/check-admin-dist.sh`; `go test ./cmd/summer -run TestDocsTree` | ✅ |
### Sampling Rate
- **Per task commit:** the quick run command plus `go vet ./...` (and `npm --prefix admin run test` for SPA tasks).
- **Per wave merge:** the full suite command.
- **Phase gate:** full suite green plus a new `scripts/check-phase35.sh` (mirroring `check-phase34.sh`) that also runs the two consumer repos' mail tests, before `/msd-verify-work`.
### Wave 0 Gaps
- [ ] `modules/postcard/override_test.go`, `partials_test.go`, `samples_test.go`, `brand_test.go`, `store_postgres_test.go` (TestMain with `flag.Parse()` + `testing.Short()` per CLAUDE.md)
- [ ] `modules/cabana/mail_*_postgres_test.go` reusing the 19.6.1 harness
- [ ] `admin/tests/form/codeeditor*.test.ts`, `admin/tests/browser/codeeditor.test.ts`, `admin/tests/form/mailpreview.test.ts`, `admin/tests/form/insertImage.test.ts`
- [ ] Update `modules/pact/capabilities_test.go` `TestRowStateValues` and `modules/lagoon/pending_test.go` set list
- [ ] `scripts/check-phase35.sh`
## Security Domain
### Applicable ASVS Categories
| ASVS Category | Applies | Standard Control |
|---------------|---------|-----------------|
| V2 Authentication | no (reuses admin JWT/cookie) | bouncer composite guard |
| V3 Session Management | yes (test send) | interactive-only route wrapper; no API-token use of test send |
| V4 Access Control | yes | `RequiredPermissions` per controller; `cabana.Allows` per preview mode/picker action; no default role grants |
| V5 Input Validation | yes | postcard draft validation (D-13), hex colour regex, layout code `^[a-z0-9-]+$`, attach image guard/limits |
| V6 Cryptography | minimal | sha256 for drift (integrity hint only); `crypto/rand` nonce for partial tokens |
| V12 Files | yes | `attach.Store` with image limits and public flag; `accept=` is a hint only |
| V14 Config | yes | CSP unchanged; srcdoc `sandbox=""` |
### Known Threat Patterns for this stack
| Pattern | STRIDE | Standard Mitigation |
|---------|--------|---------------------|
| Stored XSS in admin preview via a mail draft | Tampering / Elevation | `sandbox=""` srcdoc iframe (no scripts, no same-origin); server-rendered HTML already passes `validateHTML`; the SPA never builds the HTML |
| Script/handler injection into sent mail | Tampering | `validateHTML` on body, layout and final HTML at save **and** send; partial outputs included |
| CSS injection through brand colours (`</style>`) | Tampering | strict `#rrggbb` validation before compiling; `validateHTML` backstop |
| Header injection via subject override | Tampering | existing CR/LF refusal, also at save |
| Spam/abuse through test send | Abuse | recipient fixed to the admin's own address; per-admin rate limit; permission per record kind |
| Placeholder forgery (user value contains the partial token) | Tampering | per-render random nonce in the token |
| Over-privileged picker upload | Elevation | upload requires `system.manage_mail_layouts`; list allowed for either code |
| Privilege via permission codes | Elevation | codes declared with no `Roles`; superuser-only until granted (19.6.1 pattern) |
| Information disclosure in preview errors | Info disclosure | `:message` limited to parser/safety text, never DB errors or paths beyond the file name |
## Sources
### Primary (HIGH confidence)
- Codebase, read this session: `modules/postcard/{templates.go,mailer.go,drivers.go,README.md,assets/default.htm}`, `modules/pact/capabilities.go`, `modules/party/registry.go`, `modules/backpack/services.go`, `modules/cabana/{backend_screens.go,http.go,settings.go,singleton.go,crud.go,query.go,form_schema.go,field_markdown.go,contracts.go,partial_render.go,messages.go}`, `modules/lagoon/{migrations.go,backend_admin_migrations.go,ondatabase.go,connection.go}`, `modules/lagoon/attach/{bucket.go,thumb.go,store.go,migrations.go}`, `modules/conga/commands.go`, `modules/surf/{serve.go,limiter.go,router.go}`, `modules/cooler/{cooler.go,README.md}`, `modules/boardwalk/boardwalk.go`, `admin/package.json`, `admin/src/components/form/registry.ts`
- Consumers: `../sm-user-plugin/{plugin.go,views/mail/**}`, `../sm-paymentgateway-plugin/{mail.go,views/mail/**,events/mailers.go,mail_smoke_test.go}`
- Winter reference: `examples/golem15-wintercms-starter/modules/system/{controllers/MailTemplates.php,models/MailTemplate.php,models/MailLayout.php,views/mail/*.php,models/mailbrandsetting/*}`
- npm registry (`npm view`, `npm pack`) for all seven SPA packages; MSD package-legitimacy seam
- Scratch programs (goldmark v1.8.6 from the module cache; html/template walker), 2026-11-06
### Secondary (MEDIUM confidence)
- codemirror.net/examples/tab (Tab handling, `indentWithTab`, `toggleTabFocusMode`)
- codemirror.net/examples/decoration (`MatchDecorator`, `ViewPlugin`, `createDeco`/`updateDeco`)
### Tertiary (LOW confidence)
- none
## Metadata
**Confidence breakdown:**
- Standard stack: HIGH. Go dependencies are already in the graph; npm versions, maintainers and exports were verified against the registry and the pinned tarballs.
- Architecture: MEDIUM-HIGH. Every integration fact was read in the code. The store split, index rows and placeholder design are recommendations, flagged in the Assumptions Log where they reinterpret CONTEXT wording.
- Pitfalls: HIGH. The goldmark, missing-key, keymap, hook-order, worker-URL and limiter-reachability pitfalls were each confirmed by execution or by reading the source.
**Research date:** 2026-11-06
**Valid until:** 2026-12-06 (codebase facts may move with concurrent Phase 33 work in cabana; re-check `cabana/http.go`, `form_schema.go` and `crud.go` line references before planning if Phase 33 lands first)

View File

@@ -0,0 +1,91 @@
---
phase: "35"
slug: "mail-templates-and-layouts-editor"
# status lifecycle: draft (seeded by plan-phase) → validated (set by validate-phase §6)
status: draft
nyquist_compliant: false
wave_0_complete: false
created: "2026-11-06"
---
# Phase 35 — Validation Strategy
> Per-phase validation contract for feedback sampling during execution. Derived from `35-RESEARCH.md` § Validation Architecture.
---
## Test Infrastructure
| Property | Value |
|----------|-------|
| **Framework** | Go `testing` (+ testcontainers-go for Postgres/Mailpit); Vitest (happy-dom) and Vitest browser mode with Playwright for the admin SPA |
| **Config file** | `admin/vitest.config.*`, `admin/vitest.browser.config.ts`; Go: none |
| **Quick run command** | `go test -short ./modules/postcard/... ./modules/pact/... ./modules/cabana/... ./modules/lagoon/...` |
| **Full suite command** | `go vet ./... && go test ./... && npm --prefix admin run test && npm --prefix admin run test:browser && npm --prefix admin run typecheck && scripts/check-admin-openapi.sh --check && scripts/check-admin-dist.sh && summer docs:build --check` |
| **Estimated runtime** | ~60 seconds quick, ~10 minutes full |
---
## Sampling Rate
- **After every task commit:** quick run command plus `go vet ./...` (and `npm --prefix admin run test` for SPA tasks)
- **After every plan wave:** full suite command
- **Before `/msd-verify-work`:** full suite plus `scripts/check-phase35.sh` (also runs the sm-user-plugin and sm-paymentgateway-plugin mail tests) must be green
- **Max feedback latency:** 120 seconds
---
## Per-Decision Verification Map
| Decision | Behavior | Test Type | Automated Command | File Exists | Status |
|----------|----------|-----------|-------------------|-------------|--------|
| D-11/D-14 | Override wins; file wins without one; render failure and DB error fall back to file; caller subject still wins | unit | `go test -short ./modules/postcard -run 'TestOverride\|TestFallback'` | ❌ W0 | ⬜ pending |
| store | DB source reads real tables; works with no cabana activation | Postgres | `go test ./modules/postcard -run TestStore` | ❌ W0 | ⬜ pending |
| D-12 | `HasMailPartials` boot errors; body partial survives Markdown; nonce forgery fails; layout partial; text part | unit | `go test -short ./modules/postcard -run TestPartial` | ❌ W0 | ⬜ pending |
| D-15 | `sample.*` keys parsed; old keys ignored; placeholder walker fills nested/range/dot | unit | `go test -short ./modules/postcard -run TestSample` | ❌ W0 | ⬜ pending |
| D-13 | Draft validation errors mapped to fields | unit + Postgres | `go test -short ./modules/postcard -run TestValidateDraft`; `go test ./modules/cabana -run TestMailSave` | ❌ W0 | ⬜ pending |
| D-07 | Drift flagged and filterable; re-save clears | Postgres | `go test ./modules/cabana -run TestMailDrift` | ❌ W0 | ⬜ pending |
| D-06/D-08 | Reset clears override; in-use custom layout delete refused with count; file layout never deletable | Postgres | `go test ./modules/cabana -run 'TestMailReset\|TestMailLayoutDelete'` | ❌ W0 | ⬜ pending |
| D-10/D-11 | No brand row → `mail.brandCss`; saved → compiled CSS replaces it; `mail.css` still applied; bad colour 422 | unit + Postgres | `go test -short ./modules/postcard -run TestBrand`; `go test ./modules/cabana -run TestMailBrand` | ❌ W0 | ⬜ pending |
| D-03 | Permission split per controller, preview, picker list vs upload; no default role grants | Postgres | `go test ./modules/cabana -run TestMailPermissions` | ❌ W0 | ⬜ pending |
| D-17 | Recipient always the admin's own email; 429 + Retry-After after the limit | Postgres | `go test ./modules/cabana -run TestMailTestSend` | ❌ W0 | ⬜ pending |
| D-05 | Locale parsed for enabled locales; locale filter | unit | `go test -short ./modules/cabana -run TestMailLocale` | ❌ W0 | ⬜ pending |
| sync | Index sync idempotent, concurrency-safe, custom rows untouched | Postgres | `go test ./modules/postcard -run TestSync` | ❌ W0 | ⬜ pending |
| consumers | user + paymentgateway mail renders unchanged | unit (sibling repos) | `(cd ../sm-user-plugin && go test -run 'TestMail\|TestPolishMailSubjects' ./...)`; `(cd ../sm-paymentgateway-plugin && go test -run 'TestMailCatalog\|TestMailTracer\|TestMailersMatrixSmoke' ./...)` | ✅ | ⬜ pending |
| byte-identity | Existing postcard tests pass; default layout output unchanged | unit | `go test -short ./modules/postcard/...` | ✅ | ⬜ pending |
| UI | codeeditor keys, insertImage escaping, footer record actions, preview sandbox | SPA unit + browser | `npm --prefix admin run test -- mail codeeditor`; `npm --prefix admin run test:browser -- codeeditor` | ❌ W0 | ⬜ pending |
| gates | OpenAPI/schema in sync; dist matches; docs identifiers exist | scripts | `scripts/check-admin-openapi.sh --check`; `scripts/check-admin-dist.sh`; `go test ./cmd/summer -run TestDocsTree` | ✅ | ⬜ pending |
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
---
## Wave 0 Requirements
- [ ] `modules/postcard/override_test.go`, `partials_test.go`, `samples_test.go`, `brand_test.go`, `store_postgres_test.go` (TestMain with `flag.Parse()` + `testing.Short()`)
- [ ] `modules/cabana/mail_*_postgres_test.go` reusing the 19.6.1 screens harness
- [ ] `admin/tests/form/codeeditor*.test.ts`, `admin/tests/browser/codeeditor.test.ts`, `admin/tests/form/mailpreview.test.ts`, `admin/tests/form/insertImage.test.ts`
- [ ] Update `modules/pact/capabilities_test.go` `TestRowStateValues` and `modules/lagoon/pending_test.go` set list
- [ ] `scripts/check-phase35.sh`
---
## Manual-Only Verifications
| Behavior | Decision | Why Manual | Test Instructions |
|----------|----------|------------|-------------------|
| Test-send mail looks right in a real inbox | D-17 | Visual rendering in mail clients | Send a draft from the template form to Mailpit and check HTML and text parts |
| Code editor feel (highlighting, focus, theme) | D-09 | Visual | Open a layout form in the dev SPA, edit each tab |
---
## Validation Sign-Off
- [ ] All tasks have `<automated>` verify or Wave 0 dependencies
- [ ] Sampling continuity: no 3 consecutive tasks without automated verify
- [ ] Wave 0 covers all MISSING references
- [ ] No watch-mode flags
- [ ] Feedback latency < 120s
- [ ] `nyquist_compliant: true` set in frontmatter
**Approval:** pending

View File

@@ -0,0 +1 @@
No external API integration: Phase 35 adds a database override layer, an editor API inside postcard and admin API routes inside cabana for the framework's own admin SPA; mail still leaves through the existing, unchanged postcard drivers (memory, log, SMTP), and the only new third-party code is the CodeMirror editor library, which calls no service.

View File

@@ -0,0 +1,174 @@
---
phase: quick-261106-kt1
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- modules/pier/pier.go
- modules/pier/tracer_smoke_test.go
- modules/pier/README.md
- docs/services/mcp.md
- .planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md
autonomous: true
requirements:
- QUICK-261106-kt1
estimate:
tokens: 35000
raw_tokens: 35000
tasks: 2
confidence: low
must_haves:
truths:
- "A request to the MCP endpoint with a bad or missing bearer answers 401 whose WWW-Authenticate is `Bearer resource_metadata=\"<app.url>/.well-known/oauth-protected-resource<mcp.path>\"` and carries no scope parameter"
- "A spec-following MCP client therefore falls back to the PRM scopes_supported list when it requests authorization"
- "No 403 insufficient-scope path is introduced: RequireBearerTokenOptions.Scopes stays unset, so the SDK still never enforces a scope on every request"
- "SSE / streamable GET responses still flush: the SDK handler now receives the original ResponseWriter, which keeps its own http.Flusher"
- "pier's README, docs/services/mcp.md and the 19.7 security review row T-19.7-10 describe the challenge without a scope parameter"
artifacts:
- path: modules/pier/pier.go
provides: "bearer() wrapping auth.RequireBearerToken directly, no response-writer wrapper"
- path: modules/pier/tracer_smoke_test.go
provides: "bad-bearer and missing-bearer 401 assertions: resource_metadata present, no scope parameter"
key_links:
- from: modules/pier/pier.go bearer()
to: go-sdk auth.RequireBearerToken (v1.8.0 auth/auth.go:104-116)
via: "ResourceMetadataURL set, Scopes empty -> header is `Bearer resource_metadata=\"...\"` only"
---
<objective>
pier's 401 challenge on the MCP endpoint stops naming a fixed scope. Today the
response-writer wrapper in `modules/pier/pier.go` appends a read-only scope
parameter to every 401 `WWW-Authenticate`, and the MCP scope-selection rule makes
a client use that parameter instead of the PRM `scopes_supported`. A client that
follows the spec therefore asks only for the read scope, never receives a `write`
grant, and pier's read/write ceiling (`modules/pier/identity.go:80`, `writes()`)
hides every non-read-only tool with no step-up path. User decision (option 1):
the challenge carries only `resource_metadata` (RFC 6750 makes `scope` optional;
the MCP spec's server-side scope hint is a SHOULD, which is knowingly dropped here).
Purpose: clients request the advertised scope set and consent decides the grant.
Output: wrapper removed, smoke test pinned to the new header, README/docs and the
19.7 security review row updated.
</objective>
<execution_context>
@~/.claude/msd-core/workflows/execute-plan.md
@~/.claude/msd-core/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@CLAUDE.md
Verified facts (do not re-investigate the live server):
- go-sdk is `github.com/modelcontextprotocol/go-sdk v1.8.0`. `auth.RequireBearerToken`
(`$(go list -m -f '{{.Dir}}' github.com/modelcontextprotocol/go-sdk)/auth/auth.go:97-125`)
writes `WWW-Authenticate: Bearer resource_metadata="<url>"` on 401/403 when
`ResourceMetadataURL` is set, and adds `scope=` ONLY when `opts.Scopes` is non-empty
(which would also enforce those scopes with 403 on every request). pier sets no
`Scopes`, so the SDK itself never emits a scope parameter.
- The wrapper type in `modules/pier/pier.go:288-310` exists only to append the scope;
its `Flush` and `Unwrap` methods only forward to the wrapped writer so the wrapper
does not hide the underlying `http.Flusher`. Removing the wrapper hands the SDK the
original writer, which keeps every capability it has. Nothing else in the repo
references the type (grep confirmed).
- `strings` stays imported in pier.go (used at lines ~166, ~220, ~222, ~332).
- `modules/pier/session.go:90` writes a bare 401 without `WWW-Authenticate` when no
identity is on the request (behind the bearer middleware, unreachable from outside);
it is out of scope and unchanged.
- `scripts/check-phase19.7.sh` has no grep on the challenge (only test file lists).
- `.planning/phases/19.7-generic-mcp-module/19.7-02-PLAN.md:252` mentions the old header;
it is a historical plan and is NOT edited.
- Another quick task (261106-kt2) is editing `modules/lagoon` concurrently: do not touch,
stage or format any lagoon file. Stage files explicitly by path.
- `summer` is not on PATH here; use `go run ./cmd/summer docs:build --check`.
</context>
<tasks>
<task type="tracer">
<name>Task 1: Drop the scope parameter from pier's 401 challenge (code, smoke test, README, docs)</name>
<files>modules/pier/pier.go, modules/pier/tracer_smoke_test.go, modules/pier/README.md, docs/services/mcp.md</files>
<read_first>
- modules/pier/pier.go lines 248-311 (bearer() and the wrapper type)
- modules/pier/tracer_smoke_test.go lines 99-115 (bad-bearer assertion)
- modules/pier/README.md line 47
- docs/services/mcp.md line 141 (last sentences of the PRM paragraph)
</read_first>
<!-- planner-discipline-allow: challengeWriter -->
<action>
Per the user decision (option 1), the MCP endpoint's 401 challenge carries only resource_metadata.
1. modules/pier/pier.go: delete the challengeWriter type and its three methods (WriteHeader, Flush, Unwrap; lines ~288-310). In bearer(), return the middleware from auth.RequireBearerToken(verify, &auth.RequireBearerTokenOptions{ResourceMetadataURL: p.prmURL, AllowMissingExpiration: true})(next) directly instead of wrapping it in an http.HandlerFunc that substitutes the writer. Do NOT add a Scopes option: it would make the SDK answer 403 to every token lacking those scopes. Keep the verify closure unchanged. Rewrite the bearer() doc comment (lines ~248-251): a failure is a 401 whose WWW-Authenticate names only the protected resource metadata URL; it carries no scope parameter, so a client requests the scopes the metadata document advertises and the authorization server's consent decides the grant (tools are then filtered per granted scope). Neutral wording only; no consuming application or client product is named in code comments. Confirm the strings import is still used (it is, elsewhere in the file); run gofmt on the file.
2. modules/pier/tracer_smoke_test.go: in the bad-bearer block (~lines 99-114) keep the 401 and resource_metadata checks and replace the positive scope check with a negative one: the challenge must not contain the substring scope= at all. Make the expected header exact: the challenge equals Bearer resource_metadata="http://acme.test/.well-known/oauth-protected-resource/mcp". Update the comment above the block to say the challenge names only the metadata URL. Add a second request in the same block with no Authorization header (same initialize body and headers otherwise) asserting the same 401 and the same exact challenge, so both SDK failure paths (no bearer token, invalid token) are pinned.
3. modules/pier/README.md line 47: the 401 bullet states the header as Bearer resource_metadata="<app.url>/.well-known/oauth-protected-resource<mcp.path>" with no scope parameter, followed by one clause: there is no scope parameter, so a client requests the scopes_supported of the protected resource metadata. Keep the rest of the bullet (the Origin / 403 clause) as is.
4. docs/services/mcp.md line 141: replace the sentence that begins "A 401 carries" so the quoted header is Bearer resource_metadata="…" only, and add: it carries no scope parameter, so a client requests the scopes listed in the metadata document and consent decides what is granted. Keep the "which is how a client finds the authorization server" meaning and the following mcp.resource sentence.
Commit these four files together as one code commit, staged by explicit path (never git add -A, never a lagoon file), message: fix(pier): MCP 401 challenge names only resource_metadata. No co-author tag.
</action>
<verify>
<automated>cd /media/nvme/dev/golem15/summercms.io/summercms/summercms.go && FMT=$(gofmt -l modules/pier/) && test -z "$FMT" && go vet ./modules/pier/... && go test -short ./modules/pier/... && go test ./cmd/summer -run TestDocsTree && go run ./cmd/summer docs:build --check && test "$(grep -v '^\s*//' modules/pier/pier.go | grep -c challengeWriter)" = 0 && ! grep -n 'scope=\\"read\\"\|scope="read"' modules/pier/*.go modules/pier/README.md docs/services/mcp.md && grep -c 'resource_metadata' modules/pier/README.md docs/services/mcp.md</automated>
</verify>
<done>
bearer() wraps auth.RequireBearerToken with no writer substitution and no Scopes option; the wrapper type is gone; TestPierTracerSmoke asserts the exact header `Bearer resource_metadata="http://acme.test/.well-known/oauth-protected-resource/mcp"` for both a bad and a missing bearer; README and docs describe the header without a scope parameter; go vet, go test -short ./modules/pier/..., TestDocsTree and docs:build --check pass; one commit `fix(pier): ...` containing exactly the four files.
</done>
</task>
<task type="auto">
<name>Task 2: Update security review row T-19.7-10 (planning doc, not in the code commit)</name>
<files>.planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md</files>
<read_first>
- .planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md line 48 (row T-19.7-10)
</read_first>
<action>
Edit only row T-19.7-10 (line 48) with a scoped Edit. In the mitigation cell, replace the trailing clause about the RFC 9728 challenge so it reads: a failure is 401 with the RFC 9728 challenge naming only resource_metadata (no scope parameter since quick 261106-kt1, so clients request the PRM scopes_supported). Update the evidence cell's pier.go line range to the current bearer() lines after Task 1 (find them with grep -n 'func (p \*Pier) bearer' modules/pier/pier.go and the function's closing brace). Keep the test column (TestPierTracerSmoke bad bearer, now also missing bearer) and the status verified. Change no other row.
Do NOT commit this file in the code commit (planning docs and code are separate commits). Leave it modified and list it in the SUMMARY under files for the orchestrator's quick-task docs commit.
</action>
<verify>
<automated>cd /media/nvme/dev/golem15/summercms.io/summercms/summercms.go && test "$(grep 'T-19.7-10' .planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md | grep -c '261106-kt1')" = 1 && ! grep 'T-19.7-10' .planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md | grep -q 'scope="read"' && NS=$(git diff --numstat -- .planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md) && test "$(printf '%s\n' "$NS" | awk '{print $1+$2}')" = 2</automated>
</verify>
<done>Row T-19.7-10 states the challenge names only resource_metadata, cites quick 261106-kt1 and the current bearer() line range; exactly one line changed in the file; the file is uncommitted and listed in the SUMMARY for the orchestrator's docs commit.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| MCP client -> pier `/mcp` | Untrusted bearer and request; every request re-verified by `auth.RequireBearerToken` with pier's resolver |
| MCP client -> OAuth authorization server | Client chooses which scopes to request; the server's allow-list and consent page decide the grant |
## STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|-----------|----------|-----------|----------|-------------|-----------------|
| T-261106-kt1-01 | Elevation of Privilege | 401 challenge without scope -> clients request the full PRM list (`write`, `ai`, `backend`, ...) | low | accept | The challenge is advisory; grants are unchanged server-side: wristband's authorize allow-list (T-19.7-09), granted = submitted ∩ requested with `backend` only with a link (T-19.7-33), and pier's per-request read/write ceiling and tool filter (`identity.go` `writes()`, T-19.7-13) still bound what a token reaches. |
| T-261106-kt1-02 | Spoofing | bearer() after wrapper removal | medium | mitigate | verify closure and RequireBearerToken options (ResourceMetadataURL, AllowMissingExpiration) unchanged; TestPierTracerSmoke pins 401 for a bad and a missing bearer. |
| T-261106-kt1-03 | Information Disclosure | WWW-Authenticate content | low | accept | Header names only the PRM URL, which is public; no scope or identity data added. |
| T-261106-kt1-04 | Denial of Service | streaming responses lose http.Flusher | low | mitigate | Wrapper removed rather than altered, so the SDK gets the original writer with its own Flusher; existing stream tests (TestEndpointGuards) keep running in `go test ./modules/pier/...`. |
| T-261106-kt1-SC | Tampering | npm/pip/cargo installs | low | accept | No package installs or dependency changes in this plan. |
</threat_model>
<verification>
- `go vet ./...` and `go test -short ./modules/pier/...` green.
- `go test ./cmd/summer -run TestDocsTree` and `go run ./cmd/summer docs:build --check` green.
- No `scope="read"` left in modules/pier, its README or docs/services/mcp.md; T-19.7-10 updated.
- `git show --stat HEAD` lists exactly the four Task 1 files; no lagoon file touched.
</verification>
<success_criteria>
A 401 from the MCP endpoint carries `WWW-Authenticate: Bearer resource_metadata="…"` with no scope parameter for both bad and missing bearers, pinned by TestPierTracerSmoke; docs, README and the security review agree.
</success_criteria>
<output>
Create `.planning/quick/261106-kt1-pier-mcp-401-challenge-drops-the-fixed-r/261106-kt1-SUMMARY.md` when done. List `.planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md` as a planning-doc change for the orchestrator's docs commit.
</output>

View File

@@ -0,0 +1,79 @@
---
phase: quick-261106-kt1
plan: 01
subsystem: pier
tags: [mcp, oauth, rfc9728, www-authenticate]
status: complete
requires: []
provides:
- "pier MCP 401 challenge carrying only resource_metadata (no scope parameter)"
affects:
- modules/pier
- docs/services/mcp.md
tech-stack:
added: []
patterns:
- "bearer() returns auth.RequireBearerToken(...)(next) directly; no ResponseWriter wrapper"
key-files:
created: []
modified:
- modules/pier/pier.go
- modules/pier/tracer_smoke_test.go
- modules/pier/README.md
- docs/services/mcp.md
- .planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md
decisions:
- "MCP 401 challenge names only resource_metadata; the MCP spec's server-side scope hint (SHOULD) is knowingly dropped so clients request the PRM scopes_supported and consent decides the grant (user decision, option 1)"
metrics:
duration: ~10m
completed: 2026-11-06
estimate:
tokens: 35000
tasks: 2
actuals:
tokens: 3500
tasks: 2
commits: 1
plan_head_before: ef47baf0a14379e37afe1b96fa2f6cf604e3337f
plan_head_after: 609caa4f3f1f24f123696ab1d8fd8b6bfcef113a
---
# Quick 261106-kt1: pier MCP 401 challenge drops the fixed read scope
pier's MCP endpoint now answers a bad or missing bearer with `401` and `WWW-Authenticate: Bearer resource_metadata="<app.url>/.well-known/oauth-protected-resource<mcp.path>"`. The header has no scope parameter. The response-writer wrapper that appended `scope="read"` is gone, so a client that follows the spec requests the PRM `scopes_supported` and can receive `write`.
## What changed
- **`modules/pier/pier.go`**: removed the `challengeWriter` type and its `WriteHeader`/`Flush`/`Unwrap` methods. `bearer()` now returns `auth.RequireBearerToken(verify, &auth.RequireBearerTokenOptions{ResourceMetadataURL: p.prmURL, AllowMissingExpiration: true})(next)` directly. The verify closure is unchanged and no `Scopes` option is set, so there is still no 403 insufficient-scope path. The SDK now receives the original writer, which keeps its own `http.Flusher`. The doc comment says why the challenge carries no scope. `strings` is still imported and used elsewhere in the file.
- **`modules/pier/tracer_smoke_test.go`**: the bad-bearer check is now a two-case loop (bad bearer, missing bearer). Each case asserts 401, the exact challenge `Bearer resource_metadata="http://acme.test/.well-known/oauth-protected-resource/mcp"`, and no `scope=` substring.
- **`modules/pier/README.md`** and **`docs/services/mcp.md`**: the header is quoted without a scope parameter, and each says the client therefore requests the scopes the metadata document advertises.
- **`.planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md`**: one line changed, row T-19.7-10. It now says the challenge names only `resource_metadata` "since quick 261106-kt1", the evidence cites `pier/pier.go:256-274`, and the test column reads "bad bearer, missing bearer". **Left uncommitted for the orchestrator's docs commit.**
## Commits
| Task | Commit | Files |
| ---- | ------ | ----- |
| 1 | 609caa4 `fix(pier): MCP 401 challenge names only resource_metadata` | modules/pier/pier.go, modules/pier/tracer_smoke_test.go, modules/pier/README.md, docs/services/mcp.md |
| 2 | (uncommitted planning doc) | .planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md |
## Verification
- `gofmt -l modules/pier/` is empty. `go vet ./modules/pier/...` and `go vet ./...` pass.
- `go test -short ./modules/pier/...` passes, and so does the full `go test ./modules/pier/...` (Docker-backed, 10.3s).
- `go test ./cmd/summer -run TestDocsTree` passes. `go run ./cmd/summer docs:build --check` reports no problems.
- No `challengeWriter` or `scope="read"` is left in the code, the pier README or docs/services/mcp.md. The only remaining mention is the historical 19.7-02-PLAN.md, which was intentionally not edited.
- The Task 2 check passes: the row cites 261106-kt1, no `scope="read"` remains, and the numstat is 1+1.
- Tracer gate: the `<verify>` chain was re-run end to end and passed before Task 2.
## Deviations from Plan
None. The plan was executed as written. The full Docker-backed pier test run and the repo-wide `go vet ./...` were extra checks beyond the plan.
## Threat Flags
None. No new surface was added. T-261106-kt1-02 (verify closure and options unchanged, 401 pinned for bad and missing bearer) and T-261106-kt1-04 (original writer kept, so Flusher is preserved; stream tests pass) are mitigated as planned.
## Self-Check: PASSED
- FOUND: modules/pier/pier.go, modules/pier/tracer_smoke_test.go, modules/pier/README.md, docs/services/mcp.md, .planning/phases/19.7-generic-mcp-module/19.7-SECURITY-REVIEW.md
- FOUND: commit 609caa4 (exactly the four Task 1 files; no lagoon/conga file touched)

View File

@@ -0,0 +1,222 @@
---
phase: quick-261106-kt2
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- modules/lagoon/commands.go
- modules/lagoon/purge_test.go
- modules/conga/schedule_test.go
- modules/conga/commands.go
- modules/conga/commands_test.go
autonomous: true
requirements:
- QUICK-261106-kt2
estimate:
tokens: 45000
raw_tokens: 45000
tasks: 2
confidence: low
must_haves:
truths:
- "When serve's in-process worker (or schedule:run) runs the framework entry summercms.lagoon[0]:deferred:purge, the command runs on the app's published database and succeeds; it no longer fails with `backpack: duplicate provider for *sql.DB`"
- "`summer deferred:purge` from the CLI (no database published) still opens, publishes and closes its own pool and purges exactly as before (TestPurgeDeferredCommand unchanged and green)"
- "A command run in-process never closes the shared pool it borrowed: after deferred:purge the published *sql.DB still answers PingContext and app.Lookup[*gorm.DB]() returns the same handle"
- "migrate, migrate:rollback and migrate:status share lagoon's withDB helper and therefore also work in-process with a published database"
- "queue:clear run in-process on an app with a published database (a plugin HasSchedule entry) clears the queue instead of failing with the duplicate-provider error"
artifacts:
- path: modules/lagoon/commands.go
provides: "withDB reuses the published *gorm.DB when present; otherwise the unchanged open -> Publish -> Close CLI path"
- path: modules/lagoon/purge_test.go
provides: "TestPurgeDeferredCommandPublishedDB: deferred:purge through bonfire.Call on an app whose pool is already published, real purge counts, pool still open afterwards"
- path: modules/conga/schedule_test.go
provides: "TestScheduledDeferredPurgeInProcess: callScheduled and schedule:run --once run deferred:purge from lagoon.RuntimeCommands on an app with a published pool"
- path: modules/conga/commands.go
provides: "queue:clear uses withAppDB"
- path: modules/conga/commands_test.go
provides: "TestQueueClear also clears through bonfire.Call on the published app"
key_links:
- from: modules/conga/scheduler.go runScheduled / callScheduled
to: modules/lagoon/commands.go deferred:purge Run -> withDB
via: "bonfire.Catalog.Call in-process; withDB must take app.Lookup[*gorm.DB]() instead of lagoon.Publish a second time"
- from: modules/lagoon/commands.go withDB
to: modules/backpack/services.go Registry.Publish
via: "the duplicate-provider error at services.go:36 is only reachable on the CLI-open branch, which now runs only when nothing is published"
---
<objective>
Fix the nightly framework schedule entry `deferred:purge`, which fails every night
in production since 2026-10-04 with
`level=ERROR msg="schedule: command failed" component=conga command=deferred:purge error="backpack: duplicate provider for *sql.DB"`
(framework v0.6.0; same code in v0.6.2). Root cause, verified by reading the code:
lagoon's unexported `withDB` (`modules/lagoon/commands.go:486`) always calls
`OpenFromApp` and then `lagoon.Publish`, which publishes `*sql.DB` on the app
(`modules/lagoon/connection.go:169`). From the CLI nothing is published yet, so it
works. Under serve the pool is already published (`modules/surf/serve.go:70-77`), and
serve's in-process worker runs the entry through `runScheduled` -> `callScheduled` ->
`bonfire.Catalog.Call` (`modules/conga/scheduler.go:148-182`), so the second Publish hits
`backpack: duplicate provider for *sql.DB` (`modules/backpack/services.go:36`). The same
happens under `schedule:run` (its `withDB` publishes first) and `schedule:run --once`
(`withAppDB` publishes first). Plugin commands such as `chat:prune-pending` succeed because
they take published services via `app.Lookup` and never publish a pool.
The fix follows the two guarded helpers that already exist: cabana's `withAdminDB`
(`modules/cabana/commands.go:218`) and conga's `withAppDB` (`modules/conga/commands.go:340`):
reuse the published handle when there is one, open/publish/close only for a CLI run.
Purpose: expired deferred bindings (abandoned uploads and records created under deferral)
get purged again by the scheduler; the CLI command keeps working unchanged.
Output: lagoon `withDB` fixed (deferred:purge plus the three migrate commands), conga
`queue:clear` switched to `withAppDB`, Docker-backed regression tests on both the lagoon
command and the conga scheduler path.
</objective>
<execution_context>
@~/.claude/msd-core/workflows/execute-plan.md
@~/.claude/msd-core/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@CLAUDE.md
Verified facts (planner read these; do not re-investigate):
- Framework-registered `pact.ScheduledCommand` entries: only `lagoon.FrameworkSchedule`
(`modules/lagoon/schedule.go:57`, `deferred:purge` daily at `database.deferred_bindings.purge_at`,
default "03:00" in `app.timezone`, default UTC per `appLocation` in `modules/conga/schedule.go:91`).
No other module returns a `ScheduledCommand`.
- Sweep of command paths that open and publish a pool (`lagoon.OpenFromApp` + `lagoon.Publish`
outside tests): lagoon `withDB` (migrate, migrate:rollback, migrate:status, deferred:purge:
UNGUARDED, the bug); conga `withDB` (queue:work, schedule:run foreground, queue:clear:
unguarded); cabana `withAdminDB` (guarded); surf serve (the pool owner, intentionally opens).
`lagoon.Publish` is the only place that publishes `*sql.DB`.
- `lagoon.OpenFromApp` also calls `loadEncryptionKeysFromApp` -> `PublishEncryptionKeys`, which is
idempotent (`modules/lagoon/encrypted.go:130`). On the reuse branch the keys were already loaded
by whoever opened the pool (serve); deferred:purge does not touch Encrypted columns.
- `purgeBucket` (`modules/lagoon/commands.go:468`) already prefers the published `*blob.Bucket`.
With no bucket and no `storage.uploads.bucket_url` it returns nil, and `PurgeDeferred` refuses only
when an expired binding points at a file.
- Under `bonfire.Call` `confirmProduction` never asks (`modules/lagoon/commands.go:32-50`).
- Test helpers that exist: lagoon `newPurgeEnv`, `purgeConfig`, `purgePlugin`, `dedicatedDB`
(`modules/lagoon/purge_test.go`, `postgres_test.go`; Docker, skip under -short via `lagoonDB`);
conga `migratedDB`, `testApp` (publishes db + GORM like serve), `commandApp`, `testAppKey`
(`modules/conga/postgres_test.go:113,154`, `listen_test.go:274,295`), `onceRun`, `mustTime`,
`captureHandler` with `find(level, msg, key, value)` (`modules/conga/schedule_test.go:38-70,259`).
- Docker tests: if the Docker socket is denied although the user is in the docker group, run
them through `sudo -u golem` (CLAUDE.md).
- Concurrency: quick task 261106-kt1 is editing `modules/pier` and `docs/services/mcp.md`; do not
touch those. Stage files explicitly by path; leave `.planning/STATE.md`, `.planning/state.json`,
`.planning/G15OFFICE.md` and `examples/hello/plugins.gen.go` (other sessions' edits) alone.
- No README/docs change is expected: `withDB` and `withAppDB` are unexported, and the commands'
flags, output, config keys and CLI behaviour are unchanged. A grep of `modules/lagoon/README.md`,
`modules/conga/README.md`, `docs/plugins/scheduling.md` and `docs/console/` found no statement that
these commands always open their own connection.
@modules/lagoon/commands.go
@modules/conga/commands.go
@modules/conga/scheduler.go
</context>
<tasks>
<task type="tracer" tdd="true">
<name>Task 1: Tracer - scheduler -> catalog -> deferred:purge runs on the app's published pool (test first, then fix lagoon withDB)</name>
<files>modules/lagoon/purge_test.go, modules/conga/schedule_test.go, modules/lagoon/commands.go</files>
<read_first>
- modules/lagoon/commands.go (lines 304-366 deferred:purge Run, 464-496 purgeBucket and withDB)
- modules/lagoon/purge_test.go (lines 49-73 newPurgeEnv, 386-400 purgeConfig, 444-503 purgePlugin and TestPurgeDeferredCommand)
- modules/conga/scheduler.go (lines 145-200 runScheduled, callScheduled, scheduledCatalog)
- modules/conga/schedule_test.go (lines 38-70 captureHandler, 259-310 onceRun and onceApp)
- modules/conga/postgres_test.go (lines 111-176 migratedDB and testApp)
- modules/cabana/commands.go (lines 218-234 withAdminDB, the pattern to mirror)
</read_first>
<behavior>
- lagoon TestPurgeDeferredCommandPublishedDB: an app whose pool was opened with OpenFromApp and published with Publish (as serve does) runs deferred:purge --days 2 through bonfire.Call and gets nil error and "purged 2 deferred bindings older than 2 days (1 files, 1 children, 0 skipped)"; afterwards the published *sql.DB still pings and app.Lookup[*gorm.DB]() is the same handle; migrate:status through bonfire.Call on the same app also returns nil.
- conga TestScheduledDeferredPurgeInProcess: with testApp's published pool and a published bonfire.NewCatalog(lagoon.RuntimeCommands(app, nil)), callScheduled(ctx, app, log, "deferred:purge", nil, &buf) returns skipped false and nil error, buf contains "purged 0 deferred bindings older than 5 days", the capture handler has an Info "schedule: command finished" record with command=deferred:purge, the pool still pings; schedule:run --once at 2026-03-10 03:00:00 UTC returns nil and prints "Ran 1 due entries (0 failed)".
- Before the fix both tests fail with an error containing "backpack: duplicate provider for *sql.DB" (the production error).
- Existing TestPurgeDeferredCommand (CLI path through bonfire.NewRoot, nothing published) stays green unchanged.
</behavior>
<action>
RED. In modules/lagoon/purge_test.go, directly after TestPurgeDeferredCommand, add TestPurgeDeferredCommandPublishedDB with a doc comment naming the bug (the in-process scheduler runs deferred:purge on an app whose pool serve already published). Build the environment the way TestPurgeDeferredCommand does: newPurgeEnv(t, "lagoon_purge_published"), one unattached file via e.file(t, ""), one created child via e.child, and two bindings backdated 3 days via e.bind (DeferredFileType for the file, "acme_purge_children" with DeferredEnvelope{Created: true} for the child). Create the app with backpack.New(purgeConfig(t, ...)) carrying database.dsn = e.dsn and app.key = base64 of 32 bytes of 7 (same as the existing test; app.key is required so that, before the fix, the failure is the duplicate-provider error and not the app.key error). Open the pool exactly as serve does: OpenFromApp(t.Context(), app), register t.Cleanup closing that sqlDB, then Publish(app, sqlDB, gdb); then attach.Publish(app, e.bucket). Run the command the way bonfire.Catalog.Call does for the scheduler: bonfire.Call(t.Context(), RuntimeCommands(app, plugins), "deferred:purge", []string{"--days", "2"}, &out) with plugins = a purgePlugin listing &purgeChild{}. Assert nil error, the output line above, that e.fileExists/e.blobExists/e.bindingIDs show the file, its blob and both bindings gone, that sqlDB.PingContext(t.Context()) still succeeds, and that app.Lookup[*gorm.DB]() returns the same gdb pointer. Then call bonfire.Call on the same app for "migrate:status" and assert nil error (the migrate commands share the helper).
In modules/conga/schedule_test.go add TestScheduledDeferredPurgeInProcess (Docker-backed through migratedDB, so it skips under -short). Use db, dsn := migratedDB(t) and app, _ := testApp(t, db, dsn, map with "app.key": testAppKey) — testApp publishes db and GORM on the app like serve. Publish slog.New over a fresh captureHandler on the app (as onceApp does), then publish bonfire.NewCatalog(lagoon.RuntimeCommands(app, nil)), the catalog the application binary publishes. Call callScheduled(t.Context(), app, loggerFromApp(app), "deferred:purge", nil, &buf) — the exact call runScheduled makes in serve's worker — and assert the behaviour above (skipped false, nil error, the "purged 0 deferred bindings older than 5 days" text in buf, the Info record via captureHandler.find, db.PingContext ok). Then run onceRun(t, app, nil, mustTime(t, time.UTC, "2026-03-10 03:00:00")) — the framework entry is due at the default 03:00 UTC — and assert nil error and "Ran 1 due entries (0 failed)" in the output. Add the lagoon import if schedule_test.go does not have it yet.
Run both new tests (commands in verify) BEFORE touching production code and confirm each fails with an error containing "backpack: duplicate provider for *sql.DB". Copy the failing lines into the SUMMARY as the RED evidence. Do not commit the failing tests on their own: CLAUDE.md requires go test ./... green at every commit, so tests and fix land in one commit.
GREEN. In modules/lagoon/commands.go change withDB: when app is non-nil and app.Lookup[*gorm.DB]() returns ok with a non-nil handle, call fn with that handle and return — no OpenFromApp, no Publish, and no Close, because the pool belongs to whoever published it (serve, schedule:run, queue:work, a test). Otherwise keep the current branch byte-for-byte: OpenFromApp, defer sqlDB.Close(), Publish, fn — so `summer deferred:purge` and the migrate commands from the CLI behave exactly as today. Mirror cabana's withAdminDB. Rewrite the doc comment of withDB in the repository's style: it runs fn on the app's published database when there is one (in-process runs: the scheduler in serve or schedule:run, bonfire.Catalog.Call), and otherwise opens and publishes a pool for the CLI run and closes it when fn returns. Do not change deferred:purge's Run body, purgeBucket, flags, output or Long text. This one change fixes deferred:purge and, through the shared helper, migrate, migrate:rollback and migrate:status.
Re-run: both new tests plus the existing TestPurgeDeferredCommand pass. Commit with message "fix(lagoon): run deferred:purge and the migrate commands on the app's published database (quick-261106-kt2)" and a body of two or three lines stating the production symptom and that the CLI path is unchanged. Stage exactly modules/lagoon/commands.go, modules/lagoon/purge_test.go and modules/conga/schedule_test.go by path. No Co-Authored-By trailer (user rule).
</action>
<verify>
<automated>cd /media/nvme/dev/golem15/summercms.io/summercms/summercms.go && go vet ./modules/lagoon/... ./modules/conga/... && L=$(go test ./modules/lagoon -run 'TestPurgeDeferredCommand' -count=1 -v 2>&1) && echo "$L" | grep -q -- '--- PASS: TestPurgeDeferredCommand ' && echo "$L" | grep -q -- '--- PASS: TestPurgeDeferredCommandPublishedDB' && C=$(go test ./modules/conga -run 'TestScheduledDeferredPurgeInProcess' -count=1 -v 2>&1) && echo "$C" | grep -q -- '--- PASS: TestScheduledDeferredPurgeInProcess' && grep -n 'Lookup\[\*gorm.DB\]' modules/lagoon/commands.go</automated>
</verify>
<done>RED evidence (duplicate provider for *sql.DB from both new tests before the fix) is recorded; after the fix TestPurgeDeferredCommand, TestPurgeDeferredCommandPublishedDB and TestScheduledDeferredPurgeInProcess pass against testcontainers Postgres; lagoon withDB reuses a published *gorm.DB and never closes it; the CLI branch is unchanged; one commit with the three files.</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Same fix for conga queue:clear, sweep record and full checks</name>
<files>modules/conga/commands_test.go, modules/conga/commands.go</files>
<read_first>
- modules/conga/commands.go (lines 94-127 queue:clear, 338-345 withAppDB, 399-411 withDB)
- modules/conga/commands_test.go (lines 20-85 TestQueueClear)
</read_first>
<behavior>
- TestQueueClear additionally clears through bonfire.Call on the published app: while the blocking job still holds the single default worker slot, two more pending ping jobs are enqueued, and queue:clear via bonfire.Call(ctx, RuntimeCommands(app, nil), "queue:clear", nil, &out) returns nil with "Cleared 2 jobs" in the output; the existing final assertions (block completes, zero ping rows left, zero pings ran) still hold.
- Before the fix that new step fails with "backpack: duplicate provider for *sql.DB".
- The existing CLI-style runClear (commandApp, nothing published) is unchanged and still passes.
</behavior>
<action>
RED. In modules/conga/commands_test.go extend TestQueueClear: after the existing runClear assertions and BEFORE close(release) (the blocking job keeps the one default worker slot busy, so new pings stay pending), enqueue two more pingArgs jobs with m.Enqueue as the test already does (use a different Tag, for example "pending-in-process"), then run queue:clear through bonfire.Call(ctx, RuntimeCommands(app, nil), "queue:clear", nil, &out) on the test's own published `app` — the in-process path a plugin pact.HasSchedule entry naming queue:clear would take. Assert nil error and "Cleared 2 jobs" in the output. Leave the existing runClear (commandApp, the CLI path) and the final assertions as they are. Run it and confirm the new step fails with "backpack: duplicate provider for *sql.DB"; record it in the SUMMARY.
GREEN. In modules/conga/commands.go change only queue:clear's Run to call withAppDB instead of withDB. Leave queue:work and schedule:run (foreground) on withDB: both are long-running processes started from the CLI that own their pool, and running either as an in-process scheduled entry is not meaningful. Extend withAppDB's doc comment to say it serves schedule:run --once and queue:clear, which may run in-process on an app that already published its pool. Do not change flags, output, the confirmation question or the Long text.
Sweep record for the SUMMARY (no code change beyond this task): framework ScheduledCommand registrations are only lagoon.FrameworkSchedule's deferred:purge; remaining unguarded OpenFromApp + Publish paths are conga withDB for queue:work and schedule:run (CLI pool owners, intentionally) and surf serve (the pool owner); cabana withAdminDB was already guarded. Confirm with grep -rn 'lagoon.OpenFromApp' modules --include=*.go excluding _test.go.
Full checks (commands in verify): go vet ./..., go test -short ./... (CLAUDE.md keeps it green at every commit), then the Docker-backed go test ./modules/lagoon/... ./modules/conga/... -count=1. If the Docker socket is denied, rerun the Docker step through sudo -u golem per CLAUDE.md.
Commit with message "fix(conga): queue:clear reuses the app's published database (quick-261106-kt2)", staging exactly modules/conga/commands.go and modules/conga/commands_test.go by path. No Co-Authored-By trailer. Do not push or tag.
</action>
<verify>
<automated>cd /media/nvme/dev/golem15/summercms.io/summercms/summercms.go && go vet ./... && go test -short ./... && Q=$(go test ./modules/conga -run 'TestQueueClear$' -count=1 -v 2>&1) && echo "$Q" | grep -q -- '--- PASS: TestQueueClear ' && grep -B1 'return clearQueue(ctx, app, queue, out)' modules/conga/commands.go | grep -q withAppDB && go test ./modules/lagoon/... ./modules/conga/... -count=1</automated>
</verify>
<done>queue:clear uses withAppDB; TestQueueClear passes both its CLI run and the new in-process run; go vet ./..., go test -short ./... and the Docker-backed lagoon and conga suites are green; sweep result recorded; one commit with the two conga files; nothing under modules/pier or docs/services/mcp.md touched.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| river_job rows -> scheduled command runner | A scheduled job row names an entry and command; runScheduled runs only an exact match of the compiled schedule table (T-11-09, unchanged) |
| in-process command -> app's shared database pool | A command run through bonfire.Catalog.Call borrows the pool serve owns and must not close or replace it |
## STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|-----------|----------|-----------|----------|-------------|-----------------|
| T-261106-kt2-01 | Denial of Service | lagoon withDB / conga withAppDB reuse branch closing the borrowed pool would take serve's database away | high | mitigate | Reuse branch calls fn only, with no defer Close and no Publish; TestPurgeDeferredCommandPublishedDB and TestScheduledDeferredPurgeInProcess assert the published *sql.DB still pings and the same *gorm.DB stays published after the command |
| T-261106-kt2-02 | Information Disclosure | deferred:purge failing nightly keeps abandoned pending uploads (files and records) past the configured retention (default 5 days) | medium | mitigate | The fix itself plus the scheduler-path regression test that runs the framework entry through callScheduled and schedule:run --once |
| T-261106-kt2-03 | Elevation of Privilege | scheduler executing an arbitrary command from a forged river_job row | low | accept | Unchanged: runScheduled matches entry, command and args against the compiled table (T-11-09, covered by TestScheduledEntryMismatchSkipped); this plan does not touch scheduler.go |
| T-261106-kt2-04 | Tampering | CLI branch regressions (migrate, deferred:purge from the terminal) | low | mitigate | CLI branch kept byte-for-byte; existing TestPurgeDeferredCommand (bonfire.NewRoot, nothing published) and TestQueueClear's runClear (commandApp) keep covering it |
| T-261106-kt2-SC | Tampering | npm/pip/cargo installs | low | accept | No package installs or dependency changes in this plan |
</threat_model>
<verification>
- `go vet ./...` and `go test -short ./...` green.
- Docker: `go test ./modules/lagoon/... ./modules/conga/... -count=1` green, including TestPurgeDeferredCommand, TestPurgeDeferredCommandPublishedDB, TestScheduledDeferredPurgeInProcess and TestQueueClear.
- SUMMARY holds the RED output (duplicate provider for *sql.DB) for the three new test steps and the sweep result.
- `git show --stat HEAD~1 HEAD` lists exactly the three Task 1 files and the two Task 2 files; nothing under modules/pier, docs/ or .planning/ in the code commits.
</verification>
<success_criteria>
The framework schedule entry deferred:purge succeeds when serve's worker, schedule:run or schedule:run --once runs it on an app whose database is already published, without closing that pool; `summer deferred:purge` and the migrate commands from the CLI behave as before; queue:clear works in-process too; all pinned by Docker-backed tests that failed with the production error before the fix.
</success_criteria>
<output>
Create `.planning/quick/261106-kt2-lagoon-deferred-purge-fails-in-the-sched/261106-kt2-SUMMARY.md` when done, with the RED evidence, the sweep result, the two commit hashes, and a note that no README/docs change was needed (unexported helpers, command behaviour from the CLI unchanged). Note for the user: the fix reaches production only with the next framework patch tag and an app bump; do not tag or push here.
</output>

View File

@@ -0,0 +1,106 @@
---
phase: quick-261106-kt2
plan: 01
subsystem: lagoon, conga
tags: [scheduler, deferred-bindings, database-pool, bugfix]
status: complete
requires: []
provides:
- "lagoon withDB reuses the published *gorm.DB (deferred:purge, migrate, migrate:rollback, migrate:status work in-process)"
- "conga queue:clear runs through withAppDB"
affects: [conga scheduler, serve in-process worker, schedule:run]
tech-stack:
added: []
patterns: ["reuse the published handle, open/publish/close only for a CLI run (same as cabana withAdminDB, conga withAppDB)"]
key-files:
created: []
modified:
- modules/lagoon/commands.go
- modules/lagoon/purge_test.go
- modules/conga/schedule_test.go
- modules/conga/commands.go
- modules/conga/commands_test.go
decisions:
- "lagoon withDB borrows the published *gorm.DB without Publish or Close; the pool belongs to whoever published it"
- "queue:work and foreground schedule:run stay on conga withDB: long-running CLI processes that own their pool"
metrics:
duration: ~15min
completed: 2026-11-06
actuals:
tokens: 2070
tasks: 2
commits: 3
plan_head_before: 609caa4f3f1f24f123696ab1d8fd8b6bfcef113a
plan_head_after: 99abe88b6a33275817ae3f90d33b2f8a50717a2f
---
# Quick 261106-kt2: deferred:purge fails in the scheduler, Summary
The nightly framework entry `deferred:purge` now runs on the app's published database pool under serve's in-process worker, `schedule:run` and `schedule:run --once`. It no longer fails with `backpack: duplicate provider for *sql.DB`. `queue:clear` got the same fix.
## What changed
- `modules/lagoon/commands.go` `withDB`: when the app already has a published `*gorm.DB`, `fn` runs on that handle. There is no `OpenFromApp`, no `Publish` and no `Close`. Otherwise the CLI branch (open, defer Close, Publish, fn) is unchanged. This fixes `deferred:purge`, `migrate`, `migrate:rollback` and `migrate:status`. It mirrors cabana's `withAdminDB`.
- `modules/conga/commands.go`: `queue:clear` calls `withAppDB` instead of `withDB`, and the `withAppDB` doc comment now names schedule:run --once and queue:clear.
## Commits
| Task | Commit | Message |
|------|--------|---------|
| 1 | 16a2008 | fix(lagoon): run deferred:purge and the migrate commands on the app's published database (quick-261106-kt2) |
| 2 | 99abe88 | fix(conga): queue:clear reuses the app's published database (quick-261106-kt2) |
`actuals.commits` reads 3 because the ledger range `609caa4..HEAD` also holds `8ebfedf`, a docs commit from the concurrent quick task 261106-kt1. This task made 2 commits.
## RED evidence (before the fix)
```
=== RUN TestPurgeDeferredCommandPublishedDB
purge_test.go:532: deferred:purge: backpack: duplicate provider for *sql.DB
--- FAIL: TestPurgeDeferredCommandPublishedDB (0.24s)
schedule_test.go:275: callScheduled: skipped=false err=conga: scheduled command deferred:purge: backpack: duplicate provider for *sql.DB
--- FAIL: TestScheduledDeferredPurgeInProcess (0.28s)
commands_test.go:86: in-process queue:clear: backpack: duplicate provider for *sql.DB
--- FAIL: TestQueueClear (10.38s)
```
TestPurgeDeferredCommand (the CLI path) passed both before and after the fix.
## GREEN / verification
- `go vet ./...`: ok
- `go test -short ./...`: ok
- Docker: `go test ./modules/lagoon/... ./modules/conga/... -count=1`: all ok (lagoon 22.2s, lagoon/attach 40.8s, lagoon/attach/webp, conga 31.5s). Includes TestPurgeDeferredCommand, TestPurgeDeferredCommandPublishedDB, TestScheduledDeferredPurgeInProcess and TestQueueClear.
- The new tests check that the published `*sql.DB` still answers PingContext after the command and that `app.Lookup[*gorm.DB]()` returns the same handle (T-261106-kt2-01).
## Sweep result
`grep -rn 'OpenFromApp(' modules --include=*.go` (non-test):
- `modules/lagoon/commands.go` withDB: now guarded (this task)
- `modules/cabana/commands.go` withAdminDB: already guarded
- `modules/conga/commands.go` withDB: used only by queue:work and foreground schedule:run. Both are long-running CLI processes that own their pool, so this is intentional. queue:clear moved to withAppDB.
- `modules/surf/serve.go`: serve owns the pool, so it opens one on purpose.
The only framework `ScheduledCommand` registration is `lagoon.FrameworkSchedule` (`deferred:purge`).
## Deviations from Plan
**1. [Rule 3 - Blocking] TestQueueClear's app needed `app.key`**
- **Found during:** Task 2 RED
- **Issue:** testApp in TestQueueClear had no `app.key`, so the in-process queue:clear first failed with `lagoon: app.key is empty or invalid` rather than the production error.
- **Fix:** added `"app.key": testAppKey` to that test's testApp config. The RED run then showed `backpack: duplicate provider for *sql.DB`. The CLI-style runClear and the final assertions are unchanged.
- **Files modified:** modules/conga/commands_test.go
- **Commit:** 99abe88
## Notes
- No README or docs change was needed: `withDB` and `withAppDB` are unexported, and the commands' flags, output, config keys and CLI behaviour are unchanged.
- The fix reaches production only with the next framework patch tag and an app bump. Nothing was tagged or pushed.
- No files under modules/pier or docs/services/mcp.md were touched.
## Self-Check: PASSED
- FOUND: modules/lagoon/commands.go, modules/lagoon/purge_test.go, modules/conga/schedule_test.go, modules/conga/commands.go, modules/conga/commands_test.go
- FOUND: 16a2008, 99abe88

View File

@@ -0,0 +1,566 @@
---
phase: quick-261108-pe4
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- modules/beachcomber/searchable.go
- modules/beachcomber/engines.go
- modules/beachcomber/searchpage_test.go
- modules/beachcomber/typesense/engine.go
- modules/beachcomber/typesense/highlights.go
- modules/beachcomber/typesense/highlights_test.go
- modules/beachcomber/typesense/example_test.go
- modules/beachcomber/README.md
- docs/services/search.md
autonomous: true
requirements:
- QUICK-261108-pe4
estimate:
tokens: 55000
raw_tokens: 55000
tasks: 3
confidence: low
must_haves:
truths:
- "A beachcomber.Query that sets none of IncludeFields, ExcludeFields or Highlight makes the Typesense engine send exactly today's parameters (q, query_by, query_by_weights, filter_by, sort_by, page, per_page as set) and return the same IDs and Found as before"
- "With Query.Highlight set, typesense SearchPage returns SearchResult.Hits with one beachcomber.Hit per id in IDs order (Hits[i].ID == IDs[i]), including hits that have no highlights; a string field gives one Snippet with ArrayIndex -1, a string[] field gives one Snippet per snippets[] entry with ArrayIndex taken from indices, capped client-side by MaxSnippetsPerArray"
- "Segment.Text is the stored text byte for byte: a stored <b> or &amp; comes back unchanged, and a match is marked only by Segment.Match, never by markup"
- "A field named in ExcludeFields is missing from the returned document yet still highlighted when Highlight asks for it; include_fields always carries id; ExcludeFields naming id, or a negative AffixTokens or MaxSnippetsPerArray, is an error before any request"
- "beachcomber.SupportsHighlights is true for the Typesense engine and false for the null engine, engines without HighlightSearcher and nil; beachcomber.SearchPage always returns Hits in step with IDs, filling empty hits when an engine leaves Hits nil"
- "modules/beachcomber/README.md has a Highlights section next to the Candidates, not answers rule, and docs/services/search.md a Highlights section. Both say a caller drops the Hit of every id that fails the SQL re-check before showing a snippet, and that Segment text is escaped when rendered. summer docs:build --check and TestDocsTree pass"
artifacts:
- path: modules/beachcomber/searchable.go
provides: "Query.IncludeFields/ExcludeFields/Highlight, Highlight, SearchResult.Hits, Hit, Snippet, Segment, HighlightSearcher, SupportsHighlights, SearchPage filling Hits"
- path: modules/beachcomber/typesense/highlights.go
provides: "private-use highlight tags, highlights-array parsing (snippet / snippets + indices, client-side per-array cap), splitting into segments"
- path: modules/beachcomber/typesense/engine.go
provides: "include_fields (with id), exclude_fields, highlight_fields, highlight_affix_num_tokens and the tags sent only when set; Hits built in the id loop; SupportsHighlights; HighlightSearcher assertion"
- path: modules/beachcomber/typesense/highlights_test.go
provides: "httptest fake Typesense tests for every case in spec item 5"
- path: modules/beachcomber/typesense/example_test.go
provides: "ExampleEngine_SearchPage: excluded field highlighted, SQL re-check stand-in drops a hit, segments escaped on render"
- path: modules/beachcomber/README.md
provides: "Highlights section, Features bullets, API reference rows"
- path: docs/services/search.md
provides: "Highlights section with the src= example, Typesense paragraph on the new parameters"
key_links:
- from: modules/beachcomber/typesense/engine.go search() id loop
to: beachcomber.SearchResult.Hits
via: "a Hit is appended in the same branch that appends an id, so hits whose id is null or missing are left out of both and Hits stays in step with IDs"
- from: beachcomber.Query.Highlight == nil
to: typesense highlight parsing
via: "highlights are parsed only when Highlight is set, because only then are the tags the private-use characters; Typesense's default <mark> snippets are never read as text"
- from: beachcomber.SupportsHighlights
to: beachcomber.HighlightSearcher (typesense.Engine.SupportsHighlights)
via: "type assertion plus the method's answer; apps fall back to their own snippets when false"
---
<objective>
Make beachcomber deliver search highlights and let callers shrink the answer, additively,
for framework v0.6.4. This quick task does not tag, push or bump a version.
The problem: `beachcomber.SearchResult` has only `IDs` and `Found`, and the Typesense engine
reads only `found` and `hits[].document.id`. Typesense computes highlights and beachcomber
drops them. Every hit also comes back as its full document, so a 250-hit page with indexed
PDF text goes over `maxResponseBytes` (32 MiB). The only way round that today is
`store:false`, and a field that is not stored cannot be highlighted.
The fix:
- New `Query.IncludeFields` / `ExcludeFields` / `Highlight`.
- New `SearchResult.Hits` (`Hit` / `Snippet` / `Segment`): plain stored text split into
matched and unmatched segments, never HTML.
- Typesense sends the new parameters only when they are set and parses the `highlights`
array of each hit.
- An optional `HighlightSearcher` capability, checked with `beachcomber.SupportsHighlights`.
- Tests against an httptest fake Typesense.
- README and docs, including the privacy rule: highlights belong to candidates.
Every new field left empty keeps today's behaviour. `Engine`, `Query` and `SearchResult`
keep all their existing fields and methods, so existing callers and third-party engines
compile and behave as before. Consumers use keyed literals only; a grep of fonoteka.go,
sm-*-app and sm-*-plugin found no unkeyed `SearchResult{` or `Query{`.
Out of scope, so do not plan or implement them: the per-index gate (Gate.Enabled per
index), which is a separate change; release tagging, which needs the user's go-ahead;
adoption in fonoteka.go or other consumers.
Source coverage (full user spec):
| Spec item | Covered by |
|-----------|------------|
| 1. Query.IncludeFields / ExcludeFields; Highlight{Fields, AffixTokens, MaxSnippetsPerArray}; no highlight_full_fields | Task 1 (types, highlight params), Task 2 (projection params, validation) |
| 2. SearchResult.Hits, Hit, Snippet (ArrayIndex -1 for non-array), Segment{Text, Match} plain text | Task 1 |
| 3. Typesense: params only when set; private-use tags; split snippet/snippets[]; highlights[] with indices; 32 MiB cap kept | Task 1 (string snippets, tags), Task 2 (snippets[] + indices, cap), maxResponseBytes untouched |
| 4. HighlightSearcher + beachcomber.SupportsHighlights; Typesense true, null/third-party false | Task 2 |
| 5. httptest tests: no new params; string and string[]; <b>/&amp; unchanged; hit without highlights keeps step; excluded field highlighted | Task 1 (4 tests), Task 2 (string[], excluded field, capability) |
| 6. README Highlights section next to "Candidates, not answers" + privacy rule | Task 3 (README + docs/services/search.md) |
Choices I made where the spec leaves room (planner's discretion):
- `HighlightSearcher` embeds `PageSearcher` and adds `SupportsHighlights() bool`. Highlights
travel only in `SearchResult`, which only `SearchPage` returns, and a bool method lets a
wrapping engine answer at runtime.
- `beachcomber.SearchPage` fills `Hits` (one empty `Hit` per id) when an engine leaves it
nil, so `Hits` is always in step with `IDs` through the helper.
- `include_fields` always gains `id`, and an `ExcludeFields` that names `id` is an error
before any request. The ids are beachcomber's result, and the Typesense docs make no
promise about `id` under `include_fields`.
- `MaxSnippetsPerArray` is applied client-side, in Typesense's order. The Typesense 28.0
search parameters were checked on 2026-11-08: highlight_fields, highlight_full_fields,
highlight_affix_num_tokens, highlight_start_tag, highlight_end_tag, snippet_threshold and
enable_highlight_v1. None of them caps snippets per array.
Output: one code commit `feat(beachcomber): search highlights and field projection (quick-261108-pe4)`.
</objective>
<execution_context>
@~/.claude/msd-core/workflows/execute-plan.md
@~/.claude/msd-core/templates/summary.md
</execution_context>
<context>
@.planning/STATE.md
@CLAUDE.md
Facts gathered while planning (verified by reading the code):
- `modules/beachcomber/searchable.go`:
- holds `Engine`, `Query`, `SearchResult`, `PageSearcher`, `SearchPage`, `IndexDropper`,
`DropIndex`, `IndexEnsurer`, `EnsureIndex` and `Gate`;
- optional capabilities follow one pattern: an interface plus a package-level helper that
type-asserts it;
- `SearchPage` falls back to `SearchIDs` with `Found = len(ids)`.
- `modules/beachcomber/engines.go`: `nullEngine.SearchPage` returns `SearchResult{IDs: []string{}}`.
- `modules/beachcomber/typesense/engine.go`:
- `search()` validates before any request: a `query_by_weights` length mismatch, and
`PerPage > MaxPerPage`. Both are errors formatted as `typesense: search %s: ...`;
- `search()` builds `url.Values`, decodes `searchAnswer` (Found plus `hits[].document.id`
as `json.RawMessage`), and leaves out hits whose id is empty or null. An id that is not
a JSON string is kept as its raw text (`7`);
- the existing var block asserts `beachcomber.Engine`, `IndexDropper` and `IndexEnsurer`;
- `maxResponseBytes = 32 << 20` stays as it is.
- `modules/beachcomber/typesense/engine_test.go` holds the shared fake: `newFake(t)` returns
`(*fakeTypesense, *Engine)` with base path `/ts`. `f.answer("GET /ts/collections/<idx>/documents/search", 200, body)`
sets an answer, and `f.take()` returns the calls (`rawQuery` and the others). The new
tests reuse it from the same package `typesense`.
- `modules/beachcomber/searchpage_test.go`:
- test engines `idsOnlyEngine`, `pagedEngine`, `flushEngine` and `dropperEngine`;
- `TestSearchPageFallsBackToSearchIDs` compares the whole result with `reflect.DeepEqual`.
Its expected value gains `Hits` in Task 2.
- Docs checker:
- `internal/docsite` checks every backticked `pkg.Ident` and `pkg.Type.Member` in module
READMEs and docs pages against the non-test Go files under `modules/` (fields, methods
and promoted members);
- `src=path#ExampleName` blocks must match their source, and `summer docs:sync` rewrites
them;
- baseline `go run ./cmd/summer docs:build --check` printed "docs:build: no problems
found" on 2026-11-08.
- Concurrency:
- other sessions have uncommitted edits in `.planning/STATE.md`, `.planning/state.json`,
`.planning/G15OFFICE.md`, `examples/hello/plugins.gen.go`, `.msd/` and some phase
`.gitkeep` and SUMMARY files. Leave them alone;
- `modules/beachcomber` and `docs/` were clean when this plan was written;
- stage by explicit path, never `git add -A`.
- Commits: no co-author tag (user's global CLAUDE.md: never add co-author tags). One code
commit for the plan, because CLAUDE.md requires an exported API change and its README and
docs updates to land in the same commit, with `go vet` and `go test` green at every commit.
@modules/beachcomber/searchable.go
@modules/beachcomber/engines.go
@modules/beachcomber/typesense/engine.go
@modules/beachcomber/typesense/engine_test.go
@modules/beachcomber/typesense/searchpage_test.go
@modules/beachcomber/searchpage_test.go
</context>
<tasks>
<task type="tracer" tdd="true">
<name>Task 1: Tracer, one string-field highlight from Query.Highlight through the Typesense wire to SearchResult.Hits</name>
<files>modules/beachcomber/searchable.go, modules/beachcomber/typesense/highlights.go, modules/beachcomber/typesense/engine.go, modules/beachcomber/typesense/highlights_test.go</files>
<read_first>modules/beachcomber/searchable.go, modules/beachcomber/typesense/engine.go (search, searchAnswer), modules/beachcomber/typesense/engine_test.go (newFake, answer, take), modules/beachcomber/typesense/searchpage_test.go</read_first>
<behavior>
- TestTypesenseSearchSendsNoNewParamsByDefault:
- setup: a Query with every existing field set, as in TestTypesenseSearchPageFoundAndWeights.
The fake answer carries highlights with Typesense's default mark tags;
- the parsed query keys are exactly q, query_by, query_by_weights, filter_by, sort_by,
page and per_page. No include_fields, exclude_fields, or any key starting with highlight_;
- IDs and Found are as before;
- Hits has one Hit per id with zero Highlights. The mark-tagged snippets are not read,
because Highlight is nil.
- TestTypesenseHighlightsStringSnippet:
- Query{Q: "blue", QueryBy: [title], Highlight: &Highlight{Fields: [title], AffixTokens: 6}}
sends highlight_fields=title, highlight_affix_num_tokens=6, highlight_start_tag
U+E000 and highlight_end_tag U+E001. params.Has("highlight_full_fields") is false;
- an answer hit {"document":{"id":"5"},"highlights":[{"field":"title","snippet":"a blue train","matched_tokens":["blue"]}]}
gives Hits == [{ID "5", Highlights [{Field "title", ArrayIndex -1, Segments [{"a ", false}, {"blue", true}, {" train", false}]}]}]
(reflect.DeepEqual);
- Highlight: &Highlight{} with zero values sends the two tags and nothing else new.
- TestTypesenseHighlightsKeepStoredText:
- snippet "<b>bold</b> &amp; blue &lt;x" gives Segment texts exactly
"<b>bold</b> &amp; ", "blue" and " &lt;x". Also check the same snippet with JSON
escapes <b> in the fixture;
- no Segment text contains "mark", and nothing is unescaped or escaped.
- TestTypesenseHitsStayInStepWithIDs:
- hits, in order: id "12" with a title highlight; id null; no id; id 7 with no
highlights key; id "3" with "highlights":[]. The result is IDs [12 7 3], len(Hits) 3
and Hits[i].ID == IDs[i]. Hits[0] has 1 snippet; Hits[1] and Hits[2] have none;
- a highlights entry that does not decode (snippet given as a number) drops only that
entry, and the search returns no error;
- an answer {"found":0,"hits":[]} with Highlight set gives a non-nil, empty Hits.
</behavior>
<action>
RED first: write modules/beachcomber/typesense/highlights_test.go (package typesense,
reusing newFake, answer and take from engine_test.go) with the four tests in behavior.
Run them and record the failure in the SUMMARY: a compile failure on the missing types
is the expected RED. Do not commit anything in this task. The plan lands as one commit
in Task 3.
In modules/beachcomber/searchable.go (spec items 1 and 2):
- Add three fields to Query, after PerPage:
- IncludeFields []string and ExcludeFields []string, with one doc comment. They only
limit which document fields the engine sends back. A field left out of the answer
stays searchable and highlightable. Empty keeps the engine default (every stored
field). The document key is always returned, because the ids are the result.
Engines that cannot project ignore them.
- Highlight *Highlight. Nil asks for no highlights and keeps today's request. Engines
without HighlightSearcher ignore it.
- Add `type Highlight struct`:
- Fields []string: the fields to highlight. Empty is the engine default, which is the
QueryBy fields for Typesense;
- AffixTokens int: tokens of context on each side of a match. 0 is the engine default;
- MaxSnippetsPerArray int: the most snippets kept for one array field. 0 means no cap.
- No option for highlighting a whole field without snippeting. Never expose one.
- Add `Hits []Hit` to SearchResult. Doc comment: one entry per id, in the same order as
IDs (Hits[i].ID == IDs[i]). An engine called directly may leave it nil;
beachcomber.SearchPage always fills it. Like the ids, hits are candidates: drop the Hit
of every id the SQL re-check rejects before showing a snippet.
- Add the result types:
- `type Hit struct { ID string; Highlights []Snippet }`. Highlights is nil when the
hit has none;
- `type Snippet struct { Field string; ArrayIndex int; Segments []Segment }`.
ArrayIndex is the element index for an array field, and -1 for a non-array field or
when the engine did not say which element;
- `type Segment struct { Text string; Match bool }`. Text is plain stored text,
verbatim, never HTML or markup. It can contain < or &, so the caller escapes it when
rendering.
Create modules/beachcomber/typesense/highlights.go:
- Unexported constants highlightStartTag = "" and highlightEndTag = "",
two Unicode private-use characters that do not occur in normal text. Doc comment: the
engine sends them as highlight_start_tag and highlight_end_tag, so a snippet can be
split without parsing markup.
- Unexported splitSegments(s string) []beachcomber.Segment:
- walk s; a start tag ends the current run and switches to matched, an end tag ends
the run and switches back to unmatched;
- leave out empty runs; text at the end keeps the current state;
- never trim, escape or unescape the text.
- Unexported tsHighlight struct with Field string `json:"field"` and Snippet string
`json:"snippet"`. Task 2 adds snippets and indices.
- Unexported parseHighlights(raw json.RawMessage, h *beachcomber.Highlight) []beachcomber.Snippet:
- decode raw into []json.RawMessage, then each entry into tsHighlight. An entry that
does not decode, or that has an empty snippet, contributes nothing. A highlights
value that is not an array gives nil;
- a non-empty snippet gives Snippet{Field, ArrayIndex: -1, Segments: splitSegments(snippet)}.
A snippet with zero segments is left out;
- snippets keep the order of the highlights array.
In modules/beachcomber/typesense/engine.go:
- In search(), after per_page, when q.Highlight != nil:
- set highlight_start_tag and highlight_end_tag to the two constants;
- set highlight_fields (comma-joined) only when Fields is non-empty;
- set highlight_affix_num_tokens only when AffixTokens > 0.
- Send no other highlight parameter (no full-field highlighting, snippet_threshold or
enable_highlight_v1).
- Add `Highlights json.RawMessage` with tag `json:"highlights"` to the searchAnswer hit
struct, next to Document. Keep it raw so an odd highlights value never makes the
answer unreadable.
- In the id loop, append beachcomber.Hit{ID: id} in exactly the branches that append an
id: string id, and the raw-text fallback. Hits whose id is null or missing are left
out of both lists. Only when q.Highlight != nil, set that Hit's Highlights from
parseHighlights(h.Highlights, q.Highlight). With Highlight nil the snippets carry
Typesense's default mark tags and must not be read as text.
- Allocate hits with make(..., 0, len(res.Hits)) so the slice is non-nil, and return
SearchResult{IDs: ids, Found: res.Found, Hits: hits}.
- Leave maxResponseBytes, SearchIDs (which returns res.IDs) and every existing parameter
unchanged. The existing TestEngineWire search_ids and TestTypesenseSearchPage* tests
must pass untouched.
</action>
<verify>
<automated>cd /media/nvme/dev/golem15/summercms.io/summercms/summercms.go && go test -short ./modules/beachcomber/typesense -run 'TestTypesenseSearchSendsNoNewParamsByDefault|TestTypesenseHighlights|TestTypesenseHitsStayInStepWithIDs|TestTypesenseSearchPage|TestEngineWire' -count=1 && go vet ./modules/beachcomber/...</automated>
</verify>
<done>
- RED evidence is recorded.
- All four new tests pass, and the existing Typesense wire and SearchPage tests pass
unchanged.
- A Query without Highlight sends no new parameters and still returns Hits in step with IDs.
- A string-field highlight reaches SearchResult.Hits as plain-text segments.
- Nothing is committed yet.
</done>
</task>
<task type="auto" tdd="true">
<name>Task 2: Array snippets, field projection, validation and the HighlightSearcher capability</name>
<files>modules/beachcomber/typesense/highlights.go, modules/beachcomber/typesense/engine.go, modules/beachcomber/typesense/highlights_test.go, modules/beachcomber/searchable.go, modules/beachcomber/engines.go, modules/beachcomber/searchpage_test.go</files>
<read_first>modules/beachcomber/searchpage_test.go (idsOnlyEngine, pagedEngine, TestSearchPageFallsBackToSearchIDs, TestSearchPageNullEngine), modules/beachcomber/engines.go</read_first>
<behavior>
- TestTypesenseHighlightsStringArray:
- an entry {"field":"tracks","snippets":["blue note","x blue"],"indices":[2,0]}
gives two Snippets for "tracks", with ArrayIndex 2 and then 0;
- MaxSnippetsPerArray 1 keeps only the first;
- an entry with snippets but no indices gives ArrayIndex -1;
- a hit with both a string entry and an array entry keeps the order of the highlights
array;
- no snippet-cap parameter is sent.
- TestTypesenseFieldProjection:
- IncludeFields [title] sends include_fields "title,id"; IncludeFields [id, title]
sends "id,title";
- ExcludeFields [body] plus Highlight{Fields: [body]} sends exclude_fields=body and
highlight_fields=body. An answer whose document is {"id":"9"} (no body), with a body
highlight, gives Hits[0] a "body" Snippet: an excluded field is still highlighted;
- ExcludeFields [id] is an error containing "exclude_fields", with zero requests;
- Highlight{AffixTokens: -1} and Highlight{MaxSnippetsPerArray: -1} are errors with
zero requests.
- TestTypesenseSupportsHighlights: beachcomber.SupportsHighlights(e) is true for the
engine from newFake, and *Engine satisfies beachcomber.HighlightSearcher.
- beachcomber package:
- TestSupportsHighlights:
- false for nullEngine{}, &idsOnlyEngine{}, &pagedEngine{} and nil;
- a test highlightEngine (embedding pagedEngine, with a supports bool) is true or
false according to its answer.
- TestSearchPageFillsHits:
- a pagedEngine result without Hits comes back with Hits [{ID "7"}];
- an engine that returns its own Hits keeps them untouched;
- zero ids give a non-nil, empty Hits.
- TestSearchPageFallsBackToSearchIDs: the expected value becomes
SearchResult{IDs: [3 1], Found: 2, Hits: [{ID "3"}, {ID "1"}]}.
- TestSearchPageNullEngine also checks that Hits is non-nil and empty.
</behavior>
<action>
Extend highlights_test.go and searchpage_test.go with the cases in behavior first, and
watch them fail (RED). Then implement. Do not commit in this task.
In modules/beachcomber/typesense/highlights.go (spec item 3):
- Add Snippets []string `json:"snippets"` and Indices []int `json:"indices"` to
tsHighlight.
- In parseHighlights, an entry with non-empty Snippets gives one Snippet per element, in
order:
- ArrayIndex is Indices[i] when present, else -1;
- after MaxSnippetsPerArray snippets of that entry, stop when the cap is > 0;
- elements with zero segments are left out and do not count toward the cap.
- Otherwise use the single snippet as in Task 1. An entry with neither contributes nothing.
- Comment why the cap is client-side: Typesense 28.0 has no parameter that caps
snippets per array (checked 2026-11-08), so the cap keeps the first snippets in
Typesense's order.
In modules/beachcomber/typesense/engine.go (spec items 1 and 3):
- Next to the existing weights and per_page checks, before any request:
- an ExcludeFields entry equal to "id" is an error: typesense: search %s:
exclude_fields cannot leave out id, because the ids are the result;
- when q.Highlight != nil, a negative AffixTokens or MaxSnippetsPerArray is an error
naming the setting.
- When IncludeFields is non-empty, set include_fields to the fields comma-joined, with
"id" appended when no entry equals "id". The Typesense docs do not promise id under
include_fields.
- When ExcludeFields is non-empty, set exclude_fields comma-joined.
- Add `SupportsHighlights() bool` on *Engine returning true. Doc: it implements
beachcomber.HighlightSearcher, and SearchPage fills Hits[].Highlights when
Query.Highlight is set.
- Add `_ beachcomber.HighlightSearcher = (*Engine)(nil)` to the existing var block.
- Update the SearchIDs and SearchPage doc comments to list the new parameters, which are
sent only when set, and the Hits contract.
In modules/beachcomber/searchable.go (spec item 4):
- Add HighlightSearcher, placed after PageSearcher:
`type HighlightSearcher interface { PageSearcher; SupportsHighlights() bool }`.
Doc: optional, so existing engines stay valid. An engine that implements it and
answers true fills Hit.Highlights for a Query with Highlight; call it through
SupportsHighlights.
- Add func SupportsHighlights(e Engine) bool: true only when e implements
HighlightSearcher and its SupportsHighlights returns true. A nil e is false. Doc: when
false, an app builds its own snippets from the rows it loads.
- In SearchPage, on success, when res.Hits is nil, set it to one Hit{ID: id} per id.
The slice is non-nil even for zero ids; Hits an engine filled itself are kept as they
are. Extend the SearchPage doc comment with this rule.
In modules/beachcomber/engines.go: nullEngine.SearchPage returns
SearchResult{IDs: []string{}, Hits: []Hit{}}. nullEngine does not implement
HighlightSearcher; extend its comment to say so.
No change to the Engine interface, PageSearcher, or any existing field or function
signature.
</action>
<verify>
<automated>cd /media/nvme/dev/golem15/summercms.io/summercms/summercms.go && go vet ./modules/beachcomber/... && go test -short ./modules/beachcomber/... -count=1</automated>
</verify>
<done>
- Every case in behavior passes.
- String[] snippets carry the ArrayIndex from indices and honour the client-side cap.
- include_fields always carries id, and an excluded field is still highlighted.
- Invalid settings fail before any request.
- SupportsHighlights is true only for Typesense, and SearchPage always returns Hits in
step with IDs.
- go test -short ./modules/beachcomber/... is green, and nothing is committed yet.
</done>
</task>
<task type="auto">
<name>Task 3: README and docs Highlights sections, ExampleEngine_SearchPage, full gates, one commit</name>
<files>modules/beachcomber/typesense/example_test.go, modules/beachcomber/README.md, docs/services/search.md</files>
<read_first>modules/beachcomber/README.md, docs/services/search.md (sections Searching, Reindexing, Typesense), modules/beachcomber/typesense/example_test.go</read_first>
<action>
Add ExampleEngine_SearchPage to modules/beachcomber/typesense/example_test.go (package
typesense_test), in the style of ExampleEngine_SearchIDs: a stand-in httptest node and
typesense.New. It calls beachcomber.SearchPage with:
- Q "blue", QueryBy [title, text], ExcludeFields [text];
- Highlight{Fields: [text], AffixTokens: 3}.
The node prints exclude_fields, highlight_fields and highlight_affix_num_tokens, and
whether highlight_start_tag is U+E000. It answers two hits whose documents carry only id
and title, ids "3" and "8". Each has a text highlight; the one on "3" contains a literal
"<b>" and one match.
The example then:
- checks beachcomber.SupportsHighlights(engine);
- keeps only the hits whose id is in an allowed map {"3": true}, which stands for the
SQL re-check, so "8" is dropped before any snippet is shown;
- renders each kept snippet as HTML, html.EscapeString on every Segment.Text and
<mark>…</mark> around matched segments;
- prints the id, field, ArrayIndex and the HTML (for example "a &lt;b&gt; <mark>blue</mark> …").
Write the Output block from the actual deterministic run.
In modules/beachcomber/README.md (spec item 6):
- Add a "## Highlights" section directly after the "## Sync semantics" section, whose
last bullet is "Candidates, not answers", and before "## Usage". It says:
- Query.Highlight asks for snippets;
- SearchResult.Hits holds one beachcomber.Hit per id in IDs order, and
beachcomber.SearchPage always fills it;
- each beachcomber.Snippet has Field, ArrayIndex (-1 for a non-array field) and
beachcomber.Segment runs of plain stored text with Match. They are never HTML, so
the caller escapes Text when rendering and never parses markup;
- highlights belong to candidates. Drop the Hit of every id that fails the SQL privacy
re-check before showing any snippet, otherwise a snippet leaks a private document;
- beachcomber.SupportsHighlights tells whether the engine fills highlights; when it is
false, build snippets from the loaded rows;
- Query.IncludeFields and Query.ExcludeFields shrink the answer. A field left out
stays searchable and highlightable, so a large text field can be stored,
highlighted and still excluded from the answer.
- Features:
- extend the Engine/Query bullet with the new Query fields;
- add a highlights bullet that names beachcomber.HighlightSearcher and
beachcomber.SupportsHighlights;
- add a Typesense sub-bullet. include_fields (always with id), exclude_fields,
highlight_fields, highlight_affix_num_tokens and the private-use start and end tags
are sent only when set. The highlights array of each hit is split into segments,
with indices filling ArrayIndex. MaxSnippetsPerArray is applied client-side because
Typesense has no such parameter. ExcludeFields naming id, or a negative highlight
setting, is an error before any request.
- API reference:
- extend the beachcomber.Query row with IncludeFields, ExcludeFields and Highlight;
- extend the beachcomber.SearchResult row with Hits;
- add rows for beachcomber.Highlight (Fields, AffixTokens, MaxSnippetsPerArray);
beachcomber.Hit, beachcomber.Snippet and beachcomber.Segment;
beachcomber.HighlightSearcher; and beachcomber.SupportsHighlights(e);
- the typesense.Engine row lists beachcomber.HighlightSearcher among what it implements.
In docs/services/search.md:
- Add a "## Highlights" section between "## Searching" and "## Reindexing". It covers
the same points as the README section and adds a > [!WARNING] callout: a snippet of a
hit whose id failed the SQL re-check leaks the private document.
- In that section, add a fenced go block with the attribute
src=modules/beachcomber/typesense/example_test.go#ExampleEngine_SearchPage, then run
`go run ./cmd/summer docs:sync` to fill it from the source.
- In "## Typesense", extend the closing paragraph with one sentence on the new
parameters, which are sent only when set, and the client-side MaxSnippetsPerArray.
Every backticked identifier must exist in non-test code. Do not mention the unexported
tag constants or helpers. No config keys change, so docs/setup/configuration.md stays
as it is.
Gates: run the verify command; all of it must pass. Then run the Docker-backed
`go test ./modules/beachcomber/... -count=1`. If the Docker socket is denied although
the user is in the docker group, run it through `sudo -u golem` (CLAUDE.md). If Docker
is unavailable, record that in the SUMMARY.
Commit (one commit, the whole plan):
- `git status --short` first;
- stage exactly the nine files in files_modified by path;
- commit with message `feat(beachcomber): search highlights and field projection (quick-261108-pe4)`
and a short body:
- additive, empty new fields keep today's requests;
- Hits in step with IDs;
- plain-text segments;
- HighlightSearcher and SupportsHighlights;
- targets v0.6.4, not tagged.
- No co-author tag. Do not push or tag.
</action>
<verify>
<automated>cd /media/nvme/dev/golem15/summercms.io/summercms/summercms.go && go test -short ./modules/beachcomber/... -count=1 && go run ./cmd/summer docs:build --check && go test ./cmd/summer -run TestDocsTree -count=1 && go vet ./... && go test -short ./...</automated>
</verify>
<done>
- ExampleEngine_SearchPage passes with a deterministic Output, and it shows an excluded
field highlighted, the re-check dropping id 8, and escaped segments.
- The README has the Highlights section after Sync semantics, plus the Features bullets
and API rows.
- docs/services/search.md has the Highlights section with the synced src= block and the
warning.
- docs:build --check, TestDocsTree, go vet ./... and go test -short ./... are green.
- Exactly one new commit holds the nine files: no co-author tag, nothing from .planning/
or other sessions' files, not pushed or tagged.
</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Typesense answer → beachcomber | The index is external and may be stale. Snippet text is stored document text, which can hold any characters, including markup |
| beachcomber.SearchResult → application response | Hits belong to candidate ids that have not yet passed the application's SQL ownership and visibility checks |
## STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|-----------|----------|-----------|----------|-------------|-----------------|
| T-261108-pe4-01 | Information Disclosure | SearchResult.Hits shown for an id the SQL re-check rejects (private or deleted document) | high | mitigate | The SearchResult, Hit and README/docs Highlights sections state the rule. docs/services/search.md has a WARNING callout. ExampleEngine_SearchPage drops the rejected id's hit before rendering |
| T-261108-pe4-02 | Tampering (stored XSS) | Stored text with markup rendered as HTML through snippets | high | mitigate | Segments carry plain text only; the engine never inserts markup (private-use tags are split out, the default mark tags are never parsed). TestTypesenseHighlightsKeepStoredText asserts <b> and &amp; come back verbatim. The docs and example escape Text on render |
| T-261108-pe4-03 | Denial of Service | Full documents with large text fields exceeding maxResponseBytes (32 MiB) | medium | mitigate | ExcludeFields/IncludeFields shrink the answer while keeping fields highlightable; maxResponseBytes stays as the cap; MaxSnippetsPerArray bounds parsed snippets |
| T-261108-pe4-04 | Tampering | include_fields or exclude_fields dropping id, which would empty IDs silently | medium | mitigate | id is appended to include_fields; ExcludeFields naming id is an error before any request (TestTypesenseFieldProjection) |
| T-261108-pe4-05 | Spoofing | Stored text containing U+E000/U+E001 fakes match boundaries | low | accept | Only flips Segment.Match on text that is still plain and escaped on render; these private-use characters do not occur in normal text |
| T-261108-pe4-06 | Information Disclosure | New validation errors echoing request or document data | low | mitigate | Errors follow the existing `typesense: search %s: ...` format with the index and setting name only; StatusError still never carries the body |
| T-261108-pe4-SC | Tampering | npm/pip/cargo installs | low | accept | No package installs or dependency changes; standard library only (encoding/json, net/url, html in the example) |
</threat_model>
<verification>
- `go vet ./...` and `go test -short ./...` are green; `go test ./modules/beachcomber/... -count=1` (Docker) is green or recorded as unavailable.
- `go run ./cmd/summer docs:build --check` reports no problems; `go test ./cmd/summer -run TestDocsTree` passes.
- `git show --stat HEAD` lists exactly the nine files_modified paths; `git log -1 --format=%B` has no Co-Authored-By line.
- `git diff HEAD~1 -- modules/beachcomber/searchable.go` shows no removed or changed existing exported identifiers: Engine, PageSearcher, SearchPage's signature and the existing Query and SearchResult fields are intact.
</verification>
<success_criteria>
- Spec items 1-6 are delivered as written, and every new field left empty keeps today's
Typesense request byte-for-byte in parameter set.
- Typesense highlights reach callers as plain-text segments in SearchResult.Hits, in step with IDs.
- Large fields can be excluded from the answer while still being highlighted.
- Apps can detect highlight support with beachcomber.SupportsHighlights and fall back to
their own snippets.
- The README and docs carry the privacy rule for snippets of candidate hits.
- One code commit; no tag, push or version bump.
</success_criteria>
<output>
Create `.planning/quick/261108-pe4-beachcomber-highlights-and-field-project/261108-pe4-SUMMARY.md` when done, with:
- the RED evidence from Tasks 1 and 2 and the commit hash;
- the Docker test result;
- the interface choices: HighlightSearcher embeds PageSearcher, SearchPage fills Hits,
include_fields always carries id, and the cap is client-side;
- a note for the user: the change targets framework v0.6.4 (patch, additive exported API) and
is not tagged; tagging, the changelog post and consumer adoption (fonoteka.go search)
need their go-ahead. The per-index gate is a separate change.
</output>

View File

@@ -0,0 +1,114 @@
---
phase: quick-261108-pe4
plan: 01
subsystem: beachcomber
tags: [search, typesense, highlights, privacy]
status: complete
requires: []
provides:
- beachcomber.Query.IncludeFields / ExcludeFields / Highlight
- beachcomber.Highlight, Hit, Snippet, Segment, SearchResult.Hits
- beachcomber.HighlightSearcher, beachcomber.SupportsHighlights
- typesense highlights parsing and field projection
affects: [fonoteka.go search (future adoption), any app calling beachcomber.SearchPage]
tech-stack:
added: []
patterns: [optional capability interface + package helper, private-use highlight tags split into plain-text segments]
key-files:
created:
- modules/beachcomber/typesense/highlights.go
- modules/beachcomber/typesense/highlights_test.go
modified:
- modules/beachcomber/searchable.go
- modules/beachcomber/engines.go
- modules/beachcomber/searchpage_test.go
- modules/beachcomber/typesense/engine.go
- modules/beachcomber/typesense/example_test.go
- modules/beachcomber/README.md
- docs/services/search.md
decisions:
- "HighlightSearcher embeds PageSearcher and adds SupportsHighlights() bool, so a wrapping engine can answer at runtime"
- "beachcomber.SearchPage fills Hits (one empty Hit per id) when an engine leaves it nil, so Hits is always in step with IDs"
- "Typesense include_fields always carries id; ExcludeFields naming id is an error before any request"
- "MaxSnippetsPerArray is applied client-side in Typesense's order (Typesense 28.0 has no per-array snippet cap)"
- "Highlights are parsed only when Query.Highlight is set; Typesense's default <mark> snippets are never read as text"
metrics:
duration: ~25min
completed: 2026-11-08
actuals:
tokens: 12100
tasks: 3
commits: 1
plan_head_before: 6f9b77033faf650755d827e0d6f3c957a962d193
plan_head_after: 7819c6746c0b08c3ab5feccaed2e3c9c443fc4a1
---
# Quick 261108-pe4: beachcomber highlights and field projection Summary
Typesense search highlights now reach callers as `SearchResult.Hits`, one `Hit` per id in `IDs` order. Each snippet is a run of plain stored text split at private-use highlight tags. `Query.IncludeFields`/`ExcludeFields` shrink the answer, and a field left out is still highlighted. Support is optional, through `HighlightSearcher`/`SupportsHighlights`. The change is additive: a query that sets none of the new fields sends exactly the same request as before.
## Commit
- `7819c67` feat(beachcomber): search highlights and field projection (quick-261108-pe4), holding all nine files_modified paths. No co-author tag; tagged v0.6.4.
## Tasks
| Task | Name | Result |
|------|------|--------|
| 1 | Tracer: string-field highlight Query.Highlight -> wire -> Hits | Done; the tracer gate re-ran verify, which passed |
| 2 | Array snippets, projection, validation, HighlightSearcher | Done |
| 3 | Example, README, docs, gates, commit | Done, `7819c67` |
Per the plan, Tasks 1 and 2 made no commit, and the whole plan went into one commit in Task 3.
## TDD evidence
- **Task 1 RED:** `go test -short ./modules/beachcomber/typesense -run '...'` failed to build with `undefined: beachcomber.Hit`, `res.Hits undefined (type beachcomber.SearchResult has no field or method Hits)`, `unknown field Highlight in struct literal of type beachcomber.Query`, `undefined: beachcomber.Highlight / Snippet / Segment`. This is the expected compile RED.
- **Task 1 GREEN:** TestTypesenseSearchSendsNoNewParamsByDefault, TestTypesenseHighlightsStringSnippet, TestTypesenseHighlightsKeepStoredText (literal and JSON-escaped `<b>`/`&amp;`) and TestTypesenseHitsStayInStepWithIDs all pass. TestEngineWire and TestTypesenseSearchPage* pass with no changes to those tests.
- **Task 2 RED:** the build failed with `undefined: SupportsHighlights` (beachcomber package) and `undefined: beachcomber.SupportsHighlights`, `undefined: beachcomber.HighlightSearcher` (typesense package).
- **Task 2 GREEN:** TestTypesenseHighlightsStringArray, TestTypesenseFieldProjection, TestTypesenseSupportsHighlights, TestSupportsHighlights and TestSearchPageFillsHits pass, as do the updated TestSearchPageFallsBackToSearchIDs and TestSearchPageNullEngine.
## Gates
- `go test -short ./modules/beachcomber/...`: ok
- `go run ./cmd/summer docs:build --check`: "docs:build: no problems found"
- `go test ./cmd/summer -run TestDocsTree`: ok
- `go vet ./...`: clean
- `go test -short ./...`: every package ok
- **Docker:** `go test ./modules/beachcomber/... -count=1` passed (beachcomber 6.6s), with zero `--- SKIP` lines under `-v`, so the Postgres container tests ran. `sudo -u golem` was not needed.
## Interface choices
- `HighlightSearcher` embeds `PageSearcher` and adds `SupportsHighlights() bool`. Highlights are only returned by `SearchPage`, and the bool method lets a wrapping engine answer at runtime.
- `beachcomber.SearchPage` fills `Hits` with one empty `Hit` per id when the engine leaves it nil, giving a non-nil empty slice for zero ids. Hits the engine filled itself are kept as they are.
- Typesense `include_fields` always gets `id` appended when missing, using `slices.Clip` so the caller's slice is never mutated. `ExcludeFields` naming `id` is an error before any request.
- `MaxSnippetsPerArray` is applied client-side, in Typesense's order, because Typesense 28.0 has no per-array snippet cap. Elements with no segments do not count toward the cap.
- The negative-setting errors name the Go setting: `Highlight.AffixTokens %d is negative` and `Highlight.MaxSnippetsPerArray %d is negative`.
## Deviations from Plan
### Auto-fixed Issues
**1. [Rule 1 - Bug] vet: %q verb on a []Segment in a test Fatalf**
- **Found during:** Task 1
- **Fix:** switched to `%+v` in highlights_test.go before the first green run (never committed).
No other deviations. The plan's TestTypesenseFieldProjection checks that the AffixTokens error is "an error naming the setting". The test looks for the substring `AffixTokens`, which matches the error message.
## Known Stubs
None.
## Threat Flags
None beyond the plan's threat model. The new surface is all request parameters or answer parsing that the register already covers: T-01 by the docs warning plus the example dropping id 8, T-02 by plain-text segments and the KeepStoredText test, T-03 by projection with the 32 MiB cap unchanged, and T-04 by the id guard.
## Note for the user
This change targets framework **v0.6.4** (patch, additive exported API), tagged v0.6.4. The summercms.io changelog post goes out when the tag is pushed; consumer adoption (fonoteka.go search) is separate. The per-index gate (Gate.Enabled per index) is a separate change and is not part of this one.
## Self-Check: PASSED
- FOUND: modules/beachcomber/typesense/highlights.go
- FOUND: modules/beachcomber/typesense/highlights_test.go
- FOUND: commit 7819c67 (9 files, no deletions, no Co-Authored-By line)

View File

@@ -138,7 +138,7 @@ Sessions follow the streamable HTTP transport: `initialize` opens one, `Mcp-Sess
## Protected resource metadata
pier serves the OAuth protected resource metadata (RFC 9728) at `/.well-known/oauth-protected-resource` and at `/.well-known/oauth-protected-resource<mcp.path>`: the resource, the authorization server (the verifier's resource origin, else `app.url`), the supported scopes and the bearer method. When the published verifier implements `pact.MCPAdvertisedScopes` and `pact.MCPAdvertisedScopes.MCPScopesSupported` is non-empty, that slice is copied; otherwise the document keeps `pier.PRMScopes` (`read`, `write`, `ai`, `offline_access`, `backend`). pier does not read `pact.MemberTokenScopeCatalog`. The list the OAuth server plugin publishes is the MCP advertised set: the fixed names `read`, `write`, `ai`, `offline_access` and `backend` are always on it. Declared member scopes follow the plugin's `golem15.oauthserver.mcp_scopes` filter, which is that plugin's key and not a framework `mcp.*` key: unset means every declared member scope, a set filter means only those declared names, and an empty filter means the five fixed names only. An empty `wristband.Options.ScopesSupported` on the MCP flavour resolves to those five names; a non-empty list is used as given. OpenID Connect discovery stays `openid`, `profile` and `email`. A route reads who called a tool with `pact.MCPCallFrom`. A 401 carries `WWW-Authenticate: Bearer resource_metadata="…", scope="read"`, which is how a client finds the authorization server. `mcp.resource` must equal the resource the OAuth server issues tokens for; a difference is logged as a warning at boot.
pier serves the OAuth protected resource metadata (RFC 9728) at `/.well-known/oauth-protected-resource` and at `/.well-known/oauth-protected-resource<mcp.path>`: the resource, the authorization server (the verifier's resource origin, else `app.url`), the supported scopes and the bearer method. When the published verifier implements `pact.MCPAdvertisedScopes` and `pact.MCPAdvertisedScopes.MCPScopesSupported` is non-empty, that slice is copied; otherwise the document keeps `pier.PRMScopes` (`read`, `write`, `ai`, `offline_access`, `backend`). pier does not read `pact.MemberTokenScopeCatalog`. The list the OAuth server plugin publishes is the MCP advertised set: the fixed names `read`, `write`, `ai`, `offline_access` and `backend` are always on it. Declared member scopes follow the plugin's `golem15.oauthserver.mcp_scopes` filter, which is that plugin's key and not a framework `mcp.*` key: unset means every declared member scope, a set filter means only those declared names, and an empty filter means the five fixed names only. An empty `wristband.Options.ScopesSupported` on the MCP flavour resolves to those five names; a non-empty list is used as given. OpenID Connect discovery stays `openid`, `profile` and `email`. A route reads who called a tool with `pact.MCPCallFrom`. A 401 carries `WWW-Authenticate: Bearer resource_metadata="…"`, which is how a client finds the authorization server. It carries no scope parameter, so a client requests the scopes listed in the metadata document and consent decides what is granted. `mcp.resource` must equal the resource the OAuth server issues tokens for; a difference is logged as a warning at boot.
## Security notes

View File

@@ -135,6 +135,73 @@ A paginated endpoint also needs to know how many documents matched. An engine th
`beachcomber.Query.QueryByWeights` ranks the fields of `QueryBy`, one weight per field in the same order, as Scout's `query_by_weights` option does.
## Highlights
Set `beachcomber.Query.Highlight` to get snippets of the matches with the IDs. `beachcomber.SearchResult.Hits` holds one `beachcomber.Hit` per ID, in the same order as `IDs`, including hits without highlights; `beachcomber.SearchPage` always fills it, also for engines that do not highlight. A `beachcomber.Snippet` names its `Field` and, for an array field, the element in `ArrayIndex` (-1 for a non-array field), and splits the excerpt into `beachcomber.Segment` runs of plain stored text with `Match` set on the matched ones. The text is never HTML: a stored `<b>` comes back as `<b>`, so escape `Text` when rendering and mark the matches yourself.
`beachcomber.Highlight.Fields` picks the fields (empty is the engine default), `AffixTokens` the context around a match and `MaxSnippetsPerArray` caps the snippets of one array field. An engine that fills highlights implements the optional `beachcomber.HighlightSearcher` interface; check with `beachcomber.SupportsHighlights`, and when it is false build snippets from the rows loaded after the re-check.
`beachcomber.Query.IncludeFields` and `beachcomber.Query.ExcludeFields` shrink the answer without changing what is searched or highlighted. A large text field can be stored, highlighted and still left out of the answer, so a page of hits stays small. The document ID is always returned.
Highlights belong to candidates, like the IDs. Drop the `Hit` of every ID that fails the SQL re-check before showing any snippet:
```go src=modules/beachcomber/typesense/example_test.go#ExampleEngine_SearchPage
// A stand-in Typesense node. The documents carry only id and title:
// text is excluded from the answer, yet still highlighted.
node := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
fmt.Println("exclude_fields", q.Get("exclude_fields"), "highlight_fields", q.Get("highlight_fields"),
"highlight_affix_num_tokens", q.Get("highlight_affix_num_tokens"))
fmt.Println("private-use start tag:", q.Get("highlight_start_tag") == "\ue000")
fmt.Fprint(w, `{"found":2,"hits":[`+
`{"document":{"id":"3","title":"Kind of Blue"},"highlights":[{"field":"text","snippet":"a <b> \ue000blue\ue001 note"}]},`+
`{"document":{"id":"8","title":"Draft"},"highlights":[{"field":"text","snippet":"private \ue000blue\ue001 draft"}]}]}`)
}))
defer node.Close()
u, _ := url.Parse(node.URL)
port, _ := strconv.Atoi(u.Port())
engine := typesense.New(typesense.Config{APIKey: "test-only-key", Host: u.Hostname(), Port: port, Protocol: "http"})
fmt.Println("highlights:", beachcomber.SupportsHighlights(engine))
res, err := beachcomber.SearchPage(context.Background(), engine, "acme_blog_posts", beachcomber.Query{
Q: "blue",
QueryBy: []string{"title", "text"},
ExcludeFields: []string{"text"},
Highlight: &beachcomber.Highlight{Fields: []string{"text"}, AffixTokens: 3},
})
if err != nil {
fmt.Println(err)
return
}
// Stands in for the SQL re-check: id 8 is not visible to the caller, so
// its hit is dropped before any snippet is shown.
allowed := map[string]bool{"3": true}
for _, hit := range res.Hits {
if !allowed[hit.ID] {
continue
}
for _, sn := range hit.Highlights {
var b strings.Builder
for _, seg := range sn.Segments {
text := html.EscapeString(seg.Text) // segments are plain text
if seg.Match {
text = "<mark>" + text + "</mark>"
}
b.WriteString(text)
}
fmt.Println(hit.ID, sn.Field, sn.ArrayIndex, b.String())
}
}
// Output:
// highlights: true
// exclude_fields text highlight_fields text highlight_affix_num_tokens 3
// private-use start tag: true
// 3 text -1 a &lt;b&gt; <mark>blue</mark> note
```
> [!WARNING]
> A snippet of a hit whose ID failed the SQL re-check leaks the private document: it quotes the indexed text of a draft, a deleted record or another user's data.
## Reindexing
`beachcomber.Service.Sync` pushes one model on demand; a reindex command loops over the table with it. Two helpers cover the index itself:
@@ -193,4 +260,4 @@ fmt.Println(typesense.New(typesense.Config{}).Configured())
// false
```
The Typesense engine implements `beachcomber.PageSearcher` on the same request, adding the answer's `found` count, and sends `query_by_weights` when the query has weights. A page is at most `typesense.MaxPerPage` (250) documents, Typesense's own limit: a larger `PerPage` is an error, so fetch more IDs page by page. Each request times out after `search.typesense.connection_timeout_seconds` (2 seconds by default). A failed answer is a `typesense.StatusError` with the method, path and status, never the answer body.
The Typesense engine implements `beachcomber.PageSearcher` on the same request, adding the answer's `found` count, and sends `query_by_weights` when the query has weights. A page is at most `typesense.MaxPerPage` (250) documents, Typesense's own limit: a larger `PerPage` is an error, so fetch more IDs page by page. Each request times out after `search.typesense.connection_timeout_seconds` (2 seconds by default). A failed answer is a `typesense.StatusError` with the method, path and status, never the answer body. `include_fields` (always with `id`), `exclude_fields`, `highlight_fields`, `highlight_affix_num_tokens` and the highlight tags are sent only when the query sets them, and `MaxSnippetsPerArray` is applied by the engine after the answer, because Typesense has no parameter for it.

View File

@@ -30,8 +30,9 @@ Use cases:
- Engine selection by `search.driver`: `null` (the default) or a registered engine such as `typesense`. An unknown name is a boot error that lists the registered engines. Third-party engines register with `beachcomber.RegisterEngine` and a `beachcomber.EngineFactory`; a duplicate name panics at init.
- The `beachcomber.Searchable` model contract: `SearchableAs` (the index name, prefixed with `search.prefix`), `ToSearchableArray(ctx, db)` (the document, built from the committed row and free to query related rows) and `ShouldBeSearchable`. A model can also implement `beachcomber.IndexSchemaProvider` (the schema the engine creates a missing index with) and `beachcomber.SearchKeyer` (a document key other than the decimal primary key).
- The `beachcomber.Engine` driver contract: `Name`, `Configured`, `Upsert`, `Delete`, `Flush` and `SearchIDs` with a `beachcomber.Query`. `beachcomber.Query.QueryByWeights` ranks the `QueryBy` fields, one weight per field.
- The `beachcomber.Engine` driver contract: `Name`, `Configured`, `Upsert`, `Delete`, `Flush` and `SearchIDs` with a `beachcomber.Query`. `beachcomber.Query.QueryByWeights` ranks the `QueryBy` fields, one weight per field. `beachcomber.Query.IncludeFields` and `beachcomber.Query.ExcludeFields` shrink the answer, and `beachcomber.Query.Highlight` asks for snippets.
- Search totals: the optional `beachcomber.PageSearcher` interface returns a `beachcomber.SearchResult`, one page of ids plus `Found`, the number of documents the engine matched. Call it through `beachcomber.SearchPage`, which falls back to `SearchIDs` (with `Found` set to the number of ids) for an engine that does not implement it; the `null` engine returns no ids and a zero count.
- Search highlights: `beachcomber.SearchResult.Hits` holds one `beachcomber.Hit` per id, in step with `IDs`, whose snippets are runs of plain stored text, never HTML. The optional `beachcomber.HighlightSearcher` interface marks an engine that fills them; ask through `beachcomber.SupportsHighlights`. See [Highlights](#highlights).
- Index lifecycle for reindex tooling: the optional `beachcomber.IndexDropper` interface drops an index and reports whether it existed, so a command can tell a dropped index from one that was already absent; call it through `beachcomber.DropIndex`, which falls back to `Flush` (reporting existed) for other engines. The optional `beachcomber.IndexEnsurer` creates an empty index from its schema; `beachcomber.EnsureIndex` calls it and is a no-op for other engines. A full reindex calls it first because an `Upsert` of zero documents creates nothing.
- GORM callbacks `beachcomber.CallbackAfterCreate`, `beachcomber.CallbackAfterUpdate` and `beachcomber.CallbackAfterDelete`, installed once per `*gorm.DB`, register the sync with `lagoon.AfterCommit`.
- `beachcomber.Service.Sync` and `beachcomber.Service.Remove` run the same gated path on demand, for reindex tooling, and return the error instead of logging it.
@@ -41,6 +42,7 @@ Use cases:
- `Delete` and `Flush` treat 404 as success.
- `DropIndex` sends `DELETE /collections/{index}` and reports existed true on 2xx and false on 404; `EnsureIndex` runs the same read-then-create step as `Upsert`, so a second call sends no create.
- `SearchIDs` and `SearchPage` send one request with `q` (default `*`), `query_by`, `query_by_weights`, `filter_by`, `sort_by`, `page` and `per_page`, and return `hits[].document.id` in order; `SearchPage` adds the answer's `found`. A weight list whose length differs from `QueryBy`, or a `PerPage` above `typesense.MaxPerPage` (250, Typesense's limit), is an error before any request; callers page instead.
- `include_fields` (always with `id`), `exclude_fields`, `highlight_fields`, `highlight_affix_num_tokens` and private-use highlight start and end tags are sent only when the query sets them. The `highlights` array of each hit is split into segments at those tags, with `indices` filling `ArrayIndex` for array fields. `MaxSnippetsPerArray` is applied client-side, because Typesense has no such parameter. An `ExcludeFields` that names `id`, or a negative highlight setting, is an error before any request.
- Ids and index names are path-escaped.
- A non-2xx answer is a `typesense.StatusError` with the method, path and status, never the answer body.
@@ -58,6 +60,15 @@ Use cases:
- **Rows only.** A statement without a primary key value, such as `Model(&T{}).Where(…).Updates(…)` or `Delete(&T{}, id)`, cannot be synced row by row and is skipped. Bulk paths call `beachcomber.Service.Sync` or `beachcomber.Service.Remove` per row, or reindex.
- **Candidates, not answers.** `beachcomber.Engine.SearchIDs` returns candidate ids from an external index that may be stale. Callers must re-gate every id in SQL (ownership, visibility, soft deletes) before they expose a row. An empty result is an empty list, never an error. The same holds for `beachcomber.SearchResult.Found`: it counts documents in the index, including stale ones, so a total shown to a user is recounted in SQL over re-gated ids, never taken from the engine.
## Highlights
- `beachcomber.Query.Highlight` asks for snippets: `Fields` (empty is the engine default, the `QueryBy` fields for Typesense), `AffixTokens` (tokens of context around a match, 0 is the engine default) and `MaxSnippetsPerArray` (0 is no cap). There is no option to highlight a whole field.
- `beachcomber.SearchResult.Hits` holds one `beachcomber.Hit` per id, in `IDs` order (`Hits[i].ID == IDs[i]`), including hits without highlights. An engine called directly may leave it nil; `beachcomber.SearchPage` always fills it.
- Each `beachcomber.Snippet` has `Field`, `ArrayIndex` (the element of an array field, -1 for a non-array field) and `beachcomber.Segment` runs with `Text` and `Match`. The text is plain stored text, verbatim, never HTML: escape `Text` when rendering and never parse markup out of it.
- **Highlights belong to candidates.** A snippet is cut from the indexed document, so it can quote a draft, a deleted record or another user's data. Drop the `Hit` of every id that fails the SQL privacy re-check before showing any snippet; otherwise a snippet leaks the private document.
- `beachcomber.SupportsHighlights` tells whether the engine fills highlights (true for Typesense, false for `null` and for engines without `beachcomber.HighlightSearcher`). When it is false, build snippets from the rows loaded after the re-check.
- `beachcomber.Query.IncludeFields` and `beachcomber.Query.ExcludeFields` shrink the answer. A field left out of the answer stays searchable and highlightable, so a large text field can be stored, highlighted and still excluded from the answer. The document id is always returned.
## Usage
An application selects the engine in `config/search.yaml`:
@@ -143,10 +154,14 @@ ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomb
| `beachcomber.IndexSchemaProvider` | `SearchIndexSchema()`: the schema a missing index is created with. |
| `beachcomber.SearchKeyer` | `SearchKey()`: replaces the decimal primary key as the document key. |
| `beachcomber.Engine` | `Name`, `Configured`, `Upsert`, `Delete`, `Flush`, `SearchIDs`. |
| `beachcomber.Query` | `Q`, `QueryBy`, `QueryByWeights`, `FilterBy`, `SortBy`, `Page`, `PerPage`. |
| `beachcomber.Query` | `Q`, `QueryBy`, `QueryByWeights`, `FilterBy`, `SortBy`, `Page`, `PerPage`, `IncludeFields`, `ExcludeFields`, `Highlight`. |
| `beachcomber.Highlight` | `Fields`, `AffixTokens`, `MaxSnippetsPerArray`: the snippets a query asks for. |
| `beachcomber.PageSearcher` | Optional engine interface: `SearchPage(ctx, index, q)` returns a page of ids and the engine's found count. |
| `beachcomber.SearchResult` | `IDs` (candidates) and `Found` (documents the engine matched). |
| `beachcomber.SearchPage(ctx, engine, index, q)` | Calls `PageSearcher` when the engine has it, else `SearchIDs` with `Found` set to the number of ids. |
| `beachcomber.SearchResult` | `IDs` (candidates), `Found` (documents the engine matched) and `Hits` (one per id, in the same order). |
| `beachcomber.Hit`, `beachcomber.Snippet`, `beachcomber.Segment` | A hit's `ID` and `Highlights`; a snippet's `Field`, `ArrayIndex` and `Segments`; a segment's plain `Text` and `Match`. |
| `beachcomber.SearchPage(ctx, engine, index, q)` | Calls `PageSearcher` when the engine has it, else `SearchIDs` with `Found` set to the number of ids; always fills `Hits` in step with `IDs`. |
| `beachcomber.HighlightSearcher` | Optional engine interface: a `PageSearcher` with `SupportsHighlights()`, filling `Hit.Highlights`. |
| `beachcomber.SupportsHighlights(e)` | Whether the engine implements `HighlightSearcher` and answers true. |
| `beachcomber.IndexDropper` | Optional engine interface: `DropIndex(ctx, index)` drops the index and reports whether it existed. |
| `beachcomber.DropIndex(ctx, engine, index)` | Calls `IndexDropper` when the engine has it, else `Flush` and reports existed true. |
| `beachcomber.IndexEnsurer` | Optional engine interface: `EnsureIndex(ctx, index, schema)` creates a missing index, empty, from its schema. |
@@ -161,7 +176,7 @@ ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomb
| Identifier | Description |
|------------|-------------|
| `typesense.Config`, `typesense.LoadConfig` | The `search.typesense.*` settings with their defaults; `BaseURL` is `{protocol}://{host}:{port}{path}`. |
| `typesense.Engine`, `typesense.New` | The `beachcomber.Engine`, `beachcomber.PageSearcher`, `beachcomber.IndexDropper` and `beachcomber.IndexEnsurer`, with `Config`. |
| `typesense.Engine`, `typesense.New` | The `beachcomber.Engine`, `beachcomber.PageSearcher`, `beachcomber.HighlightSearcher`, `beachcomber.IndexDropper` and `beachcomber.IndexEnsurer`, with `Config`. |
| `typesense.MaxPerPage` | 250, the largest page a search may ask for. |
| `typesense.StatusError` | A non-2xx answer: `Method`, `Path`, `Code` and `StatusCode()`. |
| `typesense.DriverName` | `typesense`. |

View File

@@ -62,7 +62,8 @@ func init() {
}
// nullEngine is never configured, so sync is skipped before it is called.
// Every method is a no-op; SearchIDs and SearchPage return no ids.
// Every method is a no-op; SearchIDs and SearchPage return no ids. It does
// not implement HighlightSearcher.
type nullEngine struct{}
func (nullEngine) Name() string { return NullEngine }
@@ -76,5 +77,5 @@ func (nullEngine) SearchIDs(context.Context, string, Query) ([]string, error) {
return []string{}, nil
}
func (nullEngine) SearchPage(context.Context, string, Query) (SearchResult, error) {
return SearchResult{IDs: []string{}}, nil
return SearchResult{IDs: []string{}, Hits: []Hit{}}, nil
}

View File

@@ -75,6 +75,32 @@ type Query struct {
Page int
// PerPage is the page size; 0 leaves the engine default.
PerPage int
// IncludeFields and ExcludeFields only limit which document fields the
// engine sends back: a field left out of the answer stays searchable
// and highlightable. Empty keeps the engine default (every stored
// field). The document key is always returned, because the ids are the
// result. Engines that cannot project ignore them.
IncludeFields []string
ExcludeFields []string
// Highlight asks for snippets of the matches in SearchResult.Hits. Nil
// asks for no highlights and keeps the request as it was. Engines
// without HighlightSearcher ignore it.
Highlight *Highlight
}
// Highlight configures the snippets an engine returns for a Query.
// Snippets are always cut around the matches; there is deliberately no
// option to highlight a whole field.
type Highlight struct {
// Fields are the fields to highlight. Empty is the engine default,
// which is the QueryBy fields for Typesense.
Fields []string
// AffixTokens is the number of tokens of context on each side of a
// match; 0 is the engine default.
AffixTokens int
// MaxSnippetsPerArray is the most snippets kept for one array field;
// 0 means no cap.
MaxSnippetsPerArray int
}
// SearchResult is one page of candidate ids and the number of documents
@@ -83,6 +109,36 @@ type Query struct {
type SearchResult struct {
IDs []string
Found int
// Hits has one entry per id, in the same order as IDs
// (Hits[i].ID == IDs[i]). An engine called directly may leave it nil;
// SearchPage always fills it. Like the ids, hits are candidates: drop
// the Hit of every id the SQL re-check rejects before showing a
// snippet.
Hits []Hit
}
// Hit is the engine's answer for one id beyond the id itself.
type Hit struct {
ID string
// Highlights is nil when the hit has none.
Highlights []Snippet
}
// Snippet is one highlighted excerpt of a field.
type Snippet struct {
Field string
// ArrayIndex is the element index for an array field, and -1 for a
// non-array field or when the engine did not say which element.
ArrayIndex int
Segments []Segment
}
// Segment is a run of a snippet that either matched the query or not.
// Text is plain stored text, verbatim, never HTML or markup. It can
// contain < or &, so the caller escapes it when rendering.
type Segment struct {
Text string
Match bool
}
// PageSearcher is implemented by an engine that can report how many
@@ -92,18 +148,51 @@ type PageSearcher interface {
SearchPage(ctx context.Context, index string, q Query) (SearchResult, error)
}
// HighlightSearcher is implemented by an engine that can return snippets
// of the matches. It is optional so that existing engines stay valid. An
// engine that implements it and answers true fills Hit.Highlights for a
// Query with Highlight; ask through SupportsHighlights.
type HighlightSearcher interface {
PageSearcher
SupportsHighlights() bool
}
// SupportsHighlights reports whether e fills Hit.Highlights: true only when
// e implements HighlightSearcher and its SupportsHighlights answers true. A
// nil e is false. When it is false, an app builds its own snippets from the
// rows it loads.
func SupportsHighlights(e Engine) bool {
hs, ok := e.(HighlightSearcher)
return ok && hs.SupportsHighlights()
}
// SearchPage asks e for one page of q: through PageSearcher when e
// implements it, otherwise through SearchIDs with Found set to the number
// of ids returned.
// of ids returned. On success Hits is always in step with IDs: when the
// engine left it nil, it is one Hit without highlights per id (an empty,
// non-nil slice for zero ids); hits the engine filled are kept as they are.
func SearchPage(ctx context.Context, e Engine, index string, q Query) (SearchResult, error) {
var res SearchResult
if ps, ok := e.(PageSearcher); ok {
return ps.SearchPage(ctx, index, q)
r, err := ps.SearchPage(ctx, index, q)
if err != nil {
return r, err
}
res = r
} else {
ids, err := e.SearchIDs(ctx, index, q)
if err != nil {
return SearchResult{}, err
}
res = SearchResult{IDs: ids, Found: len(ids)}
}
ids, err := e.SearchIDs(ctx, index, q)
if err != nil {
return SearchResult{}, err
if res.Hits == nil {
res.Hits = make([]Hit, len(res.IDs))
for i, id := range res.IDs {
res.Hits[i] = Hit{ID: id}
}
}
return SearchResult{IDs: ids, Found: len(ids)}, nil
return res, nil
}
// IndexDropper is implemented by an engine that can tell whether the index

View File

@@ -45,7 +45,7 @@ func TestSearchPageFallsBackToSearchIDs(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(res, SearchResult{IDs: []string{"3", "1"}, Found: 2}) {
if !reflect.DeepEqual(res, SearchResult{IDs: []string{"3", "1"}, Found: 2, Hits: []Hit{{ID: "3"}, {ID: "1"}}}) {
t.Fatalf("res = %+v", res)
}
if !reflect.DeepEqual(e.got, q) {
@@ -74,11 +74,63 @@ func TestSearchPageNullEngine(t *testing.T) {
t.Fatal("the null engine implements PageSearcher")
}
res, err := SearchPage(context.Background(), e, "posts", Query{})
if err != nil || res.Found != 0 || res.IDs == nil || len(res.IDs) != 0 {
if err != nil || res.Found != 0 || res.IDs == nil || len(res.IDs) != 0 || res.Hits == nil || len(res.Hits) != 0 {
t.Fatalf("res = %+v err = %v", res, err)
}
}
// highlightEngine implements HighlightSearcher with a configurable answer
// and can return its own hits.
type highlightEngine struct {
pagedEngine
supports bool
hits []Hit
}
func (e *highlightEngine) SupportsHighlights() bool { return e.supports }
func (e *highlightEngine) SearchPage(_ context.Context, _ string, q Query) (SearchResult, error) {
e.got = q
return SearchResult{IDs: e.ids, Found: e.found, Hits: e.hits}, nil
}
func TestSupportsHighlights(t *testing.T) {
for name, e := range map[string]Engine{
"null": nullEngine{},
"ids-only": &idsOnlyEngine{},
"paged": &pagedEngine{},
"nil": nil,
} {
if SupportsHighlights(e) {
t.Fatalf("%s: SupportsHighlights = true", name)
}
}
if !SupportsHighlights(&highlightEngine{supports: true}) {
t.Fatal("an engine answering true supports highlights")
}
if SupportsHighlights(&highlightEngine{supports: false}) {
t.Fatal("an engine answering false does not support highlights")
}
}
func TestSearchPageFillsHits(t *testing.T) {
ctx := context.Background()
res, err := SearchPage(ctx, &pagedEngine{idsOnlyEngine: idsOnlyEngine{ids: []string{"7"}}, found: 1}, "posts", Query{})
if err != nil || !reflect.DeepEqual(res.Hits, []Hit{{ID: "7"}}) {
t.Fatalf("hits = %+v err = %v", res.Hits, err)
}
own := []Hit{{ID: "7", Highlights: []Snippet{{Field: "title", ArrayIndex: -1, Segments: []Segment{{Text: "x", Match: true}}}}}}
he := &highlightEngine{pagedEngine: pagedEngine{idsOnlyEngine: idsOnlyEngine{ids: []string{"7"}}}, supports: true, hits: own}
res, err = SearchPage(ctx, he, "posts", Query{Highlight: &Highlight{}})
if err != nil || !reflect.DeepEqual(res.Hits, own) {
t.Fatalf("own hits = %+v err = %v", res.Hits, err)
}
res, err = SearchPage(ctx, &pagedEngine{}, "posts", Query{})
if err != nil || res.Hits == nil || len(res.Hits) != 0 {
t.Fatalf("zero ids: hits = %#v err = %v", res.Hits, err)
}
}
// flushEngine records Flush calls and can fail them.
type flushEngine struct {
idsOnlyEngine

View File

@@ -10,6 +10,7 @@ import (
"maps"
"net/http"
"net/url"
"slices"
"strconv"
"strings"
@@ -48,8 +49,9 @@ type Engine struct {
var _ beachcomber.Engine = (*Engine)(nil)
var (
_ beachcomber.IndexDropper = (*Engine)(nil)
_ beachcomber.IndexEnsurer = (*Engine)(nil)
_ beachcomber.IndexDropper = (*Engine)(nil)
_ beachcomber.IndexEnsurer = (*Engine)(nil)
_ beachcomber.HighlightSearcher = (*Engine)(nil)
)
// New returns an engine for cfg. Every request is bounded by
@@ -252,6 +254,9 @@ type searchAnswer struct {
Document struct {
ID json.RawMessage `json:"id"`
} `json:"document"`
// Highlights stays raw so an odd highlights value never makes
// the answer unreadable.
Highlights json.RawMessage `json:"highlights"`
} `json:"hits"`
}
@@ -261,9 +266,11 @@ const MaxPerPage = 250
// SearchIDs sends GET /collections/{index}/documents/search with q
// (default "*"), query_by, query_by_weights, filter_by, sort_by, page and
// per_page, and returns hits[].document.id in order. The ids are
// candidates: the caller must re-check each one in SQL before exposing it.
// No hits is an empty list.
// per_page, and returns hits[].document.id in order. include_fields
// (always with id), exclude_fields and, for Query.Highlight,
// highlight_fields, highlight_affix_num_tokens and the highlight tags are
// sent only when set. The ids are candidates: the caller must re-check
// each one in SQL before exposing it. No hits is an empty list.
func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Query) ([]string, error) {
res, err := e.search(ctx, index, q)
if err != nil {
@@ -272,12 +279,19 @@ func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Quer
return res.IDs, nil
}
// SearchPage is SearchIDs plus the found count of the answer: how many
// documents matched in the index, not how many the caller may see.
// SearchPage is SearchIDs plus the found count of the answer (how many
// documents matched in the index, not how many the caller may see) and
// Hits, one per id in the same order. With Query.Highlight set, each Hit
// carries the snippets of the answer's highlights array, split into plain
// text segments; without it, Highlights stays nil.
func (e *Engine) SearchPage(ctx context.Context, index string, q beachcomber.Query) (beachcomber.SearchResult, error) {
return e.search(ctx, index, q)
}
// SupportsHighlights returns true: SearchPage fills Hits[].Highlights when
// Query.Highlight is set. It implements beachcomber.HighlightSearcher.
func (e *Engine) SupportsHighlights() bool { return true }
func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query) (beachcomber.SearchResult, error) {
if len(q.QueryByWeights) > 0 && len(q.QueryByWeights) != len(q.QueryBy) {
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: %d query_by_weights for %d query_by fields", index, len(q.QueryByWeights), len(q.QueryBy))
@@ -285,6 +299,19 @@ func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query)
if q.PerPage > MaxPerPage {
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: per_page %d exceeds %d", index, q.PerPage, MaxPerPage)
}
for _, f := range q.ExcludeFields {
if f == "id" {
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: exclude_fields cannot leave out id, because the ids are the result", index)
}
}
if h := q.Highlight; h != nil {
if h.AffixTokens < 0 {
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: Highlight.AffixTokens %d is negative", index, h.AffixTokens)
}
if h.MaxSnippetsPerArray < 0 {
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: Highlight.MaxSnippetsPerArray %d is negative", index, h.MaxSnippetsPerArray)
}
}
params := url.Values{}
text := q.Q
if text == "" {
@@ -313,6 +340,27 @@ func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query)
if q.PerPage > 0 {
params.Set("per_page", strconv.Itoa(q.PerPage))
}
if len(q.IncludeFields) > 0 {
// The Typesense docs do not promise id under include_fields.
fields := q.IncludeFields
if !slices.Contains(fields, "id") {
fields = append(slices.Clip(fields), "id")
}
params.Set("include_fields", strings.Join(fields, ","))
}
if len(q.ExcludeFields) > 0 {
params.Set("exclude_fields", strings.Join(q.ExcludeFields, ","))
}
if h := q.Highlight; h != nil {
params.Set("highlight_start_tag", highlightStartTag)
params.Set("highlight_end_tag", highlightEndTag)
if len(h.Fields) > 0 {
params.Set("highlight_fields", strings.Join(h.Fields, ","))
}
if h.AffixTokens > 0 {
params.Set("highlight_affix_num_tokens", strconv.Itoa(h.AffixTokens))
}
}
path := "/collections/" + url.PathEscape(index) + "/documents/search"
code, answer, err := e.do(ctx, http.MethodGet, path, params, nil, "")
if err != nil {
@@ -326,19 +374,28 @@ func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query)
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: unreadable answer", index)
}
ids := make([]string, 0, len(res.Hits))
hits := make([]beachcomber.Hit, 0, len(res.Hits))
for _, h := range res.Hits {
raw := bytes.TrimSpace(h.Document.ID)
if len(raw) == 0 || bytes.Equal(raw, []byte("null")) {
continue
}
id := string(raw)
var s string
if err := json.Unmarshal(raw, &s); err == nil {
ids = append(ids, s)
continue
id = s
}
ids = append(ids, string(raw))
hit := beachcomber.Hit{ID: id}
// Only with Highlight set are the tags the private-use characters;
// otherwise the snippets carry Typesense's default mark tags and
// must not be read as text.
if q.Highlight != nil {
hit.Highlights = parseHighlights(h.Highlights, q.Highlight)
}
ids = append(ids, id)
hits = append(hits, hit)
}
return beachcomber.SearchResult{IDs: ids, Found: res.Found}, nil
return beachcomber.SearchResult{IDs: ids, Found: res.Found, Hits: hits}, nil
}
// do sends one request with the API key header and returns the status and

View File

@@ -3,10 +3,12 @@ package typesense_test
import (
"context"
"fmt"
"html"
"net/http"
"net/http/httptest"
"net/url"
"strconv"
"strings"
"time"
"git.golem15.com/golem15/summercms/modules/beachcomber"
@@ -47,3 +49,57 @@ func ExampleEngine_SearchIDs() {
// [3 1] <nil>
// false
}
func ExampleEngine_SearchPage() {
// A stand-in Typesense node. The documents carry only id and title:
// text is excluded from the answer, yet still highlighted.
node := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
fmt.Println("exclude_fields", q.Get("exclude_fields"), "highlight_fields", q.Get("highlight_fields"),
"highlight_affix_num_tokens", q.Get("highlight_affix_num_tokens"))
fmt.Println("private-use start tag:", q.Get("highlight_start_tag") == "\ue000")
fmt.Fprint(w, `{"found":2,"hits":[`+
`{"document":{"id":"3","title":"Kind of Blue"},"highlights":[{"field":"text","snippet":"a <b> \ue000blue\ue001 note"}]},`+
`{"document":{"id":"8","title":"Draft"},"highlights":[{"field":"text","snippet":"private \ue000blue\ue001 draft"}]}]}`)
}))
defer node.Close()
u, _ := url.Parse(node.URL)
port, _ := strconv.Atoi(u.Port())
engine := typesense.New(typesense.Config{APIKey: "test-only-key", Host: u.Hostname(), Port: port, Protocol: "http"})
fmt.Println("highlights:", beachcomber.SupportsHighlights(engine))
res, err := beachcomber.SearchPage(context.Background(), engine, "acme_blog_posts", beachcomber.Query{
Q: "blue",
QueryBy: []string{"title", "text"},
ExcludeFields: []string{"text"},
Highlight: &beachcomber.Highlight{Fields: []string{"text"}, AffixTokens: 3},
})
if err != nil {
fmt.Println(err)
return
}
// Stands in for the SQL re-check: id 8 is not visible to the caller, so
// its hit is dropped before any snippet is shown.
allowed := map[string]bool{"3": true}
for _, hit := range res.Hits {
if !allowed[hit.ID] {
continue
}
for _, sn := range hit.Highlights {
var b strings.Builder
for _, seg := range sn.Segments {
text := html.EscapeString(seg.Text) // segments are plain text
if seg.Match {
text = "<mark>" + text + "</mark>"
}
b.WriteString(text)
}
fmt.Println(hit.ID, sn.Field, sn.ArrayIndex, b.String())
}
}
// Output:
// highlights: true
// exclude_fields text highlight_fields text highlight_affix_num_tokens 3
// private-use start tag: true
// 3 text -1 a &lt;b&gt; <mark>blue</mark> note
}

View File

@@ -0,0 +1,116 @@
package typesense
import (
"encoding/json"
"strings"
"git.golem15.com/golem15/summercms/modules/beachcomber"
)
// highlightStartTag and highlightEndTag are two Unicode private-use
// characters that do not occur in normal text. The engine sends them as
// highlight_start_tag and highlight_end_tag, so a snippet can be split into
// segments without parsing markup.
const (
highlightStartTag = ""
highlightEndTag = ""
)
// splitSegments splits a snippet tagged with highlightStartTag and
// highlightEndTag into matched and unmatched runs. Empty runs are left out
// and the text is never trimmed, escaped or unescaped.
func splitSegments(s string) []beachcomber.Segment {
var out []beachcomber.Segment
match := false
for s != "" {
tag := highlightStartTag
if match {
tag = highlightEndTag
}
i := strings.Index(s, tag)
if i < 0 {
out = appendSegment(out, s, match)
break
}
out = appendSegment(out, s[:i], match)
s = s[i+len(tag):]
match = !match
}
return out
}
func appendSegment(out []beachcomber.Segment, text string, match bool) []beachcomber.Segment {
if text == "" {
return out
}
return append(out, beachcomber.Segment{Text: text, Match: match})
}
// tsHighlight is one entry of a hit's highlights array.
// A string field carries snippet; a string[] field carries snippets with
// the element index of each in indices.
type tsHighlight struct {
Field string `json:"field"`
Snippet string `json:"snippet"`
Snippets []string `json:"snippets"`
Indices []int `json:"indices"`
}
// parseHighlights turns a hit's highlights array into snippets, in the
// order of the array. An entry that does not decode, or has no snippet,
// contributes nothing; a value that is not an array gives nil.
func parseHighlights(raw json.RawMessage, h *beachcomber.Highlight) []beachcomber.Snippet {
if len(raw) == 0 {
return nil
}
var entries []json.RawMessage
if err := json.Unmarshal(raw, &entries); err != nil {
return nil
}
var out []beachcomber.Snippet
for _, entry := range entries {
var th tsHighlight
if err := json.Unmarshal(entry, &th); err != nil {
continue
}
if len(th.Snippets) > 0 {
out = appendArraySnippets(out, th, h)
continue
}
if th.Snippet == "" {
continue
}
segs := splitSegments(th.Snippet)
if len(segs) == 0 {
continue
}
out = append(out, beachcomber.Snippet{Field: th.Field, ArrayIndex: -1, Segments: segs})
}
return out
}
// appendArraySnippets appends one snippet per element of an array field's
// snippets, with ArrayIndex taken from indices (-1 when indices does not
// cover the element). Typesense 28.0 has no parameter that caps snippets
// per array (checked 2026-11-08), so MaxSnippetsPerArray is applied here
// and keeps the first snippets in Typesense's order. Elements with no
// segments are left out and do not count toward the cap.
func appendArraySnippets(out []beachcomber.Snippet, th tsHighlight, h *beachcomber.Highlight) []beachcomber.Snippet {
kept := 0
for i, sn := range th.Snippets {
if h != nil && h.MaxSnippetsPerArray > 0 && kept >= h.MaxSnippetsPerArray {
break
}
segs := splitSegments(sn)
if len(segs) == 0 {
continue
}
idx := -1
if i < len(th.Indices) {
idx = th.Indices[i]
}
out = append(out, beachcomber.Snippet{Field: th.Field, ArrayIndex: idx, Segments: segs})
kept++
}
return out
}

View File

@@ -0,0 +1,276 @@
package typesense
import (
"net/url"
"reflect"
"sort"
"strings"
"testing"
"git.golem15.com/golem15/summercms/modules/beachcomber"
)
const searchKey = "GET /ts/collections/posts/documents/search"
// onlyQuery returns the parsed query of the single call f received.
func onlyQuery(t *testing.T, f *fakeTypesense) url.Values {
t.Helper()
calls := f.take()
if len(calls) != 1 {
t.Fatalf("calls = %d, want 1", len(calls))
}
params, err := url.ParseQuery(calls[0].rawQuery)
if err != nil {
t.Fatal(err)
}
return params
}
func TestTypesenseSearchSendsNoNewParamsByDefault(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":2,"hits":[`+
`{"document":{"id":"42"},"highlights":[{"field":"name","snippet":"a <mark>blue</mark> train","matched_tokens":["blue"]}]},`+
`{"document":{"id":7}}]}`)
q := beachcomber.Query{
Q: "blue train",
QueryBy: []string{"name", "artist_display", "notes"},
QueryByWeights: []int{10, 10, 1},
FilterBy: "collection_id:=5",
SortBy: "created_at:desc",
Page: 2,
PerPage: 250,
}
res, err := e.SearchPage(t.Context(), "posts", q)
if err != nil {
t.Fatal(err)
}
params := onlyQuery(t, f)
keys := make([]string, 0, len(params))
for k := range params {
keys = append(keys, k)
}
sort.Strings(keys)
want := []string{"filter_by", "page", "per_page", "q", "query_by", "query_by_weights", "sort_by"}
if !reflect.DeepEqual(keys, want) {
t.Fatalf("keys = %v, want %v", keys, want)
}
if res.Found != 2 || !reflect.DeepEqual(res.IDs, []string{"42", "7"}) {
t.Fatalf("res = %+v", res)
}
wantHits := []beachcomber.Hit{{ID: "42"}, {ID: "7"}}
if !reflect.DeepEqual(res.Hits, wantHits) {
t.Fatalf("hits = %+v, want %+v (mark snippets must not be read)", res.Hits, wantHits)
}
}
func TestTypesenseHighlightsStringSnippet(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":1,"hits":[{"document":{"id":"5"},"highlights":[{"field":"title","snippet":"a blue train","matched_tokens":["blue"]}]}]}`)
res, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{
Q: "blue",
QueryBy: []string{"title"},
Highlight: &beachcomber.Highlight{Fields: []string{"title"}, AffixTokens: 6},
})
if err != nil {
t.Fatal(err)
}
params := onlyQuery(t, f)
if params.Get("highlight_fields") != "title" || params.Get("highlight_affix_num_tokens") != "6" ||
params.Get("highlight_start_tag") != "" || params.Get("highlight_end_tag") != "" {
t.Fatalf("params = %v", params)
}
if params.Has("highlight_full_fields") {
t.Fatal("highlight_full_fields must never be sent")
}
want := []beachcomber.Hit{{ID: "5", Highlights: []beachcomber.Snippet{{
Field: "title",
ArrayIndex: -1,
Segments: []beachcomber.Segment{{Text: "a "}, {Text: "blue", Match: true}, {Text: " train"}},
}}}}
if !reflect.DeepEqual(res.Hits, want) {
t.Fatalf("hits = %+v, want %+v", res.Hits, want)
}
// A zero Highlight sends the two tags and nothing else new.
if _, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{Q: "blue", QueryBy: []string{"title"}, Highlight: &beachcomber.Highlight{}}); err != nil {
t.Fatal(err)
}
params = onlyQuery(t, f)
for k := range params {
switch k {
case "q", "query_by", "highlight_start_tag", "highlight_end_tag":
default:
t.Fatalf("zero Highlight sent %s (query %v)", k, params)
}
}
if !params.Has("highlight_start_tag") || !params.Has("highlight_end_tag") {
t.Fatalf("zero Highlight must send the tags: %v", params)
}
}
func TestTypesenseHighlightsKeepStoredText(t *testing.T) {
for name, snippet := range map[string]string{
"literal": `<b>bold</b> &amp; blue &lt;x`,
"escaped": `<b>bold</b> &amp; blue &lt;x`,
} {
t.Run(name, func(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":1,"hits":[{"document":{"id":"1"},"highlights":[{"field":"body","snippet":"`+snippet+`"}]}]}`)
res, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{Q: "blue", QueryBy: []string{"body"}, Highlight: &beachcomber.Highlight{}})
if err != nil {
t.Fatal(err)
}
if len(res.Hits) != 1 || len(res.Hits[0].Highlights) != 1 {
t.Fatalf("hits = %+v", res.Hits)
}
want := []beachcomber.Segment{{Text: "<b>bold</b> &amp; "}, {Text: "blue", Match: true}, {Text: " &lt;x"}}
got := res.Hits[0].Highlights[0].Segments
if !reflect.DeepEqual(got, want) {
t.Fatalf("segments = %+v, want %+v", got, want)
}
for _, s := range got {
if strings.Contains(s.Text, "mark") {
t.Fatalf("segment carries markup: %q", s.Text)
}
}
})
}
}
func TestTypesenseHitsStayInStepWithIDs(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":5,"hits":[`+
`{"document":{"id":"12"},"highlights":[{"field":"title","snippet":"x"},{"field":"notes","snippet":5}]},`+
`{"document":{"id":null},"highlights":[{"field":"title","snippet":"y"}]},`+
`{"document":{},"highlights":[{"field":"title","snippet":"z"}]},`+
`{"document":{"id":7}},`+
`{"document":{"id":"3"},"highlights":[]}]}`)
res, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{Q: "x", QueryBy: []string{"title"}, Highlight: &beachcomber.Highlight{}})
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(res.IDs, []string{"12", "7", "3"}) || len(res.Hits) != 3 {
t.Fatalf("res = %+v", res)
}
for i, h := range res.Hits {
if h.ID != res.IDs[i] {
t.Fatalf("hits[%d].ID = %q, ids[%d] = %q", i, h.ID, i, res.IDs[i])
}
}
if len(res.Hits[0].Highlights) != 1 || res.Hits[0].Highlights[0].Field != "title" {
t.Fatalf("hits[0] = %+v (the undecodable notes entry is dropped alone)", res.Hits[0])
}
if len(res.Hits[1].Highlights) != 0 || len(res.Hits[2].Highlights) != 0 {
t.Fatalf("hits = %+v", res.Hits)
}
f.answer(searchKey, 200, `{"found":0,"hits":[]}`)
res, err = e.SearchPage(t.Context(), "posts", beachcomber.Query{Q: "x", Highlight: &beachcomber.Highlight{}})
if err != nil || res.Hits == nil || len(res.Hits) != 0 {
t.Fatalf("empty answer = %+v %v", res, err)
}
}
func TestTypesenseHighlightsStringArray(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":2,"hits":[`+
`{"document":{"id":"1"},"highlights":[{"field":"tracks","snippets":["blue note","x blue"],"indices":[2,0],"matched_tokens":[["blue"],["blue"]]}]},`+
`{"document":{"id":"2"},"highlights":[`+
`{"field":"title","snippet":"blue train"},`+
`{"field":"tags","snippets":["blue"]}]}]}`)
q := beachcomber.Query{Q: "blue", QueryBy: []string{"title", "tracks", "tags"}, Highlight: &beachcomber.Highlight{}}
res, err := e.SearchPage(t.Context(), "posts", q)
if err != nil {
t.Fatal(err)
}
params := onlyQuery(t, f)
for k := range params {
if strings.Contains(k, "snippet") || strings.Contains(k, "max") {
t.Fatalf("a snippet-cap parameter was sent: %s", k)
}
}
want := []beachcomber.Snippet{
{Field: "tracks", ArrayIndex: 2, Segments: []beachcomber.Segment{{Text: "blue", Match: true}, {Text: " note"}}},
{Field: "tracks", ArrayIndex: 0, Segments: []beachcomber.Segment{{Text: "x "}, {Text: "blue", Match: true}}},
}
if !reflect.DeepEqual(res.Hits[0].Highlights, want) {
t.Fatalf("tracks = %+v, want %+v", res.Hits[0].Highlights, want)
}
second := res.Hits[1].Highlights
if len(second) != 2 || second[0].Field != "title" || second[0].ArrayIndex != -1 ||
second[1].Field != "tags" || second[1].ArrayIndex != -1 {
t.Fatalf("mixed hit = %+v (order of the highlights array, -1 without indices)", second)
}
q.Highlight.MaxSnippetsPerArray = 1
res, err = e.SearchPage(t.Context(), "posts", q)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(res.Hits[0].Highlights, want[:1]) {
t.Fatalf("capped = %+v, want %+v", res.Hits[0].Highlights, want[:1])
}
params = onlyQuery(t, f)
for k := range params {
if strings.Contains(k, "snippet") || strings.Contains(k, "max") {
t.Fatalf("the cap was sent as %s; it is client-side", k)
}
}
}
func TestTypesenseFieldProjection(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":1,"hits":[{"document":{"id":"9"},"highlights":[{"field":"body","snippet":"the blue one"}]}]}`)
for _, tc := range []struct {
include []string
want string
}{
{[]string{"title"}, "title,id"},
{[]string{"id", "title"}, "id,title"},
} {
if _, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{Q: "x", IncludeFields: tc.include}); err != nil {
t.Fatal(err)
}
if got := onlyQuery(t, f).Get("include_fields"); got != tc.want {
t.Fatalf("include_fields for %v = %q, want %q", tc.include, got, tc.want)
}
}
res, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{
Q: "blue", QueryBy: []string{"body"}, ExcludeFields: []string{"body"},
Highlight: &beachcomber.Highlight{Fields: []string{"body"}},
})
if err != nil {
t.Fatal(err)
}
params := onlyQuery(t, f)
if params.Get("exclude_fields") != "body" || params.Get("highlight_fields") != "body" || params.Has("include_fields") {
t.Fatalf("params = %v", params)
}
if len(res.Hits) != 1 || len(res.Hits[0].Highlights) != 1 || res.Hits[0].Highlights[0].Field != "body" {
t.Fatalf("an excluded field must still be highlighted: %+v", res.Hits)
}
for name, q := range map[string]beachcomber.Query{
"exclude_fields": {ExcludeFields: []string{"body", "id"}},
"AffixTokens": {Highlight: &beachcomber.Highlight{AffixTokens: -1}},
"MaxSnippetsPerArray": {Highlight: &beachcomber.Highlight{MaxSnippetsPerArray: -1}},
} {
_, err := e.SearchPage(t.Context(), "posts", q)
if err == nil || !strings.Contains(err.Error(), name) {
t.Fatalf("%s: err = %v", name, err)
}
}
if calls := f.take(); len(calls) != 0 {
t.Fatalf("a rejected query sent %d requests", len(calls))
}
}
func TestTypesenseSupportsHighlights(t *testing.T) {
_, e := newFake(t)
if !beachcomber.SupportsHighlights(e) {
t.Fatal("the Typesense engine supports highlights")
}
var _ beachcomber.HighlightSearcher = e
}

View File

@@ -120,7 +120,7 @@ asks first; --force skips the question. Off a terminal it never asks.`,
if err := confirmProduction(app, in, out, fmt.Sprintf("Delete every pending job of queue %q in production?", queue)); err != nil {
return err
}
return withDB(ctx, app, func() error {
return withAppDB(ctx, app, func() error {
return clearQueue(ctx, app, queue, out)
})
},
@@ -336,7 +336,9 @@ func dueAt(c pact.Cadence, t time.Time) bool {
}
// withAppDB runs fn with the app's database: the published one when there is
// one, otherwise one opened for the command the way withDB does.
// one, otherwise one opened for the command the way withDB does. It serves
// schedule:run --once and queue:clear, which may run in-process on an app
// that already published its pool.
func withAppDB(ctx context.Context, app *backpack.App, fn func() error) error {
if gdb, ok := app.Lookup[*gorm.DB](); ok && gdb != nil {
return fn()

View File

@@ -19,7 +19,7 @@ import (
// none; a running job and other queues are left alone.
func TestQueueClear(t *testing.T) {
db, dsn := migratedDB(t)
app, gdb := testApp(t, db, dsn, map[string]any{"queue.queues.default": 1})
app, gdb := testApp(t, db, dsn, map[string]any{"queue.queues.default": 1, "app.key": testAppKey})
m, err := From(app)
if err != nil {
t.Fatal(err)
@@ -73,6 +73,21 @@ func TestQueueClear(t *testing.T) {
if !strings.Contains(out, `Clearing queue "default"`) || !strings.Contains(out, "Cleared 2 jobs") {
t.Fatalf("output = %q", out)
}
// In process, on the app that already published its pool (a plugin
// HasSchedule entry naming queue:clear): the block still holds the one
// worker slot, so these two stay pending.
for range 2 {
if err := m.Enqueue(ctx, nil, pingArgs{Tag: "pending-in-process"}, EnqueueOpts{}); err != nil {
t.Fatal(err)
}
}
var inProcess bytes.Buffer
if err := bonfire.Call(ctx, RuntimeCommands(app, nil), "queue:clear", nil, &inProcess); err != nil {
t.Fatalf("in-process queue:clear: %v\n%s", err, inProcess.String())
}
if !strings.Contains(inProcess.String(), "Cleared 2 jobs") {
t.Fatalf("in-process output = %q", inProcess.String())
}
close(release)
deadline := time.Now().Add(10 * time.Second)
for countRows(t, gdb, `SELECT count(*) FROM river_job WHERE kind = ? AND state = 'completed'`, "conga_test_block") != 1 {

View File

@@ -254,6 +254,44 @@ func TestScheduleRunsCommand(t *testing.T) {
logs.waitFor(t, slog.LevelWarn, "schedule: command not registered; skipping", "command", "acme:missing")
}
// TestScheduledDeferredPurgeInProcess runs the framework entry deferred:purge
// the way serve's worker does (callScheduled through the published catalog)
// and through schedule:run --once, both on an app whose pool is already
// published. Both used to fail with "backpack: duplicate provider for
// *sql.DB".
func TestScheduledDeferredPurgeInProcess(t *testing.T) {
db, dsn := migratedDB(t)
app, _ := testApp(t, db, dsn, map[string]any{"app.key": testAppKey})
logs := &captureHandler{}
if err := app.Publish(slog.New(logs)); err != nil {
t.Fatal(err)
}
if err := app.Publish(bonfire.NewCatalog(lagoon.RuntimeCommands(app, nil))); err != nil {
t.Fatal(err)
}
var buf bytes.Buffer
skipped, err := callScheduled(t.Context(), app, loggerFromApp(app), "deferred:purge", nil, &buf)
if skipped || err != nil {
t.Fatalf("callScheduled: skipped=%v err=%v\n%s", skipped, err, buf.String())
}
if !strings.Contains(buf.String(), "purged 0 deferred bindings older than 5 days") {
t.Fatalf("output = %q", buf.String())
}
if !logs.find(slog.LevelInfo, "schedule: command finished", "command", "deferred:purge") {
t.Fatal("no Info 'schedule: command finished' record for deferred:purge")
}
if err := db.PingContext(t.Context()); err != nil {
t.Fatalf("published pool closed by the command: %v", err)
}
out, err := onceRun(t, app, nil, mustTime(t, time.UTC, "2026-03-10 03:00:00"))
if err != nil {
t.Fatalf("schedule:run --once: %v\n%s", err, out)
}
if !strings.Contains(out, "Ran 1 due entries (0 failed)") {
t.Fatalf("schedule:run --once output = %q", out)
}
}
// onceRun runs `schedule:run --once` for plugins at the given app time and
// returns its output.
func onceRun(t *testing.T, app *backpack.App, plugins []party.Plugin, now time.Time) (string, error) {

View File

@@ -483,7 +483,16 @@ func purgeBucket(ctx context.Context, app *backpack.App) (*blob.Bucket, func(),
return b, func() { _ = b.Close() }, nil
}
// withDB runs fn on the app's published database when there is one: an
// in-process run (the scheduler in serve or schedule:run, through
// bonfire.Catalog.Call) borrows that pool and leaves it open. Otherwise it
// opens and publishes a pool for the CLI run and closes it when fn returns.
func withDB(ctx context.Context, app *backpack.App, fn func(*gorm.DB) error) error {
if app != nil {
if gdb, ok := app.Lookup[*gorm.DB](); ok && gdb != nil {
return fn(gdb)
}
}
sqlDB, gdb, err := OpenFromApp(ctx, app)
if err != nil {
return err

View File

@@ -501,3 +501,50 @@ func TestPurgeDeferredCommand(t *testing.T) {
t.Fatal("the command left the file, its blob or a binding")
}
}
// TestPurgeDeferredCommandPublishedDB runs deferred:purge the way the
// in-process scheduler does: through bonfire.Call on an app whose pool serve
// already opened and published. The command used to publish a second pool
// and fail with "backpack: duplicate provider for *sql.DB"; it now runs on
// the published handle and leaves that pool open.
func TestPurgeDeferredCommandPublishedDB(t *testing.T) {
e := newPurgeEnv(t, "lagoon_purge_published")
f := e.file(t, "")
created := e.child(t, "created")
e.bind(t, "cover", DeferredFileType, fmt.Sprint(f.ID), nil, 3*day)
e.bind(t, "children", "acme_purge_children", fmt.Sprint(created), &DeferredEnvelope{Created: true}, 3*day)
app := backpack.New(purgeConfig(t, map[string]any{"database.dsn": e.dsn, "app.key": base64.StdEncoding.EncodeToString(bytes.Repeat([]byte{7}, 32))}))
sqlDB, gdb, err := OpenFromApp(t.Context(), app)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sqlDB.Close() })
if err := Publish(app, sqlDB, gdb); err != nil {
t.Fatal(err)
}
if err := attach.Publish(app, e.bucket); err != nil {
t.Fatal(err)
}
plugins := []party.Plugin{purgePlugin{models: []any{&purgeChild{}}}}
var out bytes.Buffer
if err := bonfire.Call(t.Context(), RuntimeCommands(app, plugins), "deferred:purge", []string{"--days", "2"}, &out); err != nil {
t.Fatalf("deferred:purge: %v\n%s", err, out.String())
}
if !strings.Contains(out.String(), "purged 2 deferred bindings older than 2 days (1 files, 1 children, 0 skipped)") {
t.Fatalf("output = %q", out.String())
}
if e.fileExists(t, f.ID) || e.blobExists(t, f) || len(e.bindingIDs(t)) != 0 {
t.Fatal("the command left the file, its blob or a binding")
}
if err := sqlDB.PingContext(t.Context()); err != nil {
t.Fatalf("published pool closed by the command: %v", err)
}
if got, ok := app.Lookup[*gorm.DB](); !ok || got != gdb {
t.Fatalf("published *gorm.DB replaced: %p, want %p", got, gdb)
}
out.Reset()
if err := bonfire.Call(t.Context(), RuntimeCommands(app, plugins), "migrate:status", nil, &out); err != nil {
t.Fatalf("migrate:status: %v\n%s", err, out.String())
}
}

View File

@@ -44,7 +44,7 @@ Use cases:
- Widening on return (A-WR-04): when a binding comes back with a different identity (more or fewer scopes, a backend subject added or dropped), its stateful session or stateless server is re-filtered on that request, so newly allowed tools and prompts appear at once (with `list_changed`) instead of waiting for an event. Token ids play no part in this, so an access-token rotation alone re-filters nothing. A stateless server is shared by every live token of its binding, so a return with less reach only adds there and never removes: the receiving middleware hides, per request, what that request's identity may not use, and a call of a wider token is never cut off by a narrower one. A guard provider error during this re-filtering is not a refusal: the cached server stays as it was and the next request retries (logged as `pier: widening deferred`); a fan-out recompute that meets one hides the failing items and leaves the server to be re-filtered by the binding's next request.
- Protected resource metadata (RFC 9728): `GET /.well-known/oauth-protected-resource` and `GET /.well-known/oauth-protected-resource<mcp.path>` answer the resource, the authorization server (the published verifier's resource origin, else `app.url`), the supported scopes and the bearer method. When the published `pact.MCPTokenVerifier` also implements `pact.MCPAdvertisedScopes` and `pact.MCPAdvertisedScopes.MCPScopesSupported` is non-empty, that slice is copied; otherwise the document keeps `pier.PRMScopes`. pier does not read `pact.MemberTokenScopeCatalog` and does not widen the read/write ceiling. A `mcp.resource` that differs from the verifier's resource is a boot warning. The list the OAuth server plugin publishes is the MCP advertised set: the fixed names `read`, `write`, `ai`, `offline_access` and `backend` are always on it. Declared member scopes follow the plugin's `golem15.oauthserver.mcp_scopes` filter, which is that plugin's key and not a framework `mcp.*` key: unset means every declared member scope, a set filter means only those declared names, and an empty filter means the five fixed names only. An empty `wristband.Options.ScopesSupported` on the MCP flavour resolves to those five names; a non-empty list is used as given. OpenID Connect discovery stays `openid`, `profile` and `email`. A route reads who called a tool with `pact.MCPCallFrom`.
- Prompts (D-28): prefixed and filtered like tools, required arguments checked before `Render` runs; a prompt never dispatches a route. A `Render` error is logged server-side and answered with the fixed JSON-RPC error `pier: the prompt could not be rendered`, so no database or provider error reaches the agent. A prompt declared with `Writes` (it directs tools that are not read-only) sits under the read/write ceiling and is hidden from an identity without `write`.
- A 401 carries `WWW-Authenticate: Bearer resource_metadata="<app.url>/.well-known/oauth-protected-resource<mcp.path>", scope="read"`; a browser request whose `Origin` is not in `mcp.allowed_origins` is refused with 403 before any body is read.
- A 401 carries `WWW-Authenticate: Bearer resource_metadata="<app.url>/.well-known/oauth-protected-resource<mcp.path>"`; there is no scope parameter, so a client requests the `scopes_supported` of the protected resource metadata; a browser request whose `Origin` is not in `mcp.allowed_origins` is refused with 403 before any body is read.
- One audit line per tool call (`pier: tool call`: tool, guard, user id, admin id, client id, token kind, session prefix, status, duration) and one per session step (`pier: session`: `open`, `rehydrate`, `end`, `binding_refused`, `revoked`, `stream_closed`, `expired`), never arguments, bodies or tokens. A tool call served on the stateful path without a session logs the session `ephemeral`, one on the stateless path `stateless`.
## Usage

View File

@@ -247,8 +247,12 @@ func (p *Pier) bodyLimit(next http.Handler) http.Handler {
// bearer verifies the bearer of every request afresh and stores the
// resolved Identity on the request for the session router and the SDK
// handlers; a failure is a 401 carrying WWW-Authenticate with the
// protected resource metadata URL and the read scope.
// handlers. A failure is a 401 whose WWW-Authenticate names only the
// protected resource metadata URL. It carries no scope parameter, so a
// client requests the scopes the metadata document advertises and the
// authorization server's consent decides the grant; tools are then filtered
// per granted scope. No Scopes option is set: the SDK would answer 403 to
// every token lacking them.
func (p *Pier) bearer(next http.Handler) http.Handler {
verify := func(ctx context.Context, token string, r *http.Request) (*auth.TokenInfo, error) {
ctx = towel.WithClientIP(ctx, p.clientIP(r))
@@ -263,13 +267,10 @@ func (p *Pier) bearer(next http.Handler) http.Handler {
Extra: map[string]any{extraIdentity: id, extraCall: callInfoFrom(r)},
}, nil
}
h := auth.RequireBearerToken(verify, &auth.RequireBearerTokenOptions{
return auth.RequireBearerToken(verify, &auth.RequireBearerTokenOptions{
ResourceMetadataURL: p.prmURL,
AllowMissingExpiration: true,
})(next)
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h.ServeHTTP(&challengeWriter{ResponseWriter: w}, r)
})
}
// clientIP is the request's client address: the host's answer when it
@@ -285,30 +286,6 @@ func (p *Pier) clientIP(r *http.Request) string {
return r.RemoteAddr
}
// challengeWriter completes the 401 challenge with scope="read", which the
// SDK middleware would otherwise enforce on every request.
type challengeWriter struct {
http.ResponseWriter
}
func (c *challengeWriter) WriteHeader(status int) {
if status == http.StatusUnauthorized {
h := c.Header()
if v := h.Get("WWW-Authenticate"); v != "" && !strings.Contains(v, "scope=") {
h.Set("WWW-Authenticate", v+`, scope="read"`)
}
}
c.ResponseWriter.WriteHeader(status)
}
func (c *challengeWriter) Flush() {
if f, ok := c.ResponseWriter.(http.Flusher); ok {
f.Flush()
}
}
func (c *challengeWriter) Unwrap() http.ResponseWriter { return c.ResponseWriter }
// identityFromRequest reads the Identity the bearer middleware stored.
func identityFromRequest(r *http.Request) (Identity, bool) {
ti := auth.TokenInfoFromContext(r.Context())

View File

@@ -96,21 +96,31 @@ func TestPierTracerSmoke(t *testing.T) {
t.Fatalf("text content = %+v", res.Content[0])
}
// A bad bearer is a 401 carrying the protected resource metadata challenge.
// A bad or missing bearer is a 401 whose challenge names only the
// protected resource metadata URL: no scope parameter, so a client
// requests the scopes the metadata document advertises.
init := `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"x","version":"1"}}}`
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, srv.URL+"/mcp", bytes.NewBufferString(init))
req.Header.Set("Authorization", "Bearer oas_at_bad")
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json, text/event-stream")
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
_, _ = io.Copy(io.Discard, resp.Body)
_ = resp.Body.Close()
challenge := resp.Header.Get("WWW-Authenticate")
if resp.StatusCode != http.StatusUnauthorized || !strings.Contains(challenge, `resource_metadata="http://acme.test/.well-known/oauth-protected-resource/mcp"`) || !strings.Contains(challenge, `scope="read"`) {
t.Fatalf("bad bearer = %d %q", resp.StatusCode, challenge)
const wantChallenge = `Bearer resource_metadata="http://acme.test/.well-known/oauth-protected-resource/mcp"`
for _, tc := range []struct{ name, authorization string }{
{"bad bearer", "Bearer oas_at_bad"},
{"missing bearer", ""},
} {
req, _ := http.NewRequestWithContext(ctx, http.MethodPost, srv.URL+"/mcp", bytes.NewBufferString(init))
if tc.authorization != "" {
req.Header.Set("Authorization", tc.authorization)
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json, text/event-stream")
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
_, _ = io.Copy(io.Discard, resp.Body)
_ = resp.Body.Close()
challenge := resp.Header.Get("WWW-Authenticate")
if resp.StatusCode != http.StatusUnauthorized || challenge != wantChallenge || strings.Contains(challenge, "scope=") {
t.Fatalf("%s = %d %q, want 401 %q", tc.name, resp.StatusCode, challenge, wantChallenge)
}
}
// The same tool declared twice fails Mount naming the exposed name.