diff --git a/.planning/phases/16.1-journal-admin-urgent-fixes-relation-search-list-state-post-f/16.1-RESEARCH.md b/.planning/phases/16.1-journal-admin-urgent-fixes-relation-search-list-state-post-f/16.1-RESEARCH.md new file mode 100644 index 0000000..305602f --- /dev/null +++ b/.planning/phases/16.1-journal-admin-urgent-fixes-relation-search-list-state-post-f/16.1-RESEARCH.md @@ -0,0 +1,677 @@ +# Phase 16.1: Journal admin urgent fixes: relation search, list state, post form content and excerpt - Research + +**Researched:** 2026-10-09 +**Domain:** summercms.go `modules/cabana` (GORM list queries, admin JSON API, Vue admin SPA), `sm-journal-plugin` (post save path), UAT release +**Confidence:** HIGH for the three root causes (each reproduced by a probe or read from the UAT database); MEDIUM for the list-state design (Claude's discretion area) + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions +- **D-01: List state lives in both the URL and the server session.** Sort, filters, search and page are reflected in the list URL query string (shareable, bookmarkable, back-button friendly). When the list is opened without list-state query parameters (for example from the admin navigation), the last state stored in the user's session for that list is restored. When the URL carries list-state parameters, the URL wins and also updates the session. User's words: "I may go back to Posts list through navigation and it's great if my filters were remembered + sharable URLs are nice to have." +- **D-02: The relation-search fix is a framework fix in summercms.go `modules/cabana`**, generic for belongsToMany (pivot) relations, not a Journal-specific workaround. Another session is concurrently working in `modules/`; plans must keep cabana changes in their own commits, stage files explicitly by path, and check for conflicting in-flight edits before touching shared cabana files. +- **D-03: Content and excerpt must show the stored values without loss.** Root cause is to be established by research (suspects: translatable-field overlay replacing host values with empty translations; the markdown widget not displaying imported HTML). Saving a post from the admin must not destroy imported HTML (see todo `journal-imported-html-admin-resave.md`); if a full fix needs the Blog Editor, 16.1 at least prevents data loss and 16.2 delivers the editor. +- **D-04: Ship to UAT.** Release a summercms tag carrying the cabana fixes (unless the concurrent session's release already includes them), bump `sm-grzybyfunkcjonalne-app`, redeploy to uat.grzybyfunkcjonalne.pl per its DEPLOY.md, and re-run `scripts/smoke.sh --url` on the server. + +### Claude's Discretion +- Session storage mechanism and query-parameter naming for list state, following existing cabana conventions. +- Test strategy, within the lean-mode rule that unit tests are the last plan of the phase. + +### Deferred Ideas (OUT OF SCOPE) +- Plugin-provided field types and the Blog Editor formwidget (Phase 16.2). (CONTEXT.md has no separate Deferred section; this is its "Out of scope" line.) + + + +## Phase Requirements + +No requirement IDs are mapped. The CONTEXT decisions are the requirement set. + +| ID | Description | Research Support | +|----|-------------|------------------| +| D-01 | List sort, filters, search and page in the URL and a per-user server store; restore when the list opens without params | "List state" pattern: the URL params already exist; add the `backend_list_states` table, a `GET .../list-state` restore route, a save inside the list handler, and the ListView entry logic | +| D-02 | Generic cabana fix for search over belongsToMany (and sort over to-many) relations | Root cause reproduced: `db.Joins("Categories")` on a many2many builds `"Categories"."post_id"`. Fix: correlated `EXISTS` subqueries built from GORM's parsed relationship | +| D-03 | Post form shows the stored content and excerpt, and a save never destroys imported HTML | Root cause 1: the first enabled locale is `en`, and no `en` translations exist. Fix: default locale first in `meta.enabledLocales`. Root cause 2: `FormatHTML` drops raw HTML. Fix: `html.WithUnsafe()` plus the existing gate. 2 UAT posts already damaged; repair them after the deploy | +| D-04 | Tag, bump, redeploy, smoke | Release mechanics: the host builds against the `../summercms.go` working tree through a replace. The bump is the journal submodule pointer. The new migration needs `bin/grzybyfunkcjonalne migrate` on the server | + + +## Project Constraints (from CLAUDE.md) + +- Go 1.27, standard library first; add a dependency only when the research doc or a phase decision names it. **This phase adds none.** `github.com/yuin/goldmark/renderer/html` is a sub-package of goldmark v1.8.6, which sm-journal-plugin already requires. +- `go vet` and `go test ./...` must be green at every commit. +- Compiled plugins only; no runtime plugin loading. +- API parity: do not "improve" Journal API response shapes. The `content`/`content_html` contract stays unchanged (todo file). +- Unit tests are always the **last plan** of the phase. Before writing plans, present the plan count with one-line scopes (checkpoint). +- Commits: + - no co-author tags; + - one logical change per commit; + - planning docs and code in separate commits. +- D-02 adds three rules for this phase: + - cabana changes go in their own commits; + - stage files by explicit path; + - check for in-flight edits before touching shared cabana files. +- A change to a `modules/` package's exported API, config keys, CLI commands or dependencies updates that module's `README.md` in the same change. It also updates the affected `docs/` pages (`docs/backend/lists-and-filters.md`). `go test ./cmd/summer -run TestDocsTree` and `summer docs:build --check` must pass. Every identifier named in docs must exist. +- Framework READMEs never name a consuming application. Use `blog`/`acme`, not Journal or grzybyfunkcjonalne, in cabana or lagoon docs. +- SPA changes: rebuild `modules/boardwalk/dist` (`npm --prefix admin run build`) and commit it (`scripts/check-admin-dist.sh` is the drift gate). Swag annotation changes: regenerate `admin/openapi/admin.json` and `admin/src/api/schema.d.ts` (`scripts/check-admin-openapi.sh`, `--check` to verify). +- The PHP originals are not changed. Core plugin contracts (user, blog, pages, payment) stay unchanged. + +## Summary + +Each of the four UAT defects has one confirmed cause. + +**(1) Relation search fails.** `applyListSearch` joins every searched relation with `db.Joins(field)` [VERIFIED: modules/cabana/query.go:351-357]. GORM's association `Joins` only fits to-one relations. On a many2many it emits `LEFT JOIN "golem15_journal_categories" "Categories" ON "golem15_journal_posts"."id" = "Categories"."post_id" AND "golem15_journal_posts"."category_id" = "Categories"."id"`, which is the UAT error, reproduced this session with a GORM v1.31.2 DryRun probe. + +`applyListOrder` has the same bug for sorting by a to-many column (query.go:309). The code that ran on UAT is v0.1.15. Since v0.1.9, query.go only gained the to-many display join (`relatedSelectMany`). The bug is unchanged at HEAD. + +The fix that matches Winter is a correlated `EXISTS` subquery per relation search term. Winter's Lists widget uses `orWhereHas` for search [VERIFIED: examples/golem15-wintercms-starter/modules/backend/widgets/Lists.php:529-531]. Build the subquery from GORM's own parsed relationship (`schema.Relationship.References` / `JoinTable`), not from guessed names. + +**(2) List state is lost.** The list state already lives in the URL query (`search`, `sort`, `dir`, `page`, `per_page`, `filter[]`) [VERIFIED: admin/src/app/listQuery.ts; modules/cabana/query.go:62-84]. The form's back link, close and save redirects, and the navigation all go to the bare list path (`go(listPath)`, FormView.vue:60/403/439), so the state is dropped. + +There is no server session: admin auth is a stateless JWT cookie whose jti rotates on refresh. The table that would play this role in Winter, `backend_user_preferences`, was explicitly not ported in Phase 9; a test asserts it is absent. Recommendation: a small framework table `backend_list_states` keyed (user_id, controller), a `GET .../list-state` restore route, and a best-effort save inside the existing list handler after a successful query. ListView restores before its first load whenever it is entered with no list params. + +**(3)+(4) The form shows empty content and excerpt.** The data is fine. On UAT, `golem15_translate_locales` has `en|f|t|1` and `pl|t|t|2`: en is enabled, sorted first, and not the default. There are **0** rows in `golem15_translate_attributes`. `EnabledLocales` returns codes `ORDER BY sort_order` → `["en","pl"]`. Every ML control opens on `locales[0]`, which is `en`, so the form shows the empty en values of title, slug, content and excerpt. The record heading hides this for the title, because it falls back to the first non-empty locale. + +Fix in cabana: send the default locale first in `meta.enabledLocales`. This one server function fixes the form, the preview screen and relation child modals. + +Separately, **saving destroys imported HTML**. `preparePostWrite` runs `classes.FormatHTML(post.Content)`, which is goldmark *without* `html.WithUnsafe()`. Probe this session: it turns all 100 imported pl bodies into ``. With `html.WithUnsafe()` it returns all 100 unchanged (trim-equal), and the existing 3-pattern gate still rejects script, iframe, event handlers and javascript/data URLs. This is PHP parity: Winter's `Markdown::parse` passes raw HTML through. + +UAT already has **2 posts whose `content_html` was wiped at import**. They have no pl translation and raw HTML in the base content: +- id 53 `czy-amanita-muscaria-jest-legalna` (draft); +- id 64 `ekstrakt-z-turkey-tail` (**published**, missing its author footer). + +Re-render both after the fix ships. + +**Primary recommendation:** five changes, then release v0.1.16 and redeploy: +1. cabana: relation search and sort through `EXISTS` and correlated subqueries built from `gorm.Statement.Parse` relationships. +2. cabana: default locale first in `enabledLocales`. +3. cabana and SPA: list-state table, restore route and ListView entry logic. +4. sm-journal-plugin: `FormatHTML` with `html.WithUnsafe()` behind the existing gate. +5. After the deploy: re-save posts 53 and 64. + +## Architectural Responsibility Map + +| Capability | Primary Tier | Secondary Tier | Rationale | +|------------|-------------|----------------|-----------| +| Relation search/sort SQL | API / Backend (cabana `query.go`) | Database (Postgres EXISTS) | Allowlisted SQL is built only server-side; the SPA sends only names | +| List state persistence | API / Backend (cabana list handler + new route) | Database (`backend_list_states`) + Browser (ListView entry logic, URL) | D-01 asks for server storage; the URL stays the shareable source | +| Restore-on-entry decision | Browser (ListView) | API (restore route sanitizes) | Only the SPA knows whether the list was entered from navigation or changed by the user | +| ML default locale ordering | API / Backend (cabana `enabledContentLocales`) | Browser (MLField uses `[0]`) | A single server ordering fixes every SPA consumer of `meta.enabledLocales` | +| content → content_html rendering | API / Backend (sm-journal-plugin `classes.FormatHTML`) | — | Shared by admin save, import and Journal API writes | +| Release / deploy | Host repo + server ops | — | The host compiles the framework working tree through a replace | + +## Standard Stack + +### Core (all already in go.mod / package.json; versions read this session) +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| gorm.io/gorm | v1.31.2 | `gorm.Statement.Parse` → `schema.Relationship` (Type, References, JoinTable, FieldSchema) for the EXISTS builder | Already the ORM; its parsed relationship is the only reliable source of pivot/fk names [VERIFIED: probe output below] | +| gorm.io/gorm/schema | v1.31.2 | `schema.Many2Many` ("many_to_many"), `schema.HasMany` ("has_many"), `schema.BelongsTo`, `schema.HasOne` | Relation type switch [VERIFIED: probe printed `type: many_to_many`, `has_many`] | +| github.com/yuin/goldmark | v1.8.6 (sm-journal-plugin go.mod) | `goldmark.WithRendererOptions(html.WithUnsafe())` from `github.com/yuin/goldmark/renderer/html` | Same module, new import only [VERIFIED: probe compiled against the cached v1.8.6] | +| gormigrate/v2 | v2.1.7 | New migration appended to `lagoon.BackendAdminMigrations` | Existing framework migration list (`summercms.cabana` history) [VERIFIED: modules/lagoon/backend_admin_migrations.go:21, migrations.go:75] | +| vue-router | 5.1.0 | `router.replace` for restore, as ListView already does | Existing [VERIFIED: admin/package.json] | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| EXISTS per relation | `Joins` + `DISTINCT` / `GROUP BY` | Duplicate rows inflate `Count`, unqualified outer predicates (`id IN (...)` in Journal `FilterCategories`, `user_id = ?` in `restrictToOwnPosts`) become ambiguous; not Winter parity | +| `backend_list_states` table | `backend_users.metadata` JSON key | metadata is Winter's TEXT jsonable copied at cutover (202610090001); read-modify-write races between tabs; pollutes a copied column | +| `backend_list_states` table | Winter `backend_user_preferences` | Phase 9 D-01 left it unported and `backend_admin_migrations_test.go:116` asserts it is absent; reviving it reverses a decision and its non-unique `user_item_index` cannot back an upsert | +| Server-side table | Browser `sessionStorage` | Contradicts D-01 ("server session") | +| Default-locale-first in cabana | Reorder in sm-translate-plugin `AdminWriter.EnabledLocales` | Plugin-specific; any other TranslationWriter would keep the bug | +| Default-locale-first | New `meta.defaultLocale` + SPA changes | More SPA surface and OpenAPI change; same user-visible result. Dropdown order changes from sort_order to default-first (cosmetic) | +| `html.WithUnsafe()` | Detect HTML (`^\s*<`) and store as-is through `CheckHTML` | Heuristic, and post 64 is markdown with trailing raw HTML (would still be wiped); not PHP parity | + +**Installation:** none, because no new modules are added. + +## Package Legitimacy Audit + +No external packages are installed. `github.com/yuin/goldmark/renderer/html` is a package inside the already-required `github.com/yuin/goldmark v1.8.6`. No npm packages are added. + +**Packages removed due to [SLOP] verdict:** none +**Packages flagged as suspicious [SUS]:** none + +## Root-cause evidence + +### Defect 1: relation search (D-02) + +The search code [VERIFIED: modules/cabana/query.go:345-357, read this session]: +```go + field, ok := relationFieldName(model, col.Relation) + physical := listRelationColumn(controller, col.Relation, col.Select) + ... + // GORM aliases the joined association with the Go field name. + table = field + column = physical + if _, done := joined[field]; !done { + db = db.Joins(field) +``` +The sort code does the same, `db = db.Joins(field)` [VERIFIED: query.go:309]. + +The Journal model declares the relation as `Categories []Category \`gorm:"many2many:golem15_journal_posts_categories;" json:"-"\`` [VERIFIED: sm-journal-plugin/models/post.go:34]. The posts list column is: +```yaml + categories: + label: golem15.journal::lang.post.categories + relation: categories + select: name + searchable: true + sortable: false +``` +[VERIFIED: sm-journal-plugin/models/post/columns.yaml] + +GORM v1.31.2 DryRun probe (scratch module, same model shape), output pasted: +``` +JOINS: SELECT count(*) FROM "golem15_journal_posts" LEFT JOIN "golem15_journal_categories" "Categories" ON "golem15_journal_posts"."id" = "Categories"."post_id" AND "golem15_journal_posts"."category_id" = "Categories"."id" WHERE LOWER(CAST("Categories"."name" AS TEXT)) LIKE '%x%' +type: many_to_many jointable: golem15_journal_posts_categories + ref: fk=golem15_journal_posts_categories.post_id pk=golem15_journal_posts.id ownPrimaryKey=true pvalue="" + ref: fk=golem15_journal_posts_categories.category_id pk=golem15_journal_categories.id ownPrimaryKey=false pvalue="" +hasmany type: has_many field schema: comments + ref: fk=comments.post_id pk=golem15_journal_posts.id own=true +``` +The error happens at `query.go:129`, the `Count`, which is the first statement executed. + +The log names `summercms@v0.1.9`, but that is only the version sm-journal-plugin's go.mod *requires* (`git.golem15.com/golem15/summercms v0.1.9`). The code that ran is the `../summercms.go` working tree at 296b9cd, which descends from v0.1.15 with docs-only changes since. At HEAD, query.go differs from v0.1.9 only by `relatedSelectMany` (display), so **the bug is still present at HEAD**. + +There are no uncommitted `modules/` edits right now. The checkout is 13 docs commits ahead of origin/master. + +### Defect 2: list state (D-01) + +- **URL params already exist.** `listQueryFromRequest` reads `search`, `sort`, `dir`, `page`, `per_page` and `filter[]` (`listSearchTerm = "search"` [VERIFIED: modules/cabana/list_schema.go:20]; query.go:62-84). The SPA mirrors them in `parseListQuery`/`toListQuery` [VERIFIED: admin/src/app/listQuery.ts], and ListView changes them with `router.replace` (ListView.vue:114-116). +- **Where the state is dropped.** + - FormView: `const listPath = controllerPath(controllerId) ?? '/'` (FormView.vue:60), `await go(listPath)` after save (403), `await go(backPath.value)` for back (439). + - The navigation and breadcrumb links use the bare controller path. + - Browser Back works today. Every in-app return loses the state. +- **ListView start-up.** It loads through `watch(() => listQueryKey(query.value), ..., { immediate: true })` (ListView.vue:146-153). The first request is fired at setup, before any restore could run. +- **No server session.** + - Login mints a JWT into an HttpOnly cookie (`auth.go` `writeSessionCookie`). + - Refresh rotates the token and jti (`bouncer.RefreshAudienceFor`). + - Logout only blacklists the jti. + - Phase 9 left out Winter's preference table ("`backend_user_groups` and Winter's user-preference and access-log tables: not needed by any Golem15 plugin yet" [VERIFIED: .planning/phases/09-.../09-CONTEXT.md:126]), and `backend_admin_migrations_test.go:116` fails if `backend_user_preferences` exists. +- **What Winter does.** It keeps sort (`putSession('sort', ...)`), search term (`putSession('term', ...)`), filter scopes and `lastVisitedPage` in the session, and per_page in user preferences [VERIFIED: Lists.php:233,289,1773; Search.php:158; Filter.php:959]. + +### Defects 3 and 4: empty content and excerpt (D-03) + +Read-only UAT probe (ssh `grzybyfunkcjonalne-uat`, psql over the peer socket, SELECT only), output pasted: +``` +code|is_default|is_enabled|sort_order +en|f|t|1 +pl|t|t|2 +locale|model_type|count +(0 rows) +posts|content|html|excerpt|title|content_starts_html +104|104|104|100|104|101 +``` +- **Where the order comes from.** The translate seed inserts `SELECT 'en', 'English', TRUE, TRUE, 1` and `SELECT 'pl', 'Polski', FALSE, TRUE, 2` [VERIFIED: sm-translate-plugin/updates/202610060005_seed_en_pl_locales.go:13-18]. migrate.sh only runs `UPDATE golem15_translate_locales SET is_default = (code = 'pl')` (scripts/migrate.sh:173), so en stays enabled and first. +- **The writer keeps that order.** `AdminWriter.EnabledLocales` runs `Where("is_enabled = ?", true).Order("sort_order asc")` [VERIFIED: sm-translate-plugin/classes/admin_writer.go:31-35], and cabana copies the result into the form schema (`view.Meta.EnabledLocales = append([]string(nil), codes...)`, http.go:727-729). +- **The server sends the right data.** `hydrateMLRecord` puts the host column under the default locale (`if code == defaultLocale { out[code] = host`, field_ml.go:208-211). An en key with no stored translation is `out[code] = ""` (field_ml.go:217-219). +- **The SPA opens on the wrong locale.** + - `const activeLocale = ref(locales.value[0] ?? 'en')` (MLField.vue:30). + - `function firstMLLocale(): string { return schemaEnabledLocales(schema.value?.meta)[0] ?? 'en' }` (FormView.vue:227-229). + - So every ML control (title, slug, content and excerpt) opens on `en`, which is empty. The user reported only content and excerpt, but title and slug are empty in the inputs as well. + +**Data loss on save.** +- The save path: `html, err := classes.FormatHTML(post.Content)` … `post.ContentHTML = html` (sm-journal-plugin/controllers/posts.go:213-219). The engine is `goldmark.New(goldmark.WithExtensions(extension.Footnote, extension.Table), goldmark.WithParserOptions(parser.WithAttribute()))`, with no unsafe renderer (classes/format_html.go:15-18). +- PHP parity: `formatHtml` stores `trim($input)` when `use_rich_editor` is on, otherwise `Markdown::enableFootnotes()->enableAttributes()->enableTables()->parse(trim($input))`, then `Html::clean` [VERIFIED: grzybyfunkcjonalne.pl/plugins/golem15/journal/models/Post.php:185-208]. Winter's Markdown passes raw HTML through [ASSUMED: Parsedown default, not re-read this session; consistent with the export, where 83/100 pl `content_html` equal `content` byte for byte and the other 17 differ only by `"` → `"`]. +- Probe over the 100 pl bodies of the journal export (goldmark v1.8.6, the same options), output pasted: +``` +total=100 safe_omitted=100 unsafe_exact=0 unsafe_trim_equal=100 +sample safe="\n" +sample unsafe="

a

\n

b

" +md unsafe="

T

\n

text x

\n" +``` +The last line shows why the existing `rejectUnsafe` gate must stay after the unsafe render. None of the 105 export bodies match any of the three gate patterns, so no imported post will be refused on save. + +**Already damaged on UAT** (read-only probe): +``` +wiped|edited_after_import|admins +2|0|1 +id 53 czy-amanita-muscaria-jest-legalna published=f content_html = +id 64 ekstrakt-z-turkey-tail published=t content_html ends with 3x (author