summercms/.planning/phases/05-data-layer-full-fidelity/05-CONTEXT.md

# Phase 5: Data layer full fidelity - Context

**Gathered:** 2026-09-18
**Status:** Ready for planning

<domain>
## Phase Boundary

Phase 5 ports the whole Płytarium data layer: all 25 models of `golem15.fonoteka` (plus the `golem15.user` tables they depend on) with matching table names, columns, indexes, defaults, relations, casts and lifecycle hooks, and a Go migration set whose final schema equals PHP's. Requirements DATA-03 to DATA-11 and CLI-03. Roadmap mode: mvp. The phase is security-load-bearing: mass-assignment boundaries and encrypted-at-rest credentials are named invariants in the PHP source, so the security-review agent runs on it and the DTO-vs-model convention applies from the first model.

Two repos. `summercms.go` grows `lagoon` with what the models pull in: lifecycle hooks and transactional soft-delete cascade, relation helpers with ordered results and pivot models, `Fill` with the fillable allow-list, the hidden discipline, rule-string validation with the Laravel-shaped 422 map, casts (jsonable, money as fixed 4-decimal string, `lagoon.Encrypted`), the `system_files` attachment model with blob storage and thumbnails, the pagination envelope, the GORM callback registry for cross-plugin extension, and `migrate:rollback --plugin` hardening (CLI-03). It still knows nothing about Płytarium. `fonoteka.go` restructures its two flat Phase 3 plugins into the Winter directory layout and receives the models, the migration set, the write services' fill boundary and the tests.

Not in this phase: any HTTP route, including the photo-upload endpoint (Phase 12, API-02), the guarded outbound fetch for cover URLs (HTTP-07), Discogs cover import (Phase 14), River jobs, Centrifugo broadcasting and Typesense indexing behind `Album`'s PHP traits (Phase 11), the user plugin's auth flows (Phase 7), the OAuth provider storage implementation (Phase 8; only the OAuth *models and tables* land here), admin form schemas (Phase 9), and the production data import (Phase 15).

Known drift to correct in planning: the roadmap says "27 migrations"; `updates/version.yaml` now lists 38 PHP migration files across 26 versions. The count is not an acceptance number (see D-01).

</domain>

<decisions>
## Implementation Decisions

### Migration granularity
- **D-01:** Go migrations stay squashed per final-state table (P3 D-16 stands). One create migration per table, or per small tightly-coupled table group (e.g. OAuth tables, CSV import tables, the two Discogs credential tables), in FK order, each carrying a comment naming the PHP update files it folds. The PHP rename/flatten history (areas, items, locations, tags, item_categories) is not replayed. Success criterion 1 is read as: every Go migration runs up and down individually, and the final schema matches PHP's. The roadmap's "27" is not a target; the planner should fix the wording in ROADMAP.md/REQUIREMENTS.md DATA-09 when the phase is planned.
- **D-02:** Schema equality is proven by an automated diff, not by reading. A normalized snapshot of the PHP app's final schema on Postgres (tables, columns, types, nullability, defaults, indexes, unique constraints, FKs) is produced once and committed in `fonoteka.go`; a testcontainers test migrates the Go sets up and diffs `information_schema`/`pg_catalog` against it. Intended differences live in a small, commented allow-list in the test. How the snapshot is produced (running the PHP migrations against a scratch Postgres vs. another route) is research; if the PHP migrations do not run cleanly on Postgres, research reports that before planning.
- **D-03:** The Phase 3 minimal tables are widened by one appended ALTER migration per table (`widen_users`, `widen_collections`, `widen_albums`) placed right after the two shipped Phase 3 migrations, followed by the new create-table migrations. Shipped migrations are never edited (P3 D-17). Each widen migration has a real Rollback.
- **D-04:** Reference data in the PHP history (the various-artist taxonomy seed, any settings defaults) is ported as idempotent data migrations inside the plugin's set, keyed by natural key, with a Rollback that removes them, same as the genres seed (P3 D-18). No seeder command.

### Model DX: fillable, hidden, rules
- **D-05:** Each model declares `Fillable() []string`, a mechanical copy of PHP `$fillable`. `lagoon.Fill` copies onto the model only names that are both requested by the caller and in the model's fillable list. Write services pass their own narrower list exactly as PHP does (`AlbumWriteService::FILL_FIELDS` is a strict subset of `Album::$fillable`: no `collection_id`, no `market_price_source`). The model list is the backstop, the service list is the real boundary. `json.Unmarshal` into a GORM model is never used on a write path (PITFALLS Pitfall 3).
- **D-06:** Non-fillable and unknown keys are silently ignored, as Eloquent does; they are never persisted and never produce a 422. In non-production environments the dropped keys are logged once per call site.
- **D-07:** Criterion 3 is proven at service level in this phase: the fill boundary of the PHP write services is ported now (at minimum Album, Collection and the four credential models) and fuzzed against real Postgres with random extra and server-owned keys, asserting nothing outside the list is persisted. `lagoon.Fill` has its own framework-level fuzz test on a fixture model. The HTTP-level fuzz over request DTOs is Phase 12's job; criterion 3's "every write endpoint" clause moves there and the planner notes it in the roadmap.
- **D-08:** Hidden columns carry `json:"-"` on the GORM model so an accidental marshal cannot leak them, and models also declare `Hidden() []string` (for the Phase 9 admin schema and for a lagoon test that marshals every registered model and asserts hidden names are absent). API payloads are not produced by marshaling models: they come from ported `Serialize*` functions, as PHP's `SerializesFonoteka` does. The "explicit per-call override" is a named, greppable function (`Reveal`-style), not a serializer option.
- **D-09:** Models declare `Rules() map[string]string` with the PHP rule strings copied verbatim. `lagoon` translates the Laravel rule grammar Płytarium uses (required, nullable, integer, numeric, between, min, max, in, unique:table, plus whatever research finds in the 25 models) onto go-playground/validator, runs it in a before-save callback, and returns the Laravel-shaped 422 error map with messages translated through `phrasebook`. `unique:table` is a DB check in the same step and must respect soft deletes the way Winter's does. An untranslatable rule fails loudly at boot or test time, never silently passes.

### Encrypted cast
- **D-10:** The Go cast reads and writes only its own AES-256-GCM format. Laravel CBC payloads are never on the live read path. Phase 5 ships a small, tested helper that decrypts a Laravel `encrypted` payload (base64 JSON with iv/value/mac, AES-256-CBC + HMAC under `APP_KEY`) for the Phase 15 cutover import to re-encrypt; it is not wired into the cast.
- **D-11:** The key is `app.key` in config (`SUMMER_APP__KEY`), 32 bytes base64. Missing, short or undecodable key fails boot with no default, same rule as the JWT secret (P3 D-11). The column-encryption key is derived from it with HKDF and a fixed label so `app.key` can serve other purposes later without key reuse. A `summer key:generate` command prints a fresh key.
- **D-12:** Ciphertext is versioned: a format/key-id prefix, nonce, ciphertext+tag. Config accepts `app.previous_keys` as decrypt-only keys (Laravel's `APP_PREVIOUS_KEYS` equivalent). No re-encrypt command in this phase.
- **D-13:** Encrypted columns are typed `lagoon.Encrypted` on the model: Scanner/Valuer decrypt on read and encrypt on write; `MarshalJSON`, `String()` and `GoString()` always emit a redaction; plaintext is only reachable through an explicit `.Reveal()`. Applies to `api_key` on `UserAiCredential` and `OrgAiCredential` and to the secret columns of `UserDiscogsCredential` and `OrgDiscogsCredential`; research lists the exact columns from the PHP `$casts`.

### Attachments (DATA-08)
- **D-14:** Attachments use Winter's `system_files` table shape and name (`disk_name`, `file_name`, `file_size`, `content_type`, `title`, `description`, `field`, `attachment_id`, `attachment_type`, `is_public`, `sort_order`, timestamps). `attachment_type` keeps the PHP class string (`Golem15\Fonoteka\Models\Album`) through a per-model morph name, so cutover copies rows and files verbatim and later Winter ports reuse it. The table and `File` model are framework-owned (`lagoon`) with their own migration set that runs before plugin sets.
- **D-15:** Full storage scope lands in this phase (user chose the largest option over the recommended table-only cut): `gocloud.dev/blob` is wired now, files are stored and deleted through it, and thumbnails are generated. Only the HTTP upload endpoint stays in Phase 12. Planner should size the phase accordingly.
- **D-16:** v1 bucket is `fileblob` rooted at the same `storage/app/uploads` tree PHP uses, with bucket URL and public path prefix from a `storage.uploads.*` config section mirroring `cms.php`'s `storage.uploads`. Winter's `disk_name` partition path (`public/xxx/yyy/zzz/<disk_name>`) and the relative public URL shape (`relativeMediaUrl`) are reproduced exactly. The framework serves public files with a static handler at the configured path; S3/GCS is a config change later. `memblob` in unit tests.
- **D-17:** `Thumb(w, h, mode)` reproduces Winter's thumb filename and location (next to the original, `thumb_<id>_<w>_<h>_0_0_<mode>.<ext>`; research confirms the exact rule from Winter Storm's `File` model) and generates lazily on first call through blob when the thumb is missing. Thumbs copied at cutover are reused, and `thumb_url` stays byte-identical for the parity diff. Pixel equality with PHP GD is not a goal.
- **D-18:** The resize library is research's pick, recorded in RESEARCH.md as the phase decision naming the dependency: pure Go, maintained, no cgo/libvips. stdlib `image/*` plus `golang.org/x/image/draw` is the baseline to beat.
- **D-19:** Deletion follows Winter: soft-deleting an owner keeps `system_files` rows and blobs; force delete removes rows inside the delete transaction and removes originals and thumbs from blob after commit. This runs through the same lifecycle-callback mechanism as the other cascades (DATA-03), not a special case.

### Claude's Discretion
- Lifecycle hook naming and wiring (Winter names `BeforeValidate`/`BeforeCreate`/`BeforeSave`/`BeforeDelete`/`AfterDelete` as optional interfaces vs. GORM's own hook names), the base model embed, and how soft-delete cascade runs in one transaction.
- Pivot model shape for `album_artists.sort_order` and `CollectionEditor` (GORM `SetupJoinTable` vs explicit has-many-through), as long as the 3+ artist ordering round-trip and the business columns hold.
- Money cast type and API: a named string-backed type in `models/` (or `lagoon` if generic) that reproduces `MarketPriceCast::get` (null for null/blank/non-numeric, else 4-decimal string) and leaves normalisation to the `Album` before-save hook, as PHP does. Never `float64` in the JSON path.
- Jsonable cast shape, including `[]` vs `null` behavior per column (Pitfall 4).
- Pagination helper API; the envelope is fixed: `{data, meta{current_page,last_page,per_page,total}}`, no `links`.
- Callback-registry API surface (ARCHITECTURE.md Pattern 3b) and which plugin/model pair demonstrates criterion 5's cross-plugin extension plus companion migration; a fixture plugin in tests is acceptable if no real Płytarium case fits.
- Soft-delete + unique strategy (partial unique indexes vs. matching PHP's actual constraints); D-02's schema diff decides what "matching" means, the delete-then-recreate test must pass for every soft-deletable uniquely-keyed table.
- Which `Serialize*` functions are ported now (only those needed by this phase's tests) vs. Phase 12.
- `Album`'s service-calling hook (`ArtistResolver` in `beforeSave`) becomes a callback registered from `classes/`, per the layout note; Scout/Broadcastable traits are left as seams for Phase 11.
- CLI-03 details beyond what Phase 3 shipped.
- Plan count and split, subject to the plan-count checkpoint and "unit tests are the last plan" rules. Given D-15, expect more than the usual number of plans.

### Folded Todos
- **Verify the models-leaf layout rule against keios.eu and a jz/pxpx plugin before Phase 5** (`.planning/todos/pending/verify-models-leaf-rule.md`). Runs as the first task of the phase, before 24 models are laid out under the rule: repeat the `models/ → classes/` (and any sibling) `use`-edge probe on one keios.eu stack plugin with models and services (`/media/nvme/dev/golem15/keios.eu/plugins/golem15/user` or `paymentgateway`), one jz plugin and one pxpx plugin; classify each edge as cast / service-calling hook / other. Any "other" edge means `.planning/notes/plugin-layout-winter-directories.md` is amended before the bulk port proceeds.

</decisions>

<canonical_refs>
## Canonical References

**Downstream agents MUST read these before planning or implementing.**

### The PHP data layer being ported
- `/media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/models/*.php` — the 25 models: `$table`, `$fillable`, `$guarded`, `$hidden`, `$casts`, `$jsonable`, `$dates`, `$rules`, relations, hooks. `Album.php` is the densest (constants, money cast, jsonable, soft delete, belongsToMany artists, attachMany photos, `beforeSave` with `ArtistResolver`).
- `/media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/updates/version.yaml` and `updates/v*/` — 38 migration files across 26 versions; the history D-01 squashes and the seeds D-04 ports.
- `/media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/casts/MarketPriceCast.php` — money cast semantics and the docblock explaining the blank-string case.
- `/media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/AlbumWriteService.php` — `FILL_FIELDS` and the fill/fillEmpty/overwrite boundary that D-05 and D-07 port and fuzz.
- `/media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/models/UserAiCredential.php`, `OrgAiCredential.php`, `UserDiscogsCredential.php`, `OrgDiscogsCredential.php` — `encrypted` casts and `$hidden` (D-10 to D-13).
- `/media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/traits/SerializesFonoteka.php` — payload builders; lines ~190–215 for the photo payload, `getThumb(200, 200, ['mode' => 'crop'])` and `relativeMediaUrl` (D-08, D-16, D-17).
- `/media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/models/Collection.php` lines ~91–107 — `attachMany photos` (ordered) and public `attachOne image`.
- `/media/nvme/dev/golem15/fonoteka/config/cms.php` lines ~300–335 — `storage.uploads` / `media` / `resized` disks and paths (D-16).
- Winter Storm `File` model and `Winter\Storm\Database\Attach\File` in `/media/nvme/dev/golem15/fonoteka/vendor/winter/storm` — `disk_name` partitioning, `getPath`, `getThumb` naming, delete behavior (D-14, D-17, D-19).
- `/media/nvme/dev/golem15/fonoteka/plugins/golem15/user/models/` and its `updates/` — the user-plugin tables that `widen_users` must match.

### Go code this phase builds on
- `lagoon/connection.go`, `lagoon/migrations.go`, `lagoon/commands.go`, `lagoon/order.go` — shared `*sql.DB`, per-plugin gormigrate runner, `migrate:*` commands, Polish order helper.
- `pact/capabilities.go` — `HasModels`/`HasMigrations` and friends.
- `../fonoteka.go/plugins/golem15/fonoteka/migrations.go` and `../fonoteka.go/plugins/golem15/user/migrations.go` — the shipped, never-edited Phase 3 migrations (`202609170001_create_schema`, `202609170002_seed_genres`) that D-03 appends to.
- `../fonoteka.go/plugins/golem15/fonoteka/*.go` and `../fonoteka.go/plugins/golem15/user/*.go` — the flat Phase 3 plugins to restructure into the Winter layout.
- `internal/build/` — the `models/`-is-a-leaf enforcement shipped in Phase 4 (P4 D-11) and the `make:model`/`make:migration` stubs that should match what this phase hand-writes.

### Layout and architecture decisions
- `.planning/notes/plugin-layout-winter-directories.md` — directory mapping and the models-leaf rule (casts into `models/`, service-calling hooks out to `classes/`).
- `.planning/todos/pending/verify-models-leaf-rule.md` — the folded todo with its probe commands.
- `.planning/research/ARCHITECTURE.md` §Pattern 3 (3a events, 3b GORM callback registry, 3c companion structs over shared tables), §Migration Flow, §Anti-Pattern 4. Text that says goose is superseded by gormigrate.
- `.planning/research/PITFALLS.md` §Pitfall 3 (DTO vs model), §Pitfall 4 (nil vs `[]`), §Pitfall 5 (float vs fixed-decimal string), §Pitfall 7 (tri-state bool), §Pitfall 11 (soft delete + unique), §Pitfall 12 (pivot columns), §Pitfall 13 (no AutoMigrate), §Security Mistakes, §Integration Gotchas row on gocloud.dev/blob URL shape.
- `.planning/research/STACK.md` §GORM: plain structs, §Migration tooling, §Version Compatibility (gorm v1.31.2, validator v10.30.4, gocloud.dev v0.46.0, testcontainers v0.44.0).
- `.planning/research/FEATURES.md` — model-feature inventory from the PHP source.

### Prior phase decisions that bind this phase
- `.planning/phases/03-first-vertical-slice-genres-end-to-end/03-CONTEXT.md` D-02, D-06, D-11, D-16, D-17, D-18, D-19.
- `.planning/phases/04-cli-scaffolding-i18n-and-mail/04-CONTEXT.md` D-10, D-11, D-12, D-13.
- `.planning/phases/01-framework-kernel-foundation/01-CONTEXT.md` D-06, D-08, D-10 (config namespaces, env mapping, fail-fast boot).
- `.planning/PROJECT.md` §Constraints, §Key Decisions; `.planning/REQUIREMENTS.md` DATA-03 to DATA-11, CLI-03; `.planning/ROADMAP.md` Phase 5 section.
- `CLAUDE.md` (repo root) §GSD workflow rules.

</canonical_refs>

<code_context>
## Existing Code Insights

### Reusable Assets
- `lagoon`: connection on the shared pgx-stdlib `*sql.DB`, per-plugin gormigrate runner with per-plugin history tables, `migrate`, `migrate:rollback --plugin`, `migrate:status`, the `pl-PL` ICU order helper and its testcontainers Postgres test harness.
- `phrasebook` (Phase 4) for translated validation messages; `compass` for `app.key`, `app.previous_keys` and `storage.uploads.*`; `backpack` to publish the blob bucket and file service; `bonfire` for `key:generate`.
- `internal/build` leaf-import check and `make:model`/`make:migration` stubs.
- `fonoteka.go/parity` testcontainers `TestMain`: the place the schema-diff test and the service-level fuzz tests can share a Postgres with.

### Established Patterns
- No AutoMigrate anywhere, tests included; shipped migrations are never edited; each plugin's set rolls back without touching another's history.
- Fail boot loudly on misconfiguration (now also: bad `app.key`, untranslatable rule string, unconfigured uploads bucket).
- Response DTOs and explicit serializers, never the GORM model; `[]` not `null`; no generic envelope.
- Framework never imports the app; anything Płytarium-specific (money cast semantics, write services, morph names) lives in `fonoteka.go`.
- New dependencies need naming: `gocloud.dev` is already decided; the image resize library is named by this phase's research (D-18). Nothing else without a decision note.

### Integration Points
- The Phase 3 plugins in `fonoteka.go` are flat single packages; this phase moves them to `models/`, `classes/`, `controllers/`, `middleware/`, `updates/` with `plugin.go`/`routes.go` at the root, keeping the genres route and its parity fixture green throughout.
- The framework's `system_files` migration set must order before every plugin set.
- `Album`'s Scout and Broadcastable behavior are seams only; Phase 11 attaches them through the callback registry built here.
- OAuth models and tables land here; the zitadel/oidc storage that uses them is Phase 8.

</code_context>

<specifics>
## Specific Ideas

- PHP's real mass-assignment boundary is two-layered: `$fillable` as backstop, a narrower per-service list as the working boundary. The Go port keeps both layers rather than collapsing them into DTO shape alone, so an agent can diff `Fillable()` against the PHP array line by line.
- Rule strings, fillable lists and hidden lists are copied verbatim from PHP so conversion stays mechanical; translation to Go idiom happens once, inside `lagoon`.
- Secrets should be unable to leak by accident: `json:"-"`, a redacting `lagoon.Encrypted`, and explicit `Reveal` calls the security review can grep.
- `system_files` and the uploads tree stay Winter-shaped so cutover is a copy, and so the next Winter project port gets attachments for free.
- The user chose to take blob storage and thumbnails in this phase despite the smaller recommended cut; treat it as a deliberate choice, not scope creep, and plan capacity for it.

</specifics>

<deferred>
## Deferred Ideas

- HTTP-level fuzz of request DTOs on every write endpoint — Phase 12/13 when the routes exist (second half of criterion 3).
- Photo upload endpoint, manual cover URL and Discogs cover import — Phase 12 (API-02), HTTP-07, Phase 14.
- Re-encrypt command for key rotation — after v1; the versioned format (D-12) keeps it possible.
- Cutover import that decrypts Laravel payloads and re-encrypts, and copies `system_files` rows and the uploads tree — Phase 15, using the helper from D-10.
- S3/GCS bucket for uploads — config change after cutover.
- Typesense indexing and Centrifugo broadcasting hooks on `Album` — Phase 11.
- Correcting the "27 migrations" wording in ROADMAP.md and REQUIREMENTS.md DATA-09, and moving criterion 3's endpoint clause — do at plan time for this phase.

</deferred>

---

*Phase: 5-Data layer full fidelity*
*Context gathered: 2026-09-18*