summercms/.planning/phases/05-data-layer-full-fidelity/05-CONTEXT.md

Phase 5: Data layer full fidelity - Context

Gathered: 2026-09-18 Status: Ready for planning

## Phase Boundary

Phase 5 ports the whole Płytarium data layer: all 25 models of golem15.fonoteka (plus the golem15.user tables they depend on) with matching table names, columns, indexes, defaults, relations, casts and lifecycle hooks, and a Go migration set whose final schema equals PHP's. Requirements DATA-03 to DATA-11 and CLI-03. Roadmap mode: mvp. The phase is security-load-bearing: mass-assignment boundaries and encrypted-at-rest credentials are named invariants in the PHP source, so the security-review agent runs on it and the DTO-vs-model convention applies from the first model.

Two repos. summercms.go grows lagoon with what the models pull in: lifecycle hooks and transactional soft-delete cascade, relation helpers with ordered results and pivot models, Fill with the fillable allow-list, the hidden discipline, rule-string validation with the Laravel-shaped 422 map, casts (jsonable, money as fixed 4-decimal string, lagoon.Encrypted), the system_files attachment model with blob storage and thumbnails, the pagination envelope, the GORM callback registry for cross-plugin extension, and migrate:rollback --plugin hardening (CLI-03). It still knows nothing about Płytarium. fonoteka.go restructures its two flat Phase 3 plugins into the Winter directory layout and receives the models, the migration set, the write services' fill boundary and the tests.

Not in this phase: any HTTP route, including the photo-upload endpoint (Phase 12, API-02), the guarded outbound fetch for cover URLs (HTTP-07), Discogs cover import (Phase 14), River jobs, Centrifugo broadcasting and Typesense indexing behind Album's PHP traits (Phase 11), the user plugin's auth flows (Phase 7), the OAuth provider storage implementation (Phase 8; only the OAuth models and tables land here), admin form schemas (Phase 9), and the production data import (Phase 15).

Known drift to correct in planning: the roadmap says "27 migrations"; updates/version.yaml now lists 38 PHP migration files across 26 versions. The count is not an acceptance number (see D-01).

## Implementation Decisions

Migration granularity

• D-01: Go migrations stay squashed per final-state table (P3 D-16 stands). One create migration per table, or per small tightly-coupled table group (e.g. OAuth tables, CSV import tables, the two Discogs credential tables), in FK order, each carrying a comment naming the PHP update files it folds. The PHP rename/flatten history (areas, items, locations, tags, item_categories) is not replayed. Success criterion 1 is read as: every Go migration runs up and down individually, and the final schema matches PHP's. The roadmap's "27" is not a target; the planner should fix the wording in ROADMAP.md/REQUIREMENTS.md DATA-09 when the phase is planned.
• D-02: Schema equality is proven by an automated diff, not by reading. A normalized snapshot of the PHP app's final schema on Postgres (tables, columns, types, nullability, defaults, indexes, unique constraints, FKs) is produced once and committed in fonoteka.go; a testcontainers test migrates the Go sets up and diffs information_schema/pg_catalog against it. Intended differences live in a small, commented allow-list in the test. How the snapshot is produced (running the PHP migrations against a scratch Postgres vs. another route) is research; if the PHP migrations do not run cleanly on Postgres, research reports that before planning.
• D-03: The Phase 3 minimal tables are widened by one appended ALTER migration per table (widen_users, widen_collections, widen_albums) placed right after the two shipped Phase 3 migrations, followed by the new create-table migrations. Shipped migrations are never edited (P3 D-17). Each widen migration has a real Rollback.
• D-04: Reference data in the PHP history (the various-artist taxonomy seed, any settings defaults) is ported as idempotent data migrations inside the plugin's set, keyed by natural key, with a Rollback that removes them, same as the genres seed (P3 D-18). No seeder command.

Model DX: fillable, hidden, rules

• D-05: Each model declares Fillable() []string, a mechanical copy of PHP $fillable. lagoon.Fill copies onto the model only names that are both requested by the caller and in the model's fillable list. Write services pass their own narrower list exactly as PHP does (AlbumWriteService::FILL_FIELDS is a strict subset of Album::$fillable: no collection_id, no market_price_source). The model list is the backstop, the service list is the real boundary. json.Unmarshal into a GORM model is never used on a write path (PITFALLS Pitfall 3).
• D-06: Non-fillable and unknown keys are silently ignored, as Eloquent does; they are never persisted and never produce a 422. In non-production environments the dropped keys are logged once per call site.
• D-07: Criterion 3 is proven at service level in this phase: the fill boundary of the PHP write services is ported now (at minimum Album, Collection and the four credential models) and fuzzed against real Postgres with random extra and server-owned keys, asserting nothing outside the list is persisted. lagoon.Fill has its own framework-level fuzz test on a fixture model. The HTTP-level fuzz over request DTOs is Phase 12's job; criterion 3's "every write endpoint" clause moves there and the planner notes it in the roadmap.
• D-08: Hidden columns carry json:"-" on the GORM model so an accidental marshal cannot leak them, and models also declare Hidden() []string (for the Phase 9 admin schema and for a lagoon test that marshals every registered model and asserts hidden names are absent). API payloads are not produced by marshaling models: they come from ported Serialize* functions, as PHP's SerializesFonoteka does. The "explicit per-call override" is a named, greppable function (Reveal-style), not a serializer option.
• D-09: Models declare Rules() map[string]string with the PHP rule strings copied verbatim. lagoon translates the Laravel rule grammar Płytarium uses (required, nullable, integer, numeric, between, min, max, in, unique:table, plus whatever research finds in the 25 models) onto go-playground/validator, runs it in a before-save callback, and returns the Laravel-shaped 422 error map with messages translated through phrasebook. unique:table is a DB check in the same step and must respect soft deletes the way Winter's does. An untranslatable rule fails loudly at boot or test time, never silently passes.

Encrypted cast

• D-10: The Go cast reads and writes only its own AES-256-GCM format. Laravel CBC payloads are never on the live read path. Phase 5 ships a small, tested helper that decrypts a Laravel encrypted payload (base64 JSON with iv/value/mac, AES-256-CBC + HMAC under APP_KEY) for the Phase 15 cutover import to re-encrypt; it is not wired into the cast.
• D-11: The key is app.key in config (SUMMER_APP__KEY), 32 bytes base64. Missing, short or undecodable key fails boot with no default, same rule as the JWT secret (P3 D-11). The column-encryption key is derived from it with HKDF and a fixed label so app.key can serve other purposes later without key reuse. A summer key:generate command prints a fresh key.
• D-12: Ciphertext is versioned: a format/key-id prefix, nonce, ciphertext+tag. Config accepts app.previous_keys as decrypt-only keys (Laravel's APP_PREVIOUS_KEYS equivalent). No re-encrypt command in this phase.
• D-13: Encrypted columns are typed lagoon.Encrypted on the model: Scanner/Valuer decrypt on read and encrypt on write; MarshalJSON, String() and GoString() always emit a redaction; plaintext is only reachable through an explicit .Reveal(). Applies to api_key on UserAiCredential and OrgAiCredential and to the secret columns of UserDiscogsCredential and OrgDiscogsCredential; research lists the exact columns from the PHP $casts.

Attachments (DATA-08)

• D-14: Attachments use Winter's system_files table shape and name (disk_name, file_name, file_size, content_type, title, description, field, attachment_id, attachment_type, is_public, sort_order, timestamps). attachment_type keeps the PHP class string (Golem15\Fonoteka\Models\Album) through a per-model morph name, so cutover copies rows and files verbatim and later Winter ports reuse it. The table and File model are framework-owned (lagoon) with their own migration set that runs before plugin sets.
• D-15: Full storage scope lands in this phase (user chose the largest option over the recommended table-only cut): gocloud.dev/blob is wired now, files are stored and deleted through it, and thumbnails are generated. Only the HTTP upload endpoint stays in Phase 12. Planner should size the phase accordingly.
• D-16: v1 bucket is fileblob rooted at the same storage/app/uploads tree PHP uses, with bucket URL and public path prefix from a storage.uploads.* config section mirroring cms.php's storage.uploads. Winter's disk_name partition path (public/xxx/yyy/zzz/<disk_name>) and the relative public URL shape (relativeMediaUrl) are reproduced exactly. The framework serves public files with a static handler at the configured path; S3/GCS is a config change later. memblob in unit tests.
• D-17: Thumb(w, h, mode) reproduces Winter's thumb filename and location (next to the original, thumb_<id>_<w>_<h>_0_0_<mode>.<ext>; research confirms the exact rule from Winter Storm's File model) and generates lazily on first call through blob when the thumb is missing. Thumbs copied at cutover are reused, and thumb_url stays byte-identical for the parity diff. Pixel equality with PHP GD is not a goal.
• D-18: The resize library is research's pick, recorded in RESEARCH.md as the phase decision naming the dependency: pure Go, maintained, no cgo/libvips. stdlib image/* plus golang.org/x/image/draw is the baseline to beat.
• D-19: Deletion follows Winter: soft-deleting an owner keeps system_files rows and blobs; force delete removes rows inside the delete transaction and removes originals and thumbs from blob after commit. This runs through the same lifecycle-callback mechanism as the other cascades (DATA-03), not a special case.

Claude's Discretion

• Lifecycle hook naming and wiring (Winter names BeforeValidate/BeforeCreate/BeforeSave/BeforeDelete/AfterDelete as optional interfaces vs. GORM's own hook names), the base model embed, and how soft-delete cascade runs in one transaction.
• Pivot model shape for album_artists.sort_order and CollectionEditor (GORM SetupJoinTable vs explicit has-many-through), as long as the 3+ artist ordering round-trip and the business columns hold.
• Money cast type and API: a named string-backed type in models/ (or lagoon if generic) that reproduces MarketPriceCast::get (null for null/blank/non-numeric, else 4-decimal string) and leaves normalisation to the Album before-save hook, as PHP does. Never float64 in the JSON path.
• Jsonable cast shape, including [] vs null behavior per column (Pitfall 4).
• Pagination helper API; the envelope is fixed: {data, meta{current_page,last_page,per_page,total}}, no links.
• Callback-registry API surface (ARCHITECTURE.md Pattern 3b) and which plugin/model pair demonstrates criterion 5's cross-plugin extension plus companion migration; a fixture plugin in tests is acceptable if no real Płytarium case fits.
• Soft-delete + unique strategy (partial unique indexes vs. matching PHP's actual constraints); D-02's schema diff decides what "matching" means, the delete-then-recreate test must pass for every soft-deletable uniquely-keyed table.
• Which Serialize* functions are ported now (only those needed by this phase's tests) vs. Phase 12.
• Album's service-calling hook (ArtistResolver in beforeSave) becomes a callback registered from classes/, per the layout note; Scout/Broadcastable traits are left as seams for Phase 11.
• CLI-03 details beyond what Phase 3 shipped.
• Plan count and split, subject to the plan-count checkpoint and "unit tests are the last plan" rules. Given D-15, expect more than the usual number of plans.

Folded Todos

• Verify the models-leaf layout rule against keios.eu and a jz/pxpx plugin before Phase 5 (.planning/todos/pending/verify-models-leaf-rule.md). Runs as the first task of the phase, before 24 models are laid out under the rule: repeat the models/ → classes/ (and any sibling) use-edge probe on one keios.eu stack plugin with models and services (/media/nvme/dev/golem15/keios.eu/plugins/golem15/user or paymentgateway), one jz plugin and one pxpx plugin; classify each edge as cast / service-calling hook / other. Any "other" edge means .planning/notes/plugin-layout-winter-directories.md is amended before the bulk port proceeds.

<canonical_refs>

Canonical References

Downstream agents MUST read these before planning or implementing.

The PHP data layer being ported

• /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/models/*.php — the 25 models: $table, $fillable, $guarded, $hidden, $casts, $jsonable, $dates, $rules, relations, hooks. Album.php is the densest (constants, money cast, jsonable, soft delete, belongsToMany artists, attachMany photos, beforeSave with ArtistResolver).
• /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/updates/version.yaml and updates/v*/ — 38 migration files across 26 versions; the history D-01 squashes and the seeds D-04 ports.
• /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/casts/MarketPriceCast.php — money cast semantics and the docblock explaining the blank-string case.
• /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/AlbumWriteService.php — FILL_FIELDS and the fill/fillEmpty/overwrite boundary that D-05 and D-07 port and fuzz.
• /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/models/UserAiCredential.php, OrgAiCredential.php, UserDiscogsCredential.php, OrgDiscogsCredential.php — encrypted casts and $hidden (D-10 to D-13).
• /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/traits/SerializesFonoteka.php — payload builders; lines ~190–215 for the photo payload, getThumb(200, 200, ['mode' => 'crop']) and relativeMediaUrl (D-08, D-16, D-17).
• /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/models/Collection.php lines ~91–107 — attachMany photos (ordered) and public attachOne image.
• /media/nvme/dev/golem15/fonoteka/config/cms.php lines ~300–335 — storage.uploads / media / resized disks and paths (D-16).
• Winter Storm File model and Winter\Storm\Database\Attach\File in /media/nvme/dev/golem15/fonoteka/vendor/winter/storm — disk_name partitioning, getPath, getThumb naming, delete behavior (D-14, D-17, D-19).
• /media/nvme/dev/golem15/fonoteka/plugins/golem15/user/models/ and its updates/ — the user-plugin tables that widen_users must match.

Go code this phase builds on

• lagoon/connection.go, lagoon/migrations.go, lagoon/commands.go, lagoon/order.go — shared *sql.DB, per-plugin gormigrate runner, migrate:* commands, Polish order helper.
• pact/capabilities.go — HasModels/HasMigrations and friends.
• ../fonoteka.go/plugins/golem15/fonoteka/migrations.go and ../fonoteka.go/plugins/golem15/user/migrations.go — the shipped, never-edited Phase 3 migrations (202609170001_create_schema, 202609170002_seed_genres) that D-03 appends to.
• ../fonoteka.go/plugins/golem15/fonoteka/*.go and ../fonoteka.go/plugins/golem15/user/*.go — the flat Phase 3 plugins to restructure into the Winter layout.
• internal/build/ — the models/-is-a-leaf enforcement shipped in Phase 4 (P4 D-11) and the make:model/make:migration stubs that should match what this phase hand-writes.

Layout and architecture decisions

• .planning/notes/plugin-layout-winter-directories.md — directory mapping and the models-leaf rule (casts into models/, service-calling hooks out to classes/).
• .planning/todos/pending/verify-models-leaf-rule.md — the folded todo with its probe commands.
• .planning/research/ARCHITECTURE.md §Pattern 3 (3a events, 3b GORM callback registry, 3c companion structs over shared tables), §Migration Flow, §Anti-Pattern 4. Text that says goose is superseded by gormigrate.
• .planning/research/PITFALLS.md §Pitfall 3 (DTO vs model), §Pitfall 4 (nil vs []), §Pitfall 5 (float vs fixed-decimal string), §Pitfall 7 (tri-state bool), §Pitfall 11 (soft delete + unique), §Pitfall 12 (pivot columns), §Pitfall 13 (no AutoMigrate), §Security Mistakes, §Integration Gotchas row on gocloud.dev/blob URL shape.
• .planning/research/STACK.md §GORM: plain structs, §Migration tooling, §Version Compatibility (gorm v1.31.2, validator v10.30.4, gocloud.dev v0.46.0, testcontainers v0.44.0).
• .planning/research/FEATURES.md — model-feature inventory from the PHP source.

Prior phase decisions that bind this phase

• .planning/phases/03-first-vertical-slice-genres-end-to-end/03-CONTEXT.md D-02, D-06, D-11, D-16, D-17, D-18, D-19.
• .planning/phases/04-cli-scaffolding-i18n-and-mail/04-CONTEXT.md D-10, D-11, D-12, D-13.
• .planning/phases/01-framework-kernel-foundation/01-CONTEXT.md D-06, D-08, D-10 (config namespaces, env mapping, fail-fast boot).
• .planning/PROJECT.md §Constraints, §Key Decisions; .planning/REQUIREMENTS.md DATA-03 to DATA-11, CLI-03; .planning/ROADMAP.md Phase 5 section.
• CLAUDE.md (repo root) §GSD workflow rules.

</canonical_refs>

<code_context>

Existing Code Insights

Reusable Assets

• lagoon: connection on the shared pgx-stdlib *sql.DB, per-plugin gormigrate runner with per-plugin history tables, migrate, migrate:rollback --plugin, migrate:status, the pl-PL ICU order helper and its testcontainers Postgres test harness.
• phrasebook (Phase 4) for translated validation messages; compass for app.key, app.previous_keys and storage.uploads.*; backpack to publish the blob bucket and file service; bonfire for key:generate.
• internal/build leaf-import check and make:model/make:migration stubs.
• fonoteka.go/parity testcontainers TestMain: the place the schema-diff test and the service-level fuzz tests can share a Postgres with.

Established Patterns

• No AutoMigrate anywhere, tests included; shipped migrations are never edited; each plugin's set rolls back without touching another's history.
• Fail boot loudly on misconfiguration (now also: bad app.key, untranslatable rule string, unconfigured uploads bucket).
• Response DTOs and explicit serializers, never the GORM model; [] not null; no generic envelope.
• Framework never imports the app; anything Płytarium-specific (money cast semantics, write services, morph names) lives in fonoteka.go.
• New dependencies need naming: gocloud.dev is already decided; the image resize library is named by this phase's research (D-18). Nothing else without a decision note.

Integration Points

• The Phase 3 plugins in fonoteka.go are flat single packages; this phase moves them to models/, classes/, controllers/, middleware/, updates/ with plugin.go/routes.go at the root, keeping the genres route and its parity fixture green throughout.
• The framework's system_files migration set must order before every plugin set.
• Album's Scout and Broadcastable behavior are seams only; Phase 11 attaches them through the callback registry built here.
• OAuth models and tables land here; the zitadel/oidc storage that uses them is Phase 8.

</code_context>

## Specific Ideas

• PHP's real mass-assignment boundary is two-layered: $fillable as backstop, a narrower per-service list as the working boundary. The Go port keeps both layers rather than collapsing them into DTO shape alone, so an agent can diff Fillable() against the PHP array line by line.
• Rule strings, fillable lists and hidden lists are copied verbatim from PHP so conversion stays mechanical; translation to Go idiom happens once, inside lagoon.
• Secrets should be unable to leak by accident: json:"-", a redacting lagoon.Encrypted, and explicit Reveal calls the security review can grep.
• system_files and the uploads tree stay Winter-shaped so cutover is a copy, and so the next Winter project port gets attachments for free.
• The user chose to take blob storage and thumbnails in this phase despite the smaller recommended cut; treat it as a deliberate choice, not scope creep, and plan capacity for it.

## Deferred Ideas

• HTTP-level fuzz of request DTOs on every write endpoint — Phase 12/13 when the routes exist (second half of criterion 3).
• Photo upload endpoint, manual cover URL and Discogs cover import — Phase 12 (API-02), HTTP-07, Phase 14.
• Re-encrypt command for key rotation — after v1; the versioned format (D-12) keeps it possible.
• Cutover import that decrypts Laravel payloads and re-encrypts, and copies system_files rows and the uploads tree — Phase 15, using the helper from D-10.
• S3/GCS bucket for uploads — config change after cutover.
• Typesense indexing and Centrifugo broadcasting hooks on Album — Phase 11.
• Correcting the "27 migrations" wording in ROADMAP.md and REQUIREMENTS.md DATA-09, and moving criterion 3's endpoint clause — do at plan time for this phase.

---

Phase: 5-Data layer full fidelity Context gathered: 2026-09-18