diff --git a/.planning/phases/12-p-ytarium-api-collections-and-albums/12-01-SUMMARY.md b/.planning/phases/12-p-ytarium-api-collections-and-albums/12-01-SUMMARY.md new file mode 100644 index 0000000..0a9478d --- /dev/null +++ b/.planning/phases/12-p-ytarium-api-collections-and-albums/12-01-SUMMARY.md @@ -0,0 +1,293 @@ +--- +phase: 12-p-ytarium-api-collections-and-albums +plan: 01 +subsystem: api +tags: [lagoon, validation, laravel, phrasebook, attach, webp, tide, multipart, beachcomber, typesense, user-groups] + +requires: + - phase: 05-data-layer-full-fidelity + provides: lagoon.Validate, attach File/Thumb/BlobKey, phrasebook catalogs + - phase: 11-jobs-realtime-and-search-infrastructure + provides: beachcomber Engine/Query, typesense engine, tide broadcast goldens + - phase: 07-user-plugin-and-authentication + provides: Go user plugin (sm-user-plugin) models, migrations and Postgres test harness +provides: + - lagoon.ValidateRequest with Laravel 9 semantics and the pl/en lagoon::validation catalogs + - ParseRules, In, CustomRule, UploadedFile/UploadedFileFromHeader, ErrorKeys + - lagoon.Validate min/max/numeric-between message fix (todo folded) + - attach.PublicURL, (*attach.File).URL, webp decoding (golang.org/x/image v0.46.0) + - tide multipart Parts with fixed MultipartBoundary, upload URL masking, album date masking in publications + - beachcomber PageSearcher/SearchResult/SearchPage and Query.QueryByWeights (typesense query_by_weights, MaxPerPage 250) + - user_groups/users_groups tables, models.UserGroup, User.Groups, classes.UserGroupCodes/HasGroupCode + - ROADMAP Phase 12-14 and REQUIREMENTS API-01/02/03/07, INTG-01 rewording +affects: [12-02, 12-03, 12-04, 12-05, 12.1, 13, 14] + +actuals: + tokens: 55700 + tasks: 4 + commits: 8 +plan_head_before: cfe568a214139a438ff5aecc27cc02e5f6f06892 +plan_head_after: 1bdbdf041b7deb17a776ba8ac0b12bd7d7db8b83 + +tech-stack: + added: [golang.org/x/image v0.46.0 (direct, summercms.go)] + patterns: + - "Request rule tables are package-level []lagoon.RequestRule built with ParseRules; a bad rule panics at boot" + - "Optional engine capability via a type-asserted interface plus a fallback helper (PageSearcher/SearchPage)" + - "Multipart fixtures: file bytes beside the fixture, pinned by sha256, encoded with one fixed boundary" + +key-files: + created: + - modules/lagoon/validate_request.go + - modules/lagoon/validate_rules.go + - modules/lagoon/validate_request_test.go + - modules/phrasebook/lang/pl/validation.yaml + - modules/phrasebook/lang/en/validation.yaml + - modules/lagoon/attach/url_test.go + - modules/tide/multipart.go + - modules/tide/multipart_test.go + - modules/beachcomber/searchpage_test.go + - modules/beachcomber/typesense/searchpage_test.go + - ../fonoteka.go/plugins/golem15/user/models/user_group.go + - ../fonoteka.go/plugins/golem15/user/updates/202610020001_create_user_groups.go + - ../fonoteka.go/plugins/golem15/user/updates/user_groups_test.go + - ../fonoteka.go/plugins/golem15/user/classes/user_groups.go + modified: + - modules/lagoon/validate.go + - modules/lagoon/attach/thumb.go + - modules/tide/flow.go + - modules/tide/fixture.go + - modules/tide/record.go + - modules/tide/replay.go + - modules/tide/diff.go + - modules/tide/normalize.go + - modules/tide/variables.go + - modules/tide/centrifugo_golden.go + - modules/beachcomber/searchable.go + - modules/beachcomber/engines.go + - modules/beachcomber/typesense/engine.go + - ../fonoteka.go/plugins/golem15/user/models/user.go + - ../fonoteka.go/parity/schema_diff_test.go + - go.mod + +key-decisions: + - "Wildcard rules follow Laravel exactly: a wildcard with nothing to expand adds no attribute (the plan's 'absent parent still fires required' contradicted Validator.php and the live PHP validator)" + - "ErrorKeys takes the rule table as a second argument: a Go map cannot carry the message-bag order, so the order is rebuilt from the rules (explicit attributes first, then wildcard expansions by rule, indexes ascending)" + - "exists:table,column compares CAST(column AS TEXT) with the value's PHP string form, so a non-numeric value against an integer column is a validation failure, never a Postgres error" + - "With a nil translator ValidateRequest returns Laravel's untranslated keys (validation.required); callers pass the app translator" + - "lagoon.Validate numeric between failures use the new lagoon::validation.between.numeric line; every existing lagoon::validate text is unchanged" + - "user_groups timestamps are TIMESTAMPTZ NULL, matching the plugin's other tables, rather than plain TIMESTAMP" + +patterns-established: + - "Laravel validator parity is checked against the real Winter validator (vendor autoload + Winter Factory) before pinning Go expectations" + +requirements-completed: [API-01, API-02] + +coverage: + - id: D1 + description: "lagoon.ValidateRequest reproduces Laravel 9 request validation (implicit stop, wildcards, size messages by type, character counts, decimal bounds) with pl/en Winter catalogs" + requirement: API-02 + verification: + - kind: unit + ref: "modules/lagoon/validate_request_test.go#TestValidateRequest*" + status: pass + - kind: other + ref: "php vendor Winter Validator cross-check of 11 cases (posts [], wildcards, garbage date, min.numeric, presence, order, integer, email, boolean)" + status: pass + human_judgment: false + - id: D2 + description: "lagoon.Validate answers a numeric range failure with the bound that failed (min, max or numeric between)" + requirement: API-01 + verification: + - kind: unit + ref: "modules/lagoon/validate_test.go#TestValidateNumericRangeMessagePicksFailedBound" + status: pass + - kind: integration + ref: "go -C ../fonoteka.go test ./parity -run TestUserAPINuxtFlows" + status: pass + human_judgment: false + - id: D3 + description: "attach.PublicURL/File.URL emit Winter URLs under the Winter layout config; webp originals decode and thumbnail as JPEG bytes" + requirement: API-02 + verification: + - kind: unit + ref: "modules/lagoon/attach/url_test.go#TestFileURLWinterLayout" + status: pass + - kind: unit + ref: "modules/lagoon/attach/url_test.go#TestThumbWebP" + status: pass + human_judgment: false + - id: D4 + description: "tide records and replays multipart bodies byte-identically, refuses tampered part files, masks upload URLs by shape and album dates in publications" + requirement: API-02 + verification: + - kind: unit + ref: "modules/tide/multipart_test.go#TestMultipart*" + status: pass + - kind: unit + ref: "modules/tide/multipart_test.go#TestNormalizeUploadURL*" + status: pass + - kind: unit + ref: "modules/tide/multipart_test.go#TestNormalizePublicationAlbumDates" + status: pass + human_judgment: false + - id: D5 + description: "beachcomber exposes the engine found count through PageSearcher/SearchPage and sends query_by_weights" + requirement: API-02 + verification: + - kind: unit + ref: "modules/beachcomber/searchpage_test.go#TestSearchPage*" + status: pass + - kind: unit + ref: "modules/beachcomber/typesense/searchpage_test.go#TestTypesenseSearchPage*" + status: pass + human_judgment: false + - id: D6 + description: "User groups tables, model, relation and group-code queries in sm-user-plugin with the user API unchanged" + requirement: API-01 + verification: + - kind: integration + ref: "../fonoteka.go/plugins/golem15/user/updates/user_groups_test.go#TestUserGroups*" + status: pass + - kind: integration + ref: "go -C ../fonoteka.go test ./parity -run '^(TestSchemaMatchesPHPSnapshot|TestUserAPINuxtFlows)$'" + status: pass + human_judgment: false + - id: D7 + description: "ROADMAP and REQUIREMENTS carry the D-03/D-04/D-06/D-19/D-20 Phase 12 boundary" + verification: [] + human_judgment: true + rationale: "Planning-doc wording; grep checks pass, but whether it reads as the user intended is a human call" + +duration: 39min +completed: 2026-10-02 +status: complete +--- + +# Phase 12 Plan 01: Framework gaps, user groups and Phase 12 scope Summary + +**Laravel 9 request validator with WinterCMS pl/en catalogs, Winter upload URLs and webp, multipart tide fixtures with shape-masked upload URLs, a re-gateable search found count with field weights, and Winter user groups in sm-user-plugin.** + +## Performance + +- **Duration:** 39 min +- **Started:** 2026-10-02T09:09:24Z +- **Completed:** 2026-10-02T09:48:46Z +- **Tasks:** 4 +- **Files modified:** 57 (39 summercms.go, 8 fonoteka.go, 10 sm-user-plugin) + +## Accomplishments + +- `lagoon.ValidateRequest` ports Laravel 9 request validation: wildcard expansion in Laravel's attribute order, implicit-rule stop, `bail`, `sometimes`/`nullable`/blank skipping, size messages split by type, string length in characters and exact decimal bounds. `{"posts":[]}` against `required|array|min:1` gives the single `Pole posts jest wymagane.` in Polish. The Go expectations were checked case by case against the real Winter validator run from the PHP vendor tree. +- The `lagoon::validation` pl/en catalogs are generated verbatim from Winter's `validation.php` files. Keys missing in Polish (`after_or_equal`, `before_or_equal`) fall back to English. +- The `lagoon-validate-min-message` todo is folded: `lagoon.Validate` now reports the bound that failed, and every user-plugin 422 fixture still replays. +- `attach.PublicURL` and `(*attach.File).URL` produce Winter's `File::getPath()` URLs. The webp decoder is registered, and the thumbnailer reads the image size from the header before it decodes. +- tide requests can carry multipart `parts`, encoded with the fixed `tide.MultipartBoundary`. File bytes stay beside the fixture, pinned by sha256. Upload URLs under `url`/`thumb_url` are compared by shape (default prefix `/storage/app/uploads/public`), and Carbon dates in the published album subtree are masked. +- `beachcomber.PageSearcher`, `beachcomber.SearchResult` and `beachcomber.SearchPage` expose the engine's `found` count. `Query.QueryByWeights` is sent as `query_by_weights`. A Typesense page above 250 is refused before any request is sent. +- sm-user-plugin gains `user_groups` (with the Guest and Registered seed rows) and the `users_groups` pivot, plus `models.UserGroup`, a `User.Groups` relation that is never serialized, and `classes.UserGroupCodes`/`HasGroupCode`. No controller changed. +- ROADMAP Phases 12-14 and REQUIREMENTS API-01/02/03/07 and INTG-01 now state the locked Phase 12 boundary. + +## Task Commits + +1. **Task 1: Laravel request validator (tracer)**: `f9b7f2e` (feat, summercms.go) +2. **Task 2: uploads, webp, tide multipart and masks**: `e06e0cc` (feat, summercms.go); `c65ab3c` (build, sm-user-plugin); `6d6c827` (build, fonoteka.go) +3. **Task 3: search found/weights and user groups**: `3ae49bb` (feat, summercms.go); `8697007` (feat, sm-user-plugin); `02e1aa9` (feat, fonoteka.go, pointer bump) +4. **Task 4: roadmap and requirements wording**: `1bdbdf0` (docs, summercms.go) + +Per-repo plan ledgers (commits measured with `git rev-list --count`): +- summercms.go: `cfe568a..1bdbdf0`, 4 commits +- fonoteka.go: `18576db..02e1aa9`, 2 commits +- sm-user-plugin: `44f3c4c..8697007`, 2 commits + +## Files Created/Modified + +- `modules/lagoon/validate_request.go`: ValidateRequest, RequestRule, Rule, CustomRule, In, UploadedFile, ErrorKeys, the expansion and message logic +- `modules/lagoon/validate_rules.go`: ParseRules and the rule implementations (FILTER_VALIDATE_EMAIL, the Laravel url regex rewritten for RE2, dates, exists, mimes, size) +- `modules/lagoon/validate.go`: numericRangeMessage (the min/between fix); removed the now-unused moneyInRange +- `modules/phrasebook/lang/{pl,en}/validation.yaml`: Winter catalog ports +- `modules/lagoon/attach/thumb.go`: PublicURL, File.URL, webp import, DecodeConfig before Decode +- `modules/tide/multipart.go` (+ flow, fixture, record, replay, variables, diff, normalize, centrifugo_golden): Parts, MultipartBoundary, upload and album-date masking +- `modules/beachcomber/searchable.go`, `engines.go`, `typesense/engine.go`: PageSearcher, SearchResult, SearchPage, QueryByWeights, MaxPerPage +- `../fonoteka.go/plugins/golem15/user/...`: UserGroup model, Groups relation, migration 202610020001_create_user_groups, group-code queries, README +- `../fonoteka.go/parity/schema_diff_test.go`, `parity/migrate_test.go`, `plugins/golem15/fonoteka/classes/hidden_marshal_test.go`: allow-list and count updates +- READMEs and docs: `modules/lagoon/README.md`, `modules/phrasebook/README.md`, `modules/tide/README.md`, `modules/beachcomber/README.md`, `docs/database/casts-and-validation.md`, `docs/database/attachments.md`, `docs/services/storage.md`, `docs/services/parity-testing.md`, `docs/services/search.md`, `docs/services/localization.md` + +## Decisions Made + +See `key-decisions` in the frontmatter. The two that change the plan's literal text are written up under Deviations. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Wildcard with an absent parent adds no attribute (plan said `required` still fires)** +- **Found during:** Task 1 +- **Issue:** The plan said an absent parent should expand to the literal attribute so that `required` still fires. Laravel's `ValidationRuleParser::explodeWildcardRules` drops a wildcard rule that has nothing to expand, and the real Winter validator returns `[]` for `{}` against `posts.*.title: required`. +- **Fix:** Implemented Laravel's behaviour and pinned it with `TestValidateRequestWildcardWithoutParentAddsNothing`. +- **Files modified:** modules/lagoon/validate_request.go +- **Commit:** f9b7f2e + +**2. [Rule 3 - Blocking] `ErrorKeys(errs, rules)` takes the rule table** +- **Found during:** Task 1 +- **Issue:** The planned `ErrorKeys(errs map[string][]string)` cannot recover an order from a Go map. +- **Fix:** Added the rule table as a second argument. The function rebuilds Laravel's message-bag order from the rules (cross-checked against PHP output). +- **Commit:** f9b7f2e + +**3. [Rule 3 - Blocking] Acceptance grep `grep -c 'Pole :attribute jest wymagane.'` prints 7, not 1** +- **Found during:** Task 1 +- **Issue:** The faithful port carries every Winter key (filled, required_if, required_with and others). In grep, the unescaped `.` matches the space in `wymagane gdy`, so the pattern hits 7 lines. +- **Fix:** None needed. The exact line `Pole :attribute jest wymagane."` appears for `required` and `filled`, as in Winter (`grep -cF` prints 2). The port stays faithful. + +**4. [Rule 2 - Missing critical] Thumbnailer checks dimensions before decoding** +- **Found during:** Task 2 (T-12-16) +- **Issue:** `File.Thumb` decoded the whole image before its pixel check. A small webp or png that declares a huge canvas would allocate the full image first. +- **Fix:** It now reads the bounded bytes, runs `image.DecodeConfig`, refuses anything over 4096x4096, and only then decodes. +- **Commit:** e06e0cc + +**5. [Rule 3 - Blocking] Migration and model counts in existing tests** +- **Found during:** Task 3 +- **Issue:** Adding a sixth user migration and a fourth user model broke hard-coded counts in `updates/organisations_test.go` and `updates/user_session_test.go` (sm-user-plugin), and in `parity/migrate_test.go` (three assertions) and `plugins/golem15/fonoteka/classes/hidden_marshal_test.go` (fonoteka.go). +- **Fix:** Updated the expected histories, rollback counts and model count. No production code changed. +- **Commits:** 8697007, 02e1aa9 + +**6. [Rule 3 - Blocking] tide `RecordConfig.BaseDir`, `ReplayConfig.UploadPrefix` and `tide.DefaultUploadPrefix`** +- **Found during:** Task 2 +- **Issue:** Recording needs a directory to read part files from, and the upload prefix had to be configurable. +- **Fix:** Added both fields and the default constant. All are additive and documented in the tide README. +- **Commit:** e06e0cc + +**7. [Orchestrator instruction] sm-user-plugin not pushed** +- The plan says to push sm-user-plugin `master` before the pointer bump. The orchestrator prompt says not to push anywhere, so commits `c65ab3c` and `8697007` exist locally only. The fonoteka.go pointer references them. + +--- + +**Total deviations:** 6 auto-fixed (1 bug, 1 missing critical, 4 blocking) plus 1 instruction-driven. +**Impact on plan:** All deliverables shipped. The deviations follow the PHP contract or keep existing tests honest; there is no scope creep. + +## Issues Encountered + +- One full `go test ./...` run in summercms.go failed in `modules/lighthouse`: its testcontainers Postgres did not become ready in time under parallel package load. lighthouse was not touched by this plan. It passed alone, and a second full run was green. +- All commits were made on `master` in every repo (the project uses `branching_strategy: none`, and the orchestrator dispatched a sequential executor on the main tree). The generic protected-branch guard would refuse `master`. The orchestrator's instruction and the project convention were followed instead. +- `.planning/ROADMAP.md` already held an uncommitted Phase 12.1 insertion that the executor did not make. The Task 4 commit staged only this plan's hunks. + +## Known Stubs + +None. + +## User Setup Required + +None. No external service configuration is required. The app config change that adopts the Winter upload prefix (`storage.uploads.bucket_url` / `public_path_prefix` in fonoteka.go `config/storage.yaml`) belongs to 12-02/12-04. + +## Next Phase Readiness + +- 12-02 can build the collection endpoints on `ValidateRequest`, `HasGroupCode(..., "admin")`, `attach.File.URL` and tide multipart. +- 12-04 still has to set the Winter upload layout in `config/storage.yaml` and check that avatar URLs stay correct. It uses `beachcomber.SearchPage` for the D-19 recount. +- The PHP-oracle approach (vendor autoload plus `Winter\Storm\Validation\Factory`) is available to 12-05 for table-driven validator coverage. + +## Self-Check: PASSED + +All 15 created files listed above exist on disk. All 8 task commits exist: f9b7f2e, e06e0cc, 3ae49bb and 1bdbdf0 in summercms.go; 6d6c827 and 02e1aa9 in fonoteka.go; c65ab3c and 8697007 in sm-user-plugin. Plan-level verification passed: summercms.go `go vet ./... && go test ./...` (green on rerun), `TestDocsTree`, and fonoteka.go vet plus tests for the root module, the fonoteka plugin and the user plugin. + +--- +*Phase: 12-p-ytarium-api-collections-and-albums* +*Completed: 2026-10-02*