docs(12-01): complete framework gaps and user groups plan
This commit is contained in:
Jakub Zych
@@ -0,0 +1,293 @@
|
|||||||
|
---
|
||||||
|
phase: 12-p-ytarium-api-collections-and-albums
|
||||||
|
plan: 01
|
||||||
|
subsystem: api
|
||||||
|
tags: [lagoon, validation, laravel, phrasebook, attach, webp, tide, multipart, beachcomber, typesense, user-groups]
|
||||||
|
|
||||||
|
requires:
|
||||||
|
- phase: 05-data-layer-full-fidelity
|
||||||
|
provides: lagoon.Validate, attach File/Thumb/BlobKey, phrasebook catalogs
|
||||||
|
- phase: 11-jobs-realtime-and-search-infrastructure
|
||||||
|
provides: beachcomber Engine/Query, typesense engine, tide broadcast goldens
|
||||||
|
- phase: 07-user-plugin-and-authentication
|
||||||
|
provides: Go user plugin (sm-user-plugin) models, migrations and Postgres test harness
|
||||||
|
provides:
|
||||||
|
- lagoon.ValidateRequest with Laravel 9 semantics and the pl/en lagoon::validation catalogs
|
||||||
|
- ParseRules, In, CustomRule, UploadedFile/UploadedFileFromHeader, ErrorKeys
|
||||||
|
- lagoon.Validate min/max/numeric-between message fix (todo folded)
|
||||||
|
- attach.PublicURL, (*attach.File).URL, webp decoding (golang.org/x/image v0.46.0)
|
||||||
|
- tide multipart Parts with fixed MultipartBoundary, upload URL masking, album date masking in publications
|
||||||
|
- beachcomber PageSearcher/SearchResult/SearchPage and Query.QueryByWeights (typesense query_by_weights, MaxPerPage 250)
|
||||||
|
- user_groups/users_groups tables, models.UserGroup, User.Groups, classes.UserGroupCodes/HasGroupCode
|
||||||
|
- ROADMAP Phase 12-14 and REQUIREMENTS API-01/02/03/07, INTG-01 rewording
|
||||||
|
affects: [12-02, 12-03, 12-04, 12-05, 12.1, 13, 14]
|
||||||
|
|
||||||
|
actuals:
|
||||||
|
tokens: 55700
|
||||||
|
tasks: 4
|
||||||
|
commits: 8
|
||||||
|
plan_head_before: cfe568a214139a438ff5aecc27cc02e5f6f06892
|
||||||
|
plan_head_after: 1bdbdf041b7deb17a776ba8ac0b12bd7d7db8b83
|
||||||
|
|
||||||
|
tech-stack:
|
||||||
|
added: [golang.org/x/image v0.46.0 (direct, summercms.go)]
|
||||||
|
patterns:
|
||||||
|
- "Request rule tables are package-level []lagoon.RequestRule built with ParseRules; a bad rule panics at boot"
|
||||||
|
- "Optional engine capability via a type-asserted interface plus a fallback helper (PageSearcher/SearchPage)"
|
||||||
|
- "Multipart fixtures: file bytes beside the fixture, pinned by sha256, encoded with one fixed boundary"
|
||||||
|
|
||||||
|
key-files:
|
||||||
|
created:
|
||||||
|
- modules/lagoon/validate_request.go
|
||||||
|
- modules/lagoon/validate_rules.go
|
||||||
|
- modules/lagoon/validate_request_test.go
|
||||||
|
- modules/phrasebook/lang/pl/validation.yaml
|
||||||
|
- modules/phrasebook/lang/en/validation.yaml
|
||||||
|
- modules/lagoon/attach/url_test.go
|
||||||
|
- modules/tide/multipart.go
|
||||||
|
- modules/tide/multipart_test.go
|
||||||
|
- modules/beachcomber/searchpage_test.go
|
||||||
|
- modules/beachcomber/typesense/searchpage_test.go
|
||||||
|
- ../fonoteka.go/plugins/golem15/user/models/user_group.go
|
||||||
|
- ../fonoteka.go/plugins/golem15/user/updates/202610020001_create_user_groups.go
|
||||||
|
- ../fonoteka.go/plugins/golem15/user/updates/user_groups_test.go
|
||||||
|
- ../fonoteka.go/plugins/golem15/user/classes/user_groups.go
|
||||||
|
modified:
|
||||||
|
- modules/lagoon/validate.go
|
||||||
|
- modules/lagoon/attach/thumb.go
|
||||||
|
- modules/tide/flow.go
|
||||||
|
- modules/tide/fixture.go
|
||||||
|
- modules/tide/record.go
|
||||||
|
- modules/tide/replay.go
|
||||||
|
- modules/tide/diff.go
|
||||||
|
- modules/tide/normalize.go
|
||||||
|
- modules/tide/variables.go
|
||||||
|
- modules/tide/centrifugo_golden.go
|
||||||
|
- modules/beachcomber/searchable.go
|
||||||
|
- modules/beachcomber/engines.go
|
||||||
|
- modules/beachcomber/typesense/engine.go
|
||||||
|
- ../fonoteka.go/plugins/golem15/user/models/user.go
|
||||||
|
- ../fonoteka.go/parity/schema_diff_test.go
|
||||||
|
- go.mod
|
||||||
|
|
||||||
|
key-decisions:
|
||||||
|
- "Wildcard rules follow Laravel exactly: a wildcard with nothing to expand adds no attribute (the plan's 'absent parent still fires required' contradicted Validator.php and the live PHP validator)"
|
||||||
|
- "ErrorKeys takes the rule table as a second argument: a Go map cannot carry the message-bag order, so the order is rebuilt from the rules (explicit attributes first, then wildcard expansions by rule, indexes ascending)"
|
||||||
|
- "exists:table,column compares CAST(column AS TEXT) with the value's PHP string form, so a non-numeric value against an integer column is a validation failure, never a Postgres error"
|
||||||
|
- "With a nil translator ValidateRequest returns Laravel's untranslated keys (validation.required); callers pass the app translator"
|
||||||
|
- "lagoon.Validate numeric between failures use the new lagoon::validation.between.numeric line; every existing lagoon::validate text is unchanged"
|
||||||
|
- "user_groups timestamps are TIMESTAMPTZ NULL, matching the plugin's other tables, rather than plain TIMESTAMP"
|
||||||
|
|
||||||
|
patterns-established:
|
||||||
|
- "Laravel validator parity is checked against the real Winter validator (vendor autoload + Winter Factory) before pinning Go expectations"
|
||||||
|
|
||||||
|
requirements-completed: [API-01, API-02]
|
||||||
|
|
||||||
|
coverage:
|
||||||
|
- id: D1
|
||||||
|
description: "lagoon.ValidateRequest reproduces Laravel 9 request validation (implicit stop, wildcards, size messages by type, character counts, decimal bounds) with pl/en Winter catalogs"
|
||||||
|
requirement: API-02
|
||||||
|
verification:
|
||||||
|
- kind: unit
|
||||||
|
ref: "modules/lagoon/validate_request_test.go#TestValidateRequest*"
|
||||||
|
status: pass
|
||||||
|
- kind: other
|
||||||
|
ref: "php vendor Winter Validator cross-check of 11 cases (posts [], wildcards, garbage date, min.numeric, presence, order, integer, email, boolean)"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D2
|
||||||
|
description: "lagoon.Validate answers a numeric range failure with the bound that failed (min, max or numeric between)"
|
||||||
|
requirement: API-01
|
||||||
|
verification:
|
||||||
|
- kind: unit
|
||||||
|
ref: "modules/lagoon/validate_test.go#TestValidateNumericRangeMessagePicksFailedBound"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "go -C ../fonoteka.go test ./parity -run TestUserAPINuxtFlows"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D3
|
||||||
|
description: "attach.PublicURL/File.URL emit Winter URLs under the Winter layout config; webp originals decode and thumbnail as JPEG bytes"
|
||||||
|
requirement: API-02
|
||||||
|
verification:
|
||||||
|
- kind: unit
|
||||||
|
ref: "modules/lagoon/attach/url_test.go#TestFileURLWinterLayout"
|
||||||
|
status: pass
|
||||||
|
- kind: unit
|
||||||
|
ref: "modules/lagoon/attach/url_test.go#TestThumbWebP"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D4
|
||||||
|
description: "tide records and replays multipart bodies byte-identically, refuses tampered part files, masks upload URLs by shape and album dates in publications"
|
||||||
|
requirement: API-02
|
||||||
|
verification:
|
||||||
|
- kind: unit
|
||||||
|
ref: "modules/tide/multipart_test.go#TestMultipart*"
|
||||||
|
status: pass
|
||||||
|
- kind: unit
|
||||||
|
ref: "modules/tide/multipart_test.go#TestNormalizeUploadURL*"
|
||||||
|
status: pass
|
||||||
|
- kind: unit
|
||||||
|
ref: "modules/tide/multipart_test.go#TestNormalizePublicationAlbumDates"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D5
|
||||||
|
description: "beachcomber exposes the engine found count through PageSearcher/SearchPage and sends query_by_weights"
|
||||||
|
requirement: API-02
|
||||||
|
verification:
|
||||||
|
- kind: unit
|
||||||
|
ref: "modules/beachcomber/searchpage_test.go#TestSearchPage*"
|
||||||
|
status: pass
|
||||||
|
- kind: unit
|
||||||
|
ref: "modules/beachcomber/typesense/searchpage_test.go#TestTypesenseSearchPage*"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D6
|
||||||
|
description: "User groups tables, model, relation and group-code queries in sm-user-plugin with the user API unchanged"
|
||||||
|
requirement: API-01
|
||||||
|
verification:
|
||||||
|
- kind: integration
|
||||||
|
ref: "../fonoteka.go/plugins/golem15/user/updates/user_groups_test.go#TestUserGroups*"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "go -C ../fonoteka.go test ./parity -run '^(TestSchemaMatchesPHPSnapshot|TestUserAPINuxtFlows)$'"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D7
|
||||||
|
description: "ROADMAP and REQUIREMENTS carry the D-03/D-04/D-06/D-19/D-20 Phase 12 boundary"
|
||||||
|
verification: []
|
||||||
|
human_judgment: true
|
||||||
|
rationale: "Planning-doc wording; grep checks pass, but whether it reads as the user intended is a human call"
|
||||||
|
|
||||||
|
duration: 39min
|
||||||
|
completed: 2026-10-02
|
||||||
|
status: complete
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 12 Plan 01: Framework gaps, user groups and Phase 12 scope Summary
|
||||||
|
|
||||||
|
**Laravel 9 request validator with WinterCMS pl/en catalogs, Winter upload URLs and webp, multipart tide fixtures with shape-masked upload URLs, a re-gateable search found count with field weights, and Winter user groups in sm-user-plugin.**
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
- **Duration:** 39 min
|
||||||
|
- **Started:** 2026-10-02T09:09:24Z
|
||||||
|
- **Completed:** 2026-10-02T09:48:46Z
|
||||||
|
- **Tasks:** 4
|
||||||
|
- **Files modified:** 57 (39 summercms.go, 8 fonoteka.go, 10 sm-user-plugin)
|
||||||
|
|
||||||
|
## Accomplishments
|
||||||
|
|
||||||
|
- `lagoon.ValidateRequest` ports Laravel 9 request validation: wildcard expansion in Laravel's attribute order, implicit-rule stop, `bail`, `sometimes`/`nullable`/blank skipping, size messages split by type, string length in characters and exact decimal bounds. `{"posts":[]}` against `required|array|min:1` gives the single `Pole posts jest wymagane.` in Polish. The Go expectations were checked case by case against the real Winter validator run from the PHP vendor tree.
|
||||||
|
- The `lagoon::validation` pl/en catalogs are generated verbatim from Winter's `validation.php` files. Keys missing in Polish (`after_or_equal`, `before_or_equal`) fall back to English.
|
||||||
|
- The `lagoon-validate-min-message` todo is folded: `lagoon.Validate` now reports the bound that failed, and every user-plugin 422 fixture still replays.
|
||||||
|
- `attach.PublicURL` and `(*attach.File).URL` produce Winter's `File::getPath()` URLs. The webp decoder is registered, and the thumbnailer reads the image size from the header before it decodes.
|
||||||
|
- tide requests can carry multipart `parts`, encoded with the fixed `tide.MultipartBoundary`. File bytes stay beside the fixture, pinned by sha256. Upload URLs under `url`/`thumb_url` are compared by shape (default prefix `/storage/app/uploads/public`), and Carbon dates in the published album subtree are masked.
|
||||||
|
- `beachcomber.PageSearcher`, `beachcomber.SearchResult` and `beachcomber.SearchPage` expose the engine's `found` count. `Query.QueryByWeights` is sent as `query_by_weights`. A Typesense page above 250 is refused before any request is sent.
|
||||||
|
- sm-user-plugin gains `user_groups` (with the Guest and Registered seed rows) and the `users_groups` pivot, plus `models.UserGroup`, a `User.Groups` relation that is never serialized, and `classes.UserGroupCodes`/`HasGroupCode`. No controller changed.
|
||||||
|
- ROADMAP Phases 12-14 and REQUIREMENTS API-01/02/03/07 and INTG-01 now state the locked Phase 12 boundary.
|
||||||
|
|
||||||
|
## Task Commits
|
||||||
|
|
||||||
|
1. **Task 1: Laravel request validator (tracer)**: `f9b7f2e` (feat, summercms.go)
|
||||||
|
2. **Task 2: uploads, webp, tide multipart and masks**: `e06e0cc` (feat, summercms.go); `c65ab3c` (build, sm-user-plugin); `6d6c827` (build, fonoteka.go)
|
||||||
|
3. **Task 3: search found/weights and user groups**: `3ae49bb` (feat, summercms.go); `8697007` (feat, sm-user-plugin); `02e1aa9` (feat, fonoteka.go, pointer bump)
|
||||||
|
4. **Task 4: roadmap and requirements wording**: `1bdbdf0` (docs, summercms.go)
|
||||||
|
|
||||||
|
Per-repo plan ledgers (commits measured with `git rev-list --count`):
|
||||||
|
- summercms.go: `cfe568a..1bdbdf0`, 4 commits
|
||||||
|
- fonoteka.go: `18576db..02e1aa9`, 2 commits
|
||||||
|
- sm-user-plugin: `44f3c4c..8697007`, 2 commits
|
||||||
|
|
||||||
|
## Files Created/Modified
|
||||||
|
|
||||||
|
- `modules/lagoon/validate_request.go`: ValidateRequest, RequestRule, Rule, CustomRule, In, UploadedFile, ErrorKeys, the expansion and message logic
|
||||||
|
- `modules/lagoon/validate_rules.go`: ParseRules and the rule implementations (FILTER_VALIDATE_EMAIL, the Laravel url regex rewritten for RE2, dates, exists, mimes, size)
|
||||||
|
- `modules/lagoon/validate.go`: numericRangeMessage (the min/between fix); removed the now-unused moneyInRange
|
||||||
|
- `modules/phrasebook/lang/{pl,en}/validation.yaml`: Winter catalog ports
|
||||||
|
- `modules/lagoon/attach/thumb.go`: PublicURL, File.URL, webp import, DecodeConfig before Decode
|
||||||
|
- `modules/tide/multipart.go` (+ flow, fixture, record, replay, variables, diff, normalize, centrifugo_golden): Parts, MultipartBoundary, upload and album-date masking
|
||||||
|
- `modules/beachcomber/searchable.go`, `engines.go`, `typesense/engine.go`: PageSearcher, SearchResult, SearchPage, QueryByWeights, MaxPerPage
|
||||||
|
- `../fonoteka.go/plugins/golem15/user/...`: UserGroup model, Groups relation, migration 202610020001_create_user_groups, group-code queries, README
|
||||||
|
- `../fonoteka.go/parity/schema_diff_test.go`, `parity/migrate_test.go`, `plugins/golem15/fonoteka/classes/hidden_marshal_test.go`: allow-list and count updates
|
||||||
|
- READMEs and docs: `modules/lagoon/README.md`, `modules/phrasebook/README.md`, `modules/tide/README.md`, `modules/beachcomber/README.md`, `docs/database/casts-and-validation.md`, `docs/database/attachments.md`, `docs/services/storage.md`, `docs/services/parity-testing.md`, `docs/services/search.md`, `docs/services/localization.md`
|
||||||
|
|
||||||
|
## Decisions Made
|
||||||
|
|
||||||
|
See `key-decisions` in the frontmatter. The two that change the plan's literal text are written up under Deviations.
|
||||||
|
|
||||||
|
## Deviations from Plan
|
||||||
|
|
||||||
|
### Auto-fixed Issues
|
||||||
|
|
||||||
|
**1. [Rule 1 - Bug] Wildcard with an absent parent adds no attribute (plan said `required` still fires)**
|
||||||
|
- **Found during:** Task 1
|
||||||
|
- **Issue:** The plan said an absent parent should expand to the literal attribute so that `required` still fires. Laravel's `ValidationRuleParser::explodeWildcardRules` drops a wildcard rule that has nothing to expand, and the real Winter validator returns `[]` for `{}` against `posts.*.title: required`.
|
||||||
|
- **Fix:** Implemented Laravel's behaviour and pinned it with `TestValidateRequestWildcardWithoutParentAddsNothing`.
|
||||||
|
- **Files modified:** modules/lagoon/validate_request.go
|
||||||
|
- **Commit:** f9b7f2e
|
||||||
|
|
||||||
|
**2. [Rule 3 - Blocking] `ErrorKeys(errs, rules)` takes the rule table**
|
||||||
|
- **Found during:** Task 1
|
||||||
|
- **Issue:** The planned `ErrorKeys(errs map[string][]string)` cannot recover an order from a Go map.
|
||||||
|
- **Fix:** Added the rule table as a second argument. The function rebuilds Laravel's message-bag order from the rules (cross-checked against PHP output).
|
||||||
|
- **Commit:** f9b7f2e
|
||||||
|
|
||||||
|
**3. [Rule 3 - Blocking] Acceptance grep `grep -c 'Pole :attribute jest wymagane.'` prints 7, not 1**
|
||||||
|
- **Found during:** Task 1
|
||||||
|
- **Issue:** The faithful port carries every Winter key (filled, required_if, required_with and others). In grep, the unescaped `.` matches the space in `wymagane gdy`, so the pattern hits 7 lines.
|
||||||
|
- **Fix:** None needed. The exact line `Pole :attribute jest wymagane."` appears for `required` and `filled`, as in Winter (`grep -cF` prints 2). The port stays faithful.
|
||||||
|
|
||||||
|
**4. [Rule 2 - Missing critical] Thumbnailer checks dimensions before decoding**
|
||||||
|
- **Found during:** Task 2 (T-12-16)
|
||||||
|
- **Issue:** `File.Thumb` decoded the whole image before its pixel check. A small webp or png that declares a huge canvas would allocate the full image first.
|
||||||
|
- **Fix:** It now reads the bounded bytes, runs `image.DecodeConfig`, refuses anything over 4096x4096, and only then decodes.
|
||||||
|
- **Commit:** e06e0cc
|
||||||
|
|
||||||
|
**5. [Rule 3 - Blocking] Migration and model counts in existing tests**
|
||||||
|
- **Found during:** Task 3
|
||||||
|
- **Issue:** Adding a sixth user migration and a fourth user model broke hard-coded counts in `updates/organisations_test.go` and `updates/user_session_test.go` (sm-user-plugin), and in `parity/migrate_test.go` (three assertions) and `plugins/golem15/fonoteka/classes/hidden_marshal_test.go` (fonoteka.go).
|
||||||
|
- **Fix:** Updated the expected histories, rollback counts and model count. No production code changed.
|
||||||
|
- **Commits:** 8697007, 02e1aa9
|
||||||
|
|
||||||
|
**6. [Rule 3 - Blocking] tide `RecordConfig.BaseDir`, `ReplayConfig.UploadPrefix` and `tide.DefaultUploadPrefix`**
|
||||||
|
- **Found during:** Task 2
|
||||||
|
- **Issue:** Recording needs a directory to read part files from, and the upload prefix had to be configurable.
|
||||||
|
- **Fix:** Added both fields and the default constant. All are additive and documented in the tide README.
|
||||||
|
- **Commit:** e06e0cc
|
||||||
|
|
||||||
|
**7. [Orchestrator instruction] sm-user-plugin not pushed**
|
||||||
|
- The plan says to push sm-user-plugin `master` before the pointer bump. The orchestrator prompt says not to push anywhere, so commits `c65ab3c` and `8697007` exist locally only. The fonoteka.go pointer references them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Total deviations:** 6 auto-fixed (1 bug, 1 missing critical, 4 blocking) plus 1 instruction-driven.
|
||||||
|
**Impact on plan:** All deliverables shipped. The deviations follow the PHP contract or keep existing tests honest; there is no scope creep.
|
||||||
|
|
||||||
|
## Issues Encountered
|
||||||
|
|
||||||
|
- One full `go test ./...` run in summercms.go failed in `modules/lighthouse`: its testcontainers Postgres did not become ready in time under parallel package load. lighthouse was not touched by this plan. It passed alone, and a second full run was green.
|
||||||
|
- All commits were made on `master` in every repo (the project uses `branching_strategy: none`, and the orchestrator dispatched a sequential executor on the main tree). The generic protected-branch guard would refuse `master`. The orchestrator's instruction and the project convention were followed instead.
|
||||||
|
- `.planning/ROADMAP.md` already held an uncommitted Phase 12.1 insertion that the executor did not make. The Task 4 commit staged only this plan's hunks.
|
||||||
|
|
||||||
|
## Known Stubs
|
||||||
|
|
||||||
|
None.
|
||||||
|
|
||||||
|
## User Setup Required
|
||||||
|
|
||||||
|
None. No external service configuration is required. The app config change that adopts the Winter upload prefix (`storage.uploads.bucket_url` / `public_path_prefix` in fonoteka.go `config/storage.yaml`) belongs to 12-02/12-04.
|
||||||
|
|
||||||
|
## Next Phase Readiness
|
||||||
|
|
||||||
|
- 12-02 can build the collection endpoints on `ValidateRequest`, `HasGroupCode(..., "admin")`, `attach.File.URL` and tide multipart.
|
||||||
|
- 12-04 still has to set the Winter upload layout in `config/storage.yaml` and check that avatar URLs stay correct. It uses `beachcomber.SearchPage` for the D-19 recount.
|
||||||
|
- The PHP-oracle approach (vendor autoload plus `Winter\Storm\Validation\Factory`) is available to 12-05 for table-driven validator coverage.
|
||||||
|
|
||||||
|
## Self-Check: PASSED
|
||||||
|
|
||||||
|
All 15 created files listed above exist on disk. All 8 task commits exist: f9b7f2e, e06e0cc, 3ae49bb and 1bdbdf0 in summercms.go; 6d6c827 and 02e1aa9 in fonoteka.go; c65ab3c and 8697007 in sm-user-plugin. Plan-level verification passed: summercms.go `go vet ./... && go test ./...` (green on rerun), `TestDocsTree`, and fonoteka.go vet plus tests for the root module, the fonoteka plugin and the user plugin.
|
||||||
|
|
||||||
|
---
|
||||||
|
*Phase: 12-p-ytarium-api-collections-and-albums*
|
||||||
|
*Completed: 2026-10-02*
|
||||||
Reference in New Issue
Block a user