golem15/summercms
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
77f8b371cfa93d378231cb47c39fa5c003a87d9d
summercms/.planning/phases/12-p-ytarium-api-collections-and-albums/12-RESEARCH.md
Jakub Zych 77f8b371cf docs(12): research phase domain
2026-10-02 01:08:08 +02:00

70 KiB
Raw Blame History

Phase 12: Płytarium API — Collections and Albums - Research

Researched: 2026-10-02 Domain: Go port of the PHP collection/album/household HTTP surface (parity-driven), with authorization, uploads, SSRF-guarded fetches, search re-gating and transactional mail jobs Confidence: HIGH for the PHP contract and the existing Go seams (both read line by line this session); MEDIUM for the sizing and the plan split

Summary

Phase 12 is mostly porting work, but it is much wider than CONTEXT.md's route list makes it look. The Go side has only thin stubs for most of what these routes need:

  • classes.SerializeAlbum and SerializeCollection emit 8 and 4 keys.
  • classes.ResolveArtists only checks numeric ids.
  • SaveAlbum handles neither styles, tracklist text, dates, covers nor completeness.
  • ResolveActiveCollection ignores personal-token pins, never provisions a collection and never runs the pending-invitation guard.
  • There is no request-facing AccessibleBy that adds the token's collection_ids narrowing.

On top of the controllers, the PHP routes pull in about 15 helper classes that CONTEXT.md does not list: AlbumCompletenessService, AlbumDuplicateMatcher, TracklistTextParser, AddedDateParser, OrgProvisioner, AiGate, DiscogsGate, DiscogsConfigResolver, OrgAccess, PolishOrder, DiscogsInputParser::normalizeBarcode, NotificationService::notifyAlbumAdded (fired by an eloquent.created listener on every album create), plus the full ArtistResolver (name and Discogs-id resolution with name_key firstOrCreate). Each one changes a response body or the DB state a parity replay checks.

Four findings change locked decisions or roadmap wording and must reach the user before planning:

  1. D-16 point 3 is factually wrong for the installed Scout. laravel/scout v10.25.0 recomputes total in SQL with the query callback applied. It re-fetches up to min(found, max_total_results=1000) ids from Typesense and counts the re-gated rows. So PHP's meta.total is the re-gated count, capped at 1000. It is not Typesense's found. A stale document still yields a short page, but the total is not inflated. Mirroring Scout exactly (D-16's own wording) means porting this two-step count.
  2. me/context does not return a channel name. The recorded PHP body has no channel. The channel lives at GET realtime/channels → {"data":{"collection":"collection:<id>"}}. That route is pending in the manifest and the Nuxt app calls it. API-01's wording maps onto both routes, so realtime/channels should be ported here.
  3. Error bodies for HttpException routes are Winter HTML pages, not JSON. This covers switch, household, invitations and accept. The existing accept fixture was recorded with the debug exception page (file paths and line numbers) and must be re-recorded under APP_DEBUG=false.
  4. Photo URLs don't match. PHP serves /storage/app/uploads/public/<partition>/<disk> while Go's attach default is /storage/uploads/<partition>/<disk>. P5 D-16 required the PHP shape, so this phase (the first to emit photo URLs) must fix the config/layout.

Several framework gaps in summercms.go are blocking:

  • beachcomber has no found count and no query_by_weights.
  • tide can't record or replay multipart request bodies and doesn't mask random disk names in url/thumb_url.
  • The validation layer lacks array, * wildcards, url, date, after_or_equal, before_or_equal, exists, regex, string, image, the size-type message variants (max.string/max.array/max.numeric/max.file) and Laravel's "stop after an implicit rule fails" semantics. The 422 bodies are recorded in Polish, e.g. Pole albums jest wymagane..
  • attach has no exported original-file URL helper.

Primary recommendation: Plan this as five plans:

  1. Framework gaps (summercms.go).
  2. Active context, collections and share.
  3. Household, invitations and notifications, with the mail job.
  4. Albums, search and lookups.
  5. Unit tests, the fuzz and the security leak test.

Recordings go into each feature plan. First, take the four decision corrections above to the user.

Architectural Responsibility Map

Capability Primary Tier Secondary Tier Rationale
Tenant resolution (active collection, token pin, switch, provisioning) API / Backend (classes/active_collection.go) Database (row locks on users + contexts) Server-resolved tenant. The client never selects one (C-01).
Access scoping (accessibleBy + token narrowing) API / Backend (one GORM scope) Database One chokepoint shared by album, collection, search, stats and sync.
Request validation and 422 envelopes API / Backend (request validator + phrasebook catalog) — Laravel semantics and messages are part of the contract.
Fill boundary and persistence API / Backend (AlbumWriteService/CollectionWriteService ports) Database P5 D-05..D-07 allow-lists. The fuzz target.
Search candidate retrieval External service (Typesense via beachcomber) — Candidate ids only, never authorization.
Search authorization and re-gate Database (SQL whereIn(ids) + scopes) API / Backend Pitfall 15.
Photo/cover storage and thumbs Storage (gocloud.dev/blob fileblob) CDN/Static (static handler at the public prefix) P5 D-14..D-19.
SSRF-guarded fetches (cover_urls, cover_url) API / Backend (fetchguard) — AllowHosts for Discogs covers, PublicOnly for manual URLs.
Invitation mail Background job (conga/River) Mail (postcard) Enqueued in the write transaction (D-13).
Realtime events (album broadcasts, user notifications) Background job (lighthouse broadcast job) Centrifugo After-commit publication (P11 D-06).
Rate limits API / Backend (surf inline throttle:N,M) — Per-route limits copied from routes.php.

<user_constraints>

User Constraints (from CONTEXT.md)

Locked Decisions

Route boundary

  • D-01: Collections, household and album routes listed in the domain are ported whole. Each handler is mounted once and reused on both groups wherever PHP mirrors it (the P6 D-15 genres pattern), with the exact ->where() constraints and inline throttles (throttle:10,1 on switch, share regenerate, invitation store and resend; throttle:20,1 on album photo upload; throttle:60,1 on albums/sync).
  • D-02: albums/sync, stats, value, missing and bulk are ported here: same controllers, no external service, and the Nuxt Albums UI calls them. bulk is where the single collection.bulk_updated broadcast under WithoutBroadcasting[Album] (P11 D-08) is exercised on a real endpoint.
  • D-03: Reservations move to Phase 13. In PHP they only exist as wishlist routes (AlbumReservationService, WishlistAlbumReservationController) and need the wishlist resolver. Fix the Phase 12 success criterion 2 and API-02 wording, and add them to Phase 13 / API-03.
  • D-04: The Discogs cover-price route moves to Phase 14 with INTG-01. The Discogs routes that match or use AI stay in Phase 14. Their manifest entries stay pending (P6 D-15: no 501 shells). Fix success criterion 2 and API-02 wording.
  • D-05: CoverImporter (the host-locked cover fetch behind cover_urls on album create and bulk) is ported now on the existing fetchguard AllowHosts GET helper (P6 D-11), so POST albums with cover_urls is parity-true in this phase. The pending todo fetchguard-guarded-http-client.md (POST, multipart, bearer) is not folded. It stays for Phase 14. If research finds that CoverImporter depends on DiscogsClient or needs more than a guarded GET, it reports that before planning.
  • D-06: All anonymous public-share routes move to Phase 13. Phase 12 ports only the owner-only JWT collection/share surface: show, update and regenerate, which emits the public_token. API-01 and success criterion 1 ("public token views") are reworded at plan time, and public/{token}* added to Phase 13 alongside public-wishlist/{token}*.
  • D-07: The manual cover URL branch of album photo upload ("exactly one of file or cover_url") uses fetchguard PublicOnly, as in PHP (P6 D-11/D-13).

Parity evidence

  • D-08: New recordings against the isolated PHP instance, using the Phase 2 tide capture rules (private 0600 vars, no live tokens in git):

    • nuxt-collections flow: create → switch → me/context → share show/update/regenerate → invite → accept as a second user → members → remove member.
    • nuxt-albums flow: create (with and without cover_urls) → update → rate / unrate → photo upload and delete → search → stats/value/missing/sync → bulk → delete.
    • Plus every distinct error status and body per route: 404 for foreign or missing ids, 403/404 non-owner, 410 on an unavailable invitation, 422 validation, and the personal-token 404 on owner-only household actions.

    Envelopes are reproduced per endpoint (P7 D-13).

  • D-09: Centrifugo publications are recorded from PHP during the album flows (create, update, delete, bulk) with the P11 D-10 capture tooling. The Go side, on the memory driver or a fake Centrifugo, is diffed against them with timestamp/actor normalised.

  • D-10: Token-group twins run the same handler as the JWT route and each replays its own recorded fixture. A route-table test asserts that every twin carries the right inv.scope:read|write and that JWT-only routes (switch, share, household, invitations) are absent from the token group.

  • D-11: Uploads get both recorded multipart replays (same file bytes, body diff including thumb_url) and Go tests. The Go tests assert the system_files row, the blob and the thumb exist; that the ImageContentGuard port rejects non-images; and that the upload-group MaxBytesReader cap holds (P7 D-04 pattern).

  • D-12: Search replays run with search disabled, as PHP was recorded (SCOUT_DRIVER=null). Typesense behaviour is covered by Go tests (D-16..D-18).

Invitations and mail

  • D-13: Invitation mail is sent by a River job enqueued in the same transaction as the invitation write, using the P11 job machinery. It goes out only if the invitation commits, which matches PHP's commit-before-side-effect ordering without the inline send.
    • The raw token (64 hex chars, sha256 at rest, 7-day expiry, as PHP) is present only in the job args and the mail. It is never logged and never stored in summer_jobs.metadata.
    • Locale and link follow PHP: inviter preferred_locale picks collection_invitation / collection_invitation-en and /zaproszenia/{token} / /en/invitations/{token} under app.url.
    • Asserted through postcard's memory driver.
  • D-14: Accepting an invitation ports NotificationService's write path now: the notifyInvitationAccepted row, plus any mail it sends, through the same job mechanism. The recorded accept flow then matches PHP DB state. Phase 13 adds the notifications list, read and prune API on top.
  • D-15: Only the existing-account path is ported. That means invite, resend and cancel for any email; accept by a logged-in user; and removing an editor with PHP's CollectionProvisioner context repair and organisation-membership rules. PendingInvitationRegistration rows are cleared on accept as PHP does, but consuming them on register belongs to Phase 13.

Search and the security test

  • D-16: Scout semantics are mirrored exactly. When Typesense is on and the query takes the Typesense path:

    1. Typesense returns one page of candidate ids.
    2. SQL applies whereIn(ids) + accessibleBy(user) + active collection (or token-frozen collection) + filters.
    3. meta.total/last_page come from Typesense, as Scout's paginator does. A stale document therefore yields a short page, never a leaked row. The inflated total is a documented, tested quirk, not a bug to fix.

    Rating, name and artist sorts force the SQL path, as PHP does. A Typesense error logs a warning and falls back to the SQL search.

  • D-17: The SQL text search uses ILIKE with % and _ escaped in user input, over PHP's column lists (TEXT_FIELDS_AUTHENTICATED: name, notes, artist_display, track_titles, label, catalog_number, plus artists.name). Escaping is a small, deliberate hardening: PHP passes %/_ through, so results differ only for queries containing them. Polish ordering relies on the pl-PL ICU database (P3 D-06).

  • D-18: The leak security test (success criterion 3) proves, with a scripted fake engine driver returning poisoned ids through the P11 search interface, that none of these return a row:

    1. A stale document whose album moved to a collection the caller cannot access.
    2. A mis-scoped document carrying the caller's collection_id for a foreign album.
    3. A soft-deleted album still in the index.
    4. An editor removed from a collection whose documents remain.
    5. A personal token frozen to its creation-time collection receiving hits from the user's other collections.

    No Typesense container is used.

Carried forward (locked earlier)

  • C-01: ActiveCollection (P3 D-03) is extended with switching and editor membership. me/context never exposes a raw collection_id, and the channel name is opaque (P11 D-11 naming rules).
  • C-02: Write paths go through the ported AlbumWriteService / CollectionWriteService fill boundaries (P5 D-05..D-07). The new request-DTO fuzz covers every write endpoint of this phase (unknown and server-owned keys never persisted). This inherits P5 criterion 3's HTTP half.
  • C-03: Response DTOs come from ported Serialize* functions, never from marshalled models. Use [] not null, Carbon +00:00 times, and the pagination envelope without links (P5 D-08, P6 D-17, DATA-10).
  • C-04: Album broadcasts and Typesense sync are already wired by Phase 11 (after-commit inline sync, broadcast job in the write transaction). This phase only calls them.

Claude's Discretion

  • Handler file layout under controllers/api and how the ~800-line AlbumApiController is split across Go files.
  • DTO structs and how the fuzz enumerates write endpoints (route-table driven preferred).
  • Job kind names and queue for invitation and notification mail.
  • Fake engine driver shape for tests.
  • Which errors per route need a recording and which a Go test with bodies from PHP source, where recording a case is impractical (e.g. a mid-transaction race). Recording is the default.
  • Plan count and split, subject to the plan-count checkpoint, "unit tests are the last plan" and the security-review agent (this phase touches authorization and public tokens).

Deferred Ideas (OUT OF SCOPE)

  • Reservations (wishlist/albums/{id}/reserve|reveal) → Phase 13.
  • Anonymous public-share routes public/{token}* → Phase 13.
  • Discogs cover-price, release match, apply-release, recognize, Discogs import → Phase 14.
  • Register-via-invitation and public invitation inspection → Phase 13 (API-07).
  • Notifications list, read and prune API → Phase 13 (API-04).

Reviewed Todos (not folded)

  • fetchguard-guarded-http-client.md: not folded. Discogs cover-price moved to Phase 14, and CoverImporter only needs the existing guarded GET.
  • backend-admin-api-tokens.md: unrelated to this API surface (admin auth). Stays deferred past v1.
  • redacting-slog-handler.md: Phase 14 (credentials). Worth noting that invitation tokens must not reach logs (D-13), but no handler port is needed here. </user_constraints>

<phase_requirements>

Phase Requirements

ID Description Research Support
API-01 Collections: CRUD, active-context switch (me/context returns an opaque channel name), editor invitations and acceptance, share link regeneration, public token views The collections, switch, share, household and invitations sections below. Correction: the channel name comes from GET realtime/channels, not me/context (see Finding 2). "Public token views" moves to Phase 13 (D-06).
API-02 Albums: CRUD, ratings, reservations, photo upload and manual cover URL, Discogs cover price, artists/genres/styles lookups, search that treats Typesense as a pre-filter re-gated in SQL The albums, uploads, search and lookups sections. Reservations move to Phase 13 (D-03) and Discogs cover price to Phase 14 (D-04). The search total semantics are corrected in Finding 1.
</phase_requirements>

Project Constraints (from CLAUDE.md)

  • Lean planning: few, large plans. Present the plan count with one-line scopes before writing any PLAN.md, then wait for confirmation.
  • The last plan of every phase brings full unit-test coverage. Earlier plans may carry smoke tests.
  • Use the standard library first. Add a dependency only when the research doc or a phase decision names it. go vet and go test ./... stay green at every commit.
  • Plugins are compiled in. No runtime plugin loading.
  • API parity is the acceptance test. Don't "improve" response shapes.
  • Two repos: fonoteka.go holds the app and summercms.go the framework. The framework never names the app in READMEs or docs. Planning docs stay in summercms.go/.planning.
  • Any change to a module's exported API, config keys, CLI commands or dependencies updates that module's README.md and the affected docs/ pages in the same change (go test ./cmd/summer -run TestDocsTree, summer docs:build --check). Every identifier named in docs must exist.
  • Commits: one logical change each, no co-author tags, planning docs and code in separate commits.
  • Core plugin contracts (user, blog, pages, payment) stay unbroken. This bears on any change to the Go golem15.user plugin, for example user groups (see Open Question 3).
  • The GORM, River, fetchguard, lighthouse, beachcomber, postcard and conga choices are already decided. Don't swap them.

Critical Findings (read before planning)

Finding 1: The search total is re-gated in SQL (D-16 point 3 does not hold)

Read this session from vendor/laravel/scout/src/Builder.php lines 533-556 [VERIFIED: /media/nvme/dev/golem15/fonoteka/vendor/laravel/scout/src/Builder.php:533-556]:

$totalCount = $engine->getTotalCount($results);
if (is_null($this->queryCallback)) {
    return $totalCount;
}
$ids = $engine->mapIdsFrom($results, $this->model->getScoutKeyName())->all();
if (count($ids) < $totalCount) {
    $ids = $engine->keys(tap(clone $this, function ($builder) use ($totalCount) {
        $builder->take(
            is_null($this->limit) ? $totalCount : min($this->limit, $totalCount)
        );
    }))->all();
}
return $this->model->queryScoutModelsByIds(
    $this, $ids
)->toBase()->getCountForPagination();

The installed version is "version": "v10.25.0" [VERIFIED: vendor/composer/installed.json]. AlbumSearchService::searchWithScout always sets ->query(...), so queryCallback is never null. The Typesense engine has private int $maxPerPage = 250; [VERIFIED: Engines/TypesenseEngine.php:41]. A limit of 250 or more switches to performPaginatedSearch, which loops up to $this->maxTotalResults. That value defaults to $config['max_total_results'] ?? 1000 [VERIFIED: EngineManager.php:165].

What PHP actually returns on the Typesense path:

  • Items: the Typesense page's ids, re-gated in SQL (queryScoutModelsByIds: the callback runs, then whereIntegerInRaw(id, ids)), filtered and ordered by Typesense position. Stale or foreign ids drop out, so the page can be short.
  • meta.total: when count(page ids) < found, a second Typesense search fetches min(found, 1000) ids (in pages of 250 once found is 250 or more). The total is then COUNT(*) of those ids with the same callback applied. When found fits in one page, it is the re-gated count of that page.
  • last_page is ceil(total/per_page).
  • Quirk worth mirroring and testing: the total is capped at 1000 even when more albums match.
  • Soft deletes: queryScoutModelsByIds calls withTrashed() only when the model uses Illuminate's SoftDeletes [VERIFIED: Searchable.php:322-339, 507-510]. Album uses \Winter\Storm\Database\Traits\SoftDelete [VERIFIED: models/Album.php:18], so Winter's soft-delete scope still applies and soft-deleted albums never come back.

Action: tell the user that D-16 step 3 should read "total = re-gated SQL count over min(found, 1000) Typesense ids", which is what "mirror Scout exactly" means here. The D-18 leak test then also asserts that meta.total excludes poisoned ids. The Go engine has to expose found (see Finding 5).

Finding 2: me/context carries no channel name; realtime/channels does

The recorded PHP me/context body [VERIFIED: parity/fixtures/routes/GET___fonoteka_api_v1_me_context_jwt.yaml]: {"can_use_ai":false,"can_manage_org":true,"has_organisation":true,"ai_org_lock":false,"ai_inherited":false,"is_collection_owner":true,"invitation_acceptance_required":false,"can_import_discogs":false,"market_currency_default":"EUR","market_currencies":["USD","GBP","EUR","CAD","AUD","JPY","CHF","MXN","BRL","NZD","SEK","ZAR"]}

The channel route [VERIFIED: routes.php:94-99] is Route::get('realtime/channels', ... 'collection' => 'collection:' . app(ActiveCollectionResolver::class)->resolve(...)->id. It is recorded as {"data":{"collection":"collection:{{id:style}}"}} and is status: pending in the manifest [VERIFIED: parity/manifest.yaml:93-97]. The Nuxt app calls /realtime/channels [VERIFIED: grep of vue-fonoteka-app/app].

me/context needs ports of AiGate::allows, OrgAccess::canManage, DiscogsGate::allows (→ DiscogsConfigResolver: site admin → env token, then user credential, then org credential), config('fonoteka.ai_org_lock'), and the OrgAiCredential/UserAiCredential existence checks [VERIFIED: MeContextController.php, AiGate.php, DiscogsConfigResolver.php].

Action: add GET realtime/channels (JWT only) to this phase as the "opaque channel name" half of API-01, and port the gates as read-only predicates. The channel string embeds the raw collection id, exactly as PHP does. Parity wins over P11 D-11's "no raw ids", and the PHP comment explains why that is safe: the authorizer re-validates membership on every subscribe.

Finding 3: HttpException error bodies are Winter HTML pages

ActiveCollectionResolver::switchTo, InvitationApiController::ownerCollection, CollectionMemberController, InvitationService::notFound() and accept() throw HttpException, which Winter renders as HTML. Recorded examples:

  • The members 404 is the Polish Nie znaleziono strony page with Content-Type: "text/html; charset=UTF-8".
  • The accept 410 was recorded as the debug exception page, with PHP file paths and line 140 [VERIFIED: fixtures/routes/DELETE___fonoteka_api_v1_household_members_{id}jwt.yaml, POST___fonoteka_api_v1_invitations{token}_accept_jwt.yaml].

php_parity.sh now exports export APP_DEBUG=false [VERIFIED: parity/php_parity.sh:49]. Re-record every HttpException case. The P7/P8 precedent embeds the production page bytes (controllers/api/winter_error_page.html, which contains the literal http://127.0.0.1:8423/modules/system/assets/css/styles.css) [VERIFIED: fonoteka.go/plugins/golem15/fonoteka/controllers/api/winter_error_page.html].

Two more envelopes still need a recording, because they are not the controller's own JSON shape:

  • household/invitations store validates with $request->validate([...]), which throws ValidationException.
  • normalizeEmail throws ValidationException::withMessages(['email' => 'The email must be a valid email address.']).

Finding 4: The photo URL prefix differs from PHP

  • PHP: getPublicPath() returns Config::get('cms.storage.uploads.path', '/storage/app/uploads') plus '/public' for public files [VERIFIED: modules/system/models/File.php:63-75], with 'path' => '/storage/app/uploads' [VERIFIED: config/cms.php:322].
  • Go: const defaultPublicPathPrefix = "/storage/uploads" [VERIFIED: modules/lagoon/attach/bucket.go], the app config has public_path_prefix: "/storage/uploads" and bucket_url: "file://./storage/app/uploads" [VERIFIED: fonoteka.go/config/storage.yaml], and BlobKey = partition + disk name with no public/ segment [VERIFIED: attach/thumb.go].

P5 D-16 required the exact PHP partition path public/xxx/yyy/zzz/<disk_name> and URL shape. Action: set bucket_url: file://./storage/app/uploads/public and public_path_prefix: /storage/app/uploads/public in the app config (or add a public/ sub-prefix in attach). Verify that the user avatar URLs (the same bucket) stay correct, and pin the shape with a test. There is also no exported URL helper for an original file (publicURL is unexported), so serializePhoto.url needs a small attach export, for example (*File).URL().

Finding 5: Framework gaps that block parity (summercms.go)

Gap Where Needed for
Engine.SearchIDs returns ids only, so there is no found. Query has no QueryByWeights beachcomber/searchable.go (SearchIDs(ctx, index string, q Query) ([]string, error)) [VERIFIED] Finding 1's total, and the weights PHP sends ('query_by_weights' => '10,10,5,5,3,1,3,3')
Requests can't carry multipart bodies (Body string only). Responses have BodyFile, requests don't tide/flow.go Request{Method, Path, Query, Headers, Body} [VERIFIED] D-11 recorded multipart replays
No normalizer for random disk_name in url/thumb_url tide/normalize.go masks only id/date/collection_key/client_id keys [VERIFIED] Upload replays and album bodies that carry photos
The album subtree's created_at/updated_at in publication goldens may not be masked (README says only data.timestamp, actor and ids) tide NormalizePublications [CITED: parity/README.md "Broadcast goldens"] Flipping the created/updated goldens from pending to assertion
No validation for array, * wildcards, url, date, after_or_equal, before_or_equal, exists, regex, string, image, sometimes, closure rules. The size-type messages (max.string/max.array/max.numeric/max.file) aren't split. The implicit-rule stop isn't implemented. The pl/en catalogs hold 13 keys lagoon/validate.go, phrasebook/lang/{pl,en}/validate.yaml [VERIFIED] Every 422 body. Winter's source catalog is modules/system/lang/pl/validation.php [VERIFIED]
The min failure message bug (pending todo lagoon-validate-min-message.md) lagoon/validate.go market_price_stored min:0, year between, `rating min:1
No exported original-file URL lagoon/attach serializePhoto.url
fetchguard's test seams (tlsConfig, skipReservedCheck) are unexported fetchguard/policy.go [VERIFIED] App tests for the cover fetch success path. Use an injectable fetch func in the app instead of changing fetchguard.

Standard Stack

Core (all already in the tree; no new framework choices)

Library / module Version Purpose Why Standard
net/http ServeMux via surf Go 1.27 Routing, Where constraints, inline throttle:N,M, body.limit Project rule; P6 [VERIFIED: surf/router.go]
GORM + lagoon gorm v1.31.2 Scopes, transactions (lagoon.Transaction), Fill, Paginate, OrderBy+Collate Decided [CITED: CLAUDE.md stack]
lagoon/attach framework system_files rows, blob keys, (*File).Thumb(ctx, bucket, w, h, mode) with crop P5 D-14..D-17 [VERIFIED: attach/thumb.go]
fetchguard framework Fetch(ctx, url, Policy{Mode: AllowHostsMode/PublicOnlyMode, AllowHosts, MaxBytes, Timeout}, cfg) with the invalid_url/scheme/unresolvable/private_ip/network_error/too_large reasons P6 D-11 [VERIFIED: fetchguard/fetch.go, policy.go]
beachcomber framework Engine.SearchIDs and the settings gate P11 D-19 [VERIFIED]
lighthouse framework WithoutBroadcasting[T](ctx, fn), (*Service).Emit(ctx, db, Broadcast{Channels, Event, Payload}) P11 D-06/D-08 [VERIFIED: lighthouse]
conga framework conga.Job[T](fn, OnQueue/MaxAttempts/Timeout), (*Manager).Enqueue(ctx, tx, args, EnqueueOpts) (joins the tx) P11 D-02 [VERIFIED: conga/conga.go]
postcard framework Mailer.Send(ctx, postcard.Message{Template, To, Vars}), MemoryDriver for tests P7 precedent [VERIFIED: user/controllers/api_controller.go:902]
phrasebook framework Validation catalogs (pl/en) P5 [VERIFIED]
tide framework Recording, replay and broadcast goldens P2/P11
crypto/rand, crypto/sha256, crypto/hmac, crypto/subtle stdlib Share token, invite token, fingerprint, constant-time compare stdlib

Supporting

Library Version Purpose When to Use
golang.org/x/image/webp bump golang.org/x/image from the transitive v0.0.0-20191009234506-e7c1f5e7dbb8 to v0.46.0 [VERIFIED: go list -m golang.org/x/image@latest → v0.46.0, 2026-09-08] Register the webp decoder so image.DecodeConfig (the getimagesize stand-in) and thumbs work for .webp Only if webp uploads must decode. PHP accepts mimes:jpg,jpeg,png,gif,webp. Name it in a phase decision. It is official Go, already in the module graph.

Alternatives Considered

Instead of Could Use Tradeoff
Extending beachcomber.Engine with a found count An optional interface (e.g. PagedSearcher, SearchPage(ctx, index, q) (ids []string, found int, err error)) type-asserted by the app No breaking change for other engine implementers (null, typesense, test fakes). Recommended.
A full Laravel validator port Hand-written per-endpoint checks Hand-written checks drift on message text and ordering. One validator with Laravel semantics is reused through Phase 13.
River job for invite mail Inline send after commit Locked by D-13 (job)

Installation: none for the core. The optional webp support is go get golang.org/x/image@v0.46.0 in summercms.go (where attach lives) after a phase decision names it.

Package Legitimacy Audit

Package Registry Age Downloads Source Repo Verdict Disposition
golang.org/x/image Go module proxy ~10 yrs n/a (Go sub-repository) go.googlesource.com/image OK (official Go project, already a transitive dependency) Approved, conditional on the webp decision

The gsd-tools package-legitimacy seam covers npm/pypi/crates only. The Go module was checked with go list -m against proxy.golang.org. Packages removed due to [SLOP] verdict: none. Packages flagged as suspicious [SUS]: none.

Architecture Patterns

System Architecture Diagram

HTTP request
   │
   ├─ /_fonoteka/api/v1/*  ──► jwt.auth ─► locale.from-principal ─► inv.must-change-password ─► [route throttle] ─┐
   └─ /api/v1/fonoteka/*   ──► inv_token ─► throttle:fonoteka-api-token ─► inv.scope:<read|write> (per route) ──┤
                                                                                                             ▼
                                                                   handler (one per route, shared by twins)
                                                                             │
                                  ┌──────────── request decode (JSON / multipart / query) ◄─ body.limit
                                  ▼
                       Laravel-semantics validator ──fail──► 422 {"error":"Validation failed","errors":{...}} (pl/en)
                                  │ ok
                                  ▼
            tenant resolution: token pin? ─yes─► pinned collection (404 if not accessible)
                                  │ no
                                  ▼
            pending-invitation guard (409) ─► users FOR UPDATE ─► context FOR UPDATE ─► stored|fallback|provision
                                  │
                                  ▼
            AccessibleBy(user, token) scope + collection_id re-gate ──none──► 404 JSON / Winter HTML page
                                  │
         ┌────────────────────────┼─────────────────────────────┬───────────────────────────┐
         ▼                        ▼                             ▼                           ▼
   write service (Fill       search: gate on? sort forces   uploads: content guard,     invitations: token gen,
   allow-list, tx,           SQL? ─► engine SearchPage ─►   blob write, system_files    sha256 at rest, tx ─►
   WithoutBroadcasting)      ids ─► SQL whereIn + scopes    row, Thumb(200,200,crop)    conga.Enqueue(mail job)
         │                    ─► recount (Scout)             manual URL: fetchguard       ─► commit ─► River ─►
         │                        │ error ─► SQL ILIKE path   PublicOnly; cover_urls:      postcard
         ▼                        ▼                           AllowHosts discogs.com
   commit ─► after-commit: beachcomber sync, lighthouse broadcast job (Emit single event) ─► Centrifugo
         │
         ▼
   Serialize* DTO (never the model) ─► JSON (Cache-Control: no-cache, private)

Recommended Project Structure (fonoteka.go)

plugins/golem15/fonoteka/
├── classes/
│   ├── active_collection.go      # Resolve (token pin, guard, provision), SwitchTo, DisplayActiveID
│   ├── access.go                 # AccessibleBy(user, token) = membership AND token narrowing
│   ├── collection_provisioner.go # CollectionProvisioner, OrgProvisioner
│   ├── fingerprint.go            # CollectionFingerprint (HMAC app.key)
│   ├── share_service.go          # CollectionShareService (owner surface only)
│   ├── invitation_service.go     # send/resend/cancel/accept/removeEditor
│   ├── notification_service.go   # write(), notifyInvitationAccepted, notifyAlbumAdded
│   ├── gates.go                  # OrgAccess, AiGate, DiscogsGate (read-only)
│   ├── serialize.go              # full SerializesFonoteka port (replace stubs)
│   ├── album_write_service.go    # create/apply (+styles, artists, tracklist_text, created_at, covers, completeness)
│   ├── artist_resolver.go        # full ArtistResolver + displayFor
│   ├── tracklist_text_parser.go, added_date_parser.go, duplicate_matcher.go, completeness.go
│   ├── album_search.go           # AlbumSearchService (Typesense path + SQL path)
│   ├── album_sync.go             # AlbumSyncService (delta, tombstones, cursor)
│   ├── image_guard.go, cover_importer.go, manual_cover_fetcher.go
├── controllers/api/
│   ├── collections.go, collection_share.go, me_context.go, realtime_channels.go
│   ├── invitations.go, members.go
│   ├── albums.go, albums_bulk.go, albums_photos.go, albums_stats.go (stats/value/missing), albums_sync.go, ratings.go
│   ├── lookups.go (artists, styles, genres POST)
│   └── errors.go (Winter HTML pages, notFound helpers)
├── jobs.go                       # pact.HasJobs: invitation mail job, notification mail job
├── views/mail/collection_invitation{,-en}.htm (postcard templates)
└── routes.go                     # line-by-line from routes.php

Pattern 1: Token-aware tenant resolution (ports ActiveCollectionResolver)

What: Resolve(ctx, tx, user, token *models.ApiToken):

  • Token present: require len(token.CollectionIDs)==1 && id>=1, then load it through AccessibleBy. Anything else is HttpException 404 Collection not found, a Winter HTML page.
  • No token: run guardPendingInvitation (409 invitation_acceptance_required), then in one transaction SELECT users ... FOR UPDATE, then the context FOR UPDATE, then stored-if-accessible, else the lowest-id accessible collection, else provision Moja kolekcja.

SwitchTo returns 404 for a token caller and adds kind='collection' [VERIFIED: ActiveCollectionResolver.php:20-123]. Note: PHP's fallback in resolve() uses Collection::accessibleBy($user)->orderBy('id')->first() with no kind filter (line 46), while the current Go firstAccessibleRealCollection adds kind = collection. Check which one parity needs (a wishlist with a lower id than the real collection would differ). This is an [ASSUMED] risk; record a case. Token source: cred, _ := bouncer.Credential(r.Context()); token, _ := cred.(*models.ApiToken) [VERIFIED: middleware/token_scope.go].

Pattern 2: One access chokepoint with token narrowing

// Ports Collection::scopeAccessibleBy (models/Collection.php:164-200).
func AccessibleBy(userID uint, token *models.ApiToken) func(*gorm.DB) *gorm.DB {
    return func(db *gorm.DB) *gorm.DB {
        db = AccessibleByMembership(userID)(db) // grouped owner OR editor
        if token != nil && len(token.CollectionIDs.Get()) > 0 {
            db = db.Where(`(golem15_fonoteka_collections.id IN ? OR (golem15_fonoteka_collections.owner_id = ? AND golem15_fonoteka_collections.kind = 'wishlist'))`,
                token.CollectionIDs.Get(), userID)
        }
        return db
    }
}
// Album::scopeAccessibleBy = whereHas('collection', accessibleBy): join or EXISTS on collections with deleted_at IS NULL.

ApiToken.CollectionIDs lagoon.Jsonable[[]uint] [VERIFIED: models/api_token.go:17].

Pattern 3: Scout-exact search (Finding 1)

// Typesense path (gate on, sort not in {rating,name,artist,price}, no rating filter):
ids, found, err := engine.SearchPage(ctx, index, beachcomber.Query{Q: q, QueryBy: ..., QueryByWeights: ..., FilterBy: "collection_id:=[<id>] && ...", SortBy: sortBy, Page: page, PerPage: perPage})
if err != nil { log.Warn(...); return sqlPath() }
items := hydrate(ids)   // SQL: scope(user,token) + collection_id + embeds + id IN ids; keep Typesense order
total := len(reGate(ids))
if len(ids) < found {
    all := fetchKeys(min(found, 1000)) // per_page=found if <250, else pages of 250 up to 1000
    total = countReGated(all)
}

The filter strings are verbatim from PHP: 'collection_id:=[' . $collectionId . ']', 'genre_id:=', 'style_ids:=', 'format:=', 'medium:=', 'artist_ids:=', 'year:[' . $decade . '..' . ($decade + 9) . ']', joined by ' && '. The sort is ($sort === 'year' ? 'year' : 'created_at') . ':' . $dir, prefixed with '_text_match:desc,' when q is non-empty [VERIFIED: AlbumSearchService.php:174-199].

Pattern 4: Single album broadcast (store/update) and bulk

  • Store/update: PHP wraps the whole write in Album::withoutBroadcasting, then publishes exactly one created.fonoteka.album / updated.fonoteka.album with getBroadcastPayload (fresh load with genre, styles, artists, photos) [VERIFIED: AlbumApiController.php:89-98, 489-506; models/Album.php:450-467]. In Go, use lighthouse.WithoutBroadcasting[models.Album] and one svc.Emit, with a payload built by the same albumPayload shape that realtime.go already defines, now on the full serializer.
  • Bulk: emit collection.bulk_updated {"reason":"bulk_create","count":N} only when created !== [] && $active->kind === 'collection' [VERIFIED: AlbumApiController.php:150-156].

Pattern 5: Invitation mail as a transactional job (D-13)

In one lagoon.Transaction:

  1. Generate the token with hex(rand 32).
  2. Upsert the invitation (token_hash = sha256 hex, expires_at = now+7d, accepted_at/by = nil, revoked_at = nil).
  3. Call conga.Manager.Enqueue(ctx, tx, InvitationMailArgs{InvitationID, Token}, …).

OrgProvisioner.provisionFor(actor) runs inside the same transaction [VERIFIED: InvitationService.php:48-72]. Enqueue (not Dispatch) avoids a summer_jobs row, so nothing can reach summer_jobs.metadata.

Anti-Patterns to Avoid

  • Serializing models: C-03, Pitfall 3. All responses go through Serialize*.
  • Treating a Typesense hit as authorization: Pitfall 15. Always re-gate.
  • A bulk tx.Where(...).Delete(&Album{}) for the collection cascade: this is what the current models.Collection.BeforeDelete does [VERIFIED: models/collection.go:47-51]. beachcomber.afterWrite skips zero-PK reflect values [VERIFIED: beachcomber/sync.go:51-101], so no search document is removed and no per-album deleted broadcast fires. PHP deletes album by album (foreach ($this->albums as $album) { $album->delete(); }, [VERIFIED: models/Collection.php:127-134]). Load the albums and delete the slice instead.
  • Holding a transaction open across cover downloads: up to max_covers (5) × a 10 s timeout. Commit the album first, then import covers and save cover_import_failures, then serialize and emit.
  • Trimming or nulling input strings: the Winter kernel has neither TrimStrings nor ConvertEmptyStringsToNull [VERIFIED: vendor/winter/storm/src/Foundation/Http/Kernel.php:25-41]. The CollectionShareController comment claiming "the HTTP kernel rewrites an empty string to null" does not hold for this stack. Persist strings as sent.

Don't Hand-Roll

Problem Don't Build Use Instead Why
SSRF-safe fetch A custom http.Client with an IP check fetchguard.Fetch (dial-time reserved-IP check, no redirects, byte cap) DNS-rebinding and redirect edge cases are already solved and tested
Thumbnails Your own resize or name scheme attach.(*File).Thumb(ctx, bucket, 200, 200, "crop") Winter filename parity (thumb_<id>_<w>_<h>_0_0_<mode>.<ext>)
Pagination envelope A map literal lagoon.Paginate / lagoon.PageMeta DATA-10 shape, no links
Broadcast after commit A direct publish lighthouse.Emit inside the tx, WithoutBroadcasting Rollback-safe (P11 D-06)
Job enqueue in a tx Your own outbox conga.Manager.Enqueue(ctx, tx, ...) InsertTx on the same *sql.Tx [VERIFIED]
Random tokens math/rand, uuid crypto/rand + rejection sampling (REJECT_AT = 248) Bias-free 62-symbol alphabet [VERIFIED: CollectionShareService.php:35-63]
Polish ordering Your own sort lagoon.OrderBy(..., lagoon.Collate("pl-x-icu")) P3 D-06
Validation messages Inline strings The phrasebook catalog ported from Winter validation.php (pl + en) Locale-dependent recorded bodies

Key insight: every "small" helper in this surface has a recorded byte contract (messages, URL shapes, ordering, key presence). Reuse the framework primitives and grow them where needed. Hand-rolled one-offs drift.

Runtime State Inventory

Not a rename or migration phase. Runtime state still bears on parity:

Category Items Found Action Required
Stored data golem15_fonoteka_user_collection_contexts written by resolve/switch/provision. river_job.args will hold the raw invite token until River prunes completed jobs Code. Decide the retention or encryption of the token in job args (Security, T-12-07)
Live service config The Typesense index golem15_fonoteka_albums keeps documents after a collection cascade unless per-row deletes fire Code fix in Collection.BeforeDelete
OS-registered state None. Verified: no cron/systemd entries in scope None
Secrets/env vars app.key (key: "" in fonoteka.go/config/app.yaml) feeds CollectionFingerprint. PHP HMACs the raw config string, e.g. base64:... Code. Tests use a test-only key. collection_key is masked by tide (if key == "collection_key") [VERIFIED: tide/normalize.go:57-66]
Build artifacts None None

Common Pitfalls

Pitfall 1: The token group's group-level inv.scope:read

What goes wrong: the Go token group is surf.Use("inv_token", "throttle:fonoteka-api-token", "inv.scope:read") [VERIFIED: routes.go:48]. Adding inv.scope:write per route makes write routes require both scopes. PHP routes carry exactly one scope each [VERIFIED: routes.php:454-517], so a write-only token passes in PHP and gets 403 in Go. How to avoid: move the scope to each route (g.Get("/genres", h, "inv.scope:read")) and extend the route-table test (D-10) to assert exactly one inv.scope:* per token route.

Pitfall 2: Where constrains only the immediately preceding route

What goes wrong: g.Where("id", "[0-9]+") attaches to the last registered route [VERIFIED: surf/router.go:267-284]. Routes like albums/{id}/photos/{fileId} need two Where calls right after that route. How to avoid: a route-table test asserting that every {id}, {fileId} and {collectionId} has a constraint.

Pitfall 3: Laravel validation semantics

What goes wrong:

  • Laravel stops validating an attribute after an implicit rule (required) fails. Non-implicit rules don't run on empty or absent values.
  • The size message depends on the value type (max.string / max.array / max.numeric / max.file).
  • Wildcard attributes appear as albums.0.name.
  • Messages come in the request locale.

The recorded example is {"error":"Validation failed","errors":{"albums":["Pole albums jest wymagane."]}} for {"albums":[]}: one message, even though min:1 would also fail [VERIFIED: fixtures/routes/POST___fonoteka_api_v1_albums_bulk_jwt.yaml]. after_or_equal and before_or_equal are absent from Winter's pl catalog [VERIFIED: grep of validation.php], so they fall back to English. How to avoid: one request-validator implementation with a table-driven test per recorded 422. Port the full pl/en catalogs.

Pitfall 4: PHP (int) casts and $request->boolean()

(int) $request->get('per_page', 20) takes the leading numeric prefix ("12abc" → 12, "abc" → 0 → clamped to 1). $request->boolean() is true for 1, "1", true, "true", "on", "yes". Port these as named helpers (phpInt, laravelBoolean) with tests.

Pitfall 5: Ordering differences (SQLite recording vs Postgres replay vs MariaDB production)

  • GET albums uses orderBy('name') with no collation and no tiebreaker [VERIFIED: AlbumApiController.php:52].
  • GET styles uses orderBy('name') [VERIFIED: StyleApiController.php:193].
  • GET collections sorts in PHP: ->get()->sortBy('name'), a stable byte comparison in PHP 8 [VERIFIED: CollectionApiController.php:44-50].
  • Stats genres use orderByDesc(album_count), with ties left unspecified.

How to avoid: use pl-x-icu where production MariaDB utf8mb4_polish_ci would apply, always add id ASC as a tiebreaker, sort collections in Go by name bytes with a stable sort over id-ordered rows, and keep recording data free of collation-sensitive collisions.

Pitfall 6: The album-created notification side effect

What goes wrong: PHP's Plugin::registerNotificationListeners hooks eloquent.created on Album and calls NotificationService::notifyAlbumAdded. That writes a golem15_fonoteka_notifications row for each collection member except the actor, and publishes notification:new and notification:count to user:{id} [VERIFIED: Plugin.php:87-111; NotificationService.php:52-80, 265-283]. Album create and bulk in a shared collection change DB state and Centrifugo traffic. How to avoid: port write() + notifyAlbumAdded now, next to notifyInvitationAccepted (D-14). The wishlist branch is Phase 13. Publish through lighthouse.Emit on the write tx, not synchronously as PHP does.

Pitfall 7: Invitation email validation differs

$request->validate(['email' => 'required|email']) uses Laravel's email rule, then normalizeEmail applies FILTER_VALIDATE_EMAIL after strtolower(trim()) and throws its own ValidationException [VERIFIED: InvitationApiController.php:63; InvitationService.php:249-257]. go-playground's email tag accepts a different set. Record the boundary cases and pin them in tests.

Pitfall 8: Raw token persistence and logging

The raw invite token sits in river_job.args (a JSON column) until River prunes the row. River's default completed-job retention is about 24 h [ASSUMED]. Anyone with DB read access (or a backup) during that window can accept the invitation as the invitee, provided they also have a session for the invitee's email (accept checks email === user.email). That limits the impact. How to avoid: encrypt the token in the args with the app key (lagoon.Encrypted exists [VERIFIED: lagoon/encrypted.go]) or delete or scrub the job row on completion. Never log the args.

Pitfall 9: Commit-ordering of side effects

PHP sends invite mail and runs notifyInvitationAccepted after the transaction returns [VERIFIED: InvitationService.php:74, 178-180]. In Go, enqueue inside the tx (D-13) so a rollback drops them. Album notifications in bulk happen inside PHP's transaction (synchronous publish before commit). Go publishing after commit is the intended improvement (P11 D-06) and invisible to recorded bodies.

Pitfall 10: cover_urls recordings need deterministic inputs

CoverImporter fetches from discogs.com / *.discogs.com over the network [VERIFIED: CoverImporter.php:206-226]. Record only the network-free failure cases (non-https, off-allow-list host), which yield cover_import_failures: [url]. Cover the success path with Go tests through an injectable fetch func.

Mapping note: PHP's cover_host_suffix .discogs.com maps to fetchguard AllowHosts: ["discogs.com"] (dotted-suffix match) [VERIFIED: fetchguard/fetch.go:131-143].

Pitfall 11: WebP decode and thumbs

Go's stdlib registers no webp decoder. image.DecodeConfig (the getimagesize stand-in) and Thumb fail for .webp unless golang.org/x/image/webp is imported, and imaging cannot encode webp. defaultEncodeImage falls back to JPEG bytes in a .webp-named thumb [VERIFIED: attach/thumb.go:92-104]. Decide this explicitly: register the decoder and accept JPEG-in-webp thumbs, or restrict.

Code Examples

Share token generation (port of CollectionShareService::generateToken)

// Source: CollectionShareService.php:35-63 (TOKEN_LENGTH = 16, ALPHABET, REJECT_AT = 248)
const shareAlphabet = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
func generateShareToken() (string, error) {
    out := make([]byte, 0, 16)
    buf := make([]byte, 16)
    for len(out) < 16 {
        if _, err := rand.Read(buf); err != nil { return "", err }
        for _, b := range buf {
            if len(out) == 16 { break }
            if b >= 248 { continue }
            out = append(out, shareAlphabet[int(b)%62])
        }
    }
    return string(out), nil
}

The share envelope stateFor is {enabled, name, token, path ("/k/"|"/w/"+token), generated_at} [VERIFIED: CollectionShareService.php:177-192]. Mutations lock the row (FOR UPDATE) and force-fill the share columns.

Collection fingerprint

// Source: CollectionFingerprint.php:301 — substr(hash_hmac('sha256', 'fonoteka:collection:' . $id, (string) config('app.key')), 0, 32)
func CollectionKey(appKey string, id uint) string {
    m := hmac.New(sha256.New, []byte(appKey)) // raw config string, including any "base64:" prefix
    m.Write([]byte("fonoteka:collection:" + strconv.FormatUint(uint64(id), 10)))
    return hex.EncodeToString(m.Sum(nil))[:32]
}

PHP-verbatim discrete values to reuse

  • Album search text fields [VERIFIED: AlbumSearchService.php:43-48]: 'query_by' => 'name,artist_display,style_names,genre_name,track_titles,notes,label,catalog_number', 'query_by_weights' => '10,10,5,5,3,1,3,3', 'sql_like' => ['name', 'notes', 'artist_display', 'track_titles', 'label', 'catalog_number'], 'sql_like_relations' => ['artists.name'].
  • private const SORTS = ['name', 'created_at', 'rating', 'year', 'artist', 'price']; and RATING_FILTERS = ['5', '4+', '3+', '2+', '1+', 'none'] [VERIFIED: AlbumApiController.php:41-43].
  • AlbumSyncService::DEFAULT_PER_PAGE = 100, MAX_PER_PAGE = 200. The cursor is base64_encode($syncVersion->toIso8601String() . '|' . $id) [VERIFIED: AlbumSyncService.php].
  • Completeness BASE_TAGS = ['cover', 'year', 'cat', 'label', 'genre', 'tracks', 'country'], plus CONDITION_TAG = 'cond', PRICE_TAG = 'price', RATING_TAG = 'rating' [VERIFIED: AlbumCompletenessService.php:15-37].
  • Invitation token bin2hex(random_bytes(32)), hash('sha256', $rawToken), now()->addDays(7). Mail paths '/en/invitations/' . $rawToken / '/zaproszenia/' . $rawToken. Views golem15.fonoteka::mail.collection_invitation-en / golem15.fonoteka::mail.collection_invitation [VERIFIED: InvitationService.php:59-65, 263-279].
  • Manual cover error messages per reason (invalid_url, scheme, unresolvable, private_ip, network_error, http_status, content_type, too_large, invalid_image) [VERIFIED: AlbumApiController.php:623-637].
  • MediumFamily map "LP": "vinyl", "2LP": "vinyl", "EP 7\"": "vinyl", "CD": "cd", "2CD": "cd", "MC": "cassette", "Box": "box" [VERIFIED: models/album_search.go].

State of the Art

Old Approach Current Approach When Changed Impact
Scout total = engine found Scout recounts in SQL when a query callback is set Scout 10.x (installed v10.25.0) Finding 1: the Go total must recount
Go stub serializers (P5) The full SerializesFonoteka port This phase The created/updated broadcast goldens flip to assertions

Deprecated/outdated: the "inflated total quirk" wording in D-16. The genres seed hook can be retired once POST genres is ported, since register/login were ported in P7 [CITED: parity/README.md].

Assumptions Log

# Claim Section Risk if Wrong
A1 River's default completed-job retention is about 24 h, so the raw token stays in river_job.args for about a day Pitfall 8 Longer or shorter exposure window. Mitigation advised either way.
A2 PHP resolve()'s fallback without a kind filter can pick a wishlist when its id is lower. The Go kind=collection fallback may diverge Pattern 1 Wrong active collection for users whose wishlist predates their collection. Record a case.
A3 Winter renders HttpException 410/409 under APP_DEBUG=false as the production error page with that status Finding 3 Wrong embedded bytes. Re-recording settles it.
A4 The ValidationException envelope from $request->validate() on these routes is Laravel's {"message","errors"}, or HTML Finding 3 422 body mismatch. Must record.
A5 The publication normalizer does not mask the album subtree's created_at/updated_at Finding 5 If it already does, there is less tide work
A6 isSiteAdmin can be false for everyone in Phase 12 because the Go user plugin has no users_groups Open Q3 can_manage_org/can_use_ai/can_import_discogs are wrong for site admins

Open Questions

None of these is decided yet. Each needs the user's confirmation at the plan-count checkpoint.

  1. D-16 correction (Finding 1).
    • What we know: installed Scout v10.25.0 recounts total in SQL over min(found, 1000) re-gated ids.
    • What's unclear: whether the user keeps D-16's wording ("total from Typesense") or accepts the correction.
    • Recommendation: implement the Scout recount ("mirror Scout exactly") and reword D-16 step 3 and success criterion 3. The D-18 leak test then also asserts that meta.total excludes poisoned ids.
  2. Port realtime/channels here? (Finding 2)
    • What we know: me/context has no channel. realtime/channels returns collection:<id>, is pending in the manifest, and the Nuxt app calls it.
    • Recommendation: yes. It goes in the collections plan on the JWT group only, and API-01's wording maps onto me/context plus realtime/channels.
  3. Site-admin predicate.
    • What we know: OrgAccess::isSiteAdmin reads $user->groups (code admin) through users_groups [VERIFIED: user/models/User.php:50]. The Go user plugin has neither the table nor the relation [VERIFIED: grep].
    • Options: (a) false for everyone in Phase 12, with a documented gap for Phase 13 credentials; (b) add the Winter user-groups schema to the Go user plugin, which is a core-plugin contract change and needs the user's sign-off.
    • Recommendation: (a).
  4. Photo URL prefix fix (Finding 4).
    • Recommendation: config only (bucket_url: file://./storage/app/uploads/public, public_path_prefix: /storage/app/uploads/public), plus an exported URL helper in attach and a regression check on avatar URLs.
  5. WebP (Pitfall 11).
    • Recommendation: register golang.org/x/image/webp (bump to v0.46.0 in summercms.go) and document that webp thumbs are JPEG bytes. The alternative is to accept that webp uploads fail the decode check, which would diverge from PHP.
  6. Raw token in job args (Pitfall 8).
    • Recommendation: store the token encrypted with the app key in the job args (lagoon.Encrypted), and never log the args. Scrubbing the River row after send is the fallback.

Environment Availability

Dependency Required By Available Version Fallback
Go toolchain everything ✓ go1.27.0 —
Docker (testcontainers Postgres) DB tests, parity replay ✓ 29.7.2 — (TestMain fails without it, by design)
PHP CLI (isolated parity instance) New recordings (D-08) ✓ 8.5.10 —
Node capture_clients.mjs (Nuxt capture) ✓ v22.23.2 Hand-scripted tide flows via parity:record --spec
Typesense server — not needed — Fake engine (D-18), D-12 records with search off
Centrifugo — not needed — tide fake recorder on 127.0.0.1:8424, memory driver

Missing dependencies with no fallback: none.

Validation Architecture

Test Framework

Property Value
Framework Go testing (+ testify assert/require, go1.27 fuzzing), testcontainers Postgres
Config file none. parity/parity_test.go TestMain starts Postgres.
Quick run command cd fonoteka.go && go test ./plugins/golem15/fonoteka/... -short -count=1
Full suite command cd fonoteka.go && go vet ./... && go test ./... -count=1, plus cd summercms.go && go test ./... -count=1 for the framework changes
Parity `cd fonoteka.go && go test ./parity -run 'TestParityCorpus

Phase Requirements → Test Map

Req ID Behavior Test Type Automated Command File Exists?
API-01 Collections CRUD/photos/image/switch/me-context/realtime-channels/share replays parity go test ./parity -run TestParityCorpus/.*collection ❌ fixtures for new cases (Wave 0)
API-01 nuxt-collections flow (invite → accept → members → remove) parity flow go test ./parity -run TestNuxtFlow/nuxt-collections ❌
API-01 Token pin, switch refusal for tokens, pending-invitation 409, provisioning integration go test ./plugins/golem15/fonoteka/classes -run TestActiveCollection partial (classes/postgres_test.go harness)
API-01 Invite mail is enqueued in the tx and absent on rollback; token never logged; locale/link per inviter integration go test ./plugins/golem15/fonoteka/... -run TestInvitationMail (postcard memory) ❌
API-02 Album CRUD/rating/photos/bulk/stats/value/missing/sync/search replays (search off) parity go test ./parity -run TestParityCorpus/.*albums ❌ new cases
API-02 Broadcast goldens created/updated/deleted/bulk asserted parity go test ./parity -run TestBroadcastGoldens ✅ (created/updated pending)
API-02 Leak test D-18 (5 cases, plus meta.total excluding poison) security go test ./plugins/golem15/fonoteka/... -run TestSearchLeak ❌
API-02 Upload guard, system_files row, blob, thumb, MaxBytes cap integration go test ./plugins/golem15/fonoteka/... -run TestAlbumPhotoUpload ❌
API-02 Manual cover URL reasons (private_ip, scheme, …) and cover_urls allow-list unit `go test ./plugins/golem15/fonoteka/classes -run 'TestManualCover TestCoverImporter'`
C-02 Request-DTO fuzz over every write endpoint (route-table enumerated) fuzz go test ./plugins/golem15/fonoteka -run=^$ -fuzz=FuzzWriteEndpoints -fuzztime=60s + seed corpus in go test ❌
D-10 Twin scopes are exactly one inv.scope; JWT-only routes are absent from the token group unit go test ./plugins/golem15/fonoteka -run TestRouteTable ✅ extend routes_group_test.go / routes_isolation_test.go
Framework beachcomber found/weights, tide multipart + url mask, validator semantics + catalogs unit cd summercms.go && go test ./modules/beachcomber/... ./modules/tide/... ./modules/lagoon/... ./modules/phrasebook/... ❌

Sampling Rate

  • Per task commit: the quick run command, plus go vet.
  • Per wave merge: the full suite in both repos, plus the parity run.
  • Phase gate: full suite green in both repos, parity corpus with the new routes flipped to ported and passing, check_corpus.go --require-recorded --check-secrets green, and TestDocsTree green for the framework README changes.

Wave 0 Gaps

  • Re-record the HttpException cases under APP_DEBUG=false (accept 410, switch 404, household/members 404, invitations 404, the token 404).
  • tide request body_file (multipart) + url/thumb_url disk-name normalizer + publication date masking (framework).
  • Seed hooks or flows for a second user (bob) and an outsider. id:outsider exists from P11; reuse it.
  • The fake beachcomber engine with scripted ids and found for D-18.

Security Domain

Applicable ASVS Categories

ASVS Category Applies Standard Control
V2 Authentication yes (inherited) jwt.auth, inv_token guard (P6/P8), unchanged
V3 Session Management no Stateless bearer
V4 Access Control yes AccessibleBy (membership + token narrowing), owner checks for destroy/image/share/household, token-group isolation, 404-never-403
V5 Input Validation yes The Laravel-semantics request validator, the Fill allow-lists (P5), the request-DTO fuzz
V6 Cryptography yes crypto/rand tokens, sha256 at rest, HMAC fingerprint, subtle.ConstantTimeCompare where comparing secrets
V7 Error/Logging yes No raw invite or share tokens in logs. Opaque 500s. Winter error pages without debug detail.
V11 Business Logic yes Inline throttles (switch, regenerate, invite, resend 10/min; photos 20/min; sync 60/min)
V12 Files/Resources yes ImageContentGuard (sniff + decode), body caps, fetchguard SSRF
V13 API yes Per-route inv.scope, JWT-only routes absent from the token group

Known Threat Patterns (candidate T-12-xx, each with a failing-when-broken test)

ID Pattern STRIDE Standard Mitigation
T-12-01 Cross-collection IDOR on albums/{id}, collections/{id}, photos/{fileId} Information disclosure / Tampering One AccessibleBy scope plus the collection_id re-gate. The file lookup is scoped to the owner's relation. Same 404 body for foreign and missing ids.
T-12-02 Search leak through stale or mis-scoped index docs, including meta.total Information disclosure SQL re-gate on items and the total (D-18 + Finding 1)
T-12-03 A personal token escaping its pinned collection (search, stats, value, sync, artists, genres) Elevation of privilege Token pin in Resolve + AccessibleBy narrowing. switch returns 404 for tokens.
T-12-04 A token reaching owner-only interactive surfaces (share, household, invitations, switch, sync, me/context) Elevation of privilege Absent from the token group (route-table test) plus the in-handler personalTokenIsActive second lock
T-12-05 An editor publishing or deleting someone else's collection Elevation of privilege Explicit owner_id == user checks for share, image, destroy and household
T-12-06 Invitation token reuse, expiry or email mismatch Spoofing FOR UPDATE lookup by hash, isPending, email equality → 410. Resend rotates the hash.
T-12-07 Raw invite token at rest (river_job.args) or in logs Information disclosure Encrypt the args or scrub after send. No logging of args. summer_jobs not used.
T-12-08 Share token predictability Spoofing crypto/rand + rejection sampling, unique-index loop, FOR UPDATE mutation
T-12-09 SSRF through cover_url / cover_urls Tampering / Information disclosure fetchguard PublicOnly (manual) / AllowHosts discogs.com (import), no redirects, byte cap, timeouts
T-12-10 Malicious upload (polyglot, oversized) Tampering / DoS body.limit upload bytes, a 10240 KB rule, magic-byte sniff + decode check, throttle 20/min (JWT)
T-12-11 Mass assignment (owner_id, collection_id, public_token, kind, market_price_source) Tampering Fill allow-lists + the request-DTO fuzz (C-02)
T-12-12 Context race causing double provisioning Tampering users row FOR UPDATE before the context row (PHP lock order) + the unique user_id context key
T-12-13 realtime/channels raw id treated as a tenant selector Elevation of privilege Never re-accepted on write paths. The collection authorizer re-validates membership on subscribe.

Recommended Plan Split (for the plan-count checkpoint)

Lean, five plans. Recordings go into each feature plan. The unit tests are last.

  1. 12-01 Framework gaps (summercms.go)
    • beachcomber: an optional found + QueryByWeights.
    • tide: multipart request bodies, a disk-name URL normalizer, album date masking in publication goldens.
    • A Laravel-semantics request validator with the full pl/en catalogs. Fold lagoon-validate-min-message.
    • attach: export the original-file URL.
    • The webp decoder decision.
    • READMEs and docs updated.
  2. 12-02 Active context, collections and share (fonoteka.go)
    • The token-aware Resolve/SwitchTo, provisioners, AccessibleBy, gates, CollectionFingerprint.
    • The full Serialize* port and the photo URL prefix fix.
    • Collections CRUD/photos/image/switch, me/context, realtime/channels, collection/share with show, update and regenerate.
    • Token-group restructure with per-route scopes, the Collection.BeforeDelete per-row fix, and recordings for these routes.
  3. 12-03 Household and invitations
    • InvitationService and OrgProvisioner.
    • NotificationService.write, notifyInvitationAccepted and notifyAlbumAdded.
    • The invitation mail job (encrypted args) and postcard templates.
    • Invitation, member and accept routes, the nuxt-collections flow recording and the HTML error pages.
  4. 12-04 Albums, search and lookups
    • The full AlbumWriteService (artists, styles, tracklist text, dates, duplicate matcher, completeness, CoverImporter).
    • ImageContentGuard and the manual cover fetcher.
    • Album CRUD, bulk, rating, photos, stats, value, missing and sync.
    • AlbumSearchService with the Scout-exact recount.
    • Artists, styles and genres POST.
    • The nuxt-albums flow, the upload replays, and flipping the broadcast goldens.
  5. 12-05 Unit tests and security
    • The request-DTO fuzz over route-table write endpoints.
    • The D-18 leak test with total assertions.
    • Route-table scope and constraint tests.
    • Validator, parser and completeness unit tests.
    • Coverage for everything above, plus the T-12-xx tests.

Alternative (4 plans): merge 12-03 into 12-02. That plan gets very large, so it is not recommended. Waves: 12-01 → 12-02 → 12-03 → 12-04 → 12-05. Run them sequentially: 12-03 and 12-04 both edit routes.go, serialize.go and the manifest.

Sources

Primary (HIGH confidence, read this session)

  • /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/routes.php (the full file).
  • PHP controllers: CollectionApiController, CollectionShareController, MeContextController, InvitationApiController, CollectionMemberController, AlbumApiController, AlbumSyncController, RatingApiController, ArtistApiController, StyleApiController, GenreApiController (store).
  • PHP classes: ActiveCollectionResolver, CollectionProvisioner, OrgProvisioner, OrgAccess, AiGate, DiscogsGate, DiscogsConfigResolver, CollectionFingerprint, CollectionShareService, InvitationService, NotificationService (write paths), AlbumWriteService, ArtistResolver, AlbumSearchService, AlbumSyncService, AlbumCompletenessService, AlbumDuplicateMatcher, TracklistTextParser, AddedDateParser, ImageContentGuard, ManualCoverUrlFetcher, discogs/CoverImporter, PolishOrder.
  • PHP traits and models: SerializesFonoteka, models Collection/Album/CollectionInvitation, Plugin.php notification listener.
  • PHP vendor: laravel/scout v10.25.0 Builder.php, Engines/TypesenseEngine.php, EngineManager.php, Searchable.php; winter/storm Http Kernel; modules/system/models/File.php; modules/system/lang/pl/validation.php; config/cms.php.
  • Go app: routes.go, classes/{active_collection,album_write_service,collection_write_service,artist_resolver,serialize}.go, models/{album,album_search,collection,collection_invitation,notification,api_token}.go, middleware/token_scope.go, classes/auth/token_guard.go, search.go, realtime.go, controllers/genre_controller.go, user avatar upload, config/{app,http,storage}.yaml.
  • Go framework: beachcomber (searchable.go, sync.go, typesense/engine.go), lighthouse (Emit, WithoutBroadcasting), conga (Dispatch, Enqueue, Job), fetchguard, lagoon (validate.go, paginate.go, attach/*), surf/router.go, tide (flow.go, normalize.go), phrasebook catalogs.
  • Parity corpus: manifest.yaml, README.md, php_parity.sh, capture-rules.yaml, fixtures (routes, broadcasts, nuxt).
  • Planning: 11-CONTEXT.md decisions, 05-CONTEXT D-14..D-19, PITFALLS.md Pitfall 15, pending todos.

Secondary

  • go list -m golang.org/x/image@latest → v0.46.0 (Go module proxy).

Tertiary (LOW)

  • River completed-job retention default (A1). Training knowledge, not verified this session.

Metadata

Confidence breakdown:

  • PHP contract and helpers: HIGH. Read line by line.
  • The Scout total semantics: HIGH. Read from the installed vendor source.
  • Go seams and gaps: HIGH. Read from the source.
  • Error-page and ValidationException envelopes: MEDIUM. Need re-recording.
  • Plan sizing: MEDIUM.

Research date: 2026-10-02 Valid until: 2026-11-01 (stable: PHP reference frozen, framework under active development)

Reference in New Issue View Git Blame Copy Permalink