diff --git a/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-03-SUMMARY.md b/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-03-SUMMARY.md new file mode 100644 index 0000000..01fad8c --- /dev/null +++ b/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-03-SUMMARY.md @@ -0,0 +1,331 @@ +--- +phase: 11-jobs-realtime-and-search-infrastructure +plan: 03 +subsystem: realtime +tags: [centrifugo, jwt, realtime, river, gorm-callbacks, broadcast, authorization] + +requires: + - phase: 11-jobs-realtime-and-search-infrastructure + provides: "11-01 conga (Manager.Register/Enqueue, conga.Job, StartWorker), lagoon.OnDatabase and lagoon.Transaction" + - phase: 07-user-plugin-and-authentication + provides: "jwt.auth guard and the frontend principal (bouncer.User)" + - phase: 06-http-routing-auth-groups-and-rate-limiting + provides: "surf Group/GroupRaw, named buckets (surf.BucketProvider), ClientIP" +provides: + - "modules/lighthouse: transport-neutral realtime Service (From, driver registry, null/log/memory drivers, Route/Surface/Mount)" + - "lighthouse authorizer Registry, channel rules (ParseChannel, ChannelID with PHP (int) semantics, FormatChannels) and client-id ctx" + - "Model broadcasts: Broadcastable contract, Bind[T], GORM callbacks enqueueing a summer.broadcast River job in the write tx inside a savepoint, WithoutBroadcasting[T], Service.Emit" + - "modules/lighthouse/centrifugo: net/http API client, five-generator HS256 TokenIssuer, TokenHandler, ProxyHandler, driver registered as centrifugo" + - "fonoteka.go: GET /api/realtime/token (jwt.auth, throttle:ws-api), POST /api/realtime/subscribe (raw, throttle:ws-api), ws-api bucket, collection and wishlist authorizers, Album binding, config/realtime.yaml" +affects: [11-06 parity goldens, 11-07 unit tests, 12 albums API (store/update/bulk must suppress + Emit), 13 notifications, 15 cutover env mapping] + +actuals: + tokens: 31479 + tasks: 3 + commits: 4 +plan_head_before: f1077382f537d8abefa85b4f2d7a744f09f74569 +plan_head_after: eab2b007f57d013bf2a9d1d241b4f1d9023aff8d +# fonoteka.go (separate repository) received 3 more commits: c59fd76, cb78a76, a223a72 + +tech-stack: + added: [] + patterns: + - "Drivers register from init (lighthouse.RegisterDriver, database/sql style) and are chosen by realtime.driver" + - "Drivers declare routes by surface; the app mounts them once with lighthouse.Mount and owns guard, group and bucket" + - "Model broadcasts are River jobs enqueued on the write's *sql.Tx from GORM callbacks, inside a savepoint" + - "Job args that carry byte-ordered JSON store it as a JSON string, because River's JSONB column reorders keys" + +key-files: + created: + - modules/lighthouse/lighthouse.go + - modules/lighthouse/drivers.go + - modules/lighthouse/route.go + - modules/lighthouse/users.go + - modules/lighthouse/channel.go + - modules/lighthouse/channel_test.go + - modules/lighthouse/registry.go + - modules/lighthouse/broadcast.go + - modules/lighthouse/suppress.go + - modules/lighthouse/job.go + - modules/lighthouse/README.md + - modules/lighthouse/centrifugo/config.go + - modules/lighthouse/centrifugo/client.go + - modules/lighthouse/centrifugo/token.go + - modules/lighthouse/centrifugo/handlers.go + - modules/lighthouse/centrifugo/driver.go + - ../fonoteka.go/config/realtime.yaml + - ../fonoteka.go/plugins/golem15/fonoteka/realtime.go + - ../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/ws/collection_authorizer.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/ws/wishlist_authorizer.go + modified: + - README.md + - .planning/PROJECT.md + - .planning/REQUIREMENTS.md + - ../fonoteka.go/README.md + - ../fonoteka.go/plugins/golem15/fonoteka/plugin.go + - ../fonoteka.go/plugins/golem15/fonoteka/routes.go + - ../fonoteka.go/plugins/golem15/fonoteka/plugin_boot_test.go + - ../fonoteka.go/plugins/golem15/fonoteka/routes_bucket_test.go + - ../fonoteka.go/plugins/golem15/fonoteka/go.mod + - ../fonoteka.go/plugins/golem15/fonoteka/go.sum + +key-decisions: + - "Broadcast job args store the payload as a JSON string (BroadcastArgs MarshalJSON/UnmarshalJSON): River keeps args in JSONB, which reorders object keys and would break payload key order" + - "No broadcast job is enqueued (and Emit is a no-op) when the driver publishes nothing: the null driver, or a driver whose Enabled() is false (Centrifugo without an API key)" + - "Broadcast callbacks are installed per *gorm.DB with Register-or-Replace, so a handle shared by several apps broadcasts through the most recently built service" + - "The subscribe proxy denies a missing channel and any user value that is not a JSON string or number (PHP would TypeError or cast true to 1); recorded as deliberate hardening" + - "Every test harness in fonoteka.go boots the centrifugo driver (bootConfig sets realtime.driver), so assembled route tables always carry the realtime routes" + +patterns-established: + - "lighthouse.Mount(r, driver, Surfaces{UserAuth, ServerToServer, Public, Middleware}) is the one call an app makes for realtime routes" + - "Bind[T] keeps payload code out of the models package (models stay a leaf)" + - "Bulk writes: WithoutBroadcasting[T] + lagoon.Transaction + Service.Emit" + +requirements-completed: [RT-01, RT-02, RT-03] + +coverage: + - id: D1 + description: "GET /api/realtime/token behind jwt.auth then throttle:ws-api returns {\"token\"} whose HS256 claims are exactly sub/exp(now+3600)/info{name}; info.name null for a nameless user; 503 WebSocket not configured with an empty secret; jwt.auth 401 without a bearer; concurrent requests race-clean" + requirement: RT-01 + verification: + - kind: integration + ref: "../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go#TestRealtimeTokenRoute" + status: pass + human_judgment: false + - id: D2 + description: "Centrifugo client POSTs /publish and /broadcast with Authorization: apikey, the {channel,data:{event,payload,timestamp +00:00}} body, [] for an empty payload, and sends nothing without an API key" + requirement: RT-01 + verification: + - kind: integration + ref: "../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go#TestRealtimeCentrifugoClientRequests" + status: pass + human_judgment: false + - id: D3 + description: "Subscribe proxy: constant-time secret (missing, wrong, empty-configured all deny), empty/0 user, missing channel, presence:presence:, four segments, unknown or case-mismatched namespace, owner/editor allow with exact {\"result\":{\"info\":[]}}, wishlist id under collection denies, presence:collection denies, presence allow/override merge with authorizer override winning, user as string or number, generic HTTP 200 deny with reasons only in logs and no secrets logged, editor removal flips to deny, concurrent subscribes race-clean, raw route with throttle:ws-api" + requirement: RT-02 + verification: + - kind: integration + ref: "../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go#TestRealtimeSubscribeProxy" + status: pass + - kind: integration + ref: "../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go#TestRealtimeSubscribeProxyWithoutSecretDenies" + status: pass + human_judgment: false + - id: D4 + description: "Channel rules: ChannelID matches PHP 8.5 (int) casts for 44 inputs (php -r table), ParseChannel presence/segment rules, FormatChannels lowercase + namespace, ClientID ctx, Registry validation, duplicate error and sorted Namespaces" + requirement: RT-02 + verification: + - kind: unit + ref: "modules/lighthouse/channel_test.go#TestChannelIDMatchesPHP" + status: pass + - kind: unit + ref: "modules/lighthouse/channel_test.go#TestParseChannel" + status: pass + - kind: unit + ref: "modules/lighthouse/channel_test.go#TestRegistry" + status: pass + human_judgment: false + - id: D5 + description: "Album create in lagoon.Transaction publishes once to acme:collection: (namespace applied) with created.fonoteka.album and payload {id, collection_id, action, actor{user_id,name}, timestamp, album}; rollback publishes nothing; id-only delete publishes deleted payload without album; wishlist album publishes nothing" + requirement: RT-03 + verification: + - kind: integration + ref: "../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go#TestAlbumBroadcastSmoke" + status: pass + human_judgment: false + - id: D6 + description: "Three album creates under WithoutBroadcasting[models.Album] plus one Emit publish exactly one collection.bulk_updated {\"reason\":\"bulk_create\",\"count\":3}; a write through a stale outer ctx still broadcasts" + requirement: RT-03 + verification: + - kind: integration + ref: "../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go#TestAlbumBroadcastSmoke" + status: pass + human_judgment: false + - id: D7 + description: "Centrifugo answering 500 leaves the album committed, logs realtime: broadcast failed without the API key, and the one-attempt job is not retried" + requirement: RT-03 + verification: + - kind: integration + ref: "../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go#TestAlbumBroadcastSmoke" + status: pass + human_judgment: false + - id: D8 + description: "Suppressing Album never suppresses another bound type (the plan assigns the acme-type assertion to plan 11-07)" + requirement: RT-03 + verification: [] + human_judgment: true + rationale: "Not asserted in this plan; plan 11-07 adds the acme-type unit test" + - id: D9 + description: "Delivery order across separate broadcast jobs is not guaranteed, as with PHP's queued BroadcastEventJob" + verification: [] + human_judgment: true + rationale: "Backstop truth by design (plan marks it verification: backstop); no test asserts ordering" + - id: D10 + description: "Live end to end: Centrifugo v6 with the production secret layout, fonoteka serve, the Nuxt app sees an album change" + verification: [] + human_judgment: true + rationale: "Manual smoke listed in the plan's verification, collected at /gsd-verify-work" + +duration: 12h 45m +completed: 2026-09-30 +status: complete +--- + +# Phase 11 Plan 03: lighthouse realtime and the Centrifugo driver Summary + +**The transport-neutral `lighthouse` package and its `centrifugo` driver. Płytarium's Nuxt app gets PHP-shaped connection tokens at `GET /api/realtime/token`. Every Centrifugo subscribe is re-authorized through a namespace registry with the ported collection and wishlist authorizers. Album writes publish to their collection channel only after commit, through a one-attempt River job enqueued in the write transaction.** + +## Performance + +- **Duration:** 12h 45m wall-clock (2026-09-29T21:51:18Z to 2026-09-30T10:36:24Z). The clock includes long idle gaps between tool calls; the active work was much shorter. +- **Started:** 2026-09-29T21:51:18Z +- **Completed:** 2026-09-30T10:36:24Z +- **Tasks:** 3 +- **Files modified:** 31 (19 in summercms.go, 12 in fonoteka.go) + +## Accomplishments + +- **lighthouse core (D-11, D-13).** + - `lighthouse.From` picks the driver by `realtime.driver`: `null` (the default), `log`, `memory`, or a registered driver. An unknown name fails boot and lists the registered drivers. + - `RegisterDriver` is an init-time registry. A duplicate name panics. + - `Mount` puts user and public routes in `Group` and server-to-server routes in `GroupRaw`, each with the surface middleware followed by the shared middleware. It refuses a UserAuth route when no guard is given (T-11-19). +- **Centrifugo driver (D-12).** + - The `net/http` client sends `Authorization: apikey`, times out after 5 s and treats any 2xx as success. Without an API key it sends no request and returns `ErrNotConfigured`. + - `TokenIssuer` has the five generators. They are HS256 with ordered claims, `ForIdentifier` encodes an empty `info` as `[]`, and all refuse an empty secret. + - `TokenHandler` returns the PHP 401 and 503 bodies. + - `ProxyHandler` checks the secret in constant time and always answers HTTP 200. A deny gets a generic body, with the reasons only in logs. An allow gets `info` (`[]` when empty) and, on presence channels, `allow`/`override` in PHP `array_merge` order. The body is capped at 64 KiB. +- **Channel rules (RT-02).** + - `ParseChannel` ports `parseChannel` exactly. + - `ChannelID` and `PHPInt` reproduce the PHP 8.5 `(int)` cast. A table test pins 44 inputs printed by `php -r`, including saturation and the rule that INF becomes 0. + - `FormatChannels` lowercases ASCII and applies the namespace, as `BroadcastEventJob` does. +- **Broadcasts (D-06..D-09, RT-03).** + - Models broadcast through the `Broadcastable` contract or `Bind[T]`. + - The GORM callbacks are installed through `lagoon.OnDatabase` and run in the production boot order. Each enqueues a `summer.broadcast` job on the write's `*sql.Tx` inside a savepoint. A failure is logged and never aborts the write. + - Deletes are snapshotted from a fresh read before the row goes. + - `WithoutBroadcasting[T]` suppresses one type for one ctx. `Service.Emit` enqueues one summary event. + - The job namespaces the channels, then publishes to one channel or broadcasts to several. A failure is a Warn log, and the job is never retried. +- **fonoteka.go wiring (D-14, D-16).** + - `config/realtime.yaml` holds the PHP defaults and the README maps the env names. + - The `ws-api` bucket allows 120/min, keyed by user id, else by client IP. + - A single `lighthouse.Mount` call puts `jwt.auth` before `throttle:ws-api` (user decision 5). + - The `collection` and `wishlist` authorizers are ported. + - The Album binding applies the PHP channel and payload overrides. + - PROJECT.md records that websockets is no longer an app plugin. REQUIREMENTS.md notes that RT-01 uses a hand-rolled client. + +## Task Commits + +summercms.go: +1. **Task 1: token route through the neutral package** - `cada7a4` (feat) +2. **Task 2: subscribe proxy and registry** - `79fd705` (feat) +3. **Task 3: transactional album broadcasts** - `211c413` (feat); planning docs `eab2b00` (docs) + +fonoteka.go: +1. **Task 1** - `c59fd76` (feat: realtime wiring, ws-api bucket, Mount, config, token smoke test) +2. **Task 2** - `cb78a76` (feat: collection and wishlist authorizers, proxy smoke test) +3. **Task 3** - `a223a72` (feat: Album binding, broadcast smoke test) + +## Files Created/Modified + +- `modules/lighthouse/lighthouse.go`: `Service`, `From`, config reading, job registration, callback installation, `DurationSetting` +- `modules/lighthouse/drivers.go`: `Publisher`, `Driver`, `DriverFactory`, `RegisterDriver`, the null/log drivers, `MemoryDriver`, `Publication` +- `modules/lighthouse/route.go`: `Route`, `Surface`, `Surfaces`, `Mount` +- `modules/lighthouse/users.go`: `User`, `UserLookup`, `Actor`, `SystemActor`, `Service.Actor` +- `modules/lighthouse/channel.go`, `registry.go`: the channel rules, `PHPInt`, client-id ctx, `Registry`, `Result`, `Allowed`, `Denied`, `Authorizer` +- `modules/lighthouse/broadcast.go`, `suppress.go`, `job.go`: the broadcast contract, `Bind`, the callbacks, `WithoutBroadcasting`, `Emit`, `BroadcastArgs` and the worker +- `modules/lighthouse/channel_test.go`: the PHP `(int)` table, parse/format, registry +- `modules/lighthouse/centrifugo/*.go`: `Config`/`LoadConfig`, `Client`, `TokenIssuer`, `TokenHandler`, `ProxyHandler`, `Driver` +- `modules/lighthouse/README.md` and the root `README.md` row +- fonoteka.go: `realtime.go`, `classes/ws/*`, `plugin.go` (field, Boot call, `ws-api`), `routes.go` (Mount), `config/realtime.yaml`, `README.md` (Configuration), `realtime_smoke_test.go`, test harness tweaks, go.mod/go.sum tidy + +## Decisions Made + +See `key-decisions` in the frontmatter. + +## TDD Gate Compliance + +Tasks 2 and 3 are `tdd="true"`. For each, the tests were written first against stubs with the final signatures. The RED runs were recorded with `go test -json` converted to TAP, and `gsd-tools check tdd-red-evidence` returned `RED_EVIDENCE_OK` (`target_test_failed`) for: + +- Task 2: `TestRealtimeSubscribeProxy`, where the stub proxy answered 418 and the allow body was expected; and `TestChannelIDMatchesPHP` with the other lighthouse unit tests, where the stub `ChannelID` returned -1. +- Task 3: `TestAlbumBroadcastSmoke`, where the stubs installed no callbacks and `Emit` enqueued nothing. The rollback and wishlist subtests passed at RED, as expected, because they assert that nothing is published. + +**Gate violation, flagged:** there are no separate `test(11-03)` RED commits. The project CLAUDE.md requires `go vet` and `go test ./...` to be green at every commit, so the tests and implementation landed together in the `feat` commits. Plans 11-01 and 11-02 made the same call. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] JSONB reordered broadcast payload keys** +- **Found during:** Task 3 (TestAlbumBroadcastSmoke GREEN run) +- **Issue:** River stores job args in a JSONB column, and JSONB sorts object keys. So `{"reason":"bulk_create","count":3}` arrived at Centrifugo as `{"count":3,"reason":"bulk_create"}`, which breaks the PHP byte order that the 11-06 goldens will compare. +- **Fix:** `BroadcastArgs` keeps `Payload json.RawMessage` in Go, but its `MarshalJSON`/`UnmarshalJSON` carry the payload as a JSON string, whose content JSONB leaves alone. +- **Files modified:** modules/lighthouse/job.go +- **Verification:** the bulk subtest asserts the exact bytes `{"reason":"bulk_create","count":3}`. +- **Committed in:** 211c413 + +**2. [Rule 2 - Correctness/operability] No broadcast jobs when the driver publishes nothing** +- **Found during:** Task 3 +- **Issue:** Every album write would enqueue a River job in two cases where the job would publish nothing: an install without a Centrifugo API key, and every fonoteka test harness (the centrifugo driver without a key). PHP queues the job and then logs a failed publish. +- **Fix:** The callbacks and `Emit` skip the enqueue for the null driver, and for a driver whose optional `Enabled()` reports false. `centrifugo.Driver.Enabled` reports whether an API key is set. Nothing observable is published in either case. The only difference is that no "broadcast failed" warning is logged. +- **Files modified:** modules/lighthouse/broadcast.go, suppress.go, centrifugo/driver.go +- **Committed in:** 211c413 + +**3. [Rule 3 - Test harness] fonoteka harness boots the centrifugo driver** +- **Found during:** Task 1 +- **Issue:** `bootConfig` did not set `realtime.driver`, so every assembled route table used the null driver and carried no realtime routes. `TestAllRouteGroupsBoot` and `TestFullRouteTableAuthGroupMutualExclusivity` would then prove nothing about the Mount call. +- **Fix:** `bootConfig` sets `realtime.driver: centrifugo`, following D-13 ("Płytarium and its tests always run the Centrifugo driver"). `TestAllRouteGroupsBoot` now also expects the `ws-api` bucket. Both test files are outside the plan's files list. +- **Files modified:** ../fonoteka.go/plugins/golem15/fonoteka/plugin_boot_test.go, routes_bucket_test.go +- **Committed in:** c59fd76 + +**4. [Hardening, documented] Proxy input handling beyond PHP** +- A missing or empty channel is denied. Centrifugo always sends one; PHP would return a 500 TypeError. +- A `user` that is neither a JSON string nor a number (for example `true`) counts as empty and is denied. PHP would cast `true` to 1. +- A malformed JSON body is denied with the generic body. +- Registering a namespace twice is a boot error. PHP lets the last registration win. This was already specified in the plan. +- **Committed in:** 79fd705 + +**5. [Additions] Extra exported surface, all documented in the README and checked with go doc** +- `Service.Logger`, `Namespace`, `Queue` and `Timeout` +- `DurationSetting` +- `DefaultDriver`, `DefaultQueue`, `DefaultTimeout` and `DefaultTTL` +- `PHPInt` and `WithClientID` +- `NewRegistry` +- the `Callback*` name constants +- centrifugo: `LoadConfig`, `NewDriver`, `NewTokenIssuer`, `DriverName`, `Driver.Enabled`, `Driver.Client`, `Driver.Issuer`, `Driver.Config`, `Config.TrustedProxies` (the client IP in secret-failure logs) and the `Default*` constants + +**6. [Additions] Extra tests** +- `TestRealtimeCentrifugoClientRequests`: the httptest check of the client bytes, which the plan placed in the Task 1 smoke test. +- `TestRealtimeSubscribeProxyWithoutSecretDenies`. +- lighthouse unit tests: `TestParseChannel`, `TestFormatChannels`, `TestClientID` and `TestRegistry`. + +--- + +**Total deviations:** 3 auto-fixed (1 bug, 1 correctness/operability, 1 test harness), plus 3 documented notes. +**Impact on plan:** Fix 1 is required for payload parity with PHP. Fix 2 keeps unconfigured installs and tests off River. No scope creep. + +## Issues Encountered + +- The shell aliases `rm` to interactive mode, so one command stalled on a prompt. It was rerun with `/bin/rm -f`. Plan 11-01 hit the same thing. +- The token signing error path answers 500 `{"error":"Internal server error"}`. It is unreachable in practice, because the handler checks `Configured()` first. + +## Flagged assumptions (carried from the plan) + +- Batch updates through `Model(&T{}).Where(...)` have a zero primary key and are not broadcast (Pitfall 8). A soft delete counts as a delete, and a restore is not broadcast. +- The automatic Album `created`/`updated` payload is built inside the write transaction. `SaveAlbum` syncs the artist pivot after `tx.Save`, so the payload can carry stale artists. Phase 12's store/update/bulk controllers must use `WithoutBroadcasting[models.Album]` plus `Emit`, as PHP does. `classes.SerializeAlbum` is still the minimal Phase 5 shape until Phase 12. +- Several apps that share one `*gorm.DB` broadcast through the service built last, because the callbacks are replaced. Production has one app per handle. + +## User Setup Required + +None for development. For a deployment, set the `SUMMER_REALTIME__CENTRIFUGO__*` variables listed in the fonoteka.go README (the PHP values carry over unchanged). Point Centrifugo's subscribe proxy at `POST /api/realtime/subscribe` with the same `X-Centrifugo-Secret`. + +## Next Phase Readiness + +- Plan 11-06 (tide goldens) can diff the `deleted` and `collection.bulk_updated` bytes. It can drive the Go side with the memory driver or the fake-Centrifugo pattern from `realtime_smoke_test.go`. +- Plan 11-07 should add: + - the acme-type suppression test (suppressing one type never suppresses another) + - unit tests for `Mount` validation errors, the log driver, `Presence`/`Unsubscribe`, all five token generators with an injected clock, the override merge with extra keys, the method-based `Broadcastable` path (alias, TTL, filter, payloader), and `Emit` without a transaction +- Phase 12 must route album store/update/bulk through `WithoutBroadcasting` plus `Emit` (see above), and must migrate `gdb.Transaction` call sites to `lagoon.Transaction`. + +--- +*Phase: 11-jobs-realtime-and-search-infrastructure* +*Completed: 2026-09-30*