27 KiB
27 KiB
phase, verified, status, score, covered_files, covered_digest, covered_files_note, mvp_mode_note, behavior_unverified, overrides_applied, gaps, behavior_unverified_items, human_verification
| phase | verified | status | score | covered_files | covered_digest | covered_files_note | mvp_mode_note | behavior_unverified | overrides_applied | gaps | behavior_unverified_items | human_verification | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 11-jobs-realtime-and-search-infrastructure | 2026-09-30T13:28:50Z | gaps_found | 4/5 roadmap success criteria verified (SC-5 partial); plan truths 66/70 verified, 1 failed, 3 backstop (insufficient_spec, routed to human) |
|
v2:sha256:5e6b25eb51444acda651d1e01476692917ab8ecbf999cd039c3a54ab2f8457f6 | fonoteka.go files are outside the project root and cannot be fingerprinted. They were checked at fonoteka.go HEAD c222e03 with a clean working tree and are listed in the report body. modules/cabana/crud.go is not a Phase 11 change; it is included because the gap below names it. | ROADMAP marks Phase 11 mode: mvp, but the goal is not a User Story and no 11-*-PLAN.md carries one. Following the Phase 1/3/5/8/9/10 precedent, the five ROADMAP success criteria are the contract, User Flow Coverage is derived from them, and plan must_haves are supporting evidence. | 0 | 0 |
|
|
Phase 11: Jobs, realtime and search infrastructure — Verification Report
Phase Goal: River jobs run on the correct dual-driver split, Centrifugo publishing and channel authorization match the existing server, and Typesense sync stays a re-gated pre-filter — all brought up before the API phases that depend on them.
Verified: 2026-09-30T13:28:50Z (summercms.go HEAD 61da4d1, fonoteka.go HEAD c222e03, both trees clean of code changes)
Status: gaps_found
Re-verification: No — initial verification
MVP note: ROADMAP marks this phase mode: mvp, but the goal is not a User Story and no plan carries one. Following the precedent of Phases 1, 3, 5, 8, 9 and 10, the five ROADMAP success criteria are the contract.
User Flow Coverage
Derived story: As a host application developer, I want to dispatch River jobs, schedule commands, publish realtime model events through the existing Centrifugo and keep Album search documents in Typesense, so that the API phases can rely on jobs, realtime and search without the PHP backend.
| Step | Expected | Evidence | Status |
|---|---|---|---|
| Dispatch a job and see its outcome | summer_jobs row + River job in one tx; complete/fail/skip queryable | conga.Dispatch (InsertTx on the gorm sql.Tx), Manager.Get; TestDispatchTransactional, TestOutcome pass | ✓ |
| Worker picks jobs up promptly | LISTEN wake-up, not poll latency | TestListenPickupLatency re-run: 3ms pickup with a 30s poll; poll-only control not picked up in 2s | ✓ |
Run queue:work / schedule:run |
Foreground worker; scheduler runs registered commands | conga.RuntimeCommands wired into generated main and fonoteka main.go; TestQueueWork, TestScheduleRunForeground pass | ✓ |
| Nuxt gets a token and subscribes | Same HS256 claims; proxy re-authorizes | TokenIssuer matches PHP JwtTokenGenerator claim-for-claim; ProxyHandler looks up registry per request; parity routes replay green | ✓ (live Centrifugo: human) |
| Album edit reaches Typesense correctly | Committed document, collection-scoped, gated | Scoping and gate correct; admin edit path indexes pre-commit, stale artist_ids | ✗ (CR-01) |
Goal Achievement
Roadmap Success Criteria
| # | Success criterion | Status | Evidence |
|---|---|---|---|
| 1 | River on shared *sql.DB (riverdatabasesql, transactional enqueue) + LISTEN on a small separate pgx pool (NewWithPgxListener, one client), timed pickup test; outcomes queryable via job manager | ✓ VERIFIED | modules/conga/worker.go:135-146 builds riverdatabasesql.NewWithPgxListener(sqlDB, listener) with the published *sql.DB; listenerPool sets MaxConns 1/MinConns 0. One worker client per Manager (second start errors); inserts reuse it when running. conga.go:158-198 InsertTx on tx.Statement.ConnPool.(*sql.Tx). runAttempt sets ERROR only on final attempt/panic; skip = CompleteJob {"skipped":true}; Manager.Get exposes the row. Verifier re-ran TestListenPickupLatency: listen 3.0ms, poll-only control not picked up in 2s. |
| 2 | summer queue:work runs the worker; summer schedule:run runs recurring commands |
✓ VERIFIED (WR-02 warning) | conga/commands.go queue:work, schedule:run (daemon and --once), queue:clear; appended by internal/build/build.go:115 and in ../fonoteka.go/main.go:39; bonfire.NewCatalog published (build.go:124). Scheduled worker calls commands through the catalog. Tests: TestQueueWork, TestScheduleRunsCommand, TestScheduleRunForeground, TestScheduleRunOnce, TestScheduleUniqueByPeriod pass in the gate. Caveat WR-02: a scheduled command that opens its own DB via the withDB pattern fails inside a running worker (backpack: duplicate provider for *sql.DB). |
| 3 | Token + subscription JWTs with the same secret/claims/channel names at GET /api/realtime/token; publishes to existing Centrifugo | ✓ VERIFIED (live server: human) | centrifugo/token.go ports all five PHP generators with identical claims (sub string id, exp, info.name only; anonymous 300s; identifier info []). TokenHandler 401/503/200 bodies match PHP routes.php. Client posts /publish and /broadcast with Authorization: apikey, 5s timeout. fonoteka mounts with jwt.auth + throttle:ws-api (routes.go:86). Album channels collection:<id> only for kind=collection. TestTokenClaims, TestTokenHandler, TestClientRequests, parity token/subscribe routes and TestBroadcastGoldens (deleted + bulk vs PHP goldens) pass; created/updated goldens pending Phase 12 by design. |
| 4 | Namespace authorizer registry re-validates every subscribe; broadcastable model with bulk suppression emits exactly one summary event | ✓ VERIFIED | ProxyHandler calls svc.Registry().Get(namespace) and Authorize per request, no cache; constant-time secret compare; generic deny body. fonoteka registers collection and wishlist authorizers. WithoutBroadcasting[T] + Service.Emit; TestBulkEmitsOnce asserts exactly one publication (waitPublications fails on extras) and none on rollback. Broadcast callbacks now register .Before("gorm:commit_or_rollback_transaction") (deferred item 1 fixed). TestProxy, TestWsAuthorizer, TestSuppression, TestBroadcastTx pass. |
| 5 | Typesense sync scoped by collection_id behind a settings kill-switch, degrades gracefully without DB/config | ⚠ PARTIAL | The literal clauses hold: Album.ToSearchableArray refuses collection_id 0; settingsGate reads search_use_typesense per sync and treats read errors as off; syncOne returns early with no engine/api_key or no published DB. But the sync contract underneath it (D-20, after-commit, built from the committed row) fails on the framework admin write path — see Gap 1. |
Plan must-have truths (supporting evidence)
70 truths across 7 plans. 66 verified by code reading plus the named tests the gate runs (all pass, none skipped except the two declared Phase 12 goldens); 3 are verification: backstop (11-02 multi-process leader election, 11-03 delivery order, 11-07 real clients) and route to human verification as insufficient_spec; 1 failed:
| Plan | Truth | Status | Evidence |
|---|---|---|---|
| 11-05 | D-20: sync after commit; rolled-back write sends nothing | ✗ FAILED | Holds for lagoon.Transaction and implicit single-statement transactions, fails for plain gorm.Transaction, which is how cabana admin CRUD and SaveAlbum write. Passing tests assert the pre-commit sync. |
| 11-01 | "outside any lagoon-managed transaction the callback runs immediately" | ✓ VERIFIED as written | This is the design that produces Gap 1. The two plan truths contradict each other on the framework's main write path. |
Required Artifacts
| Artifact | Status | Details |
|---|---|---|
modules/lagoon/queue_migrations.go, ondatabase.go, transaction.go |
✓ VERIFIED | Migrations wired from migrations.go; OnDatabase drained by Publish; Transaction/AfterCommit present (see gap for semantics) |
modules/conga/* (conga, worker, client, commands, schedule, scheduler, job, record) |
✓ VERIFIED | Substantive, wired into serve (surf/serve.go:57), generated main and fonoteka main |
modules/pact/capabilities.go, modules/bonfire/call.go |
✓ VERIFIED | HasSchedule/Daily/DailyAt/Every; Catalog/Call used by scheduler |
modules/lighthouse/*, modules/lighthouse/centrifugo/* |
✓ VERIFIED | Registry, Mount, Broadcastable, suppression, token issuer, proxy, HTTP client, health command |
modules/flare/* |
✓ VERIFIED | RFC 8291 encryption, VAPID, commands; registered by fonoteka Commands() |
modules/beachcomber/*, beachcomber/typesense/* |
✓ VERIFIED (exists, wired) / ✗ semantics | Wired through OnDatabase and fonoteka wireSearch; AfterCommit semantics defect (Gap 1) |
modules/tide/centrifugo.go, centrifugo_golden.go, cmd/summer/parity.go |
✓ VERIFIED | Loopback recorder, goldens, parity:broadcasts |
../fonoteka.go/plugins/golem15/fonoteka/{realtime,search,schedule,routes}.go, classes/ws/*, models/album_search.go |
✓ VERIFIED | Authorizers, Album binding, settings gate, daily prune entry, route mount |
../fonoteka.go/config/{queue,realtime,search,push}.yaml |
✓ VERIFIED | PHP defaults; push disabled |
../fonoteka.go/parity/fixtures/broadcasts/{deleted,bulk,created,updated}.yaml |
✓ VERIFIED | created/updated pending Phase 12 and never counted as passing |
scripts/check-phase11.sh |
✓ VERIFIED | Re-run by verifier: phase11 all passed (3m27s) |
Key Link Verification
| From | To | Via | Status |
|---|---|---|---|
| lagoon/migrations.go | queue_migrations.go | QueueMigrations( | ✓ WIRED |
| conga/conga.go | River | InsertTx on *sql.Tx | ✓ WIRED |
| surf/serve.go | conga/worker.go | conga.StartServeWorker | ✓ WIRED |
| internal/build/build.go | conga/commands.go, bonfire | conga.RuntimeCommands, bonfire.NewCatalog | ✓ WIRED |
| conga/worker.go | scheduler.go | cfg.PeriodicJobs | ✓ WIRED |
| fonoteka routes.go | lighthouse/route.go | lighthouse.Mount | ✓ WIRED |
| lighthouse/broadcast.go | conga | Enqueue on write tx in savepoint | ✓ WIRED |
| centrifugo/handlers.go | registry.go | Registry().Get per subscribe | ✓ WIRED |
| beachcomber/sync.go | lagoon/transaction.go | lagoon.AfterCommit | ⚠ WIRED, wrong timing inside plain gorm tx (Gap 1) |
| fonoteka plugin.go | search.go | wireSearch | ✓ WIRED |
Data-Flow Trace (Level 4)
| Artifact | Data | Source | Real data | Status |
|---|---|---|---|---|
| Album search document | artist_ids | reload inside syncOne |
Reads pivots at the moment of the callback; on cabana edits that is before syncBelongsToMany |
⚠ STALE on admin edit path |
Album updated broadcast payload |
album.artists | albumPayload Preload on tx |
Same timing; pre-edit artists on admin edits (subtree asserted only in Phase 12) | ⚠ STALE (same root cause) |
Token info.name |
user name | SetUserLookup DB read |
Real query | ✓ FLOWING |
Behavioral Spot-Checks
| Behavior | Command | Result | Status |
|---|---|---|---|
| LISTEN pickup not poll latency | go test ./modules/conga/ -run '^TestListenPickupLatency$' -v -count=1 |
listen 3.0ms (30s poll); poll-only not picked up in 2s | ✓ PASS |
| Plain gorm tx syncs before commit (gap evidence) | go test ./modules/beachcomber/ -run '^TestSyncAfterCommit$/plain_gorm_transaction' -v -count=1 |
PASS — the test asserts the immediate, pre-commit sync | ✓ PASS (confirms gap) |
| Full phase gate | bash scripts/check-phase11.sh --all |
self-test, hygiene, go, postgres, named, evidence passed; phase11 all passed |
✓ PASS |
| Pre-existing hello failure | go test ./examples/hello -run TestTypedItemRoute at HEAD and in a 718a35c worktree |
Same failure (surf: config http.body_limits.default_bytes is required) at both |
ℹ pre-existing, not a Phase 11 regression |
Probe Execution
Step 7c: no scripts/*/tests/probe-*.sh probes declared or present; the phase gate scripts/check-phase11.sh --all was run instead (above).
Requirements Coverage
| Requirement | Source Plan | Status | Evidence |
|---|---|---|---|
| JOBS-01 | 11-01, 11-07 | ✓ SATISFIED | SC-1 evidence. Note: the requirement text names riverpgxv5 for LISTEN; the implementation uses a pgxpool through NewWithPgxListener, which is what ROADMAP SC-1 specifies |
| CLI-04 | 11-02, 11-07 | ✓ SATISFIED (WR-02 warning) | SC-2 evidence |
| CLI-06 | 11-01, 11-07 | ✓ SATISFIED | queue:work in-process (serve) and foreground |
| RT-01 | 11-03, 11-04, 11-06, 11-07 | ✓ SATISFIED (live: human) | SC-3 evidence; hand-rolled client per the D-12/D-16 note |
| RT-02 | 11-03, 11-06, 11-07 | ✓ SATISFIED | SC-4 evidence; channel names are opaque collection:<id> as in PHP |
| RT-03 | 11-03, 11-06, 11-07 | ✓ SATISFIED | Bulk suppression + single Emit; rollback publishes nothing (broadcast jobs ride the write tx) |
| SRCH-01 | 11-05, 11-07 | ✗ BLOCKED (partial) | Scoping, kill-switch and degradation satisfied; "Album documents sync to Typesense" is incorrect on the admin edit path (Gap 1) |
No orphaned requirements: REQUIREMENTS.md maps exactly these seven IDs to Phase 11, and every one is claimed by a plan. REQUIREMENTS.md already marks all seven Complete; SRCH-01 should not be treated as complete until Gap 1 closes.
Prohibitions
All prohibitions are verification: test and each has a named test run by the gate; high threats additionally have RC removal checks in 11-SECURITY-REVIEW.md. None flagged. The SRCH-01 prohibitions (no index without positive collection_id; no request while the switch is off or the key is empty; engine failure never fails the write) hold, including inside plain transactions.
Anti-Patterns and Review Findings
No TBD/FIXME/XXX markers in the Phase 11 files. Code review findings weighed against the success criteria:
| Finding | File | Severity here | Impact on goal |
|---|---|---|---|
| CR-01 | lagoon/transaction.go, cabana/crud.go, fonoteka album_write_service.go | 🛑 Blocker | Gap 1 (SC-5/SRCH-01, D-20 truth) |
| WR-02 | conga/commands.go withDB, lagoon/commands.go withDB | ⚠ Warning | SC-2 holds for in-process commands; any scheduled command opening its own DB fails inside a worker. Must be fixed before Phase 14 registers fonoteka:prune-notifications |
| WR-01 | lighthouse/broadcast.go:443-449 | ⚠ Warning | Framework API trap for Binding functions that call WithContext; fonoteka bindings unaffected |
| WR-03 | lagoon/transaction.go:54-65 | ⚠ Warning | Nested Transaction over the root handle; same area as Gap 1, fix together |
| WR-04 | flare/commands.go; fonoteka.go/.gitignore | ⚠ Warning (security) | Confirmed: git check-ignore config/env/dev/overrides.yaml reports not ignored, so --update can leave the VAPID private key committable |
| WR-05 | centrifugo/token.go, handlers.go | ⚠ Warning (security) | Identifier tokens with numeric prefixes authorize as user ids; no callers today |
| WR-06 | lighthouse/broadcast.go:424-432 | ⚠ Warning | Default payload publishes in-memory model; Album overrides payload |
| WR-07 | fonoteka routes.go/plugin.go, lighthouse/route.go | ⚠ Warning | Subscribe proxy shares IP bucket with public traffic (PHP parity, DoS vector) |
| IN-01..IN-10 | various | ℹ Info | None affects a success criterion |
Human Verification Required
- Live Centrifugo v6 + unchanged Nuxt — change an album, confirm the event arrives with a Go-issued token.
- Live Typesense 26.0 upsert — enable the switch, save an album, query the collection. Re-run after Gap 1 is fixed, and edit the album's artists in the admin form to confirm artist_ids.
- Real-browser Web Push through the flare VAPID driver.
- Multi-process scheduler leader election (11-02 backstop).
- Broadcast delivery order accepted as unordered (11-03 backstop).
Gaps Summary
One root cause blocks the phase: lagoon.AfterCommit treats a plain gorm transaction as "no transaction" and runs immediately. beachcomber relies on it for every Searchable write, and both real album write paths (the framework's cabana admin CRUD, live since Phase 10, and fonoteka SaveAlbum, which Phase 12 will call) use plain transactions that save the album before writing its artist pivots. Typesense therefore receives a document built mid-transaction with stale artist_ids, and a later failure in the same transaction leaves Typesense diverged from the committed database. Passing tests lock this in. Phase 12's SQL re-gate keeps a stale document from leaking an unauthorized result, so this is a correctness and parity defect rather than a data leak. It is not deferred: no later phase owns the AfterCommit contract or the cabana write path (Phase 12 SC-3 covers the re-gate, Phase 14 only the reindex command). The fix is small and local: route cabana and SaveAlbum through lagoon.Transaction, make AfterCommit refuse a foreign transaction, invert the two tests, and add an admin artists-edit test. Fix WR-03 in the same change and WR-02 before Phase 14.
Everything else in the goal is in place and tested: River's dual-driver split with measured LISTEN pickup, the job manager, queue and schedule commands, Centrifugo tokens, the proxy, publishing matched against PHP goldens, and the authorizer registry with bulk suppression.
Verified: 2026-09-30T13:28:50Z Verifier: Claude (gsd-verifier)