docs(11): create phase plan
This commit is contained in:
Jakub Zych
@@ -505,9 +505,28 @@ Plans:
|
||||
4. A channel-namespace authorizer registry re-validates on every subscribe; a broadcastable model interface with bulk-write suppression emits exactly one summary event for a bulk operation.
|
||||
5. Typesense sync is scoped by `collection_id` behind a settings kill-switch and degrades gracefully without DB/config.
|
||||
|
||||
**Plans**: TBD
|
||||
**Plans:** 7 plans
|
||||
**Research flag:** yes
|
||||
|
||||
Plans:
|
||||
|
||||
**Wave 1**
|
||||
- [ ] 11-01-PLAN.md — conga core: River v0.47.0 on NewWithPgxListener, summer_jobs + River v7 migrations, job manager, conga.Job[T], worker in serve, queue:work/queue:clear, lagoon OnDatabase and Transaction/AfterCommit seams, fonoteka allow-lists (summercms.go, fonoteka.go)
|
||||
|
||||
**Wave 2** *(blocked on Wave 1 completion)*
|
||||
- [ ] 11-02-PLAN.md — Scheduler: pact.HasSchedule, Daily/Every, River periodic jobs, bonfire.Call, schedule:run (+ --once), fonoteka prune-notifications entry
|
||||
- [ ] 11-03-PLAN.md — Realtime: lighthouse + Centrifugo driver (token route, subscribe proxy, HTTP client, broadcasts with suppression), fonoteka authorizers, Mount, ws-api bucket, Album binding
|
||||
|
||||
**Wave 3** *(blocked on Wave 2 completion)*
|
||||
- [ ] 11-05-PLAN.md — Search: beachcomber + Typesense engine, after-commit sync, fonoteka Album Searchable and settings kill-switch
|
||||
- [ ] 11-06-PLAN.md — tide Centrifugo recorder, PHP broadcast goldens (deleted + bulk proven; created/updated pending Phase 12), realtime routes recorded and replayed
|
||||
|
||||
**Wave 4** *(blocked on Wave 3 completion)*
|
||||
- [ ] 11-04-PLAN.md — Web Push (flare VAPID driver, RFC 8291/8292) and websockets:health, websockets:generate-vapid-keys, websockets:test-push
|
||||
|
||||
**Wave 5** *(blocked on Wave 4 completion)*
|
||||
- [ ] 11-07-PLAN.md — Unit tests last: full coverage, failing-when-broken T-11 evidence, check-phase11.sh gate, security review and validation map
|
||||
|
||||
### Phase 11.1: SummerCMS documentation for humans and AI agents (INSERTED)
|
||||
|
||||
**Goal:** SummerCMS has a WinterCMS-style documentation set that serves both humans and AI agents. The Markdown source lives in `summercms.go/docs/` and a `summer` CLI command builds it into a static site with sidebar navigation, search, `llms.txt`/`llms-full.txt` and a raw `.md` per page. Every code example compiles and is tested, and a "Coming from WinterCMS" map plus an `acme/blog` porting walkthrough cover the migration path. It documents the framework as it stands after Phase 11 and never names a consuming application.
|
||||
|
||||
@@ -1,18 +1,18 @@
|
||||
---
|
||||
gsd_state_version: "1.0"
|
||||
milestone: v1.0
|
||||
current_phase: "10.1"
|
||||
current_phase_name: Runtime admin extension point
|
||||
status: verifying
|
||||
current_phase: 11
|
||||
current_phase_name: Jobs, realtime and search infrastructure
|
||||
status: executing
|
||||
stopped_at: Completed 10.1-04-PLAN.md
|
||||
last_updated: "2026-09-29T01:15:04.585Z"
|
||||
last_updated: "2026-09-29T12:34:00.518Z"
|
||||
last_activity: 2026-09-28
|
||||
last_activity_desc: Phase 10.1 execution started
|
||||
state_head: 0ddf7f8f68ed31d3b16ccc734aa480da1200224e
|
||||
state_head: d9b951fe29243cf1855b680e746778336271c6b2
|
||||
progress:
|
||||
total_phases: 19
|
||||
completed_phases: 9
|
||||
total_plans: 78
|
||||
total_plans: 85
|
||||
completed_plans: 78
|
||||
milestone_name: milestone
|
||||
---
|
||||
@@ -28,9 +28,9 @@ See: .planning/PROJECT.md (updated 2026-09-16)
|
||||
|
||||
## Current Position
|
||||
|
||||
Phase: 10.1 (Runtime admin extension point) — EXECUTING
|
||||
Phase: 11 (Jobs, realtime and search infrastructure) — READY TO EXECUTE
|
||||
Plan: 4 of 4
|
||||
Status: Phase complete — ready for verification
|
||||
Status: Ready to execute
|
||||
Last activity: 2026-09-28 — Phase 10.1 execution started
|
||||
|
||||
Progress: [██████░░░░] 60%
|
||||
|
||||
@@ -0,0 +1,371 @@
|
||||
---
|
||||
phase: 11-jobs-realtime-and-search-infrastructure
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- go.mod
|
||||
- go.sum
|
||||
- modules/lagoon/queue_migrations.go
|
||||
- modules/lagoon/migrations.go
|
||||
- modules/lagoon/ondatabase.go
|
||||
- modules/lagoon/transaction.go
|
||||
- modules/lagoon/connection.go
|
||||
- modules/lagoon/ondatabase_test.go
|
||||
- modules/lagoon/transaction_test.go
|
||||
- modules/lagoon/README.md
|
||||
- modules/conga/conga.go
|
||||
- modules/conga/record.go
|
||||
- modules/conga/job.go
|
||||
- modules/conga/client.go
|
||||
- modules/conga/worker.go
|
||||
- modules/conga/commands.go
|
||||
- modules/conga/postgres_test.go
|
||||
- modules/conga/listen_test.go
|
||||
- modules/conga/README.md
|
||||
- modules/surf/serve.go
|
||||
- modules/surf/README.md
|
||||
- internal/build/build.go
|
||||
- internal/build/build_test.go
|
||||
- internal/build/artifact.go
|
||||
- internal/build/stubs/artifacts.tmpl
|
||||
- cmd/summer/main.go
|
||||
- cmd/summer/runtime.go
|
||||
- cmd/summer/main_test.go
|
||||
- examples/hello/main.go
|
||||
- examples/hello/go.mod
|
||||
- examples/hello/go.sum
|
||||
- examples/hello/plugins/base/go.sum
|
||||
- examples/hello/plugins/greeter/go.sum
|
||||
- examples/hello/plugins/optional/go.sum
|
||||
- README.md
|
||||
- ../fonoteka.go/go.mod
|
||||
- ../fonoteka.go/go.sum
|
||||
- ../fonoteka.go/main.go
|
||||
- ../fonoteka.go/config/queue.yaml
|
||||
- ../fonoteka.go/parity/schema_diff_test.go
|
||||
- ../fonoteka.go/parity/migrate_test.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/go.mod
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/go.sum
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/plugin.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/plugin_boot_test.go
|
||||
- ../fonoteka.go/plugins/golem15/user/go.mod
|
||||
- ../fonoteka.go/plugins/golem15/user/go.sum
|
||||
autonomous: true
|
||||
requirements: [JOBS-01, CLI-06]
|
||||
estimate:
|
||||
tokens: 160000
|
||||
raw_tokens: 160000
|
||||
tasks: 3
|
||||
confidence: low
|
||||
must_haves:
|
||||
truths:
|
||||
- "Per D-01, `lagoon.Migrate` always runs a framework migration set under history id summercms.conga (history table summer_migrations_summercms_conga) that creates River's schema pinned at version 7 and the `summer_jobs` table with exactly the PHP golem15_apparatus_jobs columns (SERIAL id, label, status default 0, progress default 0, progress_max default 0, nullable user_id, is_admin default false, is_canceled default false, metadata TEXT NOT NULL, nullable created_at/updated_at) plus the internal nullable `river_job_id BIGINT`."
|
||||
- "Per D-02, `conga.From(app).Dispatch` inserts the summer_jobs row with status 1 (IN_PROGRESS, PHP JobManager.php:76, not IN_QUEUE), user_id/is_admin from the request principal and metadata `\"\"` when none is given, and enqueues the River job with InsertTx on the same *sql.Tx; rolling that transaction back leaves neither a summer_jobs row nor a river_job row."
|
||||
- "Per D-03, statuses are IN_QUEUE=0, IN_PROGRESS=1, COMPLETE=2, ERROR=3, STOPPED=4; a job that errors on a non-final attempt leaves its row IN_PROGRESS, the final failed attempt (Attempt >= MaxAttempts) or a recovered panic sets ERROR with the error text under metadata key `error`, and skip is expressed as CompleteJob with metadata {\"skipped\": true} (no extra status)."
|
||||
- "Per D-04, CancelJob sets is_canceled=true and status STOPPED and calls River JobCancel, so a queued job never starts and a running job's ctx is cancelled; StopJob (worker side) sets only STOPPED like PHP cancelJob; CheckIfCanceled reads is_canceled."
|
||||
- "Per D-17 and ROADMAP SC-1, a worker runs one `*river.Client[*sql.Tx]` built with `riverdatabasesql.NewWithPgxListener(sharedSQLDB, listenerPool)` where the listener pgxpool has MaxConns 1 and MinConns 0; with FetchPollInterval 30s, a job inserted with InsertTx from a different insert-only client and committed is picked up in under 1s, while a poll-only `riverdatabasesql.New` worker with the same poll interval does not pick it up within 2s."
|
||||
- "Per D-17, `summer serve` (surf.ServeCommand) starts the worker in-process after publishing the database unless `queue.work_in_serve` is false, and stops it on shutdown; `summer queue:work` runs a foreground worker and accepts repeatable `--queue` filters; an unknown queue name is an error listing the known queues."
|
||||
- "Per D-05, `queue:clear [queue]` deletes available, scheduled and retryable jobs of one queue (default `default`) in loops of River JobDeleteMany until a pass deletes none, prints `Cleared N jobs`, and never touches running jobs."
|
||||
- "Per RESEARCH Pattern 2, plugins register River-free jobs with `conga.Job[T pact.JobArgs](fn, opts...)`; a pact.Job not built by conga.Job fails worker start with an error naming the plugin; `summer make:job` scaffolds a typed args struct wrapped in conga.Job that compiles."
|
||||
- "Per RESEARCH Pattern 4, `lagoon.OnDatabase(app, fn)` runs fn immediately when the database is already published and otherwise when `lagoon.Publish` runs, so the fonoteka artist-resolver and join-table GORM callbacks now register under `summer serve` too (the documented Boot-order gap)."
|
||||
- "Per RESEARCH Pattern 10 and D-20's seam, `lagoon.Transaction(ctx, gdb, fn)` runs `lagoon.AfterCommit` callbacks only after a successful commit (none on rollback, inner savepoint work dropped when the inner call fails); an implicit single-statement GORM transaction flushes its AfterCommit callbacks after `gorm:commit_or_rollback_transaction`; outside any lagoon-managed transaction the callback runs immediately with the caller's handle."
|
||||
- "fonoteka.go stays green: parity/schema_diff_test.go allowedDiffs names each River table and summer_jobs with the D-01/JOBS-01 reason, and parity/migrate_test.go expects summer_migrations_summercms_conga in the history-table list; `go vet ./... && go test ./...` pass in both repositories after every task."
|
||||
artifacts:
|
||||
- path: "modules/lagoon/queue_migrations.go"
|
||||
provides: "QueueMigrations(sqlDB), QueueHistoryID, JobsTable, RiverSchemaVersion"
|
||||
contains: "summer_jobs"
|
||||
- path: "modules/lagoon/ondatabase.go"
|
||||
provides: "OnDatabase seam drained by Publish"
|
||||
contains: "func OnDatabase"
|
||||
- path: "modules/lagoon/transaction.go"
|
||||
provides: "Transaction and AfterCommit"
|
||||
contains: "func AfterCommit"
|
||||
- path: "modules/conga/conga.go"
|
||||
provides: "Manager, From, Dispatch, Enqueue, Register and the PHP JobManager methods"
|
||||
contains: "func (m *Manager) Dispatch"
|
||||
- path: "modules/conga/worker.go"
|
||||
provides: "StartWorker, StartServeWorker, Worker.Stop on NewWithPgxListener"
|
||||
contains: "NewWithPgxListener"
|
||||
- path: "modules/conga/commands.go"
|
||||
provides: "RuntimeCommands: queue:work and queue:clear"
|
||||
contains: "queue:clear"
|
||||
- path: "modules/conga/listen_test.go"
|
||||
provides: "TestListenPickupLatency and TestDispatchTransactional"
|
||||
contains: "TestListenPickupLatency"
|
||||
- path: "modules/conga/README.md"
|
||||
provides: "Module documentation with the standard structure"
|
||||
- path: "../fonoteka.go/config/queue.yaml"
|
||||
provides: "work_in_serve, max_attempts, job_timeout, queues"
|
||||
key_links:
|
||||
- from: "modules/lagoon/migrations.go"
|
||||
to: "modules/lagoon/queue_migrations.go"
|
||||
via: "Migrate runs the summercms.conga set after attach and cabana"
|
||||
pattern: "QueueMigrations\\("
|
||||
- from: "modules/conga/conga.go"
|
||||
to: "github.com/riverqueue/river"
|
||||
via: "InsertTx on tx.Statement.ConnPool.(*sql.Tx)"
|
||||
pattern: "InsertTx\\("
|
||||
- from: "modules/surf/serve.go"
|
||||
to: "modules/conga/worker.go"
|
||||
via: "StartServeWorker after lagoon.Publish, Stop on shutdown"
|
||||
pattern: "conga\\.StartServeWorker"
|
||||
- from: "internal/build/build.go"
|
||||
to: "modules/conga/commands.go"
|
||||
via: "generated main appends conga.RuntimeCommands"
|
||||
pattern: "conga\\.RuntimeCommands"
|
||||
- from: "modules/lagoon/connection.go"
|
||||
to: "modules/lagoon/ondatabase.go"
|
||||
via: "Publish drains queued OnDatabase callbacks"
|
||||
pattern: "runDatabaseHooks|drain"
|
||||
- from: "../fonoteka.go/plugins/golem15/fonoteka/plugin.go"
|
||||
to: "modules/lagoon/ondatabase.go"
|
||||
via: "classes.RegisterHooks registered through lagoon.OnDatabase"
|
||||
pattern: "lagoon\\.OnDatabase"
|
||||
prohibitions:
|
||||
- requirement_id: JOBS-01
|
||||
category: transparency
|
||||
statement: "MUST NOT report a summer_jobs row as COMPLETE (2) for work that failed, panicked or was discarded by River; a final failure ends as ERROR (3) with the error recorded in metadata"
|
||||
status: resolved
|
||||
verification: test
|
||||
- requirement_id: JOBS-01
|
||||
category: safety
|
||||
statement: "MUST NOT leave a River job or a summer_jobs row behind when the business transaction that dispatched it rolls back"
|
||||
status: resolved
|
||||
verification: test
|
||||
- requirement_id: CLI-06
|
||||
category: safety
|
||||
statement: "queue:clear MUST NOT delete, cancel or interrupt a running job; only available, scheduled and retryable jobs are removed"
|
||||
status: resolved
|
||||
verification: test
|
||||
---
|
||||
|
||||
## Phase Goal
|
||||
|
||||
ROADMAP Phase 11 goal (verbatim; it is not in user-story form, see the planner return note): River jobs run on the correct dual-driver split, Centrifugo publishing and channel authorization match the existing server, and Typesense sync stays a re-gated pre-filter — all brought up before the API phases that depend on them.
|
||||
|
||||
This plan's slice: a plugin can dispatch a job inside its write transaction, a worker running in `summer serve` or `summer queue:work` picks it up through LISTEN/NOTIFY, and the job's outcome is queryable from `summer_jobs` (JOBS-01, CLI-06).
|
||||
|
||||
<objective>
|
||||
Build the `conga` job framework package on River v0.47.0 and the two `lagoon` seams later plans need, and wire workers into `serve`, `queue:work` and `queue:clear`.
|
||||
|
||||
Purpose: CSV import (Phase 13/14), broadcasts (plan 11-03) and the scheduler (plan 11-02) all run on this. Decisions implemented: D-01, D-02, D-03, D-04, D-05, D-17; user decisions 2 (NewWithPgxListener, one client) and 4 (river_job_id column); RESEARCH Patterns 1-4 and 10, Pitfalls 1-5, 12, 13.
|
||||
Output: River dependency, framework migrations (River v7 + summer_jobs), `modules/conga` with README, lagoon OnDatabase/Transaction/AfterCommit, worker in serve, queue:work/queue:clear, generated-main and `summer` delegate changes, the make:job stub, the fonoteka.go allow-list and Boot-order fixes.
|
||||
|
||||
Repos: summercms.go (framework) and fonoteka.go (go.mod/go.sum tidy, config, allow-lists, Boot seam adoption, regenerated main.go). Commit each repository separately; fonoteka.go changes that keep its tests green land in the same task as the framework change that needs them. Planning docs and code in separate commits. Never add co-author tags.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@~/.claude/gsd-core/workflows/execute-plan.md
|
||||
@~/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/notes/apparatus-dissolved-into-framework.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-PATTERNS.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-VALIDATION.md
|
||||
@modules/lagoon/connection.go
|
||||
@modules/lagoon/migrations.go
|
||||
@modules/lagoon/backend_admin_migrations.go
|
||||
@modules/lagoon/commands.go
|
||||
@modules/lagoon/postgres_test.go
|
||||
@modules/surf/serve.go
|
||||
@modules/pact/capabilities.go
|
||||
@modules/postcard/mailer.go
|
||||
@internal/build/build.go
|
||||
@cmd/summer/runtime.go
|
||||
|
||||
<interfaces>
|
||||
Existing seams (read, do not re-derive):
|
||||
- modules/lagoon/connection.go: `Open(ctx, dsn) (*sql.DB, *gorm.DB, error)`, `Use(ctx, sqlDB) (*gorm.DB, error)`, unexported `gormFromSQL(sqlDB)` (opens GORM with `&gorm.Config{}` and checks it wraps the same *sql.DB), `OpenFromApp(ctx, app)`, `DSN(cfg *compass.Config) string` (database.dsn), `Publish(app, sqlDB, gdb) error` (publishes *sql.DB then *gorm.DB; backpack refuses a second publish of the same type). The comment "Phase 11 owns a separate pgxpool.Pool for River LISTEN/NOTIFY. Do not create that listener pool here" stays true: conga creates it.
|
||||
- modules/lagoon/migrations.go: `Migrate(gdb, plugins []party.Plugin)` runs `migrator(gdb, "summercms.attach", attach.Migrations)` then `migrator(gdb, "summercms.cabana", BackendAdminMigrations)` then plugin sets; `migrator` uses gormigrate with `UseTransaction: true`; `HistoryTableName(id)` gives `summer_migrations_<id with dots as underscores>`.
|
||||
- modules/lagoon/commands.go: `RuntimeCommands(app, plugins)`; unexported `withDB(ctx, app, fn)` opens, publishes and closes the DB for CLI commands.
|
||||
- modules/pact/capabilities.go:84-99: `JobArgs{Kind() string}`, `Job{Work(ctx, args JobArgs) error}`, `HasJobs{Jobs() []Job}` — River-free by contract ("the interface itself must not import River").
|
||||
- modules/bonfire/command.go: `Command{Name, Description, Flags, Args, Run}`, `Flag{Name, Description, Default, Bare, Repeatable}`, `Input.Flag(name)`, `Input.Flags(name) []string` (repeatable), `Input.Argument(name)`.
|
||||
- modules/bouncer/context.go: `User(ctx) (*Principal, bool)`; `Principal.ID uint`, `Principal.Backend bool`.
|
||||
- modules/surf/serve.go: `ServeCommand(app, plugins)`: OpenFromApp → lagoon.Publish → publishUploads → Assemble → signal.NotifyContext → ListenAndServe/Shutdown(10s).
|
||||
- cmd/summer/runtime.go: `delegateCommand(name, description)` forwards only positional args; `delegateRollbackCommand()` is the flag-forwarding precedent.
|
||||
- River v0.47.0 (module cache): `riverdatabasesql.New(dbPool *sql.DB) *Driver`, `riverdatabasesql.NewWithPgxListener(dbPool *sql.DB, listenerPool *pgxpool.Pool) *Driver`; `river.NewClient(driver, *river.Config)`; Config fields `Queues map[string]river.QueueConfig{MaxWorkers}`, `Workers *river.Workers`, `PeriodicJobs`, `MaxAttempts`, `JobTimeout`, `FetchPollInterval`, `FetchCooldown`, `Logger *slog.Logger`; `(*Client[TTx]).InsertTx(ctx, tx TTx, args JobArgs, opts *InsertOpts) (*rivertype.JobInsertResult, error)`, `Insert`, `JobCancel(ctx, id)`, `JobDeleteMany(ctx, *JobDeleteManyParams)`, `Start`, `Stop`, `StopAndCancel`; `river.NewJobDeleteManyParams().Queues(q).States(...).First(n)`; `river.AddWorkerSafely[T](workers, worker)`; `river.Worker[T]` = `Work(ctx, *Job[T]) error` plus `Timeout(*Job[T])`, `NextRetry`, `Middleware` (embed `river.WorkerDefaults[T]`); `river.Job[T]{*rivertype.JobRow; Args T}`; `rivertype.JobRow{ID, Attempt, MaxAttempts, Metadata []byte, Kind, Queue, State}`; `river.JobCancel(err) error`; InsertOpts `{MaxAttempts, Metadata []byte, Queue, ScheduledAt, UniqueOpts}`; `rivermigrate.New(driver, nil)`, `(*Migrator).Migrate(ctx, rivermigrate.DirectionUp, &rivermigrate.MigrateOpts{TargetVersion: 7})` (per-migration transactions; `MigrateTx` is marked Deprecated at river_migrate.go:344-346 because "Certain migrations cannot be batched together in a single transaction"); DirectionDown with TargetVersion -1 removes River entirely (river_migrate.go:239-241). Defaults: FetchPollInterval 1s, JobTimeout 1m, MaxAttempts 25.
|
||||
- PHP contract: /media/nvme/dev/golem15/fonoteka/plugins/golem15/apparatus/classes/JobManager.php (dispatch status IN_PROGRESS, json_encode metadata, startJob sets progress 0/progress_max/updated_at, updateJobState sets only progress then replaces metadata when non-empty, completeJob sets COMPLETE and progress=progress_max (1 when the row is missing) and replaces metadata only when non-empty, failJob ERROR, cancelJob STOPPED, checkIfCanceled, getMetadata `json_decode ?: []`, raw query-builder updates that never touch updated_at except dispatch/startJob), contracts/JobStatus.php, updates/create_jobs_table.php, console/QueueClearCommand.php.
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
## Artifacts this phase produces
|
||||
|
||||
(This plan's share of the phase artifacts.)
|
||||
|
||||
- Dependencies: `github.com/riverqueue/river v0.47.0`, `github.com/riverqueue/river/riverdriver/riverdatabasesql v0.47.0`, `github.com/riverqueue/river/rivertype v0.47.0` (plus the transitive riverdriver, riverpgxv5, rivershared modules River pulls in).
|
||||
- Tables: `summer_jobs` (with internal `river_job_id BIGINT NULL`), River v7 tables (`river_job`, `river_leader`, `river_queue`, `river_migration` and whatever else version 7 leaves; the executor lists the exact set from `pg_tables` after migrating), history table `summer_migrations_summercms_conga`.
|
||||
- lagoon: `QueueMigrations(sqlDB *sql.DB) []*gormigrate.Migration`, const `QueueHistoryID = "summercms.conga"`, const `JobsTable = "summer_jobs"`, const `RiverSchemaVersion = 7`, `OnDatabase(app, fn func(*sql.DB, *gorm.DB) error) error`, `Transaction(ctx, gdb, fn func(ctx context.Context, tx *gorm.DB) error) error`, `AfterCommit(ctx, db *gorm.DB, fn func(ctx context.Context, db *gorm.DB))`, GORM callback name `lagoon:after_commit`.
|
||||
- conga: `Manager`, `From(app) (*Manager, error)`, `(*Manager).Register(jobs ...pact.Job) error`, `Dispatch(ctx, db, args, DispatchOpts) (uint, error)`, `Enqueue(ctx, db, args, EnqueueOpts) error`, `StartJob`, `UpdateJobState`, `UpdateMetadata`, `CompleteJob`, `FailJob`, `CancelJob`, `StopJob`, `CheckIfCanceled`, `GetMetadata`, `Get`; `Record` (summer_jobs model); `Status` with `StatusInQueue`, `StatusInProgress`, `StatusComplete`, `StatusError`, `StatusStopped`; `DispatchOpts{Label, Count, Metadata, Queue, Delay, MaxAttempts}`; `EnqueueOpts{Queue, Delay, MaxAttempts}`; `Job[T pact.JobArgs](fn func(context.Context, T) error, opts ...JobOption) pact.Job`; `JobOption`, `OnQueue`, `MaxAttempts`, `Timeout`; `JobID(ctx) (uint, bool)`; `WorkerOptions{Queues []string}`; `StartWorker(ctx, app, plugins, WorkerOptions) (*Worker, error)`; `StartServeWorker(ctx, app, plugins) (*Worker, error)`; `(*Worker).Stop(ctx) error`; `RuntimeCommands(app, plugins) []bonfire.Command`; errors `ErrNoDatabase`, `ErrRegistrationClosed`, `ErrNotCongaJob`, `ErrUnknownQueue`.
|
||||
- CLI: app commands `queue:work [--queue name ...]`, `queue:clear [queue]`; `summer` delegates `queue:work` (forwarding repeatable `--queue`) and `queue:clear`.
|
||||
- Config keys (app-level `queue.*`): `queue.work_in_serve` (default true), `queue.max_attempts` (3), `queue.job_timeout` (300 seconds, int or duration string), `queue.queues.<name>` (MaxWorkers; default queue `default: 4`).
|
||||
- Files: `modules/conga/*`, `../fonoteka.go/config/queue.yaml`.
|
||||
|
||||
## Flagged assumptions (edge probe: unclassified)
|
||||
|
||||
- JOBS-01 and CLI-06 came back `unclassified` from the spec-less edge probe and stay `unresolved`; they are surfaced here, not auto-resolved. Planner reading for manual review: (a) concurrency — two workers never run the same River job concurrently (River row locking) and two Dispatch calls produce distinct summer_jobs ids; (b) idempotency — Dispatch is not idempotent (each call is a new job, like PHP); (c) empty — `queue:work` with no registered jobs still starts and idles; `queue:clear` on an empty queue prints `Cleared 0 jobs`. Confirm or correct at verify time.
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="tracer">
|
||||
<name>Task 1: A job dispatched in a write transaction is picked up through LISTEN and completes its summer_jobs row</name>
|
||||
<reversibility rating="one-way">D-01: summer_jobs ids become the public import_job_id/match_job_id contract and Phase 15 copies golem15_apparatus_jobs rows with ids preserved; the user chose this table and column list in CONTEXT.md (plus river_job_id at the plan-count checkpoint), so it is recorded without a checkpoint.</reversibility>
|
||||
<precondition>Docker is reachable for testcontainers: `docker info` exits 0.</precondition>
|
||||
<files>go.mod, go.sum, modules/lagoon/queue_migrations.go, modules/lagoon/migrations.go, modules/lagoon/README.md, modules/conga/conga.go, modules/conga/record.go, modules/conga/job.go, modules/conga/client.go, modules/conga/worker.go, modules/conga/postgres_test.go, modules/conga/listen_test.go, modules/conga/README.md, README.md, examples/hello/go.mod, examples/hello/go.sum, examples/hello/plugins/base/go.sum, examples/hello/plugins/greeter/go.sum, examples/hello/plugins/optional/go.sum, ../fonoteka.go/go.mod, ../fonoteka.go/go.sum, ../fonoteka.go/plugins/golem15/fonoteka/go.mod, ../fonoteka.go/plugins/golem15/fonoteka/go.sum, ../fonoteka.go/plugins/golem15/user/go.mod, ../fonoteka.go/plugins/golem15/user/go.sum, ../fonoteka.go/parity/schema_diff_test.go, ../fonoteka.go/parity/migrate_test.go</files>
|
||||
<read_first>modules/lagoon/connection.go, modules/lagoon/migrations.go, modules/lagoon/backend_admin_migrations.go, modules/lagoon/postgres_test.go (TestMain, dedicatedDB, dsnWithDB), modules/lagoon/README.md, modules/pact/capabilities.go (JobArgs/Job/HasJobs), modules/postcard/mailer.go (Activate/driverFromApp/loggerFromApp pattern), modules/bouncer/context.go, README.md (modules table), /media/nvme/dev/golem15/fonoteka/plugins/golem15/apparatus/classes/JobManager.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/apparatus/updates/create_jobs_table.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/apparatus/contracts/JobStatus.php, $(go env GOMODCACHE)/github.com/riverqueue/river/riverdriver/riverdatabasesql@v0.47.0/river_database_sql_driver.go (lines 40-140), $(go env GOMODCACHE)/github.com/riverqueue/river@v0.47.0/rivermigrate/river_migrate.go (lines 215-360), ../fonoteka.go/parity/schema_diff_test.go (allowedDiffs, publicBaseTables), ../fonoteka.go/parity/migrate_test.go (wantTables), .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Patterns 1-3, Pitfalls 1, 3, 5, 12, 13, Code Examples "Transactional dispatch")</read_first>
|
||||
<action>(1) Dependency (user decision 2, research-named): in summercms.go run `go get github.com/riverqueue/river@v0.47.0 github.com/riverqueue/river/riverdriver/riverdatabasesql@v0.47.0 github.com/riverqueue/river/rivertype@v0.47.0` and `go mod tidy`; MVS bumps testify to v1.12.1, which is expected. Then run `go mod tidy` in examples/hello and in ../fonoteka.go (root, plugins/golem15/fonoteka, plugins/golem15/user) so every module that links lagoon has River in go.sum; commit only files that changed. No other new module.
|
||||
|
||||
(2) Framework migrations (D-01, user decision 4, Pitfall 13): new modules/lagoon/queue_migrations.go with consts `QueueHistoryID = "summercms.conga"`, `JobsTable = "summer_jobs"`, `RiverSchemaVersion = 7` and `func QueueMigrations(sqlDB *sql.DB) []*gormigrate.Migration` returning two migrations. `202609290001_river_schema`: Migrate builds `rivermigrate.New(riverdatabasesql.New(sqlDB), nil)` and calls `Migrate(ctx, rivermigrate.DirectionUp, &rivermigrate.MigrateOpts{TargetVersion: RiverSchemaVersion})` on the shared pool (per-migration transactions; the Tx variant is deprecated upstream because some River migrations cannot share one transaction); Rollback calls DirectionDown with TargetVersion -1. `202609290002_summer_jobs`: `CREATE TABLE IF NOT EXISTS summer_jobs (id SERIAL PRIMARY KEY, label VARCHAR(255) NOT NULL, status INTEGER NOT NULL DEFAULT 0, progress INTEGER NOT NULL DEFAULT 0, progress_max INTEGER NOT NULL DEFAULT 0, user_id INTEGER NULL, is_admin BOOLEAN NOT NULL DEFAULT FALSE, is_canceled BOOLEAN NOT NULL DEFAULT FALSE, metadata TEXT NOT NULL, river_job_id BIGINT NULL, created_at TIMESTAMPTZ NULL, updated_at TIMESTAMPTZ NULL)` (Laravel `timestamps()` are nullable; no extra indexes, matching PHP); Rollback drops the table. In migrations.go `Migrate` runs the set with `migrator(gdb, QueueHistoryID, QueueMigrations(sqlDB))` right after the cabana set, where sqlDB comes from `gdb.DB()`; wrap errors as `lagoon: migrate queue: %w`. The set lives in lagoon for the same reason BackendAdminMigrations does (lagoon cannot import conga), and every app gets it on `migrate`.
|
||||
|
||||
(3) conga core, new package modules/conga (errors prefixed `conga: `, logger from `app.Lookup[*slog.Logger]()` else slog.Default() like postcard):
|
||||
- record.go: `type Status int` with the five PHP constants; `type Record struct` mapping every summer_jobs column (ID uint, Label string, Status Status, Progress int, ProgressMax int, UserID *uint, IsAdmin bool, IsCanceled bool, Metadata string, RiverJobID *int64, CreatedAt/UpdatedAt *time.Time), `TableName() string { return lagoon.JobsTable }`.
|
||||
- job.go: `type JobOption func(*jobConfig)`, `OnQueue(name string)`, `MaxAttempts(n int)`, `Timeout(d time.Duration)`; `func Job[T pact.JobArgs](fn func(ctx context.Context, args T) error, opts ...JobOption) pact.Job` returning an unexported `*typedJob[T]` that implements pact.Job (its Work type-asserts args to T and calls fn) plus unexported `kind() string`, `register(*river.Workers) error` (calls `river.AddWorkerSafely[T]` with an unexported riverWorker[T] embedding `river.WorkerDefaults[T]`) and `config() jobConfig`. `JobID(ctx) (uint, bool)` reads the summer_jobs id the wrapper stored in ctx.
|
||||
- conga.go: `type Manager struct` holding the app, a mutex, the registered jobs by kind, the lazily built insert-only client and the running worker client; `func From(app *backpack.App) (*Manager, error)` does lookup-or-publish of `*Manager` on the app (on a publish race, look up again); `Register(jobs ...pact.Job) error` returns `ErrNotCongaJob` for a job not built by Job, an error on a duplicate kind, and `ErrRegistrationClosed` once a client has been built; `Dispatch(ctx, db *gorm.DB, args pact.JobArgs, o DispatchOpts) (uint, error)`: when db is not inside a transaction (`db.Statement.ConnPool` is not a *sql.Tx) wrap the work in `db.WithContext(ctx).Transaction`; insert the Record with Status StatusInProgress (Pitfall 1), Label (required, error when empty), ProgressMax = o.Count, Metadata = JSON of o.Metadata or the two characters `""` when nil (PHP json_encode of ''), UserID/IsAdmin from `bouncer.User(ctx)` (IsAdmin = Principal.Backend), CreatedAt = UpdatedAt = now; then `client.InsertTx(ctx, sqlTx, args, &river.InsertOpts{Queue, MaxAttempts, ScheduledAt: now+o.Delay when Delay > 0, Metadata: {"summer_job_id": id}})` using the registered job's queue/max attempts when o leaves them zero; then set river_job_id with an UpdateColumn on the same tx; return the id. `Enqueue(ctx, db, args, EnqueueOpts)` does only the River InsertTx (Insert when db is not in a transaction) and is what plan 11-03 uses for broadcasts. `CompleteJob(ctx, id, metadata map[string]any) error` and `Get(ctx, id) (Record, error)` land here now (the rest in Task 2); CompleteJob reads progress_max (1 when the row is missing) and sets status 2 and progress with `UpdateColumns` on `Table(lagoon.JobsTable)` so updated_at is never auto-touched, replacing metadata only when the map is non-empty. The Manager resolves the database per call with `app.Lookup[*gorm.DB]()` / `*sql.DB` and returns `ErrNoDatabase` when unpublished.
|
||||
- client.go: `riverConfig(app, workers, queues)` reads `queue.max_attempts` (default 3, PHP --tries=3), `queue.job_timeout` (default 300s; accept "300s" or an int of seconds, the postcard timeout idiom; Pitfall 4) and `queue.queues.<name>` MaxWorkers (default 4); the queue set is the configured queues plus every queue a registered job names plus `default`. The insert-only client is `river.NewClient(riverdatabasesql.New(sqlDB), cfg without Queues)` built once, on first Dispatch/Enqueue/cancel when no worker client runs; the worker client is used for inserts when present.
|
||||
- worker.go: `type WorkerOptions struct { Queues []string }` (nil = every known queue) plus unexported test knobs `pollInterval time.Duration` and `pollOnly bool`; `StartWorker(ctx, app, plugins []party.Plugin, o WorkerOptions) (*Worker, error)`: registers every `pact.HasJobs` job of the plugins (a non-conga job fails with an error naming the plugin id), builds the listener pool from `lagoon.DSN(app.Config)` with `pgxpool.ParseConfig`, MaxConns 1, MinConns 0, builds `riverdatabasesql.NewWithPgxListener(sqlDB, listener)` (or `riverdatabasesql.New` when pollOnly), filters queues (`ErrUnknownQueue` listing known names), sets FetchPollInterval when the knob is set, starts the client and hands it to the Manager. `(*Worker).Stop(ctx)` calls Stop, falls back to StopAndCancel when ctx expires, and closes the listener pool. The riverWorker[T].Work wrapper: put the summer_jobs id from job.Metadata into ctx; call fn with panic recovery (a panic becomes an error); on error, when the row is STOPPED or is_canceled, return `river.JobCancel(err)`; otherwise when `job.Attempt >= job.MaxAttempts` set status 3 with metadata = current metadata plus key `error` (D-03); return the error so River retries non-final attempts; Timeout returns the job option or 0.
|
||||
|
||||
(4) Smoke tests (coverage is plan 11-07): modules/conga/postgres_test.go copies the lagoon TestMain harness (testcontainers `postgres:16-alpine`, ICU pl-PL, `-short` skips, Docker failure fails TestMain) exposing a helper that returns a dedicated *sql.DB plus its DSN migrated with `lagoon.Migrate(gdb, nil)`. modules/conga/listen_test.go: `TestListenPickupLatency` (Pitfall 5: worker with pollInterval 30s, wait until `SELECT count(*) FROM pg_stat_activity WHERE query ILIKE 'LISTEN%'` is positive, insert through a second insert-only client with InsertTx and commit, assert the job function runs within 1s; negative control with pollOnly and the same interval asserts no run within 2s) and `TestDispatchTransactional` (commit path: row status 1 then the job calls CompleteJob and Get shows status 2 and progress = progress_max; rollback path: no summer_jobs row and `SELECT count(*) FROM river_job` unchanged).
|
||||
|
||||
(5) fonoteka.go allow-lists (Pitfall 12), same task so both repos stay green: add allowedDiffs entries in parity/schema_diff_test.go for `summer_jobs` ("Framework job record (11-01, D-01); PHP golem15_apparatus_jobs is copied at cutover") and for each River table actually created (reason "River v7 schema (11-01, JOBS-01)"); determine the names by running the migration and reading pg_tables rather than from memory. Add `summer_migrations_summercms_conga` to wantTables in parity/migrate_test.go (keep the ORDER BY order).
|
||||
|
||||
(6) Docs in the same commits: new modules/conga/README.md in the standard structure (H1 `conga`, the one-sentence summary "Background jobs on River over the shared Postgres pool: transactional dispatch, a `summer_jobs` progress record, in-process or dedicated workers, and a wall-clock scheduler.", import line, Overview, Features, Usage, API reference, Configuration, CLI commands, Dependencies (River v0.47.0 and why), Testing), mentioning the PgBouncer session-pooling requirement for the listener pool; a root README.md modules-table row with the same sentence; modules/lagoon/README.md gains the QueueMigrations set in the migrations bullet. Framework text uses neutral names (acme); verify each identifier with `go doc ./modules/conga <Identifier>`.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/conga -run '^(TestListenPickupLatency|TestDispatchTransactional)$' -count=1 -v && go test ./modules/lagoon -count=1 && (cd ../fonoteka.go && go test ./parity -run '^(TestMigrateSeedsCanonicalGenres|TestSchemaMatchesPHPSnapshot)$' -count=1 -v)</automated>
|
||||
<fails_when>Any command exits non-zero; the verbose output lacks a "--- PASS" line for TestListenPickupLatency, TestDispatchTransactional, TestMigrateSeedsCanonicalGenres or TestSchemaMatchesPHPSnapshot, or prints "no tests to run" or "--- SKIP"; the schema test prints "Go extra table" for a River table or summer_jobs.</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `go list -m github.com/riverqueue/river` prints `github.com/riverqueue/river v0.47.0`.
|
||||
- `grep -c 'NewWithPgxListener' modules/conga/worker.go` prints at least 1 and `grep -c 'MaxConns' modules/conga/worker.go` prints at least 1.
|
||||
- `grep -c 'river_job_id BIGINT' modules/lagoon/queue_migrations.go` prints 1 and `grep -c 'TargetVersion: RiverSchemaVersion' modules/lagoon/queue_migrations.go` prints at least 1.
|
||||
- `grep -c 'StatusInProgress' modules/conga/conga.go` prints at least 1 (Dispatch writes status 1).
|
||||
- `grep -c 'summer_migrations_summercms_conga' ../fonoteka.go/parity/migrate_test.go` prints 1 and `grep -c '"summer_jobs"' ../fonoteka.go/parity/schema_diff_test.go` prints 1.
|
||||
- `go doc ./modules/conga Manager.Dispatch`, `go doc ./modules/conga Job`, `go doc ./modules/conga StartWorker` and `go doc ./modules/lagoon QueueMigrations` exit 0.
|
||||
- `grep -c '\[conga\](modules/conga/README.md)' README.md` prints 1.
|
||||
- TestListenPickupLatency measures pickup under 1s with a 30s poll interval and the poll-only control shows no pickup in 2s.
|
||||
</acceptance_criteria>
|
||||
<done>River is a pinned dependency, `migrate` creates River v7 and summer_jobs, a transactional Dispatch is picked up by a NewWithPgxListener worker well under the poll interval and completes its row, rollback leaves nothing, and fonoteka.go's schema and migrate tests pass with the new tables.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Job outcomes and cancellation are queryable, and workers run in serve, queue:work and queue:clear</name>
|
||||
<files>modules/conga/conga.go, modules/conga/job.go, modules/conga/worker.go, modules/conga/commands.go, modules/conga/listen_test.go, modules/conga/README.md, modules/surf/serve.go, modules/surf/README.md, internal/build/build.go, internal/build/build_test.go, internal/build/artifact.go, internal/build/stubs/artifacts.tmpl, cmd/summer/main.go, cmd/summer/runtime.go, cmd/summer/main_test.go, examples/hello/main.go, ../fonoteka.go/main.go, ../fonoteka.go/config/queue.yaml</files>
|
||||
<read_first>modules/conga/conga.go and modules/conga/worker.go (Task 1), modules/lagoon/commands.go (RuntimeCommands, withDB), modules/surf/serve.go, modules/surf/README.md, modules/bonfire/command.go, internal/build/build.go (generateMain), internal/build/build_test.go (TestGenerateMainRegistersCabanaRuntimeCommands, TestScaffoldAllArtifacts), internal/build/artifact.go (MakeJob), internal/build/stubs/artifacts.tmpl (job.go block), cmd/summer/main.go, cmd/summer/runtime.go (delegateCommand, delegateRollbackCommand), cmd/summer/main_test.go, /media/nvme/dev/golem15/fonoteka/plugins/golem15/apparatus/classes/JobManager.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/apparatus/console/QueueClearCommand.php, ../fonoteka.go/config/app.yaml (comment style), .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Pattern 3 table, Pitfalls 2-4, Don't Hand-Roll "Queue bulk delete")</read_first>
|
||||
<behavior>
|
||||
- StartJob(id, 5) sets progress 0, progress_max 5 and updated_at; UpdateJobState(id, 3, nil) sets only progress; UpdateJobState(id, 4, {"a":1}) also replaces metadata; UpdateMetadata replaces metadata without touching updated_at.
|
||||
- FailJob(id, nil) sets status 3 and keeps metadata; FailJob(id, {"x":1}) replaces it; GetMetadata returns an empty map for the stored `""` and the decoded object otherwise.
|
||||
- A job that returns an error on attempts 1 and 2 of 3 leaves the row at status 1; the third failure sets status 3 with metadata key `error`; a panicking job ends the same way.
|
||||
- CancelJob on a queued job sets is_canceled and status 4 and the job function never runs; CancelJob on a running job cancels its ctx and the row stays status 4 (not 3).
|
||||
- StopJob(id, nil) sets status 4 without is_canceled; CheckIfCanceled reflects is_canceled.
|
||||
- queue:clear on queue `default` with two available jobs and one running job deletes two, prints `Cleared 2 jobs`, and the running job completes.
|
||||
- queue:work --queue unknown exits non-zero naming the known queues.
|
||||
</behavior>
|
||||
<action>(1) Finish the PHP JobManager surface on Manager (D-02, D-03, D-04), every write through `UpdateColumns` on `Table(lagoon.JobsTable)` so GORM never auto-sets updated_at: `StartJob(ctx, id uint, total int) error` (progress 0, progress_max total, updated_at now); `UpdateJobState(ctx, id, current int, metadata map[string]any) error` (progress only, then UpdateMetadata when the map is non-empty); `UpdateMetadata(ctx, id, metadata map[string]any) error`; `FailJob(ctx, id, metadata map[string]any) error` (status 3, metadata only when non-empty); `CancelJob(ctx, id) error` (is_canceled true and status 4 in one update, then River JobCancel on river_job_id when set; a JobCancel "not found" for an already finished job is not an error); `StopJob(ctx, id, metadata map[string]any) error` (status 4 only, PHP cancelJob semantics, Pitfall 2); `CheckIfCanceled(ctx, id) (bool, error)`; `GetMetadata(ctx, id) (map[string]any, error)` (a non-object or empty decode yields an empty map). Doc comments state the Skip convention: `CompleteJob(ctx, id, map[string]any{"skipped": true})`.
|
||||
|
||||
(2) Worker wrapper refinement (D-03, D-04): after fn returns, if the row is status 4 treat any ctx-cancel error as `river.JobCancel`; a nil return with the row still at status 1 is left as is (jobs complete themselves, as in PHP). Recovered panics log the stack at Error and follow the final-attempt rule.
|
||||
|
||||
(3) Commands, new modules/conga/commands.go: `func RuntimeCommands(app *backpack.App, plugins []party.Plugin) []bonfire.Command` returning `queue:work` (description "Run background job workers in the foreground"; flag `queue` Repeatable, "Queue to work (repeatable; default all known queues)") and `queue:clear` (description "Clear all queued jobs, by deleting all pending jobs." as in PHP; optional arg `queue`, default `default`; the PHP connection argument has no Go counterpart because River uses the one database). Both open the DB the lagoon way (OpenFromApp + Publish, closing on return; mirror lagoon's unexported withDB inside conga). queue:work starts StartWorker with the filter, prints `worker started on queues: <sorted list>`, blocks on `signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM)` (surf/serve.go idiom) and stops with a 10s timeout. queue:clear prints `Clearing queue "<q>"`, loops `JobDeleteMany(ctx, river.NewJobDeleteManyParams().Queues(q).States(rivertype.JobStateAvailable, rivertype.JobStateScheduled, rivertype.JobStateRetryable).First(10000))` until a pass deletes zero, and prints `Cleared N jobs` (D-05).
|
||||
|
||||
(4) serve (D-17): in modules/surf/serve.go create the signal ctx before starting anything long-lived, then after `Assemble` call `worker, err := conga.StartServeWorker(ctx, app, plugins)`; `StartServeWorker` returns nil, nil when `queue.work_in_serve` is explicitly false and otherwise StartWorker with all queues. On every exit path after a successful start (shutdown and ListenAndServe error) stop the worker with the same 10s shutdown ctx after srv.Shutdown. surf importing conga is allowed because conga never imports surf. Update modules/surf/README.md (serve starts workers; `queue.work_in_serve`).
|
||||
|
||||
(5) Generated main and `summer` delegates: internal/build/build.go imports `git.golem15.com/golem15/summercms/modules/conga` and appends `commands = append(commands, conga.RuntimeCommands(app, plugins)...)` right after `lagoon.RuntimeCommands`; add `TestGenerateMainRegistersCongaRuntimeCommands` to build_test.go asserting that exact line appears once. Regenerate examples/hello/main.go and ../fonoteka.go/main.go with the framework CLI (`go run ./cmd/summer build` from each app directory, or `go run <path to summercms.go>/cmd/summer build` from ../fonoteka.go); bin/ output stays untracked. In cmd/summer add `delegateQueueWorkCommand()` (declares the repeatable `queue` flag and forwards every value as `--queue <v>`, modelled on delegateRollbackCommand) and `delegateCommand("queue:clear", "Clear pending queued jobs in the app binary")`, register both in toolCommands, and extend the expected list in cmd/summer/main_test.go.
|
||||
|
||||
(6) make:job stub (RESEARCH Pattern 2): the `job.go` block in internal/build/stubs/artifacts.tmpl generates `{{.Ident}}Args` with `Kind()` and `func {{.Func}}() pact.Job { return conga.Job(func(ctx context.Context, args {{.Ident}}Args) error { return nil }) }`, importing conga and pact but never River (the command description stays "Generate a plugin job without importing River"); drop the now-unused Worker field from artifact.go's data if nothing else reads it. TestScaffoldAllArtifacts must still compile the scaffolded plugin.
|
||||
|
||||
(7) fonoteka.go config: new ../fonoteka.go/config/queue.yaml with commented keys `work_in_serve: true`, `max_attempts: 3`, `job_timeout: 300`, `queues: {default: 4}` (comment: set work_in_serve false when a separate `fonoteka queue:work` process runs; the listener needs session pooling, not PgBouncer transaction pooling).
|
||||
|
||||
(8) Tests in listen_test.go for the behavior list above (the Postgres ones), kept as smoke-level; plan 11-07 adds branch coverage. Update modules/conga/README.md (API reference rows for every new method, CLI commands section, Configuration section) in the same commit.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/conga ./modules/surf ./internal/build ./cmd/summer -count=1 && go test ./internal/build -run '^(TestGenerateMainRegistersCongaRuntimeCommands|TestScaffoldAllArtifacts)$' -count=1 -v && (cd ../fonoteka.go && go vet ./... && go build ./... && SUMMER_GOLEM15__USER__JWT__SECRET=test-only-cli-secret go run . queue:clear --help)</automated>
|
||||
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS" for TestGenerateMainRegistersCongaRuntimeCommands or TestScaffoldAllArtifacts or prints "no tests to run"; the help output does not contain "queue:clear".</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `grep -c 'conga.RuntimeCommands(app, plugins)' ../fonoteka.go/main.go examples/hello/main.go` prints 1 for each file.
|
||||
- `grep -c 'conga.StartServeWorker' modules/surf/serve.go` prints 1.
|
||||
- `grep -c 'JobStateRunning' modules/conga/commands.go` prints 0 and `grep -c 'JobStateAvailable' modules/conga/commands.go` prints at least 1.
|
||||
- `grep -c 'queue:work' cmd/summer/main_test.go` prints at least 1.
|
||||
- `grep -c 'conga.Job(' internal/build/stubs/artifacts.tmpl` prints 1.
|
||||
- `go doc ./modules/conga Manager.CancelJob`, `go doc ./modules/conga Manager.StopJob` and `go doc ./modules/conga RuntimeCommands` exit 0.
|
||||
- `test -f ../fonoteka.go/config/queue.yaml` succeeds and `grep -c 'work_in_serve' ../fonoteka.go/config/queue.yaml` prints at least 1.
|
||||
</acceptance_criteria>
|
||||
<done>Every PHP JobManager operation exists with PHP semantics, retries only become ERROR on the final attempt, cancellation both stops River and marks the row, and workers run in serve (unless disabled), in queue:work with queue filters, and queue:clear removes only pending jobs.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 3: GORM callbacks registered at Boot fire under summer serve, and work can run after commit</name>
|
||||
<files>modules/lagoon/ondatabase.go, modules/lagoon/transaction.go, modules/lagoon/connection.go, modules/lagoon/ondatabase_test.go, modules/lagoon/transaction_test.go, modules/lagoon/README.md, ../fonoteka.go/plugins/golem15/fonoteka/plugin.go, ../fonoteka.go/plugins/golem15/fonoteka/plugin_boot_test.go</files>
|
||||
<read_first>modules/lagoon/connection.go (Publish, gormFromSQL), modules/lagoon/postgres_test.go, modules/lagoon/README.md, modules/backpack/services.go (Publish refuses duplicates), $(go env GOMODCACHE)/gorm.io/gorm@v1.31.2/callbacks/transaction.go, $(go env GOMODCACHE)/gorm.io/gorm@v1.31.2/callbacks.go (Register/Replace/sortCallbacks: duplicate names keep the last handler), ../fonoteka.go/plugins/golem15/fonoteka/plugin.go (Boot comment on the RegisterHooks gap), ../fonoteka.go/plugins/golem15/fonoteka/classes/registry.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/artist_resolver.go, ../fonoteka.go/plugins/golem15/fonoteka/plugin_boot_test.go, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Patterns 4 and 10, Pitfall 6)</read_first>
|
||||
<behavior>
|
||||
- OnDatabase registered before Publish runs once when Publish runs; registered after Publish runs immediately; an error from a callback makes Publish (or OnDatabase) return it.
|
||||
- Transaction commit runs AfterCommit callbacks in registration order after COMMIT; a rolled-back Transaction runs none; a nested Transaction whose fn fails drops only the callbacks it added.
|
||||
- A plain gdb.Create (implicit single-statement transaction) with an AfterCommit registered from a GORM after_create callback runs it after commit and not when the insert fails.
|
||||
- Inside gdb.Transaction (not lagoon.Transaction) AfterCommit runs immediately with the tx handle.
|
||||
- In fonoteka, activating plugins with no database published and then calling lagoon.Publish registers the `fonoteka:album_artist_resolver` callback on that *gorm.DB.
|
||||
</behavior>
|
||||
<action>(1) modules/lagoon/ondatabase.go (RESEARCH Pattern 4): `func OnDatabase(app *backpack.App, fn func(sqlDB *sql.DB, gdb *gorm.DB) error) error`. Per-app state is an unexported `*databaseHooks` service published on the app (lookup-or-publish, no package globals): when `*gorm.DB` and `*sql.DB` are already published, call fn now and return its error; otherwise append fn. In connection.go `Publish` drains the queue once after publishing both handles (`runDatabaseHooks`), returning the first error wrapped `lagoon: database hook: %w`. Update the listener-pool comment in connection.go to say conga owns it.
|
||||
|
||||
(2) modules/lagoon/transaction.go (RESEARCH Pattern 10, D-20 seam): an unexported ctx key carries an `*afterCommitBuffer`. `func Transaction(ctx context.Context, gdb *gorm.DB, fn func(ctx context.Context, tx *gorm.DB) error) error`: when ctx already carries a buffer, run a nested `tx.Transaction` (savepoint) with a child buffer merged into the parent only when fn succeeds; otherwise create a buffer, run `gdb.WithContext(txCtx).Transaction(func(tx) error { return fn(txCtx, tx) })`, and after a nil return run each buffered callback in order with `gdb.Session(&gorm.Session{NewDB: true, Context: ctx})`, recovering and logging (slog.Default Warn) any panic so a committed write is never reported as failed. `func AfterCommit(ctx context.Context, db *gorm.DB, fn func(ctx context.Context, db *gorm.DB))`: (a) buffer in ctx → append; (b) else when the statement opened its own transaction (`db.InstanceGet("gorm:started_transaction")`) → append to a statement-scoped buffer stored with `db.InstanceSet("lagoon:after_commit", ...)`; (c) else run fn now with db (the PHP `after_commit=false` fallback; inside an explicit non-lagoon GORM transaction db is that tx). In gormFromSQL register a callback named `lagoon:after_commit` with `.After("gorm:commit_or_rollback_transaction")` on the Create, Update and Delete processors that flushes the statement buffer only when `db.Error == nil`, using a fresh session on the committed connection pool. Register with `Register` when `Get(name)` is nil so repeated opens of one *gorm.DB stay idempotent.
|
||||
|
||||
(3) fonoteka.go Boot fix (flagged in RESEARCH Pattern 4 as optional; included because it closes a documented production gap): replace the `if gdb, ok := app.Lookup[*gorm.DB](); ok { classes.RegisterHooks(gdb) }` block in plugins/golem15/fonoteka/plugin.go with `lagoon.OnDatabase(app, func(_ *sql.DB, gdb *gorm.DB) error { return classes.RegisterHooks(gdb) })`, rewrite the surrounding comment to say the hooks now register whenever the database is published, and add `TestHooksRegisterWhenDatabasePublishedAfterBoot` to plugin_boot_test.go (activate with no DB published, then lagoon.Publish, then assert `gdb.Callback().Create().Get("fonoteka:album_artist_resolver")` is non-nil; use a fresh `lagoon.Use(ctx, bootSQL)` handle so the shared bootGDB is not involved).
|
||||
|
||||
(4) Tests: modules/lagoon/ondatabase_test.go `TestOnDatabaseAfterActivate` and modules/lagoon/transaction_test.go `TestTransactionAfterCommit` covering the behavior list (use the existing lagoon Postgres harness and a throwaway table). Update modules/lagoon/README.md (Features and API reference for OnDatabase, Transaction, AfterCommit and the `lagoon:after_commit` callback) in the same commit; check identifiers with `go doc ./modules/lagoon OnDatabase` etc.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/lagoon -run '^(TestOnDatabaseAfterActivate|TestTransactionAfterCommit)$' -count=1 -v && go test ./... && (cd ../fonoteka.go && go vet ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/... && go test ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/...)</automated>
|
||||
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS" for TestOnDatabaseAfterActivate or TestTransactionAfterCommit, or prints "no tests to run" or "--- SKIP"; any package in either repository reports FAIL.</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `go doc ./modules/lagoon OnDatabase`, `go doc ./modules/lagoon Transaction` and `go doc ./modules/lagoon AfterCommit` exit 0.
|
||||
- `grep -l 'lagoon:after_commit' modules/lagoon/*.go` lists at least one file.
|
||||
- `grep -c 'lagoon.OnDatabase' ../fonoteka.go/plugins/golem15/fonoteka/plugin.go` prints 1.
|
||||
- `(cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^TestHooksRegisterWhenDatabasePublishedAfterBoot$' -count=1 -v)` shows "--- PASS".
|
||||
- `grep -c 'OnDatabase' modules/lagoon/README.md` prints at least 1.
|
||||
</acceptance_criteria>
|
||||
<done>Anything registered through lagoon.OnDatabase at Boot runs when serve publishes the database, the fonoteka hooks now fire in production, and lagoon.Transaction/AfterCommit give later plans a commit-safe place to run side effects; both repositories' full suites pass.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| HTTP handler → business transaction → River | Request-driven writes enqueue jobs whose rows and River entries must share the write's fate |
|
||||
| Worker process → shared Postgres pool + listener pool | Workers hold a dedicated LISTEN connection and run plugin job code |
|
||||
| Operator CLI → queue:clear / queue:work | Destructive and long-running commands on the job tables |
|
||||
| Go module proxy → go.mod/go.sum | New third-party code (River) enters the binary |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-11-08 | Elevation of Privilege | summer_jobs ids (cancel/progress IDOR) | medium | accept | Phase 11 exposes no HTTP route over summer_jobs; the Phase 13 CSV endpoints must scope ids through the owning import (findVisible). Recorded for Phase 13. |
|
||||
| T-11-12 | Tampering | Dispatch/Enqueue (orphan jobs for rolled-back writes) | high | mitigate | Row insert and River InsertTx run on the caller's *sql.Tx (Dispatch opens one when the caller has none); TestDispatchTransactional asserts rollback leaves neither (Task 1). |
|
||||
| T-11-13 | Denial of Service | River listener pool | medium | mitigate | Dedicated pgxpool with MaxConns 1, MinConns 0 per worker; README documents session pooling for PgBouncer (Task 1). |
|
||||
| T-11-14 | Tampering | queue:clear | medium | mitigate | JobDeleteMany restricted to available, scheduled and retryable states of one queue; a running job is untouched (Task 2 behavior test). |
|
||||
| T-11-15 | Information Disclosure | job failure metadata and logs | medium | mitigate | Only the error text is written to metadata key `error`; the wrapper never logs job args; River's logger is the app logger (Task 1-2). |
|
||||
| T-11-16 | Denial of Service | panicking plugin jobs | medium | mitigate | The wrapper recovers panics into errors and applies the final-attempt ERROR rule, so a panic cannot crash the worker or strand the row (Task 2). |
|
||||
| T-11-SC | Tampering | Go module installs (River v0.47.0 and sub-modules) | high | mitigate | River is named by STACK.md and RESEARCH's legitimacy audit (Go module proxy, official docs); versions pinned in go.mod with go.sum checksums verified by GOSUMDB; no other module added. |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
After Task 3: `go vet ./... && go test ./...` in summercms.go and `(cd ../fonoteka.go && go vet ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/... && go test ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/...)` pass; `go test ./modules/conga -run '^TestListenPickupLatency$' -count=1 -v` shows LISTEN pickup under 1s with a 30s poll. [BLOCKING, schema gate equivalent] The migration set is proven against live Postgres, not only by compilation: the conga TestMain migrates a real database and fonoteka.go's TestMigrateSeedsCanonicalGenres and TestSchemaMatchesPHPSnapshot run `lagoon.Migrate` on testcontainers Postgres.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- River v0.47.0 runs on one client type with NewWithPgxListener; LISTEN pickup is proven not to be poll latency.
|
||||
- summer_jobs mirrors PHP columns plus river_job_id; Dispatch is transactional; every PHP JobManager method exists with PHP semantics; outcomes complete/fail/skip/cancel are queryable.
|
||||
- serve runs workers in-process unless `queue.work_in_serve: false`; `queue:work --queue` and `queue:clear` exist in the app binary and as `summer` delegates.
|
||||
- lagoon.OnDatabase closes the Boot-order gap; lagoon.Transaction/AfterCommit exist for plan 11-05.
|
||||
- READMEs of conga (new), lagoon and surf plus the root modules row are updated in the same commits; both repositories green.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-01-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -0,0 +1,242 @@
|
||||
---
|
||||
phase: 11-jobs-realtime-and-search-infrastructure
|
||||
plan: 02
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on: ["11-01"]
|
||||
files_modified:
|
||||
- modules/pact/capabilities.go
|
||||
- modules/pact/README.md
|
||||
- modules/bonfire/call.go
|
||||
- modules/bonfire/call_test.go
|
||||
- modules/bonfire/README.md
|
||||
- modules/conga/schedule.go
|
||||
- modules/conga/scheduler.go
|
||||
- modules/conga/worker.go
|
||||
- modules/conga/commands.go
|
||||
- modules/conga/schedule_test.go
|
||||
- modules/conga/README.md
|
||||
- internal/build/build.go
|
||||
- internal/build/build_test.go
|
||||
- cmd/summer/main.go
|
||||
- cmd/summer/runtime.go
|
||||
- cmd/summer/main_test.go
|
||||
- examples/hello/main.go
|
||||
- ../fonoteka.go/main.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/schedule.go
|
||||
autonomous: true
|
||||
requirements: [CLI-04]
|
||||
coupling_justified: ["11-03: both extend the River worker built by 11-01 (this plan adds periodic jobs and the scheduled queue inside conga; 11-03 registers the broadcast job through the conga.Manager API without editing conga files); disjoint files, order-independent"]
|
||||
estimate:
|
||||
tokens: 90000
|
||||
raw_tokens: 90000
|
||||
tasks: 2
|
||||
confidence: low
|
||||
must_haves:
|
||||
truths:
|
||||
- "Per D-18, plugins declare recurring commands through a new `pact.HasSchedule` (`Schedule() []pact.ScheduledCommand`) with `pact.Daily()`, `pact.DailyAt(hour, minute)` or `pact.Every(d)` cadences, and pact still imports no River code."
|
||||
- "Per D-18 and CLI-04, every worker (serve, queue:work) and `summer schedule:run` carries the same River PeriodicJobs built from all plugins' schedules; each scheduled run is a River job on the `scheduled` queue with MaxAttempts 1 and UniqueOpts.ByPeriod equal to the cadence period, whose worker runs the named command in-process through `bonfire.Call`."
|
||||
- "Per RESEARCH Pattern 6, `Daily` computes the next wall-clock hh:mm in the `app.timezone` location (default UTC) and `Every(d)` the next multiple of d since local midnight, so a restart never delays a daily run by up to 24h the way PeriodicInterval(24h) would."
|
||||
- "Per D-18, `schedule:run` runs a foreground scheduler-only River client (scheduled queue plus periodic jobs) until SIGINT/SIGTERM, and `schedule:run --once` runs, without River, every entry due in the current minute (Laravel schedule:run semantics for system cron), printing `Running scheduled command: <name>` per entry or `No scheduled commands are ready to run.`"
|
||||
- "Per D-18 and user decision 5, fonoteka declares `fonoteka:prune-notifications` daily; until Phase 14 registers that command, the scheduler logs a warning naming it and skips it, without failing boot or other entries."
|
||||
- "Edge (CLI-04 adjacency): `Daily(h, m).Next(t)` for t exactly at h:m:00 returns the next day's h:m, and `Every(d).Next(t)` for t exactly on a multiple of d returns t+d (strictly after)."
|
||||
- "Edge (CLI-04 empty): a plugin whose Schedule() returns nil or an empty slice adds no periodic job; an entry with an empty Command, a zero Cadence, or an Every interval that does not evenly divide 24h fails worker start with an error naming the plugin id and entry index."
|
||||
- "Edge (CLI-04 ordering): periodic jobs are registered in plugin activation order, then declaration order, with ids `<plugin id>#<index>:<command>`; entries sharing a cadence each get their own job and their relative execution order is unspecified."
|
||||
- "Edge (CLI-04 idempotency): two enqueues of one entry inside one cadence period collapse to one River job through UniqueOpts.ByPeriod; `schedule:run --once` invoked twice in the same minute runs the due entries twice, as Laravel does (no overlap lock)."
|
||||
- statement: "Edge (CLI-04 concurrency): with several worker processes, River leader election means exactly one process enqueues each period; a scheduled command interrupted mid-run is not retried (MaxAttempts 1) and the next period runs normally."
|
||||
verification: backstop
|
||||
- "The generated app main publishes the final command list as a `*bonfire.Catalog` on the app before executing, so scheduled runs can call any registered app command."
|
||||
artifacts:
|
||||
- path: "modules/pact/capabilities.go"
|
||||
provides: "HasSchedule, ScheduledCommand, Cadence, Daily, DailyAt, Every"
|
||||
contains: "type HasSchedule interface"
|
||||
- path: "modules/bonfire/call.go"
|
||||
provides: "Catalog, NewCatalog, Call, ErrUnknownCommand"
|
||||
contains: "func Call("
|
||||
- path: "modules/conga/schedule.go"
|
||||
provides: "river.PeriodicSchedule implementations for Daily and Every"
|
||||
- path: "modules/conga/scheduler.go"
|
||||
provides: "periodic job construction, ScheduledCommandArgs and its worker"
|
||||
contains: "ByPeriod"
|
||||
- path: "../fonoteka.go/plugins/golem15/fonoteka/schedule.go"
|
||||
provides: "fonoteka:prune-notifications daily entry"
|
||||
contains: "fonoteka:prune-notifications"
|
||||
key_links:
|
||||
- from: "modules/conga/worker.go"
|
||||
to: "modules/conga/scheduler.go"
|
||||
via: "StartWorker sets river.Config.PeriodicJobs from plugins' HasSchedule"
|
||||
pattern: "PeriodicJobs"
|
||||
- from: "modules/conga/scheduler.go"
|
||||
to: "modules/bonfire/call.go"
|
||||
via: "scheduled worker resolves *bonfire.Catalog from the app and calls the command"
|
||||
pattern: "Catalog"
|
||||
- from: "internal/build/build.go"
|
||||
to: "modules/bonfire/call.go"
|
||||
via: "generated main publishes bonfire.NewCatalog(commands)"
|
||||
pattern: "bonfire\\.NewCatalog"
|
||||
prohibitions:
|
||||
- requirement_id: CLI-04
|
||||
category: safety
|
||||
statement: "The scheduler MUST NOT execute a command name or argument list that does not come from a compiled pact.HasSchedule entry (never from config, database rows or request input)"
|
||||
status: resolved
|
||||
verification: test
|
||||
- requirement_id: CLI-04
|
||||
category: safety
|
||||
statement: "A scheduled entry for an unregistered command MUST NOT fail boot or stop other entries; it is skipped with a warning naming the command"
|
||||
status: resolved
|
||||
verification: test
|
||||
---
|
||||
|
||||
## Phase Goal
|
||||
|
||||
ROADMAP Phase 11 goal (verbatim, not in user-story form): River jobs run on the correct dual-driver split, Centrifugo publishing and channel authorization match the existing server, and Typesense sync stays a re-gated pre-filter — all brought up before the API phases that depend on them.
|
||||
|
||||
This plan's slice: a plugin declares a daily or interval command and it runs on schedule inside `summer serve`, in a dedicated `summer schedule:run` process, or from system cron with `--once` (CLI-04, ROADMAP SC-2).
|
||||
|
||||
<objective>
|
||||
Add the schedule capability to pact, a `bonfire.Call` in-process command runner, and a River-periodic-job scheduler in conga with `schedule:run`, then declare fonoteka's first schedule entry.
|
||||
|
||||
Purpose: Laravel's `registerSchedule` has no Go counterpart yet; Phase 14's prune-notifications command and later maintenance jobs need it. Decisions implemented: D-18; user decision 5 (unknown command warns and skips); RESEARCH Pattern 6 and its anti-pattern on PeriodicInterval(24h).
|
||||
Output: pact.HasSchedule and cadences, bonfire Catalog/Call, conga Daily/Every schedules and scheduler wiring, `schedule:run [--once]`, generated main catalog publish, `summer schedule:run` delegate, fonoteka schedule.go.
|
||||
|
||||
Repos: summercms.go (framework) and fonoteka.go (schedule.go, regenerated main.go). This plan does not edit fonoteka's plugin.go (the schedule lives in its own file so plan 11-03 can edit plugin.go in the same wave). Planning docs and code in separate commits. Never add co-author tags.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@~/.claude/gsd-core/workflows/execute-plan.md
|
||||
@~/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-01-SUMMARY.md
|
||||
@modules/pact/capabilities.go
|
||||
@modules/bonfire/root.go
|
||||
@modules/bonfire/command.go
|
||||
@modules/conga/worker.go
|
||||
@modules/conga/commands.go
|
||||
@internal/build/build.go
|
||||
|
||||
<interfaces>
|
||||
- From plan 11-01 (read its SUMMARY for the final names): `conga.From(app) (*Manager, error)`, `(*Manager).Register(jobs ...pact.Job) error`, `conga.Job[T pact.JobArgs](fn, opts...)`, `conga.OnQueue`, `conga.MaxAttempts`, `conga.StartWorker(ctx, app, plugins, WorkerOptions{Queues})`, `(*Worker).Stop(ctx)`, `conga.RuntimeCommands(app, plugins)`, unexported riverConfig/queue-set helpers in client.go.
|
||||
- modules/bonfire/root.go: `NewRoot(name, commands, out)`, `NewRootIO(name, commands, in, out, errW)` builds a cobra root; `validCommandName` requires `namespace:verb` except build/dev/serve/migrate.
|
||||
- modules/pact/capabilities.go:368-379: the "Future capability families" comment lists HasListeners and HasSchedule; the kernel type-assert list follows it.
|
||||
- River v0.47.0: `river.PeriodicSchedule` = `Next(current time.Time) time.Time`; `river.NewPeriodicJob(schedule, func() (river.JobArgs, *river.InsertOpts), &river.PeriodicJobOpts{ID string, RunOnStart bool})`; `river.Config.PeriodicJobs []*river.PeriodicJob`; `river.InsertOpts.UniqueOpts.ByPeriod time.Duration`. River's periodic enqueuer runs only on the elected leader and keeps only in-memory state (restarts reset it).
|
||||
- Laravel reference: `$schedule->command('fonoteka:prune-notifications')->daily()` in /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/Plugin.php:328-331 (daily = `0 0 * * *`); PHP config/app.php timezone is UTC.
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
## Artifacts this phase produces
|
||||
|
||||
(This plan's share.)
|
||||
|
||||
- pact: `HasSchedule` (`Schedule() []ScheduledCommand`), `ScheduledCommand{Command string; Args []string; Cadence Cadence}`, `Cadence` (opaque struct with `IsZero()`, `Interval() time.Duration`, `At() (hour, minute int, ok bool)`), constructors `Daily()`, `DailyAt(hour, minute int)`, `Every(d time.Duration)`.
|
||||
- bonfire: `Catalog`, `NewCatalog(commands []Command) *Catalog`, `(*Catalog).Has(name string) bool`, `(*Catalog).Call(ctx, name string, args []string, out io.Writer) error`, `Call(ctx, commands []Command, name string, args []string, out io.Writer) error`, `ErrUnknownCommand`.
|
||||
- conga: `Daily{Hour, Minute int; Loc *time.Location}` and `Every{Interval time.Duration; Loc *time.Location}` (both `Next(time.Time) time.Time`), `ScheduledCommandArgs{Entry, Command string; Args []string}` with Kind `summer.scheduled_command`, queue constant `QueueScheduled = "scheduled"`, the `schedule:run [--once]` command.
|
||||
- CLI: app command `schedule:run` (flag `once`, bare); `summer schedule:run` delegate forwarding `--once`.
|
||||
- Config key: `app.timezone` (read, default UTC).
|
||||
- Files: `modules/bonfire/call.go`, `modules/conga/schedule.go`, `modules/conga/scheduler.go`, `../fonoteka.go/plugins/golem15/fonoteka/schedule.go`.
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="tracer">
|
||||
<name>Task 1: A plugin's interval command runs on schedule inside a worker through bonfire.Call</name>
|
||||
<precondition>Plan 11-01 is executed: `go doc ./modules/conga StartWorker` exits 0 and `docker info` exits 0.</precondition>
|
||||
<files>modules/pact/capabilities.go, modules/pact/README.md, modules/bonfire/call.go, modules/bonfire/call_test.go, modules/bonfire/README.md, modules/conga/schedule.go, modules/conga/scheduler.go, modules/conga/worker.go, modules/conga/schedule_test.go, modules/conga/README.md, internal/build/build.go, internal/build/build_test.go, examples/hello/main.go, ../fonoteka.go/main.go</files>
|
||||
<read_first>modules/pact/capabilities.go (JobArgs/Job/HasJobs block and the closing "Future capability families" comment), modules/pact/README.md, modules/bonfire/root.go, modules/bonfire/command.go, modules/bonfire/README.md, modules/conga/worker.go, modules/conga/client.go, modules/conga/conga.go (Register), modules/conga/postgres_test.go (harness), internal/build/build.go (generateMain), internal/build/build_test.go, $(go env GOMODCACHE)/github.com/riverqueue/river@v0.47.0/periodic_job.go, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Pattern 6, Anti-Patterns, Code Examples "Daily periodic schedule")</read_first>
|
||||
<action>(1) pact (D-18): after HasJobs add `type Cadence struct` with unexported fields (kind, hour, minute, every) and exported constructors `Daily() Cadence` (00:00, Laravel daily), `DailyAt(hour, minute int) Cadence` and `Every(d time.Duration) Cadence`, plus accessors `IsZero() bool`, `Interval() time.Duration` (24h for daily cadences) and `At() (hour, minute int, ok bool)` (ok is false for an Every cadence); `type ScheduledCommand struct { Command string; Args []string; Cadence Cadence }`; `type HasSchedule interface { Schedule() []ScheduledCommand }`. Remove HasSchedule from the "Future capability families" comment and add "HasSchedule (conga workers)" to the kernel type-assert list. No River import in pact. README Features/API reference updated in the same commit.
|
||||
|
||||
(2) bonfire, new modules/bonfire/call.go: `var ErrUnknownCommand = errors.New("bonfire: unknown command")`; `func Call(ctx context.Context, commands []Command, name string, args []string, out io.Writer) error` finds the command by exact name (else `fmt.Errorf("%w: %q", ErrUnknownCommand, name)`), builds a root with `NewRootIO(name, []Command{cmd}, strings.NewReader(""), out, out)` (empty stdin so prompts take their defaults), sets args `append([]string{name}, args...)` and runs `ExecuteContext(ctx)`; `type Catalog struct` holding a copy of the commands, `NewCatalog(commands []Command) *Catalog`, `(*Catalog).Has(name) bool`, `(*Catalog).Call(ctx, name, args, out) error`. A smoke `TestCall` in call_test.go (a command receiving its args and writing to out; unknown name wraps ErrUnknownCommand). README updated.
|
||||
|
||||
(3) conga schedules, new modules/conga/schedule.go (RESEARCH Pattern 6): `type Daily struct { Hour, Minute int; Loc *time.Location }` whose Next returns the first `Hour:Minute:00` in Loc strictly after now (adding a day when not after, using time.Date so DST gaps resolve the way Go normalizes them); `type Every struct { Interval time.Duration; Loc *time.Location }` whose Next returns local midnight plus the smallest multiple of Interval strictly after now (rolling into the next day at midnight). A helper `scheduleFor(c pact.Cadence, loc *time.Location) (river.PeriodicSchedule, time.Duration, error)` returns the schedule and its period, rejecting a zero cadence, a non-positive interval and an interval that does not evenly divide 24h. Location comes from `app.timezone` (default "UTC", `time.LoadLocation`; an invalid name is a start error).
|
||||
|
||||
(4) conga scheduler, new modules/conga/scheduler.go: const `QueueScheduled = "scheduled"`; `type ScheduledCommandArgs struct { Entry string; Command string; Args []string }` with `Kind() "summer.scheduled_command"`. `periodicJobs(app, plugins) ([]*river.PeriodicJob, map[string]pact.ScheduledCommand, error)` walks plugins in activation order and each `pact.HasSchedule` entry in declaration order; entry id `<plugin id>#<index>:<command>`; an empty Command or invalid cadence is an error naming plugin id and index; each periodic job's constructor returns the args plus `&river.InsertOpts{Queue: QueueScheduled, MaxAttempts: 1, UniqueOpts: river.UniqueOpts{ByPeriod: period}}` with `&river.PeriodicJobOpts{ID: entryID}`. The built-in job (registered once through `conga.Job` on QueueScheduled, MaxAttempts 1) looks the Entry up in the compiled table and runs only when Command and Args match it exactly (T-11-09); it resolves `*bonfire.Catalog` from the app (missing catalog: warn and skip), and on `errors.Is(err, bonfire.ErrUnknownCommand)` logs Warn `schedule: command not registered; skipping` with the command attribute and returns nil (user decision 5); other command errors are logged with duration and returned (no retry because MaxAttempts is 1). Command output goes to an io.Writer that logs each line at Info with the command attribute.
|
||||
|
||||
(5) Wiring: in modules/conga/worker.go StartWorker registers the built-in job, adds QueueScheduled to the known queues (MaxWorkers from `queue.queues.scheduled`, default 1) and sets `river.Config.PeriodicJobs` from periodicJobs for every worker regardless of the queue filter (only the elected leader enqueues). In internal/build/build.go the generated main, after collecting plugin commands and before `bonfire.NewRoot`, emits `if err := app.Publish(bonfire.NewCatalog(commands)); err != nil { return err }`; add `TestGenerateMainPublishesCommandCatalog` to build_test.go; regenerate examples/hello/main.go and ../fonoteka.go/main.go with the framework CLI.
|
||||
|
||||
(6) Smoke test modules/conga/schedule_test.go: `TestScheduleNext` (Daily and Every adjacency: exactly-on-boundary returns the next occurrence; DST day in Europe/Warsaw keeps wall-clock hh:mm) and `TestScheduleRunsCommand` (Postgres harness: an acme test plugin declaring `acme:tick` with `pact.Every(time.Second)`, a catalog holding an `acme:tick` command that signals a channel, StartWorker with all queues, assert the command runs within 10s; a second entry `acme:missing` produces the skip warning in a captured slog handler). Document the scheduler in modules/conga/README.md (Usage, API reference, CLI commands).</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/pact ./modules/bonfire ./internal/build -count=1 && go test ./modules/bonfire -run '^TestCall$' -count=1 -v && go test ./modules/conga -run '^(TestScheduleNext|TestScheduleRunsCommand)$' -count=1 -v && go test ./internal/build -run '^TestGenerateMainPublishesCommandCatalog$' -count=1 -v</automated>
|
||||
<fails_when>Any command exits non-zero; a verbose run lacks "--- PASS" for TestCall, TestScheduleNext, TestScheduleRunsCommand or TestGenerateMainPublishesCommandCatalog, or prints "no tests to run" or "--- SKIP".</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `go doc ./modules/pact HasSchedule`, `go doc ./modules/pact ScheduledCommand`, `go doc ./modules/pact Every`, `go doc ./modules/bonfire Call` and `go doc ./modules/conga Daily` exit 0.
|
||||
- `grep -c 'riverqueue' modules/pact/capabilities.go` prints 0.
|
||||
- `grep -c 'ByPeriod' modules/conga/scheduler.go` prints at least 1 and `grep -c 'PeriodicJobs' modules/conga/worker.go` prints at least 1.
|
||||
- `grep -c 'bonfire.NewCatalog(commands)' ../fonoteka.go/main.go examples/hello/main.go` prints 1 for each file.
|
||||
- `grep -c 'HasSchedule' modules/pact/README.md` prints at least 1 and `grep -c 'Catalog' modules/bonfire/README.md` prints at least 1.
|
||||
</acceptance_criteria>
|
||||
<done>A plugin declares `Every(1s)` and its command runs through a River periodic job and bonfire.Call inside a worker, unknown commands are skipped with a warning, and every generated main publishes the command catalog.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: schedule:run works as a dedicated process and from system cron, and fonoteka declares its daily prune entry</name>
|
||||
<files>modules/conga/commands.go, modules/conga/scheduler.go, modules/conga/schedule_test.go, modules/conga/README.md, cmd/summer/main.go, cmd/summer/runtime.go, cmd/summer/main_test.go, ../fonoteka.go/plugins/golem15/fonoteka/schedule.go</files>
|
||||
<read_first>modules/conga/commands.go (queue:work signal loop), modules/conga/scheduler.go (Task 1), cmd/summer/runtime.go (delegateRollbackCommand, delegateQueueWorkCommand from 11-01), cmd/summer/main_test.go, ../fonoteka.go/plugins/golem15/fonoteka/plugin.go (Plugin type and compile-time assertion list, read only), /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/Plugin.php (registerSchedule), .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Pattern 6 "--once", Open Question 5)</read_first>
|
||||
<behavior>
|
||||
- `schedule:run --once` at 00:00 app time runs a `Daily()` entry and prints `Running scheduled command: acme:tick`; at 00:01 it prints `No scheduled commands are ready to run.`
|
||||
- `schedule:run --once` with an `Every(5*time.Minute)` entry runs it at 10:05 and not at 10:07.
|
||||
- `schedule:run --once` with an unregistered command prints a warning naming it, exits 0 and still runs the other due entries.
|
||||
- A ScheduledCommandArgs whose Command or Args differ from its compiled Entry is logged and skipped without running anything.
|
||||
- Two inserts of the same periodic entry inside one period leave one river_job row (ByPeriod).
|
||||
</behavior>
|
||||
<action>(1) `schedule:run` in modules/conga/commands.go (D-18), added to RuntimeCommands: description "Run the scheduler in the foreground (or once for system cron)", bare flag `once`. Without `--once`: open the DB the conga way, start `StartWorker` with `WorkerOptions{Queues: []string{QueueScheduled}}` (the scheduled queue plus periodic jobs), print `scheduler started`, block on the SIGINT/SIGTERM loop, stop with a 10s timeout. With `--once`: no River; compute the current minute in `app.timezone` through an injectable clock (unexported, for tests), select entries whose cadence is due that minute (Daily: hour and minute match; Every(d): d at or below one minute is due every run, otherwise minutes since local midnight times 60s is a multiple of d), run each in order through the app's `*bonfire.Catalog` printing `Running scheduled command: <command> <args...>`; an unknown command prints and logs the skip warning and continues; none due prints `No scheduled commands are ready to run.`; the exit status is the first command error, after all due entries ran.
|
||||
|
||||
(2) `summer` delegate: in cmd/summer/runtime.go add `delegateScheduleRunCommand()` declaring the bare `once` flag and forwarding `--once` when set; register it in cmd/summer/main.go toolCommands and extend the expected list in main_test.go.
|
||||
|
||||
(3) fonoteka.go (D-18, user decision 5): new plugins/golem15/fonoteka/schedule.go with `var _ pact.HasSchedule = (*Plugin)(nil)` and `func (p *Plugin) Schedule() []pact.ScheduledCommand { return []pact.ScheduledCommand{{Command: "fonoteka:prune-notifications", Cadence: pact.Daily()}} }` and a comment that the command itself ships in Phase 14 and is skipped with a warning until then. Do not edit plugin.go.
|
||||
|
||||
(4) Tests for the behavior list in modules/conga/schedule_test.go (`TestScheduleRunOnce`, `TestScheduledEntryMismatchSkipped`, `TestScheduleUniqueByPeriod`); the ByPeriod test inserts the same constructed args twice through the periodic constructor within one period and counts river_job rows. Update modules/conga/README.md CLI commands with `schedule:run` and `--once`.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/conga -run '^(TestScheduleRunOnce|TestScheduledEntryMismatchSkipped|TestScheduleUniqueByPeriod)$' -count=1 -v && go test ./cmd/summer -count=1 && go test ./... && (cd ../fonoteka.go && go vet ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/... && go test ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/...)</automated>
|
||||
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS" for TestScheduleRunOnce, TestScheduledEntryMismatchSkipped or TestScheduleUniqueByPeriod, or prints "no tests to run" or "--- SKIP"; any package reports FAIL.</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `grep -c '"schedule:run"' modules/conga/commands.go` prints at least 1 and `grep -c 'schedule:run' cmd/summer/main_test.go` prints at least 1.
|
||||
- `grep -c 'No scheduled commands are ready to run.' modules/conga/commands.go` prints 1.
|
||||
- `grep -c 'fonoteka:prune-notifications' ../fonoteka.go/plugins/golem15/fonoteka/schedule.go` prints 1 and `grep -c 'pact.Daily()' ../fonoteka.go/plugins/golem15/fonoteka/schedule.go` prints 1.
|
||||
- `(cd ../fonoteka.go && SUMMER_GOLEM15__USER__JWT__SECRET=test-only-cli-secret go run . schedule:run --help)` output contains "--once" (the app refuses to boot without a user JWT secret, so a test-only value is passed).
|
||||
</acceptance_criteria>
|
||||
<done>`schedule:run` runs as a scheduler-only process or once per cron minute, fonoteka declares its daily prune entry (skipped with a warning until Phase 14), and both repositories pass their full suites.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| river_job rows (database) → scheduled-command worker | Job args stored in Postgres name a command to execute in-process |
|
||||
| System cron → `schedule:run --once` | An external trigger runs due commands |
|
||||
| Go module proxy → go.mod/go.sum | No new module in this plan |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-11-09 | Tampering | scheduled-command worker | high | mitigate | Command names come only from compiled pact.HasSchedule entries; the worker runs a job only when its Entry exists in the compiled table and Command/Args match exactly, so a forged river_job row cannot run an arbitrary command (Tasks 1-2, TestScheduledEntryMismatchSkipped). |
|
||||
| T-11-17 | Denial of Service | periodic enqueue on leader failover | low | mitigate | UniqueOpts.ByPeriod dedupes an entry within its period; wall-clock Daily/Every schedules avoid the restart drift of PeriodicInterval (Task 1-2). |
|
||||
| T-11-18 | Repudiation | scheduled runs | low | mitigate | Each run, skip and failure is logged with the command name and duration; skips of unregistered commands are Warn level (Task 1). |
|
||||
| T-11-SC | Tampering | Go module installs | high | mitigate | No dependency added; River stays pinned at v0.47.0 from plan 11-01; cron parsing is not added because Daily/Every cover the cadences. |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
After Task 2: `go vet ./... && go test ./...` in summercms.go and the fonoteka.go full vet/test command pass; `go test ./modules/conga -run 'TestSchedule' -count=1 -v` passes; `fonoteka schedule:run --once` prints either a Running line or the no-commands line and exits 0.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- pact.HasSchedule exists with Daily/DailyAt/Every cadences and no River import.
|
||||
- Every worker carries the periodic jobs; scheduled runs go through bonfire.Call with ByPeriod dedupe and MaxAttempts 1.
|
||||
- `schedule:run` runs in the foreground or `--once`; unknown commands warn and skip.
|
||||
- fonoteka declares `fonoteka:prune-notifications` daily in schedule.go.
|
||||
- pact, bonfire and conga READMEs updated in the same commits.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-02-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -0,0 +1,354 @@
|
||||
---
|
||||
phase: 11-jobs-realtime-and-search-infrastructure
|
||||
plan: 03
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on: ["11-01"]
|
||||
files_modified:
|
||||
- modules/lighthouse/lighthouse.go
|
||||
- modules/lighthouse/drivers.go
|
||||
- modules/lighthouse/route.go
|
||||
- modules/lighthouse/channel.go
|
||||
- modules/lighthouse/registry.go
|
||||
- modules/lighthouse/users.go
|
||||
- modules/lighthouse/broadcast.go
|
||||
- modules/lighthouse/suppress.go
|
||||
- modules/lighthouse/job.go
|
||||
- modules/lighthouse/README.md
|
||||
- modules/lighthouse/centrifugo/config.go
|
||||
- modules/lighthouse/centrifugo/client.go
|
||||
- modules/lighthouse/centrifugo/token.go
|
||||
- modules/lighthouse/centrifugo/handlers.go
|
||||
- modules/lighthouse/centrifugo/driver.go
|
||||
- README.md
|
||||
- .planning/PROJECT.md
|
||||
- .planning/REQUIREMENTS.md
|
||||
- ../fonoteka.go/config/realtime.yaml
|
||||
- ../fonoteka.go/README.md
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/plugin.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/routes.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/realtime.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/classes/ws/collection_authorizer.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/classes/ws/wishlist_authorizer.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/go.mod
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/go.sum
|
||||
autonomous: true
|
||||
requirements: [RT-01, RT-02, RT-03]
|
||||
coupling_justified: ["11-02: this plan registers the broadcast job through the conga.Manager API from 11-01 while 11-02 adds periodic jobs inside conga; no shared files, order-independent"]
|
||||
estimate:
|
||||
tokens: 180000
|
||||
raw_tokens: 180000
|
||||
tasks: 3
|
||||
confidence: low
|
||||
must_haves:
|
||||
truths:
|
||||
- "Per D-11, `modules/lighthouse` is transport-neutral: it owns the Publisher/Driver interfaces, the channel-namespace authorizer Registry, channel rules, Route/Surface/Mount, the Broadcastable contract with WithoutBroadcasting and Emit, and the broadcast River job; drivers are chosen by `realtime.driver` (centrifugo, memory, log, null; default null) and models and authorizers never import a driver package."
|
||||
- "Per D-12 and RT-01, `GET /api/realtime/token` behind `jwt.auth` returns 200 `{\"token\":...}` signed HS256 with `realtime.centrifugo.token_secret` whose decoded claims are exactly `sub` (the user id as a string), `exp` (now + token_ttl, default 3600) and `info` = {\"name\": the user's name or null}; with an empty token_secret it returns 503 `{\"error\":\"WebSocket not configured\"}`, and without a resolvable user 401 `{\"error\":\"Unauthorized\"}`."
|
||||
- "Per D-12, the token issuer ports all five PHP JwtTokenGenerator generators (ForUser, Subscription, Anonymous with exp now+300 and sub \"\", ForIdentifier with info encoded as `[]` when empty, SubscriptionForIdentifier) and refuses to sign with an empty secret."
|
||||
- "Per D-12 and RT-02, `POST /api/realtime/subscribe` always answers HTTP 200: an empty or mismatched `X-Centrifugo-Secret` (constant-time compare; an empty configured proxy_secret denies), an empty or \"0\" user, a double `presence:` prefix, more than three segments, an unknown namespace or an authorizer denial all return exactly `{\"error\":{\"code\":403,\"message\":\"Access denied\"}}` with the reason logged only; an allow returns `{\"result\":{\"info\":[]}}` (object info when non-empty) and, for a `presence:` channel, also `allow` (default [\"prs\"]) and `override` merging presence/join_leave true and force_push_join_leave false with the authorizer's overrides."
|
||||
- "Per D-13, routes are declared by the driver as `lighthouse.Route{Name, Method, Path, Surface, Handler}` and mounted by the app with one `lighthouse.Mount(r, driver, lighthouse.Surfaces{UserAuth, ServerToServer, Public, Middleware})` call; fonoteka mounts the token route with `jwt.auth` before `throttle:ws-api` (user decision 5) and the subscribe route in a raw group with `throttle:ws-api`; Mount refuses a UserAuth route when the UserAuth surface is empty."
|
||||
- "Per D-14, fonoteka registers the `ws-api` bucket: 120 requests per minute keyed by the authenticated user id, else by the trusted-proxy client IP."
|
||||
- "Per D-16, fonoteka registers `collection` and `wishlist` authorizers ported from CollectionChannelAuthorizer (owner or editor of a kind=collection collection) and WishlistChannelAuthorizer (a wishlist_subscriptions row, or a kind=wishlist collection owned by a household peer), both parsing the id segment with PHP (int)-cast semantics; there is no separate websockets plugin and PROJECT.md records that deviation."
|
||||
- "Per D-06, D-07 and RT-03, a create, update or delete of a broadcastable model enqueues one River job on the `realtime.broadcast_queue` queue (default `broadcasts`) inside the write transaction, with MaxAttempts 1 and timeout `realtime.broadcast_timeout` (default 5s); nothing is published when the write rolls back; a failed publish or a failed enqueue is logged as a warning and never fails the write."
|
||||
- "Per D-08, `lighthouse.WithoutBroadcasting[T](ctx, fn)` silences only T's broadcasts for writes made with the ctx handed to fn (other types still broadcast, a stale outer ctx is not suppressed), and `(*lighthouse.Service).Emit` enqueues one explicit summary event in the same transaction, so N album creates under suppression plus one Emit publish exactly one `collection.bulk_updated`."
|
||||
- "Per D-09, the event name is `{action}.{alias}` lowercased (alias defaults to `<plugin>.<model>` from the Go package path and can be overridden), the default payload is `{model, actor, timestamp, ttl}` with ttl 60, a delete snapshot is taken before the row is deleted, `ShouldBroadcast(action)` can veto, one channel uses Centrifugo publish and several use broadcast, and channels are lowercased and prefixed with `realtime.broadcast_namespace` unless already prefixed."
|
||||
- "Per D-12, the Centrifugo HTTP client POSTs `{api_url}/publish` with `{\"channel\":c,\"data\":{\"event\":e,\"payload\":p,\"timestamp\":\"...+00:00\"}}`, `/broadcast` with `channels`, `/presence` and `/unsubscribe` with header `Authorization: apikey <api_key>` and a 5s timeout; success is any 2xx; with an empty api_key it sends nothing."
|
||||
- "Album broadcasts match PHP Album overrides: channels `[\"collection:<id>\"]` only when the album's collection has kind `collection` (else none), payload `{id, collection_id, action, actor, timestamp}` plus `album` (the current classes.SerializeAlbum of the reloaded row) except for deleted, actor `{user_id, name}` from the frontend principal or `{user_id: null, name: \"System\"}` for no principal or a backend admin."
|
||||
- "Edge (RT-01 concurrency): concurrent token requests for one user each get an independently signed token with no shared mutable state (race detector clean)."
|
||||
- statement: "Edge (RT-01 concurrency): delivery order across separate broadcast jobs is not guaranteed, as with PHP's queued BroadcastEventJob."
|
||||
verification: backstop
|
||||
- "Edge (RT-02 adjacency): a two-segment `collection:5` and a three-segment `ns:entity:id` channel reach their authorizer, four segments deny, one `presence:` prefix is stripped before the namespace lookup while the authorizer receives the full original channel, and `presence:presence:` denies."
|
||||
- "Edge (RT-02 empty): a missing or empty user, user \"0\", a missing or empty channel, and an empty namespace all return the generic HTTP 200 deny (a missing channel, which Centrifugo never sends, is denied instead of PHP's TypeError 500 and is recorded as deliberate hardening)."
|
||||
- "Edge (RT-02 encoding): namespace lookup is byte-exact and case-sensitive with no lowercasing or Unicode normalization, as in PHP; the id segment is parsed with PHP (int)-cast semantics confirmed with `php -r` (for example `5abc` is 5 and `abc` is 0, which denies)."
|
||||
- "Edge (RT-02 ordering): registering a namespace twice is a boot error (PHP's last-wins registry is not reproduced; fail-loud like duplicate middleware names) and `Registry.Namespaces()` returns a sorted list."
|
||||
- "Edge (RT-02 idempotency): every subscribe re-runs the authorizer against current database state with no caching, so a user removed as editor is denied on the next subscribe."
|
||||
- "Edge (RT-02 concurrency): concurrent subscribe requests are handled independently and the registry is safe for concurrent reads (race detector clean)."
|
||||
artifacts:
|
||||
- path: "modules/lighthouse/lighthouse.go"
|
||||
provides: "Service, From, config reading, callback installation"
|
||||
contains: "func From("
|
||||
- path: "modules/lighthouse/route.go"
|
||||
provides: "Route, Surface, Surfaces, Mount"
|
||||
contains: "func Mount("
|
||||
- path: "modules/lighthouse/registry.go"
|
||||
provides: "Authorizer, AuthorizerFunc, Result, Allowed, Denied, Registry"
|
||||
- path: "modules/lighthouse/broadcast.go"
|
||||
provides: "Broadcastable contract, Binding, Bind, GORM callbacks"
|
||||
- path: "modules/lighthouse/suppress.go"
|
||||
provides: "WithoutBroadcasting, Broadcast, Service.Emit"
|
||||
contains: "func WithoutBroadcasting"
|
||||
- path: "modules/lighthouse/centrifugo/handlers.go"
|
||||
provides: "TokenHandler and ProxyHandler"
|
||||
contains: "Access denied"
|
||||
- path: "modules/lighthouse/centrifugo/token.go"
|
||||
provides: "TokenIssuer with five generators"
|
||||
- path: "../fonoteka.go/plugins/golem15/fonoteka/realtime.go"
|
||||
provides: "lighthouse wiring, user lookup, Album binding, authorizer registration"
|
||||
- path: "../fonoteka.go/config/realtime.yaml"
|
||||
provides: "realtime.driver and centrifugo settings with PHP defaults"
|
||||
key_links:
|
||||
- from: "../fonoteka.go/plugins/golem15/fonoteka/routes.go"
|
||||
to: "modules/lighthouse/route.go"
|
||||
via: "lighthouse.Mount with jwt.auth then throttle:ws-api"
|
||||
pattern: "lighthouse\\.Mount"
|
||||
- from: "modules/lighthouse/broadcast.go"
|
||||
to: "modules/conga/conga.go"
|
||||
via: "Enqueue on the write's *sql.Tx inside a savepoint"
|
||||
pattern: "Enqueue\\("
|
||||
- from: "modules/lighthouse/job.go"
|
||||
to: "modules/lighthouse/centrifugo/client.go"
|
||||
via: "broadcast worker calls Driver.Publish or Driver.Broadcast"
|
||||
pattern: "Broadcast\\(|Publish\\("
|
||||
- from: "modules/lighthouse/centrifugo/handlers.go"
|
||||
to: "modules/lighthouse/registry.go"
|
||||
via: "proxy looks up the namespace authorizer on every subscribe"
|
||||
pattern: "Registry\\(\\)|\\.Get\\("
|
||||
- from: "../fonoteka.go/plugins/golem15/fonoteka/realtime.go"
|
||||
to: "../fonoteka.go/plugins/golem15/fonoteka/classes/ws/collection_authorizer.go"
|
||||
via: "Registry().Register(\"collection\", ...)"
|
||||
pattern: "Register\\(\"collection\""
|
||||
prohibitions:
|
||||
- requirement_id: RT-01
|
||||
category: privacy
|
||||
statement: "The connection token info claim MUST NOT carry anything but the display name (no email, no ids beyond sub), keeping the PHP WS-004 rule"
|
||||
status: resolved
|
||||
verification: test
|
||||
- requirement_id: RT-02
|
||||
category: privacy
|
||||
statement: "A denied subscribe MUST NOT reveal the internal reason, the namespace's existence or the user's membership in its response; the body is always the generic Access denied"
|
||||
status: resolved
|
||||
verification: test
|
||||
- requirement_id: RT-03
|
||||
category: safety
|
||||
statement: "A model broadcast MUST NOT be published for a write that rolled back"
|
||||
status: resolved
|
||||
verification: test
|
||||
- requirement_id: RT-03
|
||||
category: privacy
|
||||
statement: "An album change MUST NOT be broadcast to any channel other than its own kind=collection collection channel; wishlist albums publish nothing"
|
||||
status: resolved
|
||||
verification: test
|
||||
---
|
||||
|
||||
## Phase Goal
|
||||
|
||||
ROADMAP Phase 11 goal (verbatim, not in user-story form): River jobs run on the correct dual-driver split, Centrifugo publishing and channel authorization match the existing server, and Typesense sync stays a re-gated pre-filter — all brought up before the API phases that depend on them.
|
||||
|
||||
This plan's slice: the Nuxt app can fetch a Centrifugo connection token from `GET /api/realtime/token`, Centrifugo's subscribe proxy is re-authorized on every subscribe, and album changes are published only after their write commits (RT-01, RT-02, RT-03; ROADMAP SC-3 and SC-4).
|
||||
|
||||
<objective>
|
||||
Create the transport-neutral realtime package `lighthouse` with its Centrifugo driver sub-package, and bind Płytarium to it in fonoteka.go: config, authorizers, the Mount call, the ws-api bucket and the Album broadcast binding.
|
||||
|
||||
Purpose: notifications (Phase 13) and the Albums API (Phase 12) publish through this; the Nuxt client and Centrifugo stay unchanged. Decisions implemented: D-06, D-07, D-08, D-09, D-11, D-12, D-13, D-14, D-16; user decision 5 (jwt.auth before throttle:ws-api); RESEARCH Patterns 5, 7, 8, 9 and Pitfalls 7-11, 15.
|
||||
Output: modules/lighthouse (+ centrifugo) with README, root README row, fonoteka.go realtime wiring and smoke tests, PROJECT.md D-16 note and the RT-01 requirement note.
|
||||
|
||||
Repos: summercms.go (framework, planning docs) and fonoteka.go (application). Framework text and tests use neutral names (acme, blog); the application names stay in fonoteka.go. Planning docs and code in separate commits. Never add co-author tags.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@~/.claude/gsd-core/workflows/execute-plan.md
|
||||
@~/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-PATTERNS.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-01-SUMMARY.md
|
||||
@modules/postcard/mailer.go
|
||||
@modules/postcard/drivers.go
|
||||
@modules/wristband/server.go
|
||||
@modules/wire/response.go
|
||||
@modules/bouncer/context.go
|
||||
@modules/pact/capabilities.go
|
||||
@modules/surf/limiter.go
|
||||
@../fonoteka.go/plugins/golem15/fonoteka/plugin.go
|
||||
@../fonoteka.go/plugins/golem15/fonoteka/routes.go
|
||||
|
||||
<interfaces>
|
||||
- From plan 11-01 (see its SUMMARY): `conga.From(app) (*conga.Manager, error)`, `(*Manager).Register(jobs ...pact.Job) error` (must run before the first client is built, i.e. at Boot), `(*Manager).Enqueue(ctx, db *gorm.DB, args pact.JobArgs, conga.EnqueueOpts{Queue, Delay, MaxAttempts}) error` (InsertTx on db's *sql.Tx when in a transaction), `conga.Job[T](fn, conga.OnQueue(q), conga.MaxAttempts(1), conga.Timeout(d))`, `lagoon.OnDatabase(app, func(*sql.DB, *gorm.DB) error)`.
|
||||
- pact.Router: `Group(prefix, mw []string, fn)`, `GroupRaw(prefix, mw, fn)` (raw groups refuse house-tagged middleware only), `Get/Post(path, http.HandlerFunc, mw ...string)`; surf joinPath accepts `"/"` prefix plus an absolute path; `surf.Use(names...)`.
|
||||
- surf: `type Bucket struct { Max int; Decay time.Duration; Key func(*http.Request) string }`, `surf.BucketProvider` (fonoteka Plugin.Buckets), `surf.ClientIP(r, trusted)`, `surf.TrustedProxies(cfg)`.
|
||||
- bouncer: `User(ctx) (*Principal, bool)`; Principal.ID uint, Principal.Backend bool. `jwt.auth` is the Phase 6/7 guard name and writes the PHP jwt.auth 401 bodies itself.
|
||||
- wire: `WriteJSON(w, status, v)` (no trailing newline, no HTML escaping), `wire.Time` (Carbon `+00:00`).
|
||||
- wristband/server.go:180-199 `writeExactJSON` is the raw-JSON writer precedent.
|
||||
- postcard/mailer.go:85-134 driver selection by config key; drivers.go MemoryDriver (mutex + copying accessor), LogDriver.
|
||||
- fonoteka: `classes.AccessibleByMembership(userID uint) func(*gorm.DB) *gorm.DB` (owner OR editor predicate), `models.Collection{ID, OwnerID, Kind, DeletedAt}`, `models.WishlistSubscription{UserID, CollectionID}`, `models.Album{ID, CollectionID, ...}`, `classes.SerializeAlbum(*models.Album) map[string]any` (minimal until Phase 12), user plugin `usermodels.User{ID uint; Name *string}`; Boot already resolves *gorm.DB lazily per call (lazyInvTokenGuard precedent) because Boot runs before serve publishes the DB.
|
||||
- PHP contract files: /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/{routes.php, config/config.php, classes/JwtTokenGenerator.php, classes/CentrifugoClient.php, classes/AuthorizerRegistry.php, classes/AuthorizationResult.php, http/controllers/ProxyController.php, traits/BroadcastableModel.php, jobs/BroadcastEventJob.php}; /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/{classes/ws/CollectionChannelAuthorizer.php, classes/ws/WishlistChannelAuthorizer.php, models/Album.php (getBroadcastChannels, getBroadcastPayload, getActorMetadata), models/Collection.php (scopeWishlistsVisibleTo), controllers/api/AlbumApiController.php (bulk and publishAlbumBroadcast)}; /media/nvme/dev/golem15/fonoteka/vue-fonoteka-app/app/composables/useCentrifugo.ts.
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
## Artifacts this phase produces
|
||||
|
||||
(This plan's share.)
|
||||
|
||||
- lighthouse: `Service`, `From(app) (*Service, error)`, `(*Service).Driver()`, `(*Service).Registry()`, `(*Service).SetUserLookup(UserLookup)`, `(*Service).User(ctx, id)`, `(*Service).Actor(ctx) Actor`, `(*Service).Emit(ctx, db, Broadcast) error`; `Publisher` (`Publish`, `Broadcast`), `Driver` (`Publisher` + `Name()` + `Routes()`), `DriverFactory`, `RegisterDriver(name, DriverFactory)`, built-in drivers `memory` (`MemoryDriver`, `Publication`, `(*MemoryDriver).Publications()`), `log`, `null`; `Route{Name, Method, Path, Surface, Handler}`, `Surface` with `UserAuth`, `ServerToServer`, `Public`, `Surfaces{UserAuth, ServerToServer, Public, Middleware []string}`, `Mount(r pact.Router, d Driver, s Surfaces) error`; `ParseChannel`, `FormatChannels`, `ChannelID` (PHP (int) cast of segment 1), `ClientID(ctx)`; `Authorizer`, `AuthorizerFunc`, `Result` (`Allowed`, `Info`, `Capabilities`, `Overrides`, `Reason()`), `Allowed(info)`, `Denied(reason)`, `Registry` (`Register`, `Get`, `Namespaces`); `User{ID uint; Name *string}`, `UserLookup`, `Actor{UserID *uint; Name *string}`, `SystemActor()`; `Action` (`ActionCreated`, `ActionUpdated`, `ActionDeleted`), `Event{Action, Actor, Timestamp, TTL}`, `Broadcastable`, `BroadcastPayloader`, `BroadcastAliaser`, `BroadcastFilter`, `BroadcastTTLer`, `Binding[T]`, `Bind[T](svc, Binding[T]) error`, `WithoutBroadcasting[T](ctx, fn) error`, `Broadcast{Channels, Event, Payload}`, `BroadcastArgs` (Kind `summer.broadcast`); GORM callback names `lighthouse:snapshot`, `lighthouse:after_create`, `lighthouse:after_update`, `lighthouse:after_delete`.
|
||||
- lighthouse/centrifugo: `Config` (from `realtime.centrifugo.*`), `Client` (`Publish`, `Broadcast`, `Presence`, `Unsubscribe`, `Enabled`, `DebugInfo`), `DebugInfo`, `TokenIssuer` (`ForUser`, `Subscription`, `Anonymous`, `ForIdentifier`, `SubscriptionForIdentifier`, `Configured`), `Driver`, `TokenHandler`, `ProxyHandler`, `ErrNotConfigured`; driver name `centrifugo` registered in init.
|
||||
- Routes (fonoteka): `GET /api/realtime/token` (jwt.auth, throttle:ws-api), `POST /api/realtime/subscribe` (raw, throttle:ws-api).
|
||||
- Config keys: `realtime.driver`, `realtime.broadcast_namespace`, `realtime.broadcast_queue`, `realtime.broadcast_timeout`, `realtime.centrifugo.api_url` (default `http://127.0.0.1:8001/api`), `.api_key`, `.token_secret`, `.token_ttl` (3600), `.ws_url` (`/ws`), `.proxy_secret`, `.token_path` (`/api/realtime/token`), `.subscribe_path` (`/api/realtime/subscribe`).
|
||||
- fonoteka: bucket `ws-api`; `ws.CollectionAuthorizer`, `ws.WishlistAuthorizer` (package `classes/ws`); files `realtime.go`, `config/realtime.yaml`.
|
||||
|
||||
## Flagged assumptions (edge probe: unclassified)
|
||||
|
||||
- RT-03 came back `unclassified` from the spec-less edge probe and stays `unresolved`. Planner reading for manual review: batch updates through `Model(&T{}).Where(...)` carry a zero primary key and are skipped (Pitfall 8), so bulk paths must use WithoutBroadcasting plus Emit; a soft delete is a delete for broadcasting; a restore is not broadcast (PHP has no afterRestore hook in getBroadcastHooks).
|
||||
- The Album `created`/`updated` payload is built inside the write transaction by the GORM callback; SaveAlbum syncs artist pivots after `tx.Save`, so the automatic created payload can carry stale artists. PHP's store/update suppress automatic broadcasts and publish one explicit event after the write; Phase 12's controllers must follow that pattern (WithoutBroadcasting + Emit), which this plan makes expressible.
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="tracer">
|
||||
<name>Task 1: The Nuxt token request reaches a Centrifugo token through the neutral realtime package and the app's Mount</name>
|
||||
<reversibility rating="costly">D-11 and D-13: every broadcastable model, authorizer and driver is written against the lighthouse interfaces and the Route/Surface mount contract; the user locked both in CONTEXT.md, so the weight is flagged without a checkpoint.</reversibility>
|
||||
<precondition>Plan 11-01 is executed: `go doc ./modules/conga Manager.Enqueue` and `go doc ./modules/lagoon OnDatabase` exit 0.</precondition>
|
||||
<files>modules/lighthouse/lighthouse.go, modules/lighthouse/drivers.go, modules/lighthouse/route.go, modules/lighthouse/users.go, modules/lighthouse/README.md, modules/lighthouse/centrifugo/config.go, modules/lighthouse/centrifugo/client.go, modules/lighthouse/centrifugo/token.go, modules/lighthouse/centrifugo/handlers.go, modules/lighthouse/centrifugo/driver.go, README.md, ../fonoteka.go/config/realtime.yaml, ../fonoteka.go/README.md, ../fonoteka.go/plugins/golem15/fonoteka/plugin.go, ../fonoteka.go/plugins/golem15/fonoteka/routes.go, ../fonoteka.go/plugins/golem15/fonoteka/realtime.go, ../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go, ../fonoteka.go/plugins/golem15/fonoteka/go.mod, ../fonoteka.go/plugins/golem15/fonoteka/go.sum</files>
|
||||
<read_first>modules/postcard/mailer.go, modules/postcard/drivers.go, modules/wristband/server.go (lines 170-200), modules/wristband/token.go (golang-jwt usage), modules/wire/response.go, modules/bouncer/context.go, modules/pact/capabilities.go (Router), modules/surf/router.go (Group, GroupRaw, joinPath), modules/surf/limiter.go (Bucket, BucketProvider), ../fonoteka.go/plugins/golem15/fonoteka/plugin.go (Boot, Buckets), ../fonoteka.go/plugins/golem15/fonoteka/routes.go, ../fonoteka.go/plugins/golem15/fonoteka/plugin_boot_test.go (bootConfig, bootDB), ../fonoteka.go/plugins/golem15/user/models/user.go, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/routes.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/config/config.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/classes/JwtTokenGenerator.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/classes/CentrifugoClient.php, /media/nvme/dev/golem15/fonoteka/vue-fonoteka-app/app/composables/useCentrifugo.ts, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Patterns 7-8, Pitfalls 9-11)</read_first>
|
||||
<action>(1) lighthouse core (D-11): lighthouse.go with `type Service struct` (app, driver, registry, user lookup, namespace, queue, timeout, logger, bindings), `func From(app *backpack.App) (*Service, error)` doing lookup-or-publish of `*Service` on the app; the first call reads `realtime.driver` (default `null`), `realtime.broadcast_namespace` (default ""), `realtime.broadcast_queue` (default `broadcasts`) and `realtime.broadcast_timeout` (default 5s; int seconds or duration string, the postcard idiom), builds the driver through the registered factory (unknown name is an error listing the registered names and saying the driver package must be imported), and publishes. drivers.go: `type Publisher interface { Publish(ctx, channel, event string, payload json.RawMessage) error; Broadcast(ctx, channels []string, event string, payload json.RawMessage) error }`, `type Driver interface { Publisher; Name() string; Routes() []Route }`, `type DriverFactory func(app *backpack.App, svc *Service) (Driver, error)`, `func RegisterDriver(name string, f DriverFactory)` (init-time registry like database/sql; a duplicate name panics), and built-ins registered in init: `null` (discards), `log` (logs channel names and event at Info, never the payload), `memory` (`MemoryDriver` with a mutex-guarded `[]Publication{Method, Channels, Event, Payload, Timestamp}` and a copying `Publications()`; postcard MemoryDriver precedent). None of the built-ins declares routes. users.go: `type User struct { ID uint; Name *string }`, `type UserLookup func(ctx context.Context, id uint) (User, bool, error)`, `(*Service).SetUserLookup`, `(*Service).User(ctx, id)` (no lookup registered yields User{ID} with a nil Name), `type Actor struct` with `UserID *uint` (JSON key user_id) and `Name *string` (JSON key name), `SystemActor()` ({nil, "System"}) and `(*Service).Actor(ctx)` (no principal or `Principal.Backend` gives SystemActor, because PHP's frontend `auth()->user()` is null in backend requests; otherwise the user id and looked-up name).
|
||||
|
||||
(2) route.go (D-13): `type Surface int` with `UserAuth`, `ServerToServer`, `Public`; `type Route struct { Name, Method, Path string; Surface Surface; Handler http.HandlerFunc }`; `type Surfaces struct { UserAuth, ServerToServer, Public, Middleware []string }`; `func Mount(r pact.Router, d Driver, s Surfaces) error` registers each route under prefix "/" with the surface middleware followed by Middleware: UserAuth and Public through `r.Group`, ServerToServer through `r.GroupRaw`; a UserAuth route with an empty s.UserAuth is an error (never mount the token route without a guard, T-11-19); a nil driver or a driver without routes mounts nothing.
|
||||
|
||||
(3) centrifugo sub-package (D-12), import path `.../modules/lighthouse/centrifugo`: config.go `type Config struct` read from `realtime.centrifugo.*` with the PHP defaults (api_url `http://127.0.0.1:8001/api`, token_ttl 3600, ws_url `/ws`, token_path `/api/realtime/token`, subscribe_path `/api/realtime/subscribe`, empty api_key, token_secret and proxy_secret). client.go (RESEARCH Pattern 8): `NewClient(cfg, *http.Client)` with a 5s timeout; `Publish` POSTs `{api_url}/publish` with an ordered struct body `{channel, data: {event, payload, timestamp}}` where timestamp is `wire.Time` of now, headers `Authorization: apikey <key>` and `Content-Type: application/json`, encoder with SetEscapeHTML(false); `Broadcast` the same with `channels` to `/broadcast`; `Presence(ctx, channel) (map[string]any, error)` returns `result.presence` (empty map when disabled or on failure, logged); `Unsubscribe(ctx, userID uint, channel)` POSTs `{"user":"<id>","channel":c}` to `/unsubscribe`; success is any 2xx (Centrifugo's 200-with-error bodies count as success as in PHP; debug-log them); empty api_key returns `ErrNotConfigured` without a request; the key never appears in logs or errors; `Enabled()` and `DebugInfo()` ({APIURL, Enabled, APIKeySet}). token.go (RESEARCH Pattern 7): `TokenIssuer` with injectable clock; `ForUser(u lighthouse.User)` claims `sub` = decimal id string, `exp` = now + ttl, `info` = {"name": u.Name}; `Subscription(u, channel)` claims sub, channel, exp; `Anonymous()` sub "" and exp now+300; `ForIdentifier(identifier string, info map[string]any)` with info encoded as the JSON `[]` when empty (Pitfall 9); `SubscriptionForIdentifier(identifier, channel)`; all HS256 via golang-jwt/v5 and `ErrNotConfigured` for an empty secret; `Configured()`. handlers.go: `TokenHandler(svc, issuer) http.HandlerFunc`: no principal or no user from svc.User gives 401 `{"error":"Unauthorized"}`; empty token_secret gives 503 `{"error":"WebSocket not configured"}` (checked after the user, PHP order); else 200 `{"token":"..."}`; bodies through wire.WriteJSON with `Cache-Control: no-cache, private` (Laravel default, confirmed later against PHP by plan 11-06). driver.go: `Driver` implementing lighthouse.Driver (name `centrifugo`, Publish/Broadcast via Client, Routes returning `token` GET token_path UserAuth and, in Task 2, `subscribe`); `func init() { lighthouse.RegisterDriver("centrifugo", ...) }`.
|
||||
|
||||
(4) fonoteka.go wiring: config/realtime.yaml with `driver: centrifugo`, empty secrets and the PHP defaults (comment the SUMMER_REALTIME__CENTRIFUGO__* env names). New plugins/golem15/fonoteka/realtime.go: blank-imports the centrifugo driver package and defines `func (p *Plugin) wireRealtime(app *backpack.App) error` that calls `lighthouse.From(app)`, stores the service on the Plugin (new field `realtime *lighthouse.Service`) and sets a UserLookup that loads `usermodels.User` by id through a lazily resolved *gorm.DB (`app.Lookup[*gorm.DB]()`, lazyInvTokenGuard precedent). plugin.go Boot calls `p.wireRealtime(app)`; Buckets gains `"ws-api": {Max: 120, Decay: time.Minute, Key: ...}` keyed `wsapi:u:<id>` from `bouncer.User` else `wsapi:ip:` plus surf.ClientIP (D-14). routes.go calls `lighthouse.Mount(r, p.realtime.Driver(), lighthouse.Surfaces{UserAuth: surf.Use("jwt.auth"), ServerToServer: surf.Use(), Middleware: surf.Use("throttle:ws-api")})` once, so jwt.auth runs before the throttle (user decision 5). Run `go mod tidy` in plugins/golem15/fonoteka.
|
||||
|
||||
(5) Smoke test ../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go `TestRealtimeTokenRoute` through the assembled router (surf.BuildRouter of both plugins with a test config): a valid frontend JWT returns 200 and the token decodes with the configured secret to exactly the claims sub/exp/info{name}; an empty token_secret returns 503 with the exact body; a request without Authorization gets the jwt.auth 401; the response has Content-Type application/json and no trailing newline. Also a small httptest-backed check that Client.Publish sends the exact path, `Authorization: apikey` header and body keys.
|
||||
|
||||
(6) Docs: new modules/lighthouse/README.md in the standard structure (H1, summary "Transport-neutral realtime: a publisher interface with pluggable drivers, subscribe-time channel authorization, and model broadcasts enqueued in the write transaction.", both import lines, Overview, Features, Usage with an acme example, API reference for lighthouse and centrifugo, Configuration table, Dependencies, Testing), root README.md row with the same sentence, and a `## Configuration` section in ../fonoteka.go/README.md mapping CENTRIFUGO_API_URL, CENTRIFUGO_API_KEY, CENTRIFUGO_SECRET, CENTRIFUGO_TOKEN_TTL, CENTRIFUGO_WS_URL, CENTRIFUGO_PROXY_SECRET, BROADCAST_NAMESPACE, BROADCAST_QUEUE and BROADCAST_TIMEOUT to their SUMMER_REALTIME__* names. Check identifiers with `go doc ./modules/lighthouse <Identifier>` and `go doc ./modules/lighthouse/centrifugo <Identifier>`.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/lighthouse/... -count=1 && (cd ../fonoteka.go && go vet ./plugins/golem15/fonoteka/... && go test ./plugins/golem15/fonoteka -run '^(TestRealtimeTokenRoute|TestAllRouteGroupsBoot|TestFullRouteTableAuthGroupMutualExclusivity)$' -count=1 -race -v)</automated>
|
||||
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS" for TestRealtimeTokenRoute, TestAllRouteGroupsBoot or TestFullRouteTableAuthGroupMutualExclusivity, prints "no tests to run", "--- SKIP" or "DATA RACE".</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `go doc ./modules/lighthouse Mount`, `go doc ./modules/lighthouse Surfaces`, `go doc ./modules/lighthouse RegisterDriver` and `go doc ./modules/lighthouse/centrifugo TokenIssuer.ForUser` exit 0.
|
||||
- `grep -c 'apikey ' modules/lighthouse/centrifugo/client.go` prints at least 1.
|
||||
- `grep -c 'WebSocket not configured' modules/lighthouse/centrifugo/handlers.go` prints 1.
|
||||
- `grep -n 'lighthouse.Mount' ../fonoteka.go/plugins/golem15/fonoteka/routes.go` shows `surf.Use("jwt.auth")` as UserAuth and `surf.Use("throttle:ws-api")` as Middleware.
|
||||
- `grep -c '"ws-api"' ../fonoteka.go/plugins/golem15/fonoteka/plugin.go` prints 1.
|
||||
- `grep -c '\[lighthouse\](modules/lighthouse/README.md)' README.md` prints 1.
|
||||
- `grep -c 'SUMMER_REALTIME__CENTRIFUGO__TOKEN_SECRET' ../fonoteka.go/README.md` prints at least 1.
|
||||
</acceptance_criteria>
|
||||
<done>A logged-in client gets a PHP-shaped Centrifugo token from the mounted route, the unconfigured and unauthenticated cases keep PHP's bodies, and the driver can publish byte-shaped requests to Centrifugo.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Centrifugo's subscribe proxy re-authorizes every subscribe through the namespace registry and Płytarium's collection and wishlist authorizers</name>
|
||||
<files>modules/lighthouse/channel.go, modules/lighthouse/registry.go, modules/lighthouse/lighthouse.go, modules/lighthouse/README.md, modules/lighthouse/centrifugo/handlers.go, modules/lighthouse/centrifugo/driver.go, ../fonoteka.go/plugins/golem15/fonoteka/realtime.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/ws/collection_authorizer.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/ws/wishlist_authorizer.go, ../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go</files>
|
||||
<read_first>/media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/http/controllers/ProxyController.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/classes/AuthorizerRegistry.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/classes/AuthorizationResult.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/ws/CollectionChannelAuthorizer.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/ws/WishlistChannelAuthorizer.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/models/Collection.php (scopeWishlistsVisibleTo), ../fonoteka.go/plugins/golem15/fonoteka/classes/active_collection.go (AccessibleByMembership), ../fonoteka.go/plugins/golem15/fonoteka/models/collection.go, ../fonoteka.go/plugins/golem15/fonoteka/models/wishlist_subscription.go, modules/lighthouse/centrifugo/handlers.go (Task 1), .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Pattern 9, Pitfall 9, Known Threat Patterns T-11-01..T-11-03)</read_first>
|
||||
<behavior>
|
||||
- Missing, wrong or empty-configured proxy secret: 200 with the exact generic deny body; the log line holds the reason and the client IP but never either secret.
|
||||
- user "" or "0": deny; user "5" as a JSON string or number: the authorizer receives 5.
|
||||
- `collection:{id}` for an owner or editor of a kind=collection row: 200 `{"result":{"info":[]}}`; the same id of a kind=wishlist row: deny.
|
||||
- `presence:collection:{id}` for a member: deny (segment 1 is "collection", PHP (int) gives 0, malformed), matching PHP as is.
|
||||
- `presence:acme:room:1` with an allowing test authorizer: result has info, allow ["prs"] and the three override keys; an authorizer override of join_leave wins over the default.
|
||||
- `presence:presence:x`, `a:b:c:d`, `unknown:1`: deny.
|
||||
- `wishlist:{id}` for a wishlist_subscriptions row or for a household peer of the wishlist owner: allow; for a stranger: deny.
|
||||
- Removing an editor between two subscribes flips allow to deny.
|
||||
</behavior>
|
||||
<action>(1) channel.go (RESEARCH Pattern 9): `func ParseChannel(channel string) (namespace string, presence bool)` porting parseChannel exactly: a `presence:presence:` prefix yields "" (deny), one `presence:` prefix is stripped for the lookup (presence reported true), splitting on ":" with more than three segments yields "", otherwise segment 0; `func ChannelID(channel string) int64` returns segment 1 of the original channel (0 when missing) converted with PHP (int)-cast semantics — before coding, run `php -r 'foreach (["5","05"," 5","5abc","abc","-3","","1e3","0x1A","9999999999999999999"] as $s) var_dump((int)$s);'` and implement exactly what it prints, keeping those inputs as a table test; `func FormatChannels(namespace string, channels []string) []string` lowercases each channel and prefixes `lower(namespace)+":"` unless empty namespace or already prefixed (BroadcastEventJob.formatChannels); `ClientID(ctx) string` returns the proxy's client id placed in ctx.
|
||||
|
||||
(2) registry.go (D-11): `type Result struct { Allowed bool; Info map[string]any; Capabilities []string; Overrides map[string]any; reason string }` with `Reason() string`, constructors `Allowed(info map[string]any) Result` and `Denied(reason string) Result`; `type Authorizer interface { Authorize(ctx context.Context, userID uint, channel string) Result }` and `AuthorizerFunc`; `type Registry struct` (RWMutex) with `Register(namespace string, a Authorizer) error` (empty namespace, one containing ":", nil authorizer or a duplicate are errors), `Get(namespace) (Authorizer, bool)`, `Namespaces() []string` (sorted); `(*Service).Registry()` returns the service's registry.
|
||||
|
||||
(3) ProxyHandler in centrifugo/handlers.go (D-12, T-11-01..T-11-03): body capped at 64 KiB with http.MaxBytesReader; secret check first with `crypto/subtle.ConstantTimeCompare` against `proxy_secret` (empty configured secret denies); decode `{user, channel, client}` accepting user as a JSON string or number (anything else counts as empty); empty or "0" user denies; namespace from ParseChannel; unknown namespace denies; call the authorizer with ctx carrying the client id, the user id (PHP (int) of the user, negative mapped to 0) and the full original channel. Allow: `{"result":{"info":...}}` with info `[]` when empty; presence adds `allow` (Capabilities or ["prs"]) and `override` (defaults presence/join_leave `{"value":true}`, force_push_join_leave `{"value":false}`, then the authorizer's Overrides on top). Deny: slog Warn `Subscription denied` with reason and the PHP log context (ip for secret failures; user, channel, client and internal_reason otherwise) and the exact body `{"error":{"code":403,"message":"Access denied"}}` with HTTP 200 (Centrifugo reads non-200 as internal error 100). Always `Content-Type: application/json` and `Cache-Control: no-cache, private`, no trailing newline. A missing channel denies (Centrifugo always sends one; PHP would 500 with a TypeError), recorded as deliberate hardening in the SUMMARY. Driver.Routes adds `subscribe` POST subscribe_path ServerToServer.
|
||||
|
||||
(4) fonoteka.go authorizers (D-16), new package `classes/ws`: `CollectionAuthorizer{App}` resolves *gorm.DB lazily per call (deny with reason "database unavailable" when unpublished); id := lighthouse.ChannelID(channel), not positive denies "malformed channel"; the user must exist in users (else "user not found"); allowed when a golem15_fonoteka_collections row with that id, kind `collection`, not soft-deleted, passes `classes.AccessibleByMembership(userID)`; else "not owner/editor of collection". `WishlistAuthorizer{App}`: same id parsing; allowed when a golem15_fonoteka_wishlist_subscriptions row has (user_id, collection_id), or when a kind `wishlist` collection with that id has owner_id in the peer set (owner ids plus editor user ids of every collection the user reaches through AccessibleByMembership, mirroring scopeWishlistsVisibleTo); else "not a subscriber of this wishlist". realtime.go registers `collection` and `wishlist` on the service registry during wireRealtime.
|
||||
|
||||
(5) Smoke test `TestRealtimeSubscribeProxy` in realtime_smoke_test.go covering the behavior list against Postgres (use a fresh `lagoon.Use(ctx, bootSQL)` handle and seeded users/collections), run with -race; plus `TestChannelIDMatchesPHP` for the php -r table (the test may live in the lighthouse package as a unit test instead). Update modules/lighthouse/README.md (proxy contract, registry, channel rules).</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/lighthouse/... -count=1 -race && (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^(TestRealtimeSubscribeProxy|TestRealtimeTokenRoute)$' -count=1 -race -v)</automated>
|
||||
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS" for TestRealtimeSubscribeProxy or TestRealtimeTokenRoute, prints "no tests to run", "--- SKIP" or "DATA RACE".</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `grep -c 'ConstantTimeCompare' modules/lighthouse/centrifugo/handlers.go` prints 1.
|
||||
- `grep -c '"Access denied"' modules/lighthouse/centrifugo/handlers.go` prints 1.
|
||||
- `grep -c 'force_push_join_leave' modules/lighthouse/centrifugo/handlers.go` prints at least 1.
|
||||
- `grep -c 'Register("collection"' ../fonoteka.go/plugins/golem15/fonoteka/realtime.go` and `grep -c 'Register("wishlist"' ../fonoteka.go/plugins/golem15/fonoteka/realtime.go` each print 1.
|
||||
- `go doc ./modules/lighthouse Registry.Register`, `go doc ./modules/lighthouse ChannelID` and `go doc ./modules/lighthouse/centrifugo ProxyHandler` exit 0.
|
||||
- TestRealtimeSubscribeProxy asserts the allow body bytes `{"result":{"info":[]}}` exactly.
|
||||
</acceptance_criteria>
|
||||
<done>Every Centrifugo subscribe is re-authorized against the database through the namespace registry, denials are generic HTTP 200 bodies with reasons only in logs, and Płytarium's collection and wishlist rules are ported.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 3: Album writes publish to Centrifugo only after commit, and a suppressed bulk write emits exactly one summary event</name>
|
||||
<files>modules/lighthouse/broadcast.go, modules/lighthouse/suppress.go, modules/lighthouse/job.go, modules/lighthouse/lighthouse.go, modules/lighthouse/README.md, ../fonoteka.go/plugins/golem15/fonoteka/realtime.go, ../fonoteka.go/plugins/golem15/fonoteka/realtime_smoke_test.go, .planning/PROJECT.md, .planning/REQUIREMENTS.md</files>
|
||||
<read_first>/media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/traits/BroadcastableModel.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/jobs/BroadcastEventJob.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/models/Album.php (lines 415-495), /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/controllers/api/AlbumApiController.php (store, bulk, publishAlbumBroadcast), ../fonoteka.go/plugins/golem15/fonoteka/classes/serialize.go, ../fonoteka.go/plugins/golem15/fonoteka/models/album.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/artist_resolver.go (callback registration precedent), modules/conga/conga.go (Enqueue, Register), modules/lagoon/ondatabase.go, $(go env GOMODCACHE)/gorm.io/gorm@v1.31.2/callbacks/transaction.go, .planning/PROJECT.md (Constraints and Key Decisions), .planning/REQUIREMENTS.md (RT-01), .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Pattern 5, Pitfalls 7, 8, 15)</read_first>
|
||||
<behavior>
|
||||
- An album created inside lagoon.Transaction under a frontend principal produces one POST /publish on `collection:<id>` with data.event `created.fonoteka.album` and payload keys id, collection_id, action, actor, timestamp, album; actor is {user_id, name}.
|
||||
- The same write rolled back produces no publish within 2s.
|
||||
- A delete of an album loaded with only its id still publishes `deleted.fonoteka.album` with id, collection_id, action, actor, timestamp and no album key.
|
||||
- An album in a kind=wishlist collection publishes nothing.
|
||||
- Three album creates inside WithoutBroadcasting[models.Album] plus one svc.Emit of `collection.bulk_updated` {reason: bulk_create, count: 3} produce exactly one publish; suppressing Album never suppresses another bound type (asserted with an acme type in plan 11-07); a write through a stale outer ctx inside fn is not suppressed.
|
||||
- Centrifugo answering 500 leaves the album committed and logs a warning; the job is not retried.
|
||||
- With broadcast_namespace `acme`, the published channel is `acme:collection:<id>`.
|
||||
</behavior>
|
||||
<action>(1) broadcast.go (D-06, D-09, Pattern 5): `type Action string` with `ActionCreated`, `ActionUpdated`, `ActionDeleted`; `type Event struct { Action Action; Actor Actor; Timestamp wire.Time; TTL int }`; model-method contract `Broadcastable` (`BroadcastChannels(ctx context.Context, tx *gorm.DB) ([]string, error)`) with optional `BroadcastPayloader` (`BroadcastPayload(ctx, tx, Event) (any, error)`), `BroadcastAliaser` (`BroadcastAlias() string`), `BroadcastFilter` (`ShouldBroadcast(Action) bool`), `BroadcastTTLer` (`BroadcastTTL() int`); plus `type Binding[T any] struct { Alias string; Channels func(ctx, tx *gorm.DB, m *T) ([]string, error); Payload func(ctx, tx *gorm.DB, m *T, ev Event) (any, error); ShouldBroadcast func(Action) bool; TTL int }` and `func Bind[T any](svc *Service, b Binding[T]) error` (for payloads that need packages the models-leaf rule keeps out of models; nil Channels or a duplicate binding is an error). Default alias: last Go package path segment, or the one before it when that segment is `models`, plus "." plus the lowercased type name; event name `strings.ToLower(action + "." + alias)`; default payload `{"model": <model JSON>, "actor", "timestamp", "ttl"}` with ttl 60. Callbacks installed through `lagoon.OnDatabase` from From (idempotent per *gorm.DB: Register when `Get(name)` is nil, else Replace): `lighthouse:after_create` After `gorm:after_create`, `lighthouse:after_update` After `gorm:after_update`, `lighthouse:snapshot` Before `gorm:before_delete` (reload the full row by primary key with an unscoped fresh session when needed, compute channels and the deleted payload now and keep them with InstanceSet) and `lighthouse:after_delete` After `gorm:after_delete` (enqueue the snapshot only when db.Error is nil). Each callback: skip when the statement type has no binding or contract, when its primary key is zero (Pitfall 8; iterate slice elements for batch creates), when ctx suppresses the type, or when ShouldBroadcast vetoes; compute channels (empty means no broadcast), actor via svc.Actor(db.Statement.Context), payload, then `conga.From(app).Enqueue(ctx, db, BroadcastArgs{...}, conga.EnqueueOpts{Queue: svc.queue})`, all inside a GORM savepoint (`SavePoint`/`RollbackTo`) so a failed query or enqueue can never abort the write transaction; failures are logged Warn with channels and event, never payloads.
|
||||
|
||||
(2) suppress.go (D-08, Pitfall 7): `func WithoutBroadcasting[T any](ctx context.Context, fn func(ctx context.Context) error) error` adds `reflect.TypeFor[T]()` (pointer types normalized to their element) to an immutable set in a new ctx value and calls fn with that ctx only; doc comment: writes must use `gdb.WithContext(ctx)` with the ctx handed to fn. `type Broadcast struct { Channels []string; Event string; Payload any }` and `(*Service).Emit(ctx, db *gorm.DB, b Broadcast) error` enqueue exactly one job on the same transaction (returns the error, since the caller asked explicitly).
|
||||
|
||||
(3) job.go (D-06, D-07): `type BroadcastArgs struct { Channels []string; Event string; Payload json.RawMessage }` with Kind `summer.broadcast`; From registers it once with `conga.Job(..., conga.OnQueue(queue), conga.MaxAttempts(1), conga.Timeout(timeout))`; the worker applies FormatChannels with the namespace, returns nil for zero channels, calls driver.Publish for one channel and driver.Broadcast for several, and on error logs Warn `realtime: broadcast failed` (channels, event) and returns nil so River never retries (PHP tries = 1).
|
||||
|
||||
(4) fonoteka Album binding in realtime.go: `lighthouse.Bind[models.Album](svc, lighthouse.Binding[models.Album]{Alias: "fonoteka.album", Channels: albumChannels, Payload: albumPayload})`; albumChannels loads the collection by CollectionID through tx with the default soft-delete scope and returns `[]string{"collection:" + id}` only when Kind is `collection`; albumPayload returns an ordered struct `{id, collection_id, action, actor, timestamp, album?}` where album (omitted for deleted) is `classes.SerializeAlbum` of the album reloaded through tx with Genre, Styles and Artists preloaded.
|
||||
|
||||
(5) Smoke `TestAlbumBroadcastSmoke` in realtime_smoke_test.go for the behavior list: realtime config pointing api_url at an httptest server recording requests with api_key `test-only-api-key`, a fresh `lagoon.Use` handle, migrations, `conga.StartWorker` for the app, writes through lagoon.Transaction with `bouncer.WithUser` ctx; wait up to 5s for publishes.
|
||||
|
||||
(6) Docs: modules/lighthouse/README.md (broadcasting, suppression, Emit, Bind, queue and timeout config) in the code commit. Separate planning-docs commit in summercms.go: .planning/PROJECT.md Constraints line on the two repositories notes that websockets is not a separate app plugin (D-16: the framework realtime package plus fonoteka's config, authorizers and Mount call replace it) and a Key Decisions row for it; .planning/REQUIREMENTS.md RT-01 gets an appended note that the Centrifugo client is a hand-rolled net/http client per D-12 rather than the originally named library, so the verifier does not flag the wording.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./... && (cd ../fonoteka.go && go vet ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/... && go test ./plugins/golem15/fonoteka -run '^(TestAlbumBroadcastSmoke|TestRealtimeSubscribeProxy|TestRealtimeTokenRoute)$' -count=1 -race -v && go test ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/...)</automated>
|
||||
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS" for TestAlbumBroadcastSmoke, TestRealtimeSubscribeProxy or TestRealtimeTokenRoute, prints "no tests to run", "--- SKIP" or "DATA RACE"; any package reports FAIL.</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `go doc ./modules/lighthouse WithoutBroadcasting`, `go doc ./modules/lighthouse Bind`, `go doc ./modules/lighthouse Service.Emit` and `go doc ./modules/lighthouse BroadcastArgs` exit 0.
|
||||
- `grep -c 'SavePoint' modules/lighthouse/broadcast.go` prints at least 1.
|
||||
- `grep -l 'MaxAttempts(1)' modules/lighthouse/*.go` lists at least one file.
|
||||
- `grep -c 'Bind\[models.Album\]' ../fonoteka.go/plugins/golem15/fonoteka/realtime.go` prints 1.
|
||||
- TestAlbumBroadcastSmoke asserts exactly one publish for the suppressed bulk case and zero for the rolled-back write.
|
||||
- `grep -c 'D-16' .planning/PROJECT.md` prints at least 1 and `grep -c 'D-12' .planning/REQUIREMENTS.md` prints at least 1.
|
||||
</acceptance_criteria>
|
||||
<done>Album creates, updates and deletes publish Płytarium-shaped events to collection channels only after commit, bulk writes can suppress per-row events and emit one summary, publish failures never touch the write, and both repositories pass their full suites.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| Browser (frontend JWT) → GET /api/realtime/token | Authenticated users obtain a Centrifugo connection token |
|
||||
| Centrifugo server → POST /api/realtime/subscribe | Server-to-server call authenticated by a shared secret; body names the user and channel |
|
||||
| Write transaction → River broadcasts queue → Centrifugo HTTP API | Model data leaves the database for subscribers of a channel |
|
||||
| Operator config → driver selection and secrets | api_key, token_secret, proxy_secret |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-11-01 | Spoofing | subscribe proxy | high | mitigate | Constant-time X-Centrifugo-Secret check before any parsing; an empty configured proxy_secret denies (Task 2). |
|
||||
| T-11-02 | Elevation of Privilege | channel parsing and authorizers | high | mitigate | parseChannel ported exactly (presence:presence:, over three segments, namespace lookup); authorizers check kind (collection vs wishlist) and membership against the database on every subscribe; PHP (int) id semantics pinned by a php -r table test (Task 2). |
|
||||
| T-11-03 | Information Disclosure | deny responses | medium | mitigate | One generic 200 deny body; reasons and context only in logs (Task 2). |
|
||||
| T-11-04 | Information Disclosure | connection token info claim | medium | mitigate | ForUser puts only `name` in info; test asserts the exact claim set sub/exp/info (Task 1). |
|
||||
| T-11-05 | Information Disclosure | broadcast audience and rolled-back writes | high | mitigate | Enqueue happens inside the write transaction (rollback removes the job); Album channels only for kind=collection; payload built from SerializeAlbum, which carries no hidden or encrypted fields (Task 3). |
|
||||
| T-11-06 | Information Disclosure | logs (api key, tokens, secrets, payloads) | medium | mitigate | Client never logs the key or header; handlers never log secrets; broadcast failures log channels and event only; the log driver omits payloads (Tasks 1-3). |
|
||||
| T-11-10 | Denial of Service | token and subscribe flooding | medium | mitigate | ws-api bucket 120/min per user or IP on both routes (jwt.auth runs first on token); subscribe body capped at 64 KiB (Tasks 1-2). |
|
||||
| T-11-19 | Elevation of Privilege | Mount of a user route without a guard | high | mitigate | Mount refuses a UserAuth route when the UserAuth surface is empty (Task 1). |
|
||||
| T-11-20 | Denial of Service | broadcast failure aborting the write transaction | medium | mitigate | Broadcast queries and enqueue run inside a savepoint; errors are logged and rolled back to the savepoint only (Task 3). |
|
||||
| T-11-SC | Tampering | Go module installs | high | mitigate | No new module: golang-jwt v5.3.1 and River v0.47.0 are already pinned; the official Centrifugo Go client is deliberately not added (D-12). |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
After Task 3: `go vet ./... && go test ./...` in summercms.go and the fonoteka.go full vet/test command pass; TestRealtimeTokenRoute, TestRealtimeSubscribeProxy and TestAlbumBroadcastSmoke pass under -race.
|
||||
Manual smoke (not blocking, collected at /gsd-verify-work): start Centrifugo v6 with the production secret layout, run `fonoteka serve`, log in through the Nuxt app, change an album and see the event arrive in the browser.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- lighthouse and lighthouse/centrifugo exist with README and root row; drivers selected by `realtime.driver`.
|
||||
- The token route and subscribe proxy match PHP bodies and statuses; the registry re-validates every subscribe.
|
||||
- Album broadcasts are transactional, suppressible per type, and bulk-summarizable with one Emit.
|
||||
- fonoteka registers the ws-api bucket, both authorizers, the Mount call and the Album binding; PROJECT.md records D-16.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-03-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -0,0 +1,232 @@
|
||||
---
|
||||
phase: 11-jobs-realtime-and-search-infrastructure
|
||||
plan: 04
|
||||
type: execute
|
||||
wave: 4
|
||||
depends_on: ["11-03", "11-05"]
|
||||
files_modified:
|
||||
- modules/flare/flare.go
|
||||
- modules/flare/vapid.go
|
||||
- modules/flare/encrypt.go
|
||||
- modules/flare/commands.go
|
||||
- modules/flare/encrypt_test.go
|
||||
- modules/flare/send_test.go
|
||||
- modules/flare/commands_test.go
|
||||
- modules/flare/README.md
|
||||
- modules/lighthouse/centrifugo/client.go
|
||||
- modules/lighthouse/centrifugo/commands.go
|
||||
- modules/lighthouse/centrifugo/commands_test.go
|
||||
- modules/lighthouse/README.md
|
||||
- README.md
|
||||
- ../fonoteka.go/config/push.yaml
|
||||
- ../fonoteka.go/README.md
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/plugin.go
|
||||
autonomous: true
|
||||
requirements: [RT-01]
|
||||
estimate:
|
||||
tokens: 110000
|
||||
raw_tokens: 110000
|
||||
tasks: 2
|
||||
confidence: low
|
||||
must_haves:
|
||||
truths:
|
||||
- "Per D-15 and user decision 3, Web Push is a separate framework package `modules/flare` (not a realtime driver) with a `Pusher` interface and a stdlib VAPID driver: RFC 8291 aes128gcm payload encryption built from crypto/ecdh, crypto/hkdf, crypto/aes and crypto/cipher, reproducing the RFC 8291 Appendix A test vector byte for byte, and an RFC 8292 `Authorization: vapid t=<ES256 JWT>, k=<public key>` header whose JWT carries aud (the endpoint origin), exp (at most 24h ahead) and sub (push.subject)."
|
||||
- "Per D-15, config keys follow the PHP push block: `push.enabled` (default false), `push.public_key`, `push.private_key`, `push.subject`, plus `push.ttl` (default 2419200 seconds) and `push.allowed_hosts`; no push is sent while push.enabled is false."
|
||||
- "Per D-15 and user decision 3, `websockets:generate-vapid-keys` generates a P-256 key pair as unpadded base64url (65-byte public key, 87 chars; 32-byte private key, 43 chars), validates and prints them with manual SUMMER_PUSH__* instructions, `--show-current` shows only the configured keys truncated to first 8 + `...` + last 4 with their lengths, and `--update` writes the keys through compass Set/Persist into the 0600 environment overrides file."
|
||||
- "Per D-15 and user decision 3, `websockets:test-push <user_id> [--show-config]` shows the push configuration without key values, reads subscriptions only through an app-provided `flare.SubscriptionSource` and reports `no subscription source registered` (exit 1) when none is published; fonoteka publishes none because Płytarium has no subscription store."
|
||||
- "Per D-15 and D-12, `websockets:health` exits 1 with `Centrifugo not configured (API key missing)` when realtime.centrifugo.api_key is empty, and otherwise prints the API URL, probes Centrifugo's `info` API method and prints the PHP settings table (API URL, Enabled, API Key Set) with exit 0, or `Connection check failed: ...` with exit 1."
|
||||
- "Push endpoints are user-supplied URLs, so the VAPID driver sends only to https endpoints whose host matches `push.allowed_hosts` (default: the FCM, Mozilla autopush, Apple and Windows push services); any other endpoint is refused before a connection is made."
|
||||
- "fonoteka registers the three websockets:* commands through its plugin Commands() and ships config/push.yaml with push disabled."
|
||||
artifacts:
|
||||
- path: "modules/flare/encrypt.go"
|
||||
provides: "RFC 8291 aes128gcm encryption"
|
||||
contains: "aes128gcm"
|
||||
- path: "modules/flare/vapid.go"
|
||||
provides: "VAPID key generation, parsing and RFC 8292 header"
|
||||
contains: "vapid t="
|
||||
- path: "modules/flare/flare.go"
|
||||
provides: "Pusher, Subscription, SendOptions, SubscriptionSource, Service, From"
|
||||
contains: "type Pusher interface"
|
||||
- path: "modules/flare/commands.go"
|
||||
provides: "websockets:generate-vapid-keys and websockets:test-push"
|
||||
contains: "websockets:generate-vapid-keys"
|
||||
- path: "modules/lighthouse/centrifugo/commands.go"
|
||||
provides: "websockets:health"
|
||||
contains: "websockets:health"
|
||||
key_links:
|
||||
- from: "../fonoteka.go/plugins/golem15/fonoteka/plugin.go"
|
||||
to: "modules/flare/commands.go"
|
||||
via: "Commands() appends flare.Commands(app) and centrifugo.Commands(app)"
|
||||
pattern: "flare\\.Commands|centrifugo\\.Commands"
|
||||
- from: "modules/flare/flare.go"
|
||||
to: "modules/flare/encrypt.go"
|
||||
via: "VAPID driver encrypts every payload before POSTing"
|
||||
pattern: "encrypt"
|
||||
- from: "modules/lighthouse/centrifugo/commands.go"
|
||||
to: "modules/lighthouse/centrifugo/client.go"
|
||||
via: "health probes Client.Info"
|
||||
pattern: "\\.Info\\("
|
||||
prohibitions:
|
||||
- requirement_id: RT-01
|
||||
category: privacy
|
||||
statement: "websockets:test-push, websockets:health and --show-current MUST NOT print or log a configured VAPID private key or the Centrifugo API key; only newly generated keys are printed, and configured keys appear truncated or as set/unset"
|
||||
status: resolved
|
||||
verification: test
|
||||
- requirement_id: RT-01
|
||||
category: safety
|
||||
statement: "websockets:test-push MUST NOT send a push when no SubscriptionSource is registered or push.enabled is false"
|
||||
status: resolved
|
||||
verification: test
|
||||
---
|
||||
|
||||
## Phase Goal
|
||||
|
||||
ROADMAP Phase 11 goal (verbatim, not in user-story form): River jobs run on the correct dual-driver split, Centrifugo publishing and channel authorization match the existing server, and Typesense sync stays a re-gated pre-filter — all brought up before the API phases that depend on them.
|
||||
|
||||
This plan's slice: an operator can check Centrifugo connectivity, generate VAPID keys and test Web Push from the app binary, with the push channel ready for any app that supplies subscriptions (D-15, RT-01's websockets plugin surface).
|
||||
|
||||
<objective>
|
||||
Port the PHP websockets console commands and the Web Push seams: a `flare` package with a stdlib VAPID driver, `websockets:generate-vapid-keys`, `websockets:test-push` over an app-provided SubscriptionSource, and `websockets:health` on the Centrifugo client.
|
||||
|
||||
Purpose: D-15 keeps the full websockets command surface even though Płytarium's PHP push code is dead (minishlink/web-push is not installed and TestPushNotifications imports a plugin that is not in the repo; RESEARCH Pitfall 14). Decisions implemented: D-15, D-12 (health on the hand-rolled client); user decision 3.
|
||||
Output: modules/flare with README and root row, centrifugo commands and Info probe, fonoteka config and command registration, smoke tests including the RFC 8291 vector.
|
||||
|
||||
Repos: summercms.go (framework) and fonoteka.go (config, Commands, README). This plan runs after 11-05 because both edit fonoteka's plugin.go and the root and fonoteka READMEs. Planning docs and code in separate commits. Never add co-author tags.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@~/.claude/gsd-core/workflows/execute-plan.md
|
||||
@~/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-03-SUMMARY.md
|
||||
@modules/lighthouse/centrifugo/client.go
|
||||
@modules/compass/persist.go
|
||||
@modules/lagoon/keygen.go
|
||||
@modules/bonfire/command.go
|
||||
@modules/bonfire/prompts.go
|
||||
@../fonoteka.go/plugins/golem15/fonoteka/plugin.go
|
||||
|
||||
<interfaces>
|
||||
- compass: `(*Config).Set(path string, value any) error` (runtime override), `(*Config).Persist() error` (writes config/env/<env>/overrides.yaml atomically with restrictive permissions), `String/Int/Bool/Lookup`.
|
||||
- lagoon/keygen.go `KeyGenerateCommand()` is the key-printing command precedent.
|
||||
- bonfire: `Command{Name, Description, Flags, Args, Run}`, `Flag{Bare}`, `Output` (Info, Success, Printf, Table, confirm prompts in prompts.go with non-TTY defaults).
|
||||
- From plan 11-03: `centrifugo.Config` (realtime.centrifugo.*), `centrifugo.NewClient`, `(*Client).Enabled()`, `(*Client).DebugInfo() DebugInfo{APIURL, Enabled, APIKeySet}`, header `Authorization: apikey <key>`, `ErrNotConfigured`.
|
||||
- Go 1.27 stdlib: `crypto/ecdh` (P256 GenerateKey, NewPrivateKey, NewPublicKey, ECDH), `crypto/hkdf` (Extract, Expand), `crypto/aes` + `crypto/cipher` (GCM), `crypto/ecdsa` (`ParseRawPrivateKey(elliptic.P256(), b)` to sign ES256), `encoding/base64.RawURLEncoding`; golang-jwt/v5 `SigningMethodES256`.
|
||||
- PHP: /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/console/{GenerateVapidKeys.php, TestPushNotifications.php, CentrifugoHealthCheck.php}, config/config.php push block, classes/CentrifugoClient.php (isEnabled, getDebugInfo).
|
||||
- RFC 8291 (https://www.rfc-editor.org/rfc/rfc8291) Section 3.4 key derivation and Appendix A vector; RFC 8292 (https://www.rfc-editor.org/rfc/rfc8292) Sections 2 and 3; RFC 8188 record layout (salt 16 bytes, rs uint32, idlen, keyid).
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
## Artifacts this phase produces
|
||||
|
||||
(This plan's share.)
|
||||
|
||||
- flare: `Pusher` (`Send(ctx, sub Subscription, payload []byte, opts SendOptions) error`), `Subscription{Endpoint, P256dh, Auth string}`, `SendOptions{TTL time.Duration; Urgency, Topic string}`, `SubscriptionSource` (`Subscriptions(ctx, userID uint) ([]SubscriptionInfo, error)`), `SubscriptionInfo{Subscription; ID uint; UserAgent string; SubscribedAt, LastUsedAt *time.Time}`, `ErrUserNotFound`, `ErrPushDisabled`, `ErrEndpointNotAllowed`, `ErrSubscriptionGone`, `Service`, `From(app) (*Service, error)`, `(*Service).Pusher() Pusher`, `VAPIDKeys{PublicKey, PrivateKey string}`, `GenerateVAPIDKeys() (VAPIDKeys, error)`, `ParseVAPIDKeys(public, private string) (*ecdh.PrivateKey, error)`, `VAPIDHeader(endpoint, subject string, keys VAPIDKeys, now time.Time) (string, error)`, `Encrypt(payload []byte, sub Subscription) ([]byte, error)`, `Commands(app) []bonfire.Command`.
|
||||
- lighthouse/centrifugo: `(*Client).Info(ctx) (map[string]any, error)`, `Commands(app) []bonfire.Command`.
|
||||
- CLI: `websockets:generate-vapid-keys [--update] [--show-current]`, `websockets:test-push <user_id> [--show-config]`, `websockets:health`.
|
||||
- Config keys: `push.enabled`, `push.public_key`, `push.private_key`, `push.subject`, `push.ttl`, `push.allowed_hosts`.
|
||||
- Files: `modules/flare/*`, `../fonoteka.go/config/push.yaml`.
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="tracer">
|
||||
<name>Task 1: A payload encrypted and signed by the VAPID driver is accepted and decrypted by a push service endpoint</name>
|
||||
<files>modules/flare/flare.go, modules/flare/vapid.go, modules/flare/encrypt.go, modules/flare/encrypt_test.go, modules/flare/send_test.go, modules/flare/README.md, README.md</files>
|
||||
<read_first>modules/postcard/mailer.go (From/driver selection pattern), modules/lagoon/keygen.go, modules/lagoon/encrypted.go (stdlib crypto style, hkdf use), /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/config/config.php, RFC 8291 Section 3.4 and Appendix A, RFC 8292 Sections 2-3 (fetch the RFC text; copy vector values verbatim), .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Alternatives "Hand-rolled RFC 8291", Pitfall 14, T-11-11)</read_first>
|
||||
<action>(1) encrypt.go (RFC 8291/8188): `func Encrypt(payload []byte, sub Subscription) ([]byte, error)` decodes `P256dh` (65-byte uncompressed P-256 point) and `Auth` (16 bytes) from unpadded base64url, generates an ephemeral P-256 key and a 16-byte random salt through unexported injectable sources (so the Appendix A vector can fix both), computes ecdh_secret, `PRK_key = HKDF-Extract(auth_secret, ecdh_secret)`, `IKM = HKDF-Expand(PRK_key, "WebPush: info" 0x00 || ua_public || as_public, 32)`, `PRK = HKDF-Extract(salt, IKM)`, `CEK = HKDF-Expand(PRK, "Content-Encoding: aes128gcm" 0x00, 16)`, `NONCE = HKDF-Expand(PRK, "Content-Encoding: nonce" 0x00, 12)`, encrypts `payload || 0x02` with AES-128-GCM as a single record, and returns `salt || uint32be(4096) || 0x41 || as_public || ciphertext`; payloads above 3993 bytes are an error.
|
||||
|
||||
(2) vapid.go (RFC 8292): `VAPIDKeys{PublicKey, PrivateKey string}`; `GenerateVAPIDKeys()` via `ecdh.P256().GenerateKey(rand.Reader)` with the public key as the unpadded base64url of the 65-byte uncompressed point and the private key as the unpadded base64url of the 32-byte scalar; `ParseVAPIDKeys(public, private)` accepts padded or unpadded input and checks the pair matches; `VAPIDHeader(endpoint, subject, keys, now)` builds an ES256 JWT (golang-jwt/v5, header typ JWT) with `aud` = scheme://host of the endpoint, `exp` = now + 12h, `sub` = subject (must start with `mailto:` or `https:`), and returns `vapid t=<jwt>, k=<public key>`.
|
||||
|
||||
(3) flare.go (D-15): `Pusher`, `Subscription`, `SendOptions`, `SubscriptionSource`, `SubscriptionInfo`, the error values; `Service` and `From(app)` (lookup-or-publish) reading `push.enabled`, `push.public_key`, `push.private_key`, `push.subject`, `push.ttl` (default 2419200s) and `push.allowed_hosts` (default `fcm.googleapis.com`, `updates.push.services.mozilla.com`, `*.push.apple.com`, `*.notify.windows.com`; `*.` means any subdomain); `(*Service).Pusher()` returns the VAPID driver whose Send refuses when disabled (`ErrPushDisabled`), refuses non-https endpoints or hosts outside the allowlist (`ErrEndpointNotAllowed`, before dialing), encrypts, POSTs with headers `TTL`, `Content-Encoding: aes128gcm`, `Content-Type: application/octet-stream`, optional `Urgency`/`Topic` and `Authorization` from VAPIDHeader, uses an injectable *http.Client with a 10s timeout, treats 2xx as success, maps 404 and 410 to `ErrSubscriptionGone` and other statuses to an error with the status code; the private key is never logged.
|
||||
|
||||
(4) Smoke tests: encrypt_test.go `TestRFC8291AppendixA` fixes the Appendix A salt and application-server key pair and asserts the exact output bytes for the Appendix A plaintext and receiver keys; send_test.go `TestVAPIDSendRoundTrip` runs an `httptest.NewTLSServer` push endpoint (allowlisted host in the test config) that verifies the VAPID JWT with the public key from `k=`, checks aud and the headers, and decrypts the body with the subscription's private key (a test-side RFC 8291 decrypt helper) back to the original payload; plus `TestSendRefusesDisallowedEndpoint` for an http URL and an unlisted host.
|
||||
|
||||
(5) Docs: new modules/flare/README.md in the standard structure (H1, summary "Web Push delivery with VAPID (RFC 8292) and aes128gcm payload encryption (RFC 8291) behind a small Pusher interface.", import line, Overview stating push is a separate channel from realtime, Features, Usage with an acme SubscriptionSource, API reference, Configuration, CLI commands (filled in Task 2), Dependencies (stdlib plus golang-jwt/v5), Testing), and the root README.md row with the same sentence; identifiers checked with `go doc ./modules/flare <Identifier>`.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/flare -run '^(TestRFC8291AppendixA|TestVAPIDSendRoundTrip|TestSendRefusesDisallowedEndpoint)$' -count=1 -v</automated>
|
||||
<fails_when>Non-zero exit; the output lacks "--- PASS" for TestRFC8291AppendixA, TestVAPIDSendRoundTrip or TestSendRefusesDisallowedEndpoint, or prints "no tests to run" or "--- SKIP".</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `go doc ./modules/flare Pusher`, `go doc ./modules/flare Encrypt`, `go doc ./modules/flare VAPIDHeader` and `go doc ./modules/flare SubscriptionSource` exit 0.
|
||||
- `grep -c 'crypto/hkdf' modules/flare/encrypt.go` prints 1 and `go list -m all | grep -ci 'webpush'` prints 0.
|
||||
- `grep -c 'Content-Encoding: aes128gcm' modules/flare/encrypt.go` prints at least 1.
|
||||
- `grep -c '\[flare\](modules/flare/README.md)' README.md` prints 1.
|
||||
- TestRFC8291AppendixA compares against the RFC's published output string, not a value produced by the implementation.
|
||||
</acceptance_criteria>
|
||||
<done>The VAPID driver produces RFC 8291 ciphertext matching the RFC vector and a push endpoint can verify and decrypt its requests; disallowed endpoints are refused before any connection.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Operators run websockets:health, websockets:generate-vapid-keys and websockets:test-push from the app binary</name>
|
||||
<files>modules/flare/commands.go, modules/flare/commands_test.go, modules/flare/README.md, modules/lighthouse/centrifugo/client.go, modules/lighthouse/centrifugo/commands.go, modules/lighthouse/centrifugo/commands_test.go, modules/lighthouse/README.md, ../fonoteka.go/config/push.yaml, ../fonoteka.go/README.md, ../fonoteka.go/plugins/golem15/fonoteka/plugin.go</files>
|
||||
<read_first>/media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/console/GenerateVapidKeys.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/console/TestPushNotifications.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/console/CentrifugoHealthCheck.php, modules/compass/persist.go, modules/bonfire/prompts.go, modules/bonfire/output.go, modules/lighthouse/centrifugo/client.go, modules/lighthouse/README.md, modules/flare/flare.go (Task 1), ../fonoteka.go/plugins/golem15/fonoteka/plugin.go (Commands), ../fonoteka.go/README.md (Configuration section)</read_first>
|
||||
<behavior>
|
||||
- websockets:health with an empty api_key prints `Centrifugo not configured (API key missing)` and exits 1 without a request.
|
||||
- websockets:health against a fake Centrifugo answering `/info` 200 prints `Configuration OK` and the table rows API URL, Enabled Yes, API Key Set Yes and exits 0; answering 500 prints `Connection check failed:` and exits 1; the api key never appears in output.
|
||||
- websockets:generate-vapid-keys prints an 87-char public key and a 43-char private key that ParseVAPIDKeys accepts; `--show-current` prints only `first8...last4` forms with lengths; `--update` persists both keys so a reloaded config returns them and the overrides file mode is 0600.
|
||||
- websockets:test-push 5 with no SubscriptionSource prints `no subscription source registered` and exits 1; with a source returning ErrUserNotFound prints `User 5 not found`; with no subscriptions prints `No push subscriptions found for this user`; with push.enabled false it lists subscriptions but refuses to send; with a working source and an allowlisted TLS endpoint it sends one encrypted push per subscription.
|
||||
- No command output contains a configured private key or api key value.
|
||||
</behavior>
|
||||
<action>(1) centrifugo: add `(*Client).Info(ctx) (map[string]any, error)` POSTing `{}` to `{api_url}/info` with the same headers and timeout; new commands.go `Commands(app) []bonfire.Command` with `websockets:health` (description "Check Centrifugo connection health") porting CentrifugoHealthCheck: not enabled → error lines `Centrifugo not configured (API key missing)` and `Set SUMMER_REALTIME__CENTRIFUGO__API_KEY in the environment`, exit 1; else `Checking Centrifugo connection...`, `API URL: <url>`, Info probe (PHP's comment calls this the basic connectivity check; PHP's getDebugInfo never reached the server, so the probe is the real check), then `Configuration OK` and the Setting/Value table from DebugInfo, or `Connection check failed: <err>` and exit 1.
|
||||
|
||||
(2) flare commands.go `Commands(app) []bonfire.Command`: `websockets:generate-vapid-keys` (bare flags `update`, `show-current`) porting GenerateVapidKeys: title lines, current keys shown truncated (first 8 + `...` + last 4, char count, check mark when the decoded public key is 65 bytes and the private key 32 bytes), `--show-current` stops there; otherwise generate, validate lengths and base64url charset, print both new keys, then with `--update` call `app.Config.Set("push.public_key", ...)`, `Set("push.private_key", ...)` and `Persist()` and print the overrides path, or print manual lines `SUMMER_PUSH__PUBLIC_KEY=<key>` and `SUMMER_PUSH__PRIVATE_KEY=<key>`; framework-neutral next steps. `websockets:test-push` (required arg `user_id`, bare flag `show-config`, accepted for PHP compatibility since the configuration is always shown) porting TestPushNotifications: configuration block (enabled, public/private key set with char counts only, subject with mailto/https check), key length checks (public 87 or 88, private 43), SubscriptionSource lookup via `app.Lookup[flare.SubscriptionSource]()` (missing → `no subscription source registered`, exit 1), user and subscription reporting as in PHP (endpoint first 60 chars plus `...`, user agent, subscribed/last used), a confirm prompt `Send test notification?` defaulting to yes (non-TTY takes the default), refusal when push is disabled, and a JSON payload `{"title": "<app.name> test", "body": "This is a test push notification sent at HH:MM:SS", "data": {"test": true, "timestamp": <unix>}}` sent to each subscription with per-subscription results; exit 1 when any send fails.
|
||||
|
||||
(3) fonoteka.go: plugin.go Commands() returns the existing oauth-client command plus `centrifugo.Commands(p.app)...` and `flare.Commands(p.app)...`; new config/push.yaml with `enabled: false`, empty public_key/private_key/subject and a comment on SUMMER_PUSH__* names; README Configuration gains PUSH_ENABLED, PUSH_VAPID_PUBLIC_KEY, PUSH_VAPID_PRIVATE_KEY and PUSH_VAPID_SUBJECT mapped to SUMMER_PUSH__ENABLED, SUMMER_PUSH__PUBLIC_KEY, SUMMER_PUSH__PRIVATE_KEY and SUMMER_PUSH__SUBJECT, and a note that Płytarium registers no SubscriptionSource yet.
|
||||
|
||||
(4) Tests: modules/flare/commands_test.go and modules/lighthouse/centrifugo/commands_test.go for the behavior list, running commands through `bonfire.NewRootIO` with captured output and temp config directories. Docs: modules/flare/README.md and modules/lighthouse/README.md CLI commands sections updated in the same commit.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/flare ./modules/lighthouse/... -count=1 && go test ./... && (cd ../fonoteka.go && go vet ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/... && go test ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/... && SUMMER_GOLEM15__USER__JWT__SECRET=test-only-cli-secret go run . websockets:health 2>&1 | grep -q 'Centrifugo not configured (API key missing)')</automated>
|
||||
<fails_when>Any command exits non-zero or reports FAIL; the final grep finds no "Centrifugo not configured (API key missing)" line in the output of `fonoteka websockets:health` run with the committed empty api_key.</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `grep -c 'websockets:health' modules/lighthouse/centrifugo/commands.go` prints 1; `grep -c 'websockets:generate-vapid-keys' modules/flare/commands.go` and `grep -c 'websockets:test-push' modules/flare/commands.go` each print 1.
|
||||
- `grep -c 'no subscription source registered' modules/flare/commands.go` prints 1.
|
||||
- `grep -c 'flare.Commands(p.app)' ../fonoteka.go/plugins/golem15/fonoteka/plugin.go` prints 1 and `grep -c 'centrifugo.Commands(p.app)' ../fonoteka.go/plugins/golem15/fonoteka/plugin.go` prints 1.
|
||||
- `grep -c 'enabled: false' ../fonoteka.go/config/push.yaml` prints 1 and `grep -c 'SUMMER_PUSH__PRIVATE_KEY' ../fonoteka.go/README.md` prints at least 1.
|
||||
- `go doc ./modules/lighthouse/centrifugo Commands` and `go doc ./modules/flare Commands` exit 0.
|
||||
</acceptance_criteria>
|
||||
<done>All three PHP websockets commands exist in the app binary with PHP output shapes, never print configured secrets, and push sends only through an app-provided subscription source; both repositories pass their full suites.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| App-provided subscriptions (browser-supplied endpoint URLs) → VAPID driver | Outbound HTTPS requests to URLs that originated in browsers |
|
||||
| Operator CLI → key material | Commands generate, show and persist VAPID keys and read the Centrifugo key |
|
||||
| Config overrides file on disk | Persisted private key |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-11-11 | Information Disclosure | VAPID private key | medium | mitigate | Printed only when newly generated; `--show-current` and test-push show truncated values or lengths; never logged (Task 2 tests assert output). |
|
||||
| T-11-22 | Spoofing / SSRF | push endpoint requests | high | mitigate | https only and host allowlist (`push.allowed_hosts`, known push services by default) checked before dialing; unlisted endpoints fail with ErrEndpointNotAllowed (Task 1). |
|
||||
| T-11-23 | Information Disclosure | websockets:health output | low | mitigate | Only "API Key Set: Yes/No" is shown; the key is never printed or logged (Task 2). |
|
||||
| T-11-24 | Information Disclosure | persisted overrides file | medium | mitigate | `--update` uses compass Persist, which writes atomically with restrictive (0600) permissions; test asserts the mode (Task 2). |
|
||||
| T-11-SC | Tampering | Go module installs | high | mitigate | No push library is installed (the webpush-go alternative in RESEARCH was not recommended and is not added); only stdlib crypto and the already-pinned golang-jwt/v5. |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
After Task 2: `go vet ./... && go test ./...` in summercms.go and the fonoteka.go full vet/test command pass; TestRFC8291AppendixA matches the RFC; `fonoteka websockets:health` with the committed config exits 1 with the not-configured message.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- flare exists with README and root row; the VAPID driver matches RFC 8291's vector and RFC 8292's header format.
|
||||
- websockets:health, websockets:generate-vapid-keys and websockets:test-push run from the app binary with PHP-shaped output and no secret leakage.
|
||||
- fonoteka registers the commands and ships push disabled.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-04-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -0,0 +1,248 @@
|
||||
---
|
||||
phase: 11-jobs-realtime-and-search-infrastructure
|
||||
plan: 05
|
||||
type: execute
|
||||
wave: 3
|
||||
depends_on: ["11-01", "11-03"]
|
||||
files_modified:
|
||||
- modules/beachcomber/beachcomber.go
|
||||
- modules/beachcomber/searchable.go
|
||||
- modules/beachcomber/sync.go
|
||||
- modules/beachcomber/engines.go
|
||||
- modules/beachcomber/README.md
|
||||
- modules/beachcomber/typesense/config.go
|
||||
- modules/beachcomber/typesense/engine.go
|
||||
- README.md
|
||||
- ../fonoteka.go/config/search.yaml
|
||||
- ../fonoteka.go/README.md
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/plugin.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/search.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go
|
||||
autonomous: true
|
||||
requirements: [SRCH-01]
|
||||
estimate:
|
||||
tokens: 120000
|
||||
raw_tokens: 120000
|
||||
tasks: 2
|
||||
confidence: low
|
||||
must_haves:
|
||||
truths:
|
||||
- "Per D-19, `modules/beachcomber` defines the `Searchable` model interface (`SearchableAs() string`, `ToSearchableArray(ctx, db) (map[string]any, error)`, `ShouldBeSearchable() bool`), the `Engine` driver interface (`Upsert`, `Delete`, `Flush`, `SearchIDs`, `Configured`, `Name`) and an application `Gate`; engines are chosen by `search.driver` (`typesense` or `null`, default `null`) and the Typesense engine is a hand-rolled net/http client (the official Typesense Go client is not added)."
|
||||
- "Per D-20, sync runs after commit, inline and non-fatal through `lagoon.AfterCommit`: a create or update of a Searchable model upserts its reloaded document, a delete or soft delete removes it, a rolled-back write sends nothing, and a Typesense error or timeout is logged as a warning while the write stays committed."
|
||||
- "Per D-20, the whole sync is skipped with zero Typesense requests when the engine is null or its api_key is empty, when no database is published, or when the application Gate reports off; fonoteka's Gate reads `golem15_fonoteka_settings.search_use_typesense` and treats any read error (for example a fresh install before migrate) as off."
|
||||
- "Per D-19 and SRCH-01, every Album document carries a positive integer `collection_id`; ToSearchableArray refuses (error, logged, nothing indexed) an album whose collection_id is not positive, as PHP's LogicException does."
|
||||
- "Album documents match PHP Album::toSearchableArray field for field (id as string, collection_id, genre_id, style_ids, artist_ids in pivot sort_order, artist_display, format, medium from the Album MEDIA map, year, created_at as unix seconds, name, style_names, genre_name, notes, track_titles, label, catalog_number) in index `golem15_fonoteka_albums` (prefixed by `search.prefix`), and the collection is created from the PHP collection-schema with default_sorting_field created_at when Typesense does not have it."
|
||||
- "The Typesense wire contract follows Scout's TypesenseEngine: `X-TYPESENSE-API-KEY` header; upsert = GET /collections/{name} (POST /collections with the schema when 404) then POST /collections/{name}/documents/import?action=upsert with a JSONL body where any `success:false` line is an error; delete = DELETE /collections/{name}/documents/{id} with 404 treated as success; flush = DELETE /collections/{name}; SearchIDs returns only candidate ids (the SQL re-gate is Phase 12's job)."
|
||||
- "Batch statements without a primary key are skipped (no document can be built for them), matching the broadcast rule."
|
||||
artifacts:
|
||||
- path: "modules/beachcomber/searchable.go"
|
||||
provides: "Searchable, IndexSchemaProvider, SearchKeyer, Engine, Query, Gate, GateFunc"
|
||||
contains: "type Searchable interface"
|
||||
- path: "modules/beachcomber/sync.go"
|
||||
provides: "GORM callbacks and the after-commit sync with its gates"
|
||||
contains: "AfterCommit"
|
||||
- path: "modules/beachcomber/typesense/engine.go"
|
||||
provides: "Typesense engine: Upsert, Delete, Flush, SearchIDs"
|
||||
contains: "X-TYPESENSE-API-KEY"
|
||||
- path: "../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go"
|
||||
provides: "Album Searchable implementation, AlbumMedia, MediumFamily, SearchIndexSchema"
|
||||
contains: "golem15_fonoteka_albums"
|
||||
- path: "../fonoteka.go/plugins/golem15/fonoteka/search.go"
|
||||
provides: "settings kill-switch Gate and beachcomber wiring"
|
||||
contains: "search_use_typesense"
|
||||
key_links:
|
||||
- from: "modules/beachcomber/sync.go"
|
||||
to: "modules/lagoon/transaction.go"
|
||||
via: "callbacks register work with lagoon.AfterCommit"
|
||||
pattern: "lagoon\\.AfterCommit"
|
||||
- from: "modules/beachcomber/beachcomber.go"
|
||||
to: "modules/lagoon/ondatabase.go"
|
||||
via: "callbacks installed through lagoon.OnDatabase"
|
||||
pattern: "lagoon\\.OnDatabase"
|
||||
- from: "../fonoteka.go/plugins/golem15/fonoteka/plugin.go"
|
||||
to: "../fonoteka.go/plugins/golem15/fonoteka/search.go"
|
||||
via: "Boot calls wireSearch which sets the settings Gate"
|
||||
pattern: "wireSearch"
|
||||
prohibitions:
|
||||
- requirement_id: SRCH-01
|
||||
category: privacy
|
||||
statement: "MUST NOT index a document whose collection_id is missing or not positive"
|
||||
status: resolved
|
||||
verification: test
|
||||
- requirement_id: SRCH-01
|
||||
category: safety
|
||||
statement: "A search engine failure MUST NOT fail, roll back or hold the user's write beyond the engine timeout"
|
||||
status: resolved
|
||||
verification: test
|
||||
- requirement_id: SRCH-01
|
||||
category: consent
|
||||
statement: "MUST NOT send any request to the search engine while search_use_typesense is off or the engine API key is empty"
|
||||
status: resolved
|
||||
verification: test
|
||||
---
|
||||
|
||||
## Phase Goal
|
||||
|
||||
ROADMAP Phase 11 goal (verbatim, not in user-story form): River jobs run on the correct dual-driver split, Centrifugo publishing and channel authorization match the existing server, and Typesense sync stays a re-gated pre-filter — all brought up before the API phases that depend on them.
|
||||
|
||||
This plan's slice: when the search toggle is on, saving an album makes it findable in Typesense right after the write commits, scoped by collection_id; when the toggle is off or Typesense is unconfigured, nothing is sent and nothing breaks (SRCH-01, ROADMAP SC-5).
|
||||
|
||||
<objective>
|
||||
Create the search framework package `beachcomber` with its Typesense engine sub-package, and bind the Album model and the settings kill-switch in fonoteka.go.
|
||||
|
||||
Purpose: Phase 12's album search endpoint reads candidate ids from this index and re-gates them in SQL. Decisions implemented: D-19, D-20; RESEARCH Pattern 10 and the Typesense contract; T-11-07.
|
||||
Output: modules/beachcomber (+ typesense) with README and root row, fonoteka Album Searchable binding, settings Gate, config/search.yaml, README config mapping, smoke tests.
|
||||
|
||||
Repos: summercms.go (framework) and fonoteka.go (application). Planning docs and code in separate commits. Never add co-author tags.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@~/.claude/gsd-core/workflows/execute-plan.md
|
||||
@~/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-01-SUMMARY.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-03-SUMMARY.md
|
||||
@modules/lagoon/transaction.go
|
||||
@modules/lagoon/ondatabase.go
|
||||
@modules/lighthouse/drivers.go
|
||||
@modules/lighthouse/broadcast.go
|
||||
@../fonoteka.go/plugins/golem15/fonoteka/models/album.go
|
||||
@../fonoteka.go/plugins/golem15/fonoteka/models/settings.go
|
||||
@../fonoteka.go/plugins/golem15/fonoteka/plugin.go
|
||||
|
||||
<interfaces>
|
||||
- From plan 11-01: `lagoon.OnDatabase(app, func(*sql.DB, *gorm.DB) error) error`; `lagoon.AfterCommit(ctx, db *gorm.DB, fn func(ctx context.Context, db *gorm.DB))` (buffered to after COMMIT inside lagoon.Transaction or an implicit single-statement transaction; immediate otherwise, with the tx handle inside a plain gdb.Transaction); `lagoon.Transaction(ctx, gdb, fn)`.
|
||||
- From plan 11-03: driver registry idiom `lighthouse.RegisterDriver(name, factory)` with init-time registration by a sub-package, lookup-or-publish `From(app)`, callback installation per *gorm.DB (Register when Get(name) is nil, else Replace), zero-primary-key skip (Pitfall 8). Copy these shapes for engines.
|
||||
- fonoteka models: `Album{ID, CollectionID, GenreID *uint, Genre *Genre, Name, Notes, ArtistDisplay, Year *int, Format *string, TrackTitles, Label, CatalogNumber *string, Artists []Artist (many2many golem15_fonoteka_album_artists with sort_order), Styles []Style, DeletedAt gorm.DeletedAt, CreatedAt time.Time}`; `Settings{ID, SearchUseTypesense bool}` table golem15_fonoteka_settings (singleton row).
|
||||
- PHP contract: /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/models/Album.php (MEDIA 38-46, mediumFamily 220-227, searchableAs, shouldBeSearchable, toSearchableArray, getSearchConfig 496-570), /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/Plugin.php (guardSearchSyncing 290-308), /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/console/ReindexAlbums.php (typesenseIsConfigured), /media/nvme/dev/golem15/fonoteka/config/scout.php (driver, prefix, queue=false, after_commit=false, typesense client-settings: host localhost, port 8181, path "", protocol http, connection_timeout_seconds 2, import_action upsert), vendor Scout TypesenseEngine.php under /media/nvme/dev/golem15/fonoteka/vendor/laravel/scout/src/Engines/.
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
## Artifacts this phase produces
|
||||
|
||||
(This plan's share.)
|
||||
|
||||
- beachcomber: `Service`, `From(app) (*Service, error)`, `(*Service).SetGate(Gate)`, `(*Service).Engine() Engine`, `(*Service).Sync(ctx, db, model) ` and `(*Service).Remove(ctx, db, model)` (explicit calls for later reindex work), `Searchable`, `IndexSchemaProvider` (`SearchIndexSchema() map[string]any`), `SearchKeyer` (`SearchKey() string`), `Engine`, `EngineFactory`, `RegisterEngine(name, EngineFactory)`, `Query{Q string; QueryBy []string; FilterBy string; SortBy string; Page, PerPage int}`, `Gate` (`Enabled(ctx, db *gorm.DB) bool`), `GateFunc`, built-in engine `null`; GORM callback names `beachcomber:after_create`, `beachcomber:after_update`, `beachcomber:after_delete`.
|
||||
- beachcomber/typesense: `Config` (from `search.typesense.*`), `Engine` (`Upsert`, `Delete`, `Flush`, `SearchIDs`, `Configured`, `Name`), engine name `typesense` registered in init.
|
||||
- Config keys: `search.driver` (default null), `search.prefix` (""), `search.typesense.api_key` (""), `.host` (localhost), `.port` (8181), `.protocol` (http), `.path` (""), `.connection_timeout_seconds` (2), `.import_action` (upsert).
|
||||
- fonoteka: `(Album) SearchableAs`, `(*Album) ToSearchableArray`, `(Album) ShouldBeSearchable`, `(Album) SearchIndexSchema`, `(Album) MediumFamily() *string`, `AlbumMedia` map; `settingsGate`; `(*Plugin) wireSearch`; `config/search.yaml`.
|
||||
|
||||
## Flagged assumptions (edge probe: unclassified)
|
||||
|
||||
- SRCH-01 came back `unclassified` from the spec-less edge probe and stays `unresolved`. Planner reading for manual review: (a) restore of a soft-deleted album — PHP A4 (Winter SoftDelete firing restored) is unverified, so the Go port re-indexes on the update that clears deleted_at because it is an ordinary update; (b) concurrent saves of one album may upsert out of order, which the SQL re-gate in Phase 12 keeps safe; (c) an empty search result is an empty id list, never an error.
|
||||
- `Album.ShouldBeSearchable` returns true in Go: PHP returns the settings flag, but PHP also disables syncing entirely at boot when that flag is off, so the flag is only ever consulted when it is on. The Go Gate reproduces the kill-switch; behaviour is equivalent.
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="tracer">
|
||||
<name>Task 1: With the toggle on, a saved album is upserted into Typesense right after commit with its collection_id</name>
|
||||
<reversibility rating="costly">D-19: the Searchable and Engine interfaces are what every indexed model and future engine implements; locked by the user in CONTEXT.md, so flagged without a checkpoint.</reversibility>
|
||||
<precondition>Plans 11-01 and 11-03 are executed: `go doc ./modules/lagoon AfterCommit` and `go doc ./modules/lighthouse RegisterDriver` exit 0.</precondition>
|
||||
<files>modules/beachcomber/beachcomber.go, modules/beachcomber/searchable.go, modules/beachcomber/sync.go, modules/beachcomber/engines.go, modules/beachcomber/README.md, modules/beachcomber/typesense/config.go, modules/beachcomber/typesense/engine.go, README.md, ../fonoteka.go/config/search.yaml, ../fonoteka.go/plugins/golem15/fonoteka/plugin.go, ../fonoteka.go/plugins/golem15/fonoteka/search.go, ../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go</files>
|
||||
<read_first>modules/lagoon/transaction.go, modules/lagoon/ondatabase.go, modules/lighthouse/lighthouse.go and modules/lighthouse/drivers.go (From and registry idiom), modules/lighthouse/broadcast.go (callback registration and zero-PK skip), ../fonoteka.go/plugins/golem15/fonoteka/models/album.go, ../fonoteka.go/plugins/golem15/fonoteka/models/settings.go, ../fonoteka.go/plugins/golem15/fonoteka/models/artist.go and style.go and genre.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/join_tables.go, ../fonoteka.go/plugins/golem15/fonoteka/plugin.go (Boot), ../fonoteka.go/plugins/golem15/fonoteka/realtime.go (wiring precedent from 11-03), ../fonoteka.go/plugins/golem15/fonoteka/plugin_boot_test.go, /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/models/Album.php (MEDIA, mediumFamily, searchableAs, toSearchableArray, getSearchConfig), /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/Plugin.php (guardSearchSyncing), /media/nvme/dev/golem15/fonoteka/config/scout.php, /media/nvme/dev/golem15/fonoteka/vendor/laravel/scout/src/Engines/TypesenseEngine.php, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Pattern 10)</read_first>
|
||||
<action>(1) beachcomber core (D-19): searchable.go defines `Searchable`, optional `IndexSchemaProvider` and `SearchKeyer` (default key is the decimal primary key), `Engine` (`Name() string`, `Configured() bool`, `Upsert(ctx, index string, schema map[string]any, docs []map[string]any) error`, `Delete(ctx, index string, ids []string) error`, `Flush(ctx, index string) error`, `SearchIDs(ctx, index string, q Query) ([]string, error)`), `Query`, `Gate` (`Enabled(ctx context.Context, db *gorm.DB) bool`) and `GateFunc`. engines.go: `EngineFactory func(app *backpack.App) (Engine, error)`, `RegisterEngine(name, f)` (init-time registry; duplicate panics) and the built-in `null` engine (Configured false, every method a no-op). beachcomber.go: `Service` and `From(app)` (lookup-or-publish; reads `search.driver` default `null` and `search.prefix`; unknown driver is an error naming registered engines and the import requirement), `SetGate`, `Engine()`, and callback installation through `lagoon.OnDatabase` (`beachcomber:after_create` After `gorm:after_create`, `beachcomber:after_update` After `gorm:after_update`, `beachcomber:after_delete` After `gorm:after_delete`; idempotent per *gorm.DB).
|
||||
|
||||
(2) sync.go (D-20, Pattern 10): each callback skips non-Searchable statement types and zero primary keys (iterating slice elements for batch creates), then calls `lagoon.AfterCommit(db.Statement.Context, db, func(ctx, db) { svc.syncOne(ctx, db, type, key, op) })`. syncOne gates in order, each non-fatal and request-free: engine not Configured → skip; no *gorm.DB published → skip; Gate set and not Enabled → skip. Then it reloads a fresh instance by primary key with `Unscoped()`: not found, soft-deleted (a non-null gorm.DeletedAt) or op delete → `Delete(index, [key])`; `ShouldBeSearchable()` false → Delete; else `ToSearchableArray(ctx, db)` then `Upsert(index, schema, [doc])` where index = `search.prefix` + SearchableAs() and schema comes from IndexSchemaProvider (nil otherwise). Every error is logged Warn with index, key and operation (never the API key) and swallowed; a panic is recovered and logged. Public `(*Service).Sync(ctx, db, model)` and `Remove` run the same path directly.
|
||||
|
||||
(3) typesense sub-package (D-19): config.go reads `search.typesense.*` with Scout defaults (api_key "", host localhost, port 8181, protocol http, path "", connection_timeout_seconds 2, import_action upsert); engine.go builds base URL `{protocol}://{host}:{port}{path}`, an http.Client with the connection timeout, header `X-TYPESENSE-API-KEY` on every request; `Configured()` is api_key non-empty; `Upsert`: GET `/collections/{index}`, on 404 POST `/collections` with the schema plus `"name": index`, then POST `/collections/{index}/documents/import?action={import_action}` with one JSON object per line (Content-Type text/plain) and an error if any response line has `"success":false` (Typesense answers 200 even then) or the status is not 2xx; `Delete`: DELETE `/collections/{index}/documents/{url-escaped id}` per id, 404 counts as success; `Flush`: DELETE `/collections/{index}`, 404 success; `SearchIDs`: GET `/collections/{index}/documents/search` with q, query_by (comma-joined), filter_by, sort_by, page, per_page and returns `hits[].document.id`; `func init() { beachcomber.RegisterEngine("typesense", ...) }`.
|
||||
|
||||
(4) fonoteka Album (SRCH-01): new models/album_search.go porting PHP exactly — `var AlbumMedia = map[string]string{"LP": "vinyl", "2LP": "vinyl", "EP 7\"": "vinyl", "CD": "cd", "2CD": "cd", "MC": "cassette", "Box": "box"}`, `(a Album) MediumFamily() *string` (nil for nil/empty or unknown format), `(Album) SearchableAs() string` = `golem15_fonoteka_albums`, `(Album) ShouldBeSearchable() bool` = true (see flagged assumption), `(a *Album) ToSearchableArray(ctx, db)` that errors when CollectionID is 0, loads Genre, Styles and Artists (artists ordered by the pivot sort_order via the album_artists join table) through db, and returns the 17 PHP keys with PHP types (id string, ints for ids/year/created_at, joined style names with a space, empty strings for nil text), and `(Album) SearchIndexSchema() map[string]any` = the PHP collection-schema fields list with optional flags plus `default_sorting_field: created_at`. Keep models a leaf (framework imports only).
|
||||
|
||||
(5) fonoteka wiring: new search.go with a `settingsGate` whose Enabled reads the first golem15_fonoteka_settings row through the given db (Order id, Limit 1) and returns its SearchUseTypesense, false on any error or missing row; blank-import the typesense package; `func (p *Plugin) wireSearch(app *backpack.App) error` calls `beachcomber.From(app)` and `SetGate(settingsGate{})`; Boot in plugin.go calls it. New config/search.yaml: `driver: typesense`, `prefix: ""`, typesense block with the Scout defaults and empty api_key (comment the SUMMER_SEARCH__TYPESENSE__* names).
|
||||
|
||||
(6) Smoke `TestAlbumSearchSmoke` (search_smoke_test.go, Postgres, fresh `lagoon.Use` handle, fake Typesense via httptest recording method/path/headers/body): toggle on plus api_key set → creating an album inside lagoon.Transaction results, after commit, in GET collection (404) → POST /collections (schema has name golem15_fonoteka_albums) → POST import?action=upsert whose JSONL doc has a positive integer collection_id and string id; toggle off → zero requests; api_key empty → zero requests.
|
||||
|
||||
(7) Docs: new modules/beachcomber/README.md in the standard structure (H1, summary "Search index sync for GORM models: after-commit upserts and deletes through a pluggable engine, gated by an application kill-switch.", both import lines, Overview, Features, Usage with an acme model, API reference, Configuration, Dependencies, Testing), root README.md row with the same sentence; identifiers checked with go doc.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/beachcomber/... -count=1 && (cd ../fonoteka.go && go vet ./plugins/golem15/fonoteka/... && go test ./plugins/golem15/fonoteka -run '^TestAlbumSearchSmoke$' -count=1 -race -v)</automated>
|
||||
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS" for TestAlbumSearchSmoke, prints "no tests to run", "--- SKIP" or "DATA RACE".</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `go doc ./modules/beachcomber Searchable`, `go doc ./modules/beachcomber Engine`, `go doc ./modules/beachcomber Gate` and `go doc ./modules/beachcomber/typesense Engine.Upsert` exit 0.
|
||||
- `grep -c 'X-TYPESENSE-API-KEY' modules/beachcomber/typesense/engine.go` prints at least 1 and `grep -c 'import?action=' modules/beachcomber/typesense/engine.go` prints at least 1.
|
||||
- `grep -c 'lagoon.AfterCommit' modules/beachcomber/sync.go` prints at least 1.
|
||||
- `grep -c 'golem15_fonoteka_albums' ../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go` prints 1 and `grep -c 'default_sorting_field' ../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go` prints 1.
|
||||
- `grep -c 'wireSearch' ../fonoteka.go/plugins/golem15/fonoteka/plugin.go` prints 1.
|
||||
- `grep -c '\[beachcomber\](modules/beachcomber/README.md)' README.md` prints 1.
|
||||
</acceptance_criteria>
|
||||
<done>With the toggle on and Typesense configured, an album save produces a scoped upsert after commit; with the toggle off or no API key, Typesense is never contacted.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: Deletes remove documents, failures never break writes, and search returns candidate ids</name>
|
||||
<files>modules/beachcomber/sync.go, modules/beachcomber/typesense/engine.go, modules/beachcomber/README.md, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go, ../fonoteka.go/README.md</files>
|
||||
<read_first>modules/beachcomber/sync.go and modules/beachcomber/typesense/engine.go (Task 1), modules/lagoon/transaction.go, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go (Task 1), ../fonoteka.go/README.md (Configuration section from 11-03), /media/nvme/dev/golem15/fonoteka/vendor/laravel/scout/src/ModelObserver.php, /media/nvme/dev/golem15/fonoteka/config/scout.php</read_first>
|
||||
<behavior>
|
||||
- Soft-deleting an indexed album sends DELETE /collections/golem15_fonoteka_albums/documents/{id}; a 404 answer is not logged as a failure.
|
||||
- Typesense answering 500 on import (or an import line with success:false) leaves the album committed and logs one warning naming the index and key.
|
||||
- A Typesense endpoint that never answers is abandoned after connection_timeout_seconds and the write still returns.
|
||||
- A rolled-back lagoon.Transaction sends nothing; a write inside a plain gdb.Transaction syncs immediately through the tx handle.
|
||||
- A plain gdb.Create outside any explicit transaction syncs after its implicit commit.
|
||||
- An album whose collection_id is 0 is logged and never sent.
|
||||
- SearchIDs with filter_by `collection_id:=5` returns the ids from the fake response's hits in order.
|
||||
</behavior>
|
||||
<action>(1) Finish sync.go edge handling: explicit coverage of the delete/soft-delete branch, the plain-transaction and implicit-transaction paths provided by lagoon.AfterCommit, the timeout path (the engine's http.Client timeout bounds the inline call), and the refusal of a ToSearchableArray error (logged Warn, no request). Log lines never include the API key or document bodies.
|
||||
|
||||
(2) typesense/engine.go: confirm 404 handling for Delete and Flush, the success:false line scan for import, URL escaping of ids and index names, and SearchIDs parameter encoding (url.Values); return a typed error carrying the status code for non-2xx responses (without response bodies that could echo keys).
|
||||
|
||||
(3) Tests in search_smoke_test.go for the behavior list (`TestAlbumSearchDeleteAndFailures`), using the fake Typesense with switchable failure modes and a hanging handler for the timeout case (configure a 1s timeout in the test).
|
||||
|
||||
(4) Docs: modules/beachcomber/README.md documents the after-commit semantics, the three gates, delete-on-soft-delete and that SearchIDs results must be re-gated in SQL by callers. ../fonoteka.go/README.md Configuration gains the search mapping: SCOUT_DRIVER → SUMMER_SEARCH__DRIVER, SCOUT_PREFIX → SUMMER_SEARCH__PREFIX, TYPESENSE_API_KEY, TYPESENSE_HOST, TYPESENSE_PORT, TYPESENSE_PATH, TYPESENSE_PROTOCOL, TYPESENSE_CONNECTION_TIMEOUT_SECONDS and TYPESENSE_IMPORT_ACTION → SUMMER_SEARCH__TYPESENSE__*, plus the note that the admin setting search_use_typesense is the kill-switch.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./... && (cd ../fonoteka.go && go vet ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/... && go test ./plugins/golem15/fonoteka -run '^(TestAlbumSearchSmoke|TestAlbumSearchDeleteAndFailures)$' -count=1 -race -v && go test ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/...)</automated>
|
||||
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS" for TestAlbumSearchSmoke or TestAlbumSearchDeleteAndFailures, prints "no tests to run", "--- SKIP" or "DATA RACE"; any package reports FAIL.</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- TestAlbumSearchDeleteAndFailures asserts the album row still exists after a Typesense 500 and after a timeout.
|
||||
- `grep -c 'SUMMER_SEARCH__TYPESENSE__API_KEY' ../fonoteka.go/README.md` prints at least 1.
|
||||
- `grep -ci 're-gate' modules/beachcomber/README.md` prints at least 1.
|
||||
- `go doc ./modules/beachcomber/typesense Engine.SearchIDs` exits 0.
|
||||
</acceptance_criteria>
|
||||
<done>Album deletes clear their documents, every Typesense failure mode leaves writes committed with a warning, search returns candidate ids for Phase 12 to re-gate, and both repositories pass their full suites.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| Committed write → Typesense HTTP API | Album data, scoped by collection_id, leaves the database for an operator-run search server |
|
||||
| Typesense results → Phase 12 search endpoint | Candidate ids come back from an external index and must be re-authorized in SQL |
|
||||
| Admin setting → sync gate | The search_use_typesense toggle decides whether data is sent at all |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-11-07 | Information Disclosure | cross-collection search leak through a stale or mis-scoped index | high | mitigate | Every document carries a positive collection_id (ToSearchableArray refuses otherwise); SearchIDs returns candidates only and the README states callers re-gate in SQL (Phase 12 owns the endpoint re-gate) (Tasks 1-2). |
|
||||
| T-11-25 | Information Disclosure | Typesense API key in logs or errors | medium | mitigate | The key is only set as a request header; errors carry status codes, not bodies; sync logs name index, key and operation only (Task 2). |
|
||||
| T-11-26 | Denial of Service | search failure blocking or failing writes | medium | mitigate | Sync runs after commit, inline with the engine's connection timeout, and swallows every error and panic with a warning (Tasks 1-2). |
|
||||
| T-11-27 | Information Disclosure | data sent while the kill-switch is off | medium | mitigate | Gates run before any request: unconfigured engine, no database, and the settings Gate (errors mean off); tests assert zero requests (Task 1). |
|
||||
| T-11-SC | Tampering | Go module installs | high | mitigate | No module added; the Typesense client is hand-rolled on net/http per D-19. |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
After Task 2: `go vet ./... && go test ./...` in summercms.go and the fonoteka.go full vet/test command pass; TestAlbumSearchSmoke and TestAlbumSearchDeleteAndFailures pass under -race.
|
||||
Manual smoke (not blocking, collected at /gsd-verify-work): start `typesense/typesense:26.0`, enable search_use_typesense in the admin settings, save an album and query the collection.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- beachcomber and beachcomber/typesense exist with README and root row; engines selected by `search.driver`.
|
||||
- Album documents mirror PHP toSearchableArray and always carry a positive collection_id.
|
||||
- Sync is after-commit, inline, non-fatal, and fully skipped by the three gates.
|
||||
- fonoteka wires the Album binding and the settings Gate and documents the env mapping.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-05-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -0,0 +1,257 @@
|
||||
---
|
||||
phase: 11-jobs-realtime-and-search-infrastructure
|
||||
plan: 06
|
||||
type: execute
|
||||
wave: 3
|
||||
depends_on: ["11-01", "11-03"]
|
||||
files_modified:
|
||||
- modules/tide/centrifugo.go
|
||||
- modules/tide/centrifugo_golden.go
|
||||
- modules/tide/README.md
|
||||
- cmd/summer/parity.go
|
||||
- cmd/summer/main.go
|
||||
- cmd/summer/main_test.go
|
||||
- ../fonoteka.go/parity/php_parity.sh
|
||||
- ../fonoteka.go/parity/README.md
|
||||
- ../fonoteka.go/parity/broadcast_goldens_test.go
|
||||
- ../fonoteka.go/parity/fixtures/broadcasts/flows/album-lifecycle.yaml
|
||||
- ../fonoteka.go/parity/fixtures/broadcasts/flows/album-bulk.yaml
|
||||
- ../fonoteka.go/parity/fixtures/broadcasts/created.yaml
|
||||
- ../fonoteka.go/parity/fixtures/broadcasts/updated.yaml
|
||||
- ../fonoteka.go/parity/fixtures/broadcasts/deleted.yaml
|
||||
- ../fonoteka.go/parity/fixtures/broadcasts/bulk.yaml
|
||||
- ../fonoteka.go/parity/manifest.yaml
|
||||
- ../fonoteka.go/parity/routes.snapshot
|
||||
- ../fonoteka.go/parity/check_corpus.go
|
||||
- ../fonoteka.go/parity/parity_test.go
|
||||
- ../fonoteka.go/parity/realtime_seed_test.go
|
||||
- ../fonoteka.go/parity/capture-rules.yaml
|
||||
- ../fonoteka.go/parity/fixtures/routes/GET__api_realtime_token_realtime.yaml
|
||||
- ../fonoteka.go/parity/fixtures/routes/POST__api_realtime_subscribe_realtime.yaml
|
||||
autonomous: true
|
||||
requirements: [RT-01, RT-02, RT-03]
|
||||
estimate:
|
||||
tokens: 150000
|
||||
raw_tokens: 150000
|
||||
tasks: 3
|
||||
confidence: low
|
||||
must_haves:
|
||||
truths:
|
||||
- "Per D-10 and user decision 5, tide owns a fake Centrifugo HTTP recorder bound to loopback only: PHP's CENTRIFUGO_API_URL points at it during recorded flows, and it stores each publish/broadcast request as {method path, whether Authorization was present, JSON body} with the api key never written anywhere."
|
||||
- "Per D-10, broadcast goldens are stored as YAML under ../fonoteka.go/parity/fixtures/broadcasts with timestamps (data.timestamp and payload.timestamp) and the actor normalised, and captured ids replaced by `{{id:...}}` placeholders; the same recorder serves as the fake Centrifugo on the Go side and the structural diff ignores key order."
|
||||
- "Per D-10 and user decision 5, TestBroadcastGoldens proves the Go `deleted.fonoteka.album` publication and the Go `collection.bulk_updated` {reason: bulk_create, count} publication equal the PHP goldens after normalisation; the created and updated goldens are recorded and committed now, reported as pending Phase 12 (their `album` subtree), and never counted as passing."
|
||||
- "Per D-12 and RT-01/RT-02, the corpus gains two tracked extra routes, `GET /api/realtime/token realtime` and `POST /api/realtime/subscribe realtime`, recorded against isolated PHP and replayed green against the Go app: token 200 (the token captured into the private vars store as jwt:centrifugo, never committed) and 401 without a bearer; subscribe allow for a collection member (`{\"result\":{\"info\":[]}}`), allow on a presence channel (allow and override keys), and denies for a wrong secret, empty user, unknown namespace, `presence:presence:`, four segments and a non-member, all HTTP 200."
|
||||
- "The PHP-side test-only Centrifugo values (api key, token secret, proxy secret) are fixed constants shared by php_parity.sh and the Go replay config, the proxy secret reaches fixtures only as a `{{var}}` reference, and the fixture secret scan stays clean."
|
||||
- "check_corpus and parity_test counts move from 169 to 171 tracked routes and from 31 to 33 ported routes, with the realtime IDs listed next to the user-api extras so the 154-route routes.php digest lock is unchanged."
|
||||
artifacts:
|
||||
- path: "modules/tide/centrifugo.go"
|
||||
provides: "CentrifugoRecorder, Publication, loopback-only listener"
|
||||
contains: "type CentrifugoRecorder"
|
||||
- path: "modules/tide/centrifugo_golden.go"
|
||||
provides: "BroadcastGolden, LoadBroadcastGolden, WriteBroadcastGolden, NormalizePublications, DiffPublications"
|
||||
- path: "cmd/summer/parity.go"
|
||||
provides: "parity:broadcasts command"
|
||||
contains: "parity:broadcasts"
|
||||
- path: "../fonoteka.go/parity/broadcast_goldens_test.go"
|
||||
provides: "TestBroadcastGoldens"
|
||||
contains: "TestBroadcastGoldens"
|
||||
- path: "../fonoteka.go/parity/fixtures/broadcasts/deleted.yaml"
|
||||
provides: "PHP deleted.fonoteka.album golden"
|
||||
- path: "../fonoteka.go/parity/fixtures/broadcasts/bulk.yaml"
|
||||
provides: "PHP collection.bulk_updated golden"
|
||||
key_links:
|
||||
- from: "../fonoteka.go/parity/php_parity.sh"
|
||||
to: "modules/tide/centrifugo.go"
|
||||
via: "CENTRIFUGO_API_URL points PHP at the loopback recorder"
|
||||
pattern: "CENTRIFUGO_API_URL"
|
||||
- from: "../fonoteka.go/parity/broadcast_goldens_test.go"
|
||||
to: "modules/tide/centrifugo_golden.go"
|
||||
via: "Go publications diffed against PHP goldens"
|
||||
pattern: "DiffPublications|LoadBroadcastGolden"
|
||||
- from: "../fonoteka.go/parity/manifest.yaml"
|
||||
to: "../fonoteka.go/parity/fixtures/routes/POST__api_realtime_subscribe_realtime.yaml"
|
||||
via: "realtime auth group cases"
|
||||
pattern: "api/realtime/subscribe"
|
||||
prohibitions:
|
||||
- requirement_id: RT-03
|
||||
category: privacy
|
||||
statement: "Committed goldens and fixtures MUST NOT contain a live secret, a JWT, the Centrifugo API key or the proxy secret value"
|
||||
status: resolved
|
||||
verification: test
|
||||
- requirement_id: RT-03
|
||||
category: transparency
|
||||
statement: "Normalisation MUST NOT mask any field beyond timestamps, the actor and captured ids, and the created/updated goldens MUST NOT count as passing until Phase 12 asserts their album subtree"
|
||||
status: resolved
|
||||
verification: test
|
||||
---
|
||||
|
||||
## Phase Goal
|
||||
|
||||
ROADMAP Phase 11 goal (verbatim, not in user-story form): River jobs run on the correct dual-driver split, Centrifugo publishing and channel authorization match the existing server, and Typesense sync stays a re-gated pre-filter — all brought up before the API phases that depend on them.
|
||||
|
||||
This plan's slice: the realtime contract is proven against real PHP output, both for what Centrifugo receives when albums change and for the two HTTP routes the Nuxt app and Centrifugo call (RT-01, RT-02, RT-03 parity evidence; "API parity is the acceptance test").
|
||||
|
||||
<objective>
|
||||
Extend the framework parity toolkit `tide` with a fake Centrifugo recorder and broadcast golden files, record Płytarium's album broadcasts and realtime routes from isolated PHP, and replay them against the Go app.
|
||||
|
||||
Purpose: D-10 requires payload parity proven against real PHP output, and the project's acceptance rule is that the PHP contract is replayed, not reasoned about. Decisions implemented: D-10; user decision 5 (fake-Centrifugo recorder, created/updated recorded now and asserted in Phase 12); D-12/D-13 route contracts through recorded fixtures.
|
||||
Output: tide recorder and golden helpers with README, `summer parity:broadcasts`, fonoteka.go broadcast goldens and TestBroadcastGoldens, two realtime manifest routes with recorded fixtures replayed green.
|
||||
|
||||
Repos: summercms.go (tide, summer CLI) and fonoteka.go (parity corpus). tide stays application-agnostic (neutral names in its code, tests and README). Planning docs and code in separate commits. Never add co-author tags.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@~/.claude/gsd-core/workflows/execute-plan.md
|
||||
@~/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-03-SUMMARY.md
|
||||
@.planning/phases/02-api-parity-harness-bootstrap/02-CONTEXT.md
|
||||
@modules/tide/flow.go
|
||||
@modules/tide/record.go
|
||||
@modules/tide/proxy.go
|
||||
@modules/tide/diff.go
|
||||
@modules/tide/variables.go
|
||||
@cmd/summer/parity.go
|
||||
@../fonoteka.go/parity/README.md
|
||||
@../fonoteka.go/parity/php_parity.sh
|
||||
@../fonoteka.go/parity/parity_test.go
|
||||
@../fonoteka.go/parity/check_corpus.go
|
||||
@../fonoteka.go/parity/capture-rules.yaml
|
||||
|
||||
<interfaces>
|
||||
- tide: `Flow{Version, Name, Description, SeedHook, Steps}`, `Step{ID, RouteID, Request, Response, Capture, Normalize, Headers}`, `RecordFlow(ctx, spec Flow, RecordConfig{Target, Client, MaxBody, Store, Rules}) (Flow, error)`, `LoadRules(path)`, `Store` (0600 vars store, `{{name}}` substitution in variables.go), structural JSON diff (key order ignored, missing keys and token-type changes fail at $.path; diff.go), `requireLoopbackAddr(hostport)` and `isLoopbackHost` in proxy.go (T-02-01), goccy/go-yaml with DisallowUnknownField for fixtures.
|
||||
- cmd/summer/parity.go: `parityProxyCommand`, `parityRecordCommand`, `parityReplayCommand` (the shape to copy for a new command).
|
||||
- fonoteka parity: `newConfiguredTarget(t, db)` (app.Handler over the TestMain pool with testConfig), `seedHooks` map (`genres`, `user-api`, `synthetic-item`), `const expectedPHPRoutes = 169`, `const expectedPortedRoutes = 31`; check_corpus.go `expectedRouteCount = 169`, `userAPIRouteIDs`, `comparePHPSnapshot` (154 routes.php IDs plus tracked extras); capture-rules.yaml already captures `$.token` of `GET /api/realtime/token` as `jwt:centrifugo` (category jwt).
|
||||
- From plan 11-03: fonoteka mounts `GET /api/realtime/token` (jwt.auth, throttle:ws-api) and `POST /api/realtime/subscribe` (raw, throttle:ws-api); config keys `realtime.driver`, `realtime.centrifugo.{api_url, api_key, token_secret, proxy_secret}`; `lighthouse.WithoutBroadcasting[T]`, `(*lighthouse.Service).Emit`, Album binding with alias `fonoteka.album`; `conga.StartWorker`.
|
||||
- PHP: isolated server via `parity/php_parity.sh reset|serve|artisan` on 127.0.0.1:8423 with QUEUE_CONNECTION=sync (BroadcastEventJob runs inline); album store/update/bulk publish synchronously through publishAlbumBroadcast; destroy relies on the model's deleted event; CentrifugoClient sends nothing when CENTRIFUGO_API_KEY is empty.
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
## Artifacts this phase produces
|
||||
|
||||
(This plan's share.)
|
||||
|
||||
- tide: `CentrifugoRecorder` (`http.Handler`; `NewCentrifugoRecorder(CentrifugoRecorderOptions{APIKey string})`, `Publications() []Publication`, `Reset()`, `ListenAndServe(ctx, addr)` loopback-only), `Publication{Method, Path string; Authorization bool; Body json.RawMessage}`, `BroadcastGolden{Version, Name, Flow, Pending string; Publications []Publication}`, `LoadBroadcastGolden(path)`, `WriteBroadcastGolden(path, g)`, `NormalizePublications(pubs, store) ([]Publication, error)`, `DiffPublications(expected, actual []Publication) []Diff`.
|
||||
- CLI: `summer parity:broadcasts --flow <file> --target <php url> --vars <file> --listen 127.0.0.1:8424 --out <golden> [--name <name>]`.
|
||||
- fonoteka parity: goldens `fixtures/broadcasts/{created,updated,deleted,bulk}.yaml`, flows `fixtures/broadcasts/flows/{album-lifecycle,album-bulk}.yaml`, `TestBroadcastGoldens`, seed hook `realtime`, auth group `realtime`, manifest IDs `GET /api/realtime/token realtime` and `POST /api/realtime/subscribe realtime`, fixtures `fixtures/routes/GET__api_realtime_token_realtime.yaml` and `POST__api_realtime_subscribe_realtime.yaml`, php_parity.sh env `CENTRIFUGO_API_URL`, `CENTRIFUGO_API_KEY`, `CENTRIFUGO_SECRET`, `CENTRIFUGO_PROXY_SECRET`.
|
||||
|
||||
<!-- planner-discipline-allow: parity-centrifugo-api-key -->
|
||||
<!-- planner-discipline-allow: parity-centrifugo-proxy-secret -->
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="tracer">
|
||||
<name>Task 1: PHP's album delete publication is recorded through the tide recorder and the Go delete broadcast matches it</name>
|
||||
<precondition>The isolated PHP stack can run: `command -v php` succeeds and `test -f /media/nvme/dev/golem15/fonoteka/artisan` succeeds; plan 11-03 is executed (`go doc ./modules/lighthouse WithoutBroadcasting` exits 0); `docker info` exits 0.</precondition>
|
||||
<files>modules/tide/centrifugo.go, modules/tide/centrifugo_golden.go, modules/tide/README.md, cmd/summer/parity.go, cmd/summer/main.go, cmd/summer/main_test.go, ../fonoteka.go/parity/php_parity.sh, ../fonoteka.go/parity/broadcast_goldens_test.go, ../fonoteka.go/parity/fixtures/broadcasts/flows/album-lifecycle.yaml, ../fonoteka.go/parity/fixtures/broadcasts/deleted.yaml</files>
|
||||
<read_first>modules/tide/flow.go, modules/tide/record.go, modules/tide/proxy.go (requireLoopbackAddr, isLoopbackHost), modules/tide/diff.go, modules/tide/variables.go, modules/tide/fixture.go, modules/tide/README.md, cmd/summer/parity.go, cmd/summer/main.go, cmd/summer/main_test.go, ../fonoteka.go/parity/php_parity.sh, ../fonoteka.go/parity/README.md, ../fonoteka.go/parity/fixtures/seed/bootstrap.yaml, ../fonoteka.go/parity/fixtures/routes/DELETE___fonoteka_api_v1_albums_{id}_jwt.yaml, ../fonoteka.go/plugins/golem15/fonoteka/realtime.go and realtime_smoke_test.go (from 11-03), /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/controllers/api/AlbumApiController.php (store, update, destroy, bulk), /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/classes/CentrifugoClient.php, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Open Question 4, Pitfall 15)</read_first>
|
||||
<action>(1) tide recorder (D-10, user decision 5), new modules/tide/centrifugo.go: `type Publication struct { Method, Path string; Authorization bool; Body json.RawMessage }` (YAML field names method, path, authorization, body with the body as a literal block like other fixtures); `type CentrifugoRecorderOptions struct { APIKey string }`; `NewCentrifugoRecorder(opts) *CentrifugoRecorder` implementing http.Handler: POST paths ending in `/publish` or `/broadcast` are recorded (Authorization true only when the header equals `apikey <APIKey>`; the header value itself is never stored) and answered `200 {"result":{}}`; `/presence` answers `{"result":{"presence":{}}}`, `/unsubscribe` and `/info` answer `{"result":{}}`; other paths 404; bodies capped at 1 MiB; `Publications()` returns a copy in arrival order, `Reset()` clears; `ListenAndServe(ctx, addr)` refuses non-loopback addresses with the proxy's loopback rule (T-02-01) and stops on ctx cancel. New centrifugo_golden.go: `BroadcastGolden{Version int; Name, Flow, Pending string; Publications []Publication}` with goccy/go-yaml load (DisallowUnknownField) and write helpers; `NormalizePublications(pubs []Publication, store *Store) ([]Publication, error)` replaces `$.data.timestamp`, `$.data.payload.timestamp` with the literal `{{timestamp}}`, `$.data.payload.actor` with `{{actor}}`, and any string or number equal to a captured `id:*` value of the store (including inside channel names such as `collection:12`) with that `{{id:name}}` placeholder, and nothing else; `DiffPublications(expected, actual)` compares count, method, path, authorization and the body with tide's structural JSON diff, returning Diffs with `$[i].body...` paths.
|
||||
|
||||
(2) CLI: in cmd/summer/parity.go add `parityBroadcastsCommand()` named `parity:broadcasts` (flags flow, target, vars, listen default `127.0.0.1:8424`, out, name, api-key default from env `PARITY_CENTRIFUGO_API_KEY`): start the recorder, run the flow against the PHP target with tide.RecordFlow using the vars store (so captured ids and JWTs stay in the 0600 file), wait briefly for trailing publishes, normalise with the store and write the golden; refuse a non-loopback target like the proxy does. Register it in toolCommands and the main_test.go expected list. Document it in modules/tide/README.md (recorder, golden format, normalisation rules; neutral names only).
|
||||
|
||||
(3) PHP env: php_parity.sh export_env adds `CENTRIFUGO_API_URL=${CENTRIFUGO_API_URL:-http://127.0.0.1:8424/api}`, `CENTRIFUGO_API_KEY=${CENTRIFUGO_API_KEY:-parity-centrifugo-api-key}`, `CENTRIFUGO_SECRET=${CENTRIFUGO_SECRET:-parity-centrifugo-token-secret}` and `CENTRIFUGO_PROXY_SECRET=${CENTRIFUGO_PROXY_SECRET:-parity-centrifugo-proxy-secret}` — fixed test-only values for the isolated instance, never production secrets (Phase 3 test-secret precedent); leave BROADCAST_ENABLED as is (it only affects the Laravel broadcaster, not model broadcasts).
|
||||
|
||||
(4) Record: write flows/album-lifecycle.yaml (seed via the bootstrap identities; steps: create an album in alice's collection with POST /_fonoteka/api/v1/albums capturing its id as `id:album`, update it with PUT, delete it with DELETE, each step carrying its own golden target) — or split the lifecycle into per-step recordings, whichever keeps one golden per event; run the isolated PHP (`php_parity.sh reset`, `serve`), then `summer parity:broadcasts` to produce fixtures/broadcasts/deleted.yaml (and the created/updated goldens in Task 2). The deleted golden holds every publication PHP emitted during the delete step; if PHP emits more than the deleted event on a soft delete, the Go side must reproduce it and the SUMMARY records the finding.
|
||||
|
||||
(5) Go side, new ../fonoteka.go/parity/broadcast_goldens_test.go `TestBroadcastGoldens` with subtest `deleted`: boot the app on the TestMain pool with realtime config pointing api_url at an httptest server wrapping `tide.NewCentrifugoRecorder` (api key = the same test-only constant), start `conga.StartWorker` for the app, seed a kind=collection collection and an album for a frontend user, delete the album inside `lagoon.Transaction` with `bouncer.WithUser` ctx, wait up to 5s for the publication, normalise it with a store holding the Go ids under the golden's placeholder names, and require `DiffPublications(golden, got)` to be empty.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/tide ./cmd/summer -count=1 && (cd ../fonoteka.go && go test ./parity -run '^TestBroadcastGoldens$/^deleted$' -count=1 -v)</automated>
|
||||
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS: TestBroadcastGoldens/deleted", prints "no tests to run" or "--- SKIP"; the diff output lists any $[i] path.</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `go doc ./modules/tide CentrifugoRecorder`, `go doc ./modules/tide NormalizePublications` and `go doc ./modules/tide DiffPublications` exit 0.
|
||||
- `grep -c 'CENTRIFUGO_API_URL' ../fonoteka.go/parity/php_parity.sh` prints 1.
|
||||
- `grep -c 'deleted.fonoteka.album' ../fonoteka.go/parity/fixtures/broadcasts/deleted.yaml` prints at least 1 and `grep -c '{{timestamp}}' ../fonoteka.go/parity/fixtures/broadcasts/deleted.yaml` prints at least 1.
|
||||
- `grep -c 'parity-centrifugo-api-key' ../fonoteka.go/parity/fixtures/broadcasts/deleted.yaml` prints 0 (the key never reaches a golden).
|
||||
- `grep -c 'parity:broadcasts' cmd/summer/main_test.go` prints at least 1.
|
||||
</acceptance_criteria>
|
||||
<done>A real PHP album delete is captured as a normalised golden and the Go app's delete broadcast produces the same Centrifugo request.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Bulk-add parity is proven and the created/updated goldens are recorded for Phase 12</name>
|
||||
<files>../fonoteka.go/parity/fixtures/broadcasts/flows/album-bulk.yaml, ../fonoteka.go/parity/fixtures/broadcasts/created.yaml, ../fonoteka.go/parity/fixtures/broadcasts/updated.yaml, ../fonoteka.go/parity/fixtures/broadcasts/bulk.yaml, ../fonoteka.go/parity/broadcast_goldens_test.go, ../fonoteka.go/parity/README.md</files>
|
||||
<read_first>../fonoteka.go/parity/broadcast_goldens_test.go and fixtures/broadcasts/* (Task 1), ../fonoteka.go/parity/README.md, /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/controllers/api/AlbumApiController.php (bulk), ../fonoteka.go/parity/fixtures/routes/POST___fonoteka_api_v1_albums_bulk_jwt.yaml (request shape), modules/tide/centrifugo_golden.go</read_first>
|
||||
<action>(1) Record from isolated PHP with `summer parity:broadcasts`: created.yaml and updated.yaml from the lifecycle flow's create and update steps, and bulk.yaml from flows/album-bulk.yaml (POST /_fonoteka/api/v1/albums/bulk with two albums into alice's collection, expecting exactly one `collection.bulk_updated` publication with payload {reason: bulk_create, count: 2}). Set `pending: "Phase 12 asserts the album subtree"` in created.yaml and updated.yaml.
|
||||
|
||||
(2) TestBroadcastGoldens subtests: `bulk` creates two albums inside `lighthouse.WithoutBroadcasting[models.Album]` and `lagoon.Transaction`, then calls `svc.Emit` with channels `collection:<id>`, event `collection.bulk_updated` and payload {reason: bulk_create, count: 2} in the same transaction, and requires an empty diff against bulk.yaml (exactly one publication); `created` and `updated` load their goldens, check they parse and name `created.fonoteka.album` / `updated.fonoteka.album`, and call `t.Skip` with the golden's pending text so the run reports them pending, never as passes (D-16 of Phase 2: pending never equals passing).
|
||||
|
||||
(3) ../fonoteka.go/parity/README.md gains a "Broadcast goldens" section: the recorder port, the php_parity.sh CENTRIFUGO_* defaults, the parity:broadcasts command lines used, the normalisation rules and the Phase 12 pending note.</action>
|
||||
<verify>
|
||||
<automated>(cd ../fonoteka.go && go test ./parity -run '^TestBroadcastGoldens$' -count=1 -v)</automated>
|
||||
<fails_when>Non-zero exit; the output lacks "--- PASS: TestBroadcastGoldens/deleted" or "--- PASS: TestBroadcastGoldens/bulk"; created or updated appears as "--- PASS" instead of "--- SKIP"; the output prints "no tests to run".</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `grep -c 'collection.bulk_updated' ../fonoteka.go/parity/fixtures/broadcasts/bulk.yaml` prints at least 1 and `grep -c 'bulk_create' ../fonoteka.go/parity/fixtures/broadcasts/bulk.yaml` prints at least 1.
|
||||
- `grep -c 'Phase 12' ../fonoteka.go/parity/fixtures/broadcasts/created.yaml ../fonoteka.go/parity/fixtures/broadcasts/updated.yaml` prints at least 1 for each file.
|
||||
- `grep -c 'Broadcast goldens' ../fonoteka.go/parity/README.md` prints at least 1.
|
||||
</acceptance_criteria>
|
||||
<done>Deleted and bulk broadcasts are byte-equivalent to PHP after normalisation, and created/updated goldens are committed and visibly pending Phase 12.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: The Nuxt token route and Centrifugo's subscribe proxy replay green against recorded PHP fixtures</name>
|
||||
<files>../fonoteka.go/parity/manifest.yaml, ../fonoteka.go/parity/routes.snapshot, ../fonoteka.go/parity/check_corpus.go, ../fonoteka.go/parity/parity_test.go, ../fonoteka.go/parity/realtime_seed_test.go, ../fonoteka.go/parity/capture-rules.yaml, ../fonoteka.go/parity/fixtures/routes/GET__api_realtime_token_realtime.yaml, ../fonoteka.go/parity/fixtures/routes/POST__api_realtime_subscribe_realtime.yaml, ../fonoteka.go/parity/README.md</files>
|
||||
<read_first>../fonoteka.go/parity/manifest.yaml (auth_groups, a user-api entry), ../fonoteka.go/parity/check_corpus.go (expectedRouteCount, userAPIRouteIDs, comparePHPSnapshot), ../fonoteka.go/parity/routes.snapshot, ../fonoteka.go/parity/parity_test.go (expectedPHPRoutes, expectedPortedRoutes, seedHooks, newConfiguredTarget, testConfig), ../fonoteka.go/parity/user_api_seed_test.go (seed hook precedent), ../fonoteka.go/parity/capture-rules.yaml, ../fonoteka.go/parity/README.md (Record/Replay), /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/routes.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/http/controllers/ProxyController.php, .planning/phases/07-user-plugin-and-authentication/07-CONTEXT.md (D-11 extra manifest IDs)</read_first>
|
||||
<action>(1) Manifest (D-12, D-13; Phase 7 D-11 precedent for extra IDs): add auth group `realtime` and two routes with `status: ported` and `seed_hook: realtime`: `GET /api/realtime/token realtime` with cases `case` (alice bearer → 200, token captured by the existing jwt:centrifugo rule) and `no-bearer` (401 from jwt.auth); `POST /api/realtime/subscribe realtime` with JSON bodies shaped like Centrifugo's subscribe proxy request (`client`, `transport`, `protocol`, `encoding`, `user`, `channel`) and header `X-Centrifugo-Secret: "{{secret:centrifugo_proxy}}"` (add the header to the route's keep list in capture-rules.yaml so it is replayed, and classify the value so the fixture holds only the reference), cases: member allow on `collection:{{id:collection}}`, presence allow on `presence:` channel for a namespace whose authorizer allows presence if the PHP stack has one (otherwise record the PHP deny on `presence:collection:{{id:collection}}`, which is the ported behaviour), wrong secret, empty user, unknown namespace, `presence:presence:collection:1`, four segments, non-member collection — each response recorded as PHP returns it (all HTTP 200).
|
||||
|
||||
(2) Record against isolated PHP (php_parity.sh env from Task 1 carries the proxy secret) with `summer parity:record --manifest ... --fixtures ... --target http://127.0.0.1:8423 --vars /tmp/summercms-parity/vars.yaml --resume true --next-batch 2`, storing the proxy secret as `secret:centrifugo_proxy` in the private vars file only.
|
||||
|
||||
(3) Go replay: new realtime_seed_test.go `seedRealtime` hook (registered as `realtime` in seedHooks) that seeds alice, her kind=collection collection and a second user without membership through direct Postgres like the user-api hook, and stores `secret:centrifugo_proxy` = the php_parity.sh test-only constant plus the ids the fixtures reference; testConfig/newConfiguredTarget add realtime config (driver centrifugo, token_secret and proxy_secret = the same test-only constants, api_url pointing at an unreachable loopback port so no publish leaves the test). Update counts: check_corpus.go `expectedRouteCount` 171 with a `realtimeRouteIDs` list folded into comparePHPSnapshot next to userAPIRouteIDs; parity_test.go `expectedPHPRoutes` 171 and `expectedPortedRoutes` 33; routes.snapshot gains the two IDs without changing its PHP digest line. Record the steps in parity/README.md.
|
||||
|
||||
(4) Run TestParityCorpus and the corpus checker (including its secret scan) and fix any byte difference on the Go side (Cache-Control, Content-Type, no trailing newline, `[]` info) — never by editing recorded PHP bytes.</action>
|
||||
<verify>
|
||||
<automated>(cd ../fonoteka.go && go test ./parity -run '^(TestParityCorpus|TestParsePHPRoutesCountAndGroups|TestUniqueAndSecretScan|TestCompareIDSets|TestParityContract)$' -count=1 -v)</automated>
|
||||
<fails_when>Non-zero exit; the output lacks "--- PASS" for TestParityCorpus, TestUniqueAndSecretScan or TestParsePHPRoutesCountAndGroups; any subtest for "GET /api/realtime/token realtime" or "POST /api/realtime/subscribe realtime" reports FAIL; the coverage summary line reports "failing" above 0 or a recorded total other than 171.</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `grep -c 'api/realtime/token realtime' ../fonoteka.go/parity/manifest.yaml ../fonoteka.go/parity/routes.snapshot ../fonoteka.go/parity/check_corpus.go` prints at least 1 for each file.
|
||||
- `grep -cE 'expectedRouteCount\s*=\s*171' ../fonoteka.go/parity/check_corpus.go` prints 1 and `grep -cE 'expectedPortedRoutes\s*=\s*33' ../fonoteka.go/parity/parity_test.go` prints 1.
|
||||
- `grep -c 'parity-centrifugo-proxy-secret' ../fonoteka.go/parity/fixtures/routes/POST__api_realtime_subscribe_realtime.yaml` prints 0 and `grep -c '{{secret:centrifugo_proxy}}' ../fonoteka.go/parity/fixtures/routes/POST__api_realtime_subscribe_realtime.yaml` prints at least 1.
|
||||
- `grep -c '"info":\[\]' ../fonoteka.go/parity/fixtures/routes/POST__api_realtime_subscribe_realtime.yaml` prints at least 1.
|
||||
</acceptance_criteria>
|
||||
<done>The token route and subscribe proxy are part of the recorded corpus and the Go app replays every recorded case byte-compatibly, with no secret committed.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| Isolated PHP stack → tide recorder (loopback) | PHP sends Centrifugo API calls with an api key to a local recorder |
|
||||
| Recorded fixtures/goldens → git | Anything captured may be committed |
|
||||
| Developer shell → summer parity:broadcasts | Targets and listen addresses come from flags |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-11-28 | Information Disclosure | goldens and route fixtures | high | mitigate | The recorder stores only whether Authorization matched; JWTs and the proxy secret go to the 0600 vars store and appear as `{{var}}` references; test-only constants are used for the isolated PHP; the corpus secret scan runs in Task 3 and a grep asserts the constants are absent from fixtures. |
|
||||
| T-11-29 | Spoofing | recorder listener and parity:broadcasts target | medium | mitigate | Recorder listen address and PHP target must be loopback, reusing tide's proxy rule (T-02-01) (Task 1). |
|
||||
| T-11-30 | Repudiation | golden normalisation hiding regressions | medium | mitigate | Only timestamps, actor and captured ids are normalised; created/updated are reported pending, never passing; DiffPublications checks count, path and authorization too (Tasks 1-2). |
|
||||
| T-11-SC | Tampering | Go module installs | high | mitigate | No module added; tide keeps goccy/go-yaml v1.19.2 and stdlib net/http. |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
After Task 3: `go vet ./... && go test ./...` in summercms.go; `(cd ../fonoteka.go && go vet ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/... && go test ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/...)`; TestBroadcastGoldens shows deleted and bulk passing with created/updated skipped as pending; TestParityCorpus reports 171 tracked routes with no failures.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- tide has a loopback-only fake Centrifugo recorder, golden I/O, normalisation and diff, documented without application names.
|
||||
- PHP deleted and bulk broadcasts are replayed byte-equivalent by Go; created/updated goldens are stored and pending Phase 12.
|
||||
- The realtime token and subscribe routes are recorded from PHP and replay green; counts updated to 171/33.
|
||||
- No live or test secret value is committed in fixtures or goldens.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-06-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -0,0 +1,252 @@
|
||||
---
|
||||
phase: 11-jobs-realtime-and-search-infrastructure
|
||||
plan: 07
|
||||
type: execute
|
||||
wave: 5
|
||||
depends_on: ["11-01", "11-02", "11-03", "11-04", "11-05", "11-06"]
|
||||
files_modified:
|
||||
- modules/conga/manager_test.go
|
||||
- modules/conga/worker_test.go
|
||||
- modules/conga/commands_test.go
|
||||
- modules/conga/schedule_test.go
|
||||
- modules/conga/listen_test.go
|
||||
- modules/lagoon/ondatabase_test.go
|
||||
- modules/lagoon/transaction_test.go
|
||||
- modules/lagoon/queue_migrations_test.go
|
||||
- modules/bonfire/call_test.go
|
||||
- modules/pact/capabilities_test.go
|
||||
- modules/lighthouse/postgres_test.go
|
||||
- modules/lighthouse/lighthouse_test.go
|
||||
- modules/lighthouse/channel_test.go
|
||||
- modules/lighthouse/registry_test.go
|
||||
- modules/lighthouse/route_test.go
|
||||
- modules/lighthouse/broadcast_test.go
|
||||
- modules/lighthouse/centrifugo/token_test.go
|
||||
- modules/lighthouse/centrifugo/client_test.go
|
||||
- modules/lighthouse/centrifugo/proxy_test.go
|
||||
- modules/lighthouse/centrifugo/commands_test.go
|
||||
- modules/flare/flare_test.go
|
||||
- modules/flare/encrypt_test.go
|
||||
- modules/flare/commands_test.go
|
||||
- modules/beachcomber/postgres_test.go
|
||||
- modules/beachcomber/sync_test.go
|
||||
- modules/beachcomber/typesense/engine_test.go
|
||||
- modules/tide/centrifugo_test.go
|
||||
- scripts/check-phase11.sh
|
||||
- .planning/phases/11-jobs-realtime-and-search-infrastructure/11-SECURITY-REVIEW.md
|
||||
- .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VALIDATION.md
|
||||
- .planning/REQUIREMENTS.md
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/ws_authorizer_test.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/album_realtime_test.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/album_search_test.go
|
||||
- ../fonoteka.go/plugins/golem15/fonoteka/schedule_test.go
|
||||
autonomous: true
|
||||
requirements: [JOBS-01, CLI-04, CLI-06, RT-01, RT-02, RT-03, SRCH-01]
|
||||
estimate:
|
||||
tokens: 180000
|
||||
raw_tokens: 180000
|
||||
tasks: 3
|
||||
confidence: low
|
||||
must_haves:
|
||||
truths:
|
||||
- "Per CLAUDE.md rule 3 and QA-03, every Phase 11 package (conga, lagoon seams, pact schedule types, bonfire.Call, lighthouse, lighthouse/centrifugo, flare, beachcomber, beachcomber/typesense, the tide recorder) and every fonoteka binding (ws authorizers, Album broadcast binding, Album Searchable, settings Gate, schedule entry) has branch-level tests, with Postgres tests on testcontainers and skipped only under -short."
|
||||
- "The VALIDATION map's named tests exist and pass: TestListenPickupLatency, TestDispatchTransactional, TestOutcome*, TestCancel*, TestQueueClear, TestQueueWork, TestSchedule*, TestCall, TestToken*, TestClient*, TestProxy*, TestWsAuthorizer, TestBroadcastTx, TestSuppression, TestBulkEmitsOnce, TestOnDatabaseAfterActivate, TestBroadcastGoldens, TestSync*, TestAlbumSearchable."
|
||||
- "Every high threat marked mitigate in plans 11-01..11-06 (T-11-01, T-11-02, T-11-05, T-11-07, T-11-09, T-11-12, T-11-19, T-11-22, T-11-28) has a named test that fails when its mitigation is removed, proven by an anchor-exact removal check recorded as an RC row in 11-SECURITY-REVIEW.md (10.1-04 precedent)."
|
||||
- "`scripts/check-phase11.sh --all` runs self-test, hygiene, both repositories' vet and tests (including the fonoteka plugin module patterns and -race on the realtime packages) and the evidence check, fails closed on any skipped or missing named test, and prints `phase11 all passed`."
|
||||
- "The hygiene stage refuses application names in the new framework modules and tide's recorder files, refuses the Centrifugo, Typesense and Web Push client libraries and a direct cron library in go.mod, and requires River at v0.47.0; its self-test proves each rule refuses a planted violation for its own reason."
|
||||
- "11-VALIDATION.md is validated (nyquist_compliant true, every row mapped to a task and a passing command) and REQUIREMENTS.md marks JOBS-01, CLI-04, CLI-06, RT-01, RT-02, RT-03 and SRCH-01 complete for Phase 11."
|
||||
- statement: "Real-client behaviour (the unchanged Nuxt app receiving an album event through a real Centrifugo v6 with a Go-issued token, and a real Typesense 26.0 receiving an upsert) is confirmed manually at /gsd-verify-work."
|
||||
verification: backstop
|
||||
artifacts:
|
||||
- path: "scripts/check-phase11.sh"
|
||||
provides: "phase gate with --self-test, --hygiene, --go, --postgres, --evidence, --all"
|
||||
contains: "phase11 all passed"
|
||||
- path: ".planning/phases/11-jobs-realtime-and-search-infrastructure/11-SECURITY-REVIEW.md"
|
||||
provides: "threat rows T-11-01..T-11-30 plus T-11-SC and RC removal-check rows"
|
||||
- path: "../fonoteka.go/plugins/golem15/fonoteka/ws_authorizer_test.go"
|
||||
provides: "TestWsAuthorizer"
|
||||
contains: "TestWsAuthorizer"
|
||||
- path: "modules/lighthouse/broadcast_test.go"
|
||||
provides: "TestBroadcastTx, TestSuppression, TestBulkEmitsOnce"
|
||||
key_links:
|
||||
- from: "scripts/check-phase11.sh"
|
||||
to: ".planning/phases/11-jobs-realtime-and-search-infrastructure/11-SECURITY-REVIEW.md"
|
||||
via: "--evidence requires an RC row per high mitigated threat"
|
||||
pattern: "RC-"
|
||||
- from: "scripts/check-phase11.sh"
|
||||
to: "../fonoteka.go"
|
||||
via: "go vet/test with ./plugins/golem15/fonoteka/... and ./plugins/golem15/user/..."
|
||||
pattern: "plugins/golem15/fonoteka/\\.\\.\\."
|
||||
prohibitions:
|
||||
- requirement_id: JOBS-01
|
||||
category: transparency
|
||||
statement: "The phase gate MUST NOT report success when a named test is skipped, reports no tests to run, or a detector stage is disabled"
|
||||
status: resolved
|
||||
verification: test
|
||||
---
|
||||
|
||||
## Phase Goal
|
||||
|
||||
ROADMAP Phase 11 goal (verbatim, not in user-story form): River jobs run on the correct dual-driver split, Centrifugo publishing and channel authorization match the existing server, and Typesense sync stays a re-gated pre-filter — all brought up before the API phases that depend on them.
|
||||
|
||||
This plan's slice: the phase's code is fully covered, the security mitigations are proven to fail when removed, and one fail-closed gate proves the phase in both repositories (all seven requirements).
|
||||
|
||||
<objective>
|
||||
Bring full unit and integration coverage to everything Phase 11 added, write the failing-when-broken security evidence, and add the `scripts/check-phase11.sh` gate, following the Phase 10.1 plan 04 shape.
|
||||
|
||||
Purpose: CLAUDE.md rule 3 (unit tests are the last plan of every phase) and QA-03. Decisions exercised: D-01..D-20 through their plan tests; this plan adds branch coverage and evidence, not features.
|
||||
Output: test files across both repositories, the phase gate, 11-SECURITY-REVIEW.md, a validated 11-VALIDATION.md, REQUIREMENTS.md traceability.
|
||||
|
||||
Repos: summercms.go (framework tests, gate, planning docs) and fonoteka.go (binding tests). A test that uncovers a real bug gets a separate `fix` commit with its RED evidence in the SUMMARY (10.1-04 precedent) so every commit stays green. Planning docs and code in separate commits. Never add co-author tags.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@~/.claude/gsd-core/workflows/execute-plan.md
|
||||
@~/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-VALIDATION.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-01-SUMMARY.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-02-SUMMARY.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-03-SUMMARY.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-04-SUMMARY.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-05-SUMMARY.md
|
||||
@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-06-SUMMARY.md
|
||||
@.planning/phases/10.1-runtime-admin-extension-point/10.1-SECURITY-REVIEW.md
|
||||
@scripts/check-phase10.1.sh
|
||||
@scripts/check-phase10.2.sh
|
||||
|
||||
<interfaces>
|
||||
- Prior-phase gate shape: scripts/check-phase10.1.sh and check-phase10.2.sh (stage functions, `--self-test` with scratch plants, detector refusing "no tests to run"/SKIP, `KNOWN_APP_FAILURES=""`, fonoteka patterns `./plugins/golem15/fonoteka/... ./plugins/golem15/user/...`), and 10.1-SECURITY-REVIEW.md (threat rows `| T-... |` plus `| RC-NN | T-... |` removal-check rows produced by an anchor-exact mutation harness that restores files byte for byte).
|
||||
- Test harnesses: modules/lagoon/postgres_test.go and modules/conga/postgres_test.go (testcontainers postgres:16-alpine, ICU pl-PL, -short skip, Docker failure fails TestMain); fonoteka plugin_boot_test.go (bootSQL, bootDB, bootConfig); fonoteka parity TestMain.
|
||||
- Plan outputs to test are listed in each plan's "Artifacts this phase produces" section and SUMMARY; read the SUMMARYs for final names and deviations.
|
||||
</interfaces>
|
||||
</context>
|
||||
|
||||
## Artifacts this phase produces
|
||||
|
||||
(This plan's share.)
|
||||
|
||||
- Tests: every file listed in files_modified ending in `_test.go`, including the VALIDATION-named tests.
|
||||
- Gate: `scripts/check-phase11.sh` with `--self-test`, `--hygiene`, `--go`, `--postgres`, `--evidence`, `--all`.
|
||||
- Evidence: `11-SECURITY-REVIEW.md` (threat rows and RC rows), validated `11-VALIDATION.md`, REQUIREMENTS.md traceability rows for the seven requirement IDs.
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="tracer">
|
||||
<name>Task 1: Jobs, scheduler and lagoon seams are covered branch by branch, including the timed LISTEN proof and every job outcome</name>
|
||||
<precondition>Plans 11-01..11-06 are executed: all six SUMMARY files exist in the phase directory and `docker info` exits 0.</precondition>
|
||||
<files>modules/conga/manager_test.go, modules/conga/worker_test.go, modules/conga/commands_test.go, modules/conga/schedule_test.go, modules/conga/listen_test.go, modules/lagoon/ondatabase_test.go, modules/lagoon/transaction_test.go, modules/lagoon/queue_migrations_test.go, modules/bonfire/call_test.go, modules/pact/capabilities_test.go, ../fonoteka.go/plugins/golem15/fonoteka/schedule_test.go</files>
|
||||
<read_first>modules/conga/*.go (all), modules/lagoon/queue_migrations.go, modules/lagoon/ondatabase.go, modules/lagoon/transaction.go, modules/lagoon/connection.go, modules/bonfire/call.go, modules/pact/capabilities.go, modules/surf/serve.go, the 11-01 and 11-02 SUMMARYs (final names, deviations), .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VALIDATION.md (JOBS-01, CLI-04, CLI-06 rows), .planning/phases/11-jobs-realtime-and-search-infrastructure/11-RESEARCH.md (Pitfalls 1-5, 13)</read_first>
|
||||
<action>(1) conga (JOBS-01, CLI-06, D-01..D-05, D-17): `TestOutcomeComplete`, `TestOutcomeFailFinalAttemptOnly` (errors on attempts 1..MaxAttempts-1 keep status 1; final attempt sets 3 with metadata error), `TestOutcomePanicBecomesError`, `TestOutcomeSkipIsCompleteWithMetadata`, `TestCancelQueuedNeverRuns`, `TestCancelRunningCancelsCtx` (row stays 4, not 3), `TestStopJobFromWorker`, `TestManagerPHPSemantics` (StartJob/UpdateJobState/UpdateMetadata/CompleteJob/FailJob/GetMetadata table including the stored `""` metadata, updated_at untouched where PHP leaves it, progress_max default 1 on a missing row, river_job_id set), `TestDispatchActorFromPrincipal` (frontend principal → user_id, is_admin false; backend → is_admin true; none → NULL), `TestDispatchDelayUsesScheduledAt`, `TestRegisterRejectsForeignAndDuplicateJobs` (ErrNotCongaJob, duplicate kind, ErrRegistrationClosed), `TestStartWorkerRejectsNonCongaPluginJob`, `TestQueueWork` (filter works only the named queue; unknown queue is ErrUnknownQueue naming known queues; StartServeWorker returns nil when `queue.work_in_serve` is false), `TestQueueClear` (only available/scheduled/retryable removed across the 10000-row loop; running job untouched; output `Cleared N jobs`), config parsing tests for `queue.job_timeout` int and duration forms. Keep TestListenPickupLatency and TestDispatchTransactional and add a `-count=3` stability run in the gate.
|
||||
|
||||
(2) lagoon: `TestQueueMigrationsUpDown` (fresh DB: migrate creates River v7 tables and summer_jobs with the exact column types and defaults; RollbackLast-style gormigrate rollback of the conga set removes summer_jobs and the River schema; rerun is idempotent), extend `TestOnDatabaseAfterActivate` (errors surface from Publish; per-app isolation of two apps) and `TestTransactionAfterCommit` (nested failure drops only inner callbacks, panic in a callback is recovered, implicit single-statement flush only on success, plain gdb.Transaction fallback runs with the tx handle).
|
||||
|
||||
(3) scheduler (CLI-04, D-18): `TestScheduleNext` table (Daily at, before and after the boundary; DST spring-forward and fall-back days in Europe/Warsaw; Every(1m, 5m, 1h, 24h) boundaries), `TestScheduleValidation` (empty command, zero cadence, non-dividing interval, invalid app.timezone each name the plugin and index), `TestScheduleRunOnce` table (due/not-due for Daily and Every, unknown command warning plus exit 0, first command error returned after all ran), `TestScheduleUniqueByPeriod`, `TestScheduledEntryMismatchSkipped`, `TestScheduleMissingCatalog`, `TestScheduleOrdering` (activation then declaration order, entry ids). bonfire `TestCall` table (args passed, unknown wraps ErrUnknownCommand, empty stdin gives prompt defaults, Catalog.Has). pact `TestCadence` accessors.
|
||||
|
||||
(4) fonoteka.go schedule_test.go `TestFonotekaScheduleSkipsUnregisteredPrune` (Schedule() returns the daily prune entry; running it through the scheduler worker with the real command catalog logs `schedule: command not registered; skipping` with command fonoteka:prune-notifications until Phase 14; user decision 5).</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/conga ./modules/lagoon ./modules/bonfire ./modules/pact -count=1 && go test ./modules/conga -run '^(TestListenPickupLatency|TestDispatchTransactional|TestOutcome.*|TestCancel.*|TestQueueClear|TestQueueWork|TestSchedule.*)$' -count=1 -v && go test ./modules/lagoon -run '^(TestQueueMigrationsUpDown|TestOnDatabaseAfterActivate|TestTransactionAfterCommit)$' -count=1 -v && go test ./modules/bonfire -run '^TestCall$' -count=1 -v && (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^TestFonotekaScheduleSkipsUnregisteredPrune$' -count=1 -v)</automated>
|
||||
<fails_when>Any command exits non-zero; a verbose run lacks "--- PASS" for any test named in its -run pattern, or prints "no tests to run" or "--- SKIP".</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `go test ./modules/conga -cover -count=1` reports coverage of at least 80.0% for modules/conga.
|
||||
- `grep -c 'func TestQueueClear' modules/conga/commands_test.go` and `grep -c 'func TestCancelRunningCancelsCtx' modules/conga/manager_test.go` each print 1.
|
||||
- `grep -c 'Europe/Warsaw' modules/conga/schedule_test.go` prints at least 1.
|
||||
- `grep -c 'func TestQueueMigrationsUpDown' modules/lagoon/queue_migrations_test.go` prints 1.
|
||||
</acceptance_criteria>
|
||||
<done>Every job outcome, cancellation path, queue command, scheduler cadence and lagoon seam behaviour is pinned by a named passing test, and the LISTEN pickup proof is stable across repeated runs.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Realtime, push, search and the tide recorder are covered, and each high threat has a failing-when-broken test</name>
|
||||
<files>modules/lighthouse/postgres_test.go, modules/lighthouse/lighthouse_test.go, modules/lighthouse/channel_test.go, modules/lighthouse/registry_test.go, modules/lighthouse/route_test.go, modules/lighthouse/broadcast_test.go, modules/lighthouse/centrifugo/token_test.go, modules/lighthouse/centrifugo/client_test.go, modules/lighthouse/centrifugo/proxy_test.go, modules/lighthouse/centrifugo/commands_test.go, modules/flare/flare_test.go, modules/flare/encrypt_test.go, modules/flare/commands_test.go, modules/beachcomber/postgres_test.go, modules/beachcomber/sync_test.go, modules/beachcomber/typesense/engine_test.go, modules/tide/centrifugo_test.go, ../fonoteka.go/plugins/golem15/fonoteka/ws_authorizer_test.go, ../fonoteka.go/plugins/golem15/fonoteka/album_realtime_test.go, ../fonoteka.go/plugins/golem15/fonoteka/album_search_test.go</files>
|
||||
<read_first>modules/lighthouse/**/*.go, modules/flare/*.go, modules/beachcomber/**/*.go, modules/tide/centrifugo.go, modules/tide/centrifugo_golden.go, the 11-03..11-06 SUMMARYs, ../fonoteka.go/plugins/golem15/fonoteka/{realtime.go, search.go, classes/ws/*.go, models/album_search.go, realtime_smoke_test.go, search_smoke_test.go}, /media/nvme/dev/golem15/fonoteka/plugins/golem15/websockets/tests/security/*.php (AccessControlTest, AuthenticationTest, DataHandlingTest, InjectionTest — port their cases), .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VALIDATION.md (RT-01..RT-03, SRCH-01 rows), .planning/phases/10.1-runtime-admin-extension-point/10.1-SECURITY-REVIEW.md (removal-check format)</read_first>
|
||||
<action>(1) lighthouse (framework, acme names only; Postgres harness copied from conga): `TestFromSelectsDriver` (null default, memory, log, unknown lists names), `TestParseChannelTable` and `TestChannelIDMatchesPHP` (the php -r table from 11-03), `TestFormatChannels` (lowercase, namespace prefix once), `TestRegistry` (duplicate, empty and colon namespaces rejected, sorted Namespaces, concurrent Get under -race), `TestMountSurfaces` (UserAuth group gets UserAuth then Middleware in order; ServerToServer in a raw group; UserAuth route with empty surface is an error; null driver mounts nothing), `TestBroadcastTx` (commit publishes once through the memory driver after the worker runs; rollback publishes nothing), `TestSuppression` (WithoutBroadcasting[acme.Widget] silences Widget but not acme.Gadget; stale outer ctx not suppressed), `TestBulkEmitsOnce` (N suppressed creates plus one Emit give exactly one publication), `TestBroadcastEdges` (zero-PK batch update skipped, delete snapshot from an id-only model, ShouldBroadcast veto, default alias and event name, default payload keys and ttl 60, savepoint keeps the write alive when enqueue fails, publish failure logged and not retried).
|
||||
|
||||
(2) centrifugo: `TestTokenClaims` (ForUser exact claim set sub/exp/info{name} with a fixed clock, null name, Subscription/Anonymous/ForIdentifier `[]` info/SubscriptionForIdentifier, empty secret ErrNotConfigured), `TestTokenHandler` (401 no principal, 401 unknown user, 503 empty secret after the user check, 200 body shape, headers, no trailing newline, concurrent requests under -race), `TestClientRequests` (exact paths, apikey header, ordered body keys, timestamp +00:00, 2xx success, non-2xx error without the key, empty key sends nothing, presence/unsubscribe/info bodies), `TestProxy` table porting every PHP security test case plus the RT-02 edge truths (secret missing/wrong/empty-configured, user ""/"0"/number/string, presence:presence:, 4 segments, unknown namespace, allow `{"result":{"info":[]}}` bytes, presence allow/override merge, deny body bytes and HTTP 200, 64 KiB cap, log line never containing the secret), `TestHealthCommand`.
|
||||
|
||||
(3) flare: `TestRFC8291AppendixA` kept, `TestEncryptRejects` (bad key lengths, oversized payload), `TestVAPIDHeader` (aud origin, exp bound, subject validation), `TestSendAllowlist` (http scheme, unlisted host, wildcard subdomain match), `TestSendStatuses` (201 ok, 404/410 ErrSubscriptionGone, 500 error, disabled ErrPushDisabled), command tests extended for every branch and the no-secret-output assertion.
|
||||
|
||||
(4) beachcomber: `TestSyncGates` (null engine, empty key, no DB, Gate off: zero requests), `TestSyncAfterCommit` (commit upsert, rollback nothing, implicit tx, plain gdb.Transaction fallback), `TestSyncDeleteAndSoftDelete`, `TestSyncFailuresNonFatal` (500, success:false line, timeout, ToSearchableArray error, panic) and typesense `TestEngineWire` (collection create on 404, import JSONL, header, delete 404 success, flush, SearchIDs parameters and parsing, escaping).
|
||||
|
||||
(5) tide: `TestCentrifugoRecorder` (records publish/broadcast only, authorization flag without storing the header, non-loopback listen refused, body cap) and `TestNormalizePublications` (only timestamps, actor and captured ids masked; DiffPublications reports count/path/body differences).
|
||||
|
||||
(6) fonoteka: `TestWsAuthorizer` (member owner and editor allowed; non-member denied; wishlist id under collection namespace denied; presence-prefixed collection channel denied; wishlist subscriber and household peer allowed; stranger denied; editor removal flips to deny; malformed and unknown user denied), `TestAlbumBroadcastBinding` (channels only for kind=collection, deleted payload without album, created payload with album, actor for frontend/backend/no principal), `TestAlbumSearchable` (document fields and types against a PHP-shaped expectation, artist pivot order, medium map, collection_id 0 refused, schema fields, settings Gate on/off/error).
|
||||
|
||||
(7) Removal checks: for each high mitigated threat (T-11-01 proxy secret compare, T-11-02 parseChannel segment/presence rules and authorizer kind predicate, T-11-05 enqueue inside the write tx and the kind=collection channel rule, T-11-07 positive collection_id refusal, T-11-09 entry-match check, T-11-12 Dispatch on the caller tx, T-11-19 Mount empty-UserAuth refusal, T-11-22 push allowlist, T-11-28 fixture secret references), write an anchor-exact mutation helper (a small Go test helper or script function used by the gate) that removes the protection, runs the named test, requires it to fail, and restores the file byte for byte; record the results for Task 3.</action>
|
||||
<verify>
|
||||
<automated>go vet ./... && go test ./modules/lighthouse/... ./modules/flare ./modules/beachcomber/... ./modules/tide -count=1 -race && go test ./modules/lighthouse/... -run '^(TestBroadcastTx|TestSuppression|TestBulkEmitsOnce|TestToken.*|TestClient.*|TestProxy.*)$' -count=1 -v && go test ./modules/beachcomber/... -run '^TestSync.*$' -count=1 -v && (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^(TestWsAuthorizer|TestAlbumBroadcastBinding|TestAlbumSearchable)$' -count=1 -race -v)</automated>
|
||||
<fails_when>Any command exits non-zero; a verbose run lacks "--- PASS" for any test named in its -run pattern, or prints "no tests to run", "--- SKIP" or "DATA RACE".</fails_when>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `go test ./modules/lighthouse/... ./modules/beachcomber/... ./modules/flare -cover -count=1` reports at least 80.0% coverage for lighthouse, lighthouse/centrifugo, beachcomber, beachcomber/typesense and flare.
|
||||
- `grep -c 'func TestWsAuthorizer' ../fonoteka.go/plugins/golem15/fonoteka/ws_authorizer_test.go` prints 1 and `grep -c 'func TestProxy' modules/lighthouse/centrifugo/proxy_test.go` prints at least 1.
|
||||
- Each of the nine high threats listed in action (7) has a removal check that made its named test fail and a byte-for-byte restore (`cmp` of the mutated file against the copy saved before mutation succeeds).
|
||||
</acceptance_criteria>
|
||||
<done>Realtime, push, search and recorder code is covered, every RT/SRCH validation row has its named passing test, and each high threat's test is proven to fail when its protection is removed.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: One fail-closed phase gate, the security review, the validated test map and requirement traceability</name>
|
||||
<files>scripts/check-phase11.sh, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-SECURITY-REVIEW.md, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VALIDATION.md, .planning/REQUIREMENTS.md</files>
|
||||
<read_first>scripts/check-phase10.1.sh, scripts/check-phase10.2.sh, .planning/phases/10.1-runtime-admin-extension-point/10.1-SECURITY-REVIEW.md, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VALIDATION.md, every 11-0N-PLAN.md threat_model block (threat IDs, severities, dispositions), .planning/REQUIREMENTS.md (Traceability table)</read_first>
|
||||
<action>(1) scripts/check-phase11.sh (mode 100755, bash strict mode), modelled on check-phase10.1.sh: stages `--hygiene` (application-name regex `pl[yý]tarium|fonoteka|albumy|kolekcj|winyl|p[lł]yt[aęy]` over tracked files in modules/conga, modules/lighthouse, modules/flare, modules/beachcomber, modules/tide/centrifugo*.go; `go list -m all` must not contain the Centrifugo Go client, the Typesense Go client, a Web Push library or a direct cron library, and must pin github.com/riverqueue/river at v0.47.0; each new module has README.md and a root README row), `--go` (summercms.go `go vet ./...` and `go test ./...`; `go test -race` on modules/lighthouse/... modules/beachcomber/... modules/flare; `go test ./modules/conga -run '^TestListenPickupLatency$' -count=3`), `--postgres` (fonoteka.go `go vet ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/...` and `go test ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/...` with KNOWN_APP_FAILURES empty), `--named` (runs every VALIDATION-named test with -v and refuses any "--- SKIP", "no tests to run" or missing "--- PASS" line, except TestBroadcastGoldens/created and /updated which must be SKIP with the Phase 12 pending text), `--evidence` (11-SECURITY-REVIEW.md has one `| T-11-NN |` row per threat ID declared in the phase plans plus T-11-SC, and an `| RC-NN | T-11-NN |` row for every high mitigated threat; 11-VALIDATION.md has `nyquist_compliant: true` and no pending rows), `--self-test` (scratch copies with one planted violation per rule, each refused with its own reason, plus a clean look-alike accepted), and `--all` (self-test, hygiene, go, postgres, named, evidence) printing `phase11 all passed`. Each detector returns immediately on its first violation (10.2 fail-open lesson).
|
||||
|
||||
(2) 11-SECURITY-REVIEW.md: frontmatter with reviewer note, one row per threat T-11-01..T-11-30 and T-11-SC copying each plan's severity and disposition verbatim (T-11-08 accepted with its Phase 13 rationale), the mitigating file:function and the named test; RC-01.. rows for the nine high threats with the mutation anchor and the observed failing test.
|
||||
|
||||
(3) 11-VALIDATION.md: fill Task IDs, plans and waves for every row, flip statuses to green with the commands actually run, set `status: validated`, `nyquist_compliant: true`, `wave_0_complete: true`, keep the two manual-only rows as manual.
|
||||
|
||||
(4) REQUIREMENTS.md: mark JOBS-01, CLI-04, CLI-06, RT-01, RT-02, RT-03 and SRCH-01 `[x]` and Complete in the Traceability table (planning-docs commit separate from the gate script commit).</action>
|
||||
<verify>
|
||||
<automated>bash -n scripts/check-phase11.sh && scripts/check-phase11.sh --self-test && scripts/check-phase11.sh --all</automated>
|
||||
<fails_when>Non-zero exit from any of the three commands; `--all` output lacks the line "phase11 all passed" or contains a line starting with "refuse:".</fails_when>
|
||||
<human-check>At /gsd-verify-work, collect the two manual rows of 11-VALIDATION.md: (1) start Centrifugo v6 with the production-shaped config, run `fonoteka serve` with the real secrets, log in through the unchanged Nuxt app, edit an album and confirm the browser receives the event; (2) start `typesense/typesense:26.0`, enable search_use_typesense in the admin settings, save an album and query the golem15_fonoteka_albums collection for it.</human-check>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `test -x scripts/check-phase11.sh` succeeds.
|
||||
- `grep -cE '^\| T-11-(0[1-9]|[12][0-9]|30|SC) \|' .planning/phases/11-jobs-realtime-and-search-infrastructure/11-SECURITY-REVIEW.md` prints 31.
|
||||
- `grep -cE '^\| RC-[0-9]+ \| T-11-' .planning/phases/11-jobs-realtime-and-search-infrastructure/11-SECURITY-REVIEW.md` prints at least 9.
|
||||
- `grep -c 'nyquist_compliant: true' .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VALIDATION.md` prints 1.
|
||||
- `grep -cE '^\| (JOBS-01|CLI-04|CLI-06|RT-01|RT-02|RT-03|SRCH-01) \| Phase 11 \| Complete \|' .planning/REQUIREMENTS.md` prints 7.
|
||||
</acceptance_criteria>
|
||||
<done>One command proves Phase 11 in both repositories and refuses every planted regression; the security review ties each threat to a test and each high threat to a removal check; validation and requirement traceability are complete.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| Gate script → verification verdict | A passing gate is the phase's acceptance signal |
|
||||
| Mutation harness → working tree | Removal checks edit source files temporarily |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-11-21 | Repudiation | phase gate fail-open | medium | mitigate | Each detector returns on its first violation; `--self-test` proves every rule refuses a planted violation for its own reason; skipped or missing named tests fail the gate (Task 3). |
|
||||
| T-11-SC | Tampering | Go module installs | high | mitigate | No module added; the hygiene stage refuses the excluded client libraries and requires River v0.47.0 (Task 3). |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
`scripts/check-phase11.sh --all` prints "phase11 all passed"; `scripts/check-phase10.1.sh --all` still passes (no regression of the prior gate); the two manual rows are collected at /gsd-verify-work.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- Branch-level tests for all Phase 11 code in both repositories, with the VALIDATION-named tests passing and Postgres tests on testcontainers.
|
||||
- Nine high threats proven by removal checks; 11-SECURITY-REVIEW.md complete.
|
||||
- check-phase11.sh fail-closed with a passing self-test; 11-VALIDATION.md validated; seven requirements marked complete.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-07-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -634,24 +634,26 @@ func (d Daily) Next(now time.Time) time.Time {
|
||||
| A6 | `webpush-go` legitimacy/maintenance | Alternatives | Not recommended; irrelevant if hand-rolled |
|
||||
| A7 | Production Centrifugo is v6 (inferred from `.env.example` naming `client.token.hmac_secret_key`, `http_api.key`) | Pattern 8 | Header/proxy format is identical across v4–v6 per docs; low risk |
|
||||
|
||||
## Open Questions
|
||||
## Open Questions (RESOLVED)
|
||||
|
||||
1. **SC-1 wording vs `NewWithPgxListener`.**
|
||||
All seven were settled by the user at the plan-count checkpoint on 2026-09-29.
|
||||
|
||||
1. **SC-1 wording vs `NewWithPgxListener`.** RESOLVED: use `NewWithPgxListener`; ROADMAP SC-1 reworded (a1ed5b3).
|
||||
- What we know: River now ships an official single-driver LISTEN split, and the River GORM docs recommend it.
|
||||
- What's unclear: whether the user accepts it as meeting "a separate `riverpgxv5` client".
|
||||
- Recommendation: use `NewWithPgxListener`, state the mapping in the plan, and keep the timed test identical.
|
||||
2. **Where the River job id lives.**
|
||||
2. **Where the River job id lives.** RESOLVED: internal nullable `river_job_id BIGINT` column on `summer_jobs`.
|
||||
- What we know: D-04 needs `JobCancel(riverID)`. The D-01 column list has no slot for it.
|
||||
- Recommendation: add an internal nullable `river_job_id BIGINT` column. It is additive; cutover rows get NULL. The alternative is `JobList().Metadata('{"summer_job_id":N}')`, which is `[VERIFIED: job_list_params.go:362]` but slower.
|
||||
3. **Web Push scope (D-15).**
|
||||
3. **Web Push scope (D-15).** RESOLVED: port the seams as recommended (Pusher, stdlib VAPID, generate-vapid-keys, health, test-push via SubscriptionSource).
|
||||
- What we know: there is no PHP subscription store or dependency in Płytarium, and the Nuxt side has push seams only.
|
||||
- Recommendation: port the `Pusher` interface, the VAPID driver (RFC 8291/8292, tested with the RFC vector), `websockets:generate-vapid-keys` (P-256 via `crypto/ecdh`, base64url) and `websockets:health`. `websockets:test-push` takes subscriptions through an app-provided `SubscriptionSource` interface and reports "no subscription source" when none is registered. Confirm with the user.
|
||||
4. **tide capture mechanism (D-10 says "subscribe").**
|
||||
4. **tide capture mechanism (D-10 says "subscribe").** RESOLVED: fake Centrifugo HTTP recorder.
|
||||
- Recommendation: point PHP `CENTRIFUGO_API_URL` at a tide-owned fake Centrifugo HTTP recorder on loopback. The parity README already runs PHP with `QUEUE_CONNECTION=sync`, so `BroadcastEventJob` runs inline. Record `{path, Authorization-present, body}` as goldens with `timestamp`/`actor` normalised and the apikey redacted. No WebSocket dependency, and it captures channels. The Go side diffs its Centrifugo driver requests against a `httptest` fake the same way.
|
||||
5. **Schedule entry for a command that ships in Phase 14.**
|
||||
5. **Schedule entry for a command that ships in Phase 14.** RESOLVED: warn and skip unknown commands, with a test asserting the warning.
|
||||
- Recommendation: the scheduler logs a warning and skips unknown commands (no boot failure), and a test asserts the warning. Alternatively, land the fonoteka schedule entry in Phase 14.
|
||||
6. **ws-api ordering (Pitfall 11).** Confirm `jwt.auth` before `throttle:ws-api` on the token route. It only changes throttling of invalid-token spam.
|
||||
7. **Payload parity for created/updated** (Pitfall 15). Record the goldens now and assert the `album` subtree in Phase 12.
|
||||
6. **ws-api ordering (Pitfall 11).** RESOLVED: `jwt.auth` before `throttle:ws-api`. Confirm `jwt.auth` before `throttle:ws-api` on the token route. It only changes throttling of invalid-token spam.
|
||||
7. **Payload parity for created/updated** (Pitfall 15). RESOLVED: record goldens now; assert the `album` subtree in Phase 12. Record the goldens now and assert the `album` subtree in Phase 12.
|
||||
|
||||
## Environment Availability
|
||||
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# API Coverage — Phase 11 (Centrifugo, Typesense, Web Push services)
|
||||
|
||||
> Full coverage by default. Opt-outs are explicit, reasoned decisions. The PHP clients being ported (websockets `CentrifugoClient`, Scout `TypesenseEngine`, the websockets push block) define the baseline; D-12 and D-19 make both clients hand-rolled on net/http.
|
||||
|
||||
## Centrifugo server HTTP API (outbound, plan 11-03 and 11-04)
|
||||
|
||||
| capability | decision | reason |
|
||||
|---|---|---|
|
||||
| publish | INTEGRATE | |
|
||||
| broadcast | INTEGRATE | |
|
||||
| presence | INTEGRATE | |
|
||||
| unsubscribe | INTEGRATE | |
|
||||
| info | INTEGRATE | |
|
||||
| presence_stats | OPT-OUT | not in the PHP CentrifugoClient contract and no Phase 11-15 consumer |
|
||||
| history | OPT-OUT | not in the PHP CentrifugoClient contract; the Nuxt client does not use history recovery |
|
||||
| history_remove | OPT-OUT | not in the PHP CentrifugoClient contract and no consumer |
|
||||
| channels | OPT-OUT | not in the PHP CentrifugoClient contract and no consumer |
|
||||
| disconnect | OPT-OUT | not in the PHP CentrifugoClient contract; token revocation is not part of the port |
|
||||
| refresh | OPT-OUT | not in the PHP CentrifugoClient contract; connection tokens expire by exp as in PHP |
|
||||
| subscribe (server-side) | OPT-OUT | not in the PHP CentrifugoClient contract; clients subscribe through the subscribe proxy |
|
||||
| batch | OPT-OUT | not needed; PHP sends one request per publish or broadcast |
|
||||
|
||||
## Centrifugo proxy and token contract (inbound and signed, plan 11-03)
|
||||
|
||||
| capability | decision | reason |
|
||||
|---|---|---|
|
||||
| subscribe proxy | INTEGRATE | |
|
||||
| connection token (HS256) | INTEGRATE | |
|
||||
| subscription token (HS256) | INTEGRATE | |
|
||||
| anonymous and identifier tokens | INTEGRATE | |
|
||||
| connect proxy | OPT-OUT | explicitly out of scope — PHP authenticates connections with JWTs, not a connect proxy |
|
||||
| refresh proxy | OPT-OUT | explicitly out of scope — not configured on the existing Centrifugo server |
|
||||
| publish proxy | OPT-OUT | explicitly out of scope — clients never publish; the server publishes through the HTTP API |
|
||||
| sub_refresh proxy | OPT-OUT | explicitly out of scope — not configured on the existing Centrifugo server |
|
||||
| rpc proxy | OPT-OUT | explicitly out of scope — the Nuxt client makes no Centrifugo RPC calls |
|
||||
|
||||
## Typesense API (outbound, plan 11-05)
|
||||
|
||||
| capability | decision | reason |
|
||||
|---|---|---|
|
||||
| collection retrieve | INTEGRATE | |
|
||||
| collection create | INTEGRATE | |
|
||||
| collection delete (flush) | INTEGRATE | |
|
||||
| documents import upsert | INTEGRATE | |
|
||||
| document delete | INTEGRATE | |
|
||||
| documents search | INTEGRATE | |
|
||||
| multi_search | OPT-OUT | not needed — Scout issues single-collection searches |
|
||||
| delete documents by filter | OPT-OUT | not needed yet — Phase 14 reindex (SRCH-02) decides its own flush strategy |
|
||||
| collection update (schema alter) | OPT-OUT | not needed — schema changes go through reindex, as in PHP |
|
||||
| aliases | OPT-OUT | not needed yet — the legacy-index drop belongs to Phase 14 SRCH-02 |
|
||||
| synonyms and curation overrides | OPT-OUT | not used by the PHP app |
|
||||
| API keys management | OPT-OUT | explicitly out of scope — keys are operator-provisioned |
|
||||
| health and metrics | OPT-OUT | not needed — sync failures are logged per request and never block writes |
|
||||
|
||||
## Web Push service protocol (outbound, plan 11-04)
|
||||
|
||||
| capability | decision | reason |
|
||||
|---|---|---|
|
||||
| send notification (RFC 8030 push with RFC 8291 payload and RFC 8292 VAPID) | INTEGRATE | |
|
||||
| message urgency and topic headers | INTEGRATE | |
|
||||
| receipts (push message receipts) | OPT-OUT | not needed — PHP never requested receipts and push services rarely support them |
|
||||
| subscription management (store and prune) | OPT-OUT | explicitly out of scope — subscriptions come from an app-provided SubscriptionSource (user decision 3); Płytarium has no store |
|
||||
Reference in New Issue
Block a user