summercms/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-01-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, actuals, plan_head_before, plan_head_after, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, coverage, duration, completed, status

phase	plan	subsystem	tags	requires	provides	affects	actuals	plan_head_before	plan_head_after	tech-stack	key-files	key-decisions	patterns-established	requirements-completed	coverage	duration	completed	status
11-jobs-realtime-and-search-infrastructure	01	infra	river postgres jobs gorm listen-notify cli gormigrate	phase provides 03-first-vertical-slice lagoon shared *sql.DB pool and the reserved River listener seam phase provides 08-oauth2-1-authorization-server bonfire repeatable flags (Input.Flags) used by queue:work --queue	River v0.47.0 on the shared *sql.DB with riverdatabasesql.NewWithPgxListener (one client type, 1-connection LISTEN pool) lagoon.QueueMigrations under summercms.conga (River schema v7 + summer_jobs with river_job_id) conga package Manager (Dispatch, Enqueue, full PHP JobManager surface), conga.Job typed adapter, StartWorker/StartServeWorker, queue:work, queue:clear serve runs the job worker in-process unless queue.work_in_serve is false lagoon.OnDatabase (Boot-order seam) and lagoon.Transaction / AfterCommit / lagoon:after_commit GORM callback fonoteka GORM hooks now register under serve (Boot gap closed)	11-02 scheduler 11-03 broadcasts 11-05 search sync 11-07 unit tests 13 CSV import 14 domain jobs 15 cutover	tokens tasks commits 30550 3 4	718a35caba	6c1f94e	added patterns github.com/riverqueue/river v0.47.0 riverdriver/riverdatabasesql v0.47.0 rivertype v0.47.0 (transitive riverdriver riverpgxv5 rivershared lib/pq tidwall gjson/sjson; testify bumped to v1.12.1) Jobs are declared River-free with conga.Job[T](fn, opts...) and registered by the worker from pact.HasJobs Transactional dispatch: summer_jobs row + River InsertTx on the caller's *sql.Tx (tx.Statement.ConnPool) Row writes are raw UpdateColumns on Table(lagoon.JobsTable) so updated_at only moves on dispatch and StartJob Anything that needs *gorm.DB at Boot goes through lagoon.OnDatabase Side effects after a write use lagoon.Transaction + lagoon.AfterCommit	created modified modules/lagoon/queue_migrations.go modules/lagoon/ondatabase.go modules/lagoon/transaction.go modules/conga/conga.go modules/conga/record.go modules/conga/job.go modules/conga/client.go modules/conga/worker.go modules/conga/commands.go modules/conga/README.md ../fonoteka.go/config/queue.yaml modules/lagoon/migrations.go modules/lagoon/connection.go modules/surf/serve.go internal/build/build.go internal/build/stubs/artifacts.tmpl cmd/summer/main.go cmd/summer/runtime.go ../fonoteka.go/plugins/golem15/fonoteka/plugin.go ../fonoteka.go/parity/schema_diff_test.go ../fonoteka.go/parity/migrate_test.go	Job registration closes only while a worker client runs; the insert-only client has no Workers bundle, so inserts do not validate kinds A worker's lifetime is controlled by Worker.Stop, not the ctx passed to StartWorker (Start runs on context.WithoutCancel), so SIGTERM stops serve and queue:work gracefully instead of hard-cancelling jobs An app with no registered jobs gets an internal idle worker (kind summercms_conga_idle) because River refuses to start a client without workers lagoon.OnDatabase treats a published *gorm.DB as ready and derives the pool from it; existing harnesses publish only *gorm.DB River v7 leaves exactly river_job, river_leader, river_migration, river_notification and river_queue (read from pg_tables after migrating)	conga.Job: typed job functions wrapped for River without importing River in plugins lagoon.OnDatabase for Boot-time GORM callback registration lagoon.Transaction/AfterCommit for commit-safe side effects	JOBS-01 CLI-06	id description requirement verification human_judgment D1 River LISTEN pickup under 1s with a 30s poll interval from a separate insert-only client; poll-only control misses in 2s JOBS-01 kind ref status integration modules/conga/listen_test.go#TestListenPickupLatency pass false id description requirement verification human_judgment D2 Dispatch writes summer_jobs (status 1, principal, metadata "") and the River job in one transaction; rollback leaves neither JOBS-01 kind ref status integration modules/conga/listen_test.go#TestDispatchTransactional pass false id description requirement verification human_judgment D3 PHP JobManager operations (StartJob, UpdateJobState, UpdateMetadata, FailJob, StopJob, CancelJob, CheckIfCanceled, GetMetadata, skip via CompleteJob) with PHP semantics JOBS-01 kind ref status integration modules/conga/listen_test.go#TestJobManagerOperations pass false id description requirement verification human_judgment D4 Retries keep the row IN_PROGRESS; the final failed attempt or a panic records ERROR with metadata error; cancel stops queued and running jobs with the row at STOPPED JOBS-01 kind ref status integration modules/conga/listen_test.go#TestAttemptOutcomes pass kind ref status integration modules/conga/listen_test.go#TestCancelJob pass false id description requirement verification human_judgment D5 queue:work with --queue filters and unknown-queue error, queue:clear removes only pending jobs, serve worker honours queue.work_in_serve CLI-06 kind ref status integration modules/conga/listen_test.go#TestQueueWorkCommand pass kind ref status integration modules/conga/listen_test.go#TestQueueClear pass kind ref status integration modules/conga/listen_test.go#TestStartServeWorker pass kind ref status unit internal/build/build_test.go#TestGenerateMainRegistersCongaRuntimeCommands pass false id description verification human_judgment D6 lagoon.OnDatabase and lagoon.Transaction/AfterCommit seams; fonoteka hooks register when the database is published after Boot kind ref status integration modules/lagoon/ondatabase_test.go#TestOnDatabaseAfterActivate pass kind ref status integration modules/lagoon/transaction_test.go#TestTransactionAfterCommit pass kind ref status integration ../fonoteka.go/plugins/golem15/fonoteka/plugin_boot_test.go#TestHooksRegisterWhenDatabasePublishedAfterBoot pass false id description verification human_judgment D7 fonoteka.go schema diff and migrate tests accept the framework queue tables kind ref status integration ../fonoteka.go/parity/schema_diff_test.go#TestSchemaMatchesPHPSnapshot pass kind ref status integration ../fonoteka.go/parity/migrate_test.go#TestMigrateSeedsCanonicalGenres pass false	37min	2026-09-29	complete

Phase 11 Plan 01: conga job framework on River Summary

River v0.47.0 on the shared pool via riverdatabasesql.NewWithPgxListener (3-5 ms LISTEN pickup against a 30 s poll), transactional conga.Manager.Dispatch into a PHP-shaped summer_jobs record, workers in serve and queue:work, queue:clear, plus the lagoon.OnDatabase and after-commit seams.

Performance

• Duration: 37 min
• Started: 2026-09-29T12:48:56Z
• Completed: 2026-09-29T13:26:12Z
• Tasks: 3
• Files modified: 50 (38 in summercms.go, 12 in fonoteka.go)

Accomplishments

• lagoon.Migrate now runs the summercms.conga set: River's schema pinned at version 7 and summer_jobs with the exact apparatus columns plus the internal river_job_id BIGINT.
• conga package: Dispatch writes the row with status 1 (IN_PROGRESS, as PHP does) and enqueues on the same *sql.Tx. The full JobManager surface keeps PHP semantics: raw column updates, CancelJob vs StopJob split, and the final-attempt ERROR rule with panic recovery.
• One River client type. Workers use NewWithPgxListener with a MaxConns 1 / MinConns 0 listener pool; the timed test proves LISTEN pickup (about 5 ms) and a poll-only control proves the test is not measuring poll latency.
• serve starts the worker unless queue.work_in_serve: false. queue:work --queue ... and queue:clear [queue] exist in the app binary and as summer delegates. make:job scaffolds a conga.Job.
• lagoon.OnDatabase closes the Boot-order gap: fonoteka's slug, artist-resolver and credential-encryption callbacks now register under serve. lagoon.Transaction / lagoon.AfterCommit and the lagoon:after_commit callback are ready for plan 11-05.

Task Commits

summercms.go:

1. Task 1: transactional dispatch picked up through LISTEN - 0bc5c77 (feat); toolchain-line restore 05ba88a (fix)
2. Task 2: outcomes, cancellation, serve/queue:work/queue:clear - b319e7c (feat)
3. Task 3: OnDatabase and after-commit transactions - 6c1f94e (feat)

fonoteka.go:

1. Task 1 - 3d4895d (chore: tidy + allow-lists); toolchain-line restore 0b9a2a5 (fix)
2. Task 2 - e0b1b0d (feat: regenerated main.go, config/queue.yaml)
3. Task 3 - 36fc473 (fix: Boot registers hooks through lagoon.OnDatabase)

Files Created/Modified

• modules/lagoon/queue_migrations.go: QueueMigrations, QueueHistoryID, JobsTable, RiverSchemaVersion
• modules/lagoon/ondatabase.go: OnDatabase plus the per-app hook queue drained by Publish
• modules/lagoon/transaction.go: Transaction, AfterCommit, AfterCommitCallback (lagoon:after_commit)
• modules/lagoon/connection.go: Publish drains the hooks; gormFromSQL installs the after-commit callback
• modules/conga/*.go: Manager, Record/Status, Job adapter, client config, worker, commands
• modules/surf/serve.go: in-process worker start/stop
• internal/build/build.go, internal/build/stubs/artifacts.tmpl, internal/build/artifact.go: generated main registers conga.RuntimeCommands; the make:job stub is a conga.Job
• cmd/summer/main.go, cmd/summer/runtime.go: queue:work (forwards repeatable --queue) and queue:clear delegates
• READMEs: new modules/conga/README.md, root modules table row, modules/lagoon/README.md, modules/surf/README.md
• fonoteka.go: config/queue.yaml, regenerated main.go, plugin.go Boot fix, parity allow-lists, go.mod/go.sum tidy

Decisions Made

See key-decisions in the frontmatter. In short:

• Registration closes only while a worker runs.
• Worker lifetime is tied to Stop, not the start ctx.
• An app without jobs gets an idle worker.
• OnDatabase treats a published *gorm.DB as ready.
• The River v7 table set was read from pg_tables.

TDD Gate Compliance

Tasks 2 and 3 are tdd="true". For each, the RED run was observed and recorded before the implementation. The tests compiled against stubs that returned "not implemented", so they failed on their assertions. gsd-tools check tdd-red-evidence returned RED_EVIDENCE_OK for:

• TestJobManagerOperations, TestAttemptOutcomes, TestCancelJob, TestQueueClear, TestQueueWorkCommand and TestStartServeWorker (Task 2)
• TestOnDatabaseAfterActivate and TestTransactionAfterCommit (Task 3)

TestHooksRegisterWhenDatabasePublishedAfterBoot failed against the HEAD plugin.go and passed with the fix.

Gate violation, flagged: there are no separate test(11-01) RED commits. Tests and implementation landed together in feat/fix commits because the project CLAUDE.md requires go vet and go test ./... to be green at every commit. The CLAUDE.md rule was treated as the higher-priority constraint.

Deviations from Plan

Auto-fixed Issues

1. [Rule 3 - Blocking] River cannot start a client with no workers

• Found during: Task 2 (TestStartServeWorker, TestQueueWorkCommand)
• Issue: river.NewClient fails with "at least one Worker must be added" for an app without jobs, so serve and queue:work would fail. The plan's flagged assumption (c) says they must start and idle.
• Fix: When no job is registered, StartWorker adds an internal summercms_conga_idle worker that nothing inserts.
• Files modified: modules/conga/worker.go
• Committed in: b319e7c

2. [Rule 1 - Bug] go mod tidy dropped the toolchain go1.27.0 line

• Found during: Task 2 (reading ensureToolchain in internal/build)
• Issue: The scaffolder keeps toolchain go1.27.0 in every module. The Task 1 tidy removed it from examples/hello (app and three plugins) and from fonoteka.go/go.mod. The same tidy also picked up pre-existing indirect-requirement drift in the example plugin modules.
• Fix: Restored the toolchain line in all five files and kept the tidy requirement updates.
• Files modified: examples/hello/go.mod, examples/hello/plugins/{base,greeter,optional}/go.mod, ../fonoteka.go/go.mod
• Committed in: 05ba88a (summercms.go), 0b9a2a5 (fonoteka.go)

3. [Rule 1 - Bug] OnDatabase must accept a published *gorm.DB alone

• Found during: Task 3
• Issue: Existing fonoteka test harnesses publish only *gorm.DB before party.Activate. With the plan's literal "both handles published" check, RegisterHooks would have been queued forever in those tests, and the slug and encryption hooks would have stopped firing.
• Fix: A published *gorm.DB counts as ready, and the pool comes from gdb.DB() when no *sql.DB is published. This is covered by the gorm_only_published_runs_now subtest.
• Files modified: modules/lagoon/ondatabase.go
• Committed in: 6c1f94e

4. [Rule 1 - Bug] SIGTERM would hard-cancel running jobs

• Found during: Task 1/2 (River Start docs: cancelling the start ctx is a hard stop)
• Issue: serve and queue:work pass a signal ctx, so a SIGTERM would have cancelled every running job's ctx before Stop ran.
• Fix: StartWorker starts River with context.WithoutCancel(ctx). Worker.Stop does the graceful stop and falls back to StopAndCancel when its own ctx expires.
• Files modified: modules/conga/worker.go
• Committed in: 0bc5c77

5. [Plan wording] Registration closes while a worker runs, not "once a client has been built"

• The insert-only client is built lazily on the first Dispatch and carries no Workers bundle, so it does not depend on the job registry. Closing registration only while a worker exists lets StartWorker register plugin jobs even after an earlier Dispatch, and lets a worker restart after Stop.

6. [Additions] Small extra exported surface

• conga.Worker.Queues() is used by the queue:work and serve output.
• lagoon.AfterCommitCallback is the name constant of the lagoon:after_commit callback.
• The unexported WorkerOptions.retryPolicy test knob keeps retry tests fast.
• All of these are documented in the module READMEs.

7. [Process] Commits on master

• gsd-tools git.base-branch --is-protected master reports true. The project config sets git.branching_strategy: "none", every earlier plan committed directly to master, and the orchestrator ran this as a sequential executor on the main working tree. So the plan was committed on master in both repositories, with no branch created.

---

Total deviations: 4 auto-fixed (1 blocking, 3 bugs), plus 3 documented plan-interpretation or process notes. Impact on plan: All fixes were needed for correct boot, shutdown and green builds. No scope creep.

Issues Encountered

• The shell aliases rm and cp to interactive mode, which stalled two background commands. They were rerun with /bin/rm -f and /bin/cp -f.
• GOWORK=off builds of examples/hello fail on the toolchain line. This was already true before the plan. Workspace-mode builds, the project's norm, are green.
• fonoteka.go/parity/parity_test.go, cmd/summer/main.go and cmd/summer/runtime.go had gofmt drift at HEAD. It is out of scope and was left untouched.

Flagged assumptions (planner, edge probe)

• (a) Two Dispatch calls get distinct ids (SERIAL), and River row locking keeps one job on one worker at a time. This was not load-tested; plan 11-07 can add a concurrency case.
• (b) Dispatch is not idempotent. Each call creates a new job, as in PHP. Confirmed by design.
• (c) Confirmed by test: queue:work and serve start and idle with no registered jobs, and queue:clear on an empty queue prints Cleared 0 jobs.

User Setup Required

None. config/queue.yaml notes that a PgBouncer in front of Postgres must use session pooling (or be bypassed) for the worker's LISTEN connection.

Next Phase Readiness

• Plan 11-02 (scheduler): the worker client is the place to attach PeriodicJobs.
• Plan 11-03 (broadcasts): it can use conga.Manager.Enqueue on the write's transaction and lagoon.OnDatabase for callbacks.
• Plan 11-05 (search sync): lagoon.Transaction/AfterCommit are ready. Existing gdb.Transaction call sites (such as album_write_service.go) still need migrating in Phase 12.
• Plan 11-07: it should add branch coverage for client.go settings parsing, selectQueues, Worker.Stop's hard-stop fallback and the Enqueue non-transaction path.

Self-Check: PASSED

• All 15 key created files exist on disk.
• summercms.go commits 0bc5c77, 05ba88a, b319e7c and 6c1f94e exist, and so do fonoteka.go commits 3d4895d, 0b9a2a5, e0b1b0d and 36fc473.
• The RED stub files were removed.
• go vet ./... && go test ./... passed in summercms.go, and the vet and test commands for all three modules passed in fonoteka.go, after the final task.

---

Phase: 11-jobs-realtime-and-search-infrastructure Completed: 2026-09-29