docs(11-01): complete conga job framework plan

.planning/phases/11-jobs-realtime-and-search-infrastructure/11-01-SUMMARY.md

@@ -0,0 +1,306 @@
+---
+phase: 11-jobs-realtime-and-search-infrastructure
+plan: 01
+subsystem: infra
+tags: [river, postgres, jobs, gorm, listen-notify, cli, gormigrate]
+
+requires:
+  - phase: 03-first-vertical-slice
+    provides: lagoon shared *sql.DB pool and the reserved River listener seam
+  - phase: 08-oauth2-1-authorization-server
+    provides: bonfire repeatable flags (Input.Flags) used by queue:work --queue
+provides:
+  - River v0.47.0 on the shared *sql.DB with riverdatabasesql.NewWithPgxListener (one client type, 1-connection LISTEN pool)
+  - lagoon.QueueMigrations under summercms.conga (River schema v7 + summer_jobs with river_job_id)
+  - conga package: Manager (Dispatch, Enqueue, full PHP JobManager surface), conga.Job typed adapter, StartWorker/StartServeWorker, queue:work, queue:clear
+  - serve runs the job worker in-process unless queue.work_in_serve is false
+  - lagoon.OnDatabase (Boot-order seam) and lagoon.Transaction / AfterCommit / lagoon:after_commit GORM callback
+  - fonoteka GORM hooks now register under serve (Boot gap closed)
+affects: [11-02 scheduler, 11-03 broadcasts, 11-05 search sync, 11-07 unit tests, 13 CSV import, 14 domain jobs, 15 cutover]
+
+actuals:
+  tokens: 30550
+  tasks: 3
+  commits: 4
+plan_head_before: 718a35cabaf5bf61f2a497a2a011b3ab9b6435fc
+plan_head_after: 6c1f94e
+# fonoteka.go (separate repository) received 4 more commits: 3d4895d, 0b9a2a5, e0b1b0d, 36fc473
+
+tech-stack:
+  added: [github.com/riverqueue/river v0.47.0, riverdriver/riverdatabasesql v0.47.0, rivertype v0.47.0 (transitive riverdriver, riverpgxv5, rivershared, lib/pq, tidwall gjson/sjson; testify bumped to v1.12.1)]
+  patterns:
+    - "Jobs are declared River-free with conga.Job[T](fn, opts...) and registered by the worker from pact.HasJobs"
+    - "Transactional dispatch: summer_jobs row + River InsertTx on the caller's *sql.Tx (tx.Statement.ConnPool)"
+    - "Row writes are raw UpdateColumns on Table(lagoon.JobsTable) so updated_at only moves on dispatch and StartJob"
+    - "Anything that needs *gorm.DB at Boot goes through lagoon.OnDatabase"
+    - "Side effects after a write use lagoon.Transaction + lagoon.AfterCommit"
+
+key-files:
+  created:
+    - modules/lagoon/queue_migrations.go
+    - modules/lagoon/ondatabase.go
+    - modules/lagoon/transaction.go
+    - modules/conga/conga.go
+    - modules/conga/record.go
+    - modules/conga/job.go
+    - modules/conga/client.go
+    - modules/conga/worker.go
+    - modules/conga/commands.go
+    - modules/conga/README.md
+    - ../fonoteka.go/config/queue.yaml
+  modified:
+    - modules/lagoon/migrations.go
+    - modules/lagoon/connection.go
+    - modules/surf/serve.go
+    - internal/build/build.go
+    - internal/build/stubs/artifacts.tmpl
+    - cmd/summer/main.go
+    - cmd/summer/runtime.go
+    - ../fonoteka.go/plugins/golem15/fonoteka/plugin.go
+    - ../fonoteka.go/parity/schema_diff_test.go
+    - ../fonoteka.go/parity/migrate_test.go
+
+key-decisions:
+  - "Job registration closes only while a worker client runs; the insert-only client has no Workers bundle, so inserts do not validate kinds"
+  - "A worker's lifetime is controlled by Worker.Stop, not the ctx passed to StartWorker (Start runs on context.WithoutCancel), so SIGTERM stops serve and queue:work gracefully instead of hard-cancelling jobs"
+  - "An app with no registered jobs gets an internal idle worker (kind summercms_conga_idle) because River refuses to start a client without workers"
+  - "lagoon.OnDatabase treats a published *gorm.DB as ready and derives the pool from it; existing harnesses publish only *gorm.DB"
+  - "River v7 leaves exactly river_job, river_leader, river_migration, river_notification and river_queue (read from pg_tables after migrating)"
+
+patterns-established:
+  - "conga.Job: typed job functions wrapped for River without importing River in plugins"
+  - "lagoon.OnDatabase for Boot-time GORM callback registration"
+  - "lagoon.Transaction/AfterCommit for commit-safe side effects"
+
+requirements-completed: [JOBS-01, CLI-06]
+
+coverage:
+  - id: D1
+    description: "River LISTEN pickup under 1s with a 30s poll interval from a separate insert-only client; poll-only control misses in 2s"
+    requirement: JOBS-01
+    verification:
+      - kind: integration
+        ref: "modules/conga/listen_test.go#TestListenPickupLatency"
+        status: pass
+    human_judgment: false
+  - id: D2
+    description: "Dispatch writes summer_jobs (status 1, principal, metadata \"\") and the River job in one transaction; rollback leaves neither"
+    requirement: JOBS-01
+    verification:
+      - kind: integration
+        ref: "modules/conga/listen_test.go#TestDispatchTransactional"
+        status: pass
+    human_judgment: false
+  - id: D3
+    description: "PHP JobManager operations (StartJob, UpdateJobState, UpdateMetadata, FailJob, StopJob, CancelJob, CheckIfCanceled, GetMetadata, skip via CompleteJob) with PHP semantics"
+    requirement: JOBS-01
+    verification:
+      - kind: integration
+        ref: "modules/conga/listen_test.go#TestJobManagerOperations"
+        status: pass
+    human_judgment: false
+  - id: D4
+    description: "Retries keep the row IN_PROGRESS; the final failed attempt or a panic records ERROR with metadata error; cancel stops queued and running jobs with the row at STOPPED"
+    requirement: JOBS-01
+    verification:
+      - kind: integration
+        ref: "modules/conga/listen_test.go#TestAttemptOutcomes"
+        status: pass
+      - kind: integration
+        ref: "modules/conga/listen_test.go#TestCancelJob"
+        status: pass
+    human_judgment: false
+  - id: D5
+    description: "queue:work with --queue filters and unknown-queue error, queue:clear removes only pending jobs, serve worker honours queue.work_in_serve"
+    requirement: CLI-06
+    verification:
+      - kind: integration
+        ref: "modules/conga/listen_test.go#TestQueueWorkCommand"
+        status: pass
+      - kind: integration
+        ref: "modules/conga/listen_test.go#TestQueueClear"
+        status: pass
+      - kind: integration
+        ref: "modules/conga/listen_test.go#TestStartServeWorker"
+        status: pass
+      - kind: unit
+        ref: "internal/build/build_test.go#TestGenerateMainRegistersCongaRuntimeCommands"
+        status: pass
+    human_judgment: false
+  - id: D6
+    description: "lagoon.OnDatabase and lagoon.Transaction/AfterCommit seams; fonoteka hooks register when the database is published after Boot"
+    verification:
+      - kind: integration
+        ref: "modules/lagoon/ondatabase_test.go#TestOnDatabaseAfterActivate"
+        status: pass
+      - kind: integration
+        ref: "modules/lagoon/transaction_test.go#TestTransactionAfterCommit"
+        status: pass
+      - kind: integration
+        ref: "../fonoteka.go/plugins/golem15/fonoteka/plugin_boot_test.go#TestHooksRegisterWhenDatabasePublishedAfterBoot"
+        status: pass
+    human_judgment: false
+  - id: D7
+    description: "fonoteka.go schema diff and migrate tests accept the framework queue tables"
+    verification:
+      - kind: integration
+        ref: "../fonoteka.go/parity/schema_diff_test.go#TestSchemaMatchesPHPSnapshot"
+        status: pass
+      - kind: integration
+        ref: "../fonoteka.go/parity/migrate_test.go#TestMigrateSeedsCanonicalGenres"
+        status: pass
+    human_judgment: false
+
+duration: 37min
+completed: 2026-09-29
+status: complete
+---
+
+# Phase 11 Plan 01: conga job framework on River Summary
+
+**River v0.47.0 on the shared pool via `riverdatabasesql.NewWithPgxListener` (3-5 ms LISTEN pickup against a 30 s poll), transactional `conga.Manager.Dispatch` into a PHP-shaped `summer_jobs` record, workers in `serve` and `queue:work`, `queue:clear`, plus the `lagoon.OnDatabase` and after-commit seams.**
+
+## Performance
+
+- **Duration:** 37 min
+- **Started:** 2026-09-29T12:48:56Z
+- **Completed:** 2026-09-29T13:26:12Z
+- **Tasks:** 3
+- **Files modified:** 50 (38 in summercms.go, 12 in fonoteka.go)
+
+## Accomplishments
+
+- `lagoon.Migrate` now runs the `summercms.conga` set: River's schema pinned at version 7 and `summer_jobs` with the exact apparatus columns plus the internal `river_job_id BIGINT`.
+- `conga` package: `Dispatch` writes the row with status 1 (IN_PROGRESS, as PHP does) and enqueues on the same `*sql.Tx`. The full JobManager surface keeps PHP semantics: raw column updates, CancelJob vs StopJob split, and the final-attempt ERROR rule with panic recovery.
+- One River client type. Workers use `NewWithPgxListener` with a `MaxConns 1 / MinConns 0` listener pool; the timed test proves LISTEN pickup (about 5 ms) and a poll-only control proves the test is not measuring poll latency.
+- `serve` starts the worker unless `queue.work_in_serve: false`. `queue:work --queue ...` and `queue:clear [queue]` exist in the app binary and as `summer` delegates. `make:job` scaffolds a `conga.Job`.
+- `lagoon.OnDatabase` closes the Boot-order gap: fonoteka's slug, artist-resolver and credential-encryption callbacks now register under `serve`. `lagoon.Transaction` / `lagoon.AfterCommit` and the `lagoon:after_commit` callback are ready for plan 11-05.
+
+## Task Commits
+
+summercms.go:
+1. **Task 1: transactional dispatch picked up through LISTEN** - `0bc5c77` (feat); toolchain-line restore `05ba88a` (fix)
+2. **Task 2: outcomes, cancellation, serve/queue:work/queue:clear** - `b319e7c` (feat)
+3. **Task 3: OnDatabase and after-commit transactions** - `6c1f94e` (feat)
+
+fonoteka.go:
+1. **Task 1** - `3d4895d` (chore: tidy + allow-lists); toolchain-line restore `0b9a2a5` (fix)
+2. **Task 2** - `e0b1b0d` (feat: regenerated main.go, config/queue.yaml)
+3. **Task 3** - `36fc473` (fix: Boot registers hooks through lagoon.OnDatabase)
+
+## Files Created/Modified
+
+- `modules/lagoon/queue_migrations.go`: `QueueMigrations`, `QueueHistoryID`, `JobsTable`, `RiverSchemaVersion`
+- `modules/lagoon/ondatabase.go`: `OnDatabase` plus the per-app hook queue drained by `Publish`
+- `modules/lagoon/transaction.go`: `Transaction`, `AfterCommit`, `AfterCommitCallback` (`lagoon:after_commit`)
+- `modules/lagoon/connection.go`: `Publish` drains the hooks; `gormFromSQL` installs the after-commit callback
+- `modules/conga/*.go`: Manager, Record/Status, Job adapter, client config, worker, commands
+- `modules/surf/serve.go`: in-process worker start/stop
+- `internal/build/build.go`, `internal/build/stubs/artifacts.tmpl`, `internal/build/artifact.go`: generated main registers `conga.RuntimeCommands`; the `make:job` stub is a `conga.Job`
+- `cmd/summer/main.go`, `cmd/summer/runtime.go`: `queue:work` (forwards repeatable `--queue`) and `queue:clear` delegates
+- READMEs: new `modules/conga/README.md`, root modules table row, `modules/lagoon/README.md`, `modules/surf/README.md`
+- fonoteka.go: `config/queue.yaml`, regenerated `main.go`, `plugin.go` Boot fix, parity allow-lists, go.mod/go.sum tidy
+
+## Decisions Made
+
+See `key-decisions` in the frontmatter. In short:
+
+- Registration closes only while a worker runs.
+- Worker lifetime is tied to `Stop`, not the start ctx.
+- An app without jobs gets an idle worker.
+- `OnDatabase` treats a published `*gorm.DB` as ready.
+- The River v7 table set was read from `pg_tables`.
+
+## TDD Gate Compliance
+
+Tasks 2 and 3 are `tdd="true"`. For each, the RED run was observed and recorded before the implementation. The tests compiled against stubs that returned "not implemented", so they failed on their assertions. `gsd-tools check tdd-red-evidence` returned `RED_EVIDENCE_OK` for:
+
+- `TestJobManagerOperations`, `TestAttemptOutcomes`, `TestCancelJob`, `TestQueueClear`, `TestQueueWorkCommand` and `TestStartServeWorker` (Task 2)
+- `TestOnDatabaseAfterActivate` and `TestTransactionAfterCommit` (Task 3)
+
+`TestHooksRegisterWhenDatabasePublishedAfterBoot` failed against the HEAD `plugin.go` and passed with the fix.
+
+**Gate violation, flagged:** there are no separate `test(11-01)` RED commits. Tests and implementation landed together in `feat`/`fix` commits because the project CLAUDE.md requires `go vet` and `go test ./...` to be green at every commit. The CLAUDE.md rule was treated as the higher-priority constraint.
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 3 - Blocking] River cannot start a client with no workers**
+- **Found during:** Task 2 (TestStartServeWorker, TestQueueWorkCommand)
+- **Issue:** `river.NewClient` fails with "at least one Worker must be added" for an app without jobs, so `serve` and `queue:work` would fail. The plan's flagged assumption (c) says they must start and idle.
+- **Fix:** When no job is registered, `StartWorker` adds an internal `summercms_conga_idle` worker that nothing inserts.
+- **Files modified:** modules/conga/worker.go
+- **Committed in:** b319e7c
+
+**2. [Rule 1 - Bug] `go mod tidy` dropped the `toolchain go1.27.0` line**
+- **Found during:** Task 2 (reading `ensureToolchain` in internal/build)
+- **Issue:** The scaffolder keeps `toolchain go1.27.0` in every module. The Task 1 tidy removed it from `examples/hello` (app and three plugins) and from `fonoteka.go/go.mod`. The same tidy also picked up pre-existing indirect-requirement drift in the example plugin modules.
+- **Fix:** Restored the toolchain line in all five files and kept the tidy requirement updates.
+- **Files modified:** examples/hello/go.mod, examples/hello/plugins/{base,greeter,optional}/go.mod, ../fonoteka.go/go.mod
+- **Committed in:** 05ba88a (summercms.go), 0b9a2a5 (fonoteka.go)
+
+**3. [Rule 1 - Bug] OnDatabase must accept a published `*gorm.DB` alone**
+- **Found during:** Task 3
+- **Issue:** Existing fonoteka test harnesses publish only `*gorm.DB` before `party.Activate`. With the plan's literal "both handles published" check, `RegisterHooks` would have been queued forever in those tests, and the slug and encryption hooks would have stopped firing.
+- **Fix:** A published `*gorm.DB` counts as ready, and the pool comes from `gdb.DB()` when no `*sql.DB` is published. This is covered by the `gorm_only_published_runs_now` subtest.
+- **Files modified:** modules/lagoon/ondatabase.go
+- **Committed in:** 6c1f94e
+
+**4. [Rule 1 - Bug] SIGTERM would hard-cancel running jobs**
+- **Found during:** Task 1/2 (River `Start` docs: cancelling the start ctx is a hard stop)
+- **Issue:** `serve` and `queue:work` pass a signal ctx, so a SIGTERM would have cancelled every running job's ctx before `Stop` ran.
+- **Fix:** `StartWorker` starts River with `context.WithoutCancel(ctx)`. `Worker.Stop` does the graceful stop and falls back to `StopAndCancel` when its own ctx expires.
+- **Files modified:** modules/conga/worker.go
+- **Committed in:** 0bc5c77
+
+**5. [Plan wording] Registration closes while a worker runs, not "once a client has been built"**
+- The insert-only client is built lazily on the first Dispatch and carries no Workers bundle, so it does not depend on the job registry. Closing registration only while a worker exists lets `StartWorker` register plugin jobs even after an earlier Dispatch, and lets a worker restart after `Stop`.
+
+**6. [Additions] Small extra exported surface**
+- `conga.Worker.Queues()` is used by the `queue:work` and `serve` output.
+- `lagoon.AfterCommitCallback` is the name constant of the `lagoon:after_commit` callback.
+- The unexported `WorkerOptions.retryPolicy` test knob keeps retry tests fast.
+- All of these are documented in the module READMEs.
+
+**7. [Process] Commits on `master`**
+- `gsd-tools git.base-branch --is-protected master` reports true. The project config sets `git.branching_strategy: "none"`, every earlier plan committed directly to master, and the orchestrator ran this as a sequential executor on the main working tree. So the plan was committed on master in both repositories, with no branch created.
+
+---
+
+**Total deviations:** 4 auto-fixed (1 blocking, 3 bugs), plus 3 documented plan-interpretation or process notes.
+**Impact on plan:** All fixes were needed for correct boot, shutdown and green builds. No scope creep.
+
+## Issues Encountered
+
+- The shell aliases `rm` and `cp` to interactive mode, which stalled two background commands. They were rerun with `/bin/rm -f` and `/bin/cp -f`.
+- `GOWORK=off` builds of `examples/hello` fail on the toolchain line. This was already true before the plan. Workspace-mode builds, the project's norm, are green.
+- `fonoteka.go/parity/parity_test.go`, `cmd/summer/main.go` and `cmd/summer/runtime.go` had gofmt drift at HEAD. It is out of scope and was left untouched.
+
+## Flagged assumptions (planner, edge probe)
+
+- (a) Two Dispatch calls get distinct ids (SERIAL), and River row locking keeps one job on one worker at a time. This was not load-tested; plan 11-07 can add a concurrency case.
+- (b) Dispatch is not idempotent. Each call creates a new job, as in PHP. Confirmed by design.
+- (c) Confirmed by test: `queue:work` and `serve` start and idle with no registered jobs, and `queue:clear` on an empty queue prints `Cleared 0 jobs`.
+
+## User Setup Required
+
+None. `config/queue.yaml` notes that a PgBouncer in front of Postgres must use session pooling (or be bypassed) for the worker's LISTEN connection.
+
+## Next Phase Readiness
+
+- Plan 11-02 (scheduler): the worker client is the place to attach `PeriodicJobs`.
+- Plan 11-03 (broadcasts): it can use `conga.Manager.Enqueue` on the write's transaction and `lagoon.OnDatabase` for callbacks.
+- Plan 11-05 (search sync): `lagoon.Transaction`/`AfterCommit` are ready. Existing `gdb.Transaction` call sites (such as `album_write_service.go`) still need migrating in Phase 12.
+- Plan 11-07: it should add branch coverage for `client.go` settings parsing, `selectQueues`, `Worker.Stop`'s hard-stop fallback and the `Enqueue` non-transaction path.
+
+## Self-Check: PASSED
+
+- All 15 key created files exist on disk.
+- summercms.go commits 0bc5c77, 05ba88a, b319e7c and 6c1f94e exist, and so do fonoteka.go commits 3d4895d, 0b9a2a5, e0b1b0d and 36fc473.
+- The RED stub files were removed.
+- `go vet ./... && go test ./...` passed in summercms.go, and the vet and test commands for all three modules passed in fonoteka.go, after the final task.
+
+---
+*Phase: 11-jobs-realtime-and-search-infrastructure*
+*Completed: 2026-09-29*