feat(11-01): add the conga job framework on River with transactional dispatch

- Pin River v0.47.0 (riverdatabasesql, rivertype) and tidy the example modules
- lagoon.Migrate runs the summercms.conga set: River schema v7 and summer_jobs
  with the apparatus columns plus an internal river_job_id link
- conga.Manager.Dispatch writes the summer_jobs row (status IN_PROGRESS) and the
  River job on the caller's *sql.Tx; a rollback leaves neither
- conga.Job wraps typed job functions so plugins never import River
- conga.StartWorker runs one client on riverdatabasesql.NewWithPgxListener with
  a single-connection LISTEN pool; the final failed attempt sets ERROR
- TestListenPickupLatency: 30s poll, pickup under 1s; poll-only control 2s miss
This commit is contained in:
Jakub Zych
2026-09-29 15:02:04 +02:00
parent 718a35caba
commit 0bc5c77097
22 changed files with 2685 additions and 122 deletions

135
modules/conga/README.md Normal file
View File

@@ -0,0 +1,135 @@
# conga
Background jobs on River over the shared Postgres pool: transactional dispatch, a `summer_jobs` progress record, in-process or dedicated workers, and a wall-clock scheduler.
`import "git.golem15.com/golem15/summercms/modules/conga"`
## Overview
conga is the SummerCMS counterpart of the WinterCMS queue plus the apparatus job manager. Plugins describe background work as typed job functions wrapped by `conga.Job` and return them from `pact.HasJobs`, so plugin code never imports River. A caller dispatches a job with `conga.Manager.Dispatch` inside its own write transaction: the `summer_jobs` record row and the River job are written on the same `*sql.Tx`, so a rollback leaves neither behind. River only executes the work; the record row is what progress, outcome and cancellation reads use.
Every River client runs on the one `*sql.DB` pool that `lagoon` opens. A worker started by `conga.StartWorker` uses `riverdatabasesql.NewWithPgxListener`: all queries go through the shared pool, and only Postgres `LISTEN` goes through a dedicated pgx pool with a single connection, so a job committed by any process wakes the worker immediately instead of waiting for the poll interval.
## Features
- River-free job declarations: `conga.Job` turns `func(ctx context.Context, args T) error` into a `pact.Job`; `conga.OnQueue`, `conga.MaxAttempts` and `conga.Timeout` set per-job defaults. A `pact.Job` not built by `conga.Job` is rejected with `conga.ErrNotCongaJob`.
- Transactional dispatch: `conga.Manager.Dispatch` inserts the `summer_jobs` row with `conga.StatusInProgress`, the principal's user id and admin flag, `progress_max` from `conga.DispatchOpts.Count` and JSON metadata, then enqueues the River job in the same transaction. It opens a transaction itself when the caller has none.
- Plain enqueue: `conga.Manager.Enqueue` inserts a River job without a record row, inside the caller's transaction when there is one.
- The record row: `conga.Record` maps `summer_jobs`; `conga.Status` holds the WinterCMS status values (`conga.StatusInQueue`, `conga.StatusInProgress`, `conga.StatusComplete`, `conga.StatusError`, `conga.StatusStopped`). `conga.Manager.CompleteJob` completes a row, `conga.Manager.Get` reads one, and `conga.JobID` gives a running job its own row id.
- Outcome rules in the worker: an error on an attempt before the last leaves the row in progress so River can retry; the final failed attempt, or a recovered panic on it, sets `conga.StatusError` with the error text under the metadata key `error`. Skipped work is recorded as complete with `{"skipped": true}` metadata.
- Workers: `conga.StartWorker` registers every plugin job and starts one River client; `conga.WorkerOptions.Queues` limits it to some queues, and an unknown queue is `conga.ErrUnknownQueue` listing the known ones. `conga.Worker.Stop` stops it gracefully and cancels running jobs when its context ends.
## Usage
A plugin declares a job:
```go
package blog
import (
"context"
"git.golem15.com/golem15/summercms/modules/conga"
"git.golem15.com/golem15/summercms/modules/pact"
)
type ImportPostsArgs struct {
File string `json:"file"`
}
func (ImportPostsArgs) Kind() string { return "acme_blog_import_posts" }
func ImportPostsJob(m *conga.Manager) pact.Job {
return conga.Job(func(ctx context.Context, args ImportPostsArgs) error {
id, _ := conga.JobID(ctx)
// ... import args.File ...
return m.CompleteJob(ctx, id, nil)
}, conga.OnQueue("imports"))
}
```
and dispatches it inside the write that needs it:
```go
func startImport(ctx context.Context, app *backpack.App, gdb *gorm.DB, file string) (uint, error) {
m, err := conga.From(app)
if err != nil {
return 0, err
}
var id uint
err = gdb.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
// ... write the import record ...
id, err = m.Dispatch(ctx, tx, ImportPostsArgs{File: file}, conga.DispatchOpts{Label: "Import posts", Count: 100})
return err
})
return id, err
}
```
A worker runs in the same process or in a separate one:
```go
w, err := conga.StartWorker(ctx, app, plugins, conga.WorkerOptions{})
if err != nil {
return err
}
defer w.Stop(context.Background())
```
## API reference
| Identifier | Description |
|------------|-------------|
| `conga.From` | Returns the app's `conga.Manager`, publishing one on first use. |
| `conga.Manager` | App-scoped job manager: registration, dispatch and the `summer_jobs` record. |
| `conga.Manager.Register` | Registers jobs built by `conga.Job`; closed while a worker runs (`conga.ErrRegistrationClosed`). |
| `conga.Manager.Dispatch` | Writes the record row and enqueues the River job in one transaction; returns the row id. |
| `conga.Manager.Enqueue` | Enqueues a River job without a record row. |
| `conga.Manager.CompleteJob` | Sets `conga.StatusComplete` and progress to `progress_max`; replaces metadata when given. |
| `conga.Manager.Get` | Reads one `conga.Record`. |
| `conga.DispatchOpts` | Label, count, metadata, queue, delay and attempt limit of a dispatch. |
| `conga.EnqueueOpts` | Queue, delay and attempt limit of an enqueue. |
| `conga.Record` | The `summer_jobs` row model. |
| `conga.Status` | Record status values, matching the WinterCMS job statuses. |
| `conga.Job` | Wraps a typed job function as a `pact.Job` that conga can run on River. |
| `conga.JobOption` | Per-job option: `conga.OnQueue`, `conga.MaxAttempts`, `conga.Timeout`. |
| `conga.JobID` | Returns the record row id of the job running in a context. |
| `conga.StartWorker` | Registers plugin jobs and starts a River worker client. |
| `conga.WorkerOptions` | Selects the queues a worker runs. |
| `conga.Worker` | A running worker; `conga.Worker.Stop` stops it and `conga.Worker.Queues` lists its queues. |
| `conga.ErrNoDatabase` | The app has not published the shared database handles. |
| `conga.ErrNotCongaJob` | A registered `pact.Job` was not built by `conga.Job`. |
| `conga.ErrRegistrationClosed` | Registration was attempted while a worker runs. |
| `conga.ErrUnknownQueue` | A worker was asked for a queue nothing names. |
## Configuration
Keys are read from the compass config (`config/queue.yaml`, or `SUMMER_QUEUE__...` environment variables).
| Key | Default | Controls |
|-----|---------|----------|
| `queue.max_attempts` | `3` | Attempts per job before its record becomes `conga.StatusError`, unless the job or dispatch sets its own. |
| `queue.job_timeout` | `300` | Per-attempt deadline, in seconds or as a duration string such as `5m`, unless the job sets `conga.Timeout`. |
| `queue.queues.<name>` | `default: 4` | Concurrent workers per queue. The worker runs these queues plus every queue a registered job names plus `default`. |
| `database.dsn` | none (required) | Also opens the worker's single-connection `LISTEN` pool. With PgBouncer, that connection must use session pooling or go straight to Postgres; transaction pooling cannot hold a `LISTEN`. |
```yaml
max_attempts: 3
job_timeout: 300
queues:
default: 4
imports: 1
```
## Dependencies
- SummerCMS modules: [backpack](../backpack/README.md), [bouncer](../bouncer/README.md) (the dispatching principal), [lagoon](../lagoon/README.md) (the shared pool, `lagoon.JobsTable` and the migrations that create River's schema and `summer_jobs`), [pact](../pact/README.md), [party](../party/README.md).
- Third-party: `github.com/riverqueue/river` v0.47.0 with its `riverdriver/riverdatabasesql` and `rivertype` modules. River is the Postgres job queue the project stack names: it gives transactional inserts on the shared `*sql.DB`, retries, stuck-job rescue, leader election for periodic work and `LISTEN`/`NOTIFY` wake-ups. `github.com/jackc/pgx/v5/pgxpool` opens the listener pool; `gorm.io/gorm` writes the record rows.
## Testing
```sh
go test ./modules/conga/...
```
The tests start a `postgres:16-alpine` container through testcontainers-go and migrate a fresh database per test with `lagoon.Migrate`, so they need a running Docker daemon. `TestListenPickupLatency` sets a 30-second poll interval and checks that a job committed by a separate client is picked up in under one second, while a poll-only worker does not pick it up within two. `go test -short ./modules/conga/...` skips the database tests.

123
modules/conga/client.go Normal file
View File

@@ -0,0 +1,123 @@
package conga
import (
"database/sql"
"log/slog"
"sort"
"strings"
"time"
"git.golem15.com/golem15/summercms/modules/backpack"
"github.com/riverqueue/river"
"github.com/riverqueue/river/riverdriver/riverdatabasesql"
)
const (
// defaultQueue is River's default queue name.
defaultQueue = river.QueueDefault
// defaultMaxAttempts matches the WinterCMS queue worker's --tries=3.
defaultMaxAttempts = 3
// defaultJobTimeout matches the WinterCMS queue worker's --timeout=300.
defaultJobTimeout = 300 * time.Second
// defaultMaxWorkers is the concurrency of a queue without configuration.
defaultMaxWorkers = 4
)
// settings is the queue.* configuration of one app.
type settings struct {
maxAttempts int
jobTimeout time.Duration
queues map[string]int
workInServe bool
}
func settingsFromApp(app *backpack.App) settings {
s := settings{
maxAttempts: defaultMaxAttempts,
jobTimeout: defaultJobTimeout,
queues: map[string]int{},
workInServe: true,
}
if app == nil || app.Config == nil {
return s
}
c := app.Config
if n := c.Int("queue.max_attempts"); n > 0 {
s.maxAttempts = n
}
if raw := strings.TrimSpace(c.String("queue.job_timeout")); raw != "" {
if d, err := time.ParseDuration(raw); err == nil && d > 0 {
s.jobTimeout = d
} else if n := c.Int("queue.job_timeout"); n > 0 {
s.jobTimeout = time.Duration(n) * time.Second
}
}
if c.Has("queue.work_in_serve") {
s.workInServe = c.Bool("queue.work_in_serve")
}
if raw, ok := c.Lookup("queue.queues"); ok {
if m, ok := raw.(map[string]any); ok {
for name := range m {
name = strings.TrimSpace(name)
if name == "" {
continue
}
n := c.Int("queue.queues." + name)
if n <= 0 {
n = defaultMaxWorkers
}
s.queues[name] = n
}
}
}
return s
}
// knownQueues is the configured queues plus every queue a registered job
// names plus default, each with its MaxWorkers.
func knownQueues(s settings, jobs map[string]congaJob) map[string]int {
out := map[string]int{defaultQueue: defaultMaxWorkers}
for _, j := range jobs {
if q := j.config().queue; q != "" {
out[q] = defaultMaxWorkers
}
}
for name, n := range s.queues {
out[name] = n
}
return out
}
func sortedQueueNames(queues map[string]int) []string {
names := make([]string, 0, len(queues))
for name := range queues {
names = append(names, name)
}
sort.Strings(names)
return names
}
// baseConfig is the River client configuration shared by the insert-only
// and worker clients. Queues and Workers are set by the worker only.
func baseConfig(s settings, log *slog.Logger) *river.Config {
return &river.Config{
MaxAttempts: s.maxAttempts,
JobTimeout: s.jobTimeout,
Logger: log,
}
}
// newInsertClient builds an insert-only client on the shared pool. It is
// never started; it inserts, cancels and deletes jobs.
func newInsertClient(sqlDB *sql.DB, s settings, log *slog.Logger) (*river.Client[*sql.Tx], error) {
return river.NewClient(riverdatabasesql.New(sqlDB), baseConfig(s, log))
}
func loggerFromApp(app *backpack.App) *slog.Logger {
if app != nil {
if log, ok := app.Lookup[*slog.Logger](); ok && log != nil {
return log
}
}
return slog.Default()
}

406
modules/conga/conga.go Normal file
View File

@@ -0,0 +1,406 @@
// Package conga runs background jobs on River over the shared Postgres pool
// and keeps a queryable summer_jobs record of each dispatched job.
package conga
import (
"bytes"
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"time"
"git.golem15.com/golem15/summercms/modules/backpack"
"git.golem15.com/golem15/summercms/modules/bouncer"
"git.golem15.com/golem15/summercms/modules/lagoon"
"git.golem15.com/golem15/summercms/modules/pact"
"github.com/riverqueue/river"
"gorm.io/gorm"
)
var (
// ErrNoDatabase is returned when the app has not published the shared
// *sql.DB and *gorm.DB (see lagoon.Publish).
ErrNoDatabase = errors.New("conga: database is not published")
// ErrRegistrationClosed is returned by Register while a worker client
// runs, because River fixes its worker set when the client is built.
ErrRegistrationClosed = errors.New("conga: job registration is closed while a worker runs")
// ErrNotCongaJob is returned for a pact.Job that was not built by Job.
ErrNotCongaJob = errors.New("conga: job was not built by conga.Job")
// ErrUnknownQueue is returned when a worker is asked for a queue that no
// configuration or registered job names.
ErrUnknownQueue = errors.New("conga: unknown queue")
)
// Manager is the app-scoped job manager: it registers jobs, dispatches them
// with a summer_jobs row and records their outcome. Get one with From.
type Manager struct {
app *backpack.App
mu sync.Mutex
jobs map[string]congaJob
inserter *river.Client[*sql.Tx]
worker *river.Client[*sql.Tx]
}
// From returns the app's Manager, publishing a new one on first use.
func From(app *backpack.App) (*Manager, error) {
if app == nil {
return nil, fmt.Errorf("conga: app is nil")
}
if m, ok := app.Lookup[*Manager](); ok && m != nil {
return m, nil
}
m := &Manager{app: app, jobs: map[string]congaJob{}}
if err := app.Publish(m); err != nil {
if existing, ok := app.Lookup[*Manager](); ok && existing != nil {
return existing, nil
}
return nil, fmt.Errorf("conga: %w", err)
}
return m, nil
}
// Register adds jobs built by Job. A job not built by Job is ErrNotCongaJob,
// a second job with the same kind is an error, and registering while a
// worker runs is ErrRegistrationClosed.
func (m *Manager) Register(jobs ...pact.Job) error {
m.mu.Lock()
defer m.mu.Unlock()
return m.registerLocked(jobs...)
}
func (m *Manager) registerLocked(jobs ...pact.Job) error {
for _, j := range jobs {
cj, ok := j.(congaJob)
if !ok || cj == nil {
return fmt.Errorf("%w (got %T)", ErrNotCongaJob, j)
}
kind := cj.kind()
if strings.TrimSpace(kind) == "" {
return fmt.Errorf("conga: job args %T have an empty Kind()", j)
}
if existing, dup := m.jobs[kind]; dup {
if existing == cj {
continue
}
return fmt.Errorf("conga: job kind %q is already registered", kind)
}
if m.worker != nil {
return ErrRegistrationClosed
}
m.jobs[kind] = cj
}
return nil
}
// DispatchOpts are the parameters of Dispatch. Label is required.
type DispatchOpts struct {
// Label is stored in summer_jobs.label.
Label string
// Count is the initial progress_max.
Count int
// Metadata is stored as JSON in summer_jobs.metadata; nil stores the
// JSON string "" as the WinterCMS job manager does without metadata.
Metadata map[string]any
// Queue overrides the job's queue.
Queue string
// Delay schedules the first attempt after this duration.
Delay time.Duration
// MaxAttempts overrides the job's attempt limit.
MaxAttempts int
}
// EnqueueOpts are the parameters of Enqueue.
type EnqueueOpts struct {
Queue string
Delay time.Duration
MaxAttempts int
}
// Dispatch inserts a summer_jobs row with StatusInProgress and enqueues the
// River job in the same transaction, then returns the row id. When db is
// already inside a transaction both writes join it, so a rollback leaves
// neither behind; otherwise Dispatch opens its own. The row's user_id and
// is_admin come from the bouncer principal in ctx.
func (m *Manager) Dispatch(ctx context.Context, db *gorm.DB, args pact.JobArgs, o DispatchOpts) (uint, error) {
if args == nil {
return 0, fmt.Errorf("conga: dispatch args are nil")
}
label := strings.TrimSpace(o.Label)
if label == "" {
return 0, fmt.Errorf("conga: dispatch label is empty")
}
db, err := m.dbOr(db)
if err != nil {
return 0, err
}
client, err := m.insertClient()
if err != nil {
return 0, err
}
meta := `""`
if o.Metadata != nil {
meta, err = encodeMetadata(o.Metadata)
if err != nil {
return 0, err
}
}
var id uint
run := func(tx *gorm.DB) error {
sqlTx, ok := tx.Statement.ConnPool.(*sql.Tx)
if !ok {
return fmt.Errorf("conga: dispatch needs a *sql.Tx connection, got %T", tx.Statement.ConnPool)
}
now := time.Now()
rec := Record{
Label: label,
Status: StatusInProgress,
ProgressMax: o.Count,
Metadata: meta,
CreatedAt: &now,
UpdatedAt: &now,
}
if p, ok := bouncer.User(ctx); ok {
uid := p.ID
rec.UserID = &uid
rec.IsAdmin = p.Backend
}
if err := tx.Create(&rec).Error; err != nil {
return fmt.Errorf("conga: insert job record: %w", err)
}
opts, err := m.insertOpts(args.Kind(), o.Queue, o.MaxAttempts, o.Delay)
if err != nil {
return err
}
opts.Metadata, err = json.Marshal(map[string]any{"summer_job_id": rec.ID})
if err != nil {
return fmt.Errorf("conga: job metadata: %w", err)
}
res, err := client.InsertTx(ctx, sqlTx, args, opts)
if err != nil {
return fmt.Errorf("conga: enqueue %s: %w", args.Kind(), err)
}
if err := tx.Table(lagoon.JobsTable).Where("id = ?", rec.ID).UpdateColumn("river_job_id", res.Job.ID).Error; err != nil {
return fmt.Errorf("conga: link river job: %w", err)
}
id = rec.ID
return nil
}
if inTx(db) {
err = run(db.WithContext(ctx))
} else {
err = db.WithContext(ctx).Transaction(run)
}
if err != nil {
return 0, err
}
return id, nil
}
// Enqueue inserts a River job without a summer_jobs row. When db is inside a
// transaction the job joins it; otherwise it is inserted on its own. db may
// be nil to use the published *gorm.DB.
func (m *Manager) Enqueue(ctx context.Context, db *gorm.DB, args pact.JobArgs, o EnqueueOpts) error {
if args == nil {
return fmt.Errorf("conga: enqueue args are nil")
}
client, err := m.insertClient()
if err != nil {
return err
}
opts, err := m.insertOpts(args.Kind(), o.Queue, o.MaxAttempts, o.Delay)
if err != nil {
return err
}
if db != nil && inTx(db) {
sqlTx := db.Statement.ConnPool.(*sql.Tx)
_, err = client.InsertTx(ctx, sqlTx, args, opts)
} else {
_, err = client.Insert(ctx, args, opts)
}
if err != nil {
return fmt.Errorf("conga: enqueue %s: %w", args.Kind(), err)
}
return nil
}
// CompleteJob sets StatusComplete and progress to progress_max (1 when the
// row is missing), replacing metadata only when it is non-empty. Skipped work
// is recorded as CompleteJob(ctx, id, map[string]any{"skipped": true}); there
// is no separate skipped status. updated_at is not touched.
func (m *Manager) CompleteJob(ctx context.Context, id uint, metadata map[string]any) error {
gdb, err := m.gdb()
if err != nil {
return err
}
maxProgress := 1
var rows []struct{ ProgressMax int }
if err := gdb.WithContext(ctx).Table(lagoon.JobsTable).Select("progress_max").Where("id = ?", id).Limit(1).Scan(&rows).Error; err != nil {
return fmt.Errorf("conga: read job %d: %w", id, err)
}
if len(rows) == 1 {
maxProgress = rows[0].ProgressMax
}
cols := map[string]any{"status": int(StatusComplete), "progress": maxProgress}
if len(metadata) > 0 {
enc, err := encodeMetadata(metadata)
if err != nil {
return err
}
cols["metadata"] = enc
}
return m.updateColumns(ctx, gdb, id, cols)
}
// Get returns the summer_jobs row with id.
func (m *Manager) Get(ctx context.Context, id uint) (Record, error) {
gdb, err := m.gdb()
if err != nil {
return Record{}, err
}
var rec Record
if err := gdb.WithContext(ctx).Where("id = ?", id).Take(&rec).Error; err != nil {
return Record{}, fmt.Errorf("conga: job %d: %w", id, err)
}
return rec, nil
}
// failWithError is the worker-side final failure: StatusError with the
// current metadata plus the error text under "error".
func (m *Manager) failWithError(ctx context.Context, id uint, jobErr error) error {
gdb, err := m.gdb()
if err != nil {
return err
}
meta, err := m.metadata(ctx, gdb, id)
if err != nil {
return err
}
meta["error"] = jobErr.Error()
enc, err := encodeMetadata(meta)
if err != nil {
return err
}
return m.updateColumns(ctx, gdb, id, map[string]any{"status": int(StatusError), "metadata": enc})
}
func (m *Manager) metadata(ctx context.Context, gdb *gorm.DB, id uint) (map[string]any, error) {
var rows []struct{ Metadata string }
if err := gdb.WithContext(ctx).Table(lagoon.JobsTable).Select("metadata").Where("id = ?", id).Limit(1).Scan(&rows).Error; err != nil {
return nil, fmt.Errorf("conga: read job %d metadata: %w", id, err)
}
if len(rows) == 0 {
return nil, fmt.Errorf("conga: job %d: %w", id, gorm.ErrRecordNotFound)
}
return decodeMetadata(rows[0].Metadata), nil
}
// updateColumns writes cols with a raw column update so GORM never sets
// updated_at on its own, matching the query-builder updates of the
// WinterCMS job manager.
func (m *Manager) updateColumns(ctx context.Context, gdb *gorm.DB, id uint, cols map[string]any) error {
if err := gdb.WithContext(ctx).Table(lagoon.JobsTable).Where("id = ?", id).UpdateColumns(cols).Error; err != nil {
return fmt.Errorf("conga: update job %d: %w", id, err)
}
return nil
}
func (m *Manager) insertOpts(kind, queue string, maxAttempts int, delay time.Duration) (*river.InsertOpts, error) {
m.mu.Lock()
j := m.jobs[kind]
m.mu.Unlock()
opts := &river.InsertOpts{Queue: queue, MaxAttempts: maxAttempts}
if j != nil {
cfg := j.config()
if opts.Queue == "" {
opts.Queue = cfg.queue
}
if opts.MaxAttempts == 0 {
opts.MaxAttempts = cfg.maxAttempts
}
}
if delay > 0 {
opts.ScheduledAt = time.Now().Add(delay)
}
return opts, nil
}
// insertClient returns the running worker client, else the lazily built
// insert-only client on the shared pool.
func (m *Manager) insertClient() (*river.Client[*sql.Tx], error) {
m.mu.Lock()
defer m.mu.Unlock()
if m.worker != nil {
return m.worker, nil
}
if m.inserter != nil {
return m.inserter, nil
}
sqlDB, ok := m.app.Lookup[*sql.DB]()
if !ok || sqlDB == nil {
return nil, ErrNoDatabase
}
c, err := newInsertClient(sqlDB, settingsFromApp(m.app), loggerFromApp(m.app))
if err != nil {
return nil, fmt.Errorf("conga: river client: %w", err)
}
m.inserter = c
return c, nil
}
func (m *Manager) gdb() (*gorm.DB, error) {
gdb, ok := m.app.Lookup[*gorm.DB]()
if !ok || gdb == nil {
return nil, ErrNoDatabase
}
return gdb, nil
}
func (m *Manager) dbOr(db *gorm.DB) (*gorm.DB, error) {
if db != nil {
return db, nil
}
return m.gdb()
}
func inTx(db *gorm.DB) bool {
if db == nil || db.Statement == nil {
return false
}
_, ok := db.Statement.ConnPool.(*sql.Tx)
return ok
}
// encodeMetadata JSON-encodes metadata the way the WinterCMS job manager's
// json_encode does for an array: an empty map is [] and slashes and HTML
// characters are left as is.
func encodeMetadata(metadata map[string]any) (string, error) {
if len(metadata) == 0 {
return "[]", nil
}
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
enc.SetEscapeHTML(false)
if err := enc.Encode(metadata); err != nil {
return "", fmt.Errorf("conga: encode metadata: %w", err)
}
return strings.TrimSuffix(buf.String(), "\n"), nil
}
// decodeMetadata mirrors json_decode(...) ?: []: anything that is not a
// non-empty JSON object decodes to an empty map.
func decodeMetadata(raw string) map[string]any {
out := map[string]any{}
var v any
if err := json.Unmarshal([]byte(raw), &v); err != nil {
return out
}
if obj, ok := v.(map[string]any); ok {
return obj
}
return out
}

124
modules/conga/job.go Normal file
View File

@@ -0,0 +1,124 @@
package conga
import (
"context"
"fmt"
"time"
"git.golem15.com/golem15/summercms/modules/pact"
"github.com/riverqueue/river"
)
type jobConfig struct {
queue string
maxAttempts int
timeout time.Duration
}
// JobOption configures a job built by Job.
type JobOption func(*jobConfig)
// OnQueue sets the queue a job is inserted on when the caller does not name
// one. Empty means the River default queue, "default".
func OnQueue(name string) JobOption {
return func(c *jobConfig) { c.queue = name }
}
// MaxAttempts sets how many times River tries the job before the summer_jobs
// row is marked StatusError. Zero means queue.max_attempts.
func MaxAttempts(n int) JobOption {
return func(c *jobConfig) { c.maxAttempts = n }
}
// Timeout sets the per-attempt deadline of the job. Zero means
// queue.job_timeout.
func Timeout(d time.Duration) JobOption {
return func(c *jobConfig) { c.timeout = d }
}
// congaJob is the unexported side of a job built by Job: the parts the worker
// needs to register it on River without the plugin importing River.
type congaJob interface {
pact.Job
kind() string
config() jobConfig
register(workers *river.Workers, m *Manager) error
}
type typedJob[T pact.JobArgs] struct {
fn func(ctx context.Context, args T) error
cfg jobConfig
}
// Job wraps a typed job function as a pact.Job that conga can register on
// River. Plugins return these values from pact.HasJobs.Jobs and never import
// River. The job kind is T's Kind(); the args are stored as JSON.
func Job[T pact.JobArgs](fn func(ctx context.Context, args T) error, opts ...JobOption) pact.Job {
j := &typedJob[T]{fn: fn}
for _, o := range opts {
if o != nil {
o(&j.cfg)
}
}
return j
}
// Work runs the job function when args has the job's argument type.
func (j *typedJob[T]) Work(ctx context.Context, args pact.JobArgs) error {
if j.fn == nil {
return fmt.Errorf("conga: job %s has no function", j.kind())
}
a, ok := args.(T)
if !ok {
return fmt.Errorf("conga: job %s got args of type %T", j.kind(), args)
}
return j.fn(ctx, a)
}
func (j *typedJob[T]) kind() string {
var zero T
return zero.Kind()
}
func (j *typedJob[T]) config() jobConfig { return j.cfg }
func (j *typedJob[T]) register(workers *river.Workers, m *Manager) error {
if j.fn == nil {
return fmt.Errorf("conga: job %s has no function", j.kind())
}
return river.AddWorkerSafely[T](workers, &riverWorker[T]{job: j, m: m})
}
// riverWorker adapts a typedJob onto River and applies the summer_jobs
// outcome rules around each attempt.
type riverWorker[T pact.JobArgs] struct {
river.WorkerDefaults[T]
job *typedJob[T]
m *Manager
}
func (w *riverWorker[T]) Work(ctx context.Context, job *river.Job[T]) error {
return w.m.runAttempt(ctx, job.JobRow, func(ctx context.Context) error {
return w.job.fn(ctx, job.Args)
})
}
func (w *riverWorker[T]) Timeout(*river.Job[T]) time.Duration {
return w.job.cfg.timeout
}
type jobIDKey struct{}
// JobID returns the summer_jobs id of the dispatched job running in ctx.
// Jobs inserted with Manager.Enqueue have no row and report false.
func JobID(ctx context.Context) (uint, bool) {
if ctx == nil {
return 0, false
}
id, ok := ctx.Value(jobIDKey{}).(uint)
return id, ok && id > 0
}
func withJobID(ctx context.Context, id uint) context.Context {
return context.WithValue(ctx, jobIDKey{}, id)
}

View File

@@ -0,0 +1,269 @@
package conga
import (
"context"
"database/sql"
"errors"
"testing"
"time"
"git.golem15.com/golem15/summercms/modules/backpack"
"git.golem15.com/golem15/summercms/modules/bouncer"
"git.golem15.com/golem15/summercms/modules/pact"
"git.golem15.com/golem15/summercms/modules/party"
"github.com/riverqueue/river"
"github.com/riverqueue/river/riverdriver/riverdatabasesql"
"gorm.io/gorm"
)
type pingArgs struct {
Tag string `json:"tag"`
}
func (pingArgs) Kind() string { return "conga_test_ping" }
// jobsPlugin is a minimal compiled plugin that only declares jobs.
type jobsPlugin struct {
id string
jobs []pact.Job
}
func (p *jobsPlugin) ID() string { return p.id }
func (p *jobsPlugin) Requires() []string { return nil }
func (p *jobsPlugin) Register(app *backpack.App) error { return nil }
func (p *jobsPlugin) Boot(app *backpack.App) error { return nil }
func (p *jobsPlugin) Jobs() []pact.Job { return p.jobs }
func plugins(jobs ...pact.Job) []party.Plugin {
return []party.Plugin{&jobsPlugin{id: "acme.jobs", jobs: jobs}}
}
func startTestWorker(t *testing.T, app *backpack.App, o WorkerOptions, jobs ...pact.Job) *Worker {
t.Helper()
w, err := StartWorker(t.Context(), app, plugins(jobs...), o)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := w.Stop(ctx); err != nil {
t.Errorf("stop worker: %v", err)
}
})
return w
}
// waitForListener blocks until a backend of dbName is in LISTEN.
func waitForListener(t *testing.T, dbName string) {
t.Helper()
deadline := time.Now().Add(15 * time.Second)
for time.Now().Before(deadline) {
var n int
if err := adminDB(t).QueryRowContext(t.Context(),
`SELECT count(*) FROM pg_stat_activity WHERE datname = $1 AND query ILIKE 'LISTEN%'`, dbName).Scan(&n); err != nil {
t.Fatal(err)
}
if n > 0 {
return
}
time.Sleep(50 * time.Millisecond)
}
t.Fatalf("no LISTEN backend on %s within 15s", dbName)
}
func currentDatabase(t *testing.T, db *sql.DB) string {
t.Helper()
var name string
if err := db.QueryRowContext(t.Context(), `SELECT current_database()`).Scan(&name); err != nil {
t.Fatal(err)
}
return name
}
// insertFromOtherClient commits one ping job through a separate insert-only
// client, the way another process would, and returns the commit time.
func insertFromOtherClient(t *testing.T, db *sql.DB) time.Time {
t.Helper()
other, err := river.NewClient(riverdatabasesql.New(db), &river.Config{})
if err != nil {
t.Fatal(err)
}
tx, err := db.BeginTx(t.Context(), nil)
if err != nil {
t.Fatal(err)
}
if _, err := other.InsertTx(t.Context(), tx, pingArgs{Tag: "latency"}, nil); err != nil {
_ = tx.Rollback()
t.Fatal(err)
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
return time.Now()
}
// TestListenPickupLatency proves the worker wakes on LISTEN/NOTIFY: with a
// 30s poll interval a committed job runs within 1s, while the same worker
// without the LISTEN pool does not see it within 2s.
func TestListenPickupLatency(t *testing.T) {
const poll = 30 * time.Second
t.Run("listen", func(t *testing.T) {
db, dsn := migratedDB(t)
app, _ := testApp(t, db, dsn, nil)
ran := make(chan time.Time, 4)
startTestWorker(t, app, WorkerOptions{pollInterval: poll}, Job(func(ctx context.Context, a pingArgs) error {
ran <- time.Now()
return nil
}))
waitForListener(t, currentDatabase(t, db))
time.Sleep(500 * time.Millisecond) // let the start-up fetch pass
committed := insertFromOtherClient(t, db)
select {
case at := <-ran:
latency := at.Sub(committed)
t.Logf("LISTEN pickup latency %s with a %s poll interval", latency, poll)
if latency >= time.Second {
t.Fatalf("pickup took %s, want under 1s", latency)
}
case <-time.After(time.Second):
t.Fatal("job was not picked up within 1s of commit")
}
})
t.Run("poll_only_control", func(t *testing.T) {
db, dsn := migratedDB(t)
app, _ := testApp(t, db, dsn, nil)
ran := make(chan time.Time, 4)
startTestWorker(t, app, WorkerOptions{pollInterval: poll, pollOnly: true}, Job(func(ctx context.Context, a pingArgs) error {
ran <- time.Now()
return nil
}))
time.Sleep(1500 * time.Millisecond) // let the start-up fetch pass
insertFromOtherClient(t, db)
select {
case <-ran:
t.Fatal("poll-only worker picked the job up within 2s; the LISTEN test would not prove LISTEN")
case <-time.After(2 * time.Second):
t.Log("poll-only worker did not pick the job up within 2s")
}
})
}
func waitStatus(t *testing.T, m *Manager, id uint, want Status) Record {
t.Helper()
deadline := time.Now().Add(10 * time.Second)
var rec Record
for time.Now().Before(deadline) {
var err error
rec, err = m.Get(t.Context(), id)
if err != nil {
t.Fatal(err)
}
if rec.Status == want {
return rec
}
time.Sleep(25 * time.Millisecond)
}
t.Fatalf("job %d status = %d, want %d", id, rec.Status, want)
return rec
}
func countRows(t *testing.T, gdb *gorm.DB, query string, args ...any) int64 {
t.Helper()
var n int64
if err := gdb.Raw(query, args...).Scan(&n).Error; err != nil {
t.Fatal(err)
}
return n
}
// TestDispatchTransactional covers D-02: the row and the River job share the
// caller's transaction.
func TestDispatchTransactional(t *testing.T) {
db, dsn := migratedDB(t)
app, gdb := testApp(t, db, dsn, nil)
m, err := From(app)
if err != nil {
t.Fatal(err)
}
complete := Job(func(ctx context.Context, a pingArgs) error {
id, ok := JobID(ctx)
if !ok {
return errors.New("no summer job id in ctx")
}
return m.CompleteJob(ctx, id, nil)
})
startTestWorker(t, app, WorkerOptions{}, complete)
ctx := bouncer.WithUser(t.Context(), &bouncer.Principal{ID: 7})
t.Run("commit", func(t *testing.T) {
var id uint
err := gdb.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
var err error
id, err = m.Dispatch(ctx, tx, pingArgs{Tag: "commit"}, DispatchOpts{Label: "Import albums", Count: 5})
if err != nil {
return err
}
var rec Record
if err := tx.Where("id = ?", id).Take(&rec).Error; err != nil {
return err
}
if rec.Status != StatusInProgress {
t.Errorf("status inside tx = %d, want %d", rec.Status, StatusInProgress)
}
return nil
})
if err != nil {
t.Fatal(err)
}
rec := waitStatus(t, m, id, StatusComplete)
if rec.Progress != 5 || rec.ProgressMax != 5 {
t.Fatalf("progress = %d/%d, want 5/5", rec.Progress, rec.ProgressMax)
}
if rec.Label != "Import albums" || rec.Metadata != `""` {
t.Fatalf("label/metadata = %q/%q", rec.Label, rec.Metadata)
}
if rec.UserID == nil || *rec.UserID != 7 || rec.IsAdmin {
t.Fatalf("user_id/is_admin = %v/%v, want 7/false", rec.UserID, rec.IsAdmin)
}
if rec.RiverJobID == nil || *rec.RiverJobID <= 0 {
t.Fatalf("river_job_id = %v", rec.RiverJobID)
}
if rec.CreatedAt == nil || rec.UpdatedAt == nil {
t.Fatal("timestamps not set")
}
})
t.Run("own_transaction", func(t *testing.T) {
id, err := m.Dispatch(ctx, gdb, pingArgs{Tag: "own"}, DispatchOpts{Label: "Match albums"})
if err != nil {
t.Fatal(err)
}
rec := waitStatus(t, m, id, StatusComplete)
if rec.Progress != 0 {
t.Fatalf("progress = %d, want progress_max 0", rec.Progress)
}
})
t.Run("rollback", func(t *testing.T) {
before := countRows(t, gdb, `SELECT count(*) FROM river_job`)
rollback := errors.New("rollback")
err := gdb.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
if _, err := m.Dispatch(ctx, tx, pingArgs{Tag: "rollback"}, DispatchOpts{Label: "Rolled back"}); err != nil {
return err
}
return rollback
})
if !errors.Is(err, rollback) {
t.Fatalf("transaction err = %v", err)
}
if n := countRows(t, gdb, `SELECT count(*) FROM summer_jobs WHERE label = ?`, "Rolled back"); n != 0 {
t.Fatalf("summer_jobs rows after rollback = %d", n)
}
if after := countRows(t, gdb, `SELECT count(*) FROM river_job`); after != before {
t.Fatalf("river_job rows %d -> %d after rollback", before, after)
}
})
}

View File

@@ -0,0 +1,178 @@
package conga
import (
"context"
"database/sql"
"fmt"
"net/url"
"os"
"strings"
"sync/atomic"
"testing"
"time"
"git.golem15.com/golem15/summercms/modules/backpack"
"git.golem15.com/golem15/summercms/modules/compass"
"git.golem15.com/golem15/summercms/modules/lagoon"
_ "github.com/jackc/pgx/v5/stdlib"
"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/modules/postgres"
"gorm.io/gorm"
)
var (
congaPG *postgres.PostgresContainer
congaSQL *sql.DB
congaDSN string
congaPGErr error
dbSeq atomic.Int64
)
func TestMain(m *testing.M) {
code := 1
if !testShort() {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
congaPGErr = startCongaPostgres(ctx)
cancel()
if congaPGErr != nil {
fmt.Fprintf(os.Stderr, "conga: testcontainers postgres: %v\n", congaPGErr)
stopCongaPostgres()
os.Exit(1)
}
}
code = m.Run()
stopCongaPostgres()
os.Exit(code)
}
func testShort() bool {
for _, a := range os.Args {
if a == "-test.short" {
return true
}
}
return false
}
func startCongaPostgres(ctx context.Context) error {
ctr, err := postgres.Run(ctx,
"postgres:16-alpine",
postgres.WithDatabase("conga"),
postgres.WithUsername("conga"),
postgres.WithPassword("conga"),
postgres.BasicWaitStrategies(),
testcontainers.WithEnv(map[string]string{
"POSTGRES_INITDB_ARGS": "--locale-provider=icu --icu-locale=pl-PL --encoding=UTF8",
}),
)
if err != nil {
return err
}
congaPG = ctr
dsn, err := ctr.ConnectionString(ctx, "sslmode=disable")
if err != nil {
return err
}
db, err := sql.Open("pgx", dsn)
if err != nil {
return err
}
if err := db.PingContext(ctx); err != nil {
_ = db.Close()
return err
}
congaSQL = db
congaDSN = dsn
return nil
}
func stopCongaPostgres() {
if congaSQL != nil {
_ = congaSQL.Close()
}
if congaPG != nil {
_ = testcontainers.TerminateContainer(congaPG)
}
}
func adminDB(t *testing.T) *sql.DB {
t.Helper()
if testing.Short() {
t.Skip("requires testcontainers postgres")
}
if congaPGErr != nil {
t.Fatalf("postgres unavailable: %v", congaPGErr)
}
if congaSQL == nil {
t.Fatal("postgres unavailable: container was not started")
}
return congaSQL
}
// migratedDB returns a dedicated ICU pl-PL database migrated with
// lagoon.Migrate (River v7 and summer_jobs) plus its DSN.
func migratedDB(t *testing.T) (*sql.DB, string) {
t.Helper()
admin := adminDB(t)
ctx := t.Context()
name := fmt.Sprintf("conga_%d", dbSeq.Add(1))
if _, err := admin.ExecContext(ctx, `CREATE DATABASE `+name+` TEMPLATE template0 ENCODING 'UTF8' LOCALE_PROVIDER icu ICU_LOCALE 'pl-PL'`); err != nil && !strings.Contains(err.Error(), "already exists") {
t.Fatalf("create %s: %v", name, err)
}
dsn, err := dsnWithDB(congaDSN, name)
if err != nil {
t.Fatal(err)
}
db, err := sql.Open("pgx", dsn)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_ = db.Close()
_, _ = admin.ExecContext(context.Background(), `DROP DATABASE IF EXISTS `+name+` WITH (FORCE)`)
})
gdb, err := lagoon.Use(ctx, db)
if err != nil {
t.Fatal(err)
}
if err := lagoon.Migrate(gdb, nil); err != nil {
t.Fatal(err)
}
return db, dsn
}
func dsnWithDB(dsn, name string) (string, error) {
u, err := url.Parse(dsn)
if err != nil {
return "", err
}
u.Path = "/" + name
return u.String(), nil
}
// testApp returns an app whose config names dsn and which has published db
// and a GORM handle on it. Extra config keys are set as overrides.
func testApp(t *testing.T, db *sql.DB, dsn string, extra map[string]any) (*backpack.App, *gorm.DB) {
t.Helper()
cfg, err := compass.Open(compass.Options{Dir: t.TempDir(), Env: "testing", Environ: []string{}})
if err != nil {
t.Fatal(err)
}
if err := cfg.Set("database.dsn", dsn); err != nil {
t.Fatal(err)
}
for k, v := range extra {
if err := cfg.Set(k, v); err != nil {
t.Fatal(err)
}
}
app := backpack.New(cfg)
gdb, err := lagoon.Use(t.Context(), db)
if err != nil {
t.Fatal(err)
}
if err := lagoon.Publish(app, db, gdb); err != nil {
t.Fatal(err)
}
return app, gdb
}

45
modules/conga/record.go Normal file
View File

@@ -0,0 +1,45 @@
package conga
import (
"time"
"git.golem15.com/golem15/summercms/modules/lagoon"
)
// Status is the summer_jobs status column. The values match the WinterCMS
// apparatus JobStatus constants.
type Status int
const (
// StatusInQueue is defined for parity; Dispatch never writes it.
StatusInQueue Status = 0
// StatusInProgress is written by Dispatch and kept while River retries.
StatusInProgress Status = 1
// StatusComplete is written by CompleteJob, including skipped work.
StatusComplete Status = 2
// StatusError is written by FailJob and by the final failed attempt.
StatusError Status = 3
// StatusStopped is written by CancelJob and StopJob.
StatusStopped Status = 4
)
// Record is one summer_jobs row: the queryable outcome of a dispatched job.
// River only executes the work; API responses, progress and cancellation
// read this row.
type Record struct {
ID uint `gorm:"column:id;primaryKey"`
Label string `gorm:"column:label"`
Status Status `gorm:"column:status"`
Progress int `gorm:"column:progress"`
ProgressMax int `gorm:"column:progress_max"`
UserID *uint `gorm:"column:user_id"`
IsAdmin bool `gorm:"column:is_admin"`
IsCanceled bool `gorm:"column:is_canceled"`
Metadata string `gorm:"column:metadata"`
RiverJobID *int64 `gorm:"column:river_job_id"`
CreatedAt *time.Time `gorm:"column:created_at"`
UpdatedAt *time.Time `gorm:"column:updated_at"`
}
// TableName returns lagoon.JobsTable.
func (Record) TableName() string { return lagoon.JobsTable }

244
modules/conga/worker.go Normal file
View File

@@ -0,0 +1,244 @@
package conga
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"runtime/debug"
"strings"
"time"
"git.golem15.com/golem15/summercms/modules/backpack"
"git.golem15.com/golem15/summercms/modules/lagoon"
"git.golem15.com/golem15/summercms/modules/pact"
"git.golem15.com/golem15/summercms/modules/party"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/riverqueue/river"
"github.com/riverqueue/river/riverdriver/riverdatabasesql"
"github.com/riverqueue/river/rivertype"
)
// WorkerOptions selects what a worker runs.
type WorkerOptions struct {
// Queues limits the worker to these queues; nil means every known queue.
Queues []string
// pollInterval overrides River's FetchPollInterval (tests only).
pollInterval time.Duration
// pollOnly builds a driver without the LISTEN pool (tests only).
pollOnly bool
}
// Worker is a running River work client. Stop it with Stop.
type Worker struct {
m *Manager
client *river.Client[*sql.Tx]
listener *pgxpool.Pool
queues []string
}
// Queues returns the sorted queue names the worker runs.
func (w *Worker) Queues() []string {
if w == nil {
return nil
}
return append([]string(nil), w.queues...)
}
// StartWorker registers every pact.HasJobs job of plugins and starts one
// River client on the shared *sql.DB. Its driver is
// riverdatabasesql.NewWithPgxListener: every query runs on the shared pool
// and only Postgres LISTEN runs on a dedicated pgx pool (MaxConns 1) opened
// from database.dsn, so jobs committed by any process are picked up without
// waiting for the poll interval. The worker keeps running after ctx is
// cancelled; stop it with Worker.Stop.
func StartWorker(ctx context.Context, app *backpack.App, plugins []party.Plugin, o WorkerOptions) (*Worker, error) {
m, err := From(app)
if err != nil {
return nil, err
}
for _, p := range plugins {
hj, ok := p.(pact.HasJobs)
if !ok {
continue
}
for _, j := range hj.Jobs() {
if err := m.Register(j); err != nil {
return nil, fmt.Errorf("conga: plugin %s: %w", p.ID(), err)
}
}
}
sqlDB, ok := app.Lookup[*sql.DB]()
if !ok || sqlDB == nil {
return nil, ErrNoDatabase
}
s := settingsFromApp(app)
log := loggerFromApp(app)
m.mu.Lock()
defer m.mu.Unlock()
if m.worker != nil {
return nil, fmt.Errorf("conga: a worker is already running")
}
known := knownQueues(s, m.jobs)
queues, err := selectQueues(known, o.Queues)
if err != nil {
return nil, err
}
workers := river.NewWorkers()
for _, j := range m.jobs {
if err := j.register(workers, m); err != nil {
return nil, fmt.Errorf("conga: register %s: %w", j.kind(), err)
}
}
cfg := baseConfig(s, log)
cfg.Workers = workers
cfg.Queues = map[string]river.QueueConfig{}
for name, n := range queues {
cfg.Queues[name] = river.QueueConfig{MaxWorkers: n}
}
if o.pollInterval > 0 {
cfg.FetchPollInterval = o.pollInterval
}
var listener *pgxpool.Pool
var driver *riverdatabasesql.Driver
if o.pollOnly {
driver = riverdatabasesql.New(sqlDB)
} else {
listener, err = listenerPool(ctx, app)
if err != nil {
return nil, err
}
driver = riverdatabasesql.NewWithPgxListener(sqlDB, listener)
}
client, err := river.NewClient(driver, cfg)
if err != nil {
closePool(listener)
return nil, fmt.Errorf("conga: river client: %w", err)
}
if err := client.Start(context.WithoutCancel(ctx)); err != nil {
closePool(listener)
return nil, fmt.Errorf("conga: start worker: %w", err)
}
m.worker = client
return &Worker{m: m, client: client, listener: listener, queues: sortedQueueNames(queues)}, nil
}
// Stop stops fetching jobs and waits for running jobs to finish. When ctx
// ends first, running jobs are cancelled. The listener pool is closed.
func (w *Worker) Stop(ctx context.Context) error {
if w == nil || w.client == nil {
return nil
}
err := w.client.Stop(ctx)
if err != nil && ctx.Err() != nil {
hardCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
err = w.client.StopAndCancel(hardCtx)
cancel()
}
closePool(w.listener)
w.m.mu.Lock()
if w.m.worker == w.client {
w.m.worker = nil
}
w.m.mu.Unlock()
if err != nil {
return fmt.Errorf("conga: stop worker: %w", err)
}
return nil
}
func listenerPool(ctx context.Context, app *backpack.App) (*pgxpool.Pool, error) {
dsn := ""
if app != nil {
dsn = lagoon.DSN(app.Config)
}
if dsn == "" {
return nil, fmt.Errorf("conga: database.dsn is empty; the worker's LISTEN pool needs it")
}
cfg, err := pgxpool.ParseConfig(dsn)
if err != nil {
return nil, fmt.Errorf("conga: listener pool config: %w", err)
}
cfg.MaxConns = 1
cfg.MinConns = 0
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
return nil, fmt.Errorf("conga: listener pool: %w", err)
}
return pool, nil
}
func closePool(p *pgxpool.Pool) {
if p != nil {
p.Close()
}
}
func selectQueues(known map[string]int, want []string) (map[string]int, error) {
if len(want) == 0 {
return known, nil
}
out := map[string]int{}
for _, name := range want {
name = strings.TrimSpace(name)
n, ok := known[name]
if !ok {
names := sortedQueueNames(known)
return nil, fmt.Errorf("%w %q (known queues: %s)", ErrUnknownQueue, name, strings.Join(names, ", "))
}
out[name] = n
}
return out, nil
}
// runAttempt runs one River attempt of a job and applies the summer_jobs
// rules: the row id goes into ctx; a panic becomes an error; an error on a
// STOPPED or canceled row cancels the River job; an error on the final
// attempt sets StatusError with the error text under metadata "error". An
// error on an earlier attempt leaves the row as it is so River can retry.
func (m *Manager) runAttempt(ctx context.Context, row *rivertype.JobRow, fn func(context.Context) error) error {
id, hasRow := summerJobID(row)
if hasRow {
ctx = withJobID(ctx, id)
}
err := m.call(ctx, row, fn)
if err == nil || !hasRow {
return err
}
dbCtx := context.WithoutCancel(ctx)
if rec, gerr := m.Get(dbCtx, id); gerr == nil && (rec.Status == StatusStopped || rec.IsCanceled) {
return river.JobCancel(err)
}
if row.Attempt >= row.MaxAttempts {
if ferr := m.failWithError(dbCtx, id, err); ferr != nil {
loggerFromApp(m.app).Error("conga: record job failure", "job_id", id, "kind", row.Kind, "error", ferr)
}
}
return err
}
func (m *Manager) call(ctx context.Context, row *rivertype.JobRow, fn func(context.Context) error) (err error) {
defer func() {
if r := recover(); r != nil {
loggerFromApp(m.app).Error("conga: job panicked", "kind", row.Kind, "river_job_id", row.ID, "panic", fmt.Sprint(r), "stack", string(debug.Stack()))
err = fmt.Errorf("conga: job %s panicked: %v", row.Kind, r)
}
}()
return fn(ctx)
}
func summerJobID(row *rivertype.JobRow) (uint, bool) {
if row == nil || len(row.Metadata) == 0 {
return 0, false
}
var meta struct {
SummerJobID uint `json:"summer_job_id"`
}
if err := json.Unmarshal(row.Metadata, &meta); err != nil || meta.SummerJobID == 0 {
return 0, false
}
return meta.SummerJobID, true
}

View File

@@ -14,7 +14,7 @@ Postgres data layer: the shared GORM connection, per-plugin migrations, model he
- One shared pool: `lagoon.Open`, `lagoon.Use` and `lagoon.OpenFromApp` return a `*sql.DB` and a `*gorm.DB` built on that same pool; `lagoon.Publish` makes both available on the `backpack.App`.
- Database check at connect time: `lagoon.CheckLocale` refuses a database whose default collation is not the ICU `pl-PL` locale, so ordering matches the database default without per-query `COLLATE`.
- Per-plugin migrations: `lagoon.Migrate` runs the framework's `system_files` set (`attach.Migrations`) and backend admin identity set (`lagoon.BackendAdminMigrations`), then every `pact.HasMigrations` set in plugin activation order, each in its own `summer_migrations_<plugin_id>` history table (`lagoon.HistoryTableName`). `lagoon.RollbackLast` and `lagoon.Status` cover rollback and history.
- Per-plugin migrations: `lagoon.Migrate` runs the framework's `system_files` set (`attach.Migrations`), backend admin identity set (`lagoon.BackendAdminMigrations`) and job-queue set (`lagoon.QueueMigrations`: River's schema pinned at `lagoon.RiverSchemaVersion`, then the `lagoon.JobsTable` record table, under the `lagoon.QueueHistoryID` history), then every `pact.HasMigrations` set in plugin activation order, each in its own `summer_migrations_<plugin_id>` history table (`lagoon.HistoryTableName`). `lagoon.RollbackLast` and `lagoon.Status` cover rollback and history.
- Mass assignment: `lagoon.Fill` copies only allow-listed keys onto a model by GORM column name and silently drops the rest, logging each dropped key once outside production. A `json.Number` (from a decoder using `UseNumber`) fills integer, unsigned and float fields. A value that does not fit its column (a fraction, an exponent or an overflow for an integer field, or a value of the wrong type) is a `lagoon.FillTypeError` naming the key, so a caller can answer it as a validation failure on that field. `lagoon.HasFillable` and `lagoon.HasHidden` are the Go forms of `$fillable` and `$hidden`.
- Validation: `lagoon.Validate` accepts Laravel-style rule strings (`required`, `nullable`, `integer`, `numeric`, `between`, `min`, `max`, `in`, `unique`, `boolean`, `email`, `confirmed`, `different`, `mimes`) and returns a field-to-messages map, translated through phrasebook when a translator is given. Unknown rule tokens are an error.
- Safe ordering: `lagoon.OrderBy` appends an ORDER BY only for an allow-listed column and an `asc` or `desc` direction.
@@ -95,6 +95,10 @@ func (p *Plugin) Migrations() []*gormigrate.Migration {
| `lagoon.Status` | Lists applied migration IDs per plugin as `lagoon.StatusRow` values. |
| `lagoon.HistoryTableName` | Returns the gormigrate history table for a plugin ID. |
| `lagoon.BackendAdminMigrations` | Creates the backend user, role and admin token blacklist tables and seeds the system roles. |
| `lagoon.QueueMigrations` | Migrates River's schema to `lagoon.RiverSchemaVersion` and creates the `summer_jobs` job record table. |
| `lagoon.QueueHistoryID` | History id of the job-queue set, `summercms.conga`. |
| `lagoon.JobsTable` | Name of the job record table, `summer_jobs`. |
| `lagoon.RiverSchemaVersion` | The pinned River schema version, 7. |
| `lagoon.RuntimeCommands` | Returns the migrate, migrate:rollback, migrate:status and key:generate commands. |
| `lagoon.KeyGenerateCommand` | Returns the key:generate command on its own. |
| `lagoon.LoadAppKey` | Decodes `app.key` and `app.previous_keys`. |
@@ -155,7 +159,7 @@ storage:
## Dependencies
- SummerCMS modules: [backpack](../backpack/README.md), [bonfire](../bonfire/README.md), [compass](../compass/README.md), [pact](../pact/README.md), [party](../party/README.md), [phrasebook](../phrasebook/README.md) (validation messages).
- Third-party: `gorm.io/gorm`, `gorm.io/driver/postgres`, `github.com/jackc/pgx/v5` (stdlib driver), `github.com/go-gormigrate/gormigrate/v2`, `github.com/go-playground/validator/v10`.
- Third-party: `gorm.io/gorm`, `gorm.io/driver/postgres`, `github.com/jackc/pgx/v5` (stdlib driver), `github.com/go-gormigrate/gormigrate/v2`, `github.com/go-playground/validator/v10`, `github.com/riverqueue/river` (its `rivermigrate` and `riverdriver/riverdatabasesql` packages, for the River schema in `lagoon.QueueMigrations`).
- Third-party, `attach` only: `gocloud.dev/blob` (file and memory drivers), `github.com/disintegration/imaging` (thumbnails).
- Standard library: `database/sql`, `crypto/aes`, `crypto/cipher`, `crypto/hkdf`, `log/slog`, `image`, among others.

View File

@@ -57,8 +57,9 @@ func migrator(gdb *gorm.DB, pluginID string, migrations []*gormigrate.Migration)
}, migrations), nil
}
// Migrate runs the framework-owned system_files and backend-admin sets
// first, then each plugin's HasMigrations set in party.Activate order.
// Migrate runs the framework-owned system_files, backend-admin and job-queue
// (QueueMigrations) sets first, then each plugin's HasMigrations set in
// party.Activate order.
func Migrate(gdb *gorm.DB, plugins []party.Plugin) error {
if gdb == nil {
return fmt.Errorf("lagoon: gorm db is nil")
@@ -77,6 +78,17 @@ func Migrate(gdb *gorm.DB, plugins []party.Plugin) error {
if err := admin.Migrate(); err != nil {
return fmt.Errorf("lagoon: migrate backend admin: %w", err)
}
sqlDB, err := gdb.DB()
if err != nil {
return fmt.Errorf("lagoon: migrate queue: %w", err)
}
queue, err := migrator(gdb, QueueHistoryID, QueueMigrations(sqlDB))
if err != nil {
return err
}
if err := queue.Migrate(); err != nil {
return fmt.Errorf("lagoon: migrate queue: %w", err)
}
for _, p := range plugins {
hm, ok := p.(pact.HasMigrations)
if !ok {

View File

@@ -0,0 +1,86 @@
package lagoon
import (
"context"
"database/sql"
"fmt"
"github.com/go-gormigrate/gormigrate/v2"
"github.com/riverqueue/river/riverdriver/riverdatabasesql"
"github.com/riverqueue/river/rivermigrate"
"gorm.io/gorm"
)
const (
// QueueHistoryID is the migration history id of the framework job-queue
// set. Its history table is summer_migrations_summercms_conga.
QueueHistoryID = "summercms.conga"
// JobsTable is the framework job record table. Its columns match the
// WinterCMS apparatus jobs table plus the internal river_job_id link.
JobsTable = "summer_jobs"
// RiverSchemaVersion pins the River schema the framework migrates to, so a
// River upgrade never applies new DDL without a new framework migration.
RiverSchemaVersion = 7
)
// QueueMigrations returns the framework job-queue migration set: River's
// schema pinned at RiverSchemaVersion, then the summer_jobs record table.
// River's migrator runs each of its own steps in a separate transaction on
// sqlDB, because some River migrations cannot share one transaction.
func QueueMigrations(sqlDB *sql.DB) []*gormigrate.Migration {
return []*gormigrate.Migration{
{
ID: "202609290001_river_schema",
Migrate: func(tx *gorm.DB) error {
return migrateRiver(txContext(tx), sqlDB, rivermigrate.DirectionUp, &rivermigrate.MigrateOpts{TargetVersion: RiverSchemaVersion})
},
Rollback: func(tx *gorm.DB) error {
// TargetVersion -1 applies every down step, removing River's schema.
return migrateRiver(txContext(tx), sqlDB, rivermigrate.DirectionDown, &rivermigrate.MigrateOpts{TargetVersion: -1})
},
},
{
ID: "202609290002_summer_jobs",
Migrate: func(tx *gorm.DB) error {
return tx.Exec(`CREATE TABLE IF NOT EXISTS summer_jobs (
id SERIAL PRIMARY KEY,
label VARCHAR(255) NOT NULL,
status INTEGER NOT NULL DEFAULT 0,
progress INTEGER NOT NULL DEFAULT 0,
progress_max INTEGER NOT NULL DEFAULT 0,
user_id INTEGER NULL,
is_admin BOOLEAN NOT NULL DEFAULT FALSE,
is_canceled BOOLEAN NOT NULL DEFAULT FALSE,
metadata TEXT NOT NULL,
river_job_id BIGINT NULL,
created_at TIMESTAMPTZ NULL,
updated_at TIMESTAMPTZ NULL
)`).Error
},
Rollback: func(tx *gorm.DB) error {
return tx.Exec(`DROP TABLE IF EXISTS summer_jobs`).Error
},
},
}
}
func migrateRiver(ctx context.Context, sqlDB *sql.DB, dir rivermigrate.Direction, opts *rivermigrate.MigrateOpts) error {
if sqlDB == nil {
return fmt.Errorf("lagoon: river migrate: sql db is nil")
}
m, err := rivermigrate.New(riverdatabasesql.New(sqlDB), nil)
if err != nil {
return fmt.Errorf("lagoon: river migrate: %w", err)
}
if _, err := m.Migrate(ctx, dir, opts); err != nil {
return fmt.Errorf("lagoon: river migrate %s: %w", dir, err)
}
return nil
}
func txContext(tx *gorm.DB) context.Context {
if tx != nil && tx.Statement != nil && tx.Statement.Context != nil {
return tx.Statement.Context
}
return context.Background()
}