feat(11-01): add the conga job framework on River with transactional dispatch
- Pin River v0.47.0 (riverdatabasesql, rivertype) and tidy the example modules - lagoon.Migrate runs the summercms.conga set: River schema v7 and summer_jobs with the apparatus columns plus an internal river_job_id link - conga.Manager.Dispatch writes the summer_jobs row (status IN_PROGRESS) and the River job on the caller's *sql.Tx; a rollback leaves neither - conga.Job wraps typed job functions so plugins never import River - conga.StartWorker runs one client on riverdatabasesql.NewWithPgxListener with a single-connection LISTEN pool; the final failed attempt sets ERROR - TestListenPickupLatency: 30s poll, pickup under 1s; poll-only control 2s miss
This commit is contained in:
135
modules/conga/README.md
Normal file
135
modules/conga/README.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# conga
|
||||
|
||||
Background jobs on River over the shared Postgres pool: transactional dispatch, a `summer_jobs` progress record, in-process or dedicated workers, and a wall-clock scheduler.
|
||||
|
||||
`import "git.golem15.com/golem15/summercms/modules/conga"`
|
||||
|
||||
## Overview
|
||||
|
||||
conga is the SummerCMS counterpart of the WinterCMS queue plus the apparatus job manager. Plugins describe background work as typed job functions wrapped by `conga.Job` and return them from `pact.HasJobs`, so plugin code never imports River. A caller dispatches a job with `conga.Manager.Dispatch` inside its own write transaction: the `summer_jobs` record row and the River job are written on the same `*sql.Tx`, so a rollback leaves neither behind. River only executes the work; the record row is what progress, outcome and cancellation reads use.
|
||||
|
||||
Every River client runs on the one `*sql.DB` pool that `lagoon` opens. A worker started by `conga.StartWorker` uses `riverdatabasesql.NewWithPgxListener`: all queries go through the shared pool, and only Postgres `LISTEN` goes through a dedicated pgx pool with a single connection, so a job committed by any process wakes the worker immediately instead of waiting for the poll interval.
|
||||
|
||||
## Features
|
||||
|
||||
- River-free job declarations: `conga.Job` turns `func(ctx context.Context, args T) error` into a `pact.Job`; `conga.OnQueue`, `conga.MaxAttempts` and `conga.Timeout` set per-job defaults. A `pact.Job` not built by `conga.Job` is rejected with `conga.ErrNotCongaJob`.
|
||||
- Transactional dispatch: `conga.Manager.Dispatch` inserts the `summer_jobs` row with `conga.StatusInProgress`, the principal's user id and admin flag, `progress_max` from `conga.DispatchOpts.Count` and JSON metadata, then enqueues the River job in the same transaction. It opens a transaction itself when the caller has none.
|
||||
- Plain enqueue: `conga.Manager.Enqueue` inserts a River job without a record row, inside the caller's transaction when there is one.
|
||||
- The record row: `conga.Record` maps `summer_jobs`; `conga.Status` holds the WinterCMS status values (`conga.StatusInQueue`, `conga.StatusInProgress`, `conga.StatusComplete`, `conga.StatusError`, `conga.StatusStopped`). `conga.Manager.CompleteJob` completes a row, `conga.Manager.Get` reads one, and `conga.JobID` gives a running job its own row id.
|
||||
- Outcome rules in the worker: an error on an attempt before the last leaves the row in progress so River can retry; the final failed attempt, or a recovered panic on it, sets `conga.StatusError` with the error text under the metadata key `error`. Skipped work is recorded as complete with `{"skipped": true}` metadata.
|
||||
- Workers: `conga.StartWorker` registers every plugin job and starts one River client; `conga.WorkerOptions.Queues` limits it to some queues, and an unknown queue is `conga.ErrUnknownQueue` listing the known ones. `conga.Worker.Stop` stops it gracefully and cancels running jobs when its context ends.
|
||||
|
||||
## Usage
|
||||
|
||||
A plugin declares a job:
|
||||
|
||||
```go
|
||||
package blog
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"git.golem15.com/golem15/summercms/modules/conga"
|
||||
"git.golem15.com/golem15/summercms/modules/pact"
|
||||
)
|
||||
|
||||
type ImportPostsArgs struct {
|
||||
File string `json:"file"`
|
||||
}
|
||||
|
||||
func (ImportPostsArgs) Kind() string { return "acme_blog_import_posts" }
|
||||
|
||||
func ImportPostsJob(m *conga.Manager) pact.Job {
|
||||
return conga.Job(func(ctx context.Context, args ImportPostsArgs) error {
|
||||
id, _ := conga.JobID(ctx)
|
||||
// ... import args.File ...
|
||||
return m.CompleteJob(ctx, id, nil)
|
||||
}, conga.OnQueue("imports"))
|
||||
}
|
||||
```
|
||||
|
||||
and dispatches it inside the write that needs it:
|
||||
|
||||
```go
|
||||
func startImport(ctx context.Context, app *backpack.App, gdb *gorm.DB, file string) (uint, error) {
|
||||
m, err := conga.From(app)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var id uint
|
||||
err = gdb.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
// ... write the import record ...
|
||||
id, err = m.Dispatch(ctx, tx, ImportPostsArgs{File: file}, conga.DispatchOpts{Label: "Import posts", Count: 100})
|
||||
return err
|
||||
})
|
||||
return id, err
|
||||
}
|
||||
```
|
||||
|
||||
A worker runs in the same process or in a separate one:
|
||||
|
||||
```go
|
||||
w, err := conga.StartWorker(ctx, app, plugins, conga.WorkerOptions{})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer w.Stop(context.Background())
|
||||
```
|
||||
|
||||
## API reference
|
||||
|
||||
| Identifier | Description |
|
||||
|------------|-------------|
|
||||
| `conga.From` | Returns the app's `conga.Manager`, publishing one on first use. |
|
||||
| `conga.Manager` | App-scoped job manager: registration, dispatch and the `summer_jobs` record. |
|
||||
| `conga.Manager.Register` | Registers jobs built by `conga.Job`; closed while a worker runs (`conga.ErrRegistrationClosed`). |
|
||||
| `conga.Manager.Dispatch` | Writes the record row and enqueues the River job in one transaction; returns the row id. |
|
||||
| `conga.Manager.Enqueue` | Enqueues a River job without a record row. |
|
||||
| `conga.Manager.CompleteJob` | Sets `conga.StatusComplete` and progress to `progress_max`; replaces metadata when given. |
|
||||
| `conga.Manager.Get` | Reads one `conga.Record`. |
|
||||
| `conga.DispatchOpts` | Label, count, metadata, queue, delay and attempt limit of a dispatch. |
|
||||
| `conga.EnqueueOpts` | Queue, delay and attempt limit of an enqueue. |
|
||||
| `conga.Record` | The `summer_jobs` row model. |
|
||||
| `conga.Status` | Record status values, matching the WinterCMS job statuses. |
|
||||
| `conga.Job` | Wraps a typed job function as a `pact.Job` that conga can run on River. |
|
||||
| `conga.JobOption` | Per-job option: `conga.OnQueue`, `conga.MaxAttempts`, `conga.Timeout`. |
|
||||
| `conga.JobID` | Returns the record row id of the job running in a context. |
|
||||
| `conga.StartWorker` | Registers plugin jobs and starts a River worker client. |
|
||||
| `conga.WorkerOptions` | Selects the queues a worker runs. |
|
||||
| `conga.Worker` | A running worker; `conga.Worker.Stop` stops it and `conga.Worker.Queues` lists its queues. |
|
||||
| `conga.ErrNoDatabase` | The app has not published the shared database handles. |
|
||||
| `conga.ErrNotCongaJob` | A registered `pact.Job` was not built by `conga.Job`. |
|
||||
| `conga.ErrRegistrationClosed` | Registration was attempted while a worker runs. |
|
||||
| `conga.ErrUnknownQueue` | A worker was asked for a queue nothing names. |
|
||||
|
||||
## Configuration
|
||||
|
||||
Keys are read from the compass config (`config/queue.yaml`, or `SUMMER_QUEUE__...` environment variables).
|
||||
|
||||
| Key | Default | Controls |
|
||||
|-----|---------|----------|
|
||||
| `queue.max_attempts` | `3` | Attempts per job before its record becomes `conga.StatusError`, unless the job or dispatch sets its own. |
|
||||
| `queue.job_timeout` | `300` | Per-attempt deadline, in seconds or as a duration string such as `5m`, unless the job sets `conga.Timeout`. |
|
||||
| `queue.queues.<name>` | `default: 4` | Concurrent workers per queue. The worker runs these queues plus every queue a registered job names plus `default`. |
|
||||
| `database.dsn` | none (required) | Also opens the worker's single-connection `LISTEN` pool. With PgBouncer, that connection must use session pooling or go straight to Postgres; transaction pooling cannot hold a `LISTEN`. |
|
||||
|
||||
```yaml
|
||||
max_attempts: 3
|
||||
job_timeout: 300
|
||||
queues:
|
||||
default: 4
|
||||
imports: 1
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
- SummerCMS modules: [backpack](../backpack/README.md), [bouncer](../bouncer/README.md) (the dispatching principal), [lagoon](../lagoon/README.md) (the shared pool, `lagoon.JobsTable` and the migrations that create River's schema and `summer_jobs`), [pact](../pact/README.md), [party](../party/README.md).
|
||||
- Third-party: `github.com/riverqueue/river` v0.47.0 with its `riverdriver/riverdatabasesql` and `rivertype` modules. River is the Postgres job queue the project stack names: it gives transactional inserts on the shared `*sql.DB`, retries, stuck-job rescue, leader election for periodic work and `LISTEN`/`NOTIFY` wake-ups. `github.com/jackc/pgx/v5/pgxpool` opens the listener pool; `gorm.io/gorm` writes the record rows.
|
||||
|
||||
## Testing
|
||||
|
||||
```sh
|
||||
go test ./modules/conga/...
|
||||
```
|
||||
|
||||
The tests start a `postgres:16-alpine` container through testcontainers-go and migrate a fresh database per test with `lagoon.Migrate`, so they need a running Docker daemon. `TestListenPickupLatency` sets a 30-second poll interval and checks that a job committed by a separate client is picked up in under one second, while a poll-only worker does not pick it up within two. `go test -short ./modules/conga/...` skips the database tests.
|
||||
123
modules/conga/client.go
Normal file
123
modules/conga/client.go
Normal file
@@ -0,0 +1,123 @@
|
||||
package conga
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"log/slog"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.golem15.com/golem15/summercms/modules/backpack"
|
||||
"github.com/riverqueue/river"
|
||||
"github.com/riverqueue/river/riverdriver/riverdatabasesql"
|
||||
)
|
||||
|
||||
const (
|
||||
// defaultQueue is River's default queue name.
|
||||
defaultQueue = river.QueueDefault
|
||||
// defaultMaxAttempts matches the WinterCMS queue worker's --tries=3.
|
||||
defaultMaxAttempts = 3
|
||||
// defaultJobTimeout matches the WinterCMS queue worker's --timeout=300.
|
||||
defaultJobTimeout = 300 * time.Second
|
||||
// defaultMaxWorkers is the concurrency of a queue without configuration.
|
||||
defaultMaxWorkers = 4
|
||||
)
|
||||
|
||||
// settings is the queue.* configuration of one app.
|
||||
type settings struct {
|
||||
maxAttempts int
|
||||
jobTimeout time.Duration
|
||||
queues map[string]int
|
||||
workInServe bool
|
||||
}
|
||||
|
||||
func settingsFromApp(app *backpack.App) settings {
|
||||
s := settings{
|
||||
maxAttempts: defaultMaxAttempts,
|
||||
jobTimeout: defaultJobTimeout,
|
||||
queues: map[string]int{},
|
||||
workInServe: true,
|
||||
}
|
||||
if app == nil || app.Config == nil {
|
||||
return s
|
||||
}
|
||||
c := app.Config
|
||||
if n := c.Int("queue.max_attempts"); n > 0 {
|
||||
s.maxAttempts = n
|
||||
}
|
||||
if raw := strings.TrimSpace(c.String("queue.job_timeout")); raw != "" {
|
||||
if d, err := time.ParseDuration(raw); err == nil && d > 0 {
|
||||
s.jobTimeout = d
|
||||
} else if n := c.Int("queue.job_timeout"); n > 0 {
|
||||
s.jobTimeout = time.Duration(n) * time.Second
|
||||
}
|
||||
}
|
||||
if c.Has("queue.work_in_serve") {
|
||||
s.workInServe = c.Bool("queue.work_in_serve")
|
||||
}
|
||||
if raw, ok := c.Lookup("queue.queues"); ok {
|
||||
if m, ok := raw.(map[string]any); ok {
|
||||
for name := range m {
|
||||
name = strings.TrimSpace(name)
|
||||
if name == "" {
|
||||
continue
|
||||
}
|
||||
n := c.Int("queue.queues." + name)
|
||||
if n <= 0 {
|
||||
n = defaultMaxWorkers
|
||||
}
|
||||
s.queues[name] = n
|
||||
}
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// knownQueues is the configured queues plus every queue a registered job
|
||||
// names plus default, each with its MaxWorkers.
|
||||
func knownQueues(s settings, jobs map[string]congaJob) map[string]int {
|
||||
out := map[string]int{defaultQueue: defaultMaxWorkers}
|
||||
for _, j := range jobs {
|
||||
if q := j.config().queue; q != "" {
|
||||
out[q] = defaultMaxWorkers
|
||||
}
|
||||
}
|
||||
for name, n := range s.queues {
|
||||
out[name] = n
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func sortedQueueNames(queues map[string]int) []string {
|
||||
names := make([]string, 0, len(queues))
|
||||
for name := range queues {
|
||||
names = append(names, name)
|
||||
}
|
||||
sort.Strings(names)
|
||||
return names
|
||||
}
|
||||
|
||||
// baseConfig is the River client configuration shared by the insert-only
|
||||
// and worker clients. Queues and Workers are set by the worker only.
|
||||
func baseConfig(s settings, log *slog.Logger) *river.Config {
|
||||
return &river.Config{
|
||||
MaxAttempts: s.maxAttempts,
|
||||
JobTimeout: s.jobTimeout,
|
||||
Logger: log,
|
||||
}
|
||||
}
|
||||
|
||||
// newInsertClient builds an insert-only client on the shared pool. It is
|
||||
// never started; it inserts, cancels and deletes jobs.
|
||||
func newInsertClient(sqlDB *sql.DB, s settings, log *slog.Logger) (*river.Client[*sql.Tx], error) {
|
||||
return river.NewClient(riverdatabasesql.New(sqlDB), baseConfig(s, log))
|
||||
}
|
||||
|
||||
func loggerFromApp(app *backpack.App) *slog.Logger {
|
||||
if app != nil {
|
||||
if log, ok := app.Lookup[*slog.Logger](); ok && log != nil {
|
||||
return log
|
||||
}
|
||||
}
|
||||
return slog.Default()
|
||||
}
|
||||
406
modules/conga/conga.go
Normal file
406
modules/conga/conga.go
Normal file
@@ -0,0 +1,406 @@
|
||||
// Package conga runs background jobs on River over the shared Postgres pool
|
||||
// and keeps a queryable summer_jobs record of each dispatched job.
|
||||
package conga
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"git.golem15.com/golem15/summercms/modules/backpack"
|
||||
"git.golem15.com/golem15/summercms/modules/bouncer"
|
||||
"git.golem15.com/golem15/summercms/modules/lagoon"
|
||||
"git.golem15.com/golem15/summercms/modules/pact"
|
||||
"github.com/riverqueue/river"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrNoDatabase is returned when the app has not published the shared
|
||||
// *sql.DB and *gorm.DB (see lagoon.Publish).
|
||||
ErrNoDatabase = errors.New("conga: database is not published")
|
||||
// ErrRegistrationClosed is returned by Register while a worker client
|
||||
// runs, because River fixes its worker set when the client is built.
|
||||
ErrRegistrationClosed = errors.New("conga: job registration is closed while a worker runs")
|
||||
// ErrNotCongaJob is returned for a pact.Job that was not built by Job.
|
||||
ErrNotCongaJob = errors.New("conga: job was not built by conga.Job")
|
||||
// ErrUnknownQueue is returned when a worker is asked for a queue that no
|
||||
// configuration or registered job names.
|
||||
ErrUnknownQueue = errors.New("conga: unknown queue")
|
||||
)
|
||||
|
||||
// Manager is the app-scoped job manager: it registers jobs, dispatches them
|
||||
// with a summer_jobs row and records their outcome. Get one with From.
|
||||
type Manager struct {
|
||||
app *backpack.App
|
||||
|
||||
mu sync.Mutex
|
||||
jobs map[string]congaJob
|
||||
inserter *river.Client[*sql.Tx]
|
||||
worker *river.Client[*sql.Tx]
|
||||
}
|
||||
|
||||
// From returns the app's Manager, publishing a new one on first use.
|
||||
func From(app *backpack.App) (*Manager, error) {
|
||||
if app == nil {
|
||||
return nil, fmt.Errorf("conga: app is nil")
|
||||
}
|
||||
if m, ok := app.Lookup[*Manager](); ok && m != nil {
|
||||
return m, nil
|
||||
}
|
||||
m := &Manager{app: app, jobs: map[string]congaJob{}}
|
||||
if err := app.Publish(m); err != nil {
|
||||
if existing, ok := app.Lookup[*Manager](); ok && existing != nil {
|
||||
return existing, nil
|
||||
}
|
||||
return nil, fmt.Errorf("conga: %w", err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// Register adds jobs built by Job. A job not built by Job is ErrNotCongaJob,
|
||||
// a second job with the same kind is an error, and registering while a
|
||||
// worker runs is ErrRegistrationClosed.
|
||||
func (m *Manager) Register(jobs ...pact.Job) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
return m.registerLocked(jobs...)
|
||||
}
|
||||
|
||||
func (m *Manager) registerLocked(jobs ...pact.Job) error {
|
||||
for _, j := range jobs {
|
||||
cj, ok := j.(congaJob)
|
||||
if !ok || cj == nil {
|
||||
return fmt.Errorf("%w (got %T)", ErrNotCongaJob, j)
|
||||
}
|
||||
kind := cj.kind()
|
||||
if strings.TrimSpace(kind) == "" {
|
||||
return fmt.Errorf("conga: job args %T have an empty Kind()", j)
|
||||
}
|
||||
if existing, dup := m.jobs[kind]; dup {
|
||||
if existing == cj {
|
||||
continue
|
||||
}
|
||||
return fmt.Errorf("conga: job kind %q is already registered", kind)
|
||||
}
|
||||
if m.worker != nil {
|
||||
return ErrRegistrationClosed
|
||||
}
|
||||
m.jobs[kind] = cj
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// DispatchOpts are the parameters of Dispatch. Label is required.
|
||||
type DispatchOpts struct {
|
||||
// Label is stored in summer_jobs.label.
|
||||
Label string
|
||||
// Count is the initial progress_max.
|
||||
Count int
|
||||
// Metadata is stored as JSON in summer_jobs.metadata; nil stores the
|
||||
// JSON string "" as the WinterCMS job manager does without metadata.
|
||||
Metadata map[string]any
|
||||
// Queue overrides the job's queue.
|
||||
Queue string
|
||||
// Delay schedules the first attempt after this duration.
|
||||
Delay time.Duration
|
||||
// MaxAttempts overrides the job's attempt limit.
|
||||
MaxAttempts int
|
||||
}
|
||||
|
||||
// EnqueueOpts are the parameters of Enqueue.
|
||||
type EnqueueOpts struct {
|
||||
Queue string
|
||||
Delay time.Duration
|
||||
MaxAttempts int
|
||||
}
|
||||
|
||||
// Dispatch inserts a summer_jobs row with StatusInProgress and enqueues the
|
||||
// River job in the same transaction, then returns the row id. When db is
|
||||
// already inside a transaction both writes join it, so a rollback leaves
|
||||
// neither behind; otherwise Dispatch opens its own. The row's user_id and
|
||||
// is_admin come from the bouncer principal in ctx.
|
||||
func (m *Manager) Dispatch(ctx context.Context, db *gorm.DB, args pact.JobArgs, o DispatchOpts) (uint, error) {
|
||||
if args == nil {
|
||||
return 0, fmt.Errorf("conga: dispatch args are nil")
|
||||
}
|
||||
label := strings.TrimSpace(o.Label)
|
||||
if label == "" {
|
||||
return 0, fmt.Errorf("conga: dispatch label is empty")
|
||||
}
|
||||
db, err := m.dbOr(db)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
client, err := m.insertClient()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
meta := `""`
|
||||
if o.Metadata != nil {
|
||||
meta, err = encodeMetadata(o.Metadata)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
var id uint
|
||||
run := func(tx *gorm.DB) error {
|
||||
sqlTx, ok := tx.Statement.ConnPool.(*sql.Tx)
|
||||
if !ok {
|
||||
return fmt.Errorf("conga: dispatch needs a *sql.Tx connection, got %T", tx.Statement.ConnPool)
|
||||
}
|
||||
now := time.Now()
|
||||
rec := Record{
|
||||
Label: label,
|
||||
Status: StatusInProgress,
|
||||
ProgressMax: o.Count,
|
||||
Metadata: meta,
|
||||
CreatedAt: &now,
|
||||
UpdatedAt: &now,
|
||||
}
|
||||
if p, ok := bouncer.User(ctx); ok {
|
||||
uid := p.ID
|
||||
rec.UserID = &uid
|
||||
rec.IsAdmin = p.Backend
|
||||
}
|
||||
if err := tx.Create(&rec).Error; err != nil {
|
||||
return fmt.Errorf("conga: insert job record: %w", err)
|
||||
}
|
||||
opts, err := m.insertOpts(args.Kind(), o.Queue, o.MaxAttempts, o.Delay)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
opts.Metadata, err = json.Marshal(map[string]any{"summer_job_id": rec.ID})
|
||||
if err != nil {
|
||||
return fmt.Errorf("conga: job metadata: %w", err)
|
||||
}
|
||||
res, err := client.InsertTx(ctx, sqlTx, args, opts)
|
||||
if err != nil {
|
||||
return fmt.Errorf("conga: enqueue %s: %w", args.Kind(), err)
|
||||
}
|
||||
if err := tx.Table(lagoon.JobsTable).Where("id = ?", rec.ID).UpdateColumn("river_job_id", res.Job.ID).Error; err != nil {
|
||||
return fmt.Errorf("conga: link river job: %w", err)
|
||||
}
|
||||
id = rec.ID
|
||||
return nil
|
||||
}
|
||||
if inTx(db) {
|
||||
err = run(db.WithContext(ctx))
|
||||
} else {
|
||||
err = db.WithContext(ctx).Transaction(run)
|
||||
}
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// Enqueue inserts a River job without a summer_jobs row. When db is inside a
|
||||
// transaction the job joins it; otherwise it is inserted on its own. db may
|
||||
// be nil to use the published *gorm.DB.
|
||||
func (m *Manager) Enqueue(ctx context.Context, db *gorm.DB, args pact.JobArgs, o EnqueueOpts) error {
|
||||
if args == nil {
|
||||
return fmt.Errorf("conga: enqueue args are nil")
|
||||
}
|
||||
client, err := m.insertClient()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
opts, err := m.insertOpts(args.Kind(), o.Queue, o.MaxAttempts, o.Delay)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if db != nil && inTx(db) {
|
||||
sqlTx := db.Statement.ConnPool.(*sql.Tx)
|
||||
_, err = client.InsertTx(ctx, sqlTx, args, opts)
|
||||
} else {
|
||||
_, err = client.Insert(ctx, args, opts)
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("conga: enqueue %s: %w", args.Kind(), err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// CompleteJob sets StatusComplete and progress to progress_max (1 when the
|
||||
// row is missing), replacing metadata only when it is non-empty. Skipped work
|
||||
// is recorded as CompleteJob(ctx, id, map[string]any{"skipped": true}); there
|
||||
// is no separate skipped status. updated_at is not touched.
|
||||
func (m *Manager) CompleteJob(ctx context.Context, id uint, metadata map[string]any) error {
|
||||
gdb, err := m.gdb()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
maxProgress := 1
|
||||
var rows []struct{ ProgressMax int }
|
||||
if err := gdb.WithContext(ctx).Table(lagoon.JobsTable).Select("progress_max").Where("id = ?", id).Limit(1).Scan(&rows).Error; err != nil {
|
||||
return fmt.Errorf("conga: read job %d: %w", id, err)
|
||||
}
|
||||
if len(rows) == 1 {
|
||||
maxProgress = rows[0].ProgressMax
|
||||
}
|
||||
cols := map[string]any{"status": int(StatusComplete), "progress": maxProgress}
|
||||
if len(metadata) > 0 {
|
||||
enc, err := encodeMetadata(metadata)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cols["metadata"] = enc
|
||||
}
|
||||
return m.updateColumns(ctx, gdb, id, cols)
|
||||
}
|
||||
|
||||
// Get returns the summer_jobs row with id.
|
||||
func (m *Manager) Get(ctx context.Context, id uint) (Record, error) {
|
||||
gdb, err := m.gdb()
|
||||
if err != nil {
|
||||
return Record{}, err
|
||||
}
|
||||
var rec Record
|
||||
if err := gdb.WithContext(ctx).Where("id = ?", id).Take(&rec).Error; err != nil {
|
||||
return Record{}, fmt.Errorf("conga: job %d: %w", id, err)
|
||||
}
|
||||
return rec, nil
|
||||
}
|
||||
|
||||
// failWithError is the worker-side final failure: StatusError with the
|
||||
// current metadata plus the error text under "error".
|
||||
func (m *Manager) failWithError(ctx context.Context, id uint, jobErr error) error {
|
||||
gdb, err := m.gdb()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
meta, err := m.metadata(ctx, gdb, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
meta["error"] = jobErr.Error()
|
||||
enc, err := encodeMetadata(meta)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return m.updateColumns(ctx, gdb, id, map[string]any{"status": int(StatusError), "metadata": enc})
|
||||
}
|
||||
|
||||
func (m *Manager) metadata(ctx context.Context, gdb *gorm.DB, id uint) (map[string]any, error) {
|
||||
var rows []struct{ Metadata string }
|
||||
if err := gdb.WithContext(ctx).Table(lagoon.JobsTable).Select("metadata").Where("id = ?", id).Limit(1).Scan(&rows).Error; err != nil {
|
||||
return nil, fmt.Errorf("conga: read job %d metadata: %w", id, err)
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
return nil, fmt.Errorf("conga: job %d: %w", id, gorm.ErrRecordNotFound)
|
||||
}
|
||||
return decodeMetadata(rows[0].Metadata), nil
|
||||
}
|
||||
|
||||
// updateColumns writes cols with a raw column update so GORM never sets
|
||||
// updated_at on its own, matching the query-builder updates of the
|
||||
// WinterCMS job manager.
|
||||
func (m *Manager) updateColumns(ctx context.Context, gdb *gorm.DB, id uint, cols map[string]any) error {
|
||||
if err := gdb.WithContext(ctx).Table(lagoon.JobsTable).Where("id = ?", id).UpdateColumns(cols).Error; err != nil {
|
||||
return fmt.Errorf("conga: update job %d: %w", id, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manager) insertOpts(kind, queue string, maxAttempts int, delay time.Duration) (*river.InsertOpts, error) {
|
||||
m.mu.Lock()
|
||||
j := m.jobs[kind]
|
||||
m.mu.Unlock()
|
||||
opts := &river.InsertOpts{Queue: queue, MaxAttempts: maxAttempts}
|
||||
if j != nil {
|
||||
cfg := j.config()
|
||||
if opts.Queue == "" {
|
||||
opts.Queue = cfg.queue
|
||||
}
|
||||
if opts.MaxAttempts == 0 {
|
||||
opts.MaxAttempts = cfg.maxAttempts
|
||||
}
|
||||
}
|
||||
if delay > 0 {
|
||||
opts.ScheduledAt = time.Now().Add(delay)
|
||||
}
|
||||
return opts, nil
|
||||
}
|
||||
|
||||
// insertClient returns the running worker client, else the lazily built
|
||||
// insert-only client on the shared pool.
|
||||
func (m *Manager) insertClient() (*river.Client[*sql.Tx], error) {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if m.worker != nil {
|
||||
return m.worker, nil
|
||||
}
|
||||
if m.inserter != nil {
|
||||
return m.inserter, nil
|
||||
}
|
||||
sqlDB, ok := m.app.Lookup[*sql.DB]()
|
||||
if !ok || sqlDB == nil {
|
||||
return nil, ErrNoDatabase
|
||||
}
|
||||
c, err := newInsertClient(sqlDB, settingsFromApp(m.app), loggerFromApp(m.app))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("conga: river client: %w", err)
|
||||
}
|
||||
m.inserter = c
|
||||
return c, nil
|
||||
}
|
||||
|
||||
func (m *Manager) gdb() (*gorm.DB, error) {
|
||||
gdb, ok := m.app.Lookup[*gorm.DB]()
|
||||
if !ok || gdb == nil {
|
||||
return nil, ErrNoDatabase
|
||||
}
|
||||
return gdb, nil
|
||||
}
|
||||
|
||||
func (m *Manager) dbOr(db *gorm.DB) (*gorm.DB, error) {
|
||||
if db != nil {
|
||||
return db, nil
|
||||
}
|
||||
return m.gdb()
|
||||
}
|
||||
|
||||
func inTx(db *gorm.DB) bool {
|
||||
if db == nil || db.Statement == nil {
|
||||
return false
|
||||
}
|
||||
_, ok := db.Statement.ConnPool.(*sql.Tx)
|
||||
return ok
|
||||
}
|
||||
|
||||
// encodeMetadata JSON-encodes metadata the way the WinterCMS job manager's
|
||||
// json_encode does for an array: an empty map is [] and slashes and HTML
|
||||
// characters are left as is.
|
||||
func encodeMetadata(metadata map[string]any) (string, error) {
|
||||
if len(metadata) == 0 {
|
||||
return "[]", nil
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
enc := json.NewEncoder(&buf)
|
||||
enc.SetEscapeHTML(false)
|
||||
if err := enc.Encode(metadata); err != nil {
|
||||
return "", fmt.Errorf("conga: encode metadata: %w", err)
|
||||
}
|
||||
return strings.TrimSuffix(buf.String(), "\n"), nil
|
||||
}
|
||||
|
||||
// decodeMetadata mirrors json_decode(...) ?: []: anything that is not a
|
||||
// non-empty JSON object decodes to an empty map.
|
||||
func decodeMetadata(raw string) map[string]any {
|
||||
out := map[string]any{}
|
||||
var v any
|
||||
if err := json.Unmarshal([]byte(raw), &v); err != nil {
|
||||
return out
|
||||
}
|
||||
if obj, ok := v.(map[string]any); ok {
|
||||
return obj
|
||||
}
|
||||
return out
|
||||
}
|
||||
124
modules/conga/job.go
Normal file
124
modules/conga/job.go
Normal file
@@ -0,0 +1,124 @@
|
||||
package conga
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"git.golem15.com/golem15/summercms/modules/pact"
|
||||
"github.com/riverqueue/river"
|
||||
)
|
||||
|
||||
type jobConfig struct {
|
||||
queue string
|
||||
maxAttempts int
|
||||
timeout time.Duration
|
||||
}
|
||||
|
||||
// JobOption configures a job built by Job.
|
||||
type JobOption func(*jobConfig)
|
||||
|
||||
// OnQueue sets the queue a job is inserted on when the caller does not name
|
||||
// one. Empty means the River default queue, "default".
|
||||
func OnQueue(name string) JobOption {
|
||||
return func(c *jobConfig) { c.queue = name }
|
||||
}
|
||||
|
||||
// MaxAttempts sets how many times River tries the job before the summer_jobs
|
||||
// row is marked StatusError. Zero means queue.max_attempts.
|
||||
func MaxAttempts(n int) JobOption {
|
||||
return func(c *jobConfig) { c.maxAttempts = n }
|
||||
}
|
||||
|
||||
// Timeout sets the per-attempt deadline of the job. Zero means
|
||||
// queue.job_timeout.
|
||||
func Timeout(d time.Duration) JobOption {
|
||||
return func(c *jobConfig) { c.timeout = d }
|
||||
}
|
||||
|
||||
// congaJob is the unexported side of a job built by Job: the parts the worker
|
||||
// needs to register it on River without the plugin importing River.
|
||||
type congaJob interface {
|
||||
pact.Job
|
||||
kind() string
|
||||
config() jobConfig
|
||||
register(workers *river.Workers, m *Manager) error
|
||||
}
|
||||
|
||||
type typedJob[T pact.JobArgs] struct {
|
||||
fn func(ctx context.Context, args T) error
|
||||
cfg jobConfig
|
||||
}
|
||||
|
||||
// Job wraps a typed job function as a pact.Job that conga can register on
|
||||
// River. Plugins return these values from pact.HasJobs.Jobs and never import
|
||||
// River. The job kind is T's Kind(); the args are stored as JSON.
|
||||
func Job[T pact.JobArgs](fn func(ctx context.Context, args T) error, opts ...JobOption) pact.Job {
|
||||
j := &typedJob[T]{fn: fn}
|
||||
for _, o := range opts {
|
||||
if o != nil {
|
||||
o(&j.cfg)
|
||||
}
|
||||
}
|
||||
return j
|
||||
}
|
||||
|
||||
// Work runs the job function when args has the job's argument type.
|
||||
func (j *typedJob[T]) Work(ctx context.Context, args pact.JobArgs) error {
|
||||
if j.fn == nil {
|
||||
return fmt.Errorf("conga: job %s has no function", j.kind())
|
||||
}
|
||||
a, ok := args.(T)
|
||||
if !ok {
|
||||
return fmt.Errorf("conga: job %s got args of type %T", j.kind(), args)
|
||||
}
|
||||
return j.fn(ctx, a)
|
||||
}
|
||||
|
||||
func (j *typedJob[T]) kind() string {
|
||||
var zero T
|
||||
return zero.Kind()
|
||||
}
|
||||
|
||||
func (j *typedJob[T]) config() jobConfig { return j.cfg }
|
||||
|
||||
func (j *typedJob[T]) register(workers *river.Workers, m *Manager) error {
|
||||
if j.fn == nil {
|
||||
return fmt.Errorf("conga: job %s has no function", j.kind())
|
||||
}
|
||||
return river.AddWorkerSafely[T](workers, &riverWorker[T]{job: j, m: m})
|
||||
}
|
||||
|
||||
// riverWorker adapts a typedJob onto River and applies the summer_jobs
|
||||
// outcome rules around each attempt.
|
||||
type riverWorker[T pact.JobArgs] struct {
|
||||
river.WorkerDefaults[T]
|
||||
job *typedJob[T]
|
||||
m *Manager
|
||||
}
|
||||
|
||||
func (w *riverWorker[T]) Work(ctx context.Context, job *river.Job[T]) error {
|
||||
return w.m.runAttempt(ctx, job.JobRow, func(ctx context.Context) error {
|
||||
return w.job.fn(ctx, job.Args)
|
||||
})
|
||||
}
|
||||
|
||||
func (w *riverWorker[T]) Timeout(*river.Job[T]) time.Duration {
|
||||
return w.job.cfg.timeout
|
||||
}
|
||||
|
||||
type jobIDKey struct{}
|
||||
|
||||
// JobID returns the summer_jobs id of the dispatched job running in ctx.
|
||||
// Jobs inserted with Manager.Enqueue have no row and report false.
|
||||
func JobID(ctx context.Context) (uint, bool) {
|
||||
if ctx == nil {
|
||||
return 0, false
|
||||
}
|
||||
id, ok := ctx.Value(jobIDKey{}).(uint)
|
||||
return id, ok && id > 0
|
||||
}
|
||||
|
||||
func withJobID(ctx context.Context, id uint) context.Context {
|
||||
return context.WithValue(ctx, jobIDKey{}, id)
|
||||
}
|
||||
269
modules/conga/listen_test.go
Normal file
269
modules/conga/listen_test.go
Normal file
@@ -0,0 +1,269 @@
|
||||
package conga
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.golem15.com/golem15/summercms/modules/backpack"
|
||||
"git.golem15.com/golem15/summercms/modules/bouncer"
|
||||
"git.golem15.com/golem15/summercms/modules/pact"
|
||||
"git.golem15.com/golem15/summercms/modules/party"
|
||||
"github.com/riverqueue/river"
|
||||
"github.com/riverqueue/river/riverdriver/riverdatabasesql"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
type pingArgs struct {
|
||||
Tag string `json:"tag"`
|
||||
}
|
||||
|
||||
func (pingArgs) Kind() string { return "conga_test_ping" }
|
||||
|
||||
// jobsPlugin is a minimal compiled plugin that only declares jobs.
|
||||
type jobsPlugin struct {
|
||||
id string
|
||||
jobs []pact.Job
|
||||
}
|
||||
|
||||
func (p *jobsPlugin) ID() string { return p.id }
|
||||
func (p *jobsPlugin) Requires() []string { return nil }
|
||||
func (p *jobsPlugin) Register(app *backpack.App) error { return nil }
|
||||
func (p *jobsPlugin) Boot(app *backpack.App) error { return nil }
|
||||
func (p *jobsPlugin) Jobs() []pact.Job { return p.jobs }
|
||||
|
||||
func plugins(jobs ...pact.Job) []party.Plugin {
|
||||
return []party.Plugin{&jobsPlugin{id: "acme.jobs", jobs: jobs}}
|
||||
}
|
||||
|
||||
func startTestWorker(t *testing.T, app *backpack.App, o WorkerOptions, jobs ...pact.Job) *Worker {
|
||||
t.Helper()
|
||||
w, err := StartWorker(t.Context(), app, plugins(jobs...), o)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
if err := w.Stop(ctx); err != nil {
|
||||
t.Errorf("stop worker: %v", err)
|
||||
}
|
||||
})
|
||||
return w
|
||||
}
|
||||
|
||||
// waitForListener blocks until a backend of dbName is in LISTEN.
|
||||
func waitForListener(t *testing.T, dbName string) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(15 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
var n int
|
||||
if err := adminDB(t).QueryRowContext(t.Context(),
|
||||
`SELECT count(*) FROM pg_stat_activity WHERE datname = $1 AND query ILIKE 'LISTEN%'`, dbName).Scan(&n); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n > 0 {
|
||||
return
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("no LISTEN backend on %s within 15s", dbName)
|
||||
}
|
||||
|
||||
func currentDatabase(t *testing.T, db *sql.DB) string {
|
||||
t.Helper()
|
||||
var name string
|
||||
if err := db.QueryRowContext(t.Context(), `SELECT current_database()`).Scan(&name); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
// insertFromOtherClient commits one ping job through a separate insert-only
|
||||
// client, the way another process would, and returns the commit time.
|
||||
func insertFromOtherClient(t *testing.T, db *sql.DB) time.Time {
|
||||
t.Helper()
|
||||
other, err := river.NewClient(riverdatabasesql.New(db), &river.Config{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tx, err := db.BeginTx(t.Context(), nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := other.InsertTx(t.Context(), tx, pingArgs{Tag: "latency"}, nil); err != nil {
|
||||
_ = tx.Rollback()
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
// TestListenPickupLatency proves the worker wakes on LISTEN/NOTIFY: with a
|
||||
// 30s poll interval a committed job runs within 1s, while the same worker
|
||||
// without the LISTEN pool does not see it within 2s.
|
||||
func TestListenPickupLatency(t *testing.T) {
|
||||
const poll = 30 * time.Second
|
||||
|
||||
t.Run("listen", func(t *testing.T) {
|
||||
db, dsn := migratedDB(t)
|
||||
app, _ := testApp(t, db, dsn, nil)
|
||||
ran := make(chan time.Time, 4)
|
||||
startTestWorker(t, app, WorkerOptions{pollInterval: poll}, Job(func(ctx context.Context, a pingArgs) error {
|
||||
ran <- time.Now()
|
||||
return nil
|
||||
}))
|
||||
waitForListener(t, currentDatabase(t, db))
|
||||
time.Sleep(500 * time.Millisecond) // let the start-up fetch pass
|
||||
committed := insertFromOtherClient(t, db)
|
||||
select {
|
||||
case at := <-ran:
|
||||
latency := at.Sub(committed)
|
||||
t.Logf("LISTEN pickup latency %s with a %s poll interval", latency, poll)
|
||||
if latency >= time.Second {
|
||||
t.Fatalf("pickup took %s, want under 1s", latency)
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("job was not picked up within 1s of commit")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("poll_only_control", func(t *testing.T) {
|
||||
db, dsn := migratedDB(t)
|
||||
app, _ := testApp(t, db, dsn, nil)
|
||||
ran := make(chan time.Time, 4)
|
||||
startTestWorker(t, app, WorkerOptions{pollInterval: poll, pollOnly: true}, Job(func(ctx context.Context, a pingArgs) error {
|
||||
ran <- time.Now()
|
||||
return nil
|
||||
}))
|
||||
time.Sleep(1500 * time.Millisecond) // let the start-up fetch pass
|
||||
insertFromOtherClient(t, db)
|
||||
select {
|
||||
case <-ran:
|
||||
t.Fatal("poll-only worker picked the job up within 2s; the LISTEN test would not prove LISTEN")
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Log("poll-only worker did not pick the job up within 2s")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func waitStatus(t *testing.T, m *Manager, id uint, want Status) Record {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
var rec Record
|
||||
for time.Now().Before(deadline) {
|
||||
var err error
|
||||
rec, err = m.Get(t.Context(), id)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if rec.Status == want {
|
||||
return rec
|
||||
}
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("job %d status = %d, want %d", id, rec.Status, want)
|
||||
return rec
|
||||
}
|
||||
|
||||
func countRows(t *testing.T, gdb *gorm.DB, query string, args ...any) int64 {
|
||||
t.Helper()
|
||||
var n int64
|
||||
if err := gdb.Raw(query, args...).Scan(&n).Error; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// TestDispatchTransactional covers D-02: the row and the River job share the
|
||||
// caller's transaction.
|
||||
func TestDispatchTransactional(t *testing.T) {
|
||||
db, dsn := migratedDB(t)
|
||||
app, gdb := testApp(t, db, dsn, nil)
|
||||
m, err := From(app)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
complete := Job(func(ctx context.Context, a pingArgs) error {
|
||||
id, ok := JobID(ctx)
|
||||
if !ok {
|
||||
return errors.New("no summer job id in ctx")
|
||||
}
|
||||
return m.CompleteJob(ctx, id, nil)
|
||||
})
|
||||
startTestWorker(t, app, WorkerOptions{}, complete)
|
||||
ctx := bouncer.WithUser(t.Context(), &bouncer.Principal{ID: 7})
|
||||
|
||||
t.Run("commit", func(t *testing.T) {
|
||||
var id uint
|
||||
err := gdb.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
var err error
|
||||
id, err = m.Dispatch(ctx, tx, pingArgs{Tag: "commit"}, DispatchOpts{Label: "Import albums", Count: 5})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var rec Record
|
||||
if err := tx.Where("id = ?", id).Take(&rec).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
if rec.Status != StatusInProgress {
|
||||
t.Errorf("status inside tx = %d, want %d", rec.Status, StatusInProgress)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := waitStatus(t, m, id, StatusComplete)
|
||||
if rec.Progress != 5 || rec.ProgressMax != 5 {
|
||||
t.Fatalf("progress = %d/%d, want 5/5", rec.Progress, rec.ProgressMax)
|
||||
}
|
||||
if rec.Label != "Import albums" || rec.Metadata != `""` {
|
||||
t.Fatalf("label/metadata = %q/%q", rec.Label, rec.Metadata)
|
||||
}
|
||||
if rec.UserID == nil || *rec.UserID != 7 || rec.IsAdmin {
|
||||
t.Fatalf("user_id/is_admin = %v/%v, want 7/false", rec.UserID, rec.IsAdmin)
|
||||
}
|
||||
if rec.RiverJobID == nil || *rec.RiverJobID <= 0 {
|
||||
t.Fatalf("river_job_id = %v", rec.RiverJobID)
|
||||
}
|
||||
if rec.CreatedAt == nil || rec.UpdatedAt == nil {
|
||||
t.Fatal("timestamps not set")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("own_transaction", func(t *testing.T) {
|
||||
id, err := m.Dispatch(ctx, gdb, pingArgs{Tag: "own"}, DispatchOpts{Label: "Match albums"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := waitStatus(t, m, id, StatusComplete)
|
||||
if rec.Progress != 0 {
|
||||
t.Fatalf("progress = %d, want progress_max 0", rec.Progress)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("rollback", func(t *testing.T) {
|
||||
before := countRows(t, gdb, `SELECT count(*) FROM river_job`)
|
||||
rollback := errors.New("rollback")
|
||||
err := gdb.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
if _, err := m.Dispatch(ctx, tx, pingArgs{Tag: "rollback"}, DispatchOpts{Label: "Rolled back"}); err != nil {
|
||||
return err
|
||||
}
|
||||
return rollback
|
||||
})
|
||||
if !errors.Is(err, rollback) {
|
||||
t.Fatalf("transaction err = %v", err)
|
||||
}
|
||||
if n := countRows(t, gdb, `SELECT count(*) FROM summer_jobs WHERE label = ?`, "Rolled back"); n != 0 {
|
||||
t.Fatalf("summer_jobs rows after rollback = %d", n)
|
||||
}
|
||||
if after := countRows(t, gdb, `SELECT count(*) FROM river_job`); after != before {
|
||||
t.Fatalf("river_job rows %d -> %d after rollback", before, after)
|
||||
}
|
||||
})
|
||||
}
|
||||
178
modules/conga/postgres_test.go
Normal file
178
modules/conga/postgres_test.go
Normal file
@@ -0,0 +1,178 @@
|
||||
package conga
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.golem15.com/golem15/summercms/modules/backpack"
|
||||
"git.golem15.com/golem15/summercms/modules/compass"
|
||||
"git.golem15.com/golem15/summercms/modules/lagoon"
|
||||
_ "github.com/jackc/pgx/v5/stdlib"
|
||||
"github.com/testcontainers/testcontainers-go"
|
||||
"github.com/testcontainers/testcontainers-go/modules/postgres"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
var (
|
||||
congaPG *postgres.PostgresContainer
|
||||
congaSQL *sql.DB
|
||||
congaDSN string
|
||||
congaPGErr error
|
||||
dbSeq atomic.Int64
|
||||
)
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
code := 1
|
||||
if !testShort() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||
congaPGErr = startCongaPostgres(ctx)
|
||||
cancel()
|
||||
if congaPGErr != nil {
|
||||
fmt.Fprintf(os.Stderr, "conga: testcontainers postgres: %v\n", congaPGErr)
|
||||
stopCongaPostgres()
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
code = m.Run()
|
||||
stopCongaPostgres()
|
||||
os.Exit(code)
|
||||
}
|
||||
|
||||
func testShort() bool {
|
||||
for _, a := range os.Args {
|
||||
if a == "-test.short" {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func startCongaPostgres(ctx context.Context) error {
|
||||
ctr, err := postgres.Run(ctx,
|
||||
"postgres:16-alpine",
|
||||
postgres.WithDatabase("conga"),
|
||||
postgres.WithUsername("conga"),
|
||||
postgres.WithPassword("conga"),
|
||||
postgres.BasicWaitStrategies(),
|
||||
testcontainers.WithEnv(map[string]string{
|
||||
"POSTGRES_INITDB_ARGS": "--locale-provider=icu --icu-locale=pl-PL --encoding=UTF8",
|
||||
}),
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
congaPG = ctr
|
||||
dsn, err := ctr.ConnectionString(ctx, "sslmode=disable")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
db, err := sql.Open("pgx", dsn)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := db.PingContext(ctx); err != nil {
|
||||
_ = db.Close()
|
||||
return err
|
||||
}
|
||||
congaSQL = db
|
||||
congaDSN = dsn
|
||||
return nil
|
||||
}
|
||||
|
||||
func stopCongaPostgres() {
|
||||
if congaSQL != nil {
|
||||
_ = congaSQL.Close()
|
||||
}
|
||||
if congaPG != nil {
|
||||
_ = testcontainers.TerminateContainer(congaPG)
|
||||
}
|
||||
}
|
||||
|
||||
func adminDB(t *testing.T) *sql.DB {
|
||||
t.Helper()
|
||||
if testing.Short() {
|
||||
t.Skip("requires testcontainers postgres")
|
||||
}
|
||||
if congaPGErr != nil {
|
||||
t.Fatalf("postgres unavailable: %v", congaPGErr)
|
||||
}
|
||||
if congaSQL == nil {
|
||||
t.Fatal("postgres unavailable: container was not started")
|
||||
}
|
||||
return congaSQL
|
||||
}
|
||||
|
||||
// migratedDB returns a dedicated ICU pl-PL database migrated with
|
||||
// lagoon.Migrate (River v7 and summer_jobs) plus its DSN.
|
||||
func migratedDB(t *testing.T) (*sql.DB, string) {
|
||||
t.Helper()
|
||||
admin := adminDB(t)
|
||||
ctx := t.Context()
|
||||
name := fmt.Sprintf("conga_%d", dbSeq.Add(1))
|
||||
if _, err := admin.ExecContext(ctx, `CREATE DATABASE `+name+` TEMPLATE template0 ENCODING 'UTF8' LOCALE_PROVIDER icu ICU_LOCALE 'pl-PL'`); err != nil && !strings.Contains(err.Error(), "already exists") {
|
||||
t.Fatalf("create %s: %v", name, err)
|
||||
}
|
||||
dsn, err := dsnWithDB(congaDSN, name)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
db, err := sql.Open("pgx", dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
_ = db.Close()
|
||||
_, _ = admin.ExecContext(context.Background(), `DROP DATABASE IF EXISTS `+name+` WITH (FORCE)`)
|
||||
})
|
||||
gdb, err := lagoon.Use(ctx, db)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := lagoon.Migrate(gdb, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return db, dsn
|
||||
}
|
||||
|
||||
func dsnWithDB(dsn, name string) (string, error) {
|
||||
u, err := url.Parse(dsn)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
u.Path = "/" + name
|
||||
return u.String(), nil
|
||||
}
|
||||
|
||||
// testApp returns an app whose config names dsn and which has published db
|
||||
// and a GORM handle on it. Extra config keys are set as overrides.
|
||||
func testApp(t *testing.T, db *sql.DB, dsn string, extra map[string]any) (*backpack.App, *gorm.DB) {
|
||||
t.Helper()
|
||||
cfg, err := compass.Open(compass.Options{Dir: t.TempDir(), Env: "testing", Environ: []string{}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := cfg.Set("database.dsn", dsn); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for k, v := range extra {
|
||||
if err := cfg.Set(k, v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
app := backpack.New(cfg)
|
||||
gdb, err := lagoon.Use(t.Context(), db)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := lagoon.Publish(app, db, gdb); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return app, gdb
|
||||
}
|
||||
45
modules/conga/record.go
Normal file
45
modules/conga/record.go
Normal file
@@ -0,0 +1,45 @@
|
||||
package conga
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"git.golem15.com/golem15/summercms/modules/lagoon"
|
||||
)
|
||||
|
||||
// Status is the summer_jobs status column. The values match the WinterCMS
|
||||
// apparatus JobStatus constants.
|
||||
type Status int
|
||||
|
||||
const (
|
||||
// StatusInQueue is defined for parity; Dispatch never writes it.
|
||||
StatusInQueue Status = 0
|
||||
// StatusInProgress is written by Dispatch and kept while River retries.
|
||||
StatusInProgress Status = 1
|
||||
// StatusComplete is written by CompleteJob, including skipped work.
|
||||
StatusComplete Status = 2
|
||||
// StatusError is written by FailJob and by the final failed attempt.
|
||||
StatusError Status = 3
|
||||
// StatusStopped is written by CancelJob and StopJob.
|
||||
StatusStopped Status = 4
|
||||
)
|
||||
|
||||
// Record is one summer_jobs row: the queryable outcome of a dispatched job.
|
||||
// River only executes the work; API responses, progress and cancellation
|
||||
// read this row.
|
||||
type Record struct {
|
||||
ID uint `gorm:"column:id;primaryKey"`
|
||||
Label string `gorm:"column:label"`
|
||||
Status Status `gorm:"column:status"`
|
||||
Progress int `gorm:"column:progress"`
|
||||
ProgressMax int `gorm:"column:progress_max"`
|
||||
UserID *uint `gorm:"column:user_id"`
|
||||
IsAdmin bool `gorm:"column:is_admin"`
|
||||
IsCanceled bool `gorm:"column:is_canceled"`
|
||||
Metadata string `gorm:"column:metadata"`
|
||||
RiverJobID *int64 `gorm:"column:river_job_id"`
|
||||
CreatedAt *time.Time `gorm:"column:created_at"`
|
||||
UpdatedAt *time.Time `gorm:"column:updated_at"`
|
||||
}
|
||||
|
||||
// TableName returns lagoon.JobsTable.
|
||||
func (Record) TableName() string { return lagoon.JobsTable }
|
||||
244
modules/conga/worker.go
Normal file
244
modules/conga/worker.go
Normal file
@@ -0,0 +1,244 @@
|
||||
package conga
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"runtime/debug"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.golem15.com/golem15/summercms/modules/backpack"
|
||||
"git.golem15.com/golem15/summercms/modules/lagoon"
|
||||
"git.golem15.com/golem15/summercms/modules/pact"
|
||||
"git.golem15.com/golem15/summercms/modules/party"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/riverqueue/river"
|
||||
"github.com/riverqueue/river/riverdriver/riverdatabasesql"
|
||||
"github.com/riverqueue/river/rivertype"
|
||||
)
|
||||
|
||||
// WorkerOptions selects what a worker runs.
|
||||
type WorkerOptions struct {
|
||||
// Queues limits the worker to these queues; nil means every known queue.
|
||||
Queues []string
|
||||
|
||||
// pollInterval overrides River's FetchPollInterval (tests only).
|
||||
pollInterval time.Duration
|
||||
// pollOnly builds a driver without the LISTEN pool (tests only).
|
||||
pollOnly bool
|
||||
}
|
||||
|
||||
// Worker is a running River work client. Stop it with Stop.
|
||||
type Worker struct {
|
||||
m *Manager
|
||||
client *river.Client[*sql.Tx]
|
||||
listener *pgxpool.Pool
|
||||
queues []string
|
||||
}
|
||||
|
||||
// Queues returns the sorted queue names the worker runs.
|
||||
func (w *Worker) Queues() []string {
|
||||
if w == nil {
|
||||
return nil
|
||||
}
|
||||
return append([]string(nil), w.queues...)
|
||||
}
|
||||
|
||||
// StartWorker registers every pact.HasJobs job of plugins and starts one
|
||||
// River client on the shared *sql.DB. Its driver is
|
||||
// riverdatabasesql.NewWithPgxListener: every query runs on the shared pool
|
||||
// and only Postgres LISTEN runs on a dedicated pgx pool (MaxConns 1) opened
|
||||
// from database.dsn, so jobs committed by any process are picked up without
|
||||
// waiting for the poll interval. The worker keeps running after ctx is
|
||||
// cancelled; stop it with Worker.Stop.
|
||||
func StartWorker(ctx context.Context, app *backpack.App, plugins []party.Plugin, o WorkerOptions) (*Worker, error) {
|
||||
m, err := From(app)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, p := range plugins {
|
||||
hj, ok := p.(pact.HasJobs)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
for _, j := range hj.Jobs() {
|
||||
if err := m.Register(j); err != nil {
|
||||
return nil, fmt.Errorf("conga: plugin %s: %w", p.ID(), err)
|
||||
}
|
||||
}
|
||||
}
|
||||
sqlDB, ok := app.Lookup[*sql.DB]()
|
||||
if !ok || sqlDB == nil {
|
||||
return nil, ErrNoDatabase
|
||||
}
|
||||
s := settingsFromApp(app)
|
||||
log := loggerFromApp(app)
|
||||
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if m.worker != nil {
|
||||
return nil, fmt.Errorf("conga: a worker is already running")
|
||||
}
|
||||
known := knownQueues(s, m.jobs)
|
||||
queues, err := selectQueues(known, o.Queues)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
workers := river.NewWorkers()
|
||||
for _, j := range m.jobs {
|
||||
if err := j.register(workers, m); err != nil {
|
||||
return nil, fmt.Errorf("conga: register %s: %w", j.kind(), err)
|
||||
}
|
||||
}
|
||||
cfg := baseConfig(s, log)
|
||||
cfg.Workers = workers
|
||||
cfg.Queues = map[string]river.QueueConfig{}
|
||||
for name, n := range queues {
|
||||
cfg.Queues[name] = river.QueueConfig{MaxWorkers: n}
|
||||
}
|
||||
if o.pollInterval > 0 {
|
||||
cfg.FetchPollInterval = o.pollInterval
|
||||
}
|
||||
|
||||
var listener *pgxpool.Pool
|
||||
var driver *riverdatabasesql.Driver
|
||||
if o.pollOnly {
|
||||
driver = riverdatabasesql.New(sqlDB)
|
||||
} else {
|
||||
listener, err = listenerPool(ctx, app)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
driver = riverdatabasesql.NewWithPgxListener(sqlDB, listener)
|
||||
}
|
||||
client, err := river.NewClient(driver, cfg)
|
||||
if err != nil {
|
||||
closePool(listener)
|
||||
return nil, fmt.Errorf("conga: river client: %w", err)
|
||||
}
|
||||
if err := client.Start(context.WithoutCancel(ctx)); err != nil {
|
||||
closePool(listener)
|
||||
return nil, fmt.Errorf("conga: start worker: %w", err)
|
||||
}
|
||||
m.worker = client
|
||||
return &Worker{m: m, client: client, listener: listener, queues: sortedQueueNames(queues)}, nil
|
||||
}
|
||||
|
||||
// Stop stops fetching jobs and waits for running jobs to finish. When ctx
|
||||
// ends first, running jobs are cancelled. The listener pool is closed.
|
||||
func (w *Worker) Stop(ctx context.Context) error {
|
||||
if w == nil || w.client == nil {
|
||||
return nil
|
||||
}
|
||||
err := w.client.Stop(ctx)
|
||||
if err != nil && ctx.Err() != nil {
|
||||
hardCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
|
||||
err = w.client.StopAndCancel(hardCtx)
|
||||
cancel()
|
||||
}
|
||||
closePool(w.listener)
|
||||
w.m.mu.Lock()
|
||||
if w.m.worker == w.client {
|
||||
w.m.worker = nil
|
||||
}
|
||||
w.m.mu.Unlock()
|
||||
if err != nil {
|
||||
return fmt.Errorf("conga: stop worker: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func listenerPool(ctx context.Context, app *backpack.App) (*pgxpool.Pool, error) {
|
||||
dsn := ""
|
||||
if app != nil {
|
||||
dsn = lagoon.DSN(app.Config)
|
||||
}
|
||||
if dsn == "" {
|
||||
return nil, fmt.Errorf("conga: database.dsn is empty; the worker's LISTEN pool needs it")
|
||||
}
|
||||
cfg, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("conga: listener pool config: %w", err)
|
||||
}
|
||||
cfg.MaxConns = 1
|
||||
cfg.MinConns = 0
|
||||
pool, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("conga: listener pool: %w", err)
|
||||
}
|
||||
return pool, nil
|
||||
}
|
||||
|
||||
func closePool(p *pgxpool.Pool) {
|
||||
if p != nil {
|
||||
p.Close()
|
||||
}
|
||||
}
|
||||
|
||||
func selectQueues(known map[string]int, want []string) (map[string]int, error) {
|
||||
if len(want) == 0 {
|
||||
return known, nil
|
||||
}
|
||||
out := map[string]int{}
|
||||
for _, name := range want {
|
||||
name = strings.TrimSpace(name)
|
||||
n, ok := known[name]
|
||||
if !ok {
|
||||
names := sortedQueueNames(known)
|
||||
return nil, fmt.Errorf("%w %q (known queues: %s)", ErrUnknownQueue, name, strings.Join(names, ", "))
|
||||
}
|
||||
out[name] = n
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// runAttempt runs one River attempt of a job and applies the summer_jobs
|
||||
// rules: the row id goes into ctx; a panic becomes an error; an error on a
|
||||
// STOPPED or canceled row cancels the River job; an error on the final
|
||||
// attempt sets StatusError with the error text under metadata "error". An
|
||||
// error on an earlier attempt leaves the row as it is so River can retry.
|
||||
func (m *Manager) runAttempt(ctx context.Context, row *rivertype.JobRow, fn func(context.Context) error) error {
|
||||
id, hasRow := summerJobID(row)
|
||||
if hasRow {
|
||||
ctx = withJobID(ctx, id)
|
||||
}
|
||||
err := m.call(ctx, row, fn)
|
||||
if err == nil || !hasRow {
|
||||
return err
|
||||
}
|
||||
dbCtx := context.WithoutCancel(ctx)
|
||||
if rec, gerr := m.Get(dbCtx, id); gerr == nil && (rec.Status == StatusStopped || rec.IsCanceled) {
|
||||
return river.JobCancel(err)
|
||||
}
|
||||
if row.Attempt >= row.MaxAttempts {
|
||||
if ferr := m.failWithError(dbCtx, id, err); ferr != nil {
|
||||
loggerFromApp(m.app).Error("conga: record job failure", "job_id", id, "kind", row.Kind, "error", ferr)
|
||||
}
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
func (m *Manager) call(ctx context.Context, row *rivertype.JobRow, fn func(context.Context) error) (err error) {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
loggerFromApp(m.app).Error("conga: job panicked", "kind", row.Kind, "river_job_id", row.ID, "panic", fmt.Sprint(r), "stack", string(debug.Stack()))
|
||||
err = fmt.Errorf("conga: job %s panicked: %v", row.Kind, r)
|
||||
}
|
||||
}()
|
||||
return fn(ctx)
|
||||
}
|
||||
|
||||
func summerJobID(row *rivertype.JobRow) (uint, bool) {
|
||||
if row == nil || len(row.Metadata) == 0 {
|
||||
return 0, false
|
||||
}
|
||||
var meta struct {
|
||||
SummerJobID uint `json:"summer_job_id"`
|
||||
}
|
||||
if err := json.Unmarshal(row.Metadata, &meta); err != nil || meta.SummerJobID == 0 {
|
||||
return 0, false
|
||||
}
|
||||
return meta.SummerJobID, true
|
||||
}
|
||||
@@ -14,7 +14,7 @@ Postgres data layer: the shared GORM connection, per-plugin migrations, model he
|
||||
|
||||
- One shared pool: `lagoon.Open`, `lagoon.Use` and `lagoon.OpenFromApp` return a `*sql.DB` and a `*gorm.DB` built on that same pool; `lagoon.Publish` makes both available on the `backpack.App`.
|
||||
- Database check at connect time: `lagoon.CheckLocale` refuses a database whose default collation is not the ICU `pl-PL` locale, so ordering matches the database default without per-query `COLLATE`.
|
||||
- Per-plugin migrations: `lagoon.Migrate` runs the framework's `system_files` set (`attach.Migrations`) and backend admin identity set (`lagoon.BackendAdminMigrations`), then every `pact.HasMigrations` set in plugin activation order, each in its own `summer_migrations_<plugin_id>` history table (`lagoon.HistoryTableName`). `lagoon.RollbackLast` and `lagoon.Status` cover rollback and history.
|
||||
- Per-plugin migrations: `lagoon.Migrate` runs the framework's `system_files` set (`attach.Migrations`), backend admin identity set (`lagoon.BackendAdminMigrations`) and job-queue set (`lagoon.QueueMigrations`: River's schema pinned at `lagoon.RiverSchemaVersion`, then the `lagoon.JobsTable` record table, under the `lagoon.QueueHistoryID` history), then every `pact.HasMigrations` set in plugin activation order, each in its own `summer_migrations_<plugin_id>` history table (`lagoon.HistoryTableName`). `lagoon.RollbackLast` and `lagoon.Status` cover rollback and history.
|
||||
- Mass assignment: `lagoon.Fill` copies only allow-listed keys onto a model by GORM column name and silently drops the rest, logging each dropped key once outside production. A `json.Number` (from a decoder using `UseNumber`) fills integer, unsigned and float fields. A value that does not fit its column (a fraction, an exponent or an overflow for an integer field, or a value of the wrong type) is a `lagoon.FillTypeError` naming the key, so a caller can answer it as a validation failure on that field. `lagoon.HasFillable` and `lagoon.HasHidden` are the Go forms of `$fillable` and `$hidden`.
|
||||
- Validation: `lagoon.Validate` accepts Laravel-style rule strings (`required`, `nullable`, `integer`, `numeric`, `between`, `min`, `max`, `in`, `unique`, `boolean`, `email`, `confirmed`, `different`, `mimes`) and returns a field-to-messages map, translated through phrasebook when a translator is given. Unknown rule tokens are an error.
|
||||
- Safe ordering: `lagoon.OrderBy` appends an ORDER BY only for an allow-listed column and an `asc` or `desc` direction.
|
||||
@@ -95,6 +95,10 @@ func (p *Plugin) Migrations() []*gormigrate.Migration {
|
||||
| `lagoon.Status` | Lists applied migration IDs per plugin as `lagoon.StatusRow` values. |
|
||||
| `lagoon.HistoryTableName` | Returns the gormigrate history table for a plugin ID. |
|
||||
| `lagoon.BackendAdminMigrations` | Creates the backend user, role and admin token blacklist tables and seeds the system roles. |
|
||||
| `lagoon.QueueMigrations` | Migrates River's schema to `lagoon.RiverSchemaVersion` and creates the `summer_jobs` job record table. |
|
||||
| `lagoon.QueueHistoryID` | History id of the job-queue set, `summercms.conga`. |
|
||||
| `lagoon.JobsTable` | Name of the job record table, `summer_jobs`. |
|
||||
| `lagoon.RiverSchemaVersion` | The pinned River schema version, 7. |
|
||||
| `lagoon.RuntimeCommands` | Returns the migrate, migrate:rollback, migrate:status and key:generate commands. |
|
||||
| `lagoon.KeyGenerateCommand` | Returns the key:generate command on its own. |
|
||||
| `lagoon.LoadAppKey` | Decodes `app.key` and `app.previous_keys`. |
|
||||
@@ -155,7 +159,7 @@ storage:
|
||||
## Dependencies
|
||||
|
||||
- SummerCMS modules: [backpack](../backpack/README.md), [bonfire](../bonfire/README.md), [compass](../compass/README.md), [pact](../pact/README.md), [party](../party/README.md), [phrasebook](../phrasebook/README.md) (validation messages).
|
||||
- Third-party: `gorm.io/gorm`, `gorm.io/driver/postgres`, `github.com/jackc/pgx/v5` (stdlib driver), `github.com/go-gormigrate/gormigrate/v2`, `github.com/go-playground/validator/v10`.
|
||||
- Third-party: `gorm.io/gorm`, `gorm.io/driver/postgres`, `github.com/jackc/pgx/v5` (stdlib driver), `github.com/go-gormigrate/gormigrate/v2`, `github.com/go-playground/validator/v10`, `github.com/riverqueue/river` (its `rivermigrate` and `riverdriver/riverdatabasesql` packages, for the River schema in `lagoon.QueueMigrations`).
|
||||
- Third-party, `attach` only: `gocloud.dev/blob` (file and memory drivers), `github.com/disintegration/imaging` (thumbnails).
|
||||
- Standard library: `database/sql`, `crypto/aes`, `crypto/cipher`, `crypto/hkdf`, `log/slog`, `image`, among others.
|
||||
|
||||
|
||||
@@ -57,8 +57,9 @@ func migrator(gdb *gorm.DB, pluginID string, migrations []*gormigrate.Migration)
|
||||
}, migrations), nil
|
||||
}
|
||||
|
||||
// Migrate runs the framework-owned system_files and backend-admin sets
|
||||
// first, then each plugin's HasMigrations set in party.Activate order.
|
||||
// Migrate runs the framework-owned system_files, backend-admin and job-queue
|
||||
// (QueueMigrations) sets first, then each plugin's HasMigrations set in
|
||||
// party.Activate order.
|
||||
func Migrate(gdb *gorm.DB, plugins []party.Plugin) error {
|
||||
if gdb == nil {
|
||||
return fmt.Errorf("lagoon: gorm db is nil")
|
||||
@@ -77,6 +78,17 @@ func Migrate(gdb *gorm.DB, plugins []party.Plugin) error {
|
||||
if err := admin.Migrate(); err != nil {
|
||||
return fmt.Errorf("lagoon: migrate backend admin: %w", err)
|
||||
}
|
||||
sqlDB, err := gdb.DB()
|
||||
if err != nil {
|
||||
return fmt.Errorf("lagoon: migrate queue: %w", err)
|
||||
}
|
||||
queue, err := migrator(gdb, QueueHistoryID, QueueMigrations(sqlDB))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := queue.Migrate(); err != nil {
|
||||
return fmt.Errorf("lagoon: migrate queue: %w", err)
|
||||
}
|
||||
for _, p := range plugins {
|
||||
hm, ok := p.(pact.HasMigrations)
|
||||
if !ok {
|
||||
|
||||
86
modules/lagoon/queue_migrations.go
Normal file
86
modules/lagoon/queue_migrations.go
Normal file
@@ -0,0 +1,86 @@
|
||||
package lagoon
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
|
||||
"github.com/go-gormigrate/gormigrate/v2"
|
||||
"github.com/riverqueue/river/riverdriver/riverdatabasesql"
|
||||
"github.com/riverqueue/river/rivermigrate"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
const (
|
||||
// QueueHistoryID is the migration history id of the framework job-queue
|
||||
// set. Its history table is summer_migrations_summercms_conga.
|
||||
QueueHistoryID = "summercms.conga"
|
||||
// JobsTable is the framework job record table. Its columns match the
|
||||
// WinterCMS apparatus jobs table plus the internal river_job_id link.
|
||||
JobsTable = "summer_jobs"
|
||||
// RiverSchemaVersion pins the River schema the framework migrates to, so a
|
||||
// River upgrade never applies new DDL without a new framework migration.
|
||||
RiverSchemaVersion = 7
|
||||
)
|
||||
|
||||
// QueueMigrations returns the framework job-queue migration set: River's
|
||||
// schema pinned at RiverSchemaVersion, then the summer_jobs record table.
|
||||
// River's migrator runs each of its own steps in a separate transaction on
|
||||
// sqlDB, because some River migrations cannot share one transaction.
|
||||
func QueueMigrations(sqlDB *sql.DB) []*gormigrate.Migration {
|
||||
return []*gormigrate.Migration{
|
||||
{
|
||||
ID: "202609290001_river_schema",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
return migrateRiver(txContext(tx), sqlDB, rivermigrate.DirectionUp, &rivermigrate.MigrateOpts{TargetVersion: RiverSchemaVersion})
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
// TargetVersion -1 applies every down step, removing River's schema.
|
||||
return migrateRiver(txContext(tx), sqlDB, rivermigrate.DirectionDown, &rivermigrate.MigrateOpts{TargetVersion: -1})
|
||||
},
|
||||
},
|
||||
{
|
||||
ID: "202609290002_summer_jobs",
|
||||
Migrate: func(tx *gorm.DB) error {
|
||||
return tx.Exec(`CREATE TABLE IF NOT EXISTS summer_jobs (
|
||||
id SERIAL PRIMARY KEY,
|
||||
label VARCHAR(255) NOT NULL,
|
||||
status INTEGER NOT NULL DEFAULT 0,
|
||||
progress INTEGER NOT NULL DEFAULT 0,
|
||||
progress_max INTEGER NOT NULL DEFAULT 0,
|
||||
user_id INTEGER NULL,
|
||||
is_admin BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
is_canceled BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
metadata TEXT NOT NULL,
|
||||
river_job_id BIGINT NULL,
|
||||
created_at TIMESTAMPTZ NULL,
|
||||
updated_at TIMESTAMPTZ NULL
|
||||
)`).Error
|
||||
},
|
||||
Rollback: func(tx *gorm.DB) error {
|
||||
return tx.Exec(`DROP TABLE IF EXISTS summer_jobs`).Error
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func migrateRiver(ctx context.Context, sqlDB *sql.DB, dir rivermigrate.Direction, opts *rivermigrate.MigrateOpts) error {
|
||||
if sqlDB == nil {
|
||||
return fmt.Errorf("lagoon: river migrate: sql db is nil")
|
||||
}
|
||||
m, err := rivermigrate.New(riverdatabasesql.New(sqlDB), nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("lagoon: river migrate: %w", err)
|
||||
}
|
||||
if _, err := m.Migrate(ctx, dir, opts); err != nil {
|
||||
return fmt.Errorf("lagoon: river migrate %s: %w", dir, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func txContext(tx *gorm.DB) context.Context {
|
||||
if tx != nil && tx.Statement != nil && tx.Statement.Context != nil {
|
||||
return tx.Statement.Context
|
||||
}
|
||||
return context.Background()
|
||||
}
|
||||
Reference in New Issue
Block a user