golem15/summercms
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(11-01): run job workers in serve and queue:work, add queue:clear

Browse Source 
- Manager gains the apparatus JobManager surface: StartJob, UpdateJobState,
  UpdateMetadata, FailJob, CancelJob (is_canceled + STOPPED + River JobCancel),
  StopJob (STOPPED only), CheckIfCanceled and GetMetadata, all raw column
  writes so updated_at is untouched
- serve starts the in-process worker unless queue.work_in_serve is false and
  stops it on shutdown; an app without jobs gets an idle worker
- queue:work runs a foreground worker with repeatable --queue filters;
  queue:clear deletes available, scheduled and retryable jobs of one queue
- the generated main appends conga.RuntimeCommands; summer delegates
  queue:work and queue:clear; make:job scaffolds a conga.Job
This commit is contained in:
Jakub Zych
2026-09-29 15:20:14 +02:00
parent 05ba88a54a
commit b319e7cc61
15 changed files with 939 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
modules/conga/README.md
View File

@@ -15,9 +15,12 @@ Every River client runs on the one `*sql.DB` pool that `lagoon` opens. A worker
- River-free job declarations: `conga.Job` turns `func(ctx context.Context, args T) error` into a `pact.Job`; `conga.OnQueue`, `conga.MaxAttempts` and `conga.Timeout` set per-job defaults. A `pact.Job` not built by `conga.Job` is rejected with `conga.ErrNotCongaJob`.
- Transactional dispatch: `conga.Manager.Dispatch` inserts the `summer_jobs` row with `conga.StatusInProgress`, the principal's user id and admin flag, `progress_max` from `conga.DispatchOpts.Count` and JSON metadata, then enqueues the River job in the same transaction. It opens a transaction itself when the caller has none.
- Plain enqueue: `conga.Manager.Enqueue` inserts a River job without a record row, inside the caller's transaction when there is one.
- The record row: `conga.Record` maps `summer_jobs`; `conga.Status` holds the WinterCMS status values (`conga.StatusInQueue`, `conga.StatusInProgress`, `conga.StatusComplete`, `conga.StatusError`, `conga.StatusStopped`). `conga.Manager.CompleteJob` completes a row, `conga.Manager.Get` reads one, and `conga.JobID` gives a running job its own row id.
- Outcome rules in the worker: an error on an attempt before the last leaves the row in progress so River can retry; the final failed attempt, or a recovered panic on it, sets `conga.StatusError` with the error text under the metadata key `error`. Skipped work is recorded as complete with `{"skipped": true}` metadata.
- Workers: `conga.StartWorker` registers every plugin job and starts one River client; `conga.WorkerOptions.Queues` limits it to some queues, and an unknown queue is `conga.ErrUnknownQueue` listing the known ones. `conga.Worker.Stop` stops it gracefully and cancels running jobs when its context ends.
- The record row: `conga.Record` maps `summer_jobs`; `conga.Status` holds the WinterCMS status values (`conga.StatusInQueue`, `conga.StatusInProgress`, `conga.StatusComplete`, `conga.StatusError`, `conga.StatusStopped`). `conga.JobID` gives a running job its own row id.
- The WinterCMS job manager operations with the same semantics: `conga.Manager.StartJob`, `conga.Manager.UpdateJobState`, `conga.Manager.UpdateMetadata`, `conga.Manager.CompleteJob`, `conga.Manager.FailJob`, `conga.Manager.CheckIfCanceled` and `conga.Manager.GetMetadata`. Updates are raw column writes, so `updated_at` changes only on dispatch and `conga.Manager.StartJob`.
- Cancellation in two parts: `conga.Manager.CancelJob` is the outside cancel (sets `is_canceled` and `conga.StatusStopped`, then cancels the River job, so a queued job never starts and a running job's context is cancelled); `conga.Manager.StopJob` is what a job calls on its own row after `conga.Manager.CheckIfCanceled` reports true (status only, the WinterCMS `cancelJob`).
- Outcome rules in the worker: an error on an attempt before the last leaves the row in progress so River can retry; the final failed attempt, or a recovered panic on it, sets `conga.StatusError` with the error text under the metadata key `error`; an error on a row that was stopped or cancelled cancels the River job instead. A job that returns nil without completing its row leaves it as it is. Skipped work is recorded as complete with `{"skipped": true}` metadata.
- Workers: `conga.StartWorker` registers every plugin job and starts one River client; `conga.WorkerOptions.Queues` limits it to some queues, and an unknown queue is `conga.ErrUnknownQueue` listing the known ones. `conga.Worker.Stop` stops it gracefully and cancels running jobs when its context ends. `conga.StartServeWorker` is the variant the `serve` command uses: it starts nothing when `queue.work_in_serve` is false. An app without jobs still gets a worker that starts and idles.
- Commands: `conga.RuntimeCommands` adds `queue:work` and `queue:clear` to the application binary.
## Usage
@@ -66,6 +69,31 @@ func startImport(ctx context.Context, app *backpack.App, gdb *gorm.DB, file stri
}
```
A long job reports progress and honours cancellation between items:
```go
func importPosts(ctx context.Context, m *conga.Manager, files []string) error {
id, _ := conga.JobID(ctx)
if err := m.StartJob(ctx, id, len(files)); err != nil {
return err
}
for i, f := range files {
if canceled, err := m.CheckIfCanceled(ctx, id); err != nil || canceled {
if err != nil {
return err
}
return m.StopJob(ctx, id, nil)
}
// ... import f ...
_ = f
if err := m.UpdateJobState(ctx, id, i+1, nil); err != nil {
return err
}
}
return m.CompleteJob(ctx, id, map[string]any{"imported": len(files)})
}
```
A worker runs in the same process or in a separate one:
```go
@@ -85,7 +113,15 @@ defer w.Stop(context.Background())
| `conga.Manager.Register` | Registers jobs built by `conga.Job`; closed while a worker runs (`conga.ErrRegistrationClosed`). |
| `conga.Manager.Dispatch` | Writes the record row and enqueues the River job in one transaction; returns the row id. |
| `conga.Manager.Enqueue` | Enqueues a River job without a record row. |
| `conga.Manager.CompleteJob` | Sets `conga.StatusComplete` and progress to `progress_max`; replaces metadata when given. |
| `conga.Manager.StartJob` | Sets progress to 0, `progress_max` to the total and `updated_at` to now. |
| `conga.Manager.UpdateJobState` | Sets progress; replaces metadata when given. |
| `conga.Manager.UpdateMetadata` | Replaces metadata. |
| `conga.Manager.CompleteJob` | Sets `conga.StatusComplete` and progress to `progress_max`; replaces metadata when given. Skipped work passes `{"skipped": true}`. |
| `conga.Manager.FailJob` | Sets `conga.StatusError`; replaces metadata when given. |
| `conga.Manager.CancelJob` | Sets `is_canceled` and `conga.StatusStopped` and cancels the River job. |
| `conga.Manager.StopJob` | Sets `conga.StatusStopped` only; called by a job on its own row. |
| `conga.Manager.CheckIfCanceled` | Reports `is_canceled`. |
| `conga.Manager.GetMetadata` | Decodes metadata; an empty or non-object value is an empty map. |
| `conga.Manager.Get` | Reads one `conga.Record`. |
| `conga.DispatchOpts` | Label, count, metadata, queue, delay and attempt limit of a dispatch. |
| `conga.EnqueueOpts` | Queue, delay and attempt limit of an enqueue. |
@@ -95,7 +131,9 @@ defer w.Stop(context.Background())
| `conga.JobOption` | Per-job option: `conga.OnQueue`, `conga.MaxAttempts`, `conga.Timeout`. |
| `conga.JobID` | Returns the record row id of the job running in a context. |
| `conga.StartWorker` | Registers plugin jobs and starts a River worker client. |
| `conga.StartServeWorker` | The worker of the `serve` command; nil when `queue.work_in_serve` is false. |
| `conga.WorkerOptions` | Selects the queues a worker runs. |
| `conga.RuntimeCommands` | Returns the `queue:work` and `queue:clear` commands. |
| `conga.Worker` | A running worker; `conga.Worker.Stop` stops it and `conga.Worker.Queues` lists its queues. |
| `conga.ErrNoDatabase` | The app has not published the shared database handles. |
| `conga.ErrNotCongaJob` | A registered `pact.Job` was not built by `conga.Job`. |
@@ -108,12 +146,14 @@ Keys are read from the compass config (`config/queue.yaml`, or `SUMMER_QUEUE__..
| Key | Default | Controls |
|-----|---------|----------|
| `queue.work_in_serve` | `true` | Whether the `serve` command runs the job worker in its own process. Set `false` when a separate `queue:work` process runs the jobs. |
| `queue.max_attempts` | `3` | Attempts per job before its record becomes `conga.StatusError`, unless the job or dispatch sets its own. |
| `queue.job_timeout` | `300` | Per-attempt deadline, in seconds or as a duration string such as `5m`, unless the job sets `conga.Timeout`. |
| `queue.queues.<name>` | `default: 4` | Concurrent workers per queue. The worker runs these queues plus every queue a registered job names plus `default`. |
| `database.dsn` | none (required) | Also opens the worker's single-connection `LISTEN` pool. With PgBouncer, that connection must use session pooling or go straight to Postgres; transaction pooling cannot hold a `LISTEN`. |
```yaml
work_in_serve: true
max_attempts: 3
job_timeout: 300
queues:
@@ -121,6 +161,15 @@ queues:
imports: 1
```
## CLI commands
`conga.RuntimeCommands` adds these commands to the application binary. Both open the database through `lagoon.OpenFromApp`, so they need `database.dsn` and `app.key`.
| Command | Arguments and flags | Description |
|---------|---------------------|-------------|
| `queue:work` | `--queue <name>`, repeatable | Runs a job worker in the foreground on the named queues (default: every known queue) until SIGINT or SIGTERM, then stops it within 10 seconds. An unknown queue is an error that lists the known ones. |
| `queue:clear` | `[queue]` (default `default`) | Deletes the available, scheduled and retryable jobs of one queue in batches until none are left and prints `Cleared N jobs`. Running jobs are never touched. |
## Dependencies
- SummerCMS modules: [backpack](../backpack/README.md), [bouncer](../bouncer/README.md) (the dispatching principal), [lagoon](../lagoon/README.md) (the shared pool, `lagoon.JobsTable` and the migrations that create River's schema and `summer_jobs`), [pact](../pact/README.md), [party](../party/README.md).