docs(11-02): complete scheduler plan

2026-09-29 22:54:28 +02:00
parent 2237a640d2
commit f1077382f5
3 changed files with 311 additions and 10 deletions
--- a/.planning/ROADMAP.md
+++ b/.planning/ROADMAP.md
@@ -505,7 +505,7 @@ Plans:
  4. A channel-namespace authorizer registry re-validates on every subscribe; a broadcastable model interface with bulk-write suppression emits exactly one summary event for a bulk operation.
  5. Typesense sync is scoped by `collection_id` behind a settings kill-switch and degrades gracefully without DB/config.

-**Plans:** 1/7 plans executed
+**Plans:** 2/7 plans executed
 **Research flag:** yes

 Plans:
@@ -514,7 +514,7 @@ Plans:
 - [x] 11-01-PLAN.md — conga core: River v0.47.0 on NewWithPgxListener, summer_jobs + River v7 migrations, job manager, conga.Job[T], worker in serve, queue:work/queue:clear, lagoon OnDatabase and Transaction/AfterCommit seams, fonoteka allow-lists (summercms.go, fonoteka.go)

 **Wave 2** *(blocked on Wave 1 completion)*
- [ ] 11-02-PLAN.md — Scheduler: pact.HasSchedule, Daily/Every, River periodic jobs, bonfire.Call, schedule:run (+ --once), fonoteka prune-notifications entry
+- [x] 11-02-PLAN.md — Scheduler: pact.HasSchedule, Daily/Every, River periodic jobs, bonfire.Call, schedule:run (+ --once), fonoteka prune-notifications entry
 - [ ] 11-03-PLAN.md — Realtime: lighthouse + Centrifugo driver (token route, subscribe proxy, HTTP client, broadcasts with suppression), fonoteka authorizers, Mount, ws-api bucket, Album binding

 **Wave 3** *(blocked on Wave 2 completion)*
@@ -654,7 +654,7 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 →
 | 8. OAuth2.1 authorization server | 10/10 | Complete   | 2026-09-23 |
 | 9. Backend admin authentication and schema pipeline | 12/12 | In Progress|  |
 | 10. Admin Vue SPA | 5/5 | Complete    | 2026-09-27 |
-| 11. Jobs, realtime and search infrastructure | 1/7 | In Progress|  |
+| 11. Jobs, realtime and search infrastructure | 2/7 | In Progress|  |
 | 11.1. SummerCMS documentation for humans and AI agents | 0/TBD | Not started | - |
 | 11.2. Ready to share: summercms.io website and newsletter plugin | 0/TBD | Not started | - |
 | 12. Płytarium API — Collections and Albums | 0/TBD | Not started | - |
--- a/.planning/STATE.md
+++ b/.planning/STATE.md
@@ -4,16 +4,16 @@ milestone: v1.0
 current_phase: 11
 current_phase_name: Jobs, realtime and search infrastructure
 status: executing
-stopped_at: Completed 11-01-PLAN.md
-last_updated: "2026-09-29T13:27:45.169Z"
+stopped_at: Completed 11-02-PLAN.md
+last_updated: "2026-09-29T20:54:27.609Z"
 last_activity: 2026-09-29
 last_activity_desc: Phase 11 execution started
-state_head: 144c54bb2c682a95b37f9c1c68ec2ec6b68b8dbe
+state_head: 2237a640d2400e337d9e6c638a7b9534cd9c77b9
 progress:
  total_phases: 19
  completed_phases: 9
  total_plans: 85
-  completed_plans: 79
+  completed_plans: 80
 milestone_name: milestone
 ---

@@ -29,7 +29,7 @@ See: .planning/PROJECT.md (updated 2026-09-16)
 ## Current Position

 Phase: 11 (Jobs, realtime and search infrastructure) — EXECUTING
-Plan: 2 of 7
+Plan: 3 of 7
 Status: Ready to execute
 Last activity: 2026-09-29 — Phase 11 execution started

@@ -134,6 +134,7 @@ Progress: [██████░░░░] 60%
 | Phase 10.1 P03 | 11min | 3 tasks | 16 files |
 | Phase 10.1 P04 | 42min | 3 tasks | 32 files |
 | Phase 11 P01 | 37 min | 3 tasks | 50 files |
+| Phase 11 P02 | 385 min | 2 tasks | 21 files |

 ## Accumulated Context

@@ -394,6 +395,6 @@ Items acknowledged and carried forward from previous milestone close:

 ## Session Continuity

-Last session: 2026-09-29T13:27:44.754Z
-Stopped at: Completed 11-01-PLAN.md
+Last session: 2026-09-29T20:54:27.162Z
+Stopped at: Completed 11-02-PLAN.md
 Resume file: None
--- a/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-02-SUMMARY.md
+++ b/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-02-SUMMARY.md
@@ -0,0 +1,300 @@
+---
+phase: 11-jobs-realtime-and-search-infrastructure
+plan: 02
+subsystem: infra
+tags: [river, scheduler, periodic-jobs, cli, cron, bonfire]
+
+requires:
+  - phase: 11-jobs-realtime-and-search-infrastructure
+    provides: "11-01 conga worker (StartWorker, Manager, conga.Job, queue settings) on River v0.47.0"
+provides:
+  - "pact.HasSchedule, ScheduledCommand, Cadence with Daily/DailyAt/Every (no River import)"
+  - "bonfire.Call, Catalog, NewCatalog, ErrUnknownCommand for in-process command runs"
+  - "conga.Daily / conga.Every wall-clock river.PeriodicSchedule in app.timezone"
+  - "Every worker carries plugin schedules as River PeriodicJobs; runs go to the scheduled queue (MaxAttempts 1, unique by args within the cadence period)"
+  - "schedule:run (scheduler-only worker) and schedule:run --once (Laravel minute-match for system cron), plus the summer delegate"
+  - "Generated app main publishes bonfire.NewCatalog(commands)"
+  - "fonoteka declares fonoteka:prune-notifications daily in schedule.go"
+affects: [11-03 broadcasts, 11-07 unit tests, 14 domain jobs (prune-notifications command), 15 cutover (cron/process layout)]
+
+actuals:
+  tokens: 13984
+  tasks: 2
+  commits: 2
+plan_head_before: 77b8ff177ac96457d64dbb2cbddc6bcaabbea21c
+plan_head_after: 2237a640d2400e337d9e6c638a7b9534cd9c77b9
+# fonoteka.go (separate repository) received 2 more commits: 37133f9, 94e95de
+
+tech-stack:
+  added: []
+  patterns:
+    - "Plugins declare recurring commands with pact.HasSchedule; conga compiles them into River periodic jobs with ids <plugin id>[<index>]:<command>"
+    - "Scheduled runs call commands in-process through the app's *bonfire.Catalog, published by the generated main"
+    - "A scheduled job runs only when it matches a compiled schedule entry exactly (T-11-09)"
+
+key-files:
+  created:
+    - modules/bonfire/call.go
+    - modules/bonfire/call_test.go
+    - modules/conga/schedule.go
+    - modules/conga/scheduler.go
+    - modules/conga/schedule_test.go
+    - ../fonoteka.go/plugins/golem15/fonoteka/schedule.go
+  modified:
+    - modules/pact/capabilities.go
+    - modules/pact/README.md
+    - modules/bonfire/README.md
+    - modules/conga/conga.go
+    - modules/conga/worker.go
+    - modules/conga/commands.go
+    - modules/conga/listen_test.go
+    - modules/conga/README.md
+    - internal/build/build.go
+    - internal/build/build_test.go
+    - cmd/summer/main.go
+    - cmd/summer/runtime.go
+    - cmd/summer/main_test.go
+    - examples/hello/main.go
+    - ../fonoteka.go/main.go
+
+key-decisions:
+  - "Periodic job ids are <plugin id>[<index>]:<command>, not <plugin id>#<index>:<command>: River's PeriodicJob.ID regex rejects '#'"
+  - "Scheduled runs are unique by args and period (UniqueOpts{ByArgs: true, ByPeriod}); ByPeriod alone would collapse different entries of the shared summer.scheduled_command kind within one period"
+  - "Every(d) requires d >= 1s (River's ByPeriod floor) and 24h % d == 0; DailyAt rejects hours outside 0-23 and minutes outside 0-59"
+  - "schedule:run --once opens the database only when an entry is due, and reuses a published *gorm.DB when there is one; plugin commands look the DB up from the app"
+  - "Every(d) in --once is due every run for d <= 1 minute, otherwise when wall-clock minutes since midnight are a multiple of d"
+
+patterns-established:
+  - "bonfire.Catalog: the final command list is an app service, so non-CLI code can run commands with Call"
+  - "Forged-row defence: jobs whose args name code to execute are checked against a compiled table before running"
+
+requirements-completed: [CLI-04]
+
+coverage:
+  - id: D1
+    description: "pact.HasSchedule with Daily/DailyAt/Every cadences and no River import; conga.Daily/Every compute wall-clock next runs (adjacency, DST, midnight rollover) and invalid cadences are rejected"
+    requirement: CLI-04
+    verification:
+      - kind: unit
+        ref: "modules/conga/schedule_test.go#TestScheduleNext"
+        status: pass
+      - kind: unit
+        ref: "modules/conga/schedule_test.go#TestScheduleEntries"
+        status: pass
+    human_judgment: false
+  - id: D2
+    description: "bonfire.Call and Catalog run a named command in-process with args; unknown names wrap ErrUnknownCommand"
+    requirement: CLI-04
+    verification:
+      - kind: unit
+        ref: "modules/bonfire/call_test.go#TestCall"
+        status: pass
+    human_judgment: false
+  - id: D3
+    description: "A plugin's Every(1s) entry runs through a River periodic job and bonfire.Call inside a worker; an unregistered command is skipped with a Warn log"
+    requirement: CLI-04
+    verification:
+      - kind: integration
+        ref: "modules/conga/schedule_test.go#TestScheduleRunsCommand"
+        status: pass
+    human_judgment: false
+  - id: D4
+    description: "Generated app main publishes bonfire.NewCatalog(commands) after plugin commands and before NewRoot; hello and fonoteka mains regenerated"
+    requirement: CLI-04
+    verification:
+      - kind: unit
+        ref: "internal/build/build_test.go#TestGenerateMainPublishesCommandCatalog"
+        status: pass
+    human_judgment: false
+  - id: D5
+    description: "schedule:run --once minute-match semantics (daily, Every(5m), app.timezone), unknown command warns and others run, first error returned after all ran; schedule:run foreground runs a scheduler-only worker"
+    requirement: CLI-04
+    verification:
+      - kind: unit
+        ref: "modules/conga/schedule_test.go#TestScheduleRunOnce"
+        status: pass
+      - kind: integration
+        ref: "modules/conga/schedule_test.go#TestScheduleRunForeground"
+        status: pass
+      - kind: unit
+        ref: "cmd/summer/main_test.go#TestToolCommandNames"
+        status: pass
+    human_judgment: false
+  - id: D6
+    description: "T-11-09: a scheduled job whose command/args differ from its compiled entry, or whose entry is unknown, is skipped without running anything, including a forged river_job row"
+    requirement: CLI-04
+    verification:
+      - kind: integration
+        ref: "modules/conga/schedule_test.go#TestScheduledEntryMismatchSkipped"
+        status: pass
+    human_judgment: false
+  - id: D7
+    description: "T-11-17: two inserts of one entry inside its period leave one river_job row; a different entry in the same period gets its own row"
+    requirement: CLI-04
+    verification:
+      - kind: integration
+        ref: "modules/conga/schedule_test.go#TestScheduleUniqueByPeriod"
+        status: pass
+    human_judgment: false
+  - id: D8
+    description: "fonoteka declares fonoteka:prune-notifications daily; `fonoteka schedule:run --once` prints the no-commands line and exits 0; --help shows --once"
+    requirement: CLI-04
+    verification:
+      - kind: other
+        ref: "cd ../fonoteka.go && SUMMER_GOLEM15__USER__JWT__SECRET=test-only-cli-secret go run . schedule:run --once"
+        status: pass
+    human_judgment: false
+  - id: D9
+    description: "Concurrency backstop: with several worker processes River leader election means one process enqueues each period"
+    verification: []
+    human_judgment: true
+    rationale: "Multi-process leader election is River's guarantee and is not exercised by a test here (plan marks it verification: backstop)"
+
+duration: 385min
+completed: 2026-09-29
+status: complete
+---
+
+# Phase 11 Plan 02: scheduler on River periodic jobs Summary
+
+**`pact.HasSchedule` with `Daily`/`DailyAt`/`Every` cadences becomes River periodic jobs on wall-clock schedules in `app.timezone`. Each run is a one-attempt job on the `scheduled` queue that calls the command in-process through a `bonfire.Catalog` the generated main publishes. `schedule:run` runs a scheduler process, and `schedule:run --once` serves system cron.**
+
+## Performance
+
+- **Duration:** 385 min wall-clock. The clock ran from 14:09Z to 20:34Z; the active work was much shorter, and most of the span was idle time between tool calls.
+- **Started:** 2026-09-29T14:09:06Z
+- **Completed:** 2026-09-29T20:34:36Z
+- **Tasks:** 2
+- **Files modified:** 21 (19 in summercms.go, 2 in fonoteka.go)
+
+## Accomplishments
+
+- `pact` has the schedule capability: `HasSchedule`, `ScheduledCommand`, the opaque `Cadence` (`IsZero`, `Interval`, `At`) and the `Daily`, `DailyAt` and `Every` constructors. pact still imports no River code.
+- `bonfire.Call` and `bonfire.Catalog` run any registered command in-process with empty stdin. The generated app main publishes `bonfire.NewCatalog(commands)` before building the root. The hello and fonoteka mains were regenerated.
+- `conga.Daily` and `conga.Every` are `river.PeriodicSchedule` implementations:
+  - `Daily` fires at the next wall-clock `hh:mm`. On DST days the runs are 23h or 25h apart but stay at 12:00 local.
+  - `Every` fires at the next multiple of its interval since local midnight and rolls over to the next midnight.
+  - Both fire strictly after an exact boundary.
+- Every worker carries one periodic job per schedule entry, with id `<plugin id>[<index>]:<command>`. Runs go to the `scheduled` queue (`queue.queues.scheduled`, default 1) with MaxAttempts 1 and uniqueness by args within the cadence period.
+- The worker runs a job only when it matches a compiled entry exactly (T-11-09). A missing catalog or an unregistered command is logged at Warn and skipped (user decision 5). Command output is logged line by line with a `command` attribute.
+- `schedule:run` runs a scheduler-only worker until SIGINT or SIGTERM. `schedule:run --once` runs the entries due this minute without River, prints `Running scheduled command: …` or `No scheduled commands are ready to run.`, and returns the first command error. The `summer` delegate forwards `--once`.
+- fonoteka declares `fonoteka:prune-notifications` daily in `schedule.go`. `plugin.go` is untouched, so plan 11-03 can edit it.
+
+## Task Commits
+
+summercms.go:
+1. **Task 1: an interval command runs on schedule inside a worker through bonfire.Call** - `d9f939a` (feat)
+2. **Task 2: schedule:run as a dedicated process and from system cron** - `2237a64` (feat)
+
+fonoteka.go:
+1. **Task 1** - `37133f9` (feat: regenerated main.go publishes the catalog)
+2. **Task 2** - `94e95de` (feat: schedule.go with the daily prune entry)
+
+## Files Created/Modified
+
+- `modules/pact/capabilities.go`, `modules/pact/README.md`: the schedule capability and cadences
+- `modules/bonfire/call.go`, `call_test.go`, `README.md`: `Call`, `Catalog`, `NewCatalog` and `ErrUnknownCommand`
+- `modules/conga/schedule.go`: `Daily`, `Every`, `scheduleFor` and `appLocation`
+- `modules/conga/scheduler.go`: `QueueScheduled`, `ScheduledCommandArgs`, entry compilation, periodic jobs, the built-in job, the T-11-09 check and the log writer
+- `modules/conga/worker.go`, `conga.go`: `StartWorker` sets `PeriodicJobs`, registers the built-in job and adds the `scheduled` queue
+- `modules/conga/commands.go`: `schedule:run [--once]`
+- `modules/conga/schedule_test.go`: seven tests. `listen_test.go`: the serve worker now also lists `scheduled`
+- `internal/build/build.go`, `build_test.go`: the catalog publish in the generated main
+- `cmd/summer/main.go`, `runtime.go`, `main_test.go`: the `schedule:run` delegate
+- `examples/hello/main.go` and `../fonoteka.go/main.go` (regenerated), and `../fonoteka.go/plugins/golem15/fonoteka/schedule.go`
+
+## Decisions Made
+
+See `key-decisions` in the frontmatter.
+
+## TDD Gate Compliance
+
+Task 2 is `tdd="true"`. The tests were written first against a stub `schedule:run` that returned "not implemented".
+
+- **RED:** `TestScheduleRunOnce` failed on its assertions in all 6 subtests. `gsd-tools check tdd-red-evidence` returned `RED_EVIDENCE_OK` (`target_test_failed`). The command was `go test ./modules/conga -run '^(TestScheduleRunOnce|TestScheduledEntryMismatchSkipped|TestScheduleUniqueByPeriod)$' -count=1 -json`, converted to TAP.
+- `TestScheduledEntryMismatchSkipped` and `TestScheduleUniqueByPeriod` passed at RED. Task 1's action required the exact-match check (T-11-09) and the ByPeriod insert options, so both mitigations already existed. To prove the tests are real, I ran a mutation check and restored the code afterwards:
+  - Running the job's own command instead of the compiled entry failed `TestScheduledEntryMismatchSkipped` ("no skip warning").
+  - Dropping `ByArgs` failed `TestScheduleUniqueByPeriod` ("river_job rows = 1, want 2").
+- **Gate violation, flagged:** there is no separate `test(11-02)` RED commit. The project CLAUDE.md requires `go vet` and `go test ./...` to be green at every commit, so the tests and implementation landed together in `2237a64`. Plan 11-01 made the same call.
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 1 - Bug] Periodic job id separator**
+- **Found during:** Task 1 (reading River's `PeriodicJob.validate`)
+- **Issue:** River requires `PeriodicJob.ID` to match `\A[\w][\w\-\[\]<>\/.·:+]+\z`, which has no `#`. The plan's `<plugin id>#<index>:<command>` would have failed every worker start that had a schedule.
+- **Fix:** Ids use `<plugin id>[<index>]:<command>`, for example `golem15.fonoteka[0]:fonoteka:prune-notifications`. Ordering, uniqueness and readability are unchanged.
+- **Files modified:** modules/conga/scheduler.go
+- **Verification:** `TestScheduleEntries` checks the ids. `TestScheduleRunsCommand` starts a worker with them.
+- **Committed in:** d9f939a
+
+**2. [Rule 1 - Bug] ByPeriod alone deduplicates across entries**
+- **Found during:** Task 1 (River `UniqueOpts` docs: without `ByArgs`, uniqueness applies to all jobs of the kind)
+- **Issue:** Every entry shares the kind `summer.scheduled_command`, so `ByPeriod` alone would drop the second entry due in the same period.
+- **Fix:** `UniqueOpts{ByArgs: true, ByPeriod: period}`. The args hold the entry id, so the dedupe is per entry.
+- **Files modified:** modules/conga/scheduler.go
+- **Verification:** `TestScheduleUniqueByPeriod` checks both parts: the same entry twice gives 1 row, and a second entry gives 2 rows. The mutation check confirms the second part catches the bug.
+- **Committed in:** d9f939a
+
+**3. [Rule 2 - Missing validation] Cadence bounds**
+- **Found during:** Task 1
+- **Issue:** River rejects `ByPeriod` under 1s. `DailyAt(25, 0)` would silently normalize to 01:00 the next day.
+- **Fix:** `scheduleFor` also rejects intervals under 1s and daily times outside 00:00-23:59, with the plugin id and entry index in the error.
+- **Files modified:** modules/conga/schedule.go
+- **Verification:** `TestScheduleNext` covers the invalid cadences.
+- **Committed in:** d9f939a
+
+**4. [Rule 3 - Blocking] `schedule:run --once` needs the database for real commands**
+- **Found during:** Task 2 (fonoteka commands call `app.Lookup[*gorm.DB]()`)
+- **Issue:** The plan says "no River" for `--once` but does not say how commands reach the database.
+- **Fix:** When an entry is due, `--once` opens and publishes the database the way `queue:work` does, or reuses an already published `*gorm.DB`. With nothing due it touches no database.
+- **Files modified:** modules/conga/commands.go
+- **Verification:** The `TestScheduleRunOnce` subtests. `fonoteka schedule:run --once` exits 0.
+- **Committed in:** 2237a64
+
+**5. [Behaviour change, documented] Serve and queue:work workers list the `scheduled` queue**
+- The plan requires the `scheduled` queue on every known-queue set, so `TestStartServeWorker` now expects `[default scheduled]`. The unknown-queue error lists it too. `TestQueueWorkCommand` still passes because the list starts with `default`.
+
+**6. [Additions] Extra tests and small internals**
+- Added `TestScheduleEntries` (the empty, invalid and ordering edges) and `TestScheduleRunForeground` (the foreground `schedule:run`).
+- The unexported `scheduleNow` clock serves the `--once` tests.
+- `gofmt` reordered the pre-existing out-of-order imports of `cmd/summer/main.go` and `runtime.go`. Both files were already being edited.
+
+---
+
+**Total deviations:** 4 auto-fixed (2 bugs, 1 missing validation, 1 blocking), plus 2 documented notes.
+**Impact on plan:** Fixes 1 and 2 were needed for schedules to start at all and to run every entry. No scope creep.
+
+## Issues Encountered
+
+- `internal/build/registry.go` still has gofmt drift from before this plan. It is out of scope and was left untouched.
+
+## Known Stubs
+
+None. The fonoteka entry names `fonoteka:prune-notifications`, which Phase 14 registers. Until then the scheduler warns and skips it by design (user decision 5, research Open Question 5).
+
+## User Setup Required
+
+None. For system cron, run `* * * * * cd /app && ./bin/<app> schedule:run --once`. Alternatively run a `schedule:run` process, or rely on the worker inside `serve`/`queue:work`, which already processes the `scheduled` queue.
+
+## Next Phase Readiness
+
+- Plan 11-03 can register the broadcast job through `conga.Manager` and edit fonoteka `plugin.go`. This plan left both alone.
+- Plan 11-07 should add coverage for:
+  - `logWriter` partial-line flushing
+  - the missing-catalog skip path
+  - `dueAt` with `Every(d)` over 1 minute that does not divide an hour (for example 90s)
+  - the `summer schedule:run --once` argv forwarding
+- In Phase 14, registering `fonoteka:prune-notifications` makes the daily entry run with no scheduler change.
+
+## Self-Check: PASSED
+
+- All six created files exist on disk.
+- summercms.go commits d9f939a and 2237a64 exist, and so do fonoteka.go commits 37133f9 and 94e95de.
+- After Task 2, `go vet ./... && go test ./...` passed in summercms.go, and the fonoteka.go vet and test command for all three modules passed.
+- All acceptance-criteria greps and `go doc` checks for both tasks passed.
+
+---
+*Phase: 11-jobs-realtime-and-search-infrastructure*
+*Completed: 2026-09-29*