From f1077382f537d8abefa85b4f2d7a744f09f74569 Mon Sep 17 00:00:00 2001 From: Jakub Zych Date: Tue, 29 Sep 2026 22:54:28 +0200 Subject: [PATCH] docs(11-02): complete scheduler plan --- .planning/ROADMAP.md | 6 +- .planning/STATE.md | 15 +- .../11-02-SUMMARY.md | 300 ++++++++++++++++++ 3 files changed, 311 insertions(+), 10 deletions(-) create mode 100644 .planning/phases/11-jobs-realtime-and-search-infrastructure/11-02-SUMMARY.md diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index cf14da8..854ba28 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -505,7 +505,7 @@ Plans: 4. A channel-namespace authorizer registry re-validates on every subscribe; a broadcastable model interface with bulk-write suppression emits exactly one summary event for a bulk operation. 5. Typesense sync is scoped by `collection_id` behind a settings kill-switch and degrades gracefully without DB/config. -**Plans:** 1/7 plans executed +**Plans:** 2/7 plans executed **Research flag:** yes Plans: @@ -514,7 +514,7 @@ Plans: - [x] 11-01-PLAN.md — conga core: River v0.47.0 on NewWithPgxListener, summer_jobs + River v7 migrations, job manager, conga.Job[T], worker in serve, queue:work/queue:clear, lagoon OnDatabase and Transaction/AfterCommit seams, fonoteka allow-lists (summercms.go, fonoteka.go) **Wave 2** *(blocked on Wave 1 completion)* -- [ ] 11-02-PLAN.md — Scheduler: pact.HasSchedule, Daily/Every, River periodic jobs, bonfire.Call, schedule:run (+ --once), fonoteka prune-notifications entry +- [x] 11-02-PLAN.md — Scheduler: pact.HasSchedule, Daily/Every, River periodic jobs, bonfire.Call, schedule:run (+ --once), fonoteka prune-notifications entry - [ ] 11-03-PLAN.md — Realtime: lighthouse + Centrifugo driver (token route, subscribe proxy, HTTP client, broadcasts with suppression), fonoteka authorizers, Mount, ws-api bucket, Album binding **Wave 3** *(blocked on Wave 2 completion)* @@ -654,7 +654,7 @@ Phases execute in numeric order: 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → | 8. OAuth2.1 authorization server | 10/10 | Complete | 2026-09-23 | | 9. Backend admin authentication and schema pipeline | 12/12 | In Progress| | | 10. Admin Vue SPA | 5/5 | Complete | 2026-09-27 | -| 11. Jobs, realtime and search infrastructure | 1/7 | In Progress| | +| 11. Jobs, realtime and search infrastructure | 2/7 | In Progress| | | 11.1. SummerCMS documentation for humans and AI agents | 0/TBD | Not started | - | | 11.2. Ready to share: summercms.io website and newsletter plugin | 0/TBD | Not started | - | | 12. Płytarium API — Collections and Albums | 0/TBD | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index ab9e4d9..1e23e61 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -4,16 +4,16 @@ milestone: v1.0 current_phase: 11 current_phase_name: Jobs, realtime and search infrastructure status: executing -stopped_at: Completed 11-01-PLAN.md -last_updated: "2026-09-29T13:27:45.169Z" +stopped_at: Completed 11-02-PLAN.md +last_updated: "2026-09-29T20:54:27.609Z" last_activity: 2026-09-29 last_activity_desc: Phase 11 execution started -state_head: 144c54bb2c682a95b37f9c1c68ec2ec6b68b8dbe +state_head: 2237a640d2400e337d9e6c638a7b9534cd9c77b9 progress: total_phases: 19 completed_phases: 9 total_plans: 85 - completed_plans: 79 + completed_plans: 80 milestone_name: milestone --- @@ -29,7 +29,7 @@ See: .planning/PROJECT.md (updated 2026-09-16) ## Current Position Phase: 11 (Jobs, realtime and search infrastructure) — EXECUTING -Plan: 2 of 7 +Plan: 3 of 7 Status: Ready to execute Last activity: 2026-09-29 — Phase 11 execution started @@ -134,6 +134,7 @@ Progress: [██████░░░░] 60% | Phase 10.1 P03 | 11min | 3 tasks | 16 files | | Phase 10.1 P04 | 42min | 3 tasks | 32 files | | Phase 11 P01 | 37 min | 3 tasks | 50 files | +| Phase 11 P02 | 385 min | 2 tasks | 21 files | ## Accumulated Context @@ -394,6 +395,6 @@ Items acknowledged and carried forward from previous milestone close: ## Session Continuity -Last session: 2026-09-29T13:27:44.754Z -Stopped at: Completed 11-01-PLAN.md +Last session: 2026-09-29T20:54:27.162Z +Stopped at: Completed 11-02-PLAN.md Resume file: None diff --git a/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-02-SUMMARY.md b/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-02-SUMMARY.md new file mode 100644 index 0000000..2757b50 --- /dev/null +++ b/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-02-SUMMARY.md @@ -0,0 +1,300 @@ +--- +phase: 11-jobs-realtime-and-search-infrastructure +plan: 02 +subsystem: infra +tags: [river, scheduler, periodic-jobs, cli, cron, bonfire] + +requires: + - phase: 11-jobs-realtime-and-search-infrastructure + provides: "11-01 conga worker (StartWorker, Manager, conga.Job, queue settings) on River v0.47.0" +provides: + - "pact.HasSchedule, ScheduledCommand, Cadence with Daily/DailyAt/Every (no River import)" + - "bonfire.Call, Catalog, NewCatalog, ErrUnknownCommand for in-process command runs" + - "conga.Daily / conga.Every wall-clock river.PeriodicSchedule in app.timezone" + - "Every worker carries plugin schedules as River PeriodicJobs; runs go to the scheduled queue (MaxAttempts 1, unique by args within the cadence period)" + - "schedule:run (scheduler-only worker) and schedule:run --once (Laravel minute-match for system cron), plus the summer delegate" + - "Generated app main publishes bonfire.NewCatalog(commands)" + - "fonoteka declares fonoteka:prune-notifications daily in schedule.go" +affects: [11-03 broadcasts, 11-07 unit tests, 14 domain jobs (prune-notifications command), 15 cutover (cron/process layout)] + +actuals: + tokens: 13984 + tasks: 2 + commits: 2 +plan_head_before: 77b8ff177ac96457d64dbb2cbddc6bcaabbea21c +plan_head_after: 2237a640d2400e337d9e6c638a7b9534cd9c77b9 +# fonoteka.go (separate repository) received 2 more commits: 37133f9, 94e95de + +tech-stack: + added: [] + patterns: + - "Plugins declare recurring commands with pact.HasSchedule; conga compiles them into River periodic jobs with ids []:" + - "Scheduled runs call commands in-process through the app's *bonfire.Catalog, published by the generated main" + - "A scheduled job runs only when it matches a compiled schedule entry exactly (T-11-09)" + +key-files: + created: + - modules/bonfire/call.go + - modules/bonfire/call_test.go + - modules/conga/schedule.go + - modules/conga/scheduler.go + - modules/conga/schedule_test.go + - ../fonoteka.go/plugins/golem15/fonoteka/schedule.go + modified: + - modules/pact/capabilities.go + - modules/pact/README.md + - modules/bonfire/README.md + - modules/conga/conga.go + - modules/conga/worker.go + - modules/conga/commands.go + - modules/conga/listen_test.go + - modules/conga/README.md + - internal/build/build.go + - internal/build/build_test.go + - cmd/summer/main.go + - cmd/summer/runtime.go + - cmd/summer/main_test.go + - examples/hello/main.go + - ../fonoteka.go/main.go + +key-decisions: + - "Periodic job ids are []:, not #:: River's PeriodicJob.ID regex rejects '#'" + - "Scheduled runs are unique by args and period (UniqueOpts{ByArgs: true, ByPeriod}); ByPeriod alone would collapse different entries of the shared summer.scheduled_command kind within one period" + - "Every(d) requires d >= 1s (River's ByPeriod floor) and 24h % d == 0; DailyAt rejects hours outside 0-23 and minutes outside 0-59" + - "schedule:run --once opens the database only when an entry is due, and reuses a published *gorm.DB when there is one; plugin commands look the DB up from the app" + - "Every(d) in --once is due every run for d <= 1 minute, otherwise when wall-clock minutes since midnight are a multiple of d" + +patterns-established: + - "bonfire.Catalog: the final command list is an app service, so non-CLI code can run commands with Call" + - "Forged-row defence: jobs whose args name code to execute are checked against a compiled table before running" + +requirements-completed: [CLI-04] + +coverage: + - id: D1 + description: "pact.HasSchedule with Daily/DailyAt/Every cadences and no River import; conga.Daily/Every compute wall-clock next runs (adjacency, DST, midnight rollover) and invalid cadences are rejected" + requirement: CLI-04 + verification: + - kind: unit + ref: "modules/conga/schedule_test.go#TestScheduleNext" + status: pass + - kind: unit + ref: "modules/conga/schedule_test.go#TestScheduleEntries" + status: pass + human_judgment: false + - id: D2 + description: "bonfire.Call and Catalog run a named command in-process with args; unknown names wrap ErrUnknownCommand" + requirement: CLI-04 + verification: + - kind: unit + ref: "modules/bonfire/call_test.go#TestCall" + status: pass + human_judgment: false + - id: D3 + description: "A plugin's Every(1s) entry runs through a River periodic job and bonfire.Call inside a worker; an unregistered command is skipped with a Warn log" + requirement: CLI-04 + verification: + - kind: integration + ref: "modules/conga/schedule_test.go#TestScheduleRunsCommand" + status: pass + human_judgment: false + - id: D4 + description: "Generated app main publishes bonfire.NewCatalog(commands) after plugin commands and before NewRoot; hello and fonoteka mains regenerated" + requirement: CLI-04 + verification: + - kind: unit + ref: "internal/build/build_test.go#TestGenerateMainPublishesCommandCatalog" + status: pass + human_judgment: false + - id: D5 + description: "schedule:run --once minute-match semantics (daily, Every(5m), app.timezone), unknown command warns and others run, first error returned after all ran; schedule:run foreground runs a scheduler-only worker" + requirement: CLI-04 + verification: + - kind: unit + ref: "modules/conga/schedule_test.go#TestScheduleRunOnce" + status: pass + - kind: integration + ref: "modules/conga/schedule_test.go#TestScheduleRunForeground" + status: pass + - kind: unit + ref: "cmd/summer/main_test.go#TestToolCommandNames" + status: pass + human_judgment: false + - id: D6 + description: "T-11-09: a scheduled job whose command/args differ from its compiled entry, or whose entry is unknown, is skipped without running anything, including a forged river_job row" + requirement: CLI-04 + verification: + - kind: integration + ref: "modules/conga/schedule_test.go#TestScheduledEntryMismatchSkipped" + status: pass + human_judgment: false + - id: D7 + description: "T-11-17: two inserts of one entry inside its period leave one river_job row; a different entry in the same period gets its own row" + requirement: CLI-04 + verification: + - kind: integration + ref: "modules/conga/schedule_test.go#TestScheduleUniqueByPeriod" + status: pass + human_judgment: false + - id: D8 + description: "fonoteka declares fonoteka:prune-notifications daily; `fonoteka schedule:run --once` prints the no-commands line and exits 0; --help shows --once" + requirement: CLI-04 + verification: + - kind: other + ref: "cd ../fonoteka.go && SUMMER_GOLEM15__USER__JWT__SECRET=test-only-cli-secret go run . schedule:run --once" + status: pass + human_judgment: false + - id: D9 + description: "Concurrency backstop: with several worker processes River leader election means one process enqueues each period" + verification: [] + human_judgment: true + rationale: "Multi-process leader election is River's guarantee and is not exercised by a test here (plan marks it verification: backstop)" + +duration: 385min +completed: 2026-09-29 +status: complete +--- + +# Phase 11 Plan 02: scheduler on River periodic jobs Summary + +**`pact.HasSchedule` with `Daily`/`DailyAt`/`Every` cadences becomes River periodic jobs on wall-clock schedules in `app.timezone`. Each run is a one-attempt job on the `scheduled` queue that calls the command in-process through a `bonfire.Catalog` the generated main publishes. `schedule:run` runs a scheduler process, and `schedule:run --once` serves system cron.** + +## Performance + +- **Duration:** 385 min wall-clock. The clock ran from 14:09Z to 20:34Z; the active work was much shorter, and most of the span was idle time between tool calls. +- **Started:** 2026-09-29T14:09:06Z +- **Completed:** 2026-09-29T20:34:36Z +- **Tasks:** 2 +- **Files modified:** 21 (19 in summercms.go, 2 in fonoteka.go) + +## Accomplishments + +- `pact` has the schedule capability: `HasSchedule`, `ScheduledCommand`, the opaque `Cadence` (`IsZero`, `Interval`, `At`) and the `Daily`, `DailyAt` and `Every` constructors. pact still imports no River code. +- `bonfire.Call` and `bonfire.Catalog` run any registered command in-process with empty stdin. The generated app main publishes `bonfire.NewCatalog(commands)` before building the root. The hello and fonoteka mains were regenerated. +- `conga.Daily` and `conga.Every` are `river.PeriodicSchedule` implementations: + - `Daily` fires at the next wall-clock `hh:mm`. On DST days the runs are 23h or 25h apart but stay at 12:00 local. + - `Every` fires at the next multiple of its interval since local midnight and rolls over to the next midnight. + - Both fire strictly after an exact boundary. +- Every worker carries one periodic job per schedule entry, with id `[]:`. Runs go to the `scheduled` queue (`queue.queues.scheduled`, default 1) with MaxAttempts 1 and uniqueness by args within the cadence period. +- The worker runs a job only when it matches a compiled entry exactly (T-11-09). A missing catalog or an unregistered command is logged at Warn and skipped (user decision 5). Command output is logged line by line with a `command` attribute. +- `schedule:run` runs a scheduler-only worker until SIGINT or SIGTERM. `schedule:run --once` runs the entries due this minute without River, prints `Running scheduled command: …` or `No scheduled commands are ready to run.`, and returns the first command error. The `summer` delegate forwards `--once`. +- fonoteka declares `fonoteka:prune-notifications` daily in `schedule.go`. `plugin.go` is untouched, so plan 11-03 can edit it. + +## Task Commits + +summercms.go: +1. **Task 1: an interval command runs on schedule inside a worker through bonfire.Call** - `d9f939a` (feat) +2. **Task 2: schedule:run as a dedicated process and from system cron** - `2237a64` (feat) + +fonoteka.go: +1. **Task 1** - `37133f9` (feat: regenerated main.go publishes the catalog) +2. **Task 2** - `94e95de` (feat: schedule.go with the daily prune entry) + +## Files Created/Modified + +- `modules/pact/capabilities.go`, `modules/pact/README.md`: the schedule capability and cadences +- `modules/bonfire/call.go`, `call_test.go`, `README.md`: `Call`, `Catalog`, `NewCatalog` and `ErrUnknownCommand` +- `modules/conga/schedule.go`: `Daily`, `Every`, `scheduleFor` and `appLocation` +- `modules/conga/scheduler.go`: `QueueScheduled`, `ScheduledCommandArgs`, entry compilation, periodic jobs, the built-in job, the T-11-09 check and the log writer +- `modules/conga/worker.go`, `conga.go`: `StartWorker` sets `PeriodicJobs`, registers the built-in job and adds the `scheduled` queue +- `modules/conga/commands.go`: `schedule:run [--once]` +- `modules/conga/schedule_test.go`: seven tests. `listen_test.go`: the serve worker now also lists `scheduled` +- `internal/build/build.go`, `build_test.go`: the catalog publish in the generated main +- `cmd/summer/main.go`, `runtime.go`, `main_test.go`: the `schedule:run` delegate +- `examples/hello/main.go` and `../fonoteka.go/main.go` (regenerated), and `../fonoteka.go/plugins/golem15/fonoteka/schedule.go` + +## Decisions Made + +See `key-decisions` in the frontmatter. + +## TDD Gate Compliance + +Task 2 is `tdd="true"`. The tests were written first against a stub `schedule:run` that returned "not implemented". + +- **RED:** `TestScheduleRunOnce` failed on its assertions in all 6 subtests. `gsd-tools check tdd-red-evidence` returned `RED_EVIDENCE_OK` (`target_test_failed`). The command was `go test ./modules/conga -run '^(TestScheduleRunOnce|TestScheduledEntryMismatchSkipped|TestScheduleUniqueByPeriod)$' -count=1 -json`, converted to TAP. +- `TestScheduledEntryMismatchSkipped` and `TestScheduleUniqueByPeriod` passed at RED. Task 1's action required the exact-match check (T-11-09) and the ByPeriod insert options, so both mitigations already existed. To prove the tests are real, I ran a mutation check and restored the code afterwards: + - Running the job's own command instead of the compiled entry failed `TestScheduledEntryMismatchSkipped` ("no skip warning"). + - Dropping `ByArgs` failed `TestScheduleUniqueByPeriod` ("river_job rows = 1, want 2"). +- **Gate violation, flagged:** there is no separate `test(11-02)` RED commit. The project CLAUDE.md requires `go vet` and `go test ./...` to be green at every commit, so the tests and implementation landed together in `2237a64`. Plan 11-01 made the same call. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Periodic job id separator** +- **Found during:** Task 1 (reading River's `PeriodicJob.validate`) +- **Issue:** River requires `PeriodicJob.ID` to match `\A[\w][\w\-\[\]<>\/.·:+]+\z`, which has no `#`. The plan's `#:` would have failed every worker start that had a schedule. +- **Fix:** Ids use `[]:`, for example `golem15.fonoteka[0]:fonoteka:prune-notifications`. Ordering, uniqueness and readability are unchanged. +- **Files modified:** modules/conga/scheduler.go +- **Verification:** `TestScheduleEntries` checks the ids. `TestScheduleRunsCommand` starts a worker with them. +- **Committed in:** d9f939a + +**2. [Rule 1 - Bug] ByPeriod alone deduplicates across entries** +- **Found during:** Task 1 (River `UniqueOpts` docs: without `ByArgs`, uniqueness applies to all jobs of the kind) +- **Issue:** Every entry shares the kind `summer.scheduled_command`, so `ByPeriod` alone would drop the second entry due in the same period. +- **Fix:** `UniqueOpts{ByArgs: true, ByPeriod: period}`. The args hold the entry id, so the dedupe is per entry. +- **Files modified:** modules/conga/scheduler.go +- **Verification:** `TestScheduleUniqueByPeriod` checks both parts: the same entry twice gives 1 row, and a second entry gives 2 rows. The mutation check confirms the second part catches the bug. +- **Committed in:** d9f939a + +**3. [Rule 2 - Missing validation] Cadence bounds** +- **Found during:** Task 1 +- **Issue:** River rejects `ByPeriod` under 1s. `DailyAt(25, 0)` would silently normalize to 01:00 the next day. +- **Fix:** `scheduleFor` also rejects intervals under 1s and daily times outside 00:00-23:59, with the plugin id and entry index in the error. +- **Files modified:** modules/conga/schedule.go +- **Verification:** `TestScheduleNext` covers the invalid cadences. +- **Committed in:** d9f939a + +**4. [Rule 3 - Blocking] `schedule:run --once` needs the database for real commands** +- **Found during:** Task 2 (fonoteka commands call `app.Lookup[*gorm.DB]()`) +- **Issue:** The plan says "no River" for `--once` but does not say how commands reach the database. +- **Fix:** When an entry is due, `--once` opens and publishes the database the way `queue:work` does, or reuses an already published `*gorm.DB`. With nothing due it touches no database. +- **Files modified:** modules/conga/commands.go +- **Verification:** The `TestScheduleRunOnce` subtests. `fonoteka schedule:run --once` exits 0. +- **Committed in:** 2237a64 + +**5. [Behaviour change, documented] Serve and queue:work workers list the `scheduled` queue** +- The plan requires the `scheduled` queue on every known-queue set, so `TestStartServeWorker` now expects `[default scheduled]`. The unknown-queue error lists it too. `TestQueueWorkCommand` still passes because the list starts with `default`. + +**6. [Additions] Extra tests and small internals** +- Added `TestScheduleEntries` (the empty, invalid and ordering edges) and `TestScheduleRunForeground` (the foreground `schedule:run`). +- The unexported `scheduleNow` clock serves the `--once` tests. +- `gofmt` reordered the pre-existing out-of-order imports of `cmd/summer/main.go` and `runtime.go`. Both files were already being edited. + +--- + +**Total deviations:** 4 auto-fixed (2 bugs, 1 missing validation, 1 blocking), plus 2 documented notes. +**Impact on plan:** Fixes 1 and 2 were needed for schedules to start at all and to run every entry. No scope creep. + +## Issues Encountered + +- `internal/build/registry.go` still has gofmt drift from before this plan. It is out of scope and was left untouched. + +## Known Stubs + +None. The fonoteka entry names `fonoteka:prune-notifications`, which Phase 14 registers. Until then the scheduler warns and skips it by design (user decision 5, research Open Question 5). + +## User Setup Required + +None. For system cron, run `* * * * * cd /app && ./bin/ schedule:run --once`. Alternatively run a `schedule:run` process, or rely on the worker inside `serve`/`queue:work`, which already processes the `scheduled` queue. + +## Next Phase Readiness + +- Plan 11-03 can register the broadcast job through `conga.Manager` and edit fonoteka `plugin.go`. This plan left both alone. +- Plan 11-07 should add coverage for: + - `logWriter` partial-line flushing + - the missing-catalog skip path + - `dueAt` with `Every(d)` over 1 minute that does not divide an hour (for example 90s) + - the `summer schedule:run --once` argv forwarding +- In Phase 14, registering `fonoteka:prune-notifications` makes the daily entry run with no scheduler change. + +## Self-Check: PASSED + +- All six created files exist on disk. +- summercms.go commits d9f939a and 2237a64 exist, and so do fonoteka.go commits 37133f9 and 94e95de. +- After Task 2, `go vet ./... && go test ./...` passed in summercms.go, and the fonoteka.go vet and test command for all three modules passed. +- All acceptance-criteria greps and `go doc` checks for both tasks passed. + +--- +*Phase: 11-jobs-realtime-and-search-infrastructure* +*Completed: 2026-09-29*