summercms/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-07-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, actuals, plan_head_before, plan_head_after, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, coverage, duration, completed, status

phase	plan	subsystem	tags	requires	provides	affects	actuals	plan_head_before	plan_head_after	tech-stack	key-files	key-decisions	patterns-established	requirements-completed	coverage	duration	completed	status
11-jobs-realtime-and-search-infrastructure	07	testing	river conga lighthouse centrifugo flare beachcomber typesense tide testcontainers gate security-review	phase provides 11-jobs-realtime-and-search-infrastructure 11-01..11-06: conga jobs and scheduler, lagoon seams, lighthouse and the Centrifugo driver, flare Web Push, beachcomber and the Typesense engine, tide broadcast goldens, and the fonoteka bindings	Branch-level tests for every Phase 11 package (conga 92.5%, lighthouse 91.7%, lighthouse/centrifugo 92.4%, flare 90.0%, beachcomber 88.7%, beachcomber/typesense 95.6%) and every fonoteka binding Fixes for three Phase 11 defects, each with a failing-when-broken test: single-statement broadcasts enqueued after commit, after-commit handles carrying the write's statement, and savepoints released after a swallowed read failure scripts/check-phase11.sh: fail-closed gate (--self-test, --hygiene, --go, --postgres, --named, --evidence, --all) plus the --removal mutation harness 11-SECURITY-REVIEW.md with 31 threat rows and 13 removal checks; validated 11-VALIDATION.md check-phase10.1.sh accepts exactly the Phase 12 pending goldens, so the prior gate passes again	12 albums API (pending created/updated goldens SearchIDs re-gate) 13 CSV import (job id scoping T-11-08) 14 domain jobs (prune-notifications) 15 cutover	tokens tasks commits 66283 3 12	6562d94ef3	73cfed74ed	added patterns Removal checks are anchor-exact mutations run by the phase gate (--removal): the anchor must occur once, the named test must fail on an assertion (a build failure does not count), and cmp proves the byte-for-byte restore A savepoint helper rolls back when its RELEASE fails, because a caller-swallowed read error still aborts the Postgres transaction Code handed a GORM callback's handle gets a clean statement (Session NewDB+Context, Clauses(), Session NewDB), in lagoon as in beachcomber Side-effect GORM callbacks that must run inside the write's transaction declare Before(gorm:commit_or_rollback_transaction) as well as their After anchor Gate detectors accept a documented pending skip only by exact test name and pending text	created modified scripts/check-phase11.sh .planning/phases/11-jobs-realtime-and-search-infrastructure/11-SECURITY-REVIEW.md modules/conga/manager_test.go modules/conga/worker_test.go modules/conga/commands_test.go modules/lagoon/queue_migrations_test.go modules/pact/cadence_test.go modules/lighthouse/postgres_test.go modules/lighthouse/broadcast_test.go modules/lighthouse/lighthouse_test.go modules/lighthouse/registry_test.go modules/lighthouse/route_test.go modules/lighthouse/centrifugo/token_test.go modules/lighthouse/centrifugo/client_test.go modules/lighthouse/centrifugo/proxy_test.go modules/flare/flare_test.go modules/beachcomber/postgres_test.go modules/beachcomber/sync_test.go modules/beachcomber/typesense/engine_test.go ../fonoteka.go/plugins/golem15/fonoteka/schedule_test.go ../fonoteka.go/plugins/golem15/fonoteka/ws_authorizer_test.go ../fonoteka.go/plugins/golem15/fonoteka/album_realtime_test.go ../fonoteka.go/plugins/golem15/fonoteka/album_search_test.go modules/lagoon/transaction.go modules/lagoon/README.md modules/lighthouse/broadcast.go modules/lighthouse/README.md modules/beachcomber/sync.go modules/beachcomber/README.md scripts/check-phase10.1.sh .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VALIDATION.md modules/conga/listen_test.go modules/conga/schedule_test.go modules/lagoon/transaction_test.go modules/lagoon/ondatabase_test.go modules/bonfire/call_test.go modules/lighthouse/channel_test.go modules/flare/encrypt_test.go modules/tide/centrifugo_test.go	The removal harness lives in the gate (--removal) instead of a Go helper, so the RC rows are reproducible with one command; it is kept out of --all because it edits tracked source while it runs T-11-SC counts as a high mitigated threat with its own removal check (RC-13: the client-library hygiene rule disabled in a copy of the gate makes --self-test fail) The gate's go test runs use -json and the detector refuses skips, zero tests and 'no tests to run'; only TestBroadcastGoldens/created and /updated may skip, and only with 'pending: Phase 12' The cron hygiene rule reads direct go.mod requirements: robfig/cron/v3 is in the module graph only as River's test dependency check-phase10.1.sh was taught the same exact-name pending-skip rule rather than weakening it to accept skips in general JWT claims containing <, > or & are HTML-escaped by golang-jwt's json.Marshal (PHP leaves them raw); the token is opaque to Centrifugo and the decoded claims are equal, so this is recorded, not changed	Phase gates prove each security mitigation with a scripted removal check, not only with a review note	JOBS-01 CLI-04 CLI-06 RT-01 RT-02 RT-03 SRCH-01	id description requirement verification human_judgment D1 Jobs, scheduler and lagoon seams covered branch by branch: every job outcome, both cancellation paths, StopJob from a worker, PHP JobManager semantics, principal/delay/registration rules, queue:work filtering, queue:clear batches and states, scheduler validation/ordering/DST/missing catalog, River v7 + summer_jobs migrations up/down/idempotent JOBS-01 kind ref status integration go test ./modules/conga -run '^(TestListenPickupLatency|TestDispatchTransactional|TestOutcome.*|TestCancel.*|TestQueueClear|TestQueueWork|TestSchedule.*)$' -count=1 -v pass kind ref status integration go test ./modules/lagoon -run '^(TestQueueMigrationsUpDown|TestOnDatabaseAfterActivate|TestTransactionAfterCommit)$' -count=1 -v pass kind ref status unit go test ./modules/bonfire -run '^TestCall$' -count=1 -v pass false id description requirement verification human_judgment D2 fonoteka schedule entry: the daily fonoteka:prune-notifications run goes through the scheduler worker with the real command catalog and is skipped with the not-registered warning until Phase 14 CLI-04 kind ref status integration ../fonoteka.go/plugins/golem15/fonoteka/schedule_test.go#TestFonotekaScheduleSkipsUnregisteredPrune pass false id description requirement verification human_judgment D3 Realtime covered: TestBroadcastTx, TestSuppression (Widget silenced, Gadget not), TestBulkEmitsOnce, TestBroadcastEdges, TestMountSurfaces, TestRegistry under -race, token claims and handler, client requests, and a TestProxy table porting the WinterCMS WS-005/WS-007/WS-013 cases RT-02 kind ref status integration go test ./modules/lighthouse/... -run '^(TestBroadcastTx|TestSuppression|TestBulkEmitsOnce|TestToken.*|TestClient.*|TestProxy.*)$' -count=1 -v pass kind ref status integration go test ./modules/lighthouse/... ./modules/flare ./modules/beachcomber/... ./modules/tide -count=1 -race pass false id description requirement verification human_judgment D4 fonoteka bindings: TestWsAuthorizer (collection and wishlist rules, editor removal), TestAlbumBroadcastBinding (kind=collection rule, deleted/created payloads, actor), TestAlbumSearchable (PHP document keys and types, collection_id refusal, schema, settings gate) RT-03 kind ref status integration cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^(TestWsAuthorizer|TestAlbumBroadcastBinding|TestAlbumSearchable)$' -count=1 -race -v pass false id description requirement verification human_judgment D5 Search sync covered: TestSyncGates, TestSyncAfterCommit, TestSyncDeleteAndSoftDelete, TestSyncFailuresNonFatal and the Typesense wire contract (TestEngineWire) SRCH-01 kind ref status integration go test ./modules/beachcomber/... -run '^TestSync.*$' -count=1 -v pass kind ref status unit modules/beachcomber/typesense/engine_test.go#TestEngineWire pass false id description requirement verification human_judgment D6 Push and recorder covered: TestVAPIDHeader, TestSendAllowlist, TestSendStatuses, TestEncryptRejects, TestCentrifugoRecorder RT-01 kind ref status unit go test ./modules/flare ./modules/tide -count=1 -race pass false id description requirement verification human_judgment D7 Three defects fixed with RED-then-GREEN tests: single-statement broadcast enqueue ordering, the after-commit handle's statement, the savepoint left aborted by a swallowed read failure RT-03 kind ref status integration modules/lighthouse/broadcast_test.go#TestBroadcastTx/single_statement_write_enqueues_in_its_own_transaction pass kind ref status integration modules/lagoon/transaction_test.go#TestTransactionAfterCommit/callback_handle_has_a_clean_statement pass kind ref status integration modules/beachcomber/sync_test.go#TestSyncFailuresNonFatal/failed_gate_read_keeps_the_callers_transaction pass kind ref status integration modules/lighthouse/broadcast_test.go#TestBroadcastSwallowedReadFailure pass false id description requirement verification human_judgment D8 Fail-closed phase gate in both repositories, 13 removal checks for the high mitigated threats, security review and validated test map JOBS-01 kind ref status other bash -n scripts/check-phase11.sh && scripts/check-phase11.sh --self-test && scripts/check-phase11.sh --all pass kind ref status other scripts/check-phase11.sh --removal pass kind ref status other scripts/check-phase10.1.sh --all pass false id description verification human_judgment rationale D9 Real-client behaviour: the unchanged Nuxt app receives an album event through a real Centrifugo v6 with a Go-issued token, and a real Typesense 26.0 receives an upsert true Needs the running Nuxt app, Centrifugo and Typesense servers; the plan marks it a backstop collected at /gsd-verify-work (the two manual rows of 11-VALIDATION.md)	59min	2026-09-30	complete

Phase 11 Plan 07: Phase 11 unit tests, security evidence and the phase gate Summary

Phase 11 now has branch-level tests in both repositories: 80-96% coverage in every new package, with Postgres tests on testcontainers. Three real defects were found and fixed (broadcast jobs enqueued after a single write's commit, after-commit handles that carried the write's statement, savepoints left aborted by swallowed read errors). A fail-closed check-phase11.sh --all prints "phase11 all passed", and 13 scripted removal checks prove that every high threat's test fails when its protection is removed.

Performance

• Duration: 59 min
• Started: 2026-09-30T11:56:28Z
• Completed: 2026-09-30T12:55:22Z
• Tasks: 3
• Files modified: 40 (32 in summercms.go, 4 test files in fonoteka.go, plus 4 planning files)

Accomplishments

• Task 1: jobs, scheduler and lagoon seams.
  • conga tests are split into manager_test.go, worker_test.go and commands_test.go. They cover every outcome rule, both cancellation paths, StopJob from inside a job, the PHP JobManager semantics, principals, delays, registration and worker errors, queue settings, queue:work filtering, and queue:clear over two 10000-job batches. Coverage is 92.5%.
  • Scheduler tests cover validation, ordering, a missing catalog, the log writer and dueAt for Every(90s). The DST cases in Europe/Warsaw were already there.
  • TestQueueMigrationsUpDown covers the River v7 table set, the summer_jobs column types and defaults, rollback one step at a time, and an idempotent rerun.
  • Also added: bonfire TestCallEdges and pact TestCadence.
  • fonoteka TestFonotekaScheduleSkipsUnregisteredPrune pins the Phase 14 skip.
• Task 2: realtime, push, search and recorder.
  • lighthouse has a Postgres harness and TestBroadcastTx, TestSuppression, TestBulkEmitsOnce, TestBroadcastEdges, TestBroadcastPublishFailure and TestMountSurfaces.
  • centrifugo has TestTokenClaims, TestTokenHandler, TestClientRequests and a 29-case TestProxy that ports the PHP security tests.
  • flare, beachcomber, typesense and the tide recorder got their own test sets.
  • fonoteka has TestWsAuthorizer, TestAlbumBroadcastBinding and TestAlbumSearchable.
• Task 3: gate and evidence.
  • scripts/check-phase11.sh runs the detector, hygiene, both repositories' suites (including -race and a -count=3 LISTEN run), every named test and the evidence check. Its --removal harness backs 13 RC rows.
  • 11-SECURITY-REVIEW.md has 31 threat rows, 13 removal checks and a table of the fixes.
  • 11-VALIDATION.md is validated.
  • REQUIREMENTS.md marks the seven requirement IDs complete.

Task Commits

summercms.go:

1. c544319: fix(11-07), lagoon after-commit callbacks get a clean handle (deferred item b)
2. 6f50b6c: fix(11-07), broadcast jobs are enqueued before GORM commits a single write (deferred item a), plus the lighthouse harness
3. 35ac96d: test(11-07), Task 1: conga, lagoon, bonfire and pact
4. 33194a1: test(11-07), Task 2: lighthouse and centrifugo
5. 6dadbf6: test(11-07), Task 2: flare
6. 6df43d4: fix(11-07), savepoint rollback after a swallowed read failure, plus the beachcomber harness and sync tests
7. 18d3097: test(11-07), Task 2: Typesense wire contract
8. 11b5b4c: test(11-07), Task 2: tide recorder edges
9. e55d234: test(11-07), Task 2: keyless Typesense engine registration
10. 7743487: test(11-07), Task 3: check-phase11.sh and the removal harness
11. a34c6ec: docs(11-07), Task 3: security review and validation
12. 73cfed7: fix(11-07), check-phase10.1.sh accepts the Phase 12 pending goldens

fonoteka.go:

1. 1657cc1: test(11-07), Task 1: schedule entry skip
2. c222e03: test(11-07), Task 2: ws authorizers and the Album realtime and search bindings

Files Created/Modified

See key-files in the frontmatter. The production changes are modules/lagoon/transaction.go, modules/lighthouse/broadcast.go, modules/beachcomber/sync.go and their READMEs, plus scripts/check-phase10.1.sh. Everything else is tests, the gate and planning docs.

Decisions Made

See key-decisions in the frontmatter.

Deviations from Plan

Auto-fixed Issues

1. [Rule 1 - Bug] Broadcast jobs of single-statement writes were enqueued after GORM's commit (deferred item a from 11-05)

• Found during: Task 2, first lighthouse test
• Issue: lighthouse:after_create, after_update and after_delete named only an After anchor, so GORM appended them past gorm:commit_or_rollback_transaction. A plain gdb.Create therefore enqueued its broadcast job on the pool after the commit.
• RED: TestBroadcastTx/single_statement_write_enqueues_in_its_own_transaction reported that the channels ran on [false] rather than the write's *sql.Tx.
• Fix: each callback also declares Before("gorm:commit_or_rollback_transaction"). The README now states that this holds for single writes too.
• Commit: 6f50b6c. RC-05 re-proves it.

2. [Rule 1 - Bug] After-commit callbacks got a handle carrying the written model's statement (deferred item b from 11-05)

• Found during: Task 1
• Issue: flushStatementAfterCommit, Transaction and the immediate AfterCommit path passed Session{NewDB, Context} or the callback's own handle, and both keep the write's statement.
• RED: TestTransactionAfterCommit/callback_handle_has_a_clean_statement ran SELECT "lagoon_ac_items"."id","lagoon_ac_items"."label" for a query on another model. In the plain-transaction case it also aborted the caller's transaction.
• Fix: one cleanHandle helper (Session{NewDB, Context}, Clauses(), Session{NewDB}) serves all three paths. The old identity assertion in plain_gorm_transaction_runs_now now checks for the same connection instead. README updated.
• Commit: c544319

3. [Rule 1 - Bug] A swallowed read failure left the caller's transaction aborted

• Found during: Task 2 (TestSyncFailuresNonFatal)
• Issue: beachcomber's and lighthouse's savepoint helpers rolled back only when the inner function returned an error. fonoteka's settings Gate treats a failed read as "off" and returns no error, and so do channel functions and the lighthouse delete snapshot. In those cases the helper released the savepoint on an aborted transaction, and the caller's next statement failed with 25P02. Both READMEs promised that a failed read never aborts the caller's transaction.
• RED: failed_gate_read_keeps_the_callers_transaction and TestBroadcastSwallowedReadFailure both failed with 25P02.
• Fix: when RELEASE SAVEPOINT fails, the helper rolls back to the savepoint and then releases it.
• Commit: 6df43d4. This commit also carries the beachcomber harness and sync tests, because those tests hold the RED case and every commit must stay green.

4. [Rule 3 - Blocking] check-phase10.1.sh --all failed on the 11-06 pending goldens

• Found during: plan verification ("check-phase10.1.sh --all still passes")
• Issue: the 10.1 detector refuses every skip. Since 11-06, TestBroadcastGoldens/created and /updated skip by design until Phase 12.
• Fix: the 10.1 detector accepts exactly those two skips, and only when their output carries "pending: Phase 12". Two self-test cases cover this. The gate is not weakened in general.
• Commit: 73cfed7

Plan wording and additions

• TestQueueClear kept its name. The new batch and state test is named TestClearQueueStatesAndBatches, so that grep -c 'func TestQueueClear' still prints 1.
• A TestSync*-named test (TestSyncEngineRegistration) was added to beachcomber/typesense. The plan's go test ./modules/beachcomber/... -run '^TestSync.*$' covers that package too and would otherwise print "no tests to run", which the plan's fails_when rejects.
• The gate's --named stage uses go test -json instead of -v text. It gives the same per-test pass and skip evidence and is parsed more reliably.
• --removal is an extra gate mode, and T-11-SC gets a removal check (RC-13) in addition to the nine threats the plan listed. T-11-02 and T-11-05 get several checks each: 13 RC rows in total.
• TestRegistry moved from channel_test.go to registry_test.go, and the conga tests moved out of listen_test.go. TestJobManagerOperations, TestAttemptOutcomes, TestCancelJob and TestQueueWorkCommand became the plan's TestManagerPHPSemantics, TestOutcome*, TestCancel* and TestQueueWork.

---

Total deviations: 4 auto-fixed (3 bugs, 1 blocking), plus the notes above. Impact on plan: the fixes change behaviour only where the code broke its documented contract. No exported API changed, and each affected README was updated in the same commit.

TDD Gate Compliance

This is a test plan for code that already existed, so most tests passed on their first run by design. Their failing-when-broken evidence is the 13 removal checks. Each of the three behaviour fixes went RED first, with the failure recorded above, and then GREEN in a single fix commit, because green-at-every-commit forbids a failing test commit.

Issues Encountered

• golang-jwt HTML-escapes <, > and & inside claims, while PHP leaves them raw. The token is opaque to Centrifugo and the decoded claims are equal, so the token test uses a/b. This is recorded as a decision, not changed.
• gofmt -l modules/ still lists pre-existing drift in compass, party, tide and wristband files that this plan did not touch. It is out of scope.

Known Stubs

None. The created and updated broadcast goldens remain pending for Phase 12, as 11-06 intended (already in .planning/WINDOWS.md).

Threat Flags

None. No new endpoint, auth path or schema was added; the only production changes harden existing transaction handling.

User Setup Required

None.

Next Phase Readiness

• Phase 12 must:
  • turn TestBroadcastGoldens/created and /updated into assertions, then remove them from GOLDEN_SKIPS in check-phase11.sh and APP_PENDING_SKIPS in check-phase10.1.sh (both gates refuse a pending skip that passes);
  • re-gate SearchIDs results in SQL.
• Phase 13 must scope summer_jobs ids through the owning import (T-11-08, accepted).
• Manual backstops for /gsd-verify-work: the Nuxt app receiving an album event through a real Centrifugo v6, and a real Typesense 26.0 receiving an upsert.

Self-Check: PASSED

• All 23 created files and 16 modified files listed in key-files exist on disk.
• summercms.go commits c544319, 6f50b6c, 35ac96d, 33194a1, 6dadbf6, 6df43d4, 18d3097, 11b5b4c, e55d234, 7743487, a34c6ec and 73cfed7 exist, and so do fonoteka.go commits 1657cc1 and c222e03.
• scripts/check-phase11.sh --all printed "phase11 all passed" (rc 0). scripts/check-phase11.sh --removal printed "phase11 removal passed" with 13 of 13 failing as required. scripts/check-phase10.1.sh --all printed "phase10.1 all passed".
• Acceptance greps: 31 threat rows, 13 RC rows, nyquist_compliant: true once, TestWsAuthorizer 1, TestProxy 1, func TestQueueClear 1, func TestCancelRunningCancelsCtx 1, func TestQueueMigrationsUpDown 1, Europe/Warsaw 2; conga coverage 92.5%.