summercms/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-08-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, actuals, plan_head_before, plan_head_after, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, coverage, duration, completed, status

phase	plan	subsystem	tags	requires	provides	affects	actuals	plan_head_before	plan_head_after	tech-stack	key-files	key-decisions	patterns-established	requirements-completed	coverage	duration	completed	status
11-jobs-realtime-and-search-infrastructure	08	database	lagoon after-commit cabana beachcomber typesense gorm transactions gap-closure	phase provides 11-jobs-realtime-and-search-infrastructure 11-05 beachcomber sync over lagoon.AfterCommit and the fonoteka Album binding; 11-07 clean after-commit handles, the savepoint fix and check-phase11.sh	Cabana create/update/delete/bulk-delete and relation Link/Unlink run in lagoon.Transaction, so after-commit work sees the committed row and pivots fonoteka SaveAlbum runs in lagoon.Transaction; the buffered search reload happens after the ordered artist pivots commit lagoon.AfterCommit warns and skips inside a foreign plain GORM *sql.Tx; a nested lagoon.Transaction over a root handle returns an error (WR-03) Regression tests: TestAlbumsAdminSearchUsesCommittedArtists (assembled admin router) and TestSaveAlbumDefersAfterCommitUntilArtistsSync; inverted plain-transaction tests in lagoon, beachcomber and fonoteka T-11-31 and T-11-32 in 11-SECURITY-REVIEW.md, with removal check RC-14 in check-phase11.sh --removal	11 verification re-run (CR-01) 12 albums API (SearchIDs SQL re-gate) any code that writes searchable models inside its own plain GORM transaction	tokens tasks commits 5988 3 7	9033d81721	e6777bda97	added patterns Write services that trigger after-commit work run their whole unit (row plus pivots) in lagoon.Transaction and use the callback's ctx and tx throughout lagoon.AfterCommit has four documented cases: lagoon-managed (buffered), implicit single statement (buffered on the statement), foreign *sql.Tx (warned and skipped), outside a transaction (immediate)	created modified .planning/phases/11-jobs-realtime-and-search-infrastructure/11-08-SUMMARY.md modules/cabana/crud.go modules/cabana/relation.go modules/lagoon/transaction.go modules/lagoon/transaction_test.go modules/lagoon/README.md modules/beachcomber/sync_test.go modules/beachcomber/README.md scripts/check-phase11.sh .planning/phases/11-jobs-realtime-and-search-infrastructure/11-SECURITY-REVIEW.md ../fonoteka.go/plugins/golem15/fonoteka/admin_albums_test.go ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service_db_test.go	lagoon.AfterCommit inside a foreign plain GORM transaction warns and skips the callback instead of running it immediately: Lagoon cannot observe that commit, and running early leaked uncommitted or rolled-back state to Typesense A nested lagoon.Transaction given a root handle returns an error rather than being treated as a top-level transaction (WR-03 option 'return an error') T-11-31 gets its own removal check (RC-14) so check-phase11.sh --evidence accepts the 11-08 threat register; T-11-32 (medium) is proven by two hand-run mutations recorded in the review	A gap plan that adds threat IDs must also add their rows to the phase security review, because the phase gate reads every 11-0*-PLAN.md threat register	SRCH-01	id description requirement verification human_judgment D1 A Cabana admin edit that replaces an Album's artists commits the submitted pivot order and sends Typesense exactly one import whose artist_ids equal that committed order SRCH-01 kind ref status integration ../fonoteka.go/plugins/golem15/fonoteka/admin_albums_test.go#TestAlbumsAdminSearchUsesCommittedArtists pass kind ref status integration go test ./modules/cabana -count=1 pass false id description requirement verification human_judgment D2 lagoon.AfterCommit refuses a foreign plain GORM transaction with a warning; a rolled-back plain transaction reaches no search engine; lagoon-managed and implicit single-statement commits still sync; a nested lagoon.Transaction over a root handle errors without running SRCH-01 kind ref status integration go test ./modules/lagoon -run '^(TestTransactionAfterCommit|TestTransactionEdges)$' -count=1 -v pass kind ref status integration go test ./modules/beachcomber -run '^TestSyncAfterCommit$' -count=1 -v pass kind ref status integration ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go#TestAlbumSearchDeleteAndFailures pass kind ref status other PHASE11_RC=RC-14 scripts/check-phase11.sh --removal pass false id description requirement verification human_judgment D3 SaveAlbum runs after-commit callbacks only after the ordered artist pivots commit, and none on rollback; validation and market-price provenance tests unchanged and green SRCH-01 kind ref status integration cd ../fonoteka.go && go test ./plugins/golem15/fonoteka/classes -run '^TestSaveAlbum' -count=1 -v pass false id description verification human_judgment D4 Phase 11 gate green in both repositories with the 11-08 threats in the security review kind ref status other bash scripts/check-phase11.sh --all pass false	unknown (executor stopped before its summary step; see Performance)	2026-09-30	complete

Phase 11 Plan 08: Commit-safe Album search sync (gap CR-01) Summary

Cabana writes and fonoteka SaveAlbum now run in lagoon.Transaction, so Typesense receives an Album only after its row and ordered artist pivots commit. lagoon.AfterCommit warns and skips inside a foreign plain GORM transaction, so a rollback there never reaches the search engine. A nested lagoon.Transaction over a root handle is an error. check-phase11.sh --all prints "phase11 all passed".

This summary was written in a manual close-out. The 11-08 executor committed all three tasks and then stopped before its summary step. The close-out re-ran every verify command, the phase gate and both repositories' full suites. It fixed the gaps it found in three separate fix(11-08) commits.

Performance

• Duration: not measured. The executor recorded no start time or commit ledger. Its task commits are timestamped 2026-09-30T18:14:23Z to 18:14:42Z. The follow-up test commit is at 18:49:31Z and the close-out ran from about 18:50Z.
• Completed: 2026-09-30
• Tasks: 3 of 3
• Files modified: 13 (9 in summercms.go including the security review, 4 in fonoteka.go), plus this summary and the planning state files

Accomplishments

• Task 1 (tracer): an admin artist edit indexes the committed ordered pivots. CRUDService.save, Delete and BulkDelete (crud.go) and RelationService.Link and Unlink (relation.go) call lagoon.Transaction and use its ctx and tx. The read-only ShowRecord (crud.go:255) and relation query (relation.go:449) transactions are unchanged. TestAlbumsAdminSearchUsesCommittedArtists drives the router built by surf.Assemble with a PUT that replaces the artists. It asserts that the committed pivot order and the single Typesense import's artist_ids are the same.
• Task 2: refuse external after-commit work inside unmanaged transactions. AfterCommit keeps its lagoon-buffer and implicit-transaction branches. When the statement's connection pool is a gorm.TxCommitter that neither branch covers, it now logs lagoon: after-commit callback skipped inside unmanaged transaction and returns. Transaction returns lagoon: nested transaction requires the parent transaction handle when a parent buffer exists but the handle is not transactional. Tests that used to expect immediate plain-transaction sync now assert the opposite in lagoon, beachcomber and fonoteka. The lagoon README describes all four cases.
• Task 3: SaveAlbum defers sync until the ordered artists commit. SaveAlbum uses lagoon.Transaction, and the save-before-pivot order is unchanged. TestSaveAlbumDefersAfterCommitUntilArtistsSync registers a create callback that reads the pivots after commit and sees [second, first]. Its duplicate-pivot rollback case runs no callback and commits no Album.

Task Commits

summercms.go:

1. f7b6b0c: fix(11-08), make cabana writes commit-safe (Task 1)
2. a33b1ad: fix(11-08), refuse unmanaged after-commit work (Task 2)
3. 2766f34: test(11-08), keep sync failure coverage managed (Task 2 follow-up: TestSyncFailuresNonFatal/failed_document_read_inside_a_transaction moved to lagoon.Transaction; with a33b1ad alone that test fails)
4. 8e0083e: fix(11-08), document foreign and nested transaction refusal (close-out)
5. d4fc957: fix(11-08), add the T-11-31 removal check to the phase gate (close-out)
6. e6777bd: fix(11-08), record T-11-31 and T-11-32 in the security review (close-out)

fonoteka.go:

1. 1c88199: fix(11-08), defer album sync until committed (Tasks 1 and 3 tests, Task 2 fonoteka test, SaveAlbum)

Not part of 11-08: 6f57604 docs(11.1): create phase plan, which landed between the task commits.

Plan metadata: docs(11-08): complete gap plan summary (this commit)

Files Created/Modified

See key-files in the frontmatter. The production changes are modules/cabana/crud.go, modules/cabana/relation.go, modules/lagoon/transaction.go and ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go. The rest are tests, the lagoon and beachcomber READMEs, the gate script and the security review.

Verification (run in the close-out, 2026-09-30)

Postgres-backed tests ran on testcontainers (Docker). Nothing was skipped except the two broadcast goldens that the gate allows (TestBroadcastGoldens/created and /updated, pending Phase 12).

Command	Result
Task 1: go test ./modules/cabana -count=1	ok .../modules/cabana 17.754s
Task 1: (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^TestAlbumsAdminSearchUsesCommittedArtists$' -count=1 -v)	--- PASS: TestAlbumsAdminSearchUsesCommittedArtists (0.86s)
Task 2: go test ./modules/lagoon -run '^(TestTransactionAfterCommit|TestTransactionEdges)$' -count=1 -v	--- PASS: TestTransactionAfterCommit (0.13s), --- PASS: TestTransactionEdges (0.08s)
Task 2: go test ./modules/beachcomber -run '^TestSyncAfterCommit$' -count=1 -v	PASS, 6 subtests including plain_gorm_transaction_rollback_sends_nothing
Task 2: (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^TestAlbumSearchDeleteAndFailures$' -count=1 -v)	PASS, 10 subtests including a_foreign_gorm_transaction_rollback_never_reaches_Typesense
Task 3: (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka/classes -run '^TestSaveAlbum' -count=1 -v)	9 PASS: TestSaveAlbumDefersAfterCommitUntilArtistsSync plus the 8 existing year and market-price provenance tests
Task 3: go test ./plugins/golem15/fonoteka -run '^(TestAlbumsAdminSearchUsesCommittedArtists|TestAlbumSearchDeleteAndFailures)$'	both PASS
go vet ./... and go test ./... -count=1 in summercms.go	vet clean; 29 packages ok, 0 FAIL
go vet and go test ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/... -count=1 in fonoteka.go	vet clean; 15 packages ok, 7 with no test files, 0 FAIL
bash scripts/check-phase11.sh --all (first run, on the executor's commits)	self-test, hygiene, go, postgres and named passed; evidence refused: "review has 0 threat rows for T-11-31, want 1"
bash scripts/check-phase11.sh --all (after 8e0083e, d4fc957, e6777bd)	phase11 all passed
PHASE11_RC=RC-14 bash scripts/check-phase11.sh --removal	RC-14 T-11-31 modules/lagoon/transaction.go: fails as required: TestTransactionAfterCommit, .../plain_gorm_transaction_is_refused, .../callback_handle_has_a_clean_statement; restored
Hand mutation: Cabana save back to a plain s.DB.Transaction	TestAlbumsAdminSearchUsesCommittedArtists fails: imports=[] calls=[], want one document; file restored
Hand mutation: SaveAlbum back to a plain gdb.Transaction	TestSaveAlbumDefersAfterCommitUntilArtistsSync fails: callbacks=0 observed=[], want one callback with [3 2]; file restored

Acceptance criteria check:

• Task 1: crud.go has exactly three lagoon.Transaction write entry points and relation.go has two. The read-only sites are untouched. The test uses the assembled HTTP router, and it compares the import with pivots read on the committed connection after the request. Pass.
• Task 2: the plain-transaction tests assert zero callbacks or engine calls, and the lagoon one also asserts the warning. The rollback tests see no external request. The nested root-handle case errors without entering its function. The lagoon README covers lagoon-managed, implicit statement, foreign transaction and outside a transaction. Pass. The nested root-handle error was missing from the README and the Transaction doc comment until 8e0083e.
• Task 3: SaveAlbum calls lagoon.Transaction with its callback ctx. The new test sees the submitted order and no callback on rollback. The existing validation and provenance tests are green. Pass.

Decisions Made

See key-decisions in the frontmatter.

Deviations from Plan

Auto-fixed Issues

1. [Rule 1 - Bug] TestSyncFailuresNonFatal still relied on the old plain-transaction behaviour

• Found during: Task 2, after a33b1ad
• Issue: failed_document_read_inside_a_transaction wrote inside gdb.Transaction and expected a "build document" warning. After the refusal no sync ran, so the test failed. This was confirmed in the close-out by restoring the a33b1ad version of the file.
• Fix: the subtest writes through lagoon.Transaction. This was left uncommitted when the executor stopped and committed as 2766f34 during the close-out window.
• Commit: 2766f34

2. [Rule 2 - Missing critical] Documentation contradicted the new behaviour (CLAUDE.md module README rule)

• Found during: close-out review of Task 2
• Issue: modules/beachcomber/README.md still said the sync "runs immediately through the transaction's handle" inside a plain GORM transaction. lagoon.Transaction's doc comment and the lagoon README did not mention the new nested root-handle error.
• Fix: the beachcomber README now says the sync is warned and skipped there, and points to lagoon.Transaction or an explicit Sync after commit. The lagoon README and doc comment state the nested root-handle error.
• Files modified: modules/beachcomber/README.md, modules/lagoon/README.md, modules/lagoon/transaction.go (comment only)
• Commit: 8e0083e

3. [Rule 3 - Blocking] The phase gate refused 11-08's threat register

• Found during: plan verification (check-phase11.sh --all)
• Issue: the evidence stage reads the threat table of every 11-0*-PLAN.md and needs one review row per threat, plus a removal check for each high mitigated threat. T-11-31 (high) and T-11-32 (medium) had no rows.
• Fix: RC-14 in removal_table. It removes the foreign-transaction refusal and requires TestTransactionAfterCommit to fail, which it does. The review got rows for T-11-31 and T-11-32, the RC-14 row, and a CR-01 row in the fixes table.
• Commits: d4fc957 (script), e6777bd (review)

---

Total deviations: 3 (1 bug, 1 missing documentation, 1 blocking gate failure). No production behaviour was changed in the close-out; the only code change is a doc comment.

TDD Gate Compliance

The executor committed each task as a single fix commit, with its tests in the same commit, so the RED run is not in history. The close-out replaced that evidence with mutations. RC-14 covers the Task 2 refusal. Hand mutations of the Cabana save transaction and of SaveAlbum make the Task 1 and Task 3 tests fail. Restoring the a33b1ad version of sync_test.go makes TestSyncFailuresNonFatal fail.

Issues Encountered

• The fonoteka subtest an album without a positive collection_id is logged and never sent now asserts that no warning is logged, because its write happens in a rolled-back foreign transaction. Its name no longer matches its assertion. The zero-collection refusal is still covered by ToSearchableArray in the same subtest and by TestAlbumSearchable/collection_id_zero_is_refused (RC-07). The warning path for a failed build is covered by beachcomber TestSyncFailuresNonFatal. The subtest was not renamed; this is a cosmetic follow-up.
• Code that writes searchable models inside its own plain gorm transaction now gets no search sync. A Warn log appears instead of a silent skip. No production call site was found besides the Cabana and SaveAlbum paths this plan converted, and the full suites in both repositories pass.

Known Stubs

None.

Threat Flags

None. No new endpoint, auth path, file access or schema. The changes narrow when data leaves an open transaction.

User Setup Required

None.

Next Phase Readiness

• Phase 11 verification can be re-run to re-check CR-01. This close-out did not run it and did not mark the phase complete.
• Phase 12 still owns the SQL re-gate of SearchIDs results. Concurrent commits for the same Album are not serialized (flagged SRCH-01 assumption).
• RT-03 is not newly claimed: the Album updated broadcast payload timing on admin edits keeps the assumptions recorded in 11-03.

Self-Check: PASSED

• All modified files listed in key-files exist on disk.
• summercms.go commits f7b6b0c, a33b1ad, 2766f34, 8e0083e, d4fc957 and e6777bd exist, and so does fonoteka.go commit 1c88199.