summercms/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-08-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, gap_closure, requirements, estimate, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	gap_closure	requirements	estimate	must_haves
11-jobs-realtime-and-search-infrastructure	08	execute	6	11-07	modules/cabana/crud.go modules/cabana/relation.go modules/lagoon/transaction.go modules/lagoon/transaction_test.go modules/lagoon/README.md modules/beachcomber/sync_test.go ../fonoteka.go/plugins/golem15/fonoteka/admin_albums_test.go ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service_db_test.go	true	true	SRCH-01	tokens raw_tokens tasks confidence 50000 50000 3 low	truths artifacts key_links prohibitions Per D-20 and SRCH-01, a Cabana album edit that changes artists sends Typesense exactly the committed ordered artist_ids, after the album row and pivots commit. Per D-20, Cabana create/update/delete/bulk-delete and relation Link/Unlink writes, plus fonoteka SaveAlbum, use lagoon.Transaction and pass its transaction context through every write and callback. lagoon.AfterCommit never runs external work inside a foreign plain GORM *sql.Tx: it warns and skips; a rollback therefore produces zero search-engine calls. Lagoon-managed transactions and implicit single-statement GORM transactions retain their existing after-commit behavior, while a nested lagoon.Transaction over a root handle fails instead of opening an independent transaction under the parent buffer. path provides contains modules/cabana/crud.go Commit-safe Cabana create, update, delete, and bulk-delete writes lagoon.Transaction path provides contains modules/cabana/relation.go Commit-safe Cabana relation Link and Unlink writes lagoon.Transaction path provides contains modules/lagoon/transaction.go Unmanaged *sql.Tx refusal in AfterCommit and safe nested-transaction detection func AfterCommit path provides contains ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go SaveAlbum with after-commit work delayed until ordered pivots commit lagoon.Transaction from to via pattern modules/cabana/crud.go modules/lagoon/transaction.go CRUDService write methods call lagoon.Transaction with the callback-supplied context lagoon.Transaction from to via pattern modules/beachcomber/sync.go modules/lagoon/transaction.go Search callbacks register through lagoon.AfterCommit and reload only after commit lagoon.AfterCommit from to via pattern ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go ../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go SaveAlbum commits artist pivots before the buffered search reload builds artist_ids syncArtists requirement_id category statement status flagged SRCH-01 consent Search synchronization MUST NOT send an external request for database state that is still uncommitted or is later rolled back unverified true

Phase Goal

ROADMAP Phase 11 goal remains the contract; it is not written as a user story. This gap slice closes its failed Typesense criterion by making the framework's real Album write paths commit-safe.

Close verifier gap CR-01: make Album search synchronization observe committed relation state and make unmanaged transaction use fail safe.

Purpose: D-20 promises an inline, non-fatal sync built from the committed row. Cabana and SaveAlbum currently save the Album before artist pivots and use plain GORM transactions, so Typesense can receive stale or rolled-back state. Output: commit-safe Cabana and SaveAlbum transactions, a foreign-transaction guard in lagoon.AfterCommit, and regression tests for committed artist_ids and rollback silence.

<execution_context> @/home/jin/.codex/gsd-core/workflows/execute-plan.md @/home/jin/.codex/gsd-core/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md @.planning/phases/11-jobs-realtime-and-search-infrastructure/11-VERIFICATION.md @.planning/phases/11-jobs-realtime-and-search-infrastructure/11-01-SUMMARY.md @.planning/phases/11-jobs-realtime-and-search-infrastructure/11-05-SUMMARY.md @.planning/phases/11-jobs-realtime-and-search-infrastructure/11-07-SUMMARY.md

Artifacts this phase produces

No new production files, schema objects, exported symbols, or dependencies. Changed symbols: cabana.CRUDService.save, cabana.CRUDService.Delete, cabana.CRUDService.BulkDelete, cabana.RelationService.Link, cabana.RelationService.Unlink, lagoon.Transaction, lagoon.AfterCommit, and classes.SaveAlbum. New test symbols: TestAlbumsAdminSearchUsesCommittedArtists and TestSaveAlbumDefersAfterCommitUntilArtistsSync.

Flagged assumptions (spec-less fallback: unresolved)

• SRCH-01 remains unclassified: this closure guarantees that each sync reads its own committed row and ordered pivots, but does not serialize concurrent commits for the same Album; the Phase 12 SQL re-gate remains the safety boundary.
• RT-03 remains unclassified and is not newly claimed by this plan. The same transaction buffering corrects timing for Album update callbacks, but batch zero-primary-key writes and restore behavior retain the unresolved assumptions already recorded in 11-03.

Task 1: Admin artist edit indexes the committed ordered pivots modules/cabana/crud.go, modules/cabana/relation.go, ../fonoteka.go/plugins/golem15/fonoteka/admin_albums_test.go, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go modules/cabana/crud.go, modules/cabana/relation.go, modules/cabana/relation_field.go, modules/lagoon/transaction.go, ../fonoteka.go/plugins/golem15/fonoteka/admin_albums_test.go, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go, ../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VERIFICATION.md - A real Cabana HTTP update replacing an Album's artists returns success, commits the submitted pivot order, and produces one Typesense upsert whose artist_ids equal that committed order. - Create/update/delete/bulk-delete and relation Link/Unlink keep their existing response, validation, scoping, lifecycle-hook, and rollback semantics. Write `TestAlbumsAdminSearchUsesCommittedArtists` red first. Reuse the fake Typesense and search app harness; let that harness expose its activated plugins so the test can assemble the real admin router. Create an allowed backend admin, its active collection, two artists, and an Album with an initial artist; reset recorded engine calls; update the Album through the Cabana HTTP route with a different ordered artist list. Assert both the committed pivot rows and the import document's `artist_ids` have the submitted order.

Per D-20, replace only Cabana's write transactions with lagoon.Transaction: CRUDService.save, Delete, BulkDelete, and RelationService.Link/Unlink. Use the callback-supplied context for loads, validation, lifecycle hooks, relation scope checks, pivot writes, and deletes so lagoon.AfterCommit can see the buffer. Leave read-only ShowRecord and relation queries unchanged. go test ./modules/cabana -count=1 && (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^TestAlbumsAdminSearchUsesCommittedArtists$' -count=1 -v) <fails_when>Either command exits non-zero; the named admin test is absent, skipped, reports no tests to run, or its indexed artist_ids differ from the committed pivot order.</fails_when> <acceptance_criteria> - Cabana has exactly three write transaction entry points in crud.go and two in relation.go, all through lagoon.Transaction; read-only transaction sites remain unchanged. - TestAlbumsAdminSearchUsesCommittedArtists exercises the assembled HTTP router, not CRUDService directly. - The test proves the engine call occurs only after the ordered pivots are visible on the committed connection. </acceptance_criteria> An admin can change an Album's artists and the resulting Typesense document contains the committed ordered artist_ids.

Task 2: Refuse external after-commit work inside unmanaged transactions modules/lagoon/transaction.go, modules/lagoon/transaction_test.go, modules/lagoon/README.md, modules/beachcomber/sync_test.go, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go modules/lagoon/transaction.go, modules/lagoon/transaction_test.go, modules/lagoon/README.md, modules/beachcomber/sync.go, modules/beachcomber/sync_test.go, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-REVIEW.md (CR-01 and WR-03) - AfterCommit inside a foreign plain GORM *sql.Tx logs a warning and does not invoke the callback. - Rolling that foreign transaction back produces zero engine calls. - Lagoon-managed commits and implicit single-statement commits still flush once; rollbacks flush none; direct calls outside a transaction still run immediately. - A nested lagoon.Transaction given the root handle while its context carries a parent buffer returns an error and neither commits independent work nor attaches callbacks to the parent. Preserve the lagoon-buffer branch and GORM implicit-transaction branch. Before the immediate fallback, detect a foreign `*sql.Tx` through the statement connection pool, emit a stable Warn without row data or secrets, and return without running the callback. Tighten `Transaction` so a parent buffer uses the nested/savepoint branch only when the supplied handle is actually transactional; reject a root handle rather than opening an independently committed transaction under the parent's callback buffer (WR-03, directly implicated by adopting lagoon.Transaction in services). Update the API comment and README.

Invert the tests that currently bless immediate plain-GORM sync. In lagoon, assert callback suppression plus the warning. In beachcomber, use a plain GORM transaction that writes and then rolls back and assert zero fake-engine calls. In fonoteka's search smoke test, assert a foreign transaction never sends to Typesense. Retain explicit regression coverage for the implicit single-statement and lagoon-managed paths. go test ./modules/lagoon -run '^(TestTransactionAfterCommit|TestTransactionEdges)$' -count=1 -v && go test ./modules/beachcomber -run '^TestSyncAfterCommit$' -count=1 -v && (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^TestAlbumSearchDeleteAndFailures$' -count=1 -v) <fails_when>Any command exits non-zero; a named test is absent, skipped, or reports no tests to run; a foreign transaction invokes an after-commit callback or records an engine call; an implicit or lagoon-managed commit stops syncing.</fails_when> <acceptance_criteria> - The plain-transaction tests assert zero callbacks/engine calls and a warning, replacing the old immediate-sync expectation. - The rollback test observes no committed row and no external request. - Nested lagoon.Transaction with a root handle cannot independently commit under a parent buffer. - modules/lagoon/README.md describes all four cases: lagoon-managed, implicit statement, foreign transaction, and outside a transaction. </acceptance_criteria> Unmanaged transactions cannot leak uncommitted state externally, while every supported after-commit path keeps its documented behavior.

Task 3: SaveAlbum defers sync until ordered artists commit ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service_db_test.go ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service_db_test.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/artist_resolver.go, modules/lagoon/transaction.go, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VERIFICATION.md - SaveAlbum validates, resolves artists, saves the Album, replaces pivots, commits, and only then runs after-commit callbacks. - A pivot or validation failure rolls back the Album and runs no callback. Per D-20, replace SaveAlbum's plain GORM transaction with `lagoon.Transaction`, using the callback-supplied context throughout validation and writes. Keep the existing save-before-pivot order; buffering now makes the post-commit reload occur after `syncArtists` and commit. Add `TestSaveAlbumDefersAfterCommitUntilArtistsSync`, registering a test callback that proves it observes the committed ordered pivots, plus a rollback case proving it never runs. Do not add a second save/touch and do not change validation, price provenance, or artist resolution semantics. (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka/classes -run '^TestSaveAlbum' -count=1 -v && go test ./plugins/golem15/fonoteka -run '^(TestAlbumsAdminSearchUsesCommittedArtists|TestAlbumSearchDeleteAndFailures)$' -count=1 -v) Either command exits non-zero; the new SaveAlbum test or either gap regression test is absent, skipped, reports no tests to run, sees pre-pivot artist state, or runs a callback after rollback. - SaveAlbum calls lagoon.Transaction and uses its callback context. - TestSaveAlbumDefersAfterCommitUntilArtistsSync observes the submitted pivot order from the callback and zero callbacks on rollback. - Existing SaveAlbum validation and market-price provenance tests remain green. Both real Album write paths expose only committed row-and-pivot state to after-commit consumers.

<threat_model>

Trust Boundaries

Boundary	Description
Open SQL transaction -> external search engine	Album and relation data must not leave the database before commit
Cabana admin request -> Album row and pivots	One user edit spans scalar and ordered relation writes that must commit atomically

STRIDE Threat Register (ASVS L1; blocking threshold high)

Threat ID	Category	Component	Severity	Disposition	Mitigation Plan
T-11-31	Information Disclosure	lagoon.AfterCommit / beachcomber sync	high	mitigate	Buffer supported writes until commit, refuse foreign *sql.Tx callbacks, and prove a rollback causes zero engine calls (Tasks 1-2).
T-11-32	Tampering	Cabana and SaveAlbum ordered artist pivots	medium	mitigate	Route complete row-and-pivot units through lagoon.Transaction and compare the indexed artist_ids with committed pivot order (Tasks 1 and 3).
T-11-SC	Tampering	npm/pip/cargo installs	high	mitigate	No package-manager install occurs in this gap plan; existing dependency lockfiles remain unchanged.
</threat_model>

Run the three focused task commands, then run `bash scripts/check-phase11.sh --all` from the project root.

The phase gate fails if either repository's vet/tests fail, a named test is skipped or missing, or it does not print phase11 all passed.

<success_criteria>

• The Cabana admin artist-edit regression indexes the committed ordered pivots.
• Plain GORM transaction rollback reaches no search engine, and AfterCommit warns instead of executing in that foreign transaction.
• Cabana write methods and SaveAlbum use lagoon.Transaction without changing their HTTP, validation, lifecycle, or relation contracts.
• All focused tests and the existing Phase 11 gate pass; no UI-SPEC, UI file, schema file, dependency, or runtime mirror is created. </success_criteria>

Multi-Source Coverage Audit (gap-closure scope)

Source	ID	Gap item	Plan	Status
GOAL	SC-5	Typesense sync reflects committed Album state	11-08	COVERED
REQ	SRCH-01	Album documents sync correctly and safely	11-08	COVERED
RESEARCH	Pattern 10	after-commit buffer and rollback silence	11-08	COVERED
CONTEXT	D-20	after-commit, inline, non-fatal committed-row sync	11-08	COVERED
VERIFICATION	CR-01	Cabana/SaveAlbum transaction paths, foreign-tx refusal, regression tests	11-08	COVERED

All other Phase 11 goals, requirements, research items, and D-01..D-19 decisions are excluded from this gap-closure audit because 11-VERIFICATION.md already marks them verified or out of this failed truth's scope. Deferred ideas remain excluded.

Create `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-08-SUMMARY.md` when done.