diff --git a/.planning/STATE.md b/.planning/STATE.md index 46d4085..aa15245 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -3,16 +3,16 @@ gsd_state_version: "1.0" milestone: v1.0 current_phase: 11 current_phase_name: Jobs, realtime and search infrastructure -status: verifying +status: executing stopped_at: Completed 11-07-PLAN.md -last_updated: "2026-09-30T12:57:19.775Z" +last_updated: "2026-09-30T14:09:27.946Z" last_activity: 2026-09-29 last_activity_desc: Phase 11 execution started -state_head: 45fb00af1a32f9fa48f9fe8cd122da3857091234 +state_head: dd97fd12a60a31067c6a0d254161d31e36b90c6c progress: total_phases: 19 completed_phases: 9 - total_plans: 85 + total_plans: 86 completed_plans: 85 milestone_name: milestone --- @@ -28,9 +28,9 @@ See: .planning/PROJECT.md (updated 2026-09-16) ## Current Position -Phase: 11 (Jobs, realtime and search infrastructure) — EXECUTING +Phase: 11 (Jobs, realtime and search infrastructure) — READY TO EXECUTE Plan: 7 of 7 -Status: Phase complete — ready for verification +Status: Ready to execute Last activity: 2026-09-29 — Phase 11 execution started Progress: [██████░░░░] 60% diff --git a/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-08-PLAN.md b/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-08-PLAN.md new file mode 100644 index 0000000..0de1928 --- /dev/null +++ b/.planning/phases/11-jobs-realtime-and-search-infrastructure/11-08-PLAN.md @@ -0,0 +1,220 @@ +--- +phase: 11-jobs-realtime-and-search-infrastructure +plan: 08 +type: execute +wave: 6 +depends_on: ["11-07"] +files_modified: + - modules/cabana/crud.go + - modules/cabana/relation.go + - modules/lagoon/transaction.go + - modules/lagoon/transaction_test.go + - modules/lagoon/README.md + - modules/beachcomber/sync_test.go + - ../fonoteka.go/plugins/golem15/fonoteka/admin_albums_test.go + - ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service_db_test.go +autonomous: true +gap_closure: true +requirements: [SRCH-01] +estimate: + tokens: 50000 + raw_tokens: 50000 + tasks: 3 + confidence: low +must_haves: + truths: + - "Per D-20 and SRCH-01, a Cabana album edit that changes artists sends Typesense exactly the committed ordered artist_ids, after the album row and pivots commit." + - "Per D-20, Cabana create/update/delete/bulk-delete and relation Link/Unlink writes, plus fonoteka SaveAlbum, use lagoon.Transaction and pass its transaction context through every write and callback." + - "lagoon.AfterCommit never runs external work inside a foreign plain GORM *sql.Tx: it warns and skips; a rollback therefore produces zero search-engine calls." + - "Lagoon-managed transactions and implicit single-statement GORM transactions retain their existing after-commit behavior, while a nested lagoon.Transaction over a root handle fails instead of opening an independent transaction under the parent buffer." + artifacts: + - path: "modules/cabana/crud.go" + provides: "Commit-safe Cabana create, update, delete, and bulk-delete writes" + contains: "lagoon.Transaction" + - path: "modules/cabana/relation.go" + provides: "Commit-safe Cabana relation Link and Unlink writes" + contains: "lagoon.Transaction" + - path: "modules/lagoon/transaction.go" + provides: "Unmanaged *sql.Tx refusal in AfterCommit and safe nested-transaction detection" + contains: "func AfterCommit" + - path: "../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go" + provides: "SaveAlbum with after-commit work delayed until ordered pivots commit" + contains: "lagoon.Transaction" + key_links: + - from: "modules/cabana/crud.go" + to: "modules/lagoon/transaction.go" + via: "CRUDService write methods call lagoon.Transaction with the callback-supplied context" + pattern: "lagoon\\.Transaction" + - from: "modules/beachcomber/sync.go" + to: "modules/lagoon/transaction.go" + via: "Search callbacks register through lagoon.AfterCommit and reload only after commit" + pattern: "lagoon\\.AfterCommit" + - from: "../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go" + to: "../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go" + via: "SaveAlbum commits artist pivots before the buffered search reload builds artist_ids" + pattern: "syncArtists" + prohibitions: + - requirement_id: SRCH-01 + category: consent + statement: "Search synchronization MUST NOT send an external request for database state that is still uncommitted or is later rolled back" + status: unverified + flagged: true +--- + +## Phase Goal + +ROADMAP Phase 11 goal remains the contract; it is not written as a user story. This gap slice closes its failed Typesense criterion by making the framework's real Album write paths commit-safe. + + +Close verifier gap CR-01: make Album search synchronization observe committed relation state and make unmanaged transaction use fail safe. + +Purpose: D-20 promises an inline, non-fatal sync built from the committed row. Cabana and SaveAlbum currently save the Album before artist pivots and use plain GORM transactions, so Typesense can receive stale or rolled-back state. +Output: commit-safe Cabana and SaveAlbum transactions, a foreign-transaction guard in lagoon.AfterCommit, and regression tests for committed artist_ids and rollback silence. + + + +@/home/jin/.codex/gsd-core/workflows/execute-plan.md +@/home/jin/.codex/gsd-core/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md +@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-VERIFICATION.md +@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-01-SUMMARY.md +@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-05-SUMMARY.md +@.planning/phases/11-jobs-realtime-and-search-infrastructure/11-07-SUMMARY.md + + +## Artifacts this phase produces + +No new production files, schema objects, exported symbols, or dependencies. Changed symbols: `cabana.CRUDService.save`, `cabana.CRUDService.Delete`, `cabana.CRUDService.BulkDelete`, `cabana.RelationService.Link`, `cabana.RelationService.Unlink`, `lagoon.Transaction`, `lagoon.AfterCommit`, and `classes.SaveAlbum`. New test symbols: `TestAlbumsAdminSearchUsesCommittedArtists` and `TestSaveAlbumDefersAfterCommitUntilArtistsSync`. + +## Flagged assumptions (spec-less fallback: unresolved) + +- SRCH-01 remains `unclassified`: this closure guarantees that each sync reads its own committed row and ordered pivots, but does not serialize concurrent commits for the same Album; the Phase 12 SQL re-gate remains the safety boundary. +- RT-03 remains `unclassified` and is not newly claimed by this plan. The same transaction buffering corrects timing for Album update callbacks, but batch zero-primary-key writes and restore behavior retain the unresolved assumptions already recorded in 11-03. + + + + + Task 1: Admin artist edit indexes the committed ordered pivots + modules/cabana/crud.go, modules/cabana/relation.go, ../fonoteka.go/plugins/golem15/fonoteka/admin_albums_test.go, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go + modules/cabana/crud.go, modules/cabana/relation.go, modules/cabana/relation_field.go, modules/lagoon/transaction.go, ../fonoteka.go/plugins/golem15/fonoteka/admin_albums_test.go, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go, ../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VERIFICATION.md + + - A real Cabana HTTP update replacing an Album's artists returns success, commits the submitted pivot order, and produces one Typesense upsert whose artist_ids equal that committed order. + - Create/update/delete/bulk-delete and relation Link/Unlink keep their existing response, validation, scoping, lifecycle-hook, and rollback semantics. + + Write `TestAlbumsAdminSearchUsesCommittedArtists` red first. Reuse the fake Typesense and search app harness; let that harness expose its activated plugins so the test can assemble the real admin router. Create an allowed backend admin, its active collection, two artists, and an Album with an initial artist; reset recorded engine calls; update the Album through the Cabana HTTP route with a different ordered artist list. Assert both the committed pivot rows and the import document's `artist_ids` have the submitted order. + +Per D-20, replace only Cabana's write transactions with `lagoon.Transaction`: `CRUDService.save`, `Delete`, `BulkDelete`, and `RelationService.Link`/`Unlink`. Use the callback-supplied context for loads, validation, lifecycle hooks, relation scope checks, pivot writes, and deletes so `lagoon.AfterCommit` can see the buffer. Leave read-only `ShowRecord` and relation queries unchanged. + + go test ./modules/cabana -count=1 && (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^TestAlbumsAdminSearchUsesCommittedArtists$' -count=1 -v) + Either command exits non-zero; the named admin test is absent, skipped, reports no tests to run, or its indexed artist_ids differ from the committed pivot order. + + + - Cabana has exactly three write transaction entry points in crud.go and two in relation.go, all through lagoon.Transaction; read-only transaction sites remain unchanged. + - TestAlbumsAdminSearchUsesCommittedArtists exercises the assembled HTTP router, not CRUDService directly. + - The test proves the engine call occurs only after the ordered pivots are visible on the committed connection. + + An admin can change an Album's artists and the resulting Typesense document contains the committed ordered artist_ids. + + + + Task 2: Refuse external after-commit work inside unmanaged transactions + modules/lagoon/transaction.go, modules/lagoon/transaction_test.go, modules/lagoon/README.md, modules/beachcomber/sync_test.go, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go + modules/lagoon/transaction.go, modules/lagoon/transaction_test.go, modules/lagoon/README.md, modules/beachcomber/sync.go, modules/beachcomber/sync_test.go, ../fonoteka.go/plugins/golem15/fonoteka/search_smoke_test.go, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-REVIEW.md (CR-01 and WR-03) + + - AfterCommit inside a foreign plain GORM *sql.Tx logs a warning and does not invoke the callback. + - Rolling that foreign transaction back produces zero engine calls. + - Lagoon-managed commits and implicit single-statement commits still flush once; rollbacks flush none; direct calls outside a transaction still run immediately. + - A nested lagoon.Transaction given the root handle while its context carries a parent buffer returns an error and neither commits independent work nor attaches callbacks to the parent. + + Preserve the lagoon-buffer branch and GORM implicit-transaction branch. Before the immediate fallback, detect a foreign `*sql.Tx` through the statement connection pool, emit a stable Warn without row data or secrets, and return without running the callback. Tighten `Transaction` so a parent buffer uses the nested/savepoint branch only when the supplied handle is actually transactional; reject a root handle rather than opening an independently committed transaction under the parent's callback buffer (WR-03, directly implicated by adopting lagoon.Transaction in services). Update the API comment and README. + +Invert the tests that currently bless immediate plain-GORM sync. In lagoon, assert callback suppression plus the warning. In beachcomber, use a plain GORM transaction that writes and then rolls back and assert zero fake-engine calls. In fonoteka's search smoke test, assert a foreign transaction never sends to Typesense. Retain explicit regression coverage for the implicit single-statement and lagoon-managed paths. + + go test ./modules/lagoon -run '^(TestTransactionAfterCommit|TestTransactionEdges)$' -count=1 -v && go test ./modules/beachcomber -run '^TestSyncAfterCommit$' -count=1 -v && (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka -run '^TestAlbumSearchDeleteAndFailures$' -count=1 -v) + Any command exits non-zero; a named test is absent, skipped, or reports no tests to run; a foreign transaction invokes an after-commit callback or records an engine call; an implicit or lagoon-managed commit stops syncing. + + + - The plain-transaction tests assert zero callbacks/engine calls and a warning, replacing the old immediate-sync expectation. + - The rollback test observes no committed row and no external request. + - Nested lagoon.Transaction with a root handle cannot independently commit under a parent buffer. + - modules/lagoon/README.md describes all four cases: lagoon-managed, implicit statement, foreign transaction, and outside a transaction. + + Unmanaged transactions cannot leak uncommitted state externally, while every supported after-commit path keeps its documented behavior. + + + + Task 3: SaveAlbum defers sync until ordered artists commit + ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service_db_test.go + ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service_db_test.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/artist_resolver.go, modules/lagoon/transaction.go, .planning/phases/11-jobs-realtime-and-search-infrastructure/11-VERIFICATION.md + + - SaveAlbum validates, resolves artists, saves the Album, replaces pivots, commits, and only then runs after-commit callbacks. + - A pivot or validation failure rolls back the Album and runs no callback. + + Per D-20, replace SaveAlbum's plain GORM transaction with `lagoon.Transaction`, using the callback-supplied context throughout validation and writes. Keep the existing save-before-pivot order; buffering now makes the post-commit reload occur after `syncArtists` and commit. Add `TestSaveAlbumDefersAfterCommitUntilArtistsSync`, registering a test callback that proves it observes the committed ordered pivots, plus a rollback case proving it never runs. Do not add a second save/touch and do not change validation, price provenance, or artist resolution semantics. + + (cd ../fonoteka.go && go test ./plugins/golem15/fonoteka/classes -run '^TestSaveAlbum' -count=1 -v && go test ./plugins/golem15/fonoteka -run '^(TestAlbumsAdminSearchUsesCommittedArtists|TestAlbumSearchDeleteAndFailures)$' -count=1 -v) + Either command exits non-zero; the new SaveAlbum test or either gap regression test is absent, skipped, reports no tests to run, sees pre-pivot artist state, or runs a callback after rollback. + + + - SaveAlbum calls lagoon.Transaction and uses its callback context. + - TestSaveAlbumDefersAfterCommitUntilArtistsSync observes the submitted pivot order from the callback and zero callbacks on rollback. + - Existing SaveAlbum validation and market-price provenance tests remain green. + + Both real Album write paths expose only committed row-and-pivot state to after-commit consumers. + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| Open SQL transaction -> external search engine | Album and relation data must not leave the database before commit | +| Cabana admin request -> Album row and pivots | One user edit spans scalar and ordered relation writes that must commit atomically | + +## STRIDE Threat Register (ASVS L1; blocking threshold high) + +| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan | +|-----------|----------|-----------|----------|-------------|-----------------| +| T-11-31 | Information Disclosure | lagoon.AfterCommit / beachcomber sync | high | mitigate | Buffer supported writes until commit, refuse foreign *sql.Tx callbacks, and prove a rollback causes zero engine calls (Tasks 1-2). | +| T-11-32 | Tampering | Cabana and SaveAlbum ordered artist pivots | medium | mitigate | Route complete row-and-pivot units through lagoon.Transaction and compare the indexed artist_ids with committed pivot order (Tasks 1 and 3). | +| T-11-SC | Tampering | npm/pip/cargo installs | high | mitigate | No package-manager install occurs in this gap plan; existing dependency lockfiles remain unchanged. | + + + +Run the three focused task commands, then run `bash scripts/check-phase11.sh --all` from the project root. + +The phase gate fails if either repository's vet/tests fail, a named test is skipped or missing, or it does not print `phase11 all passed`. + + + +- The Cabana admin artist-edit regression indexes the committed ordered pivots. +- Plain GORM transaction rollback reaches no search engine, and AfterCommit warns instead of executing in that foreign transaction. +- Cabana write methods and SaveAlbum use lagoon.Transaction without changing their HTTP, validation, lifecycle, or relation contracts. +- All focused tests and the existing Phase 11 gate pass; no UI-SPEC, UI file, schema file, dependency, or runtime mirror is created. + + +## Multi-Source Coverage Audit (gap-closure scope) + +| Source | ID | Gap item | Plan | Status | +|--------|----|----------|------|--------| +| GOAL | SC-5 | Typesense sync reflects committed Album state | 11-08 | COVERED | +| REQ | SRCH-01 | Album documents sync correctly and safely | 11-08 | COVERED | +| RESEARCH | Pattern 10 | after-commit buffer and rollback silence | 11-08 | COVERED | +| CONTEXT | D-20 | after-commit, inline, non-fatal committed-row sync | 11-08 | COVERED | +| VERIFICATION | CR-01 | Cabana/SaveAlbum transaction paths, foreign-tx refusal, regression tests | 11-08 | COVERED | + +All other Phase 11 goals, requirements, research items, and D-01..D-19 decisions are excluded from this gap-closure audit because 11-VERIFICATION.md already marks them verified or out of this failed truth's scope. Deferred ideas remain excluded. + + +Create `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-08-SUMMARY.md` when done. +