28 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, estimate, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | estimate | must_haves | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 11-jobs-realtime-and-search-infrastructure | 05 | execute | 3 |
|
|
true |
|
|
|
Phase Goal
ROADMAP Phase 11 goal (verbatim, not in user-story form): River jobs run on the correct dual-driver split, Centrifugo publishing and channel authorization match the existing server, and Typesense sync stays a re-gated pre-filter — all brought up before the API phases that depend on them.
This plan's slice: when the search toggle is on, saving an album makes it findable in Typesense right after the write commits, scoped by collection_id; when the toggle is off or Typesense is unconfigured, nothing is sent and nothing breaks (SRCH-01, ROADMAP SC-5).
Create the search framework package `beachcomber` with its Typesense engine sub-package, and bind the Album model and the settings kill-switch in fonoteka.go.Purpose: Phase 12's album search endpoint reads candidate ids from this index and re-gates them in SQL. Decisions implemented: D-19, D-20; RESEARCH Pattern 10 and the Typesense contract; T-11-07. Output: modules/beachcomber (+ typesense) with README and root row, fonoteka Album Searchable binding, settings Gate, config/search.yaml, README config mapping, smoke tests.
Repos: summercms.go (framework) and fonoteka.go (application). Planning docs and code in separate commits. Never add co-author tags.
<execution_context>
@/.claude/gsd-core/workflows/execute-plan.md
@/.claude/gsd-core/templates/summary.md
</execution_context>
Artifacts this phase produces
(This plan's share.)
- beachcomber:
Service,From(app) (*Service, error),(*Service).SetGate(Gate),(*Service).Engine() Engine,(*Service).Sync(ctx, db, model)and(*Service).Remove(ctx, db, model)(explicit calls for later reindex work),Searchable,IndexSchemaProvider(SearchIndexSchema() map[string]any),SearchKeyer(SearchKey() string),Engine,EngineFactory,RegisterEngine(name, EngineFactory),Query{Q string; QueryBy []string; FilterBy string; SortBy string; Page, PerPage int},Gate(Enabled(ctx, db *gorm.DB) bool),GateFunc, built-in enginenull; GORM callback namesbeachcomber:after_create,beachcomber:after_update,beachcomber:after_delete. - beachcomber/typesense:
Config(fromsearch.typesense.*),Engine(Upsert,Delete,Flush,SearchIDs,Configured,Name), engine nametypesenseregistered in init. - Config keys:
search.driver(default null),search.prefix(""),search.typesense.api_key(""),.host(localhost),.port(8181),.protocol(http),.path(""),.connection_timeout_seconds(2),.import_action(upsert). - fonoteka:
(Album) SearchableAs,(*Album) ToSearchableArray,(Album) ShouldBeSearchable,(Album) SearchIndexSchema,(Album) MediumFamily() *string,AlbumMediamap;settingsGate;(*Plugin) wireSearch;config/search.yaml.
Flagged assumptions (edge probe: unclassified)
- SRCH-01 came back
unclassifiedfrom the spec-less edge probe and staysunresolved. Planner reading for manual review: (a) restore of a soft-deleted album — PHP A4 (Winter SoftDelete firing restored) is unverified, so the Go port re-indexes on the update that clears deleted_at because it is an ordinary update; (b) concurrent saves of one album may upsert out of order, which the SQL re-gate in Phase 12 keeps safe; (c) an empty search result is an empty id list, never an error. Album.ShouldBeSearchablereturns true in Go: PHP returns the settings flag, but PHP also disables syncing entirely at boot when that flag is off, so the flag is only ever consulted when it is on. The Go Gate reproduces the kill-switch; behaviour is equivalent.
(2) sync.go (D-20, Pattern 10): each callback skips non-Searchable statement types and zero primary keys (iterating slice elements for batch creates), then calls lagoon.AfterCommit(db.Statement.Context, db, func(ctx, db) { svc.syncOne(ctx, db, type, key, op) }). syncOne gates in order, each non-fatal and request-free: engine not Configured → skip; no *gorm.DB published → skip; Gate set and not Enabled → skip. Then it reloads a fresh instance by primary key with Unscoped(): not found, soft-deleted (a non-null gorm.DeletedAt) or op delete → Delete(index, [key]); ShouldBeSearchable() false → Delete; else ToSearchableArray(ctx, db) then Upsert(index, schema, [doc]) where index = search.prefix + SearchableAs() and schema comes from IndexSchemaProvider (nil otherwise). Every error is logged Warn with index, key and operation (never the API key) and swallowed; a panic is recovered and logged. Public (*Service).Sync(ctx, db, model) and Remove run the same path directly.
(3) typesense sub-package (D-19): config.go reads search.typesense.* with Scout defaults (api_key "", host localhost, port 8181, protocol http, path "", connection_timeout_seconds 2, import_action upsert); engine.go builds base URL {protocol}://{host}:{port}{path}, an http.Client with the connection timeout, header X-TYPESENSE-API-KEY on every request; Configured() is api_key non-empty; Upsert: GET /collections/{index}, on 404 POST /collections with the schema plus "name": index, then POST /collections/{index}/documents/import?action={import_action} with one JSON object per line (Content-Type text/plain) and an error if any response line has "success":false (Typesense answers 200 even then) or the status is not 2xx; Delete: DELETE /collections/{index}/documents/{url-escaped id} per id, 404 counts as success; Flush: DELETE /collections/{index}, 404 success; SearchIDs: GET /collections/{index}/documents/search with q, query_by (comma-joined), filter_by, sort_by, page, per_page and returns hits[].document.id; func init() { beachcomber.RegisterEngine("typesense", ...) }.
(4) fonoteka Album (SRCH-01): new models/album_search.go porting PHP exactly — var AlbumMedia = map[string]string{"LP": "vinyl", "2LP": "vinyl", "EP 7\"": "vinyl", "CD": "cd", "2CD": "cd", "MC": "cassette", "Box": "box"}, (a Album) MediumFamily() *string (nil for nil/empty or unknown format), (Album) SearchableAs() string = golem15_fonoteka_albums, (Album) ShouldBeSearchable() bool = true (see flagged assumption), (a *Album) ToSearchableArray(ctx, db) that errors when CollectionID is 0, loads Genre, Styles and Artists (artists ordered by the pivot sort_order via the album_artists join table) through db, and returns the 17 PHP keys with PHP types (id string, ints for ids/year/created_at, joined style names with a space, empty strings for nil text), and (Album) SearchIndexSchema() map[string]any = the PHP collection-schema fields list with optional flags plus default_sorting_field: created_at. Keep models a leaf (framework imports only).
(5) fonoteka wiring: new search.go with a settingsGate whose Enabled reads the first golem15_fonoteka_settings row through the given db (Order id, Limit 1) and returns its SearchUseTypesense, false on any error or missing row; blank-import the typesense package; func (p *Plugin) wireSearch(app *backpack.App) error calls beachcomber.From(app) and SetGate(settingsGate{}); Boot in plugin.go calls it. New config/search.yaml: driver: typesense, prefix: "", typesense block with the Scout defaults and empty api_key (comment the SUMMER_SEARCH__TYPESENSE__* names).
(6) Smoke TestAlbumSearchSmoke (search_smoke_test.go, Postgres, fresh lagoon.Use handle, fake Typesense via httptest recording method/path/headers/body): toggle on plus api_key set → creating an album inside lagoon.Transaction results, after commit, in GET collection (404) → POST /collections (schema has name golem15_fonoteka_albums) → POST import?action=upsert whose JSONL doc has a positive integer collection_id and string id; toggle off → zero requests; api_key empty → zero requests.
(7) Docs: new modules/beachcomber/README.md in the standard structure (H1, summary "Search index sync for GORM models: after-commit upserts and deletes through a pluggable engine, gated by an application kill-switch.", both import lines, Overview, Features, Usage with an acme model, API reference, Configuration, Dependencies, Testing), root README.md row with the same sentence; identifiers checked with go doc.
go vet ./... && go test ./modules/beachcomber/... -count=1 && (cd ../fonoteka.go && go vet ./plugins/golem15/fonoteka/... && go test ./plugins/golem15/fonoteka -run '^TestAlbumSearchSmoke$' -count=1 -race -v)
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS" for TestAlbumSearchSmoke, prints "no tests to run", "--- SKIP" or "DATA RACE".</fails_when>
<acceptance_criteria>
- go doc ./modules/beachcomber Searchable, go doc ./modules/beachcomber Engine, go doc ./modules/beachcomber Gate and go doc ./modules/beachcomber/typesense Engine.Upsert exit 0.
- grep -c 'X-TYPESENSE-API-KEY' modules/beachcomber/typesense/engine.go prints at least 1 and grep -c 'import?action=' modules/beachcomber/typesense/engine.go prints at least 1.
- grep -c 'lagoon.AfterCommit' modules/beachcomber/sync.go prints at least 1.
- grep -c 'golem15_fonoteka_albums' ../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go prints 1 and grep -c 'default_sorting_field' ../fonoteka.go/plugins/golem15/fonoteka/models/album_search.go prints 1.
- grep -c 'wireSearch' ../fonoteka.go/plugins/golem15/fonoteka/plugin.go prints 1.
- grep -c '\[beachcomber\](modules/beachcomber/README.md)' README.md prints 1.
</acceptance_criteria>
With the toggle on and Typesense configured, an album save produces a scoped upsert after commit; with the toggle off or no API key, Typesense is never contacted.
(2) typesense/engine.go: confirm 404 handling for Delete and Flush, the success:false line scan for import, URL escaping of ids and index names, and SearchIDs parameter encoding (url.Values); return a typed error carrying the status code for non-2xx responses (without response bodies that could echo keys).
(3) Tests in search_smoke_test.go for the behavior list (TestAlbumSearchDeleteAndFailures), using the fake Typesense with switchable failure modes and a hanging handler for the timeout case (configure a 1s timeout in the test).
(4) Docs: modules/beachcomber/README.md documents the after-commit semantics, the three gates, delete-on-soft-delete and that SearchIDs results must be re-gated in SQL by callers. ../fonoteka.go/README.md Configuration gains the search mapping: SCOUT_DRIVER → SUMMER_SEARCH__DRIVER, SCOUT_PREFIX → SUMMER_SEARCH__PREFIX, TYPESENSE_API_KEY, TYPESENSE_HOST, TYPESENSE_PORT, TYPESENSE_PATH, TYPESENSE_PROTOCOL, TYPESENSE_CONNECTION_TIMEOUT_SECONDS and TYPESENSE_IMPORT_ACTION → SUMMER_SEARCH__TYPESENSE__*, plus the note that the admin setting search_use_typesense is the kill-switch.
go vet ./... && go test ./... && (cd ../fonoteka.go && go vet ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/... && go test ./plugins/golem15/fonoteka -run '^(TestAlbumSearchSmoke|TestAlbumSearchDeleteAndFailures)$' -count=1 -race -v && go test ./... ./plugins/golem15/fonoteka/... ./plugins/golem15/user/...)
<fails_when>Any command exits non-zero; the verbose run lacks "--- PASS" for TestAlbumSearchSmoke or TestAlbumSearchDeleteAndFailures, prints "no tests to run", "--- SKIP" or "DATA RACE"; any package reports FAIL.</fails_when>
<acceptance_criteria>
- TestAlbumSearchDeleteAndFailures asserts the album row still exists after a Typesense 500 and after a timeout.
- grep -c 'SUMMER_SEARCH__TYPESENSE__API_KEY' ../fonoteka.go/README.md prints at least 1.
- grep -ci 're-gate' modules/beachcomber/README.md prints at least 1.
- go doc ./modules/beachcomber/typesense Engine.SearchIDs exits 0.
</acceptance_criteria>
Album deletes clear their documents, every Typesense failure mode leaves writes committed with a warning, search returns candidate ids for Phase 12 to re-gate, and both repositories pass their full suites.
<threat_model>
Trust Boundaries
| Boundary | Description |
|---|---|
| Committed write → Typesense HTTP API | Album data, scoped by collection_id, leaves the database for an operator-run search server |
| Typesense results → Phase 12 search endpoint | Candidate ids come back from an external index and must be re-authorized in SQL |
| Admin setting → sync gate | The search_use_typesense toggle decides whether data is sent at all |
STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|---|---|---|---|---|---|
| T-11-07 | Information Disclosure | cross-collection search leak through a stale or mis-scoped index | high | mitigate | Every document carries a positive collection_id (ToSearchableArray refuses otherwise); SearchIDs returns candidates only and the README states callers re-gate in SQL (Phase 12 owns the endpoint re-gate) (Tasks 1-2). |
| T-11-25 | Information Disclosure | Typesense API key in logs or errors | medium | mitigate | The key is only set as a request header; errors carry status codes, not bodies; sync logs name index, key and operation only (Task 2). |
| T-11-26 | Denial of Service | search failure blocking or failing writes | medium | mitigate | Sync runs after commit, inline with the engine's connection timeout, and swallows every error and panic with a warning (Tasks 1-2). |
| T-11-27 | Information Disclosure | data sent while the kill-switch is off | medium | mitigate | Gates run before any request: unconfigured engine, no database, and the settings Gate (errors mean off); tests assert zero requests (Task 1). |
| T-11-SC | Tampering | Go module installs | high | mitigate | No module added; the Typesense client is hand-rolled on net/http per D-19. |
| </threat_model> |
<success_criteria>
- beachcomber and beachcomber/typesense exist with README and root row; engines selected by
search.driver. - Album documents mirror PHP toSearchableArray and always carry a positive collection_id.
- Sync is after-commit, inline, non-fatal, and fully skipped by the three gates.
- fonoteka wires the Album binding and the settings Gate and documents the env mapping. </success_criteria>