summercms/modules/beachcomber

beachcomber

Search index sync for GORM models: after-commit upserts and deletes through a pluggable engine, gated by an application kill-switch.

import "git.golem15.com/golem15/summercms/modules/beachcomber"

import _ "git.golem15.com/golem15/summercms/modules/beachcomber/typesense"

Overview

beachcomber is the SummerCMS counterpart of Laravel Scout as WinterCMS applications use it with queue=false. A model opts in by implementing beachcomber.Searchable. After a create, update or delete of such a model commits, the row is reloaded by primary key and its document is upserted into the engine's index, or removed from it. Sync runs inline in the writing goroutine, bounded by the engine's request timeout, and is never fatal: a failure is logged and the write stays committed.

The package itself knows no search server. An engine package registers itself from its init function, the way database/sql drivers do, and the application picks one with search.driver. The built-in null engine indexes nothing. The typesense sub-package is a hand-rolled net/http client for Typesense that follows the Scout TypesenseEngine wire contract.

beachcomber.From builds the app-scoped beachcomber.Service on first use and publishes it on the app. It installs the sync GORM callbacks through lagoon.OnDatabase, so they reach the production handle even though plugins boot before serve publishes the database. The application installs a beachcomber.Gate, its kill-switch, with beachcomber.Service.SetGate.

Features

• Engine selection by search.driver: null (the default) or a registered engine such as typesense. An unknown name is a boot error that lists the registered engines. Third-party engines register with beachcomber.RegisterEngine and a beachcomber.EngineFactory; a duplicate name panics at init.
• The beachcomber.Searchable model contract: SearchableAs (the index name, prefixed with search.prefix), ToSearchableArray(ctx, db) (the document, built from the committed row and free to query related rows) and ShouldBeSearchable. A model can also implement beachcomber.IndexSchemaProvider (the schema the engine creates a missing index with) and beachcomber.SearchKeyer (a document key other than the decimal primary key).
• The beachcomber.Engine driver contract: Name, Configured, Upsert, Delete, Flush and SearchIDs with a beachcomber.Query.
• GORM callbacks beachcomber.CallbackAfterCreate, beachcomber.CallbackAfterUpdate and beachcomber.CallbackAfterDelete, installed once per *gorm.DB, register the sync with lagoon.AfterCommit.
• beachcomber.Service.Sync and beachcomber.Service.Remove run the same gated path on demand, for reindex tooling, and return the error instead of logging it.
• Typesense engine (typesense.Engine, engine name typesense):
  • Every request carries the X-TYPESENSE-API-KEY header.
  • Upsert reads the collection and creates it from the schema on 404. A 409 on create counts as success, and a model without a schema gets an auto-typed collection. It then imports the documents as JSON lines (Content-Type: text/plain) with action=upsert. Typesense answers 200 even when a document fails, so every answer line is checked and any "success":false line is an error.
  • Delete and Flush treat 404 as success.
  • SearchIDs sends q (default *), query_by, filter_by, sort_by, page and per_page, and returns hits[].document.id in order.
  • Ids and index names are path-escaped.
  • A non-2xx answer is a typesense.StatusError with the method, path and status, never the answer body.

Sync semantics

• After commit. The callbacks register the sync with lagoon.AfterCommit. Inside lagoon.Transaction it runs after that transaction commits, and not at all when it rolls back. A single-statement write, for which GORM opens its own transaction, syncs after that commit and not when the write fails. Inside a plain gorm transaction there is no commit hook, so the sync runs immediately through the transaction's handle. Its reads run in a savepoint, so a failed read never aborts the caller's transaction, including a read that the application Gate swallows and counts as off.

• Inline and non-fatal. The sync runs in the writing goroutine, after the commit, so a create followed by a search sees the document. Every engine request is bounded by the engine's timeout (search.typesense.connection_timeout_seconds), and the caller's context cancellation does not abandon it. A failure, a timeout or a panic is logged at Warn as search: sync failed with the index, key and operation. The write is already committed and stays so. The log never carries the document or the API key.

• Three gates, before any request. Nothing is sent when:

  1. the engine is not configured (the null engine, or Typesense with an empty search.typesense.api_key);
  2. no *gorm.DB is published on the app (a fresh install);
  3. the application beachcomber.Gate reports off. A gate must treat a read error as off.

  When the engine is not configured, the callbacks do not even register work.

• Reload, then decide. The row is reloaded by primary key, including soft-deleted rows. A delete, a row that is gone, a soft-deleted row (a set gorm.DeletedAt) or a row whose ShouldBeSearchable is false has its document deleted. Restoring a soft-deleted row is an ordinary update and indexes it again. An error from ToSearchableArray is logged and nothing is sent, which lets a model refuse a document that would break scoping. A document without an id gets the key.

• Rows only. A statement without a primary key value, such as Model(&T{}).Where(…).Updates(…) or Delete(&T{}, id), cannot be synced row by row and is skipped. Bulk paths call beachcomber.Service.Sync or beachcomber.Service.Remove per row, or reindex.

• Candidates, not answers. beachcomber.Engine.SearchIDs returns candidate ids from an external index that may be stale. Callers must re-gate every id in SQL (ownership, visibility, soft deletes) before they expose a row. An empty result is an empty list, never an error.

Usage

An application selects the engine in config/search.yaml:

driver: typesense
typesense:
  api_key: ""   # set with SUMMER_SEARCH__TYPESENSE__API_KEY

A model implements beachcomber.Searchable without importing beachcomber:

package models

func (Post) SearchableAs() string     { return "acme_blog_posts" }
func (Post) ShouldBeSearchable() bool { return true }

func (p *Post) ToSearchableArray(ctx context.Context, db *gorm.DB) (map[string]any, error) {
	return map[string]any{
		"id":      strconv.FormatUint(uint64(p.ID), 10),
		"blog_id": int64(p.BlogID),
		"title":   p.Title,
	}, nil
}

func (Post) SearchIndexSchema() map[string]any {
	return map[string]any{
		"fields": []map[string]any{
			{"name": "id", "type": "string"},
			{"name": "blog_id", "type": "int64"},
			{"name": "title", "type": "string"},
		},
	}
}

The plugin imports the engine package for its side effect, builds the service at Boot and installs its kill-switch:

package acme

import (
	"context"

	"git.golem15.com/golem15/summercms/modules/backpack"
	"git.golem15.com/golem15/summercms/modules/beachcomber"
	_ "git.golem15.com/golem15/summercms/modules/beachcomber/typesense"
	"gorm.io/gorm"
)

func (p *Plugin) Boot(app *backpack.App) error {
	svc, err := beachcomber.From(app)
	if err != nil {
		return err
	}
	svc.SetGate(beachcomber.GateFunc(func(ctx context.Context, db *gorm.DB) bool {
		return acmeSearchEnabled(ctx, db) // false on any read error
	}))
	return nil
}

A search endpoint asks the engine for candidate ids:

ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomber.Query{
	Q:        term,
	QueryBy:  []string{"title"},
	FilterBy: "blog_id:=" + strconv.FormatUint(uint64(blogID), 10),
})

API reference

beachcomber

Identifier	Description
beachcomber.From(app)	The app's *beachcomber.Service, built and published on first use.
beachcomber.Service	The search service: SetGate, Engine, Prefix, IndexName, Logger, Sync, Remove.
beachcomber.Searchable	SearchableAs(), ToSearchableArray(ctx, db), ShouldBeSearchable().
beachcomber.IndexSchemaProvider	SearchIndexSchema(): the schema a missing index is created with.
beachcomber.SearchKeyer	SearchKey(): replaces the decimal primary key as the document key.
beachcomber.Engine	Name, Configured, Upsert, Delete, Flush, SearchIDs.
beachcomber.Query	Q, QueryBy, FilterBy, SortBy, Page, PerPage.
beachcomber.Gate, beachcomber.GateFunc	The application kill-switch: Enabled(ctx, db) bool.
beachcomber.EngineFactory, beachcomber.RegisterEngine(name, factory)	Registers an engine from an init function.
beachcomber.NullEngine, beachcomber.DefaultDriver	The name of the built-in engine that indexes nothing, and the default of search.driver.
beachcomber.CallbackAfterCreate, beachcomber.CallbackAfterUpdate, beachcomber.CallbackAfterDelete	Names of the GORM callbacks.

beachcomber/typesense

Identifier	Description
typesense.Config, typesense.LoadConfig	The search.typesense.* settings with their defaults; BaseURL is {protocol}://{host}:{port}{path}.
typesense.Engine, typesense.New	The beachcomber.Engine, with Config.
typesense.StatusError	A non-2xx answer: Method, Path, Code and StatusCode().
typesense.DriverName	typesense.
typesense.DefaultHost, typesense.DefaultPort, typesense.DefaultProtocol, typesense.DefaultConnectionTimeout, typesense.DefaultImportAction	Defaults of the configuration keys.

Configuration

Key	Default	Description
search.driver	null	null, or a registered engine such as typesense.
search.prefix	""	Prefix applied to every index name.
search.typesense.api_key	""	API key; empty means nothing is ever sent.
search.typesense.host	localhost	Typesense node host.
search.typesense.port	8181	Typesense node port.
search.typesense.protocol	http	http or https.
search.typesense.path	""	Path prefix of the node.
search.typesense.connection_timeout_seconds	2	Per-request timeout, in seconds or as a duration string.
search.typesense.import_action	upsert	The action of document imports.

Dependencies

• backpack, compass and lagoon (callback installation and lagoon.AfterCommit) from this repository.
• gorm.io/gorm (sync callbacks and reloads).
• The Typesense client is plain net/http; no Typesense SDK is used.

Testing

go test ./modules/beachcomber/...

A test points search.typesense.host and search.typesense.port at an httptest server to see the exact Typesense requests. Sync is inline, so the requests have arrived when the write returns.