summercms/modules/beachcomber

beachcomber

Search index sync for GORM models: after-commit upserts and deletes through a pluggable engine, gated by an application kill-switch.

import "git.golem15.com/golem15/summercms/modules/beachcomber"

import _ "git.golem15.com/golem15/summercms/modules/beachcomber/typesense"

Overview

beachcomber is the SummerCMS counterpart of Laravel Scout as WinterCMS applications use it with queue=false. A model opts in by implementing beachcomber.Searchable. After a create, update or delete of such a model commits, the row is reloaded by primary key and its document is upserted into the engine's index, or removed from it. Sync runs inline in the writing goroutine, bounded by the engine's request timeout, and is never fatal: a failure is logged and the write stays committed.

The package itself knows no search server. An engine package registers itself from its init function, the way database/sql drivers do, and the application picks one with search.driver. The built-in null engine indexes nothing. The typesense sub-package is a hand-rolled net/http client for Typesense that follows the Scout TypesenseEngine wire contract.

beachcomber.From builds the app-scoped beachcomber.Service on first use and publishes it on the app. It installs the sync GORM callbacks through lagoon.OnDatabase, so they reach the production handle even though plugins boot before serve publishes the database. The application installs a beachcomber.Gate, its kill-switch, with beachcomber.Service.SetGate.

Features

• Engine selection by search.driver: null (the default) or a registered engine such as typesense. An unknown name is a boot error that lists the registered engines. Third-party engines register with beachcomber.RegisterEngine and a beachcomber.EngineFactory; a duplicate name panics at init.
• The beachcomber.Searchable model contract: SearchableAs (the index name, prefixed with search.prefix), ToSearchableArray(ctx, db) (the document, built from the committed row and free to query related rows) and ShouldBeSearchable. A model can also implement beachcomber.IndexSchemaProvider (the schema the engine creates a missing index with) and beachcomber.SearchKeyer (a document key other than the decimal primary key).
• The beachcomber.Engine driver contract: Name, Configured, Upsert, Delete, Flush and SearchIDs with a beachcomber.Query.
• GORM callbacks beachcomber.CallbackAfterCreate, beachcomber.CallbackAfterUpdate and beachcomber.CallbackAfterDelete, installed once per *gorm.DB, register the sync with lagoon.AfterCommit.
• beachcomber.Service.Sync and beachcomber.Service.Remove run the same gated path on demand, for reindex tooling, and return the error instead of logging it.
• Typesense engine (typesense.Engine, engine name typesense): the X-TYPESENSE-API-KEY header on every request; Upsert reads the collection and creates it from the schema on 404, then imports JSON lines with action=upsert; Delete and Flush treat 404 as success; SearchIDs returns hits[].document.id.

Usage

An application selects the engine in config/search.yaml:

driver: typesense
typesense:
  api_key: ""   # set with SUMMER_SEARCH__TYPESENSE__API_KEY

A model implements beachcomber.Searchable without importing beachcomber:

package models

func (Post) SearchableAs() string     { return "acme_blog_posts" }
func (Post) ShouldBeSearchable() bool { return true }

func (p *Post) ToSearchableArray(ctx context.Context, db *gorm.DB) (map[string]any, error) {
	return map[string]any{
		"id":      strconv.FormatUint(uint64(p.ID), 10),
		"blog_id": int64(p.BlogID),
		"title":   p.Title,
	}, nil
}

func (Post) SearchIndexSchema() map[string]any {
	return map[string]any{
		"fields": []map[string]any{
			{"name": "id", "type": "string"},
			{"name": "blog_id", "type": "int64"},
			{"name": "title", "type": "string"},
		},
	}
}

The plugin imports the engine package for its side effect, builds the service at Boot and installs its kill-switch:

package acme

import (
	"context"

	"git.golem15.com/golem15/summercms/modules/backpack"
	"git.golem15.com/golem15/summercms/modules/beachcomber"
	_ "git.golem15.com/golem15/summercms/modules/beachcomber/typesense"
	"gorm.io/gorm"
)

func (p *Plugin) Boot(app *backpack.App) error {
	svc, err := beachcomber.From(app)
	if err != nil {
		return err
	}
	svc.SetGate(beachcomber.GateFunc(func(ctx context.Context, db *gorm.DB) bool {
		return acmeSearchEnabled(ctx, db) // false on any read error
	}))
	return nil
}

A search endpoint asks the engine for candidate ids:

ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomber.Query{
	Q:        term,
	QueryBy:  []string{"title"},
	FilterBy: "blog_id:=" + strconv.FormatUint(uint64(blogID), 10),
})

API reference

beachcomber

Identifier	Description
beachcomber.From(app)	The app's *beachcomber.Service, built and published on first use.
beachcomber.Service	The search service: SetGate, Engine, Prefix, IndexName, Logger, Sync, Remove.
beachcomber.Searchable	SearchableAs(), ToSearchableArray(ctx, db), ShouldBeSearchable().
beachcomber.IndexSchemaProvider	SearchIndexSchema(): the schema a missing index is created with.
beachcomber.SearchKeyer	SearchKey(): replaces the decimal primary key as the document key.
beachcomber.Engine	Name, Configured, Upsert, Delete, Flush, SearchIDs.
beachcomber.Query	Q, QueryBy, FilterBy, SortBy, Page, PerPage.
beachcomber.Gate, beachcomber.GateFunc	The application kill-switch: Enabled(ctx, db) bool.
beachcomber.EngineFactory, beachcomber.RegisterEngine(name, factory)	Registers an engine from an init function.
beachcomber.NullEngine, beachcomber.DefaultDriver	The name of the built-in engine that indexes nothing, and the default of search.driver.
beachcomber.CallbackAfterCreate, beachcomber.CallbackAfterUpdate, beachcomber.CallbackAfterDelete	Names of the GORM callbacks.

beachcomber/typesense

Identifier	Description
typesense.Config, typesense.LoadConfig	The search.typesense.* settings with their defaults; BaseURL is {protocol}://{host}:{port}{path}.
typesense.Engine, typesense.New	The beachcomber.Engine, with Config.
typesense.DriverName	typesense.
typesense.DefaultHost, typesense.DefaultPort, typesense.DefaultProtocol, typesense.DefaultConnectionTimeout, typesense.DefaultImportAction	Defaults of the configuration keys.

Configuration

Key	Default	Description
search.driver	null	null, or a registered engine such as typesense.
search.prefix	""	Prefix applied to every index name.
search.typesense.api_key	""	API key; empty means nothing is ever sent.
search.typesense.host	localhost	Typesense node host.
search.typesense.port	8181	Typesense node port.
search.typesense.protocol	http	http or https.
search.typesense.path	""	Path prefix of the node.
search.typesense.connection_timeout_seconds	2	Per-request timeout, in seconds or as a duration string.
search.typesense.import_action	upsert	The action of document imports.

Dependencies

• backpack, compass and lagoon (callback installation and lagoon.AfterCommit) from this repository.
• gorm.io/gorm (sync callbacks and reloads).
• The Typesense client is plain net/http; no Typesense SDK is used.

Testing

go test ./modules/beachcomber/...

A test points search.typesense.host and search.typesense.port at an httptest server to see the exact Typesense requests. Sync is inline, so the requests have arrived when the write returns.