feat(11-05): add the beachcomber search sync package and its Typesense engine
- Searchable, Engine, Gate and an init-time engine registry with the null engine - GORM callbacks installed through lagoon.OnDatabase register an after-commit sync that reloads the row and upserts or deletes its document - Gates run before any request: engine configured, database published, app Gate - hand-rolled net/http Typesense engine following the Scout wire contract - module README and root modules row
This commit is contained in:
Jakub Zych
152
modules/beachcomber/README.md
Normal file
152
modules/beachcomber/README.md
Normal file
@@ -0,0 +1,152 @@
|
||||
# beachcomber
|
||||
|
||||
Search index sync for GORM models: after-commit upserts and deletes through a pluggable engine, gated by an application kill-switch.
|
||||
|
||||
`import "git.golem15.com/golem15/summercms/modules/beachcomber"`
|
||||
|
||||
`import _ "git.golem15.com/golem15/summercms/modules/beachcomber/typesense"`
|
||||
|
||||
## Overview
|
||||
|
||||
beachcomber is the SummerCMS counterpart of Laravel Scout as WinterCMS applications use it with `queue=false`. A model opts in by implementing `beachcomber.Searchable`. After a create, update or delete of such a model commits, the row is reloaded by primary key and its document is upserted into the engine's index, or removed from it. Sync runs inline in the writing goroutine, bounded by the engine's request timeout, and is never fatal: a failure is logged and the write stays committed.
|
||||
|
||||
The package itself knows no search server. An engine package registers itself from its `init` function, the way `database/sql` drivers do, and the application picks one with `search.driver`. The built-in `null` engine indexes nothing. The `typesense` sub-package is a hand-rolled `net/http` client for Typesense that follows the Scout `TypesenseEngine` wire contract.
|
||||
|
||||
`beachcomber.From` builds the app-scoped `beachcomber.Service` on first use and publishes it on the app. It installs the sync GORM callbacks through `lagoon.OnDatabase`, so they reach the production handle even though plugins boot before `serve` publishes the database. The application installs a `beachcomber.Gate`, its kill-switch, with `beachcomber.Service.SetGate`.
|
||||
|
||||
## Features
|
||||
|
||||
- Engine selection by `search.driver`: `null` (the default) or a registered engine such as `typesense`. An unknown name is a boot error that lists the registered engines. Third-party engines register with `beachcomber.RegisterEngine` and a `beachcomber.EngineFactory`; a duplicate name panics at init.
|
||||
- The `beachcomber.Searchable` model contract: `SearchableAs` (the index name, prefixed with `search.prefix`), `ToSearchableArray(ctx, db)` (the document, built from the committed row and free to query related rows) and `ShouldBeSearchable`. A model can also implement `beachcomber.IndexSchemaProvider` (the schema the engine creates a missing index with) and `beachcomber.SearchKeyer` (a document key other than the decimal primary key).
|
||||
- The `beachcomber.Engine` driver contract: `Name`, `Configured`, `Upsert`, `Delete`, `Flush` and `SearchIDs` with a `beachcomber.Query`.
|
||||
- GORM callbacks `beachcomber.CallbackAfterCreate`, `beachcomber.CallbackAfterUpdate` and `beachcomber.CallbackAfterDelete`, installed once per `*gorm.DB`, register the sync with `lagoon.AfterCommit`.
|
||||
- `beachcomber.Service.Sync` and `beachcomber.Service.Remove` run the same gated path on demand, for reindex tooling, and return the error instead of logging it.
|
||||
- Typesense engine (`typesense.Engine`, engine name `typesense`): the `X-TYPESENSE-API-KEY` header on every request; `Upsert` reads the collection and creates it from the schema on 404, then imports JSON lines with `action=upsert`; `Delete` and `Flush` treat 404 as success; `SearchIDs` returns `hits[].document.id`.
|
||||
|
||||
## Usage
|
||||
|
||||
An application selects the engine in `config/search.yaml`:
|
||||
|
||||
```yaml
|
||||
driver: typesense
|
||||
typesense:
|
||||
api_key: "" # set with SUMMER_SEARCH__TYPESENSE__API_KEY
|
||||
```
|
||||
|
||||
A model implements `beachcomber.Searchable` without importing beachcomber:
|
||||
|
||||
```go
|
||||
package models
|
||||
|
||||
func (Post) SearchableAs() string { return "acme_blog_posts" }
|
||||
func (Post) ShouldBeSearchable() bool { return true }
|
||||
|
||||
func (p *Post) ToSearchableArray(ctx context.Context, db *gorm.DB) (map[string]any, error) {
|
||||
return map[string]any{
|
||||
"id": strconv.FormatUint(uint64(p.ID), 10),
|
||||
"blog_id": int64(p.BlogID),
|
||||
"title": p.Title,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (Post) SearchIndexSchema() map[string]any {
|
||||
return map[string]any{
|
||||
"fields": []map[string]any{
|
||||
{"name": "id", "type": "string"},
|
||||
{"name": "blog_id", "type": "int64"},
|
||||
{"name": "title", "type": "string"},
|
||||
},
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The plugin imports the engine package for its side effect, builds the service at Boot and installs its kill-switch:
|
||||
|
||||
```go
|
||||
package acme
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"git.golem15.com/golem15/summercms/modules/backpack"
|
||||
"git.golem15.com/golem15/summercms/modules/beachcomber"
|
||||
_ "git.golem15.com/golem15/summercms/modules/beachcomber/typesense"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
func (p *Plugin) Boot(app *backpack.App) error {
|
||||
svc, err := beachcomber.From(app)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
svc.SetGate(beachcomber.GateFunc(func(ctx context.Context, db *gorm.DB) bool {
|
||||
return acmeSearchEnabled(ctx, db) // false on any read error
|
||||
}))
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
A search endpoint asks the engine for candidate ids:
|
||||
|
||||
```go
|
||||
ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomber.Query{
|
||||
Q: term,
|
||||
QueryBy: []string{"title"},
|
||||
FilterBy: "blog_id:=" + strconv.FormatUint(uint64(blogID), 10),
|
||||
})
|
||||
```
|
||||
|
||||
## API reference
|
||||
|
||||
### beachcomber
|
||||
|
||||
| Identifier | Description |
|
||||
|------------|-------------|
|
||||
| `beachcomber.From(app)` | The app's `*beachcomber.Service`, built and published on first use. |
|
||||
| `beachcomber.Service` | The search service: `SetGate`, `Engine`, `Prefix`, `IndexName`, `Logger`, `Sync`, `Remove`. |
|
||||
| `beachcomber.Searchable` | `SearchableAs()`, `ToSearchableArray(ctx, db)`, `ShouldBeSearchable()`. |
|
||||
| `beachcomber.IndexSchemaProvider` | `SearchIndexSchema()`: the schema a missing index is created with. |
|
||||
| `beachcomber.SearchKeyer` | `SearchKey()`: replaces the decimal primary key as the document key. |
|
||||
| `beachcomber.Engine` | `Name`, `Configured`, `Upsert`, `Delete`, `Flush`, `SearchIDs`. |
|
||||
| `beachcomber.Query` | `Q`, `QueryBy`, `FilterBy`, `SortBy`, `Page`, `PerPage`. |
|
||||
| `beachcomber.Gate`, `beachcomber.GateFunc` | The application kill-switch: `Enabled(ctx, db) bool`. |
|
||||
| `beachcomber.EngineFactory`, `beachcomber.RegisterEngine(name, factory)` | Registers an engine from an `init` function. |
|
||||
| `beachcomber.NullEngine`, `beachcomber.DefaultDriver` | The name of the built-in engine that indexes nothing, and the default of `search.driver`. |
|
||||
| `beachcomber.CallbackAfterCreate`, `beachcomber.CallbackAfterUpdate`, `beachcomber.CallbackAfterDelete` | Names of the GORM callbacks. |
|
||||
|
||||
### beachcomber/typesense
|
||||
|
||||
| Identifier | Description |
|
||||
|------------|-------------|
|
||||
| `typesense.Config`, `typesense.LoadConfig` | The `search.typesense.*` settings with their defaults; `BaseURL` is `{protocol}://{host}:{port}{path}`. |
|
||||
| `typesense.Engine`, `typesense.New` | The `beachcomber.Engine`, with `Config`. |
|
||||
| `typesense.DriverName` | `typesense`. |
|
||||
| `typesense.DefaultHost`, `typesense.DefaultPort`, `typesense.DefaultProtocol`, `typesense.DefaultConnectionTimeout`, `typesense.DefaultImportAction` | Defaults of the configuration keys. |
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Default | Description |
|
||||
|-----|---------|-------------|
|
||||
| `search.driver` | `null` | `null`, or a registered engine such as `typesense`. |
|
||||
| `search.prefix` | `""` | Prefix applied to every index name. |
|
||||
| `search.typesense.api_key` | `""` | API key; empty means nothing is ever sent. |
|
||||
| `search.typesense.host` | `localhost` | Typesense node host. |
|
||||
| `search.typesense.port` | `8181` | Typesense node port. |
|
||||
| `search.typesense.protocol` | `http` | `http` or `https`. |
|
||||
| `search.typesense.path` | `""` | Path prefix of the node. |
|
||||
| `search.typesense.connection_timeout_seconds` | `2` | Per-request timeout, in seconds or as a duration string. |
|
||||
| `search.typesense.import_action` | `upsert` | The `action` of document imports. |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `backpack`, `compass` and `lagoon` (callback installation and `lagoon.AfterCommit`) from this repository.
|
||||
- `gorm.io/gorm` (sync callbacks and reloads).
|
||||
- The Typesense client is plain `net/http`; no Typesense SDK is used.
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
go test ./modules/beachcomber/...
|
||||
```
|
||||
|
||||
A test points `search.typesense.host` and `search.typesense.port` at an `httptest` server to see the exact Typesense requests. Sync is inline, so the requests have arrived when the write returns.
|
||||
Reference in New Issue
Block a user