# beachcomber

Search index sync for GORM models: after-commit upserts and deletes through a pluggable engine, gated by an application kill-switch.

`import "git.golem15.com/golem15/summercms/modules/beachcomber"`

`import _ "git.golem15.com/golem15/summercms/modules/beachcomber/typesense"`

## Overview

beachcomber is the SummerCMS counterpart of Laravel Scout as WinterCMS applications use it with `queue=false`. A model opts in by implementing `beachcomber.Searchable`. After a create, update or delete of such a model commits, the row is reloaded by primary key and its document is upserted into the engine's index, or removed from it. Sync runs inline in the writing goroutine, bounded by the engine's request timeout, and is never fatal: a failure is logged and the write stays committed.

The package itself knows no search server. An engine package registers itself from its `init` function, the way `database/sql` drivers do, and the application picks one with `search.driver`. The built-in `null` engine indexes nothing. The `typesense` sub-package is a hand-rolled `net/http` client for Typesense that follows the Scout `TypesenseEngine` wire contract.

`beachcomber.From` builds the app-scoped `beachcomber.Service` on first use and publishes it on the app. It installs the sync GORM callbacks through `lagoon.OnDatabase`, so they reach the production handle even though plugins boot before `serve` publishes the database. The application installs a `beachcomber.Gate`, its kill-switch, with `beachcomber.Service.SetGate`.

## Features

- Engine selection by `search.driver`: `null` (the default) or a registered engine such as `typesense`. An unknown name is a boot error that lists the registered engines. Third-party engines register with `beachcomber.RegisterEngine` and a `beachcomber.EngineFactory`; a duplicate name panics at init.
- The `beachcomber.Searchable` model contract: `SearchableAs` (the index name, prefixed with `search.prefix`), `ToSearchableArray(ctx, db)` (the document, built from the committed row and free to query related rows) and `ShouldBeSearchable`. A model can also implement `beachcomber.IndexSchemaProvider` (the schema the engine creates a missing index with) and `beachcomber.SearchKeyer` (a document key other than the decimal primary key).
- The `beachcomber.Engine` driver contract: `Name`, `Configured`, `Upsert`, `Delete`, `Flush` and `SearchIDs` with a `beachcomber.Query`. `beachcomber.Query.QueryByWeights` ranks the `QueryBy` fields, one weight per field.
- Search totals: the optional `beachcomber.PageSearcher` interface returns a `beachcomber.SearchResult`, one page of ids plus `Found`, the number of documents the engine matched. Call it through `beachcomber.SearchPage`, which falls back to `SearchIDs` (with `Found` set to the number of ids) for an engine that does not implement it; the `null` engine returns no ids and a zero count.
- GORM callbacks `beachcomber.CallbackAfterCreate`, `beachcomber.CallbackAfterUpdate` and `beachcomber.CallbackAfterDelete`, installed once per `*gorm.DB`, register the sync with `lagoon.AfterCommit`.
- `beachcomber.Service.Sync` and `beachcomber.Service.Remove` run the same gated path on demand, for reindex tooling, and return the error instead of logging it.
- Typesense engine (`typesense.Engine`, engine name `typesense`):
  - Every request carries the `X-TYPESENSE-API-KEY` header.
  - `Upsert` reads the collection and creates it from the schema on 404. A 409 on create counts as success, and a model without a schema gets an auto-typed collection. It then imports the documents as JSON lines (`Content-Type: text/plain`) with `action=upsert`. Typesense answers 200 even when a document fails, so every answer line is checked and any `"success":false` line is an error.
  - `Delete` and `Flush` treat 404 as success.
  - `SearchIDs` and `SearchPage` send one request with `q` (default `*`), `query_by`, `query_by_weights`, `filter_by`, `sort_by`, `page` and `per_page`, and return `hits[].document.id` in order; `SearchPage` adds the answer's `found`. A weight list whose length differs from `QueryBy`, or a `PerPage` above `typesense.MaxPerPage` (250, Typesense's limit), is an error before any request; callers page instead.
  - Ids and index names are path-escaped.
  - A non-2xx answer is a `typesense.StatusError` with the method, path and status, never the answer body.

## Sync semantics

- **After commit.** The callbacks register the sync with `lagoon.AfterCommit`. Inside `lagoon.Transaction` it runs after that transaction commits, and not at all when it rolls back. A single-statement write, for which GORM opens its own transaction, syncs after that commit and not when the write fails. Inside a plain `gorm` transaction Lagoon cannot observe the commit, so `lagoon.AfterCommit` logs a warning and the sync is skipped; wrap such writes in `lagoon.Transaction`, or call `Sync` after the commit. The sync's reads run in a savepoint, so when `Sync` or `Remove` is handed a transaction a failed read never aborts it, including a read that the application Gate swallows and counts as off.
- **Inline and non-fatal.** The sync runs in the writing goroutine, after the commit, so a create followed by a search sees the document. Every engine request is bounded by the engine's timeout (`search.typesense.connection_timeout_seconds`), and the caller's context cancellation does not abandon it. A failure, a timeout or a panic is logged at Warn as `search: sync failed` with the index, key and operation. The write is already committed and stays so. The log never carries the document or the API key.
- **Three gates, before any request.** Nothing is sent when:
  1. the engine is not configured (the `null` engine, or Typesense with an empty `search.typesense.api_key`);
  2. no `*gorm.DB` is published on the app (a fresh install);
  3. the application `beachcomber.Gate` reports off. A gate must treat a read error as off.

  When the engine is not configured, the callbacks do not even register work.
- **Reload, then decide.** The row is reloaded by primary key, including soft-deleted rows. A delete, a row that is gone, a soft-deleted row (a set `gorm.DeletedAt`) or a row whose `ShouldBeSearchable` is false has its document deleted. Restoring a soft-deleted row is an ordinary update and indexes it again. An error from `ToSearchableArray` is logged and nothing is sent, which lets a model refuse a document that would break scoping. A document without an `id` gets the key.
- **Rows only.** A statement without a primary key value, such as `Model(&T{}).Where(…).Updates(…)` or `Delete(&T{}, id)`, cannot be synced row by row and is skipped. Bulk paths call `beachcomber.Service.Sync` or `beachcomber.Service.Remove` per row, or reindex.
- **Candidates, not answers.** `beachcomber.Engine.SearchIDs` returns candidate ids from an external index that may be stale. Callers must re-gate every id in SQL (ownership, visibility, soft deletes) before they expose a row. An empty result is an empty list, never an error. The same holds for `beachcomber.SearchResult.Found`: it counts documents in the index, including stale ones, so a total shown to a user is recounted in SQL over re-gated ids, never taken from the engine.

## Usage

An application selects the engine in `config/search.yaml`:

```yaml
driver: typesense
typesense:
  api_key: ""   # set with SUMMER_SEARCH__TYPESENSE__API_KEY
```

A model implements `beachcomber.Searchable` without importing beachcomber:

```go
package models

func (Post) SearchableAs() string     { return "acme_blog_posts" }
func (Post) ShouldBeSearchable() bool { return true }

func (p *Post) ToSearchableArray(ctx context.Context, db *gorm.DB) (map[string]any, error) {
	return map[string]any{
		"id":      strconv.FormatUint(uint64(p.ID), 10),
		"blog_id": int64(p.BlogID),
		"title":   p.Title,
	}, nil
}

func (Post) SearchIndexSchema() map[string]any {
	return map[string]any{
		"fields": []map[string]any{
			{"name": "id", "type": "string"},
			{"name": "blog_id", "type": "int64"},
			{"name": "title", "type": "string"},
		},
	}
}
```

The plugin imports the engine package for its side effect, builds the service at Boot and installs its kill-switch:

```go
package acme

import (
	"context"

	"git.golem15.com/golem15/summercms/modules/backpack"
	"git.golem15.com/golem15/summercms/modules/beachcomber"
	_ "git.golem15.com/golem15/summercms/modules/beachcomber/typesense"
	"gorm.io/gorm"
)

func (p *Plugin) Boot(app *backpack.App) error {
	svc, err := beachcomber.From(app)
	if err != nil {
		return err
	}
	svc.SetGate(beachcomber.GateFunc(func(ctx context.Context, db *gorm.DB) bool {
		return acmeSearchEnabled(ctx, db) // false on any read error
	}))
	return nil
}
```

A search endpoint asks the engine for candidate ids:

```go
ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomber.Query{
	Q:        term,
	QueryBy:  []string{"title"},
	FilterBy: "blog_id:=" + strconv.FormatUint(uint64(blogID), 10),
})
```

## API reference

### beachcomber

| Identifier | Description |
|------------|-------------|
| `beachcomber.From(app)` | The app's `*beachcomber.Service`, built and published on first use. |
| `beachcomber.Service` | The search service: `SetGate`, `Engine`, `Prefix`, `IndexName`, `Logger`, `Sync`, `Remove`. |
| `beachcomber.Searchable` | `SearchableAs()`, `ToSearchableArray(ctx, db)`, `ShouldBeSearchable()`. |
| `beachcomber.IndexSchemaProvider` | `SearchIndexSchema()`: the schema a missing index is created with. |
| `beachcomber.SearchKeyer` | `SearchKey()`: replaces the decimal primary key as the document key. |
| `beachcomber.Engine` | `Name`, `Configured`, `Upsert`, `Delete`, `Flush`, `SearchIDs`. |
| `beachcomber.Query` | `Q`, `QueryBy`, `QueryByWeights`, `FilterBy`, `SortBy`, `Page`, `PerPage`. |
| `beachcomber.PageSearcher` | Optional engine interface: `SearchPage(ctx, index, q)` returns a page of ids and the engine's found count. |
| `beachcomber.SearchResult` | `IDs` (candidates) and `Found` (documents the engine matched). |
| `beachcomber.SearchPage(ctx, engine, index, q)` | Calls `PageSearcher` when the engine has it, else `SearchIDs` with `Found` set to the number of ids. |
| `beachcomber.Gate`, `beachcomber.GateFunc` | The application kill-switch: `Enabled(ctx, db) bool`. |
| `beachcomber.EngineFactory`, `beachcomber.RegisterEngine(name, factory)` | Registers an engine from an `init` function. |
| `beachcomber.NullEngine`, `beachcomber.DefaultDriver` | The name of the built-in engine that indexes nothing, and the default of `search.driver`. |
| `beachcomber.CallbackAfterCreate`, `beachcomber.CallbackAfterUpdate`, `beachcomber.CallbackAfterDelete` | Names of the GORM callbacks. |

### beachcomber/typesense

| Identifier | Description |
|------------|-------------|
| `typesense.Config`, `typesense.LoadConfig` | The `search.typesense.*` settings with their defaults; `BaseURL` is `{protocol}://{host}:{port}{path}`. |
| `typesense.Engine`, `typesense.New` | The `beachcomber.Engine` and `beachcomber.PageSearcher`, with `Config`. |
| `typesense.MaxPerPage` | 250, the largest page a search may ask for. |
| `typesense.StatusError` | A non-2xx answer: `Method`, `Path`, `Code` and `StatusCode()`. |
| `typesense.DriverName` | `typesense`. |
| `typesense.DefaultHost`, `typesense.DefaultPort`, `typesense.DefaultProtocol`, `typesense.DefaultConnectionTimeout`, `typesense.DefaultImportAction` | Defaults of the configuration keys. |

## Configuration

| Key | Default | Description |
|-----|---------|-------------|
| `search.driver` | `null` | `null`, or a registered engine such as `typesense`. |
| `search.prefix` | `""` | Prefix applied to every index name. |
| `search.typesense.api_key` | `""` | API key; empty means nothing is ever sent. |
| `search.typesense.host` | `localhost` | Typesense node host. |
| `search.typesense.port` | `8181` | Typesense node port. |
| `search.typesense.protocol` | `http` | `http` or `https`. |
| `search.typesense.path` | `""` | Path prefix of the node. |
| `search.typesense.connection_timeout_seconds` | `2` | Per-request timeout, in seconds or as a duration string. |
| `search.typesense.import_action` | `upsert` | The `action` of document imports. |

## Dependencies

- `backpack`, `compass` and `lagoon` (callback installation and `lagoon.AfterCommit`) from this repository.
- `gorm.io/gorm` (sync callbacks and reloads).
- The Typesense client is plain `net/http`; no Typesense SDK is used.

## Testing

```bash
go test ./modules/beachcomber/...
```

A test points `search.typesense.host` and `search.typesense.port` at an `httptest` server to see the exact Typesense requests. Sync is inline, so the requests have arrived when the write returns.