golem15/summercms
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(beachcomber): search highlights and field projection (quick-261108-pe4)

Browse Source 
- Additive: Query.IncludeFields, ExcludeFields and Highlight; left empty,
  the Typesense request is the same as before.
- SearchResult.Hits holds one Hit per id, in step with IDs; SearchPage
  always fills it.
- Snippets are plain-text Segment runs split at private-use highlight tags,
  never HTML; MaxSnippetsPerArray is applied client-side.
- Optional HighlightSearcher capability, checked with SupportsHighlights
  (true for Typesense, false for null and third-party engines).
- README and docs: Highlights sections with the rule that a caller drops
  the hit of every id failing the SQL re-check before showing a snippet.

Targets framework v0.6.4 (additive, no breaking change).
This commit is contained in:
Jakub Zych
2026-11-08 18:31:27 +01:00
parent 6f9b77033f
commit 7819c6746c
9 changed files with 756 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

69
docs/services/search.md
View File

@@ -135,6 +135,73 @@ A paginated endpoint also needs to know how many documents matched. An engine th
`beachcomber.Query.QueryByWeights` ranks the fields of `QueryBy`, one weight per field in the same order, as Scout's `query_by_weights` option does. `beachcomber.Query.QueryByWeights` ranks the fields of `QueryBy`, one weight per field in the same order, as Scout's `query_by_weights` option does.
## Highlights
Set `beachcomber.Query.Highlight` to get snippets of the matches with the IDs. `beachcomber.SearchResult.Hits` holds one `beachcomber.Hit` per ID, in the same order as `IDs`, including hits without highlights; `beachcomber.SearchPage` always fills it, also for engines that do not highlight. A `beachcomber.Snippet` names its `Field` and, for an array field, the element in `ArrayIndex` (-1 for a non-array field), and splits the excerpt into `beachcomber.Segment` runs of plain stored text with `Match` set on the matched ones. The text is never HTML: a stored `<b>` comes back as `<b>`, so escape `Text` when rendering and mark the matches yourself.
`beachcomber.Highlight.Fields` picks the fields (empty is the engine default), `AffixTokens` the context around a match and `MaxSnippetsPerArray` caps the snippets of one array field. An engine that fills highlights implements the optional `beachcomber.HighlightSearcher` interface; check with `beachcomber.SupportsHighlights`, and when it is false build snippets from the rows loaded after the re-check.
`beachcomber.Query.IncludeFields` and `beachcomber.Query.ExcludeFields` shrink the answer without changing what is searched or highlighted. A large text field can be stored, highlighted and still left out of the answer, so a page of hits stays small. The document ID is always returned.
Highlights belong to candidates, like the IDs. Drop the `Hit` of every ID that fails the SQL re-check before showing any snippet:
```go src=modules/beachcomber/typesense/example_test.go#ExampleEngine_SearchPage
// A stand-in Typesense node. The documents carry only id and title:
// text is excluded from the answer, yet still highlighted.
node := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
fmt.Println("exclude_fields", q.Get("exclude_fields"), "highlight_fields", q.Get("highlight_fields"),
"highlight_affix_num_tokens", q.Get("highlight_affix_num_tokens"))
fmt.Println("private-use start tag:", q.Get("highlight_start_tag") == "\ue000")
fmt.Fprint(w, `{"found":2,"hits":[`+
`{"document":{"id":"3","title":"Kind of Blue"},"highlights":[{"field":"text","snippet":"a <b> \ue000blue\ue001 note"}]},`+
`{"document":{"id":"8","title":"Draft"},"highlights":[{"field":"text","snippet":"private \ue000blue\ue001 draft"}]}]}`)
}))
defer node.Close()
u, _ := url.Parse(node.URL)
port, _ := strconv.Atoi(u.Port())
engine := typesense.New(typesense.Config{APIKey: "test-only-key", Host: u.Hostname(), Port: port, Protocol: "http"})
fmt.Println("highlights:", beachcomber.SupportsHighlights(engine))
res, err := beachcomber.SearchPage(context.Background(), engine, "acme_blog_posts", beachcomber.Query{
Q: "blue",
QueryBy: []string{"title", "text"},
ExcludeFields: []string{"text"},
Highlight: &beachcomber.Highlight{Fields: []string{"text"}, AffixTokens: 3},
})
if err != nil {
fmt.Println(err)
return
}
// Stands in for the SQL re-check: id 8 is not visible to the caller, so
// its hit is dropped before any snippet is shown.
allowed := map[string]bool{"3": true}
for _, hit := range res.Hits {
if !allowed[hit.ID] {
continue
}
for _, sn := range hit.Highlights {
var b strings.Builder
for _, seg := range sn.Segments {
text := html.EscapeString(seg.Text) // segments are plain text
if seg.Match {
text = "<mark>" + text + "</mark>"
}
b.WriteString(text)
}
fmt.Println(hit.ID, sn.Field, sn.ArrayIndex, b.String())
}
}
// Output:
// highlights: true
// exclude_fields text highlight_fields text highlight_affix_num_tokens 3
// private-use start tag: true
// 3 text -1 a &lt;b&gt; <mark>blue</mark> note
```
> [!WARNING]
> A snippet of a hit whose ID failed the SQL re-check leaks the private document: it quotes the indexed text of a draft, a deleted record or another user's data.
## Reindexing ## Reindexing
`beachcomber.Service.Sync` pushes one model on demand; a reindex command loops over the table with it. Two helpers cover the index itself: `beachcomber.Service.Sync` pushes one model on demand; a reindex command loops over the table with it. Two helpers cover the index itself:
@@ -193,4 +260,4 @@ fmt.Println(typesense.New(typesense.Config{}).Configured())
// false // false
``` ```
The Typesense engine implements `beachcomber.PageSearcher` on the same request, adding the answer's `found` count, and sends `query_by_weights` when the query has weights. A page is at most `typesense.MaxPerPage` (250) documents, Typesense's own limit: a larger `PerPage` is an error, so fetch more IDs page by page. Each request times out after `search.typesense.connection_timeout_seconds` (2 seconds by default). A failed answer is a `typesense.StatusError` with the method, path and status, never the answer body. The Typesense engine implements `beachcomber.PageSearcher` on the same request, adding the answer's `found` count, and sends `query_by_weights` when the query has weights. A page is at most `typesense.MaxPerPage` (250) documents, Typesense's own limit: a larger `PerPage` is an error, so fetch more IDs page by page. Each request times out after `search.typesense.connection_timeout_seconds` (2 seconds by default). A failed answer is a `typesense.StatusError` with the method, path and status, never the answer body. `include_fields` (always with `id`), `exclude_fields`, `highlight_fields`, `highlight_affix_num_tokens` and the highlight tags are sent only when the query sets them, and `MaxSnippetsPerArray` is applied by the engine after the answer, because Typesense has no parameter for it.

25
modules/beachcomber/README.md
View File

@@ -30,8 +30,9 @@ Use cases:
- Engine selection by `search.driver`: `null` (the default) or a registered engine such as `typesense`. An unknown name is a boot error that lists the registered engines. Third-party engines register with `beachcomber.RegisterEngine` and a `beachcomber.EngineFactory`; a duplicate name panics at init. - Engine selection by `search.driver`: `null` (the default) or a registered engine such as `typesense`. An unknown name is a boot error that lists the registered engines. Third-party engines register with `beachcomber.RegisterEngine` and a `beachcomber.EngineFactory`; a duplicate name panics at init.
- The `beachcomber.Searchable` model contract: `SearchableAs` (the index name, prefixed with `search.prefix`), `ToSearchableArray(ctx, db)` (the document, built from the committed row and free to query related rows) and `ShouldBeSearchable`. A model can also implement `beachcomber.IndexSchemaProvider` (the schema the engine creates a missing index with) and `beachcomber.SearchKeyer` (a document key other than the decimal primary key). - The `beachcomber.Searchable` model contract: `SearchableAs` (the index name, prefixed with `search.prefix`), `ToSearchableArray(ctx, db)` (the document, built from the committed row and free to query related rows) and `ShouldBeSearchable`. A model can also implement `beachcomber.IndexSchemaProvider` (the schema the engine creates a missing index with) and `beachcomber.SearchKeyer` (a document key other than the decimal primary key).
- The `beachcomber.Engine` driver contract: `Name`, `Configured`, `Upsert`, `Delete`, `Flush` and `SearchIDs` with a `beachcomber.Query`. `beachcomber.Query.QueryByWeights` ranks the `QueryBy` fields, one weight per field. - The `beachcomber.Engine` driver contract: `Name`, `Configured`, `Upsert`, `Delete`, `Flush` and `SearchIDs` with a `beachcomber.Query`. `beachcomber.Query.QueryByWeights` ranks the `QueryBy` fields, one weight per field. `beachcomber.Query.IncludeFields` and `beachcomber.Query.ExcludeFields` shrink the answer, and `beachcomber.Query.Highlight` asks for snippets.
- Search totals: the optional `beachcomber.PageSearcher` interface returns a `beachcomber.SearchResult`, one page of ids plus `Found`, the number of documents the engine matched. Call it through `beachcomber.SearchPage`, which falls back to `SearchIDs` (with `Found` set to the number of ids) for an engine that does not implement it; the `null` engine returns no ids and a zero count. - Search totals: the optional `beachcomber.PageSearcher` interface returns a `beachcomber.SearchResult`, one page of ids plus `Found`, the number of documents the engine matched. Call it through `beachcomber.SearchPage`, which falls back to `SearchIDs` (with `Found` set to the number of ids) for an engine that does not implement it; the `null` engine returns no ids and a zero count.
- Search highlights: `beachcomber.SearchResult.Hits` holds one `beachcomber.Hit` per id, in step with `IDs`, whose snippets are runs of plain stored text, never HTML. The optional `beachcomber.HighlightSearcher` interface marks an engine that fills them; ask through `beachcomber.SupportsHighlights`. See [Highlights](#highlights).
- Index lifecycle for reindex tooling: the optional `beachcomber.IndexDropper` interface drops an index and reports whether it existed, so a command can tell a dropped index from one that was already absent; call it through `beachcomber.DropIndex`, which falls back to `Flush` (reporting existed) for other engines. The optional `beachcomber.IndexEnsurer` creates an empty index from its schema; `beachcomber.EnsureIndex` calls it and is a no-op for other engines. A full reindex calls it first because an `Upsert` of zero documents creates nothing. - Index lifecycle for reindex tooling: the optional `beachcomber.IndexDropper` interface drops an index and reports whether it existed, so a command can tell a dropped index from one that was already absent; call it through `beachcomber.DropIndex`, which falls back to `Flush` (reporting existed) for other engines. The optional `beachcomber.IndexEnsurer` creates an empty index from its schema; `beachcomber.EnsureIndex` calls it and is a no-op for other engines. A full reindex calls it first because an `Upsert` of zero documents creates nothing.
- GORM callbacks `beachcomber.CallbackAfterCreate`, `beachcomber.CallbackAfterUpdate` and `beachcomber.CallbackAfterDelete`, installed once per `*gorm.DB`, register the sync with `lagoon.AfterCommit`. - GORM callbacks `beachcomber.CallbackAfterCreate`, `beachcomber.CallbackAfterUpdate` and `beachcomber.CallbackAfterDelete`, installed once per `*gorm.DB`, register the sync with `lagoon.AfterCommit`.
- `beachcomber.Service.Sync` and `beachcomber.Service.Remove` run the same gated path on demand, for reindex tooling, and return the error instead of logging it. - `beachcomber.Service.Sync` and `beachcomber.Service.Remove` run the same gated path on demand, for reindex tooling, and return the error instead of logging it.
@@ -41,6 +42,7 @@ Use cases:
- `Delete` and `Flush` treat 404 as success. - `Delete` and `Flush` treat 404 as success.
- `DropIndex` sends `DELETE /collections/{index}` and reports existed true on 2xx and false on 404; `EnsureIndex` runs the same read-then-create step as `Upsert`, so a second call sends no create. - `DropIndex` sends `DELETE /collections/{index}` and reports existed true on 2xx and false on 404; `EnsureIndex` runs the same read-then-create step as `Upsert`, so a second call sends no create.
- `SearchIDs` and `SearchPage` send one request with `q` (default `*`), `query_by`, `query_by_weights`, `filter_by`, `sort_by`, `page` and `per_page`, and return `hits[].document.id` in order; `SearchPage` adds the answer's `found`. A weight list whose length differs from `QueryBy`, or a `PerPage` above `typesense.MaxPerPage` (250, Typesense's limit), is an error before any request; callers page instead. - `SearchIDs` and `SearchPage` send one request with `q` (default `*`), `query_by`, `query_by_weights`, `filter_by`, `sort_by`, `page` and `per_page`, and return `hits[].document.id` in order; `SearchPage` adds the answer's `found`. A weight list whose length differs from `QueryBy`, or a `PerPage` above `typesense.MaxPerPage` (250, Typesense's limit), is an error before any request; callers page instead.
- `include_fields` (always with `id`), `exclude_fields`, `highlight_fields`, `highlight_affix_num_tokens` and private-use highlight start and end tags are sent only when the query sets them. The `highlights` array of each hit is split into segments at those tags, with `indices` filling `ArrayIndex` for array fields. `MaxSnippetsPerArray` is applied client-side, because Typesense has no such parameter. An `ExcludeFields` that names `id`, or a negative highlight setting, is an error before any request.
- Ids and index names are path-escaped. - Ids and index names are path-escaped.
- A non-2xx answer is a `typesense.StatusError` with the method, path and status, never the answer body. - A non-2xx answer is a `typesense.StatusError` with the method, path and status, never the answer body.
@@ -58,6 +60,15 @@ Use cases:
- **Rows only.** A statement without a primary key value, such as `Model(&T{}).Where(…).Updates(…)` or `Delete(&T{}, id)`, cannot be synced row by row and is skipped. Bulk paths call `beachcomber.Service.Sync` or `beachcomber.Service.Remove` per row, or reindex. - **Rows only.** A statement without a primary key value, such as `Model(&T{}).Where(…).Updates(…)` or `Delete(&T{}, id)`, cannot be synced row by row and is skipped. Bulk paths call `beachcomber.Service.Sync` or `beachcomber.Service.Remove` per row, or reindex.
- **Candidates, not answers.** `beachcomber.Engine.SearchIDs` returns candidate ids from an external index that may be stale. Callers must re-gate every id in SQL (ownership, visibility, soft deletes) before they expose a row. An empty result is an empty list, never an error. The same holds for `beachcomber.SearchResult.Found`: it counts documents in the index, including stale ones, so a total shown to a user is recounted in SQL over re-gated ids, never taken from the engine. - **Candidates, not answers.** `beachcomber.Engine.SearchIDs` returns candidate ids from an external index that may be stale. Callers must re-gate every id in SQL (ownership, visibility, soft deletes) before they expose a row. An empty result is an empty list, never an error. The same holds for `beachcomber.SearchResult.Found`: it counts documents in the index, including stale ones, so a total shown to a user is recounted in SQL over re-gated ids, never taken from the engine.
## Highlights
- `beachcomber.Query.Highlight` asks for snippets: `Fields` (empty is the engine default, the `QueryBy` fields for Typesense), `AffixTokens` (tokens of context around a match, 0 is the engine default) and `MaxSnippetsPerArray` (0 is no cap). There is no option to highlight a whole field.
- `beachcomber.SearchResult.Hits` holds one `beachcomber.Hit` per id, in `IDs` order (`Hits[i].ID == IDs[i]`), including hits without highlights. An engine called directly may leave it nil; `beachcomber.SearchPage` always fills it.
- Each `beachcomber.Snippet` has `Field`, `ArrayIndex` (the element of an array field, -1 for a non-array field) and `beachcomber.Segment` runs with `Text` and `Match`. The text is plain stored text, verbatim, never HTML: escape `Text` when rendering and never parse markup out of it.
- **Highlights belong to candidates.** A snippet is cut from the indexed document, so it can quote a draft, a deleted record or another user's data. Drop the `Hit` of every id that fails the SQL privacy re-check before showing any snippet; otherwise a snippet leaks the private document.
- `beachcomber.SupportsHighlights` tells whether the engine fills highlights (true for Typesense, false for `null` and for engines without `beachcomber.HighlightSearcher`). When it is false, build snippets from the rows loaded after the re-check.
- `beachcomber.Query.IncludeFields` and `beachcomber.Query.ExcludeFields` shrink the answer. A field left out of the answer stays searchable and highlightable, so a large text field can be stored, highlighted and still excluded from the answer. The document id is always returned.
## Usage ## Usage
An application selects the engine in `config/search.yaml`: An application selects the engine in `config/search.yaml`:
@@ -143,10 +154,14 @@ ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomb
| `beachcomber.IndexSchemaProvider` | `SearchIndexSchema()`: the schema a missing index is created with. | | `beachcomber.IndexSchemaProvider` | `SearchIndexSchema()`: the schema a missing index is created with. |
| `beachcomber.SearchKeyer` | `SearchKey()`: replaces the decimal primary key as the document key. | | `beachcomber.SearchKeyer` | `SearchKey()`: replaces the decimal primary key as the document key. |
| `beachcomber.Engine` | `Name`, `Configured`, `Upsert`, `Delete`, `Flush`, `SearchIDs`. | | `beachcomber.Engine` | `Name`, `Configured`, `Upsert`, `Delete`, `Flush`, `SearchIDs`. |
| `beachcomber.Query` | `Q`, `QueryBy`, `QueryByWeights`, `FilterBy`, `SortBy`, `Page`, `PerPage`. | | `beachcomber.Query` | `Q`, `QueryBy`, `QueryByWeights`, `FilterBy`, `SortBy`, `Page`, `PerPage`, `IncludeFields`, `ExcludeFields`, `Highlight`. |
| `beachcomber.Highlight` | `Fields`, `AffixTokens`, `MaxSnippetsPerArray`: the snippets a query asks for. |
| `beachcomber.PageSearcher` | Optional engine interface: `SearchPage(ctx, index, q)` returns a page of ids and the engine's found count. | | `beachcomber.PageSearcher` | Optional engine interface: `SearchPage(ctx, index, q)` returns a page of ids and the engine's found count. |
| `beachcomber.SearchResult` | `IDs` (candidates) and `Found` (documents the engine matched). | | `beachcomber.SearchResult` | `IDs` (candidates), `Found` (documents the engine matched) and `Hits` (one per id, in the same order). |
| `beachcomber.SearchPage(ctx, engine, index, q)` | Calls `PageSearcher` when the engine has it, else `SearchIDs` with `Found` set to the number of ids. | | `beachcomber.Hit`, `beachcomber.Snippet`, `beachcomber.Segment` | A hit's `ID` and `Highlights`; a snippet's `Field`, `ArrayIndex` and `Segments`; a segment's plain `Text` and `Match`. |
| `beachcomber.SearchPage(ctx, engine, index, q)` | Calls `PageSearcher` when the engine has it, else `SearchIDs` with `Found` set to the number of ids; always fills `Hits` in step with `IDs`. |
| `beachcomber.HighlightSearcher` | Optional engine interface: a `PageSearcher` with `SupportsHighlights()`, filling `Hit.Highlights`. |
| `beachcomber.SupportsHighlights(e)` | Whether the engine implements `HighlightSearcher` and answers true. |
| `beachcomber.IndexDropper` | Optional engine interface: `DropIndex(ctx, index)` drops the index and reports whether it existed. | | `beachcomber.IndexDropper` | Optional engine interface: `DropIndex(ctx, index)` drops the index and reports whether it existed. |
| `beachcomber.DropIndex(ctx, engine, index)` | Calls `IndexDropper` when the engine has it, else `Flush` and reports existed true. | | `beachcomber.DropIndex(ctx, engine, index)` | Calls `IndexDropper` when the engine has it, else `Flush` and reports existed true. |
| `beachcomber.IndexEnsurer` | Optional engine interface: `EnsureIndex(ctx, index, schema)` creates a missing index, empty, from its schema. | | `beachcomber.IndexEnsurer` | Optional engine interface: `EnsureIndex(ctx, index, schema)` creates a missing index, empty, from its schema. |
@@ -161,7 +176,7 @@ ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomb
| Identifier | Description | | Identifier | Description |
|------------|-------------| |------------|-------------|
| `typesense.Config`, `typesense.LoadConfig` | The `search.typesense.*` settings with their defaults; `BaseURL` is `{protocol}://{host}:{port}{path}`. | | `typesense.Config`, `typesense.LoadConfig` | The `search.typesense.*` settings with their defaults; `BaseURL` is `{protocol}://{host}:{port}{path}`. |
| `typesense.Engine`, `typesense.New` | The `beachcomber.Engine`, `beachcomber.PageSearcher`, `beachcomber.IndexDropper` and `beachcomber.IndexEnsurer`, with `Config`. | | `typesense.Engine`, `typesense.New` | The `beachcomber.Engine`, `beachcomber.PageSearcher`, `beachcomber.HighlightSearcher`, `beachcomber.IndexDropper` and `beachcomber.IndexEnsurer`, with `Config`. |
| `typesense.MaxPerPage` | 250, the largest page a search may ask for. | | `typesense.MaxPerPage` | 250, the largest page a search may ask for. |
| `typesense.StatusError` | A non-2xx answer: `Method`, `Path`, `Code` and `StatusCode()`. | | `typesense.StatusError` | A non-2xx answer: `Method`, `Path`, `Code` and `StatusCode()`. |
| `typesense.DriverName` | `typesense`. | | `typesense.DriverName` | `typesense`. |

5
modules/beachcomber/engines.go
View File

@@ -62,7 +62,8 @@ func init() {
} }
// nullEngine is never configured, so sync is skipped before it is called. // nullEngine is never configured, so sync is skipped before it is called.
// Every method is a no-op; SearchIDs and SearchPage return no ids. // Every method is a no-op; SearchIDs and SearchPage return no ids. It does
// not implement HighlightSearcher.
type nullEngine struct{} type nullEngine struct{}
func (nullEngine) Name() string { return NullEngine } func (nullEngine) Name() string { return NullEngine }
@@ -76,5 +77,5 @@ func (nullEngine) SearchIDs(context.Context, string, Query) ([]string, error) {
return []string{}, nil return []string{}, nil
} }
func (nullEngine) SearchPage(context.Context, string, Query) (SearchResult, error) { func (nullEngine) SearchPage(context.Context, string, Query) (SearchResult, error) {
return SearchResult{IDs: []string{}}, nil return SearchResult{IDs: []string{}, Hits: []Hit{}}, nil
} }

101
modules/beachcomber/searchable.go
View File

@@ -75,6 +75,32 @@ type Query struct {
Page int Page int
// PerPage is the page size; 0 leaves the engine default. // PerPage is the page size; 0 leaves the engine default.
PerPage int PerPage int
// IncludeFields and ExcludeFields only limit which document fields the
// engine sends back: a field left out of the answer stays searchable
// and highlightable. Empty keeps the engine default (every stored
// field). The document key is always returned, because the ids are the
// result. Engines that cannot project ignore them.
IncludeFields []string
ExcludeFields []string
// Highlight asks for snippets of the matches in SearchResult.Hits. Nil
// asks for no highlights and keeps the request as it was. Engines
// without HighlightSearcher ignore it.
Highlight *Highlight
}
// Highlight configures the snippets an engine returns for a Query.
// Snippets are always cut around the matches; there is deliberately no
// option to highlight a whole field.
type Highlight struct {
// Fields are the fields to highlight. Empty is the engine default,
// which is the QueryBy fields for Typesense.
Fields []string
// AffixTokens is the number of tokens of context on each side of a
// match; 0 is the engine default.
AffixTokens int
// MaxSnippetsPerArray is the most snippets kept for one array field;
// 0 means no cap.
MaxSnippetsPerArray int
} }
// SearchResult is one page of candidate ids and the number of documents // SearchResult is one page of candidate ids and the number of documents
@@ -83,6 +109,36 @@ type Query struct {
type SearchResult struct { type SearchResult struct {
IDs []string IDs []string
Found int Found int
// Hits has one entry per id, in the same order as IDs
// (Hits[i].ID == IDs[i]). An engine called directly may leave it nil;
// SearchPage always fills it. Like the ids, hits are candidates: drop
// the Hit of every id the SQL re-check rejects before showing a
// snippet.
Hits []Hit
}
// Hit is the engine's answer for one id beyond the id itself.
type Hit struct {
ID string
// Highlights is nil when the hit has none.
Highlights []Snippet
}
// Snippet is one highlighted excerpt of a field.
type Snippet struct {
Field string
// ArrayIndex is the element index for an array field, and -1 for a
// non-array field or when the engine did not say which element.
ArrayIndex int
Segments []Segment
}
// Segment is a run of a snippet that either matched the query or not.
// Text is plain stored text, verbatim, never HTML or markup. It can
// contain < or &, so the caller escapes it when rendering.
type Segment struct {
Text string
Match bool
} }
// PageSearcher is implemented by an engine that can report how many // PageSearcher is implemented by an engine that can report how many
@@ -92,18 +148,51 @@ type PageSearcher interface {
SearchPage(ctx context.Context, index string, q Query) (SearchResult, error) SearchPage(ctx context.Context, index string, q Query) (SearchResult, error)
} }
// HighlightSearcher is implemented by an engine that can return snippets
// of the matches. It is optional so that existing engines stay valid. An
// engine that implements it and answers true fills Hit.Highlights for a
// Query with Highlight; ask through SupportsHighlights.
type HighlightSearcher interface {
PageSearcher
SupportsHighlights() bool
}
// SupportsHighlights reports whether e fills Hit.Highlights: true only when
// e implements HighlightSearcher and its SupportsHighlights answers true. A
// nil e is false. When it is false, an app builds its own snippets from the
// rows it loads.
func SupportsHighlights(e Engine) bool {
hs, ok := e.(HighlightSearcher)
return ok && hs.SupportsHighlights()
}
// SearchPage asks e for one page of q: through PageSearcher when e // SearchPage asks e for one page of q: through PageSearcher when e
// implements it, otherwise through SearchIDs with Found set to the number // implements it, otherwise through SearchIDs with Found set to the number
// of ids returned. // of ids returned. On success Hits is always in step with IDs: when the
// engine left it nil, it is one Hit without highlights per id (an empty,
// non-nil slice for zero ids); hits the engine filled are kept as they are.
func SearchPage(ctx context.Context, e Engine, index string, q Query) (SearchResult, error) { func SearchPage(ctx context.Context, e Engine, index string, q Query) (SearchResult, error) {
var res SearchResult
if ps, ok := e.(PageSearcher); ok { if ps, ok := e.(PageSearcher); ok {
return ps.SearchPage(ctx, index, q) r, err := ps.SearchPage(ctx, index, q)
if err != nil {
return r, err
}
res = r
} else {
ids, err := e.SearchIDs(ctx, index, q)
if err != nil {
return SearchResult{}, err
}
res = SearchResult{IDs: ids, Found: len(ids)}
} }
ids, err := e.SearchIDs(ctx, index, q) if res.Hits == nil {
if err != nil { res.Hits = make([]Hit, len(res.IDs))
return SearchResult{}, err for i, id := range res.IDs {
res.Hits[i] = Hit{ID: id}
}
} }
return SearchResult{IDs: ids, Found: len(ids)}, nil return res, nil
} }
// IndexDropper is implemented by an engine that can tell whether the index // IndexDropper is implemented by an engine that can tell whether the index

56
modules/beachcomber/searchpage_test.go
View File

@@ -45,7 +45,7 @@ func TestSearchPageFallsBackToSearchIDs(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if !reflect.DeepEqual(res, SearchResult{IDs: []string{"3", "1"}, Found: 2}) { if !reflect.DeepEqual(res, SearchResult{IDs: []string{"3", "1"}, Found: 2, Hits: []Hit{{ID: "3"}, {ID: "1"}}}) {
t.Fatalf("res = %+v", res) t.Fatalf("res = %+v", res)
} }
if !reflect.DeepEqual(e.got, q) { if !reflect.DeepEqual(e.got, q) {
@@ -74,11 +74,63 @@ func TestSearchPageNullEngine(t *testing.T) {
t.Fatal("the null engine implements PageSearcher") t.Fatal("the null engine implements PageSearcher")
} }
res, err := SearchPage(context.Background(), e, "posts", Query{}) res, err := SearchPage(context.Background(), e, "posts", Query{})
if err != nil || res.Found != 0 || res.IDs == nil || len(res.IDs) != 0 { if err != nil || res.Found != 0 || res.IDs == nil || len(res.IDs) != 0 || res.Hits == nil || len(res.Hits) != 0 {
t.Fatalf("res = %+v err = %v", res, err) t.Fatalf("res = %+v err = %v", res, err)
} }
} }
// highlightEngine implements HighlightSearcher with a configurable answer
// and can return its own hits.
type highlightEngine struct {
pagedEngine
supports bool
hits []Hit
}
func (e *highlightEngine) SupportsHighlights() bool { return e.supports }
func (e *highlightEngine) SearchPage(_ context.Context, _ string, q Query) (SearchResult, error) {
e.got = q
return SearchResult{IDs: e.ids, Found: e.found, Hits: e.hits}, nil
}
func TestSupportsHighlights(t *testing.T) {
for name, e := range map[string]Engine{
"null": nullEngine{},
"ids-only": &idsOnlyEngine{},
"paged": &pagedEngine{},
"nil": nil,
} {
if SupportsHighlights(e) {
t.Fatalf("%s: SupportsHighlights = true", name)
}
}
if !SupportsHighlights(&highlightEngine{supports: true}) {
t.Fatal("an engine answering true supports highlights")
}
if SupportsHighlights(&highlightEngine{supports: false}) {
t.Fatal("an engine answering false does not support highlights")
}
}
func TestSearchPageFillsHits(t *testing.T) {
ctx := context.Background()
res, err := SearchPage(ctx, &pagedEngine{idsOnlyEngine: idsOnlyEngine{ids: []string{"7"}}, found: 1}, "posts", Query{})
if err != nil || !reflect.DeepEqual(res.Hits, []Hit{{ID: "7"}}) {
t.Fatalf("hits = %+v err = %v", res.Hits, err)
}
own := []Hit{{ID: "7", Highlights: []Snippet{{Field: "title", ArrayIndex: -1, Segments: []Segment{{Text: "x", Match: true}}}}}}
he := &highlightEngine{pagedEngine: pagedEngine{idsOnlyEngine: idsOnlyEngine{ids: []string{"7"}}}, supports: true, hits: own}
res, err = SearchPage(ctx, he, "posts", Query{Highlight: &Highlight{}})
if err != nil || !reflect.DeepEqual(res.Hits, own) {
t.Fatalf("own hits = %+v err = %v", res.Hits, err)
}
res, err = SearchPage(ctx, &pagedEngine{}, "posts", Query{})
if err != nil || res.Hits == nil || len(res.Hits) != 0 {
t.Fatalf("zero ids: hits = %#v err = %v", res.Hits, err)
}
}
// flushEngine records Flush calls and can fail them. // flushEngine records Flush calls and can fail them.
type flushEngine struct { type flushEngine struct {
idsOnlyEngine idsOnlyEngine

79
modules/beachcomber/typesense/engine.go
View File

@@ -10,6 +10,7 @@ import (
"maps" "maps"
"net/http" "net/http"
"net/url" "net/url"
"slices"
"strconv" "strconv"
"strings" "strings"
@@ -48,8 +49,9 @@ type Engine struct {
var _ beachcomber.Engine = (*Engine)(nil) var _ beachcomber.Engine = (*Engine)(nil)
var ( var (
_ beachcomber.IndexDropper = (*Engine)(nil) _ beachcomber.IndexDropper = (*Engine)(nil)
_ beachcomber.IndexEnsurer = (*Engine)(nil) _ beachcomber.IndexEnsurer = (*Engine)(nil)
_ beachcomber.HighlightSearcher = (*Engine)(nil)
) )
// New returns an engine for cfg. Every request is bounded by // New returns an engine for cfg. Every request is bounded by
@@ -252,6 +254,9 @@ type searchAnswer struct {
Document struct { Document struct {
ID json.RawMessage `json:"id"` ID json.RawMessage `json:"id"`
} `json:"document"` } `json:"document"`
// Highlights stays raw so an odd highlights value never makes
// the answer unreadable.
Highlights json.RawMessage `json:"highlights"`
} `json:"hits"` } `json:"hits"`
} }
@@ -261,9 +266,11 @@ const MaxPerPage = 250
// SearchIDs sends GET /collections/{index}/documents/search with q // SearchIDs sends GET /collections/{index}/documents/search with q
// (default "*"), query_by, query_by_weights, filter_by, sort_by, page and // (default "*"), query_by, query_by_weights, filter_by, sort_by, page and
// per_page, and returns hits[].document.id in order. The ids are // per_page, and returns hits[].document.id in order. include_fields
// candidates: the caller must re-check each one in SQL before exposing it. // (always with id), exclude_fields and, for Query.Highlight,
// No hits is an empty list. // highlight_fields, highlight_affix_num_tokens and the highlight tags are
// sent only when set. The ids are candidates: the caller must re-check
// each one in SQL before exposing it. No hits is an empty list.
func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Query) ([]string, error) { func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Query) ([]string, error) {
res, err := e.search(ctx, index, q) res, err := e.search(ctx, index, q)
if err != nil { if err != nil {
@@ -272,12 +279,19 @@ func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Quer
return res.IDs, nil return res.IDs, nil
} }
// SearchPage is SearchIDs plus the found count of the answer: how many // SearchPage is SearchIDs plus the found count of the answer (how many
// documents matched in the index, not how many the caller may see. // documents matched in the index, not how many the caller may see) and
// Hits, one per id in the same order. With Query.Highlight set, each Hit
// carries the snippets of the answer's highlights array, split into plain
// text segments; without it, Highlights stays nil.
func (e *Engine) SearchPage(ctx context.Context, index string, q beachcomber.Query) (beachcomber.SearchResult, error) { func (e *Engine) SearchPage(ctx context.Context, index string, q beachcomber.Query) (beachcomber.SearchResult, error) {
return e.search(ctx, index, q) return e.search(ctx, index, q)
} }
// SupportsHighlights returns true: SearchPage fills Hits[].Highlights when
// Query.Highlight is set. It implements beachcomber.HighlightSearcher.
func (e *Engine) SupportsHighlights() bool { return true }
func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query) (beachcomber.SearchResult, error) { func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query) (beachcomber.SearchResult, error) {
if len(q.QueryByWeights) > 0 && len(q.QueryByWeights) != len(q.QueryBy) { if len(q.QueryByWeights) > 0 && len(q.QueryByWeights) != len(q.QueryBy) {
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: %d query_by_weights for %d query_by fields", index, len(q.QueryByWeights), len(q.QueryBy)) return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: %d query_by_weights for %d query_by fields", index, len(q.QueryByWeights), len(q.QueryBy))
@@ -285,6 +299,19 @@ func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query)
if q.PerPage > MaxPerPage { if q.PerPage > MaxPerPage {
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: per_page %d exceeds %d", index, q.PerPage, MaxPerPage) return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: per_page %d exceeds %d", index, q.PerPage, MaxPerPage)
} }
for _, f := range q.ExcludeFields {
if f == "id" {
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: exclude_fields cannot leave out id, because the ids are the result", index)
}
}
if h := q.Highlight; h != nil {
if h.AffixTokens < 0 {
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: Highlight.AffixTokens %d is negative", index, h.AffixTokens)
}
if h.MaxSnippetsPerArray < 0 {
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: Highlight.MaxSnippetsPerArray %d is negative", index, h.MaxSnippetsPerArray)
}
}
params := url.Values{} params := url.Values{}
text := q.Q text := q.Q
if text == "" { if text == "" {
@@ -313,6 +340,27 @@ func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query)
if q.PerPage > 0 { if q.PerPage > 0 {
params.Set("per_page", strconv.Itoa(q.PerPage)) params.Set("per_page", strconv.Itoa(q.PerPage))
} }
if len(q.IncludeFields) > 0 {
// The Typesense docs do not promise id under include_fields.
fields := q.IncludeFields
if !slices.Contains(fields, "id") {
fields = append(slices.Clip(fields), "id")
}
params.Set("include_fields", strings.Join(fields, ","))
}
if len(q.ExcludeFields) > 0 {
params.Set("exclude_fields", strings.Join(q.ExcludeFields, ","))
}
if h := q.Highlight; h != nil {
params.Set("highlight_start_tag", highlightStartTag)
params.Set("highlight_end_tag", highlightEndTag)
if len(h.Fields) > 0 {
params.Set("highlight_fields", strings.Join(h.Fields, ","))
}
if h.AffixTokens > 0 {
params.Set("highlight_affix_num_tokens", strconv.Itoa(h.AffixTokens))
}
}
path := "/collections/" + url.PathEscape(index) + "/documents/search" path := "/collections/" + url.PathEscape(index) + "/documents/search"
code, answer, err := e.do(ctx, http.MethodGet, path, params, nil, "") code, answer, err := e.do(ctx, http.MethodGet, path, params, nil, "")
if err != nil { if err != nil {
@@ -326,19 +374,28 @@ func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query)
return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: unreadable answer", index) return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: unreadable answer", index)
} }
ids := make([]string, 0, len(res.Hits)) ids := make([]string, 0, len(res.Hits))
hits := make([]beachcomber.Hit, 0, len(res.Hits))
for _, h := range res.Hits { for _, h := range res.Hits {
raw := bytes.TrimSpace(h.Document.ID) raw := bytes.TrimSpace(h.Document.ID)
if len(raw) == 0 || bytes.Equal(raw, []byte("null")) { if len(raw) == 0 || bytes.Equal(raw, []byte("null")) {
continue continue
} }
id := string(raw)
var s string var s string
if err := json.Unmarshal(raw, &s); err == nil { if err := json.Unmarshal(raw, &s); err == nil {
ids = append(ids, s) id = s
continue
} }
ids = append(ids, string(raw)) hit := beachcomber.Hit{ID: id}
// Only with Highlight set are the tags the private-use characters;
// otherwise the snippets carry Typesense's default mark tags and
// must not be read as text.
if q.Highlight != nil {
hit.Highlights = parseHighlights(h.Highlights, q.Highlight)
}
ids = append(ids, id)
hits = append(hits, hit)
} }
return beachcomber.SearchResult{IDs: ids, Found: res.Found}, nil return beachcomber.SearchResult{IDs: ids, Found: res.Found, Hits: hits}, nil
} }
// do sends one request with the API key header and returns the status and // do sends one request with the API key header and returns the status and

56
modules/beachcomber/typesense/example_test.go
View File

@@ -3,10 +3,12 @@ package typesense_test
import ( import (
"context" "context"
"fmt" "fmt"
"html"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/url" "net/url"
"strconv" "strconv"
"strings"
"time" "time"
"git.golem15.com/golem15/summercms/modules/beachcomber" "git.golem15.com/golem15/summercms/modules/beachcomber"
@@ -47,3 +49,57 @@ func ExampleEngine_SearchIDs() {
// [3 1] <nil> // [3 1] <nil>
// false // false
} }
func ExampleEngine_SearchPage() {
// A stand-in Typesense node. The documents carry only id and title:
// text is excluded from the answer, yet still highlighted.
node := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
fmt.Println("exclude_fields", q.Get("exclude_fields"), "highlight_fields", q.Get("highlight_fields"),
"highlight_affix_num_tokens", q.Get("highlight_affix_num_tokens"))
fmt.Println("private-use start tag:", q.Get("highlight_start_tag") == "\ue000")
fmt.Fprint(w, `{"found":2,"hits":[`+
`{"document":{"id":"3","title":"Kind of Blue"},"highlights":[{"field":"text","snippet":"a <b> \ue000blue\ue001 note"}]},`+
`{"document":{"id":"8","title":"Draft"},"highlights":[{"field":"text","snippet":"private \ue000blue\ue001 draft"}]}]}`)
}))
defer node.Close()
u, _ := url.Parse(node.URL)
port, _ := strconv.Atoi(u.Port())
engine := typesense.New(typesense.Config{APIKey: "test-only-key", Host: u.Hostname(), Port: port, Protocol: "http"})
fmt.Println("highlights:", beachcomber.SupportsHighlights(engine))
res, err := beachcomber.SearchPage(context.Background(), engine, "acme_blog_posts", beachcomber.Query{
Q: "blue",
QueryBy: []string{"title", "text"},
ExcludeFields: []string{"text"},
Highlight: &beachcomber.Highlight{Fields: []string{"text"}, AffixTokens: 3},
})
if err != nil {
fmt.Println(err)
return
}
// Stands in for the SQL re-check: id 8 is not visible to the caller, so
// its hit is dropped before any snippet is shown.
allowed := map[string]bool{"3": true}
for _, hit := range res.Hits {
if !allowed[hit.ID] {
continue
}
for _, sn := range hit.Highlights {
var b strings.Builder
for _, seg := range sn.Segments {
text := html.EscapeString(seg.Text) // segments are plain text
if seg.Match {
text = "<mark>" + text + "</mark>"
}
b.WriteString(text)
}
fmt.Println(hit.ID, sn.Field, sn.ArrayIndex, b.String())
}
}
// Output:
// highlights: true
// exclude_fields text highlight_fields text highlight_affix_num_tokens 3
// private-use start tag: true
// 3 text -1 a &lt;b&gt; <mark>blue</mark> note
}

116
modules/beachcomber/typesense/highlights.go Normal file
View File

@@ -0,0 +1,116 @@
package typesense
import (
"encoding/json"
"strings"
"git.golem15.com/golem15/summercms/modules/beachcomber"
)
// highlightStartTag and highlightEndTag are two Unicode private-use
// characters that do not occur in normal text. The engine sends them as
// highlight_start_tag and highlight_end_tag, so a snippet can be split into
// segments without parsing markup.
const (
highlightStartTag = ""
highlightEndTag = ""
)
// splitSegments splits a snippet tagged with highlightStartTag and
// highlightEndTag into matched and unmatched runs. Empty runs are left out
// and the text is never trimmed, escaped or unescaped.
func splitSegments(s string) []beachcomber.Segment {
var out []beachcomber.Segment
match := false
for s != "" {
tag := highlightStartTag
if match {
tag = highlightEndTag
}
i := strings.Index(s, tag)
if i < 0 {
out = appendSegment(out, s, match)
break
}
out = appendSegment(out, s[:i], match)
s = s[i+len(tag):]
match = !match
}
return out
}
func appendSegment(out []beachcomber.Segment, text string, match bool) []beachcomber.Segment {
if text == "" {
return out
}
return append(out, beachcomber.Segment{Text: text, Match: match})
}
// tsHighlight is one entry of a hit's highlights array.
// A string field carries snippet; a string[] field carries snippets with
// the element index of each in indices.
type tsHighlight struct {
Field string `json:"field"`
Snippet string `json:"snippet"`
Snippets []string `json:"snippets"`
Indices []int `json:"indices"`
}
// parseHighlights turns a hit's highlights array into snippets, in the
// order of the array. An entry that does not decode, or has no snippet,
// contributes nothing; a value that is not an array gives nil.
func parseHighlights(raw json.RawMessage, h *beachcomber.Highlight) []beachcomber.Snippet {
if len(raw) == 0 {
return nil
}
var entries []json.RawMessage
if err := json.Unmarshal(raw, &entries); err != nil {
return nil
}
var out []beachcomber.Snippet
for _, entry := range entries {
var th tsHighlight
if err := json.Unmarshal(entry, &th); err != nil {
continue
}
if len(th.Snippets) > 0 {
out = appendArraySnippets(out, th, h)
continue
}
if th.Snippet == "" {
continue
}
segs := splitSegments(th.Snippet)
if len(segs) == 0 {
continue
}
out = append(out, beachcomber.Snippet{Field: th.Field, ArrayIndex: -1, Segments: segs})
}
return out
}
// appendArraySnippets appends one snippet per element of an array field's
// snippets, with ArrayIndex taken from indices (-1 when indices does not
// cover the element). Typesense 28.0 has no parameter that caps snippets
// per array (checked 2026-11-08), so MaxSnippetsPerArray is applied here
// and keeps the first snippets in Typesense's order. Elements with no
// segments are left out and do not count toward the cap.
func appendArraySnippets(out []beachcomber.Snippet, th tsHighlight, h *beachcomber.Highlight) []beachcomber.Snippet {
kept := 0
for i, sn := range th.Snippets {
if h != nil && h.MaxSnippetsPerArray > 0 && kept >= h.MaxSnippetsPerArray {
break
}
segs := splitSegments(sn)
if len(segs) == 0 {
continue
}
idx := -1
if i < len(th.Indices) {
idx = th.Indices[i]
}
out = append(out, beachcomber.Snippet{Field: th.Field, ArrayIndex: idx, Segments: segs})
kept++
}
return out
}

276
modules/beachcomber/typesense/highlights_test.go Normal file
View File

@@ -0,0 +1,276 @@
package typesense
import (
"net/url"
"reflect"
"sort"
"strings"
"testing"
"git.golem15.com/golem15/summercms/modules/beachcomber"
)
const searchKey = "GET /ts/collections/posts/documents/search"
// onlyQuery returns the parsed query of the single call f received.
func onlyQuery(t *testing.T, f *fakeTypesense) url.Values {
t.Helper()
calls := f.take()
if len(calls) != 1 {
t.Fatalf("calls = %d, want 1", len(calls))
}
params, err := url.ParseQuery(calls[0].rawQuery)
if err != nil {
t.Fatal(err)
}
return params
}
func TestTypesenseSearchSendsNoNewParamsByDefault(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":2,"hits":[`+
`{"document":{"id":"42"},"highlights":[{"field":"name","snippet":"a <mark>blue</mark> train","matched_tokens":["blue"]}]},`+
`{"document":{"id":7}}]}`)
q := beachcomber.Query{
Q: "blue train",
QueryBy: []string{"name", "artist_display", "notes"},
QueryByWeights: []int{10, 10, 1},
FilterBy: "collection_id:=5",
SortBy: "created_at:desc",
Page: 2,
PerPage: 250,
}
res, err := e.SearchPage(t.Context(), "posts", q)
if err != nil {
t.Fatal(err)
}
params := onlyQuery(t, f)
keys := make([]string, 0, len(params))
for k := range params {
keys = append(keys, k)
}
sort.Strings(keys)
want := []string{"filter_by", "page", "per_page", "q", "query_by", "query_by_weights", "sort_by"}
if !reflect.DeepEqual(keys, want) {
t.Fatalf("keys = %v, want %v", keys, want)
}
if res.Found != 2 || !reflect.DeepEqual(res.IDs, []string{"42", "7"}) {
t.Fatalf("res = %+v", res)
}
wantHits := []beachcomber.Hit{{ID: "42"}, {ID: "7"}}
if !reflect.DeepEqual(res.Hits, wantHits) {
t.Fatalf("hits = %+v, want %+v (mark snippets must not be read)", res.Hits, wantHits)
}
}
func TestTypesenseHighlightsStringSnippet(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":1,"hits":[{"document":{"id":"5"},"highlights":[{"field":"title","snippet":"a blue train","matched_tokens":["blue"]}]}]}`)
res, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{
Q: "blue",
QueryBy: []string{"title"},
Highlight: &beachcomber.Highlight{Fields: []string{"title"}, AffixTokens: 6},
})
if err != nil {
t.Fatal(err)
}
params := onlyQuery(t, f)
if params.Get("highlight_fields") != "title" || params.Get("highlight_affix_num_tokens") != "6" ||
params.Get("highlight_start_tag") != "" || params.Get("highlight_end_tag") != "" {
t.Fatalf("params = %v", params)
}
if params.Has("highlight_full_fields") {
t.Fatal("highlight_full_fields must never be sent")
}
want := []beachcomber.Hit{{ID: "5", Highlights: []beachcomber.Snippet{{
Field: "title",
ArrayIndex: -1,
Segments: []beachcomber.Segment{{Text: "a "}, {Text: "blue", Match: true}, {Text: " train"}},
}}}}
if !reflect.DeepEqual(res.Hits, want) {
t.Fatalf("hits = %+v, want %+v", res.Hits, want)
}
// A zero Highlight sends the two tags and nothing else new.
if _, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{Q: "blue", QueryBy: []string{"title"}, Highlight: &beachcomber.Highlight{}}); err != nil {
t.Fatal(err)
}
params = onlyQuery(t, f)
for k := range params {
switch k {
case "q", "query_by", "highlight_start_tag", "highlight_end_tag":
default:
t.Fatalf("zero Highlight sent %s (query %v)", k, params)
}
}
if !params.Has("highlight_start_tag") || !params.Has("highlight_end_tag") {
t.Fatalf("zero Highlight must send the tags: %v", params)
}
}
func TestTypesenseHighlightsKeepStoredText(t *testing.T) {
for name, snippet := range map[string]string{
"literal": `<b>bold</b> &amp; blue &lt;x`,
"escaped": `<b>bold</b> &amp; blue &lt;x`,
} {
t.Run(name, func(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":1,"hits":[{"document":{"id":"1"},"highlights":[{"field":"body","snippet":"`+snippet+`"}]}]}`)
res, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{Q: "blue", QueryBy: []string{"body"}, Highlight: &beachcomber.Highlight{}})
if err != nil {
t.Fatal(err)
}
if len(res.Hits) != 1 || len(res.Hits[0].Highlights) != 1 {
t.Fatalf("hits = %+v", res.Hits)
}
want := []beachcomber.Segment{{Text: "<b>bold</b> &amp; "}, {Text: "blue", Match: true}, {Text: " &lt;x"}}
got := res.Hits[0].Highlights[0].Segments
if !reflect.DeepEqual(got, want) {
t.Fatalf("segments = %+v, want %+v", got, want)
}
for _, s := range got {
if strings.Contains(s.Text, "mark") {
t.Fatalf("segment carries markup: %q", s.Text)
}
}
})
}
}
func TestTypesenseHitsStayInStepWithIDs(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":5,"hits":[`+
`{"document":{"id":"12"},"highlights":[{"field":"title","snippet":"x"},{"field":"notes","snippet":5}]},`+
`{"document":{"id":null},"highlights":[{"field":"title","snippet":"y"}]},`+
`{"document":{},"highlights":[{"field":"title","snippet":"z"}]},`+
`{"document":{"id":7}},`+
`{"document":{"id":"3"},"highlights":[]}]}`)
res, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{Q: "x", QueryBy: []string{"title"}, Highlight: &beachcomber.Highlight{}})
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(res.IDs, []string{"12", "7", "3"}) || len(res.Hits) != 3 {
t.Fatalf("res = %+v", res)
}
for i, h := range res.Hits {
if h.ID != res.IDs[i] {
t.Fatalf("hits[%d].ID = %q, ids[%d] = %q", i, h.ID, i, res.IDs[i])
}
}
if len(res.Hits[0].Highlights) != 1 || res.Hits[0].Highlights[0].Field != "title" {
t.Fatalf("hits[0] = %+v (the undecodable notes entry is dropped alone)", res.Hits[0])
}
if len(res.Hits[1].Highlights) != 0 || len(res.Hits[2].Highlights) != 0 {
t.Fatalf("hits = %+v", res.Hits)
}
f.answer(searchKey, 200, `{"found":0,"hits":[]}`)
res, err = e.SearchPage(t.Context(), "posts", beachcomber.Query{Q: "x", Highlight: &beachcomber.Highlight{}})
if err != nil || res.Hits == nil || len(res.Hits) != 0 {
t.Fatalf("empty answer = %+v %v", res, err)
}
}
func TestTypesenseHighlightsStringArray(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":2,"hits":[`+
`{"document":{"id":"1"},"highlights":[{"field":"tracks","snippets":["blue note","x blue"],"indices":[2,0],"matched_tokens":[["blue"],["blue"]]}]},`+
`{"document":{"id":"2"},"highlights":[`+
`{"field":"title","snippet":"blue train"},`+
`{"field":"tags","snippets":["blue"]}]}]}`)
q := beachcomber.Query{Q: "blue", QueryBy: []string{"title", "tracks", "tags"}, Highlight: &beachcomber.Highlight{}}
res, err := e.SearchPage(t.Context(), "posts", q)
if err != nil {
t.Fatal(err)
}
params := onlyQuery(t, f)
for k := range params {
if strings.Contains(k, "snippet") || strings.Contains(k, "max") {
t.Fatalf("a snippet-cap parameter was sent: %s", k)
}
}
want := []beachcomber.Snippet{
{Field: "tracks", ArrayIndex: 2, Segments: []beachcomber.Segment{{Text: "blue", Match: true}, {Text: " note"}}},
{Field: "tracks", ArrayIndex: 0, Segments: []beachcomber.Segment{{Text: "x "}, {Text: "blue", Match: true}}},
}
if !reflect.DeepEqual(res.Hits[0].Highlights, want) {
t.Fatalf("tracks = %+v, want %+v", res.Hits[0].Highlights, want)
}
second := res.Hits[1].Highlights
if len(second) != 2 || second[0].Field != "title" || second[0].ArrayIndex != -1 ||
second[1].Field != "tags" || second[1].ArrayIndex != -1 {
t.Fatalf("mixed hit = %+v (order of the highlights array, -1 without indices)", second)
}
q.Highlight.MaxSnippetsPerArray = 1
res, err = e.SearchPage(t.Context(), "posts", q)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(res.Hits[0].Highlights, want[:1]) {
t.Fatalf("capped = %+v, want %+v", res.Hits[0].Highlights, want[:1])
}
params = onlyQuery(t, f)
for k := range params {
if strings.Contains(k, "snippet") || strings.Contains(k, "max") {
t.Fatalf("the cap was sent as %s; it is client-side", k)
}
}
}
func TestTypesenseFieldProjection(t *testing.T) {
f, e := newFake(t)
f.answer(searchKey, 200, `{"found":1,"hits":[{"document":{"id":"9"},"highlights":[{"field":"body","snippet":"the blue one"}]}]}`)
for _, tc := range []struct {
include []string
want string
}{
{[]string{"title"}, "title,id"},
{[]string{"id", "title"}, "id,title"},
} {
if _, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{Q: "x", IncludeFields: tc.include}); err != nil {
t.Fatal(err)
}
if got := onlyQuery(t, f).Get("include_fields"); got != tc.want {
t.Fatalf("include_fields for %v = %q, want %q", tc.include, got, tc.want)
}
}
res, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{
Q: "blue", QueryBy: []string{"body"}, ExcludeFields: []string{"body"},
Highlight: &beachcomber.Highlight{Fields: []string{"body"}},
})
if err != nil {
t.Fatal(err)
}
params := onlyQuery(t, f)
if params.Get("exclude_fields") != "body" || params.Get("highlight_fields") != "body" || params.Has("include_fields") {
t.Fatalf("params = %v", params)
}
if len(res.Hits) != 1 || len(res.Hits[0].Highlights) != 1 || res.Hits[0].Highlights[0].Field != "body" {
t.Fatalf("an excluded field must still be highlighted: %+v", res.Hits)
}
for name, q := range map[string]beachcomber.Query{
"exclude_fields": {ExcludeFields: []string{"body", "id"}},
"AffixTokens": {Highlight: &beachcomber.Highlight{AffixTokens: -1}},
"MaxSnippetsPerArray": {Highlight: &beachcomber.Highlight{MaxSnippetsPerArray: -1}},
} {
_, err := e.SearchPage(t.Context(), "posts", q)
if err == nil || !strings.Contains(err.Error(), name) {
t.Fatalf("%s: err = %v", name, err)
}
}
if calls := f.take(); len(calls) != 0 {
t.Fatalf("a rejected query sent %d requests", len(calls))
}
}
func TestTypesenseSupportsHighlights(t *testing.T) {
_, e := newFake(t)
if !beachcomber.SupportsHighlights(e) {
t.Fatal("the Typesense engine supports highlights")
}
var _ beachcomber.HighlightSearcher = e
}