feat(12-01): report the search engine's found count and field weights

- optional beachcomber.PageSearcher returns a page of candidate ids plus
  the engine's found count; beachcomber.SearchPage falls back to
  SearchIDs for engines without it, so Engine is unchanged
- Query.QueryByWeights is sent to Typesense as query_by_weights; a
  mismatched weight list or a page above typesense.MaxPerPage (250) is
  refused before any request

modules/beachcomber/README.md

@@ -18,14 +18,15 @@ The package itself knows no search server. An engine package registers itself fr
 
 - Engine selection by `search.driver`: `null` (the default) or a registered engine such as `typesense`. An unknown name is a boot error that lists the registered engines. Third-party engines register with `beachcomber.RegisterEngine` and a `beachcomber.EngineFactory`; a duplicate name panics at init.
 - The `beachcomber.Searchable` model contract: `SearchableAs` (the index name, prefixed with `search.prefix`), `ToSearchableArray(ctx, db)` (the document, built from the committed row and free to query related rows) and `ShouldBeSearchable`. A model can also implement `beachcomber.IndexSchemaProvider` (the schema the engine creates a missing index with) and `beachcomber.SearchKeyer` (a document key other than the decimal primary key).
-- The `beachcomber.Engine` driver contract: `Name`, `Configured`, `Upsert`, `Delete`, `Flush` and `SearchIDs` with a `beachcomber.Query`.
+- The `beachcomber.Engine` driver contract: `Name`, `Configured`, `Upsert`, `Delete`, `Flush` and `SearchIDs` with a `beachcomber.Query`. `beachcomber.Query.QueryByWeights` ranks the `QueryBy` fields, one weight per field.
+- Search totals: the optional `beachcomber.PageSearcher` interface returns a `beachcomber.SearchResult`, one page of ids plus `Found`, the number of documents the engine matched. Call it through `beachcomber.SearchPage`, which falls back to `SearchIDs` (with `Found` set to the number of ids) for an engine that does not implement it; the `null` engine returns no ids and a zero count.
 - GORM callbacks `beachcomber.CallbackAfterCreate`, `beachcomber.CallbackAfterUpdate` and `beachcomber.CallbackAfterDelete`, installed once per `*gorm.DB`, register the sync with `lagoon.AfterCommit`.
 - `beachcomber.Service.Sync` and `beachcomber.Service.Remove` run the same gated path on demand, for reindex tooling, and return the error instead of logging it.
 - Typesense engine (`typesense.Engine`, engine name `typesense`):
   - Every request carries the `X-TYPESENSE-API-KEY` header.
   - `Upsert` reads the collection and creates it from the schema on 404. A 409 on create counts as success, and a model without a schema gets an auto-typed collection. It then imports the documents as JSON lines (`Content-Type: text/plain`) with `action=upsert`. Typesense answers 200 even when a document fails, so every answer line is checked and any `"success":false` line is an error.
   - `Delete` and `Flush` treat 404 as success.
-  - `SearchIDs` sends `q` (default `*`), `query_by`, `filter_by`, `sort_by`, `page` and `per_page`, and returns `hits[].document.id` in order.
+  - `SearchIDs` and `SearchPage` send one request with `q` (default `*`), `query_by`, `query_by_weights`, `filter_by`, `sort_by`, `page` and `per_page`, and return `hits[].document.id` in order; `SearchPage` adds the answer's `found`. A weight list whose length differs from `QueryBy`, or a `PerPage` above `typesense.MaxPerPage` (250, Typesense's limit), is an error before any request; callers page instead.
   - Ids and index names are path-escaped.
   - A non-2xx answer is a `typesense.StatusError` with the method, path and status, never the answer body.
 
@@ -41,7 +42,7 @@ The package itself knows no search server. An engine package registers itself fr
   When the engine is not configured, the callbacks do not even register work.
 - **Reload, then decide.** The row is reloaded by primary key, including soft-deleted rows. A delete, a row that is gone, a soft-deleted row (a set `gorm.DeletedAt`) or a row whose `ShouldBeSearchable` is false has its document deleted. Restoring a soft-deleted row is an ordinary update and indexes it again. An error from `ToSearchableArray` is logged and nothing is sent, which lets a model refuse a document that would break scoping. A document without an `id` gets the key.
 - **Rows only.** A statement without a primary key value, such as `Model(&T{}).Where(…).Updates(…)` or `Delete(&T{}, id)`, cannot be synced row by row and is skipped. Bulk paths call `beachcomber.Service.Sync` or `beachcomber.Service.Remove` per row, or reindex.
-- **Candidates, not answers.** `beachcomber.Engine.SearchIDs` returns candidate ids from an external index that may be stale. Callers must re-gate every id in SQL (ownership, visibility, soft deletes) before they expose a row. An empty result is an empty list, never an error.
+- **Candidates, not answers.** `beachcomber.Engine.SearchIDs` returns candidate ids from an external index that may be stale. Callers must re-gate every id in SQL (ownership, visibility, soft deletes) before they expose a row. An empty result is an empty list, never an error. The same holds for `beachcomber.SearchResult.Found`: it counts documents in the index, including stale ones, so a total shown to a user is recounted in SQL over re-gated ids, never taken from the engine.
 
 ## Usage
 
@@ -128,7 +129,10 @@ ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomb
 | `beachcomber.IndexSchemaProvider` | `SearchIndexSchema()`: the schema a missing index is created with. |
 | `beachcomber.SearchKeyer` | `SearchKey()`: replaces the decimal primary key as the document key. |
 | `beachcomber.Engine` | `Name`, `Configured`, `Upsert`, `Delete`, `Flush`, `SearchIDs`. |
-| `beachcomber.Query` | `Q`, `QueryBy`, `FilterBy`, `SortBy`, `Page`, `PerPage`. |
+| `beachcomber.Query` | `Q`, `QueryBy`, `QueryByWeights`, `FilterBy`, `SortBy`, `Page`, `PerPage`. |
+| `beachcomber.PageSearcher` | Optional engine interface: `SearchPage(ctx, index, q)` returns a page of ids and the engine's found count. |
+| `beachcomber.SearchResult` | `IDs` (candidates) and `Found` (documents the engine matched). |
+| `beachcomber.SearchPage(ctx, engine, index, q)` | Calls `PageSearcher` when the engine has it, else `SearchIDs` with `Found` set to the number of ids. |
 | `beachcomber.Gate`, `beachcomber.GateFunc` | The application kill-switch: `Enabled(ctx, db) bool`. |
 | `beachcomber.EngineFactory`, `beachcomber.RegisterEngine(name, factory)` | Registers an engine from an `init` function. |
 | `beachcomber.NullEngine`, `beachcomber.DefaultDriver` | The name of the built-in engine that indexes nothing, and the default of `search.driver`. |
@@ -139,7 +143,8 @@ ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomb
 | Identifier | Description |
 |------------|-------------|
 | `typesense.Config`, `typesense.LoadConfig` | The `search.typesense.*` settings with their defaults; `BaseURL` is `{protocol}://{host}:{port}{path}`. |
-| `typesense.Engine`, `typesense.New` | The `beachcomber.Engine`, with `Config`. |
+| `typesense.Engine`, `typesense.New` | The `beachcomber.Engine` and `beachcomber.PageSearcher`, with `Config`. |
+| `typesense.MaxPerPage` | 250, the largest page a search may ask for. |
 | `typesense.StatusError` | A non-2xx answer: `Method`, `Path`, `Code` and `StatusCode()`. |
 | `typesense.DriverName` | `typesense`. |
 | `typesense.DefaultHost`, `typesense.DefaultPort`, `typesense.DefaultProtocol`, `typesense.DefaultConnectionTimeout`, `typesense.DefaultImportAction` | Defaults of the configuration keys. |

modules/beachcomber/engines.go

@@ -62,7 +62,7 @@ func init() {
 }
 
 // nullEngine is never configured, so sync is skipped before it is called.
-// Every method is a no-op; SearchIDs returns an empty list.
+// Every method is a no-op; SearchIDs and SearchPage return no ids.
 type nullEngine struct{}
 
 func (nullEngine) Name() string     { return NullEngine }
@@ -75,3 +75,6 @@ func (nullEngine) Flush(context.Context, string) error            { return nil }
 func (nullEngine) SearchIDs(context.Context, string, Query) ([]string, error) {
 	return []string{}, nil
 }
+func (nullEngine) SearchPage(context.Context, string, Query) (SearchResult, error) {
+	return SearchResult{IDs: []string{}}, nil
+}

modules/beachcomber/searchable.go

@@ -62,6 +62,10 @@ type Query struct {
 	Q string
 	// QueryBy lists the fields Q is matched against.
 	QueryBy []string
+	// QueryByWeights ranks the QueryBy fields, one weight per field in the
+	// same order (Typesense query_by_weights). Empty leaves the engine
+	// default; a length different from QueryBy is an error.
+	QueryByWeights []int
 	// FilterBy is an engine filter expression, for example
 	// "collection_id:=5".
 	FilterBy string
@@ -73,6 +77,35 @@ type Query struct {
 	PerPage int
 }
 
+// SearchResult is one page of candidate ids and the number of documents
+// the engine found for the query. Both are candidates only: callers re-gate
+// the ids, and recount any total they expose, in SQL.
+type SearchResult struct {
+	IDs   []string
+	Found int
+}
+
+// PageSearcher is implemented by an engine that can report how many
+// documents matched besides the page of ids. It is optional so that
+// existing Engine implementations stay valid; use SearchPage to call it.
+type PageSearcher interface {
+	SearchPage(ctx context.Context, index string, q Query) (SearchResult, error)
+}
+
+// SearchPage asks e for one page of q: through PageSearcher when e
+// implements it, otherwise through SearchIDs with Found set to the number
+// of ids returned.
+func SearchPage(ctx context.Context, e Engine, index string, q Query) (SearchResult, error) {
+	if ps, ok := e.(PageSearcher); ok {
+		return ps.SearchPage(ctx, index, q)
+	}
+	ids, err := e.SearchIDs(ctx, index, q)
+	if err != nil {
+		return SearchResult{}, err
+	}
+	return SearchResult{IDs: ids, Found: len(ids)}, nil
+}
+
 // Gate is the application kill-switch consulted before every sync. It
 // reads its setting through db, a clean session on the write's connection
 // (inside a caller's plain transaction, a savepoint of it); any error must

modules/beachcomber/searchpage_test.go

@@ -0,0 +1,80 @@
+package beachcomber
+
+import (
+	"context"
+	"errors"
+	"reflect"
+	"testing"
+)
+
+// idsOnlyEngine implements Engine without PageSearcher.
+type idsOnlyEngine struct {
+	ids []string
+	err error
+	got Query
+}
+
+func (*idsOnlyEngine) Name() string     { return "ids-only" }
+func (*idsOnlyEngine) Configured() bool { return true }
+func (*idsOnlyEngine) Upsert(context.Context, string, map[string]any, []map[string]any) error {
+	return nil
+}
+func (*idsOnlyEngine) Delete(context.Context, string, []string) error { return nil }
+func (*idsOnlyEngine) Flush(context.Context, string) error            { return nil }
+
+func (e *idsOnlyEngine) SearchIDs(_ context.Context, _ string, q Query) ([]string, error) {
+	e.got = q
+	return e.ids, e.err
+}
+
+// pagedEngine implements PageSearcher.
+type pagedEngine struct {
+	idsOnlyEngine
+	found int
+}
+
+func (e *pagedEngine) SearchPage(_ context.Context, _ string, q Query) (SearchResult, error) {
+	e.got = q
+	return SearchResult{IDs: e.ids, Found: e.found}, nil
+}
+
+func TestSearchPageFallsBackToSearchIDs(t *testing.T) {
+	e := &idsOnlyEngine{ids: []string{"3", "1"}}
+	q := Query{Q: "blue", QueryBy: []string{"name", "notes"}, QueryByWeights: []int{10, 1}, PerPage: 20}
+	res, err := SearchPage(context.Background(), e, "posts", q)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !reflect.DeepEqual(res, SearchResult{IDs: []string{"3", "1"}, Found: 2}) {
+		t.Fatalf("res = %+v", res)
+	}
+	if !reflect.DeepEqual(e.got, q) {
+		t.Fatalf("query = %+v", e.got)
+	}
+	e.err = errors.New("down")
+	if _, err := SearchPage(context.Background(), e, "posts", q); err == nil {
+		t.Fatal("engine error must surface")
+	}
+}
+
+func TestSearchPageUsesPageSearcher(t *testing.T) {
+	e := &pagedEngine{idsOnlyEngine: idsOnlyEngine{ids: []string{"7"}}, found: 1234}
+	res, err := SearchPage(context.Background(), e, "posts", Query{Q: "x"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if res.Found != 1234 || !reflect.DeepEqual(res.IDs, []string{"7"}) {
+		t.Fatalf("res = %+v", res)
+	}
+}
+
+func TestSearchPageNullEngine(t *testing.T) {
+	var e Engine = nullEngine{}
+	if _, ok := e.(PageSearcher); !ok {
+		t.Fatal("the null engine implements PageSearcher")
+	}
+	res, err := SearchPage(context.Background(), e, "posts", Query{})
+	if err != nil || res.Found != 0 || res.IDs == nil || len(res.IDs) != 0 {
+		t.Fatalf("res = %+v err = %v", res, err)
+	}
+}

modules/beachcomber/typesense/engine.go

@@ -214,21 +214,46 @@ func (e *Engine) Flush(ctx context.Context, index string) error {
 	return nil
 }
 
-// searchAnswer is the part of a search answer SearchIDs reads.
+// searchAnswer is the part of a search answer SearchIDs and SearchPage read.
 type searchAnswer struct {
-	Hits []struct {
+	Found int `json:"found"`
+	Hits  []struct {
 		Document struct {
 			ID json.RawMessage `json:"id"`
 		} `json:"document"`
 	} `json:"hits"`
 }
 
+// MaxPerPage is the largest page Typesense answers (Scout's maxPerPage);
+// a larger Query.PerPage is an error, so callers page instead.
+const MaxPerPage = 250
+
 // SearchIDs sends GET /collections/{index}/documents/search with q
-// (default "*"), query_by, filter_by, sort_by, page and per_page, and
-// returns hits[].document.id in order. The ids are candidates: the caller
-// must re-check each one in SQL before exposing it. No hits is an empty
-// list.
+// (default "*"), query_by, query_by_weights, filter_by, sort_by, page and
+// per_page, and returns hits[].document.id in order. The ids are
+// candidates: the caller must re-check each one in SQL before exposing it.
+// No hits is an empty list.
 func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Query) ([]string, error) {
+	res, err := e.search(ctx, index, q)
+	if err != nil {
+		return nil, err
+	}
+	return res.IDs, nil
+}
+
+// SearchPage is SearchIDs plus the found count of the answer: how many
+// documents matched in the index, not how many the caller may see.
+func (e *Engine) SearchPage(ctx context.Context, index string, q beachcomber.Query) (beachcomber.SearchResult, error) {
+	return e.search(ctx, index, q)
+}
+
+func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query) (beachcomber.SearchResult, error) {
+	if len(q.QueryByWeights) > 0 && len(q.QueryByWeights) != len(q.QueryBy) {
+		return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: %d query_by_weights for %d query_by fields", index, len(q.QueryByWeights), len(q.QueryBy))
+	}
+	if q.PerPage > MaxPerPage {
+		return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: per_page %d exceeds %d", index, q.PerPage, MaxPerPage)
+	}
 	params := url.Values{}
 	text := q.Q
 	if text == "" {
@@ -238,6 +263,13 @@ func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Quer
 	if len(q.QueryBy) > 0 {
 		params.Set("query_by", strings.Join(q.QueryBy, ","))
 	}
+	if len(q.QueryByWeights) > 0 {
+		weights := make([]string, len(q.QueryByWeights))
+		for i, w := range q.QueryByWeights {
+			weights[i] = strconv.Itoa(w)
+		}
+		params.Set("query_by_weights", strings.Join(weights, ","))
+	}
 	if q.FilterBy != "" {
 		params.Set("filter_by", q.FilterBy)
 	}
@@ -253,14 +285,14 @@ func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Quer
 	path := "/collections/" + url.PathEscape(index) + "/documents/search"
 	code, answer, err := e.do(ctx, http.MethodGet, path, params, nil, "")
 	if err != nil {
-		return nil, err
+		return beachcomber.SearchResult{}, err
 	}
 	if !ok2xx(code) {
-		return nil, statusError(http.MethodGet, path, code)
+		return beachcomber.SearchResult{}, statusError(http.MethodGet, path, code)
 	}
 	var res searchAnswer
 	if err := json.Unmarshal(answer, &res); err != nil {
-		return nil, fmt.Errorf("typesense: search %s: unreadable answer", index)
+		return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: unreadable answer", index)
 	}
 	ids := make([]string, 0, len(res.Hits))
 	for _, h := range res.Hits {
@@ -275,7 +307,7 @@ func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Quer
 		}
 		ids = append(ids, string(raw))
 	}
-	return ids, nil
+	return beachcomber.SearchResult{IDs: ids, Found: res.Found}, nil
 }
 
 // do sends one request with the API key header and returns the status and

modules/beachcomber/typesense/searchpage_test.go

@@ -0,0 +1,79 @@
+package typesense
+
+import (
+	"net/url"
+	"reflect"
+	"strings"
+	"testing"
+
+	"git.golem15.com/golem15/summercms/modules/beachcomber"
+)
+
+func TestTypesenseSearchPageFoundAndWeights(t *testing.T) {
+	f, e := newFake(t)
+	f.answer("GET /ts/collections/posts/documents/search", 200,
+		`{"found":1234,"out_of":5000,"page":2,"hits":[{"document":{"id":"42"}},{"document":{"id":7}}]}`)
+	q := beachcomber.Query{
+		Q:              "blue train",
+		QueryBy:        []string{"name", "artist_display", "notes"},
+		QueryByWeights: []int{10, 10, 1},
+		FilterBy:       "collection_id:=5",
+		Page:           2,
+		PerPage:        250,
+	}
+	res, err := beachcomber.SearchPage(t.Context(), e, "posts", q)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if res.Found != 1234 || !reflect.DeepEqual(res.IDs, []string{"42", "7"}) {
+		t.Fatalf("res = %+v", res)
+	}
+	calls := f.take()
+	if len(calls) != 1 {
+		t.Fatalf("calls = %d", len(calls))
+	}
+	params, err := url.ParseQuery(calls[0].rawQuery)
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := map[string]string{
+		"q": "blue train", "query_by": "name,artist_display,notes", "query_by_weights": "10,10,1",
+		"filter_by": "collection_id:=5", "page": "2", "per_page": "250",
+	}
+	for k, v := range want {
+		if params.Get(k) != v {
+			t.Fatalf("%s = %q, want %q (query %s)", k, params.Get(k), v, calls[0].rawQuery)
+		}
+	}
+	// SearchIDs shares the request and returns the same page.
+	ids, err := e.SearchIDs(t.Context(), "posts", q)
+	if err != nil || !reflect.DeepEqual(ids, []string{"42", "7"}) {
+		t.Fatalf("ids = %v err = %v", ids, err)
+	}
+	if got := f.take(); len(got) != 1 || got[0].rawQuery != calls[0].rawQuery {
+		t.Fatalf("SearchIDs query = %+v", got)
+	}
+}
+
+func TestTypesenseSearchPageRejectsBadQueries(t *testing.T) {
+	f, e := newFake(t)
+	f.answer("GET /ts/collections/posts/documents/search", 200, `{"found":0,"hits":[]}`)
+	_, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{QueryBy: []string{"name", "notes"}, QueryByWeights: []int{10}})
+	if err == nil || !strings.Contains(err.Error(), "query_by_weights") {
+		t.Fatalf("weights mismatch: %v", err)
+	}
+	_, err = e.SearchPage(t.Context(), "posts", beachcomber.Query{PerPage: MaxPerPage + 1})
+	if err == nil || !strings.Contains(err.Error(), "per_page") {
+		t.Fatalf("per_page: %v", err)
+	}
+	if calls := f.take(); len(calls) != 0 {
+		t.Fatalf("a rejected query sent %d requests", len(calls))
+	}
+	res, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{QueryBy: []string{"name"}})
+	if err != nil || res.Found != 0 || len(res.IDs) != 0 {
+		t.Fatalf("empty answer = %+v %v", res, err)
+	}
+	if got := f.take(); strings.Contains(got[0].rawQuery, "query_by_weights") {
+		t.Fatalf("weights sent without QueryByWeights: %s", got[0].rawQuery)
+	}
+}