feat(12-01): report the search engine's found count and field weights

- optional beachcomber.PageSearcher returns a page of candidate ids plus
  the engine's found count; beachcomber.SearchPage falls back to
  SearchIDs for engines without it, so Engine is unchanged
- Query.QueryByWeights is sent to Typesense as query_by_weights; a
  mismatched weight list or a page above typesense.MaxPerPage (250) is
  refused before any request

docs/services/search.md

@@ -131,6 +131,10 @@ An empty result is an empty list, never an error.
 > [!WARNING]
 > Never return rows, or even counts, straight from search IDs. A stale index would otherwise show a draft, a deleted record or another user's data.
 
+A paginated endpoint also needs to know how many documents matched. An engine that can tell implements the optional `beachcomber.PageSearcher` interface; call it through `beachcomber.SearchPage`, which returns a `beachcomber.SearchResult` with the page of `IDs` and `Found`, the number of documents the engine matched. For an engine without `PageSearcher` the helper calls `SearchIDs` and sets `Found` to the number of IDs returned, so code written against it works with every engine. `Found` is a candidate count like the IDs: it includes stale and foreign documents. Use it to decide how many IDs to fetch, then recount the total in SQL with the same conditions as the rows, as Laravel Scout does when a query callback is set.
+
+`beachcomber.Query.QueryByWeights` ranks the fields of `QueryBy`, one weight per field in the same order, as Scout's `query_by_weights` option does.
+
 ## Typesense
 
 The Typesense engine follows the Scout Typesense wire contract, so indexes built by a WinterCMS application can be searched by the port:
@@ -180,4 +184,4 @@ fmt.Println(typesense.New(typesense.Config{}).Configured())
 // false
 ```
 
-Each request times out after `search.typesense.connection_timeout_seconds` (2 seconds by default). A failed answer is a `typesense.StatusError` with the method, path and status, never the answer body.
+The Typesense engine implements `beachcomber.PageSearcher` on the same request, adding the answer's `found` count, and sends `query_by_weights` when the query has weights. A page is at most `typesense.MaxPerPage` (250) documents, Typesense's own limit: a larger `PerPage` is an error, so fetch more IDs page by page. Each request times out after `search.typesense.connection_timeout_seconds` (2 seconds by default). A failed answer is a `typesense.StatusError` with the method, path and status, never the answer body.

modules/beachcomber/README.md

@@ -18,14 +18,15 @@ The package itself knows no search server. An engine package registers itself fr
 
 - Engine selection by `search.driver`: `null` (the default) or a registered engine such as `typesense`. An unknown name is a boot error that lists the registered engines. Third-party engines register with `beachcomber.RegisterEngine` and a `beachcomber.EngineFactory`; a duplicate name panics at init.
 - The `beachcomber.Searchable` model contract: `SearchableAs` (the index name, prefixed with `search.prefix`), `ToSearchableArray(ctx, db)` (the document, built from the committed row and free to query related rows) and `ShouldBeSearchable`. A model can also implement `beachcomber.IndexSchemaProvider` (the schema the engine creates a missing index with) and `beachcomber.SearchKeyer` (a document key other than the decimal primary key).
-- The `beachcomber.Engine` driver contract: `Name`, `Configured`, `Upsert`, `Delete`, `Flush` and `SearchIDs` with a `beachcomber.Query`.
+- The `beachcomber.Engine` driver contract: `Name`, `Configured`, `Upsert`, `Delete`, `Flush` and `SearchIDs` with a `beachcomber.Query`. `beachcomber.Query.QueryByWeights` ranks the `QueryBy` fields, one weight per field.
+- Search totals: the optional `beachcomber.PageSearcher` interface returns a `beachcomber.SearchResult`, one page of ids plus `Found`, the number of documents the engine matched. Call it through `beachcomber.SearchPage`, which falls back to `SearchIDs` (with `Found` set to the number of ids) for an engine that does not implement it; the `null` engine returns no ids and a zero count.
 - GORM callbacks `beachcomber.CallbackAfterCreate`, `beachcomber.CallbackAfterUpdate` and `beachcomber.CallbackAfterDelete`, installed once per `*gorm.DB`, register the sync with `lagoon.AfterCommit`.
 - `beachcomber.Service.Sync` and `beachcomber.Service.Remove` run the same gated path on demand, for reindex tooling, and return the error instead of logging it.
 - Typesense engine (`typesense.Engine`, engine name `typesense`):
   - Every request carries the `X-TYPESENSE-API-KEY` header.
   - `Upsert` reads the collection and creates it from the schema on 404. A 409 on create counts as success, and a model without a schema gets an auto-typed collection. It then imports the documents as JSON lines (`Content-Type: text/plain`) with `action=upsert`. Typesense answers 200 even when a document fails, so every answer line is checked and any `"success":false` line is an error.
   - `Delete` and `Flush` treat 404 as success.
-  - `SearchIDs` sends `q` (default `*`), `query_by`, `filter_by`, `sort_by`, `page` and `per_page`, and returns `hits[].document.id` in order.
+  - `SearchIDs` and `SearchPage` send one request with `q` (default `*`), `query_by`, `query_by_weights`, `filter_by`, `sort_by`, `page` and `per_page`, and return `hits[].document.id` in order; `SearchPage` adds the answer's `found`. A weight list whose length differs from `QueryBy`, or a `PerPage` above `typesense.MaxPerPage` (250, Typesense's limit), is an error before any request; callers page instead.
   - Ids and index names are path-escaped.
   - A non-2xx answer is a `typesense.StatusError` with the method, path and status, never the answer body.
 
@@ -41,7 +42,7 @@ The package itself knows no search server. An engine package registers itself fr
   When the engine is not configured, the callbacks do not even register work.
 - **Reload, then decide.** The row is reloaded by primary key, including soft-deleted rows. A delete, a row that is gone, a soft-deleted row (a set `gorm.DeletedAt`) or a row whose `ShouldBeSearchable` is false has its document deleted. Restoring a soft-deleted row is an ordinary update and indexes it again. An error from `ToSearchableArray` is logged and nothing is sent, which lets a model refuse a document that would break scoping. A document without an `id` gets the key.
 - **Rows only.** A statement without a primary key value, such as `Model(&T{}).Where(…).Updates(…)` or `Delete(&T{}, id)`, cannot be synced row by row and is skipped. Bulk paths call `beachcomber.Service.Sync` or `beachcomber.Service.Remove` per row, or reindex.
-- **Candidates, not answers.** `beachcomber.Engine.SearchIDs` returns candidate ids from an external index that may be stale. Callers must re-gate every id in SQL (ownership, visibility, soft deletes) before they expose a row. An empty result is an empty list, never an error.
+- **Candidates, not answers.** `beachcomber.Engine.SearchIDs` returns candidate ids from an external index that may be stale. Callers must re-gate every id in SQL (ownership, visibility, soft deletes) before they expose a row. An empty result is an empty list, never an error. The same holds for `beachcomber.SearchResult.Found`: it counts documents in the index, including stale ones, so a total shown to a user is recounted in SQL over re-gated ids, never taken from the engine.
 
 ## Usage
 
@@ -128,7 +129,10 @@ ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomb
 | `beachcomber.IndexSchemaProvider` | `SearchIndexSchema()`: the schema a missing index is created with. |
 | `beachcomber.SearchKeyer` | `SearchKey()`: replaces the decimal primary key as the document key. |
 | `beachcomber.Engine` | `Name`, `Configured`, `Upsert`, `Delete`, `Flush`, `SearchIDs`. |
-| `beachcomber.Query` | `Q`, `QueryBy`, `FilterBy`, `SortBy`, `Page`, `PerPage`. |
+| `beachcomber.Query` | `Q`, `QueryBy`, `QueryByWeights`, `FilterBy`, `SortBy`, `Page`, `PerPage`. |
+| `beachcomber.PageSearcher` | Optional engine interface: `SearchPage(ctx, index, q)` returns a page of ids and the engine's found count. |
+| `beachcomber.SearchResult` | `IDs` (candidates) and `Found` (documents the engine matched). |
+| `beachcomber.SearchPage(ctx, engine, index, q)` | Calls `PageSearcher` when the engine has it, else `SearchIDs` with `Found` set to the number of ids. |
 | `beachcomber.Gate`, `beachcomber.GateFunc` | The application kill-switch: `Enabled(ctx, db) bool`. |
 | `beachcomber.EngineFactory`, `beachcomber.RegisterEngine(name, factory)` | Registers an engine from an `init` function. |
 | `beachcomber.NullEngine`, `beachcomber.DefaultDriver` | The name of the built-in engine that indexes nothing, and the default of `search.driver`. |
@@ -139,7 +143,8 @@ ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&models.Post{}), beachcomb
 | Identifier | Description |
 |------------|-------------|
 | `typesense.Config`, `typesense.LoadConfig` | The `search.typesense.*` settings with their defaults; `BaseURL` is `{protocol}://{host}:{port}{path}`. |
-| `typesense.Engine`, `typesense.New` | The `beachcomber.Engine`, with `Config`. |
+| `typesense.Engine`, `typesense.New` | The `beachcomber.Engine` and `beachcomber.PageSearcher`, with `Config`. |
+| `typesense.MaxPerPage` | 250, the largest page a search may ask for. |
 | `typesense.StatusError` | A non-2xx answer: `Method`, `Path`, `Code` and `StatusCode()`. |
 | `typesense.DriverName` | `typesense`. |
 | `typesense.DefaultHost`, `typesense.DefaultPort`, `typesense.DefaultProtocol`, `typesense.DefaultConnectionTimeout`, `typesense.DefaultImportAction` | Defaults of the configuration keys. |

modules/beachcomber/engines.go

@@ -62,7 +62,7 @@ func init() {
 }
 
 // nullEngine is never configured, so sync is skipped before it is called.
-// Every method is a no-op; SearchIDs returns an empty list.
+// Every method is a no-op; SearchIDs and SearchPage return no ids.
 type nullEngine struct{}
 
 func (nullEngine) Name() string     { return NullEngine }
@@ -75,3 +75,6 @@ func (nullEngine) Flush(context.Context, string) error            { return nil }
 func (nullEngine) SearchIDs(context.Context, string, Query) ([]string, error) {
 	return []string{}, nil
 }
+func (nullEngine) SearchPage(context.Context, string, Query) (SearchResult, error) {
+	return SearchResult{IDs: []string{}}, nil
+}

modules/beachcomber/searchable.go

@@ -62,6 +62,10 @@ type Query struct {
 	Q string
 	// QueryBy lists the fields Q is matched against.
 	QueryBy []string
+	// QueryByWeights ranks the QueryBy fields, one weight per field in the
+	// same order (Typesense query_by_weights). Empty leaves the engine
+	// default; a length different from QueryBy is an error.
+	QueryByWeights []int
 	// FilterBy is an engine filter expression, for example
 	// "collection_id:=5".
 	FilterBy string
@@ -73,6 +77,35 @@ type Query struct {
 	PerPage int
 }
 
+// SearchResult is one page of candidate ids and the number of documents
+// the engine found for the query. Both are candidates only: callers re-gate
+// the ids, and recount any total they expose, in SQL.
+type SearchResult struct {
+	IDs   []string
+	Found int
+}
+
+// PageSearcher is implemented by an engine that can report how many
+// documents matched besides the page of ids. It is optional so that
+// existing Engine implementations stay valid; use SearchPage to call it.
+type PageSearcher interface {
+	SearchPage(ctx context.Context, index string, q Query) (SearchResult, error)
+}
+
+// SearchPage asks e for one page of q: through PageSearcher when e
+// implements it, otherwise through SearchIDs with Found set to the number
+// of ids returned.
+func SearchPage(ctx context.Context, e Engine, index string, q Query) (SearchResult, error) {
+	if ps, ok := e.(PageSearcher); ok {
+		return ps.SearchPage(ctx, index, q)
+	}
+	ids, err := e.SearchIDs(ctx, index, q)
+	if err != nil {
+		return SearchResult{}, err
+	}
+	return SearchResult{IDs: ids, Found: len(ids)}, nil
+}
+
 // Gate is the application kill-switch consulted before every sync. It
 // reads its setting through db, a clean session on the write's connection
 // (inside a caller's plain transaction, a savepoint of it); any error must

modules/beachcomber/searchpage_test.go

@@ -0,0 +1,80 @@
+package beachcomber
+
+import (
+	"context"
+	"errors"
+	"reflect"
+	"testing"
+)
+
+// idsOnlyEngine implements Engine without PageSearcher.
+type idsOnlyEngine struct {
+	ids []string
+	err error
+	got Query
+}
+
+func (*idsOnlyEngine) Name() string     { return "ids-only" }
+func (*idsOnlyEngine) Configured() bool { return true }
+func (*idsOnlyEngine) Upsert(context.Context, string, map[string]any, []map[string]any) error {
+	return nil
+}
+func (*idsOnlyEngine) Delete(context.Context, string, []string) error { return nil }
+func (*idsOnlyEngine) Flush(context.Context, string) error            { return nil }
+
+func (e *idsOnlyEngine) SearchIDs(_ context.Context, _ string, q Query) ([]string, error) {
+	e.got = q
+	return e.ids, e.err
+}
+
+// pagedEngine implements PageSearcher.
+type pagedEngine struct {
+	idsOnlyEngine
+	found int
+}
+
+func (e *pagedEngine) SearchPage(_ context.Context, _ string, q Query) (SearchResult, error) {
+	e.got = q
+	return SearchResult{IDs: e.ids, Found: e.found}, nil
+}
+
+func TestSearchPageFallsBackToSearchIDs(t *testing.T) {
+	e := &idsOnlyEngine{ids: []string{"3", "1"}}
+	q := Query{Q: "blue", QueryBy: []string{"name", "notes"}, QueryByWeights: []int{10, 1}, PerPage: 20}
+	res, err := SearchPage(context.Background(), e, "posts", q)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !reflect.DeepEqual(res, SearchResult{IDs: []string{"3", "1"}, Found: 2}) {
+		t.Fatalf("res = %+v", res)
+	}
+	if !reflect.DeepEqual(e.got, q) {
+		t.Fatalf("query = %+v", e.got)
+	}
+	e.err = errors.New("down")
+	if _, err := SearchPage(context.Background(), e, "posts", q); err == nil {
+		t.Fatal("engine error must surface")
+	}
+}
+
+func TestSearchPageUsesPageSearcher(t *testing.T) {
+	e := &pagedEngine{idsOnlyEngine: idsOnlyEngine{ids: []string{"7"}}, found: 1234}
+	res, err := SearchPage(context.Background(), e, "posts", Query{Q: "x"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if res.Found != 1234 || !reflect.DeepEqual(res.IDs, []string{"7"}) {
+		t.Fatalf("res = %+v", res)
+	}
+}
+
+func TestSearchPageNullEngine(t *testing.T) {
+	var e Engine = nullEngine{}
+	if _, ok := e.(PageSearcher); !ok {
+		t.Fatal("the null engine implements PageSearcher")
+	}
+	res, err := SearchPage(context.Background(), e, "posts", Query{})
+	if err != nil || res.Found != 0 || res.IDs == nil || len(res.IDs) != 0 {
+		t.Fatalf("res = %+v err = %v", res, err)
+	}
+}

modules/beachcomber/typesense/engine.go

@@ -214,21 +214,46 @@ func (e *Engine) Flush(ctx context.Context, index string) error {
 	return nil
 }
 
-// searchAnswer is the part of a search answer SearchIDs reads.
+// searchAnswer is the part of a search answer SearchIDs and SearchPage read.
 type searchAnswer struct {
-	Hits []struct {
+	Found int `json:"found"`
+	Hits  []struct {
 		Document struct {
 			ID json.RawMessage `json:"id"`
 		} `json:"document"`
 	} `json:"hits"`
 }
 
+// MaxPerPage is the largest page Typesense answers (Scout's maxPerPage);
+// a larger Query.PerPage is an error, so callers page instead.
+const MaxPerPage = 250
+
 // SearchIDs sends GET /collections/{index}/documents/search with q
-// (default "*"), query_by, filter_by, sort_by, page and per_page, and
+// (default "*"), query_by, query_by_weights, filter_by, sort_by, page and
-// returns hits[].document.id in order. The ids are candidates: the caller
+// per_page, and returns hits[].document.id in order. The ids are
-// must re-check each one in SQL before exposing it. No hits is an empty
+// candidates: the caller must re-check each one in SQL before exposing it.
-// list.
+// No hits is an empty list.
 func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Query) ([]string, error) {
+	res, err := e.search(ctx, index, q)
+	if err != nil {
+		return nil, err
+	}
+	return res.IDs, nil
+}
+
+// SearchPage is SearchIDs plus the found count of the answer: how many
+// documents matched in the index, not how many the caller may see.
+func (e *Engine) SearchPage(ctx context.Context, index string, q beachcomber.Query) (beachcomber.SearchResult, error) {
+	return e.search(ctx, index, q)
+}
+
+func (e *Engine) search(ctx context.Context, index string, q beachcomber.Query) (beachcomber.SearchResult, error) {
+	if len(q.QueryByWeights) > 0 && len(q.QueryByWeights) != len(q.QueryBy) {
+		return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: %d query_by_weights for %d query_by fields", index, len(q.QueryByWeights), len(q.QueryBy))
+	}
+	if q.PerPage > MaxPerPage {
+		return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: per_page %d exceeds %d", index, q.PerPage, MaxPerPage)
+	}
 	params := url.Values{}
 	text := q.Q
 	if text == "" {
@@ -238,6 +263,13 @@ func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Quer
 	if len(q.QueryBy) > 0 {
 		params.Set("query_by", strings.Join(q.QueryBy, ","))
 	}
+	if len(q.QueryByWeights) > 0 {
+		weights := make([]string, len(q.QueryByWeights))
+		for i, w := range q.QueryByWeights {
+			weights[i] = strconv.Itoa(w)
+		}
+		params.Set("query_by_weights", strings.Join(weights, ","))
+	}
 	if q.FilterBy != "" {
 		params.Set("filter_by", q.FilterBy)
 	}
@@ -253,14 +285,14 @@ func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Quer
 	path := "/collections/" + url.PathEscape(index) + "/documents/search"
 	code, answer, err := e.do(ctx, http.MethodGet, path, params, nil, "")
 	if err != nil {
-		return nil, err
+		return beachcomber.SearchResult{}, err
 	}
 	if !ok2xx(code) {
-		return nil, statusError(http.MethodGet, path, code)
+		return beachcomber.SearchResult{}, statusError(http.MethodGet, path, code)
 	}
 	var res searchAnswer
 	if err := json.Unmarshal(answer, &res); err != nil {
-		return nil, fmt.Errorf("typesense: search %s: unreadable answer", index)
+		return beachcomber.SearchResult{}, fmt.Errorf("typesense: search %s: unreadable answer", index)
 	}
 	ids := make([]string, 0, len(res.Hits))
 	for _, h := range res.Hits {
@@ -275,7 +307,7 @@ func (e *Engine) SearchIDs(ctx context.Context, index string, q beachcomber.Quer
 		}
 		ids = append(ids, string(raw))
 	}
-	return ids, nil
+	return beachcomber.SearchResult{IDs: ids, Found: res.Found}, nil
 }
 
 // do sends one request with the API key header and returns the status and

modules/beachcomber/typesense/searchpage_test.go

@@ -0,0 +1,79 @@
+package typesense
+
+import (
+	"net/url"
+	"reflect"
+	"strings"
+	"testing"
+
+	"git.golem15.com/golem15/summercms/modules/beachcomber"
+)
+
+func TestTypesenseSearchPageFoundAndWeights(t *testing.T) {
+	f, e := newFake(t)
+	f.answer("GET /ts/collections/posts/documents/search", 200,
+		`{"found":1234,"out_of":5000,"page":2,"hits":[{"document":{"id":"42"}},{"document":{"id":7}}]}`)
+	q := beachcomber.Query{
+		Q:              "blue train",
+		QueryBy:        []string{"name", "artist_display", "notes"},
+		QueryByWeights: []int{10, 10, 1},
+		FilterBy:       "collection_id:=5",
+		Page:           2,
+		PerPage:        250,
+	}
+	res, err := beachcomber.SearchPage(t.Context(), e, "posts", q)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if res.Found != 1234 || !reflect.DeepEqual(res.IDs, []string{"42", "7"}) {
+		t.Fatalf("res = %+v", res)
+	}
+	calls := f.take()
+	if len(calls) != 1 {
+		t.Fatalf("calls = %d", len(calls))
+	}
+	params, err := url.ParseQuery(calls[0].rawQuery)
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := map[string]string{
+		"q": "blue train", "query_by": "name,artist_display,notes", "query_by_weights": "10,10,1",
+		"filter_by": "collection_id:=5", "page": "2", "per_page": "250",
+	}
+	for k, v := range want {
+		if params.Get(k) != v {
+			t.Fatalf("%s = %q, want %q (query %s)", k, params.Get(k), v, calls[0].rawQuery)
+		}
+	}
+	// SearchIDs shares the request and returns the same page.
+	ids, err := e.SearchIDs(t.Context(), "posts", q)
+	if err != nil || !reflect.DeepEqual(ids, []string{"42", "7"}) {
+		t.Fatalf("ids = %v err = %v", ids, err)
+	}
+	if got := f.take(); len(got) != 1 || got[0].rawQuery != calls[0].rawQuery {
+		t.Fatalf("SearchIDs query = %+v", got)
+	}
+}
+
+func TestTypesenseSearchPageRejectsBadQueries(t *testing.T) {
+	f, e := newFake(t)
+	f.answer("GET /ts/collections/posts/documents/search", 200, `{"found":0,"hits":[]}`)
+	_, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{QueryBy: []string{"name", "notes"}, QueryByWeights: []int{10}})
+	if err == nil || !strings.Contains(err.Error(), "query_by_weights") {
+		t.Fatalf("weights mismatch: %v", err)
+	}
+	_, err = e.SearchPage(t.Context(), "posts", beachcomber.Query{PerPage: MaxPerPage + 1})
+	if err == nil || !strings.Contains(err.Error(), "per_page") {
+		t.Fatalf("per_page: %v", err)
+	}
+	if calls := f.take(); len(calls) != 0 {
+		t.Fatalf("a rejected query sent %d requests", len(calls))
+	}
+	res, err := e.SearchPage(t.Context(), "posts", beachcomber.Query{QueryBy: []string{"name"}})
+	if err != nil || res.Found != 0 || len(res.IDs) != 0 {
+		t.Fatalf("empty answer = %+v %v", res, err)
+	}
+	if got := f.take(); strings.Contains(got[0].rawQuery, "query_by_weights") {
+		t.Fatalf("weights sent without QueryByWeights: %s", got[0].rawQuery)
+	}
+}