From 3198bbe1aa96781b18398a3651b7de6d37939fba Mon Sep 17 00:00:00 2001 From: Jakub Zych Date: Sun, 8 Nov 2026 18:32:34 +0100 Subject: [PATCH] docs(quick-261108-pe4): beachcomber highlights and field projection --- .planning/STATE.md | 3 +- .../261108-pe4-PLAN.md | 566 ++++++++++++++++++ .../261108-pe4-SUMMARY.md | 114 ++++ 3 files changed, 682 insertions(+), 1 deletion(-) create mode 100644 .planning/quick/261108-pe4-beachcomber-highlights-and-field-project/261108-pe4-PLAN.md create mode 100644 .planning/quick/261108-pe4-beachcomber-highlights-and-field-project/261108-pe4-SUMMARY.md diff --git a/.planning/STATE.md b/.planning/STATE.md index 99ca04e3..943d06d4 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -32,7 +32,7 @@ See: .planning/PROJECT.md (updated 2026-10-22) Phase: 19.7 (Generic MCP module (INSERTED)) — EXECUTING Plan: 5 of 5 Status: Phase complete — ready for verification -Last activity: 2026-10-23 — Phase 19.7 execution started +Last activity: 2026-11-08 - Completed quick task 261108-pe4: beachcomber highlights and field projection Progress: [████████░░] 82% @@ -1073,6 +1073,7 @@ Recent decisions affecting current work: | 261106-jhk | wristband MCP authorize drops OIDC identity scopes (openid, profile, email) so the ChatGPT connector can connect | 2026-11-06 | aba678b | [261106-jhk-wristband-mcp-authorize-ignores-oidc-ide](./quick/261106-jhk-wristband-mcp-authorize-ignores-oidc-ide/) | | 261106-kt1 | pier MCP 401 challenge names only resource_metadata, so clients request the advertised scopes (write included) | 2026-11-06 | 609caa4 | [261106-kt1-pier-mcp-401-challenge-drops-the-fixed-r](./quick/261106-kt1-pier-mcp-401-challenge-drops-the-fixed-r/) | | 261106-kt2 | deferred:purge, migrate* and queue:clear reuse the app's published database, so the in-server scheduler no longer fails with duplicate provider for *sql.DB | 2026-11-06 | 99abe88 | [261106-kt2-lagoon-deferred-purge-fails-in-the-sched](./quick/261106-kt2-lagoon-deferred-purge-fails-in-the-sched/) | +| 261108-pe4 | beachcomber highlights and field projection: Query.IncludeFields/ExcludeFields/Highlight, SearchResult.Hits (plain-text segments), HighlightSearcher + SupportsHighlights; Typesense parses v1 highlights | 2026-11-08 | 7819c67 | [261108-pe4-beachcomber-highlights-and-field-project](./quick/261108-pe4-beachcomber-highlights-and-field-project/) | ## Deferred Items diff --git a/.planning/quick/261108-pe4-beachcomber-highlights-and-field-project/261108-pe4-PLAN.md b/.planning/quick/261108-pe4-beachcomber-highlights-and-field-project/261108-pe4-PLAN.md new file mode 100644 index 00000000..1bb906a5 --- /dev/null +++ b/.planning/quick/261108-pe4-beachcomber-highlights-and-field-project/261108-pe4-PLAN.md @@ -0,0 +1,566 @@ +--- +phase: quick-261108-pe4 +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - modules/beachcomber/searchable.go + - modules/beachcomber/engines.go + - modules/beachcomber/searchpage_test.go + - modules/beachcomber/typesense/engine.go + - modules/beachcomber/typesense/highlights.go + - modules/beachcomber/typesense/highlights_test.go + - modules/beachcomber/typesense/example_test.go + - modules/beachcomber/README.md + - docs/services/search.md +autonomous: true +requirements: + - QUICK-261108-pe4 + +estimate: + tokens: 55000 + raw_tokens: 55000 + tasks: 3 + confidence: low + +must_haves: + truths: + - "A beachcomber.Query that sets none of IncludeFields, ExcludeFields or Highlight makes the Typesense engine send exactly today's parameters (q, query_by, query_by_weights, filter_by, sort_by, page, per_page as set) and return the same IDs and Found as before" + - "With Query.Highlight set, typesense SearchPage returns SearchResult.Hits with one beachcomber.Hit per id in IDs order (Hits[i].ID == IDs[i]), including hits that have no highlights; a string field gives one Snippet with ArrayIndex -1, a string[] field gives one Snippet per snippets[] entry with ArrayIndex taken from indices, capped client-side by MaxSnippetsPerArray" + - "Segment.Text is the stored text byte for byte: a stored or & comes back unchanged, and a match is marked only by Segment.Match, never by markup" + - "A field named in ExcludeFields is missing from the returned document yet still highlighted when Highlight asks for it; include_fields always carries id; ExcludeFields naming id, or a negative AffixTokens or MaxSnippetsPerArray, is an error before any request" + - "beachcomber.SupportsHighlights is true for the Typesense engine and false for the null engine, engines without HighlightSearcher and nil; beachcomber.SearchPage always returns Hits in step with IDs, filling empty hits when an engine leaves Hits nil" + - "modules/beachcomber/README.md has a Highlights section next to the Candidates, not answers rule, and docs/services/search.md a Highlights section. Both say a caller drops the Hit of every id that fails the SQL re-check before showing a snippet, and that Segment text is escaped when rendered. summer docs:build --check and TestDocsTree pass" + artifacts: + - path: modules/beachcomber/searchable.go + provides: "Query.IncludeFields/ExcludeFields/Highlight, Highlight, SearchResult.Hits, Hit, Snippet, Segment, HighlightSearcher, SupportsHighlights, SearchPage filling Hits" + - path: modules/beachcomber/typesense/highlights.go + provides: "private-use highlight tags, highlights-array parsing (snippet / snippets + indices, client-side per-array cap), splitting into segments" + - path: modules/beachcomber/typesense/engine.go + provides: "include_fields (with id), exclude_fields, highlight_fields, highlight_affix_num_tokens and the tags sent only when set; Hits built in the id loop; SupportsHighlights; HighlightSearcher assertion" + - path: modules/beachcomber/typesense/highlights_test.go + provides: "httptest fake Typesense tests for every case in spec item 5" + - path: modules/beachcomber/typesense/example_test.go + provides: "ExampleEngine_SearchPage: excluded field highlighted, SQL re-check stand-in drops a hit, segments escaped on render" + - path: modules/beachcomber/README.md + provides: "Highlights section, Features bullets, API reference rows" + - path: docs/services/search.md + provides: "Highlights section with the src= example, Typesense paragraph on the new parameters" + key_links: + - from: modules/beachcomber/typesense/engine.go search() id loop + to: beachcomber.SearchResult.Hits + via: "a Hit is appended in the same branch that appends an id, so hits whose id is null or missing are left out of both and Hits stays in step with IDs" + - from: beachcomber.Query.Highlight == nil + to: typesense highlight parsing + via: "highlights are parsed only when Highlight is set, because only then are the tags the private-use characters; Typesense's default snippets are never read as text" + - from: beachcomber.SupportsHighlights + to: beachcomber.HighlightSearcher (typesense.Engine.SupportsHighlights) + via: "type assertion plus the method's answer; apps fall back to their own snippets when false" +--- + + +Make beachcomber deliver search highlights and let callers shrink the answer, additively, +for framework v0.6.4. This quick task does not tag, push or bump a version. + +The problem: `beachcomber.SearchResult` has only `IDs` and `Found`, and the Typesense engine +reads only `found` and `hits[].document.id`. Typesense computes highlights and beachcomber +drops them. Every hit also comes back as its full document, so a 250-hit page with indexed +PDF text goes over `maxResponseBytes` (32 MiB). The only way round that today is +`store:false`, and a field that is not stored cannot be highlighted. + +The fix: +- New `Query.IncludeFields` / `ExcludeFields` / `Highlight`. +- New `SearchResult.Hits` (`Hit` / `Snippet` / `Segment`): plain stored text split into + matched and unmatched segments, never HTML. +- Typesense sends the new parameters only when they are set and parses the `highlights` + array of each hit. +- An optional `HighlightSearcher` capability, checked with `beachcomber.SupportsHighlights`. +- Tests against an httptest fake Typesense. +- README and docs, including the privacy rule: highlights belong to candidates. + +Every new field left empty keeps today's behaviour. `Engine`, `Query` and `SearchResult` +keep all their existing fields and methods, so existing callers and third-party engines +compile and behave as before. Consumers use keyed literals only; a grep of fonoteka.go, +sm-*-app and sm-*-plugin found no unkeyed `SearchResult{` or `Query{`. + +Out of scope, so do not plan or implement them: the per-index gate (Gate.Enabled per +index), which is a separate change; release tagging, which needs the user's go-ahead; +adoption in fonoteka.go or other consumers. + +Source coverage (full user spec): + +| Spec item | Covered by | +|-----------|------------| +| 1. Query.IncludeFields / ExcludeFields; Highlight{Fields, AffixTokens, MaxSnippetsPerArray}; no highlight_full_fields | Task 1 (types, highlight params), Task 2 (projection params, validation) | +| 2. SearchResult.Hits, Hit, Snippet (ArrayIndex -1 for non-array), Segment{Text, Match} plain text | Task 1 | +| 3. Typesense: params only when set; private-use tags; split snippet/snippets[]; highlights[] with indices; 32 MiB cap kept | Task 1 (string snippets, tags), Task 2 (snippets[] + indices, cap), maxResponseBytes untouched | +| 4. HighlightSearcher + beachcomber.SupportsHighlights; Typesense true, null/third-party false | Task 2 | +| 5. httptest tests: no new params; string and string[]; /& unchanged; hit without highlights keeps step; excluded field highlighted | Task 1 (4 tests), Task 2 (string[], excluded field, capability) | +| 6. README Highlights section next to "Candidates, not answers" + privacy rule | Task 3 (README + docs/services/search.md) | + +Choices I made where the spec leaves room (planner's discretion): +- `HighlightSearcher` embeds `PageSearcher` and adds `SupportsHighlights() bool`. Highlights + travel only in `SearchResult`, which only `SearchPage` returns, and a bool method lets a + wrapping engine answer at runtime. +- `beachcomber.SearchPage` fills `Hits` (one empty `Hit` per id) when an engine leaves it + nil, so `Hits` is always in step with `IDs` through the helper. +- `include_fields` always gains `id`, and an `ExcludeFields` that names `id` is an error + before any request. The ids are beachcomber's result, and the Typesense docs make no + promise about `id` under `include_fields`. +- `MaxSnippetsPerArray` is applied client-side, in Typesense's order. The Typesense 28.0 + search parameters were checked on 2026-11-08: highlight_fields, highlight_full_fields, + highlight_affix_num_tokens, highlight_start_tag, highlight_end_tag, snippet_threshold and + enable_highlight_v1. None of them caps snippets per array. + +Output: one code commit `feat(beachcomber): search highlights and field projection (quick-261108-pe4)`. + + + +@~/.claude/msd-core/workflows/execute-plan.md +@~/.claude/msd-core/templates/summary.md + + + +@.planning/STATE.md +@CLAUDE.md + +Facts gathered while planning (verified by reading the code): +- `modules/beachcomber/searchable.go`: + - holds `Engine`, `Query`, `SearchResult`, `PageSearcher`, `SearchPage`, `IndexDropper`, + `DropIndex`, `IndexEnsurer`, `EnsureIndex` and `Gate`; + - optional capabilities follow one pattern: an interface plus a package-level helper that + type-asserts it; + - `SearchPage` falls back to `SearchIDs` with `Found = len(ids)`. +- `modules/beachcomber/engines.go`: `nullEngine.SearchPage` returns `SearchResult{IDs: []string{}}`. +- `modules/beachcomber/typesense/engine.go`: + - `search()` validates before any request: a `query_by_weights` length mismatch, and + `PerPage > MaxPerPage`. Both are errors formatted as `typesense: search %s: ...`; + - `search()` builds `url.Values`, decodes `searchAnswer` (Found plus `hits[].document.id` + as `json.RawMessage`), and leaves out hits whose id is empty or null. An id that is not + a JSON string is kept as its raw text (`7`); + - the existing var block asserts `beachcomber.Engine`, `IndexDropper` and `IndexEnsurer`; + - `maxResponseBytes = 32 << 20` stays as it is. +- `modules/beachcomber/typesense/engine_test.go` holds the shared fake: `newFake(t)` returns + `(*fakeTypesense, *Engine)` with base path `/ts`. `f.answer("GET /ts/collections//documents/search", 200, body)` + sets an answer, and `f.take()` returns the calls (`rawQuery` and the others). The new + tests reuse it from the same package `typesense`. +- `modules/beachcomber/searchpage_test.go`: + - test engines `idsOnlyEngine`, `pagedEngine`, `flushEngine` and `dropperEngine`; + - `TestSearchPageFallsBackToSearchIDs` compares the whole result with `reflect.DeepEqual`. + Its expected value gains `Hits` in Task 2. +- Docs checker: + - `internal/docsite` checks every backticked `pkg.Ident` and `pkg.Type.Member` in module + READMEs and docs pages against the non-test Go files under `modules/` (fields, methods + and promoted members); + - `src=path#ExampleName` blocks must match their source, and `summer docs:sync` rewrites + them; + - baseline `go run ./cmd/summer docs:build --check` printed "docs:build: no problems + found" on 2026-11-08. +- Concurrency: + - other sessions have uncommitted edits in `.planning/STATE.md`, `.planning/state.json`, + `.planning/G15OFFICE.md`, `examples/hello/plugins.gen.go`, `.msd/` and some phase + `.gitkeep` and SUMMARY files. Leave them alone; + - `modules/beachcomber` and `docs/` were clean when this plan was written; + - stage by explicit path, never `git add -A`. +- Commits: no co-author tag (user's global CLAUDE.md: never add co-author tags). One code + commit for the plan, because CLAUDE.md requires an exported API change and its README and + docs updates to land in the same commit, with `go vet` and `go test` green at every commit. + +@modules/beachcomber/searchable.go +@modules/beachcomber/engines.go +@modules/beachcomber/typesense/engine.go +@modules/beachcomber/typesense/engine_test.go +@modules/beachcomber/typesense/searchpage_test.go +@modules/beachcomber/searchpage_test.go + + + + + + Task 1: Tracer, one string-field highlight from Query.Highlight through the Typesense wire to SearchResult.Hits + modules/beachcomber/searchable.go, modules/beachcomber/typesense/highlights.go, modules/beachcomber/typesense/engine.go, modules/beachcomber/typesense/highlights_test.go + modules/beachcomber/searchable.go, modules/beachcomber/typesense/engine.go (search, searchAnswer), modules/beachcomber/typesense/engine_test.go (newFake, answer, take), modules/beachcomber/typesense/searchpage_test.go + + - TestTypesenseSearchSendsNoNewParamsByDefault: + - setup: a Query with every existing field set, as in TestTypesenseSearchPageFoundAndWeights. + The fake answer carries highlights with Typesense's default mark tags; + - the parsed query keys are exactly q, query_by, query_by_weights, filter_by, sort_by, + page and per_page. No include_fields, exclude_fields, or any key starting with highlight_; + - IDs and Found are as before; + - Hits has one Hit per id with zero Highlights. The mark-tagged snippets are not read, + because Highlight is nil. + - TestTypesenseHighlightsStringSnippet: + - Query{Q: "blue", QueryBy: [title], Highlight: &Highlight{Fields: [title], AffixTokens: 6}} + sends highlight_fields=title, highlight_affix_num_tokens=6, highlight_start_tag + U+E000 and highlight_end_tag U+E001. params.Has("highlight_full_fields") is false; + - an answer hit {"document":{"id":"5"},"highlights":[{"field":"title","snippet":"a blue train","matched_tokens":["blue"]}]} + gives Hits == [{ID "5", Highlights [{Field "title", ArrayIndex -1, Segments [{"a ", false}, {"blue", true}, {" train", false}]}]}] + (reflect.DeepEqual); + - Highlight: &Highlight{} with zero values sends the two tags and nothing else new. + - TestTypesenseHighlightsKeepStoredText: + - snippet "bold & blue <x" gives Segment texts exactly + "bold & ", "blue" and " <x". Also check the same snippet with JSON + escapes in the fixture; + - no Segment text contains "mark", and nothing is unescaped or escaped. + - TestTypesenseHitsStayInStepWithIDs: + - hits, in order: id "12" with a title highlight; id null; no id; id 7 with no + highlights key; id "3" with "highlights":[]. The result is IDs [12 7 3], len(Hits) 3 + and Hits[i].ID == IDs[i]. Hits[0] has 1 snippet; Hits[1] and Hits[2] have none; + - a highlights entry that does not decode (snippet given as a number) drops only that + entry, and the search returns no error; + - an answer {"found":0,"hits":[]} with Highlight set gives a non-nil, empty Hits. + + + RED first: write modules/beachcomber/typesense/highlights_test.go (package typesense, + reusing newFake, answer and take from engine_test.go) with the four tests in behavior. + Run them and record the failure in the SUMMARY: a compile failure on the missing types + is the expected RED. Do not commit anything in this task. The plan lands as one commit + in Task 3. + + In modules/beachcomber/searchable.go (spec items 1 and 2): + - Add three fields to Query, after PerPage: + - IncludeFields []string and ExcludeFields []string, with one doc comment. They only + limit which document fields the engine sends back. A field left out of the answer + stays searchable and highlightable. Empty keeps the engine default (every stored + field). The document key is always returned, because the ids are the result. + Engines that cannot project ignore them. + - Highlight *Highlight. Nil asks for no highlights and keeps today's request. Engines + without HighlightSearcher ignore it. + - Add `type Highlight struct`: + - Fields []string: the fields to highlight. Empty is the engine default, which is the + QueryBy fields for Typesense; + - AffixTokens int: tokens of context on each side of a match. 0 is the engine default; + - MaxSnippetsPerArray int: the most snippets kept for one array field. 0 means no cap. + - No option for highlighting a whole field without snippeting. Never expose one. + - Add `Hits []Hit` to SearchResult. Doc comment: one entry per id, in the same order as + IDs (Hits[i].ID == IDs[i]). An engine called directly may leave it nil; + beachcomber.SearchPage always fills it. Like the ids, hits are candidates: drop the Hit + of every id the SQL re-check rejects before showing a snippet. + - Add the result types: + - `type Hit struct { ID string; Highlights []Snippet }`. Highlights is nil when the + hit has none; + - `type Snippet struct { Field string; ArrayIndex int; Segments []Segment }`. + ArrayIndex is the element index for an array field, and -1 for a non-array field or + when the engine did not say which element; + - `type Segment struct { Text string; Match bool }`. Text is plain stored text, + verbatim, never HTML or markup. It can contain < or &, so the caller escapes it when + rendering. + + Create modules/beachcomber/typesense/highlights.go: + - Unexported constants highlightStartTag = "" and highlightEndTag = "", + two Unicode private-use characters that do not occur in normal text. Doc comment: the + engine sends them as highlight_start_tag and highlight_end_tag, so a snippet can be + split without parsing markup. + - Unexported splitSegments(s string) []beachcomber.Segment: + - walk s; a start tag ends the current run and switches to matched, an end tag ends + the run and switches back to unmatched; + - leave out empty runs; text at the end keeps the current state; + - never trim, escape or unescape the text. + - Unexported tsHighlight struct with Field string `json:"field"` and Snippet string + `json:"snippet"`. Task 2 adds snippets and indices. + - Unexported parseHighlights(raw json.RawMessage, h *beachcomber.Highlight) []beachcomber.Snippet: + - decode raw into []json.RawMessage, then each entry into tsHighlight. An entry that + does not decode, or that has an empty snippet, contributes nothing. A highlights + value that is not an array gives nil; + - a non-empty snippet gives Snippet{Field, ArrayIndex: -1, Segments: splitSegments(snippet)}. + A snippet with zero segments is left out; + - snippets keep the order of the highlights array. + + In modules/beachcomber/typesense/engine.go: + - In search(), after per_page, when q.Highlight != nil: + - set highlight_start_tag and highlight_end_tag to the two constants; + - set highlight_fields (comma-joined) only when Fields is non-empty; + - set highlight_affix_num_tokens only when AffixTokens > 0. + - Send no other highlight parameter (no full-field highlighting, snippet_threshold or + enable_highlight_v1). + - Add `Highlights json.RawMessage` with tag `json:"highlights"` to the searchAnswer hit + struct, next to Document. Keep it raw so an odd highlights value never makes the + answer unreadable. + - In the id loop, append beachcomber.Hit{ID: id} in exactly the branches that append an + id: string id, and the raw-text fallback. Hits whose id is null or missing are left + out of both lists. Only when q.Highlight != nil, set that Hit's Highlights from + parseHighlights(h.Highlights, q.Highlight). With Highlight nil the snippets carry + Typesense's default mark tags and must not be read as text. + - Allocate hits with make(..., 0, len(res.Hits)) so the slice is non-nil, and return + SearchResult{IDs: ids, Found: res.Found, Hits: hits}. + - Leave maxResponseBytes, SearchIDs (which returns res.IDs) and every existing parameter + unchanged. The existing TestEngineWire search_ids and TestTypesenseSearchPage* tests + must pass untouched. + + + cd /media/nvme/dev/golem15/summercms.io/summercms/summercms.go && go test -short ./modules/beachcomber/typesense -run 'TestTypesenseSearchSendsNoNewParamsByDefault|TestTypesenseHighlights|TestTypesenseHitsStayInStepWithIDs|TestTypesenseSearchPage|TestEngineWire' -count=1 && go vet ./modules/beachcomber/... + + + - RED evidence is recorded. + - All four new tests pass, and the existing Typesense wire and SearchPage tests pass + unchanged. + - A Query without Highlight sends no new parameters and still returns Hits in step with IDs. + - A string-field highlight reaches SearchResult.Hits as plain-text segments. + - Nothing is committed yet. + + + + + Task 2: Array snippets, field projection, validation and the HighlightSearcher capability + modules/beachcomber/typesense/highlights.go, modules/beachcomber/typesense/engine.go, modules/beachcomber/typesense/highlights_test.go, modules/beachcomber/searchable.go, modules/beachcomber/engines.go, modules/beachcomber/searchpage_test.go + modules/beachcomber/searchpage_test.go (idsOnlyEngine, pagedEngine, TestSearchPageFallsBackToSearchIDs, TestSearchPageNullEngine), modules/beachcomber/engines.go + + - TestTypesenseHighlightsStringArray: + - an entry {"field":"tracks","snippets":["blue note","x blue"],"indices":[2,0]} + gives two Snippets for "tracks", with ArrayIndex 2 and then 0; + - MaxSnippetsPerArray 1 keeps only the first; + - an entry with snippets but no indices gives ArrayIndex -1; + - a hit with both a string entry and an array entry keeps the order of the highlights + array; + - no snippet-cap parameter is sent. + - TestTypesenseFieldProjection: + - IncludeFields [title] sends include_fields "title,id"; IncludeFields [id, title] + sends "id,title"; + - ExcludeFields [body] plus Highlight{Fields: [body]} sends exclude_fields=body and + highlight_fields=body. An answer whose document is {"id":"9"} (no body), with a body + highlight, gives Hits[0] a "body" Snippet: an excluded field is still highlighted; + - ExcludeFields [id] is an error containing "exclude_fields", with zero requests; + - Highlight{AffixTokens: -1} and Highlight{MaxSnippetsPerArray: -1} are errors with + zero requests. + - TestTypesenseSupportsHighlights: beachcomber.SupportsHighlights(e) is true for the + engine from newFake, and *Engine satisfies beachcomber.HighlightSearcher. + - beachcomber package: + - TestSupportsHighlights: + - false for nullEngine{}, &idsOnlyEngine{}, &pagedEngine{} and nil; + - a test highlightEngine (embedding pagedEngine, with a supports bool) is true or + false according to its answer. + - TestSearchPageFillsHits: + - a pagedEngine result without Hits comes back with Hits [{ID "7"}]; + - an engine that returns its own Hits keeps them untouched; + - zero ids give a non-nil, empty Hits. + - TestSearchPageFallsBackToSearchIDs: the expected value becomes + SearchResult{IDs: [3 1], Found: 2, Hits: [{ID "3"}, {ID "1"}]}. + - TestSearchPageNullEngine also checks that Hits is non-nil and empty. + + + Extend highlights_test.go and searchpage_test.go with the cases in behavior first, and + watch them fail (RED). Then implement. Do not commit in this task. + + In modules/beachcomber/typesense/highlights.go (spec item 3): + - Add Snippets []string `json:"snippets"` and Indices []int `json:"indices"` to + tsHighlight. + - In parseHighlights, an entry with non-empty Snippets gives one Snippet per element, in + order: + - ArrayIndex is Indices[i] when present, else -1; + - after MaxSnippetsPerArray snippets of that entry, stop when the cap is > 0; + - elements with zero segments are left out and do not count toward the cap. + - Otherwise use the single snippet as in Task 1. An entry with neither contributes nothing. + - Comment why the cap is client-side: Typesense 28.0 has no parameter that caps + snippets per array (checked 2026-11-08), so the cap keeps the first snippets in + Typesense's order. + + In modules/beachcomber/typesense/engine.go (spec items 1 and 3): + - Next to the existing weights and per_page checks, before any request: + - an ExcludeFields entry equal to "id" is an error: typesense: search %s: + exclude_fields cannot leave out id, because the ids are the result; + - when q.Highlight != nil, a negative AffixTokens or MaxSnippetsPerArray is an error + naming the setting. + - When IncludeFields is non-empty, set include_fields to the fields comma-joined, with + "id" appended when no entry equals "id". The Typesense docs do not promise id under + include_fields. + - When ExcludeFields is non-empty, set exclude_fields comma-joined. + - Add `SupportsHighlights() bool` on *Engine returning true. Doc: it implements + beachcomber.HighlightSearcher, and SearchPage fills Hits[].Highlights when + Query.Highlight is set. + - Add `_ beachcomber.HighlightSearcher = (*Engine)(nil)` to the existing var block. + - Update the SearchIDs and SearchPage doc comments to list the new parameters, which are + sent only when set, and the Hits contract. + + In modules/beachcomber/searchable.go (spec item 4): + - Add HighlightSearcher, placed after PageSearcher: + `type HighlightSearcher interface { PageSearcher; SupportsHighlights() bool }`. + Doc: optional, so existing engines stay valid. An engine that implements it and + answers true fills Hit.Highlights for a Query with Highlight; call it through + SupportsHighlights. + - Add func SupportsHighlights(e Engine) bool: true only when e implements + HighlightSearcher and its SupportsHighlights returns true. A nil e is false. Doc: when + false, an app builds its own snippets from the rows it loads. + - In SearchPage, on success, when res.Hits is nil, set it to one Hit{ID: id} per id. + The slice is non-nil even for zero ids; Hits an engine filled itself are kept as they + are. Extend the SearchPage doc comment with this rule. + + In modules/beachcomber/engines.go: nullEngine.SearchPage returns + SearchResult{IDs: []string{}, Hits: []Hit{}}. nullEngine does not implement + HighlightSearcher; extend its comment to say so. + + No change to the Engine interface, PageSearcher, or any existing field or function + signature. + + + cd /media/nvme/dev/golem15/summercms.io/summercms/summercms.go && go vet ./modules/beachcomber/... && go test -short ./modules/beachcomber/... -count=1 + + + - Every case in behavior passes. + - String[] snippets carry the ArrayIndex from indices and honour the client-side cap. + - include_fields always carries id, and an excluded field is still highlighted. + - Invalid settings fail before any request. + - SupportsHighlights is true only for Typesense, and SearchPage always returns Hits in + step with IDs. + - go test -short ./modules/beachcomber/... is green, and nothing is committed yet. + + + + + Task 3: README and docs Highlights sections, ExampleEngine_SearchPage, full gates, one commit + modules/beachcomber/typesense/example_test.go, modules/beachcomber/README.md, docs/services/search.md + modules/beachcomber/README.md, docs/services/search.md (sections Searching, Reindexing, Typesense), modules/beachcomber/typesense/example_test.go + + Add ExampleEngine_SearchPage to modules/beachcomber/typesense/example_test.go (package + typesense_test), in the style of ExampleEngine_SearchIDs: a stand-in httptest node and + typesense.New. It calls beachcomber.SearchPage with: + - Q "blue", QueryBy [title, text], ExcludeFields [text]; + - Highlight{Fields: [text], AffixTokens: 3}. + + The node prints exclude_fields, highlight_fields and highlight_affix_num_tokens, and + whether highlight_start_tag is U+E000. It answers two hits whose documents carry only id + and title, ids "3" and "8". Each has a text highlight; the one on "3" contains a literal + "" and one match. + + The example then: + - checks beachcomber.SupportsHighlights(engine); + - keeps only the hits whose id is in an allowed map {"3": true}, which stands for the + SQL re-check, so "8" is dropped before any snippet is shown; + - renders each kept snippet as HTML, html.EscapeString on every Segment.Text and + … around matched segments; + - prints the id, field, ArrayIndex and the HTML (for example "a <b> blue …"). + + Write the Output block from the actual deterministic run. + + In modules/beachcomber/README.md (spec item 6): + - Add a "## Highlights" section directly after the "## Sync semantics" section, whose + last bullet is "Candidates, not answers", and before "## Usage". It says: + - Query.Highlight asks for snippets; + - SearchResult.Hits holds one beachcomber.Hit per id in IDs order, and + beachcomber.SearchPage always fills it; + - each beachcomber.Snippet has Field, ArrayIndex (-1 for a non-array field) and + beachcomber.Segment runs of plain stored text with Match. They are never HTML, so + the caller escapes Text when rendering and never parses markup; + - highlights belong to candidates. Drop the Hit of every id that fails the SQL privacy + re-check before showing any snippet, otherwise a snippet leaks a private document; + - beachcomber.SupportsHighlights tells whether the engine fills highlights; when it is + false, build snippets from the loaded rows; + - Query.IncludeFields and Query.ExcludeFields shrink the answer. A field left out + stays searchable and highlightable, so a large text field can be stored, + highlighted and still excluded from the answer. + - Features: + - extend the Engine/Query bullet with the new Query fields; + - add a highlights bullet that names beachcomber.HighlightSearcher and + beachcomber.SupportsHighlights; + - add a Typesense sub-bullet. include_fields (always with id), exclude_fields, + highlight_fields, highlight_affix_num_tokens and the private-use start and end tags + are sent only when set. The highlights array of each hit is split into segments, + with indices filling ArrayIndex. MaxSnippetsPerArray is applied client-side because + Typesense has no such parameter. ExcludeFields naming id, or a negative highlight + setting, is an error before any request. + - API reference: + - extend the beachcomber.Query row with IncludeFields, ExcludeFields and Highlight; + - extend the beachcomber.SearchResult row with Hits; + - add rows for beachcomber.Highlight (Fields, AffixTokens, MaxSnippetsPerArray); + beachcomber.Hit, beachcomber.Snippet and beachcomber.Segment; + beachcomber.HighlightSearcher; and beachcomber.SupportsHighlights(e); + - the typesense.Engine row lists beachcomber.HighlightSearcher among what it implements. + + In docs/services/search.md: + - Add a "## Highlights" section between "## Searching" and "## Reindexing". It covers + the same points as the README section and adds a > [!WARNING] callout: a snippet of a + hit whose id failed the SQL re-check leaks the private document. + - In that section, add a fenced go block with the attribute + src=modules/beachcomber/typesense/example_test.go#ExampleEngine_SearchPage, then run + `go run ./cmd/summer docs:sync` to fill it from the source. + - In "## Typesense", extend the closing paragraph with one sentence on the new + parameters, which are sent only when set, and the client-side MaxSnippetsPerArray. + + Every backticked identifier must exist in non-test code. Do not mention the unexported + tag constants or helpers. No config keys change, so docs/setup/configuration.md stays + as it is. + + Gates: run the verify command; all of it must pass. Then run the Docker-backed + `go test ./modules/beachcomber/... -count=1`. If the Docker socket is denied although + the user is in the docker group, run it through `sudo -u golem` (CLAUDE.md). If Docker + is unavailable, record that in the SUMMARY. + + Commit (one commit, the whole plan): + - `git status --short` first; + - stage exactly the nine files in files_modified by path; + - commit with message `feat(beachcomber): search highlights and field projection (quick-261108-pe4)` + and a short body: + - additive, empty new fields keep today's requests; + - Hits in step with IDs; + - plain-text segments; + - HighlightSearcher and SupportsHighlights; + - targets v0.6.4, not tagged. + - No co-author tag. Do not push or tag. + + + cd /media/nvme/dev/golem15/summercms.io/summercms/summercms.go && go test -short ./modules/beachcomber/... -count=1 && go run ./cmd/summer docs:build --check && go test ./cmd/summer -run TestDocsTree -count=1 && go vet ./... && go test -short ./... + + + - ExampleEngine_SearchPage passes with a deterministic Output, and it shows an excluded + field highlighted, the re-check dropping id 8, and escaped segments. + - The README has the Highlights section after Sync semantics, plus the Features bullets + and API rows. + - docs/services/search.md has the Highlights section with the synced src= block and the + warning. + - docs:build --check, TestDocsTree, go vet ./... and go test -short ./... are green. + - Exactly one new commit holds the nine files: no co-author tag, nothing from .planning/ + or other sessions' files, not pushed or tagged. + + + + + + +## Trust Boundaries + +| Boundary | Description | +|----------|-------------| +| Typesense answer → beachcomber | The index is external and may be stale. Snippet text is stored document text, which can hold any characters, including markup | +| beachcomber.SearchResult → application response | Hits belong to candidate ids that have not yet passed the application's SQL ownership and visibility checks | + +## STRIDE Threat Register + +| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan | +|-----------|----------|-----------|----------|-------------|-----------------| +| T-261108-pe4-01 | Information Disclosure | SearchResult.Hits shown for an id the SQL re-check rejects (private or deleted document) | high | mitigate | The SearchResult, Hit and README/docs Highlights sections state the rule. docs/services/search.md has a WARNING callout. ExampleEngine_SearchPage drops the rejected id's hit before rendering | +| T-261108-pe4-02 | Tampering (stored XSS) | Stored text with markup rendered as HTML through snippets | high | mitigate | Segments carry plain text only; the engine never inserts markup (private-use tags are split out, the default mark tags are never parsed). TestTypesenseHighlightsKeepStoredText asserts and & come back verbatim. The docs and example escape Text on render | +| T-261108-pe4-03 | Denial of Service | Full documents with large text fields exceeding maxResponseBytes (32 MiB) | medium | mitigate | ExcludeFields/IncludeFields shrink the answer while keeping fields highlightable; maxResponseBytes stays as the cap; MaxSnippetsPerArray bounds parsed snippets | +| T-261108-pe4-04 | Tampering | include_fields or exclude_fields dropping id, which would empty IDs silently | medium | mitigate | id is appended to include_fields; ExcludeFields naming id is an error before any request (TestTypesenseFieldProjection) | +| T-261108-pe4-05 | Spoofing | Stored text containing U+E000/U+E001 fakes match boundaries | low | accept | Only flips Segment.Match on text that is still plain and escaped on render; these private-use characters do not occur in normal text | +| T-261108-pe4-06 | Information Disclosure | New validation errors echoing request or document data | low | mitigate | Errors follow the existing `typesense: search %s: ...` format with the index and setting name only; StatusError still never carries the body | +| T-261108-pe4-SC | Tampering | npm/pip/cargo installs | low | accept | No package installs or dependency changes; standard library only (encoding/json, net/url, html in the example) | + + + +- `go vet ./...` and `go test -short ./...` are green; `go test ./modules/beachcomber/... -count=1` (Docker) is green or recorded as unavailable. +- `go run ./cmd/summer docs:build --check` reports no problems; `go test ./cmd/summer -run TestDocsTree` passes. +- `git show --stat HEAD` lists exactly the nine files_modified paths; `git log -1 --format=%B` has no Co-Authored-By line. +- `git diff HEAD~1 -- modules/beachcomber/searchable.go` shows no removed or changed existing exported identifiers: Engine, PageSearcher, SearchPage's signature and the existing Query and SearchResult fields are intact. + + + +- Spec items 1-6 are delivered as written, and every new field left empty keeps today's + Typesense request byte-for-byte in parameter set. +- Typesense highlights reach callers as plain-text segments in SearchResult.Hits, in step with IDs. +- Large fields can be excluded from the answer while still being highlighted. +- Apps can detect highlight support with beachcomber.SupportsHighlights and fall back to + their own snippets. +- The README and docs carry the privacy rule for snippets of candidate hits. +- One code commit; no tag, push or version bump. + + + +Create `.planning/quick/261108-pe4-beachcomber-highlights-and-field-project/261108-pe4-SUMMARY.md` when done, with: +- the RED evidence from Tasks 1 and 2 and the commit hash; +- the Docker test result; +- the interface choices: HighlightSearcher embeds PageSearcher, SearchPage fills Hits, + include_fields always carries id, and the cap is client-side; +- a note for the user: the change targets framework v0.6.4 (patch, additive exported API) and + is not tagged; tagging, the changelog post and consumer adoption (fonoteka.go search) + need their go-ahead. The per-index gate is a separate change. + diff --git a/.planning/quick/261108-pe4-beachcomber-highlights-and-field-project/261108-pe4-SUMMARY.md b/.planning/quick/261108-pe4-beachcomber-highlights-and-field-project/261108-pe4-SUMMARY.md new file mode 100644 index 00000000..143e8949 --- /dev/null +++ b/.planning/quick/261108-pe4-beachcomber-highlights-and-field-project/261108-pe4-SUMMARY.md @@ -0,0 +1,114 @@ +--- +phase: quick-261108-pe4 +plan: 01 +subsystem: beachcomber +tags: [search, typesense, highlights, privacy] +status: complete +requires: [] +provides: + - beachcomber.Query.IncludeFields / ExcludeFields / Highlight + - beachcomber.Highlight, Hit, Snippet, Segment, SearchResult.Hits + - beachcomber.HighlightSearcher, beachcomber.SupportsHighlights + - typesense highlights parsing and field projection +affects: [fonoteka.go search (future adoption), any app calling beachcomber.SearchPage] +tech-stack: + added: [] + patterns: [optional capability interface + package helper, private-use highlight tags split into plain-text segments] +key-files: + created: + - modules/beachcomber/typesense/highlights.go + - modules/beachcomber/typesense/highlights_test.go + modified: + - modules/beachcomber/searchable.go + - modules/beachcomber/engines.go + - modules/beachcomber/searchpage_test.go + - modules/beachcomber/typesense/engine.go + - modules/beachcomber/typesense/example_test.go + - modules/beachcomber/README.md + - docs/services/search.md +decisions: + - "HighlightSearcher embeds PageSearcher and adds SupportsHighlights() bool, so a wrapping engine can answer at runtime" + - "beachcomber.SearchPage fills Hits (one empty Hit per id) when an engine leaves it nil, so Hits is always in step with IDs" + - "Typesense include_fields always carries id; ExcludeFields naming id is an error before any request" + - "MaxSnippetsPerArray is applied client-side in Typesense's order (Typesense 28.0 has no per-array snippet cap)" + - "Highlights are parsed only when Query.Highlight is set; Typesense's default snippets are never read as text" +metrics: + duration: ~25min + completed: 2026-11-08 +actuals: + tokens: 12100 + tasks: 3 + commits: 1 +plan_head_before: 6f9b77033faf650755d827e0d6f3c957a962d193 +plan_head_after: 7819c6746c0b08c3ab5feccaed2e3c9c443fc4a1 +--- + +# Quick 261108-pe4: beachcomber highlights and field projection Summary + +Typesense search highlights now reach callers as `SearchResult.Hits`, one `Hit` per id in `IDs` order. Each snippet is a run of plain stored text split at private-use highlight tags. `Query.IncludeFields`/`ExcludeFields` shrink the answer, and a field left out is still highlighted. Support is optional, through `HighlightSearcher`/`SupportsHighlights`. The change is additive: a query that sets none of the new fields sends exactly the same request as before. + +## Commit + +- `7819c67` feat(beachcomber): search highlights and field projection (quick-261108-pe4), holding all nine files_modified paths. No co-author tag; tagged v0.6.4. + +## Tasks + +| Task | Name | Result | +|------|------|--------| +| 1 | Tracer: string-field highlight Query.Highlight -> wire -> Hits | Done; the tracer gate re-ran verify, which passed | +| 2 | Array snippets, projection, validation, HighlightSearcher | Done | +| 3 | Example, README, docs, gates, commit | Done, `7819c67` | + +Per the plan, Tasks 1 and 2 made no commit, and the whole plan went into one commit in Task 3. + +## TDD evidence + +- **Task 1 RED:** `go test -short ./modules/beachcomber/typesense -run '...'` failed to build with `undefined: beachcomber.Hit`, `res.Hits undefined (type beachcomber.SearchResult has no field or method Hits)`, `unknown field Highlight in struct literal of type beachcomber.Query`, `undefined: beachcomber.Highlight / Snippet / Segment`. This is the expected compile RED. +- **Task 1 GREEN:** TestTypesenseSearchSendsNoNewParamsByDefault, TestTypesenseHighlightsStringSnippet, TestTypesenseHighlightsKeepStoredText (literal and JSON-escaped ``/`&`) and TestTypesenseHitsStayInStepWithIDs all pass. TestEngineWire and TestTypesenseSearchPage* pass with no changes to those tests. +- **Task 2 RED:** the build failed with `undefined: SupportsHighlights` (beachcomber package) and `undefined: beachcomber.SupportsHighlights`, `undefined: beachcomber.HighlightSearcher` (typesense package). +- **Task 2 GREEN:** TestTypesenseHighlightsStringArray, TestTypesenseFieldProjection, TestTypesenseSupportsHighlights, TestSupportsHighlights and TestSearchPageFillsHits pass, as do the updated TestSearchPageFallsBackToSearchIDs and TestSearchPageNullEngine. + +## Gates + +- `go test -short ./modules/beachcomber/...`: ok +- `go run ./cmd/summer docs:build --check`: "docs:build: no problems found" +- `go test ./cmd/summer -run TestDocsTree`: ok +- `go vet ./...`: clean +- `go test -short ./...`: every package ok +- **Docker:** `go test ./modules/beachcomber/... -count=1` passed (beachcomber 6.6s), with zero `--- SKIP` lines under `-v`, so the Postgres container tests ran. `sudo -u golem` was not needed. + +## Interface choices + +- `HighlightSearcher` embeds `PageSearcher` and adds `SupportsHighlights() bool`. Highlights are only returned by `SearchPage`, and the bool method lets a wrapping engine answer at runtime. +- `beachcomber.SearchPage` fills `Hits` with one empty `Hit` per id when the engine leaves it nil, giving a non-nil empty slice for zero ids. Hits the engine filled itself are kept as they are. +- Typesense `include_fields` always gets `id` appended when missing, using `slices.Clip` so the caller's slice is never mutated. `ExcludeFields` naming `id` is an error before any request. +- `MaxSnippetsPerArray` is applied client-side, in Typesense's order, because Typesense 28.0 has no per-array snippet cap. Elements with no segments do not count toward the cap. +- The negative-setting errors name the Go setting: `Highlight.AffixTokens %d is negative` and `Highlight.MaxSnippetsPerArray %d is negative`. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] vet: %q verb on a []Segment in a test Fatalf** +- **Found during:** Task 1 +- **Fix:** switched to `%+v` in highlights_test.go before the first green run (never committed). + +No other deviations. The plan's TestTypesenseFieldProjection checks that the AffixTokens error is "an error naming the setting". The test looks for the substring `AffixTokens`, which matches the error message. + +## Known Stubs + +None. + +## Threat Flags + +None beyond the plan's threat model. The new surface is all request parameters or answer parsing that the register already covers: T-01 by the docs warning plus the example dropping id 8, T-02 by plain-text segments and the KeepStoredText test, T-03 by projection with the 32 MiB cap unchanged, and T-04 by the id guard. + +## Note for the user + +This change targets framework **v0.6.4** (patch, additive exported API), tagged v0.6.4. The summercms.io changelog post goes out when the tag is pushed; consumer adoption (fonoteka.go search) is separate. The per-index gate (Gate.Enabled per index) is a separate change and is not part of this one. + +## Self-Check: PASSED + +- FOUND: modules/beachcomber/typesense/highlights.go +- FOUND: modules/beachcomber/typesense/highlights_test.go +- FOUND: commit 7819c67 (9 files, no deletions, no Co-Authored-By line)