feat: search the management list and the MCP material tools

- GET /api/v1/bm/materials takes q: candidates over every material, in rank order, with the snippet of the student list; 503 when the engine cannot answer - bm_materials and bm_manage_list_materials take q; under MCP the snippet is plain text with ellipses where text was cut - MCPMaterialPrivacy no longer claims no file text is ever returned - README documents q, the snippet, the 503, the Search section and bm:materials-reindex
2026-11-08 19:53:26 +01:00
parent ffd19342b2
commit dd13795e6f
7 changed files with 126 additions and 17 deletions
--- a/README.md
+++ b/README.md
@@ -46,7 +46,7 @@ Every route except the calendar feed itself needs `Authorization: Bearer <golem1
 | DELETE | `/_bm/api/v1/readings/{id}/progress` | | Clears the mark (idempotent) |
 | PUT | `/_bm/api/v1/readings/{id}/owned` | | Marks the reading owned (idempotent) |
 | DELETE | `/_bm/api/v1/readings/{id}/owned` | | Clears the owned mark (idempotent) |
-| GET | `/_bm/api/v1/materials` | `course`, `meeting`, `tag` (slug), `type`, `page`, `per_page` (1–100) | One page of the published materials the caller may see |
+| GET | `/_bm/api/v1/materials` | `q` (search, 2–100 characters), `course`, `meeting`, `tag` (slug), `type`, `page`, `per_page` (1–100) | One page of the published materials the caller may see, by relevance with `q`, see [Search](#search) |
 | GET | `/_bm/api/v1/materials/{id}` | | One published material the caller may see, with body, tags, files and `can_manage` |
 | GET | `/_bm/api/v1/checklists` | `meeting` (ID) | The caller's checklists with progress, without items |
 | POST | `/_bm/api/v1/checklists` | | Creates a checklist for the caller; 201 |
@@ -189,6 +189,15 @@ Newest publication first. `meeting` keeps the materials assigned to that zjazd (
  "can_manage": false}}
 ```

+`q` searches (see [Search](#search)): a trimmed `q` of fewer than 2 characters is ignored, one over 100 is a 422. With `q` the page is in relevance order, combined with the other filters (AND); `meta.total` is counted in SQL over the materials the caller may see, never taken from the engine. A row whose match is in the body carries one `snippet`:
+
+```json
+{"snippet": {"source": "body", "file_name": null, "lead": true, "trail": true,
+  "segments": [{"text": "proces ", "hit": false}, {"text": "globalizacji", "hit": true}, {"text": " zmienia państwa", "hit": false}]}}
+```
+
+`segments` are the engine's plain text, verbatim, never markup: render each `text` as text and mark the `hit` ones. `lead` and `trail` say that text was cut before or after the excerpt. The key is absent when there is no snippet, and always without `q`, so the list without `q` is byte for byte as before. When the search engine is not configured or does not answer, a request with `q` is a 503 `{"error": true, "message": "Search is unavailable."}`; the list without `q` keeps working and never calls the engine.
+
 ### Private materials

 A private material (`is_private`) is visible only to its author (`author_user_id`) and to the users it is shared with (`jz_bm_material_shares`); a public one to every student. The rule is one SQL fragment (`visibleMaterialSQL` in `controllers/api/material_access.go`) applied to every student read: `/materials`, `/materials/{id}`, the materials of `/courses/{id}`, the `materials` counters of `/courses` and `/courses/{id}`, the material linked from a checklist item, and through the token twins the student MCP tools. For anyone else a private material is a 404 and is not counted.
@@ -243,7 +252,7 @@ The calendar is RFC 5545 (CRLF lines folded at 75 octets, escaped TEXT) with tim
 | Method | Route | Scope, permission | Answer |
 |--------|-------|-------------------|--------|
 | GET | `/api/v1/bm/me` | read | The token owner's `user_id` and the token (`id`, `name`, `scopes`, `expires_at`). |
-| GET | `/api/v1/bm/materials` | read, materials | Every material, drafts included, newest first. |
+| GET | `/api/v1/bm/materials` | read, materials | Every material, drafts and private ones included, newest first. `q` (2–100 characters) searches all of them, in relevance order, with the `snippet` of the student list; a 503 `Search is unavailable.` when the engine cannot answer. |
 | GET | `/api/v1/bm/materials/{id}` | read, materials | One material, draft or not, with its public `files`. 404 when unknown. |
 | POST | `/api/v1/bm/materials` | write, materials | Upsert on `title` + `course_id`: 201 `{"data", "created": true}` for a new material, 200 `{"data", "created": false}` for an updated one. |
 | PUT | `/api/v1/bm/materials/{id}` | write, materials | The same body applied to that material, so the title or course can change. 404 when unknown; 422 on `title` when another material already has the new title and course. |
@@ -329,7 +338,8 @@ The file tools return metadata and links, never file contents, and their descrip
 **The MCP view.** Only an answer to an MCP tool call (a request `pier` dispatched in-process, carrying the `api_token` dispatch marker, which an HTTP request can never carry) is changed; plain HTTP answers are byte for byte as before.

 - Links: the middleware `jz.bm.mcp-links` (on the token twins and the management API) prefixes every JSON string under a key named `url` or ending in `_url` that starts with a single `/` with `app.url`, else `jz.bm.public_url`, else the request's scheme and Host, so file and syllabus links are ready to follow.
- Material bodies (possible transcript text): `bm_material`, `bm_manage_list_materials` and `bm_manage_show_material` leave `body` out unless `include_body=true`, and mark an omitted non-empty body with `body_omitted: true`; the material write answers never echo it. Material tools return metadata and links, never file contents.
+- Material bodies (possible transcript text): `bm_material`, `bm_manage_list_materials` and `bm_manage_show_material` leave `body` out unless `include_body=true`, and mark an omitted non-empty body with `body_omitted: true`; the material write answers never echo it. Material tools return metadata and links, never whole file contents.
+- Search: `bm_materials` and `bm_manage_list_materials` take `q` as the HTTP lists do. With `q` the results come by relevance and a material may carry `snippet` as a plain string, the matched passage with `…` where text was cut; it can come from the body or, once file text is indexed, from an attached file.
 - Throttling: `jz.bm.manage` keys an MCP call by its user (`jz.bm.manage.user|<id>`) before the bearer and IP keys.

 ## Student app
@@ -360,8 +370,27 @@ Defaults in `config/config.yaml`, read as `jz.bm.<key>`:
 | `jz.bm.dashboard_limit` | `5` | Sessions and deadlines on `/dashboard`. |
 | `jz.bm.public_url` | `""` | Absolute base URL of the application (`https://bm.jakubzych.com`), used in the calendar feed URL and the course links inside the feed. Empty means the scheme and Host of the request; set it behind a TLS-terminating proxy. The application overrides it in `config/jz.bm.yaml`. |

+## Search
+
+Materials are searched through the framework's `beachcomber` module with the Typesense engine (`search.driver: typesense`; the blank import of `beachcomber/typesense` is in `search.go`). `models.Material` is a `beachcomber.Searchable` (`models/material_search.go`) kept in the index `jz_bm_materials` (after `search.prefix`). A document holds `id`, `title`, `source`, `url`, `tags` and `tag_slugs`, `course` (name and code), `course_id`, `meeting_id`, `meeting_number`, `type`, `published_at` (Unix seconds, 0 for a draft), the body with its Markdown syntax removed and `file_texts` (the text of attached files). No field sets a locale, so the default one folds diacritics and `lodz` finds `Łódź`.
+
+The index holds every material, drafts and private ones included, and no visibility: the engine's ids are candidates only. A search asks for at most 250 of them (title weighted over tags, source and course, then body, link and file text), and SQL re-gates them with the same visibility rule and filters as the list, counts them and pages over them in the engine's order. A snippet is read only for a row SQL returned, so a material the caller may not see never reaches an answer, its count or a snippet.
+
+The beachcomber service is app-wide, and so is its Gate: jz.bm installs one that is on only for contexts it marks (`classes.MarkMaterialSearch`), so `golem15.chatchannels` messages and DMs are never indexed. Every material write syncs explicitly after it commits: the admin form (create, update, delete), `POST`, `PUT` and `DELETE /api/v1/bm/materials`. A failed sync is logged with the material id and never fails the write.
+
+Configuration, in the app's `config/search.yaml`: `search.driver`, `search.prefix`, `search.typesense.host`, `port`, `protocol` and `connection_timeout_seconds`. The API key goes only in the environment, `SUMMER_SEARCH__TYPESENSE__API_KEY`; with no key the engine is not configured, nothing is sent and every search answers 503.
+
 ## CLI commands

+### bm:materials-reindex
+
+```sh
+./bin/bm bm:materials-reindex
+./bin/bm bm:materials-reindex --fresh
+```
+
+Ensures the `jz_bm_materials` collection and syncs every material through the same path as the sync after a save, ending with `Reindexed: N synced, M failed`; it exits non-zero when any material failed. `--fresh` first drops the collection and recreates it from `models.Material.SearchIndexSchema`: use it on the first deploy and after a schema change. Without an engine the command refuses with an error naming `search.driver` and `search.typesense.api_key`; `--if-configured` makes that a notice and a zero exit. The plugin schedules `bm:materials-reindex --if-configured` every night at 04:15 (`Schedule`, run by `serve`'s scheduler), because course renames, meeting renumbering, failed syncs and Typesense restarts can leave the index stale. Run it by hand after a Typesense restart.
+
 ### bm:seed

 ```sh
@@ -407,7 +436,7 @@ The file format (every top-level array is optional, and nullable fields may be l

 ## Dependencies

- SummerCMS framework modules: `pact`, `party`, `backpack`, `surf`, `wire`, `bouncer`, `lagoon` (with `lagoon/attach`), `cabana`, `bonfire`, `phrasebook`.
+- SummerCMS framework modules (v0.6.4): `pact`, `party`, `backpack`, `surf`, `wire`, `bouncer`, `lagoon` (with `lagoon/attach`), `cabana`, `bonfire`, `phrasebook`, `beachcomber` (with `beachcomber/typesense`).
 - The `golem15.user` plugin (`git.golem15.com/golem15/sm-user-plugin`): the `users` table and the `jwt.auth` middleware. `jz.bm` declares it in `Requires`, so its migrations run first.
 - GORM and gormigrate. Tests also use testcontainers-go and pgx.

--- a/controllers/api/manage.go
+++ b/controllers/api/manage.go
@@ -155,6 +155,9 @@ type managedMaterialJSON struct {
 	AuthorUserID *uint         `json:"author_user_id"`
 	Author       *userRefJSON  `json:"author"`
 	Shares       []userRefJSON `json:"shares"`
+	// Snippet is the matched passage of a search result, as on the
+	// student list; nil, and the key absent, without q.
+	Snippet any `json:"snippet,omitempty"`
 }

 // materialPeople are the authors and shares of a batch of materials, as
@@ -206,27 +209,58 @@ func newManagedMaterialJSON(m models.Material, people materialPeople) managedMat
 	return out
 }

-// ManagedMaterials answers GET /materials: every material, drafts
-// included, newest first. An MCP call gets the bodies only with
-// include_body (see managedMaterialView).
+// ManagedMaterials answers GET /materials?q=&include_body=: every material,
+// drafts and private ones included, newest first. With q (2 to 100
+// characters) the engine answers candidates, which are loaded in rank order
+// with at most one snippet each; when the engine cannot answer the request
+// is a 503. An MCP call gets the bodies only with include_body (see
+// managedMaterialView) and the snippets as plain text.
 func (h *Handlers) ManagedMaterials(w http.ResponseWriter, r *http.Request) {
+	in := query(r, "q", "include_body")
+	if !h.validate(w, r, in, manageMaterialsRules) {
+		return
+	}
+	var cands *classes.MaterialCandidates
+	if q := searchQuery(in); q != "" {
+		c, err := h.MaterialIndex.Search(r.Context(), classes.MaterialSearch{Q: q})
+		if err != nil {
+			writeSearchUnavailable(w)
+			return
+		}
+		cands = &c
+	}
 	db, ok := h.db(w, r)
 	if !ok {
 		return
 	}
 	var rows []models.Material
-	if err := preloadMaterial(db).Order("jz_bm_materials.id DESC").Find(&rows).Error; err != nil {
-		writeOpaque500(w)
-		return
+	if cands == nil || len(cands.IDs) > 0 {
+		q := preloadMaterial(db)
+		if cands != nil {
+			q = q.Where("jz_bm_materials.id IN ?", cands.IDs).Order(rankOrder(cands.IDs))
+		} else {
+			q = q.Order("jz_bm_materials.id DESC")
+		}
+		if err := q.Find(&rows).Error; err != nil {
+			writeOpaque500(w)
+			return
+		}
 	}
 	people, err := loadMaterialPeople(db, rows...)
 	if err != nil {
 		writeOpaque500(w)
 		return
 	}
+	mcp := viaMCP(r)
 	out := make([]any, 0, len(rows))
 	for _, m := range rows {
-		out = append(out, managedMaterialView(newManagedMaterialJSON(m, people), viaMCP(r), includeBody(r)))
+		row := newManagedMaterialJSON(m, people)
+		if cands != nil {
+			if hit, ok := cands.Hits[m.ID]; ok {
+				row.Snippet = snippetValue(pickSnippet(hit, m), mcp)
+			}
+		}
+		out = append(out, managedMaterialView(row, mcp, includeBody(r)))
 	}
 	writeData(w, out)
 }
--- a/controllers/api/material_search_test.go
+++ b/controllers/api/material_search_test.go
@@ -198,3 +198,43 @@ func TestMaterialsSearchTrace(t *testing.T) {
 		t.Errorf("total %d data %v, want the dry SQL's 0 (never the engine's found)", out.Meta.Total, out.Data)
 	}
 }
+
+func TestManagedMaterialsSearch(t *testing.T) {
+	h := noDB()
+	user := &bouncer.Principal{ID: 7}
+	code, body := serve(t, h.ManagedMaterials, http.MethodGet, "/?q=globalizacja", user, nil)
+	if code != http.StatusServiceUnavailable || body["error"] != true || body["message"] != "Search is unavailable." {
+		t.Errorf("q without an index = %d %v, want 503 Search is unavailable.", code, body)
+	}
+	code, body = serve(t, h.ManagedMaterials, http.MethodGet, "/?q="+strings.Repeat("a", 101), user, nil)
+	if code != http.StatusUnprocessableEntity {
+		t.Errorf("q over 100 characters = %d %v, want 422", code, body)
+	} else if errs, _ := body["errors"].(map[string]any); errs["q"] == nil {
+		t.Errorf("422 errors %v have no q message", errs)
+	}
+	code, body = serve(t, h.ManagedMaterials, http.MethodGet, "/?q=a&include_body=true", user, nil)
+	if code != http.StatusServiceUnavailable || body["message"] != "Database unavailable." {
+		t.Errorf("one-rune q = %d %v, want the no-database answer", code, body)
+	}
+}
+
+func TestMaterialSnippetMCP(t *testing.T) {
+	body := "Wstęp. Proces globalizacji zmienia państwa. Koniec."
+	m := models.Material{Body: &body}
+	mid := api.PickSnippet(beachcomber.Hit{Highlights: []beachcomber.Snippet{{Field: "body", ArrayIndex: -1, Segments: segments("Proces ", false, "globalizacji", true, " zmienia", false)}}}, m)
+	if got := api.FlattenSnippet(mid); got != "…Proces globalizacji zmienia…" {
+		t.Errorf("middle = %q, want ellipses on both sides", got)
+	}
+	head := api.PickSnippet(beachcomber.Hit{Highlights: []beachcomber.Snippet{{Field: "body", ArrayIndex: -1, Segments: segments("Wstęp", true, ". Proces", false)}}}, m)
+	if got := api.FlattenSnippet(head); got != "Wstęp. Proces…" {
+		t.Errorf("head = %q, want a trailing ellipsis only", got)
+	}
+	tail := api.PickSnippet(beachcomber.Hit{Highlights: []beachcomber.Snippet{{Field: "body", ArrayIndex: -1, Segments: segments("państwa. ", false, "Koniec", true, ".", false)}}}, m)
+	if got := api.FlattenSnippet(tail); got != "…państwa. Koniec." {
+		t.Errorf("tail = %q, want a leading ellipsis only", got)
+	}
+	whole := api.PickSnippet(beachcomber.Hit{Highlights: []beachcomber.Snippet{{Field: "body", ArrayIndex: -1, Segments: segments(body, false)}}}, m)
+	if got := api.FlattenSnippet(whole); got != body {
+		t.Errorf("whole = %q, want no ellipsis", got)
+	}
+}
--- a/controllers/api/params.go
+++ b/controllers/api/params.go
@@ -39,6 +39,10 @@ var (
 		{Field: "per_page", Rules: lagoon.ParseRules("nullable|integer|between:1,100")},
 		{Field: "q", Rules: lagoon.ParseRules("nullable|string|max:100")},
 	}
+	// manageMaterialsRules check the search of the management list.
+	manageMaterialsRules = []lagoon.RequestRule{
+		{Field: "q", Rules: lagoon.ParseRules("nullable|string|max:100")},
+	}
 )

 // query collects the named query parameters that are present into the input
--- a/mcp_manage.go
+++ b/mcp_manage.go
@@ -185,7 +185,8 @@ type updateDeadlineInput struct {

 // listManagedMaterialsInput is the input of bm_manage_list_materials.
 type listManagedMaterialsInput struct {
-	IncludeBody bool `json:"include_body,omitempty" query:"include_body" jsonschema:"true to include each material's body (it may hold transcript text)"`
+	Q           string `json:"q,omitempty" query:"q" jsonschema:"full-text search over the title, tags, source, course, link, body and the text of attached PDFs, 2 to 100 characters; results then come by relevance, each with a short plain-text snippet when the match is in the body or a file"`
+	IncludeBody bool   `json:"include_body,omitempty" query:"include_body" jsonschema:"true to include each material's body (it may hold transcript text)"`
 }

 // showManagedMaterialInput is the input of bm_manage_show_material.
@@ -327,7 +328,7 @@ func manageTools() []pact.MCPTool {
 			d, "/deadlines/{id}", idInput{}),

 		tool("manage_list_materials", "List materials (manage)",
-			"Lists every material, drafts included, newest first, with course, zjazd, type, source, link, publication time and tags. "+MCPMaterialPrivacy+readMaterials,
+			"Lists every material, drafts and private ones included, newest first, or by relevance with q, with course, zjazd, type, source, link, publication time and tags (and the snippet with q); 503 when search is unavailable. "+MCPMaterialPrivacy+readMaterials,
 			g, "/materials", listManagedMaterialsInput{}),
 		tool("manage_show_material", "Show a material (manage)",
 			"Returns one material, draft or not, with its public files (id, absolute link, name, size and type); 404 when unknown. "+MCPMaterialPrivacy+readMaterials,
--- a/mcp_student.go
+++ b/mcp_student.go
@@ -8,7 +8,7 @@ import (

 // MCPMaterialPrivacy is the sentence every material tool description
 // carries (D-17, D-21).
-const MCPMaterialPrivacy = "Returns metadata and links only, never file contents; a material's body (possible transcript text) is left out unless include_body is true."
+const MCPMaterialPrivacy = "Returns metadata and links, never whole file contents; with q a material may carry a short plain-text snippet of the matched passage, possibly from an attached file; a material's body (possible transcript text) is left out unless include_body is true."

 // The sentences every student tool description ends with (D-15).
 const (
@@ -47,6 +47,7 @@ type readingsInput struct {

 // materialsInput is the input of bm_materials.
 type materialsInput struct {
+	Q       string `json:"q,omitempty" query:"q" jsonschema:"full-text search over the title, tags, source, course, link, body and the text of attached PDFs, 2 to 100 characters; results then come by relevance, each with a short plain-text snippet when the match is in the body or a file"`
 	Course  uint   `json:"course,omitempty" query:"course" jsonschema:"only materials of this course id"`
 	Meeting uint   `json:"meeting,omitempty" query:"meeting" jsonschema:"only materials of this zjazd (meeting) id"`
 	Tag     string `json:"tag,omitempty" query:"tag" jsonschema:"only materials with this tag slug, at most 100 characters"`
@@ -175,7 +176,7 @@ func studentTools() []pact.MCPTool {
 		{
 			Name:        "materials",
 			Title:       "BM materials",
-			Description: "Lists a page of the published materials the student may see (public ones and private ones they wrote or that are shared with them), newest first, with title, type, course, zjazd, source, link, tags and file count, plus the page meta. " + MCPMaterialPrivacy + needsScopeRead,
+			Description: "Lists a page of the published materials the student may see (public ones and private ones they wrote or that are shared with them), newest first, or by relevance with q, with title, type, course, zjazd, source, link, tags and file count (and the snippet with q), plus the page meta. " + MCPMaterialPrivacy + needsScopeRead,
 			Guard:       "user",
 			Method:      http.MethodGet,
 			Path:        TokenPrefix + "/materials",
--- a/mcp_test.go
+++ b/mcp_test.go
@@ -120,14 +120,14 @@ var queryKeys = map[string][]string{
 	"manage_list_class_sessions": {"course_id", "meeting_id"},
 	"manage_list_readings":       {"course_id", "meeting_id", "is_required"},
 	"manage_list_deadlines":      {"course_id", "kind"},
-	"manage_list_materials":      {"include_body"},
+	"manage_list_materials":      {"q", "include_body"},
 	"manage_show_material":       {"include_body"},
 	"courses":                    {"semester"},
 	"schedule":                   {"from", "to"},
 	"deadlines":                  {"from", "to"},
 	"readings":                   {"course"},
 	"checklists":                 {"meeting"},
-	"materials":                  {"course", "meeting", "tag", "type", "page", "per_page"},
+	"materials":                  {"q", "course", "meeting", "tag", "type", "page", "per_page"},
 	"material":                   {"include_body"},
 }