feat: search the management list and the MCP material tools
- GET /api/v1/bm/materials takes q: candidates over every material, in rank order, with the snippet of the student list; 503 when the engine cannot answer - bm_materials and bm_manage_list_materials take q; under MCP the snippet is plain text with ellipses where text was cut - MCPMaterialPrivacy no longer claims no file text is ever returned - README documents q, the snippet, the 503, the Search section and bm:materials-reindex
This commit is contained in:
37
README.md
37
README.md
@@ -46,7 +46,7 @@ Every route except the calendar feed itself needs `Authorization: Bearer <golem1
|
||||
| DELETE | `/_bm/api/v1/readings/{id}/progress` | | Clears the mark (idempotent) |
|
||||
| PUT | `/_bm/api/v1/readings/{id}/owned` | | Marks the reading owned (idempotent) |
|
||||
| DELETE | `/_bm/api/v1/readings/{id}/owned` | | Clears the owned mark (idempotent) |
|
||||
| GET | `/_bm/api/v1/materials` | `course`, `meeting`, `tag` (slug), `type`, `page`, `per_page` (1–100) | One page of the published materials the caller may see |
|
||||
| GET | `/_bm/api/v1/materials` | `q` (search, 2–100 characters), `course`, `meeting`, `tag` (slug), `type`, `page`, `per_page` (1–100) | One page of the published materials the caller may see, by relevance with `q`, see [Search](#search) |
|
||||
| GET | `/_bm/api/v1/materials/{id}` | | One published material the caller may see, with body, tags, files and `can_manage` |
|
||||
| GET | `/_bm/api/v1/checklists` | `meeting` (ID) | The caller's checklists with progress, without items |
|
||||
| POST | `/_bm/api/v1/checklists` | | Creates a checklist for the caller; 201 |
|
||||
@@ -189,6 +189,15 @@ Newest publication first. `meeting` keeps the materials assigned to that zjazd (
|
||||
"can_manage": false}}
|
||||
```
|
||||
|
||||
`q` searches (see [Search](#search)): a trimmed `q` of fewer than 2 characters is ignored, one over 100 is a 422. With `q` the page is in relevance order, combined with the other filters (AND); `meta.total` is counted in SQL over the materials the caller may see, never taken from the engine. A row whose match is in the body carries one `snippet`:
|
||||
|
||||
```json
|
||||
{"snippet": {"source": "body", "file_name": null, "lead": true, "trail": true,
|
||||
"segments": [{"text": "proces ", "hit": false}, {"text": "globalizacji", "hit": true}, {"text": " zmienia państwa", "hit": false}]}}
|
||||
```
|
||||
|
||||
`segments` are the engine's plain text, verbatim, never markup: render each `text` as text and mark the `hit` ones. `lead` and `trail` say that text was cut before or after the excerpt. The key is absent when there is no snippet, and always without `q`, so the list without `q` is byte for byte as before. When the search engine is not configured or does not answer, a request with `q` is a 503 `{"error": true, "message": "Search is unavailable."}`; the list without `q` keeps working and never calls the engine.
|
||||
|
||||
### Private materials
|
||||
|
||||
A private material (`is_private`) is visible only to its author (`author_user_id`) and to the users it is shared with (`jz_bm_material_shares`); a public one to every student. The rule is one SQL fragment (`visibleMaterialSQL` in `controllers/api/material_access.go`) applied to every student read: `/materials`, `/materials/{id}`, the materials of `/courses/{id}`, the `materials` counters of `/courses` and `/courses/{id}`, the material linked from a checklist item, and through the token twins the student MCP tools. For anyone else a private material is a 404 and is not counted.
|
||||
@@ -243,7 +252,7 @@ The calendar is RFC 5545 (CRLF lines folded at 75 octets, escaped TEXT) with tim
|
||||
| Method | Route | Scope, permission | Answer |
|
||||
|--------|-------|-------------------|--------|
|
||||
| GET | `/api/v1/bm/me` | read | The token owner's `user_id` and the token (`id`, `name`, `scopes`, `expires_at`). |
|
||||
| GET | `/api/v1/bm/materials` | read, materials | Every material, drafts included, newest first. |
|
||||
| GET | `/api/v1/bm/materials` | read, materials | Every material, drafts and private ones included, newest first. `q` (2–100 characters) searches all of them, in relevance order, with the `snippet` of the student list; a 503 `Search is unavailable.` when the engine cannot answer. |
|
||||
| GET | `/api/v1/bm/materials/{id}` | read, materials | One material, draft or not, with its public `files`. 404 when unknown. |
|
||||
| POST | `/api/v1/bm/materials` | write, materials | Upsert on `title` + `course_id`: 201 `{"data", "created": true}` for a new material, 200 `{"data", "created": false}` for an updated one. |
|
||||
| PUT | `/api/v1/bm/materials/{id}` | write, materials | The same body applied to that material, so the title or course can change. 404 when unknown; 422 on `title` when another material already has the new title and course. |
|
||||
@@ -329,7 +338,8 @@ The file tools return metadata and links, never file contents, and their descrip
|
||||
**The MCP view.** Only an answer to an MCP tool call (a request `pier` dispatched in-process, carrying the `api_token` dispatch marker, which an HTTP request can never carry) is changed; plain HTTP answers are byte for byte as before.
|
||||
|
||||
- Links: the middleware `jz.bm.mcp-links` (on the token twins and the management API) prefixes every JSON string under a key named `url` or ending in `_url` that starts with a single `/` with `app.url`, else `jz.bm.public_url`, else the request's scheme and Host, so file and syllabus links are ready to follow.
|
||||
- Material bodies (possible transcript text): `bm_material`, `bm_manage_list_materials` and `bm_manage_show_material` leave `body` out unless `include_body=true`, and mark an omitted non-empty body with `body_omitted: true`; the material write answers never echo it. Material tools return metadata and links, never file contents.
|
||||
- Material bodies (possible transcript text): `bm_material`, `bm_manage_list_materials` and `bm_manage_show_material` leave `body` out unless `include_body=true`, and mark an omitted non-empty body with `body_omitted: true`; the material write answers never echo it. Material tools return metadata and links, never whole file contents.
|
||||
- Search: `bm_materials` and `bm_manage_list_materials` take `q` as the HTTP lists do. With `q` the results come by relevance and a material may carry `snippet` as a plain string, the matched passage with `…` where text was cut; it can come from the body or, once file text is indexed, from an attached file.
|
||||
- Throttling: `jz.bm.manage` keys an MCP call by its user (`jz.bm.manage.user|<id>`) before the bearer and IP keys.
|
||||
|
||||
## Student app
|
||||
@@ -360,8 +370,27 @@ Defaults in `config/config.yaml`, read as `jz.bm.<key>`:
|
||||
| `jz.bm.dashboard_limit` | `5` | Sessions and deadlines on `/dashboard`. |
|
||||
| `jz.bm.public_url` | `""` | Absolute base URL of the application (`https://bm.jakubzych.com`), used in the calendar feed URL and the course links inside the feed. Empty means the scheme and Host of the request; set it behind a TLS-terminating proxy. The application overrides it in `config/jz.bm.yaml`. |
|
||||
|
||||
## Search
|
||||
|
||||
Materials are searched through the framework's `beachcomber` module with the Typesense engine (`search.driver: typesense`; the blank import of `beachcomber/typesense` is in `search.go`). `models.Material` is a `beachcomber.Searchable` (`models/material_search.go`) kept in the index `jz_bm_materials` (after `search.prefix`). A document holds `id`, `title`, `source`, `url`, `tags` and `tag_slugs`, `course` (name and code), `course_id`, `meeting_id`, `meeting_number`, `type`, `published_at` (Unix seconds, 0 for a draft), the body with its Markdown syntax removed and `file_texts` (the text of attached files). No field sets a locale, so the default one folds diacritics and `lodz` finds `Łódź`.
|
||||
|
||||
The index holds every material, drafts and private ones included, and no visibility: the engine's ids are candidates only. A search asks for at most 250 of them (title weighted over tags, source and course, then body, link and file text), and SQL re-gates them with the same visibility rule and filters as the list, counts them and pages over them in the engine's order. A snippet is read only for a row SQL returned, so a material the caller may not see never reaches an answer, its count or a snippet.
|
||||
|
||||
The beachcomber service is app-wide, and so is its Gate: jz.bm installs one that is on only for contexts it marks (`classes.MarkMaterialSearch`), so `golem15.chatchannels` messages and DMs are never indexed. Every material write syncs explicitly after it commits: the admin form (create, update, delete), `POST`, `PUT` and `DELETE /api/v1/bm/materials`. A failed sync is logged with the material id and never fails the write.
|
||||
|
||||
Configuration, in the app's `config/search.yaml`: `search.driver`, `search.prefix`, `search.typesense.host`, `port`, `protocol` and `connection_timeout_seconds`. The API key goes only in the environment, `SUMMER_SEARCH__TYPESENSE__API_KEY`; with no key the engine is not configured, nothing is sent and every search answers 503.
|
||||
|
||||
## CLI commands
|
||||
|
||||
### bm:materials-reindex
|
||||
|
||||
```sh
|
||||
./bin/bm bm:materials-reindex
|
||||
./bin/bm bm:materials-reindex --fresh
|
||||
```
|
||||
|
||||
Ensures the `jz_bm_materials` collection and syncs every material through the same path as the sync after a save, ending with `Reindexed: N synced, M failed`; it exits non-zero when any material failed. `--fresh` first drops the collection and recreates it from `models.Material.SearchIndexSchema`: use it on the first deploy and after a schema change. Without an engine the command refuses with an error naming `search.driver` and `search.typesense.api_key`; `--if-configured` makes that a notice and a zero exit. The plugin schedules `bm:materials-reindex --if-configured` every night at 04:15 (`Schedule`, run by `serve`'s scheduler), because course renames, meeting renumbering, failed syncs and Typesense restarts can leave the index stale. Run it by hand after a Typesense restart.
|
||||
|
||||
### bm:seed
|
||||
|
||||
```sh
|
||||
@@ -407,7 +436,7 @@ The file format (every top-level array is optional, and nullable fields may be l
|
||||
|
||||
## Dependencies
|
||||
|
||||
- SummerCMS framework modules: `pact`, `party`, `backpack`, `surf`, `wire`, `bouncer`, `lagoon` (with `lagoon/attach`), `cabana`, `bonfire`, `phrasebook`.
|
||||
- SummerCMS framework modules (v0.6.4): `pact`, `party`, `backpack`, `surf`, `wire`, `bouncer`, `lagoon` (with `lagoon/attach`), `cabana`, `bonfire`, `phrasebook`, `beachcomber` (with `beachcomber/typesense`).
|
||||
- The `golem15.user` plugin (`git.golem15.com/golem15/sm-user-plugin`): the `users` table and the `jwt.auth` middleware. `jz.bm` declares it in `Requires`, so its migrations run first.
|
||||
- GORM and gormigrate. Tests also use testcontainers-go and pgx.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user