golem15/sm-journal-plugin
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: relevance-ordered search, suggest rows and journal:reindex

Browse Source 
- Engine search asks Typesense for the first 250 candidates, re-gates them
  in SQL, counts the gated rows and pages over them in rank order; pages
  after the first are no longer empty. An explicit sort overrides the rank.
- The settings row gates engine search, so it works straight after boot.
- ?fields=suggest serves id, title, slug, a plain-text excerpt,
  featured_image_url and published_at.
- Content is indexed as plain text; Post declares SearchIndexSchema.
- journal:reindex [--fresh] syncs every post into the index.
This commit is contained in:
Jakub Zych
2026-10-30 17:59:57 +01:00
parent 5b38a3b20d
commit 08e3d82ecd
9 changed files with 768 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
README.md
View File

@@ -22,6 +22,7 @@ WinterCMS theme components such as `journalPost` and `journalPosts` are a later
- **Categories and tags.** `models.Category` is translatable (name, description, slug). It also stores `content`, an HTML host column that is not translatable, edited in the admin with the `blogeditor` field type. Fillable is name, slug, code, description, content and parent_id; nest_* columns exist in DDL and are not fillable. `models.Tag` is not translatable. Fillable is name, slug, description. Pivots are `golem15_journal_posts_categories` and `golem15_journal_posts_tags`.
- **Menu and sitemap item types.** The plugin registers its five PHP menu item types (`journal-category`, `all-journal-categories`, `journal-post`, `all-journal-posts`, `category-journal-posts`) and `journal-tag` with the framework's menu item type registry, so menus and sitemaps can point at posts, categories and tags. Posts, categories and tags carry content descriptors, so plugins that work on every content type (sitemaps, SEO, search) describe, page and match Journal records without the Journal importing them. See [menu item types](#menu-item-types) and [content descriptors](#content-descriptors).
- **Settings.** Singleton row ID=1 on `golem15_journal_settings`, not PHP `system_settings`. `search_use_typesense` defaults to false. A fresh save never contacts Typesense.
- **Search.** With a configured search engine and `search_use_typesense` on, published posts are kept in the Typesense collection `golem15_journal_posts` after every committed save, and `GET /posts?search=` answers in relevance order. Otherwise search is a SQL `ILIKE`. `?fields=suggest` serves short rows for a search box, and `journal:reindex` fills the index. See [search](#search).
- **Public HTTP.** Anonymous GET `/_journal/api/v1/posts`, `/posts/{slug}`, `/categories`, `/tags`, `/rss`, `/authors/{author}`, and `/authors/{author}/posts`. `/posts/{slug}` finds the post by its default-locale slug, then by its slug in the request locale (the router's locale, from `Accept-Language` or the translate plugin's resolver), then by id, so a translated slug resolves. Writes under the same prefix require a cabana backend JWT (`aud=backend`) or a personal admin API token (`g15_…`) whose scope covers the Journal.
- **Record events.** Post writes through the admin fire the framework's record events (`pact.RecordSaved`, `pact.RecordDeleted`) after the commit, like every admin controller. The `/_journal/api/v1` post routes, which MCP tools call, fire the same events with `cabana.EmitRecordSaved` and `cabana.EmitRecordDeleted`: `POST /posts` a RecordSaved with `Created`, `PUT /posts/{id}` a RecordSaved whose changes hold the post's changed columns and each submitted translation that changed (a translated slug per locale), `DELETE /posts/{id}` a RecordDeleted. A tag `POST` or `PUT` creates from a name in `tags` fires its own RecordSaved with `Created`. `POST` and `PUT` write the row, its categories, tags and translations in one transaction and fire the event after it commits, so a write that fails part-way changes nothing and fires nothing. Their `Controller` is empty, as they do not go through an admin form. Another plugin (an SEO plugin creating redirects, a search indexer) therefore hears about a slug changed over MCP without importing the Journal.
- **MCP tools.** `journal.Plugin.MCP` declares seven tools and one prompt for the framework's MCP endpoint (pier, a binary built with the `summercms_mcp` tag): `journal_list_posts`, `journal_show_post`, `journal_list_categories`, `journal_list_tags`, `journal_create_post`, `journal_update_post`, `journal_delete_post` and the prompt `journal_draft_post`. Each runs the matching `/_journal/api/v1` route as the backend account linked to the MCP user (guard `backend`) and is listed only when that account has `golem15.journal.access_posts`; the route checks its own permissions on every call. See [MCP tools](#mcp-tools).
@@ -142,6 +143,34 @@ Content-Type: application/json
{"title":"Hello","content":"A blog post"}
```
### Search
`GET /_journal/api/v1/posts?search=<term>` filters the list by a term of 3 or more characters; a shorter term is ignored and the list comes back unfiltered.
- **SQL.** By default, and whenever the engine cannot be used, the term is matched with `ILIKE` against title, slug, excerpt and content, and the rows keep the list order (`sort`, default newest first).
- **Typesense.** When `search.driver` is `typesense` with an API key and the Journal setting `search_use_typesense` is on, the engine is asked for the first 250 candidates (`typesense.MaxPerPage`), ranked by text match with the setting weights for title, excerpt and content (default 5, 3, 1), then by date. The candidates are re-gated in SQL with the same filters as any list (published, `category`, `tag`, `author`), so a stale index entry never shows. `meta.total` counts the gated rows, the rows keep the engine's rank, with pinned posts first when `pinned_first` is set, and `page` and `per_page` page over them as over any list. An explicit `sort` overrides the rank. A search that matches more than 250 posts keeps the first 250. When the engine fails, the request is answered from SQL and the failure is logged at Warn.
The setting row is read on every engine search, so turning `search_use_typesense` off takes effect on the next request, without a restart.
`?fields=suggest` replaces each row with the short shape a search box needs, under the same pagination envelope, filters and limits:
| Key | Value |
|-----|-------|
| `id` | Post id. |
| `title` | Default-locale title, as in the full row. |
| `slug` | Default-locale slug, as in the full row. |
| `excerpt` | The excerpt as plain text, cut to about 160 characters on a word boundary with `…`; the content as plain text when the post has no excerpt. |
| `featured_image_url` | The first featured image URL, or `null`. |
| `published_at` | As in the full row. |
Any other `fields` value, or none, serves the full rows.
```http
GET /_journal/api/v1/posts?search=borowik&fields=suggest&per_page=5
```
The search document holds the default-locale title, excerpt and content; content is indexed as plain text, from `content_html` (or `content` when nothing was rendered) with tags removed and entities decoded. The collection is created from `models.Post.SearchIndexSchema` when it does not exist. A collection created before the schema existed keeps its auto-detected fields until `journal:reindex --fresh` recreates it.
### Personal API tokens
The plugin registers two scopes through `journal.Plugin.TokenScopes` (`pact.HasTokenScopes`):
@@ -333,7 +362,7 @@ The request is a `POST` to the URL with `Authorization: Bearer <token>`, `Conten
| `journal.Plugin.LangFS` | Embedded English and Polish UI phrasebooks. |
| `journal.Plugin.Migrations` | gormigrate steps registered by this plugin. |
| `journal.Plugin.Models` | Models registered by this plugin. |
| `journal.Plugin.Commands` | Registers `journal:export-posts` and `journal:import-posts`. |
| `journal.Plugin.Commands` | Registers `journal:export-posts`, `journal:import-posts` and `journal:reindex`. |
| `journal.Plugin.Routes` | Public GETs (posts, categories, tags, RSS, authors) and backend-Bearer writes under `/_journal/api/v1`. |
| `journal.Plugin.MCP` | The MCP tools and the `draft_post` prompt (`pact.HasMCP`), guard `backend`, permission `golem15.journal.access_posts`. |
| `journal.Plugin.TokenScopes` | Personal API token scopes `journal` and `journal:read` (`pact.HasTokenScopes`). |
@@ -346,7 +375,9 @@ The request is a `POST` to the URL with `Authorization: Bearer <token>`, `Conten
| `models.Post.AttachRelations` | `featured_images` and `content_images` attachMany. |
| `models.Post.SearchableAs` | Index name `golem15_journal_posts`. |
| `models.Post.ShouldBeSearchable` | False when unpublished or `search_use_typesense` is off. |
| `models.Post.ToSearchableArray` | Typesense document built after a committed save. |
| `models.Post.ToSearchableArray` | Typesense document built after a committed save; content is plain text. |
| `models.Post.SearchIndexSchema` | The Typesense collection schema a missing collection is created with (`beachcomber.IndexSchemaProvider`). |
| `models.PlainText` | HTML to plain text: tags dropped, entities decoded, whitespace collapsed. |
| `models.Post.CanEdit` | Owner or `golem15.journal.access_other_posts`. |
| `models.Category` | Category row; `TableName` is `golem15_journal_categories`; `Content` holds the HTML body served as `content`. |
| `models.Category.MorphName` | PHP class string `Golem15\Journal\Models\Category`. |
@@ -383,17 +414,19 @@ The request is a `POST` to the URL with `Authorization: Bearer <token>`, `Conten
## CLI commands
Both commands write JSON files (one per post slug), matching the PHP `journal:export-posts` / `journal:import-posts` tools. They are not a CSV ImportExport framework. A blog application binary registers them through `Plugin.Commands`.
The export and import commands write JSON files (one per post slug), matching the PHP `journal:export-posts` / `journal:import-posts` tools. They are not a CSV ImportExport framework. A blog application binary registers them through `Plugin.Commands`.
| Command | Arguments and flags | Effect |
|---------|---------------------|--------|
| `journal:export-posts` | `--path` (default `journal-export`), `--dry-run` | Writes one JSON file per post, including non-default-locale translations, with a progress bar over the posts. Attachments are omitted. Ends with `Exported N posts to: <path>`. |
| `journal:import-posts` | `--path` (default `journal-export`), `--force`, `--dry-run` | Creates posts from those files, one line per post. Existing slugs are skipped unless `--force` updates them. Ends with `Results: N created, N updated, N skipped, N failed`. |
| `journal:reindex` | `--fresh` | Syncs every post into the search index in batches of 200: published posts are upserted, the others removed. `--fresh` first drops the collection and recreates it from the schema. Refuses, sending nothing, when the engine is not configured or `search_use_typesense` is off. Ends with `Reindexed: N synced, N removed, N failed` and exits non-zero on any failure. |
```sh
./bin/acme journal:export-posts --path journal-export
./bin/acme journal:import-posts --path journal-export --dry-run
./bin/acme journal:import-posts --path journal-export --force
./bin/acme journal:reindex --fresh
```
`./bin/acme help journal:import-posts` shows each command's long help and examples. The plugin logs through the component `journal` (`log.components.journal` in `config/log.yaml`).