docs(14): capture phase context

.planning/phases/14-domain-jobs-and-external-integrations/14-CONTEXT.md

@@ -0,0 +1,150 @@
+# Phase 14: Domain jobs and external integrations - Context
+
+**Gathered:** 2026-10-03
+**Status:** Ready for planning
+
+<domain>
+## Phase Boundary
+
+Phase 14 fills in the Płytarium work that talks to the outside world or runs in the background, on top of the Phase 11 jobs/realtime/search infrastructure and the Phase 13 API surface:
+
+- **Job workers** for the kinds Phase 13 already enqueues: CSV import write, CSV/Discogs match (240 s timeout, re-enqueues itself with a delay on a Discogs rate-limit error), and wishlist digest (30-minute coalescing window, digest mail, queue-row delete).
+- **Discogs client** (proactive rate threshold, bounded in-request wait budget, retry-after fallback, host-locked cover fetch) and every route that uses it.
+- **AI cover recognition** through a new shared `sm-golem-plugin` (Anthropic and OpenAI-compatible adapters, per-credential model/base-URL overrides, backend global vision model).
+- **Commands:** `fonoteka:prune-notifications` and `fonoteka:reindex` (asserts zero `collection_id`-0 documents before and after, flag to drop the legacy index). `fonoteka:oauth-client` already shipped in Phase 8.
+- **Feedback plugin**, ported in full to a new shared `sm-feedback-plugin`.
+- **Framework helpers** folded in from todos: a guarded outbound `http.Client` in fetchguard and a slog handler that redacts credentials.
+
+Routes that leave `pending` in this phase (12): `wishlist/albums/{id}/match`, `wishlist/albums/{id}/apply-release`, `ai-credential/test`, `discogs-credential/test`, `albums/match`, `albums/{id}/match`, `albums/{id}/apply-release`, `albums/recognize` (JWT group), `albums/import/discogs`, `/api/v1/fonoteka/albums/recognize` (token group, `inv.scope:ai`), `/api/v1/fonoteka/albums/{id}/cover-price/discogs` (token group, `inv.scope:write`, `throttle:12,1`), plus the `selected_discogs_id` success case of the CSV row-edit route. The feedback routes are new to the manifest.
+
+Out of scope: sitemap (dropped for Płytarium, D-14), `oauth-identities` GET/DELETE and `GET /api/v1/fonoteka/me` (D-09), the Golem face/CompreFace services (D-04).
+
+</domain>
+
+<decisions>
+## Implementation Decisions
+
+### AI layer (sm-golem-plugin)
+- **D-01:** The Go port of PHP `Golem15.Golem` lives in its own shared core-plugin repo, `git@git.golem15.com:golem15/sm-golem-plugin.git` (created empty by the user). Module `git.golem15.com/golem15/sm-golem-plugin`, mounted in fonoteka.go at `plugins/golem15/golem` as a git submodule, following `.planning/notes/core-plugins-own-repos.md`. The plugin ID stays `golem15.golem`, with the same settings key (`golem15_golem_settings`) and lang namespace, so migrated PHP data loads as-is. — **Reversibility:** costly — the plugin ID, settings key and module path are what every consuming application and the migrated settings row use.
+- **D-02:** The provider adapters (Anthropic Messages, OpenAI-compatible chat completions) are hand-rolled JSON mappings over the guarded outbound client (folded todo fetchguard-guarded-http-client). No Anthropic or OpenAI SDK dependency. PHP's adapters are raw curl, which makes request parity easier.
+- **D-03:** Port the Golem Settings model in full: the `models` repeater (name, adapter openai|anthropic, encrypted `api_key`, `base_url`, `model`, `system_prompt`, `is_enabled`, `is_default`, `generates_images`, `accepts_images`, `has_files_endpoint`, `max_completion_tokens`) plus its admin settings screen, reading the existing settings row. `getVisionModel()` fills fonoteka's `classes.AdminVisionModel` seam (Phase 13), so the admin tier of `ResolveAIConfig` and the site-admin branch of `AIAllowed` light up.
+- **D-04:** Port all of AIService except faces: `send`, `sendStream`, `sendFile`/`sendFilePath` (files endpoint), `sendWithModel`, `sendToImageModel`, `sendToVisionModel`, `generateImage`, `ask`, the model system-prompt application, `Prompt`/`AIResponse` value objects and `PromptFactory`. CompreFace/FaceService and ChatContextCollector are not ported. The SSRFGuard is ported as part of the guarded-client work.
+- **D-05:** Base-URL guard: user and org credential `base_url` overrides always go through the private/reserved-IP dial guard. Admin-configured Settings models are trusted (they may point at a LAN/local endpoint such as Ollama). The researcher confirms this split against PHP's `SSRFGuard` and reports any difference before planning.
+- **D-06:** REQUIREMENTS INTG-02 ("Anthropic Go SDK") is reworded at plan time to "Anthropic and OpenAI-compatible adapters over the guarded client". PROJECT.md's repository paragraph gains sm-golem-plugin and sm-feedback-plugin, and drops feedback and sitemap from the application-plugin list.
+
+### Route scope
+- **D-07:** All 6 album Discogs/AI routes that Phase 12 D-04 moved here are in scope (`albums/match`, `albums/{id}/match`, `albums/{id}/apply-release`, `albums/recognize` on the JWT and token groups, `albums/import/discogs`), together with the routes the criteria already name. ROADMAP SC4/SC5 are reworded at plan time to list them. They share ReleaseMatchScorer, AlbumReleaseApplicator, DiscogsImportResolver, DiscogsMapper and AlbumRecognitionService with the named routes.
+- **D-08:** The Phase 13 `ReleaseFetcher` seam behind the CSV row-edit `selected_discogs_id` pick gets the real Discogs `getRelease` + `DiscogsMapper::mapRelease` implementation, so a pick resolves exactly as in PHP.
+- **D-09:** `GET/DELETE oauth-identities` (social login, deferred since Phase 7) and `GET /api/v1/fonoteka/me` (the minimal endpoint from Phase 8 D-20) are not part of Phase 14. They stay `pending` and are flagged for the roadmap (see Deferred).
+- **D-10:** WR-02 from the Phase 13 review is fixed before the CSV workers go live: the `mapping`, row-edit and `cancel` writes lock the import row or compare-and-swap on its status, as `commit` already does, so none of them can race a commit and queue a second import job. Response shapes stay the same. This is a deliberate deviation from PHP, which has the same race.
+- **D-11:** The researcher checks `routes.php` and the Nuxt/MCP callers for any recorded case that is still missing, for example a successful `selected_discogs_id` pick, the cover-price route, and the recognize success and truncation paths. It records those against the PHP backend, together with their upstream exchanges (D-15).
+
+### Feedback plugin (sm-feedback-plugin)
+- **D-12:** Feedback is a shared core plugin in its own repo, `git@git.golem15.com:golem15/sm-feedback-plugin.git` (created by the user). Module `git.golem15.com/golem15/sm-feedback-plugin`, mounted in fonoteka.go at `plugins/golem15/feedback`, plugin ID `golem15.feedback`, with the same tables (`feedback_submissions`, user preferences) and the same settings key. — **Reversibility:** costly — the plugin ID, tables and settings key become the contract other applications mount.
+- **D-13:** Full port. It covers `GET /_feedback/api/v1/{key}/config` (throttle 60/min/IP), `POST {key}/submit` (multipart, 10/min/IP, Origin allowlist gate, ImageContentGuard), the `OPTIONS {any}` 204 preflight, and JWT `PUT me/hidden`. `embed.js` is served at the same `/plugins/golem15/feedback/assets/js/embed.js` path that the Nuxt proxy expects. The plugin also ports the Settings model (enabled, widget_key, allow_hide, position, allowed_origins, colors, pl/en labels) with its admin screen, the submissions admin list, and the `golem15.user.getApiArray` hook that adds `feedback_widget_hidden` to the user payload. The user plugin already references this field. `SyncFeedbackToG15Office` becomes a River job on the guarded client (JSON + multipart). Feedback routes are recorded as new parity fixtures. API-08 is reworded to feedback only.
+- **D-14:** Sitemap is dropped for Płytarium. Nuxt builds its own sitemap with `@nuxtjs/seo`, and fonoteka registers no menu item types. A todo records a 1:1 sitemap plugin port for the next blog project (grzybyfunkcjonalne.pl or golem15.com).
+
+### Vendor testing
+- **D-15:** Upstream HTTP in tests: the upstream Discogs/Anthropic/OpenAI/G15Office exchanges are recorded once as a sidecar to each parity case and served from an `httptest` fake that the guarded client points at during replay. Replay is deterministic and offline. The fake also asserts the request Go sends upstream (method, path, headers including User-Agent and auth, body) against PHP's recorded request. CI makes no live vendor calls, and G15Office is verified only through the fake.
+- **D-16:** Time is injected. The Discogs rate limiter, its wait budget and retry-after handling, and the match job's re-enqueue all take a clock/sleeper interface. Tests advance a fake clock without real sleeps, and assert the 240 s timeout and the re-enqueue delay as values.
+- **D-17:** The researcher reads `DiscogsRateLimiter.php` and decides where its state lives, either process memory or Postgres (an UNLOGGED table or an advisory lock), depending on whether PHP's cross-worker guarantee is still needed when one binary runs HTTP and River workers. The choice and its reason go in RESEARCH.md.
+
+### Claude's Discretion
+- Wishlist digest mail content, locale and template: a straight port of `WishlistDigestJob` and its PHP mail view on the existing postcard mail pipeline.
+- `prune-notifications` and `reindex` options, output and exit codes: a straight port of `PruneNotifications.php` and `ReindexAlbums.php`, on bonfire with the existing `schedule.go` entry.
+- Exact package layout inside sm-golem-plugin and sm-feedback-plugin, which follows `.planning/notes/plugin-layout-winter-directories.md`.
+- Plan split, subject to the lean-mode plan-count checkpoint.
+
+### Folded Todos
+- **fetchguard-guarded-http-client** (`.planning/todos/pending/fetchguard-guarded-http-client.md`, high): extend fetchguard from a guarded HTTPS GET into a guarded outbound `http.Client`/constructor (POST, PUT, multipart upload, bearer auth), keeping the dial-time private/reserved-IP rejection, with an explicit trusted mode for admin-configured endpoints (D-05). It replaces Apparatus `RequestSender`. Its consumers are the Discogs client, the Golem adapters and feedback's G15OfficeClient. Framework change in summercms.go: update `modules/fetchguard/README.md` and `docs/`.
+- **redacting-slog-handler** (`.planning/todos/pending/redacting-slog-handler.md`, medium): a framework `slog.Handler` wrapper porting Apparatus `RedactCredentialsTap`. It redacts the keys api_key, apikey, authorization, bearer, password, secret, token, webhook_secret and admin_password (case-insensitive, nested groups) and scrubs messages with the PHP regex patterns. It also checks whether surf's error path already gives `SafeExceptionResponse` behaviour. This is the first phase to send user credentials to outside services.
+
+</decisions>
+
+<canonical_refs>
+## Canonical References
+
+**Downstream agents MUST read these before planning or implementing.**
+
+### Planning
+- `.planning/ROADMAP.md` § Phase 14 — goal and SC1–SC6 (SC4/SC5/SC6 reworded per D-06, D-07, D-14)
+- `.planning/REQUIREMENTS.md` — JOBS-02, JOBS-03, SRCH-02, INTG-01, INTG-02 (reword per D-06), API-08 (reword per D-13/D-14), CLI-05
+- `.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-CONTEXT.md` — D-01..D-08: job kinds, args and queues; seams; route deferrals
+- `.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-VERIFICATION.md` — WR-02 and the Phase 14 hand-off table
+- `.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-REVIEW.md` — WR-02 detail
+- `.planning/phases/12-p-ytarium-api-collections-and-albums/12-CONTEXT.md` — D-04/D-05: album Discogs routes deferred, CoverImporter on fetchguard
+- `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md` — conga/River, scheduling, search (reindex) decisions
+- `.planning/notes/core-plugins-own-repos.md` — sm-*-plugin repo, submodule and module-path workflow (D-01, D-12)
+- `.planning/notes/apparatus-dissolved-into-framework.md` — RequestSender and RedactCredentialsTap porting targets
+- `.planning/notes/plugin-layout-winter-directories.md` — plugin directory layout
+- `.planning/todos/pending/fetchguard-guarded-http-client.md`, `.planning/todos/pending/redacting-slog-handler.md` — folded todos
+
+### PHP reference (Płytarium, `/media/nvme/dev/golem15/fonoteka`)
+- `plugins/golem15/fonoteka/routes.php` lines 207-246, 443-505 — the routes in scope and their groups, scopes and throttles
+- `plugins/golem15/fonoteka/jobs/{AlbumCsvImportJob,AlbumCsvMatchJob,WishlistDigestJob}.php`
+- `plugins/golem15/fonoteka/console/{PruneNotifications,ReindexAlbums}.php`
+- `plugins/golem15/fonoteka/classes/discogs/*` — DiscogsClient, DiscogsRateLimiter, AlbumCoverFetcher, AlbumReleaseApplicator, DiscogsImportResolver, DiscogsInputParser, DiscogsMapper, PriceSuggestionResolver, ReleaseMatchScorer, exceptions
+- `plugins/golem15/fonoteka/classes/{AlbumRecognitionService,RecognitionTruncatedException,AiDefaults,AiGate,DiscogsGate}.php`
+- `plugins/golem15/fonoteka/controllers/api/{AlbumReleaseMatchController,WishlistReleaseMatchController,RecognizeApiController,DiscogsImportController,AlbumCoverFetchController}.php` and the credential `test` actions
+- `plugins/golem15/golem/` — `classes/services/AIService.php`, `classes/providers/*`, `classes/security/SSRFGuard.php`, `classes/valueobjects/*`, `classes/factories/PromptFactory.php`, `models/Settings.php`, `models/settings/fields.yaml`
+- `plugins/golem15/feedback/` — `routes.php`, `controllers/api/*`, `jobs/SyncFeedbackToG15Office.php`, `classes/{G15OfficeClient,ImageContentGuard}.php`, `models/*`, `updates/*`, `assets/js/embed.js`, `Plugin.php` (getApiArray hook, settings)
+- `plugins/golem15/sitemap/` — reference only for the deferred todo (D-14)
+- `vue-fonoteka-app/nuxt.config.ts` (feedback proxy paths), `vue-fonoteka-app/app/stores/auth.ts` (`me/hidden`), `fonoteka-mcp` (recognize and cover-price callers)
+
+### Go code
+- `../fonoteka.go/parity/manifest.yaml` — pending entries to flip
+- `../fonoteka.go/plugins/golem15/fonoteka/{jobs.go,schedule.go,search.go,routes.go}`
+- `../fonoteka.go/plugins/golem15/fonoteka/classes/{csv_import_service.go,job_contract.go,ai_config_resolver.go,gates.go,cover_importer.go,wishlist_notifications.go}`
+- `modules/fetchguard`, `modules/conga`, `modules/postcard`, `modules/bonfire` — framework modules this phase builds on
+
+</canonical_refs>
+
+<code_context>
+## Existing Code Insights
+
+### Reusable Assets
+- `classes/job_contract.go` plus the Phase 13 enqueue sites (`csv_import_service.go` dispatches `CsvMatchArgs`/`CsvImportArgs`, and `wishlist_notifications.go` dispatches `WishlistDigestArgs` at 1800 s). The workers register against these exact kinds and args.
+- `jobs.go` registers the invitation and purchase-mail workers. New workers follow the same pattern, and `job_contract_worker_test.go` is where they are tested.
+- `classes.ReleaseFetcher` seam (Phase 13 D-05) and `classes.AdminVisionModel` seam (Phase 13): replace their defaults.
+- `ResolveAIConfig` / `ResolveDiscogsConfig` / `AIAllowed` are already ported tier by tier.
+- `cover_importer.go` and `image_guard.go` already use fetchguard's `AllowHosts` GET, which the host-locked cover fetch reuses.
+- `schedule.go` already schedules `fonoteka:prune-notifications`, so only the command is missing.
+- `console/oauth_client.go` is the bonfire command pattern for the new commands.
+- `search.go` holds the Typesense collection setup that the reindex command builds on.
+
+### Established Patterns
+- Pending manifest routes are absent from the router: no 501 shells (P6 D-15). `TestRouteTablePhase13` asserts that the Phase 14 routes are absent, so update it when they land.
+- Secrets are `lagoon.Encrypted` with `json:"-"`.
+- Core plugins are developed inside their submodule checkout: commit and push there first, then bump the pointer in fonoteka.go as a separate commit.
+- Framework README/docs rules from CLAUDE.md apply to fetchguard and the new slog handler.
+
+### Integration Points
+- fonoteka.go `go.work`, `go.mod` (require + local replace), `summer.yaml` and `plugins.gen.go` for the two new submodule plugins.
+- The fonoteka plugin requires `golem15.golem` for recognition. Feedback hooks into the user plugin's `getApiArray` event.
+
+</code_context>
+
+<specifics>
+## Specific Ideas
+
+- The user created both repos up front: `sm-golem-plugin` (the user first created it as `sm-ai-plugin` and then renamed it so it matches the PHP plugin name) and `sm-feedback-plugin`.
+- After Płytarium, a blog project (grzybyfunkcjonalne.pl or golem15.com) is ported next and needs the sitemap plugin 1:1.
+
+</specifics>
+
+<deferred>
+## Deferred Ideas
+
+- **Sitemap plugin 1:1 port** for the next blog project: a new todo is written alongside this context.
+- **`oauth-identities` GET/DELETE (social login) and `GET /api/v1/fonoteka/me`**: the routes stay `pending` without a phase. The roadmap needs a home for them before Phase 15, whose criterion is all routes green. A new todo is written.
+- Golem face detection (CompreFace/FaceService) and ChatContextCollector: not ported until a consumer needs them.
+
+### Reviewed Todos (not folded)
+- `backend-admin-api-tokens`, `bonfire-duplicate-command-names`, `lagoon-readme-after-commit-callback-order`, `nest-framework-packages-under-modules`, `per-module-readmes-after-nest`, `readme-go-fences-src`, `refresh-fonoteka-readme`, `rewrite-summercms-readme`, `scaffold-*` (3), `wristband-neutral-resource-default`, `2026-10-01-benchmark-…`: these matched on keywords only and are unrelated to the phase domain.
+
+</deferred>
+
+---
+
+*Phase: 14-domain-jobs-and-external-integrations*
+*Context gathered: 2026-10-03*

.planning/phases/14-domain-jobs-and-external-integrations/14-DISCUSSION-LOG.md

@@ -0,0 +1,59 @@
+# Phase 14: Domain jobs and external integrations - Discussion Log
+
+> **Audit trail only.** Do not use as input to planning, research, or execution agents.
+> Decisions are captured in CONTEXT.md — this log preserves the alternatives considered.
+
+**Date:** 2026-10-03
+**Phase:** 14-domain-jobs-and-external-integrations
+**Areas discussed:** AI layer home + SDKs, Route scope, Feedback + sitemap, Vendor testing
+**Todos folded:** fetchguard-guarded-http-client, redacting-slog-handler
+
+---
+
+## AI layer home + SDKs
+
+| Question | Options | Selected |
+|---|---|---|
+| Where the Golem AI port lives | sm-golem-plugin repo / framework module / inside fonoteka | sm plugin repo. The user created `sm-ai-plugin`, then renamed it to `sm-golem-plugin` |
+| Plugin ID | keep golem15.golem / rename golem15.ai | Keep `golem15.golem`, given the repo rename to sm-golem-plugin |
+| Adapter transport | hand-rolled on guarded client / Anthropic SDK + plain OpenAI / both SDKs | Hand-rolled on guarded client |
+| Global vision model settings | models repeater full / model only / config key | Models repeater, full, with admin screen |
+| AIService surface | only what recognition needs / full minus faces | Full AIService minus faces |
+| INTG-02 wording | reword at plan time / leave | Reword at plan time |
+| base_url SSRF guard | guard user/org, trust admin / guard everything / mirror PHP | Guard user/org, trust admin |
+
+## Route scope
+
+| Question | Options | Selected |
+|---|---|---|
+| 6 album Discogs/AI routes from Phase 12 | all in, reword SC4/SC5 / named routes only | All in |
+| oauth-identities and /api/v1/fonoteka/me | out, flag for roadmap / fold in | Out; flag for roadmap |
+| WR-02 CSV race | row lock/CAS / idempotent worker / both | Fix with row lock/CAS |
+| Missing recordings | researcher checks / existing fixtures only | Researcher checks |
+
+## Feedback + sitemap
+
+| Question | Options | Selected |
+|---|---|---|
+| Feedback repo | new sm-feedback-plugin / fonoteka.go app plugin | sm-feedback-plugin (repo created by the user) |
+| Feedback depth | full / API + sync, no admin | Full |
+| Sitemap | faithful minimal / full with admin editor / drop for Płytarium | Drop for Płytarium, and add a todo: the next port is a blog project (grzybyfunkcjonalne.pl or golem15.com) that needs sitemap 1:1 |
+| G15Office verification | fake server only / plus live UAT | Fake server in tests only |
+
+## Vendor testing
+
+| Question | Options | Selected |
+|---|---|---|
+| Upstream answers in replay | recorded upstream fakes / live keys behind tag / fakes + live smoke | Recorded upstream fakes |
+| Time in tests | injected clock / real short timings | Injected clock |
+| Rate-limit state store | researcher decides / memory / Postgres | Researcher decides against PHP |
+
+## Claude's Discretion
+
+- Digest mail content, prune-notifications and reindex details (straight PHP ports), plugin package layout, plan split.
+
+## Deferred Ideas
+
+- Sitemap 1:1 port for the next blog project (todo).
+- oauth-identities + /api/v1/fonoteka/me need a roadmap home before Phase 15 (todo).
+- Golem faces/CompreFace, ChatContextCollector.