summercms/.planning/phases/14-domain-jobs-and-external-integrations/14-CONTEXT.md

# Phase 14: Domain jobs and external integrations - Context

**Gathered:** 2026-10-03
**Status:** Ready for planning

<domain>
## Phase Boundary

Phase 14 fills in the Płytarium work that talks to the outside world or runs in the background, on top of the Phase 11 jobs/realtime/search infrastructure and the Phase 13 API surface:

- **Job workers** for the kinds Phase 13 already enqueues: CSV import write, CSV/Discogs match (240 s timeout, re-enqueues itself with a delay on a Discogs rate-limit error), and wishlist digest (30-minute coalescing window, digest mail, queue-row delete).
- **Discogs client** (proactive rate threshold, bounded in-request wait budget, retry-after fallback, host-locked cover fetch) and every route that uses it.
- **AI cover recognition** through a new shared `sm-golem-plugin` (Anthropic and OpenAI-compatible adapters, per-credential model/base-URL overrides, backend global vision model).
- **Commands:** `fonoteka:prune-notifications` and `fonoteka:reindex` (asserts zero `collection_id`-0 documents before and after, flag to drop the legacy index). `fonoteka:oauth-client` already shipped in Phase 8.
- **Feedback plugin**, ported in full to a new shared `sm-feedback-plugin`.
- **Framework helpers** folded in from todos: a guarded outbound `http.Client` in fetchguard and a slog handler that redacts credentials.

Routes that leave `pending` in this phase (12): `wishlist/albums/{id}/match`, `wishlist/albums/{id}/apply-release`, `ai-credential/test`, `discogs-credential/test`, `albums/match`, `albums/{id}/match`, `albums/{id}/apply-release`, `albums/recognize` (JWT group), `albums/import/discogs`, `/api/v1/fonoteka/albums/recognize` (token group, `inv.scope:ai`), `/api/v1/fonoteka/albums/{id}/cover-price/discogs` (token group, `inv.scope:write`, `throttle:12,1`), plus the `selected_discogs_id` success case of the CSV row-edit route. The feedback routes are new to the manifest.

Out of scope: sitemap (dropped for Płytarium, D-14), `oauth-identities` GET/DELETE and `GET /api/v1/fonoteka/me` (D-09), the Golem face/CompreFace services (D-04).

</domain>

<decisions>
## Implementation Decisions

### AI layer (sm-golem-plugin)
- **D-01:** The Go port of PHP `Golem15.Golem` lives in its own shared core-plugin repo, `git@git.golem15.com:golem15/sm-golem-plugin.git` (created empty by the user). Module `git.golem15.com/golem15/sm-golem-plugin`, mounted in fonoteka.go at `plugins/golem15/golem` as a git submodule, following `.planning/notes/core-plugins-own-repos.md`. The plugin ID stays `golem15.golem`, with the same settings key (`golem15_golem_settings`) and lang namespace, so migrated PHP data loads as-is. — **Reversibility:** costly — the plugin ID, settings key and module path are what every consuming application and the migrated settings row use.
- **D-02:** The provider adapters (Anthropic Messages, OpenAI-compatible chat completions) are hand-rolled JSON mappings over the guarded outbound client (folded todo fetchguard-guarded-http-client). No Anthropic or OpenAI SDK dependency. PHP's adapters are raw curl, which makes request parity easier.
- **D-03:** Port the Golem Settings model in full: the `models` repeater (name, adapter openai|anthropic, encrypted `api_key`, `base_url`, `model`, `system_prompt`, `is_enabled`, `is_default`, `generates_images`, `accepts_images`, `has_files_endpoint`, `max_completion_tokens`) plus its admin settings screen, reading the existing settings row. `getVisionModel()` fills fonoteka's `classes.AdminVisionModel` seam (Phase 13), so the admin tier of `ResolveAIConfig` and the site-admin branch of `AIAllowed` light up.
- **D-04:** Port all of AIService except faces: `send`, `sendStream`, `sendFile`/`sendFilePath` (files endpoint), `sendWithModel`, `sendToImageModel`, `sendToVisionModel`, `generateImage`, `ask`, the model system-prompt application, `Prompt`/`AIResponse` value objects and `PromptFactory`. CompreFace/FaceService and ChatContextCollector are not ported. The SSRFGuard is ported as part of the guarded-client work.
- **D-05:** Base-URL guard: user and org credential `base_url` overrides always go through the private/reserved-IP dial guard. Admin-configured Settings models are trusted (they may point at a LAN/local endpoint such as Ollama). The researcher confirms this split against PHP's `SSRFGuard` and reports any difference before planning.
- **D-06:** REQUIREMENTS INTG-02 ("Anthropic Go SDK") is reworded at plan time to "Anthropic and OpenAI-compatible adapters over the guarded client". PROJECT.md's repository paragraph gains sm-golem-plugin and sm-feedback-plugin, and drops feedback and sitemap from the application-plugin list.

### Route scope
- **D-07:** All 6 album Discogs/AI routes that Phase 12 D-04 moved here are in scope (`albums/match`, `albums/{id}/match`, `albums/{id}/apply-release`, `albums/recognize` on the JWT and token groups, `albums/import/discogs`), together with the routes the criteria already name. ROADMAP SC4/SC5 are reworded at plan time to list them. They share ReleaseMatchScorer, AlbumReleaseApplicator, DiscogsImportResolver, DiscogsMapper and AlbumRecognitionService with the named routes.
- **D-08:** The Phase 13 `ReleaseFetcher` seam behind the CSV row-edit `selected_discogs_id` pick gets the real Discogs `getRelease` + `DiscogsMapper::mapRelease` implementation, so a pick resolves exactly as in PHP.
- **D-09:** `GET/DELETE oauth-identities` (social login, deferred since Phase 7) and `GET /api/v1/fonoteka/me` (the minimal endpoint from Phase 8 D-20) are not part of Phase 14. They stay `pending` and are flagged for the roadmap (see Deferred).
- **D-10:** WR-02 from the Phase 13 review is fixed before the CSV workers go live: the `mapping`, row-edit and `cancel` writes lock the import row or compare-and-swap on its status, as `commit` already does, so none of them can race a commit and queue a second import job. Response shapes stay the same. This is a deliberate deviation from PHP, which has the same race.
- **D-11:** The researcher checks `routes.php` and the Nuxt/MCP callers for any recorded case that is still missing, for example a successful `selected_discogs_id` pick, the cover-price route, and the recognize success and truncation paths. It records those against the PHP backend, together with their upstream exchanges (D-15).

### Feedback plugin (sm-feedback-plugin)
- **D-12:** Feedback is a shared core plugin in its own repo, `git@git.golem15.com:golem15/sm-feedback-plugin.git` (created by the user). Module `git.golem15.com/golem15/sm-feedback-plugin`, mounted in fonoteka.go at `plugins/golem15/feedback`, plugin ID `golem15.feedback`, with the same tables (`feedback_submissions`, user preferences) and the same settings key. — **Reversibility:** costly — the plugin ID, tables and settings key become the contract other applications mount.
- **D-13:** Full port. It covers `GET /_feedback/api/v1/{key}/config` (throttle 60/min/IP), `POST {key}/submit` (multipart, 10/min/IP, Origin allowlist gate, ImageContentGuard), the `OPTIONS {any}` 204 preflight, and JWT `PUT me/hidden`. `embed.js` is served at the same `/plugins/golem15/feedback/assets/js/embed.js` path that the Nuxt proxy expects. The plugin also ports the Settings model (enabled, widget_key, allow_hide, position, allowed_origins, colors, pl/en labels) with its admin screen, the submissions admin list, and the `golem15.user.getApiArray` hook that adds `feedback_widget_hidden` to the user payload. The user plugin already references this field. `SyncFeedbackToG15Office` becomes a River job on the guarded client (JSON + multipart). Feedback routes are recorded as new parity fixtures. API-08 is reworded to feedback only.
- **D-14:** Sitemap is dropped for Płytarium. Nuxt builds its own sitemap with `@nuxtjs/seo`, and fonoteka registers no menu item types. A todo records a 1:1 sitemap plugin port for the next blog project (grzybyfunkcjonalne.pl or golem15.com).

### Vendor testing
- **D-15:** Upstream HTTP in tests: the upstream Discogs/Anthropic/OpenAI/G15Office exchanges are recorded once as a sidecar to each parity case and served from an `httptest` fake that the guarded client points at during replay. Replay is deterministic and offline. The fake also asserts the request Go sends upstream (method, path, headers including User-Agent and auth, body) against PHP's recorded request. CI makes no live vendor calls, and G15Office is verified only through the fake.
- **D-16:** Time is injected. The Discogs rate limiter, its wait budget and retry-after handling, and the match job's re-enqueue all take a clock/sleeper interface. Tests advance a fake clock without real sleeps, and assert the 240 s timeout and the re-enqueue delay as values.
- **D-17:** The researcher reads `DiscogsRateLimiter.php` and decides where its state lives, either process memory or Postgres (an UNLOGGED table or an advisory lock), depending on whether PHP's cross-worker guarantee is still needed when one binary runs HTTP and River workers. The choice and its reason go in RESEARCH.md.

### Plan-time resolutions (2026-10-03, after research)
- **D-18:** Golem models are stored in a dedicated `golem15_golem_models` table with a `lagoon.Encrypted` `api_key` (`json:"-"`), an admin list/form, and a one-time importer from the PHP settings row. The PHP settings code is `golem_settings`, not `golem15_golem_settings` (this corrects D-01), and PHP stores `models[].api_key` in plaintext. Feedback settings use the same pattern: a typed singleton table plus an importer, because cabana rejects `colorpicker`/`readOnly`. — **Reversibility:** costly — the table schema becomes the plugin contract.
- **D-19:** Upstream sidecars (D-15) are captured with a recording HTTPS proxy in tide (`summer parity:upstream`). PHP runs through the proxy while cases are recorded, so the sidecars are the real exchanges. `ai-credential/test` and `discogs-credential/test` are re-recorded through it.
- **D-20:** SSRF parity: Go ports PHP `SSRFGuard` exactly. That covers the https requirement, the host allowlist (default `.openai.com` plus the DALL-E blob host, override `GOLEM15_SSRF_ALLOWED_HOSTS`) and the uncaught-exception 500 page on guard failure. Go also adds the fetchguard dial-time private-IP guard on top. Admin Settings models stay trusted (D-05 confirmed).
- **D-21:** Discogs rate-limiter state lives in Postgres: an UNLOGGED table plus one atomic `INSERT … ON CONFLICT DO UPDATE … WHERE … RETURNING`, with the clock passed in (resolves D-17, see RESEARCH).

### Claude's Discretion
- Wishlist digest mail content, locale and template: a straight port of `WishlistDigestJob` and its PHP mail view on the existing postcard mail pipeline.
- `prune-notifications` and `reindex` options, output and exit codes: a straight port of `PruneNotifications.php` and `ReindexAlbums.php`, on bonfire with the existing `schedule.go` entry.
- Exact package layout inside sm-golem-plugin and sm-feedback-plugin, which follows `.planning/notes/plugin-layout-winter-directories.md`.
- Plan split, subject to the lean-mode plan-count checkpoint.

### Folded Todos
- **fetchguard-guarded-http-client** (`.planning/todos/pending/fetchguard-guarded-http-client.md`, high): extend fetchguard from a guarded HTTPS GET into a guarded outbound `http.Client`/constructor (POST, PUT, multipart upload, bearer auth), keeping the dial-time private/reserved-IP rejection, with an explicit trusted mode for admin-configured endpoints (D-05). It replaces Apparatus `RequestSender`. Its consumers are the Discogs client, the Golem adapters and feedback's G15OfficeClient. Framework change in summercms.go: update `modules/fetchguard/README.md` and `docs/`.
- **redacting-slog-handler** (`.planning/todos/pending/redacting-slog-handler.md`, medium): a framework `slog.Handler` wrapper porting Apparatus `RedactCredentialsTap`. It redacts the keys api_key, apikey, authorization, bearer, password, secret, token, webhook_secret and admin_password (case-insensitive, nested groups) and scrubs messages with the PHP regex patterns. It also checks whether surf's error path already gives `SafeExceptionResponse` behaviour. This is the first phase to send user credentials to outside services.

</decisions>

<canonical_refs>
## Canonical References

**Downstream agents MUST read these before planning or implementing.**

### Planning
- `.planning/ROADMAP.md` § Phase 14 — goal and SC1–SC6 (SC4/SC5/SC6 reworded per D-06, D-07, D-14)
- `.planning/REQUIREMENTS.md` — JOBS-02, JOBS-03, SRCH-02, INTG-01, INTG-02 (reword per D-06), API-08 (reword per D-13/D-14), CLI-05
- `.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-CONTEXT.md` — D-01..D-08: job kinds, args and queues; seams; route deferrals
- `.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-VERIFICATION.md` — WR-02 and the Phase 14 hand-off table
- `.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-REVIEW.md` — WR-02 detail
- `.planning/phases/12-p-ytarium-api-collections-and-albums/12-CONTEXT.md` — D-04/D-05: album Discogs routes deferred, CoverImporter on fetchguard
- `.planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md` — conga/River, scheduling, search (reindex) decisions
- `.planning/notes/core-plugins-own-repos.md` — sm-*-plugin repo, submodule and module-path workflow (D-01, D-12)
- `.planning/notes/apparatus-dissolved-into-framework.md` — RequestSender and RedactCredentialsTap porting targets
- `.planning/notes/plugin-layout-winter-directories.md` — plugin directory layout
- `.planning/todos/pending/fetchguard-guarded-http-client.md`, `.planning/todos/pending/redacting-slog-handler.md` — folded todos

### PHP reference (Płytarium, `/media/nvme/dev/golem15/fonoteka`)
- `plugins/golem15/fonoteka/routes.php` lines 207-246, 443-505 — the routes in scope and their groups, scopes and throttles
- `plugins/golem15/fonoteka/jobs/{AlbumCsvImportJob,AlbumCsvMatchJob,WishlistDigestJob}.php`
- `plugins/golem15/fonoteka/console/{PruneNotifications,ReindexAlbums}.php`
- `plugins/golem15/fonoteka/classes/discogs/*` — DiscogsClient, DiscogsRateLimiter, AlbumCoverFetcher, AlbumReleaseApplicator, DiscogsImportResolver, DiscogsInputParser, DiscogsMapper, PriceSuggestionResolver, ReleaseMatchScorer, exceptions
- `plugins/golem15/fonoteka/classes/{AlbumRecognitionService,RecognitionTruncatedException,AiDefaults,AiGate,DiscogsGate}.php`
- `plugins/golem15/fonoteka/controllers/api/{AlbumReleaseMatchController,WishlistReleaseMatchController,RecognizeApiController,DiscogsImportController,AlbumCoverFetchController}.php` and the credential `test` actions
- `plugins/golem15/golem/` — `classes/services/AIService.php`, `classes/providers/*`, `classes/security/SSRFGuard.php`, `classes/valueobjects/*`, `classes/factories/PromptFactory.php`, `models/Settings.php`, `models/settings/fields.yaml`
- `plugins/golem15/feedback/` — `routes.php`, `controllers/api/*`, `jobs/SyncFeedbackToG15Office.php`, `classes/{G15OfficeClient,ImageContentGuard}.php`, `models/*`, `updates/*`, `assets/js/embed.js`, `Plugin.php` (getApiArray hook, settings)
- `plugins/golem15/sitemap/` — reference only for the deferred todo (D-14)
- `vue-fonoteka-app/nuxt.config.ts` (feedback proxy paths), `vue-fonoteka-app/app/stores/auth.ts` (`me/hidden`), `fonoteka-mcp` (recognize and cover-price callers)

### Go code
- `../fonoteka.go/parity/manifest.yaml` — pending entries to flip
- `../fonoteka.go/plugins/golem15/fonoteka/{jobs.go,schedule.go,search.go,routes.go}`
- `../fonoteka.go/plugins/golem15/fonoteka/classes/{csv_import_service.go,job_contract.go,ai_config_resolver.go,gates.go,cover_importer.go,wishlist_notifications.go}`
- `modules/fetchguard`, `modules/conga`, `modules/postcard`, `modules/bonfire` — framework modules this phase builds on

</canonical_refs>

<code_context>
## Existing Code Insights

### Reusable Assets
- `classes/job_contract.go` plus the Phase 13 enqueue sites (`csv_import_service.go` dispatches `CsvMatchArgs`/`CsvImportArgs`, and `wishlist_notifications.go` dispatches `WishlistDigestArgs` at 1800 s). The workers register against these exact kinds and args.
- `jobs.go` registers the invitation and purchase-mail workers. New workers follow the same pattern, and `job_contract_worker_test.go` is where they are tested.
- `classes.ReleaseFetcher` seam (Phase 13 D-05) and `classes.AdminVisionModel` seam (Phase 13): replace their defaults.
- `ResolveAIConfig` / `ResolveDiscogsConfig` / `AIAllowed` are already ported tier by tier.
- `cover_importer.go` and `image_guard.go` already use fetchguard's `AllowHosts` GET, which the host-locked cover fetch reuses.
- `schedule.go` already schedules `fonoteka:prune-notifications`, so only the command is missing.
- `console/oauth_client.go` is the bonfire command pattern for the new commands.
- `search.go` holds the Typesense collection setup that the reindex command builds on.

### Established Patterns
- Pending manifest routes are absent from the router: no 501 shells (P6 D-15). `TestRouteTablePhase13` asserts that the Phase 14 routes are absent, so update it when they land.
- Secrets are `lagoon.Encrypted` with `json:"-"`.
- Core plugins are developed inside their submodule checkout: commit and push there first, then bump the pointer in fonoteka.go as a separate commit.
- Framework README/docs rules from CLAUDE.md apply to fetchguard and the new slog handler.

### Integration Points
- fonoteka.go `go.work`, `go.mod` (require + local replace), `summer.yaml` and `plugins.gen.go` for the two new submodule plugins.
- The fonoteka plugin requires `golem15.golem` for recognition. Feedback hooks into the user plugin's `getApiArray` event.

</code_context>

<specifics>
## Specific Ideas

- The user created both repos up front: `sm-golem-plugin` (the user first created it as `sm-ai-plugin` and then renamed it so it matches the PHP plugin name) and `sm-feedback-plugin`.
- After Płytarium, a blog project (grzybyfunkcjonalne.pl or golem15.com) is ported next and needs the sitemap plugin 1:1.

</specifics>

<deferred>
## Deferred Ideas

- **Sitemap plugin 1:1 port** for the next blog project: a new todo is written alongside this context.
- **`oauth-identities` GET/DELETE (social login) and `GET /api/v1/fonoteka/me`**: the routes stay `pending` without a phase. The roadmap needs a home for them before Phase 15, whose criterion is all routes green. A new todo is written.
- Golem face detection (CompreFace/FaceService) and ChatContextCollector: not ported until a consumer needs them.

### Reviewed Todos (not folded)
- `backend-admin-api-tokens`, `bonfire-duplicate-command-names`, `lagoon-readme-after-commit-callback-order`, `nest-framework-packages-under-modules`, `per-module-readmes-after-nest`, `readme-go-fences-src`, `refresh-fonoteka-readme`, `rewrite-summercms-readme`, `scaffold-*` (3), `wristband-neutral-resource-default`, `2026-10-01-benchmark-…`: these matched on keywords only and are unrelated to the phase domain.

</deferred>

---

*Phase: 14-domain-jobs-and-external-integrations*
*Context gathered: 2026-10-03*