summercms/.planning/phases/14-domain-jobs-and-external-integrations/14-CONTEXT.md

Phase 14: Domain jobs and external integrations - Context

Gathered: 2026-10-03 Status: Ready for planning

## Phase Boundary

Phase 14 fills in the Płytarium work that talks to the outside world or runs in the background, on top of the Phase 11 jobs/realtime/search infrastructure and the Phase 13 API surface:

• Job workers for the kinds Phase 13 already enqueues: CSV import write, CSV/Discogs match (240 s timeout, re-enqueues itself with a delay on a Discogs rate-limit error), and wishlist digest (30-minute coalescing window, digest mail, queue-row delete).
• Discogs client (proactive rate threshold, bounded in-request wait budget, retry-after fallback, host-locked cover fetch) and every route that uses it.
• AI cover recognition through a new shared sm-golem-plugin (Anthropic and OpenAI-compatible adapters, per-credential model/base-URL overrides, backend global vision model).
• Commands: fonoteka:prune-notifications and fonoteka:reindex (asserts zero collection_id-0 documents before and after, flag to drop the legacy index). fonoteka:oauth-client already shipped in Phase 8.
• Feedback plugin, ported in full to a new shared sm-feedback-plugin.
• Framework helpers folded in from todos: a guarded outbound http.Client in fetchguard and a slog handler that redacts credentials.

Routes that leave pending in this phase (12): wishlist/albums/{id}/match, wishlist/albums/{id}/apply-release, ai-credential/test, discogs-credential/test, albums/match, albums/{id}/match, albums/{id}/apply-release, albums/recognize (JWT group), albums/import/discogs, /api/v1/fonoteka/albums/recognize (token group, inv.scope:ai), /api/v1/fonoteka/albums/{id}/cover-price/discogs (token group, inv.scope:write, throttle:12,1), plus the selected_discogs_id success case of the CSV row-edit route. The feedback routes are new to the manifest.

Out of scope: sitemap (dropped for Płytarium, D-14), oauth-identities GET/DELETE and GET /api/v1/fonoteka/me (D-09), the Golem face/CompreFace services (D-04).

## Implementation Decisions

AI layer (sm-golem-plugin)

• D-01: The Go port of PHP Golem15.Golem lives in its own shared core-plugin repo, git@git.golem15.com:golem15/sm-golem-plugin.git (created empty by the user). Module git.golem15.com/golem15/sm-golem-plugin, mounted in fonoteka.go at plugins/golem15/golem as a git submodule, following .planning/notes/core-plugins-own-repos.md. The plugin ID stays golem15.golem, with the same settings key (golem15_golem_settings) and lang namespace, so migrated PHP data loads as-is. — Reversibility: costly — the plugin ID, settings key and module path are what every consuming application and the migrated settings row use.
• D-02: The provider adapters (Anthropic Messages, OpenAI-compatible chat completions) are hand-rolled JSON mappings over the guarded outbound client (folded todo fetchguard-guarded-http-client). No Anthropic or OpenAI SDK dependency. PHP's adapters are raw curl, which makes request parity easier.
• D-03: Port the Golem Settings model in full: the models repeater (name, adapter openai|anthropic, encrypted api_key, base_url, model, system_prompt, is_enabled, is_default, generates_images, accepts_images, has_files_endpoint, max_completion_tokens) plus its admin settings screen, reading the existing settings row. getVisionModel() fills fonoteka's classes.AdminVisionModel seam (Phase 13), so the admin tier of ResolveAIConfig and the site-admin branch of AIAllowed light up.
• D-04: Port all of AIService except faces: send, sendStream, sendFile/sendFilePath (files endpoint), sendWithModel, sendToImageModel, sendToVisionModel, generateImage, ask, the model system-prompt application, Prompt/AIResponse value objects and PromptFactory. CompreFace/FaceService and ChatContextCollector are not ported. The SSRFGuard is ported as part of the guarded-client work.
• D-05: Base-URL guard: user and org credential base_url overrides always go through the private/reserved-IP dial guard. Admin-configured Settings models are trusted (they may point at a LAN/local endpoint such as Ollama). The researcher confirms this split against PHP's SSRFGuard and reports any difference before planning.
• D-06: REQUIREMENTS INTG-02 ("Anthropic Go SDK") is reworded at plan time to "Anthropic and OpenAI-compatible adapters over the guarded client". PROJECT.md's repository paragraph gains sm-golem-plugin and sm-feedback-plugin, and drops feedback and sitemap from the application-plugin list.

Route scope

• D-07: All 6 album Discogs/AI routes that Phase 12 D-04 moved here are in scope (albums/match, albums/{id}/match, albums/{id}/apply-release, albums/recognize on the JWT and token groups, albums/import/discogs), together with the routes the criteria already name. ROADMAP SC4/SC5 are reworded at plan time to list them. They share ReleaseMatchScorer, AlbumReleaseApplicator, DiscogsImportResolver, DiscogsMapper and AlbumRecognitionService with the named routes.
• D-08: The Phase 13 ReleaseFetcher seam behind the CSV row-edit selected_discogs_id pick gets the real Discogs getRelease + DiscogsMapper::mapRelease implementation, so a pick resolves exactly as in PHP.
• D-09: GET/DELETE oauth-identities (social login, deferred since Phase 7) and GET /api/v1/fonoteka/me (the minimal endpoint from Phase 8 D-20) are not part of Phase 14. They stay pending and are flagged for the roadmap (see Deferred).
• D-10: WR-02 from the Phase 13 review is fixed before the CSV workers go live: the mapping, row-edit and cancel writes lock the import row or compare-and-swap on its status, as commit already does, so none of them can race a commit and queue a second import job. Response shapes stay the same. This is a deliberate deviation from PHP, which has the same race.
• D-11: The researcher checks routes.php and the Nuxt/MCP callers for any recorded case that is still missing, for example a successful selected_discogs_id pick, the cover-price route, and the recognize success and truncation paths. It records those against the PHP backend, together with their upstream exchanges (D-15).

Feedback plugin (sm-feedback-plugin)

• D-12: Feedback is a shared core plugin in its own repo, git@git.golem15.com:golem15/sm-feedback-plugin.git (created by the user). Module git.golem15.com/golem15/sm-feedback-plugin, mounted in fonoteka.go at plugins/golem15/feedback, plugin ID golem15.feedback, with the same tables (feedback_submissions, user preferences) and the same settings key. — Reversibility: costly — the plugin ID, tables and settings key become the contract other applications mount.
• D-13: Full port. It covers GET /_feedback/api/v1/{key}/config (throttle 60/min/IP), POST {key}/submit (multipart, 10/min/IP, Origin allowlist gate, ImageContentGuard), the OPTIONS {any} 204 preflight, and JWT PUT me/hidden. embed.js is served at the same /plugins/golem15/feedback/assets/js/embed.js path that the Nuxt proxy expects. The plugin also ports the Settings model (enabled, widget_key, allow_hide, position, allowed_origins, colors, pl/en labels) with its admin screen, the submissions admin list, and the golem15.user.getApiArray hook that adds feedback_widget_hidden to the user payload. The user plugin already references this field. SyncFeedbackToG15Office becomes a River job on the guarded client (JSON + multipart). Feedback routes are recorded as new parity fixtures. API-08 is reworded to feedback only.
• D-14: Sitemap is dropped for Płytarium. Nuxt builds its own sitemap with @nuxtjs/seo, and fonoteka registers no menu item types. A todo records a 1:1 sitemap plugin port for the next blog project (grzybyfunkcjonalne.pl or golem15.com).

Vendor testing

• D-15: Upstream HTTP in tests: the upstream Discogs/Anthropic/OpenAI/G15Office exchanges are recorded once as a sidecar to each parity case and served from an httptest fake that the guarded client points at during replay. Replay is deterministic and offline. The fake also asserts the request Go sends upstream (method, path, headers including User-Agent and auth, body) against PHP's recorded request. CI makes no live vendor calls, and G15Office is verified only through the fake.
• D-16: Time is injected. The Discogs rate limiter, its wait budget and retry-after handling, and the match job's re-enqueue all take a clock/sleeper interface. Tests advance a fake clock without real sleeps, and assert the 240 s timeout and the re-enqueue delay as values.
• D-17: The researcher reads DiscogsRateLimiter.php and decides where its state lives, either process memory or Postgres (an UNLOGGED table or an advisory lock), depending on whether PHP's cross-worker guarantee is still needed when one binary runs HTTP and River workers. The choice and its reason go in RESEARCH.md.

Plan-time resolutions (2026-10-03, after research)

• D-18: Golem models are stored in a dedicated golem15_golem_models table with a lagoon.Encrypted api_key (json:"-"), an admin list/form, and a one-time importer from the PHP settings row. The PHP settings code is golem_settings, not golem15_golem_settings (this corrects D-01), and PHP stores models[].api_key in plaintext. Feedback settings use the same pattern: a typed singleton table plus an importer, because cabana rejects colorpicker/readOnly. — Reversibility: costly — the table schema becomes the plugin contract.
• D-19: Upstream sidecars (D-15) are captured with a recording HTTPS proxy in tide (summer parity:upstream). PHP runs through the proxy while cases are recorded, so the sidecars are the real exchanges. ai-credential/test and discogs-credential/test are re-recorded through it.
• D-20: SSRF parity: Go ports PHP SSRFGuard exactly. That covers the https requirement, the host allowlist (default .openai.com plus the DALL-E blob host, override GOLEM15_SSRF_ALLOWED_HOSTS) and the uncaught-exception 500 page on guard failure. Go also adds the fetchguard dial-time private-IP guard on top. Admin Settings models stay trusted (D-05 confirmed).
• D-21: Discogs rate-limiter state lives in Postgres: an UNLOGGED table plus one atomic INSERT … ON CONFLICT DO UPDATE … WHERE … RETURNING, with the clock passed in (resolves D-17, see RESEARCH).

Claude's Discretion

• Wishlist digest mail content, locale and template: a straight port of WishlistDigestJob and its PHP mail view on the existing postcard mail pipeline.
• prune-notifications and reindex options, output and exit codes: a straight port of PruneNotifications.php and ReindexAlbums.php, on bonfire with the existing schedule.go entry.
• Exact package layout inside sm-golem-plugin and sm-feedback-plugin, which follows .planning/notes/plugin-layout-winter-directories.md.
• Plan split, subject to the lean-mode plan-count checkpoint.

Folded Todos

• fetchguard-guarded-http-client (.planning/todos/pending/fetchguard-guarded-http-client.md, high): extend fetchguard from a guarded HTTPS GET into a guarded outbound http.Client/constructor (POST, PUT, multipart upload, bearer auth), keeping the dial-time private/reserved-IP rejection, with an explicit trusted mode for admin-configured endpoints (D-05). It replaces Apparatus RequestSender. Its consumers are the Discogs client, the Golem adapters and feedback's G15OfficeClient. Framework change in summercms.go: update modules/fetchguard/README.md and docs/.
• redacting-slog-handler (.planning/todos/pending/redacting-slog-handler.md, medium): a framework slog.Handler wrapper porting Apparatus RedactCredentialsTap. It redacts the keys api_key, apikey, authorization, bearer, password, secret, token, webhook_secret and admin_password (case-insensitive, nested groups) and scrubs messages with the PHP regex patterns. It also checks whether surf's error path already gives SafeExceptionResponse behaviour. This is the first phase to send user credentials to outside services.

<canonical_refs>

Canonical References

Downstream agents MUST read these before planning or implementing.

Planning

• .planning/ROADMAP.md § Phase 14 — goal and SC1–SC6 (SC4/SC5/SC6 reworded per D-06, D-07, D-14)
• .planning/REQUIREMENTS.md — JOBS-02, JOBS-03, SRCH-02, INTG-01, INTG-02 (reword per D-06), API-08 (reword per D-13/D-14), CLI-05
• .planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-CONTEXT.md — D-01..D-08: job kinds, args and queues; seams; route deferrals
• .planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-VERIFICATION.md — WR-02 and the Phase 14 hand-off table
• .planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-REVIEW.md — WR-02 detail
• .planning/phases/12-p-ytarium-api-collections-and-albums/12-CONTEXT.md — D-04/D-05: album Discogs routes deferred, CoverImporter on fetchguard
• .planning/phases/11-jobs-realtime-and-search-infrastructure/11-CONTEXT.md — conga/River, scheduling, search (reindex) decisions
• .planning/notes/core-plugins-own-repos.md — sm-*-plugin repo, submodule and module-path workflow (D-01, D-12)
• .planning/notes/apparatus-dissolved-into-framework.md — RequestSender and RedactCredentialsTap porting targets
• .planning/notes/plugin-layout-winter-directories.md — plugin directory layout
• .planning/todos/pending/fetchguard-guarded-http-client.md, .planning/todos/pending/redacting-slog-handler.md — folded todos

PHP reference (Płytarium, /media/nvme/dev/golem15/fonoteka)

• plugins/golem15/fonoteka/routes.php lines 207-246, 443-505 — the routes in scope and their groups, scopes and throttles
• plugins/golem15/fonoteka/jobs/{AlbumCsvImportJob,AlbumCsvMatchJob,WishlistDigestJob}.php
• plugins/golem15/fonoteka/console/{PruneNotifications,ReindexAlbums}.php
• plugins/golem15/fonoteka/classes/discogs/* — DiscogsClient, DiscogsRateLimiter, AlbumCoverFetcher, AlbumReleaseApplicator, DiscogsImportResolver, DiscogsInputParser, DiscogsMapper, PriceSuggestionResolver, ReleaseMatchScorer, exceptions
• plugins/golem15/fonoteka/classes/{AlbumRecognitionService,RecognitionTruncatedException,AiDefaults,AiGate,DiscogsGate}.php
• plugins/golem15/fonoteka/controllers/api/{AlbumReleaseMatchController,WishlistReleaseMatchController,RecognizeApiController,DiscogsImportController,AlbumCoverFetchController}.php and the credential test actions
• plugins/golem15/golem/ — classes/services/AIService.php, classes/providers/*, classes/security/SSRFGuard.php, classes/valueobjects/*, classes/factories/PromptFactory.php, models/Settings.php, models/settings/fields.yaml
• plugins/golem15/feedback/ — routes.php, controllers/api/*, jobs/SyncFeedbackToG15Office.php, classes/{G15OfficeClient,ImageContentGuard}.php, models/*, updates/*, assets/js/embed.js, Plugin.php (getApiArray hook, settings)
• plugins/golem15/sitemap/ — reference only for the deferred todo (D-14)
• vue-fonoteka-app/nuxt.config.ts (feedback proxy paths), vue-fonoteka-app/app/stores/auth.ts (me/hidden), fonoteka-mcp (recognize and cover-price callers)

Go code

• ../fonoteka.go/parity/manifest.yaml — pending entries to flip
• ../fonoteka.go/plugins/golem15/fonoteka/{jobs.go,schedule.go,search.go,routes.go}
• ../fonoteka.go/plugins/golem15/fonoteka/classes/{csv_import_service.go,job_contract.go,ai_config_resolver.go,gates.go,cover_importer.go,wishlist_notifications.go}
• modules/fetchguard, modules/conga, modules/postcard, modules/bonfire — framework modules this phase builds on

</canonical_refs>

<code_context>

Existing Code Insights

Reusable Assets

• classes/job_contract.go plus the Phase 13 enqueue sites (csv_import_service.go dispatches CsvMatchArgs/CsvImportArgs, and wishlist_notifications.go dispatches WishlistDigestArgs at 1800 s). The workers register against these exact kinds and args.
• jobs.go registers the invitation and purchase-mail workers. New workers follow the same pattern, and job_contract_worker_test.go is where they are tested.
• classes.ReleaseFetcher seam (Phase 13 D-05) and classes.AdminVisionModel seam (Phase 13): replace their defaults.
• ResolveAIConfig / ResolveDiscogsConfig / AIAllowed are already ported tier by tier.
• cover_importer.go and image_guard.go already use fetchguard's AllowHosts GET, which the host-locked cover fetch reuses.
• schedule.go already schedules fonoteka:prune-notifications, so only the command is missing.
• console/oauth_client.go is the bonfire command pattern for the new commands.
• search.go holds the Typesense collection setup that the reindex command builds on.

Established Patterns

• Pending manifest routes are absent from the router: no 501 shells (P6 D-15). TestRouteTablePhase13 asserts that the Phase 14 routes are absent, so update it when they land.
• Secrets are lagoon.Encrypted with json:"-".
• Core plugins are developed inside their submodule checkout: commit and push there first, then bump the pointer in fonoteka.go as a separate commit.
• Framework README/docs rules from CLAUDE.md apply to fetchguard and the new slog handler.

Integration Points

• fonoteka.go go.work, go.mod (require + local replace), summer.yaml and plugins.gen.go for the two new submodule plugins.
• The fonoteka plugin requires golem15.golem for recognition. Feedback hooks into the user plugin's getApiArray event.

</code_context>

## Specific Ideas

• The user created both repos up front: sm-golem-plugin (the user first created it as sm-ai-plugin and then renamed it so it matches the PHP plugin name) and sm-feedback-plugin.
• After Płytarium, a blog project (grzybyfunkcjonalne.pl or golem15.com) is ported next and needs the sitemap plugin 1:1.

## Deferred Ideas

• Sitemap plugin 1:1 port for the next blog project: a new todo is written alongside this context.
• oauth-identities GET/DELETE (social login) and GET /api/v1/fonoteka/me: the routes stay pending without a phase. The roadmap needs a home for them before Phase 15, whose criterion is all routes green. A new todo is written.
• Golem face detection (CompreFace/FaceService) and ChatContextCollector: not ported until a consumer needs them.

Reviewed Todos (not folded)

• backend-admin-api-tokens, bonfire-duplicate-command-names, lagoon-readme-after-commit-callback-order, nest-framework-packages-under-modules, per-module-readmes-after-nest, readme-go-fences-src, refresh-fonoteka-readme, rewrite-summercms-readme, scaffold-* (3), wristband-neutral-resource-default, 2026-10-01-benchmark-…: these matched on keywords only and are unrelated to the phase domain.

---

Phase: 14-domain-jobs-and-external-integrations Context gathered: 2026-10-03