97 KiB
Phase 14: Domain jobs and external integrations - Research
Researched: 2026-10-03
Domain: Go port of Płytarium's outbound integrations (Discogs, Anthropic/OpenAI-compatible AI, G15Office), River job workers, console commands, and two new shared core plugins (sm-golem-plugin, sm-feedback-plugin)
Confidence: HIGH for the PHP behaviour and the Go seams (all read this session); MEDIUM for the proposed new framework designs (fetchguard client, upstream sidecars); LOW only where marked [ASSUMED].
<user_constraints>
User Constraints (from CONTEXT.md)
Locked Decisions
AI layer (sm-golem-plugin)
- D-01: The Go port of PHP
Golem15.Golemlives in its own shared core-plugin repo,git@git.golem15.com:golem15/sm-golem-plugin.git(created empty by the user). Modulegit.golem15.com/golem15/sm-golem-plugin, mounted in fonoteka.go atplugins/golem15/golemas a git submodule, following.planning/notes/core-plugins-own-repos.md. The plugin ID staysgolem15.golem, with the same settings key (golem15_golem_settings) and lang namespace, so migrated PHP data loads as-is. — Reversibility: costly — the plugin ID, settings key and module path are what every consuming application and the migrated settings row use. - D-02: The provider adapters (Anthropic Messages, OpenAI-compatible chat completions) are hand-rolled JSON mappings over the guarded outbound client (folded todo fetchguard-guarded-http-client). No Anthropic or OpenAI SDK dependency. PHP's adapters are raw curl, which makes request parity easier.
- D-03: Port the Golem Settings model in full: the
modelsrepeater (name, adapter openai|anthropic, encryptedapi_key,base_url,model,system_prompt,is_enabled,is_default,generates_images,accepts_images,has_files_endpoint,max_completion_tokens) plus its admin settings screen, reading the existing settings row.getVisionModel()fills fonoteka'sclasses.AdminVisionModelseam (Phase 13), so the admin tier ofResolveAIConfigand the site-admin branch ofAIAllowedlight up. - D-04: Port all of AIService except faces:
send,sendStream,sendFile/sendFilePath(files endpoint),sendWithModel,sendToImageModel,sendToVisionModel,generateImage,ask, the model system-prompt application,Prompt/AIResponsevalue objects andPromptFactory. CompreFace/FaceService and ChatContextCollector are not ported. The SSRFGuard is ported as part of the guarded-client work. - D-05: Base-URL guard: user and org credential
base_urloverrides always go through the private/reserved-IP dial guard. Admin-configured Settings models are trusted (they may point at a LAN/local endpoint such as Ollama). The researcher confirms this split against PHP'sSSRFGuardand reports any difference before planning. - D-06: REQUIREMENTS INTG-02 ("Anthropic Go SDK") is reworded at plan time to "Anthropic and OpenAI-compatible adapters over the guarded client". PROJECT.md's repository paragraph gains sm-golem-plugin and sm-feedback-plugin, and drops feedback and sitemap from the application-plugin list.
Route scope
- D-07: All 6 album Discogs/AI routes that Phase 12 D-04 moved here are in scope (
albums/match,albums/{id}/match,albums/{id}/apply-release,albums/recognizeon the JWT and token groups,albums/import/discogs), together with the routes the criteria already name. ROADMAP SC4/SC5 are reworded at plan time to list them. They share ReleaseMatchScorer, AlbumReleaseApplicator, DiscogsImportResolver, DiscogsMapper and AlbumRecognitionService with the named routes. - D-08: The Phase 13
ReleaseFetcherseam behind the CSV row-editselected_discogs_idpick gets the real DiscogsgetRelease+DiscogsMapper::mapReleaseimplementation, so a pick resolves exactly as in PHP. - D-09:
GET/DELETE oauth-identities(social login, deferred since Phase 7) andGET /api/v1/fonoteka/me(the minimal endpoint from Phase 8 D-20) are not part of Phase 14. They staypendingand are flagged for the roadmap (see Deferred). - D-10: WR-02 from the Phase 13 review is fixed before the CSV workers go live: the
mapping, row-edit andcancelwrites lock the import row or compare-and-swap on its status, ascommitalready does, so none of them can race a commit and queue a second import job. Response shapes stay the same. This is a deliberate deviation from PHP, which has the same race. - D-11: The researcher checks
routes.phpand the Nuxt/MCP callers for any recorded case that is still missing, for example a successfulselected_discogs_idpick, the cover-price route, and the recognize success and truncation paths. It records those against the PHP backend, together with their upstream exchanges (D-15).
Feedback plugin (sm-feedback-plugin)
- D-12: Feedback is a shared core plugin in its own repo,
git@git.golem15.com:golem15/sm-feedback-plugin.git(created by the user). Modulegit.golem15.com/golem15/sm-feedback-plugin, mounted in fonoteka.go atplugins/golem15/feedback, plugin IDgolem15.feedback, with the same tables (feedback_submissions, user preferences) and the same settings key. — Reversibility: costly — the plugin ID, tables and settings key become the contract other applications mount. - D-13: Full port. It covers
GET /_feedback/api/v1/{key}/config(throttle 60/min/IP),POST {key}/submit(multipart, 10/min/IP, Origin allowlist gate, ImageContentGuard), theOPTIONS {any}204 preflight, and JWTPUT me/hidden.embed.jsis served at the same/plugins/golem15/feedback/assets/js/embed.jspath that the Nuxt proxy expects. The plugin also ports the Settings model (enabled, widget_key, allow_hide, position, allowed_origins, colors, pl/en labels) with its admin screen, the submissions admin list, and thegolem15.user.getApiArrayhook that addsfeedback_widget_hiddento the user payload. The user plugin already references this field.SyncFeedbackToG15Officebecomes a River job on the guarded client (JSON + multipart). Feedback routes are recorded as new parity fixtures. API-08 is reworded to feedback only. - D-14: Sitemap is dropped for Płytarium. Nuxt builds its own sitemap with
@nuxtjs/seo, and fonoteka registers no menu item types. A todo records a 1:1 sitemap plugin port for the next blog project (grzybyfunkcjonalne.pl or golem15.com).
Vendor testing
- D-15: Upstream HTTP in tests: the upstream Discogs/Anthropic/OpenAI/G15Office exchanges are recorded once as a sidecar to each parity case and served from an
httptestfake that the guarded client points at during replay. Replay is deterministic and offline. The fake also asserts the request Go sends upstream (method, path, headers including User-Agent and auth, body) against PHP's recorded request. CI makes no live vendor calls, and G15Office is verified only through the fake. - D-16: Time is injected. The Discogs rate limiter, its wait budget and retry-after handling, and the match job's re-enqueue all take a clock/sleeper interface. Tests advance a fake clock without real sleeps, and assert the 240 s timeout and the re-enqueue delay as values.
- D-17: The researcher reads
DiscogsRateLimiter.phpand decides where its state lives, either process memory or Postgres (an UNLOGGED table or an advisory lock), depending on whether PHP's cross-worker guarantee is still needed when one binary runs HTTP and River workers. The choice and its reason go in RESEARCH.md.
Folded Todos
- fetchguard-guarded-http-client (
.planning/todos/pending/fetchguard-guarded-http-client.md, high): extend fetchguard from a guarded HTTPS GET into a guarded outboundhttp.Client/constructor (POST, PUT, multipart upload, bearer auth), keeping the dial-time private/reserved-IP rejection, with an explicit trusted mode for admin-configured endpoints (D-05). It replaces ApparatusRequestSender. Its consumers are the Discogs client, the Golem adapters and feedback's G15OfficeClient. Framework change in summercms.go: updatemodules/fetchguard/README.mdanddocs/. - redacting-slog-handler (
.planning/todos/pending/redacting-slog-handler.md, medium): a frameworkslog.Handlerwrapper porting ApparatusRedactCredentialsTap. It redacts the keys api_key, apikey, authorization, bearer, password, secret, token, webhook_secret and admin_password (case-insensitive, nested groups) and scrubs messages with the PHP regex patterns. It also checks whether surf's error path already givesSafeExceptionResponsebehaviour. This is the first phase to send user credentials to outside services.
Claude's Discretion
- Wishlist digest mail content, locale and template: a straight port of
WishlistDigestJoband its PHP mail view on the existing postcard mail pipeline. prune-notificationsandreindexoptions, output and exit codes: a straight port ofPruneNotifications.phpandReindexAlbums.php, on bonfire with the existingschedule.goentry.- Exact package layout inside sm-golem-plugin and sm-feedback-plugin, which follows
.planning/notes/plugin-layout-winter-directories.md. - Plan split, subject to the lean-mode plan-count checkpoint.
Deferred Ideas (OUT OF SCOPE)
- Sitemap plugin 1:1 port for the next blog project: a new todo is written alongside this context.
oauth-identitiesGET/DELETE (social login) andGET /api/v1/fonoteka/me: the routes staypendingwithout a phase. The roadmap needs a home for them before Phase 15, whose criterion is all routes green. A new todo is written.- Golem face detection (CompreFace/FaceService) and ChatContextCollector: not ported until a consumer needs them.
Reviewed Todos (not folded)
backend-admin-api-tokens,bonfire-duplicate-command-names,lagoon-readme-after-commit-callback-order,nest-framework-packages-under-modules,per-module-readmes-after-nest,readme-go-fences-src,refresh-fonoteka-readme,rewrite-summercms-readme,scaffold-*(3),wristband-neutral-resource-default,2026-10-01-benchmark-…: these matched on keywords only and are unrelated to the phase domain. </user_constraints>
<phase_requirements>
Phase Requirements
| ID | Description (REQUIREMENTS.md, before the D-06/D-13/D-14 rewording) | Research Support |
|---|---|---|
| JOBS-02 | CSV import write job and Discogs match job (240 s timeout) are ported; the match job re-enqueues itself with a delay on a Discogs rate-limit error instead of failing the batch | §"The three PHP jobs", §D-17, §"WR-02 fix", Pattern 3 (job worker), Pitfalls 4-7 |
| JOBS-03 | Wishlist digest coalesces notifications in a 30-minute window and deletes its queue row on completion | §"The three PHP jobs" (WishlistDigestJob), Pattern 3 |
| SRCH-02 | The reindex command asserts zero documents with collection_id 0 before and after, and can drop the legacy index | §"Console commands" (ReindexAlbums), beachcomber gap (DropIndex) |
| INTG-01 | Discogs client with proactive rate threshold, bounded in-request wait budget, retry-after fallback, and host-locked cover fetch, plus the cover-price route, the wishlist match/apply-release routes, discogs-credential/test and the CSV row-edit pick | §"Discogs client", §D-17, §D-11 case list, §"Inbound limiters" |
| INTG-02 | AI cover recognition through provider adapters (reword: Anthropic and OpenAI-compatible adapters over the guarded client) with per-credential model and base URL overrides, plus ai-credential/test and the backend global vision model | §"Golem plugin surface", §D-05, §"Settings storage gap" |
| API-08 | Feedback submissions and sitemap output (reword: feedback only, D-13/D-14) | §"Feedback plugin surface" |
| CLI-05 | Płytarium commands: oauth-client (already shipped Phase 8), prune-notifications, reindex with drop-old-index flag | §"Console commands" |
| </phase_requirements> |
Project Constraints (from CLAUDE.md)
From summercms.go/CLAUDE.md, fonoteka.go/CLAUDE.md, the parent CLAUDE.md and the user's global CLAUDE.md:
- Lean planning: few, large plans; a plan-count checkpoint before PLAN.md files are written; unit tests are always the last plan of the phase.
- Go conventions: standard library first; add a dependency only when the research doc or a phase decision names it.
go vetandgo test ./...green at every commit. D-02 forbids an Anthropic/OpenAI SDK: this phase adds no new third-party Go dependency. - Compiled plugins only; no runtime plugin loading.
- API parity is the acceptance test. Do not "improve" response shapes. D-10 (WR-02) is the one sanctioned deviation and keeps response shapes.
- Commits: never add co-author tags; one logical change per commit; planning docs and code in separate commits.
- Framework docs rules: any change to a
modules/package's exported API, config keys, CLI commands or dependencies updates that module'sREADME.mdand the affecteddocs/pages in the same change; a new module ships a README with the standard structure and a row in the root README modules table; framework READMEs never name a consuming application; every identifier named in a README/docs page must exist (go test ./cmd/summer -run TestDocsTree,summer docs:build --check). Config keys in docs are checked by hand. - Two repositories:
summercms.gois framework only and knows nothing about Płytarium; app code goes tofonoteka.go. Planning docs stay insummercms.go/.planning. - Core plugin contracts: user, blog, pages, payment PHP plugins are shared; Go ports preserve contracts; PHP originals are not changed. sm-user-plugin changes must be additive.
- Submodule workflow (fonoteka.go/CLAUDE.md, core-plugins-own-repos.md): change a core plugin inside its checkout, commit (and push) there first, then commit the bumped pointer in fonoteka.go as a separate commit; never stage submodule files from the app repo.
ssuis available for submodule management. - GSD workflow enforcement: file changes go through GSD commands.
Summary
Phase 14 is the largest port so far by PHP volume: about 2,250 lines of Discogs classes, 900 lines of Discogs/AI controllers, 650 lines of jobs, 450 lines of AlbumRecognitionService, 900 lines of the Golem AI layer and 1,650 lines of the feedback plugin (including embed.js). Almost everything it needs from the framework already exists: conga (River) with Dispatch/Delay/Timeout/StopJob/CompleteJob, the bonfire command pattern, beachcomber's Typesense engine, lagoon file attachments, surf.MemoryStore for in-controller rate limiters, the user plugin's GetApiArrayEvent, and the Phase 13 seams (job_contract.go kinds and args, ReleaseFetcher, AdminVisionModel, ResolveAIConfig, DiscogsAllowed). The framework gaps are concrete and small: fetchguard is GET-only with test hooks that are unexported, cabana settings cannot express a repeater, beachcomber cannot report whether a dropped index existed, and tide has no notion of upstream-exchange sidecars.
The three researcher tasks resolve as follows. D-05: the admin-trusted / user-and-org-guarded split matches PHP, but PHP's SSRFGuard::assertSafeUrl is stricter and different in kind: https only, a host allowlist (default only .openai.com and the DALL-E blob host, env-overridable), and a resolve-time private-IP check. Its failure is an uncaught RuntimeException, so PHP answers with the Winter 500 HTML page, not {ok:false}. The Go port must keep the allowlist and the 500 to stay parity-true, and add the dial-time guard on top. D-17: the limiter state goes in Postgres (an UNLOGGED table and one atomic upsert), because conga explicitly supports a separate queue:work process (queue.work_in_serve: false) and the 50/60-per-minute budget is shared by every process. D-11: every currently recorded Phase 14 case is a negative path (404, 503 disabled, 403, 422, a live-vendor 401). All success, rate-limit, truncation and pick cases are still missing, and the two live-vendor cases must be re-recorded with a sidecar.
Two findings contradict CONTEXT and need user confirmation before planning (see Open Questions). First, the PHP Golem settings code is golem_settings, not golem15_golem_settings. Second, the PHP models[].api_key is stored in plaintext, not encrypted. The project's settings convention (Phase 5, user-resolved) is a dedicated typed table, not Winter system_settings, and cabana has no repeater field type. So "reading the existing settings row" needs a storage decision (recommended: a golem15_golem_models table, edited as an admin list/form, plus an import from the PHP JSON at cutover).
Primary recommendation: Build the framework pieces first (guarded fetchguard.Client with a trusted mode and a test transport seam, the redacting slog handler, tide upstream sidecars plus a recording proxy, beachcomber DropIndex). Then port in vertical slices: Discogs core and jobs, Discogs routes, sm-golem-plugin with AI recognition, sm-feedback-plugin. Finish with a unit-test plan and a check-phase14.sh gate.
Architectural Responsibility Map
| Capability | Primary Tier | Secondary Tier | Rationale |
|---|---|---|---|
| Guarded outbound HTTP (POST/PUT/multipart/bearer, dial-time IP guard, trusted mode) | Framework (summercms.go fetchguard) |
— | Shared by Discogs, Golem and G15Office; SSRF policy is a framework concern |
| Credential redaction in logs | Framework (new slog handler) | App wiring (slog.SetDefault in serve/queue:work) |
Nothing publishes a *slog.Logger today; every module falls back to slog.Default() |
| Upstream exchange sidecars (record + replay fake + request assertion) | Framework (tide) |
App (parity/ harness wiring) |
tide owns fixtures; the app owns which routes use sidecars |
| Discogs client, limiter, mapper, scorer, applicator, resolvers | App (fonoteka plugin classes/discogs) |
DB (UNLOGGED limiter table) | Płytarium-specific domain |
| Discogs/AI/credential-test routes | App (fonoteka controllers/api, routes.go) |
— | API parity surface |
| CSV match/import, digest workers | App (fonoteka plugin jobs.go or jobs/) |
Framework (conga/River) | Kinds and args are fixed by job_contract.go |
| AI providers, Prompt/AIResponse, model settings | Core plugin (sm-golem-plugin) | Framework (fetchguard) | Shared across Golem15 apps (D-01) |
| Feedback widget API, settings, G15Office sync | Core plugin (sm-feedback-plugin) | User plugin event (GetApiArrayEvent) |
Shared core plugin (D-12) |
fonoteka:prune-notifications, fonoteka:reindex |
App (fonoteka console/) |
Framework (bonfire, conga schedule, beachcomber) | Straight ports |
Standard Stack
Core (all already in the build; nothing new to install)
| Library / module | Version | Purpose | Why standard here |
|---|---|---|---|
Go stdlib net/http, mime/multipart, encoding/json, crypto/hmac, crypto/sha256, log/slog |
Go 1.27.0 (go version this session) |
Adapters, multipart upload, HMAC bucket id, redaction handler | D-02: hand-rolled JSON over the guarded client [VERIFIED: go version] |
modules/fetchguard |
in-repo | Base of the guarded client | Existing dial-time classifier (ip.go) is a strict superset of PHP's lists [VERIFIED: modules/fetchguard/ip.go:5-20] |
modules/conga (River v0.47.0) |
in-repo | Workers, Delay re-dispatch, Timeout, record rows | conga.Timeout, DispatchOpts.Delay, StopJob, CompleteJob exist [VERIFIED: modules/conga/job.go:21-37, conga.go:112-133] |
modules/beachcomber + beachcomber/typesense |
in-repo | Reindex: Upsert/Flush/SearchPage | Engine has Upsert, Delete, Flush, SearchIDs; PageSearcher gives Found [VERIFIED: modules/beachcomber/searchable.go:39-57] |
modules/surf MemoryStore |
in-repo | In-controller limiters (fonoteka-discogs-missing:, fonoteka-recognize:, fonoteka-discogs-import:) |
Attempt(key, max, decay) is atomic check+increment and returns retryAfter [VERIFIED: modules/surf/limiter_store.go:11-13] |
modules/lagoon/attach |
in-repo | Feedback screenshot attachOne |
attach.Relation{Name, Many, Public} [VERIFIED: modules/lagoon/attach/relation.go] |
modules/postcard |
in-repo | Digest mail templates | Existing mail jobs use postcard.Mailer.Send [VERIFIED: fonoteka plugin jobs.go:119-126] |
modules/cabana |
in-repo | Admin settings and list screens | Settings are scalar-only singleton rows (see gap) [VERIFIED: modules/cabana/settings.go:37-95] |
Supporting
| Library | Version | Purpose | When to Use |
|---|---|---|---|
| testcontainers-go postgres | v0.44.0 (already in go.mod) | Limiter SQL, worker and migration tests | Postgres-dependent tests, -short skippable |
pgx/v5 |
v5.10.0 (already in go.mod) | — | Unchanged |
Alternatives Considered
| Instead of | Could Use | Tradeoff |
|---|---|---|
| Hand-rolled adapters | anthropic-sdk-go / openai-go | Locked out by D-02 |
| Postgres limiter table | Process-memory limiter (like PubfailCounter) |
Breaks the cross-process budget when queue.work_in_serve: false (see D-17) |
| MITM recording proxy for sidecars | Hand-written sidecars from vendor docs | Hand-written sidecars cannot capture what PHP actually sent; D-15 asks for PHP's recorded request |
Installation: none. go.mod changes are limited to the two new submodule modules (require plus local replace) and their own go.mod files.
Package Legitimacy Audit
This phase installs no external packages (D-02 rules out vendor SDKs; every capability is stdlib or in-repo). The seam check was not needed.
| Package | Registry | Age | Downloads | Source Repo | Verdict | Disposition |
|---|---|---|---|---|---|---|
| (none) | — | — | — | — | — | — |
Packages removed due to [SLOP] verdict: none Packages flagged as suspicious [SUS]: none
Existing Go seams and enqueue sites (inventory for the planner)
| Seam / file | What exists | Phase 14 action |
|---|---|---|
fonoteka/classes/job_contract.go:29-62 |
CsvImportKind = "golem15.fonoteka.csv_import", CsvImportQueue = "fonoteka_csv_import", CsvImportLabel = "fonoteka.csv.import", CsvMatchKind = "golem15.fonoteka.csv_match", CsvMatchQueue = "fonoteka_csv_match", CsvMatchLabel = "fonoteka.csv.match", WishlistDigestKind = "golem15.fonoteka.wishlist_digest", WishlistDigestJobQueue = "fonoteka_wishlist_digest", WishlistDigestLabel = "wishlist_digest", WishlistDigestDelay = 1800 * time.Second; args CsvImportArgs{CsvImportID uint \json:"csv_import_id"`}, CsvMatchArgs{CsvImportID}, WishlistDigestArgs{SubscriberID `json:"subscriber_id"`, WishlistCollectionID `json:"wishlist_collection_id"`}` [VERIFIED: job_contract.go:29-88] |
Register workers with exactly these kinds and queues. Never rename (queued rows carry them) |
fonoteka/jobs.go:32-41 |
Jobs() returns invitation mail and purchase mail on conga.OnQueue(...), conga.MaxAttempts(...); comment says "The CSV and wishlist digest kinds have no worker until Phase 14 (D-04)" |
Add three conga.Job(...) entries. Update the comment and TestJobContract* (they assert the queues are unserved) |
classes/csv_import_service.go:519-587 UpdateCsvMapping |
Dispatches CsvMatchArgs on CsvMatchQueue, label, Count: RowCount, metadata {"csv_import_id":N} |
WR-02 lock (below) |
csv_import_service.go:589-628 ReleaseFetcher + SetReleaseFetcher (atomic box) |
Default always returns ErrDiscogsUnavailable |
Install the real fetcher at plugin Boot (D-08). ErrDiscogsRateLimited maps to discogs_rate_limited 422 |
csv_import_service.go:639-705 UpdateCsvRow |
Pick path: digits check, candidate allow-list, DiscogsAllowed, fetch, resolved |
WR-02 lock. Success now writes the draft |
csv_import_service.go:825-882 CommitCsvImport |
CAS UPDATE ... WHERE id = ? AND status = ? preview→importing, then Dispatch CsvImportArgs |
Unchanged (reference CAS) |
csv_import_service.go:889-901 CancelCsvImport |
Cancels MatchJobID/ImportJobID from the stale read |
WR-02 lock |
classes/wishlist_notifications.go:130-153 EnqueueWishlistDigest |
Upsert golem15_fonoteka_wishlist_digest_queue with RETURNING (xmax = 0), Dispatch WishlistDigestArgs with Delay: WishlistDigestDelay only on first insert |
Worker only |
classes/ai_config_resolver.go:40-45 |
var AdminVisionModel = func(ctx context.Context) (*AIConfig, error) { return nil, nil }; AIConfig{Adapter, APIKey, BaseURL, Model} all json:"-"; comment: "the outbound client that consumes it (Phase 14) owns the SSRF guard PHP applies" |
Replace from the golem plugin at Boot. Add a trust marker (for example Trusted bool \json:"-"``) so the admin tier selects trusted mode (D-05) |
classes/gates.go:60-90 AIAllowed |
Admin branch calls AdminVisionModel(ctx) |
Lights up automatically |
classes/gates.go:136 ResolveDiscogsConfig, :177 DiscogsAllowed, :188 MarketCurrency |
Ported tiers | Reuse in the client forUser |
classes/cover_importer.go:34-94 |
Fetch seam; fetchguard.Fetch with AllowHostsMode |
Reuse for the host-locked cover fetch in AlbumCoverFetcher and the applicator |
classes/manual_cover_fetcher.go:84 |
Fetch func(ctx, rawURL, policy) seam pattern |
Pattern precedent for injectable fetchers |
classes/public_share.go PubfailCounter |
In-memory per-process limiter with an injected clock | Precedent: in-request inbound limiters stay in memory |
fonoteka/search.go:19-47 |
settingsGate reads search_use_typesense; wireSearch installs the gate |
Reuse in reindex's "configured" check |
fonoteka/schedule.go:13-17 |
{Command: "fonoteka:prune-notifications", Cadence: pact.Daily()} |
Only the command is missing |
fonoteka/console/oauth_client.go |
bonfire.Command{Name, Description, Args, Flags (Bare), Run} pattern |
Template for the two new commands |
controllers/albums_admin_controller.go:57-90 |
Phase 10.1 admin discogsLookup/discogsSync stubs, documented as "Phase 14 replaces" |
Not in CONTEXT; see Open Question 5 |
routes_table_phase13_test.go:270-282 phase14-routes-absent |
Asserts the Phase 14 routes are not mounted | Update as routes land |
sm-user-plugin controllers/api_controller.go:795-806 |
"feedback_widget_hidden": false default, then app.Events.Collect(ctx, &classes.GetApiArrayEvent{User}) merges listener keys |
Feedback plugin listens (fonoteka's own listener at plugin.go:93 is the template) |
fonoteka/plugins/golem15/fonoteka/config/config.yaml:22-36 |
discogs.token, market_currency, max_covers, cover_max_bytes, cover_timeout_seconds, cover_host_suffix |
Missing: user_agent, rate_threshold, wait_budget_seconds, retry_after_fallback_seconds. Add them under golem15.fonoteka.discogs.*. Keep base_uri a code constant (PHP comment: "literal, never env()/user input") |
fonoteka.go/config/http.yaml |
CORS paths already include _feedback/api/* |
Nothing to add |
The three PHP jobs (port targets)
AlbumCsvMatchJob (jobs/AlbumCsvMatchJob.php) [VERIFIED: read this session]
public int $timeout = 240;(line 36), so the Go job getsconga.Timeout(240 * time.Second).private const MAX_CANDIDATES = 10;(line 38).- Flow: import gone →
failJob {"error":"import_not_found"}.checkIfCanceled→cancelJob(Go:StopJob).isTerminalOrSuperseded(status in canceled/done/importing/preview/failed, ormatchingwith a differentmatch_job_id) →completeJob {"skipped": "<status string>"}. Then status=matching,startJob(total=row_count),updateJobState(alreadyDone)when resuming, per pending row (ordered byrow_index): cancel check →matchRow→updateJobState(done). End: statuspreview,error_message=null,completeJob {"matched": done}. matchRow: duplicate in the collection →matched_csv, candidates[],matched_album_id. Discogs gate off →matched_csv,[]. ElsesearchByQuery("artist title"), thensearchByBarcodewhen empty. 0 results →matched,[]. 1 result →getRelease+mapReleaseintodraft_json, candidates[mapSearchResult],matched. More → the first 10mapSearchResult,draft_json=null,matched_ambiguous.DiscogsRateLimitException→pauseAndReschedule:delay = max(e.retryAfterSeconds, limiter.forToken(token).secondsUntilAvailable()), pending count, status staysmatching, Dispatch a new match job (labelfonoteka.csv.match, countpending, metadata{csv_import_id, resumed_after_rate_limit: true, retry_after: delay}, delay), writematch_job_id = next, thencompleteJob {"paused":"discogs_rate_limited","retry_after":delay,"next_job_id":next}. The$reschedulingstatic guard exists only for Laravel's sync driver; Go does not need it.- Any other throwable → status
failed,error_message,failJob {"error": msg}and return normally (no retry). In Go, callFailJoband return nil, or River retries up to 3 times.
AlbumCsvImportJob (jobs/AlbumCsvImportJob.php) [VERIFIED]
- No Discogs network.
importerCanWrite(Collection::accessibleBy(user)->whereKey(collection_id)) before start and per row; on failure statuscanceled+cancelJob. Rows notwritten/skipped, ordered byrow_index;startJob(count). - Per row inside
Album::withoutBroadcasting: canonical rows (matched_csvwith rawid) →CsvCanonicalIdMatcher::match, thenapplyCsvOverwrite/applyCsvFillorcreateCsv. Other rows →inputForRow(draft wins over CSV-mapped fields when there is a draft and the row is selected/matched/resolved;genre→genre_idviaresolveGenreId),writeOptions(allow_null_format, first cover URL +import_covers),matcher->find→fillEmptyorcreate,syncCsvRating. Per-row exception → rowerror,error_code=write_failed. - End: status
done. WhenwrittenCount > 0, publishcollection:{id}eventcollection.bulk_updated{"reason":"csv_import","count":N}.completeJob {"written": done}. Outer throwable →failed+failJob. - Go gap:
album_write_service.gohasCreateAlbum/UpdateAlbum/SaveAlbumbut no CSV variants. PortfillEmpty,applyCsvFill,createCsv,applyCsvOverwrite,resolveGenreId,syncCsvRatingandcreate($options)(PHPAlbumWriteService.php:64-373).classes/csv/canonical_id.goonly hasCanonicalID(row); the scoped matcher query (CsvCanonicalIdMatcher::match) also needs a port. Bulk-broadcast suppression exists in lighthouse (Phase 11 SC4).
WishlistDigestJob (jobs/WishlistDigestJob.php, 49 lines) [VERIFIED]
- Read queue row (user_id, collection_id). Missing or
item_count <= 0→completeJob {"skipped": true}. Else delete the row, thenNotificationService::mailWishlistDigest(subscriber, wishlist, count), thencompleteJob {"sent": count}. mailWishlistDigest(NotificationService.php:204-232): wishlist must exist withkind === 'wishlist'and subscriber must exist. Localeenwhenpreferred_locale === 'en', elsepl. Viewgolem15.fonoteka::mail.wishlist_subscription_digest(-en). VarsownerName(wishlist owner name),itemCount,wishlistName. To: subscriber email. Copyviews/mail/wishlist_subscription_digest.htmand-en.htm(absent in Goviews/mail) and register them inmail.go.- Order matters: PHP deletes the row before sending. A failed send after the delete loses that digest. Keep the order for parity, and put the delete and the
CompleteJobin one transaction where possible.
Discogs client (INTG-01)
DiscogsClient.php [VERIFIED]: TIMEOUT_SECONDS = 10. Headers Authorization: Discogs token=<token>, User-Agent: config('fonoteka.discogs.user_agent'), Accept: application/json. URL = rtrim(base_uri,'/') . path, with ids cast to int in the path and free text only in the query. Endpoints: getRelease(id) → /releases/{id}?curr_abbr=<marketCurrency>; getPriceSuggestions(id) → /marketplace/price_suggestions/{id} (an empty object is normal); getIdentity() → /oauth/identity; getMasterVersions(id) → /masters/{id}/versions?per_page=10; searchByBarcode → /database/search?barcode=&type=release; searchByQuery → /database/search?q=&type=release.
Request loop: limiter->acquire() once, then loop. Each response → syncFromHeaders(X-Discogs-Ratelimit-Remaining). 200 → json; 404 → null; 401/403 → DiscogsTokenRejectedException; 429 → wait = registerRetryAfter(Retry-After); if wait > remaining budget throw DiscogsRateLimitException(max(1, wait)), else sleep and retry. Any other status → ApplicationException("Discogs request failed with HTTP status N."). Logs carry only status and path.
Config values [VERIFIED: plugins/golem15/fonoteka/config/fonoteka.php:18-30]: 'base_uri' => 'https://api.discogs.com', 'user_agent' => env('DISCOGS_USER_AGENT', 'FonotekaApp/1.0 +https://github.com/golem15com/wn-fonoteka-plugin'), 'rate_threshold' => 50, 'wait_budget_seconds' => 15, 'retry_after_fallback_seconds' => 10.
Discogs itself throttles by source IP: 60/min authenticated, 25/min unauthenticated, as a 60 s moving average, with X-Discogs-Ratelimit, -Used and -Remaining headers [CITED: Discogs developer docs via search summary; the docs page returned 403 to direct fetch, LOW].
Domain classes to port alongside (PHP line counts): DiscogsMapper (434: mapRelease, mapSearchResult, mapMasterVersion), ReleaseMatchScorer (210, depends on completeness; Go completeness.go exists), AlbumReleaseApplicator (178, apply(album, mapped, overwriteAll, coverOnly, dryRun), uses CoverImporter), DiscogsImportResolver (209), DiscogsInputParser (133, static parse, normalizeBarcode, alternateBarcode), PriceSuggestionResolver (88), AlbumCoverFetcher (575). The PHP suite has unit tests for every one of these (tests/unit/Discogs*Test.php, ReleaseMatchScorerTest.php, PriceSuggestionResolverTest.php, AlbumReleaseApplicator*Test.php, CoverImporterTest.php, DiscogsRateLimiterTest.php, DiscogsClientTest.php). Use them as test vectors. The pure classes (DiscogsMapper, DiscogsInputParser, ReleaseMatchScorer, PriceSuggestionResolver) suit PHP-generated truth tables, the precedent being parity/csv_truth_tables.php, which runs the PHP classes directly and writes testdata/php_*.json.
Inbound (per-route) limiters — separate from the outbound Discogs limiter
| Key (PHP) | Limit | Response when over | Routes |
|---|---|---|---|
fonoteka-discogs-missing: + user id or IP [VERIFIED: AlbumReleaseMatchController.php:31] |
tooManyAttempts(key, 60), hit(key, 60) |
429 {"result":"error","code":"too_many_requests","retry_after":availableIn} |
albums match/apply (3 routes) and wishlist match/apply (same key, shared on purpose) |
fonoteka-discogs-import: + id [VERIFIED: DiscogsImportController.php:68] |
20 per 60 s | 429 {"result":"error","code":"too_many_requests"} (no retry_after) |
albums/import/discogs (checked after validation) |
fonoteka-recognize: + id [VERIFIED: RecognizeApiController.php:53-57] |
10 per 60 s | 429 {"error":"Too many requests"} |
recognize, both groups (after AiGate, before validation) |
route throttle:12,1 |
12/min | surf throttle 429 | token cover-price |
Use one surf.MemoryStore per plugin (process memory, precedent PubfailCounter). Attempt matches Laravel's tooManyAttempts+hit pair, and its retryAfter is availableIn. Order of checks per route is parity-visible. Copy it exactly (match/apply: album scope 404 → gate 503 → limiter → validation; import: gate → validation → limiter; recognize: AiGate 403 → limiter → validation → image guard).
D-17: where the Discogs limiter state lives — Postgres, UNLOGGED table, single atomic statement
PHP behaviour [VERIFIED: DiscogsRateLimiter.php:23-233]: a fixed window (WINDOW_SECONDS = 60) per bucket. Bucket id = substr(hash_hmac('sha256', token, app.key), 0, 32), never logged. Cache keys discogs:ratelimit:<bucket>:{count,window_start,lock}. Every read-modify-write runs under Cache::lock(..., 5)->block(3, ...) "regardless of driver" so that two near-simultaneous workers cannot both pass. acquire() loops: when a slot is free it increments and returns. Otherwise, if the wait exceeds the remaining wait budget (wait_budget_seconds, 15) it throws DiscogsRateLimitException(max(1, windowRemainder)), else it sleeps. secondsUntilAvailable() is read-only (1 when free). syncFromHeaders() only tightens: implied = max(0, threshold - Remaining) raises count when a window is active. registerRetryAfter() returns the numeric Retry-After or the fallback (10).
Decision: Postgres. Reasons:
- The cross-process guarantee is still needed. conga supports
queue.work_in_serve: falsewith a separatefonoteka queue:workprocess (systemd), which is documented in the app'sconfig/queue.yamland the conga README [VERIFIED: fonoteka.go/config/queue.yaml; modules/conga/README.md "Configuration"]. Rolling restarts also briefly run two binaries. A process-memory limiter would give each process its own 50/min. Discogs throttles by source IP at 60/min, so two processes on one host could overshoot by up to 40/min, and the CSV match job would degrade to 429 handling. - The cost is negligible: one short statement per Discogs request, which itself takes about 100-500 ms and is capped at 50/min per token.
- UNLOGGED gives cache semantics (no WAL, emptied after a crash), the same as PHP's file/redis cache. A lost window only means one fresh window.
- One atomic
INSERT ... ON CONFLICT DO UPDATE ... WHERE ... RETURNINGreplaces PHP's lock. Postgres guarantees an atomic insert-or-update under concurrency, and a row "locked but not updated because anON CONFLICT DO UPDATE ... WHEREcondition was not satisfied ... will not be returned" [CITED: postgresql.org/docs/current/sql-insert.html]. So "a row returned" means "slot granted", with no advisory lock and no explicit transaction. - Clock injection (D-16) still works: pass
nowfrom the injected clock as a parameter instead of SQLnow(), so tests drive the window with a fake clock. The unit tests of the waiting logic use an in-memoryRateStorefake. One testcontainers test proves the SQL.
Shape (migration in the fonoteka plugin; the table is Go-only, so add it to parity/schema_diff_test.go's Go-only allow-list as golem15_fonoteka_settings was) [ASSUMED design]:
CREATE UNLOGGED TABLE golem15_fonoteka_discogs_rate_windows (
bucket TEXT PRIMARY KEY, -- 32-hex HMAC id, never the token
window_start TIMESTAMPTZ NOT NULL,
hits INTEGER NOT NULL
);
-- tryAcquire($1 bucket, $2 now, $3 threshold): a returned row = granted
INSERT INTO golem15_fonoteka_discogs_rate_windows AS w (bucket, window_start, hits)
VALUES ($1, $2, 1)
ON CONFLICT (bucket) DO UPDATE SET
window_start = CASE WHEN $2 - w.window_start >= interval '60 seconds' THEN $2 ELSE w.window_start END,
hits = CASE WHEN $2 - w.window_start >= interval '60 seconds' THEN 1 ELSE w.hits + 1 END
WHERE $2 - w.window_start >= interval '60 seconds' OR w.hits < $3
RETURNING hits;
-- not granted: SELECT window_start → wait = max(1, 60 - floor(now - window_start))
-- syncFromHeaders: UPDATE ... SET hits = GREATEST(hits, $implied) WHERE bucket = $1 AND $now - window_start < interval '60 seconds'
Keep the HMAC key as app.key (the bucket id is not shared with PHP, so byte-compatibility does not matter, but secrecy does). The exception's retryAfterSeconds is the window remainder, not the leftover budget, because the CSV match job uses it as the re-dispatch delay.
D-05: base-URL guard split, confirmed against PHP with differences
PHP SSRFGuard::assertSafeUrl [VERIFIED: plugins/golem15/golem/classes/security/SSRFGuard.php:40-67, config/ssrf.php:16-20]:
parse_urlmust give scheme and host, elseRuntimeException('Invalid URL').- Scheme must be
https(Only https:// scheme allowed). - Host allowlist
config('golem15.golem::ssrf.allowed_hosts'). A leading.is a suffix match; otherwise exact, case-insensitive. Default:'oaidalleapiprodscus.blob.core.windows.net'and'.openai.com', overridable byGOLEM15_SSRF_ALLOWED_HOSTS(comma list). Failure:Host not in allowlist: <host>. - Resolve A/AAAA (fallback
gethostbynamel). No answer →Cannot resolve host. Any private IP →Resolves to private/loopback IP(127/8, 10/8, 172.16/12, 192.168/16, 169.254/16, 0/8, ::1, fe80::/10, fc00::/7, plusfilter_var NO_PRIV_RANGE|NO_RES_RANGE).
Callers [VERIFIED by grep + read]: UserAiConfig::fromCredential (line 43), OrgAiConfig::fromOrg (line 42) and AiCredentialController::resolveTestConfig (line 150, inline body base_url). All of them apply it only when a base_url override is present; the default provider URLs are never checked. Plugin.php:82 also guards the AI-returned DALL-E image URL before File::fromUrl. Admin Golem Settings models go through AIService::send → RequestSender::sendPostRequest, which has no URL validation (validateUrl runs only on GET/download) [VERIFIED: apparatus/classes/RequestSender.php:37-57, 123-169].
The split matches D-05 (admin trusted, user and org guarded). Differences to carry into the plan:
| # | PHP | D-05 as written | Recommendation |
|---|---|---|---|
| 1 | User/org guard is https-only + host allowlist + resolve-time IP check. The allowlist is the primary defence (the class comment says so) | "private/reserved-IP dial guard" only | Port the allowlist (config key golem15.golem.ssrf.allowed_hosts in sm-golem-plugin, same default, env override) and keep the dial-time guard. Without the allowlist Go accepts https://api.groq.com, which PHP rejects: a parity and security regression |
| 2 | With the default allowlist, any user/org Claude credential with an explicit base_url (even https://api.anthropic.com/v1) is rejected, because .anthropic.com is not listed. Unknown whether production sets GOLEM15_SSRF_ALLOWED_HOSTS (.env is secret-guarded) |
— | Open Question 1 |
| 3 | Guard failure throws RuntimeException. AiCredentialController::test catches only ApplicationException, and recognize calls the resolver outside its try, so PHP answers the Winter 500 HTML page (the recorded POST ai-credential __bad-base-url fixture shows that page for store) |
— | Go must return the same 500 page (controllers/api/winter_500.html exists) on guard failure, not {ok:false}. Record cases (D-11 list) |
| 4 | PHP's IP check is resolve-time (DNS-rebind TOCTOU is acknowledged in the class doc) | dial-time | Go's dial-time check is strictly stronger and invisible to parity |
| 5 | RequestSender POST sets CURLOPT_FOLLOWLOCATION and no timeout |
— | Guarded mode: never follow redirects (or re-guard each hop). Trusted mode: follow is acceptable. Set an explicit AI timeout of 120 s (PHP's sendStream uses CURLOPT_TIMEOUT 120) [ASSUMED value for non-stream] |
| 6 | generateImage returns a URL that callers must assertSafeUrl before fetching |
— | Port AssertSafeURL as an exported helper in sm-golem-plugin. Fetch any AI-returned URL only via fetchguard AllowHosts mode with the same allowlist |
D-11: missing parity cases and their upstream sidecars
All 14 pending routes already have one fixture each, and every one is a negative path recorded with DISCOGS_TOKEN= empty and no AI configured [VERIFIED: parity/manifest.yaml pending entries; fixture bodies read]:
| Route (pending) | Recorded today | Notes |
|---|---|---|
POST wishlist/albums/{id}/match jwt |
404 {"error":"Album not found"} |
|
POST wishlist/albums/{id}/apply-release jwt |
404 | |
POST albums/match jwt |
503 discogs_disabled |
|
POST albums/{id}/match jwt |
404 | |
POST albums/{id}/apply-release jwt |
404 | |
POST albums/recognize jwt |
403 {"error":"AI features not available"} |
Manifest says status: 422, the fixture says 403. Fix the manifest when flipping to ported |
POST albums/import/discogs jwt |
503 discogs_disabled |
request body {"query":...} (the real field is input) |
POST ai-credential/test jwt |
200 {"ok":false,"error":"Incorrect API key provided: sk-parit******real. ..."} |
Live OpenAI call, no sidecar. Re-record |
POST discogs-credential/test jwt |
200 {"ok":false,"error":"Token Discogs jest nieprawidłowy lub wygasł."} |
Live Discogs 401, no sidecar. Re-record |
POST /api/v1/fonoteka/albums/recognize token |
422 photo required | |
POST /api/v1/fonoteka/albums/{id}/cover-price/discogs token |
200 {"fetched":false,"reason":"no_source"} |
|
PATCH import/csv/{id}/rows/{rowId} (ported route) |
discogs-off 422 exists; no success pick |
Cases to record against PHP (each with an upstream sidecar unless marked "none"), from routes.php, the Nuxt store (app/stores/fonoteka.ts:315-440, 601-620, 1011, 1156) and the MCP client (fonoteka-mcp/src/client.ts:277-337):
| # | Route | Case | Upstream sidecar |
|---|---|---|---|
| 1 | PATCH import/csv/{id}/rows/{rowId} |
pick success → 200 resolved + draft (Nuxt sends {selected_discogs_id: "<id>"}) |
Discogs GET /releases/{id}?curr_abbr=EUR 200 |
| 2 | same | pick rate-limited → 422 discogs_rate_limited |
Discogs 429 (+/- Retry-After) |
| 3 | same | release 404 → 422 discogs_unavailable |
Discogs 404 |
| 4 | POST albums/{id}/match |
200 candidates with q, year, year_delta, medium |
/database/search?q= 200 |
| 5 | same | 422 validation (missing q) |
none (stops before upstream; needs a Discogs-enabled user) |
| 6 | same | 429 too_many_requests inbound (61st call) |
none, or 60 sidecar entries; prefer a Go unit test |
| 7 | same | 502 discogs_token_rejected |
Discogs 401 |
| 8 | same | 429 discogs_rate_limited (budget exhausted) |
Discogs 429 with Retry-After > 15 |
| 9 | POST albums/match |
200 candidates against draft |
search 200 |
| 10 | POST albums/{id}/apply-release |
200 {data, filled, remaining, draft}, dry_run:true, cover_only:true, overwrite_all (Nuxt MatchReleaseDialog sends all three) |
/releases/{id} 200 + cover image GET (i.discogs.com) |
| 11 | same | 200 discogs_no_match (release 404) |
Discogs 404 |
| 12 | POST wishlist/albums/{id}/match, .../apply-release |
200 success twins (no dry_run; response has no draft key) |
as 4 / 10 |
| 13 | POST albums/import/discogs |
draft (release URL/id), candidates (barcode with many hits), no_match (barcode, zero hits: 200 with barcode), 422 validation, 429 inbound |
search and/or release, master versions |
| 14 | POST /api/v1/fonoteka/albums/{id}/cover-price/discogs |
fetched:true (discogs_id path, cover + price suggestion), nothing_missing, ambiguous, rate_limited, refresh_price:true (MCP sends {refresh_price:true} or {}) |
release, price_suggestions, cover GET |
| 15 | POST discogs-credential/test |
{token} body ok → {"ok":true}; stored token rejected (re-record); rate limited; disabled (no body, gate off: no upstream) |
/oauth/identity 200/401/429 |
| 16 | POST ai-credential/test |
inline {provider:"claude", api_key, model} ok; inline OpenAI 401 (re-record the existing case); stored credential; unsafe base_url → 500 Winter page (no upstream); non-allowlisted host → 500; no credential → {"ok":false,"error":"No AI credential configured."} |
Anthropic POST /v1/messages, OpenAI POST /v1/chat/completions |
| 17 | POST albums/recognize jwt |
200 albums (BYOK user), 200 {"albums":[],"code":"recognition_truncated"} (both responses non-JSON, retry stop_reason:max_tokens/finish_reason:length), 200 {"albums":[]} (unparseable twice, not truncated), 502 provider error, 422 non-image bytes (ImageContentGuard), 429 inbound, admin-tier with global vision model |
1-2 AI exchanges per case |
| 18 | token albums/recognize |
200 success with MCP multipart (photo filename cover, locale) |
AI exchange |
| 19 | feedback (new manifest section) | GET {key}/config 200 / 404 bad key / 403 origin; POST {key}/submit 202 / 422 / 422 bad image / 403; OPTIONS 204; PUT me/hidden 200 / 422 |
none at request time. G15Office exchanges are job-side: record them as a job sidecar (create task + attachment) and assert from the River worker test |
| 20 | CSV flow | match job paused on 429 and resumed; import job writing rows | Discogs exchanges; job rows golden (summer_jobs) |
How sidecars fit the existing layout [VERIFIED: parity/README.md, php_parity.sh, parity_test.go:233-297, fixtures/routes/*.yaml]:
- Fixtures are tide flow YAML (
version: 1,steps[]ofrequest/response/normalize). Route cases live underfixtures/routes/<METHOD>_<path>_<group>__<case>.yaml, flows underfixtures/nuxt|mcp, and DB goldens next to them as*.rows.json(precedent:nuxt-csv.rows.json, compared infonoteka_flows_test.go:107-155). - Proposed sidecar:
<fixture>.upstream.yamlnext to each fixture, holding an ordered list of exchanges{step, host, request:{method, path, query, headers (allow-listed: User-Agent, Accept, Content-Type, anthropic-version, anthropic-beta; auth headers masked to a placeholder), body (JSON-normalized; base64 image replaced by a sha256)}, response:{status, headers (Retry-After, X-Discogs-Ratelimit-*), body}}. tide loads it with the flow.replayPortedRouteserves it through an in-process fake (aRoundTripperor an httptest server) that asserts each Go request against the recorded one and fails on unconsumed or extra exchanges. - Recording PHP's side: PHP's Discogs base URI is a code literal (
'base_uri' => 'https://api.discogs.com', no env), and the AI path is raw curl. So the only way to capture what PHP really sends, without editing PHP, is a recording HTTPS proxy. Guzzle honoursHTTPS_PROXYin any SAPI and libcurl honourshttps_proxy/HTTPS_PROXY[ASSUMED: Guzzle/libcurl env-proxy behaviour]. Add asummer parity:upstreamcommand to tide that (a) acts as a CONNECT proxy, (b) terminates TLS with a locally generated CA (stdlibcrypto/x509), (c) in script mode answers from a hand-authored vendor-response file (deterministic, no real tokens), or in forward mode passes through to the real vendor once, and (d) writes the sidecar. Extendphp_parity.sh servewithHTTPS_PROXY=http://127.0.0.1:8425and-d curl.cainfo=<parity CA>(the Centrifugo recorder on127.0.0.1:8424is the precedent). Seed fake BYOK credentials (Discogs token matching^[A-Za-z0-9_\-]{10,255}$, AI keysk-parity-…) inparity/fonoteka_reset.php.check_corpus --check-secretsmust stay green, so mask the auth headers in sidecars. - If the proxy is judged too large, the fallback is: sidecars authored by hand from vendor docs, and the request assertion compares against a request derived from the PHP source rather than a recorded one. This weakens D-15 and needs user sign-off.
WR-02 fix (D-10)
[VERIFIED: 13-REVIEW.md:131-158; csv_import_service.go:519-587, 639-705, 889-901]. In one lagoon.Transaction, SELECT ... FOR UPDATE the import row (clause.Locking{Strength: "UPDATE"}), re-check csvBeforeCommit(cur.Status) on the locked row, then cancel cur.MatchJobID/cur.ImportJobID from the locked row, write, and dispatch. Apply this to UpdateCsvMapping, UpdateCsvRow (the save closure) and CancelCsvImport. UpdateCsvRow's Discogs fetch (up to 15 s wait budget) must happen outside the lock: fetch first, then lock, re-check status, and save. Otherwise a slow Discogs call holds the row lock and blocks commit. Add the review's interleaving test (commit vs mapping: at most one import job). Recommended, not mandated by D-10: make the match/import workers' own status writes (matching, preview, done, canceled) conditional on the locked row too. PHP's worker can otherwise resurrect a canceled import to matching. Response shapes are unaffected.
jobs.CancelJob runs on its own connection while the transaction holds the csv_imports row lock. It touches only summer_jobs and river_job rows, so no deadlock [ASSUMED: no FK or trigger from those tables back to csv_imports].
fetchguard: what the guarded client needs
Current API [VERIFIED: modules/fetchguard/fetch.go, policy.go]: Fetch(ctx, rawURL, Policy, *compass.Config) (*Result, error), GET only. Policy{Mode (AllowHostsMode|PublicOnlyMode), AllowHosts, MaxBytes, Timeout} plus unexported tlsConfig and skipReservedCheck test hooks, reachable only from the package's own tests (withTestLoopback). Redirects are never followed. Proxy: nil. The dial Control rejects reserved IPs (including NAT64/6to4-embedded v4 and zoned v6). Typed Error{Reason} with reasons invalid_url|scheme|unresolvable|private_ip|network_error|too_large.
Needed (design [ASSUMED], to be fixed in the plan):
fetchguard.NewClient(policy ClientPolicy, cfg *compass.Config) *Client(or*http.Clientplus helpers) withDo(req), plus helpersPostJSON(ctx, url, headers, v),PutJSON,PostMultipart(ctx, url, headers, fields, file{field, name, mime, reader}), and aBearer(token)header helper. Responses are capped atMaxBytes, typed errors are kept, and the status code is returned, not judged (PHP callers inspect bodies, not status).- Modes:
AllowHostsMode,PublicOnlyMode, andTrustedMode(no dial guard, no host check, http allowed, for admin-configured endpoints such as LAN Ollama). - Test transport seam that production input cannot reach: for example
fetchguard.WithTransport(ctx, rt)or aClientoption set only by code, which the parity harness uses to routeapi.discogs.com/api.anthropic.com/api.openai.com/G15Office to the sidecar fake while keeping the original host in the asserted request. Production URLs stay code literals (Discogsbase_uri). - Request bodies (a base64 photo of up to about 14 MB) are not capped by fetchguard. Set the timeout per consumer: Discogs 10 s (PHP
TIMEOUT_SECONDS = 10), AI 120 s [ASSUMED], G15Office 30 s [ASSUMED; PHP has none]. - Update
modules/fetchguard/README.md,docs/services/outbound-http.md, plus mentions indocs/architecture/introduction.mdanddocs/setup/coming-from-wintercms.md(RequestSender is now covered) [VERIFIED: grep docs].
Redacting slog handler
PHP RedactCredentialsTap [VERIFIED: apparatus/classes/logging/RedactCredentialsTap.php]: keys api_key, apikey, authorization, bearer, password, secret, token, webhook_secret, admin_password, OPENAI_API_KEY, ANTHROPIC_API_KEY, PERPLEXITY_API_KEY (the todo omits the last three; include them, the comparison is case-insensitive). Values become [REDACTED] recursively; string values are also scrubbed with the patterns /Bearer\s+[A-Za-z0-9._\-+\/=]+/i → 'Bearer [REDACTED]', /sk-[A-Za-z0-9]{20,}/ → 'sk-[REDACTED]', /x-api-key:\s*[^\s,]+/i → 'x-api-key: [REDACTED]'. Go: a slog.Handler wrapper that rewrites the record message and every Attr (recursing into slog.KindGroup and handling WithAttrs/WithGroup). Nothing publishes a *slog.Logger today: beachcomber, lighthouse, postcard, conga and flare all do app.Lookup[*slog.Logger]() then fall back to slog.Default() [VERIFIED: grep]. So install it via slog.SetDefault(slog.New(redact.Wrap(base))) in the serve, queue:work and schedule:run entry points (or have backpack publish one). Host it in an existing module or a new small module; a new module needs a README and a root-table row. SafeExceptionResponse: surf already has recoverJSON/recoverBare panic recovery [VERIFIED: modules/surf/router.go:435-443, 638]. Whether it hides messages outside debug is unverified. A test in the plan should pin it.
Golem plugin surface (sm-golem-plugin)
[VERIFIED: plugins/golem15/golem/*, read this session]
AIService::send(Prompt, ?config): config defaults toSettings::getDefaultModel(). No config →failure('No AI model configured. Please add a model in Settings > AI.'). Empty key →failure('API key is not configured for the selected model.'). Model defaults togpt-4o.applyModelSystemPromptsets the model'ssystem_promptonly when the prompt has none. POST JSON (adapter headers).false→failure('Failed to connect to AI service.'). Invalid JSON →failure('Invalid response from AI service: <json error>', ['raw'=>body]). Otherwiseadapter->parseChatResponse(status code ignored). Exceptions go throughsafeExceptionMessage("Internal server error" outside debug).- Anthropic adapter [VERIFIED: AnthropicAdapter.php:13-15, 23, 59-61]: headers
x-api-key: <key>,anthropic-version: 2023-06-01,anthropic-beta: files-api-2025-04-14. Endpointrtrim(base_url) . '/messages'. Payload{model: prompt.model ?? modelId, messages[], system?, max_tokens: options.max_tokens ?? options.max_completion_tokens ?? 4096, ...other options except response_format and max_completion_tokens}. Image blocks:data:URL →{type:image, source:{type:base64, media_type, data}}, else{source:{type:url}}; file blocks →document/file_id. Response:error.message→ failure;content[0].text; usage mapped to prompt/completion/total;raw= the whole body (recognition readsraw.stop_reason). - OpenAI adapter [VERIFIED: OpenAIAdapter.php:13, 21, 50-62]:
Authorization: Bearer <key>;/chat/completions;toApiPayload(system message first) minusresponse_format;max_tokens→max_completion_tokens;reasoning_effort: lowfor/^(o\d|gpt-5)/iunless set. Response:choices[0].message.content;raw.choices[0].finish_reason. - Content-Type: RequestSender adds
Content-Type: application/jsonfirst; adapters append their headers. PHP json_encode escapes/and non-ASCII (ł) unless flagged. Upstream request-body parity: compare JSON semantically in the sidecar assertion, not byte-for-byte. Settings[VERIFIED: models/Settings.php:13, 19-118]:$settingsCode = 'golem_settings'(notgolem15_golem_settingsas D-01 states).getModels()=array_filter(models, is_enabled), which preserves keys, sogetDefaultModel()'sreturn $models[0] ?? nullfallback returns null when model 0 is disabled even if others are enabled. Port the quirk.getVisionModel()= first enabled withaccepts_images; alsogetImageModel(generates_images),getModelByName,getFileModel(has_files_endpoint).fields.yamlrepeater fields:name, adapter (openai|anthropic, default openai), api_key (text, password attribute), base_url (default https://api.openai.com/v1), model, system_prompt, is_enabled, is_default, generates_images, accepts_images, has_files_endpoint, max_completion_tokens (default 4096).api_keyis stored in plaintext: Settings has no encryption and the only encryption migration targets FaceSettingsadmin_password[VERIFIED: updates/version.yaml, encrypt_face_settings.php].- Recognition (
AlbumRecognitionService.php):MAX_ALBUMS = 30,MAX_COMPLETION_TOKENS = 8192. Admin path with no config and no vision model →ApplicationException('No vision model configured in Golem AI settings.'). System prompt text (lines 237-292) with formats, genre hint (up to 20 distinct genres via albums in the collection, newest first) and language directive (en/plfrom BCP-47). User message with the image. Strip fences, json_decode. On failure, one retry with the appended "IMPORTANT: Respond with raw JSON only…". A retry parse failure with finish reasonlength|max_tokens→RecognitionTruncatedException(controller → 200{"albums":[],"code":"recognition_truncated"}), else[]. Normalisation: drop nameless rows; nullableString fields; year 1889-2100; format must be inAlbum::FORMATS; tracklist viaTracklistTextParser(Gotracklist_text_parser.goexists). - Not ported: FaceService, CompreFaceClient, ChatContextCollector, Conversations/Messages/Chat components (D-04).
routes.phphas no routes. - Settings storage gap (needs a decision, Open Question 2): cabana settings are typed singleton rows (id=1), scalar fields only.
nestedValueis rejected, there is norepeatertype informFieldTypes, and unknown YAML keys fail boot [VERIFIED: cabana/settings.go:66-95, form_schema.go:22-44]. The project convention (Phase 5, user-resolved for fonoteka) is a dedicated typed table, notsystem_settings. Recommendation: tablegolem15_golem_models(one row per repeater item,sort_order,api_key lagoon.Encrypted json:"-"), edited through a cabana list+form controller under Settings › AI, plus an idempotent importer for the PHPsystem_settingsrowitem='golem_settings'(a migration that runs only whensystem_settingsexists, or a cutover command). This also reconciles D-03's "encrypted" with PHP's plaintext.
Feedback plugin surface (sm-feedback-plugin)
[VERIFIED: plugins/golem15/feedback/*, read this session]
- Routes: group
/_feedback/api/v1withjson.response:GET {key}/config(throttle:feedback-config= 60/min by IP),POST {key}/submit(throttle:feedback-submit= 10/min by IP),OPTIONS {any}→response('', 204)withwhere('any','.*'). A separate group withjwt.auth:PUT me/hidden. CORSpathsinclude_feedback/api/*in both PHP and Go configs. config:keyMatches=enabled && widget_key !== '' && hash_equals, else 404{"error":true,"message":"Not found"}. Origin gate: no Origin header → allowed; unparseable host → false; empty allow-list → fail closed 403{"error":true,"message":"Origin not allowed"}; host compared lowercase againstgetAllowedOriginHosts()(lines split on[\r\n]+, trimmed,parse_urlhost or the raw line, lowercased, unique). 200{"success":true,"data":{"position","allowHide","colors":{"primary","accent"},"labels":{"title","placeholder","success"}}}with?lang=enselecting_en, anything else_pl.submit: validation (message required|string|max:5000,type required|in:bug,feature,other,email nullable|email|max:255,page_url required|string|max:2000,user_agent nullable|max:500,console_log nullable|max:20000,screenshot nullable|image|mimes:jpg,jpeg,png,gif,webp|max:10240) → 422{"error":true,"message":"Validation failed","details":errors}. ImageContentGuard (finfo sniff in jpeg/png/gif/webp +getimagesize) → 422 withdetails.screenshot = ["The file is not a valid image."]. Create the submission, attach the screenshot (File,is_public = true, attachOnescreenshot), dispatchSyncFeedbackToG15Office, 202{"success":true}. Goclasses/image_guard.go(IsAllowedImage,SniffImageMIME) can be copied into the plugin (PHP keeps per-plugin copies on purpose).me/hidden:hidden required|boolean→ 422{"error":"Validation failed","errors":...};UserPreference::setWidgetHidden(updateOrCreate) → 200{"hidden":bool}.getApiArraylistener:feedback_widget_hidden => UserPreference::isWidgetHidden(id)(value ofhidden, false when no row).- Settings:
$settingsCode = 'golem15_feedback_settings'(line 14).initSettingsData:enabled=true, allow_hide=false, widget_key='wk_'.Str::random(32), position='bottom-right', g15office_task_priority='normal', and the pl/en label defaults. Rules are all nullable (positionin:bottom-right,bottom-left,middle-right,middle-left, priorityin:low,normal,high,urgent). Fields usecolorpickerandreadOnly, which cabana does not support (unknown keys fail boot). Adapt the Gofields.yaml(text fields,attributes: {readonly: …}) or add the types to cabana. Scalar-only, so a dedicated typed singleton table works with cabana settings.widget_keymust be generated on first creation, because Winter'sinstance()persists defaults on first access [ASSUMED Winter behaviour]. - Models and tables:
golem15_feedback_submissions(id, message text, type string default 'bug', email, page_url 2000, user_agent, console_log longtext, status default 'pending' indexed, g15office_task_id, g15office_error text, timestamps).golem15_feedback_user_preferences(id, user_id unique, hidden bool default false, timestamps). Status values pending/sent/failed. Type map bug→Bug, feature→Feature, other→Task. SyncFeedbackToG15Office:$tries = 3,$backoff = 30. Configfeedback.g15_office.{base_url, token, project}from envG15_OFFICE_BASE_URL/TOKEN/PROJECT.createTaskPOST JSON{base}/_support/api/v1/projects/{project}/taskswith Bearer,Accept: application/json, bodyarray_filter({title:"Feedback: "+Str::limit(collapsed message,80), description, type, status?, priority ?: 'normal'}). ThenattachFilemultipart POST{base}/_support/api/v1/tasks/{hashId}/attachments(fieldfile). Decode:false→ "connection error"; non-JSON → "invalid response body";errortruthy → message. Success: statussent, task id stored; failure: statusfailed,g15office_error, rethrow (retry). Description markdown format at lines 198-226. In Go:conga.MaxAttempts(3); River's backoff differs from 30 s (acceptable, not parity-visible) [ASSUMED].embed.js(643 lines) is served at/plugins/golem15/feedback/assets/js/embed.js, which the Nuxt proxy uses (nuxt.config.ts:254, 369). No framework capability serves public plugin assets [VERIFIED: pact/capabilities.go interface list], so the plugin registers a GET route serving the embedded file.- No admin submissions list exists in PHP (no controller). D-13's list is a net-new cabana list controller, which is fine but not parity-checked.
- Parity: feedback routes are outside fonoteka's
routes.php(154 ids). Follow theuserAPIRouteIDs/realtimeRouteIDsprecedent inparity/check_corpus.go:619-630androutes.snapshot(172 lines) by adding afeedbackRouteIDsset and an auth group.
Console commands
fonoteka:prune-notifications[VERIFIED: PruneNotifications.php:20-21]:Notification::where('created_at', '<', now()->subDays(90))->delete();then$this->info("Pruned {$deleted} notifications older than 90 days.");, exit SUCCESS. Go tablegolem15_fonoteka_notifications. Register inCommands()(plugin.go:267). The scheduler entry already exists.fonoteka:reindex {--drop-old-items-index}[VERIFIED: ReindexAlbums.php:16-17, 49-103]: not configured (scout.driver === typesense && Settings search_use_typesense && api_key) → errorTypesense reindex skipped: enable Fonoteka Typesense search and configure TYPESENSE_API_KEY.FAILURE. Any album withcollection_id IS NULL OR <= 0→Album reindex aborted: every active Album must have a positive collection_id.FAILURE. Thenscout:flushandscout:import(chunks of 500 [ASSUMED Scout default]). If the album table is empty, create the collection from the schema. Then searchq=*, query_by=name, filter_by=collection_id:=0, per_page=1;found != 0→Album reindex failed integrity check: collection_id:=0 documents exist.FAILURE. With--drop-old-items-index, deletegolem15_fonoteka_items(Deleted legacy Typesense collection: …or, on ObjectNotFound,Legacy Typesense collection already absent: …). Success messageAlbum index rebuilt; collection_id:=0 document count is zero.Go:beachcomber.From(app);Engine().Name(),Configured(),settingsGate{}.Enabled;Engine().Flush(index); batchedUpsert(index, schema, docs)viaalbum.ToSearchableArray;beachcomber.SearchPageforFound. Gap:Flushtreats a missing index as nil, so the "already absent" message cannot be told apart. Add an optionalbeachcomber.IndexDropper(DropIndex(ctx, index) (existed bool, err)) to the framework (README + docs), or accept a single message (Open Question 6). "Before" per SRCH-02 is the DB check in PHP; "after" is the Typesensefoundcheck.fonoteka:oauth-clientshipped in Phase 8 (console/oauth_client.go).
Submodule workflow for the two new repos
- Both remotes are reachable and empty:
git ls-remote git@git.golem15.com:golem15/sm-golem-plugin.gitand.../sm-feedback-plugin.gitexit 0 with no refs, while a nonexistent repo errors with "Cannot find repository" [VERIFIED: probe this session]. Neither is mounted yet;.gitmoduleslists onlyplugins/golem15/user[VERIFIED]. git submodule addof an empty remote cannot stage a gitlink (there is no commit to check out) [ASSUMED git behaviour]. Recommended sequence:git initthe plugin directory atplugins/golem15/golem, write and commit the initial module there,git remote add origin …, pushmaster(a user action in prior phases; executors were told not to push, see 13-VERIFICATION human item), then in fonoteka.go rungit submodule add git@git.golem15.com:golem15/sm-golem-plugin.git plugins/golem15/golem(an existing repository at the path is staged without cloning). Commit.gitmodulesand the gitlink separately.- Wiring (precedent: sm-user-plugin): plugin
go.modwithmodule git.golem15.com/golem15/sm-golem-plugin,replace git.golem15.com/golem15/summercms => ../../../../summercms.go(feedback alsoreplace git.golem15.com/golem15/sm-user-plugin => ../user). Appgo.workuse ./plugins/golem15/golemand./plugins/golem15/feedback. Appgo.modrequire+replace … => ./plugins/golem15/<name>. The fonoteka plugin'sgo.modrequires sm-golem-plugin withreplace => ../golem.summer.yamladds idsgolem15.golemandgolem15.feedback(golem before fonoteka). Regenerateplugins.gen.go. - sm-user-plugin
masteris still 5 commits ahead of origin [VERIFIED:git status -sb], so a fresh clone cannot check out the pointer. That is unrelated to this phase but blocks CI clones. - README rules apply to plugin READMEs (never name the consuming application).
Architecture Patterns
System Architecture Diagram
Nuxt SPA / fonoteka-mcp / embed.js
│ HTTPS
▼
surf router ── group middleware (jwt | inv_token+scope | public throttle) ──► controllers/api
│ │
│ in-controller limiter (surf.MemoryStore) ◄────────────────────┤
│ ▼
│ classes: gates/resolvers (DiscogsAllowed, ResolveAIConfig)
│ │ │
│ ┌──────────────────────┘ └──────────────┐
│ ▼ ▼
│ Discogs client ──► rate limiter ──► Postgres UNLOGGED sm-golem-plugin AIService
│ │ (atomic upsert) │ adapter (anthropic|openai)
│ ▼ ▼
│ fetchguard.Client (guarded) ──dial guard──► api.discogs.com fetchguard.Client (guarded | trusted)
│ │ │
│ └──► CoverImporter (AllowHosts *.discogs.com) ──► storage │──► AI provider
▼
DB transaction ── conga.Dispatch ──► river_job + summer_jobs
│ LISTEN/NOTIFY
▼
River worker (serve or queue:work)
┌──────────────┬───────────────┬──────────────────┐
▼ ▼ ▼ ▼
CSV match job CSV import job digest job feedback G15Office job
(240 s; 429 → (write service, (delete row, (fetchguard POST JSON +
re-Dispatch bulk broadcast) postcard mail) multipart, 3 attempts)
with Delay)
Recommended layout (per .planning/notes/plugin-layout-winter-directories.md)
fonoteka/plugins/golem15/fonoteka/
├── classes/discogs/ # client.go, rate_limiter.go, mapper.go, scorer.go, applicator.go,
│ # import_resolver.go, input_parser.go, price_suggestion.go, cover_fetcher.go, errors.go
├── classes/recognition.go # AlbumRecognitionService port (+ truncated error)
├── controllers/api/ # release_match_controller.go, wishlist_release_match_controller.go,
│ # recognize_controller.go, discogs_import_controller.go, album_cover_fetch_controller.go
├── jobs.go (or jobs/) # csv match/import, digest workers
├── console/ # prune_notifications.go, reindex.go
└── updates/ # discogs rate windows UNLOGGED table
plugins/golem15/golem/ (sm-golem-plugin)
├── plugin.go routes.go(no routes) classes/{service.go, providers/*.go, prompt.go, response.go,
│ prompt_factory.go, ssrf_guard.go} models/{ai_model.go} controllers/ (admin list/form) updates/ lang/en
plugins/golem15/feedback/ (sm-feedback-plugin)
├── plugin.go routes.go controllers/api/{feedback_api_controller.go, me_hidden_controller.go}
├── classes/{g15office_client.go, image_guard.go} jobs/sync_g15office.go models/ updates/ lang/{en,pl}
└── assets/js/embed.js (embedded, served by a route)
models/ stays a leaf package (the rule is enforced by tooling).
Pattern 1: Worker with summer_jobs outcome semantics
What: Translate PHP JobManager calls one-to-one: startJob → m.StartJob(ctx, id, total), updateJobState → UpdateJobState, completeJob(meta) → CompleteJob, failJob(meta) → FailJob then return nil (PHP does not retry), cancelJob inside a job → StopJob, checkIfCanceled → CheckIfCanceled. conga.JobID(ctx) gives the row id.
// Source: modules/conga/README.md Usage + fonoteka jobs.go pattern
conga.Job(p.matchCsv, conga.OnQueue(classes.CsvMatchQueue), conga.Timeout(240*time.Second))
// self re-dispatch on rate limit (inside the job):
next, err := m.Dispatch(ctx, gdb, classes.CsvMatchArgs{CsvImportID: imp.ID}, conga.DispatchOpts{
Label: classes.CsvMatchLabel, Queue: classes.CsvMatchQueue, Count: pending,
Metadata: map[string]any{"csv_import_id": imp.ID, "resumed_after_rate_limit": true, "retry_after": delay},
Delay: time.Duration(delay) * time.Second,
})
Pattern 2: Injected clock (D-16)
type Clock interface {
Now() time.Time
Sleep(ctx context.Context, d time.Duration) error // returns ctx.Err() on cancel
}
The limiter's Acquire, the client's 429 loop and the job's delay computation take it. Tests use a fake whose Sleep advances Now. The Sleep respects ctx so the 240 s job timeout and request cancellation interrupt waits.
Pattern 3: Tier-aware AI config
classes.AIConfig gains a trust marker that only the AdminVisionModel adapter sets. Recognition and ai-credential/test call golem Send(ctx, prompt, cfg). The golem plugin selects fetchguard TrustedMode for admin configs and guarded PublicOnly for everything else, after AssertSafeURL(baseURL) (allowlist) when a user/org override is present.
Anti-Patterns to Avoid
- Sleeping out the Discogs window inside the job: PHP explicitly never does this. Re-dispatch with
Delay. - Making the Discogs base URI configurable: PHP keeps it a code literal for SSRF reasons. Inject the test transport instead.
{ok:false}for an unsafe base_url: PHP answers 500.- Marshalling multi-key job metadata via Go maps and comparing strings: see Pitfall 4.
- Holding the CSV row lock across a Discogs call: see the WR-02 section.
Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---|---|---|---|
| Private-IP / reserved-range checks | New CIDR tables per consumer | fetchguard dial Control (isReservedOrPrivate) |
Already a superset of PHP's lists, with NAT64/6to4 and zone handling |
| Inbound per-key counters | Another bespoke map+mutex | surf.MemoryStore.Attempt |
Atomic check+increment, returns retryAfter |
| Cover download allow-listing | New downloader | classes.CoverImporter / fetchguard AllowHostsMode |
Ported and parity-tested in Phase 12 |
| Job progress/cancel bookkeeping | Custom status table | conga summer_jobs API |
job_contract.go and the parity row goldens already depend on it |
| Image sniffing | New magic-byte parser | classes/image_guard.go (copy into feedback) |
Matches PHP finfo + getimagesize parity cases |
| Cross-process rate window | Advisory-lock dance | One INSERT … ON CONFLICT DO UPDATE … WHERE … RETURNING |
Atomic per Postgres docs |
| Tracklist parsing in recognition | New parser | tracklist_text_parser.go |
PHP reuses one parser (D-22) |
Key insight: almost every primitive exists. Phase risk lies in faithful edge-case parity (ordering of checks, error-to-status maps, metadata shapes) and in the test harness for upstream calls, not in new infrastructure.
Common Pitfalls
Pitfall 1: Allowlist omitted from the user/org base_url guard
What goes wrong: Go accepts an https://api.groq.com override that PHP rejects (500); parity diff and SSRF surface widen. Avoid: port SSRFGuard fully (scheme, allowlist, IP) before the dial guard. Warning sign: no allowed_hosts config key in sm-golem-plugin.
Pitfall 2: PHP 500 vs {ok:false} on guard failure
What goes wrong: Go's "nicer" error handling returns 200 {ok:false}. Avoid: record cases 16 (unsafe and non-allowlisted base_url) and assert the Winter 500 page.
Pitfall 3: Settings key and plaintext key mismatch
What goes wrong: Importing from system_settings with item golem15_golem_settings finds nothing; PHP's is golem_settings. Or the importer expects ciphertext. Avoid: Open Question 2 resolved before planning the storage.
Pitfall 4: Job metadata key order
What goes wrong: conga Metadata map[string]any marshals keys sorted, while PHP preserves insertion order ({"paused","retry_after","next_job_id"}). nuxt-csv.rows.json compares the metadata string [VERIFIED: parity/fonoteka_flows_test.go:84-95]. Avoid: compare metadata JSON semantically in new goldens, or extend conga to accept an ordered value. Single-key metadata is unaffected.
Pitfall 5: Unregistered-kind assertions break when workers land
What goes wrong: TestJobContractDispatchWhileWorkerRuns/TestCsvJobRows and conga's ErrUnregisteredKindQueue assumptions (queues "unserved") flip once the three kinds register. Avoid: update the tests in the same plan. Check that nuxt-csv still expects cancelled River states (the parity app handler does not start a worker).
Pitfall 6: Retrying jobs PHP never retries
What goes wrong: returning an error after FailJob makes River retry, which double-processes rows. Avoid: PHP catches Throwable and returns; Go returns nil after FailJob. Only the 240 s timeout or a panic should surface as a River retry.
Pitfall 7: Discogs lock-held fetch in row edit
See the WR-02 section: fetch outside the lock, then re-check under the lock.
Pitfall 8: getDefaultModel key-preservation quirk
array_filter keeps keys, so $models[0] ?? null can be null while enabled models exist. Port the quirk deliberately and test it.
Pitfall 9: Recognize manifest status mismatch
The manifest case says 422 but the fixture shows 403 for POST albums/recognize jwt. TestCheckCorpusPortedCaseStatus will fail on flip unless fixed.
Pitfall 10: Upstream JSON byte equality
PHP json_encode escapes / and Unicode. Go escapes differently (json/v2 in Go 1.27). Assert upstream bodies semantically (normalise, then compare). Response bodies to our clients stay byte-compared via the existing wire helpers.
Pitfall 11: cabana rejects unknown YAML keys
The feedback colorpicker/readOnly and the golem repeater fail boot. Adapt the YAML or add the types to cabana (a framework change with docs).
Code Examples
Redacting handler skeleton
// Source: log/slog Handler contract (stdlib); key list from RedactCredentialsTap.php
type redactHandler struct{ next slog.Handler }
func (h redactHandler) Enabled(ctx context.Context, l slog.Level) bool { return h.next.Enabled(ctx, l) }
func (h redactHandler) Handle(ctx context.Context, r slog.Record) error {
out := slog.NewRecord(r.Time, r.Level, scrub(r.Message), r.PC)
r.Attrs(func(a slog.Attr) bool { out.AddAttrs(redactAttr(a)); return true })
return h.next.Handle(ctx, out)
}
func (h redactHandler) WithAttrs(as []slog.Attr) slog.Handler { /* redact each, then h.next.WithAttrs */ }
func (h redactHandler) WithGroup(n string) slog.Handler { return redactHandler{h.next.WithGroup(n)} }
Inbound limiter (Laravel tooManyAttempts + hit)
// Source: modules/surf/limiter_store.go Store.Attempt
ok, _, retry := store.Attempt("fonoteka-discogs-missing:"+strconv.FormatUint(uint64(user.ID), 10), 60, time.Minute)
if !ok {
return wire.JSON(w, 429, ordered("result","error","code","too_many_requests","retry_after", int(math.Ceil(retry.Seconds()))))
}
(wire.JSON/ordered stand for the app's existing ordered-JSON helpers; use whatever the Phase 12/13 controllers use.)
State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|---|---|---|---|
Apparatus RequestSender (curl, resolve-time check, follows redirects, no POST guard) |
fetchguard client with dial-time guard |
This phase | Closes DNS-rebind and redirect SSRF gaps |
Winter SettingsModel over system_settings |
Dedicated typed tables + cabana settings | Phase 5 decision | Golem repeater and feedback settings need storage choices |
| Laravel sync/redis queue | River via conga (LISTEN/NOTIFY) | Phase 11 | Delayed re-dispatch is native (Delay) |
Deprecated/outdated: REQUIREMENTS.md INTG-02 "Anthropic Go SDK" (reword per D-06); API-08 "sitemap" (reword per D-13/D-14); ROADMAP SC4/SC5/SC6 (reword per D-06/D-07/D-14).
Assumptions Log
| # | Claim | Section | Risk if Wrong |
|---|---|---|---|
| A1 | Guzzle honours HTTPS_PROXY in non-CLI SAPIs and libcurl honours https_proxy for raw curl, so a recording proxy captures PHP's Discogs and AI calls |
D-11 | Recording proxy needs another capture route (Laravel Http events for Discogs; admin-tier local base_url for AI) |
| A2 | git submodule add cannot stage an empty remote; pre-initialising the directory works |
Submodules | Minor workflow change |
| A3 | Winter SettingsModel::instance() persists initSettingsData() defaults on first access (widget_key stable) |
Feedback | widget_key generation timing differs |
| A4 | AI request timeout 120 s, G15Office 30 s | fetchguard | Hung upstream holds a request or worker longer or shorter |
| A5 | Scout imports in chunks of 500 | Reindex | Batch size only |
| A6 | River backoff instead of a fixed 30 s for the G15Office job is acceptable | Feedback | Not parity-visible |
| A7 | The UNLOGGED table schema and SQL shown for the limiter | D-17 | Design detail to settle in the plan |
| A8 | No FK or trigger links summer_jobs/river_job back to csv_imports (no deadlock under the row lock) | WR-02 | Lock ordering issue |
| A9 | The fetchguard client API shape (NewClient, modes, helpers, transport seam) | fetchguard | Plan-level design |
Open Questions (RESOLVED)
- Is
GOLEM15_SSRF_ALLOWED_HOSTSset in production? We know the default allowlist rejects any user/org base_url outside*.openai.com(Anthropic included). The.envis secret-guarded. Recommendation: ask the user for the production value (names only) and make it the Go config default for the application; keep the PHP default in the plugin. DEFERRED (Phase 15 cutover): 14-04 keeps the PHP default allowlist and honours the env override. The operator confirms the production value at cutover. - Golem settings storage and key. We know the PHP code is
golem_settingswith a plaintextapi_keyinside a repeater JSON, and cabana has no repeater. D-01 saysgolem15_golem_settings; D-03 says "encrypted" and "reading the existing settings row". Recommendation: confirm a dedicatedgolem15_golem_modelstable (encrypted key, admin list/form) plus an importer fromsystem_settingsitemgolem_settings; correct D-01's key string. RESOLVED: D-18 (user, 2026-10-03): thegolem15_golem_modelstable with an encrypted key, plus an importer fromgolem_settings. - Feedback settings storage. The same convention question for
golem15_feedback_settings(scalar, so a typed singleton table fits cabana). Confirm a typed table plus an importer. RESOLVED: D-18: a typed singleton table plus an importer (14-05). - Recording-proxy scope (D-15). Building
summer parity:upstream(MITM CA, script/forward modes) is the only faithful way to capture PHP's actual upstream requests. Confirm it is in scope, or accept hand-authored sidecars. RESOLVED: D-19 (user):summer parity:upstreamis in scope (14-01). - Phase 10.1 admin Discogs stubs (
discogsLookup,discogsSync) say "Phase 14 replaces". PHP has no such admin actions. In or out of scope? RESOLVED: out of scope. The stubs are left unchanged and 14-03 records this as an assumption. - Reindex "already absent" message: add
beachcomber.IndexDropper(framework change) or accept one message for both outcomes? RESOLVED:beachcomber.IndexDropperandEnsureIndexare added in 14-01. - Roadmap home for D-09 routes (oauth-identities,
/api/v1/fonoteka/me): Phase 15 requires all routes green; a todo exists, and a phase needs to be named. DEFERRED: tracked in.planning/todos/pending/orphan-pending-routes.md. A phase must be inserted before Phase 15, for example with/gsd-phase --insert 14.1. The 14-06 gate pins exactly these 3 routes as pending.
Environment Availability
| Dependency | Required By | Available | Version | Fallback |
|---|---|---|---|---|
| Go | everything | ✓ | go1.27.0 | — |
| Docker daemon | testcontainers (Postgres tests) | ✓ | 29.7.2 | -short skips |
| PHP CLI | recording PHP parity cases | ✓ | 8.5.10 | — |
| sqlite3 | php_parity.sh rows goldens |
✓ | 3.53.4 | — |
| node | capture_clients.mjs |
✓ | v22.23.2 | — |
| psql | manual DB checks | ✓ | 18.6 | — |
| ssu | submodule management | ✓ | /home/jin/.local/bin/ssu | git submodule |
| git.golem15.com sm-golem-plugin / sm-feedback-plugin | D-01 / D-12 | ✓ reachable, empty | — | — |
| Live Discogs / Anthropic / OpenAI / G15Office | one-time forward-mode recording only | not probed | — | script-mode sidecars (no live calls) |
| fonoteka.go builds | baseline | ✓ go build ./... ok |
— | — |
Missing dependencies with no fallback: none. Note: pushing the new plugin repos and sm-user-plugin (5 ahead) is a user action.
Validation Architecture
Test Framework
| Property | Value |
|---|---|
| Framework | Go testing (+ testify assertions in existing tests), testcontainers-go v0.44.0 postgres, tide replay |
| Config file | none (Go); gate script summercms.go/scripts/check-phase14.sh (new, modelled on check-phase13.sh: --self-test --go --parity --named --coverage --evidence --all, EXPECTED_PORTED/EXPECTED_PENDING, coverage floor 80, -race on app packages) |
| Quick run command | go -C ../fonoteka.go test ./plugins/golem15/fonoteka/... -count=1 -short |
| Full suite command | go test ./... -count=1 && go -C ../fonoteka.go test ./... -count=1 -race && bash scripts/check-phase14.sh --all |
Prior verify-command pattern (Phase 13 plans): go -C ../fonoteka.go vet ./... && go -C ../fonoteka.go test ./plugins/golem15/fonoteka -run '^(TestX|TestY)$' -count=1 -race -v && go -C ../fonoteka.go test ./parity -run '^(TestParityCorpus|TestFonotekaNuxtFlows)$' -count=1 -v && go -C ../fonoteka.go run ./parity/check_corpus.go --manifest parity/manifest.yaml --routes /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/routes.php --require-recorded --check-secrets. Framework tasks add go test ./cmd/summer -count=1 -run '^(TestDocsTree|TestDocsCommandsMirrorGeneratedMain)$' && go run ./cmd/summer docs:build --check.
Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|---|---|---|---|---|
| JOBS-02 | match job: 0/1/many results, gate off, duplicate, cancel, superseded, 429 → re-dispatch with delay = max(retryAfter, secondsUntilAvailable), timeout 240 s as a value | unit + Postgres | go -C ../fonoteka.go test ./plugins/golem15/fonoteka -run '^TestCsvMatchJob' -count=1 -race |
❌ Wave 0 |
| JOBS-02 | import job: canonical overwrite/fill/create, draft vs CSV merge, write_failed row, unauthorized → canceled, bulk_updated publish once | unit + Postgres | ... -run '^TestCsvImportJob' |
❌ |
| JOBS-02 | WR-02: commit vs mapping interleave → at most one import job | Postgres | ... -run '^TestCsvWR02' |
❌ |
| JOBS-03 | digest: skipped row, sent count, row deleted, en/pl template, wishlist kind guard | unit + Postgres | ... -run '^TestWishlistDigestJob' |
❌ |
| SRCH-02 | reindex: not configured, zero-tenant DB abort, found≠0 failure, drop legacy present/absent | unit (fake engine) | ... -run '^TestReindexCommand' (fake engine exists: fake_engine_test.go) |
❌ |
| CLI-05 | prune deletes >90 d, message text, schedule entry resolves | unit + Postgres | `... -run '^(TestPruneNotifications | TestSchedule)'` |
| INTG-01 | limiter: threshold, budget exhaustion carries window remainder, tighten-only header sync, retry-after fallback, fake clock | unit + Postgres SQL | go -C ../fonoteka.go test ./plugins/golem15/fonoteka/classes/discogs -count=1 -race |
❌ |
| INTG-01 | mapper/parser/scorer/price resolver vs PHP truth tables | unit | ... ./classes/discogs -run '^TestPHPTruth' |
❌ (generator parity/discogs_truth_tables.php) |
| INTG-01 | routes parity with sidecars (cases 1-15) | parity | `go -C ../fonoteka.go test ./parity -run '^(TestParityCorpus | TestCheckCorpusPortedCaseStatus)$' -count=1` |
| INTG-02 | adapters build PHP payloads (headers, max_tokens mapping, reasoning_effort, image blocks), parse errors verbatim | unit | go -C ../fonoteka.go test ./plugins/golem15/golem/... -count=1 |
❌ |
| INTG-02 | SSRF guard: scheme, allowlist suffix, private IP; admin trusted bypass; 500 page on guard failure | unit + parity | ... -run '^TestSSRFGuard' + parity case 16 |
❌ |
| INTG-02 | recognition: retry, truncation code, normalisation caps, admin tier | unit + parity | go -C ../fonoteka.go test ./plugins/golem15/fonoteka -run '^TestRecognize' |
❌ |
| API-08 | feedback config/submit/options/me-hidden, origin gate, image guard, getApiArray key, G15Office job via fake | unit + parity | go -C ../fonoteka.go test ./plugins/golem15/feedback/... -count=1 -race |
❌ |
| framework | fetchguard client modes, redirects, caps, transport seam; redact handler; tide sidecar replay asserts | unit | go test ./modules/fetchguard/... ./modules/tide/... -count=1 |
❌ |
Sampling Rate
- Per task commit: the task's named
-runtests plusgo vetin the touched repo. - Per wave merge:
go test ./...in both repos plusTestParityCorpus/TestFonotekaNuxtFlows. - Phase gate:
scripts/check-phase14.sh --allgreen before/gsd-verify-work.
Wave 0 Gaps
fonoteka.go/parity/discogs_truth_tables.php(truth-table generator,csv_truth_tables.phppattern)- tide sidecar loader and fake; the
replayPortedRoutehook for*.upstream.yaml - Fake clock helper shared by limiter, client and job tests
- Test harness to run a conga worker function directly (call the job func with a ctx carrying
JobID) without a live River client scripts/check-phase14.sh(copy the check-phase13 structure; updateEXPECTED_PORTED/EXPECTED_PENDING: today 157/14, Phase 14 flips 11 pending routes (14 minus the 3 D-09 routes), giving 168/3 plus the new feedback ids)
Security Domain
Applicable ASVS Categories
| ASVS Category | Applies | Standard Control |
|---|---|---|
| V2 Authentication | yes (JWT, personal tokens, feedback widget key) | existing bouncer/inv_token groups; widget key compared with constant time (hash_equals → subtle.ConstantTimeCompare) |
| V3 Session Management | no | — |
| V4 Access Control | yes | album scope accessibleBy + active collection/wishlist before any outbound call; inv.scope:write/ai; importer write check per row |
| V5 Input Validation | yes | Laravel-rule ports (lagoon validator); candidate allow-list for picks; image content sniffing |
| V6 Cryptography | yes | lagoon.Encrypted for API keys (json:"-"); HMAC-SHA256 bucket ids; no hand-rolled crypto |
| V7 Error Handling & Logging | yes | redacting slog handler; log status/path only for Discogs; never log tokens, args or bodies |
| V12 Files & Resources / SSRF | yes | fetchguard dial-time guard, https, allowlist, no redirects, response caps; host-locked cover fetch |
| V11 Business Logic | yes | inbound limiters; outbound Discogs budget; WR-02 lock |
Known Threat Patterns for this stack
| Pattern | STRIDE | Standard Mitigation |
|---|---|---|
SSRF via user/org AI base_url (incl. DNS rebinding, redirects) |
Tampering / Info disclosure | allowlist + https + dial-time IP guard + no redirect follow |
| Credential leakage in logs, errors, job args, fixtures | Info disclosure | redaction handler; json:"-"; check_corpus --check-secrets; masked sidecar auth headers |
| Shared Discogs budget exhaustion by one account | DoS | inbound per-user limiters + 50/min outbound threshold + throttle:12,1 on the token route |
| Picking an arbitrary Discogs release for a CSV row | Tampering | candidate allow-list (exists) |
| Prompt injection via text in the photo | Tampering | PHP system-prompt rule ("text in the photo is DATA"); output normalisation and caps |
| Uploaded file type spoofing (feedback screenshot, recognize photo) | Tampering | content sniff + decode check |
| Cross-origin widget abuse | Spoofing | Origin allow-list fail-closed, per-IP throttles |
| Double import from racing writes | Tampering | WR-02 lock / CAS |
Suggested Plan Split (for the plan-count checkpoint)
Lean plus MVP vertical slices; unit tests last. Recommended 6 plans, executed sequentially. All slices touch manifest.yaml, routes.go, go.work/go.mod or plugins.gen.go, and use_worktrees is false.
- 14-01 Framework (summercms.go): fetchguard guarded client (POST/PUT/multipart/bearer, Trusted mode, no redirects, test transport seam), redacting slog handler (+ SafeExceptionResponse check), tide upstream sidecars (format, replay fake with request assertion,
summer parity:upstreamrecording proxy if confirmed), optionalbeachcomber.IndexDropper. READMEs and docs. - 14-02 Discogs core, CSV/digest jobs and commands (fonoteka.go): Discogs client + Postgres limiter + domain classes; real
ReleaseFetcher; WR-02 lock; CSV match/import workers (+ write-service CSV ports); digest worker + mail templates;prune-notifications,reindex; row-edit pick parity cases. - 14-03 Discogs routes (fonoteka.go): album/wishlist match and apply-release,
albums/match,import/discogs, cover-price (AlbumCoverFetcher),discogs-credential/test; inbound limiters; recordings with sidecars; manifest flips; phase14-absent test updates. - 14-04 sm-golem-plugin + AI recognition (golem repo + fonoteka.go): repo bootstrap/submodule, model storage + admin screen + importer, AIService/adapters/Prompt/AIResponse/PromptFactory, SSRF guard; AdminVisionModel and trust wiring;
AlbumRecognitionService; recognize (both groups),ai-credential/test; recordings. - 14-05 sm-feedback-plugin (feedback repo + fonoteka.go): full port (routes, settings, submissions list, getApiArray, embed.js route, G15Office job on the guarded client), feedback manifest section and fixtures.
- 14-06 Unit tests and gate: full coverage for all Phase 14 code, PHP truth tables,
check-phase14.sh, requirement and roadmap rewording checks, validation evidence.
Alternative (7 plans): split 14-02 into "Discogs core + CSV jobs" and "digest + commands" if 14-02 proves too large at planning time.
Sources
Primary (HIGH confidence, read this session)
- PHP:
fonoteka/plugins/golem15/fonoteka/{routes.php, jobs/*, console/*, classes/discogs/*, classes/{AlbumRecognitionService,AiConfigResolver,AiGate,UserAiConfig,OrgAiConfig,AiDefaults,NotificationService}.php, controllers/api/{AlbumReleaseMatch,WishlistReleaseMatch,RecognizeApi,DiscogsImport,AlbumCoverFetch,AiCredential,DiscogsCredential,CsvImportApi}Controller.php, config/fonoteka.php} - PHP:
golem/{Plugin.php, models/Settings.php, models/settings/fields.yaml, classes/services/AIService.php, classes/providers/*, classes/valueobjects/{Prompt,AIResponse}.php, classes/security/SSRFGuard.php, config/ssrf.php, updates/*} - PHP:
feedback/*(all files),apparatus/classes/{RequestSender.php, traits/SafeExceptionResponse.php, logging/RedactCredentialsTap.php} - Nuxt
vue-fonoteka-app/{nuxt.config.ts, app/stores/fonoteka.ts, app/stores/auth.ts}, MCPfonoteka-mcp/src/client.ts - Go:
fonoteka.go(job_contract.go,jobs.go,csv_import_service.go,wishlist_notifications.go,ai_config_resolver.go,gates.go,cover_importer.go,search.go,schedule.go,console/oauth_client.go,routes.go,parity/*),summercms.go/modules/{fetchguard,conga,beachcomber,cabana,surf,lagoon/attach,pact} - Planning: 13-REVIEW.md (WR-02), 13-VERIFICATION.md, 12/13-CONTEXT.md, notes (core-plugins-own-repos, plugin-layout, apparatus-dissolved), todos
- postgresql.org/docs/current/sql-insert.html: ON CONFLICT atomicity and RETURNING of non-updated rows
- pkg.go.dev/net/http#Client: redirect semantics, GetBody, ErrUseLastResponse
Secondary (MEDIUM)
- Remote reachability probe (
git ls-remote) of both new repos
Tertiary (LOW)
- Discogs rate limits (search summary of discogs.com/developers; direct fetch returned 403)
Metadata
Confidence breakdown:
- Standard stack: HIGH (no new dependencies; every module read)
- Architecture: MEDIUM-HIGH (seams verified; new framework API shapes are proposals)
- Pitfalls: HIGH (each grounded in a file read this session)
Research date: 2026-10-03 Valid until: 2026-11-02 (stable in-repo facts); re-check vendor API details at implementation.