708 lines
97 KiB
Markdown
708 lines
97 KiB
Markdown
# Phase 14: Domain jobs and external integrations - Research
|
||
|
||
**Researched:** 2026-10-03
|
||
**Domain:** Go port of Płytarium's outbound integrations (Discogs, Anthropic/OpenAI-compatible AI, G15Office), River job workers, console commands, and two new shared core plugins (sm-golem-plugin, sm-feedback-plugin)
|
||
**Confidence:** HIGH for the PHP behaviour and the Go seams (all read this session); MEDIUM for the proposed new framework designs (fetchguard client, upstream sidecars); LOW only where marked `[ASSUMED]`.
|
||
|
||
<user_constraints>
|
||
## User Constraints (from CONTEXT.md)
|
||
|
||
### Locked Decisions
|
||
|
||
#### AI layer (sm-golem-plugin)
|
||
- **D-01:** The Go port of PHP `Golem15.Golem` lives in its own shared core-plugin repo, `git@git.golem15.com:golem15/sm-golem-plugin.git` (created empty by the user). Module `git.golem15.com/golem15/sm-golem-plugin`, mounted in fonoteka.go at `plugins/golem15/golem` as a git submodule, following `.planning/notes/core-plugins-own-repos.md`. The plugin ID stays `golem15.golem`, with the same settings key (`golem15_golem_settings`) and lang namespace, so migrated PHP data loads as-is. — **Reversibility:** costly — the plugin ID, settings key and module path are what every consuming application and the migrated settings row use.
|
||
- **D-02:** The provider adapters (Anthropic Messages, OpenAI-compatible chat completions) are hand-rolled JSON mappings over the guarded outbound client (folded todo fetchguard-guarded-http-client). No Anthropic or OpenAI SDK dependency. PHP's adapters are raw curl, which makes request parity easier.
|
||
- **D-03:** Port the Golem Settings model in full: the `models` repeater (name, adapter openai|anthropic, encrypted `api_key`, `base_url`, `model`, `system_prompt`, `is_enabled`, `is_default`, `generates_images`, `accepts_images`, `has_files_endpoint`, `max_completion_tokens`) plus its admin settings screen, reading the existing settings row. `getVisionModel()` fills fonoteka's `classes.AdminVisionModel` seam (Phase 13), so the admin tier of `ResolveAIConfig` and the site-admin branch of `AIAllowed` light up.
|
||
- **D-04:** Port all of AIService except faces: `send`, `sendStream`, `sendFile`/`sendFilePath` (files endpoint), `sendWithModel`, `sendToImageModel`, `sendToVisionModel`, `generateImage`, `ask`, the model system-prompt application, `Prompt`/`AIResponse` value objects and `PromptFactory`. CompreFace/FaceService and ChatContextCollector are not ported. The SSRFGuard is ported as part of the guarded-client work.
|
||
- **D-05:** Base-URL guard: user and org credential `base_url` overrides always go through the private/reserved-IP dial guard. Admin-configured Settings models are trusted (they may point at a LAN/local endpoint such as Ollama). The researcher confirms this split against PHP's `SSRFGuard` and reports any difference before planning.
|
||
- **D-06:** REQUIREMENTS INTG-02 ("Anthropic Go SDK") is reworded at plan time to "Anthropic and OpenAI-compatible adapters over the guarded client". PROJECT.md's repository paragraph gains sm-golem-plugin and sm-feedback-plugin, and drops feedback and sitemap from the application-plugin list.
|
||
|
||
#### Route scope
|
||
- **D-07:** All 6 album Discogs/AI routes that Phase 12 D-04 moved here are in scope (`albums/match`, `albums/{id}/match`, `albums/{id}/apply-release`, `albums/recognize` on the JWT and token groups, `albums/import/discogs`), together with the routes the criteria already name. ROADMAP SC4/SC5 are reworded at plan time to list them. They share ReleaseMatchScorer, AlbumReleaseApplicator, DiscogsImportResolver, DiscogsMapper and AlbumRecognitionService with the named routes.
|
||
- **D-08:** The Phase 13 `ReleaseFetcher` seam behind the CSV row-edit `selected_discogs_id` pick gets the real Discogs `getRelease` + `DiscogsMapper::mapRelease` implementation, so a pick resolves exactly as in PHP.
|
||
- **D-09:** `GET/DELETE oauth-identities` (social login, deferred since Phase 7) and `GET /api/v1/fonoteka/me` (the minimal endpoint from Phase 8 D-20) are not part of Phase 14. They stay `pending` and are flagged for the roadmap (see Deferred).
|
||
- **D-10:** WR-02 from the Phase 13 review is fixed before the CSV workers go live: the `mapping`, row-edit and `cancel` writes lock the import row or compare-and-swap on its status, as `commit` already does, so none of them can race a commit and queue a second import job. Response shapes stay the same. This is a deliberate deviation from PHP, which has the same race.
|
||
- **D-11:** The researcher checks `routes.php` and the Nuxt/MCP callers for any recorded case that is still missing, for example a successful `selected_discogs_id` pick, the cover-price route, and the recognize success and truncation paths. It records those against the PHP backend, together with their upstream exchanges (D-15).
|
||
|
||
#### Feedback plugin (sm-feedback-plugin)
|
||
- **D-12:** Feedback is a shared core plugin in its own repo, `git@git.golem15.com:golem15/sm-feedback-plugin.git` (created by the user). Module `git.golem15.com/golem15/sm-feedback-plugin`, mounted in fonoteka.go at `plugins/golem15/feedback`, plugin ID `golem15.feedback`, with the same tables (`feedback_submissions`, user preferences) and the same settings key. — **Reversibility:** costly — the plugin ID, tables and settings key become the contract other applications mount.
|
||
- **D-13:** Full port. It covers `GET /_feedback/api/v1/{key}/config` (throttle 60/min/IP), `POST {key}/submit` (multipart, 10/min/IP, Origin allowlist gate, ImageContentGuard), the `OPTIONS {any}` 204 preflight, and JWT `PUT me/hidden`. `embed.js` is served at the same `/plugins/golem15/feedback/assets/js/embed.js` path that the Nuxt proxy expects. The plugin also ports the Settings model (enabled, widget_key, allow_hide, position, allowed_origins, colors, pl/en labels) with its admin screen, the submissions admin list, and the `golem15.user.getApiArray` hook that adds `feedback_widget_hidden` to the user payload. The user plugin already references this field. `SyncFeedbackToG15Office` becomes a River job on the guarded client (JSON + multipart). Feedback routes are recorded as new parity fixtures. API-08 is reworded to feedback only.
|
||
- **D-14:** Sitemap is dropped for Płytarium. Nuxt builds its own sitemap with `@nuxtjs/seo`, and fonoteka registers no menu item types. A todo records a 1:1 sitemap plugin port for the next blog project (grzybyfunkcjonalne.pl or golem15.com).
|
||
|
||
#### Vendor testing
|
||
- **D-15:** Upstream HTTP in tests: the upstream Discogs/Anthropic/OpenAI/G15Office exchanges are recorded once as a sidecar to each parity case and served from an `httptest` fake that the guarded client points at during replay. Replay is deterministic and offline. The fake also asserts the request Go sends upstream (method, path, headers including User-Agent and auth, body) against PHP's recorded request. CI makes no live vendor calls, and G15Office is verified only through the fake.
|
||
- **D-16:** Time is injected. The Discogs rate limiter, its wait budget and retry-after handling, and the match job's re-enqueue all take a clock/sleeper interface. Tests advance a fake clock without real sleeps, and assert the 240 s timeout and the re-enqueue delay as values.
|
||
- **D-17:** The researcher reads `DiscogsRateLimiter.php` and decides where its state lives, either process memory or Postgres (an UNLOGGED table or an advisory lock), depending on whether PHP's cross-worker guarantee is still needed when one binary runs HTTP and River workers. The choice and its reason go in RESEARCH.md.
|
||
|
||
#### Folded Todos
|
||
- **fetchguard-guarded-http-client** (`.planning/todos/pending/fetchguard-guarded-http-client.md`, high): extend fetchguard from a guarded HTTPS GET into a guarded outbound `http.Client`/constructor (POST, PUT, multipart upload, bearer auth), keeping the dial-time private/reserved-IP rejection, with an explicit trusted mode for admin-configured endpoints (D-05). It replaces Apparatus `RequestSender`. Its consumers are the Discogs client, the Golem adapters and feedback's G15OfficeClient. Framework change in summercms.go: update `modules/fetchguard/README.md` and `docs/`.
|
||
- **redacting-slog-handler** (`.planning/todos/pending/redacting-slog-handler.md`, medium): a framework `slog.Handler` wrapper porting Apparatus `RedactCredentialsTap`. It redacts the keys api_key, apikey, authorization, bearer, password, secret, token, webhook_secret and admin_password (case-insensitive, nested groups) and scrubs messages with the PHP regex patterns. It also checks whether surf's error path already gives `SafeExceptionResponse` behaviour. This is the first phase to send user credentials to outside services.
|
||
|
||
### Claude's Discretion
|
||
- Wishlist digest mail content, locale and template: a straight port of `WishlistDigestJob` and its PHP mail view on the existing postcard mail pipeline.
|
||
- `prune-notifications` and `reindex` options, output and exit codes: a straight port of `PruneNotifications.php` and `ReindexAlbums.php`, on bonfire with the existing `schedule.go` entry.
|
||
- Exact package layout inside sm-golem-plugin and sm-feedback-plugin, which follows `.planning/notes/plugin-layout-winter-directories.md`.
|
||
- Plan split, subject to the lean-mode plan-count checkpoint.
|
||
|
||
### Deferred Ideas (OUT OF SCOPE)
|
||
- **Sitemap plugin 1:1 port** for the next blog project: a new todo is written alongside this context.
|
||
- **`oauth-identities` GET/DELETE (social login) and `GET /api/v1/fonoteka/me`**: the routes stay `pending` without a phase. The roadmap needs a home for them before Phase 15, whose criterion is all routes green. A new todo is written.
|
||
- Golem face detection (CompreFace/FaceService) and ChatContextCollector: not ported until a consumer needs them.
|
||
|
||
#### Reviewed Todos (not folded)
|
||
- `backend-admin-api-tokens`, `bonfire-duplicate-command-names`, `lagoon-readme-after-commit-callback-order`, `nest-framework-packages-under-modules`, `per-module-readmes-after-nest`, `readme-go-fences-src`, `refresh-fonoteka-readme`, `rewrite-summercms-readme`, `scaffold-*` (3), `wristband-neutral-resource-default`, `2026-10-01-benchmark-…`: these matched on keywords only and are unrelated to the phase domain.
|
||
</user_constraints>
|
||
|
||
<phase_requirements>
|
||
## Phase Requirements
|
||
|
||
| ID | Description (REQUIREMENTS.md, before the D-06/D-13/D-14 rewording) | Research Support |
|
||
|----|-------------|------------------|
|
||
| JOBS-02 | CSV import write job and Discogs match job (240 s timeout) are ported; the match job re-enqueues itself with a delay on a Discogs rate-limit error instead of failing the batch | §"The three PHP jobs", §D-17, §"WR-02 fix", Pattern 3 (job worker), Pitfalls 4-7 |
|
||
| JOBS-03 | Wishlist digest coalesces notifications in a 30-minute window and deletes its queue row on completion | §"The three PHP jobs" (WishlistDigestJob), Pattern 3 |
|
||
| SRCH-02 | The reindex command asserts zero documents with collection_id 0 before and after, and can drop the legacy index | §"Console commands" (ReindexAlbums), beachcomber gap (DropIndex) |
|
||
| INTG-01 | Discogs client with proactive rate threshold, bounded in-request wait budget, retry-after fallback, and host-locked cover fetch, plus the cover-price route, the wishlist match/apply-release routes, discogs-credential/test and the CSV row-edit pick | §"Discogs client", §D-17, §D-11 case list, §"Inbound limiters" |
|
||
| INTG-02 | AI cover recognition through provider adapters (reword: Anthropic and OpenAI-compatible adapters over the guarded client) with per-credential model and base URL overrides, plus ai-credential/test and the backend global vision model | §"Golem plugin surface", §D-05, §"Settings storage gap" |
|
||
| API-08 | Feedback submissions and sitemap output (reword: feedback only, D-13/D-14) | §"Feedback plugin surface" |
|
||
| CLI-05 | Płytarium commands: oauth-client (already shipped Phase 8), prune-notifications, reindex with drop-old-index flag | §"Console commands" |
|
||
</phase_requirements>
|
||
|
||
## Project Constraints (from CLAUDE.md)
|
||
|
||
From `summercms.go/CLAUDE.md`, `fonoteka.go/CLAUDE.md`, the parent `CLAUDE.md` and the user's global `CLAUDE.md`:
|
||
|
||
- **Lean planning:** few, large plans; a plan-count checkpoint before PLAN.md files are written; **unit tests are always the last plan** of the phase.
|
||
- **Go conventions:** standard library first; add a dependency only when the research doc or a phase decision names it. `go vet` and `go test ./...` green at every commit. D-02 forbids an Anthropic/OpenAI SDK: this phase adds **no** new third-party Go dependency.
|
||
- **Compiled plugins only;** no runtime plugin loading.
|
||
- **API parity is the acceptance test.** Do not "improve" response shapes. D-10 (WR-02) is the one sanctioned deviation and keeps response shapes.
|
||
- **Commits:** never add co-author tags; one logical change per commit; planning docs and code in separate commits.
|
||
- **Framework docs rules:** any change to a `modules/` package's exported API, config keys, CLI commands or dependencies updates that module's `README.md` and the affected `docs/` pages in the same change; a new module ships a README with the standard structure and a row in the root README modules table; framework READMEs never name a consuming application; every identifier named in a README/docs page must exist (`go test ./cmd/summer -run TestDocsTree`, `summer docs:build --check`). Config keys in docs are checked by hand.
|
||
- **Two repositories:** `summercms.go` is framework only and knows nothing about Płytarium; app code goes to `fonoteka.go`. Planning docs stay in `summercms.go/.planning`.
|
||
- **Core plugin contracts:** user, blog, pages, payment PHP plugins are shared; Go ports preserve contracts; PHP originals are not changed. sm-user-plugin changes must be additive.
|
||
- **Submodule workflow (fonoteka.go/CLAUDE.md, core-plugins-own-repos.md):** change a core plugin inside its checkout, commit (and push) there first, then commit the bumped pointer in fonoteka.go as a separate commit; never stage submodule files from the app repo. `ssu` is available for submodule management.
|
||
- **GSD workflow enforcement:** file changes go through GSD commands.
|
||
|
||
## Summary
|
||
|
||
Phase 14 is the largest port so far by PHP volume: about 2,250 lines of Discogs classes, 900 lines of Discogs/AI controllers, 650 lines of jobs, 450 lines of `AlbumRecognitionService`, 900 lines of the Golem AI layer and 1,650 lines of the feedback plugin (including `embed.js`). Almost everything it needs from the framework already exists: conga (River) with `Dispatch`/`Delay`/`Timeout`/`StopJob`/`CompleteJob`, the bonfire command pattern, beachcomber's Typesense engine, lagoon file attachments, `surf.MemoryStore` for in-controller rate limiters, the user plugin's `GetApiArrayEvent`, and the Phase 13 seams (`job_contract.go` kinds and args, `ReleaseFetcher`, `AdminVisionModel`, `ResolveAIConfig`, `DiscogsAllowed`). The framework gaps are concrete and small: fetchguard is GET-only with test hooks that are unexported, cabana settings cannot express a repeater, beachcomber cannot report whether a dropped index existed, and tide has no notion of upstream-exchange sidecars.
|
||
|
||
The three researcher tasks resolve as follows. **D-05:** the admin-trusted / user-and-org-guarded split matches PHP, but PHP's `SSRFGuard::assertSafeUrl` is stricter and different in kind: https only, a **host allowlist** (default only `.openai.com` and the DALL-E blob host, env-overridable), and a resolve-time private-IP check. Its failure is an uncaught `RuntimeException`, so PHP answers with the Winter 500 HTML page, not `{ok:false}`. The Go port must keep the allowlist and the 500 to stay parity-true, and add the dial-time guard on top. **D-17:** the limiter state goes in Postgres (an UNLOGGED table and one atomic upsert), because conga explicitly supports a separate `queue:work` process (`queue.work_in_serve: false`) and the 50/60-per-minute budget is shared by every process. **D-11:** every currently recorded Phase 14 case is a negative path (404, 503 disabled, 403, 422, a live-vendor 401). All success, rate-limit, truncation and pick cases are still missing, and the two live-vendor cases must be re-recorded with a sidecar.
|
||
|
||
Two findings contradict CONTEXT and need user confirmation before planning (see Open Questions). First, the PHP Golem settings code is `golem_settings`, not `golem15_golem_settings`. Second, the PHP `models[].api_key` is stored in **plaintext**, not encrypted. The project's settings convention (Phase 5, user-resolved) is a dedicated typed table, not Winter `system_settings`, and cabana has no `repeater` field type. So "reading the existing settings row" needs a storage decision (recommended: a `golem15_golem_models` table, edited as an admin list/form, plus an import from the PHP JSON at cutover).
|
||
|
||
**Primary recommendation:** Build the framework pieces first (guarded `fetchguard.Client` with a trusted mode and a test transport seam, the redacting slog handler, tide upstream sidecars plus a recording proxy, beachcomber `DropIndex`). Then port in vertical slices: Discogs core and jobs, Discogs routes, sm-golem-plugin with AI recognition, sm-feedback-plugin. Finish with a unit-test plan and a `check-phase14.sh` gate.
|
||
|
||
## Architectural Responsibility Map
|
||
|
||
| Capability | Primary Tier | Secondary Tier | Rationale |
|
||
|------------|-------------|----------------|-----------|
|
||
| Guarded outbound HTTP (POST/PUT/multipart/bearer, dial-time IP guard, trusted mode) | Framework (summercms.go `fetchguard`) | — | Shared by Discogs, Golem and G15Office; SSRF policy is a framework concern |
|
||
| Credential redaction in logs | Framework (new slog handler) | App wiring (`slog.SetDefault` in serve/queue:work) | Nothing publishes a `*slog.Logger` today; every module falls back to `slog.Default()` |
|
||
| Upstream exchange sidecars (record + replay fake + request assertion) | Framework (`tide`) | App (`parity/` harness wiring) | tide owns fixtures; the app owns which routes use sidecars |
|
||
| Discogs client, limiter, mapper, scorer, applicator, resolvers | App (fonoteka plugin `classes/discogs`) | DB (UNLOGGED limiter table) | Płytarium-specific domain |
|
||
| Discogs/AI/credential-test routes | App (fonoteka `controllers/api`, `routes.go`) | — | API parity surface |
|
||
| CSV match/import, digest workers | App (fonoteka plugin `jobs.go` or `jobs/`) | Framework (conga/River) | Kinds and args are fixed by `job_contract.go` |
|
||
| AI providers, Prompt/AIResponse, model settings | Core plugin (sm-golem-plugin) | Framework (fetchguard) | Shared across Golem15 apps (D-01) |
|
||
| Feedback widget API, settings, G15Office sync | Core plugin (sm-feedback-plugin) | User plugin event (`GetApiArrayEvent`) | Shared core plugin (D-12) |
|
||
| `fonoteka:prune-notifications`, `fonoteka:reindex` | App (fonoteka `console/`) | Framework (bonfire, conga schedule, beachcomber) | Straight ports |
|
||
|
||
## Standard Stack
|
||
|
||
### Core (all already in the build; nothing new to install)
|
||
|
||
| Library / module | Version | Purpose | Why standard here |
|
||
|---------|---------|---------|--------------|
|
||
| Go stdlib `net/http`, `mime/multipart`, `encoding/json`, `crypto/hmac`, `crypto/sha256`, `log/slog` | Go 1.27.0 (`go version` this session) | Adapters, multipart upload, HMAC bucket id, redaction handler | D-02: hand-rolled JSON over the guarded client [VERIFIED: go version] |
|
||
| `modules/fetchguard` | in-repo | Base of the guarded client | Existing dial-time classifier (`ip.go`) is a strict superset of PHP's lists [VERIFIED: modules/fetchguard/ip.go:5-20] |
|
||
| `modules/conga` (River v0.47.0) | in-repo | Workers, Delay re-dispatch, Timeout, record rows | `conga.Timeout`, `DispatchOpts.Delay`, `StopJob`, `CompleteJob` exist [VERIFIED: modules/conga/job.go:21-37, conga.go:112-133] |
|
||
| `modules/beachcomber` + `beachcomber/typesense` | in-repo | Reindex: Upsert/Flush/SearchPage | Engine has `Upsert`, `Delete`, `Flush`, `SearchIDs`; `PageSearcher` gives `Found` [VERIFIED: modules/beachcomber/searchable.go:39-57] |
|
||
| `modules/surf` `MemoryStore` | in-repo | In-controller limiters (`fonoteka-discogs-missing:`, `fonoteka-recognize:`, `fonoteka-discogs-import:`) | `Attempt(key, max, decay)` is atomic check+increment and returns retryAfter [VERIFIED: modules/surf/limiter_store.go:11-13] |
|
||
| `modules/lagoon/attach` | in-repo | Feedback screenshot `attachOne` | `attach.Relation{Name, Many, Public}` [VERIFIED: modules/lagoon/attach/relation.go] |
|
||
| `modules/postcard` | in-repo | Digest mail templates | Existing mail jobs use `postcard.Mailer.Send` [VERIFIED: fonoteka plugin jobs.go:119-126] |
|
||
| `modules/cabana` | in-repo | Admin settings and list screens | Settings are scalar-only singleton rows (see gap) [VERIFIED: modules/cabana/settings.go:37-95] |
|
||
|
||
### Supporting
|
||
|
||
| Library | Version | Purpose | When to Use |
|
||
|---------|---------|---------|-------------|
|
||
| testcontainers-go postgres | v0.44.0 (already in go.mod) | Limiter SQL, worker and migration tests | Postgres-dependent tests, `-short` skippable |
|
||
| `pgx/v5` | v5.10.0 (already in go.mod) | — | Unchanged |
|
||
|
||
### Alternatives Considered
|
||
|
||
| Instead of | Could Use | Tradeoff |
|
||
|------------|-----------|----------|
|
||
| Hand-rolled adapters | anthropic-sdk-go / openai-go | Locked out by D-02 |
|
||
| Postgres limiter table | Process-memory limiter (like `PubfailCounter`) | Breaks the cross-process budget when `queue.work_in_serve: false` (see D-17) |
|
||
| MITM recording proxy for sidecars | Hand-written sidecars from vendor docs | Hand-written sidecars cannot capture what PHP actually sent; D-15 asks for PHP's recorded request |
|
||
|
||
**Installation:** none. `go.mod` changes are limited to the two new submodule modules (require plus local replace) and their own `go.mod` files.
|
||
|
||
## Package Legitimacy Audit
|
||
|
||
This phase installs **no** external packages (D-02 rules out vendor SDKs; every capability is stdlib or in-repo). The seam check was not needed.
|
||
|
||
| Package | Registry | Age | Downloads | Source Repo | Verdict | Disposition |
|
||
|---------|----------|-----|-----------|-------------|---------|-------------|
|
||
| (none) | — | — | — | — | — | — |
|
||
|
||
**Packages removed due to [SLOP] verdict:** none
|
||
**Packages flagged as suspicious [SUS]:** none
|
||
|
||
## Existing Go seams and enqueue sites (inventory for the planner)
|
||
|
||
| Seam / file | What exists | Phase 14 action |
|
||
|---|---|---|
|
||
| `fonoteka/classes/job_contract.go:29-62` | `CsvImportKind = "golem15.fonoteka.csv_import"`, `CsvImportQueue = "fonoteka_csv_import"`, `CsvImportLabel = "fonoteka.csv.import"`, `CsvMatchKind = "golem15.fonoteka.csv_match"`, `CsvMatchQueue = "fonoteka_csv_match"`, `CsvMatchLabel = "fonoteka.csv.match"`, `WishlistDigestKind = "golem15.fonoteka.wishlist_digest"`, `WishlistDigestJobQueue = "fonoteka_wishlist_digest"`, `WishlistDigestLabel = "wishlist_digest"`, `WishlistDigestDelay = 1800 * time.Second`; args `CsvImportArgs{CsvImportID uint \`json:"csv_import_id"\`}`, `CsvMatchArgs{CsvImportID}`, `WishlistDigestArgs{SubscriberID \`json:"subscriber_id"\`, WishlistCollectionID \`json:"wishlist_collection_id"\`}` [VERIFIED: job_contract.go:29-88] | Register workers with exactly these kinds and queues. Never rename (queued rows carry them) |
|
||
| `fonoteka/jobs.go:32-41` | `Jobs()` returns invitation mail and purchase mail on `conga.OnQueue(...)`, `conga.MaxAttempts(...)`; comment says "The CSV and wishlist digest kinds have no worker until Phase 14 (D-04)" | Add three `conga.Job(...)` entries. Update the comment and `TestJobContract*` (they assert the queues are unserved) |
|
||
| `classes/csv_import_service.go:519-587` `UpdateCsvMapping` | Dispatches `CsvMatchArgs` on `CsvMatchQueue`, label, `Count: RowCount`, metadata `{"csv_import_id":N}` | WR-02 lock (below) |
|
||
| `csv_import_service.go:589-628` `ReleaseFetcher` + `SetReleaseFetcher` (atomic box) | Default always returns `ErrDiscogsUnavailable` | Install the real fetcher at plugin Boot (D-08). `ErrDiscogsRateLimited` maps to `discogs_rate_limited` 422 |
|
||
| `csv_import_service.go:639-705` `UpdateCsvRow` | Pick path: digits check, candidate allow-list, `DiscogsAllowed`, fetch, `resolved` | WR-02 lock. Success now writes the draft |
|
||
| `csv_import_service.go:825-882` `CommitCsvImport` | CAS `UPDATE ... WHERE id = ? AND status = ?` preview→importing, then Dispatch `CsvImportArgs` | Unchanged (reference CAS) |
|
||
| `csv_import_service.go:889-901` `CancelCsvImport` | Cancels `MatchJobID`/`ImportJobID` from the stale read | WR-02 lock |
|
||
| `classes/wishlist_notifications.go:130-153` `EnqueueWishlistDigest` | Upsert `golem15_fonoteka_wishlist_digest_queue` with `RETURNING (xmax = 0)`, Dispatch `WishlistDigestArgs` with `Delay: WishlistDigestDelay` only on first insert | Worker only |
|
||
| `classes/ai_config_resolver.go:40-45` | `var AdminVisionModel = func(ctx context.Context) (*AIConfig, error) { return nil, nil }`; `AIConfig{Adapter, APIKey, BaseURL, Model}` all `json:"-"`; comment: "the outbound client that consumes it (Phase 14) owns the SSRF guard PHP applies" | Replace from the golem plugin at Boot. **Add a trust marker** (for example `Trusted bool \`json:"-"\``) so the admin tier selects trusted mode (D-05) |
|
||
| `classes/gates.go:60-90` `AIAllowed` | Admin branch calls `AdminVisionModel(ctx)` | Lights up automatically |
|
||
| `classes/gates.go:136` `ResolveDiscogsConfig`, `:177` `DiscogsAllowed`, `:188` `MarketCurrency` | Ported tiers | Reuse in the client `forUser` |
|
||
| `classes/cover_importer.go:34-94` | `Fetch` seam; `fetchguard.Fetch` with `AllowHostsMode` | Reuse for the host-locked cover fetch in `AlbumCoverFetcher` and the applicator |
|
||
| `classes/manual_cover_fetcher.go:84` | `Fetch func(ctx, rawURL, policy)` seam pattern | Pattern precedent for injectable fetchers |
|
||
| `classes/public_share.go` `PubfailCounter` | In-memory per-process limiter with an injected clock | Precedent: in-request inbound limiters stay in memory |
|
||
| `fonoteka/search.go:19-47` | `settingsGate` reads `search_use_typesense`; `wireSearch` installs the gate | Reuse in reindex's "configured" check |
|
||
| `fonoteka/schedule.go:13-17` | `{Command: "fonoteka:prune-notifications", Cadence: pact.Daily()}` | Only the command is missing |
|
||
| `fonoteka/console/oauth_client.go` | `bonfire.Command{Name, Description, Args, Flags (Bare), Run}` pattern | Template for the two new commands |
|
||
| `controllers/albums_admin_controller.go:57-90` | Phase 10.1 admin `discogsLookup`/`discogsSync` **stubs**, documented as "Phase 14 replaces" | Not in CONTEXT; see Open Question 5 |
|
||
| `routes_table_phase13_test.go:270-282` `phase14-routes-absent` | Asserts the Phase 14 routes are not mounted | Update as routes land |
|
||
| `sm-user-plugin controllers/api_controller.go:795-806` | `"feedback_widget_hidden": false` default, then `app.Events.Collect(ctx, &classes.GetApiArrayEvent{User})` merges listener keys | Feedback plugin listens (fonoteka's own listener at `plugin.go:93` is the template) |
|
||
| `fonoteka/plugins/golem15/fonoteka/config/config.yaml:22-36` | `discogs.token`, `market_currency`, `max_covers`, `cover_max_bytes`, `cover_timeout_seconds`, `cover_host_suffix` | **Missing:** `user_agent`, `rate_threshold`, `wait_budget_seconds`, `retry_after_fallback_seconds`. Add them under `golem15.fonoteka.discogs.*`. Keep `base_uri` a code constant (PHP comment: "literal, never env()/user input") |
|
||
| `fonoteka.go/config/http.yaml` | CORS `paths` already include `_feedback/api/*` | Nothing to add |
|
||
|
||
## The three PHP jobs (port targets)
|
||
|
||
**AlbumCsvMatchJob** (`jobs/AlbumCsvMatchJob.php`) [VERIFIED: read this session]
|
||
- `public int $timeout = 240;` (line 36), so the Go job gets `conga.Timeout(240 * time.Second)`. `private const MAX_CANDIDATES = 10;` (line 38).
|
||
- Flow: import gone → `failJob {"error":"import_not_found"}`. `checkIfCanceled` → `cancelJob` (Go: `StopJob`). `isTerminalOrSuperseded` (status in canceled/done/importing/preview/failed, or `matching` with a different `match_job_id`) → `completeJob {"skipped": "<status string>"}`. Then status=`matching`, `startJob(total=row_count)`, `updateJobState(alreadyDone)` when resuming, per pending row (ordered by `row_index`): cancel check → `matchRow` → `updateJobState(done)`. End: status `preview`, `error_message=null`, `completeJob {"matched": done}`.
|
||
- `matchRow`: duplicate in the collection → `matched_csv`, candidates `[]`, `matched_album_id`. Discogs gate off → `matched_csv`, `[]`. Else `searchByQuery("artist title")`, then `searchByBarcode` when empty. 0 results → `matched`, `[]`. 1 result → `getRelease` + `mapRelease` into `draft_json`, candidates `[mapSearchResult]`, `matched`. More → the first 10 `mapSearchResult`, `draft_json=null`, `matched_ambiguous`.
|
||
- `DiscogsRateLimitException` → `pauseAndReschedule`: `delay = max(e.retryAfterSeconds, limiter.forToken(token).secondsUntilAvailable())`, pending count, status stays `matching`, Dispatch a new match job (label `fonoteka.csv.match`, count `pending`, metadata `{csv_import_id, resumed_after_rate_limit: true, retry_after: delay}`, **delay**), write `match_job_id = next`, then `completeJob {"paused":"discogs_rate_limited","retry_after":delay,"next_job_id":next}`. The `$rescheduling` static guard exists only for Laravel's sync driver; Go does not need it.
|
||
- Any other throwable → status `failed`, `error_message`, `failJob {"error": msg}` and **return normally** (no retry). In Go, call `FailJob` and return nil, or River retries up to 3 times.
|
||
|
||
**AlbumCsvImportJob** (`jobs/AlbumCsvImportJob.php`) [VERIFIED]
|
||
- No Discogs network. `importerCanWrite` (`Collection::accessibleBy(user)->whereKey(collection_id)`) before start and per row; on failure status `canceled` + `cancelJob`. Rows not `written`/`skipped`, ordered by `row_index`; `startJob(count)`.
|
||
- Per row inside `Album::withoutBroadcasting`: canonical rows (`matched_csv` with raw `id`) → `CsvCanonicalIdMatcher::match`, then `applyCsvOverwrite`/`applyCsvFill` or `createCsv`. Other rows → `inputForRow` (draft wins over CSV-mapped fields when there is a draft and the row is selected/matched/resolved; `genre` → `genre_id` via `resolveGenreId`), `writeOptions` (`allow_null_format`, first cover URL + `import_covers`), `matcher->find` → `fillEmpty` or `create`, `syncCsvRating`. Per-row exception → row `error`, `error_code=write_failed`.
|
||
- End: status `done`. When `writtenCount > 0`, publish `collection:{id}` event `collection.bulk_updated` `{"reason":"csv_import","count":N}`. `completeJob {"written": done}`. Outer throwable → `failed` + `failJob`.
|
||
- **Go gap:** `album_write_service.go` has `CreateAlbum`/`UpdateAlbum`/`SaveAlbum` but **no** CSV variants. Port `fillEmpty`, `applyCsvFill`, `createCsv`, `applyCsvOverwrite`, `resolveGenreId`, `syncCsvRating` and `create($options)` (PHP `AlbumWriteService.php:64-373`). `classes/csv/canonical_id.go` only has `CanonicalID(row)`; the scoped matcher query (`CsvCanonicalIdMatcher::match`) also needs a port. Bulk-broadcast suppression exists in lighthouse (Phase 11 SC4).
|
||
|
||
**WishlistDigestJob** (`jobs/WishlistDigestJob.php`, 49 lines) [VERIFIED]
|
||
- Read queue row (user_id, collection_id). Missing or `item_count <= 0` → `completeJob {"skipped": true}`. Else delete the row, then `NotificationService::mailWishlistDigest(subscriber, wishlist, count)`, then `completeJob {"sent": count}`.
|
||
- `mailWishlistDigest` (NotificationService.php:204-232): wishlist must exist with `kind === 'wishlist'` and subscriber must exist. Locale `en` when `preferred_locale === 'en'`, else `pl`. View `golem15.fonoteka::mail.wishlist_subscription_digest(-en)`. Vars `ownerName` (wishlist owner name), `itemCount`, `wishlistName`. To: subscriber email. **Copy** `views/mail/wishlist_subscription_digest.htm` and `-en.htm` (absent in Go `views/mail`) and register them in `mail.go`.
|
||
- Order matters: PHP deletes the row **before** sending. A failed send after the delete loses that digest. Keep the order for parity, and put the delete and the `CompleteJob` in one transaction where possible.
|
||
|
||
## Discogs client (INTG-01)
|
||
|
||
`DiscogsClient.php` [VERIFIED]: `TIMEOUT_SECONDS = 10`. Headers `Authorization: Discogs token=<token>`, `User-Agent: config('fonoteka.discogs.user_agent')`, `Accept: application/json`. URL = `rtrim(base_uri,'/') . path`, with ids cast to int in the path and free text only in the query. Endpoints: `getRelease(id)` → `/releases/{id}?curr_abbr=<marketCurrency>`; `getPriceSuggestions(id)` → `/marketplace/price_suggestions/{id}` (an empty object is normal); `getIdentity()` → `/oauth/identity`; `getMasterVersions(id)` → `/masters/{id}/versions?per_page=10`; `searchByBarcode` → `/database/search?barcode=&type=release`; `searchByQuery` → `/database/search?q=&type=release`.
|
||
|
||
Request loop: `limiter->acquire()` once, then loop. Each response → `syncFromHeaders(X-Discogs-Ratelimit-Remaining)`. 200 → json; 404 → null; 401/403 → `DiscogsTokenRejectedException`; 429 → `wait = registerRetryAfter(Retry-After)`; if `wait > remaining budget` throw `DiscogsRateLimitException(max(1, wait))`, else sleep and retry. Any other status → `ApplicationException("Discogs request failed with HTTP status N.")`. Logs carry only status and path.
|
||
|
||
Config values [VERIFIED: plugins/golem15/fonoteka/config/fonoteka.php:18-30]: `'base_uri' => 'https://api.discogs.com'`, `'user_agent' => env('DISCOGS_USER_AGENT', 'FonotekaApp/1.0 +https://github.com/golem15com/wn-fonoteka-plugin')`, `'rate_threshold' => 50`, `'wait_budget_seconds' => 15`, `'retry_after_fallback_seconds' => 10`.
|
||
|
||
Discogs itself throttles **by source IP**: 60/min authenticated, 25/min unauthenticated, as a 60 s moving average, with `X-Discogs-Ratelimit`, `-Used` and `-Remaining` headers [CITED: Discogs developer docs via search summary; the docs page returned 403 to direct fetch, LOW].
|
||
|
||
Domain classes to port alongside (PHP line counts): `DiscogsMapper` (434: `mapRelease`, `mapSearchResult`, `mapMasterVersion`), `ReleaseMatchScorer` (210, depends on completeness; Go `completeness.go` exists), `AlbumReleaseApplicator` (178, `apply(album, mapped, overwriteAll, coverOnly, dryRun)`, uses CoverImporter), `DiscogsImportResolver` (209), `DiscogsInputParser` (133, static `parse`, `normalizeBarcode`, `alternateBarcode`), `PriceSuggestionResolver` (88), `AlbumCoverFetcher` (575). The PHP suite has unit tests for every one of these (`tests/unit/Discogs*Test.php`, `ReleaseMatchScorerTest.php`, `PriceSuggestionResolverTest.php`, `AlbumReleaseApplicator*Test.php`, `CoverImporterTest.php`, `DiscogsRateLimiterTest.php`, `DiscogsClientTest.php`). Use them as test vectors. The pure classes (`DiscogsMapper`, `DiscogsInputParser`, `ReleaseMatchScorer`, `PriceSuggestionResolver`) suit **PHP-generated truth tables**, the precedent being `parity/csv_truth_tables.php`, which runs the PHP classes directly and writes `testdata/php_*.json`.
|
||
|
||
### Inbound (per-route) limiters — separate from the outbound Discogs limiter
|
||
|
||
| Key (PHP) | Limit | Response when over | Routes |
|
||
|---|---|---|---|
|
||
| `fonoteka-discogs-missing:` + user id or IP [VERIFIED: AlbumReleaseMatchController.php:31] | `tooManyAttempts(key, 60)`, `hit(key, 60)` | 429 `{"result":"error","code":"too_many_requests","retry_after":availableIn}` | albums match/apply (3 routes) **and** wishlist match/apply (same key, shared on purpose) |
|
||
| `fonoteka-discogs-import:` + id [VERIFIED: DiscogsImportController.php:68] | 20 per 60 s | 429 `{"result":"error","code":"too_many_requests"}` (no retry_after) | `albums/import/discogs` (checked **after** validation) |
|
||
| `fonoteka-recognize:` + id [VERIFIED: RecognizeApiController.php:53-57] | 10 per 60 s | 429 `{"error":"Too many requests"}` | recognize, both groups (after AiGate, before validation) |
|
||
| route `throttle:12,1` | 12/min | surf throttle 429 | token cover-price |
|
||
|
||
Use one `surf.MemoryStore` per plugin (process memory, precedent `PubfailCounter`). `Attempt` matches Laravel's `tooManyAttempts`+`hit` pair, and its `retryAfter` is `availableIn`. Order of checks per route is parity-visible. Copy it exactly (match/apply: album scope 404 → gate 503 → limiter → validation; import: gate → validation → limiter; recognize: AiGate 403 → limiter → validation → image guard).
|
||
|
||
## D-17: where the Discogs limiter state lives — **Postgres, UNLOGGED table, single atomic statement**
|
||
|
||
**PHP behaviour** [VERIFIED: DiscogsRateLimiter.php:23-233]: a fixed window (`WINDOW_SECONDS = 60`) per bucket. Bucket id = `substr(hash_hmac('sha256', token, app.key), 0, 32)`, never logged. Cache keys `discogs:ratelimit:<bucket>:{count,window_start,lock}`. Every read-modify-write runs under `Cache::lock(..., 5)->block(3, ...)` "regardless of driver" so that **two near-simultaneous workers** cannot both pass. `acquire()` loops: when a slot is free it increments and returns. Otherwise, if the wait exceeds the remaining wait budget (`wait_budget_seconds`, 15) it throws `DiscogsRateLimitException(max(1, windowRemainder))`, else it sleeps. `secondsUntilAvailable()` is read-only (1 when free). `syncFromHeaders()` **only tightens**: `implied = max(0, threshold - Remaining)` raises `count` when a window is active. `registerRetryAfter()` returns the numeric Retry-After or the fallback (10).
|
||
|
||
**Decision: Postgres.** Reasons:
|
||
1. The cross-process guarantee is still needed. conga supports `queue.work_in_serve: false` with a separate `fonoteka queue:work` process (systemd), which is documented in the app's `config/queue.yaml` and the conga README [VERIFIED: fonoteka.go/config/queue.yaml; modules/conga/README.md "Configuration"]. Rolling restarts also briefly run two binaries. A process-memory limiter would give each process its own 50/min. Discogs throttles by **source IP** at 60/min, so two processes on one host could overshoot by up to 40/min, and the CSV match job would degrade to 429 handling.
|
||
2. The cost is negligible: one short statement per Discogs request, which itself takes about 100-500 ms and is capped at 50/min per token.
|
||
3. UNLOGGED gives cache semantics (no WAL, emptied after a crash), the same as PHP's file/redis cache. A lost window only means one fresh window.
|
||
4. One atomic `INSERT ... ON CONFLICT DO UPDATE ... WHERE ... RETURNING` replaces PHP's lock. Postgres guarantees an atomic insert-or-update under concurrency, and a row "locked but not updated because an `ON CONFLICT DO UPDATE ... WHERE` condition was not satisfied ... will not be returned" [CITED: postgresql.org/docs/current/sql-insert.html]. So "a row returned" means "slot granted", with no advisory lock and no explicit transaction.
|
||
5. Clock injection (D-16) still works: pass `now` from the injected clock as a parameter instead of SQL `now()`, so tests drive the window with a fake clock. The unit tests of the waiting logic use an in-memory `RateStore` fake. One testcontainers test proves the SQL.
|
||
|
||
**Shape** (migration in the fonoteka plugin; the table is Go-only, so add it to `parity/schema_diff_test.go`'s Go-only allow-list as `golem15_fonoteka_settings` was) [ASSUMED design]:
|
||
```sql
|
||
CREATE UNLOGGED TABLE golem15_fonoteka_discogs_rate_windows (
|
||
bucket TEXT PRIMARY KEY, -- 32-hex HMAC id, never the token
|
||
window_start TIMESTAMPTZ NOT NULL,
|
||
hits INTEGER NOT NULL
|
||
);
|
||
-- tryAcquire($1 bucket, $2 now, $3 threshold): a returned row = granted
|
||
INSERT INTO golem15_fonoteka_discogs_rate_windows AS w (bucket, window_start, hits)
|
||
VALUES ($1, $2, 1)
|
||
ON CONFLICT (bucket) DO UPDATE SET
|
||
window_start = CASE WHEN $2 - w.window_start >= interval '60 seconds' THEN $2 ELSE w.window_start END,
|
||
hits = CASE WHEN $2 - w.window_start >= interval '60 seconds' THEN 1 ELSE w.hits + 1 END
|
||
WHERE $2 - w.window_start >= interval '60 seconds' OR w.hits < $3
|
||
RETURNING hits;
|
||
-- not granted: SELECT window_start → wait = max(1, 60 - floor(now - window_start))
|
||
-- syncFromHeaders: UPDATE ... SET hits = GREATEST(hits, $implied) WHERE bucket = $1 AND $now - window_start < interval '60 seconds'
|
||
```
|
||
Keep the HMAC key as `app.key` (the bucket id is not shared with PHP, so byte-compatibility does not matter, but secrecy does). The exception's `retryAfterSeconds` is the window remainder, not the leftover budget, because the CSV match job uses it as the re-dispatch delay.
|
||
|
||
## D-05: base-URL guard split, confirmed against PHP with differences
|
||
|
||
**PHP `SSRFGuard::assertSafeUrl`** [VERIFIED: plugins/golem15/golem/classes/security/SSRFGuard.php:40-67, config/ssrf.php:16-20]:
|
||
1. `parse_url` must give scheme and host, else `RuntimeException('Invalid URL')`.
|
||
2. Scheme must be `https` (`Only https:// scheme allowed`).
|
||
3. **Host allowlist** `config('golem15.golem::ssrf.allowed_hosts')`. A leading `.` is a suffix match; otherwise exact, case-insensitive. Default: `'oaidalleapiprodscus.blob.core.windows.net'` and `'.openai.com'`, overridable by `GOLEM15_SSRF_ALLOWED_HOSTS` (comma list). Failure: `Host not in allowlist: <host>`.
|
||
4. Resolve A/AAAA (fallback `gethostbynamel`). No answer → `Cannot resolve host`. Any private IP → `Resolves to private/loopback IP` (`127/8, 10/8, 172.16/12, 192.168/16, 169.254/16, 0/8, ::1, fe80::/10, fc00::/7`, plus `filter_var NO_PRIV_RANGE|NO_RES_RANGE`).
|
||
|
||
**Callers** [VERIFIED by grep + read]: `UserAiConfig::fromCredential` (line 43), `OrgAiConfig::fromOrg` (line 42) and `AiCredentialController::resolveTestConfig` (line 150, inline body `base_url`). All of them apply it **only when a base_url override is present**; the default provider URLs are never checked. `Plugin.php:82` also guards the AI-returned DALL-E image URL before `File::fromUrl`. Admin Golem Settings models go through `AIService::send` → `RequestSender::sendPostRequest`, which has **no** URL validation (`validateUrl` runs only on GET/download) [VERIFIED: apparatus/classes/RequestSender.php:37-57, 123-169].
|
||
|
||
**The split matches D-05 (admin trusted, user and org guarded). Differences to carry into the plan:**
|
||
|
||
| # | PHP | D-05 as written | Recommendation |
|
||
|---|---|---|---|
|
||
| 1 | User/org guard is **https-only + host allowlist + resolve-time IP check**. The allowlist is the primary defence (the class comment says so) | "private/reserved-IP dial guard" only | Port the allowlist (config key `golem15.golem.ssrf.allowed_hosts` in sm-golem-plugin, same default, env override) **and** keep the dial-time guard. Without the allowlist Go accepts `https://api.groq.com`, which PHP rejects: a parity and security regression |
|
||
| 2 | With the default allowlist, **any user/org Claude credential with an explicit `base_url` (even `https://api.anthropic.com/v1`) is rejected**, because `.anthropic.com` is not listed. Unknown whether production sets `GOLEM15_SSRF_ALLOWED_HOSTS` (`.env` is secret-guarded) | — | Open Question 1 |
|
||
| 3 | Guard failure throws `RuntimeException`. `AiCredentialController::test` catches only `ApplicationException`, and recognize calls the resolver outside its try, so PHP answers the **Winter 500 HTML page** (the recorded `POST ai-credential __bad-base-url` fixture shows that page for store) | — | Go must return the same 500 page (`controllers/api/winter_500.html` exists) on guard failure, not `{ok:false}`. Record cases (D-11 list) |
|
||
| 4 | PHP's IP check is resolve-time (DNS-rebind TOCTOU is acknowledged in the class doc) | dial-time | Go's dial-time check is strictly stronger and invisible to parity |
|
||
| 5 | `RequestSender` POST sets `CURLOPT_FOLLOWLOCATION` and **no timeout** | — | Guarded mode: never follow redirects (or re-guard each hop). Trusted mode: follow is acceptable. Set an explicit AI timeout of 120 s (PHP's `sendStream` uses `CURLOPT_TIMEOUT 120`) [ASSUMED value for non-stream] |
|
||
| 6 | `generateImage` returns a URL that callers must `assertSafeUrl` before fetching | — | Port `AssertSafeURL` as an exported helper in sm-golem-plugin. Fetch any AI-returned URL only via `fetchguard` AllowHosts mode with the same allowlist |
|
||
|
||
## D-11: missing parity cases and their upstream sidecars
|
||
|
||
All 14 `pending` routes already have **one** fixture each, and every one is a negative path recorded with `DISCOGS_TOKEN=` empty and no AI configured [VERIFIED: parity/manifest.yaml pending entries; fixture bodies read]:
|
||
|
||
| Route (pending) | Recorded today | Notes |
|
||
|---|---|---|
|
||
| `POST wishlist/albums/{id}/match` jwt | 404 `{"error":"Album not found"}` | |
|
||
| `POST wishlist/albums/{id}/apply-release` jwt | 404 | |
|
||
| `POST albums/match` jwt | 503 `discogs_disabled` | |
|
||
| `POST albums/{id}/match` jwt | 404 | |
|
||
| `POST albums/{id}/apply-release` jwt | 404 | |
|
||
| `POST albums/recognize` jwt | 403 `{"error":"AI features not available"}` | **Manifest says `status: 422`, the fixture says 403.** Fix the manifest when flipping to ported |
|
||
| `POST albums/import/discogs` jwt | 503 `discogs_disabled` | request body `{"query":...}` (the real field is `input`) |
|
||
| `POST ai-credential/test` jwt | 200 `{"ok":false,"error":"Incorrect API key provided: sk-parit******real. ..."}` | **Live OpenAI call, no sidecar.** Re-record |
|
||
| `POST discogs-credential/test` jwt | 200 `{"ok":false,"error":"Token Discogs jest nieprawidłowy lub wygasł."}` | **Live Discogs 401, no sidecar.** Re-record |
|
||
| `POST /api/v1/fonoteka/albums/recognize` token | 422 photo required | |
|
||
| `POST /api/v1/fonoteka/albums/{id}/cover-price/discogs` token | 200 `{"fetched":false,"reason":"no_source"}` | |
|
||
| `PATCH import/csv/{id}/rows/{rowId}` (ported route) | `discogs-off` 422 exists; **no success pick** | |
|
||
|
||
**Cases to record against PHP (each with an upstream sidecar unless marked "none")**, from routes.php, the Nuxt store (`app/stores/fonoteka.ts:315-440, 601-620, 1011, 1156`) and the MCP client (`fonoteka-mcp/src/client.ts:277-337`):
|
||
|
||
| # | Route | Case | Upstream sidecar |
|
||
|---|---|---|---|
|
||
| 1 | `PATCH import/csv/{id}/rows/{rowId}` | pick success → 200 `resolved` + draft (Nuxt sends `{selected_discogs_id: "<id>"}`) | Discogs `GET /releases/{id}?curr_abbr=EUR` 200 |
|
||
| 2 | same | pick rate-limited → 422 `discogs_rate_limited` | Discogs 429 (+/- Retry-After) |
|
||
| 3 | same | release 404 → 422 `discogs_unavailable` | Discogs 404 |
|
||
| 4 | `POST albums/{id}/match` | 200 candidates with `q`, `year`, `year_delta`, `medium` | `/database/search?q=` 200 |
|
||
| 5 | same | 422 validation (missing `q`) | none (stops before upstream; needs a Discogs-enabled user) |
|
||
| 6 | same | 429 `too_many_requests` inbound (61st call) | none, or 60 sidecar entries; prefer a Go unit test |
|
||
| 7 | same | 502 `discogs_token_rejected` | Discogs 401 |
|
||
| 8 | same | 429 `discogs_rate_limited` (budget exhausted) | Discogs 429 with Retry-After > 15 |
|
||
| 9 | `POST albums/match` | 200 candidates against `draft` | search 200 |
|
||
| 10 | `POST albums/{id}/apply-release` | 200 `{data, filled, remaining, draft}`, `dry_run:true`, `cover_only:true`, `overwrite_all` (Nuxt MatchReleaseDialog sends all three) | `/releases/{id}` 200 + cover image GET (i.discogs.com) |
|
||
| 11 | same | 200 `discogs_no_match` (release 404) | Discogs 404 |
|
||
| 12 | `POST wishlist/albums/{id}/match`, `.../apply-release` | 200 success twins (no `dry_run`; response has no `draft` key) | as 4 / 10 |
|
||
| 13 | `POST albums/import/discogs` | `draft` (release URL/id), `candidates` (barcode with many hits), `no_match` (barcode, zero hits: 200 with `barcode`), 422 validation, 429 inbound | search and/or release, master versions |
|
||
| 14 | `POST /api/v1/fonoteka/albums/{id}/cover-price/discogs` | `fetched:true` (discogs_id path, cover + price suggestion), `nothing_missing`, `ambiguous`, `rate_limited`, `refresh_price:true` (MCP sends `{refresh_price:true}` or `{}`) | release, price_suggestions, cover GET |
|
||
| 15 | `POST discogs-credential/test` | `{token}` body ok → `{"ok":true}`; stored token rejected (re-record); rate limited; disabled (no body, gate off: no upstream) | `/oauth/identity` 200/401/429 |
|
||
| 16 | `POST ai-credential/test` | inline `{provider:"claude", api_key, model}` ok; inline OpenAI 401 (re-record the existing case); stored credential; **unsafe `base_url` → 500 Winter page** (no upstream); **non-allowlisted host → 500**; no credential → `{"ok":false,"error":"No AI credential configured."}` | Anthropic `POST /v1/messages`, OpenAI `POST /v1/chat/completions` |
|
||
| 17 | `POST albums/recognize` jwt | 200 albums (BYOK user), 200 `{"albums":[],"code":"recognition_truncated"}` (both responses non-JSON, retry `stop_reason:max_tokens`/`finish_reason:length`), 200 `{"albums":[]}` (unparseable twice, not truncated), 502 provider error, 422 non-image bytes (ImageContentGuard), 429 inbound, admin-tier with global vision model | 1-2 AI exchanges per case |
|
||
| 18 | token `albums/recognize` | 200 success with MCP multipart (`photo` filename `cover`, `locale`) | AI exchange |
|
||
| 19 | feedback (new manifest section) | `GET {key}/config` 200 / 404 bad key / 403 origin; `POST {key}/submit` 202 / 422 / 422 bad image / 403; `OPTIONS` 204; `PUT me/hidden` 200 / 422 | none at request time. **G15Office exchanges are job-side**: record them as a job sidecar (create task + attachment) and assert from the River worker test |
|
||
| 20 | CSV flow | match job paused on 429 and resumed; import job writing rows | Discogs exchanges; job rows golden (`summer_jobs`) |
|
||
|
||
**How sidecars fit the existing layout** [VERIFIED: parity/README.md, php_parity.sh, parity_test.go:233-297, fixtures/routes/*.yaml]:
|
||
- Fixtures are tide flow YAML (`version: 1`, `steps[]` of `request`/`response`/`normalize`). Route cases live under `fixtures/routes/<METHOD>_<path>_<group>__<case>.yaml`, flows under `fixtures/nuxt|mcp`, and DB goldens next to them as `*.rows.json` (precedent: `nuxt-csv.rows.json`, compared in `fonoteka_flows_test.go:107-155`).
|
||
- Proposed sidecar: `<fixture>.upstream.yaml` next to each fixture, holding an ordered list of exchanges `{step, host, request:{method, path, query, headers (allow-listed: User-Agent, Accept, Content-Type, anthropic-version, anthropic-beta; auth headers masked to a placeholder), body (JSON-normalized; base64 image replaced by a sha256)}, response:{status, headers (Retry-After, X-Discogs-Ratelimit-*), body}}`. tide loads it with the flow. `replayPortedRoute` serves it through an in-process fake (a `RoundTripper` or an httptest server) that **asserts** each Go request against the recorded one and fails on unconsumed or extra exchanges.
|
||
- **Recording PHP's side:** PHP's Discogs base URI is a code literal (`'base_uri' => 'https://api.discogs.com'`, no env), and the AI path is raw curl. So the only way to capture what PHP really sends, without editing PHP, is a recording HTTPS proxy. Guzzle honours `HTTPS_PROXY` in any SAPI and libcurl honours `https_proxy`/`HTTPS_PROXY` [ASSUMED: Guzzle/libcurl env-proxy behaviour]. Add a `summer parity:upstream` command to tide that (a) acts as a CONNECT proxy, (b) terminates TLS with a locally generated CA (stdlib `crypto/x509`), (c) in **script mode** answers from a hand-authored vendor-response file (deterministic, no real tokens), or in **forward mode** passes through to the real vendor once, and (d) writes the sidecar. Extend `php_parity.sh serve` with `HTTPS_PROXY=http://127.0.0.1:8425` and `-d curl.cainfo=<parity CA>` (the Centrifugo recorder on `127.0.0.1:8424` is the precedent). Seed fake BYOK credentials (Discogs token matching `^[A-Za-z0-9_\-]{10,255}$`, AI key `sk-parity-…`) in `parity/fonoteka_reset.php`. `check_corpus --check-secrets` must stay green, so mask the auth headers in sidecars.
|
||
- If the proxy is judged too large, the fallback is: sidecars authored by hand from vendor docs, and the request assertion compares against a request derived from the PHP source rather than a recorded one. This weakens D-15 and needs user sign-off.
|
||
|
||
## WR-02 fix (D-10)
|
||
|
||
[VERIFIED: 13-REVIEW.md:131-158; csv_import_service.go:519-587, 639-705, 889-901]. In one `lagoon.Transaction`, `SELECT ... FOR UPDATE` the import row (`clause.Locking{Strength: "UPDATE"}`), re-check `csvBeforeCommit(cur.Status)` on the **locked** row, then cancel `cur.MatchJobID`/`cur.ImportJobID` from the locked row, write, and dispatch. Apply this to `UpdateCsvMapping`, `UpdateCsvRow` (the `save` closure) and `CancelCsvImport`. `UpdateCsvRow`'s Discogs fetch (up to 15 s wait budget) must happen **outside** the lock: fetch first, then lock, re-check status, and save. Otherwise a slow Discogs call holds the row lock and blocks commit. Add the review's interleaving test (commit vs mapping: at most one import job). Recommended, not mandated by D-10: make the match/import workers' own status writes (`matching`, `preview`, `done`, `canceled`) conditional on the locked row too. PHP's worker can otherwise resurrect a `canceled` import to `matching`. Response shapes are unaffected.
|
||
|
||
`jobs.CancelJob` runs on its own connection while the transaction holds the `csv_imports` row lock. It touches only `summer_jobs` and `river_job` rows, so no deadlock [ASSUMED: no FK or trigger from those tables back to csv_imports].
|
||
|
||
## fetchguard: what the guarded client needs
|
||
|
||
Current API [VERIFIED: modules/fetchguard/fetch.go, policy.go]: `Fetch(ctx, rawURL, Policy, *compass.Config) (*Result, error)`, GET only. `Policy{Mode (AllowHostsMode|PublicOnlyMode), AllowHosts, MaxBytes, Timeout}` plus **unexported** `tlsConfig` and `skipReservedCheck` test hooks, reachable only from the package's own tests (`withTestLoopback`). Redirects are never followed. `Proxy: nil`. The dial `Control` rejects reserved IPs (including NAT64/6to4-embedded v4 and zoned v6). Typed `Error{Reason}` with reasons `invalid_url|scheme|unresolvable|private_ip|network_error|too_large`.
|
||
|
||
Needed (design [ASSUMED], to be fixed in the plan):
|
||
- `fetchguard.NewClient(policy ClientPolicy, cfg *compass.Config) *Client` (or `*http.Client` plus helpers) with `Do(req)`, plus helpers `PostJSON(ctx, url, headers, v)`, `PutJSON`, `PostMultipart(ctx, url, headers, fields, file{field, name, mime, reader})`, and a `Bearer(token)` header helper. Responses are capped at `MaxBytes`, typed errors are kept, and the status code is returned, not judged (PHP callers inspect bodies, not status).
|
||
- Modes: `AllowHostsMode`, `PublicOnlyMode`, and **`TrustedMode`** (no dial guard, no host check, http allowed, for admin-configured endpoints such as LAN Ollama).
|
||
- **Test transport seam** that production input cannot reach: for example `fetchguard.WithTransport(ctx, rt)` or a `Client` option set only by code, which the parity harness uses to route `api.discogs.com`/`api.anthropic.com`/`api.openai.com`/G15Office to the sidecar fake while keeping the original host in the asserted request. Production URLs stay code literals (Discogs `base_uri`).
|
||
- Request bodies (a base64 photo of up to about 14 MB) are not capped by fetchguard. Set the timeout per consumer: Discogs 10 s (PHP `TIMEOUT_SECONDS = 10`), AI 120 s [ASSUMED], G15Office 30 s [ASSUMED; PHP has none].
|
||
- Update `modules/fetchguard/README.md`, `docs/services/outbound-http.md`, plus mentions in `docs/architecture/introduction.md` and `docs/setup/coming-from-wintercms.md` (RequestSender is now covered) [VERIFIED: grep docs].
|
||
|
||
## Redacting slog handler
|
||
|
||
PHP `RedactCredentialsTap` [VERIFIED: apparatus/classes/logging/RedactCredentialsTap.php]: keys `api_key, apikey, authorization, bearer, password, secret, token, webhook_secret, admin_password, OPENAI_API_KEY, ANTHROPIC_API_KEY, PERPLEXITY_API_KEY` (the todo omits the last three; include them, the comparison is case-insensitive). Values become `[REDACTED]` recursively; string values are also scrubbed with the patterns `/Bearer\s+[A-Za-z0-9._\-+\/=]+/i → 'Bearer [REDACTED]'`, `/sk-[A-Za-z0-9]{20,}/ → 'sk-[REDACTED]'`, `/x-api-key:\s*[^\s,]+/i → 'x-api-key: [REDACTED]'`. Go: a `slog.Handler` wrapper that rewrites the record message and every `Attr` (recursing into `slog.KindGroup` and handling `WithAttrs`/`WithGroup`). Nothing publishes a `*slog.Logger` today: beachcomber, lighthouse, postcard, conga and flare all do `app.Lookup[*slog.Logger]()` then fall back to `slog.Default()` [VERIFIED: grep]. So install it via `slog.SetDefault(slog.New(redact.Wrap(base)))` in the serve, queue:work and schedule:run entry points (or have backpack publish one). Host it in an existing module or a new small module; a new module needs a README and a root-table row. **SafeExceptionResponse:** surf already has `recoverJSON`/`recoverBare` panic recovery [VERIFIED: modules/surf/router.go:435-443, 638]. Whether it hides messages outside debug is unverified. A test in the plan should pin it.
|
||
|
||
## Golem plugin surface (sm-golem-plugin)
|
||
|
||
[VERIFIED: plugins/golem15/golem/*, read this session]
|
||
- `AIService::send(Prompt, ?config)`: config defaults to `Settings::getDefaultModel()`. No config → `failure('No AI model configured. Please add a model in Settings > AI.')`. Empty key → `failure('API key is not configured for the selected model.')`. Model defaults to `gpt-4o`. `applyModelSystemPrompt` sets the model's `system_prompt` only when the prompt has none. POST JSON (adapter headers). `false` → `failure('Failed to connect to AI service.')`. Invalid JSON → `failure('Invalid response from AI service: <json error>', ['raw'=>body])`. Otherwise `adapter->parseChatResponse` (status code ignored). Exceptions go through `safeExceptionMessage` ("Internal server error" outside debug).
|
||
- Anthropic adapter [VERIFIED: AnthropicAdapter.php:13-15, 23, 59-61]: headers `x-api-key: <key>`, `anthropic-version: 2023-06-01`, `anthropic-beta: files-api-2025-04-14`. Endpoint `rtrim(base_url) . '/messages'`. Payload `{model: prompt.model ?? modelId, messages[], system?, max_tokens: options.max_tokens ?? options.max_completion_tokens ?? 4096, ...other options except response_format and max_completion_tokens}`. Image blocks: `data:` URL → `{type:image, source:{type:base64, media_type, data}}`, else `{source:{type:url}}`; file blocks → `document/file_id`. Response: `error.message` → failure; `content[0].text`; usage mapped to prompt/completion/total; `raw` = the whole body (recognition reads `raw.stop_reason`).
|
||
- OpenAI adapter [VERIFIED: OpenAIAdapter.php:13, 21, 50-62]: `Authorization: Bearer <key>`; `/chat/completions`; `toApiPayload` (system message first) minus `response_format`; `max_tokens` → `max_completion_tokens`; `reasoning_effort: low` for `/^(o\d|gpt-5)/i` unless set. Response: `choices[0].message.content`; `raw.choices[0].finish_reason`.
|
||
- Content-Type: RequestSender adds `Content-Type: application/json` first; adapters append their headers. PHP json_encode escapes `/` and non-ASCII (`ł`) unless flagged. **Upstream request-body parity:** compare JSON semantically in the sidecar assertion, not byte-for-byte.
|
||
- `Settings` [VERIFIED: models/Settings.php:13, 19-118]: `$settingsCode = 'golem_settings'` (not `golem15_golem_settings` as D-01 states). `getModels()` = `array_filter(models, is_enabled)`, which **preserves keys**, so `getDefaultModel()`'s `return $models[0] ?? null` fallback returns null when model 0 is disabled even if others are enabled. Port the quirk. `getVisionModel()` = first enabled with `accepts_images`; also `getImageModel` (generates_images), `getModelByName`, `getFileModel` (has_files_endpoint). `fields.yaml` repeater fields: `name, adapter (openai|anthropic, default openai), api_key (text, password attribute), base_url (default https://api.openai.com/v1), model, system_prompt, is_enabled, is_default, generates_images, accepts_images, has_files_endpoint, max_completion_tokens (default 4096)`. **`api_key` is stored in plaintext**: Settings has no encryption and the only encryption migration targets FaceSettings `admin_password` [VERIFIED: updates/version.yaml, encrypt_face_settings.php].
|
||
- Recognition (`AlbumRecognitionService.php`): `MAX_ALBUMS = 30`, `MAX_COMPLETION_TOKENS = 8192`. Admin path with no config and no vision model → `ApplicationException('No vision model configured in Golem AI settings.')`. System prompt text (lines 237-292) with formats, genre hint (up to 20 distinct genres via albums in the collection, newest first) and language directive (`en`/`pl` from BCP-47). User message with the image. Strip fences, json_decode. On failure, one retry with the appended "IMPORTANT: Respond with raw JSON only…". A retry parse failure with finish reason `length|max_tokens` → `RecognitionTruncatedException` (controller → 200 `{"albums":[],"code":"recognition_truncated"}`), else `[]`. Normalisation: drop nameless rows; nullableString fields; year 1889-2100; format must be in `Album::FORMATS`; tracklist via `TracklistTextParser` (Go `tracklist_text_parser.go` exists).
|
||
- Not ported: FaceService, CompreFaceClient, ChatContextCollector, Conversations/Messages/Chat components (D-04). `routes.php` has no routes.
|
||
- **Settings storage gap (needs a decision, Open Question 2):** cabana settings are typed singleton rows (id=1), scalar fields only. `nestedValue` is rejected, there is no `repeater` type in `formFieldTypes`, and unknown YAML keys fail boot [VERIFIED: cabana/settings.go:66-95, form_schema.go:22-44]. The project convention (Phase 5, user-resolved for fonoteka) is a dedicated typed table, not `system_settings`. Recommendation: table `golem15_golem_models` (one row per repeater item, `sort_order`, `api_key lagoon.Encrypted json:"-"`), edited through a cabana list+form controller under Settings › AI, plus an idempotent importer for the PHP `system_settings` row `item='golem_settings'` (a migration that runs only when `system_settings` exists, or a cutover command). This also reconciles D-03's "encrypted" with PHP's plaintext.
|
||
|
||
## Feedback plugin surface (sm-feedback-plugin)
|
||
|
||
[VERIFIED: plugins/golem15/feedback/*, read this session]
|
||
- Routes: group `/_feedback/api/v1` with `json.response`: `GET {key}/config` (`throttle:feedback-config` = 60/min by IP), `POST {key}/submit` (`throttle:feedback-submit` = 10/min by IP), `OPTIONS {any}` → `response('', 204)` with `where('any','.*')`. A separate group with `jwt.auth`: `PUT me/hidden`. CORS `paths` include `_feedback/api/*` in both PHP and Go configs.
|
||
- `config`: `keyMatches` = `enabled && widget_key !== '' && hash_equals`, else 404 `{"error":true,"message":"Not found"}`. Origin gate: no Origin header → allowed; unparseable host → false; empty allow-list → **fail closed** 403 `{"error":true,"message":"Origin not allowed"}`; host compared lowercase against `getAllowedOriginHosts()` (lines split on `[\r\n]+`, trimmed, `parse_url` host or the raw line, lowercased, unique). 200 `{"success":true,"data":{"position","allowHide","colors":{"primary","accent"},"labels":{"title","placeholder","success"}}}` with `?lang=en` selecting `_en`, anything else `_pl`.
|
||
- `submit`: validation (`message required|string|max:5000`, `type required|in:bug,feature,other`, `email nullable|email|max:255`, `page_url required|string|max:2000`, `user_agent nullable|max:500`, `console_log nullable|max:20000`, `screenshot nullable|image|mimes:jpg,jpeg,png,gif,webp|max:10240`) → 422 `{"error":true,"message":"Validation failed","details":errors}`. ImageContentGuard (finfo sniff in jpeg/png/gif/webp + `getimagesize`) → 422 with `details.screenshot = ["The file is not a valid image."]`. Create the submission, attach the screenshot (`File`, `is_public = true`, attachOne `screenshot`), dispatch `SyncFeedbackToG15Office`, 202 `{"success":true}`. Go `classes/image_guard.go` (`IsAllowedImage`, `SniffImageMIME`) can be copied into the plugin (PHP keeps per-plugin copies on purpose).
|
||
- `me/hidden`: `hidden required|boolean` → 422 `{"error":"Validation failed","errors":...}`; `UserPreference::setWidgetHidden` (updateOrCreate) → 200 `{"hidden":bool}`.
|
||
- `getApiArray` listener: `feedback_widget_hidden => UserPreference::isWidgetHidden(id)` (value of `hidden`, false when no row).
|
||
- Settings: `$settingsCode = 'golem15_feedback_settings'` (line 14). `initSettingsData`: `enabled=true, allow_hide=false, widget_key='wk_'.Str::random(32), position='bottom-right', g15office_task_priority='normal'`, and the pl/en label defaults. Rules are all nullable (position `in:bottom-right,bottom-left,middle-right,middle-left`, priority `in:low,normal,high,urgent`). Fields use `colorpicker` and `readOnly`, which cabana does **not** support (unknown keys fail boot). Adapt the Go `fields.yaml` (text fields, `attributes: {readonly: …}`) or add the types to cabana. Scalar-only, so a dedicated typed singleton table works with cabana settings. `widget_key` must be generated on first creation, because Winter's `instance()` persists defaults on first access [ASSUMED Winter behaviour].
|
||
- Models and tables: `golem15_feedback_submissions` (id, message text, type string default 'bug', email, page_url 2000, user_agent, console_log longtext, status default 'pending' indexed, g15office_task_id, g15office_error text, timestamps). `golem15_feedback_user_preferences` (id, user_id unique, hidden bool default false, timestamps). Status values pending/sent/failed. Type map bug→Bug, feature→Feature, other→Task.
|
||
- `SyncFeedbackToG15Office`: `$tries = 3`, `$backoff = 30`. Config `feedback.g15_office.{base_url, token, project}` from env `G15_OFFICE_BASE_URL/TOKEN/PROJECT`. `createTask` POST JSON `{base}/_support/api/v1/projects/{project}/tasks` with Bearer, `Accept: application/json`, body `array_filter({title:"Feedback: "+Str::limit(collapsed message,80), description, type, status?, priority ?: 'normal'})`. Then `attachFile` multipart POST `{base}/_support/api/v1/tasks/{hashId}/attachments` (field `file`). Decode: `false` → "connection error"; non-JSON → "invalid response body"; `error` truthy → message. Success: status `sent`, task id stored; failure: status `failed`, `g15office_error`, rethrow (retry). Description markdown format at lines 198-226. In Go: `conga.MaxAttempts(3)`; River's backoff differs from 30 s (acceptable, not parity-visible) [ASSUMED].
|
||
- `embed.js` (643 lines) is served at `/plugins/golem15/feedback/assets/js/embed.js`, which the Nuxt proxy uses (`nuxt.config.ts:254, 369`). No framework capability serves public plugin assets [VERIFIED: pact/capabilities.go interface list], so the plugin registers a GET route serving the embedded file.
|
||
- **No admin submissions list exists in PHP** (no controller). D-13's list is a net-new cabana list controller, which is fine but not parity-checked.
|
||
- Parity: feedback routes are outside fonoteka's `routes.php` (154 ids). Follow the `userAPIRouteIDs`/`realtimeRouteIDs` precedent in `parity/check_corpus.go:619-630` and `routes.snapshot` (172 lines) by adding a `feedbackRouteIDs` set and an auth group.
|
||
|
||
## Console commands
|
||
|
||
- `fonoteka:prune-notifications` [VERIFIED: PruneNotifications.php:20-21]: `Notification::where('created_at', '<', now()->subDays(90))->delete();` then `$this->info("Pruned {$deleted} notifications older than 90 days.");`, exit SUCCESS. Go table `golem15_fonoteka_notifications`. Register in `Commands()` (plugin.go:267). The scheduler entry already exists.
|
||
- `fonoteka:reindex {--drop-old-items-index}` [VERIFIED: ReindexAlbums.php:16-17, 49-103]: not configured (`scout.driver === typesense && Settings search_use_typesense && api_key`) → error `Typesense reindex skipped: enable Fonoteka Typesense search and configure TYPESENSE_API_KEY.` FAILURE. Any album with `collection_id IS NULL OR <= 0` → `Album reindex aborted: every active Album must have a positive collection_id.` FAILURE. Then `scout:flush` and `scout:import` (chunks of 500 [ASSUMED Scout default]). If the album table is empty, create the collection from the schema. Then search `q=*, query_by=name, filter_by=collection_id:=0, per_page=1`; `found != 0` → `Album reindex failed integrity check: collection_id:=0 documents exist.` FAILURE. With `--drop-old-items-index`, delete `golem15_fonoteka_items` (`Deleted legacy Typesense collection: …` or, on ObjectNotFound, `Legacy Typesense collection already absent: …`). Success message `Album index rebuilt; collection_id:=0 document count is zero.` Go: `beachcomber.From(app)`; `Engine().Name()`, `Configured()`, `settingsGate{}.Enabled`; `Engine().Flush(index)`; batched `Upsert(index, schema, docs)` via `album.ToSearchableArray`; `beachcomber.SearchPage` for `Found`. **Gap:** `Flush` treats a missing index as nil, so the "already absent" message cannot be told apart. Add an optional `beachcomber.IndexDropper` (`DropIndex(ctx, index) (existed bool, err)`) to the framework (README + docs), or accept a single message (Open Question 6). "Before" per SRCH-02 is the DB check in PHP; "after" is the Typesense `found` check.
|
||
- `fonoteka:oauth-client` shipped in Phase 8 (`console/oauth_client.go`).
|
||
|
||
## Submodule workflow for the two new repos
|
||
|
||
- Both remotes **are reachable and empty**: `git ls-remote git@git.golem15.com:golem15/sm-golem-plugin.git` and `.../sm-feedback-plugin.git` exit 0 with no refs, while a nonexistent repo errors with "Cannot find repository" [VERIFIED: probe this session]. Neither is mounted yet; `.gitmodules` lists only `plugins/golem15/user` [VERIFIED].
|
||
- `git submodule add` of an **empty** remote cannot stage a gitlink (there is no commit to check out) [ASSUMED git behaviour]. Recommended sequence: `git init` the plugin directory at `plugins/golem15/golem`, write and commit the initial module there, `git remote add origin …`, push `master` (a user action in prior phases; executors were told not to push, see 13-VERIFICATION human item), then in fonoteka.go run `git submodule add git@git.golem15.com:golem15/sm-golem-plugin.git plugins/golem15/golem` (an existing repository at the path is staged without cloning). Commit `.gitmodules` and the gitlink separately.
|
||
- Wiring (precedent: sm-user-plugin): plugin `go.mod` with `module git.golem15.com/golem15/sm-golem-plugin`, `replace git.golem15.com/golem15/summercms => ../../../../summercms.go` (feedback also `replace git.golem15.com/golem15/sm-user-plugin => ../user`). App `go.work` `use ./plugins/golem15/golem` and `./plugins/golem15/feedback`. App `go.mod` `require` + `replace … => ./plugins/golem15/<name>`. The fonoteka plugin's `go.mod` requires sm-golem-plugin with `replace => ../golem`. `summer.yaml` adds ids `golem15.golem` and `golem15.feedback` (golem before fonoteka). Regenerate `plugins.gen.go`.
|
||
- sm-user-plugin `master` is still 5 commits ahead of origin [VERIFIED: `git status -sb`], so a fresh clone cannot check out the pointer. That is unrelated to this phase but blocks CI clones.
|
||
- README rules apply to plugin READMEs (never name the consuming application).
|
||
|
||
## Architecture Patterns
|
||
|
||
### System Architecture Diagram
|
||
|
||
```
|
||
Nuxt SPA / fonoteka-mcp / embed.js
|
||
│ HTTPS
|
||
▼
|
||
surf router ── group middleware (jwt | inv_token+scope | public throttle) ──► controllers/api
|
||
│ │
|
||
│ in-controller limiter (surf.MemoryStore) ◄────────────────────┤
|
||
│ ▼
|
||
│ classes: gates/resolvers (DiscogsAllowed, ResolveAIConfig)
|
||
│ │ │
|
||
│ ┌──────────────────────┘ └──────────────┐
|
||
│ ▼ ▼
|
||
│ Discogs client ──► rate limiter ──► Postgres UNLOGGED sm-golem-plugin AIService
|
||
│ │ (atomic upsert) │ adapter (anthropic|openai)
|
||
│ ▼ ▼
|
||
│ fetchguard.Client (guarded) ──dial guard──► api.discogs.com fetchguard.Client (guarded | trusted)
|
||
│ │ │
|
||
│ └──► CoverImporter (AllowHosts *.discogs.com) ──► storage │──► AI provider
|
||
▼
|
||
DB transaction ── conga.Dispatch ──► river_job + summer_jobs
|
||
│ LISTEN/NOTIFY
|
||
▼
|
||
River worker (serve or queue:work)
|
||
┌──────────────┬───────────────┬──────────────────┐
|
||
▼ ▼ ▼ ▼
|
||
CSV match job CSV import job digest job feedback G15Office job
|
||
(240 s; 429 → (write service, (delete row, (fetchguard POST JSON +
|
||
re-Dispatch bulk broadcast) postcard mail) multipart, 3 attempts)
|
||
with Delay)
|
||
```
|
||
|
||
### Recommended layout (per `.planning/notes/plugin-layout-winter-directories.md`)
|
||
```
|
||
fonoteka/plugins/golem15/fonoteka/
|
||
├── classes/discogs/ # client.go, rate_limiter.go, mapper.go, scorer.go, applicator.go,
|
||
│ # import_resolver.go, input_parser.go, price_suggestion.go, cover_fetcher.go, errors.go
|
||
├── classes/recognition.go # AlbumRecognitionService port (+ truncated error)
|
||
├── controllers/api/ # release_match_controller.go, wishlist_release_match_controller.go,
|
||
│ # recognize_controller.go, discogs_import_controller.go, album_cover_fetch_controller.go
|
||
├── jobs.go (or jobs/) # csv match/import, digest workers
|
||
├── console/ # prune_notifications.go, reindex.go
|
||
└── updates/ # discogs rate windows UNLOGGED table
|
||
plugins/golem15/golem/ (sm-golem-plugin)
|
||
├── plugin.go routes.go(no routes) classes/{service.go, providers/*.go, prompt.go, response.go,
|
||
│ prompt_factory.go, ssrf_guard.go} models/{ai_model.go} controllers/ (admin list/form) updates/ lang/en
|
||
plugins/golem15/feedback/ (sm-feedback-plugin)
|
||
├── plugin.go routes.go controllers/api/{feedback_api_controller.go, me_hidden_controller.go}
|
||
├── classes/{g15office_client.go, image_guard.go} jobs/sync_g15office.go models/ updates/ lang/{en,pl}
|
||
└── assets/js/embed.js (embedded, served by a route)
|
||
```
|
||
`models/` stays a leaf package (the rule is enforced by tooling).
|
||
|
||
### Pattern 1: Worker with summer_jobs outcome semantics
|
||
**What:** Translate PHP JobManager calls one-to-one: `startJob` → `m.StartJob(ctx, id, total)`, `updateJobState` → `UpdateJobState`, `completeJob(meta)` → `CompleteJob`, `failJob(meta)` → `FailJob` **then return nil** (PHP does not retry), `cancelJob` inside a job → `StopJob`, `checkIfCanceled` → `CheckIfCanceled`. `conga.JobID(ctx)` gives the row id.
|
||
```go
|
||
// Source: modules/conga/README.md Usage + fonoteka jobs.go pattern
|
||
conga.Job(p.matchCsv, conga.OnQueue(classes.CsvMatchQueue), conga.Timeout(240*time.Second))
|
||
// self re-dispatch on rate limit (inside the job):
|
||
next, err := m.Dispatch(ctx, gdb, classes.CsvMatchArgs{CsvImportID: imp.ID}, conga.DispatchOpts{
|
||
Label: classes.CsvMatchLabel, Queue: classes.CsvMatchQueue, Count: pending,
|
||
Metadata: map[string]any{"csv_import_id": imp.ID, "resumed_after_rate_limit": true, "retry_after": delay},
|
||
Delay: time.Duration(delay) * time.Second,
|
||
})
|
||
```
|
||
|
||
### Pattern 2: Injected clock (D-16)
|
||
```go
|
||
type Clock interface {
|
||
Now() time.Time
|
||
Sleep(ctx context.Context, d time.Duration) error // returns ctx.Err() on cancel
|
||
}
|
||
```
|
||
The limiter's `Acquire`, the client's 429 loop and the job's delay computation take it. Tests use a fake whose `Sleep` advances `Now`. The `Sleep` respects ctx so the 240 s job timeout and request cancellation interrupt waits.
|
||
|
||
### Pattern 3: Tier-aware AI config
|
||
`classes.AIConfig` gains a trust marker that only the AdminVisionModel adapter sets. Recognition and ai-credential/test call golem `Send(ctx, prompt, cfg)`. The golem plugin selects `fetchguard` TrustedMode for admin configs and guarded PublicOnly for everything else, after `AssertSafeURL(baseURL)` (allowlist) when a user/org override is present.
|
||
|
||
### Anti-Patterns to Avoid
|
||
- **Sleeping out the Discogs window inside the job:** PHP explicitly never does this. Re-dispatch with `Delay`.
|
||
- **Making the Discogs base URI configurable:** PHP keeps it a code literal for SSRF reasons. Inject the test transport instead.
|
||
- **`{ok:false}` for an unsafe base_url:** PHP answers 500.
|
||
- **Marshalling multi-key job metadata via Go maps and comparing strings:** see Pitfall 4.
|
||
- **Holding the CSV row lock across a Discogs call:** see the WR-02 section.
|
||
|
||
## Don't Hand-Roll
|
||
|
||
| Problem | Don't Build | Use Instead | Why |
|
||
|---------|-------------|-------------|-----|
|
||
| Private-IP / reserved-range checks | New CIDR tables per consumer | `fetchguard` dial `Control` (`isReservedOrPrivate`) | Already a superset of PHP's lists, with NAT64/6to4 and zone handling |
|
||
| Inbound per-key counters | Another bespoke map+mutex | `surf.MemoryStore.Attempt` | Atomic check+increment, returns retryAfter |
|
||
| Cover download allow-listing | New downloader | `classes.CoverImporter` / `fetchguard` AllowHostsMode | Ported and parity-tested in Phase 12 |
|
||
| Job progress/cancel bookkeeping | Custom status table | conga `summer_jobs` API | `job_contract.go` and the parity row goldens already depend on it |
|
||
| Image sniffing | New magic-byte parser | `classes/image_guard.go` (copy into feedback) | Matches PHP finfo + getimagesize parity cases |
|
||
| Cross-process rate window | Advisory-lock dance | One `INSERT … ON CONFLICT DO UPDATE … WHERE … RETURNING` | Atomic per Postgres docs |
|
||
| Tracklist parsing in recognition | New parser | `tracklist_text_parser.go` | PHP reuses one parser (D-22) |
|
||
|
||
**Key insight:** almost every primitive exists. Phase risk lies in faithful edge-case parity (ordering of checks, error-to-status maps, metadata shapes) and in the test harness for upstream calls, not in new infrastructure.
|
||
|
||
## Common Pitfalls
|
||
|
||
### Pitfall 1: Allowlist omitted from the user/org base_url guard
|
||
**What goes wrong:** Go accepts an `https://api.groq.com` override that PHP rejects (500); parity diff and SSRF surface widen. **Avoid:** port `SSRFGuard` fully (scheme, allowlist, IP) before the dial guard. **Warning sign:** no `allowed_hosts` config key in sm-golem-plugin.
|
||
|
||
### Pitfall 2: PHP 500 vs `{ok:false}` on guard failure
|
||
**What goes wrong:** Go's "nicer" error handling returns 200 `{ok:false}`. **Avoid:** record cases 16 (unsafe and non-allowlisted base_url) and assert the Winter 500 page.
|
||
|
||
### Pitfall 3: Settings key and plaintext key mismatch
|
||
**What goes wrong:** Importing from `system_settings` with item `golem15_golem_settings` finds nothing; PHP's is `golem_settings`. Or the importer expects ciphertext. **Avoid:** Open Question 2 resolved before planning the storage.
|
||
|
||
### Pitfall 4: Job metadata key order
|
||
**What goes wrong:** conga `Metadata map[string]any` marshals keys sorted, while PHP preserves insertion order (`{"paused","retry_after","next_job_id"}`). `nuxt-csv.rows.json` compares the `metadata` **string** [VERIFIED: parity/fonoteka_flows_test.go:84-95]. **Avoid:** compare metadata JSON semantically in new goldens, or extend conga to accept an ordered value. Single-key metadata is unaffected.
|
||
|
||
### Pitfall 5: Unregistered-kind assertions break when workers land
|
||
**What goes wrong:** `TestJobContractDispatchWhileWorkerRuns`/`TestCsvJobRows` and conga's `ErrUnregisteredKindQueue` assumptions (queues "unserved") flip once the three kinds register. **Avoid:** update the tests in the same plan. Check that `nuxt-csv` still expects `cancelled` River states (the parity app handler does not start a worker).
|
||
|
||
### Pitfall 6: Retrying jobs PHP never retries
|
||
**What goes wrong:** returning an error after `FailJob` makes River retry, which double-processes rows. **Avoid:** PHP catches Throwable and returns; Go returns nil after `FailJob`. Only the 240 s timeout or a panic should surface as a River retry.
|
||
|
||
### Pitfall 7: Discogs lock-held fetch in row edit
|
||
See the WR-02 section: fetch outside the lock, then re-check under the lock.
|
||
|
||
### Pitfall 8: `getDefaultModel` key-preservation quirk
|
||
`array_filter` keeps keys, so `$models[0] ?? null` can be null while enabled models exist. Port the quirk deliberately and test it.
|
||
|
||
### Pitfall 9: Recognize manifest status mismatch
|
||
The manifest case says 422 but the fixture shows 403 for `POST albums/recognize jwt`. `TestCheckCorpusPortedCaseStatus` will fail on flip unless fixed.
|
||
|
||
### Pitfall 10: Upstream JSON byte equality
|
||
PHP `json_encode` escapes `/` and Unicode. Go escapes differently (json/v2 in Go 1.27). Assert upstream bodies semantically (normalise, then compare). Response bodies to our clients stay byte-compared via the existing `wire` helpers.
|
||
|
||
### Pitfall 11: cabana rejects unknown YAML keys
|
||
The feedback `colorpicker`/`readOnly` and the golem `repeater` fail boot. Adapt the YAML or add the types to cabana (a framework change with docs).
|
||
|
||
## Code Examples
|
||
|
||
### Redacting handler skeleton
|
||
```go
|
||
// Source: log/slog Handler contract (stdlib); key list from RedactCredentialsTap.php
|
||
type redactHandler struct{ next slog.Handler }
|
||
func (h redactHandler) Enabled(ctx context.Context, l slog.Level) bool { return h.next.Enabled(ctx, l) }
|
||
func (h redactHandler) Handle(ctx context.Context, r slog.Record) error {
|
||
out := slog.NewRecord(r.Time, r.Level, scrub(r.Message), r.PC)
|
||
r.Attrs(func(a slog.Attr) bool { out.AddAttrs(redactAttr(a)); return true })
|
||
return h.next.Handle(ctx, out)
|
||
}
|
||
func (h redactHandler) WithAttrs(as []slog.Attr) slog.Handler { /* redact each, then h.next.WithAttrs */ }
|
||
func (h redactHandler) WithGroup(n string) slog.Handler { return redactHandler{h.next.WithGroup(n)} }
|
||
```
|
||
|
||
### Inbound limiter (Laravel tooManyAttempts + hit)
|
||
```go
|
||
// Source: modules/surf/limiter_store.go Store.Attempt
|
||
ok, _, retry := store.Attempt("fonoteka-discogs-missing:"+strconv.FormatUint(uint64(user.ID), 10), 60, time.Minute)
|
||
if !ok {
|
||
return wire.JSON(w, 429, ordered("result","error","code","too_many_requests","retry_after", int(math.Ceil(retry.Seconds()))))
|
||
}
|
||
```
|
||
(`wire.JSON`/`ordered` stand for the app's existing ordered-JSON helpers; use whatever the Phase 12/13 controllers use.)
|
||
|
||
## State of the Art
|
||
|
||
| Old Approach | Current Approach | When Changed | Impact |
|
||
|--------------|------------------|--------------|--------|
|
||
| Apparatus `RequestSender` (curl, resolve-time check, follows redirects, no POST guard) | `fetchguard` client with dial-time guard | This phase | Closes DNS-rebind and redirect SSRF gaps |
|
||
| Winter `SettingsModel` over `system_settings` | Dedicated typed tables + cabana settings | Phase 5 decision | Golem repeater and feedback settings need storage choices |
|
||
| Laravel sync/redis queue | River via conga (LISTEN/NOTIFY) | Phase 11 | Delayed re-dispatch is native (`Delay`) |
|
||
|
||
**Deprecated/outdated:** `REQUIREMENTS.md` INTG-02 "Anthropic Go SDK" (reword per D-06); API-08 "sitemap" (reword per D-13/D-14); ROADMAP SC4/SC5/SC6 (reword per D-06/D-07/D-14).
|
||
|
||
## Assumptions Log
|
||
|
||
| # | Claim | Section | Risk if Wrong |
|
||
|---|-------|---------|---------------|
|
||
| A1 | Guzzle honours `HTTPS_PROXY` in non-CLI SAPIs and libcurl honours `https_proxy` for raw curl, so a recording proxy captures PHP's Discogs and AI calls | D-11 | Recording proxy needs another capture route (Laravel Http events for Discogs; admin-tier local base_url for AI) |
|
||
| A2 | `git submodule add` cannot stage an empty remote; pre-initialising the directory works | Submodules | Minor workflow change |
|
||
| A3 | Winter `SettingsModel::instance()` persists `initSettingsData()` defaults on first access (widget_key stable) | Feedback | widget_key generation timing differs |
|
||
| A4 | AI request timeout 120 s, G15Office 30 s | fetchguard | Hung upstream holds a request or worker longer or shorter |
|
||
| A5 | Scout imports in chunks of 500 | Reindex | Batch size only |
|
||
| A6 | River backoff instead of a fixed 30 s for the G15Office job is acceptable | Feedback | Not parity-visible |
|
||
| A7 | The UNLOGGED table schema and SQL shown for the limiter | D-17 | Design detail to settle in the plan |
|
||
| A8 | No FK or trigger links summer_jobs/river_job back to csv_imports (no deadlock under the row lock) | WR-02 | Lock ordering issue |
|
||
| A9 | The fetchguard client API shape (NewClient, modes, helpers, transport seam) | fetchguard | Plan-level design |
|
||
|
||
## Open Questions (RESOLVED)
|
||
|
||
1. **Is `GOLEM15_SSRF_ALLOWED_HOSTS` set in production?** We know the default allowlist rejects any user/org base_url outside `*.openai.com` (Anthropic included). The `.env` is secret-guarded. Recommendation: ask the user for the production value (names only) and make it the Go config default for the application; keep the PHP default in the plugin. **DEFERRED (Phase 15 cutover):** 14-04 keeps the PHP default allowlist and honours the env override. The operator confirms the production value at cutover.
|
||
2. **Golem settings storage and key.** We know the PHP code is `golem_settings` with a plaintext `api_key` inside a repeater JSON, and cabana has no repeater. D-01 says `golem15_golem_settings`; D-03 says "encrypted" and "reading the existing settings row". Recommendation: confirm a dedicated `golem15_golem_models` table (encrypted key, admin list/form) plus an importer from `system_settings` item `golem_settings`; correct D-01's key string. **RESOLVED:** D-18 (user, 2026-10-03): the `golem15_golem_models` table with an encrypted key, plus an importer from `golem_settings`.
|
||
3. **Feedback settings storage.** The same convention question for `golem15_feedback_settings` (scalar, so a typed singleton table fits cabana). Confirm a typed table plus an importer. **RESOLVED:** D-18: a typed singleton table plus an importer (14-05).
|
||
4. **Recording-proxy scope (D-15).** Building `summer parity:upstream` (MITM CA, script/forward modes) is the only faithful way to capture PHP's actual upstream requests. Confirm it is in scope, or accept hand-authored sidecars. **RESOLVED:** D-19 (user): `summer parity:upstream` is in scope (14-01).
|
||
5. **Phase 10.1 admin Discogs stubs** (`discogsLookup`, `discogsSync`) say "Phase 14 replaces". PHP has no such admin actions. In or out of scope? **RESOLVED:** out of scope. The stubs are left unchanged and 14-03 records this as an assumption.
|
||
6. **Reindex "already absent" message:** add `beachcomber.IndexDropper` (framework change) or accept one message for both outcomes? **RESOLVED:** `beachcomber.IndexDropper` and `EnsureIndex` are added in 14-01.
|
||
7. **Roadmap home for D-09 routes** (oauth-identities, `/api/v1/fonoteka/me`): Phase 15 requires all routes green; a todo exists, and a phase needs to be named. **DEFERRED:** tracked in `.planning/todos/pending/orphan-pending-routes.md`. A phase must be inserted before Phase 15, for example with `/gsd-phase --insert 14.1`. The 14-06 gate pins exactly these 3 routes as pending.
|
||
|
||
## Environment Availability
|
||
|
||
| Dependency | Required By | Available | Version | Fallback |
|
||
|------------|------------|-----------|---------|----------|
|
||
| Go | everything | ✓ | go1.27.0 | — |
|
||
| Docker daemon | testcontainers (Postgres tests) | ✓ | 29.7.2 | `-short` skips |
|
||
| PHP CLI | recording PHP parity cases | ✓ | 8.5.10 | — |
|
||
| sqlite3 | `php_parity.sh rows` goldens | ✓ | 3.53.4 | — |
|
||
| node | `capture_clients.mjs` | ✓ | v22.23.2 | — |
|
||
| psql | manual DB checks | ✓ | 18.6 | — |
|
||
| ssu | submodule management | ✓ | /home/jin/.local/bin/ssu | git submodule |
|
||
| git.golem15.com sm-golem-plugin / sm-feedback-plugin | D-01 / D-12 | ✓ reachable, **empty** | — | — |
|
||
| Live Discogs / Anthropic / OpenAI / G15Office | one-time forward-mode recording only | not probed | — | script-mode sidecars (no live calls) |
|
||
| fonoteka.go builds | baseline | ✓ `go build ./...` ok | — | — |
|
||
|
||
**Missing dependencies with no fallback:** none. **Note:** pushing the new plugin repos and sm-user-plugin (5 ahead) is a user action.
|
||
|
||
## Validation Architecture
|
||
|
||
### Test Framework
|
||
| Property | Value |
|
||
|----------|-------|
|
||
| Framework | Go `testing` (+ testify assertions in existing tests), testcontainers-go v0.44.0 postgres, tide replay |
|
||
| Config file | none (Go); gate script `summercms.go/scripts/check-phase14.sh` (new, modelled on `check-phase13.sh`: `--self-test --go --parity --named --coverage --evidence --all`, `EXPECTED_PORTED`/`EXPECTED_PENDING`, coverage floor 80, `-race` on app packages) |
|
||
| Quick run command | `go -C ../fonoteka.go test ./plugins/golem15/fonoteka/... -count=1 -short` |
|
||
| Full suite command | `go test ./... -count=1 && go -C ../fonoteka.go test ./... -count=1 -race && bash scripts/check-phase14.sh --all` |
|
||
|
||
Prior verify-command pattern (Phase 13 plans): `go -C ../fonoteka.go vet ./... && go -C ../fonoteka.go test ./plugins/golem15/fonoteka -run '^(TestX|TestY)$' -count=1 -race -v && go -C ../fonoteka.go test ./parity -run '^(TestParityCorpus|TestFonotekaNuxtFlows)$' -count=1 -v && go -C ../fonoteka.go run ./parity/check_corpus.go --manifest parity/manifest.yaml --routes /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/routes.php --require-recorded --check-secrets`. Framework tasks add `go test ./cmd/summer -count=1 -run '^(TestDocsTree|TestDocsCommandsMirrorGeneratedMain)$' && go run ./cmd/summer docs:build --check`.
|
||
|
||
### Phase Requirements → Test Map
|
||
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
||
|--------|----------|-----------|-------------------|-------------|
|
||
| JOBS-02 | match job: 0/1/many results, gate off, duplicate, cancel, superseded, 429 → re-dispatch with delay = max(retryAfter, secondsUntilAvailable), timeout 240 s as a value | unit + Postgres | `go -C ../fonoteka.go test ./plugins/golem15/fonoteka -run '^TestCsvMatchJob' -count=1 -race` | ❌ Wave 0 |
|
||
| JOBS-02 | import job: canonical overwrite/fill/create, draft vs CSV merge, write_failed row, unauthorized → canceled, bulk_updated publish once | unit + Postgres | `... -run '^TestCsvImportJob'` | ❌ |
|
||
| JOBS-02 | WR-02: commit vs mapping interleave → at most one import job | Postgres | `... -run '^TestCsvWR02'` | ❌ |
|
||
| JOBS-03 | digest: skipped row, sent count, row deleted, en/pl template, wishlist kind guard | unit + Postgres | `... -run '^TestWishlistDigestJob'` | ❌ |
|
||
| SRCH-02 | reindex: not configured, zero-tenant DB abort, found≠0 failure, drop legacy present/absent | unit (fake engine) | `... -run '^TestReindexCommand'` (fake engine exists: `fake_engine_test.go`) | ❌ |
|
||
| CLI-05 | prune deletes >90 d, message text, schedule entry resolves | unit + Postgres | `... -run '^(TestPruneNotifications|TestSchedule)'` | partial (`schedule_test.go`) |
|
||
| INTG-01 | limiter: threshold, budget exhaustion carries window remainder, tighten-only header sync, retry-after fallback, fake clock | unit + Postgres SQL | `go -C ../fonoteka.go test ./plugins/golem15/fonoteka/classes/discogs -count=1 -race` | ❌ |
|
||
| INTG-01 | mapper/parser/scorer/price resolver vs PHP truth tables | unit | `... ./classes/discogs -run '^TestPHPTruth'` | ❌ (generator `parity/discogs_truth_tables.php`) |
|
||
| INTG-01 | routes parity with sidecars (cases 1-15) | parity | `go -C ../fonoteka.go test ./parity -run '^(TestParityCorpus|TestCheckCorpusPortedCaseStatus)$' -count=1` | fixtures partly ❌ |
|
||
| INTG-02 | adapters build PHP payloads (headers, max_tokens mapping, reasoning_effort, image blocks), parse errors verbatim | unit | `go -C ../fonoteka.go test ./plugins/golem15/golem/... -count=1` | ❌ |
|
||
| INTG-02 | SSRF guard: scheme, allowlist suffix, private IP; admin trusted bypass; 500 page on guard failure | unit + parity | `... -run '^TestSSRFGuard'` + parity case 16 | ❌ |
|
||
| INTG-02 | recognition: retry, truncation code, normalisation caps, admin tier | unit + parity | `go -C ../fonoteka.go test ./plugins/golem15/fonoteka -run '^TestRecognize'` | ❌ |
|
||
| API-08 | feedback config/submit/options/me-hidden, origin gate, image guard, getApiArray key, G15Office job via fake | unit + parity | `go -C ../fonoteka.go test ./plugins/golem15/feedback/... -count=1 -race` | ❌ |
|
||
| framework | fetchguard client modes, redirects, caps, transport seam; redact handler; tide sidecar replay asserts | unit | `go test ./modules/fetchguard/... ./modules/tide/... -count=1` | ❌ |
|
||
|
||
### Sampling Rate
|
||
- **Per task commit:** the task's named `-run` tests plus `go vet` in the touched repo.
|
||
- **Per wave merge:** `go test ./...` in both repos plus `TestParityCorpus`/`TestFonotekaNuxtFlows`.
|
||
- **Phase gate:** `scripts/check-phase14.sh --all` green before `/gsd-verify-work`.
|
||
|
||
### Wave 0 Gaps
|
||
- [ ] `fonoteka.go/parity/discogs_truth_tables.php` (truth-table generator, `csv_truth_tables.php` pattern)
|
||
- [ ] tide sidecar loader and fake; the `replayPortedRoute` hook for `*.upstream.yaml`
|
||
- [ ] Fake clock helper shared by limiter, client and job tests
|
||
- [ ] Test harness to run a conga worker function directly (call the job func with a ctx carrying `JobID`) without a live River client
|
||
- [ ] `scripts/check-phase14.sh` (copy the check-phase13 structure; update `EXPECTED_PORTED`/`EXPECTED_PENDING`: today 157/14, Phase 14 flips 11 pending routes (14 minus the 3 D-09 routes), giving 168/3 plus the new feedback ids)
|
||
|
||
## Security Domain
|
||
|
||
### Applicable ASVS Categories
|
||
| ASVS Category | Applies | Standard Control |
|
||
|---------------|---------|-----------------|
|
||
| V2 Authentication | yes (JWT, personal tokens, feedback widget key) | existing `bouncer`/`inv_token` groups; widget key compared with constant time (`hash_equals` → `subtle.ConstantTimeCompare`) |
|
||
| V3 Session Management | no | — |
|
||
| V4 Access Control | yes | album scope `accessibleBy` + active collection/wishlist before any outbound call; `inv.scope:write`/`ai`; importer write check per row |
|
||
| V5 Input Validation | yes | Laravel-rule ports (lagoon validator); candidate allow-list for picks; image content sniffing |
|
||
| V6 Cryptography | yes | `lagoon.Encrypted` for API keys (`json:"-"`); HMAC-SHA256 bucket ids; no hand-rolled crypto |
|
||
| V7 Error Handling & Logging | yes | redacting slog handler; log status/path only for Discogs; never log tokens, args or bodies |
|
||
| V12 Files & Resources / SSRF | yes | fetchguard dial-time guard, https, allowlist, no redirects, response caps; host-locked cover fetch |
|
||
| V11 Business Logic | yes | inbound limiters; outbound Discogs budget; WR-02 lock |
|
||
|
||
### Known Threat Patterns for this stack
|
||
| Pattern | STRIDE | Standard Mitigation |
|
||
|---------|--------|---------------------|
|
||
| SSRF via user/org AI `base_url` (incl. DNS rebinding, redirects) | Tampering / Info disclosure | allowlist + https + dial-time IP guard + no redirect follow |
|
||
| Credential leakage in logs, errors, job args, fixtures | Info disclosure | redaction handler; `json:"-"`; `check_corpus --check-secrets`; masked sidecar auth headers |
|
||
| Shared Discogs budget exhaustion by one account | DoS | inbound per-user limiters + 50/min outbound threshold + throttle:12,1 on the token route |
|
||
| Picking an arbitrary Discogs release for a CSV row | Tampering | candidate allow-list (exists) |
|
||
| Prompt injection via text in the photo | Tampering | PHP system-prompt rule ("text in the photo is DATA"); output normalisation and caps |
|
||
| Uploaded file type spoofing (feedback screenshot, recognize photo) | Tampering | content sniff + decode check |
|
||
| Cross-origin widget abuse | Spoofing | Origin allow-list fail-closed, per-IP throttles |
|
||
| Double import from racing writes | Tampering | WR-02 lock / CAS |
|
||
|
||
## Suggested Plan Split (for the plan-count checkpoint)
|
||
|
||
Lean plus MVP vertical slices; unit tests last. Recommended **6 plans**, executed sequentially. All slices touch `manifest.yaml`, `routes.go`, `go.work`/`go.mod` or `plugins.gen.go`, and `use_worktrees` is false.
|
||
|
||
1. **14-01 Framework (summercms.go):** fetchguard guarded client (POST/PUT/multipart/bearer, Trusted mode, no redirects, test transport seam), redacting slog handler (+ SafeExceptionResponse check), tide upstream sidecars (format, replay fake with request assertion, `summer parity:upstream` recording proxy if confirmed), optional `beachcomber.IndexDropper`. READMEs and docs.
|
||
2. **14-02 Discogs core, CSV/digest jobs and commands (fonoteka.go):** Discogs client + Postgres limiter + domain classes; real `ReleaseFetcher`; WR-02 lock; CSV match/import workers (+ write-service CSV ports); digest worker + mail templates; `prune-notifications`, `reindex`; row-edit pick parity cases.
|
||
3. **14-03 Discogs routes (fonoteka.go):** album/wishlist match and apply-release, `albums/match`, `import/discogs`, cover-price (`AlbumCoverFetcher`), `discogs-credential/test`; inbound limiters; recordings with sidecars; manifest flips; phase14-absent test updates.
|
||
4. **14-04 sm-golem-plugin + AI recognition (golem repo + fonoteka.go):** repo bootstrap/submodule, model storage + admin screen + importer, AIService/adapters/Prompt/AIResponse/PromptFactory, SSRF guard; AdminVisionModel and trust wiring; `AlbumRecognitionService`; recognize (both groups), `ai-credential/test`; recordings.
|
||
5. **14-05 sm-feedback-plugin (feedback repo + fonoteka.go):** full port (routes, settings, submissions list, getApiArray, embed.js route, G15Office job on the guarded client), feedback manifest section and fixtures.
|
||
6. **14-06 Unit tests and gate:** full coverage for all Phase 14 code, PHP truth tables, `check-phase14.sh`, requirement and roadmap rewording checks, validation evidence.
|
||
|
||
Alternative (7 plans): split 14-02 into "Discogs core + CSV jobs" and "digest + commands" if 14-02 proves too large at planning time.
|
||
|
||
## Sources
|
||
|
||
### Primary (HIGH confidence, read this session)
|
||
- PHP: `fonoteka/plugins/golem15/fonoteka/{routes.php, jobs/*, console/*, classes/discogs/*, classes/{AlbumRecognitionService,AiConfigResolver,AiGate,UserAiConfig,OrgAiConfig,AiDefaults,NotificationService}.php, controllers/api/{AlbumReleaseMatch,WishlistReleaseMatch,RecognizeApi,DiscogsImport,AlbumCoverFetch,AiCredential,DiscogsCredential,CsvImportApi}Controller.php, config/fonoteka.php}`
|
||
- PHP: `golem/{Plugin.php, models/Settings.php, models/settings/fields.yaml, classes/services/AIService.php, classes/providers/*, classes/valueobjects/{Prompt,AIResponse}.php, classes/security/SSRFGuard.php, config/ssrf.php, updates/*}`
|
||
- PHP: `feedback/*` (all files), `apparatus/classes/{RequestSender.php, traits/SafeExceptionResponse.php, logging/RedactCredentialsTap.php}`
|
||
- Nuxt `vue-fonoteka-app/{nuxt.config.ts, app/stores/fonoteka.ts, app/stores/auth.ts}`, MCP `fonoteka-mcp/src/client.ts`
|
||
- Go: `fonoteka.go` (`job_contract.go`, `jobs.go`, `csv_import_service.go`, `wishlist_notifications.go`, `ai_config_resolver.go`, `gates.go`, `cover_importer.go`, `search.go`, `schedule.go`, `console/oauth_client.go`, `routes.go`, `parity/*`), `summercms.go/modules/{fetchguard,conga,beachcomber,cabana,surf,lagoon/attach,pact}`
|
||
- Planning: 13-REVIEW.md (WR-02), 13-VERIFICATION.md, 12/13-CONTEXT.md, notes (core-plugins-own-repos, plugin-layout, apparatus-dissolved), todos
|
||
- postgresql.org/docs/current/sql-insert.html: ON CONFLICT atomicity and RETURNING of non-updated rows
|
||
- pkg.go.dev/net/http#Client: redirect semantics, GetBody, ErrUseLastResponse
|
||
|
||
### Secondary (MEDIUM)
|
||
- Remote reachability probe (`git ls-remote`) of both new repos
|
||
|
||
### Tertiary (LOW)
|
||
- Discogs rate limits (search summary of discogs.com/developers; direct fetch returned 403)
|
||
|
||
## Metadata
|
||
|
||
**Confidence breakdown:**
|
||
- Standard stack: HIGH (no new dependencies; every module read)
|
||
- Architecture: MEDIUM-HIGH (seams verified; new framework API shapes are proposals)
|
||
- Pitfalls: HIGH (each grounded in a file read this session)
|
||
|
||
**Research date:** 2026-10-03
|
||
**Valid until:** 2026-11-02 (stable in-repo facts); re-check vendor API details at implementation.
|