57 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, requirements, estimate, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | requirements | estimate | must_haves | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 14-domain-jobs-and-external-integrations | 02 | execute | 2 |
|
|
true |
|
|
|
Phase Goal
ROADMAP Phase 14 goal (verbatim, not in user-story form): The domain-specific River jobs (CSV import write, Discogs match, wishlist digest), the reindex command, the Discogs client and AI cover recognition are ported on top of the Phase 11 jobs/realtime/search infrastructure and the Phase 13 API surface they serve.
This plan's slice: a collector's CSV import now finishes. The match job finds Discogs candidates and survives Discogs rate limits, a picked candidate resolves to the release PHP would store, the import job writes the albums, wishlist subscribers get their digest mail, and the operator can prune notifications and rebuild the search index (JOBS-02, JOBS-03, SRCH-02, CLI-05, the Discogs half of INTG-01; ROADMAP SC1-SC3 and SC6's commands).
Port the Discogs client with its Postgres limiter and the pure Discogs domain classes, install the real ReleaseFetcher, fix WR-02, add the CSV match and import workers with the CSV write-service variants, the digest worker with its mail templates, and the prune and reindex commands; wire upstream sidecars into the parity harness and record the row-edit pick cases.Purpose: Phase 13 queued these jobs with no worker (13 D-04); this plan makes them run exactly as PHP does. Decisions: D-08, D-10, D-11, D-15, D-16, D-17, D-21; research Pitfalls 4-7.
Output: classes/discogs package, three workers, two commands, migration, mail templates, parity sidecar hook and recordings.
Repo: fonoteka.go only. Commits path-scoped; never add co-author tags. Assumptions recorded below.
<execution_context>
@/.claude/gsd-core/workflows/execute-plan.md
@/.claude/gsd-core/templates/summary.md
</execution_context>
Artifacts this phase produces
(This plan's share.)
- Package
classes/discogs(packagediscogs):BaseURI,Clock,SystemClock,Client,ClientConfig,NewClient,ForUser,(*Client).GetRelease,GetPriceSuggestions,GetIdentity,GetMasterVersions,SearchByBarcode,SearchByQuery,RateLimiter,NewRateLimiter,LimiterConfig,(*RateLimiter).Acquire,SecondsUntilAvailable,SyncFromHeaders,RegisterRetryAfter,BucketID,RateStore,PostgresRateStore,MemoryRateStore,RateLimitError{RetryAfterSeconds},ErrTokenRejected,RequestError{Status},MapRelease,MapSearchResult,MapMasterVersion,ParseInput,NormalizeBarcode,AlternateBarcode,ScoreRelease(ReleaseMatchScorer),Applicator,ApplyResult,ImportResolver,ResolvePriceSuggestion. - Package
classes/discogs/discogstest:FakeClock,NewFakeClock. - classes:
ReleaseFetcher.FetchReleasenow returns*csv.Map; CSV write-service portsFillEmptyFromCsv,ApplyCsvFill,ApplyCsvOverwrite,CreateCsvAlbum,ResolveGenreID,SyncCsvRating(album_write_service.go);MatchCanonicalCsvAlbum(csv_canonical_matcher.go);CsvJobRunnerinterface. - Root package:
discogsReleaseFetcher(discogs_wiring.go), workersmatchCsv/deliverCsvMatch,importCsv/deliverCsvImport,sendWishlistDigest/deliverWishlistDigest. - Console:
PruneNotificationsCommand(app)(fonoteka:prune-notifications),ReindexCommand(app)(fonoteka:reindex,--drop-old-items-index). - Migration
202610030001_create_discogs_rate_windows(UNLOGGEDgolem15_fonoteka_discogs_rate_windows). - Config keys:
golem15.fonoteka.discogs.user_agent,golem15.fonoteka.discogs.rate_threshold,golem15.fonoteka.discogs.wait_budget_seconds,golem15.fonoteka.discogs.retry_after_fallback_seconds. - Mail templates
golem15.fonoteka::mail.wishlist_subscription_digestand-en. - Parity: sidecar replay hook in parity_test.go,
parity/upstream/scripts/vendor script files, row-edit pick cases with*.upstream.yaml,fixtures/jobs/csv-match.upstream.yamlandcsv-match.rows.json, sidecars scanned bycheck_corpus --check-secrets. - Tests:
TestClientGetRelease,TestDiscogsClientStatuses,TestRateLimiterBudget,TestRateLimiterSyncFromHeaders,TestRegisterRetryAfter,TestDiscogsRateWindowConcurrent,TestMapReleaseFixture,TestDiscogsDomainVectors,TestCsvRowPickResolves,TestCsvRowPickSeam,TestCsvWR02,TestCsvMatchJob,TestCsvImportJob,TestJobContractDispatchWhileWorkerRuns,TestWishlistDigestJob,TestPruneNotifications,TestReindexCommand,TestFonotekaSchedulePrune.
Assumptions
- EDGE-UNCLASSIFIED (JOBS-03, flagged): the edge probe could not classify the digest requirement; this plan assumes the coalescing window is entirely the 1800 s dispatch delay Phase 13 already sets on the first queue-row insert, and the worker only reads, deletes and mails. Not auto-resolved.
- EDGE-UNCLASSIFIED (SRCH-02, flagged): the probe could not classify the reindex requirement; this plan assumes "before" is PHP's database check (no album with collection_id NULL or ≤ 0) and "after" is the Typesense
foundcheck oncollection_id:=0. Not auto-resolved. - A5: Scout imports in chunks of 500; the Go reindex uses 500.
- A7/A8: the limiter SQL is the research shape;
jobs.CancelJobtouches only summer_jobs and river_job, so it cannot deadlock against the csv_imports row lock. - A1: PHP's curl honours HTTPS_PROXY and the parity CA passed with
-d curl.cainfoand-d openssl.cafile; Task 1's precondition checks it before any recording. - Pitfall 4: new job goldens compare summer_jobs metadata JSON semantically, because conga marshals map keys sorted and PHP keeps insertion order.
<assumption_delta_decision> signal: pluralization term: fallback decision: no-change rationale: "retry-after fallback" is the single Discogs Retry-After default (retry_after_fallback_seconds = 10) used when the header is absent or non-numeric; it is not a second identity or a plural concept. </assumption_delta_decision>
Task 1: A collector picks a Discogs candidate on a CSV row and the row resolves with the draft PHP stores, replayed offline from PHP's recorded Discogs exchange The UNLOGGED limiter table and the `.upstream.yaml` sidecar layout become shared by every later Discogs, AI and G15Office case; both are re-creatable, so no checkpoint. `php -v` exits 0, the isolated PHP instance resets with `parity/php_parity.sh reset`, and a probe `HTTPS_PROXY=http://127.0.0.1:8425 php -d curl.cainfo= -r` request to https://api.discogs.com reaches a running `summer parity:upstream` (research assumption A1). ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/client.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/rate_limiter.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/rate_store.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/clock.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/errors.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/mapper.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/discogs_test.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/postgres_test.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/testdata/, ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/discogstest/fake_clock.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/csv_import_service.go, ../fonoteka.go/plugins/golem15/fonoteka/updates/22_discogs_rate_windows.go, ../fonoteka.go/plugins/golem15/fonoteka/config/config.yaml, ../fonoteka.go/plugins/golem15/fonoteka/discogs_wiring.go, ../fonoteka.go/plugins/golem15/fonoteka/plugin.go, ../fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go, ../fonoteka.go/parity/parity_test.go, ../fonoteka.go/parity/upstream_replay_test.go, ../fonoteka.go/parity/fonoteka_seed_test.go, ../fonoteka.go/parity/fonoteka_reset.php, ../fonoteka.go/parity/php_parity.sh, ../fonoteka.go/parity/check_corpus.go, ../fonoteka.go/parity/check_corpus_test.go, ../fonoteka.go/parity/schema_diff_test.go, ../fonoteka.go/parity/manifest.yaml, ../fonoteka.go/parity/fixtures/routes/, ../fonoteka.go/parity/upstream/scripts/, ../fonoteka.go/parity/README.md /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/discogs/DiscogsClient.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/discogs/DiscogsRateLimiter.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/discogs/DiscogsMapper.php (mapRelease), /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/discogs/DiscogsRateLimitException.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/classes/discogs/DiscogsTokenRejectedException.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/controllers/api/CsvImportApiController.php (updateRow), /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/config/fonoteka.php (lines 18-30), /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/tests/unit/DiscogsMapperTest.php, ../fonoteka.go/plugins/golem15/fonoteka/classes/csv_import_service.go (ReleaseFetcher lines 589-628, UpdateCsvRow 639-705, CommitCsvImport 825-882), ../fonoteka.go/plugins/golem15/fonoteka/classes/cover_importer.go (config constants and FromConfig), ../fonoteka.go/plugins/golem15/fonoteka/classes/csv/ordered.go, ../fonoteka.go/plugins/golem15/fonoteka/classes/gates.go (ResolveDiscogsConfig, DiscogsAllowed, MarketCurrency), ../fonoteka.go/plugins/golem15/fonoteka/updates/11_secrets_slice.go, ../fonoteka.go/plugins/golem15/fonoteka/updates/registry.go, ../fonoteka.go/plugins/golem15/fonoteka/plugin.go (Boot), ../fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go (TestCsvRowPickSeam, fakeReleaseFetcher), ../fonoteka.go/parity/parity_test.go (replayPortedRoute), ../fonoteka.go/parity/php_parity.sh, ../fonoteka.go/parity/fonoteka_reset.php, ../fonoteka.go/parity/fonoteka_seed_test.go (seedParityCredentials, seedParityCSVImports), ../fonoteka.go/parity/check_corpus.go (secret scan), ../fonoteka.go/parity/schema_diff_test.go (Go-only allow-list), .planning/phases/14-domain-jobs-and-external-integrations/14-RESEARCH.md (sections "Discogs client", "D-17", "WR-02 fix", "D-11") Per D-08, D-11, D-15, D-16, D-19, D-21 (the thinnest path: route → service → fetcher → client → limiter table → guarded client → sidecar fake).(1) classes/discogs: clock.go Clock interface (Now() time.Time, Sleep(ctx, d) error returning ctx.Err() on cancel) and SystemClock; discogstest/fake_clock.go FakeClock whose Sleep advances Now. errors.go RateLimitError{RetryAfterSeconds int}, ErrTokenRejected, RequestError{Status int} with PHP's message. rate_store.go RateStore (TryAcquire(ctx, bucket string, now time.Time, threshold int) (granted bool, windowStart time.Time, err error), Tighten(ctx, bucket, now, implied int) error, WindowStart(ctx, bucket) (time.Time, bool, error)), PostgresRateStore{DB *gorm.DB} with the research single-statement upsert (now passed as a parameter, never SQL now()), and MemoryRateStore for unit tests. rate_limiter.go LimiterConfig{Threshold int; WaitBudget time.Duration; RetryAfterFallback int}, NewRateLimiter(store, clock, appKey []byte, cfg), BucketID(token, appKey) (first 32 hex of HMAC-SHA256), (*RateLimiter).Acquire(ctx, token) error (granted → nil; otherwise wait = max(1, 60 − floor(now − window_start)); a wait beyond the remaining budget returns *RateLimitError{max(1, wait)}, else Sleep and retry). client.go BaseURI = "https://api.discogs.com" (code constant), ClientConfig{Token, UserAgent, MarketCurrency string; HTTP *fetchguard.Client; Limiter *RateLimiter; Clock Clock; WaitBudget time.Duration}, NewClient, ForUser(ctx, db, cfg, user) (ResolveDiscogsConfig, MarketCurrency(cfg), fetchguard AllowHostsMode api.discogs.com 10 s), and GetRelease(ctx, id int) (map[string]any, error) on the PHP request loop (acquire once; 200 decodes with UseNumber, 404 returns nil, 401/403 ErrTokenRejected, 429 handled in Task 2, others RequestError); the request context reaches fetchguard so WithTransport applies. mapper.go MapRelease(release map[string]any) *csv.Map builds PHP's draft keys in PHP order.
(2) Migration updates/22_discogs_rate_windows.go ID 202610030001_create_discogs_rate_windows: CREATE UNLOGGED TABLE golem15_fonoteka_discogs_rate_windows (bucket TEXT PRIMARY KEY, window_start TIMESTAMPTZ NOT NULL, hits INTEGER NOT NULL), rollback drops it; register through updates.Register; add the table to schema_diff_test's Go-only allow-list with the D-21 reason. config.yaml under discogs: adds user_agent: "FonotekaApp/1.0 +https://github.com/golem15com/wn-fonoteka-plugin", rate_threshold: 50, wait_budget_seconds: 15, retry_after_fallback_seconds: 10 with comments naming the SUMMER_ override; the base URI stays code.
(3) ReleaseFetcher: change FetchRelease to return *csv.Map (the ordered draft) and update the default and the test fake. discogs_wiring.go (root package) discogsReleaseFetcher{app} resolves db and config per call, builds the client with ForUser, maps nil to ErrDiscogsUnavailable and *RateLimitError to ErrDiscogsRateLimited; plugin.go Boot installs it with SetReleaseFetcher. In UpdateCsvRow the fetch stays before any lock; the save closure runs in one lagoon.Transaction that re-reads the import with clause.Locking{Strength: "UPDATE"}, re-checks csvBeforeCommit and the row, then writes resolved, the selected id and draft_json (D-10 for this path).
(4) Parity: parity_test.go — for each case, when tide.UpstreamPath(fixture) exists, load it, build tide.NewUpstreamFake(sidecar, store), serve the case through a handler wrapper that puts fetchguard.WithTransport on each request context, and fail the case when Verify reports anything; upstream_replay_test.go TestUpstreamSidecarsAreReplayed asserts every sidecar on disk belongs to a manifest case or a job golden. check_corpus --check-secrets scans *.upstream.yaml too (check_corpus_test covers a planted Discogs token). php_parity.sh serve honours PARITY_UPSTREAM_CA: when set it exports HTTPS_PROXY and https_proxy as http://127.0.0.1:8425 and passes the CA with -d curl.cainfo and -d openssl.cafile. fonoteka_reset.php and the Go seed give alice a BYOK Discogs token held as {{secret:discogs-token}} (matching ^[A-Za-z0-9_\-]{10,255}$) and a CSV import row whose candidates_json offers a fixed release id. parity/upstream/scripts/discogs-release.yaml holds the scripted GET /releases/{id}?curr_abbr=EUR 200 response built from the DiscogsMapperTest release fixture. Record with summer parity:upstream --mode script running and PHP serving through it: the pick-success case of PATCH /_fonoteka/api/v1/import/csv/{id}/rows/{rowId} jwt plus its sidecar; add the case to the manifest (the route is already ported). README gains the "Upstream sidecars" recipe.
(5) Tests: discogs_test.go TestClientGetRelease (headers, URL, UseNumber decode, 404 nil, through a tide fake), TestMapReleaseFixture (DiscogsMapperTest's release maps to PHP's draft bytes); postgres_test.go TestRateLimiterPostgresAcquire (testcontainers Postgres, migration applied, threshold reached then a new window); csv_smoke_test.go TestCsvRowPickResolves (fake fetcher draft is written, row resolved, PHP key order kept) and TestCsvRowPickSeam updated (Discogs disallowed still answers discogs_unavailable; a non-candidate id still validation_failed).
go -C ../fonoteka.go vet ./... && go -C ../fonoteka.go test ./plugins/golem15/fonoteka/classes/discogs/... -count=1 -race -v -run '^(TestClientGetRelease|TestMapReleaseFixture|TestRateLimiterPostgresAcquire)$' && go -C ../fonoteka.go test ./plugins/golem15/fonoteka -count=1 -race -v -run '^(TestCsvRowPickResolves|TestCsvRowPickSeam)$' && go -C ../fonoteka.go test ./parity -count=1 -v -run '^(TestParityCorpus|TestCheckCorpusPortedCaseStatus|TestUpstreamSidecarsAreReplayed|TestSchemaMatchesPHPSnapshot)$' && go -C ../fonoteka.go run ./parity/check_corpus.go --manifest parity/manifest.yaml --routes /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/routes.php --require-recorded --check-secrets
<fails_when>Any command exits non-zero; a verbose run prints "--- FAIL", "no tests to run", "--- SKIP" or "DATA RACE", or lacks "--- PASS" for TestClientGetRelease, TestMapReleaseFixture, TestRateLimiterPostgresAcquire, TestCsvRowPickResolves, TestUpstreamSidecarsAreReplayed and "--- PASS: TestParityCorpus/coverage"; check_corpus reports a secret, an unrecorded route or a ported case-status mismatch.</fails_when>
<acceptance_criteria>
- grep -c 'CREATE UNLOGGED TABLE golem15_fonoteka_discogs_rate_windows' ../fonoteka.go/plugins/golem15/fonoteka/updates/22_discogs_rate_windows.go prints 1.
- grep -c 'SetReleaseFetcher' ../fonoteka.go/plugins/golem15/fonoteka/plugin.go prints at least 1.
- ls ../fonoteka.go/parity/fixtures/routes/ | grep -c 'upstream.yaml' prints at least 1, and that sidecar contains {{secret:discogs-token}} and no 10+ character literal token after Discogs token=.
- grep -c 'PARITY_UPSTREAM_CA' ../fonoteka.go/parity/php_parity.sh prints at least 1.
- grep -c 'golem15_fonoteka_discogs_rate_windows' ../fonoteka.go/parity/schema_diff_test.go prints 1.
</acceptance_criteria>
A pick on a CSV row resolves through the real Discogs client and the shared limiter exactly as PHP did, and the harness replays PHP's recorded Discogs exchange offline while asserting Go's request.
(1) Client: the full PHP loop — acquire once, then per response SyncFromHeaders(X-Discogs-Ratelimit-Remaining); 429 computes wait = RegisterRetryAfter(Retry-After) and returns *RateLimitError{max(1, wait)} when wait exceeds the remaining WaitBudget, else Clock.Sleep and retry. Add GetPriceSuggestions(ctx, id) (/marketplace/price_suggestions/{id}, an empty object is normal), GetIdentity(ctx) (/oauth/identity), GetMasterVersions(ctx, id) (/masters/{id}/versions?per_page=10), SearchByBarcode(ctx, barcode) and SearchByQuery(ctx, q) (/database/search?…&type=release); ids are ints in the path, free text only in the query. Logs through the resolved logger carry status and path only. Limiter: SecondsUntilAvailable(ctx, token) (int, error) (read-only, 1 when free), SyncFromHeaders(ctx, token, remaining string) (tighten-only per the research SQL), RegisterRetryAfter(header string) int (integer seconds else the fallback).
(2) Domain ports, each a straight port with PHP key order via *csv.Map where output is JSON: mapper.go MapSearchResult, MapMasterVersion; input_parser.go ParseInput, NormalizeBarcode, AlternateBarcode; scorer.go ScoreRelease (ReleaseMatchScorer, using completeness.go); applicator.go Applicator{CoverImporter *classes.CoverImporter; DB *gorm.DB} with Apply(ctx, album, mapped, overwriteAll, coverOnly, dryRun) (ApplyResult, error) (ApplyResult{Filled, Remaining []string; Draft *csv.Map}), importing covers only through CoverImporter's AllowHosts fetch; import_resolver.go ImportResolver (input URL or id → release draft, barcode → candidates or no_match, master → versions); price_suggestion.go ResolvePriceSuggestion. These have no route yet; plan 14-03 mounts them.
(3) Tests: discogs_test.go TestDiscogsClientStatuses (429 inside budget retries after a fake-clock sleep, 429 beyond budget returns RateLimitError with max(1, wait), 401 and 403 ErrTokenRejected, 500 RequestError message, headers tighten the limiter), TestRateLimiterBudget (threshold 50: 50th granted, 51st waits; wait equal to the remaining budget sleeps, one second more errors; window rollover), TestRateLimiterSyncFromHeaders (only raises), TestRegisterRetryAfter (numeric, absent, non-numeric), TestDiscogsDomainVectors (table cases transcribed from each PHP unit test named in read_first for parser, scorer, mapper, applicator dry-run and cover-only, import resolver and price resolver); postgres_test.go TestDiscogsRateWindowConcurrent (two sessions at threshold − 1, exactly one granted).
(4) Record the two remaining pick cases with sidecars (script files for a 404 release and a 429 with Retry-After: 30): 422 discogs_unavailable and 422 discogs_rate_limited; add them to the manifest.
go -C ../fonoteka.go vet ./... && go -C ../fonoteka.go test ./plugins/golem15/fonoteka/classes/discogs/... -count=1 -race -v -run '^(TestDiscogsClientStatuses|TestRateLimiterBudget|TestRateLimiterSyncFromHeaders|TestRegisterRetryAfter|TestDiscogsDomainVectors|TestDiscogsRateWindowConcurrent|TestClientGetRelease|TestMapReleaseFixture)$' && go -C ../fonoteka.go test ./parity -count=1 -v -run '^(TestParityCorpus|TestCheckCorpusPortedCaseStatus|TestUpstreamSidecarsAreReplayed)$'
<fails_when>Any command exits non-zero; a verbose run prints "--- FAIL", "no tests to run", "--- SKIP" or "DATA RACE", or lacks "--- PASS" for TestDiscogsClientStatuses, TestRateLimiterBudget, TestDiscogsDomainVectors and TestDiscogsRateWindowConcurrent; the parity run reports FAIL for a row-edit case or lacks "--- PASS: TestParityCorpus/coverage".</fails_when>
<acceptance_criteria>
- grep -c 'time.Sleep' ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/client.go ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/rate_limiter.go prints 0 for both files (all waits go through Clock).
- grep -c 'func ScoreRelease(' ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/scorer.go and grep -c 'func ResolvePriceSuggestion(' ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/price_suggestion.go each print 1.
- The manifest's row-edit route lists the pick-success, discogs_unavailable and discogs_rate_limited cases, each with a sidecar on disk.
</acceptance_criteria>
The Discogs client, limiter and domain rules behave as PHP's for every status and vector its unit tests pin, ready for the routes in 14-03.
(1) WR-02 first: UpdateCsvMapping and CancelCsvImport run inside one lagoon.Transaction that re-reads the import FOR UPDATE, re-checks csvBeforeCommit (mapping) on the locked row, cancels the job ids read from the locked row, writes and dispatches; response shapes unchanged. TestCsvWR02 (Postgres, goroutines with a barrier) races commit against mapping, against a row save and against cancel, and asserts at most one import job dispatched and no import left in a state PHP cannot reach.
(2) Write-service ports in album_write_service.go: FillEmptyFromCsv, ApplyCsvFill, ApplyCsvOverwrite, CreateCsvAlbum (with PHP's options: allow_null_format, first cover URL plus import_covers through CoverImporter), ResolveGenreID, SyncCsvRating; csv_canonical_matcher.go MatchCanonicalCsvAlbum(ctx, tx, importer, collectionID, row) ports CsvCanonicalIdMatcher's scoped query.
(3) Workers, each a thin p.xxx resolver plus a testable deliverXxx(ctx, deps, jobID uint, args) free function (the wrapper passes conga.JobID(ctx)): classes gains CsvJobRunner (Dispatch, CancelJob, StartJob, UpdateJobState, CompleteJob, FailJob, StopJob, CheckIfCanceled; *conga.Manager satisfies it). csv_match_job.go ports AlbumCsvMatchJob line by line: import gone → FailJob {"error":"import_not_found"}; cancel check → StopJob; terminal or superseded → CompleteJob {"skipped":"<status>"}; status matching, StartJob(row_count), resume progress; per pending row (by row_index) cancel check, matchRow (duplicate → matched_csv; gate off → matched_csv; SearchByQuery then SearchByBarcode; 0/1/many with MAX_CANDIDATES 10, GetRelease plus MapRelease for exactly one) and UpdateJobState; end preview, error_message null, CompleteJob {"matched":N}. A *discogs.RateLimitError pauses: delay = max(RetryAfterSeconds, SecondsUntilAvailable), Dispatch a new CsvMatchArgs (label fonoteka.csv.match, queue fonoteka_csv_match, Count pending, Metadata csv_import_id, resumed_after_rate_limit true, retry_after delay, Delay that many seconds), store match_job_id, CompleteJob {"paused":"discogs_rate_limited","retry_after":delay,"next_job_id":id}. Any other error: status failed, error_message, FailJob {"error":msg}, return nil. Worker status writes are conditional on the locked row so a canceled import is never resurrected. csv_import_job.go ports AlbumCsvImportJob: importerCanWrite before start and per row (else canceled plus StopJob), rows not written or skipped by row_index, StartJob(count), per row inside album broadcast suppression the canonical or draft/CSV merge path, per-row failure → row error with error_code write_failed; end done, one collection.bulk_updated publication {"reason":"csv_import","count":N} on collection:{id} when N > 0, CompleteJob {"written":N}; outer failure → failed plus FailJob, return nil. jobs.go registers conga.Job(p.matchCsv, conga.OnQueue(classes.CsvMatchQueue), conga.Timeout(240*time.Second)) and conga.Job(p.importCsv, conga.OnQueue(classes.CsvImportQueue)), and its doc comment now says all queued kinds have workers.
(4) Tests: TestCsvJobRows and TestCsvCancel keep their dispatch and cancel assertions but drop any assumption that the CSV queues are unserved. job_contract_worker_test.go inverts the unserved-queue assertions: the worker serves fonoteka_csv_import, fonoteka_csv_match and fonoteka_wishlist_digest, and a dispatched import for a missing id ends failed with import_not_found. csv_jobs_test.go TestCsvMatchJob (0, 1, 2 and 11 results, gate off, duplicate, cancel mid-run, superseded, a 429 beyond budget re-dispatches with the computed delay and the 240 s timeout value is asserted from the registered job) driven by a FakeClock and tide fakes; TestCsvImportJob (canonical overwrite and fill, create, draft wins over CSV for selected rows, write_failed row, lost access → canceled, exactly one bulk_updated publication). Record PHP's match job under the sync queue: a non-canonical mapping PATCH with the proxy running in script mode captures the job's Discogs exchanges into parity/fixtures/jobs/csv-match.upstream.yaml, and php_parity.sh rows dumps csv_import_rows and the job row into csv-match.rows.json; TestCsvMatchJob/php-recorded replays that sidecar through deliverCsvMatch and compares row statuses, candidates and draft JSON, and job metadata semantically (Pitfall 4).
go -C ../fonoteka.go vet ./... && go -C ../fonoteka.go test ./plugins/golem15/fonoteka -count=1 -race -v -run '^(TestCsvWR02|TestCsvMatchJob|TestCsvImportJob|TestJobContractDispatchWhileWorkerRuns|TestCsvCommitCAS|TestCsvCancel|TestCsvJobRows)$' && go -C ../fonoteka.go test ./parity -count=1 -v -run '^(TestParityCorpus|TestFonotekaNuxtFlows)$'
<fails_when>Any command exits non-zero; a verbose run prints "--- FAIL", "no tests to run", "--- SKIP" or "DATA RACE", or lacks "--- PASS" for TestCsvWR02, TestCsvMatchJob, "TestCsvMatchJob/php-recorded", TestCsvImportJob and TestJobContractDispatchWhileWorkerRuns; the parity run reports FAIL for nuxt-csv or lacks "--- PASS: TestParityCorpus/coverage".</fails_when>
<acceptance_criteria>
- grep -c 'conga.Timeout(240 \* time.Second)' ../fonoteka.go/plugins/golem15/fonoteka/jobs.go prints 1.
- grep -c 'until Phase 14' ../fonoteka.go/plugins/golem15/fonoteka/jobs.go ../fonoteka.go/plugins/golem15/fonoteka/job_contract_worker_test.go prints 0 for both files.
- grep -c 'Locking' ../fonoteka.go/plugins/golem15/fonoteka/classes/csv_import_service.go prints at least 3.
- ls ../fonoteka.go/parity/fixtures/jobs/csv-match.upstream.yaml ../fonoteka.go/parity/fixtures/jobs/csv-match.rows.json succeeds.
</acceptance_criteria>
CSV imports match and write end to end in Go with PHP's job outcomes, rate limits pause and resume the batch, and no racing write can queue a second import.
(1) Digest: copy both PHP templates verbatim into views/mail and register golem15.fonoteka::mail.wishlist_subscription_digest and -en in mail.go. wishlist_digest_job.go sendWishlistDigest plus deliverWishlistDigest(ctx, gdb, mailer, jobs, log, jobID, args): read the golem15_fonoteka_wishlist_digest_queue row for (subscriber, wishlist); missing or item_count ≤ 0 → CompleteJob {"skipped":true}; else delete the row, then mailWishlistDigest (wishlist must exist with kind wishlist and the subscriber must exist; locale en when preferred_locale is en else pl; vars ownerName, itemCount, wishlistName; to the subscriber's email), then CompleteJob {"sent":N}; a mail error is returned (River retries and the next run finds no row). jobs.go registers conga.Job(p.sendWishlistDigest, conga.OnQueue(classes.WishlistDigestJobQueue)). Tests TestWishlistDigestJob: missing row, zero count, sent count with the row deleted, en and pl templates with the three vars, a non-wishlist list and a missing subscriber send nothing, two dispatches for one window mail once.
(2) console/prune_notifications.go PruneNotificationsCommand(app) named fonoteka:prune-notifications: one DELETE on golem15_fonoteka_notifications where created_at is before now minus 90 days, prints Pruned N notifications older than 90 days., exit 0; the cutoff comes from an unexported pruneNotifications(ctx, db, now) so tests pin it. console/reindex.go ReindexCommand(app) named fonoteka:reindex with Bare flag drop-old-items-index: not configured (engine name typesense, settingsGate enabled, engine Configured) → error line Typesense reindex skipped: enable Fonoteka Typesense search and configure TYPESENSE_API_KEY. and non-zero exit; any album with collection_id NULL or ≤ 0 → Album reindex aborted: every active Album must have a positive collection_id.; then Flush the album index, EnsureIndex from the album schema, upsert all albums ordered by id in batches of 500 through the existing searchable document; then SearchPage q=*, query_by=name, filter_by=collection_id:=0, per_page=1 and found != 0 → Album reindex failed integrity check: collection_id:=0 documents exist.; with the flag, DropIndex of the legacy index name (search prefix plus golem15_fonoteka_items) → Deleted legacy Typesense collection: <name> or Legacy Typesense collection already absent: <name>; success line Album index rebuilt; collection_id:=0 document count is zero. Register both in Commands() beside the oauth-client command; schedule.go's comment no longer says the command is missing.
(3) Tests console/phase14_commands_test.go: TestPruneNotifications (row at exactly 90 days kept, older deleted, empty run prints 0, second run deletes 0), TestReindexCommand (fake engine: not configured, tenantless album abort, empty table creates the index, found≠0 failure, drop present then absent, second reindex same count, only the legacy name is dropped); schedule_test.go's TestFonotekaScheduleSkipsUnregisteredPrune is replaced by TestFonotekaSchedulePrune, which asserts the daily entry now resolves to the registered command.
go -C ../fonoteka.go vet ./... && go -C ../fonoteka.go test ./plugins/golem15/fonoteka ./plugins/golem15/fonoteka/console -count=1 -race -v -run '^(TestWishlistDigestJob|TestPruneNotifications|TestReindexCommand|TestFonotekaSchedulePrune|TestJobContractDispatchWhileWorkerRuns)$' && go -C ../fonoteka.go test ./... -count=1 -short
<fails_when>Any command exits non-zero; a verbose run prints "--- FAIL", "no tests to run", "--- SKIP" or "DATA RACE", or lacks "--- PASS" for TestWishlistDigestJob, TestPruneNotifications, TestReindexCommand and TestFonotekaSchedulePrune; the short suite reports FAIL in any package.</fails_when>
<acceptance_criteria>
- grep -c 'fonoteka:prune-notifications' ../fonoteka.go/plugins/golem15/fonoteka/console/prune_notifications.go and grep -c 'fonoteka:reindex' ../fonoteka.go/plugins/golem15/fonoteka/console/reindex.go each print at least 1.
- grep -c 'PruneNotificationsCommand\|ReindexCommand' ../fonoteka.go/plugins/golem15/fonoteka/plugin.go prints at least 2.
- diff /media/nvme/dev/golem15/fonoteka/plugins/golem15/fonoteka/views/mail/wishlist_subscription_digest.htm ../fonoteka.go/plugins/golem15/fonoteka/views/mail/wishlist_subscription_digest.htm exits 0 (verbatim copy; same for -en).
- grep -c 'Album index rebuilt; collection_id:=0 document count is zero.' ../fonoteka.go/plugins/golem15/fonoteka/console/reindex.go prints 1.
</acceptance_criteria>
The digest job mails each subscriber once per window in their language, and both console commands behave as their PHP originals including messages and exit codes.
Canon referrals (not minted as prohibitions)
- Token leakage through logs and errors is canon (OWASP logging) — covered by /gsd-secure-phase and the 14-01 sunscreen handler; not minted here.
- SSRF through the Discogs host is canon — the base URI is a code constant behind fetchguard AllowHostsMode; covered by /gsd-secure-phase.
<threat_model>
Trust Boundaries
| Boundary | Description |
|---|---|
| Application → api.discogs.com | A user's or the site's Discogs token leaves the process |
| Concurrent processes → limiter table | serve and queue:work share one budget |
| Request writes ↔ job writes | Mapping, row edits, cancel and commit race the workers |
| Job → collection | The importer writes albums on the user's behalf |
| Job → subscriber mailbox | The digest mails outside the request path |
| Console → search engine | Reindex deletes and rebuilds indexes |
STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|---|---|---|---|---|---|
| T-14-08 | Information Disclosure | Discogs token | high | mitigate | Bucket id is an HMAC with the app key, token never stored in the limiter or job args, logs carry status and path only, sidecars masked; TestRateLimiterPostgresAcquire and check_corpus --check-secrets (Task 1). |
| T-14-09 | Denial of Service | shared Discogs budget | medium | mitigate | One atomic UNLOGGED upsert across processes, threshold 50, 15 s wait budget; TestRateLimiterBudget, TestDiscogsRateWindowConcurrent (Task 2). |
| T-14-10 | Tampering | racing CSV writes | high | mitigate | Row lock or CAS on mapping, row save and cancel (D-10); TestCsvWR02 (Task 3). |
| T-14-11 | Tampering | CSV row pick | high | mitigate | Candidate allow-list, fetch before the lock and re-check under it; TestCsvRowPickSeam, TestCsvRowPickResolves (Task 1). |
| T-14-12 | Tampering | job retries | medium | mitigate | FailJob then return nil; only timeout or panic retries; TestCsvMatchJob, TestCsvImportJob (Task 3). |
| T-14-13 | Elevation of Privilege | import writes | high | mitigate | importerCanWrite before start and per row → canceled; TestCsvImportJob lost-access case (Task 3). |
| T-14-14 | Repudiation | digest mail | medium | mitigate | Wishlist-kind and subscriber guards, row deleted before send, one mail per row; TestWishlistDigestJob (Task 4). |
| T-14-15 | Information Disclosure | reindex | high | mitigate | Abort on tenantless albums, integrity check after import, drop only the legacy index; TestReindexCommand (Task 4). |
| T-14-SC | Tampering | package installs | low | accept | No new module or package; only in-repo framework packages and stdlib. |
| </threat_model> |
<success_criteria>
- Discogs client, limiter and domain classes ported with PHP vectors.
- CSV match and import workers, digest worker and both commands run as in PHP.
- WR-02 closed; row-edit pick cases recorded with sidecars and replayed offline. </success_criteria>