diff --git a/.planning/phases/14-domain-jobs-and-external-integrations/14-02-SUMMARY.md b/.planning/phases/14-domain-jobs-and-external-integrations/14-02-SUMMARY.md new file mode 100644 index 0000000..6b4f7ec --- /dev/null +++ b/.planning/phases/14-domain-jobs-and-external-integrations/14-02-SUMMARY.md @@ -0,0 +1,226 @@ +--- +phase: 14-domain-jobs-and-external-integrations +plan: 02 +subsystem: jobs +tags: [discogs, rate-limiter, unlogged-table, river, conga, csv-import, wishlist-digest, reindex, typesense, parity, upstream-sidecar] + +requires: + - phase: 14-domain-jobs-and-external-integrations + provides: "14-01 fetchguard.Client, tide upstream sidecars and summer parity:upstream, beachcomber DropIndex/EnsureIndex" + - phase: 13-p-ytarium-api-wishlist-notifications-csv-credentials-public + provides: "CSV import routes and job contract (kinds, queues, args), ReleaseFetcher seam, wishlist digest queue" +provides: + - "classes/discogs: guarded Discogs client, Postgres per-token rate limiter (UNLOGGED table), DiscogsMapper, input parser, scorer, price resolver, import resolver, release applicator" + - "Real CSV row Discogs pick (D-08) recorded against PHP with upstream sidecars" + - "CSV match and import workers, wishlist digest worker" + - "fonoteka:prune-notifications and fonoteka:reindex commands" + - "WR-02 fix: mapping, row save and cancel lock the import row" + - "Parity: per-case upstream sidecar replay, csv-discogs seed state, vendor-credential secret scan" + - "conga.Describe (framework) and a graceful-stop fix for summer parity:upstream" +affects: [14-03 discogs routes, 14-04 golem adapters, 14-05 feedback G15Office job, 14-06 unit tests] + +actuals: + tokens: 100700 + tasks: 4 + commits: 2 # measured in summercms.go (plan_head_before..plan_head_after) + app_repo_commits: 6 # measured in fonoteka.go (app_repo_head_before..app_repo_head_after) +plan_head_before: 43b869d613188dc1b1339a18d6f692dd3a09f2be +plan_head_after: cdd8d710facb8e87537eac36c54c00f0ba5b5773 +app_repo_head_before: 5738e82c5ee683d37498bc04c9453a17dd41986c +app_repo_head_after: ef34015a223a67e649e99bac410fb728337c0ee3 + +tech-stack: + added: [] + patterns: + - "Vendor-calling parity cases carry .upstream.yaml; replayPortedRoute serves every case through an asserting upstream fake (an empty one when there is no sidecar), so no case can dial out" + - "Pure PHP classes are pinned by PHP-generated truth tables (parity/discogs_truth_tables.php boots Winter on the parity instance)" + - "Workers are a thin p.xxx resolver plus a deliverXxx free function taking the summer_jobs id; FailJob then return nil, never a River retry" + - "Worker writes to the import status are conditional on the row (CAS), so a cancelled or remapped import is never resurrected" + +key-files: + created: + - ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/client.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/rate_limiter.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/rate_store.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/mapper.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/input_parser.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/scorer.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/applicator.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/import_resolver.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/price_suggestion.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/discogs/discogstest/fake_clock.go + - ../fonoteka.go/plugins/golem15/fonoteka/updates/22_discogs_rate_windows_table.go + - ../fonoteka.go/plugins/golem15/fonoteka/discogs_wiring.go + - ../fonoteka.go/plugins/golem15/fonoteka/csv_match_job.go + - ../fonoteka.go/plugins/golem15/fonoteka/csv_import_job.go + - ../fonoteka.go/plugins/golem15/fonoteka/wishlist_digest_job.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/csv_canonical_matcher.go + - ../fonoteka.go/plugins/golem15/fonoteka/console/prune_notifications.go + - ../fonoteka.go/plugins/golem15/fonoteka/console/reindex.go + - ../fonoteka.go/parity/upstream_replay_test.go + - ../fonoteka.go/parity/discogs_truth_tables.php + - ../fonoteka.go/parity/upstream/scripts/ + - ../fonoteka.go/parity/fixtures/jobs/csv-match.upstream.yaml + - ../fonoteka.go/parity/fixtures/jobs/csv-match.rows.json + modified: + - ../fonoteka.go/plugins/golem15/fonoteka/classes/csv_import_service.go + - ../fonoteka.go/plugins/golem15/fonoteka/classes/album_write_service.go + - ../fonoteka.go/plugins/golem15/fonoteka/plugin.go + - ../fonoteka.go/plugins/golem15/fonoteka/jobs.go + - ../fonoteka.go/plugins/golem15/fonoteka/mail.go + - ../fonoteka.go/plugins/golem15/fonoteka/config/config.yaml + - ../fonoteka.go/parity/parity_test.go + - ../fonoteka.go/parity/php_parity.sh + - ../fonoteka.go/parity/fonoteka_reset.php + - ../fonoteka.go/parity/check_corpus.go + - ../fonoteka.go/parity/manifest.yaml + - ../fonoteka.go/parity/README.md + - cmd/summer/parity.go + - modules/conga/job.go + +key-decisions: + - "ReleaseFetcher.FetchRelease takes the request's db and config and returns *csv.Map: one stateless fetcher serves every app, no app is captured at Boot" + - "The Discogs pick cases use a new csv-discogs seed state (csv plus alice's own token) instead of giving alice a token in the csv state, so the existing discogs-off fixture and every other csv case stay valid" + - "Every parity case replays through an upstream fake; a case without a sidecar gets an empty one, so an unrecorded vendor call fails the case instead of reaching the internet" + - "The digest worker mails only if its DELETE removed the queue row: two runs for one window mail once" + - "Reindex uses the search prefix for both the album and the legacy index (PHP hardcodes unprefixed names; identical with the default empty prefix)" + - "INTG-01 stays Pending: its Discogs routes land in 14-03" + +patterns-established: + - "php_parity.sh serve with PARITY_UPSTREAM_CA records PHP's vendor calls through summer parity:upstream (assumption A1 verified)" + - "php_parity.sh script runs a Winter-booting truth-table generator against the isolated instance" + +requirements-completed: [JOBS-02, JOBS-03, SRCH-02, CLI-05] + +duration: 80min +completed: 2026-10-03 +status: complete +--- + +# Phase 14 Plan 02: Discogs core, CSV and digest workers, prune and reindex Summary + +**A CSV import now finishes in Go. The match job finds Discogs candidates through a guarded client and a cross-process Postgres limiter, and pauses on rate limits. A picked candidate resolves to the draft PHP stores, replayed offline from PHP's recorded Discogs exchange. The import job writes the albums, subscribers get their digest, and the operator can prune notifications and rebuild the search index.** + +## Performance + +- **Duration:** 80 min +- **Started:** 2026-10-03T18:04:23Z +- **Completed:** 2026-10-03T19:24:36Z +- **Tasks:** 4 +- **Commits:** 6 in fonoteka.go, 2 code commits in summercms.go (plus this summary) +- **Files:** 74 changed in fonoteka.go (+9109/-140), 7 in summercms.go + +## Accomplishments + +- **Discogs client and limiter (INTG-01, Discogs half).** The `classes/discogs` package covers several pieces: + - PHP's request loop through `fetchguard` AllowHostsMode `api.discogs.com` with a 10 s timeout, PHP's headers and a code-constant base URI. A 404 is nil, 401/403 is `ErrTokenRejected`, a 429 waits Retry-After or the fallback inside the 15 s budget or fails with `RateLimitError`, and any other status is `RequestError`. + - Per-token buckets: the first 32 hex characters of an HMAC with the app key. Window state lives in the UNLOGGED `golem15_fonoteka_discogs_rate_windows` table, and one `INSERT … ON CONFLICT … WHERE … RETURNING` grants each slot. All waits go through an injected `Clock`. +- **Domain ports, each checked against PHP-generated truth tables.** `MapRelease`, `MapSearchResult` and `MapMasterVersion` keep PHP's key order. Also ported: `ParseInput`, `NormalizeBarcode` and `AlternateBarcode`, `ScoreRelease` (with a byte-exact `similar_text`), `ResolvePriceSuggestion`, `ImportResolver` and the release `Applicator` (dry run, cover only, host-locked covers). +- **Real row pick (D-08).** Boot installs the Discogs `ReleaseFetcher`. Three cases were recorded against live PHP through `summer parity:upstream` and replay offline with request assertions: `PATCH import/csv/{id}/rows/{rowId}` `discogs-pick` (200 with the mapped draft), `discogs-missing` (422) and `discogs-rate-limited` (422). +- **WR-02 (D-10).** Mapping, row save and cancel each lock the import row (`SELECT … FOR UPDATE`), re-check it, and cancel the jobs that the locked row names. The Discogs fetch happens before the lock. `TestCsvWR02` races commit against each of the three. +- **Workers (JOBS-02, JOBS-03).** + - The match pass has a 240 s timeout. It handles duplicate, gate-off and 0/1/many(≤10) results. A rate limit pauses it: it re-dispatches with delay = max(Retry-After, window remainder), so a 0.4 s remainder becomes 1 s. + - The import pass writes canonical rows by overwrite, fill or create, and writes draft or CSV rows with rating and first cover. Failed rows are marked `write_failed`, the pass cancels when the importer loses access, and it publishes one `collection.bulk_updated`. + - The digest worker uses PHP's templates and locale pick and mails once per queue row. + - PHP's real match run was recorded, and `TestCsvMatchJob/php-recorded` replays it: rows, drafts, candidates and job metadata match. +- **Commands (CLI-05, SRCH-02).** `fonoteka:prune-notifications` removes rows older than 90 days in a single DELETE. The daily schedule entry now resolves to it. `fonoteka:reindex [--drop-old-items-index]` refuses or aborts with PHP's messages, rebuilds by id in batches of 500, runs the `collection_id:=0` integrity check, and drops only the prefixed legacy index. + +## Task Commits + +fonoteka.go: +1. **Task 1 (tracer): real Discogs pick, limiter, mapper, sidecar replay**: `0496bb3` (feat) +2. **Task 2: client endpoints, limiter rules, domain ports**: `9227ec9` (feat) +3. **Task 3: WR-02 locks**: `ce4dd3a` (fix). **CSV match and import workers**: `fe24cbf` (feat) +4. **Task 4: digest worker**: `6aa0481` (feat). **prune and reindex commands**: `ef34015` (feat) + +summercms.go: +- `0a7635b` (fix): `summer parity:upstream` writes its sidecar on SIGINT/SIGTERM (Task 1 blocker) +- `cdd8d71` (feat): `conga.Describe`, used to assert the registered 240 s timeout (Task 3) + +The tracer gate after Task 1 re-ran its `` end to end. It passed, so the expansion tasks went ahead. + +## Assumption A1: verified + +With `HTTPS_PROXY=http://127.0.0.1:8425` and the parity CA passed as `-d curl.cainfo`/`-d openssl.cafile`, both raw curl and Guzzle (Laravel `Http`) reach the recording proxy. This holds from the CLI and from PHP's built-in web server. `artisan serve` does not forward `-d` to its child process, so `php_parity.sh serve` now starts `php -S` directly when `PARITY_UPSTREAM_CA` is set. The recorded sidecars come from real PHP runs, and no fixture was written by hand. + +## Deviations from Plan + +### Auto-fixed issues + +**1. [Rule 3 - Blocking] `summer parity:upstream` never wrote its sidecar** +- **Found during:** Task 1 precondition +- **Issue:** The bonfire command context had no signal handling. A backgrounded proxy (where the shell ignores SIGINT) could not be stopped gracefully, so `Flush` never ran. +- **Fix:** `signal.NotifyContext(ctx, os.Interrupt, SIGTERM)` in `runParityUpstream`. The tide README and the parity-testing docs now mention SIGTERM. +- **Commit:** 0a7635b (summercms.go) + +**2. [Rule 1 - Bug in plan] The migration file name `22_discogs_rate_windows.go` is a Windows-only build constraint** +- **Issue:** Go treats the `_windows.go` suffix as GOOS=windows, so the migration silently did not compile on Linux. +- **Fix:** The file is named `updates/22_discogs_rate_windows_table.go` and carries a comment. The acceptance grep passes against the new name. +- **Commit:** 0496bb3 + +**3. [Rule 2 - Correctness] `ReleaseFetcher` takes db and config** +- **Issue:** A fetcher that captured one app at Boot would serve every app in a process from that app's database. It also made tests reach the real api.discogs.com. Phase 13's T-13-16 test did exactly that and got release 1234567 back. +- **Fix:** `FetchRelease(ctx, db, cfg, user, id) (*csv.Map, error)`. The fetcher is stateless. T-13-16 and `TestCsvRowPickSeam` now go through scripted upstream fakes. +- **Commit:** 0496bb3 + +**4. [Rule 2 - Correctness] A separate `csv-discogs` seed state** +- **Issue:** Giving alice a token in the shared `csv` state would have flipped the recorded `discogs-off` case and other credential-dependent fixtures. +- **Fix:** The pick cases use a new `csv-discogs` extra in both `fonoteka_reset.php` and the Go seed. The token is the existing `secret:discogs-token` (`parityNewDiscogsToken`, which matches PHP's token rule). The rate window table is cleared per case. +- **Commit:** 0496bb3 + +**5. [Scope] All three pick cases recorded in Task 1** +- The plan put the 404 and 429 cases in Task 2. They share the recording harness, so they were recorded together with the success case and committed with Task 1. + +**6. [Rule 1 - Plan conflict] The digest templates are not byte copies** +- **Issue:** postcard renders Go templates, so PHP's Twig `{{ ownerName }}` fails to parse. The existing ported templates already use `{{ .var }}`. +- **Fix:** Subject, description, layout and text are verbatim; only `{{ x }}` became `{{ .x }}`. The acceptance check `diff … exits 0` therefore cannot pass: the diff is exactly the three variables. +- **Commit:** 6aa0481 + +**7. [Rule 2] Digest mailing is gated on the DELETE** +- PHP reads, deletes, then mails, so two racing runs could both mail. Go mails only when its DELETE removed the row. This enforces the plan's prohibition "never more than once per queue row". + +**8. [Plan detail] `RateStore.Window` instead of `WindowStart`** +- `SecondsUntilAvailable` needs both hits and window start, so the read method returns both. + +**9. [Plan detail] Test helpers and extra coverage** +- `TestCsvWR02` lives in its own file (`csv_wr02_test.go`) so the WR-02 fix is a separate commit. +- Extra tests beyond the plan: `TestBucketID` (pinned to PHP's HMAC), `TestNumberFormat4`, `TestCheckCorpusUpstreamCredential` and `TestWishlistDigestWorkerRegistered`. + +**10. [Framework] `conga.Describe`** +- The plan asks for the 240 s timeout to be asserted "from the registered job". conga had no accessor for it, so `conga.Describe(pact.Job) (JobInfo, bool)` was added, with README, docs and an example. +- The registration keeps the literal `conga.Timeout(240 * time.Second)` in a local variable. gofmt would otherwise rewrite it to `240*time.Second` inside the nested call, and the acceptance grep would fail. + +**Total deviations:** 10. Five are fixes, two are plan conflicts that could not be met as written (the migration name and the template diff), and three are refinements. **Impact:** no scope creep. Both framework changes are small and documented. + +## Issues Encountered + +- The test environment needs `FORCE_COLOR=` and `TMPDIR`/`GOTMPDIR` set to `~/.cache/gotest-tmp`, as noted in 14-01. +- `household_smoke_test.go` was already not gofmt-clean before this plan. It is out of scope and was left untouched. +- The CSV upload made during recording was written into the PHP checkout's `storage/app/fonoteka-csv/` and removed afterwards, as the README instructs. + +## Verification + +- fonoteka.go: `go vet ./...` passes. `go test ./... -count=1` passes in the root module, including parity. `go test ./plugins/golem15/fonoteka/... -count=1` passes in every package. Each task's `` command ran with `-race` and showed `--- PASS` for every named test, with no SKIP and no DATA RACE. +- Parity corpus: 157 routes ported and passing, now including the three new row-edit cases. `TestParityCorpus/coverage`, `TestUpstreamSidecarsAreReplayed`, `TestSchemaMatchesPHPSnapshot` and `TestFonotekaNuxtFlows` all pass. `check_corpus --routes … --require-recorded --check-secrets` passes (171/171 recorded). +- summercms.go: `go vet ./...` and `go test ./... -count=1` pass. `TestDocsTree` and `summer docs:build --check` pass. + +## Known Stubs + +None. + +## Threat Flags + +None. The new surface was already in the plan's threat model: the Discogs egress through fetchguard AllowHostsMode, the limiter table and the CSV workers. + +## Next Phase Readiness + +- 14-03 can mount the Discogs routes on `discogs.ForUser`, `ImportResolver`, `ScoreRelease`, `Applicator` and `ResolvePriceSuggestion`. The sidecar recording recipe is in `parity/README.md` ("Upstream sidecars"). +- INTG-01 stays Pending until those routes land. + +--- +*Phase: 14-domain-jobs-and-external-integrations* +*Completed: 2026-10-03* + +## Self-Check: PASSED + +All key files exist; commits 0a7635b and cdd8d71 (summercms.go) and 0496bb3, 9227ec9, ce4dd3a, fe24cbf, 6aa0481 and ef34015 (fonoteka.go) are present.