16 KiB
16 KiB
phase, plan, subsystem, tags, requires, provides, affects, actuals, plan_head_before, plan_head_after, app_repo_head_before, app_repo_head_after, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, duration, completed, status
| phase | plan | subsystem | tags | requires | provides | affects | actuals | plan_head_before | plan_head_after | app_repo_head_before | app_repo_head_after | tech-stack | key-files | key-decisions | patterns-established | requirements-completed | duration | completed | status | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 14-domain-jobs-and-external-integrations | 02 | jobs |
|
|
|
|
|
43b869d613 |
cdd8d710fa |
5738e82c5ee683d37498bc04c9453a17dd41986c | ef34015a223a67e649e99bac410fb728337c0ee3 |
|
|
|
|
|
80min | 2026-10-03 | complete |
Phase 14 Plan 02: Discogs core, CSV and digest workers, prune and reindex Summary
A CSV import now finishes in Go. The match job finds Discogs candidates through a guarded client and a cross-process Postgres limiter, and pauses on rate limits. A picked candidate resolves to the draft PHP stores, replayed offline from PHP's recorded Discogs exchange. The import job writes the albums, subscribers get their digest, and the operator can prune notifications and rebuild the search index.
Performance
- Duration: 80 min
- Started: 2026-10-03T18:04:23Z
- Completed: 2026-10-03T19:24:36Z
- Tasks: 4
- Commits: 6 in fonoteka.go, 2 code commits in summercms.go (plus this summary)
- Files: 74 changed in fonoteka.go (+9109/-140), 7 in summercms.go
Accomplishments
- Discogs client and limiter (INTG-01, Discogs half). The
classes/discogspackage covers several pieces:- PHP's request loop through
fetchguardAllowHostsModeapi.discogs.comwith a 10 s timeout, PHP's headers and a code-constant base URI. A 404 is nil, 401/403 isErrTokenRejected, a 429 waits Retry-After or the fallback inside the 15 s budget or fails withRateLimitError, and any other status isRequestError. - Per-token buckets: the first 32 hex characters of an HMAC with the app key. Window state lives in the UNLOGGED
golem15_fonoteka_discogs_rate_windowstable, and oneINSERT … ON CONFLICT … WHERE … RETURNINGgrants each slot. All waits go through an injectedClock.
- PHP's request loop through
- Domain ports, each checked against PHP-generated truth tables.
MapRelease,MapSearchResultandMapMasterVersionkeep PHP's key order. Also ported:ParseInput,NormalizeBarcodeandAlternateBarcode,ScoreRelease(with a byte-exactsimilar_text),ResolvePriceSuggestion,ImportResolverand the releaseApplicator(dry run, cover only, host-locked covers). - Real row pick (D-08). Boot installs the Discogs
ReleaseFetcher. Three cases were recorded against live PHP throughsummer parity:upstreamand replay offline with request assertions:PATCH import/csv/{id}/rows/{rowId}discogs-pick(200 with the mapped draft),discogs-missing(422) anddiscogs-rate-limited(422). - WR-02 (D-10). Mapping, row save and cancel each lock the import row (
SELECT … FOR UPDATE), re-check it, and cancel the jobs that the locked row names. The Discogs fetch happens before the lock.TestCsvWR02races commit against each of the three. - Workers (JOBS-02, JOBS-03).
- The match pass has a 240 s timeout. It handles duplicate, gate-off and 0/1/many(≤10) results. A rate limit pauses it: it re-dispatches with delay = max(Retry-After, window remainder), so a 0.4 s remainder becomes 1 s.
- The import pass writes canonical rows by overwrite, fill or create, and writes draft or CSV rows with rating and first cover. Failed rows are marked
write_failed, the pass cancels when the importer loses access, and it publishes onecollection.bulk_updated. - The digest worker uses PHP's templates and locale pick and mails once per queue row.
- PHP's real match run was recorded, and
TestCsvMatchJob/php-recordedreplays it: rows, drafts, candidates and job metadata match.
- Commands (CLI-05, SRCH-02).
fonoteka:prune-notificationsremoves rows older than 90 days in a single DELETE. The daily schedule entry now resolves to it.fonoteka:reindex [--drop-old-items-index]refuses or aborts with PHP's messages, rebuilds by id in batches of 500, runs thecollection_id:=0integrity check, and drops only the prefixed legacy index.
Task Commits
fonoteka.go:
- Task 1 (tracer): real Discogs pick, limiter, mapper, sidecar replay:
0496bb3(feat) - Task 2: client endpoints, limiter rules, domain ports:
9227ec9(feat) - Task 3: WR-02 locks:
ce4dd3a(fix). CSV match and import workers:fe24cbf(feat) - Task 4: digest worker:
6aa0481(feat). prune and reindex commands:ef34015(feat)
summercms.go:
0a7635b(fix):summer parity:upstreamwrites its sidecar on SIGINT/SIGTERM (Task 1 blocker)cdd8d71(feat):conga.Describe, used to assert the registered 240 s timeout (Task 3)
The tracer gate after Task 1 re-ran its <verify> end to end. It passed, so the expansion tasks went ahead.
Assumption A1: verified
With HTTPS_PROXY=http://127.0.0.1:8425 and the parity CA passed as -d curl.cainfo/-d openssl.cafile, both raw curl and Guzzle (Laravel Http) reach the recording proxy. This holds from the CLI and from PHP's built-in web server. artisan serve does not forward -d to its child process, so php_parity.sh serve now starts php -S directly when PARITY_UPSTREAM_CA is set. The recorded sidecars come from real PHP runs, and no fixture was written by hand.
Deviations from Plan
Auto-fixed issues
1. [Rule 3 - Blocking] summer parity:upstream never wrote its sidecar
- Found during: Task 1 precondition
- Issue: The bonfire command context had no signal handling. A backgrounded proxy (where the shell ignores SIGINT) could not be stopped gracefully, so
Flushnever ran. - Fix:
signal.NotifyContext(ctx, os.Interrupt, SIGTERM)inrunParityUpstream. The tide README and the parity-testing docs now mention SIGTERM. - Commit:
0a7635b(summercms.go)
2. [Rule 1 - Bug in plan] The migration file name 22_discogs_rate_windows.go is a Windows-only build constraint
- Issue: Go treats the
_windows.gosuffix as GOOS=windows, so the migration silently did not compile on Linux. - Fix: The file is named
updates/22_discogs_rate_windows_table.goand carries a comment. The acceptance grep passes against the new name. - Commit: 0496bb3
3. [Rule 2 - Correctness] ReleaseFetcher takes db and config
- Issue: A fetcher that captured one app at Boot would serve every app in a process from that app's database. It also made tests reach the real api.discogs.com. Phase 13's T-13-16 test did exactly that and got release 1234567 back.
- Fix:
FetchRelease(ctx, db, cfg, user, id) (*csv.Map, error). The fetcher is stateless. T-13-16 andTestCsvRowPickSeamnow go through scripted upstream fakes. - Commit: 0496bb3
4. [Rule 2 - Correctness] A separate csv-discogs seed state
- Issue: Giving alice a token in the shared
csvstate would have flipped the recordeddiscogs-offcase and other credential-dependent fixtures. - Fix: The pick cases use a new
csv-discogsextra in bothfonoteka_reset.phpand the Go seed. The token is the existingsecret:discogs-token(parityNewDiscogsToken, which matches PHP's token rule). The rate window table is cleared per case. - Commit: 0496bb3
5. [Scope] All three pick cases recorded in Task 1
- The plan put the 404 and 429 cases in Task 2. They share the recording harness, so they were recorded together with the success case and committed with Task 1.
6. [Rule 1 - Plan conflict] The digest templates are not byte copies
- Issue: postcard renders Go templates, so PHP's Twig
{{ ownerName }}fails to parse. The existing ported templates already use{{ .var }}. - Fix: Subject, description, layout and text are verbatim; only
{{ x }}became{{ .x }}. The acceptance checkdiff … exits 0therefore cannot pass: the diff is exactly the three variables. - Commit: 6aa0481
7. [Rule 2] Digest mailing is gated on the DELETE
- PHP reads, deletes, then mails, so two racing runs could both mail. Go mails only when its DELETE removed the row. This enforces the plan's prohibition "never more than once per queue row".
8. [Plan detail] RateStore.Window instead of WindowStart
SecondsUntilAvailableneeds both hits and window start, so the read method returns both.
9. [Plan detail] Test helpers and extra coverage
TestCsvWR02lives in its own file (csv_wr02_test.go) so the WR-02 fix is a separate commit.- Extra tests beyond the plan:
TestBucketID(pinned to PHP's HMAC),TestNumberFormat4,TestCheckCorpusUpstreamCredentialandTestWishlistDigestWorkerRegistered.
10. [Framework] conga.Describe
- The plan asks for the 240 s timeout to be asserted "from the registered job". conga had no accessor for it, so
conga.Describe(pact.Job) (JobInfo, bool)was added, with README, docs and an example. - The registration keeps the literal
conga.Timeout(240 * time.Second)in a local variable. gofmt would otherwise rewrite it to240*time.Secondinside the nested call, and the acceptance grep would fail.
Total deviations: 10. Five are fixes, two are plan conflicts that could not be met as written (the migration name and the template diff), and three are refinements. Impact: no scope creep. Both framework changes are small and documented.
Issues Encountered
- The test environment needs
FORCE_COLOR=andTMPDIR/GOTMPDIRset to~/.cache/gotest-tmp, as noted in 14-01. household_smoke_test.gowas already not gofmt-clean before this plan. It is out of scope and was left untouched.- The CSV upload made during recording was written into the PHP checkout's
storage/app/fonoteka-csv/and removed afterwards, as the README instructs.
Verification
- fonoteka.go:
go vet ./...passes.go test ./... -count=1passes in the root module, including parity.go test ./plugins/golem15/fonoteka/... -count=1passes in every package. Each task's<verify>command ran with-raceand showed--- PASSfor every named test, with no SKIP and no DATA RACE. - Parity corpus: 157 routes ported and passing, now including the three new row-edit cases.
TestParityCorpus/coverage,TestUpstreamSidecarsAreReplayed,TestSchemaMatchesPHPSnapshotandTestFonotekaNuxtFlowsall pass.check_corpus --routes … --require-recorded --check-secretspasses (171/171 recorded). - summercms.go:
go vet ./...andgo test ./... -count=1pass.TestDocsTreeandsummer docs:build --checkpass.
Known Stubs
None.
Threat Flags
None. The new surface was already in the plan's threat model: the Discogs egress through fetchguard AllowHostsMode, the limiter table and the CSV workers.
Next Phase Readiness
- 14-03 can mount the Discogs routes on
discogs.ForUser,ImportResolver,ScoreRelease,ApplicatorandResolvePriceSuggestion. The sidecar recording recipe is inparity/README.md("Upstream sidecars"). - INTG-01 stays Pending until those routes land.
Phase: 14-domain-jobs-and-external-integrations Completed: 2026-10-03
Self-Check: PASSED
All key files exist; commits 0a7635b and cdd8d71 (summercms.go) and 0496bb3, 9227ec9, ce4dd3a, fe24cbf, 6aa0481 and ef34015 (fonoteka.go) are present.