summercms/.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-04-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, actuals, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, coverage, duration, completed, status, plan_head_before, plan_head_after

phase	plan	subsystem	tags	requires	provides	affects	actuals	tech-stack	key-files	key-decisions	patterns-established	requirements-completed	coverage	duration	completed	status	plan_head_before	plan_head_after
13-p-ytarium-api-wishlist-notifications-csv-credentials-public	04	api	csv import export fputcsv fgetcsv charmap gocloud-blob conga river parity tide	phase provides 13-01 job contract (CsvImport*/CsvMatch* kinds, queues, labels, args), conga workerless kinds, tide Content-Disposition date mask, php_parity.sh QUEUE_CONNECTION override and rows dump phase provides 13-03 jobDB fresh-statement handle for job queues on a write transaction, JobDispatcher wiring phase provides 12 album search SQL path and filters, album embeds serializer, fonoteka seed hook and recording driver	classes/csv package: Headers, ExportRow, FormulaSafe, WriteRecord (fputcsv port), PipeEncode/PipeDecode, Parse (CsvAlbumParser with a byte-exact fgetcsv/str_getcsv port), DetectColumns, DetectFields, GuessDelimiter, LooksCombined, MapColumns, ParseCanonicalRow, CanonicalID, ParseError, ordered Map with DecodeOrdered/AsMap, MaxRows, MaxBytes classes: ExportAlbums, StoreCsvImport, CsvImportFor, SerializeCsvImport, ShowCsvImport, UpdateCsvMapping, UpdateCsvRow, CommitCsvImport, CancelCsvImport, CsvJobs, ReleaseFetcher, SetReleaseFetcher, CsvImportError with ErrCsvUnreadable/TooLarge/ValidationFailed/AlreadyCommitted/MatchNotReady, ErrDiscogsUnavailable, ErrDiscogsRateLimited 8 CSV routes ported (151 ported): JWT export/csv, import/csv store (throttle:10,1), show, mapping, rows/{rowId}, commit, cancel; token-group export/csv (inv.scope:read) Private CSV bucket golem15.fonoteka.csv.bucket_url (default file://./storage/app), keys fonoteka-csv/<user id>/<uuid>.csv Parity: csv seed state on both sides (albums, cover, 7 imports, rows, job rows, aligned id sequences), 67 route fixtures for the 8 routes, nuxt-csv flow with its job-row golden, PHP truth tables from parity/csv_truth_tables.php	13-05 13-06 14	tokens tasks commits 132000 3 3	added patterns golang.org/x/text (direct requirement of the fonoteka plugin module; charmap) Byte-level PHP ports are proven by differential runs against PHP (16k fgetcsv/str_getcsv inputs, 1.5k parser files) and pinned by committed truth tables that PHP generates Client JSON whose key order PHP keeps (column_map) is decoded into an ordered csv.Map, never a Go map Uploads PHP stores on Laravel's local disk go to a private gocloud bucket opened per app, separate from the public uploads bucket Parity states that create rows restart their id sequences at the same floors on both sides, so uncaptured ids in bodies match	created modified fonoteka.go/plugins/golem15/fonoteka/classes/csv/contract.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/writer.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/pipe_codec.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/fgetcsv.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/parser.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/detector.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/mapper.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/canonical_id.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/errors.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/ordered.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/php.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/csv_test.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/truth_table_test.go fonoteka.go/plugins/golem15/fonoteka/classes/csv/testdata/ (22 input CSVs, php_fputcsv/contract/parser/detector.json, README.md) fonoteka.go/plugins/golem15/fonoteka/classes/csv_export.go fonoteka.go/plugins/golem15/fonoteka/classes/csv_import_service.go fonoteka.go/plugins/golem15/fonoteka/controllers/api/csv_export_controller.go fonoteka.go/plugins/golem15/fonoteka/controllers/api/csv_import_controller.go fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go fonoteka.go/parity/csv_truth_tables.php fonoteka.go/parity/fixtures/nuxt/nuxt-csv.yaml fonoteka.go/parity/fixtures/nuxt/nuxt-csv.rows.json fonoteka.go/parity/fixtures/nuxt/files/csv-plain.csv fonoteka.go/parity/fixtures/nuxt/files/csv-canonical.csv fonoteka.go/parity/fixtures/routes/files/csv-*.csv, csv-cover.png (8 upload parts) fonoteka.go/plugins/golem15/fonoteka/models/csv_import.go fonoteka.go/plugins/golem15/fonoteka/models/csv_import_row.go fonoteka.go/plugins/golem15/fonoteka/controllers/api/request.go fonoteka.go/plugins/golem15/fonoteka/config/config.yaml fonoteka.go/plugins/golem15/fonoteka/routes.go fonoteka.go/plugins/golem15/fonoteka/go.mod fonoteka.go/plugins/golem15/fonoteka/phase08_coverage_test.go fonoteka.go/plugins/golem15/fonoteka/phase12_security_test.go fonoteka.go/parity/manifest.yaml fonoteka.go/parity/fixtures/routes/ (8 CSV routes, 67 route fixtures) fonoteka.go/parity/fonoteka_reset.php fonoteka.go/parity/fonoteka_seed_test.go fonoteka.go/parity/fonoteka_flows_test.go fonoteka.go/parity/migrate_test.go fonoteka.go/parity/parity_contract_test.go fonoteka.go/parity/parity_test.go fonoteka.go/parity/README.md	An ISO-8859-2 file is decoded as Windows-1250, as PHP does: iconv('Windows-1250', 'UTF-8//IGNORE') drops the five bytes Windows-1250 leaves undefined instead of failing, so the ISO-8859-2 branch only runs when that yields an empty string. Letters both encodings share survive; ź, ą, ś and the other differing letters do not (Łódź in ISO-8859-2 imports as ŁódĽ). The plan expected the ISO-8859-2 twin to match its UTF-8 file; the truth table records PHP's real result and Go follows it. ReleaseFetcher.FetchRelease returns the mapped draft (Discogs getRelease plus DiscogsMapper::mapRelease in one seam). The Phase 13 default fails with ErrDiscogsUnavailable, so a pick answers 422 discogs_unavailable and writes nothing; only a test-installed fetcher resolves a row, and Phase 14 installs the real one. Export cover URLs are absolute like Winter's File::getPath(): app.url plus the upload's public path (PHP prefixes the request origin, which equals APP_URL on the isolated instance and app.url in production). CSV model JSON columns (column_map, raw_json, candidates_json, draft_json) are Jsonable[json.RawMessage]: PHP stores a list in raw_json for plain files, and column_map is returned with the key order PHP wrote. The csv parity state restarts the csv_imports, csv_import_rows and job id sequences at 80000100, 81000100 and 90000100 on both sides, so the import, row and job ids a recording creates (and the {"csv_import_id":N} job metadata) match without capturing every one.	PHP byte-format ports: differential-test against PHP on random inputs while porting, then commit curated truth tables generated by a PHP script in parity/ Routes whose responses carry string ids under *_id keys disable tide's id mask for those keys per step (discogs_id, selected_discogs_id)	API-05	id description requirement verification human_judgment D1 CSV export on both groups: BOM, canonical header row, one fputcsv-quoted row per album of the active collection with the album index's filters and default order; formula cells neutralised; dated attachment filename API-05 kind ref status unit fonoteka.go/plugins/golem15/fonoteka/classes/csv/csv_test.go#TestPHPFputcsv pass kind ref status integration fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvExport pass kind ref status integration fonoteka.go/parity/parity_test.go#TestParityCorpus (GET export/csv jwt, GET export/csv personal_token) pass false id description requirement verification human_judgment D2 CSV parser, detector, mapper and canonical row parser match PHP across encodings (UTF-8, Windows-1250, ISO-8859-2), BOM, CRLF and multi-line cells, header aliases, combined values, overrides, NUL bytes, empty and letterless files, and 5000 vs 5001 rows API-05 kind ref status unit fonoteka.go/plugins/golem15/fonoteka/classes/csv/truth_table_test.go#TestCsvParserTruthTable pass kind ref status unit fonoteka.go/plugins/golem15/fonoteka/classes/csv/truth_table_test.go#TestCsvDetectorTruthTable pass false id description requirement verification human_judgment D3 Store and show: private bucket under fonoteka-csv/<uid>/, preview vs mapping, PHP's error envelopes (Polish for controller checks, English parse messages), 5242880 vs 5242881 bytes, per_page clamped 1..50, summary [] without rows, imports visible only to their creator while the collection is reachable API-05 kind ref status integration fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvStoreAndShow pass kind ref status integration fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvImportScope pass kind ref status integration fonoteka.go/parity/parity_test.go#TestParityCorpus (POST import/csv, GET import/csv/{id}) pass false id description requirement verification human_judgment D4 Mapping, row edit, commit and cancel: match and import jobs queued with the D-03 contract on unserved queues and left unworked (D-04), single compare-and-swap commit with replay, cancel stopping both jobs, D-05 pick seam answering discogs_unavailable without writing API-05 kind ref status integration fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvJobRows pass kind ref status integration fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvCommitCAS pass kind ref status integration fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvCancel pass kind ref status integration fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvRowPickSeam pass kind ref status integration fonoteka.go/parity/parity_test.go#TestParityCorpus (mapping, rows, commit, cancel) pass false id description requirement verification human_judgment D5 Recorded Nuxt import wizard journey with QUEUE_CONNECTION=database; Go job rows equal PHP's golem15_apparatus_jobs rows; corpus at 151 ported; secrets check green API-05 kind ref status e2e fonoteka.go/parity/fonoteka_flows_test.go#TestFonotekaNuxtFlows/nuxt-csv pass kind ref status integration fonoteka.go/parity/parity_test.go#TestParityCorpus/coverage (151 ported, 151 passing) pass kind ref status other go run ./parity/check_corpus.go --manifest parity/manifest.yaml --routes .../routes.php --require-recorded --check-secrets pass false	80min	2026-10-03	complete	37405f014878e249bd501ec93b1b7ffaa632772b	c540102eea33d68733ee4a5366be28c035954255

Phase 13 Plan 04: CSV export and import Summary

The collection backup downloads as PHP's exact CSV on both groups. The import wizard runs in Go up to a queued import job: upload in UTF-8, Windows-1250 or ISO-8859-2, preview or column mapping, row fixes, commit and cancel. The match and import jobs wait for their Phase 14 workers, and a Discogs pick never invents release data. The parser is a byte-level port of PHP's fgetcsv and CSV classes, checked against PHP on thousands of generated files. The corpus is at 151 ported routes.

Performance

• Duration: 80 min
• Started: 2026-10-03T06:30:14Z
• Completed: 2026-10-03T07:50:37Z
• Tasks: 3 of 3
• Files modified: 141 in fonoteka.go: 24 source, test and script files, 22 parser inputs, 4 truth tables and a README, 8 upload parts, 2 flow parts, 67 route fixtures, the flow and its rows golden, and the manifest, seeds, README and module file

Accomplishments

• Export (Task 1).
  • csv.WriteRecord ports fputcsv with an empty escape. A field with a comma, quote, CR, LF, tab or space is quoted; null and false are empty, true is 1, floats use PHP's form. It never uses the standard library's CSV writer.
  • csv.ExportRow ports CsvAlbumContract::exportRow. It applies the FORMULA_PATTERN apostrophe guard, pipe-encodes styles, artists and covers, and tab-encodes the tracklist.
  • classes.ExportAlbums runs the album index's SQL path and filters in the index's default order, in batches of 100.
  • CsvExport writes the BOM, the headers and Content-Disposition: attachment; filename=plytarium-kolekcja-<Y-m-d>.csv. The route is mounted on the JWT group and on the token group (inv.scope:read).
• Parser (Task 2).
  • fgetcsv and str_getcsv are ported from php_fgetcsv, including its reads past the line end. Over 16,088 random record reads, 0 differ from PHP.
  • csv.Parse ports CsvAlbumParser. It decodes as iconv //IGNORE does, runs the \p{L} check and strips the BOM. It resolves the delimiter with strtok, detects or overrides the map, backfills headers, and enforces the 5000-row cap and NUL rejection.
  • The detector, mapper and canonical row parser are ported too. Over 1,507 random files (key order included), 0 differ from PHP.
  • The committed truth tables come from parity/csv_truth_tables.php.
• Import session (Tasks 2 and 3).
  • Store checks the extension and size, then parses. Invalid canonical rows are validation_failed. The file goes to the private bucket. The import is created in preview or mapping, with its rows; canonical ids are matched only inside the importer's collection.
  • Show paginates rows by row_index. summary is the count per status, or [] without rows. progress comes from the row counts, or from the summer_jobs row.
  • Mapping keeps the client's column_map key order and re-parses the stored file. It cancels the previous match job, replaces the rows, and queues CsvMatchArgs on the import's transaction.
  • Row edit handles skip, accept_csv and the candidate allow-list pick through the ReleaseFetcher seam.
  • Commit is a single UPDATE ... WHERE status = 'preview' that queues one CsvImportArgs job. A replay answers the current job.
  • Cancel stops both jobs, the summer_jobs rows and the River jobs.
• Parity.
  • The csv state exists on both sides: albums with a cover, seven imports in every needed status, their rows and job rows, and aligned id sequences.
  • 67 route fixtures cover the 8 routes. The 5 Task 3 routes were recorded with QUEUE_CONNECTION=database.
  • The nuxt-csv flow (17 steps) and its job-row golden replay green.

Task Commits

fonoteka.go (master, not pushed):

1. Task 1: CSV export with the fputcsv port - 643c9dd (feat)
2. Task 2: parser package, private bucket, store and show - 8af008e (feat)
3. Task 3: mapping, rows, commit, cancel, token export, nuxt-csv flow - c540102 (feat)

Decisions Made

See key-decisions. The user may want to look at the first one. PHP decodes an ISO-8859-2 upload as Windows-1250, so Polish letters that differ between the two encodings come out wrong (Łódź imports as ŁódĽ). The Go port keeps that behaviour, because the PHP truth table is the contract. Fixing it would be a deliberate change in both implementations.

Deviations from Plan

Auto-fixed Issues

1. [Rule 1 - Plan premise] ISO-8859-2 files do not match their UTF-8 twin in PHP

• Found during: Task 2 (truth-table generation)
• Issue: The plan's encoding edge expected a Windows-1250 file and an ISO-8859-2 file with Łódź to import with the UTF-8 values. PHP's iconv //IGNORE never fails on Windows-1250, so ISO-8859-2 bytes are read as Windows-1250.
• Fix: The port follows PHP. The truth table pins all three files (lodz_*.csv), plus an ISO-8859-2 file that uses only shared letters and does match its twin (shared_*.csv).
• Commit: 8af008e

2. [Rule 3 - Blocking] The CSV model JSON columns could not hold PHP's data

• Issue: raw_json holds a list for non-canonical files, and column_map must keep its key order.
• Fix: ColumnMap, RawJSON, CandidatesJSON and DraftJSON are now lagoon.Jsonable[json.RawMessage]. No other code used them.
• Commit: 8af008e

3. [Rule 1 - Parity] tide masks *_id keys as integer ids

• Issue: Discogs ids are strings, so the mask failed discogs_id and selected_discogs_id.
• Fix: CSV steps disable the mask for those two keys with step normalize rules.
• Commit: 8af008e

4. [Recording] The 5 MiB size case is not recorded

• Issue: The isolated PHP runs with php.ini's default upload_max_filesize of 2M, which rejects such a file before the controller does. The Go parity replay also caps uploads at 1 MiB.
• Coverage instead: TestCsvStoreAndShow checks 5242880 bytes (accepted) and 5242881 bytes (csv_too_large). The 5001-row case is recorded.

5. [Structure] API names differ from the plan's list

• ExportAlbums(ctx, db, AlbumSearchParams, coverBase, fn func([]string) error) lives in package classes. It reuses searchSQL, orderSearch and resolveSearchSort directly, so album_search.go is unchanged.
• MatchCanonicalID is split in two:
  • csv.CanonicalID is the pure check;
  • classes.matchCanonicalAlbum is the scoped lookup, since the csv package has no database.
• Other API changes:
  • CommitCsvImport takes the mode and whether it was sent.
  • ShowCsvImport was added.
  • CsvJobs covers Dispatch and CancelJob.
• Route upload parts are in fixtures/routes/files/, because tide resolves part files relative to the fixture. The flow parts are in fixtures/nuxt/files/.

6. [Rule 3 - Test inventories] Two guard tests named the old state

• The Phase 8 coverage subtest for CSV now asserts the real surfaces: export on both groups, the six import routes on JWT only.
• The T-12-17 part-file check accepts small csv-* uploads that contain no secret shape.
• Commits: 643c9dd, 8af008e, c540102

7. [Tooling] go work sync side effects reverted

• Promoting golang.org/x/text changed only the plugin go.mod; its hash was already in go.sum.
• go work sync also reordered go.work and dropped the root toolchain line. Both were reverted, so the root go.mod, go.sum and go.work.sum are unchanged.

---

Total deviations: 7: 1 plan premise corrected by the PHP truth, 2 blocking model or harness issues, 1 recording limitation covered by Go tests, 1 structural API naming change, 1 test-inventory update and 1 tooling revert. Impact: every recorded body, header and job row matches PHP.

Issues Encountered

• TestPhase09SecurityRoutes (12.2 cabana routes) still fails in the fonoteka plugin package; it is pre-existing and logged in deferred-items.md. Every other suite is green: the root module, parity (TestParityCorpus 151/151, TestFonotekaNuxtFlows including nuxt-csv), and the plugin's subpackages with -race.
• The shell aliases rm to rm -i, which stalled two commands; later commands used command rm or -f.
• The isolated PHP server was started for recording (sync, then QUEUE_CONNECTION=database) and stopped before returning. Only the CSV uploads written today were removed from the PHP checkout's storage/app/fonoteka-csv/; an older file there was left in place.

Known Stubs

None. The CSV import and match jobs have no worker by design (D-04, Phase 14 JOBS-02). The Discogs pick seam fails by design until Phase 14 (D-05, INTG-01). Neither is a stub.

Threat Flags

None beyond the plan's register:

• T-13-12: CsvImportFor (creator plus reachable collection) and TestCsvImportScope.
• T-13-13: the private bucket, server-side keys, and the public-bucket absence check in TestCsvStoreAndShow.
• T-13-14: FormulaSafe on every exported cell.
• T-13-15: the 5 MiB cap, 5000 rows, NUL rejection and throttle:10,1.
• T-13-16: the candidate allow-list and the failing fetcher.
• T-13-17: the single compare-and-swap, checked by TestCsvCommitCAS.
• T-13-31: unserved queues and cancel stopping River jobs.
• T-13-SC: x/text v0.42.0 pinned by go.sum.

User Setup Required

None. In production, golem15.fonoteka.csv.bucket_url defaults to file://./storage/app (Laravel's local disk, never served). Point it at another private location if the binary's working directory differs.

Next Phase Readiness

• 13-05 can rely on the token-group route table: export/csv is now mounted with inv.scope:read.
• 13-06 should add the CSV write routes (store, mapping, rows, commit, cancel) to write_endpoints_fuzz_test.go and the 8 routes to the Phase 13 route-table test.
• Phase 14:
  • It registers workers for golem15.fonoteka.csv_import and golem15.fonoteka.csv_match on fonoteka_csv_import and fonoteka_csv_match. The rows carry {"csv_import_id":N}.
  • It installs a real ReleaseFetcher with SetReleaseFetcher, which ports DiscogsClient::getRelease and DiscogsMapper::mapRelease.
  • It records the successful-pick case.

Self-Check: PASSED

• Created files exist: every key-files.created path checked with test -e.
• Commits exist in fonoteka.go: 643c9dd, 8af008e, c540102.
• Plan verification:
  • fonoteka.go go vet ./... is clean (root and plugin module).
  • go test ./... is green for the root module and the plugin subpackages; the plugin package fails only the pre-existing TestPhase09SecurityRoutes.
  • TestParityCorpus has 151 routes ported and passing.
  • TestFonotekaNuxtFlows/nuxt-csv passes.
  • check_corpus --require-recorded --check-secrets is green.
• Acceptance checks:
  • encoding/csv appears 0 times in writer.go.
  • plytarium-kolekcja- appears once in the export controller.
  • x/text is a direct requirement of the plugin module, and parser.go imports charmap.
  • The plugin config.yaml has csv.bucket_url and no uploads/public.
  • testdata has Windows-1250 and ISO-8859-2 samples.
  • jobs.go names no CSV kind.
  • nuxt-csv.rows.json holds fonoteka.csv.import.
  • The service names CsvImportLabel, CsvMatchLabel and discogs_unavailable.