docs(13-04): complete CSV export and import plan
This commit is contained in:
Jakub Zych
@@ -0,0 +1,324 @@
|
|||||||
|
---
|
||||||
|
phase: 13-p-ytarium-api-wishlist-notifications-csv-credentials-public
|
||||||
|
plan: 04
|
||||||
|
subsystem: api
|
||||||
|
tags: [csv, import, export, fputcsv, fgetcsv, charmap, gocloud-blob, conga, river, parity, tide]
|
||||||
|
|
||||||
|
requires:
|
||||||
|
- phase: 13-01
|
||||||
|
provides: job contract (CsvImport*/CsvMatch* kinds, queues, labels, args), conga workerless kinds, tide Content-Disposition date mask, php_parity.sh QUEUE_CONNECTION override and rows dump
|
||||||
|
- phase: 13-03
|
||||||
|
provides: jobDB fresh-statement handle for job queues on a write transaction, JobDispatcher wiring
|
||||||
|
- phase: 12
|
||||||
|
provides: album search SQL path and filters, album embeds serializer, fonoteka seed hook and recording driver
|
||||||
|
provides:
|
||||||
|
- "classes/csv package: Headers, ExportRow, FormulaSafe, WriteRecord (fputcsv port), PipeEncode/PipeDecode, Parse (CsvAlbumParser with a byte-exact fgetcsv/str_getcsv port), DetectColumns, DetectFields, GuessDelimiter, LooksCombined, MapColumns, ParseCanonicalRow, CanonicalID, ParseError, ordered Map with DecodeOrdered/AsMap, MaxRows, MaxBytes"
|
||||||
|
- "classes: ExportAlbums, StoreCsvImport, CsvImportFor, SerializeCsvImport, ShowCsvImport, UpdateCsvMapping, UpdateCsvRow, CommitCsvImport, CancelCsvImport, CsvJobs, ReleaseFetcher, SetReleaseFetcher, CsvImportError with ErrCsvUnreadable/TooLarge/ValidationFailed/AlreadyCommitted/MatchNotReady, ErrDiscogsUnavailable, ErrDiscogsRateLimited"
|
||||||
|
- "8 CSV routes ported (151 ported): JWT export/csv, import/csv store (throttle:10,1), show, mapping, rows/{rowId}, commit, cancel; token-group export/csv (inv.scope:read)"
|
||||||
|
- "Private CSV bucket golem15.fonoteka.csv.bucket_url (default file://./storage/app), keys fonoteka-csv/<user id>/<uuid>.csv"
|
||||||
|
- "Parity: csv seed state on both sides (albums, cover, 7 imports, rows, job rows, aligned id sequences), 67 route fixtures for the 8 routes, nuxt-csv flow with its job-row golden, PHP truth tables from parity/csv_truth_tables.php"
|
||||||
|
affects: [13-05, 13-06, 14]
|
||||||
|
|
||||||
|
actuals:
|
||||||
|
tokens: 132000
|
||||||
|
tasks: 3
|
||||||
|
commits: 3
|
||||||
|
|
||||||
|
tech-stack:
|
||||||
|
added: ["golang.org/x/text (direct requirement of the fonoteka plugin module; charmap)"]
|
||||||
|
patterns:
|
||||||
|
- "Byte-level PHP ports are proven by differential runs against PHP (16k fgetcsv/str_getcsv inputs, 1.5k parser files) and pinned by committed truth tables that PHP generates"
|
||||||
|
- "Client JSON whose key order PHP keeps (column_map) is decoded into an ordered csv.Map, never a Go map"
|
||||||
|
- "Uploads PHP stores on Laravel's local disk go to a private gocloud bucket opened per app, separate from the public uploads bucket"
|
||||||
|
- "Parity states that create rows restart their id sequences at the same floors on both sides, so uncaptured ids in bodies match"
|
||||||
|
|
||||||
|
key-files:
|
||||||
|
created:
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/contract.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/writer.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/pipe_codec.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/fgetcsv.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/parser.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/detector.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/mapper.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/canonical_id.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/errors.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/ordered.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/php.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/csv_test.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/truth_table_test.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv/testdata/ (22 input CSVs, php_fputcsv/contract/parser/detector.json, README.md)
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv_export.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/classes/csv_import_service.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/controllers/api/csv_export_controller.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/controllers/api/csv_import_controller.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go
|
||||||
|
- fonoteka.go/parity/csv_truth_tables.php
|
||||||
|
- fonoteka.go/parity/fixtures/nuxt/nuxt-csv.yaml
|
||||||
|
- fonoteka.go/parity/fixtures/nuxt/nuxt-csv.rows.json
|
||||||
|
- fonoteka.go/parity/fixtures/nuxt/files/csv-plain.csv
|
||||||
|
- fonoteka.go/parity/fixtures/nuxt/files/csv-canonical.csv
|
||||||
|
- fonoteka.go/parity/fixtures/routes/files/csv-*.csv, csv-cover.png (8 upload parts)
|
||||||
|
modified:
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/models/csv_import.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/models/csv_import_row.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/controllers/api/request.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/config/config.yaml
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/routes.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/go.mod
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/phase08_coverage_test.go
|
||||||
|
- fonoteka.go/plugins/golem15/fonoteka/phase12_security_test.go
|
||||||
|
- fonoteka.go/parity/manifest.yaml
|
||||||
|
- fonoteka.go/parity/fixtures/routes/ (8 CSV routes, 67 route fixtures)
|
||||||
|
- fonoteka.go/parity/fonoteka_reset.php
|
||||||
|
- fonoteka.go/parity/fonoteka_seed_test.go
|
||||||
|
- fonoteka.go/parity/fonoteka_flows_test.go
|
||||||
|
- fonoteka.go/parity/migrate_test.go
|
||||||
|
- fonoteka.go/parity/parity_contract_test.go
|
||||||
|
- fonoteka.go/parity/parity_test.go
|
||||||
|
- fonoteka.go/parity/README.md
|
||||||
|
|
||||||
|
key-decisions:
|
||||||
|
- "An ISO-8859-2 file is decoded as Windows-1250, as PHP does: iconv('Windows-1250', 'UTF-8//IGNORE') drops the five bytes Windows-1250 leaves undefined instead of failing, so the ISO-8859-2 branch only runs when that yields an empty string. Letters both encodings share survive; ź, ą, ś and the other differing letters do not (Łódź in ISO-8859-2 imports as ŁódĽ). The plan expected the ISO-8859-2 twin to match its UTF-8 file; the truth table records PHP's real result and Go follows it."
|
||||||
|
- "ReleaseFetcher.FetchRelease returns the mapped draft (Discogs getRelease plus DiscogsMapper::mapRelease in one seam). The Phase 13 default fails with ErrDiscogsUnavailable, so a pick answers 422 discogs_unavailable and writes nothing; only a test-installed fetcher resolves a row, and Phase 14 installs the real one."
|
||||||
|
- "Export cover URLs are absolute like Winter's File::getPath(): app.url plus the upload's public path (PHP prefixes the request origin, which equals APP_URL on the isolated instance and app.url in production)."
|
||||||
|
- "CSV model JSON columns (column_map, raw_json, candidates_json, draft_json) are Jsonable[json.RawMessage]: PHP stores a list in raw_json for plain files, and column_map is returned with the key order PHP wrote."
|
||||||
|
- "The csv parity state restarts the csv_imports, csv_import_rows and job id sequences at 80000100, 81000100 and 90000100 on both sides, so the import, row and job ids a recording creates (and the {\"csv_import_id\":N} job metadata) match without capturing every one."
|
||||||
|
|
||||||
|
patterns-established:
|
||||||
|
- "PHP byte-format ports: differential-test against PHP on random inputs while porting, then commit curated truth tables generated by a PHP script in parity/"
|
||||||
|
- "Routes whose responses carry string ids under *_id keys disable tide's id mask for those keys per step (discogs_id, selected_discogs_id)"
|
||||||
|
|
||||||
|
requirements-completed: [API-05]
|
||||||
|
|
||||||
|
coverage:
|
||||||
|
- id: D1
|
||||||
|
description: "CSV export on both groups: BOM, canonical header row, one fputcsv-quoted row per album of the active collection with the album index's filters and default order; formula cells neutralised; dated attachment filename"
|
||||||
|
requirement: API-05
|
||||||
|
verification:
|
||||||
|
- kind: unit
|
||||||
|
ref: "fonoteka.go/plugins/golem15/fonoteka/classes/csv/csv_test.go#TestPHPFputcsv"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvExport"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/parity/parity_test.go#TestParityCorpus (GET export/csv jwt, GET export/csv personal_token)"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D2
|
||||||
|
description: "CSV parser, detector, mapper and canonical row parser match PHP across encodings (UTF-8, Windows-1250, ISO-8859-2), BOM, CRLF and multi-line cells, header aliases, combined values, overrides, NUL bytes, empty and letterless files, and 5000 vs 5001 rows"
|
||||||
|
requirement: API-05
|
||||||
|
verification:
|
||||||
|
- kind: unit
|
||||||
|
ref: "fonoteka.go/plugins/golem15/fonoteka/classes/csv/truth_table_test.go#TestCsvParserTruthTable"
|
||||||
|
status: pass
|
||||||
|
- kind: unit
|
||||||
|
ref: "fonoteka.go/plugins/golem15/fonoteka/classes/csv/truth_table_test.go#TestCsvDetectorTruthTable"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D3
|
||||||
|
description: "Store and show: private bucket under fonoteka-csv/<uid>/, preview vs mapping, PHP's error envelopes (Polish for controller checks, English parse messages), 5242880 vs 5242881 bytes, per_page clamped 1..50, summary [] without rows, imports visible only to their creator while the collection is reachable"
|
||||||
|
requirement: API-05
|
||||||
|
verification:
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvStoreAndShow"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvImportScope"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/parity/parity_test.go#TestParityCorpus (POST import/csv, GET import/csv/{id})"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D4
|
||||||
|
description: "Mapping, row edit, commit and cancel: match and import jobs queued with the D-03 contract on unserved queues and left unworked (D-04), single compare-and-swap commit with replay, cancel stopping both jobs, D-05 pick seam answering discogs_unavailable without writing"
|
||||||
|
requirement: API-05
|
||||||
|
verification:
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvJobRows"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvCommitCAS"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvCancel"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/plugins/golem15/fonoteka/csv_smoke_test.go#TestCsvRowPickSeam"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/parity/parity_test.go#TestParityCorpus (mapping, rows, commit, cancel)"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D5
|
||||||
|
description: "Recorded Nuxt import wizard journey with QUEUE_CONNECTION=database; Go job rows equal PHP's golem15_apparatus_jobs rows; corpus at 151 ported; secrets check green"
|
||||||
|
requirement: API-05
|
||||||
|
verification:
|
||||||
|
- kind: e2e
|
||||||
|
ref: "fonoteka.go/parity/fonoteka_flows_test.go#TestFonotekaNuxtFlows/nuxt-csv"
|
||||||
|
status: pass
|
||||||
|
- kind: integration
|
||||||
|
ref: "fonoteka.go/parity/parity_test.go#TestParityCorpus/coverage (151 ported, 151 passing)"
|
||||||
|
status: pass
|
||||||
|
- kind: other
|
||||||
|
ref: "go run ./parity/check_corpus.go --manifest parity/manifest.yaml --routes .../routes.php --require-recorded --check-secrets"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
|
||||||
|
duration: 80min
|
||||||
|
completed: 2026-10-03
|
||||||
|
status: complete
|
||||||
|
plan_head_before: 37405f014878e249bd501ec93b1b7ffaa632772b
|
||||||
|
plan_head_after: c540102eea33d68733ee4a5366be28c035954255
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 13 Plan 04: CSV export and import Summary
|
||||||
|
|
||||||
|
**The collection backup downloads as PHP's exact CSV on both groups. The import wizard runs in Go up to a queued import job: upload in UTF-8, Windows-1250 or ISO-8859-2, preview or column mapping, row fixes, commit and cancel. The match and import jobs wait for their Phase 14 workers, and a Discogs pick never invents release data. The parser is a byte-level port of PHP's `fgetcsv` and CSV classes, checked against PHP on thousands of generated files. The corpus is at 151 ported routes.**
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
- **Duration:** 80 min
|
||||||
|
- **Started:** 2026-10-03T06:30:14Z
|
||||||
|
- **Completed:** 2026-10-03T07:50:37Z
|
||||||
|
- **Tasks:** 3 of 3
|
||||||
|
- **Files modified:** 141 in fonoteka.go: 24 source, test and script files, 22 parser inputs, 4 truth tables and a README, 8 upload parts, 2 flow parts, 67 route fixtures, the flow and its rows golden, and the manifest, seeds, README and module file
|
||||||
|
|
||||||
|
## Accomplishments
|
||||||
|
|
||||||
|
- **Export (Task 1).**
|
||||||
|
- `csv.WriteRecord` ports `fputcsv` with an empty escape. A field with a comma, quote, CR, LF, tab or space is quoted; null and false are empty, true is `1`, floats use PHP's form. It never uses the standard library's CSV writer.
|
||||||
|
- `csv.ExportRow` ports `CsvAlbumContract::exportRow`. It applies the `FORMULA_PATTERN` apostrophe guard, pipe-encodes styles, artists and covers, and tab-encodes the tracklist.
|
||||||
|
- `classes.ExportAlbums` runs the album index's SQL path and filters in the index's default order, in batches of 100.
|
||||||
|
- `CsvExport` writes the BOM, the headers and `Content-Disposition: attachment; filename=plytarium-kolekcja-<Y-m-d>.csv`. The route is mounted on the JWT group and on the token group (`inv.scope:read`).
|
||||||
|
- **Parser (Task 2).**
|
||||||
|
- `fgetcsv` and `str_getcsv` are ported from `php_fgetcsv`, including its reads past the line end. Over 16,088 random record reads, 0 differ from PHP.
|
||||||
|
- `csv.Parse` ports `CsvAlbumParser`. It decodes as `iconv //IGNORE` does, runs the `\p{L}` check and strips the BOM. It resolves the delimiter with `strtok`, detects or overrides the map, backfills headers, and enforces the 5000-row cap and NUL rejection.
|
||||||
|
- The detector, mapper and canonical row parser are ported too. Over 1,507 random files (key order included), 0 differ from PHP.
|
||||||
|
- The committed truth tables come from `parity/csv_truth_tables.php`.
|
||||||
|
- **Import session (Tasks 2 and 3).**
|
||||||
|
- **Store** checks the extension and size, then parses. Invalid canonical rows are `validation_failed`. The file goes to the private bucket. The import is created in preview or mapping, with its rows; canonical ids are matched only inside the importer's collection.
|
||||||
|
- **Show** paginates rows by `row_index`. `summary` is the count per status, or `[]` without rows. `progress` comes from the row counts, or from the `summer_jobs` row.
|
||||||
|
- **Mapping** keeps the client's `column_map` key order and re-parses the stored file. It cancels the previous match job, replaces the rows, and queues `CsvMatchArgs` on the import's transaction.
|
||||||
|
- **Row edit** handles skip, accept_csv and the candidate allow-list pick through the `ReleaseFetcher` seam.
|
||||||
|
- **Commit** is a single `UPDATE ... WHERE status = 'preview'` that queues one `CsvImportArgs` job. A replay answers the current job.
|
||||||
|
- **Cancel** stops both jobs, the `summer_jobs` rows and the River jobs.
|
||||||
|
- **Parity.**
|
||||||
|
- The `csv` state exists on both sides: albums with a cover, seven imports in every needed status, their rows and job rows, and aligned id sequences.
|
||||||
|
- 67 route fixtures cover the 8 routes. The 5 Task 3 routes were recorded with `QUEUE_CONNECTION=database`.
|
||||||
|
- The `nuxt-csv` flow (17 steps) and its job-row golden replay green.
|
||||||
|
|
||||||
|
## Task Commits
|
||||||
|
|
||||||
|
fonoteka.go (master, not pushed):
|
||||||
|
1. **Task 1: CSV export with the fputcsv port** - `643c9dd` (feat)
|
||||||
|
2. **Task 2: parser package, private bucket, store and show** - `8af008e` (feat)
|
||||||
|
3. **Task 3: mapping, rows, commit, cancel, token export, nuxt-csv flow** - `c540102` (feat)
|
||||||
|
|
||||||
|
## Decisions Made
|
||||||
|
|
||||||
|
See `key-decisions`. The user may want to look at the first one. PHP decodes an ISO-8859-2 upload as Windows-1250, so Polish letters that differ between the two encodings come out wrong (Łódź imports as ŁódĽ). The Go port keeps that behaviour, because the PHP truth table is the contract. Fixing it would be a deliberate change in both implementations.
|
||||||
|
|
||||||
|
## Deviations from Plan
|
||||||
|
|
||||||
|
### Auto-fixed Issues
|
||||||
|
|
||||||
|
**1. [Rule 1 - Plan premise] ISO-8859-2 files do not match their UTF-8 twin in PHP**
|
||||||
|
- **Found during:** Task 2 (truth-table generation)
|
||||||
|
- **Issue:** The plan's encoding edge expected a Windows-1250 file and an ISO-8859-2 file with `Łódź` to import with the UTF-8 values. PHP's `iconv //IGNORE` never fails on Windows-1250, so ISO-8859-2 bytes are read as Windows-1250.
|
||||||
|
- **Fix:** The port follows PHP. The truth table pins all three files (`lodz_*.csv`), plus an ISO-8859-2 file that uses only shared letters and does match its twin (`shared_*.csv`).
|
||||||
|
- **Commit:** 8af008e
|
||||||
|
|
||||||
|
**2. [Rule 3 - Blocking] The CSV model JSON columns could not hold PHP's data**
|
||||||
|
- **Issue:** `raw_json` holds a list for non-canonical files, and `column_map` must keep its key order.
|
||||||
|
- **Fix:** `ColumnMap`, `RawJSON`, `CandidatesJSON` and `DraftJSON` are now `lagoon.Jsonable[json.RawMessage]`. No other code used them.
|
||||||
|
- **Commit:** 8af008e
|
||||||
|
|
||||||
|
**3. [Rule 1 - Parity] tide masks `*_id` keys as integer ids**
|
||||||
|
- **Issue:** Discogs ids are strings, so the mask failed `discogs_id` and `selected_discogs_id`.
|
||||||
|
- **Fix:** CSV steps disable the mask for those two keys with step `normalize` rules.
|
||||||
|
- **Commit:** 8af008e
|
||||||
|
|
||||||
|
**4. [Recording] The 5 MiB size case is not recorded**
|
||||||
|
- **Issue:** The isolated PHP runs with php.ini's default `upload_max_filesize` of 2M, which rejects such a file before the controller does. The Go parity replay also caps uploads at 1 MiB.
|
||||||
|
- **Coverage instead:** `TestCsvStoreAndShow` checks 5242880 bytes (accepted) and 5242881 bytes (csv_too_large). The 5001-row case is recorded.
|
||||||
|
|
||||||
|
**5. [Structure] API names differ from the plan's list**
|
||||||
|
- `ExportAlbums(ctx, db, AlbumSearchParams, coverBase, fn func([]string) error)` lives in package `classes`. It reuses `searchSQL`, `orderSearch` and `resolveSearchSort` directly, so `album_search.go` is unchanged.
|
||||||
|
- `MatchCanonicalID` is split in two:
|
||||||
|
- `csv.CanonicalID` is the pure check;
|
||||||
|
- `classes.matchCanonicalAlbum` is the scoped lookup, since the csv package has no database.
|
||||||
|
- Other API changes:
|
||||||
|
- `CommitCsvImport` takes the mode and whether it was sent.
|
||||||
|
- `ShowCsvImport` was added.
|
||||||
|
- `CsvJobs` covers Dispatch and CancelJob.
|
||||||
|
- Route upload parts are in `fixtures/routes/files/`, because tide resolves part files relative to the fixture. The flow parts are in `fixtures/nuxt/files/`.
|
||||||
|
|
||||||
|
**6. [Rule 3 - Test inventories] Two guard tests named the old state**
|
||||||
|
- The Phase 8 coverage subtest for CSV now asserts the real surfaces: export on both groups, the six import routes on JWT only.
|
||||||
|
- The T-12-17 part-file check accepts small `csv-*` uploads that contain no secret shape.
|
||||||
|
- **Commits:** 643c9dd, 8af008e, c540102
|
||||||
|
|
||||||
|
**7. [Tooling] `go work sync` side effects reverted**
|
||||||
|
- Promoting `golang.org/x/text` changed only the plugin `go.mod`; its hash was already in `go.sum`.
|
||||||
|
- `go work sync` also reordered `go.work` and dropped the root toolchain line. Both were reverted, so the root `go.mod`, `go.sum` and `go.work.sum` are unchanged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Total deviations:** 7: 1 plan premise corrected by the PHP truth, 2 blocking model or harness issues, 1 recording limitation covered by Go tests, 1 structural API naming change, 1 test-inventory update and 1 tooling revert. **Impact:** every recorded body, header and job row matches PHP.
|
||||||
|
|
||||||
|
## Issues Encountered
|
||||||
|
|
||||||
|
- `TestPhase09SecurityRoutes` (12.2 cabana routes) still fails in the fonoteka plugin package; it is pre-existing and logged in deferred-items.md. Every other suite is green: the root module, parity (`TestParityCorpus` 151/151, `TestFonotekaNuxtFlows` including nuxt-csv), and the plugin's subpackages with `-race`.
|
||||||
|
- The shell aliases `rm` to `rm -i`, which stalled two commands; later commands used `command rm` or `-f`.
|
||||||
|
- The isolated PHP server was started for recording (sync, then `QUEUE_CONNECTION=database`) and stopped before returning. Only the CSV uploads written today were removed from the PHP checkout's `storage/app/fonoteka-csv/`; an older file there was left in place.
|
||||||
|
|
||||||
|
## Known Stubs
|
||||||
|
|
||||||
|
None. The CSV import and match jobs have no worker by design (D-04, Phase 14 JOBS-02). The Discogs pick seam fails by design until Phase 14 (D-05, INTG-01). Neither is a stub.
|
||||||
|
|
||||||
|
## Threat Flags
|
||||||
|
|
||||||
|
None beyond the plan's register:
|
||||||
|
- **T-13-12:** `CsvImportFor` (creator plus reachable collection) and `TestCsvImportScope`.
|
||||||
|
- **T-13-13:** the private bucket, server-side keys, and the public-bucket absence check in `TestCsvStoreAndShow`.
|
||||||
|
- **T-13-14:** `FormulaSafe` on every exported cell.
|
||||||
|
- **T-13-15:** the 5 MiB cap, 5000 rows, NUL rejection and `throttle:10,1`.
|
||||||
|
- **T-13-16:** the candidate allow-list and the failing fetcher.
|
||||||
|
- **T-13-17:** the single compare-and-swap, checked by `TestCsvCommitCAS`.
|
||||||
|
- **T-13-31:** unserved queues and cancel stopping River jobs.
|
||||||
|
- **T-13-SC:** x/text v0.42.0 pinned by `go.sum`.
|
||||||
|
|
||||||
|
## User Setup Required
|
||||||
|
|
||||||
|
None. In production, `golem15.fonoteka.csv.bucket_url` defaults to `file://./storage/app` (Laravel's local disk, never served). Point it at another private location if the binary's working directory differs.
|
||||||
|
|
||||||
|
## Next Phase Readiness
|
||||||
|
|
||||||
|
- 13-05 can rely on the token-group route table: export/csv is now mounted with `inv.scope:read`.
|
||||||
|
- 13-06 should add the CSV write routes (store, mapping, rows, commit, cancel) to `write_endpoints_fuzz_test.go` and the 8 routes to the Phase 13 route-table test.
|
||||||
|
- Phase 14:
|
||||||
|
- It registers workers for `golem15.fonoteka.csv_import` and `golem15.fonoteka.csv_match` on `fonoteka_csv_import` and `fonoteka_csv_match`. The rows carry `{"csv_import_id":N}`.
|
||||||
|
- It installs a real `ReleaseFetcher` with `SetReleaseFetcher`, which ports DiscogsClient::getRelease and DiscogsMapper::mapRelease.
|
||||||
|
- It records the successful-pick case.
|
||||||
|
|
||||||
|
## Self-Check: PASSED
|
||||||
|
|
||||||
|
- Created files exist: every `key-files.created` path checked with `test -e`.
|
||||||
|
- Commits exist in fonoteka.go: 643c9dd, 8af008e, c540102.
|
||||||
|
- Plan verification:
|
||||||
|
- fonoteka.go `go vet ./...` is clean (root and plugin module).
|
||||||
|
- `go test ./...` is green for the root module and the plugin subpackages; the plugin package fails only the pre-existing `TestPhase09SecurityRoutes`.
|
||||||
|
- `TestParityCorpus` has 151 routes ported and passing.
|
||||||
|
- `TestFonotekaNuxtFlows/nuxt-csv` passes.
|
||||||
|
- `check_corpus --require-recorded --check-secrets` is green.
|
||||||
|
- Acceptance checks:
|
||||||
|
- `encoding/csv` appears 0 times in writer.go.
|
||||||
|
- `plytarium-kolekcja-` appears once in the export controller.
|
||||||
|
- x/text is a direct requirement of the plugin module, and parser.go imports `charmap`.
|
||||||
|
- The plugin `config.yaml` has `csv.bucket_url` and no `uploads/public`.
|
||||||
|
- testdata has Windows-1250 and ISO-8859-2 samples.
|
||||||
|
- `jobs.go` names no CSV kind.
|
||||||
|
- `nuxt-csv.rows.json` holds `fonoteka.csv.import`.
|
||||||
|
- The service names `CsvImportLabel`, `CsvMatchLabel` and `discogs_unavailable`.
|
||||||
Reference in New Issue
Block a user