diff --git a/.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-06-SUMMARY.md b/.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-06-SUMMARY.md new file mode 100644 index 0000000..4c6813c --- /dev/null +++ b/.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-06-SUMMARY.md @@ -0,0 +1,326 @@ +--- +phase: 13-p-ytarium-api-wishlist-notifications-csv-credentials-public +plan: 06 +subsystem: testing +tags: [route-table, fuzz, mass-assignment, threats, removal-harness, coverage, gate, security-review, validation, sm-user-plugin] + +requires: + - phase: 13-01 + provides: surf overlap families, conga workerless kinds, lagoon prohibited, tide masks, job contract + - phase: 13-02 + provides: notifications, credentials, onboarding, inspection, register hook and the user plugin's registration exports + - phase: 13-03 + provides: the 30 wishlist routes, reservations, subscriptions, digest and purchase side effects + - phase: 13-04 + provides: the CSV export and import session with its jobs and private bucket + - phase: 13-05 + provides: the six public share routes, PubfailCounter and the D-14 layout + - phase: 12 + provides: phase12RouteTable, FuzzWriteEndpoints, check-phase12.sh and the 12-SECURITY-REVIEW.md format +provides: + - "TestRouteTablePhase13: the 58 Phase 13 routes pinned to routes.php groups, scopes, constraints, throttles and buckets; Phase 14 routes absent; overlap 404/405" + - "FuzzWriteEndpoints over 70 write routes (41 Phase 12 + 29 Phase 13) with the Phase 13 server-owned keys and 70 committed synthetic seeds" + - "TestPhase13Threats: 27 subtests, one per mitigated application threat" + - "TestPhase13EmptyBodies, TestPhase13Boundaries, TestPhase13HandlersFailClosed and package unit tests in both repos and sm-user-plugin" + - "scripts/check-phase13.sh: --self-test, --go, --parity, --named, --removal (31 checks), --coverage, --evidence, --all" + - "13-SECURITY-REVIEW.md, validated 13-VALIDATION.md, API-03..API-07 complete" +affects: [13 verification, 14, 15] + +actuals: + tokens: 80500 + tasks: 3 + commits: 13 + +tech-stack: + added: [] + patterns: + - "Route-level tests that queue River jobs run on a dedicated database, so a later test's worker on the shared database never drains them" + - "A test process that boots several apps boots the one whose writes dispatch jobs last (the job dispatcher is process-wide)" + - "Removal rows may mutate a framework file and run the application's test (repo kind rootapp)" + - "The gate reads the parity replay's own coverage line instead of counting manifest lines alone" + +key-files: + created: + - fonoteka.go/plugins/golem15/fonoteka/routes_table_phase13_test.go + - fonoteka.go/plugins/golem15/fonoteka/phase13_security_test.go + - fonoteka.go/plugins/golem15/fonoteka/phase13_controllers_test.go + - fonoteka.go/plugins/golem15/fonoteka/classes/phase13_classes_test.go + - fonoteka.go/plugins/golem15/fonoteka/classes/csv/csv_edges_test.go + - fonoteka.go/plugins/golem15/fonoteka/controllers/api/phase13_controllers_test.go + - fonoteka.go/plugins/golem15/fonoteka/testdata/fuzz/FuzzWriteEndpoints/ (29 Phase 13 seeds) + - fonoteka.go/plugins/golem15/user/registration_test.go + - summercms.go/modules/surf/overlap_edges_test.go + - summercms.go/scripts/check-phase13.sh + - summercms.go/.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-SECURITY-REVIEW.md + modified: + - fonoteka.go/plugins/golem15/fonoteka/write_endpoints_fuzz_test.go + - fonoteka.go/plugins/golem15/fonoteka/handler_failures_test.go + - fonoteka.go/plugins/golem15/fonoteka/admin_phase09_security_test.go + - fonoteka.go/plugins/golem15/fonoteka/middleware/public_share_headers_test.go + - fonoteka.go/plugins/golem15/fonoteka/controllers/api/csv_export_controller.go + - fonoteka.go/plugins/golem15/user (submodule pointer) + - summercms.go/modules/conga/unregistered_kind_test.go + - summercms.go/modules/lagoon/validate_request_test.go + - summercms.go/modules/tide/normalize_phase13_test.go + - summercms.go/.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/13-VALIDATION.md + - summercms.go/.planning/phases/13-p-ytarium-api-wishlist-notifications-csv-credentials-public/deferred-items.md + - summercms.go/.planning/REQUIREMENTS.md + +key-decisions: + - "T-13-SC is reviewed at its strictest declaration (13-04: medium, mitigate) and the gate pins golang.org/x/text at v0.42.0; the evidence stage merges repeated threat ids to the strictest severity and disposition." + - "The surf constraint skip cannot be observed through the routes.php pairs (each member's handler re-checks its Where constraints and the pairs differ in literals), so RC-01 relies on the new TestOverlapConstraintFallsThrough and the application checks mutate the literal match instead." + - "TestPhase13EmptyBodies and TestPhase13Boundaries live in the plugin package, not controllers/api, because they need the assembled router and the Postgres harness; controllers/api gets a unit test of mappingInput." + - "The user plugin's controllers package floor is its pre-phase value, 68.8%, measured at sm-user-plugin c258e9f; Phase 13 raised it to 70.6%." + - "The CSV export now logs a failure after its headers are sent (Laravel reports exceptions thrown inside a streamed response); the status still cannot change, as in PHP." + +patterns-established: + - "One subtest per threat id in TestPhase13Threats, each named T-13-NN, with the protecting function named in its comment" + - "Fuzz specs for a phase keyed by the full route key so a phase's write surface is greppable" + +requirements-completed: [API-03, API-04, API-05, API-06, API-07] + +coverage: + - id: D1 + description: "The Phase 13 route surface is pinned to routes.php through the assembled router: 58 routes on their groups with routes.php lines, one inv.scope per token route, inline throttles and public buckets exactly where PHP puts them, numeric parameters refused at the router, {token} unconstrained, the four Phase 14 routes absent, overlap pairs dispatching with 404/405" + requirement: API-07 + verification: + - kind: unit + ref: "fonoteka.go/plugins/golem15/fonoteka/routes_table_phase13_test.go#TestRouteTablePhase13" + status: pass + - kind: other + ref: "scripts/check-phase13.sh --removal (RC-03, RC-22)" + status: pass + human_judgment: false + - id: D2 + description: "No Phase 13 write endpoint persists a server-owned key: the fuzz enumerates all 29 Phase 13 write routes beside the Phase 12 ones, sends every server-owned key with a hostile value and compares Postgres snapshots; 60 s of fuzzing found nothing" + requirement: API-06 + verification: + - kind: integration + ref: "fonoteka.go/plugins/golem15/fonoteka/write_endpoints_fuzz_test.go#FuzzWriteEndpoints" + status: pass + - kind: other + ref: "go -C ../fonoteka.go test ./plugins/golem15/fonoteka -run '^$' -fuzz '^FuzzWriteEndpoints$' -fuzztime 60s" + status: pass + human_judgment: false + - id: D3 + description: "Every mitigated Phase 13 threat has a named test that fails with its protection removed: 27 TestPhase13Threats subtests plus framework tests, 31 anchor-exact removal checks all failing as required with byte-identical restore" + requirement: API-03 + verification: + - kind: integration + ref: "fonoteka.go/plugins/golem15/fonoteka/phase13_security_test.go#TestPhase13Threats" + status: pass + - kind: other + ref: "scripts/check-phase13.sh --removal" + status: pass + human_judgment: false + - id: D4 + description: "Phase 13 code is unit tested to the 80% floor in both repos and the user plugin (surf 85.8, conga 92.8, lagoon 85.1, tide 81.7, classes 85.8, classes/csv 94.5, controllers/api 81.9, middleware 100, user classes 88.7, registration.go functions at least 87.5), with empty bodies, boundaries and fail-closed handlers covered" + requirement: API-05 + verification: + - kind: unit + ref: "scripts/check-phase13.sh --coverage" + status: pass + - kind: integration + ref: "fonoteka.go/plugins/golem15/fonoteka/phase13_controllers_test.go#TestPhase13EmptyBodies" + status: pass + - kind: integration + ref: "fonoteka.go/plugins/golem15/fonoteka/phase13_controllers_test.go#TestPhase13Boundaries" + status: pass + - kind: integration + ref: "fonoteka.go/plugins/golem15/fonoteka/phase13_controllers_test.go#TestPhase13HandlersFailClosed" + status: pass + - kind: unit + ref: "fonoteka.go/plugins/golem15/user/registration_test.go#TestRegisterUserExports" + status: pass + human_judgment: false + - id: D5 + description: "The fail-closed gate passes: vet and tests in both repos, 157 ported routes passing and 14 not ported read from the replay's coverage line, all flows and goldens, docs checks, named tests, coverage floors and evidence; the self-test proves each detector fails closed" + requirement: API-04 + verification: + - kind: other + ref: "scripts/check-phase13.sh --self-test && scripts/check-phase13.sh --all" + status: pass + human_judgment: false + - id: D6 + description: "13-SECURITY-REVIEW.md maps every T-13 id to category, severity, disposition, protecting code, named test and removal result; 13-VALIDATION.md is validated; API-03..API-07 complete" + requirement: API-07 + verification: + - kind: other + ref: "scripts/check-phase13.sh --evidence" + status: pass + human_judgment: true + rationale: "The review was self-performed by the executor (08-10 precedent); a human should read its accepted risks (per-process pubfail counter, base_url stored for Phase 14) before shipping." + +duration: 70min +completed: 2026-10-03 +status: complete +plan_head_before: 5d0ce7f5bba89bae5737c00b6f9611d64a25a81d +plan_head_after: 931c02db311254f548e5b17c95b668b9f3c1c53e +fonoteka_plan_head_before: 3c8b7cd25ac996921a745fa7e98e3a81414c7072 +fonoteka_plan_head_after: 40c575e72703c7e7fb4684e1477ce73496586e54 +user_plugin_head_before: d80d7990f6f8b743c4df2f0fa9538d9e2acf780e +user_plugin_head_after: 8a658903edb1b4957cde073fe82a475b978384bb +--- + +# Phase 13 Plan 06: Unit tests, fuzz, threats, gate and sign-off Summary + +**Phase 13 is now closed by evidence:** +- **Route table:** a test pins all 58 routes to routes.php through the real router. +- **Fuzz:** the request fuzz covers all 70 write routes with every server-owned key. +- **Threats:** 27 subtests pin each application threat, and 31 removal checks prove each protection is load-bearing. +- **Coverage:** every Phase 13 package in both repos and the user plugin is at or above the floor. +- **Gate:** `scripts/check-phase13.sh` fails closed, and its `--all` and `--removal` modes both pass. +- **Fixes:** the review found one silent failure in the CSV export (now logged) and one stale route inventory (now updated). + +## Performance + +- **Duration:** 70 min +- **Started:** 2026-10-03T08:32:57Z +- **Completed:** 2026-10-03T09:43:22Z +- **Tasks:** 3 of 3 +- **Files modified:** 59. In fonoteka.go: 10 test and source files, 29 fuzz seeds and the submodule pointer. In the user plugin: 1 test file. In summercms.go: 5 test and script files and 5 planning docs (this summary included), plus the STATE and ROADMAP updates. + +## Accomplishments + +- **Task 1: route table.** `TestRouteTablePhase13` boots the assembled router and checks the following against routes.php: + - **groups:** 58 routes, each on its group (JWT, personal token, onboarding, invitation inspection, public share), with its routes.php line; + - **scopes:** exactly one `inv.scope` per token route; + - **throttles:** the inline throttles on share regenerate, CSV store, onboarding, inspection and the two resolves, and the public buckets on the four albums routes; + - **constraints:** 24 numeric parameters refuse `abc` and `1x` at the router and reach the handler with `12`, and 10 `{token}` parameters are unconstrained; + - **absent routes:** the four Phase 14 routes answer the router's 404; + - **overlap pairs:** they dispatch to the real handlers, with the 404 and the sorted 405 `Allow`. +- **Task 2: fuzz and threats.** + - **Fuzz:** `FuzzWriteEndpoints` gained 29 Phase 13 specs keyed by full route key, a Phase 13 world (shared wishlist, reservation, subscriptions, notification, credentials, CSV imports in mapping and preview), snapshots of nine more tables and the Phase 13 server-owned keys. The onboarding bootstrap fuzzes on its own database with every user soft-deleted. Each request gets its own client address. 29 synthetic seed files were committed. A 60 s fuzzing session found nothing. + - **Threats:** `TestPhase13Threats` covers T-13-01..10, 12..21, 23, 27..31 and 33. For example, T-13-18 holds the advisory lock, lets a bootstrap wait on it, inserts a user and requires 409 with nothing written. +- **Task 3: coverage, gate and sign-off.** + - **summercms.go tests:** surf edges (a transitive family, HEAD and Allow, a family across plugins, refused shapes, constraint fall-through), conga refusal messages and delayed workerless dispatch, lagoon `prohibited` under wildcards, dotted paths and bail, and tide quoted, RFC 5987 and multi-date names. + - **fonoteka tests:** + - `TestPhase13EmptyBodies` covers 29 routes × 2 body kinds, each with PHP's status; + - `TestPhase13Boundaries` covers CSV bytes and rows, `per_page`, Discogs token length, the bell cap and pubfail; + - `TestPhase13HandlersFailClosed` renames 13 tables in turn; + - the classes, csv, api and middleware unit tests; + - `TestRegisterUserExports` in sm-user-plugin. + - **Gate:** `scripts/check-phase13.sh` follows check-phase12's structure and adds the corpus coverage-line parser, a function-floor check for `registration.go`, the `rootapp` removal kind and a signal-safe restore. It also unsets `FORCE_COLOR` itself. + - **Docs:** 13-SECURITY-REVIEW.md and 13-VALIDATION.md are written, and API-03..API-07 are complete. + +## Task Commits + +fonoteka.go (master, not pushed): +1. **Pre-task: Phase 9 admin route inventory** - `549840d` (test) +2. **Task 1: route table** - `23cc540` (test) +3. **Task 2: fuzz and threats** - `4190a4a` (test) +4. **Task 3: CSV export logging fix** - `4dec779` (fix) +5. **Task 3: application coverage tests** - `f25e09d` (test) +6. **Task 3: sm-user-plugin pointer bump** - `e121295` (chore) +7. **Task 3: T-13-01 lockout through the albums routes** - `40c575e` (test) + +sm-user-plugin (master, not pushed): +1. **Task 3: registration export tests** - `8a65890` (test) + +summercms.go (master): +1. **Task 3: framework edge tests** - `4ed45c1` (test) +2. **Task 3: constraint fall-through test** - `22a5ebd` (test) +3. **Task 3: the gate** - `12b961b` (test) +4. **Task 3: the gate's named surf test** - `ad79b3c` (test) +5. **Task 3: security review, validation, requirements** - `931c02d` (docs) + +## Decisions Made + +See `key-decisions` above. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] The CSV export dropped a failure after its headers were sent** +- **Found during:** Task 3 (TestPhase13HandlersFailClosed) +- **Issue:** With the albums table unreachable, `GET export/csv` answered 200 with the BOM and the header row only, and logged nothing. Laravel reports an exception thrown inside a streamed response. +- **Fix:** `CsvExport` logs the error with the user id. The status cannot change after the headers, as in PHP. +- **Files modified:** controllers/api/csv_export_controller.go +- **Commit:** 4dec779 + +**2. [Rule 3 - Blocking] TestPhase09SecurityRoutes failed on an outdated route inventory** +- **Found during:** pre-task (the `--all` gate runs the plugin package) +- **Issue:** The 19 cabana relation child, pivot and file routes added by Phase 12.2 were missing from the expected set. All 19 carry the backend guard, so this is not a security gap. +- **Fix:** The expected set lists them. +- **Commit:** 549840d + +**3. [Rule 1 - Test] The first T-13-01 subtest passed with the pubfail check removed** +- **Found during:** Task 3 (`--removal`, RC-08 survived) +- **Issue:** The resolve route's inline `throttle:10,1` answered the eleventh request with 429 on its own. +- **Fix:** The failures now go through the albums routes, whose buckets are wide. +- **Commit:** 40c575e + +**4. [Rule 1 - Test] The surf constraint-skip mutation was not observable** +- **Found during:** Task 3 (`--removal`, RC-01 survived) +- **Issue:** Each member's handler is wrapped with its own constraints, and the routes.php pairs differ in literals. +- **Fix:** `TestOverlapConstraintFallsThrough` covers a family where an earlier member matches the literals but not its constraint. The application checks mutate the literal match through the new `rootapp` removal kind. +- **Commits:** 22a5ebd, 12b961b, ad79b3c + +**5. [Rule 3 - Test isolation] Queued River jobs broke TestAlbumBroadcastSmoke** +- **Found during:** Task 3 (full plugin run) +- **Issue:** The threat and empty-body worlds left hundreds of publication jobs on the shared database, and the broadcast smoke test's worker drained them. +- **Fix:** Both tests run on databases of their own (`dedicatedHarness`, which now takes `testing.TB`). +- **Commit:** f25e09d + +**6. [Rule 3 - Test harness] The process-wide job dispatcher** +- **Found during:** Task 2 +- **Issue:** `phase12RouteTable` boots an app without a database. When it booted last, every digest dispatch failed with "conga: database is not published". +- **Fix:** FuzzWriteEndpoints and TestPhase13Threats boot the app whose writes dispatch jobs last. The underlying design is logged in deferred-items.md. +- **Commit:** 4190a4a + +**7. [Structure] Plan wording versus the code** +- **Test locations:** + - `TestPhase13EmptyBodies` and `TestPhase13Boundaries` are in `plugins/golem15/fonoteka/phase13_controllers_test.go`, because they need the plugin harness; + - `controllers/api/phase13_controllers_test.go` holds `TestCsvMappingInput`. +- **Pubfail removal:** the handlers call `PubfailCounter.Begin`, so "TooMany always false" is two removal rows: + - RC-08 removes the check in `Begin`; + - RC-09 removes the check in `TooMany`, which `TestPubfailCounter` pins. +- **Extra removal rows:** they cover every high threat (T-13-04, 06, 09, 13, 19) and the gate's own threats (T-13-34, 35, 36, SC). + +--- + +**Total deviations:** 7: 1 production bug, 1 stale test inventory, 2 tests strengthened after a survived removal, 2 harness or isolation fixes, 1 structural note. **Impact:** no response contract changed; the only production change adds a log line. + +## Issues Encountered + +- `/tmp` on this host is a nearly full tmpfs: `go test ./...` failed to link with "disk quota exceeded". The suites and the gate ran with `TMPDIR`/`GOTMPDIR` on the home disk (logged in deferred-items.md). +- `FORCE_COLOR` was unset for every run; the gate now unsets it itself. +- No PHP server was started in this plan. + +## Known Stubs + +None. + +## Threat Flags + +None. This plan added no endpoint, auth path or schema. T-13-34, T-13-35 and T-13-36 are mitigated and removal-proven (RC-29 to RC-31). T-13-SC holds: no dependency was added. + +## User Setup Required + +None. To run the gate on a host with a small `/tmp`, set `TMPDIR` and `GOTMPDIR` to a larger disk. + +## Next Phase Readiness + +- Phase 13 is ready for `/gsd-verify-work 13`: `scripts/check-phase13.sh --self-test && scripts/check-phase13.sh --all` and `--removal` are green. +- Phase 14 inherits the following from this plan: + - the absent-route assertions in `TestRouteTablePhase13` (`phase14Absent`), which must be moved into a Phase 14 table when those routes are mounted; + - `EXPECTED_PORTED`/`EXPECTED_PENDING` in the gate; + - the deferred items: the process-wide job dispatcher, the int4 `pathID` gap on the older routes, and `cleanSession`. +- The sm-user-plugin commit `8a65890` is local to the submodule; push it with `ssu` when the user asks. + +## Self-Check: PASSED + +- Created files exist: routes_table_phase13_test.go, phase13_security_test.go, phase13_controllers_test.go, classes/phase13_classes_test.go, classes/csv/csv_edges_test.go, controllers/api/phase13_controllers_test.go, user/registration_test.go, surf/overlap_edges_test.go, scripts/check-phase13.sh, 13-SECURITY-REVIEW.md; 29 new seed files (70 in total). +- Commits exist: fonoteka.go 549840d, 23cc540, 4190a4a, 4dec779, f25e09d, e121295, 40c575e; sm-user-plugin 8a65890; summercms.go 4ed45c1, 22a5ebd, 12b961b, ad79b3c, 931c02d. +- Plan verification: + - `scripts/check-phase13.sh --self-test`, `--all` and `--removal` exited 0. + - Both repos' `go vet` and `go test` are green inside the gate's `--go` stage. + - TestDocsTree and `docs:build --check` are green. +- Acceptance greps: + - TestRouteTablePhase13 has 58 table entries, and the absent list has 6 matches. + - There are 27 `t.Run("T-13-` subtests. + - The fuzz file has 29 Phase 13 write keys, and there are 70 seed files (41 + 29). + - `EXPECTED_PORTED=157` appears once, `nyquist_compliant: true` once and `⬜ pending` 0 times. + - Every T-13 id of the plans appears in the review. + - 5 `API-0[3-7] | Phase 13 | Complete` rows.