diff --git a/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/19.3-03-SUMMARY.md b/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/19.3-03-SUMMARY.md new file mode 100644 index 0000000..b2c7da9 --- /dev/null +++ b/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/19.3-03-SUMMARY.md @@ -0,0 +1,226 @@ +--- +phase: 19.3-framework-transport-and-infrastructure-for-wavepath +plan: 03 +subsystem: storage +tags: [lagoon, attach, s3, gocloud, build-tags, summer-build, stored-xss, cdn] + +requires: + - phase: 19.3-01 + provides: "decision note .planning/notes/2026-10-15-cooler-go-redis-otter.md naming gocloud.dev/blob/s3blob and its AWS SDK v2 modules" +provides: + - "summer.yaml build.tags (Manifest.Tags, ValidateBuildTag, goBuildArgs, build tags output line)" + - "attach.S3BuildTag (summercms_s3): s3blob linked only by s3_on.go; an untagged binary refuses s3:// at boot naming the tag" + - "attach.DiskExtension: stored extension follows the sniffed content (stored-XSS fix)" + - "storage.uploads.public_url and attach.PublicURLBase for direct bucket or CDN URLs" +affects: [19.3-05, fonoteka-upload-call-sites-follow-up, apps-on-s3] + +actuals: + tokens: 15400 + tasks: 3 + commits: 3 +plan_head_before: b1fab9930b11090d7a5515f05a3842e15be06809 +plan_head_after: 2e87efae54141a12a197f1e8de844a1996ffba32 + +tech-stack: + added: ["gocloud.dev/blob/s3blob (sub-package of gocloud.dev v0.46.0, summercms_s3 tag only)", "github.com/aws/aws-sdk-go-v2 v1.41.9 and 17 sibling modules (indirect)", "github.com/aws/smithy-go v1.26.0 (indirect)", "github.com/google/wire v0.7.0 (indirect)"] + patterns: + - "Feature build tag split: s3_on.go (//go:build summercms_s3) blank-imports the driver and sets a const; s3_off.go sets it false; the boot guard names the tag" + - "Manifest keys must be written back by SaveManifest so summer plugin:add round-trips them" + - "Extension derived from content through a fixed in-code table, never package mime" + +key-files: + created: + - internal/build/manifest_tags_smoke_test.go + - modules/lagoon/attach/s3_on.go + - modules/lagoon/attach/s3_off.go + - modules/lagoon/attach/diskext.go + - modules/lagoon/attach/infra_smoke_test.go + modified: + - go.mod + - go.sum + - internal/build/manifest.go + - internal/build/build.go + - modules/lagoon/attach/bucket.go + - modules/lagoon/attach/store.go + - modules/lagoon/attach/store_test.go + - modules/lagoon/attach/thumb.go + - modules/lagoon/README.md + - docs/services/storage.md + - docs/database/attachments.md + - docs/architecture/go-modules-and-workspaces.md + - docs/architecture/performance-and-scaling.md + - docs/console/scaffolding.md + - docs/setup/configuration.md + +key-decisions: + - "Build tag summercms_s3 (attach.S3BuildTag), declared in summer.yaml as build.tags; tags match ^[a-z0-9_]+$, duplicates dropped, passed to go build as one -tags= exec argument; summer build prints a build tags line" + - "DiskExtension: specific sniffed types keep a matching client extension or use the canonical one; generic sniffs keep the client extension unless script-capable (txt for text/*, bin otherwise); Store passes the sniffed type before its octet-stream extension fallback" + - "storage.uploads.public_url applies to every URL attach builds, private files included; the start-up info line and the docs carry the public-files-only rule" + - "public_url is validated before the bucket opens (absolute http/https, host, no user info, query or fragment); errors name the key and SUMMER_STORAGE__UPLOADS__PUBLIC_URL without echoing the value" + +patterns-established: + - "Tag-mode smoke tests branch on the unexported const (s3Enabled) so one test passes in both tag modes" + +requirements-completed: [SC-3, SC-4, D-12, D-13, D-14, D-15, D-19] + +coverage: + - id: D1 + description: "summer.yaml build.tags loads (dedup, validation), round-trips through SaveManifest, and reaches go build as -tags before -o" + requirement: D-13 + verification: + - kind: unit + ref: "internal/build/manifest_tags_smoke_test.go#TestManifestBuildTagsSmoke" + status: pass + human_judgment: false + - id: D2 + description: "Untagged binary refuses s3:// naming summercms_s3, build.tags and -tags; tagged binary opens s3://smoke-bucket without network" + requirement: D-13 + verification: + - kind: unit + ref: "modules/lagoon/attach/infra_smoke_test.go#TestOpenBucketS3Smoke (go test -short, and go test -short -tags summercms_s3)" + status: pass + human_judgment: false + - id: D3 + description: "Real upload to an S3-compatible bucket (R2/MinIO/Hetzner) through the tagged binary" + requirement: SC-3 + human_judgment: true + rationale: "No real S3 or MinIO available; manual UAT item in 19.3-VALIDATION.md" + - id: D4 + description: "PNG uploaded as x.html is stored and served as .png with file_name x.html; report.docx keeps .docx; HTML body as note.txt keeps .txt" + requirement: D-15 + verification: + - kind: integration + ref: "modules/lagoon/attach/store_test.go#TestStoreDiskExtensionSmoke (Docker, postgres:16-alpine)" + status: pass + human_judgment: false + - id: D5 + description: "public_url set gives base plus partition path for PublicURL and File.URL; unset keeps /storage/uploads; ftp://, user info, a bare path and a query fail naming the key" + requirement: D-12 + verification: + - kind: unit + ref: "modules/lagoon/attach/infra_smoke_test.go#TestPublicURLSmoke" + status: pass + human_judgment: false + - id: D6 + description: "Docs: S3 section, Public URL base, deploy note (nosniff, allow-list), disk-name paragraph, build.tags in manifest/scaffolding docs, configuration rows, scaling note, lagoon README" + requirement: SC-4 + verification: + - kind: other + ref: "go test ./cmd/summer -run TestDocsTree; go run ./cmd/summer docs:build --check" + status: pass + human_judgment: false + +duration: 10min +completed: 2026-10-15 +status: complete +--- + +# Phase 19.3 Plan 03: S3 build tag, content-derived disk names and public_url Summary + +**`summer.yaml` `build.tags` with the `summercms_s3` tag that links gocloud's s3blob only into tagged binaries (untagged ones name the tag at boot), `attach.DiskExtension` so a PNG uploaded as `x.html` is stored and served as `.png`, and `storage.uploads.public_url` for direct bucket or CDN file URLs** + +## Performance + +- **Duration:** 10 min +- **Started:** 2026-10-15T16:03:46Z +- **Completed:** 2026-10-15T16:13:06Z +- **Tasks:** 3 (Task 1 tracer, Tasks 2 and 3 expansion) +- **Files modified:** 20 + +## Accomplishments + +- `internal/build`: `Manifest.Tags` read from the optional `build.tags` list (trimmed, `ValidateBuildTag` against `^[a-z0-9_]+$`, duplicates dropped, a non-mapping `build` or non-list `tags` is an error naming `build.tags`). `SaveManifest` writes the `build:` block after the plugins, so `summer plugin:add` keeps it. `goBuildArgs` puts `-tags=` before `-o`, and `App` prints `build tags: …`. `summer dev` inherits this through `build.App`. +- `attach`: `s3_on.go` (`//go:build summercms_s3`) is the only importer of `gocloud.dev/blob/s3blob`. `OpenBucket` refuses an `s3://` URL (case-insensitive scheme) in an untagged binary. The error starts with `attach: storage.uploads.bucket_url uses s3://` and names `summercms_s3`, `build.tags` in summer.yaml, `-tags summercms_s3` and `GOFLAGS`. No `s3.*` keys exist. The S3 configuration lives in the gocloud `bucket_url` query, and credentials come from the AWS environment. +- `attach.DiskExtension(contentType, clientExt)` uses a fixed table of 22 specific sniffed types and a list of script-capable extensions. `Store` names the blob with `newDiskName(DiskExtension(sniffed, ext))`. `file_name`, the allow-list and the stored `content_type` rule are unchanged. +- `storage.uploads.public_url`: OpenBucket validates and records it (`PublicURLBase()`) next to the path prefix under `prefixMu` and logs one info line naming the host and the public-files-only rule. `PublicURL` (and therefore `File.URL` and thumbnail URLs) uses the base when it is set. Otherwise the URLs are unchanged. +- Docs: `docs/services/storage.md` gained the sections "S3 and S3-compatible buckets" and "Public URL base", a deploy note on nosniff and allow-lists, and an s3 row in the bucket URL table. Also updated: the disk-name paragraph and URL sentence in `attachments.md`, `build.tags` in `go-modules-and-workspaces.md`, the `summer build`/`summer dev`/`summer.yaml` rows in `scaffolding.md`, the `bucket_url` and `public_url` rows in `configuration.md`, the shared-uploads note in `performance-and-scaling.md`, and the lagoon README (overview, features, API rows `S3BuildTag`, `DiskExtension`, `PublicURLBase`, `PublicURL`, `OpenBucket`, `Store`, config rows, dependencies). + +## Task Commits + +1. **Task 1 (tracer): build.tags and the summercms_s3 S3 driver tag** - `2382040` (feat) +2. **Task 2: content-derived disk names through attach.DiskExtension** - `2e22e9f` (fix) +3. **Task 3: storage.uploads.public_url** - `2e87efa` (feat) + +**Plan metadata:** the final docs(19.3-03) commit. + +## Identifiers and keys added + +- internal/build: `Manifest.Tags`, `ValidateBuildTag`, unexported `goBuildArgs(binRel string, tags []string) []string` and `loadBuildTags`; summer.yaml key `build.tags`; output line `build tags: `. +- attach: `S3BuildTag` (`summercms_s3`), unexported `s3Enabled` and `checkS3Driver`; `DiskExtension`; `PublicURLBase`, unexported `setPublicURLBase` and `parsePublicURL`; config key `storage.uploads.public_url` (env `SUMMER_STORAGE__UPLOADS__PUBLIC_URL`). +- Tests: `TestManifestBuildTagsSmoke`, `TestOpenBucketS3Smoke` (both tag modes), `TestStoreDiskExtensionSmoke` (Postgres), `TestPublicURLSmoke`. + +## go.mod additions + +`go mod tidy` added indirect requirements only: `github.com/aws/aws-sdk-go-v2` v1.41.9 with `config`, `credentials`, `feature/ec2/imds`, `feature/s3/transfermanager`, `internal/configsources`, `internal/endpoints/v2`, `internal/v4a`, `aws/protocol/eventstream`, `service/s3` v1.102.2, `service/internal/{accept-encoding,checksum,presigned-url,s3shared}`, `service/{signin,sso,ssooidc,sts}`, `github.com/aws/smithy-go` v1.26.0 and `github.com/google/wire` v0.7.0. go.sum also gained hashes for `google/go-replayers` and `google/martian/v3` (test-only modules of gocloud). None of them is linked into a binary without the tag. `grep -rln gocloud.dev/blob/s3blob` over modules, internal and cmd lists only `s3_on.go`. + +## Docker smoke tests + +Run on this host (the Docker daemon answered): `TestStoreDiskExtensionSmoke` PASS (postgres:16-alpine). The non-short `go test ./modules/lagoon/...` run after Task 2 was green, and so were the non-short `TestPublicURL|TestFileURL|TestStatic|TestOpenBucket` runs after Task 3. No real S3 or MinIO was available: the S3 smoke only opens the bucket (s3blob connects lazily), so a real upload stays a manual UAT item (19.3-VALIDATION.md). + +## Verification + +- Task 1: `go vet ./...` and `go vet -tags summercms_s3 ./modules/lagoon/...` pass. `go build -tags summercms_s3 ./...` passes. Both smokes passed in both tag modes, and the short suites and TestDocsTree passed. +- Task 2: vet, the lagoon short suites, the Postgres regression smoke and TestDocsTree passed. +- Task 3: both smokes passed (2 PASS lines). The whole `go test -short ./...` run, TestDocsTree and `summer docs:build --check` ("no problems found") passed. +- Wave sampling: `go vet -tags summercms_s3 ./modules/lagoon/... && go test -short -tags summercms_s3 ./modules/lagoon/attach` is green. + +## Decisions Made + +See key-decisions in the frontmatter. All come from the plan's "Claude's discretion" list. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 3 - Blocking] The Store regression smoke lives in store_test.go, not infra_smoke_test.go** +- **Found during:** Task 2 +- **Issue:** The Postgres harness `attachGorm` is in the external test package `attach_test` (lifecycle_test.go). `infra_smoke_test.go` has to be in the internal package `attach`, because `TestOpenBucketS3Smoke` reads the unexported `s3Enabled` and `TestPublicURLSmoke` resets the package URL state. A file cannot belong to both packages. +- **Fix:** `TestStoreDiskExtensionSmoke` is appended to `modules/lagoon/attach/store_test.go` next to `TestStoreSmoke`, whose harness it reuses. The test name and the `-run` filter in the plan's verify command are unchanged. PATTERNS.md suggested this file too. +- **Files modified:** modules/lagoon/attach/store_test.go +- **Committed in:** 2e22e9f + +**2. [Rule 1 - Acceptance gate] A comment in bucket.go named the s3blob import path** +- **Found during:** Task 1 acceptance criteria (only `s3_on.go` may mention `gocloud.dev/blob/s3blob`) +- **Fix:** The `S3BuildTag` doc comment now says "s3blob, imported only by s3_on.go". The fix was made before the Task 1 commit. +- **Committed in:** 2382040 + +**3. [Rule 2 - Hardening] DiskExtension normalises its inputs** +- **Found during:** Task 2 +- **Issue:** `DiskExtension` is exported, and the deferred app call sites will pass `filepath.Ext` output and raw content types. +- **Fix:** It strips media-type parameters, lower-cases both inputs and trims a leading dot. A client extension that fails the `[a-z0-9]{1,10}` pattern falls back to `txt` or `bin` like a script-capable one. `Store`'s inputs are already normalised, so its behaviour does not change. +- **Committed in:** 2e22e9f + +### Notes + +- `github.com/google/wire` v0.7.0 is an indirect requirement of s3blob that the plan 01 decision note does not name (the note names the AWS SDK v2 modules). It comes with the already-required gocloud.dev v0.46.0 and is linked only under the tag. +- The Task 3 acceptance grep for consumer names over `modules/lagoon` matches three pre-existing test files that this plan does not touch. This plan added no matching line (`git diff` count 0). The finding is logged in `deferred-items.md`, together with a pre-existing gofmt issue in `internal/build/registry.go`. + +--- + +**Total deviations:** 3 auto-fixed (1 blocking, 1 acceptance gate, 1 hardening). +**Impact on plan:** None on scope. The identifiers, keys and behaviour match the plan. + +## Issues Encountered + +None. + +## Known Stubs + +None. + +## Threat Flags + +None. All new surface is in the plan's threat register: T-19.3-18 to T-19.3-22 and T-19.3-SC. + +## User Setup Required + +None for the framework. An app that switches to S3 adds `summercms_s3` under `build.tags`, sets an `s3://` `bucket_url` and provides AWS credentials in the environment. An app with a second raw `go build` in its build script must also add `-tags summercms_s3` there. + +## Next Phase Readiness + +- Plan 05 owns full unit coverage: a `DiskExtension` table test (every table type, the script-capable list, input normalisation), `ValidateBuildTag` and loader edge cases (non-mapping `build`, non-list `tags`), and `parsePublicURL` cases. +- Follow-up outside this phase: move the app upload call sites (`storeUpload`, `StorePublicFile`) to `attach.DiskExtension`. The app build script needs `-tags summercms_s3` when the app moves to S3. + +--- +*Phase: 19.3-framework-transport-and-infrastructure-for-wavepath* +*Completed: 2026-10-15* diff --git a/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/deferred-items.md b/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/deferred-items.md new file mode 100644 index 0000000..e22bd1d --- /dev/null +++ b/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/deferred-items.md @@ -0,0 +1,8 @@ +# Phase 19.3 deferred items + +Out-of-scope findings logged by executors. Not fixed in the plan that found them. + +## From 19.3-03 + +- `internal/build/registry.go` is not gofmt-clean (const block alignment, `registryFileName` / `generatedMark`). Pre-existing, untouched by 19.3-03; `go vet` is unaffected. +- The 19.3-03 Task 3 acceptance grep `grep -rniE 'wavepath|fonoteka|grzyby' modules/lagoon ...` matches pre-existing test files outside the plan's scope: `modules/lagoon/hidden_marshal_test.go` (comments naming the consumer app), `modules/lagoon/migrations_test.go` (`golem15.fonoteka` as a sample plugin ID) and `modules/lagoon/attach/lifecycle_test.go` (`Golem15\Fonoteka\Models\Album` morph type). CLAUDE.md says framework code never names a consuming application; a follow-up quick task could switch these to neutral names (`acme`). 19.3-03 added no such names.