docs(19.3-03): complete S3 build tag, content-derived disk names and public_url plan

2026-10-15 18:14:18 +02:00
parent 2e87efae54
commit 17054f9f85
2 changed files with 234 additions and 0 deletions
--- a/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/19.3-03-SUMMARY.md
+++ b/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/19.3-03-SUMMARY.md
@@ -0,0 +1,226 @@
+---
+phase: 19.3-framework-transport-and-infrastructure-for-wavepath
+plan: 03
+subsystem: storage
+tags: [lagoon, attach, s3, gocloud, build-tags, summer-build, stored-xss, cdn]
+
+requires:
+  - phase: 19.3-01
+    provides: "decision note .planning/notes/2026-10-15-cooler-go-redis-otter.md naming gocloud.dev/blob/s3blob and its AWS SDK v2 modules"
+provides:
+  - "summer.yaml build.tags (Manifest.Tags, ValidateBuildTag, goBuildArgs, build tags output line)"
+  - "attach.S3BuildTag (summercms_s3): s3blob linked only by s3_on.go; an untagged binary refuses s3:// at boot naming the tag"
+  - "attach.DiskExtension: stored extension follows the sniffed content (stored-XSS fix)"
+  - "storage.uploads.public_url and attach.PublicURLBase for direct bucket or CDN URLs"
+affects: [19.3-05, fonoteka-upload-call-sites-follow-up, apps-on-s3]
+
+actuals:
+  tokens: 15400
+  tasks: 3
+  commits: 3
+plan_head_before: b1fab9930b11090d7a5515f05a3842e15be06809
+plan_head_after: 2e87efae54141a12a197f1e8de844a1996ffba32
+
+tech-stack:
+  added: ["gocloud.dev/blob/s3blob (sub-package of gocloud.dev v0.46.0, summercms_s3 tag only)", "github.com/aws/aws-sdk-go-v2 v1.41.9 and 17 sibling modules (indirect)", "github.com/aws/smithy-go v1.26.0 (indirect)", "github.com/google/wire v0.7.0 (indirect)"]
+  patterns:
+    - "Feature build tag split: s3_on.go (//go:build summercms_s3) blank-imports the driver and sets a const; s3_off.go sets it false; the boot guard names the tag"
+    - "Manifest keys must be written back by SaveManifest so summer plugin:add round-trips them"
+    - "Extension derived from content through a fixed in-code table, never package mime"
+
+key-files:
+  created:
+    - internal/build/manifest_tags_smoke_test.go
+    - modules/lagoon/attach/s3_on.go
+    - modules/lagoon/attach/s3_off.go
+    - modules/lagoon/attach/diskext.go
+    - modules/lagoon/attach/infra_smoke_test.go
+  modified:
+    - go.mod
+    - go.sum
+    - internal/build/manifest.go
+    - internal/build/build.go
+    - modules/lagoon/attach/bucket.go
+    - modules/lagoon/attach/store.go
+    - modules/lagoon/attach/store_test.go
+    - modules/lagoon/attach/thumb.go
+    - modules/lagoon/README.md
+    - docs/services/storage.md
+    - docs/database/attachments.md
+    - docs/architecture/go-modules-and-workspaces.md
+    - docs/architecture/performance-and-scaling.md
+    - docs/console/scaffolding.md
+    - docs/setup/configuration.md
+
+key-decisions:
+  - "Build tag summercms_s3 (attach.S3BuildTag), declared in summer.yaml as build.tags; tags match ^[a-z0-9_]+$, duplicates dropped, passed to go build as one -tags= exec argument; summer build prints a build tags line"
+  - "DiskExtension: specific sniffed types keep a matching client extension or use the canonical one; generic sniffs keep the client extension unless script-capable (txt for text/*, bin otherwise); Store passes the sniffed type before its octet-stream extension fallback"
+  - "storage.uploads.public_url applies to every URL attach builds, private files included; the start-up info line and the docs carry the public-files-only rule"
+  - "public_url is validated before the bucket opens (absolute http/https, host, no user info, query or fragment); errors name the key and SUMMER_STORAGE__UPLOADS__PUBLIC_URL without echoing the value"
+
+patterns-established:
+  - "Tag-mode smoke tests branch on the unexported const (s3Enabled) so one test passes in both tag modes"
+
+requirements-completed: [SC-3, SC-4, D-12, D-13, D-14, D-15, D-19]
+
+coverage:
+  - id: D1
+    description: "summer.yaml build.tags loads (dedup, validation), round-trips through SaveManifest, and reaches go build as -tags before -o"
+    requirement: D-13
+    verification:
+      - kind: unit
+        ref: "internal/build/manifest_tags_smoke_test.go#TestManifestBuildTagsSmoke"
+        status: pass
+    human_judgment: false
+  - id: D2
+    description: "Untagged binary refuses s3:// naming summercms_s3, build.tags and -tags; tagged binary opens s3://smoke-bucket without network"
+    requirement: D-13
+    verification:
+      - kind: unit
+        ref: "modules/lagoon/attach/infra_smoke_test.go#TestOpenBucketS3Smoke (go test -short, and go test -short -tags summercms_s3)"
+        status: pass
+    human_judgment: false
+  - id: D3
+    description: "Real upload to an S3-compatible bucket (R2/MinIO/Hetzner) through the tagged binary"
+    requirement: SC-3
+    human_judgment: true
+    rationale: "No real S3 or MinIO available; manual UAT item in 19.3-VALIDATION.md"
+  - id: D4
+    description: "PNG uploaded as x.html is stored and served as .png with file_name x.html; report.docx keeps .docx; HTML body as note.txt keeps .txt"
+    requirement: D-15
+    verification:
+      - kind: integration
+        ref: "modules/lagoon/attach/store_test.go#TestStoreDiskExtensionSmoke (Docker, postgres:16-alpine)"
+        status: pass
+    human_judgment: false
+  - id: D5
+    description: "public_url set gives base plus partition path for PublicURL and File.URL; unset keeps /storage/uploads; ftp://, user info, a bare path and a query fail naming the key"
+    requirement: D-12
+    verification:
+      - kind: unit
+        ref: "modules/lagoon/attach/infra_smoke_test.go#TestPublicURLSmoke"
+        status: pass
+    human_judgment: false
+  - id: D6
+    description: "Docs: S3 section, Public URL base, deploy note (nosniff, allow-list), disk-name paragraph, build.tags in manifest/scaffolding docs, configuration rows, scaling note, lagoon README"
+    requirement: SC-4
+    verification:
+      - kind: other
+        ref: "go test ./cmd/summer -run TestDocsTree; go run ./cmd/summer docs:build --check"
+        status: pass
+    human_judgment: false
+
+duration: 10min
+completed: 2026-10-15
+status: complete
+---
+
+# Phase 19.3 Plan 03: S3 build tag, content-derived disk names and public_url Summary
+
+**`summer.yaml` `build.tags` with the `summercms_s3` tag that links gocloud's s3blob only into tagged binaries (untagged ones name the tag at boot), `attach.DiskExtension` so a PNG uploaded as `x.html` is stored and served as `.png`, and `storage.uploads.public_url` for direct bucket or CDN file URLs**
+
+## Performance
+
+- **Duration:** 10 min
+- **Started:** 2026-10-15T16:03:46Z
+- **Completed:** 2026-10-15T16:13:06Z
+- **Tasks:** 3 (Task 1 tracer, Tasks 2 and 3 expansion)
+- **Files modified:** 20
+
+## Accomplishments
+
+- `internal/build`: `Manifest.Tags` read from the optional `build.tags` list (trimmed, `ValidateBuildTag` against `^[a-z0-9_]+$`, duplicates dropped, a non-mapping `build` or non-list `tags` is an error naming `build.tags`). `SaveManifest` writes the `build:` block after the plugins, so `summer plugin:add` keeps it. `goBuildArgs` puts `-tags=<comma list>` before `-o`, and `App` prints `build tags: …`. `summer dev` inherits this through `build.App`.
+- `attach`: `s3_on.go` (`//go:build summercms_s3`) is the only importer of `gocloud.dev/blob/s3blob`. `OpenBucket` refuses an `s3://` URL (case-insensitive scheme) in an untagged binary. The error starts with `attach: storage.uploads.bucket_url uses s3://` and names `summercms_s3`, `build.tags` in summer.yaml, `-tags summercms_s3` and `GOFLAGS`. No `s3.*` keys exist. The S3 configuration lives in the gocloud `bucket_url` query, and credentials come from the AWS environment.
+- `attach.DiskExtension(contentType, clientExt)` uses a fixed table of 22 specific sniffed types and a list of script-capable extensions. `Store` names the blob with `newDiskName(DiskExtension(sniffed, ext))`. `file_name`, the allow-list and the stored `content_type` rule are unchanged.
+- `storage.uploads.public_url`: OpenBucket validates and records it (`PublicURLBase()`) next to the path prefix under `prefixMu` and logs one info line naming the host and the public-files-only rule. `PublicURL` (and therefore `File.URL` and thumbnail URLs) uses the base when it is set. Otherwise the URLs are unchanged.
+- Docs: `docs/services/storage.md` gained the sections "S3 and S3-compatible buckets" and "Public URL base", a deploy note on nosniff and allow-lists, and an s3 row in the bucket URL table. Also updated: the disk-name paragraph and URL sentence in `attachments.md`, `build.tags` in `go-modules-and-workspaces.md`, the `summer build`/`summer dev`/`summer.yaml` rows in `scaffolding.md`, the `bucket_url` and `public_url` rows in `configuration.md`, the shared-uploads note in `performance-and-scaling.md`, and the lagoon README (overview, features, API rows `S3BuildTag`, `DiskExtension`, `PublicURLBase`, `PublicURL`, `OpenBucket`, `Store`, config rows, dependencies).
+
+## Task Commits
+
+1. **Task 1 (tracer): build.tags and the summercms_s3 S3 driver tag** - `2382040` (feat)
+2. **Task 2: content-derived disk names through attach.DiskExtension** - `2e22e9f` (fix)
+3. **Task 3: storage.uploads.public_url** - `2e87efa` (feat)
+
+**Plan metadata:** the final docs(19.3-03) commit.
+
+## Identifiers and keys added
+
+- internal/build: `Manifest.Tags`, `ValidateBuildTag`, unexported `goBuildArgs(binRel string, tags []string) []string` and `loadBuildTags`; summer.yaml key `build.tags`; output line `build tags: <list>`.
+- attach: `S3BuildTag` (`summercms_s3`), unexported `s3Enabled` and `checkS3Driver`; `DiskExtension`; `PublicURLBase`, unexported `setPublicURLBase` and `parsePublicURL`; config key `storage.uploads.public_url` (env `SUMMER_STORAGE__UPLOADS__PUBLIC_URL`).
+- Tests: `TestManifestBuildTagsSmoke`, `TestOpenBucketS3Smoke` (both tag modes), `TestStoreDiskExtensionSmoke` (Postgres), `TestPublicURLSmoke`.
+
+## go.mod additions
+
+`go mod tidy` added indirect requirements only: `github.com/aws/aws-sdk-go-v2` v1.41.9 with `config`, `credentials`, `feature/ec2/imds`, `feature/s3/transfermanager`, `internal/configsources`, `internal/endpoints/v2`, `internal/v4a`, `aws/protocol/eventstream`, `service/s3` v1.102.2, `service/internal/{accept-encoding,checksum,presigned-url,s3shared}`, `service/{signin,sso,ssooidc,sts}`, `github.com/aws/smithy-go` v1.26.0 and `github.com/google/wire` v0.7.0. go.sum also gained hashes for `google/go-replayers` and `google/martian/v3` (test-only modules of gocloud). None of them is linked into a binary without the tag. `grep -rln gocloud.dev/blob/s3blob` over modules, internal and cmd lists only `s3_on.go`.
+
+## Docker smoke tests
+
+Run on this host (the Docker daemon answered): `TestStoreDiskExtensionSmoke` PASS (postgres:16-alpine). The non-short `go test ./modules/lagoon/...` run after Task 2 was green, and so were the non-short `TestPublicURL|TestFileURL|TestStatic|TestOpenBucket` runs after Task 3. No real S3 or MinIO was available: the S3 smoke only opens the bucket (s3blob connects lazily), so a real upload stays a manual UAT item (19.3-VALIDATION.md).
+
+## Verification
+
+- Task 1: `go vet ./...` and `go vet -tags summercms_s3 ./modules/lagoon/...` pass. `go build -tags summercms_s3 ./...` passes. Both smokes passed in both tag modes, and the short suites and TestDocsTree passed.
+- Task 2: vet, the lagoon short suites, the Postgres regression smoke and TestDocsTree passed.
+- Task 3: both smokes passed (2 PASS lines). The whole `go test -short ./...` run, TestDocsTree and `summer docs:build --check` ("no problems found") passed.
+- Wave sampling: `go vet -tags summercms_s3 ./modules/lagoon/... && go test -short -tags summercms_s3 ./modules/lagoon/attach` is green.
+
+## Decisions Made
+
+See key-decisions in the frontmatter. All come from the plan's "Claude's discretion" list.
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 3 - Blocking] The Store regression smoke lives in store_test.go, not infra_smoke_test.go**
+- **Found during:** Task 2
+- **Issue:** The Postgres harness `attachGorm` is in the external test package `attach_test` (lifecycle_test.go). `infra_smoke_test.go` has to be in the internal package `attach`, because `TestOpenBucketS3Smoke` reads the unexported `s3Enabled` and `TestPublicURLSmoke` resets the package URL state. A file cannot belong to both packages.
+- **Fix:** `TestStoreDiskExtensionSmoke` is appended to `modules/lagoon/attach/store_test.go` next to `TestStoreSmoke`, whose harness it reuses. The test name and the `-run` filter in the plan's verify command are unchanged. PATTERNS.md suggested this file too.
+- **Files modified:** modules/lagoon/attach/store_test.go
+- **Committed in:** 2e22e9f
+
+**2. [Rule 1 - Acceptance gate] A comment in bucket.go named the s3blob import path**
+- **Found during:** Task 1 acceptance criteria (only `s3_on.go` may mention `gocloud.dev/blob/s3blob`)
+- **Fix:** The `S3BuildTag` doc comment now says "s3blob, imported only by s3_on.go". The fix was made before the Task 1 commit.
+- **Committed in:** 2382040
+
+**3. [Rule 2 - Hardening] DiskExtension normalises its inputs**
+- **Found during:** Task 2
+- **Issue:** `DiskExtension` is exported, and the deferred app call sites will pass `filepath.Ext` output and raw content types.
+- **Fix:** It strips media-type parameters, lower-cases both inputs and trims a leading dot. A client extension that fails the `[a-z0-9]{1,10}` pattern falls back to `txt` or `bin` like a script-capable one. `Store`'s inputs are already normalised, so its behaviour does not change.
+- **Committed in:** 2e22e9f
+
+### Notes
+
+- `github.com/google/wire` v0.7.0 is an indirect requirement of s3blob that the plan 01 decision note does not name (the note names the AWS SDK v2 modules). It comes with the already-required gocloud.dev v0.46.0 and is linked only under the tag.
+- The Task 3 acceptance grep for consumer names over `modules/lagoon` matches three pre-existing test files that this plan does not touch. This plan added no matching line (`git diff` count 0). The finding is logged in `deferred-items.md`, together with a pre-existing gofmt issue in `internal/build/registry.go`.
+
+---
+
+**Total deviations:** 3 auto-fixed (1 blocking, 1 acceptance gate, 1 hardening).
+**Impact on plan:** None on scope. The identifiers, keys and behaviour match the plan.
+
+## Issues Encountered
+
+None.
+
+## Known Stubs
+
+None.
+
+## Threat Flags
+
+None. All new surface is in the plan's threat register: T-19.3-18 to T-19.3-22 and T-19.3-SC.
+
+## User Setup Required
+
+None for the framework. An app that switches to S3 adds `summercms_s3` under `build.tags`, sets an `s3://` `bucket_url` and provides AWS credentials in the environment. An app with a second raw `go build` in its build script must also add `-tags summercms_s3` there.
+
+## Next Phase Readiness
+
+- Plan 05 owns full unit coverage: a `DiskExtension` table test (every table type, the script-capable list, input normalisation), `ValidateBuildTag` and loader edge cases (non-mapping `build`, non-list `tags`), and `parsePublicURL` cases.
+- Follow-up outside this phase: move the app upload call sites (`storeUpload`, `StorePublicFile`) to `attach.DiskExtension`. The app build script needs `-tags summercms_s3` when the app moves to S3.
+
+---
+*Phase: 19.3-framework-transport-and-infrastructure-for-wavepath*
+*Completed: 2026-10-15*
--- a/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/deferred-items.md
+++ b/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/deferred-items.md
@@ -0,0 +1,8 @@
+# Phase 19.3 deferred items
+
+Out-of-scope findings logged by executors. Not fixed in the plan that found them.
+
+## From 19.3-03
+
+- `internal/build/registry.go` is not gofmt-clean (const block alignment, `registryFileName` / `generatedMark`). Pre-existing, untouched by 19.3-03; `go vet` is unaffected.
+- The 19.3-03 Task 3 acceptance grep `grep -rniE 'wavepath|fonoteka|grzyby' modules/lagoon ...` matches pre-existing test files outside the plan's scope: `modules/lagoon/hidden_marshal_test.go` (comments naming the consumer app), `modules/lagoon/migrations_test.go` (`golem15.fonoteka` as a sample plugin ID) and `modules/lagoon/attach/lifecycle_test.go` (`Golem15\Fonoteka\Models\Album` morph type). CLAUDE.md says framework code never names a consuming application; a follow-up quick task could switch these to neutral names (`acme`). 19.3-03 added no such names.