fix(lagoon/attach): store non-inline uploads with Content-Disposition attachment (quick-261028-rqe)

- attach.IsInlineContentType: the shared inline raster list (JPEG, PNG,
  GIF, WebP, AVIF), parameters and case ignored, false for empty or
  unparsable values
- writeBlob stores Content-Disposition: attachment for every other type,
  so a public bucket or CDN makes the browser download HTML, SVG, PDF or
  plain text; content_type and thumbnails are unchanged
- driver tests on memblob and fileblob, Postgres-backed Store assertion,
  lagoon README and the storage and attachments pages

docs/database/attachments.md

@@ -97,6 +97,8 @@ The client file name never reaches a blob key. The disk name is 22 random hexade
 
 The disk name's extension follows the content, not the client's name, because a static web server in front of the uploads directory picks the Content-Type from the extension. A public bucket, or a CDN that pulls from it, sends the type stored with the object instead; see the [deploy note](../services/storage.md#deploy-note) for both cases. `attach.DiskExtension` derives the extension from the sniffed type and the client's extension. For a specific type such as PNG, JPEG, PDF, MP3 or WebM, the client's extension is kept when it names that type, else the type's usual extension is used: a PNG uploaded as `x.html` (with `html` allowed) is stored as `<hex>.png`, and its URL ends in `.png`. For a generic sniff (plain text, HTML, XML, a zip container or unknown bytes) the client's extension is kept, so `report.docx` stays `.docx` and an HTML body uploaded as `note.txt` stays `.txt`, unless the extension is one a browser renders or runs (html, svg, xml, js, css, php and the like), which becomes `.txt` for text and `.bin` otherwise. `file_name` always keeps the client's name, so downloads still carry it.
 
+`attach.Store` writes every file whose type `attach.IsInlineContentType` does not report as a raster image (JPEG, PNG, GIF, WebP, AVIF) with `Content-Disposition: attachment`, so a browser that opens it from a public bucket downloads it instead of rendering it. `<img>`, `<video>` and `<audio>` embeds and the `content_type` column are unaffected, and thumbnails stay inline.
+
 The content type is sniffed from the first bytes of the body, never taken from the client. `Limits.MIMETypes` narrows it further: an entry with a slash is a MIME pattern such as `image/png` or `image/*`, an entry without one is an extension, and a file that matches no entry returns `attach.ErrMIMEType`. `Limits.MaxBytes` is enforced while the body streams into the bucket; a body one byte over the limit returns `attach.ErrTooLarge` and leaves no blob and no row behind.
 
 With `Limits.Image` set, the content must pass `attach.IsAllowedImage`: the bytes must sniff as one of `attach.AllowedImageMIMEs` (JPEG, PNG, GIF or WebP), the header must decode as that format, and the image may not exceed `attach.MaxImagePixels` (4096 by 4096), so every accepted image can be thumbnailed. Anything else, an SVG for example, returns `attach.ErrNotImage`.

docs/services/storage.md

@@ -157,7 +157,11 @@ server {
 
 Extend the extension list only with types the application accepts (see `attach.DefaultFileExtensions`), and serve anything that is not a raster image as a download (`Content-Disposition: attachment`) or from a separate origin, the way the built-in route does.
 
-**A public S3 bucket, or a CDN that pulls from it, serving the files under `storage.uploads.public_url`.** It does not look at the extension: it sends the Content-Type stored with the object, and `attach.Store` stores the sniffed type. An HTML body uploaded as `note.txt` (txt is in `attach.DefaultFileExtensions`) is stored as `<hex>.txt` but served as `text/html`. Serve such a bucket from its own origin, never the application's, and set `X-Content-Type-Options: nosniff` and, for everything but raster images, `Content-Disposition: attachment` through the bucket's or the CDN's response-header settings.
+**A public S3 bucket, or a CDN that pulls from it, serving the files under `storage.uploads.public_url`.** It does not look at the extension: it sends the headers stored with the object. The stored Content-Type is the sniffed type, so an HTML body uploaded as `note.txt` (txt is in `attach.DefaultFileExtensions`) is stored as `<hex>.txt` with `text/html`. `attach.Store` therefore also stores `Content-Disposition: attachment` with every object whose type `attach.IsInlineContentType` does not report as a raster image (JPEG, PNG, GIF, WebP, AVIF). S3, and a CDN that forwards the stored headers, make a browser that opens the URL download HTML, SVG, PDF or plain text instead of rendering it. `<img>`, `<video>` and `<audio>` embeds are not affected, and thumbnails, being images, carry no disposition.
+
+Objects written before this change, or by code that bypasses `attach.Store`, carry no disposition: set it on them, or add the rule in the CDN. A CDN that rewrites or drops the origin's headers needs the rule itself. Either way, serve such a bucket from its own origin, never the application's, and set `X-Content-Type-Options: nosniff` through the bucket's or the CDN's response-header settings, since S3 cannot store that header on an object.
+
+A `file://` bucket keeps the disposition in the file driver's `.attrs` sidecar files, which a static server does not read, so the static-server case above still applies to it.
 
 ## Upload size
 

modules/lagoon/README.md

@@ -39,7 +39,7 @@ Use cases:
 - Deferred binding: WinterCMS's `deferred_bindings` table holds the uploads and related-record changes of a form whose record is not saved yet. Every operation takes a `lagoon.DeferredKey` (the form's session key, the owning backend admin's id and the master record's morph type from `lagoon.MorphType`) and never reads or changes another admin's rows, since each row stores `backend_user_id`. `lagoon.DeferredBind` and `lagoon.DeferredUnbind` port WinterCMS's duplicate and cancel rules: a repeated bind writes nothing, and an unbind of a slave with a pending bind deletes that bind and returns it so the caller can remove what it created. `lagoon.DeferredBindings` reads and locks a session's bindings for the save that commits them, `lagoon.DeferredForget` deletes them once applied, and `lagoon.DeferredSlaves` is the subquery a list uses to include pending rows. A child created under deferral carries the `lagoon.DeferredEnvelope` (`{"created":true,"pivot":{...},"upload_id":"..."}`) in `pivot_data`. `lagoon.PurgeDeferred` removes expired bindings: it deletes an unattached `system_files` row a bind points at, and its blobs only after the commit, deletes a child only when its binding carries the created envelope, keeps records that were only linked, and locks each batch with `FOR UPDATE SKIP LOCKED`.
 - Logging: GORM's lines no longer go to stdout through GORM's own logger. They arrive through [diary](../diary/README.md) as component `db`: slow queries (over 200 ms) and errors at warn, while a missing record (`gorm.ErrRecordNotFound`) is no longer logged. At verbosity 3 (`-vvv`) every query is traced at info, with placeholders instead of the bound values. lagoon's own warnings log as component `lagoon`, and `attach` as component `attach`.
 - Imports from Laravel: `lagoon.DecryptLaravelPayload` decrypts Laravel `encrypted` payloads with the old application key, for one-off data imports.
-- Attachments (`attach`): the `attach.File` model for `system_files` rows, WinterCMS-compatible partitioned storage keys (`attach.BlobKey`, `attach.PartitionDirectory`), an S3 bucket option behind the `attach.S3BuildTag` build tag (an untagged binary refuses an `s3://` bucket URL at boot with an error naming the tag), public URLs (`attach.PublicURL` for any key, under `storage.uploads.public_url` when a public bucket or CDN serves the files (`attach.PublicURLBase`), `attach.File.URL` for an original, matching WinterCMS's `File::getPath()` under the WinterCMS layout), on-demand thumbnails through `attach.File.Thumb` for JPEG, PNG, GIF and WebP originals (a WebP original's thumbnail is JPEG bytes under its `.webp` name, since WebP cannot be encoded; a missing, undecodable or oversized original gets WinterCMS's broken-image picture, `attach.BrokenImagePNG`, as its thumbnail, as `File::makeThumb` does), storing uploads through `attach.Store` (a server-generated disk name whose extension follows the sniffed content through `attach.DiskExtension`, so a PNG named `x.html` is stored as `.png`, an extension allow-list with `attach.DefaultImageExtensions` and `attach.DefaultFileExtensions` as defaults, a MIME filter, a size limit enforced while streaming, in image mode the `attach.IsAllowedImage` content guard, and the refusal of protected uploads while `storage.uploads.public_url` serves the bucket publicly, `attach.ErrProtectedNeedsPrivateBucket`), attachment relation declarations (`attach.Relation`, `attach.HasRelations`), static serving with an optional `is_public` gate (`attach.StaticHandlerPublic`), and a two-phase delete that removes blobs only after the database transaction commits (`attach.DeleteForOwner`, `attach.DeleteKeys`).
+- Attachments (`attach`): the `attach.File` model for `system_files` rows, WinterCMS-compatible partitioned storage keys (`attach.BlobKey`, `attach.PartitionDirectory`), an S3 bucket option behind the `attach.S3BuildTag` build tag (an untagged binary refuses an `s3://` bucket URL at boot with an error naming the tag), public URLs (`attach.PublicURL` for any key, under `storage.uploads.public_url` when a public bucket or CDN serves the files (`attach.PublicURLBase`), `attach.File.URL` for an original, matching WinterCMS's `File::getPath()` under the WinterCMS layout), on-demand thumbnails through `attach.File.Thumb` for JPEG, PNG, GIF and WebP originals (a WebP original's thumbnail is JPEG bytes under its `.webp` name, since WebP cannot be encoded; a missing, undecodable or oversized original gets WinterCMS's broken-image picture, `attach.BrokenImagePNG`, as its thumbnail, as `File::makeThumb` does), storing uploads through `attach.Store` (a server-generated disk name whose extension follows the sniffed content through `attach.DiskExtension`, so a PNG named `x.html` is stored as `.png`, an extension allow-list with `attach.DefaultImageExtensions` and `attach.DefaultFileExtensions` as defaults, a MIME filter, a size limit enforced while streaming, in image mode the `attach.IsAllowedImage` content guard, the refusal of protected uploads while `storage.uploads.public_url` serves the bucket publicly, `attach.ErrProtectedNeedsPrivateBucket`, and `Content-Disposition: attachment` on the blob for every type `attach.IsInlineContentType` does not report as an inline raster image (JPEG, PNG, GIF, WebP, AVIF), while `content_type` keeps the sniffed type), attachment relation declarations (`attach.Relation`, `attach.HasRelations`), static serving with an optional `is_public` gate (`attach.StaticHandlerPublic`), and a two-phase delete that removes blobs only after the database transaction commits (`attach.DeleteForOwner`, `attach.DeleteKeys`).
 
 ## Usage
 
@@ -206,7 +206,7 @@ func (p *Plugin) Migrations() []*gormigrate.Migration {
 | `attach.DeleteForOwner` | Deletes an owner's attachment rows in a transaction and reports their blob keys. |
 | `attach.DeleteKeys` | Deletes blobs, including thumbnails, after the transaction commits. |
 | `attach.Migrations` | Creates the `system_files` table. |
-| `attach.Store` | Stores an upload as an unattached `system_files` row with `sort_order` equal to its id, after the type, size and image checks; the disk name's extension comes from `attach.DiskExtension`, `file_name` keeps the client's name. |
+| `attach.Store` | Stores an upload as an unattached `system_files` row with `sort_order` equal to its id, after the type, size and image checks; the disk name's extension comes from `attach.DiskExtension`, `file_name` keeps the client's name, and a blob of a non-inline type is stored with `Content-Disposition: attachment`. |
 | `attach.DiskExtension` | The stored extension for a sniffed content type and the client's extension: the client's when it matches a specific type, else that type's canonical one; under a generic sniff the client's unless it is script-capable (then `txt` or `bin`). |
 | `attach.Upload` | The client file name, body and public flag of one upload. |
 | `attach.Limits` | Size limit, allowed extensions, allowed MIME types and image mode for `attach.Store`. |
@@ -219,6 +219,7 @@ func (p *Plugin) Migrations() []*gormigrate.Migration {
 | `attach.DefaultFileExtensions` | File-mode extensions when none are given: WinterCMS's default list without the script-capable types. |
 | `attach.AllowedImageMIMEs` | The sniffed content types the image guard accepts. |
 | `attach.IsAllowedImage` | The image guard: sniffed type, decoded header and a pixel ceiling, failing closed. |
+| `attach.IsInlineContentType` | Reports whether a content type, parameters ignored, is a raster image a browser may show inline (JPEG, PNG, GIF, WebP, AVIF); `attach.Store` stores every other type with `Content-Disposition: attachment`, and the surf uploads route serves it as a download. |
 | `attach.MaxImagePixels` | The image guard's pixel ceiling, 4096 by 4096. |
 | `attach.Relation` | One attachOne or attachMany relation: `Name`, `Many` and `Public`. |
 | `attach.HasRelations` | Implemented by an owner model that declares its attachment relations. |

modules/lagoon/attach/inline.go

@@ -0,0 +1,44 @@
+package attach
+
+import (
+	"mime"
+	"strings"
+
+	"gocloud.dev/blob"
+)
+
+// inlineContentTypes are the raster image types a browser may show inline
+// from an upload.
+var inlineContentTypes = map[string]bool{
+	"image/avif": true,
+	"image/gif":  true,
+	"image/jpeg": true,
+	"image/png":  true,
+	"image/webp": true,
+}
+
+// attachmentDisposition is the Content-Disposition stored with a non-inline
+// upload. It is the bare value with no filename parameter: the client's file
+// name never reaches blob metadata or a response header.
+const attachmentDisposition = "attachment"
+
+// IsInlineContentType reports whether contentType, with parameters and case
+// ignored, is a raster image a browser may show inline from an upload (JPEG,
+// PNG, GIF, WebP, AVIF). An empty or unparsable value is not. Store stores
+// every other type, SVG, PDF, HTML and plain text included, with
+// Content-Disposition: attachment, and the surf uploads route serves it as a
+// download.
+func IsInlineContentType(contentType string) bool {
+	mediaType, _, err := mime.ParseMediaType(contentType)
+	return err == nil && inlineContentTypes[strings.ToLower(mediaType)]
+}
+
+// uploadWriterOptions are the blob writer options of an upload: the content
+// type, plus the attachment disposition when the type is not inline.
+func uploadWriterOptions(contentType string) *blob.WriterOptions {
+	opts := &blob.WriterOptions{ContentType: contentType}
+	if !IsInlineContentType(contentType) {
+		opts.ContentDisposition = attachmentDisposition
+	}
+	return opts
+}

modules/lagoon/attach/inline_test.go

@@ -0,0 +1,116 @@
+package attach
+
+import (
+	"strings"
+	"testing"
+
+	"gocloud.dev/blob"
+	"gocloud.dev/blob/fileblob"
+	"gocloud.dev/blob/memblob"
+)
+
+func TestIsInlineContentType(t *testing.T) {
+	cases := []struct {
+		in   string
+		want bool
+	}{
+		{"image/jpeg", true},
+		{"image/png", true},
+		{"image/gif", true},
+		{"image/webp", true},
+		{"image/avif", true},
+		{"IMAGE/WEBP; charset=binary", true},
+		{"text/html; charset=utf-8", false},
+		{"image/svg+xml", false},
+		{"application/pdf", false},
+		{"text/plain", false},
+		{"application/octet-stream", false},
+		{"application/zip", false},
+		{"image/bmp", false},
+		{"image/x-icon", false},
+		{"video/mp4", false},
+		{"audio/mpeg", false},
+		{"", false},
+		{"not a media type;;", false},
+	}
+	for _, tc := range cases {
+		if got := IsInlineContentType(tc.in); got != tc.want {
+			t.Errorf("IsInlineContentType(%q) = %v, want %v", tc.in, got, tc.want)
+		}
+	}
+}
+
+// TestWriteBlobDisposition is the driver half of the public-bucket fix: a
+// public bucket serves the headers Store stores with the object, so the
+// attachment disposition must survive the write on the real drivers. The
+// metadata-skip fileblob bucket shows that a driver which does not keep
+// attributes still accepts the option without an error.
+func TestWriteBlobDisposition(t *testing.T) {
+	cases := []struct {
+		name, ct, disposition string
+	}{
+		{"html", "text/html", "attachment"},
+		{"text", "text/plain", "attachment"},
+		{"pdf", "application/pdf", "attachment"},
+		{"svg", "image/svg+xml", "attachment"},
+		{"zip", "application/zip", "attachment"},
+		{"png", "image/png", ""},
+		{"jpeg", "image/jpeg", ""},
+		{"avif", "image/avif", ""},
+	}
+	sidecar, err := fileblob.OpenBucket(t.TempDir(), nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	noMeta, err := fileblob.OpenBucket(t.TempDir(), &fileblob.Options{Metadata: fileblob.MetadataDontWrite})
+	if err != nil {
+		t.Fatal(err)
+	}
+	buckets := []struct {
+		name       string
+		bucket     *blob.Bucket
+		keepsAttrs bool
+	}{
+		{"memblob", memblob.OpenBucket(nil), true},
+		{"fileblob sidecar", sidecar, true},
+		{"fileblob metadata skipped", noMeta, false},
+	}
+	const body = "<html><script>alert(1)</script></html>"
+	for _, b := range buckets {
+		bucket := b.bucket
+		t.Cleanup(func() { _ = bucket.Close() })
+		t.Run(b.name, func(t *testing.T) {
+			ctx := t.Context()
+			for _, tc := range cases {
+				key := "abc/def/012/" + tc.name + "-file.bin"
+				n, err := writeBlob(ctx, bucket, key, strings.NewReader(body), tc.ct, 0)
+				if err != nil {
+					t.Fatalf("%s: writeBlob: %v", tc.name, err)
+				}
+				if n != int64(len(body)) {
+					t.Fatalf("%s: wrote %d bytes, want %d", tc.name, n, len(body))
+				}
+				got, err := bucket.ReadAll(ctx, key)
+				if err != nil {
+					t.Fatalf("%s: ReadAll: %v", tc.name, err)
+				}
+				if string(got) != body {
+					t.Fatalf("%s: body = %q, want %q", tc.name, got, body)
+				}
+				if !b.keepsAttrs {
+					continue
+				}
+				attrs, err := bucket.Attributes(ctx, key)
+				if err != nil {
+					t.Fatalf("%s: Attributes: %v", tc.name, err)
+				}
+				if attrs.ContentType != tc.ct {
+					t.Errorf("%s: ContentType = %q, want %q", tc.name, attrs.ContentType, tc.ct)
+				}
+				if attrs.ContentDisposition != tc.disposition {
+					t.Errorf("%s: ContentDisposition = %q, want %q", tc.name, attrs.ContentDisposition, tc.disposition)
+				}
+			}
+		})
+	}
+}

modules/lagoon/attach/store.go

@@ -106,9 +106,13 @@ type Limits struct {
 // is then streamed into bucket at BlobKey of a server-generated disk name (22
 // random lowercase hex characters, a dot and the extension DiskExtension
 // derives from the sniffed type and the client extension, so a PNG named
-// x.html is stored as .png; file_name keeps the client's name); a body longer
-// than Limits.MaxBytes aborts the write, deletes the key and returns
-// ErrTooLarge. Finally it inserts the row with empty attachment columns,
+// x.html is stored as .png; file_name keeps the client's name). The blob's
+// Content-Type is the stored content type; a type IsInlineContentType does
+// not accept also gets Content-Disposition: attachment, so a public bucket or
+// CDN that sends the stored headers makes a browser download the file instead
+// of rendering it. The row's content_type holds the same type either way. A
+// body longer than Limits.MaxBytes aborts the write, deletes the key and
+// returns ErrTooLarge. Finally it inserts the row with empty attachment columns,
 // is_public from Upload.Public, the byte size and the content type, and sets
 // sort_order to the new id as WinterCMS's Sortable trait does. When the row
 // cannot be written the blob is deleted again.
@@ -201,7 +205,7 @@ func Store(ctx context.Context, db *gorm.DB, bucket *blob.Bucket, in Upload, lim
 func writeBlob(ctx context.Context, bucket *blob.Bucket, key string, r io.Reader, contentType string, limit int64) (int64, error) {
 	writeCtx, cancel := context.WithCancel(ctx)
 	defer cancel()
-	w, err := bucket.NewWriter(writeCtx, key, &blob.WriterOptions{ContentType: contentType})
+	w, err := bucket.NewWriter(writeCtx, key, uploadWriterOptions(contentType))
 	if err != nil {
 		return 0, fmt.Errorf("attach: blob writer: %w", err)
 	}

modules/lagoon/attach/store_test.go

@@ -506,13 +506,31 @@ func TestStoreDiskExtension(t *testing.T) {
 			if err := gdb.First(&stored, f.ID).Error; err != nil {
 				t.Fatal(err)
 			}
-			if stored.DiskName != f.DiskName || stored.FileName != tc.fileName {
+			if stored.DiskName != f.DiskName || stored.FileName != tc.fileName || stored.ContentType != f.ContentType {
 				t.Fatalf("stored row %+v", stored)
 			}
 			got, err := bucket.ReadAll(ctx, attach.BlobKey(f.DiskName))
 			if err != nil || !bytes.Equal(got, tc.body) {
 				t.Fatalf("blob under %s: %v", attach.BlobKey(f.DiskName), err)
 			}
+			attrs, err := bucket.Attributes(ctx, attach.BlobKey(f.DiskName))
+			if err != nil {
+				t.Fatalf("blob attributes: %v", err)
+			}
+			if attrs.ContentType != f.ContentType {
+				t.Fatalf("blob Content-Type %q, want the stored %q", attrs.ContentType, f.ContentType)
+			}
+			// An HTML body uploaded as note.txt keeps text/html as its type
+			// but is stored with an attachment disposition, so a public
+			// bucket makes the browser download it. Only raster images stay
+			// inline.
+			wantDisposition := "attachment"
+			if tc.ct == "image/png" || tc.ct == "image/jpeg" {
+				wantDisposition = ""
+			}
+			if attrs.ContentDisposition != wantDisposition {
+				t.Fatalf("blob Content-Disposition %q, want %q", attrs.ContentDisposition, wantDisposition)
+			}
 		})
 	}
 }

modules/lagoon/attach/thumb_test.go

@@ -128,6 +128,14 @@ func TestFileThumbResizesOnce(t *testing.T) {
 	if url1 != want || url2 != want {
 		t.Fatalf("urls = %q %q, want %q", url1, url2, want)
 	}
+	// Thumbnails stay inline: an image type and no disposition.
+	attrs, err := bucket.Attributes(ctx, strings.TrimPrefix(url1, "/storage/uploads/"))
+	if err != nil {
+		t.Fatalf("thumbnail Attributes: %v", err)
+	}
+	if attrs.ContentType != "image/jpeg" || attrs.ContentDisposition != "" {
+		t.Fatalf("thumbnail attrs = %q %q, want image/jpeg and no disposition", attrs.ContentType, attrs.ContentDisposition)
+	}
 }
 
 func TestFileThumbEncodeFailureDoesNotCache(t *testing.T) {