caa5b5d8ff
- attach.Store returns attach.ErrProtectedNeedsPrivateBucket for an is_public false upload while storage.uploads.public_url is set, before any blob is written - OpenBucket's public_url line becomes a warning naming the refusal
127 lines
10 KiB
Markdown
127 lines
10 KiB
Markdown
---
|
|
title: Storage
|
|
description: Configure the uploads bucket that serve opens, choose file, memory or S3 bucket URLs, serve stored files, and size upload routes.
|
|
section: services
|
|
order: 90
|
|
---
|
|
# Storage
|
|
|
|
WinterCMS stores uploads on a Laravel filesystem disk. SummerCMS stores them in one [gocloud.dev](https://gocloud.dev/howto/blob/) bucket, opened from `storage.uploads.bucket_url` by the `attach` package of [lagoon](../../modules/lagoon/README.md). The `serve` command opens the bucket at start-up, before it accepts requests, and publishes it on the application; an empty `bucket_url` stops the start-up.
|
|
|
|
## Bucket URLs
|
|
|
|
| URL | Stores |
|
|
|-----|--------|
|
|
| `file:///var/lib/acme/uploads` | In a directory on the server. |
|
|
| `mem://` | In memory, for tests. Everything is lost when the process exits. |
|
|
| `s3://acme-uploads?region=eu-central-1` | In an S3 or S3-compatible bucket. The binary must be built with the `summercms_s3` tag; see [S3 and S3-compatible buckets](#s3-and-s3-compatible-buckets). |
|
|
|
|
The keys go in `config/storage.yaml`:
|
|
|
|
```yaml
|
|
uploads:
|
|
bucket_url: file:///var/lib/acme/uploads
|
|
public_path_prefix: /storage/uploads
|
|
```
|
|
|
|
Files are laid out as WinterCMS lays out its uploads disk, so a copy of a WinterCMS `storage/app/uploads/public` directory can serve as the bucket after a port. `public_path_prefix` is the URL prefix that file and thumbnail URLs start with. `attach.PublicURL` builds the URL of any blob key from it, and `attach.File.URL` the URL of an original.
|
|
|
|
### The WinterCMS layout
|
|
|
|
A port whose clients already store or compare upload URLs keeps WinterCMS's URLs too. WinterCMS serves public uploads from `storage/app/uploads/public` under the URL path `/storage/app/uploads/public` (`cms.storage.uploads.path` plus `/public`). Root the bucket at that directory and use the same prefix:
|
|
|
|
```yaml
|
|
uploads:
|
|
bucket_url: file://./storage/app/uploads/public
|
|
public_path_prefix: /storage/app/uploads/public
|
|
```
|
|
|
|
An original then has the URL `/storage/app/uploads/public/<partition>/<disk_name>`, as WinterCMS's `File::getPath()` returns, and a 200 by 200 cropped thumbnail `/storage/app/uploads/public/<partition>/thumb_<id>_200_200_0_0_crop.<ext>`, as `getThumb()` returns. The framework default stays `/storage/uploads`, so only an application that sets these keys changes its URLs.
|
|
|
|
Application code gets the bucket with `app.Lookup[*blob.Bucket]()` and reads and writes it through the `gocloud.dev/blob` API. Model attachments, thumbnails and deleting files after commit are covered in [Attachments](../database/attachments.md).
|
|
|
|
## S3 and S3-compatible buckets
|
|
|
|
The S3 driver (`gocloud.dev/blob/s3blob`) is not linked into every binary, since it brings in the AWS SDK. A binary links it when it is built with the Go build tag `summercms_s3` (`attach.S3BuildTag`). The application never imports the driver itself: list the tag under `build.tags` in `summer.yaml` and `summer build` passes it to `go build`:
|
|
|
|
```yaml
|
|
module: git.golem15.com/acme/acme
|
|
binary: acme
|
|
plugins:
|
|
- id: acme.blog
|
|
module: git.golem15.com/acme/acme/plugins/blog
|
|
build:
|
|
tags:
|
|
- summercms_s3
|
|
```
|
|
|
|
`summer build` prints `build tags: summercms_s3` before it builds, and `summer dev` builds the same way. A build script that runs its own `go build` after `summer build`, for example to cross-compile a release, must repeat the tag: `go build -tags summercms_s3 -o bin/acme .`, or `GOFLAGS=-tags=summercms_s3` in its environment. A binary built without the tag refuses an `s3://` bucket URL at start-up:
|
|
|
|
```text
|
|
attach: storage.uploads.bucket_url uses s3:// but this binary was built without the summercms_s3 build tag; list summercms_s3 under build.tags in summer.yaml and rebuild with summer build (a raw go build needs -tags summercms_s3, or GOFLAGS=-tags=summercms_s3)
|
|
```
|
|
|
|
The bucket is configured only through `bucket_url` and its query parameters, as gocloud.dev defines them; there are no separate `s3.*` keys. For AWS:
|
|
|
|
```yaml
|
|
uploads:
|
|
bucket_url: s3://acme-uploads?region=eu-central-1
|
|
```
|
|
|
|
For an S3-compatible service (MinIO, Cloudflare R2, Hetzner Object Storage and others), add the service's `endpoint`, path-style addressing and, for services that do not support the newer AWS checksums, the two checksum parameters:
|
|
|
|
```yaml
|
|
uploads:
|
|
bucket_url: s3://acme-uploads?region=auto&endpoint=https://storage.example.com&use_path_style=true&request_checksum_calculation=when_required&response_checksum_validation=when_required
|
|
```
|
|
|
|
Credentials come from the standard AWS environment: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, a profile (`AWS_PROFILE`, or `profile=` in the URL) or an instance role. Never put keys in `bucket_url` or in a config file. The bucket opens at start-up without a request to the service, so wrong credentials or a missing bucket show up on the first upload or read.
|
|
|
|
The [Serving files](#serving-files) route reads through the bucket, so it serves an S3 bucket exactly as it serves a directory.
|
|
|
|
## Public URL base
|
|
|
|
By default every file and thumbnail URL starts with `public_path_prefix`, a path on the application's own host. When a public bucket or a CDN serves the files directly, set `storage.uploads.public_url` (environment `SUMMER_STORAGE__UPLOADS__PUBLIC_URL`) to its base URL:
|
|
|
|
```yaml
|
|
uploads:
|
|
bucket_url: s3://acme-uploads?region=eu-central-1
|
|
public_url: https://cdn.example.com/uploads
|
|
```
|
|
|
|
`attach.PublicURL`, `attach.File.URL` and thumbnail URLs then are that base, one slash and the blob key, which keeps the partition layout: `https://cdn.example.com/uploads/5f1/d0c/2e9/5f1d0c2e9a7b4c3d8e6f.jpg`. `attach.PublicURLBase` returns the base. The value must be an absolute `http` or `https` URL with a host and without user info, query or fragment; anything else stops the start-up with an error naming the key. Thumbnails are written to the bucket when they are first requested through `attach.File.Thumb`, so their URLs work behind the base as well.
|
|
|
|
The `/storage/uploads` route checks `is_public` before it serves a file; a public bucket or a CDN serves any key it holds. A bucket behind `public_url` therefore holds public files only: while `public_url` is set, `attach.Store` refuses a protected (`is_public` false) upload with `attach.ErrProtectedNeedsPrivateBucket` before it writes anything. Keep protected relations on an application whose bucket is private, with `public_url` unset, or make the relation public. `serve` logs one warning at start-up naming the host and this rule.
|
|
|
|
Leave `public_url` unset to keep the URLs under `public_path_prefix`. With `storage.uploads.serve` on, those URLs go through the [Serving files](#serving-files) route, which works for an S3 bucket too, and private files stay protected.
|
|
|
|
## Serving files
|
|
|
|
The framework mounts no file route unless you ask for one. Set `storage.uploads.serve` to `true` (environment `SUMMER_STORAGE__UPLOADS__SERVE=true`, default `false`) and the router registers `GET <public_path_prefix>/{path...}` on `attach.StaticHandlerPublic`:
|
|
|
|
```yaml
|
|
storage:
|
|
uploads:
|
|
bucket_url: file://./storage/app/uploads/public
|
|
public_path_prefix: /storage/app/uploads/public
|
|
serve: true
|
|
```
|
|
|
|
The route serves originals and thumbnails and answers 404 for a missing file, an invalid path and a file whose `system_files` row is not public (`is_public` false). The stored content type is the one sniffed from the file's bytes at upload, so an uploaded text file that starts with `<html>` is stored as `text/html`; `X-Content-Type-Options: nosniff` alone does not stop a browser from rendering a file whose declared type is already HTML. The route therefore protects the application origin in three ways:
|
|
|
|
- Every response carries `X-Content-Type-Options: nosniff`.
|
|
- Every response carries `Content-Security-Policy: default-src 'none'; img-src 'self'; media-src 'self'; sandbox`. A file opened as a page runs no script and gets an opaque origin, so it cannot read the application's cookies or storage. The policy does not affect an `<img>` or `<video>` that embeds the file.
|
|
- A file whose type is not a raster image (`image/jpeg`, `image/png`, `image/gif`, `image/webp` or `image/avif`) is sent with `Content-Disposition: attachment`, so the browser downloads it instead of showing it. That includes HTML, SVG, PDF and plain text.
|
|
|
|
With `serve` on, `public_path_prefix` must be an absolute URL path such as `/storage/uploads`: a leading slash, no scheme or host, no query, no empty, `.` or `..` segment and no `{` or `}`. Start-up fails with an error naming the key otherwise, for example for the WinterCMS disk path `storage/app/uploads/public` or a CDN URL. With `serve` off the key only builds file URLs, so a CDN origin is fine there.
|
|
|
|
The route looks up the bucket and the database on each request: `route:list` lists it without opening either, and it answers 404 until `serve` has published them. The route owns the whole prefix, so start-up fails when a plugin registers a route at or under it.
|
|
|
|
Without the key, the application decides where files are served: mount `attach.StaticHandlerPublic` itself, or put a web server or CDN in front of the bucket directory. Either way, serve uploads from a separate origin when you can; the [Attachments](../database/attachments.md) page explains why. The built-in route is meant for development and for deployments that cannot split the origin yet.
|
|
|
|
> **Deploy note.** When a static server such as nginx, or a CDN, serves the uploads directory or bucket directly, it picks the Content-Type from the file extension and sends none of the route's headers. `attach.Store` names each file after its sniffed content (`attach.DiskExtension`), so a PNG uploaded as `x.html` is stored as `.png`, but configure the server as the route behaves anyway: send `X-Content-Type-Options: nosniff` on every upload, and serve only the extensions the application accepts (for example the image and document types in `attach.DefaultFileExtensions`), answering 404 for anything else.
|
|
|
|
## Upload size
|
|
|
|
Every non-raw route has a request body limit of `http.body_limits.default_bytes`. A route that accepts uploads raises its own limit with the `body.limit:<bytes>` middleware; see [Routing](routing.md). `http.body_limits.upload_bytes` is required and validated at start-up, but the framework applies it to no route; a plugin that wants its upload routes to follow it reads it in `Register` and puts the value in the route's `body.limit`.
|