Jakub Zych
1806e5a36d
When a public image's thumb_url or url fails to load, the fileupload field fetches its preview once through the authenticated thumb route (session key in the header) and shows it from an object URL, revoked when the item is forgotten and on unmount. The preview is reported as unavailable only when that fetch or the object-URL image fails too. Protected fields behave as before.
129 lines
10 KiB
Markdown
129 lines
10 KiB
Markdown
---
|
|
title: Storage
|
|
description: Configure the uploads bucket that serve opens, choose file, memory or S3 bucket URLs, serve stored files, and size upload routes.
|
|
section: services
|
|
order: 90
|
|
---
|
|
# Storage
|
|
|
|
WinterCMS stores uploads on a Laravel filesystem disk. SummerCMS stores them in one [gocloud.dev](https://gocloud.dev/howto/blob/) bucket, opened from `storage.uploads.bucket_url` by the `attach` package of [lagoon](../../modules/lagoon/README.md). The `serve` command opens the bucket at start-up, before it accepts requests, and publishes it on the application; an empty `bucket_url` stops the start-up.
|
|
|
|
## Bucket URLs
|
|
|
|
| URL | Stores |
|
|
|-----|--------|
|
|
| `file:///var/lib/acme/uploads` | In a directory on the server. |
|
|
| `mem://` | In memory, for tests. Everything is lost when the process exits. |
|
|
| `s3://acme-uploads?region=eu-central-1` | In an S3 or S3-compatible bucket. The binary must be built with the `summercms_s3` tag; see [S3 and S3-compatible buckets](#s3-and-s3-compatible-buckets). |
|
|
|
|
The keys go in `config/storage.yaml`:
|
|
|
|
```yaml
|
|
uploads:
|
|
bucket_url: file:///var/lib/acme/uploads
|
|
public_path_prefix: /storage/uploads
|
|
```
|
|
|
|
Files are laid out as WinterCMS lays out its uploads disk, so a copy of a WinterCMS `storage/app/uploads/public` directory can serve as the bucket after a port. `public_path_prefix` is the URL prefix that file and thumbnail URLs start with. `attach.PublicURL` builds the URL of any blob key from it, and `attach.File.URL` the URL of an original.
|
|
|
|
### The WinterCMS layout
|
|
|
|
A port whose clients already store or compare upload URLs keeps WinterCMS's URLs too. WinterCMS serves public uploads from `storage/app/uploads/public` under the URL path `/storage/app/uploads/public` (`cms.storage.uploads.path` plus `/public`). Root the bucket at that directory and use the same prefix:
|
|
|
|
```yaml
|
|
uploads:
|
|
bucket_url: file://./storage/app/uploads/public
|
|
public_path_prefix: /storage/app/uploads/public
|
|
```
|
|
|
|
An original then has the URL `/storage/app/uploads/public/<partition>/<disk_name>`, as WinterCMS's `File::getPath()` returns, and a 200 by 200 cropped thumbnail `/storage/app/uploads/public/<partition>/thumb_<id>_200_200_0_0_crop.<ext>`, as `getThumb()` returns. The framework default stays `/storage/uploads`, so only an application that sets these keys changes its URLs.
|
|
|
|
Application code gets the bucket with `app.Lookup[*blob.Bucket]()` and reads and writes it through the `gocloud.dev/blob` API. Model attachments, thumbnails and deleting files after commit are covered in [Attachments](../database/attachments.md).
|
|
|
|
## S3 and S3-compatible buckets
|
|
|
|
The S3 driver (`gocloud.dev/blob/s3blob`) is not linked into every binary, since it brings in the AWS SDK. A binary links it when it is built with the Go build tag `summercms_s3` (`attach.S3BuildTag`). The application never imports the driver itself: list the tag under `build.tags` in `summer.yaml` and `summer build` passes it to `go build`:
|
|
|
|
```yaml
|
|
module: git.golem15.com/acme/acme
|
|
binary: acme
|
|
plugins:
|
|
- id: acme.blog
|
|
module: git.golem15.com/acme/acme/plugins/blog
|
|
build:
|
|
tags:
|
|
- summercms_s3
|
|
```
|
|
|
|
`summer build` prints `build tags: summercms_s3` before it builds, and `summer dev` builds the same way. A build script that runs its own `go build` after `summer build`, for example to cross-compile a release, must repeat the tag: `go build -tags summercms_s3 -o bin/acme .`, or `GOFLAGS=-tags=summercms_s3` in its environment. A binary built without the tag refuses an `s3://` bucket URL at start-up:
|
|
|
|
```text
|
|
attach: storage.uploads.bucket_url uses s3:// but this binary was built without the summercms_s3 build tag; list summercms_s3 under build.tags in summer.yaml and rebuild with summer build (a raw go build needs -tags summercms_s3, or GOFLAGS=-tags=summercms_s3)
|
|
```
|
|
|
|
The bucket is configured only through `bucket_url` and its query parameters, as gocloud.dev defines them; there are no separate `s3.*` keys. For AWS:
|
|
|
|
```yaml
|
|
uploads:
|
|
bucket_url: s3://acme-uploads?region=eu-central-1
|
|
```
|
|
|
|
For an S3-compatible service (MinIO, Cloudflare R2, Hetzner Object Storage and others), add the service's `endpoint`, path-style addressing and, for services that do not support the newer AWS checksums, the two checksum parameters:
|
|
|
|
```yaml
|
|
uploads:
|
|
bucket_url: s3://acme-uploads?region=auto&endpoint=https://storage.example.com&use_path_style=true&request_checksum_calculation=when_required&response_checksum_validation=when_required
|
|
```
|
|
|
|
Credentials come from the standard AWS environment: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, a profile (`AWS_PROFILE`, or `profile=` in the URL) or an instance role. Never put keys in `bucket_url` or in a config file. The bucket opens at start-up without a request to the service, so wrong credentials or a missing bucket show up on the first upload or read.
|
|
|
|
The [Serving files](#serving-files) route reads through the bucket, so it serves an S3 bucket exactly as it serves a directory.
|
|
|
|
## Public URL base
|
|
|
|
By default every file and thumbnail URL starts with `public_path_prefix`, a path on the application's own host. When a public bucket or a CDN serves the files directly, set `storage.uploads.public_url` (environment `SUMMER_STORAGE__UPLOADS__PUBLIC_URL`) to its base URL:
|
|
|
|
```yaml
|
|
uploads:
|
|
bucket_url: s3://acme-uploads?region=eu-central-1
|
|
public_url: https://cdn.example.com/uploads
|
|
```
|
|
|
|
`attach.PublicURL`, `attach.File.URL` and thumbnail URLs then are that base, one slash and the blob key, which keeps the partition layout: `https://cdn.example.com/uploads/5f1/d0c/2e9/5f1d0c2e9a7b4c3d8e6f.jpg`. `attach.PublicURLBase` returns the base. The value must be an absolute `http` or `https` URL with a host and without user info, query or fragment; anything else stops the start-up with an error naming the key. Thumbnails are written to the bucket when they are first requested through `attach.File.Thumb`, so their URLs work behind the base as well.
|
|
|
|
The `/storage/uploads` route checks `is_public` before it serves a file; a public bucket or a CDN serves any key it holds. A bucket behind `public_url` therefore holds public files only: while `public_url` is set, `attach.Store` refuses a protected (`is_public` false) upload with `attach.ErrProtectedNeedsPrivateBucket` before it writes anything. Keep protected relations on an application whose bucket is private, with `public_url` unset, or make the relation public. `serve` logs one warning at start-up naming the host and this rule.
|
|
|
|
Leave `public_url` unset to keep the URLs under `public_path_prefix`. With `storage.uploads.serve` on, those URLs go through the [Serving files](#serving-files) route, which works for an S3 bucket too, and private files stay protected.
|
|
|
|
## Serving files
|
|
|
|
The framework mounts no file route unless you ask for one. Set `storage.uploads.serve` to `true` (environment `SUMMER_STORAGE__UPLOADS__SERVE=true`, default `false`) and the router registers `GET <public_path_prefix>/{path...}` on `attach.StaticHandlerPublic`:
|
|
|
|
```yaml
|
|
storage:
|
|
uploads:
|
|
bucket_url: file://./storage/app/uploads/public
|
|
public_path_prefix: /storage/app/uploads/public
|
|
serve: true
|
|
```
|
|
|
|
The route serves originals and thumbnails and answers 404 for a missing file, an invalid path and a file whose `system_files` row is not public (`is_public` false). The stored content type is the one sniffed from the file's bytes at upload, so an uploaded text file that starts with `<html>` is stored as `text/html`; `X-Content-Type-Options: nosniff` alone does not stop a browser from rendering a file whose declared type is already HTML. The route therefore protects the application origin in three ways:
|
|
|
|
- Every response carries `X-Content-Type-Options: nosniff`.
|
|
- Every response carries `Content-Security-Policy: default-src 'none'; img-src 'self'; media-src 'self'; sandbox`. A file opened as a page runs no script and gets an opaque origin, so it cannot read the application's cookies or storage. The policy does not affect an `<img>` or `<video>` that embeds the file.
|
|
- A file whose type is not a raster image (`image/jpeg`, `image/png`, `image/gif`, `image/webp` or `image/avif`) is sent with `Content-Disposition: attachment`, so the browser downloads it instead of showing it. That includes HTML, SVG, PDF and plain text.
|
|
|
|
With `serve` on, `public_path_prefix` must be an absolute URL path such as `/storage/uploads`: a leading slash, no scheme or host, no query, no empty, `.` or `..` segment and no `{` or `}`. Start-up fails with an error naming the key otherwise, for example for the WinterCMS disk path `storage/app/uploads/public` or a CDN URL. With `serve` off the key only builds file URLs, so a CDN origin is fine there.
|
|
|
|
Admin fileupload previews do not depend on this route: when a public image's URL does not load, the admin falls back to its authenticated thumb route (see [File uploads](../backend/forms.md#file-uploads)).
|
|
|
|
The route looks up the bucket and the database on each request: `route:list` lists it without opening either, and it answers 404 until `serve` has published them. The route owns the whole prefix, so start-up fails when a plugin registers a route at or under it.
|
|
|
|
Without the key, the application decides where files are served: mount `attach.StaticHandlerPublic` itself, or put a web server or CDN in front of the bucket directory. Either way, serve uploads from a separate origin when you can; the [Attachments](../database/attachments.md) page explains why. The built-in route is meant for development and for deployments that cannot split the origin yet.
|
|
|
|
> **Deploy note.** When a static server such as nginx, or a CDN, serves the uploads directory or bucket directly, it picks the Content-Type from the file extension and sends none of the route's headers. `attach.Store` names each file after its sniffed content (`attach.DiskExtension`), so a PNG uploaded as `x.html` is stored as `.png`, but configure the server as the route behaves anyway: send `X-Content-Type-Options: nosniff` on every upload, and serve only the extensions the application accepts (for example the image and document types in `attach.DefaultFileExtensions`), answering 404 for anything else.
|
|
|
|
## Upload size
|
|
|
|
Every non-raw route has a request body limit of `http.body_limits.default_bytes`. A route that accepts uploads raises its own limit with the `body.limit:<bytes>` middleware; see [Routing](routing.md). `http.body_limits.upload_bytes` is required and validated at start-up, but the framework applies it to no route; a plugin that wants its upload routes to follow it reads it in `Register` and puts the value in the route's `body.limit`.
|