feat(19.3-03): summer.yaml build.tags and the summercms_s3 tag that links the S3 uploads driver

- Manifest.Tags from build.tags (^[a-z0-9_]+$, duplicates dropped), round-tripped by SaveManifest
- summer build passes -tags=<list> through goBuildArgs and prints the build tags line
- attach: s3blob imported only by s3_on.go (//go:build summercms_s3); an untagged binary
  refuses an s3:// bucket_url naming the tag, build.tags and -tags
- go mod tidy adds the AWS SDK v2 modules as indirect requirements
- storage, modules, scaffolding, configuration and scaling docs plus the lagoon README
This commit is contained in:
Jakub Zych
2026-10-15 18:07:05 +02:00
parent b1fab9930b
commit 2382040b25
15 changed files with 346 additions and 18 deletions

View File

@@ -10,7 +10,7 @@ Postgres data layer: the shared GORM connection, per-plugin migrations, model he
## Overview
`lagoon` is the WinterCMS models and migrations counterpart: it opens the one `*sql.DB` pool (pgx stdlib driver) that GORM and the rest of the application share, runs each plugin's gormigrate set in its own history table, and ports the Eloquent model conventions that WinterCMS plugins rely on, such as `$fillable`, `$hidden`, `$jsonable`, encrypted casts and Laravel-style validation rules. Its `attach` subpackage ports WinterCMS's `system_files` attachments, storing originals and lazily generated thumbnails in a gocloud.dev blob bucket. The application binary gets the migrate and key commands from `lagoon.RuntimeCommands`.
`lagoon` is the WinterCMS models and migrations counterpart: it opens the one `*sql.DB` pool (pgx stdlib driver) that GORM and the rest of the application share, runs each plugin's gormigrate set in its own history table, and ports the Eloquent model conventions that WinterCMS plugins rely on, such as `$fillable`, `$hidden`, `$jsonable`, encrypted casts and Laravel-style validation rules. Its `attach` subpackage ports WinterCMS's `system_files` attachments, storing originals and lazily generated thumbnails in a gocloud.dev blob bucket (a directory, memory, or S3 in binaries built with the `summercms_s3` tag). The application binary gets the migrate and key commands from `lagoon.RuntimeCommands`.
## Role in the framework
@@ -38,7 +38,7 @@ Use cases:
- Lifecycle and relations: hook interfaces matching GORM's native method names (`lagoon.HasBeforeCreate`, `lagoon.HasBeforeSave`, `lagoon.HasBeforeDelete`, `lagoon.HasAfterDelete`) plus `lagoon.HasBeforeValidate`; `lagoon.WithSoftDeleteCascade` runs a cascade inside the parent delete; `lagoon.RegisterJoinTable` wires pivot models with business columns.
- Deferred binding: WinterCMS's `deferred_bindings` table holds the uploads and related-record changes of a form whose record is not saved yet. Every operation takes a `lagoon.DeferredKey` (the form's session key, the owning backend admin's id and the master record's morph type from `lagoon.MorphType`) and never reads or changes another admin's rows, since each row stores `backend_user_id`. `lagoon.DeferredBind` and `lagoon.DeferredUnbind` port WinterCMS's duplicate and cancel rules: a repeated bind writes nothing, and an unbind of a slave with a pending bind deletes that bind and returns it so the caller can remove what it created. `lagoon.DeferredBindings` reads and locks a session's bindings for the save that commits them, `lagoon.DeferredForget` deletes them once applied, and `lagoon.DeferredSlaves` is the subquery a list uses to include pending rows. A child created under deferral carries the `lagoon.DeferredEnvelope` (`{"created":true,"pivot":{...},"upload_id":"..."}`) in `pivot_data`. `lagoon.PurgeDeferred` removes expired bindings: it deletes an unattached `system_files` row a bind points at, and its blobs only after the commit, deletes a child only when its binding carries the created envelope, keeps records that were only linked, and locks each batch with `FOR UPDATE SKIP LOCKED`.
- Imports from Laravel: `lagoon.DecryptLaravelPayload` decrypts Laravel `encrypted` payloads with the old application key, for one-off data imports.
- Attachments (`attach`): the `attach.File` model for `system_files` rows, WinterCMS-compatible partitioned storage keys (`attach.BlobKey`, `attach.PartitionDirectory`), public URLs (`attach.PublicURL` for any key, `attach.File.URL` for an original, matching WinterCMS's `File::getPath()` under the WinterCMS layout), on-demand thumbnails through `attach.File.Thumb` for JPEG, PNG, GIF and WebP originals (a WebP original's thumbnail is JPEG bytes under its `.webp` name, since WebP cannot be encoded; a missing, undecodable or oversized original gets WinterCMS's broken-image picture, `attach.BrokenImagePNG`, as its thumbnail, as `File::makeThumb` does), storing uploads through `attach.Store` (a server-generated disk name, an extension allow-list with `attach.DefaultImageExtensions` and `attach.DefaultFileExtensions` as defaults, a MIME filter, a size limit enforced while streaming and, in image mode, the `attach.IsAllowedImage` content guard), attachment relation declarations (`attach.Relation`, `attach.HasRelations`), static serving with an optional `is_public` gate (`attach.StaticHandlerPublic`), and a two-phase delete that removes blobs only after the database transaction commits (`attach.DeleteForOwner`, `attach.DeleteKeys`).
- Attachments (`attach`): the `attach.File` model for `system_files` rows, WinterCMS-compatible partitioned storage keys (`attach.BlobKey`, `attach.PartitionDirectory`), an S3 bucket option behind the `attach.S3BuildTag` build tag (an untagged binary refuses an `s3://` bucket URL at boot with an error naming the tag), public URLs (`attach.PublicURL` for any key, `attach.File.URL` for an original, matching WinterCMS's `File::getPath()` under the WinterCMS layout), on-demand thumbnails through `attach.File.Thumb` for JPEG, PNG, GIF and WebP originals (a WebP original's thumbnail is JPEG bytes under its `.webp` name, since WebP cannot be encoded; a missing, undecodable or oversized original gets WinterCMS's broken-image picture, `attach.BrokenImagePNG`, as its thumbnail, as `File::makeThumb` does), storing uploads through `attach.Store` (a server-generated disk name, an extension allow-list with `attach.DefaultImageExtensions` and `attach.DefaultFileExtensions` as defaults, a MIME filter, a size limit enforced while streaming and, in image mode, the `attach.IsAllowedImage` content guard), attachment relation declarations (`attach.Relation`, `attach.HasRelations`), static serving with an optional `is_public` gate (`attach.StaticHandlerPublic`), and a two-phase delete that removes blobs only after the database transaction commits (`attach.DeleteForOwner`, `attach.DeleteKeys`).
## Usage
@@ -194,7 +194,8 @@ func (p *Plugin) Migrations() []*gormigrate.Migration {
| `attach.PublicURL` | Public URL of a blob key under `storage.uploads.public_path_prefix`. |
| `attach.BrokenImagePNG` | WinterCMS's broken-image picture, stored as the thumbnail of an unusable original. |
| `attach.Owner` | Implemented by models that own attachments; returns the stored morph type name. |
| `attach.OpenBucket` | Opens the uploads bucket from config. |
| `attach.OpenBucket` | Opens the uploads bucket from config; refuses `s3://` in a binary built without `attach.S3BuildTag`. |
| `attach.S3BuildTag` | The build tag, `summercms_s3`, that links the S3 driver; list it under `build.tags` in `summer.yaml`. |
| `attach.Publish` | Stores the bucket on the `backpack.App`. |
| `attach.StaticHandler` | Serves stored files and thumbnails under a URL prefix. |
| `attach.StaticHandlerPublic` | `attach.StaticHandler` plus a 404 for rows that are not public. |
@@ -227,7 +228,7 @@ Keys are read from the compass config; the `SUMMER_` environment overlay maps a
| `database.dsn` | none (required) | Postgres connection string used by `lagoon.OpenFromApp` and every database command. |
| `app.key` | none (required) | Base64 encoding of 32 random bytes; the source of the `lagoon.Encrypted` column key. Generate one with `key:generate`. |
| `app.previous_keys` | empty | List of earlier base64 keys still accepted when decrypting, for key rotation. |
| `storage.uploads.bucket_url` | none (required by `attach.OpenBucket`) | Uploads bucket URL, `file://` or `mem://`. |
| `storage.uploads.bucket_url` | none (required by `attach.OpenBucket`) | Uploads bucket URL, `file://`, `mem://` or `s3://` (with `attach.S3BuildTag`; query parameters such as `region`, `endpoint` and `use_path_style` per gocloud.dev, credentials from the AWS environment). |
| `storage.uploads.public_path_prefix` | `/storage/uploads` | URL prefix used when building public file and thumbnail URLs. |
| `database.deferred_bindings.purge_days` | `5` | Age in days after which `deferred:purge` removes a deferred binding, WinterCMS's `cleanUp(5)`; the `--days` flag overrides it. |
| `database.deferred_bindings.purge_at` | `03:00` | Daily `HH:MM` (in `app.timezone`) at which the scheduler runs `deferred:purge`; an empty value disables the framework entry. |
@@ -258,7 +259,7 @@ storage:
- SummerCMS modules: [backpack](../backpack/README.md), [bonfire](../bonfire/README.md), [compass](../compass/README.md), [pact](../pact/README.md), [party](../party/README.md), [phrasebook](../phrasebook/README.md) (validation messages).
- Third-party: `gorm.io/gorm`, `gorm.io/driver/postgres`, `github.com/jackc/pgx/v5` (stdlib driver), `github.com/go-gormigrate/gormigrate/v2`, `github.com/go-playground/validator/v10`, `github.com/riverqueue/river` (its `rivermigrate` and `riverdriver/riverdatabasesql` packages, for the River schema in `lagoon.QueueMigrations`).
- Third-party, `attach` only: `gocloud.dev/blob` (file and memory drivers), `github.com/disintegration/imaging` (thumbnails), `golang.org/x/image/webp` (WebP decoding).
- Third-party, `attach` only: `gocloud.dev/blob` (file and memory drivers; the S3 driver `gocloud.dev/blob/s3blob` and the AWS SDK v2 only under the `summercms_s3` build tag), `github.com/disintegration/imaging` (thumbnails), `golang.org/x/image/webp` (WebP decoding).
- Standard library: `database/sql`, `crypto/aes`, `crypto/cipher`, `crypto/hkdf`, `log/slog`, `image`, among others.
## Testing

View File

@@ -16,6 +16,11 @@ import (
const defaultPublicPathPrefix = "/storage/uploads"
// S3BuildTag is the Go build tag that links the gocloud S3 driver
// (s3blob, imported only by s3_on.go). List it under build.tags in summer.yaml so
// summer build passes it to go build; a raw go build needs -tags summercms_s3.
const S3BuildTag = "summercms_s3"
var (
prefixMu sync.RWMutex
publicPathPrefix = defaultPublicPathPrefix
@@ -57,8 +62,24 @@ func normalizeBucketURL(raw string) (string, error) {
}
}
// OpenBucket opens storage.uploads.bucket_url (file:// or mem://) and
// records storage.uploads.public_path_prefix. An empty bucket_url fails boot.
// checkS3Driver refuses an s3:// bucket URL when the binary was built
// without S3BuildTag, naming the tag and where to set it.
func checkS3Driver(bucketURL string) error {
if s3Enabled {
return nil
}
scheme, _, ok := strings.Cut(bucketURL, "://")
if !ok || !strings.EqualFold(scheme, "s3") {
return nil
}
return fmt.Errorf("attach: storage.uploads.bucket_url uses s3:// but this binary was built without the %s build tag; "+
"list %s under build.tags in summer.yaml and rebuild with summer build (a raw go build needs -tags %s, or GOFLAGS=-tags=%s)",
S3BuildTag, S3BuildTag, S3BuildTag, S3BuildTag)
}
// OpenBucket opens storage.uploads.bucket_url (file://, mem://, or s3:// in a
// binary built with S3BuildTag) and records
// storage.uploads.public_path_prefix. An empty bucket_url fails boot.
func OpenBucket(ctx context.Context, cfg *compass.Config) (*blob.Bucket, error) {
if cfg == nil {
return nil, fmt.Errorf("attach: config is nil")
@@ -71,6 +92,9 @@ func OpenBucket(ctx context.Context, cfg *compass.Config) (*blob.Bucket, error)
if err != nil {
return nil, err
}
if err := checkS3Driver(bucketURL); err != nil {
return nil, err
}
bucket, err := blob.OpenBucket(ctx, bucketURL)
if err != nil {
return nil, fmt.Errorf("attach: open bucket %q: %w", raw, err)

View File

@@ -0,0 +1,60 @@
package attach
import (
"os"
"path/filepath"
"strings"
"testing"
"git.golem15.com/golem15/summercms/modules/compass"
)
// openTestConfig writes storage.yaml into a temp config dir and opens it.
func openTestConfig(t *testing.T, storageYAML string) *compass.Config {
t.Helper()
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(dir, "storage.yaml"), []byte(storageYAML), 0o644); err != nil {
t.Fatal(err)
}
cfg, err := compass.Open(compass.Options{
Dir: dir,
Env: "development",
Environ: []string{"SUMMER_ENV=development"},
})
if err != nil {
t.Fatal(err)
}
return cfg
}
// TestOpenBucketS3Smoke runs in both tag modes: without summercms_s3 an
// s3:// bucket_url fails boot naming the tag, with it the bucket opens
// (s3blob connects lazily, so no network is needed).
func TestOpenBucketS3Smoke(t *testing.T) {
t.Setenv("AWS_ACCESS_KEY_ID", "smoke-access-key")
t.Setenv("AWS_SECRET_ACCESS_KEY", "smoke-secret-key")
t.Setenv("AWS_REGION", "us-east-1")
t.Setenv("AWS_EC2_METADATA_DISABLED", "true")
t.Cleanup(func() { setPublicPathPrefix(defaultPublicPathPrefix) })
cfg := openTestConfig(t, "uploads:\n bucket_url: \"s3://smoke-bucket?region=us-east-1\"\n")
bucket, err := OpenBucket(t.Context(), cfg)
if !s3Enabled {
if err == nil {
_ = bucket.Close()
t.Fatal("OpenBucket opened s3:// without the summercms_s3 tag")
}
for _, want := range []string{S3BuildTag, "build.tags", "-tags " + S3BuildTag} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("error %q does not name %q", err, want)
}
}
return
}
if err != nil {
t.Fatalf("OpenBucket with the summercms_s3 tag: %v", err)
}
if err := bucket.Close(); err != nil {
t.Fatalf("close: %v", err)
}
}

View File

@@ -0,0 +1,7 @@
//go:build !summercms_s3
package attach
// Without the summercms_s3 tag the S3 driver is not linked; OpenBucket refuses
// an s3:// bucket_url with an error naming the tag.
const s3Enabled = false

View File

@@ -0,0 +1,9 @@
//go:build summercms_s3
package attach
// Binaries built with the summercms_s3 tag (summer.yaml build.tags) link the
// gocloud S3 driver, so storage.uploads.bucket_url may use s3://.
import _ "gocloud.dev/blob/s3blob"
const s3Enabled = true