golem15/summercms
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(11.1-04): add the Backend section, the remaining Services pages and the concept map links

Browse Source 
- docs/backend: admin controllers, forms, lists and filters, relation
  manager, users and permissions, settings, partials and widgets, admin SPA
- docs/services: storage, outbound HTTP, realtime, Web Push, search, parity
  testing and the Frontend and AJAX (not provided) page
- Examples for cabana (with testdata/docs YAML), fetchguard, lighthouse and
  its centrifugo driver, flare, beachcomber and typesense, tide; lighthouse
  and beachcomber TestDocs* regions run on their Postgres harnesses
- concept map rows link their guide pages and the not-provided rows the
  Frontend and AJAX page; index lists Backend, Database and Services
- TestDocsRequiredPages asserts the D-08 section order
This commit is contained in:
Jakub Zych
2026-09-30 23:18:35 +02:00
parent f7dfe68707
commit 44bd1446f5
40 changed files with 2882 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/services/authentication.md
View File

@@ -125,4 +125,4 @@ fmt.Println(bouncer.NeedsRehash(hash, 12))
Rehash a password on the next successful login when `bouncer.NeedsRehash` reports true.
Admin sign-in, admin permissions and the admin user commands are covered in the Backend section.
Admin sign-in, admin permissions and the admin user commands are covered in [Users and permissions](../backend/users-and-permissions.md).

2
docs/services/configuration.md
View File

@@ -91,7 +91,7 @@ The application opens its configuration once with `compass.Load("config")` in th
`compass.Config.Set` changes a value in memory. `compass.Config.Persist` saves every value set at runtime to `config/env/<environment>/overrides.yaml`, keeping the keys already saved there. It replaces the file atomically, creates it readable only by its owner and refuses any path outside the config directory. `compass.Config.Reload` rereads every source and discards values that were set but not persisted.
Values that admins edit in the backend are not configuration: settings pages store them in a database row.
Values that admins edit in the backend are not configuration: settings pages store them in a database row; see [Settings](../backend/settings.md).
> [!WARNING]
> `overrides.yaml` is written by the running application. Keep it out of version control, and keep secrets in environment variables rather than in values the application persists.

37
docs/services/frontend-and-ajax.md Normal file
View File

@@ -0,0 +1,37 @@
---
title: Frontend and AJAX (not provided)
description: SummerCMS is headless, so CMS pages, themes, components, the AJAX framework and Snowboard are not provided; build the frontend as a separate application.
section: services
order: 160
---
# Frontend and AJAX (not provided)
WinterCMS renders its frontend on the server: CMS pages and layouts in a theme, partials, components that plugins attach to pages, and the AJAX framework with Snowboard for handlers such as `onSave` that update parts of a page without a reload. SummerCMS provides none of these. It is headless: it serves a JSON API, realtime channels and the admin SPA, and the frontend is a separate application that talks to it.
## What is not provided
| WinterCMS | In SummerCMS |
|-----------|--------------|
| CMS pages, layouts and partials in `themes/` | Not provided. The frontend application renders every page. |
| Themes and the theme customisation form | Not provided. |
| Components and `componentDetails`, `defineProperties`, `onRun` | Not provided. Expose the data a component loaded as a JSON route. |
| The AJAX framework (`data-request`, `$this->page`, AJAX handlers) | Not provided. Call JSON routes with the frontend's own HTTP client. |
| Snowboard and its plugins | Not provided. |
| Twig and the Twig filters and functions | Not provided. |
| Sessions and flash messages | Not provided. The API is stateless and authenticates each request with a token. |
A WinterCMS plugin that shipped components and AJAX handlers is ported as routes: each component's data loading and each handler becomes a JSON endpoint declared through `pact.HasRoutes`.
## Building the frontend
Build the frontend with any framework that can call a JSON API, as its own project with its own build and deployment:
- **Data:** call the plugins' JSON routes. [Routing](routing.md) shows how routes, auth groups and JSON responses are declared, and [Queries and pagination](../database/queries-and-pagination.md) the list envelope.
- **Signing in:** the user plugin issues JWTs; send them as a bearer token or in the cookie the guard reads. See [Authentication](authentication.md).
- **Live updates:** instead of polling an AJAX handler, subscribe to realtime channels. The frontend connects to Centrifugo with a token from the token route and receives model broadcasts and explicit events. See [Realtime](realtime.md).
- **Cross-origin calls:** when the frontend runs on another origin, allow it in `http.cors`, as [Routing](routing.md) describes.
- **Push notifications:** see [Web Push](push.md).
The admin is the one frontend SummerCMS ships. It is a single-page app built the same way, against the admin API; see [Admin SPA](../backend/admin-spa.md).
The full map of what carries over from WinterCMS, and what does not, is on [Coming from WinterCMS](../setup/coming-from-wintercms.md).

72
docs/services/outbound-http.md Normal file
View File

@@ -0,0 +1,72 @@
---
title: Outbound HTTP
description: Fetch URLs that users or third parties supply through fetchguard, which allows HTTPS only, blocks private addresses at dial time and limits size and time.
section: services
order: 100
---
# Outbound HTTP
A WinterCMS plugin fetches a remote URL with the Laravel HTTP client or Guzzle, and checks the URL by hand when it came from a user. When a URL comes from outside the application, such as a remote image address, fetch it with [fetchguard](../../modules/fetchguard/README.md). It is the framework's guard against server-side request forgery: a request that a user can aim at the application's own network, a cloud metadata service or an internal admin panel.
For calls to services the application itself chose, such as a payment provider's API, the standard `net/http` client is fine.
## Policies
Every call takes a `fetchguard.Policy`:
- `fetchguard.AllowHostsMode` allows only the hosts in `AllowHosts`, matched exactly or as a dotted suffix.
- `fetchguard.PublicOnlyMode` allows any public host.
In both modes only `https` is allowed, and the resolved IP address is checked when the connection is dialled, so a DNS name that resolves into the network is refused too. The check covers private, loopback, link-local, carrier-grade NAT, documentation, multicast and other reserved IPv4 and IPv6 ranges, including IPv4 addresses inside NAT64 and 6to4 addresses. Environment proxy settings are ignored, so the check always sees the real target.
```go src=modules/fetchguard/example_test.go#ExampleFetch
ctx := context.Background()
// Only the application's image host, at most 5 MiB within 5 seconds.
images := fetchguard.Policy{
Mode: fetchguard.AllowHostsMode,
AllowHosts: []string{"images.example.com"},
MaxBytes: 5 << 20,
Timeout: 5 * time.Second,
}
// Any public host, for a URL a user pasted.
public := fetchguard.Policy{Mode: fetchguard.PublicOnlyMode}
for _, c := range []struct {
url string
policy fetchguard.Policy
}{
{"http://images.example.com/cover.jpg", images},
{"https://cdn.attacker.example/cover.jpg", images},
{"https://127.0.0.1/admin", public},
{"https://169.254.169.254/latest/meta-data/", public},
{"https://[::ffff:10.0.0.1]/", public},
{"https://%zz", public},
} {
// The last argument is the application's config (app.Config), for
// limits the policy leaves at zero; nil uses the framework defaults.
_, err := fetchguard.Fetch(ctx, c.url, c.policy, nil)
var fe *fetchguard.Error
if errors.As(err, &fe) {
fmt.Println(fe.Reason, c.url)
}
}
fmt.Println(fetchguard.Defaults())
// Output:
// scheme http://images.example.com/cover.jpg
// invalid_url https://cdn.attacker.example/cover.jpg
// private_ip https://127.0.0.1/admin
// private_ip https://169.254.169.254/latest/meta-data/
// private_ip https://[::ffff:10.0.0.1]/
// invalid_url https://%zz
// 10485760 10s
```
A failure is always a `fetchguard.Error` with one `fetchguard.Reason` from a closed set, so a handler can map it to a stable API error code. A host outside the allow list is reported as `invalid_url`, as the example shows.
## Responses and limits
`fetchguard.Fetch` returns a `fetchguard.Result` with the body, the content type and the status code for any response the server completed, including 4xx and 5xx. Check the status yourself.
Redirects are never followed: a 3xx response is returned as a result. To follow it, call `fetchguard.Fetch` again with the `Location` URL, which runs every check again.
The body is capped at the policy's `MaxBytes` (a larger body is `fetchguard.ReasonTooLarge`) and the call at its `Timeout`. A limit left at zero falls back to `http.fetch.max_bytes` and `http.fetch.timeout_seconds` from the configuration you pass, then to the framework defaults of 10 MiB and 10 seconds (`fetchguard.Defaults`). A configured value of zero or less is an error, not a way to turn a limit off.

105
docs/services/parity-testing.md Normal file
View File

@@ -0,0 +1,105 @@
---
title: Parity testing
description: Record the reference backend's responses and broadcasts with tide, replay them against the Go port and diff them after masking IDs and timestamps.
section: services
order: 150
---
# Parity testing
When a plugin is ported from WinterCMS, its existing clients define the contract: the Go port must answer every request the way the PHP backend did. [tide](../../modules/tide/README.md) turns that rule into tests. It records the reference backend's real responses as YAML fixtures, replays the same requests against the port, and diffs the responses after masking the values that legitimately differ. WinterCMS has no counterpart.
## Flows
A fixture is a `tide.Flow`: a versioned, ordered list of steps, each a request with its recorded response. You write a spec, a flow with requests only:
```yaml src=modules/tide/testdata/docs/posts-spec.yaml
version: 1
name: blog-posts
description: List the posts of one blog
steps:
- id: list-posts
route_id: GET /api/blog/posts
request:
method: GET
path: /api/blog/posts?page=1
```
Recording sends each request to the reference backend and fills in the responses. Replaying sends them to the port and compares status, a fixed set of contract headers and the body. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies byte for byte:
```go src=modules/tide/example_test.go#ExampleReplayFlow
ctx := context.Background()
// The reference (PHP) backend and two Go ports. IDs and *_at timestamps
// legitimately differ; the second port changed a title.
reference := backend(`{"data":[{"id":12,"title":"Hello","created_at":"2026-09-30T10:00:00+00:00"}]}`)
defer reference.Close()
port := backend(`{"data":[{"id":3,"title":"Hello","created_at":"2026-10-01T08:30:00+00:00"}]}`)
defer port.Close()
broken := backend(`{"data":[{"id":3,"title":"hello","created_at":"2026-10-01T08:30:00+00:00"}]}`)
defer broken.Close()
raw, err := os.ReadFile("testdata/docs/posts-spec.yaml")
if err != nil {
fmt.Println(err)
return
}
spec, err := tide.ParseFlow(raw)
if err != nil {
fmt.Println(err)
return
}
// Record the reference once; the flow is what testdata/parity keeps.
flow, err := tide.RecordFlow(ctx, spec, tide.RecordConfig{Target: reference.URL})
if err != nil {
fmt.Println(err)
return
}
for _, target := range []string{port.URL, broken.URL} {
res, err := tide.ReplayFlow(ctx, flow, tide.ReplayConfig{Target: target})
var mismatch *tide.MismatchError
if errors.As(err, &mismatch) {
res = mismatch.Result // a difference is an error carrying the result
} else if err != nil {
fmt.Println(err)
return
}
fmt.Println("ok:", res.OK)
for _, step := range res.Steps {
for _, d := range step.Diffs {
fmt.Printf(" %s %s: want %s, got %s\n", step.ID, d.Path, d.Expected, d.Actual)
}
}
}
// Output:
// ok: true
// ok: false
// list-posts $.data[0].title: want "Hello", got "hello"
```
The first port returns other IDs and timestamps and passes; the second changed a title and fails with the JSON path of the difference. A difference makes `tide.ReplayFlow` return a `tide.MismatchError` that carries the full `tide.Result`.
## The parity commands
The `summer` CLI wraps tide. Run the reference backend and the port on loopback addresses:
```sh
summer parity:record --spec testdata/parity/posts.spec.yaml --target http://127.0.0.1:8000 --output testdata/parity/posts.yaml --vars /tmp/parity/vars.yaml
summer parity:replay --fixtures testdata/parity --target http://127.0.0.1:8080 --vars /tmp/parity/vars.yaml
```
`parity:proxy` records a real client instead: it runs a reverse proxy in front of the reference backend, and each named session of traffic becomes one fixture. Point the existing frontend at the proxy and click through a feature. The proxy binds to and forwards to loopback addresses only.
Values captured during a flow, such as tokens and created IDs, live in a variables file (`--vars`) readable only by its owner, and fixtures refer to them as `{{name}}` placeholders. Recording refuses to write a fixture that still holds a token- or password-shaped value, so credentials do not end up in committed fixtures. Keep the variables file outside the repository.
A route manifest (`tide.Manifest`) lists a plugin's routes with their auth groups, status and cases; `parity:record --manifest` records the missing cases in batches, and `parity:replay --manifest` reports coverage. The [Console utilities](../console/utilities.md) page lists every flag.
## Broadcast goldens
Realtime side effects are part of the contract too. `summer parity:broadcasts` runs a flow against the reference backend while a fake Centrifugo server (`tide.NewCentrifugoRecorder`) records the publications the backend sends, and writes them to a golden file:
```sh
summer parity:broadcasts --flow testdata/broadcasts/flows/post-lifecycle.yaml --step delete --name deleted --target http://127.0.0.1:8000 --vars /tmp/parity/vars.yaml --out testdata/broadcasts/deleted.yaml
```
Point the reference backend's Centrifugo API URL at the recorder (`127.0.0.1:8424` by default). `--step` keeps only the publications of one step, running the earlier steps as setup. Timestamps, the actor and captured IDs are masked (`tide.NormalizePublications`), so the Go port's publications, recorded the same way, compare with `tide.DiffPublications`. A golden with `--pending` set is recorded but not yet asserted.
On the Go side, the memory realtime driver records publications the same way in tests; see [Realtime](realtime.md).

137
docs/services/push.md Normal file
View File

@@ -0,0 +1,137 @@
---
title: Web Push
description: Send browser push notifications with flare over VAPID, only to https push service hosts on push.allowed_hosts and without redirects, and manage VAPID keys.
section: services
order: 130
---
# Web Push
[flare](../../modules/flare/README.md) sends browser push notifications. Push is a different channel from realtime: realtime reaches pages that hold an open connection, while a push goes to the browser vendor's push service, which wakes the browser even when no page is open. flare is written on the standard library: the RFC 8291 payload encryption and the RFC 8292 VAPID authorization are implemented in the package, with no Web Push library.
## Subscriptions belong to the application
When a browser subscribes, the frontend posts its `PushSubscription` (the endpoint URL and the `p256dh` and `auth` keys) to an application route, and the application stores it in its own table. flare never reads the database. Code that sends a push passes a `flare.Subscription` to the `flare.Pusher` that `flare.From` returns through `flare.Service.Pusher`.
For the operator commands below, the application also publishes a `flare.SubscriptionSource` on the app, which reads a user's stored subscriptions.
## Sending
`flare.Pusher.Send` encrypts the payload for the subscriber and posts it to the endpoint with the VAPID `Authorization` header. `flare.SendOptions` sets the `TTL` (default `push.ttl`), `Urgency` and `Topic` headers:
```go src=modules/flare/example_test.go#ExampleVAPIDPusher_Send
// A stand-in push service: 201 for a live subscription, 410 for one the
// browser dropped.
push := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/gone" {
w.WriteHeader(http.StatusGone)
return
}
fmt.Println("push service got", r.Header.Get("Content-Encoding"), r.Header.Get("TTL"), r.Header.Get("Urgency"))
w.WriteHeader(http.StatusCreated)
}))
defer push.Close()
keys, err := flare.GenerateVAPIDKeys() // websockets:generate-vapid-keys
if err != nil {
fmt.Println(err)
return
}
cfg := flare.Config{
Enabled: true,
PublicKey: keys.PublicKey,
PrivateKey: keys.PrivateKey,
Subject: "mailto:admin@example.com",
TTL: time.Hour,
AllowedHosts: []string{"127.0.0.1"}, // production keeps the default push services
}
// The test server's client trusts its certificate.
pusher := flare.NewVAPIDPusher(cfg, push.Client())
ctx := context.Background()
payload := []byte(`{"title":"New comment","body":"Someone replied to your post"}`)
for _, endpoint := range []string{
push.URL + "/live",
push.URL + "/gone",
"https://push.attacker.example/steal",
"http://127.0.0.1/plain",
} {
sub, err := browserSubscription(endpoint)
if err != nil {
fmt.Println(err)
return
}
err = pusher.Send(ctx, sub, payload, flare.SendOptions{Urgency: "normal"})
switch {
case err == nil:
fmt.Println("sent")
case errors.Is(err, flare.ErrSubscriptionGone):
fmt.Println("gone: delete the subscription")
case errors.Is(err, flare.ErrEndpointNotAllowed):
fmt.Println("refused before connecting")
default:
fmt.Println("error:", err)
}
}
// Formatting the keys never prints the private key.
fmt.Println(strings.Contains(fmt.Sprintf("%v %#v", keys, keys), keys.PrivateKey))
// Output:
// push service got aes128gcm 3600 normal
// sent
// gone: delete the subscription
// refused before connecting
// refused before connecting
// false
```
- A 2xx answer is success.
- 404 and 410 return `flare.ErrSubscriptionGone`: the browser unsubscribed, so delete the stored subscription.
- Any other status returns a `flare.StatusError` with the code, never the response body.
- A payload over `flare.MaxPayloadSize` (3993 bytes) returns `flare.ErrPayloadTooLarge`.
- While `push.enabled` is false, nothing is sent and `flare.ErrPushDisabled` is returned.
A send is one HTTP request with a 10-second timeout. Send from a queued job when a request would otherwise wait for it; see [Queued jobs](jobs.md).
## Endpoint safety
Endpoints come from browsers, so they are untrusted URLs. flare sends only to `https` endpoints whose host is on `push.allowed_hosts`, checks this before it opens a connection, and never follows a redirect, so a push service cannot bounce the request to another host. A refused endpoint returns `flare.ErrEndpointNotAllowed`, which names the host but never the endpoint path.
The default allowlist, `flare.DefaultAllowedHosts`, covers Firebase Cloud Messaging, Mozilla autopush, Apple and Windows push. `*.example.com` matches any subdomain but not `example.com` itself:
```go src=modules/flare/example_test.go#ExampleHostAllowed
allowed := []string{"fcm.googleapis.com", "*.push.apple.com"}
for _, host := range []string{"fcm.googleapis.com", "api.push.apple.com", "push.apple.com", "evil.example"} {
fmt.Println(host, flare.HostAllowed(host, allowed))
}
// Output:
// fcm.googleapis.com true
// api.push.apple.com true
// push.apple.com false
// evil.example false
```
Keep the default unless you know a browser your users run pushes through another service.
## VAPID keys
A push service accepts a push only when it carries a token signed with the application's VAPID key pair. Generate the pair once:
```sh
./bin/acme websockets:generate-vapid-keys
```
It prints `SUMMER_PUSH__PUBLIC_KEY=...` and `SUMMER_PUSH__PRIVATE_KEY=...` lines to set in the environment. With `--update` it saves the keys to the environment's `overrides.yaml` instead. Keep the private key out of committed files. flare never writes the private key to a log or an error, and `flare.VAPIDKeys` and `flare.Config` redact it when printed.
The frontend needs the public key to subscribe; serve it from a route of your own. Set `push.subject` to a `mailto:` or `https:` contact address for the push services, and `push.enabled` to `true`:
```yaml
enabled: true
subject: mailto:admin@example.com
```
in `config/push.yaml`.
`websockets:test-push <user_id>` prints the push configuration without the key values, lists the user's subscriptions from the published `flare.SubscriptionSource`, and sends each one a test notification:
```sh
./bin/acme websockets:test-push 42
```

246
docs/services/realtime.md Normal file
View File

@@ -0,0 +1,246 @@
---
title: Realtime
description: Publish model changes and events to realtime channels with lighthouse, authorize subscriptions per channel namespace, and run the Centrifugo driver.
section: services
order: 120
---
# Realtime
[lighthouse](../../modules/lighthouse/README.md) is the SummerCMS counterpart of the WinterCMS websockets plugin. The application publishes events to named channels; the frontend holds a connection to a realtime server, subscribes to channels and receives the events. SummerCMS does not run the connection server itself. The Centrifugo driver publishes to a Centrifugo server through its HTTP API, issues the connection tokens the frontend needs, and answers Centrifugo's subscribe checks.
## Drivers
`realtime.driver` selects the driver:
| Driver | Publishes |
|--------|-----------|
| `null` (default) | Nothing. |
| `log` | To the log: channel names and the event, never the payload. |
| `memory` | Into memory, readable with `lighthouse.MemoryDriver.Publications`, for tests. |
| `centrifugo` | To Centrifugo, from the `lighthouse/centrifugo` package. |
A driver registers itself from its package's `init` function, as `database/sql` drivers do, so the application imports the driver package for its side effect: `_ ".../modules/lighthouse/centrifugo"`. An unknown driver name stops the start-up with the list of registered drivers. `lighthouse.From` returns the application's `lighthouse.Service`, which holds the driver, the authorizer registry and the broadcast settings.
## Channels and authorization
A channel name is `namespace:entity:id`, optionally prefixed once with `presence:`. A plugin registers a `lighthouse.Authorizer` per namespace on the service's `lighthouse.Registry`. The driver asks the namespace's authorizer on every subscribe, so a user who loses access is refused the next time the client subscribes; nothing is cached:
```go src=modules/lighthouse/example_test.go#ExampleRegistry_Register
app, err := newApp(map[string]any{"realtime.driver": "memory"})
if err != nil {
fmt.Println(err)
return
}
svc, err := lighthouse.From(app)
if err != nil {
fmt.Println(err)
return
}
// blog:{entity}:{id} channels are open to members of the blog only. The
// authorizer runs on every subscribe; nothing is cached.
err = svc.Registry().Register("blog", lighthouse.AuthorizerFunc(
func(ctx context.Context, userID uint, channel string) lighthouse.Result {
if isMember(ctx, userID, lighthouse.ChannelID(channel)) {
return lighthouse.Allowed(nil)
}
return lighthouse.Denied("not a member of the blog")
}))
if err != nil {
fmt.Println(err)
return
}
for _, sub := range []struct {
user uint
channel string
}{{42, "blog:7"}, {42, "blog:8"}, {42, "presence:blog:7"}, {42, "shop:7"}} {
ns, presence := lighthouse.ParseChannel(sub.channel)
auth, ok := svc.Registry().Get(ns)
if !ok {
fmt.Println(sub.channel, "no authorizer")
continue
}
res := auth.Authorize(context.Background(), sub.user, sub.channel)
fmt.Printf("%d %s namespace=%s presence=%v allowed=%v reason=%q\n", sub.user, sub.channel, ns, presence, res.Allowed, res.Reason())
}
fmt.Println(lighthouse.ChannelID("blog:12abc"), lighthouse.FormatChannels("acme", []string{"Blog:7"}))
// Output:
// 42 blog:7 namespace=blog presence=false allowed=true reason=""
// 42 blog:8 namespace=blog presence=false allowed=false reason="not a member of the blog"
// 42 presence:blog:7 namespace=blog presence=true allowed=false reason="not a member of the blog"
// shop:7 no authorizer
// 12 [acme:blog:7]
```
The channel rules follow the WinterCMS plugin byte for byte:
- `lighthouse.ParseChannel` returns the namespace and whether the channel is a presence channel. A doubled `presence:` prefix or more than three segments give an empty namespace, which no authorizer matches.
- `lighthouse.ChannelID` reads segment 1 with PHP's `(int)` cast: `12abc` is 12. For a `presence:` channel, segment 1 is the namespace, so the ID is 0, as the presence line of the example shows. An authorizer for presence channels must parse the ID itself.
- `lighthouse.FormatChannels` lowercases channel names and applies the `realtime.broadcast_namespace` prefix.
A denial's reason goes to the log only; the client always sees the same refusal.
## Mounting the driver's routes
A driver may need HTTP routes. The Centrifugo driver has two: the token route, which signed-in users call, and the subscribe proxy, which Centrifugo calls. The application mounts them once, from a plugin's `Routes`, with `lighthouse.Mount`, choosing the middleware per surface:
```go src=modules/lighthouse/centrifugo/example_test.go#ExampleDriver_Routes
app, err := newApp(map[string]any{"realtime.driver": "centrifugo"})
if err != nil {
fmt.Println(err)
return
}
svc, err := lighthouse.From(app)
if err != nil {
fmt.Println(err)
return
}
// In a plugin's Routes method, r is the router the plugin receives.
r := surf.New(nil)
err = lighthouse.Mount(r, svc.Driver(), lighthouse.Surfaces{
UserAuth: surf.Use("acme.auth"),
Middleware: surf.Use("throttle:60,1"),
})
if err != nil {
fmt.Println(err)
return
}
for _, rt := range r.Routes() {
fmt.Println(rt.Method, rt.Pattern, rt.Middleware, "raw:", rt.Raw)
}
// A user route without a guard is refused.
err = lighthouse.Mount(surf.New(nil), svc.Driver(), lighthouse.Surfaces{})
fmt.Println(err != nil)
// Output:
// GET /api/realtime/token [acme.auth throttle:60,1] raw: false
// POST /api/realtime/subscribe [throttle:60,1] raw: true
// true
```
`lighthouse.UserAuth` routes get the `UserAuth` middleware, `lighthouse.ServerToServer` routes are mounted in a raw group, and `Middleware` is added to every route after the surface's own. A user route without a guard is refused, so the token route can never be exposed to anonymous callers. Switching drivers never changes the application's route declarations.
## Model broadcasts
A model broadcasts its creates, updates and deletes when a `lighthouse.Binding` is registered for it, or when its pointer type implements `lighthouse.Broadcastable`. A binding keeps realtime code out of the models package:
```go src=modules/lighthouse/example_test.go#bind
return lighthouse.Bind[Post](svc, lighthouse.Binding[Post]{
Alias: "blog.post",
Channels: func(ctx context.Context, tx *gorm.DB, p *Post) ([]string, error) {
return []string{"blog:" + strconv.FormatUint(uint64(p.BlogID), 10)}, nil
},
})
```
The event name is `{action}.{alias}`, here `created.blog.post`, and the default payload is `{"model":...,"actor":...,"timestamp":"...+00:00","ttl":60}`. A binding's `Payload`, `ShouldBroadcast` and `TTL` fields, or the matching model methods, replace the defaults.
Delivery is transactional. The write enqueues a broadcast job, through [conga](../../modules/conga/README.md), inside its own transaction, so nothing is published for a write that rolls back, and the job publishes after the commit:
```go src=modules/lighthouse/example_test.go#write
return lagoon.Transaction(ctx, db, func(ctx context.Context, tx *gorm.DB) error {
if err := tx.Create(&Post{BlogID: 7, Title: "Hello"}).Error; err != nil {
return err
}
// The broadcast job is now queued in this transaction. It is
// published only if the transaction commits.
if fail {
return fmt.Errorf("rolled back")
}
return nil
})
```
The job runs once, best effort: a failed publish is logged as `realtime: broadcast failed` and never affects the write. Delivery order across separate jobs is not guaranteed. A job worker must be running, in `serve` or in `queue:work`; see [Queued jobs](jobs.md). A write without a primary key value, such as `Model(&Post{}).Where(...).Updates(...)`, is not broadcast.
## Bulk writes
`lighthouse.WithoutBroadcasting` silences one model type for writes made with the context it hands to its function; other types still broadcast. `lighthouse.Service.Emit` enqueues one explicit event on the caller's transaction. Together they turn a thousand row events into one summary:
```go src=modules/lighthouse/example_test.go#bulk
return lighthouse.WithoutBroadcasting[Post](ctx, func(ctx context.Context) error {
return lagoon.Transaction(ctx, db, func(ctx context.Context, tx *gorm.DB) error {
for _, title := range titles {
if err := tx.Create(&Post{BlogID: 7, Title: title}).Error; err != nil {
return err
}
}
return svc.Emit(ctx, tx, lighthouse.Broadcast{
Channels: []string{"blog:7"},
Event: "blog.posts_imported",
Payload: struct {
Count int `json:"count"`
}{len(titles)},
})
})
})
```
Only writes that use the context passed to the function are silenced, so write through it, as `lagoon.Transaction` does above.
## The Centrifugo driver
The driver reads `realtime.centrifugo.*` from `config/realtime.yaml`. Secrets go in the environment:
```yaml
driver: centrifugo
centrifugo:
api_url: http://127.0.0.1:8001/api
ws_url: /ws
```
with `SUMMER_REALTIME__CENTRIFUGO__API_KEY`, `SUMMER_REALTIME__CENTRIFUGO__TOKEN_SECRET` and `SUMMER_REALTIME__CENTRIFUGO__PROXY_SECRET` set.
- The token route (`realtime.centrifugo.token_path`, `/api/realtime/token` by default) answers a signed-in user with `{"token":"..."}`, an HS256 connection token signed with the token secret. It answers 401 without a user and 503 when the token secret is empty.
- The subscribe proxy (`realtime.centrifugo.subscribe_path`) accepts a call only when its `X-Centrifugo-Secret` header equals the proxy secret, compared in constant time; an empty proxy secret refuses every subscribe. It then asks the channel's authorizer. Every answer is HTTP 200, as Centrifugo requires, with the decision in the body.
- Publishing uses the HTTP API with the API key. With an empty API key nothing is sent and no broadcast jobs are queued.
The subscribe proxy runs the same authorizers as above:
```go src=modules/lighthouse/centrifugo/example_test.go#ExampleProxyHandler
svc, err := lighthouse.From(backpack.New(nil))
if err != nil {
fmt.Println(err)
return
}
// Members of blog 7 may subscribe to its channels.
err = svc.Registry().Register("blog", lighthouse.AuthorizerFunc(
func(ctx context.Context, userID uint, channel string) lighthouse.Result {
if userID == 42 && lighthouse.ChannelID(channel) == 7 {
return lighthouse.Allowed(nil)
}
return lighthouse.Denied("not a member of the blog")
}))
if err != nil {
fmt.Println(err)
return
}
// realtime.centrifugo.proxy_secret; set it through the environment.
proxy := centrifugo.ProxyHandler(svc, centrifugo.Config{ProxySecret: "test-only-proxy-secret"})
// What Centrifugo posts to the subscribe proxy.
subscribe := func(secret, user, channel string) {
body := fmt.Sprintf(`{"client":"c1","user":%q,"channel":%q}`, user, channel)
req := httptest.NewRequest(http.MethodPost, "/api/realtime/subscribe", strings.NewReader(body))
req.Header.Set("X-Centrifugo-Secret", secret)
rec := httptest.NewRecorder()
proxy.ServeHTTP(rec, req)
fmt.Println(rec.Code, strings.TrimSpace(rec.Body.String()))
}
subscribe("test-only-proxy-secret", "42", "blog:7")
subscribe("test-only-proxy-secret", "5", "blog:7")
subscribe("wrong-secret", "42", "blog:7")
// Output:
// 200 {"result":{"info":[]}}
// 200 {"error":{"code":403,"message":"Access denied"}}
// 200 {"error":{"code":403,"message":"Access denied"}}
```
Configure Centrifugo to call the subscribe proxy with the same secret, and keep its HTTP API on a private address.
`websockets:health` checks the connection to Centrifugo and prints the settings, never the API key:
```sh
./bin/acme websockets:health
```

183
docs/services/search.md Normal file
View File

@@ -0,0 +1,183 @@
---
title: Search
description: Keep models in a search index with beachcomber, synced after commit behind a kill-switch, and re-check the candidate IDs a search returns in SQL.
section: services
order: 140
---
# Search
[beachcomber](../../modules/beachcomber/README.md) is the SummerCMS counterpart of Laravel Scout, as WinterCMS applications use it without a queue. A model opts in by implementing `beachcomber.Searchable`; after a write of such a model commits, its row is reloaded and its document written to, or removed from, the search index. A search asks the index for matching IDs, and the application loads the rows from the database.
## Engines
`search.driver` selects the engine. The default, `null`, indexes nothing and finds nothing. The `typesense` engine, from the `beachcomber/typesense` package, talks to a Typesense server over its HTTP API. An engine registers itself from its package's `init` function, so the application imports the engine package for its side effect; an unknown name stops the start-up. `search.prefix` is prepended to every index name, for example to keep staging and production apart on one server:
```go src=modules/beachcomber/example_test.go#ExampleFrom
cfg, err := compass.Open(compass.Options{Dir: "config", Env: "development", Environ: []string{}})
if err != nil {
fmt.Println(err)
return
}
_ = cfg.Set("search.prefix", "staging_")
svc, err := beachcomber.From(backpack.New(cfg)) // search.driver defaults to null
if err != nil {
fmt.Println(err)
return
}
fmt.Println(svc.Engine().Name(), svc.Engine().Configured(), svc.IndexName(&Post{}))
ids, err := svc.Engine().SearchIDs(context.Background(), svc.IndexName(&Post{}), beachcomber.Query{Q: "go"})
fmt.Println(ids, err)
_ = cfg.Set("search.driver", "elastic")
_, err = beachcomber.From(backpack.New(cfg))
fmt.Println(err != nil)
// Output:
// null false staging_acme_blog_posts
// [] <nil>
// true
```
An engine implements `beachcomber.Engine` and registers with `beachcomber.RegisterEngine`. The examples on this page use a small in-memory engine that matches titles:
```go src=modules/beachcomber/example_test.go#memoryEngine.SearchIDs
func (e *memoryEngine) SearchIDs(ctx context.Context, index string, q beachcomber.Query) ([]string, error) {
e.mu.Lock()
defer e.mu.Unlock()
ids := []string{}
for id, d := range e.docs[index] {
if strings.Contains(strings.ToLower(fmt.Sprint(d["title"])), strings.ToLower(q.Q)) {
ids = append(ids, id)
}
}
slices.Sort(ids)
return ids, nil
}
```
## Searchable models
A model implements `beachcomber.Searchable` with three methods, and needs no import of beachcomber to do so:
```go src=modules/beachcomber/example_test.go#Post.SearchableAs
// SearchableAs is the index name, before search.prefix.
func (Post) SearchableAs() string { return "acme_blog_posts" }
```
```go src=modules/beachcomber/example_test.go#Post.ShouldBeSearchable
// ShouldBeSearchable keeps drafts out of the index.
func (p *Post) ShouldBeSearchable() bool { return p.Published }
```
```go src=modules/beachcomber/example_test.go#Post.ToSearchableArray
// ToSearchableArray builds the document from the committed row.
func (p *Post) ToSearchableArray(ctx context.Context, db *gorm.DB) (map[string]any, error) {
return map[string]any{
"id": strconv.FormatUint(uint64(p.ID), 10),
"blog_id": int64(p.BlogID),
"title": p.Title,
}, nil
}
```
`ToSearchableArray` runs after the commit on a fresh copy of the row, so it may query related rows. A row whose `ShouldBeSearchable` is false, a soft-deleted row and a deleted row have their documents removed. A model can also implement `beachcomber.IndexSchemaProvider`, the schema the engine creates a missing index with, and `beachcomber.SearchKeyer`, a document key other than the primary key.
## Sync after commit
`beachcomber.From` installs GORM callbacks that register the sync with `lagoon.AfterCommit`:
- Inside `lagoon.Transaction`, the sync runs after the commit, and never after a rollback.
- A single-statement write syncs after GORM commits it.
- Inside a plain GORM transaction the sync is skipped with a warning, because the commit cannot be observed. Wrap such writes in `lagoon.Transaction`, or call `beachcomber.Service.Sync` after the commit.
The sync runs inline in the writing goroutine, so a search right after a save finds the document, and it is bounded by the engine's timeout. It is never fatal: a failure is logged as `search: sync failed` with the index, key and operation, never the document or the API key, and the write stays committed. A write without a primary key value, such as `Model(&Post{}).Where(...).Updates(...)`, cannot be synced row by row; bulk paths call `beachcomber.Service.Sync` and `beachcomber.Service.Remove` per row, or reindex.
## The kill-switch
Nothing is sent when the engine is not configured (the `null` engine, or Typesense without an API key), when no database is published, or when the application's `beachcomber.Gate` reports off. Install the gate from a plugin's `Boot`. A gate must treat a read error as off:
```go src=modules/beachcomber/example_test.go#gate
svc.SetGate(beachcomber.GateFunc(func(ctx context.Context, db *gorm.DB) bool {
var enabled bool
err := db.WithContext(ctx).Raw(`SELECT search_enabled FROM acme_blog_settings WHERE id = 1`).Scan(&enabled).Error
return err == nil && enabled
}))
```
## Searching
`beachcomber.Engine.SearchIDs` returns the IDs of matching documents, in the engine's order. They are candidates, not answers: the index can be stale (a write it missed, a document from before a permission change) and its filters are only as good as the document. Re-check every ID in SQL, with the same ownership, visibility and soft-delete conditions the rest of the API applies, before a row reaches a response:
```go src=modules/beachcomber/example_test.go#search
ids, err := svc.Engine().SearchIDs(ctx, svc.IndexName(&Post{}), beachcomber.Query{
Q: term,
QueryBy: []string{"title"},
FilterBy: "blog_id:=" + strconv.FormatUint(uint64(blogID), 10),
})
if err != nil {
return nil, err
}
// The ids are candidates from an index that may be stale or loosely
// filtered: re-check every one in SQL before exposing a row.
var posts []Post
err = db.WithContext(ctx).
Where("id IN ? AND blog_id = ? AND published", ids, blogID).
Order("id").
Find(&posts).Error
return posts, err
```
An empty result is an empty list, never an error.
> [!WARNING]
> Never return rows, or even counts, straight from search IDs. A stale index would otherwise show a draft, a deleted record or another user's data.
## Typesense
The Typesense engine follows the Scout Typesense wire contract, so indexes built by a WinterCMS application can be searched by the port:
```yaml
driver: typesense
typesense:
host: 127.0.0.1
port: 8108
protocol: http
```
in `config/search.yaml`, with the key in `SUMMER_SEARCH__TYPESENSE__API_KEY`. Without an API key nothing is ever sent. A search is one request to the collection's search endpoint, and the engine returns the hit IDs in Typesense's order:
```go src=modules/beachcomber/typesense/example_test.go#ExampleEngine_SearchIDs
// A stand-in Typesense node.
node := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
fmt.Println(r.Method, r.URL.Path, "key sent:", r.Header.Get("X-TYPESENSE-API-KEY") != "")
fmt.Println("q", q.Get("q"), "query_by", q.Get("query_by"), "filter_by", q.Get("filter_by"))
fmt.Fprint(w, `{"hits":[{"document":{"id":"3"}},{"document":{"id":"1"}}]}`)
}))
defer node.Close()
u, _ := url.Parse(node.URL)
port, _ := strconv.Atoi(u.Port())
engine := typesense.New(typesense.Config{
APIKey: "test-only-key", // SUMMER_SEARCH__TYPESENSE__API_KEY
Host: u.Hostname(),
Port: port,
Protocol: "http",
ConnectionTimeout: 2 * time.Second,
})
ids, err := engine.SearchIDs(context.Background(), "acme_blog_posts", beachcomber.Query{
Q: "go",
QueryBy: []string{"title"},
FilterBy: "blog_id:=7",
})
fmt.Println(ids, err)
// Without an API key the engine is not configured and nothing is sent.
fmt.Println(typesense.New(typesense.Config{}).Configured())
// Output:
// GET /collections/acme_blog_posts/documents/search key sent: true
// q go query_by title filter_by blog_id:=7
// [3 1] <nil>
// false
```
Each request times out after `search.typesense.connection_timeout_seconds` (2 seconds by default). A failed answer is a `typesense.StatusError` with the method, path and status, never the answer body.

36
docs/services/storage.md Normal file
View File

@@ -0,0 +1,36 @@
---
title: Storage
description: Configure the uploads bucket that serve opens, choose file or memory bucket URLs, serve stored files, and size upload routes.
section: services
order: 90
---
# Storage
WinterCMS stores uploads on a Laravel filesystem disk. SummerCMS stores them in one [gocloud.dev](https://gocloud.dev/howto/blob/) bucket, opened from `storage.uploads.bucket_url` by the `attach` package of [lagoon](../../modules/lagoon/README.md). The `serve` command opens the bucket at start-up, before it accepts requests, and publishes it on the application; an empty `bucket_url` stops the start-up.
## Bucket URLs
| URL | Stores |
|-----|--------|
| `file:///var/lib/acme/uploads` | In a directory on the server. |
| `mem://` | In memory, for tests. Everything is lost when the process exits. |
The keys go in `config/storage.yaml`:
```yaml
uploads:
bucket_url: file:///var/lib/acme/uploads
public_path_prefix: /storage/uploads
```
Files are laid out as WinterCMS lays out its uploads disk, so a copy of a WinterCMS `storage/app/uploads/public` directory can serve as the bucket after a port. `public_path_prefix` is the URL prefix that file and thumbnail URLs start with.
Application code gets the bucket with `app.Lookup[*blob.Bucket]()` and reads and writes it through the `gocloud.dev/blob` API. Model attachments, thumbnails and deleting files after commit are covered in [Attachments](../database/attachments.md).
## Serving files
The framework does not mount a file route by itself. The application decides where files are served: mount `attach.StaticHandlerPublic` under `public_path_prefix` to serve originals and thumbnails and answer 404 for files whose row is not public, or put a web server or CDN in front of the bucket directory. Serve uploads from a separate origin when you can; the [Attachments](../database/attachments.md) page explains why.
## Upload size
Every non-raw route has a request body limit of `http.body_limits.default_bytes`. A route that accepts uploads raises its own limit with the `body.limit:<bytes>` middleware; see [Routing](routing.md). `http.body_limits.upload_bytes` is required and validated at start-up, but the framework applies it to no route; a plugin that wants its upload routes to follow it reads it in `Register` and puts the value in the route's `body.limit`.