golem15/summercms
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(13-01): add the prohibited rule and dated-download and notification masks

Browse Source 
- lagoon.ValidateRequest supports Laravel 9 prohibited (!required, not
  implicit); with no catalog line its message is validation.prohibited
- tide compares Content-Disposition with real calendar dates masked on both
  sides; a different name, an invalid date or a one-sided date still diffs
- tide.NormalizePublications masks a Carbon +00:00 $.data.payload.created_at
  and an uncaptured positive integer $.data.payload.id as {{id}}
- the album-date test's outside-album sibling moves off payload.created_at,
  which now has its own mask
- READMEs and docs describe the rule and both masks
This commit is contained in:
Jakub Zych
2026-10-03 06:32:46 +02:00
parent 55a4092019
commit 2b94dfd2d2
10 changed files with 243 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
modules/tide/README.md
View File

@@ -16,10 +16,10 @@ HTTP parity toolkit that records request and response fixtures from a reference
- Recording proxy: `tide.NewProxy` builds a reverse proxy that only binds to and forwards to loopback addresses, groups traffic into named sessions (from the `tide.SessionHeader` request header or a default session) and writes one fixture per complete session on `tide.Proxy.Flush`.
- Capture rules: `tide.Rules` (loaded with `tide.LoadRules`) decide which request and response headers are kept per route and which values are captured into variables, from response JSON paths, headers, redirect query strings or form fields.
- Variables: `tide.Store` holds captured values such as tokens and IDs in a mode-0600 file, `tide.Store.Expand` substitutes `{{name}}` placeholders before a request is sent, and `tide.ScrubStep` puts placeholders back into fixtures. Scrubbing fails when a step still holds an unclassified token- or password-shaped value, so credentials do not leak into committed fixtures.
- Replay and diff: `tide.ReplayFlow` re-sends each step, compares status, a fixed set of contract headers and the body, and returns `tide.Result` with per-step `tide.Diff` entries. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies are compared byte for byte. Uploaded-file URLs under `url` and `thumb_url` keys are compared by shape: under the uploads prefix (`tide.ReplayConfig` `UploadPrefix`, default `tide.DefaultUploadPrefix`, `/storage/app/uploads/public`) an original must be `<prefix>/xxx/yyy/zzz/<disk_name>` with the partition taken from the disk name, and a thumbnail `<prefix>/xxx/yyy/zzz/thumb_<id>_<w>_<h>_<ox>_<oy>_<mode>.<ext>`. The partition, disk name and file id are masked; the prefix, size, offsets, mode and extension stay, so a thumbnail of another size or an upload URL under another prefix is still a difference.
- Replay and diff: `tide.ReplayFlow` re-sends each step, compares status, a fixed set of contract headers and the body, and returns `tide.Result` with per-step `tide.Diff` entries. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies are compared byte for byte. A `Content-Disposition` header is compared after masking its `YYYY-MM-DD` dates on both sides, so a download named after the day it was made, such as `attachment; filename=export-2026-09-17.csv`, replays on a later day; each masked token must be a real calendar date, and a different name, an invalid date or a date on one side only is still a difference. Uploaded-file URLs under `url` and `thumb_url` keys are compared by shape: under the uploads prefix (`tide.ReplayConfig` `UploadPrefix`, default `tide.DefaultUploadPrefix`, `/storage/app/uploads/public`) an original must be `<prefix>/xxx/yyy/zzz/<disk_name>` with the partition taken from the disk name, and a thumbnail `<prefix>/xxx/yyy/zzz/thumb_<id>_<w>_<h>_<ox>_<oy>_<mode>.<ext>`. The partition, disk name and file id are masked; the prefix, size, offsets, mode and extension stay, so a thumbnail of another size or an upload URL under another prefix is still a difference.
- Fake Centrifugo: `tide.NewCentrifugoRecorder` returns an `http.Handler` that records every POST to a path ending in `/publish` or `/broadcast` as a `tide.Publication` (method, path, whether `Authorization: apikey <key>` carried the configured key, JSON body) and answers `{"result":{}}`. Paths ending in `/presence` answer `{"result":{"presence":{}}}`, `/unsubscribe` and `/info` answer `{"result":{}}`, anything else is 404. Bodies are capped at `tide.MaxPublicationBody` (1 MiB). The API key is only compared, never stored. `tide.CentrifugoRecorder.ListenAndServe` binds loopback addresses only, like the recording proxy.
- Broadcast goldens: `tide.RecordBroadcasts` runs a flow against a loopback reference backend whose Centrifugo API URL points at a recorder on `tide.DefaultCentrifugoListen` (`127.0.0.1:8424`). With `tide.BroadcastConfig` `Step` set, earlier steps run as setup and only that step's publications are kept. The result is a `tide.BroadcastGolden`, written with `tide.WriteBroadcastGolden` (which refuses token-shaped bodies) and read strictly with `tide.LoadBroadcastGolden`. A golden with `pending` set is recorded but not yet asserted.
- Broadcast normalisation: `tide.NormalizePublications` masks only `$.data.timestamp` and `$.data.payload.timestamp` (ISO 8601 with an offset) as `"{{timestamp}}"`, `$.data.payload.actor` (an object of exactly `user_id` and `name`) as `"{{actor}}"`, and values equal to an `id:*` variable of a `tide.Store`: numbers or strings under `id`, `*_id` or `*_ids` keys, and the numeric last segment of a channel name such as `room:12`. Carbon `+00:00` values of `*_at` keys anywhere under `$.data.payload.album` become `"{{datetime}}"`; a date of another shape is left as it is, so a format change shows as a difference. A masked number is written as a bare `{{id:name}}`, so a number that becomes a string still differs. A value matching two id variables is an error. `tide.DiffPublications` compares the count, method, path, authorization flag and body (structurally, key order ignored) and reports paths such as `$[0].body.data.payload.id`.
- Broadcast normalisation: `tide.NormalizePublications` masks only `$.data.timestamp` and `$.data.payload.timestamp` (ISO 8601 with an offset) as `"{{timestamp}}"`, `$.data.payload.actor` (an object of exactly `user_id` and `name`) as `"{{actor}}"`, and values equal to an `id:*` variable of a `tide.Store`: numbers or strings under `id`, `*_id` or `*_ids` keys, and the numeric last segment of a channel name such as `room:12`. Carbon `+00:00` values of `*_at` keys anywhere under `$.data.payload.album` become `"{{datetime}}"`. A notification publication's own row fields are masked too: a Carbon `+00:00` `$.data.payload.created_at` becomes `"{{datetime}}"`, and a positive integer `$.data.payload.id` that no `id:*` variable names becomes a bare `{{id}}` (a captured one keeps `{{id:name}}`); a date of another shape is left as it is, so a format change shows as a difference. A masked number is written as a bare `{{id:name}}`, so a number that becomes a string still differs. A value matching two id variables is an error. `tide.DiffPublications` compares the count, method, path, authorization flag and body (structurally, key order ignored) and reports paths such as `$[0].body.data.payload.id`.
- Manifests: `tide.Manifest` lists routes with auth groups, a pending or ported status, cases and fixture paths; `tide.RecordManifest` records missing cases in batches of at most `tide.MaxBatch`, and `tide.ReplayManifest` replays every recorded case into a `tide.Coverage` table.
## Usage

20
modules/tide/centrifugo_golden.go
View File

@@ -275,7 +275,27 @@ func (n *normalizer) lookup(v string) (string, bool) {
}
}
// positiveIntRe is a JSON number literal holding a positive integer.
var positiveIntRe = regexp.MustCompile(`^[1-9][0-9]*$`)
func (n *normalizer) walk(path, key string, v *onode) *onode {
// A notification publication carries the new row's id and created_at
// under $.data.payload. A Carbon +00:00 created_at is masked like an album
// date; a positive integer id keeps its captured variable's placeholder
// and is {{id}} when no captured variable names it. Any other shape at
// these paths stays visible.
switch {
case path == "$.data.payload.created_at" && v.kind == kindString && carbonOffsetRe.MatchString(v.str):
return &onode{kind: kindString, str: "{{datetime}}"}
case path == "$.data.payload.id" && v.kind == kindNumber && positiveIntRe.MatchString(v.str):
if name, ok := n.lookup(v.str); ok {
return &onode{kind: kindPlaceholder, str: name}
}
if len(n.ids[v.str]) == 0 {
return &onode{kind: kindPlaceholder, str: "id"}
}
return v
}
switch path {
case "$.data.timestamp", "$.data.payload.timestamp":
if v.kind == kindString && isoOffsetRe.MatchString(v.str) {

34
modules/tide/diff.go
View File

@@ -6,8 +6,10 @@ import (
"fmt"
"mime"
"net/http"
"regexp"
"strconv"
"strings"
"time"
"unicode"
"unicode/utf8"
)
@@ -81,6 +83,9 @@ func compareHeaders(want, got, extra map[string]string) []Diff {
if gv == wv {
continue
}
if ck == "Content-Disposition" && sameDispositionButDates(wv, gv) {
continue
}
if gv == "" {
gv = "<missing>"
}
@@ -290,3 +295,32 @@ func quotePrintable(b []byte) string {
buf.WriteByte('"')
return buf.String()
}
// dispositionDateRe finds YYYY-MM-DD tokens in a Content-Disposition value,
// such as the date in a download name built from the current day.
var dispositionDateRe = regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}\b`)
// maskDispositionDates replaces every date token with one placeholder and
// reports how many it replaced. ok is false when a token is not a real
// calendar date, so the caller falls back to the byte comparison.
func maskDispositionDates(s string) (masked string, n int, ok bool) {
ok = true
masked = dispositionDateRe.ReplaceAllStringFunc(s, func(m string) string {
if _, err := time.Parse("2006-01-02", m); err != nil {
ok = false
return m
}
n++
return "{{date}}"
})
return masked, n, ok
}
// sameDispositionButDates reports whether two Content-Disposition values
// differ only in their dates: both carry the same number of real calendar
// dates and are equal once those are masked. Anything else stays a Diff.
func sameDispositionButDates(want, got string) bool {
wm, wn, wok := maskDispositionDates(want)
gm, gn, gok := maskDispositionDates(got)
return wok && gok && wn > 0 && wn == gn && wm == gm
}

8
modules/tide/multipart_test.go
View File

@@ -265,7 +265,7 @@ func TestNormalizePublicationAlbumDates(t *testing.T) {
pub := func(created, updated string) []Publication {
return []Publication{{Method: "POST", Path: "/api/publish", Body: json.RawMessage(
`{"channel":"c","data":{"payload":{"album":{"name":"x","created_at":"` + created + `","updated_at":"` + updated +
`","market_price_checked_at":null,"photos":[{"created_at":"` + created + `"}]},"created_at":"2026-01-01T00:00:00+00:00"}}}`)}}
`","market_price_checked_at":null,"photos":[{"created_at":"` + created + `"}]},"published_at":"2026-01-01T00:00:00+00:00"}}}`)}}
}
a, err := NormalizePublications(pub("2026-09-30T11:21:55+00:00", "2026-09-30T11:21:56+00:00"), store)
if err != nil {
@@ -281,8 +281,10 @@ func TestNormalizePublicationAlbumDates(t *testing.T) {
if !strings.Contains(string(a[0].Body), `"updated_at":"{{datetime}}"`) || !strings.Contains(string(a[0].Body), `"market_price_checked_at":null`) {
t.Fatalf("album dates not masked: %s", a[0].Body)
}
// Only the album subtree is masked; a payload date outside it stays.
if !strings.Contains(string(a[0].Body), `},"created_at":"2026-01-01T00:00:00+00:00"`) {
// Only the album subtree is masked; a payload date outside it stays
// ($.data.payload.created_at itself is the notification row's date and
// has its own mask, see TestNormalizeNotificationPublication).
if !strings.Contains(string(a[0].Body), `},"published_at":"2026-01-01T00:00:00+00:00"`) {
t.Fatalf("over-normalised: %s", a[0].Body)
}
// A Z-suffixed album date keeps its value, so a format change is a diff.

124
modules/tide/normalize_phase13_test.go Normal file
View File

@@ -0,0 +1,124 @@
package tide
import (
"encoding/json"
"strings"
"testing"
)
// TestNormalizeContentDispositionDate: a download name built from the day
// of the request replays on a later day, while a different stem, an invalid
// date, a date on one side only or a missing header still diff.
func TestNormalizeContentDispositionDate(t *testing.T) {
const recorded = "attachment; filename=export-2026-09-17.csv"
cases := []struct {
name string
got map[string]string
diff bool
}{
{"same stem on a later day", map[string]string{"Content-Disposition": "attachment; filename=export-2026-10-03.csv"}, false},
{"same day", map[string]string{"Content-Disposition": recorded}, false},
{"header name case", map[string]string{"content-disposition": "attachment; filename=export-2027-01-01.csv"}, false},
{"different stem", map[string]string{"Content-Disposition": "attachment; filename=report-2026-10-03.csv"}, true},
{"invalid date", map[string]string{"Content-Disposition": "attachment; filename=export-2026-13-45.csv"}, true},
{"date on one side only", map[string]string{"Content-Disposition": "attachment; filename=export.csv"}, true},
{"inline instead of attachment", map[string]string{"Content-Disposition": "inline; filename=export-2026-10-03.csv"}, true},
{"header absent", map[string]string{}, true},
}
for _, c := range cases {
diffs := compareHeaders(map[string]string{"Content-Disposition": recorded}, c.got, nil)
if (len(diffs) > 0) != c.diff {
t.Errorf("%s: diffs = %+v, want diff=%v", c.name, diffs, c.diff)
}
}
// A recorded value that holds an invalid date keeps the byte comparison.
odd := "attachment; filename=export-2026-02-30.csv"
if diffs := compareHeaders(map[string]string{"Content-Disposition": odd}, map[string]string{"Content-Disposition": "attachment; filename=export-2026-02-28.csv"}, nil); len(diffs) != 1 {
t.Errorf("invalid recorded date: diffs = %+v, want one", diffs)
}
// Two dates must both be real and both present.
two := "attachment; filename=export-2026-09-01-2026-09-17.csv"
if diffs := compareHeaders(map[string]string{"Content-Disposition": two}, map[string]string{"Content-Disposition": "attachment; filename=export-2026-10-01-2026-10-03.csv"}, nil); len(diffs) != 0 {
t.Errorf("two dates: diffs = %+v, want none", diffs)
}
// The mask applies to Content-Disposition only.
if diffs := compareHeaders(map[string]string{"Location": "/files/2026-09-17"}, map[string]string{"Location": "/files/2026-10-03"}, nil); len(diffs) != 1 {
t.Errorf("Location: diffs = %+v, want one", diffs)
}
if _, n, ok := maskDispositionDates("x12026-09-170"); n != 0 || !ok {
t.Errorf("a date inside a longer digit run is not a date token")
}
}
// TestNormalizeNotificationPublication: a notification:new publication's id
// and created_at under $.data.payload normalize equal across two runs, while
// a Z date, a string id and every other path stay visible.
func TestNormalizeNotificationPublication(t *testing.T) {
store := mustMemoryStore()
pub := func(id, created string) []Publication {
return []Publication{{Method: "POST", Path: "/api/publish", Body: json.RawMessage(
`{"channel":"acme#5","data":{"event":"notification:new","payload":{"id":` + id +
`,"type":"item_added","payload":{"album_id":3,"created_at":"2026-01-01T00:00:00+00:00"},"read_at":null,"created_at":"` + created + `"}}}`)}}
}
a, err := NormalizePublications(pub("10000001", "2026-09-30T11:21:55+00:00"), store)
if err != nil {
t.Fatal(err)
}
b, err := NormalizePublications(pub("42", "2026-10-03T08:00:00+00:00"), store)
if err != nil {
t.Fatal(err)
}
if diffs := DiffPublications(a, b); len(diffs) != 0 {
t.Fatalf("diffs = %+v", diffs)
}
body := string(a[0].Body)
if !strings.Contains(body, `"payload":{"id":{{id}},`) || !strings.HasSuffix(body, `"read_at":null,"created_at":"{{datetime}}"}}}`) {
t.Fatalf("notification not masked: %s", body)
}
// Only $.data.payload.created_at is masked, not a nested payload date.
if !strings.Contains(body, `"album_id":3,"created_at":"2026-01-01T00:00:00+00:00"`) {
t.Fatalf("over-normalised: %s", body)
}
// A Z date, a zero or negative id and a string id stay visible.
for _, c := range []struct{ id, created, want string }{
{"42", "2026-10-03T08:00:00Z", `"created_at":"2026-10-03T08:00:00Z"`},
{`"42"`, "2026-10-03T08:00:00+00:00", `"payload":{"id":"42",`},
{"0", "2026-10-03T08:00:00+00:00", `"payload":{"id":0,`},
{"-3", "2026-10-03T08:00:00+00:00", `"payload":{"id":-3,`},
{"4.5", "2026-10-03T08:00:00+00:00", `"payload":{"id":4.5,`},
} {
got, err := NormalizePublications(pub(c.id, c.created), store)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(got[0].Body), c.want) {
t.Errorf("id %s created %s: %s lacks %s", c.id, c.created, got[0].Body, c.want)
}
if diffs := DiffPublications(a, got); len(diffs) == 0 {
t.Errorf("id %s created %s must diff against the masked publication", c.id, c.created)
}
}
// A captured id keeps its own placeholder.
store.Set("id:note", "77")
got, err := NormalizePublications(pub("77", "2026-10-03T08:00:00+00:00"), store)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(got[0].Body), `"payload":{"id":{{id:note}},`) {
t.Fatalf("captured id: %s", got[0].Body)
}
// An album publication's existing masks are unchanged.
album := []Publication{{Method: "POST", Path: "/api/publish", Body: json.RawMessage(
`{"channel":"c","data":{"payload":{"album":{"created_at":"2026-09-30T11:21:55+00:00"},"actor":{"user_id":1,"name":null},"timestamp":"2026-09-30T11:21:55+00:00"}}}`)}}
got, err = NormalizePublications(album, store)
if err != nil {
t.Fatal(err)
}
want := `{"channel":"c","data":{"payload":{"album":{"created_at":"{{datetime}}"},"actor":"{{actor}}","timestamp":"{{timestamp}}"}}}`
if string(got[0].Body) != want {
t.Fatalf("album publication\n got %s\nwant %s", got[0].Body, want)
}
}