From 2b94dfd2d275131834470ad4673784efb3b5c48c Mon Sep 17 00:00:00 2001 From: Jakub Zych Date: Sat, 3 Oct 2026 06:32:46 +0200 Subject: [PATCH] feat(13-01): add the prohibited rule and dated-download and notification masks - lagoon.ValidateRequest supports Laravel 9 prohibited (!required, not implicit); with no catalog line its message is validation.prohibited - tide compares Content-Disposition with real calendar dates masked on both sides; a different name, an invalid date or a one-sided date still diffs - tide.NormalizePublications masks a Carbon +00:00 $.data.payload.created_at and an uncaptured positive integer $.data.payload.id as {{id}} - the album-date test's outside-album sibling moves off payload.created_at, which now has its own mask - READMEs and docs describe the rule and both masks --- docs/database/casts-and-validation.md | 1 + docs/services/parity-testing.md | 4 +- modules/lagoon/README.md | 2 +- modules/lagoon/validate_request_test.go | 49 ++++++++++ modules/lagoon/validate_rules.go | 6 +- modules/tide/README.md | 4 +- modules/tide/centrifugo_golden.go | 20 ++++ modules/tide/diff.go | 34 +++++++ modules/tide/multipart_test.go | 8 +- modules/tide/normalize_phase13_test.go | 124 ++++++++++++++++++++++++ 10 files changed, 243 insertions(+), 9 deletions(-) create mode 100644 modules/tide/normalize_phase13_test.go diff --git a/docs/database/casts-and-validation.md b/docs/database/casts-and-validation.md index 4270b37..3876a3c 100644 --- a/docs/database/casts-and-validation.md +++ b/docs/database/casts-and-validation.md @@ -127,6 +127,7 @@ The behaviour follows Laravel: - The other rules run only when there is a value: they skip an absent attribute, a blank string, `null` under `nullable` and an absent key under `sometimes`. - `min`, `max`, `size` and `between` measure what Laravel measures: the number on an `integer` or `numeric` attribute, compared exactly as a decimal; the number of elements of an array; the size of an uploaded file in kilobytes; otherwise the length in characters, not bytes. Each picks the matching message, such as `max.string` or `max.file`. - `email` is PHP's `FILTER_VALIDATE_EMAIL`, the check WinterCMS uses by default, so `user@localhost` is refused. `date` accepts ISO 8601 dates and date-times, `Y/m/d`, `m/d/Y`, `d.m.Y` and `d-m-Y`, and refuses dates that do not exist. `after_or_equal` and `before_or_equal` take a date, `today`, `tomorrow`, `yesterday` or `now` (in UTC), or the name of another field. +- `prohibited` refuses a field the endpoint must never accept: it fails for any value that `required` would accept, including `0`, `false` and a non-empty array. Like the other non-implicit rules it never runs for an absent key or a blank string, and `null` or an empty array passes. Laravel 9's catalogs have no line for it, so the message is the key itself, `validation.prohibited`, as the PHP endpoint answers. - `exists:table,column` counts matching rows in the database, so it needs the transaction handle. Like Laravel it does not skip soft-deleted rows, and it does not run once the attribute already has a message. - File rules (`file`, `image`, `mimes` and the size rules) take a `lagoon.UploadedFile`; `lagoon.UploadedFileFromHeader` builds one from a parsed multipart part. The type is read from the file content, not from its name. diff --git a/docs/services/parity-testing.md b/docs/services/parity-testing.md index bdfe26a..eadcaa8 100644 --- a/docs/services/parity-testing.md +++ b/docs/services/parity-testing.md @@ -24,7 +24,7 @@ steps: path: /api/blog/posts?page=1 ``` -Recording sends each request to the reference backend and fills in the responses. Replaying sends them to the port and compares status, a fixed set of contract headers and the body. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies byte for byte: +Recording sends each request to the reference backend and fills in the responses. Replaying sends them to the port and compares status, a fixed set of contract headers and the body. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies byte for byte. Among the headers, `Content-Disposition` is compared with its dates masked: a CSV download named `export-2026-09-17.csv` on the recording day replays as `export-2026-10-03.csv` a few weeks later. Each date must be a real calendar date on both sides, so a renamed file, a broken date or a date that disappears still fails: ```go src=modules/tide/example_test.go#ExampleReplayFlow ctx := context.Background() @@ -121,6 +121,6 @@ Realtime side effects are part of the contract too. `summer parity:broadcasts` r summer parity:broadcasts --flow testdata/broadcasts/flows/post-lifecycle.yaml --step delete --name deleted --target http://127.0.0.1:8000 --vars /tmp/parity/vars.yaml --out testdata/broadcasts/deleted.yaml ``` -Point the reference backend's Centrifugo API URL at the recorder (`127.0.0.1:8424` by default). `--step` keeps only the publications of one step, running the earlier steps as setup. Timestamps, the actor, the `*_at` dates inside a published album and captured IDs are masked (`tide.NormalizePublications`), so the Go port's publications, recorded the same way, compare with `tide.DiffPublications`. A golden with `--pending` set is recorded but not yet asserted. +Point the reference backend's Centrifugo API URL at the recorder (`127.0.0.1:8424` by default). `--step` keeps only the publications of one step, running the earlier steps as setup. Timestamps, the actor, the `*_at` dates inside a published album, a notification's own `created_at` and `id` under `$.data.payload` (the id only when it is a positive integer, as `{{id}}` unless a captured ID names it) and captured IDs are masked (`tide.NormalizePublications`), so the Go port's publications, recorded the same way, compare with `tide.DiffPublications`. A golden with `--pending` set is recorded but not yet asserted. On the Go side, the memory realtime driver records publications the same way in tests; see [Realtime](realtime.md). diff --git a/modules/lagoon/README.md b/modules/lagoon/README.md index d9cdeea..74036f6 100644 --- a/modules/lagoon/README.md +++ b/modules/lagoon/README.md @@ -18,7 +18,7 @@ Postgres data layer: the shared GORM connection, per-plugin migrations, model he - Per-plugin migrations: `lagoon.Migrate` runs the framework's `system_files` set (`attach.Migrations`), backend admin identity set (`lagoon.BackendAdminMigrations`), `deferred_bindings` set (`lagoon.DeferredBindingMigrations`, under the `lagoon.DeferredHistoryID` history) and job-queue set (`lagoon.QueueMigrations`: River's schema pinned at `lagoon.RiverSchemaVersion`, then the `lagoon.JobsTable` record table, under the `lagoon.QueueHistoryID` history), then every `pact.HasMigrations` set in plugin activation order, each in its own `summer_migrations_` history table (`lagoon.HistoryTableName`). `lagoon.RollbackLast` and `lagoon.Status` cover rollback and history. - Mass assignment: `lagoon.Fill` copies only allow-listed keys onto a model by GORM column name and silently drops the rest, logging each dropped key once outside production. A `json.Number` (from a decoder using `UseNumber`) fills integer, unsigned and float fields. A value that does not fit its column (a fraction, an exponent or an overflow for an integer field, or a value of the wrong type) is a `lagoon.FillTypeError` naming the key, so a caller can answer it as a validation failure on that field. `lagoon.HasFillable` and `lagoon.HasHidden` are the Go forms of `$fillable` and `$hidden`. - Validation: `lagoon.Validate` (where `required` fails on a zero date or time) accepts Laravel-style rule strings (`required`, `nullable`, `integer`, `numeric`, `between`, `min`, `max`, `in`, `unique`, `boolean`, `email`, `confirmed`, `different`, `mimes`) and returns a field-to-messages map, translated through phrasebook when a translator is given. Unknown rule tokens are an error. A failed numeric range reports the bound that failed: the `min` message below the lower bound, the `max` message above the upper one, and the numeric `between` message when the bound came from `between`. -- Request validation: `lagoon.ValidateRequest` reproduces Laravel 9 request validation for ported API endpoints, so a 422 body matches the PHP one message for message. It takes the decoded input and an ordered `lagoon.RequestRule` table (attribute names may hold `*` wildcards, expanded against the input to `posts.0.title`), runs the rules of each attribute in order and stops an attribute after a failed implicit rule (`required`, `present`, `filled`, `accepted`) or, under `bail`, after any failure. A non-implicit rule is skipped for an absent attribute, a blank string, a null value under `nullable` and an absent key under `sometimes`. Supported rules: `required`, `present`, `filled`, `accepted`, `nullable`, `sometimes`, `bail`, `array`, `string`, `integer`, `numeric`, `boolean`, `email` (PHP `FILTER_VALIDATE_EMAIL`, WinterCMS's default), `url`, `date`, `after`, `after_or_equal`, `before`, `before_or_equal` (a date, a relative word such as `tomorrow`, or another field), `exists:table,column`, `regex`, `not_regex`, `in`, `not_in`, `file`, `image`, `mimes`, `min`, `max`, `size` and `between`, plus closure rules built with `lagoon.CustomRule`. The size rules compare the number under `numeric` or `integer` (exactly, as decimals), the element count of an array, kilobytes of a `lagoon.UploadedFile`, and otherwise the length in characters, and pick the matching message. Messages come from the `lagoon::validation` catalog in the request locale; `lagoon.ErrorKeys` gives the attribute order of PHP's message bag. +- Request validation: `lagoon.ValidateRequest` reproduces Laravel 9 request validation for ported API endpoints, so a 422 body matches the PHP one message for message. It takes the decoded input and an ordered `lagoon.RequestRule` table (attribute names may hold `*` wildcards, expanded against the input to `posts.0.title`), runs the rules of each attribute in order and stops an attribute after a failed implicit rule (`required`, `present`, `filled`, `accepted`) or, under `bail`, after any failure. A non-implicit rule is skipped for an absent attribute, a blank string, a null value under `nullable` and an absent key under `sometimes`. Supported rules: `required`, `present`, `filled`, `accepted`, `nullable`, `sometimes`, `bail`, `array`, `string`, `integer`, `numeric`, `boolean`, `email` (PHP `FILTER_VALIDATE_EMAIL`, WinterCMS's default), `url`, `date`, `after`, `after_or_equal`, `before`, `before_or_equal` (a date, a relative word such as `tomorrow`, or another field), `exists:table,column`, `regex`, `not_regex`, `in`, `not_in`, `file`, `image`, `mimes`, `min`, `max`, `size`, `between` and `prohibited` (fails for any value `required` would accept, `0` and `false` included; not implicit, so an absent key, `null`, a blank string and an empty array pass, and with no catalog line its message is the key `validation.prohibited`), plus closure rules built with `lagoon.CustomRule`. The size rules compare the number under `numeric` or `integer` (exactly, as decimals), the element count of an array, kilobytes of a `lagoon.UploadedFile`, and otherwise the length in characters, and pick the matching message. Messages come from the `lagoon::validation` catalog in the request locale; `lagoon.ErrorKeys` gives the attribute order of PHP's message bag. - Safe ordering: `lagoon.OrderBy` appends an ORDER BY only for an allow-listed column and an `asc` or `desc` direction, and `lagoon.Collate` adds a validated `COLLATE` clause for language-specific text order (for example the ICU collation `pl-x-icu`); lagoon puts no requirement on the database's default locale. - Pagination: `lagoon.Paginate` builds a `lagoon.Page` with `data` and `meta` (`current_page`, `last_page`, `per_page`, `total`). - Date and time columns: `lagoon.Date` (a `DATE` column, JSON `"2026-10-02"`) and `lagoon.TimeOfDay` (a `TIME` column, JSON `"14:30:00"`) implement `sql.Scanner`, `driver.Valuer`, JSON and text marshalling, and store NULL for their zero value; `*lagoon.Date` and `*lagoon.TimeOfDay` are the nullable variants, next to `time.Time` and `*time.Time` for `timestamptz`. Build them with `lagoon.NewDate`, `lagoon.DateOf`, `lagoon.ParseDate`, `lagoon.NewTimeOfDay` and `lagoon.ParseTimeOfDay`. `lagoon.Fill` fills all six from JSON strings (RFC 3339 for `time.Time`) through their text unmarshalling, after every conversion it already made. Behaviour change: `required` now treats a zero `time.Time`, `lagoon.Date` or `lagoon.TimeOfDay` (or a pointer to one) as empty, so declare optional dates as pointer fields. diff --git a/modules/lagoon/validate_request_test.go b/modules/lagoon/validate_request_test.go index 17d6e84..f8dafca 100644 --- a/modules/lagoon/validate_request_test.go +++ b/modules/lagoon/validate_request_test.go @@ -664,3 +664,52 @@ func TestValidateRequestExistsRule(t *testing.T) { t.Error("a missing table must be an error for arrays too") } } + +// TestValidateRequestProhibited ports Laravel 9's prohibited rule: it is +// !validateRequired and not implicit, so an absent key and a blank string +// are never validated, null and an empty array pass, and any value required +// would accept fails, 0 and false included. Neither catalog has a +// validation.prohibited line, so the message is the key itself. +func TestValidateRequestProhibited(t *testing.T) { + rules := []RequestRule{{Field: "condition", Rules: ParseRules("prohibited")}} + pass := map[string]map[string]any{ + "absent": {}, + "null": {"condition": nil}, + "empty string": {"condition": ""}, + "blank string": {"condition": " "}, + "empty array": {"condition": []any{}}, + "empty map": {"condition": map[string]any{}}, + } + for name, input := range pass { + if got := mustValidate(t, inLocale("pl"), input, rules); len(got) != 0 { + t.Errorf("%s: errors = %v, want none", name, got) + } + } + fail := map[string]any{ + "string": "VG", + "zero": json.Number("0"), + "float zero": float64(0), + "false": false, + "true": true, + "non-empty array": []any{"x"}, + "non-empty map": map[string]any{"a": 1}, + } + for name, value := range fail { + for _, loc := range []string{"pl", "en"} { + got := mustValidate(t, inLocale(loc), map[string]any{"condition": value}, rules) + want := map[string][]string{"condition": {"validation.prohibited"}} + if !reflect.DeepEqual(got, want) { + t.Errorf("%s (%s): errors = %#v, want %#v", name, loc, got, want) + } + } + } + // Next to other rules, prohibited fails like any non-implicit rule. + mixed := []RequestRule{ + {Field: "name", Rules: ParseRules("required|string")}, + {Field: "shelf", Rules: ParseRules("prohibited")}, + } + got := mustValidate(t, inLocale("en"), map[string]any{"name": "Kind of Blue", "shelf": "A1"}, mixed) + if want := map[string][]string{"shelf": {"validation.prohibited"}}; !reflect.DeepEqual(got, want) { + t.Errorf("mixed = %#v, want %#v", got, want) + } +} diff --git a/modules/lagoon/validate_rules.go b/modules/lagoon/validate_rules.go index 79531ec..3340be8 100644 --- a/modules/lagoon/validate_rules.go +++ b/modules/lagoon/validate_rules.go @@ -34,7 +34,7 @@ var imageExtensions = []string{"jpg", "jpeg", "png", "gif", "bmp", "svg", "webp" // parameter count: -1 any number (at least one), 0 none. var requestRuleArity = map[string]int{ "required": 0, "present": 0, "filled": 0, "accepted": 0, - "nullable": 0, "sometimes": 0, "bail": 0, + "nullable": 0, "sometimes": 0, "bail": 0, "prohibited": 0, "array": -2, "string": 0, "integer": 0, "numeric": 0, "boolean": 0, "email": 0, "url": 0, "date": 0, "after": 1, "after_or_equal": 1, "before": 1, "before_or_equal": 1, @@ -278,6 +278,10 @@ func (v *requestValidator) passes(rule Rule, attr string, value any, present boo return !present || validateRequired(value), nil case "accepted": return validateRequired(value) && isAccepted(value), nil + case "prohibited": + // Laravel 9 validateProhibited: !validateRequired. Not implicit, so it + // never runs for an absent key or a blank string. + return !validateRequired(value), nil case "nullable", "sometimes", "bail": return true, nil case "array": diff --git a/modules/tide/README.md b/modules/tide/README.md index de4164f..1649c0b 100644 --- a/modules/tide/README.md +++ b/modules/tide/README.md @@ -16,10 +16,10 @@ HTTP parity toolkit that records request and response fixtures from a reference - Recording proxy: `tide.NewProxy` builds a reverse proxy that only binds to and forwards to loopback addresses, groups traffic into named sessions (from the `tide.SessionHeader` request header or a default session) and writes one fixture per complete session on `tide.Proxy.Flush`. - Capture rules: `tide.Rules` (loaded with `tide.LoadRules`) decide which request and response headers are kept per route and which values are captured into variables, from response JSON paths, headers, redirect query strings or form fields. - Variables: `tide.Store` holds captured values such as tokens and IDs in a mode-0600 file, `tide.Store.Expand` substitutes `{{name}}` placeholders before a request is sent, and `tide.ScrubStep` puts placeholders back into fixtures. Scrubbing fails when a step still holds an unclassified token- or password-shaped value, so credentials do not leak into committed fixtures. -- Replay and diff: `tide.ReplayFlow` re-sends each step, compares status, a fixed set of contract headers and the body, and returns `tide.Result` with per-step `tide.Diff` entries. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies are compared byte for byte. Uploaded-file URLs under `url` and `thumb_url` keys are compared by shape: under the uploads prefix (`tide.ReplayConfig` `UploadPrefix`, default `tide.DefaultUploadPrefix`, `/storage/app/uploads/public`) an original must be `/xxx/yyy/zzz/` with the partition taken from the disk name, and a thumbnail `/xxx/yyy/zzz/thumb______.`. The partition, disk name and file id are masked; the prefix, size, offsets, mode and extension stay, so a thumbnail of another size or an upload URL under another prefix is still a difference. +- Replay and diff: `tide.ReplayFlow` re-sends each step, compares status, a fixed set of contract headers and the body, and returns `tide.Result` with per-step `tide.Diff` entries. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies are compared byte for byte. A `Content-Disposition` header is compared after masking its `YYYY-MM-DD` dates on both sides, so a download named after the day it was made, such as `attachment; filename=export-2026-09-17.csv`, replays on a later day; each masked token must be a real calendar date, and a different name, an invalid date or a date on one side only is still a difference. Uploaded-file URLs under `url` and `thumb_url` keys are compared by shape: under the uploads prefix (`tide.ReplayConfig` `UploadPrefix`, default `tide.DefaultUploadPrefix`, `/storage/app/uploads/public`) an original must be `/xxx/yyy/zzz/` with the partition taken from the disk name, and a thumbnail `/xxx/yyy/zzz/thumb______.`. The partition, disk name and file id are masked; the prefix, size, offsets, mode and extension stay, so a thumbnail of another size or an upload URL under another prefix is still a difference. - Fake Centrifugo: `tide.NewCentrifugoRecorder` returns an `http.Handler` that records every POST to a path ending in `/publish` or `/broadcast` as a `tide.Publication` (method, path, whether `Authorization: apikey ` carried the configured key, JSON body) and answers `{"result":{}}`. Paths ending in `/presence` answer `{"result":{"presence":{}}}`, `/unsubscribe` and `/info` answer `{"result":{}}`, anything else is 404. Bodies are capped at `tide.MaxPublicationBody` (1 MiB). The API key is only compared, never stored. `tide.CentrifugoRecorder.ListenAndServe` binds loopback addresses only, like the recording proxy. - Broadcast goldens: `tide.RecordBroadcasts` runs a flow against a loopback reference backend whose Centrifugo API URL points at a recorder on `tide.DefaultCentrifugoListen` (`127.0.0.1:8424`). With `tide.BroadcastConfig` `Step` set, earlier steps run as setup and only that step's publications are kept. The result is a `tide.BroadcastGolden`, written with `tide.WriteBroadcastGolden` (which refuses token-shaped bodies) and read strictly with `tide.LoadBroadcastGolden`. A golden with `pending` set is recorded but not yet asserted. -- Broadcast normalisation: `tide.NormalizePublications` masks only `$.data.timestamp` and `$.data.payload.timestamp` (ISO 8601 with an offset) as `"{{timestamp}}"`, `$.data.payload.actor` (an object of exactly `user_id` and `name`) as `"{{actor}}"`, and values equal to an `id:*` variable of a `tide.Store`: numbers or strings under `id`, `*_id` or `*_ids` keys, and the numeric last segment of a channel name such as `room:12`. Carbon `+00:00` values of `*_at` keys anywhere under `$.data.payload.album` become `"{{datetime}}"`; a date of another shape is left as it is, so a format change shows as a difference. A masked number is written as a bare `{{id:name}}`, so a number that becomes a string still differs. A value matching two id variables is an error. `tide.DiffPublications` compares the count, method, path, authorization flag and body (structurally, key order ignored) and reports paths such as `$[0].body.data.payload.id`. +- Broadcast normalisation: `tide.NormalizePublications` masks only `$.data.timestamp` and `$.data.payload.timestamp` (ISO 8601 with an offset) as `"{{timestamp}}"`, `$.data.payload.actor` (an object of exactly `user_id` and `name`) as `"{{actor}}"`, and values equal to an `id:*` variable of a `tide.Store`: numbers or strings under `id`, `*_id` or `*_ids` keys, and the numeric last segment of a channel name such as `room:12`. Carbon `+00:00` values of `*_at` keys anywhere under `$.data.payload.album` become `"{{datetime}}"`. A notification publication's own row fields are masked too: a Carbon `+00:00` `$.data.payload.created_at` becomes `"{{datetime}}"`, and a positive integer `$.data.payload.id` that no `id:*` variable names becomes a bare `{{id}}` (a captured one keeps `{{id:name}}`); a date of another shape is left as it is, so a format change shows as a difference. A masked number is written as a bare `{{id:name}}`, so a number that becomes a string still differs. A value matching two id variables is an error. `tide.DiffPublications` compares the count, method, path, authorization flag and body (structurally, key order ignored) and reports paths such as `$[0].body.data.payload.id`. - Manifests: `tide.Manifest` lists routes with auth groups, a pending or ported status, cases and fixture paths; `tide.RecordManifest` records missing cases in batches of at most `tide.MaxBatch`, and `tide.ReplayManifest` replays every recorded case into a `tide.Coverage` table. ## Usage diff --git a/modules/tide/centrifugo_golden.go b/modules/tide/centrifugo_golden.go index 6153c4e..7a19796 100644 --- a/modules/tide/centrifugo_golden.go +++ b/modules/tide/centrifugo_golden.go @@ -275,7 +275,27 @@ func (n *normalizer) lookup(v string) (string, bool) { } } +// positiveIntRe is a JSON number literal holding a positive integer. +var positiveIntRe = regexp.MustCompile(`^[1-9][0-9]*$`) + func (n *normalizer) walk(path, key string, v *onode) *onode { + // A notification publication carries the new row's id and created_at + // under $.data.payload. A Carbon +00:00 created_at is masked like an album + // date; a positive integer id keeps its captured variable's placeholder + // and is {{id}} when no captured variable names it. Any other shape at + // these paths stays visible. + switch { + case path == "$.data.payload.created_at" && v.kind == kindString && carbonOffsetRe.MatchString(v.str): + return &onode{kind: kindString, str: "{{datetime}}"} + case path == "$.data.payload.id" && v.kind == kindNumber && positiveIntRe.MatchString(v.str): + if name, ok := n.lookup(v.str); ok { + return &onode{kind: kindPlaceholder, str: name} + } + if len(n.ids[v.str]) == 0 { + return &onode{kind: kindPlaceholder, str: "id"} + } + return v + } switch path { case "$.data.timestamp", "$.data.payload.timestamp": if v.kind == kindString && isoOffsetRe.MatchString(v.str) { diff --git a/modules/tide/diff.go b/modules/tide/diff.go index 4baee7f..1e86b00 100644 --- a/modules/tide/diff.go +++ b/modules/tide/diff.go @@ -6,8 +6,10 @@ import ( "fmt" "mime" "net/http" + "regexp" "strconv" "strings" + "time" "unicode" "unicode/utf8" ) @@ -81,6 +83,9 @@ func compareHeaders(want, got, extra map[string]string) []Diff { if gv == wv { continue } + if ck == "Content-Disposition" && sameDispositionButDates(wv, gv) { + continue + } if gv == "" { gv = "" } @@ -290,3 +295,32 @@ func quotePrintable(b []byte) string { buf.WriteByte('"') return buf.String() } + +// dispositionDateRe finds YYYY-MM-DD tokens in a Content-Disposition value, +// such as the date in a download name built from the current day. +var dispositionDateRe = regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}\b`) + +// maskDispositionDates replaces every date token with one placeholder and +// reports how many it replaced. ok is false when a token is not a real +// calendar date, so the caller falls back to the byte comparison. +func maskDispositionDates(s string) (masked string, n int, ok bool) { + ok = true + masked = dispositionDateRe.ReplaceAllStringFunc(s, func(m string) string { + if _, err := time.Parse("2006-01-02", m); err != nil { + ok = false + return m + } + n++ + return "{{date}}" + }) + return masked, n, ok +} + +// sameDispositionButDates reports whether two Content-Disposition values +// differ only in their dates: both carry the same number of real calendar +// dates and are equal once those are masked. Anything else stays a Diff. +func sameDispositionButDates(want, got string) bool { + wm, wn, wok := maskDispositionDates(want) + gm, gn, gok := maskDispositionDates(got) + return wok && gok && wn > 0 && wn == gn && wm == gm +} diff --git a/modules/tide/multipart_test.go b/modules/tide/multipart_test.go index 0455436..eb24456 100644 --- a/modules/tide/multipart_test.go +++ b/modules/tide/multipart_test.go @@ -265,7 +265,7 @@ func TestNormalizePublicationAlbumDates(t *testing.T) { pub := func(created, updated string) []Publication { return []Publication{{Method: "POST", Path: "/api/publish", Body: json.RawMessage( `{"channel":"c","data":{"payload":{"album":{"name":"x","created_at":"` + created + `","updated_at":"` + updated + - `","market_price_checked_at":null,"photos":[{"created_at":"` + created + `"}]},"created_at":"2026-01-01T00:00:00+00:00"}}}`)}} + `","market_price_checked_at":null,"photos":[{"created_at":"` + created + `"}]},"published_at":"2026-01-01T00:00:00+00:00"}}}`)}} } a, err := NormalizePublications(pub("2026-09-30T11:21:55+00:00", "2026-09-30T11:21:56+00:00"), store) if err != nil { @@ -281,8 +281,10 @@ func TestNormalizePublicationAlbumDates(t *testing.T) { if !strings.Contains(string(a[0].Body), `"updated_at":"{{datetime}}"`) || !strings.Contains(string(a[0].Body), `"market_price_checked_at":null`) { t.Fatalf("album dates not masked: %s", a[0].Body) } - // Only the album subtree is masked; a payload date outside it stays. - if !strings.Contains(string(a[0].Body), `},"created_at":"2026-01-01T00:00:00+00:00"`) { + // Only the album subtree is masked; a payload date outside it stays + // ($.data.payload.created_at itself is the notification row's date and + // has its own mask, see TestNormalizeNotificationPublication). + if !strings.Contains(string(a[0].Body), `},"published_at":"2026-01-01T00:00:00+00:00"`) { t.Fatalf("over-normalised: %s", a[0].Body) } // A Z-suffixed album date keeps its value, so a format change is a diff. diff --git a/modules/tide/normalize_phase13_test.go b/modules/tide/normalize_phase13_test.go new file mode 100644 index 0000000..2329fdd --- /dev/null +++ b/modules/tide/normalize_phase13_test.go @@ -0,0 +1,124 @@ +package tide + +import ( + "encoding/json" + "strings" + "testing" +) + +// TestNormalizeContentDispositionDate: a download name built from the day +// of the request replays on a later day, while a different stem, an invalid +// date, a date on one side only or a missing header still diff. +func TestNormalizeContentDispositionDate(t *testing.T) { + const recorded = "attachment; filename=export-2026-09-17.csv" + cases := []struct { + name string + got map[string]string + diff bool + }{ + {"same stem on a later day", map[string]string{"Content-Disposition": "attachment; filename=export-2026-10-03.csv"}, false}, + {"same day", map[string]string{"Content-Disposition": recorded}, false}, + {"header name case", map[string]string{"content-disposition": "attachment; filename=export-2027-01-01.csv"}, false}, + {"different stem", map[string]string{"Content-Disposition": "attachment; filename=report-2026-10-03.csv"}, true}, + {"invalid date", map[string]string{"Content-Disposition": "attachment; filename=export-2026-13-45.csv"}, true}, + {"date on one side only", map[string]string{"Content-Disposition": "attachment; filename=export.csv"}, true}, + {"inline instead of attachment", map[string]string{"Content-Disposition": "inline; filename=export-2026-10-03.csv"}, true}, + {"header absent", map[string]string{}, true}, + } + for _, c := range cases { + diffs := compareHeaders(map[string]string{"Content-Disposition": recorded}, c.got, nil) + if (len(diffs) > 0) != c.diff { + t.Errorf("%s: diffs = %+v, want diff=%v", c.name, diffs, c.diff) + } + } + // A recorded value that holds an invalid date keeps the byte comparison. + odd := "attachment; filename=export-2026-02-30.csv" + if diffs := compareHeaders(map[string]string{"Content-Disposition": odd}, map[string]string{"Content-Disposition": "attachment; filename=export-2026-02-28.csv"}, nil); len(diffs) != 1 { + t.Errorf("invalid recorded date: diffs = %+v, want one", diffs) + } + // Two dates must both be real and both present. + two := "attachment; filename=export-2026-09-01-2026-09-17.csv" + if diffs := compareHeaders(map[string]string{"Content-Disposition": two}, map[string]string{"Content-Disposition": "attachment; filename=export-2026-10-01-2026-10-03.csv"}, nil); len(diffs) != 0 { + t.Errorf("two dates: diffs = %+v, want none", diffs) + } + // The mask applies to Content-Disposition only. + if diffs := compareHeaders(map[string]string{"Location": "/files/2026-09-17"}, map[string]string{"Location": "/files/2026-10-03"}, nil); len(diffs) != 1 { + t.Errorf("Location: diffs = %+v, want one", diffs) + } + if _, n, ok := maskDispositionDates("x12026-09-170"); n != 0 || !ok { + t.Errorf("a date inside a longer digit run is not a date token") + } +} + +// TestNormalizeNotificationPublication: a notification:new publication's id +// and created_at under $.data.payload normalize equal across two runs, while +// a Z date, a string id and every other path stay visible. +func TestNormalizeNotificationPublication(t *testing.T) { + store := mustMemoryStore() + pub := func(id, created string) []Publication { + return []Publication{{Method: "POST", Path: "/api/publish", Body: json.RawMessage( + `{"channel":"acme#5","data":{"event":"notification:new","payload":{"id":` + id + + `,"type":"item_added","payload":{"album_id":3,"created_at":"2026-01-01T00:00:00+00:00"},"read_at":null,"created_at":"` + created + `"}}}`)}} + } + a, err := NormalizePublications(pub("10000001", "2026-09-30T11:21:55+00:00"), store) + if err != nil { + t.Fatal(err) + } + b, err := NormalizePublications(pub("42", "2026-10-03T08:00:00+00:00"), store) + if err != nil { + t.Fatal(err) + } + if diffs := DiffPublications(a, b); len(diffs) != 0 { + t.Fatalf("diffs = %+v", diffs) + } + body := string(a[0].Body) + if !strings.Contains(body, `"payload":{"id":{{id}},`) || !strings.HasSuffix(body, `"read_at":null,"created_at":"{{datetime}}"}}}`) { + t.Fatalf("notification not masked: %s", body) + } + // Only $.data.payload.created_at is masked, not a nested payload date. + if !strings.Contains(body, `"album_id":3,"created_at":"2026-01-01T00:00:00+00:00"`) { + t.Fatalf("over-normalised: %s", body) + } + + // A Z date, a zero or negative id and a string id stay visible. + for _, c := range []struct{ id, created, want string }{ + {"42", "2026-10-03T08:00:00Z", `"created_at":"2026-10-03T08:00:00Z"`}, + {`"42"`, "2026-10-03T08:00:00+00:00", `"payload":{"id":"42",`}, + {"0", "2026-10-03T08:00:00+00:00", `"payload":{"id":0,`}, + {"-3", "2026-10-03T08:00:00+00:00", `"payload":{"id":-3,`}, + {"4.5", "2026-10-03T08:00:00+00:00", `"payload":{"id":4.5,`}, + } { + got, err := NormalizePublications(pub(c.id, c.created), store) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(got[0].Body), c.want) { + t.Errorf("id %s created %s: %s lacks %s", c.id, c.created, got[0].Body, c.want) + } + if diffs := DiffPublications(a, got); len(diffs) == 0 { + t.Errorf("id %s created %s must diff against the masked publication", c.id, c.created) + } + } + + // A captured id keeps its own placeholder. + store.Set("id:note", "77") + got, err := NormalizePublications(pub("77", "2026-10-03T08:00:00+00:00"), store) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(got[0].Body), `"payload":{"id":{{id:note}},`) { + t.Fatalf("captured id: %s", got[0].Body) + } + + // An album publication's existing masks are unchanged. + album := []Publication{{Method: "POST", Path: "/api/publish", Body: json.RawMessage( + `{"channel":"c","data":{"payload":{"album":{"created_at":"2026-09-30T11:21:55+00:00"},"actor":{"user_id":1,"name":null},"timestamp":"2026-09-30T11:21:55+00:00"}}}`)}} + got, err = NormalizePublications(album, store) + if err != nil { + t.Fatal(err) + } + want := `{"channel":"c","data":{"payload":{"album":{"created_at":"{{datetime}}"},"actor":"{{actor}}","timestamp":"{{timestamp}}"}}}` + if string(got[0].Body) != want { + t.Fatalf("album publication\n got %s\nwant %s", got[0].Body, want) + } +}