feat(13-01): add the prohibited rule and dated-download and notification masks
- lagoon.ValidateRequest supports Laravel 9 prohibited (!required, not
implicit); with no catalog line its message is validation.prohibited
- tide compares Content-Disposition with real calendar dates masked on both
sides; a different name, an invalid date or a one-sided date still diffs
- tide.NormalizePublications masks a Carbon +00:00 $.data.payload.created_at
and an uncaptured positive integer $.data.payload.id as {{id}}
- the album-date test's outside-album sibling moves off payload.created_at,
which now has its own mask
- READMEs and docs describe the rule and both masks
This commit is contained in:
@@ -127,6 +127,7 @@ The behaviour follows Laravel:
|
|||||||
- The other rules run only when there is a value: they skip an absent attribute, a blank string, `null` under `nullable` and an absent key under `sometimes`.
|
- The other rules run only when there is a value: they skip an absent attribute, a blank string, `null` under `nullable` and an absent key under `sometimes`.
|
||||||
- `min`, `max`, `size` and `between` measure what Laravel measures: the number on an `integer` or `numeric` attribute, compared exactly as a decimal; the number of elements of an array; the size of an uploaded file in kilobytes; otherwise the length in characters, not bytes. Each picks the matching message, such as `max.string` or `max.file`.
|
- `min`, `max`, `size` and `between` measure what Laravel measures: the number on an `integer` or `numeric` attribute, compared exactly as a decimal; the number of elements of an array; the size of an uploaded file in kilobytes; otherwise the length in characters, not bytes. Each picks the matching message, such as `max.string` or `max.file`.
|
||||||
- `email` is PHP's `FILTER_VALIDATE_EMAIL`, the check WinterCMS uses by default, so `user@localhost` is refused. `date` accepts ISO 8601 dates and date-times, `Y/m/d`, `m/d/Y`, `d.m.Y` and `d-m-Y`, and refuses dates that do not exist. `after_or_equal` and `before_or_equal` take a date, `today`, `tomorrow`, `yesterday` or `now` (in UTC), or the name of another field.
|
- `email` is PHP's `FILTER_VALIDATE_EMAIL`, the check WinterCMS uses by default, so `user@localhost` is refused. `date` accepts ISO 8601 dates and date-times, `Y/m/d`, `m/d/Y`, `d.m.Y` and `d-m-Y`, and refuses dates that do not exist. `after_or_equal` and `before_or_equal` take a date, `today`, `tomorrow`, `yesterday` or `now` (in UTC), or the name of another field.
|
||||||
|
- `prohibited` refuses a field the endpoint must never accept: it fails for any value that `required` would accept, including `0`, `false` and a non-empty array. Like the other non-implicit rules it never runs for an absent key or a blank string, and `null` or an empty array passes. Laravel 9's catalogs have no line for it, so the message is the key itself, `validation.prohibited`, as the PHP endpoint answers.
|
||||||
- `exists:table,column` counts matching rows in the database, so it needs the transaction handle. Like Laravel it does not skip soft-deleted rows, and it does not run once the attribute already has a message.
|
- `exists:table,column` counts matching rows in the database, so it needs the transaction handle. Like Laravel it does not skip soft-deleted rows, and it does not run once the attribute already has a message.
|
||||||
- File rules (`file`, `image`, `mimes` and the size rules) take a `lagoon.UploadedFile`; `lagoon.UploadedFileFromHeader` builds one from a parsed multipart part. The type is read from the file content, not from its name.
|
- File rules (`file`, `image`, `mimes` and the size rules) take a `lagoon.UploadedFile`; `lagoon.UploadedFileFromHeader` builds one from a parsed multipart part. The type is read from the file content, not from its name.
|
||||||
|
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ steps:
|
|||||||
path: /api/blog/posts?page=1
|
path: /api/blog/posts?page=1
|
||||||
```
|
```
|
||||||
|
|
||||||
Recording sends each request to the reference backend and fills in the responses. Replaying sends them to the port and compares status, a fixed set of contract headers and the body. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies byte for byte:
|
Recording sends each request to the reference backend and fills in the responses. Replaying sends them to the port and compares status, a fixed set of contract headers and the body. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies byte for byte. Among the headers, `Content-Disposition` is compared with its dates masked: a CSV download named `export-2026-09-17.csv` on the recording day replays as `export-2026-10-03.csv` a few weeks later. Each date must be a real calendar date on both sides, so a renamed file, a broken date or a date that disappears still fails:
|
||||||
|
|
||||||
```go src=modules/tide/example_test.go#ExampleReplayFlow
|
```go src=modules/tide/example_test.go#ExampleReplayFlow
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
@@ -121,6 +121,6 @@ Realtime side effects are part of the contract too. `summer parity:broadcasts` r
|
|||||||
summer parity:broadcasts --flow testdata/broadcasts/flows/post-lifecycle.yaml --step delete --name deleted --target http://127.0.0.1:8000 --vars /tmp/parity/vars.yaml --out testdata/broadcasts/deleted.yaml
|
summer parity:broadcasts --flow testdata/broadcasts/flows/post-lifecycle.yaml --step delete --name deleted --target http://127.0.0.1:8000 --vars /tmp/parity/vars.yaml --out testdata/broadcasts/deleted.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
Point the reference backend's Centrifugo API URL at the recorder (`127.0.0.1:8424` by default). `--step` keeps only the publications of one step, running the earlier steps as setup. Timestamps, the actor, the `*_at` dates inside a published album and captured IDs are masked (`tide.NormalizePublications`), so the Go port's publications, recorded the same way, compare with `tide.DiffPublications`. A golden with `--pending` set is recorded but not yet asserted.
|
Point the reference backend's Centrifugo API URL at the recorder (`127.0.0.1:8424` by default). `--step` keeps only the publications of one step, running the earlier steps as setup. Timestamps, the actor, the `*_at` dates inside a published album, a notification's own `created_at` and `id` under `$.data.payload` (the id only when it is a positive integer, as `{{id}}` unless a captured ID names it) and captured IDs are masked (`tide.NormalizePublications`), so the Go port's publications, recorded the same way, compare with `tide.DiffPublications`. A golden with `--pending` set is recorded but not yet asserted.
|
||||||
|
|
||||||
On the Go side, the memory realtime driver records publications the same way in tests; see [Realtime](realtime.md).
|
On the Go side, the memory realtime driver records publications the same way in tests; see [Realtime](realtime.md).
|
||||||
|
|||||||
@@ -18,7 +18,7 @@ Postgres data layer: the shared GORM connection, per-plugin migrations, model he
|
|||||||
- Per-plugin migrations: `lagoon.Migrate` runs the framework's `system_files` set (`attach.Migrations`), backend admin identity set (`lagoon.BackendAdminMigrations`), `deferred_bindings` set (`lagoon.DeferredBindingMigrations`, under the `lagoon.DeferredHistoryID` history) and job-queue set (`lagoon.QueueMigrations`: River's schema pinned at `lagoon.RiverSchemaVersion`, then the `lagoon.JobsTable` record table, under the `lagoon.QueueHistoryID` history), then every `pact.HasMigrations` set in plugin activation order, each in its own `summer_migrations_<plugin_id>` history table (`lagoon.HistoryTableName`). `lagoon.RollbackLast` and `lagoon.Status` cover rollback and history.
|
- Per-plugin migrations: `lagoon.Migrate` runs the framework's `system_files` set (`attach.Migrations`), backend admin identity set (`lagoon.BackendAdminMigrations`), `deferred_bindings` set (`lagoon.DeferredBindingMigrations`, under the `lagoon.DeferredHistoryID` history) and job-queue set (`lagoon.QueueMigrations`: River's schema pinned at `lagoon.RiverSchemaVersion`, then the `lagoon.JobsTable` record table, under the `lagoon.QueueHistoryID` history), then every `pact.HasMigrations` set in plugin activation order, each in its own `summer_migrations_<plugin_id>` history table (`lagoon.HistoryTableName`). `lagoon.RollbackLast` and `lagoon.Status` cover rollback and history.
|
||||||
- Mass assignment: `lagoon.Fill` copies only allow-listed keys onto a model by GORM column name and silently drops the rest, logging each dropped key once outside production. A `json.Number` (from a decoder using `UseNumber`) fills integer, unsigned and float fields. A value that does not fit its column (a fraction, an exponent or an overflow for an integer field, or a value of the wrong type) is a `lagoon.FillTypeError` naming the key, so a caller can answer it as a validation failure on that field. `lagoon.HasFillable` and `lagoon.HasHidden` are the Go forms of `$fillable` and `$hidden`.
|
- Mass assignment: `lagoon.Fill` copies only allow-listed keys onto a model by GORM column name and silently drops the rest, logging each dropped key once outside production. A `json.Number` (from a decoder using `UseNumber`) fills integer, unsigned and float fields. A value that does not fit its column (a fraction, an exponent or an overflow for an integer field, or a value of the wrong type) is a `lagoon.FillTypeError` naming the key, so a caller can answer it as a validation failure on that field. `lagoon.HasFillable` and `lagoon.HasHidden` are the Go forms of `$fillable` and `$hidden`.
|
||||||
- Validation: `lagoon.Validate` (where `required` fails on a zero date or time) accepts Laravel-style rule strings (`required`, `nullable`, `integer`, `numeric`, `between`, `min`, `max`, `in`, `unique`, `boolean`, `email`, `confirmed`, `different`, `mimes`) and returns a field-to-messages map, translated through phrasebook when a translator is given. Unknown rule tokens are an error. A failed numeric range reports the bound that failed: the `min` message below the lower bound, the `max` message above the upper one, and the numeric `between` message when the bound came from `between`.
|
- Validation: `lagoon.Validate` (where `required` fails on a zero date or time) accepts Laravel-style rule strings (`required`, `nullable`, `integer`, `numeric`, `between`, `min`, `max`, `in`, `unique`, `boolean`, `email`, `confirmed`, `different`, `mimes`) and returns a field-to-messages map, translated through phrasebook when a translator is given. Unknown rule tokens are an error. A failed numeric range reports the bound that failed: the `min` message below the lower bound, the `max` message above the upper one, and the numeric `between` message when the bound came from `between`.
|
||||||
- Request validation: `lagoon.ValidateRequest` reproduces Laravel 9 request validation for ported API endpoints, so a 422 body matches the PHP one message for message. It takes the decoded input and an ordered `lagoon.RequestRule` table (attribute names may hold `*` wildcards, expanded against the input to `posts.0.title`), runs the rules of each attribute in order and stops an attribute after a failed implicit rule (`required`, `present`, `filled`, `accepted`) or, under `bail`, after any failure. A non-implicit rule is skipped for an absent attribute, a blank string, a null value under `nullable` and an absent key under `sometimes`. Supported rules: `required`, `present`, `filled`, `accepted`, `nullable`, `sometimes`, `bail`, `array`, `string`, `integer`, `numeric`, `boolean`, `email` (PHP `FILTER_VALIDATE_EMAIL`, WinterCMS's default), `url`, `date`, `after`, `after_or_equal`, `before`, `before_or_equal` (a date, a relative word such as `tomorrow`, or another field), `exists:table,column`, `regex`, `not_regex`, `in`, `not_in`, `file`, `image`, `mimes`, `min`, `max`, `size` and `between`, plus closure rules built with `lagoon.CustomRule`. The size rules compare the number under `numeric` or `integer` (exactly, as decimals), the element count of an array, kilobytes of a `lagoon.UploadedFile`, and otherwise the length in characters, and pick the matching message. Messages come from the `lagoon::validation` catalog in the request locale; `lagoon.ErrorKeys` gives the attribute order of PHP's message bag.
|
- Request validation: `lagoon.ValidateRequest` reproduces Laravel 9 request validation for ported API endpoints, so a 422 body matches the PHP one message for message. It takes the decoded input and an ordered `lagoon.RequestRule` table (attribute names may hold `*` wildcards, expanded against the input to `posts.0.title`), runs the rules of each attribute in order and stops an attribute after a failed implicit rule (`required`, `present`, `filled`, `accepted`) or, under `bail`, after any failure. A non-implicit rule is skipped for an absent attribute, a blank string, a null value under `nullable` and an absent key under `sometimes`. Supported rules: `required`, `present`, `filled`, `accepted`, `nullable`, `sometimes`, `bail`, `array`, `string`, `integer`, `numeric`, `boolean`, `email` (PHP `FILTER_VALIDATE_EMAIL`, WinterCMS's default), `url`, `date`, `after`, `after_or_equal`, `before`, `before_or_equal` (a date, a relative word such as `tomorrow`, or another field), `exists:table,column`, `regex`, `not_regex`, `in`, `not_in`, `file`, `image`, `mimes`, `min`, `max`, `size`, `between` and `prohibited` (fails for any value `required` would accept, `0` and `false` included; not implicit, so an absent key, `null`, a blank string and an empty array pass, and with no catalog line its message is the key `validation.prohibited`), plus closure rules built with `lagoon.CustomRule`. The size rules compare the number under `numeric` or `integer` (exactly, as decimals), the element count of an array, kilobytes of a `lagoon.UploadedFile`, and otherwise the length in characters, and pick the matching message. Messages come from the `lagoon::validation` catalog in the request locale; `lagoon.ErrorKeys` gives the attribute order of PHP's message bag.
|
||||||
- Safe ordering: `lagoon.OrderBy` appends an ORDER BY only for an allow-listed column and an `asc` or `desc` direction, and `lagoon.Collate` adds a validated `COLLATE` clause for language-specific text order (for example the ICU collation `pl-x-icu`); lagoon puts no requirement on the database's default locale.
|
- Safe ordering: `lagoon.OrderBy` appends an ORDER BY only for an allow-listed column and an `asc` or `desc` direction, and `lagoon.Collate` adds a validated `COLLATE` clause for language-specific text order (for example the ICU collation `pl-x-icu`); lagoon puts no requirement on the database's default locale.
|
||||||
- Pagination: `lagoon.Paginate` builds a `lagoon.Page` with `data` and `meta` (`current_page`, `last_page`, `per_page`, `total`).
|
- Pagination: `lagoon.Paginate` builds a `lagoon.Page` with `data` and `meta` (`current_page`, `last_page`, `per_page`, `total`).
|
||||||
- Date and time columns: `lagoon.Date` (a `DATE` column, JSON `"2026-10-02"`) and `lagoon.TimeOfDay` (a `TIME` column, JSON `"14:30:00"`) implement `sql.Scanner`, `driver.Valuer`, JSON and text marshalling, and store NULL for their zero value; `*lagoon.Date` and `*lagoon.TimeOfDay` are the nullable variants, next to `time.Time` and `*time.Time` for `timestamptz`. Build them with `lagoon.NewDate`, `lagoon.DateOf`, `lagoon.ParseDate`, `lagoon.NewTimeOfDay` and `lagoon.ParseTimeOfDay`. `lagoon.Fill` fills all six from JSON strings (RFC 3339 for `time.Time`) through their text unmarshalling, after every conversion it already made. Behaviour change: `required` now treats a zero `time.Time`, `lagoon.Date` or `lagoon.TimeOfDay` (or a pointer to one) as empty, so declare optional dates as pointer fields.
|
- Date and time columns: `lagoon.Date` (a `DATE` column, JSON `"2026-10-02"`) and `lagoon.TimeOfDay` (a `TIME` column, JSON `"14:30:00"`) implement `sql.Scanner`, `driver.Valuer`, JSON and text marshalling, and store NULL for their zero value; `*lagoon.Date` and `*lagoon.TimeOfDay` are the nullable variants, next to `time.Time` and `*time.Time` for `timestamptz`. Build them with `lagoon.NewDate`, `lagoon.DateOf`, `lagoon.ParseDate`, `lagoon.NewTimeOfDay` and `lagoon.ParseTimeOfDay`. `lagoon.Fill` fills all six from JSON strings (RFC 3339 for `time.Time`) through their text unmarshalling, after every conversion it already made. Behaviour change: `required` now treats a zero `time.Time`, `lagoon.Date` or `lagoon.TimeOfDay` (or a pointer to one) as empty, so declare optional dates as pointer fields.
|
||||||
|
|||||||
@@ -664,3 +664,52 @@ func TestValidateRequestExistsRule(t *testing.T) {
|
|||||||
t.Error("a missing table must be an error for arrays too")
|
t.Error("a missing table must be an error for arrays too")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestValidateRequestProhibited ports Laravel 9's prohibited rule: it is
|
||||||
|
// !validateRequired and not implicit, so an absent key and a blank string
|
||||||
|
// are never validated, null and an empty array pass, and any value required
|
||||||
|
// would accept fails, 0 and false included. Neither catalog has a
|
||||||
|
// validation.prohibited line, so the message is the key itself.
|
||||||
|
func TestValidateRequestProhibited(t *testing.T) {
|
||||||
|
rules := []RequestRule{{Field: "condition", Rules: ParseRules("prohibited")}}
|
||||||
|
pass := map[string]map[string]any{
|
||||||
|
"absent": {},
|
||||||
|
"null": {"condition": nil},
|
||||||
|
"empty string": {"condition": ""},
|
||||||
|
"blank string": {"condition": " "},
|
||||||
|
"empty array": {"condition": []any{}},
|
||||||
|
"empty map": {"condition": map[string]any{}},
|
||||||
|
}
|
||||||
|
for name, input := range pass {
|
||||||
|
if got := mustValidate(t, inLocale("pl"), input, rules); len(got) != 0 {
|
||||||
|
t.Errorf("%s: errors = %v, want none", name, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fail := map[string]any{
|
||||||
|
"string": "VG",
|
||||||
|
"zero": json.Number("0"),
|
||||||
|
"float zero": float64(0),
|
||||||
|
"false": false,
|
||||||
|
"true": true,
|
||||||
|
"non-empty array": []any{"x"},
|
||||||
|
"non-empty map": map[string]any{"a": 1},
|
||||||
|
}
|
||||||
|
for name, value := range fail {
|
||||||
|
for _, loc := range []string{"pl", "en"} {
|
||||||
|
got := mustValidate(t, inLocale(loc), map[string]any{"condition": value}, rules)
|
||||||
|
want := map[string][]string{"condition": {"validation.prohibited"}}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Errorf("%s (%s): errors = %#v, want %#v", name, loc, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Next to other rules, prohibited fails like any non-implicit rule.
|
||||||
|
mixed := []RequestRule{
|
||||||
|
{Field: "name", Rules: ParseRules("required|string")},
|
||||||
|
{Field: "shelf", Rules: ParseRules("prohibited")},
|
||||||
|
}
|
||||||
|
got := mustValidate(t, inLocale("en"), map[string]any{"name": "Kind of Blue", "shelf": "A1"}, mixed)
|
||||||
|
if want := map[string][]string{"shelf": {"validation.prohibited"}}; !reflect.DeepEqual(got, want) {
|
||||||
|
t.Errorf("mixed = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -34,7 +34,7 @@ var imageExtensions = []string{"jpg", "jpeg", "png", "gif", "bmp", "svg", "webp"
|
|||||||
// parameter count: -1 any number (at least one), 0 none.
|
// parameter count: -1 any number (at least one), 0 none.
|
||||||
var requestRuleArity = map[string]int{
|
var requestRuleArity = map[string]int{
|
||||||
"required": 0, "present": 0, "filled": 0, "accepted": 0,
|
"required": 0, "present": 0, "filled": 0, "accepted": 0,
|
||||||
"nullable": 0, "sometimes": 0, "bail": 0,
|
"nullable": 0, "sometimes": 0, "bail": 0, "prohibited": 0,
|
||||||
"array": -2, "string": 0, "integer": 0, "numeric": 0, "boolean": 0,
|
"array": -2, "string": 0, "integer": 0, "numeric": 0, "boolean": 0,
|
||||||
"email": 0, "url": 0, "date": 0,
|
"email": 0, "url": 0, "date": 0,
|
||||||
"after": 1, "after_or_equal": 1, "before": 1, "before_or_equal": 1,
|
"after": 1, "after_or_equal": 1, "before": 1, "before_or_equal": 1,
|
||||||
@@ -278,6 +278,10 @@ func (v *requestValidator) passes(rule Rule, attr string, value any, present boo
|
|||||||
return !present || validateRequired(value), nil
|
return !present || validateRequired(value), nil
|
||||||
case "accepted":
|
case "accepted":
|
||||||
return validateRequired(value) && isAccepted(value), nil
|
return validateRequired(value) && isAccepted(value), nil
|
||||||
|
case "prohibited":
|
||||||
|
// Laravel 9 validateProhibited: !validateRequired. Not implicit, so it
|
||||||
|
// never runs for an absent key or a blank string.
|
||||||
|
return !validateRequired(value), nil
|
||||||
case "nullable", "sometimes", "bail":
|
case "nullable", "sometimes", "bail":
|
||||||
return true, nil
|
return true, nil
|
||||||
case "array":
|
case "array":
|
||||||
|
|||||||
@@ -16,10 +16,10 @@ HTTP parity toolkit that records request and response fixtures from a reference
|
|||||||
- Recording proxy: `tide.NewProxy` builds a reverse proxy that only binds to and forwards to loopback addresses, groups traffic into named sessions (from the `tide.SessionHeader` request header or a default session) and writes one fixture per complete session on `tide.Proxy.Flush`.
|
- Recording proxy: `tide.NewProxy` builds a reverse proxy that only binds to and forwards to loopback addresses, groups traffic into named sessions (from the `tide.SessionHeader` request header or a default session) and writes one fixture per complete session on `tide.Proxy.Flush`.
|
||||||
- Capture rules: `tide.Rules` (loaded with `tide.LoadRules`) decide which request and response headers are kept per route and which values are captured into variables, from response JSON paths, headers, redirect query strings or form fields.
|
- Capture rules: `tide.Rules` (loaded with `tide.LoadRules`) decide which request and response headers are kept per route and which values are captured into variables, from response JSON paths, headers, redirect query strings or form fields.
|
||||||
- Variables: `tide.Store` holds captured values such as tokens and IDs in a mode-0600 file, `tide.Store.Expand` substitutes `{{name}}` placeholders before a request is sent, and `tide.ScrubStep` puts placeholders back into fixtures. Scrubbing fails when a step still holds an unclassified token- or password-shaped value, so credentials do not leak into committed fixtures.
|
- Variables: `tide.Store` holds captured values such as tokens and IDs in a mode-0600 file, `tide.Store.Expand` substitutes `{{name}}` placeholders before a request is sent, and `tide.ScrubStep` puts placeholders back into fixtures. Scrubbing fails when a step still holds an unclassified token- or password-shaped value, so credentials do not leak into committed fixtures.
|
||||||
- Replay and diff: `tide.ReplayFlow` re-sends each step, compares status, a fixed set of contract headers and the body, and returns `tide.Result` with per-step `tide.Diff` entries. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies are compared byte for byte. Uploaded-file URLs under `url` and `thumb_url` keys are compared by shape: under the uploads prefix (`tide.ReplayConfig` `UploadPrefix`, default `tide.DefaultUploadPrefix`, `/storage/app/uploads/public`) an original must be `<prefix>/xxx/yyy/zzz/<disk_name>` with the partition taken from the disk name, and a thumbnail `<prefix>/xxx/yyy/zzz/thumb_<id>_<w>_<h>_<ox>_<oy>_<mode>.<ext>`. The partition, disk name and file id are masked; the prefix, size, offsets, mode and extension stay, so a thumbnail of another size or an upload URL under another prefix is still a difference.
|
- Replay and diff: `tide.ReplayFlow` re-sends each step, compares status, a fixed set of contract headers and the body, and returns `tide.Result` with per-step `tide.Diff` entries. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies are compared byte for byte. A `Content-Disposition` header is compared after masking its `YYYY-MM-DD` dates on both sides, so a download named after the day it was made, such as `attachment; filename=export-2026-09-17.csv`, replays on a later day; each masked token must be a real calendar date, and a different name, an invalid date or a date on one side only is still a difference. Uploaded-file URLs under `url` and `thumb_url` keys are compared by shape: under the uploads prefix (`tide.ReplayConfig` `UploadPrefix`, default `tide.DefaultUploadPrefix`, `/storage/app/uploads/public`) an original must be `<prefix>/xxx/yyy/zzz/<disk_name>` with the partition taken from the disk name, and a thumbnail `<prefix>/xxx/yyy/zzz/thumb_<id>_<w>_<h>_<ox>_<oy>_<mode>.<ext>`. The partition, disk name and file id are masked; the prefix, size, offsets, mode and extension stay, so a thumbnail of another size or an upload URL under another prefix is still a difference.
|
||||||
- Fake Centrifugo: `tide.NewCentrifugoRecorder` returns an `http.Handler` that records every POST to a path ending in `/publish` or `/broadcast` as a `tide.Publication` (method, path, whether `Authorization: apikey <key>` carried the configured key, JSON body) and answers `{"result":{}}`. Paths ending in `/presence` answer `{"result":{"presence":{}}}`, `/unsubscribe` and `/info` answer `{"result":{}}`, anything else is 404. Bodies are capped at `tide.MaxPublicationBody` (1 MiB). The API key is only compared, never stored. `tide.CentrifugoRecorder.ListenAndServe` binds loopback addresses only, like the recording proxy.
|
- Fake Centrifugo: `tide.NewCentrifugoRecorder` returns an `http.Handler` that records every POST to a path ending in `/publish` or `/broadcast` as a `tide.Publication` (method, path, whether `Authorization: apikey <key>` carried the configured key, JSON body) and answers `{"result":{}}`. Paths ending in `/presence` answer `{"result":{"presence":{}}}`, `/unsubscribe` and `/info` answer `{"result":{}}`, anything else is 404. Bodies are capped at `tide.MaxPublicationBody` (1 MiB). The API key is only compared, never stored. `tide.CentrifugoRecorder.ListenAndServe` binds loopback addresses only, like the recording proxy.
|
||||||
- Broadcast goldens: `tide.RecordBroadcasts` runs a flow against a loopback reference backend whose Centrifugo API URL points at a recorder on `tide.DefaultCentrifugoListen` (`127.0.0.1:8424`). With `tide.BroadcastConfig` `Step` set, earlier steps run as setup and only that step's publications are kept. The result is a `tide.BroadcastGolden`, written with `tide.WriteBroadcastGolden` (which refuses token-shaped bodies) and read strictly with `tide.LoadBroadcastGolden`. A golden with `pending` set is recorded but not yet asserted.
|
- Broadcast goldens: `tide.RecordBroadcasts` runs a flow against a loopback reference backend whose Centrifugo API URL points at a recorder on `tide.DefaultCentrifugoListen` (`127.0.0.1:8424`). With `tide.BroadcastConfig` `Step` set, earlier steps run as setup and only that step's publications are kept. The result is a `tide.BroadcastGolden`, written with `tide.WriteBroadcastGolden` (which refuses token-shaped bodies) and read strictly with `tide.LoadBroadcastGolden`. A golden with `pending` set is recorded but not yet asserted.
|
||||||
- Broadcast normalisation: `tide.NormalizePublications` masks only `$.data.timestamp` and `$.data.payload.timestamp` (ISO 8601 with an offset) as `"{{timestamp}}"`, `$.data.payload.actor` (an object of exactly `user_id` and `name`) as `"{{actor}}"`, and values equal to an `id:*` variable of a `tide.Store`: numbers or strings under `id`, `*_id` or `*_ids` keys, and the numeric last segment of a channel name such as `room:12`. Carbon `+00:00` values of `*_at` keys anywhere under `$.data.payload.album` become `"{{datetime}}"`; a date of another shape is left as it is, so a format change shows as a difference. A masked number is written as a bare `{{id:name}}`, so a number that becomes a string still differs. A value matching two id variables is an error. `tide.DiffPublications` compares the count, method, path, authorization flag and body (structurally, key order ignored) and reports paths such as `$[0].body.data.payload.id`.
|
- Broadcast normalisation: `tide.NormalizePublications` masks only `$.data.timestamp` and `$.data.payload.timestamp` (ISO 8601 with an offset) as `"{{timestamp}}"`, `$.data.payload.actor` (an object of exactly `user_id` and `name`) as `"{{actor}}"`, and values equal to an `id:*` variable of a `tide.Store`: numbers or strings under `id`, `*_id` or `*_ids` keys, and the numeric last segment of a channel name such as `room:12`. Carbon `+00:00` values of `*_at` keys anywhere under `$.data.payload.album` become `"{{datetime}}"`. A notification publication's own row fields are masked too: a Carbon `+00:00` `$.data.payload.created_at` becomes `"{{datetime}}"`, and a positive integer `$.data.payload.id` that no `id:*` variable names becomes a bare `{{id}}` (a captured one keeps `{{id:name}}`); a date of another shape is left as it is, so a format change shows as a difference. A masked number is written as a bare `{{id:name}}`, so a number that becomes a string still differs. A value matching two id variables is an error. `tide.DiffPublications` compares the count, method, path, authorization flag and body (structurally, key order ignored) and reports paths such as `$[0].body.data.payload.id`.
|
||||||
- Manifests: `tide.Manifest` lists routes with auth groups, a pending or ported status, cases and fixture paths; `tide.RecordManifest` records missing cases in batches of at most `tide.MaxBatch`, and `tide.ReplayManifest` replays every recorded case into a `tide.Coverage` table.
|
- Manifests: `tide.Manifest` lists routes with auth groups, a pending or ported status, cases and fixture paths; `tide.RecordManifest` records missing cases in batches of at most `tide.MaxBatch`, and `tide.ReplayManifest` replays every recorded case into a `tide.Coverage` table.
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|||||||
@@ -275,7 +275,27 @@ func (n *normalizer) lookup(v string) (string, bool) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// positiveIntRe is a JSON number literal holding a positive integer.
|
||||||
|
var positiveIntRe = regexp.MustCompile(`^[1-9][0-9]*$`)
|
||||||
|
|
||||||
func (n *normalizer) walk(path, key string, v *onode) *onode {
|
func (n *normalizer) walk(path, key string, v *onode) *onode {
|
||||||
|
// A notification publication carries the new row's id and created_at
|
||||||
|
// under $.data.payload. A Carbon +00:00 created_at is masked like an album
|
||||||
|
// date; a positive integer id keeps its captured variable's placeholder
|
||||||
|
// and is {{id}} when no captured variable names it. Any other shape at
|
||||||
|
// these paths stays visible.
|
||||||
|
switch {
|
||||||
|
case path == "$.data.payload.created_at" && v.kind == kindString && carbonOffsetRe.MatchString(v.str):
|
||||||
|
return &onode{kind: kindString, str: "{{datetime}}"}
|
||||||
|
case path == "$.data.payload.id" && v.kind == kindNumber && positiveIntRe.MatchString(v.str):
|
||||||
|
if name, ok := n.lookup(v.str); ok {
|
||||||
|
return &onode{kind: kindPlaceholder, str: name}
|
||||||
|
}
|
||||||
|
if len(n.ids[v.str]) == 0 {
|
||||||
|
return &onode{kind: kindPlaceholder, str: "id"}
|
||||||
|
}
|
||||||
|
return v
|
||||||
|
}
|
||||||
switch path {
|
switch path {
|
||||||
case "$.data.timestamp", "$.data.payload.timestamp":
|
case "$.data.timestamp", "$.data.payload.timestamp":
|
||||||
if v.kind == kindString && isoOffsetRe.MatchString(v.str) {
|
if v.kind == kindString && isoOffsetRe.MatchString(v.str) {
|
||||||
|
|||||||
@@ -6,8 +6,10 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"mime"
|
"mime"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"regexp"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
"unicode"
|
"unicode"
|
||||||
"unicode/utf8"
|
"unicode/utf8"
|
||||||
)
|
)
|
||||||
@@ -81,6 +83,9 @@ func compareHeaders(want, got, extra map[string]string) []Diff {
|
|||||||
if gv == wv {
|
if gv == wv {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
if ck == "Content-Disposition" && sameDispositionButDates(wv, gv) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
if gv == "" {
|
if gv == "" {
|
||||||
gv = "<missing>"
|
gv = "<missing>"
|
||||||
}
|
}
|
||||||
@@ -290,3 +295,32 @@ func quotePrintable(b []byte) string {
|
|||||||
buf.WriteByte('"')
|
buf.WriteByte('"')
|
||||||
return buf.String()
|
return buf.String()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// dispositionDateRe finds YYYY-MM-DD tokens in a Content-Disposition value,
|
||||||
|
// such as the date in a download name built from the current day.
|
||||||
|
var dispositionDateRe = regexp.MustCompile(`\b\d{4}-\d{2}-\d{2}\b`)
|
||||||
|
|
||||||
|
// maskDispositionDates replaces every date token with one placeholder and
|
||||||
|
// reports how many it replaced. ok is false when a token is not a real
|
||||||
|
// calendar date, so the caller falls back to the byte comparison.
|
||||||
|
func maskDispositionDates(s string) (masked string, n int, ok bool) {
|
||||||
|
ok = true
|
||||||
|
masked = dispositionDateRe.ReplaceAllStringFunc(s, func(m string) string {
|
||||||
|
if _, err := time.Parse("2006-01-02", m); err != nil {
|
||||||
|
ok = false
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
n++
|
||||||
|
return "{{date}}"
|
||||||
|
})
|
||||||
|
return masked, n, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
// sameDispositionButDates reports whether two Content-Disposition values
|
||||||
|
// differ only in their dates: both carry the same number of real calendar
|
||||||
|
// dates and are equal once those are masked. Anything else stays a Diff.
|
||||||
|
func sameDispositionButDates(want, got string) bool {
|
||||||
|
wm, wn, wok := maskDispositionDates(want)
|
||||||
|
gm, gn, gok := maskDispositionDates(got)
|
||||||
|
return wok && gok && wn > 0 && wn == gn && wm == gm
|
||||||
|
}
|
||||||
|
|||||||
@@ -265,7 +265,7 @@ func TestNormalizePublicationAlbumDates(t *testing.T) {
|
|||||||
pub := func(created, updated string) []Publication {
|
pub := func(created, updated string) []Publication {
|
||||||
return []Publication{{Method: "POST", Path: "/api/publish", Body: json.RawMessage(
|
return []Publication{{Method: "POST", Path: "/api/publish", Body: json.RawMessage(
|
||||||
`{"channel":"c","data":{"payload":{"album":{"name":"x","created_at":"` + created + `","updated_at":"` + updated +
|
`{"channel":"c","data":{"payload":{"album":{"name":"x","created_at":"` + created + `","updated_at":"` + updated +
|
||||||
`","market_price_checked_at":null,"photos":[{"created_at":"` + created + `"}]},"created_at":"2026-01-01T00:00:00+00:00"}}}`)}}
|
`","market_price_checked_at":null,"photos":[{"created_at":"` + created + `"}]},"published_at":"2026-01-01T00:00:00+00:00"}}}`)}}
|
||||||
}
|
}
|
||||||
a, err := NormalizePublications(pub("2026-09-30T11:21:55+00:00", "2026-09-30T11:21:56+00:00"), store)
|
a, err := NormalizePublications(pub("2026-09-30T11:21:55+00:00", "2026-09-30T11:21:56+00:00"), store)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -281,8 +281,10 @@ func TestNormalizePublicationAlbumDates(t *testing.T) {
|
|||||||
if !strings.Contains(string(a[0].Body), `"updated_at":"{{datetime}}"`) || !strings.Contains(string(a[0].Body), `"market_price_checked_at":null`) {
|
if !strings.Contains(string(a[0].Body), `"updated_at":"{{datetime}}"`) || !strings.Contains(string(a[0].Body), `"market_price_checked_at":null`) {
|
||||||
t.Fatalf("album dates not masked: %s", a[0].Body)
|
t.Fatalf("album dates not masked: %s", a[0].Body)
|
||||||
}
|
}
|
||||||
// Only the album subtree is masked; a payload date outside it stays.
|
// Only the album subtree is masked; a payload date outside it stays
|
||||||
if !strings.Contains(string(a[0].Body), `},"created_at":"2026-01-01T00:00:00+00:00"`) {
|
// ($.data.payload.created_at itself is the notification row's date and
|
||||||
|
// has its own mask, see TestNormalizeNotificationPublication).
|
||||||
|
if !strings.Contains(string(a[0].Body), `},"published_at":"2026-01-01T00:00:00+00:00"`) {
|
||||||
t.Fatalf("over-normalised: %s", a[0].Body)
|
t.Fatalf("over-normalised: %s", a[0].Body)
|
||||||
}
|
}
|
||||||
// A Z-suffixed album date keeps its value, so a format change is a diff.
|
// A Z-suffixed album date keeps its value, so a format change is a diff.
|
||||||
|
|||||||
124
modules/tide/normalize_phase13_test.go
Normal file
124
modules/tide/normalize_phase13_test.go
Normal file
@@ -0,0 +1,124 @@
|
|||||||
|
package tide
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestNormalizeContentDispositionDate: a download name built from the day
|
||||||
|
// of the request replays on a later day, while a different stem, an invalid
|
||||||
|
// date, a date on one side only or a missing header still diff.
|
||||||
|
func TestNormalizeContentDispositionDate(t *testing.T) {
|
||||||
|
const recorded = "attachment; filename=export-2026-09-17.csv"
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
got map[string]string
|
||||||
|
diff bool
|
||||||
|
}{
|
||||||
|
{"same stem on a later day", map[string]string{"Content-Disposition": "attachment; filename=export-2026-10-03.csv"}, false},
|
||||||
|
{"same day", map[string]string{"Content-Disposition": recorded}, false},
|
||||||
|
{"header name case", map[string]string{"content-disposition": "attachment; filename=export-2027-01-01.csv"}, false},
|
||||||
|
{"different stem", map[string]string{"Content-Disposition": "attachment; filename=report-2026-10-03.csv"}, true},
|
||||||
|
{"invalid date", map[string]string{"Content-Disposition": "attachment; filename=export-2026-13-45.csv"}, true},
|
||||||
|
{"date on one side only", map[string]string{"Content-Disposition": "attachment; filename=export.csv"}, true},
|
||||||
|
{"inline instead of attachment", map[string]string{"Content-Disposition": "inline; filename=export-2026-10-03.csv"}, true},
|
||||||
|
{"header absent", map[string]string{}, true},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
diffs := compareHeaders(map[string]string{"Content-Disposition": recorded}, c.got, nil)
|
||||||
|
if (len(diffs) > 0) != c.diff {
|
||||||
|
t.Errorf("%s: diffs = %+v, want diff=%v", c.name, diffs, c.diff)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// A recorded value that holds an invalid date keeps the byte comparison.
|
||||||
|
odd := "attachment; filename=export-2026-02-30.csv"
|
||||||
|
if diffs := compareHeaders(map[string]string{"Content-Disposition": odd}, map[string]string{"Content-Disposition": "attachment; filename=export-2026-02-28.csv"}, nil); len(diffs) != 1 {
|
||||||
|
t.Errorf("invalid recorded date: diffs = %+v, want one", diffs)
|
||||||
|
}
|
||||||
|
// Two dates must both be real and both present.
|
||||||
|
two := "attachment; filename=export-2026-09-01-2026-09-17.csv"
|
||||||
|
if diffs := compareHeaders(map[string]string{"Content-Disposition": two}, map[string]string{"Content-Disposition": "attachment; filename=export-2026-10-01-2026-10-03.csv"}, nil); len(diffs) != 0 {
|
||||||
|
t.Errorf("two dates: diffs = %+v, want none", diffs)
|
||||||
|
}
|
||||||
|
// The mask applies to Content-Disposition only.
|
||||||
|
if diffs := compareHeaders(map[string]string{"Location": "/files/2026-09-17"}, map[string]string{"Location": "/files/2026-10-03"}, nil); len(diffs) != 1 {
|
||||||
|
t.Errorf("Location: diffs = %+v, want one", diffs)
|
||||||
|
}
|
||||||
|
if _, n, ok := maskDispositionDates("x12026-09-170"); n != 0 || !ok {
|
||||||
|
t.Errorf("a date inside a longer digit run is not a date token")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNormalizeNotificationPublication: a notification:new publication's id
|
||||||
|
// and created_at under $.data.payload normalize equal across two runs, while
|
||||||
|
// a Z date, a string id and every other path stay visible.
|
||||||
|
func TestNormalizeNotificationPublication(t *testing.T) {
|
||||||
|
store := mustMemoryStore()
|
||||||
|
pub := func(id, created string) []Publication {
|
||||||
|
return []Publication{{Method: "POST", Path: "/api/publish", Body: json.RawMessage(
|
||||||
|
`{"channel":"acme#5","data":{"event":"notification:new","payload":{"id":` + id +
|
||||||
|
`,"type":"item_added","payload":{"album_id":3,"created_at":"2026-01-01T00:00:00+00:00"},"read_at":null,"created_at":"` + created + `"}}}`)}}
|
||||||
|
}
|
||||||
|
a, err := NormalizePublications(pub("10000001", "2026-09-30T11:21:55+00:00"), store)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
b, err := NormalizePublications(pub("42", "2026-10-03T08:00:00+00:00"), store)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if diffs := DiffPublications(a, b); len(diffs) != 0 {
|
||||||
|
t.Fatalf("diffs = %+v", diffs)
|
||||||
|
}
|
||||||
|
body := string(a[0].Body)
|
||||||
|
if !strings.Contains(body, `"payload":{"id":{{id}},`) || !strings.HasSuffix(body, `"read_at":null,"created_at":"{{datetime}}"}}}`) {
|
||||||
|
t.Fatalf("notification not masked: %s", body)
|
||||||
|
}
|
||||||
|
// Only $.data.payload.created_at is masked, not a nested payload date.
|
||||||
|
if !strings.Contains(body, `"album_id":3,"created_at":"2026-01-01T00:00:00+00:00"`) {
|
||||||
|
t.Fatalf("over-normalised: %s", body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A Z date, a zero or negative id and a string id stay visible.
|
||||||
|
for _, c := range []struct{ id, created, want string }{
|
||||||
|
{"42", "2026-10-03T08:00:00Z", `"created_at":"2026-10-03T08:00:00Z"`},
|
||||||
|
{`"42"`, "2026-10-03T08:00:00+00:00", `"payload":{"id":"42",`},
|
||||||
|
{"0", "2026-10-03T08:00:00+00:00", `"payload":{"id":0,`},
|
||||||
|
{"-3", "2026-10-03T08:00:00+00:00", `"payload":{"id":-3,`},
|
||||||
|
{"4.5", "2026-10-03T08:00:00+00:00", `"payload":{"id":4.5,`},
|
||||||
|
} {
|
||||||
|
got, err := NormalizePublications(pub(c.id, c.created), store)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(got[0].Body), c.want) {
|
||||||
|
t.Errorf("id %s created %s: %s lacks %s", c.id, c.created, got[0].Body, c.want)
|
||||||
|
}
|
||||||
|
if diffs := DiffPublications(a, got); len(diffs) == 0 {
|
||||||
|
t.Errorf("id %s created %s must diff against the masked publication", c.id, c.created)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A captured id keeps its own placeholder.
|
||||||
|
store.Set("id:note", "77")
|
||||||
|
got, err := NormalizePublications(pub("77", "2026-10-03T08:00:00+00:00"), store)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(got[0].Body), `"payload":{"id":{{id:note}},`) {
|
||||||
|
t.Fatalf("captured id: %s", got[0].Body)
|
||||||
|
}
|
||||||
|
|
||||||
|
// An album publication's existing masks are unchanged.
|
||||||
|
album := []Publication{{Method: "POST", Path: "/api/publish", Body: json.RawMessage(
|
||||||
|
`{"channel":"c","data":{"payload":{"album":{"created_at":"2026-09-30T11:21:55+00:00"},"actor":{"user_id":1,"name":null},"timestamp":"2026-09-30T11:21:55+00:00"}}}`)}}
|
||||||
|
got, err = NormalizePublications(album, store)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
want := `{"channel":"c","data":{"payload":{"album":{"created_at":"{{datetime}}"},"actor":"{{actor}}","timestamp":"{{timestamp}}"}}}`
|
||||||
|
if string(got[0].Body) != want {
|
||||||
|
t.Fatalf("album publication\n got %s\nwant %s", got[0].Body, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user