feat(14-03): tide keeps binary upstream bodies and compares every *_url upload by shape
- A response body that is not valid UTF-8 (a cover image) is written as a YAML !!binary scalar, never masked and replayed byte for byte - The upload URL normalizer covers every key ending in _url (cover_url), not only url and thumb_url - README and parity-testing docs updated
This commit is contained in:
@@ -16,11 +16,11 @@ HTTP parity toolkit that records request and response fixtures from a reference
|
||||
- Recording proxy: `tide.NewProxy` builds a reverse proxy that only binds to and forwards to loopback addresses, groups traffic into named sessions (from the `tide.SessionHeader` request header or a default session) and writes one fixture per complete session on `tide.Proxy.Flush`.
|
||||
- Capture rules: `tide.Rules` (loaded with `tide.LoadRules`) decide which request and response headers are kept per route and which values are captured into variables, from response JSON paths, headers, redirect query strings or form fields.
|
||||
- Variables: `tide.Store` holds captured values such as tokens and IDs in a mode-0600 file, `tide.Store.Expand` substitutes `{{name}}` placeholders before a request is sent, and `tide.ScrubStep` puts placeholders back into fixtures. Scrubbing fails when a step still holds an unclassified token- or password-shaped value, so credentials do not leak into committed fixtures.
|
||||
- Replay and diff: `tide.ReplayFlow` re-sends each step, compares status, a fixed set of contract headers and the body, and returns `tide.Result` with per-step `tide.Diff` entries. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies are compared byte for byte. A `Content-Disposition` header is compared after masking its `YYYY-MM-DD` dates on both sides, so a download named after the day it was made, such as `attachment; filename=export-2026-09-17.csv`, replays on a later day; each masked token must be a real calendar date, and a different name, an invalid date or a date on one side only is still a difference. Uploaded-file URLs under `url` and `thumb_url` keys are compared by shape: under the uploads prefix (`tide.ReplayConfig` `UploadPrefix`, default `tide.DefaultUploadPrefix`, `/storage/app/uploads/public`) an original must be `<prefix>/xxx/yyy/zzz/<disk_name>` with the partition taken from the disk name, and a thumbnail `<prefix>/xxx/yyy/zzz/thumb_<id>_<w>_<h>_<ox>_<oy>_<mode>.<ext>`. The partition, disk name and file id are masked; the prefix, size, offsets, mode and extension stay, so a thumbnail of another size or an upload URL under another prefix is still a difference.
|
||||
- Replay and diff: `tide.ReplayFlow` re-sends each step, compares status, a fixed set of contract headers and the body, and returns `tide.Result` with per-step `tide.Diff` entries. JSON bodies are compared structurally after masking `id`, `*_id` and `*_ids` values and `*_at` timestamps; other bodies are compared byte for byte. A `Content-Disposition` header is compared after masking its `YYYY-MM-DD` dates on both sides, so a download named after the day it was made, such as `attachment; filename=export-2026-09-17.csv`, replays on a later day; each masked token must be a real calendar date, and a different name, an invalid date or a date on one side only is still a difference. Uploaded-file URLs under the `url` key and every key ending in `_url` (`thumb_url`, `cover_url`) are compared by shape: under the uploads prefix (`tide.ReplayConfig` `UploadPrefix`, default `tide.DefaultUploadPrefix`, `/storage/app/uploads/public`) an original must be `<prefix>/xxx/yyy/zzz/<disk_name>` with the partition taken from the disk name, and a thumbnail `<prefix>/xxx/yyy/zzz/thumb_<id>_<w>_<h>_<ox>_<oy>_<mode>.<ext>`. The partition, disk name and file id are masked; the prefix, size, offsets, mode and extension stay, so a thumbnail of another size or an upload URL under another prefix is still a difference.
|
||||
- Fake Centrifugo: `tide.NewCentrifugoRecorder` returns an `http.Handler` that records every POST to a path ending in `/publish` or `/broadcast` as a `tide.Publication` (method, path, whether `Authorization: apikey <key>` carried the configured key, JSON body) and answers `{"result":{}}`. Paths ending in `/presence` answer `{"result":{"presence":{}}}`, `/unsubscribe` and `/info` answer `{"result":{}}`, anything else is 404. Bodies are capped at `tide.MaxPublicationBody` (1 MiB). The API key is only compared, never stored. `tide.CentrifugoRecorder.ListenAndServe` binds loopback addresses only, like the recording proxy.
|
||||
- Broadcast goldens: `tide.RecordBroadcasts` runs a flow against a loopback reference backend whose Centrifugo API URL points at a recorder on `tide.DefaultCentrifugoListen` (`127.0.0.1:8424`). With `tide.BroadcastConfig` `Step` set, earlier steps run as setup and only that step's publications are kept. The result is a `tide.BroadcastGolden`, written with `tide.WriteBroadcastGolden` (which refuses token-shaped bodies) and read strictly with `tide.LoadBroadcastGolden`. A golden with `pending` set is recorded but not yet asserted.
|
||||
- Broadcast normalisation: `tide.NormalizePublications` masks only `$.data.timestamp` and `$.data.payload.timestamp` (ISO 8601 with an offset) as `"{{timestamp}}"`, `$.data.payload.actor` (an object of exactly `user_id` and `name`) as `"{{actor}}"`, and values equal to an `id:*` variable of a `tide.Store`: numbers or strings under `id`, `*_id` or `*_ids` keys, and the numeric last segment of a channel name such as `room:12`. Carbon `+00:00` values of `*_at` keys anywhere under `$.data.payload.album` become `"{{datetime}}"`. A notification publication's own row fields are masked too: a Carbon `+00:00` `$.data.payload.created_at` becomes `"{{datetime}}"`, and a positive integer `$.data.payload.id` that no `id:*` variable names becomes a bare `{{id}}` (a captured one keeps `{{id:name}}`); a date of another shape is left as it is, so a format change shows as a difference. A masked number is written as a bare `{{id:name}}`, so a number that becomes a string still differs. A value matching two id variables is an error. `tide.DiffPublications` compares the count, method, path, authorization flag and body (structurally, key order ignored) and reports paths such as `$[0].body.data.payload.id`.
|
||||
- Upstream sidecars: `tide.UpstreamSidecar` (version 1) holds the ordered `tide.UpstreamExchange` values a backend sent to outside services during one fixture, stored as `<fixture>.upstream.yaml` (`tide.UpstreamPath`). `tide.LoadUpstream` reads one strictly (a missing file wraps `fs.ErrNotExist`). `tide.WriteUpstream` masks every variable value as `{{name}}`, replaces a JSON string longer than 1024 characters that decodes as base64 (also behind a `data:` URL prefix) with `{{sha256:<hex>}}`, and refuses an Authorization or X-Api-Key value that no variable masks. A multipart request keeps ordered `tide.UpstreamPart` values: plain fields by value, files by SHA-256. Bodies are capped at `tide.MaxUpstreamBody` (32 MiB).
|
||||
- Upstream sidecars: `tide.UpstreamSidecar` (version 1) holds the ordered `tide.UpstreamExchange` values a backend sent to outside services during one fixture, stored as `<fixture>.upstream.yaml` (`tide.UpstreamPath`). `tide.LoadUpstream` reads one strictly (a missing file wraps `fs.ErrNotExist`). `tide.WriteUpstream` masks every variable value as `{{name}}`, replaces a JSON string longer than 1024 characters that decodes as base64 (also behind a `data:` URL prefix) with `{{sha256:<hex>}}`, and refuses an Authorization or X-Api-Key value that no variable masks. A multipart request keeps ordered `tide.UpstreamPart` values: plain fields by value, files by SHA-256. A response body that is not valid UTF-8, such as an image, is written as a YAML `!!binary` scalar, never masked and replayed byte for byte. Bodies are capped at `tide.MaxUpstreamBody` (32 MiB).
|
||||
- Upstream fake: `tide.NewUpstreamFake` returns an `http.RoundTripper` that answers each request from the next recorded exchange without dialing and asserts it: method, scheme, host, path, query (order-insensitive), the `tide.UpstreamCompareHeaders` with placeholders expanded from a `tide.Store`, and the body (JSON semantically, multipart part by part, hashed base64 by digest). Mismatch messages never print credential header values. `tide.UpstreamFake.Verify` joins every mismatch, extra request and unconsumed exchange. Hand the fake to code under test with `fetchguard.WithTransport`.
|
||||
- Upstream recording proxy: `tide.NewUpstreamProxy` builds a loopback-only CONNECT proxy (`tide.DefaultUpstreamProxyListen`, `127.0.0.1:8425`) that terminates TLS with per-host certificates signed by a local parity CA (`tide.EnsureParityCA`: ECDSA P-256, certificate mode 0644, key mode 0600, standard library crypto only). In script mode it answers from a `tide.UpstreamScript` (the first unused entry whose method, host and path match; no match answers 599 and fails the recording); in forward mode it sends each request once to the vendor through a `fetchguard.Client` in `fetchguard.PublicOnlyMode`. `tide.UpstreamProxy.Flush` writes the sidecar with `tide.WriteUpstream` and refuses when any request failed. The CA directory and the vars file must be outside the sidecar's directory.
|
||||
- Manifests: `tide.Manifest` lists routes with auth groups, a pending or ported status, cases and fixture paths; `tide.RecordManifest` records missing cases in batches of at most `tide.MaxBatch`, and `tide.ReplayManifest` replays every recorded case into a `tide.Coverage` table.
|
||||
|
||||
Reference in New Issue
Block a user