feat(14-01): record vendor calls with summer parity:upstream and replay them offline

- WriteUpstream masks vars, hashes long base64 JSON strings and refuses unmasked Authorization/X-Api-Key
- multipart requests recorded as ordered parts; the fake compares parts and hashed payloads
- loopback CONNECT recording proxy with a local ECDSA parity CA, script and forward modes
- parity:upstream command, README and parity docs
This commit is contained in:
Jakub Zych
2026-10-03 19:55:42 +02:00
parent e6a67134d1
commit ee0004fb65
11 changed files with 1647 additions and 15 deletions

View File

@@ -6,7 +6,7 @@ HTTP parity toolkit that records request and response fixtures from a reference
## Overview
`tide` is the acceptance-test engine for porting an existing WinterCMS or other PHP backend to SummerCMS: the reference backend's real responses define the contract, and the Go port must reproduce them. It records YAML fixtures (flows of request and response steps) either by driving a spec against a target or by sitting as a loopback reverse proxy in front of the reference backend while a real client uses it, then replays those fixtures against the port and diffs the responses after masking values that legitimately differ, such as IDs and timestamps. It also checks realtime side effects: a fake Centrifugo recorder captures the publications a backend sends while a flow runs, and broadcast golden files hold them normalised for comparison. The `summer` CLI's parity:proxy, parity:record, parity:replay and parity:broadcasts commands are thin wrappers around this package. It has no WinterCMS counterpart.
`tide` is the acceptance-test engine for porting an existing WinterCMS or other PHP backend to SummerCMS: the reference backend's real responses define the contract, and the Go port must reproduce them. It records YAML fixtures (flows of request and response steps) either by driving a spec against a target or by sitting as a loopback reverse proxy in front of the reference backend while a real client uses it, then replays those fixtures against the port and diffs the responses after masking values that legitimately differ, such as IDs and timestamps. It also checks realtime side effects: a fake Centrifugo recorder captures the publications a backend sends while a flow runs, and broadcast golden files hold them normalised for comparison. Upstream sidecars record the calls a backend makes to outside vendors, through a loopback HTTPS recording proxy, and replay them offline through an asserting fake. The `summer` CLI's parity:proxy, parity:record, parity:replay, parity:broadcasts and parity:upstream commands are thin wrappers around this package. It has no WinterCMS counterpart.
## Features
@@ -20,6 +20,9 @@ HTTP parity toolkit that records request and response fixtures from a reference
- Fake Centrifugo: `tide.NewCentrifugoRecorder` returns an `http.Handler` that records every POST to a path ending in `/publish` or `/broadcast` as a `tide.Publication` (method, path, whether `Authorization: apikey <key>` carried the configured key, JSON body) and answers `{"result":{}}`. Paths ending in `/presence` answer `{"result":{"presence":{}}}`, `/unsubscribe` and `/info` answer `{"result":{}}`, anything else is 404. Bodies are capped at `tide.MaxPublicationBody` (1 MiB). The API key is only compared, never stored. `tide.CentrifugoRecorder.ListenAndServe` binds loopback addresses only, like the recording proxy.
- Broadcast goldens: `tide.RecordBroadcasts` runs a flow against a loopback reference backend whose Centrifugo API URL points at a recorder on `tide.DefaultCentrifugoListen` (`127.0.0.1:8424`). With `tide.BroadcastConfig` `Step` set, earlier steps run as setup and only that step's publications are kept. The result is a `tide.BroadcastGolden`, written with `tide.WriteBroadcastGolden` (which refuses token-shaped bodies) and read strictly with `tide.LoadBroadcastGolden`. A golden with `pending` set is recorded but not yet asserted.
- Broadcast normalisation: `tide.NormalizePublications` masks only `$.data.timestamp` and `$.data.payload.timestamp` (ISO 8601 with an offset) as `"{{timestamp}}"`, `$.data.payload.actor` (an object of exactly `user_id` and `name`) as `"{{actor}}"`, and values equal to an `id:*` variable of a `tide.Store`: numbers or strings under `id`, `*_id` or `*_ids` keys, and the numeric last segment of a channel name such as `room:12`. Carbon `+00:00` values of `*_at` keys anywhere under `$.data.payload.album` become `"{{datetime}}"`. A notification publication's own row fields are masked too: a Carbon `+00:00` `$.data.payload.created_at` becomes `"{{datetime}}"`, and a positive integer `$.data.payload.id` that no `id:*` variable names becomes a bare `{{id}}` (a captured one keeps `{{id:name}}`); a date of another shape is left as it is, so a format change shows as a difference. A masked number is written as a bare `{{id:name}}`, so a number that becomes a string still differs. A value matching two id variables is an error. `tide.DiffPublications` compares the count, method, path, authorization flag and body (structurally, key order ignored) and reports paths such as `$[0].body.data.payload.id`.
- Upstream sidecars: `tide.UpstreamSidecar` (version 1) holds the ordered `tide.UpstreamExchange` values a backend sent to outside services during one fixture, stored as `<fixture>.upstream.yaml` (`tide.UpstreamPath`). `tide.LoadUpstream` reads one strictly (a missing file wraps `fs.ErrNotExist`). `tide.WriteUpstream` masks every variable value as `{{name}}`, replaces a JSON string longer than 1024 characters that decodes as base64 (also behind a `data:` URL prefix) with `{{sha256:<hex>}}`, and refuses an Authorization or X-Api-Key value that no variable masks. A multipart request keeps ordered `tide.UpstreamPart` values: plain fields by value, files by SHA-256. Bodies are capped at `tide.MaxUpstreamBody` (32 MiB).
- Upstream fake: `tide.NewUpstreamFake` returns an `http.RoundTripper` that answers each request from the next recorded exchange without dialing and asserts it: method, scheme, host, path, query (order-insensitive), the `tide.UpstreamCompareHeaders` with placeholders expanded from a `tide.Store`, and the body (JSON semantically, multipart part by part, hashed base64 by digest). Mismatch messages never print credential header values. `tide.UpstreamFake.Verify` joins every mismatch, extra request and unconsumed exchange. Hand the fake to code under test with `fetchguard.WithTransport`.
- Upstream recording proxy: `tide.NewUpstreamProxy` builds a loopback-only CONNECT proxy (`tide.DefaultUpstreamProxyListen`, `127.0.0.1:8425`) that terminates TLS with per-host certificates signed by a local parity CA (`tide.EnsureParityCA`: ECDSA P-256, certificate mode 0644, key mode 0600, standard library crypto only). In script mode it answers from a `tide.UpstreamScript` (the first unused entry whose method, host and path match; no match answers 599 and fails the recording); in forward mode it sends each request once to the vendor through a `fetchguard.Client` in `fetchguard.PublicOnlyMode`. `tide.UpstreamProxy.Flush` writes the sidecar with `tide.WriteUpstream` and refuses when any request failed. The CA directory and the vars file must be outside the sidecar's directory.
- Manifests: `tide.Manifest` lists routes with auth groups, a pending or ported status, cases and fixture paths; `tide.RecordManifest` records missing cases in batches of at most `tide.MaxBatch`, and `tide.ReplayManifest` replays every recorded case into a `tide.Coverage` table.
## Usage
@@ -119,6 +122,25 @@ for _, d := range tide.DiffPublications(golden.Publications, norm) {
| `tide.NormalizePublications` | Masks timestamps, the actor, album dates and captured ids in publication bodies. |
| `tide.DiffPublications` | Structural diff of two publication lists. |
| `tide.DefaultCentrifugoListen` | Default recorder address, `127.0.0.1:8424`. |
| `tide.UpstreamSidecar` | Versioned list of recorded vendor exchanges for one fixture. |
| `tide.UpstreamExchange` | One recorded vendor request and its response. |
| `tide.UpstreamRequest` | Method, absolute URL, compared headers and a body or multipart parts. |
| `tide.UpstreamPart` | One recorded multipart part: name and value, or filename, content type and SHA-256. |
| `tide.UpstreamResponse` | Status, headers and body the fake replays. |
| `tide.UpstreamPath` | Maps `x.yaml` to its sidecar path `x.upstream.yaml`. |
| `tide.LoadUpstream` | Reads a version-1 sidecar strictly. |
| `tide.WriteUpstream` | Masks variables, hashes long base64 strings, refuses unmasked credentials and writes the sidecar (mode 0644). |
| `tide.UpstreamCompareHeaders` | The request headers the fake asserts. |
| `tide.MaxUpstreamBody` | Body cap for the fake and the recording proxy, 32 MiB. |
| `tide.UpstreamFake` | Asserting `http.RoundTripper` over a sidecar: `tide.UpstreamFake.RoundTrip`, `tide.UpstreamFake.Verify`. |
| `tide.NewUpstreamFake` | Builds the fake from a sidecar and a variables store. |
| `tide.UpstreamProxyConfig` | Listen address, CA directory, output path, mode, script path and vars path for the recording proxy. |
| `tide.UpstreamProxy` | The recording proxy: `tide.UpstreamProxy.ListenAndServe`, `tide.UpstreamProxy.Flush`. |
| `tide.NewUpstreamProxy` | Validates the config, loads the script and vars, and creates or reuses the parity CA. |
| `tide.UpstreamScript` | Hand-authored vendor responses for script mode. |
| `tide.UpstreamScriptResponse` | One scripted answer: method, host, path and `tide.UpstreamResponse`; each answers once. |
| `tide.EnsureParityCA` | Creates or reuses the parity CA in a directory and returns the certificate path. |
| `tide.DefaultUpstreamProxyListen` | Default recording proxy address, `127.0.0.1:8425`. |
| `tide.Coverage` | Recorded, passing, failing and unrecorded counts, with table rows and a summary line. |
## CLI commands
@@ -149,11 +171,23 @@ publications:
{"channel":"room:{{id:room}}","data":{"event":"deleted","payload":{"id":{{id:item}},"actor":"{{actor}}","timestamp":"{{timestamp}}"},"timestamp":"{{timestamp}}"}}
```
`summer parity:upstream` wraps `tide.NewUpstreamProxy` and `tide.UpstreamProxy.Flush`:
```sh
summer parity:upstream \
--ca-dir /tmp/parity/ca \
--vars /tmp/parity/vars.yaml \
--script testdata/parity/vendor-script.yaml \
--out testdata/parity/routes/POST_items_jwt__ok.upstream.yaml
```
`--listen` defaults to `127.0.0.1:8425` and `--mode` to `script`. The command prints the CA certificate path; run the reference backend with `HTTPS_PROXY` pointing at the proxy and that certificate as its CA file, then interrupt the command to write the sidecar. Security rules: the proxy listens on loopback only; the CA key stays mode 0600 in `--ca-dir`, outside the fixtures tree; the vars file is outside the sidecar's directory and every value in it is masked; an unmasked Authorization or X-Api-Key value refuses the write; `--mode forward` reaches the real vendor and is for hand recording only, never CI.
## Dependencies
- SummerCMS modules: none.
- Third-party: `github.com/goccy/go-yaml` (fixture, rules and manifest parsing).
- Standard library: `net/http`, `net/http/httputil`, `encoding/json`, `crypto/sha256`, `crypto/subtle`, among others.
- SummerCMS modules: [fetchguard](../fetchguard/README.md) (the recording proxy's forward mode).
- Third-party: `github.com/goccy/go-yaml` (fixture, rules, manifest and sidecar parsing).
- Standard library: `net/http`, `net/http/httputil`, `encoding/json`, `crypto/sha256`, `crypto/subtle`, `crypto/tls`, `crypto/x509`, `crypto/ecdsa`, among others.
## Testing
@@ -161,4 +195,4 @@ publications:
go test ./modules/tide/...
```
The tests run recording, the proxy, replay and the fake Centrifugo recorder against local `net/http/httptest` servers and use the sample spec in `modules/tide/testdata/`; they need no external services.
The tests run recording, the proxies, replay, the upstream fake and the fake Centrifugo recorder against local `net/http/httptest` servers and use the sample spec in `modules/tide/testdata/`; they need no external services.