summercms/modules/tide/README.md

tide

HTTP parity toolkit that records request and response fixtures from a reference backend, replays them against a new one and reports normalized differences.

import "git.golem15.com/golem15/summercms/modules/tide"

Overview

tide is the acceptance-test engine for porting an existing WinterCMS or other PHP backend to SummerCMS: the reference backend's real responses define the contract, and the Go port must reproduce them. It records YAML fixtures (flows of request and response steps) either by driving a spec against a target or by sitting as a loopback reverse proxy in front of the reference backend while a real client uses it, then replays those fixtures against the port and diffs the responses after masking values that legitimately differ, such as IDs and timestamps. It also checks realtime side effects: a fake Centrifugo recorder captures the publications a backend sends while a flow runs, and broadcast golden files hold them normalised for comparison. Upstream sidecars record the calls a backend makes to outside vendors, through a loopback HTTPS recording proxy, and replay them offline through an asserting fake. The summer CLI's parity:proxy, parity:record, parity:replay, parity:broadcasts and parity:upstream commands are thin wrappers around this package. It has no WinterCMS counterpart.

Features

• Flow fixtures: tide.Flow is a versioned, ordered list of tide.Step values, loaded strictly (unknown fields rejected) with tide.LoadFlow and tide.ParseFlow, and written atomically with tide.SaveFlow or tide.SaveFlowExclusive. Response bodies can live in sidecar files, confined to the fixture directory and checked against an optional SHA-256 digest.
• Recording: tide.RecordFlow executes a spec flow against a target URL and fills in the responses, with a bounded body size (tide.DefaultMaxBody, 8 MiB).
• Multipart uploads: a request can carry parts (tide.Part: a text field with value, or a file field with file, filename, content_type and sha256) instead of a body. File bytes stay in files beside the fixture (for example files/cover.png, read from tide.RecordConfig or tide.ReplayConfig BaseDir) and are never inlined into the YAML. Parts are encoded in declaration order with one fixed boundary, tide.MultipartBoundary, so the reference backend and the port receive byte-identical bodies; the encoded Content-Type replaces a recorded multipart Content-Type. Variables expand in part values, never in file bytes. tide.LoadFlow fails when a part file is missing or its SHA-256 differs, naming the part, and a request with both body and parts is invalid.
• Recording proxy: tide.NewProxy builds a reverse proxy that only binds to and forwards to loopback addresses, groups traffic into named sessions (from the tide.SessionHeader request header or a default session) and writes one fixture per complete session on tide.Proxy.Flush.
• Capture rules: tide.Rules (loaded with tide.LoadRules) decide which request and response headers are kept per route and which values are captured into variables, from response JSON paths, headers, redirect query strings or form fields.
• Variables: tide.Store holds captured values such as tokens and IDs in a mode-0600 file, tide.Store.Expand substitutes {{name}} placeholders before a request is sent, and tide.ScrubStep puts placeholders back into fixtures. Scrubbing fails when a step still holds an unclassified token- or password-shaped value, so credentials do not leak into committed fixtures.
• Replay and diff: tide.ReplayFlow re-sends each step, compares status, a fixed set of contract headers and the body, and returns tide.Result with per-step tide.Diff entries. JSON bodies are compared structurally after masking id, *_id and *_ids values and *_at timestamps; other bodies are compared byte for byte. A Content-Disposition header is compared after masking its YYYY-MM-DD dates on both sides, so a download named after the day it was made, such as attachment; filename=export-2026-09-17.csv, replays on a later day; each masked token must be a real calendar date, and a different name, an invalid date or a date on one side only is still a difference. Uploaded-file URLs under the url key and every key ending in _url (thumb_url, cover_url) are compared by shape: under the uploads prefix (tide.ReplayConfig UploadPrefix, default tide.DefaultUploadPrefix, /storage/app/uploads/public) an original must be <prefix>/xxx/yyy/zzz/<disk_name> with the partition taken from the disk name, and a thumbnail <prefix>/xxx/yyy/zzz/thumb_<id>_<w>_<h>_<ox>_<oy>_<mode>.<ext>. The partition, disk name and file id are masked; the prefix, size, offsets, mode and extension stay, so a thumbnail of another size or an upload URL under another prefix is still a difference.
• Fake Centrifugo: tide.NewCentrifugoRecorder returns an http.Handler that records every POST to a path ending in /publish or /broadcast as a tide.Publication (method, path, whether Authorization: apikey <key> carried the configured key, JSON body) and answers {"result":{}}. Paths ending in /presence answer {"result":{"presence":{}}}, /unsubscribe and /info answer {"result":{}}, anything else is 404. Bodies are capped at tide.MaxPublicationBody (1 MiB). The API key is only compared, never stored. tide.CentrifugoRecorder.ListenAndServe binds loopback addresses only, like the recording proxy.
• Broadcast goldens: tide.RecordBroadcasts runs a flow against a loopback reference backend whose Centrifugo API URL points at a recorder on tide.DefaultCentrifugoListen (127.0.0.1:8424). With tide.BroadcastConfig Step set, earlier steps run as setup and only that step's publications are kept. The result is a tide.BroadcastGolden, written with tide.WriteBroadcastGolden (which refuses token-shaped bodies) and read strictly with tide.LoadBroadcastGolden. A golden with pending set is recorded but not yet asserted.
• Broadcast normalisation: tide.NormalizePublications masks only $.data.timestamp and $.data.payload.timestamp (ISO 8601 with an offset) as "{{timestamp}}", $.data.payload.actor (an object of exactly user_id and name) as "{{actor}}", and values equal to an id:* variable of a tide.Store: numbers or strings under id, *_id or *_ids keys, and the numeric last segment of a channel name such as room:12. Carbon +00:00 values of *_at keys anywhere under $.data.payload.album become "{{datetime}}". A notification publication's own row fields are masked too: a Carbon +00:00 $.data.payload.created_at becomes "{{datetime}}", and a positive integer $.data.payload.id that no id:* variable names becomes a bare {{id}} (a captured one keeps {{id:name}}); a date of another shape is left as it is, so a format change shows as a difference. A masked number is written as a bare {{id:name}}, so a number that becomes a string still differs. A value matching two id variables is an error. tide.DiffPublications compares the count, method, path, authorization flag and body (structurally, key order ignored) and reports paths such as $[0].body.data.payload.id.
• Upstream sidecars: tide.UpstreamSidecar (version 1) holds the ordered tide.UpstreamExchange values a backend sent to outside services during one fixture, stored as <fixture>.upstream.yaml (tide.UpstreamPath). tide.LoadUpstream reads one strictly (a missing file wraps fs.ErrNotExist). tide.WriteUpstream masks every variable value as {{name}}, replaces a JSON string longer than 1024 characters that decodes as base64 (also behind a data: URL prefix) with {{sha256:<hex>}}, and refuses an Authorization or X-Api-Key value that no variable masks. A multipart request keeps ordered tide.UpstreamPart values: plain fields by value, files by SHA-256. A response body that is not valid UTF-8, such as an image, is written as a YAML !!binary scalar, never masked and replayed byte for byte. Bodies are capped at tide.MaxUpstreamBody (32 MiB).
• Upstream fake: tide.NewUpstreamFake returns an http.RoundTripper that answers each request from the next recorded exchange without dialing and asserts it: method, scheme, host, path, query (order-insensitive), the tide.UpstreamCompareHeaders with placeholders expanded from a tide.Store, and the body (JSON semantically, multipart part by part, hashed base64 by digest). Mismatch messages never print credential header values. tide.UpstreamFake.Verify joins every mismatch, extra request and unconsumed exchange. Hand the fake to code under test with fetchguard.WithTransport.
• Upstream recording proxy: tide.NewUpstreamProxy builds a loopback-only CONNECT proxy (tide.DefaultUpstreamProxyListen, 127.0.0.1:8425) that terminates TLS with per-host certificates signed by a local parity CA (tide.EnsureParityCA: ECDSA P-256, certificate mode 0644, key mode 0600, standard library crypto only). In script mode it answers from a tide.UpstreamScript (the first unused entry whose method, host and path match; no match answers 599 and fails the recording); in forward mode it sends each request once to the vendor through a fetchguard.Client in fetchguard.PublicOnlyMode. tide.UpstreamProxy.Flush writes the sidecar with tide.WriteUpstream and refuses when any request failed. The CA directory and the vars file must be outside the sidecar's directory.
• Manifests: tide.Manifest lists routes with auth groups, a pending or ported status, cases and fixture paths; tide.RecordManifest records missing cases in batches of at most tide.MaxBatch, and tide.ReplayManifest replays every recorded case into a tide.Coverage table.

Usage

Record a flow once against the reference backend, then replay it against the port:

spec, err := tide.LoadFlow("testdata/parity/posts.spec.yaml")
if err != nil {
	return err
}
flow, err := tide.RecordFlow(ctx, spec, tide.RecordConfig{Target: "http://127.0.0.1:8000"})
if err != nil {
	return err
}
if err := tide.SaveFlow("testdata/parity/posts.yaml", flow); err != nil {
	return err
}

res, err := tide.ReplayFlow(ctx, flow, tide.ReplayConfig{
	Target:  "http://127.0.0.1:8080",
	BaseDir: "testdata/parity",
})
if err != nil {
	return err
}
for _, step := range res.Steps {
	for _, d := range step.Diffs {
		fmt.Printf("%s %s: want %s, got %s\n", step.ID, d.Path, d.Expected, d.Actual)
	}
}

Record the publications a reference backend sends for one flow step, then compare a new backend's publications with them:

golden, err := tide.RecordBroadcasts(ctx, spec, tide.BroadcastConfig{
	Target: "http://127.0.0.1:8000",
	APIKey: "test-only-key",
	Store:  store,
	Step:   "delete",
	IDs:    []string{"id:room", "id:item"},
})
if err != nil {
	return err
}
if err := tide.WriteBroadcastGolden("testdata/broadcasts/deleted.yaml", golden); err != nil {
	return err
}

// Later, with the new backend publishing to rec (a tide.CentrifugoRecorder):
norm, err := tide.NormalizePublications(rec.Publications(), idsOfTheNewBackend)
if err != nil {
	return err
}
for _, d := range tide.DiffPublications(golden.Publications, norm) {
	fmt.Printf("%s: want %s, got %s\n", d.Path, d.Expected, d.Actual)
}

API reference

Identifier	Description
tide.Flow	Versioned, ordered list of request and response steps; the fixture format.
tide.Step	One request and response pair, with capture and normalizer overrides.
tide.LoadFlow	Reads and validates a flow file.
tide.SaveFlow	Writes a validated flow atomically.
tide.RecordFlow	Executes a spec flow against a target and returns the recorded flow.
tide.Request	The outbound call of a step: method, path, query, headers and a Body or multipart Parts.
tide.Part	One multipart field: a text value, or a file beside the fixture pinned by its SHA-256.
tide.MultipartBoundary	The fixed boundary multipart request bodies are encoded with.
tide.RecordConfig	Target, client, body cap, vars store, capture rules and part-file directory for tide.RecordFlow.
tide.ReplayConfig	Target, client, body cap, vars store, fixture directory and upload URL prefix for tide.ReplayFlow.
tide.DefaultUploadPrefix	The default uploads URL prefix, /storage/app/uploads/public.
tide.ReplayFlow	Replays a recorded flow against a target and diffs every step.
tide.Result	Replay outcome: overall status and per-step tide.StepResult values.
tide.Diff	One structural JSON or byte-level mismatch.
tide.MismatchError	Error that carries a failing tide.Result.
tide.NewProxy	Builds the loopback recording reverse proxy from a tide.ProxyConfig.
tide.Proxy	The recording proxy: tide.Proxy.Handler, tide.Proxy.ListenAndServe, tide.Proxy.Flush.
tide.Rules	Header keep lists and capture rules for proxy sessions.
tide.Store	Named capture variables, optionally persisted to a private file.
tide.OpenStore	Opens a file-backed store, or a memory-only store for an empty path.
tide.Manifest	Route list with auth groups, status, cases and fixture paths.
tide.ValidateManifest	Checks a manifest in tide.ModeAllowIncomplete or tide.ModeRequireRecorded mode.
tide.RecordManifest	Records a manifest's seed flow and missing route cases in batches.
tide.ReplayManifest	Replays every recorded route case and builds a coverage table.
tide.CentrifugoRecorder	Fake Centrifugo HTTP API: tide.CentrifugoRecorder.Publications, tide.CentrifugoRecorder.Reset, tide.CentrifugoRecorder.ListenAndServe.
tide.NewCentrifugoRecorder	Builds a recorder from tide.CentrifugoRecorderOptions (the expected API key).
tide.Publication	One recorded publish or broadcast request: method, path, authorization flag, JSON body.
tide.RecordBroadcasts	Runs a flow (or one step of it) against a loopback backend and returns its normalised publications as a golden.
tide.BroadcastConfig	Target, recorder address, API key, vars store, step, id variables and settle time for tide.RecordBroadcasts.
tide.BroadcastGolden	Versioned broadcast golden: name, flow, optional pending reason, publications.
tide.LoadBroadcastGolden	Reads a golden strictly and checks every body parses.
tide.WriteBroadcastGolden	Writes a golden atomically, refusing token-shaped bodies.
tide.NormalizePublications	Masks timestamps, the actor, album dates and captured ids in publication bodies.
tide.DiffPublications	Structural diff of two publication lists.
tide.DefaultCentrifugoListen	Default recorder address, 127.0.0.1:8424.
tide.UpstreamSidecar	Versioned list of recorded vendor exchanges for one fixture.
tide.UpstreamExchange	One recorded vendor request and its response.
tide.UpstreamRequest	Method, absolute URL, compared headers and a body or multipart parts.
tide.UpstreamPart	One recorded multipart part: name and value, or filename, content type and SHA-256.
tide.UpstreamResponse	Status, headers and body the fake replays.
tide.UpstreamPath	Maps x.yaml to its sidecar path x.upstream.yaml.
tide.LoadUpstream	Reads a version-1 sidecar strictly.
tide.WriteUpstream	Masks variables, hashes long base64 strings, refuses unmasked credentials and writes the sidecar (mode 0644).
tide.UpstreamCompareHeaders	The request headers the fake asserts.
tide.MaxUpstreamBody	Body cap for the fake and the recording proxy, 32 MiB.
tide.UpstreamFake	Asserting http.RoundTripper over a sidecar: tide.UpstreamFake.RoundTrip, tide.UpstreamFake.Verify.
tide.NewUpstreamFake	Builds the fake from a sidecar and a variables store.
tide.UpstreamProxyConfig	Listen address, CA directory, output path, mode, script path and vars path for the recording proxy.
tide.UpstreamProxy	The recording proxy: tide.UpstreamProxy.ListenAndServe, tide.UpstreamProxy.Flush.
tide.NewUpstreamProxy	Validates the config, loads the script and vars, and creates or reuses the parity CA.
tide.UpstreamScript	Hand-authored vendor responses for script mode.
tide.UpstreamScriptResponse	One scripted answer: method, host, path and tide.UpstreamResponse; each answers once.
tide.EnsureParityCA	Creates or reuses the parity CA in a directory and returns the certificate path.
tide.DefaultUpstreamProxyListen	Default recording proxy address, 127.0.0.1:8425.
tide.Coverage	Recorded, passing, failing and unrecorded counts, with table rows and a summary line.

CLI commands

summer parity:broadcasts wraps tide.RecordBroadcasts and tide.WriteBroadcastGolden:

summer parity:broadcasts \
  --flow testdata/broadcasts/flows/item-lifecycle.yaml \
  --step delete --name deleted --ids id:room,id:item \
  --target http://127.0.0.1:8000 \
  --vars /tmp/parity/vars.yaml \
  --api-key test-only-key \
  --out testdata/broadcasts/deleted.yaml

--listen defaults to 127.0.0.1:8424, --api-key to $PARITY_CENTRIFUGO_API_KEY and --settle to 500ms. --ids defaults to the id:* variables the flow mentions. --pending stores a reason the golden is not asserted yet. The target and the listen address must be loopback, and the vars file must be outside the golden's directory. The command writes:

version: 1
name: deleted
flow: "items/lifecycle#delete"
publications:
  - method: POST
    path: /api/publish
    authorization: true
    body: |-
      {"channel":"room:{{id:room}}","data":{"event":"deleted","payload":{"id":{{id:item}},"actor":"{{actor}}","timestamp":"{{timestamp}}"},"timestamp":"{{timestamp}}"}}

summer parity:upstream wraps tide.NewUpstreamProxy and tide.UpstreamProxy.Flush:

summer parity:upstream \
  --ca-dir /tmp/parity/ca \
  --vars /tmp/parity/vars.yaml \
  --script testdata/parity/vendor-script.yaml \
  --out testdata/parity/routes/POST_items_jwt__ok.upstream.yaml

--listen defaults to 127.0.0.1:8425 and --mode to script. The command prints the CA certificate path; run the reference backend with HTTPS_PROXY pointing at the proxy and that certificate as its CA file, then interrupt the command (Ctrl-C, or SIGTERM when it runs in the background) to write the sidecar. Security rules: the proxy listens on loopback only; the CA key stays mode 0600 in --ca-dir, outside the fixtures tree; the vars file is outside the sidecar's directory and every value in it is masked; an unmasked Authorization or X-Api-Key value refuses the write; --mode forward reaches the real vendor and is for hand recording only, never CI.

Dependencies

• SummerCMS modules: fetchguard (the recording proxy's forward mode).
• Third-party: github.com/goccy/go-yaml (fixture, rules, manifest and sidecar parsing).
• Standard library: net/http, net/http/httputil, encoding/json, crypto/sha256, crypto/subtle, crypto/tls, crypto/x509, crypto/ecdsa, among others.

Testing

go test ./modules/tide/...

The tests run recording, the proxies, replay, the upstream fake and the fake Centrifugo recorder against local net/http/httptest servers and use the sample spec in modules/tide/testdata/; they need no external services.