golem15/summercms
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(14-01): fetchguard client covers PUT, multipart, bearer and a trusted mode

Browse Source 
- TrustedMode (declared after PublicOnlyMode) lifts the scheme, host and dial checks for Client only
- PutJSON, PostMultipart with FormField/FormFile, Bearer
- tests for modes, redirects, multipart order, body cap and the scheme guard
- README, root modules row and outbound HTTP docs describe the client and its test seam
This commit is contained in:
Jakub Zych
2026-10-03 19:42:37 +02:00
parent 93b7142059
commit e6a67134d1
10 changed files with 543 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
modules/fetchguard/README.md
View File

@@ -1,6 +1,6 @@
# fetchguard
Guarded outbound HTTPS fetcher that blocks private and reserved addresses and enforces host, size and timeout limits.
Guarded outbound HTTP client and fetcher that blocks private and reserved addresses, enforces host, size and timeout limits, and offers an explicit trusted mode for operator-configured endpoints.
`import "git.golem15.com/golem15/summercms/modules/fetchguard"`
@@ -8,13 +8,18 @@ Guarded outbound HTTPS fetcher that blocks private and reserved addresses and en
`fetchguard` is the framework's server-side request forgery guard for fetching URLs that come from users or third parties, such as a remote image address. Every call takes a `fetchguard.Policy` that either restricts the target to an allow list of hosts or permits any public host; in both modes the dial-time check refuses private, loopback, link-local, carrier-grade NAT, documentation, multicast and other reserved IPv4 and IPv6 ranges, including IPv4 embedded in NAT64 and 6to4 addresses. Failures come back as a `fetchguard.Error` carrying one `fetchguard.Reason` from a closed set, so callers can map them onto stable API error codes. WinterCMS has no dedicated counterpart; plugins there typically used Guzzle with hand-written checks.
`fetchguard.Fetch` is a one-shot guarded GET. For calling a service API (any method, JSON or multipart bodies, bearer credentials) build one `fetchguard.Client` per vendor with `fetchguard.NewClient`; it applies the same policy to every request it sends and replaces a plugin's own curl request sender.
## Features
- HTTPS only: any other scheme fails with `fetchguard.ReasonScheme`.
- Two modes: `fetchguard.AllowHostsMode` (exact or dotted-suffix host match against `fetchguard.Policy.AllowHosts`) and `fetchguard.PublicOnlyMode` (any public host).
- HTTPS only in the guarded modes: any other scheme fails with `fetchguard.ReasonScheme`.
- Two guarded modes: `fetchguard.AllowHostsMode` (exact or dotted-suffix host match against `fetchguard.Policy.AllowHosts`) and `fetchguard.PublicOnlyMode` (any public host).
- `fetchguard.TrustedMode` for endpoints an operator configured in Go code or admin settings, such as a model server on the local network: a `fetchguard.Client` in this mode accepts http and https, checks no host list and runs no dial guard, but still caps the body and never follows redirects. Only Go code that builds the policy can choose it; `fetchguard.Fetch` guards as `fetchguard.PublicOnlyMode` when given it.
- `fetchguard.Client` sends any method: `fetchguard.Client.Do` and `fetchguard.Client.Send` take an `http.Request`, and `fetchguard.Client.Get`, `fetchguard.Client.PostJSON`, `fetchguard.Client.PutJSON` and `fetchguard.Client.PostMultipart` cover the common shapes. `fetchguard.Bearer` builds an Authorization value. Response status and headers come back unjudged in `fetchguard.Result`.
- Code-only transport seam: `fetchguard.WithTransport` routes a context's requests to a test `http.RoundTripper`. No policy field, config key, environment variable or header can set it.
- The private and reserved address check runs on the resolved IP at dial time, so DNS answers that point inside the network are refused (`fetchguard.ReasonPrivateIP`); environment proxies are ignored so the check sees the real target.
- Redirects are never followed: a 3xx response is returned as a successful `fetchguard.Result`, and a caller that wants to follow the Location header calls `fetchguard.Fetch` again, which re-runs the guard.
- The response body is capped at the policy's byte limit (`fetchguard.ReasonTooLarge` when exceeded), with a per-call timeout.
- Redirects are never followed, in any mode: a 3xx response is returned as a successful `fetchguard.Result`, and a caller that wants to follow the Location header sends a new request, which re-runs the guard.
- The response body is capped at the policy's byte limit (`fetchguard.ReasonTooLarge` when exceeded), with a per-call timeout. Request bodies are not capped; callers bound their own inputs.
- Limits left at zero in the policy fall back to config keys, then to framework defaults of 10 MiB and 10 seconds (`fetchguard.Defaults`, `fetchguard.DefaultsFromConfig`).
- Typed failure reasons: `fetchguard.ReasonInvalidURL`, `fetchguard.ReasonScheme`, `fetchguard.ReasonUnresolvable`, `fetchguard.ReasonPrivateIP`, `fetchguard.ReasonNetworkError`, `fetchguard.ReasonTooLarge`.
@@ -42,14 +47,57 @@ if res.StatusCode != http.StatusOK {
image := res.Body
```
Calling a service API with a reusable client:
```go
vendor, err := fetchguard.NewClient(fetchguard.Policy{
Mode: fetchguard.AllowHostsMode,
AllowHosts: []string{"api.example.com"},
Timeout: 10 * time.Second,
}, app.Config)
if err != nil {
return err
}
header := http.Header{}
header.Set("Authorization", fetchguard.Bearer(token))
res, err := vendor.PostJSON(ctx, "https://api.example.com/v1/items", header, payload)
if err != nil {
return err
}
if res.StatusCode == http.StatusTooManyRequests {
wait := res.Header.Get("Retry-After")
// ...
}
// Multipart upload: fields first, then files, each in slice order.
res, err = vendor.PostMultipart(ctx, "https://api.example.com/v1/files", header,
[]fetchguard.FormField{{Name: "title", Value: "Report"}},
[]fetchguard.FormFile{{Field: "file", Filename: "report.pdf", ContentType: "application/pdf", Body: f}},
)
```
## API reference
| Identifier | Description |
|------------|-------------|
| `fetchguard.Fetch` | Validates the URL against the policy and performs the guarded HTTPS GET; a non-nil error is always a `fetchguard.Error`. |
| `fetchguard.Policy` | Per-call settings: mode, allowed hosts, byte limit and timeout (zero means use the configured default). |
| `fetchguard.Mode` | Selects `fetchguard.AllowHostsMode` or `fetchguard.PublicOnlyMode`. |
| `fetchguard.Result` | Response body, Content-Type header value and status code of any completed response, including 3xx and non-2xx. |
| `fetchguard.Mode` | Selects `fetchguard.AllowHostsMode`, `fetchguard.PublicOnlyMode` or `fetchguard.TrustedMode`. |
| `fetchguard.TrustedMode` | Client mode for operator-configured endpoints: http and https, no host list, no dial guard; body cap and no redirects still apply. |
| `fetchguard.Result` | Response body, Content-Type header value, status code and all headers (`Header`) of any completed response, including 3xx and non-2xx. |
| `fetchguard.NewClient` | Resolves the policy's limits once and builds a reusable `fetchguard.Client`. |
| `fetchguard.Client` | Guarded outbound client, safe for concurrent use; one per vendor keeps connections alive. |
| `fetchguard.Client.Do` | Validates the request URL and sends it; the response body fails with `too_large` past the byte limit. The caller closes the body. |
| `fetchguard.Client.Send` | `fetchguard.Client.Do` plus reading the capped body into a `fetchguard.Result`. |
| `fetchguard.Client.Get` | Sends a GET with the given headers. |
| `fetchguard.Client.PostJSON` | Marshals the body as JSON and POSTs it; `Content-Type: application/json` is set first, then the caller's headers are copied over it. |
| `fetchguard.Client.PutJSON` | `fetchguard.Client.PostJSON` with the PUT method. |
| `fetchguard.Client.PostMultipart` | POSTs a multipart/form-data body: fields, then files, in slice order. |
| `fetchguard.FormField` | One plain multipart field (`Name`, `Value`). |
| `fetchguard.FormFile` | One multipart file part (`Field`, `Filename`, `ContentType`, `Body`); the content type defaults to `application/octet-stream`. |
| `fetchguard.Bearer` | Returns the Authorization value `Bearer <token>`. |
| `fetchguard.WithTransport` | Returns a context whose client requests go to the given `http.RoundTripper`; the test and parity-replay seam, settable only from code. |
| `fetchguard.Error` | Failure carrying a `fetchguard.Reason` and the underlying error for logging. |
| `fetchguard.Reason` | Closed set of failure reasons (`invalid_url`, `scheme`, `unresolvable`, `private_ip`, `network_error`, `too_large`). |
| `fetchguard.Defaults` | Framework fallback limits: 10 MiB and 10 seconds. |
@@ -57,7 +105,7 @@ image := res.Body
## Configuration
`fetchguard.Fetch` and `fetchguard.DefaultsFromConfig` read these keys from the `compass.Config` passed to them. They apply only when the policy leaves the matching limit at zero, and an explicitly configured zero or negative value is an error.
`fetchguard.Fetch`, `fetchguard.NewClient` and `fetchguard.DefaultsFromConfig` read these keys from the `compass.Config` passed to them. They apply only when the policy leaves the matching limit at zero, and an explicitly configured zero or negative value is an error.
| Key | Default | Controls |
|-----|---------|----------|
@@ -75,7 +123,7 @@ http:
- SummerCMS modules: [compass](../compass/README.md) (config lookup).
- Third-party: none.
- Standard library: `context`, `crypto/tls`, `errors`, `fmt`, `io`, `math`, `net`, `net/http`, `net/netip`, `net/url`, `strings`, `syscall`, `time`.
- Standard library: `bytes`, `context`, `crypto/tls`, `encoding/json`, `errors`, `fmt`, `io`, `math`, `mime/multipart`, `net`, `net/http`, `net/netip`, `net/textproto`, `net/url`, `strings`, `syscall`, `time`.
## Testing
@@ -83,4 +131,4 @@ http:
go test ./modules/fetchguard/...
```
The tests run against local `net/http/httptest` TLS servers and cover the address classifier directly; they need no external services.
The tests run against local `net/http/httptest` servers and cover the address classifier directly; they need no external services. To test code that calls a vendor through a `fetchguard.Client`, pass a context from `fetchguard.WithTransport` with a fake `http.RoundTripper`, such as the [tide](../tide/README.md) upstream fake that replays recorded vendor exchanges and asserts every request.