87 lines
4.6 KiB
Markdown
87 lines
4.6 KiB
Markdown
# fetchguard
|
|
|
|
Guarded outbound HTTPS fetcher that blocks private and reserved addresses and enforces host, size and timeout limits.
|
|
|
|
`import "git.golem15.com/golem15/summercms/modules/fetchguard"`
|
|
|
|
## Overview
|
|
|
|
`fetchguard` is the framework's server-side request forgery guard for fetching URLs that come from users or third parties, such as a remote image address. Every call takes a `fetchguard.Policy` that either restricts the target to an allow list of hosts or permits any public host; in both modes the dial-time check refuses private, loopback, link-local, carrier-grade NAT, documentation, multicast and other reserved IPv4 and IPv6 ranges, including IPv4 embedded in NAT64 and 6to4 addresses. Failures come back as a `fetchguard.Error` carrying one `fetchguard.Reason` from a closed set, so callers can map them onto stable API error codes. WinterCMS has no dedicated counterpart; plugins there typically used Guzzle with hand-written checks.
|
|
|
|
## Features
|
|
|
|
- HTTPS only: any other scheme fails with `fetchguard.ReasonScheme`.
|
|
- Two modes: `fetchguard.AllowHostsMode` (exact or dotted-suffix host match against `fetchguard.Policy.AllowHosts`) and `fetchguard.PublicOnlyMode` (any public host).
|
|
- The private and reserved address check runs on the resolved IP at dial time, so DNS answers that point inside the network are refused (`fetchguard.ReasonPrivateIP`); environment proxies are ignored so the check sees the real target.
|
|
- Redirects are never followed: a 3xx response is returned as a successful `fetchguard.Result`, and a caller that wants to follow the Location header calls `fetchguard.Fetch` again, which re-runs the guard.
|
|
- The response body is capped at the policy's byte limit (`fetchguard.ReasonTooLarge` when exceeded), with a per-call timeout.
|
|
- Limits left at zero in the policy fall back to config keys, then to framework defaults of 10 MiB and 10 seconds (`fetchguard.Defaults`, `fetchguard.DefaultsFromConfig`).
|
|
- Typed failure reasons: `fetchguard.ReasonInvalidURL`, `fetchguard.ReasonScheme`, `fetchguard.ReasonUnresolvable`, `fetchguard.ReasonPrivateIP`, `fetchguard.ReasonNetworkError`, `fetchguard.ReasonTooLarge`.
|
|
|
|
## Usage
|
|
|
|
```go
|
|
policy := fetchguard.Policy{
|
|
Mode: fetchguard.AllowHostsMode,
|
|
AllowHosts: []string{"images.example.com"},
|
|
MaxBytes: 5 << 20,
|
|
Timeout: 5 * time.Second,
|
|
}
|
|
|
|
res, err := fetchguard.Fetch(ctx, imageURL, policy, app.Config)
|
|
if err != nil {
|
|
var fe *fetchguard.Error
|
|
if errors.As(err, &fe) && fe.Reason == fetchguard.ReasonPrivateIP {
|
|
return errRejectedURL
|
|
}
|
|
return err
|
|
}
|
|
if res.StatusCode != http.StatusOK {
|
|
return fmt.Errorf("image fetch: status %d", res.StatusCode)
|
|
}
|
|
image := res.Body
|
|
```
|
|
|
|
## API reference
|
|
|
|
| Identifier | Description |
|
|
|------------|-------------|
|
|
| `fetchguard.Fetch` | Validates the URL against the policy and performs the guarded HTTPS GET; a non-nil error is always a `fetchguard.Error`. |
|
|
| `fetchguard.Policy` | Per-call settings: mode, allowed hosts, byte limit and timeout (zero means use the configured default). |
|
|
| `fetchguard.Mode` | Selects `fetchguard.AllowHostsMode` or `fetchguard.PublicOnlyMode`. |
|
|
| `fetchguard.Result` | Response body, Content-Type header value and status code of any completed response, including 3xx and non-2xx. |
|
|
| `fetchguard.Error` | Failure carrying a `fetchguard.Reason` and the underlying error for logging. |
|
|
| `fetchguard.Reason` | Closed set of failure reasons (`invalid_url`, `scheme`, `unresolvable`, `private_ip`, `network_error`, `too_large`). |
|
|
| `fetchguard.Defaults` | Framework fallback limits: 10 MiB and 10 seconds. |
|
|
| `fetchguard.DefaultsFromConfig` | Reads the limits from a `compass.Config`, falling back to `fetchguard.Defaults` for absent keys. |
|
|
|
|
## Configuration
|
|
|
|
`fetchguard.Fetch` and `fetchguard.DefaultsFromConfig` read these keys from the `compass.Config` passed to them. They apply only when the policy leaves the matching limit at zero, and an explicitly configured zero or negative value is an error.
|
|
|
|
| Key | Default | Controls |
|
|
|-----|---------|----------|
|
|
| `http.fetch.max_bytes` | `10485760` (10 MiB) | Maximum response body size in bytes. |
|
|
| `http.fetch.timeout_seconds` | `10` | Dial and overall request timeout, in seconds. |
|
|
|
|
```yaml
|
|
http:
|
|
fetch:
|
|
max_bytes: 5242880
|
|
timeout_seconds: 5
|
|
```
|
|
|
|
## Dependencies
|
|
|
|
- SummerCMS modules: [compass](../compass/README.md) (config lookup).
|
|
- Third-party: none.
|
|
- Standard library: `context`, `crypto/tls`, `errors`, `fmt`, `io`, `math`, `net`, `net/http`, `net/netip`, `net/url`, `strings`, `syscall`, `time`.
|
|
|
|
## Testing
|
|
|
|
```sh
|
|
go test ./modules/fetchguard/...
|
|
```
|
|
|
|
The tests run against local `net/http/httptest` TLS servers and cover the address classifier directly; they need no external services.
|