summercms/modules/fetchguard/README.md

fetchguard

Guarded outbound HTTP client and fetcher that blocks private and reserved addresses, enforces host, size and timeout limits, and offers an explicit trusted mode for operator-configured endpoints.

import "git.golem15.com/golem15/summercms/modules/fetchguard"

Overview

fetchguard is the framework's server-side request forgery guard for fetching URLs that come from users or third parties, such as a remote image address. Every call takes a fetchguard.Policy that either restricts the target to an allow list of hosts or permits any public host; in both modes the dial-time check refuses private, loopback, link-local, carrier-grade NAT, documentation, multicast and other reserved IPv4 and IPv6 ranges, including IPv4 embedded in NAT64 and 6to4 addresses. Failures come back as a fetchguard.Error carrying one fetchguard.Reason from a closed set, so callers can map them onto stable API error codes. WinterCMS has no dedicated counterpart; plugins there typically used Guzzle with hand-written checks.

fetchguard.Fetch is a one-shot guarded GET. For calling a service API (any method, JSON or multipart bodies, bearer credentials) build one fetchguard.Client per vendor with fetchguard.NewClient; it applies the same policy to every request it sends and replaces a plugin's own curl request sender.

Features

• HTTPS only in the guarded modes: any other scheme fails with fetchguard.ReasonScheme.
• Two guarded modes: fetchguard.AllowHostsMode (exact or dotted-suffix host match against fetchguard.Policy.AllowHosts) and fetchguard.PublicOnlyMode (any public host).
• fetchguard.TrustedMode for endpoints an operator configured in Go code or admin settings, such as a model server on the local network: a fetchguard.Client in this mode accepts http and https, checks no host list and runs no dial guard, but still caps the body and never follows redirects. Only Go code that builds the policy can choose it; fetchguard.Fetch guards as fetchguard.PublicOnlyMode when given it.
• fetchguard.Client sends any method: fetchguard.Client.Do and fetchguard.Client.Send take an http.Request, and fetchguard.Client.Get, fetchguard.Client.PostJSON, fetchguard.Client.PutJSON and fetchguard.Client.PostMultipart cover the common shapes. fetchguard.Bearer builds an Authorization value. Response status and headers come back unjudged in fetchguard.Result.
• Code-only transport seam: fetchguard.WithTransport routes a context's requests to a test http.RoundTripper. No policy field, config key, environment variable or header can set it.
• The private and reserved address check runs on the resolved IP at dial time, so DNS answers that point inside the network are refused (fetchguard.ReasonPrivateIP); environment proxies are ignored so the check sees the real target.
• Redirects are never followed, in any mode: a 3xx response is returned as a successful fetchguard.Result, and a caller that wants to follow the Location header sends a new request, which re-runs the guard.
• The response body is capped at the policy's byte limit (fetchguard.ReasonTooLarge when exceeded), with a per-call timeout. Request bodies are not capped; callers bound their own inputs.
• Limits left at zero in the policy fall back to config keys, then to framework defaults of 10 MiB and 10 seconds (fetchguard.Defaults, fetchguard.DefaultsFromConfig).
• Typed failure reasons: fetchguard.ReasonInvalidURL, fetchguard.ReasonScheme, fetchguard.ReasonUnresolvable, fetchguard.ReasonPrivateIP, fetchguard.ReasonNetworkError, fetchguard.ReasonTooLarge.

Usage

policy := fetchguard.Policy{
	Mode:       fetchguard.AllowHostsMode,
	AllowHosts: []string{"images.example.com"},
	MaxBytes:   5 << 20,
	Timeout:    5 * time.Second,
}

res, err := fetchguard.Fetch(ctx, imageURL, policy, app.Config)
if err != nil {
	var fe *fetchguard.Error
	if errors.As(err, &fe) && fe.Reason == fetchguard.ReasonPrivateIP {
		return errRejectedURL
	}
	return err
}
if res.StatusCode != http.StatusOK {
	return fmt.Errorf("image fetch: status %d", res.StatusCode)
}
image := res.Body

Calling a service API with a reusable client:

vendor, err := fetchguard.NewClient(fetchguard.Policy{
	Mode:       fetchguard.AllowHostsMode,
	AllowHosts: []string{"api.example.com"},
	Timeout:    10 * time.Second,
}, app.Config)
if err != nil {
	return err
}

header := http.Header{}
header.Set("Authorization", fetchguard.Bearer(token))
res, err := vendor.PostJSON(ctx, "https://api.example.com/v1/items", header, payload)
if err != nil {
	return err
}
if res.StatusCode == http.StatusTooManyRequests {
	wait := res.Header.Get("Retry-After")
	// ...
}

// Multipart upload: fields first, then files, each in slice order.
res, err = vendor.PostMultipart(ctx, "https://api.example.com/v1/files", header,
	[]fetchguard.FormField{{Name: "title", Value: "Report"}},
	[]fetchguard.FormFile{{Field: "file", Filename: "report.pdf", ContentType: "application/pdf", Body: f}},
)

API reference

Identifier	Description
fetchguard.Fetch	Validates the URL against the policy and performs the guarded HTTPS GET; a non-nil error is always a fetchguard.Error.
fetchguard.Policy	Per-call settings: mode, allowed hosts, byte limit and timeout (zero means use the configured default).
fetchguard.Mode	Selects fetchguard.AllowHostsMode, fetchguard.PublicOnlyMode or fetchguard.TrustedMode.
fetchguard.TrustedMode	Client mode for operator-configured endpoints: http and https, no host list, no dial guard; body cap and no redirects still apply.
fetchguard.Result	Response body, Content-Type header value, status code and all headers (Header) of any completed response, including 3xx and non-2xx.
fetchguard.NewClient	Resolves the policy's limits once and builds a reusable fetchguard.Client.
fetchguard.Client	Guarded outbound client, safe for concurrent use; one per vendor keeps connections alive.
fetchguard.Client.Do	Validates the request URL and sends it; the response body fails with too_large past the byte limit. The caller closes the body.
fetchguard.Client.Send	fetchguard.Client.Do plus reading the capped body into a fetchguard.Result.
fetchguard.Client.Get	Sends a GET with the given headers.
fetchguard.Client.PostJSON	Marshals the body as JSON and POSTs it; Content-Type: application/json is set first, then the caller's headers are copied over it.
fetchguard.Client.PutJSON	fetchguard.Client.PostJSON with the PUT method.
fetchguard.Client.PostMultipart	POSTs a multipart/form-data body: fields, then files, in slice order.
fetchguard.FormField	One plain multipart field (Name, Value).
fetchguard.FormFile	One multipart file part (Field, Filename, ContentType, Body); the content type defaults to application/octet-stream.
fetchguard.Bearer	Returns the Authorization value Bearer <token>.
fetchguard.WithTransport	Returns a context whose client requests go to the given http.RoundTripper; the test and parity-replay seam, settable only from code.
fetchguard.Error	Failure carrying a fetchguard.Reason and the underlying error for logging.
fetchguard.Reason	Closed set of failure reasons (invalid_url, scheme, unresolvable, private_ip, network_error, too_large).
fetchguard.Defaults	Framework fallback limits: 10 MiB and 10 seconds.
fetchguard.DefaultsFromConfig	Reads the limits from a compass.Config, falling back to fetchguard.Defaults for absent keys.

Configuration

fetchguard.Fetch, fetchguard.NewClient and fetchguard.DefaultsFromConfig read these keys from the compass.Config passed to them. They apply only when the policy leaves the matching limit at zero, and an explicitly configured zero or negative value is an error.

Key	Default	Controls
http.fetch.max_bytes	10485760 (10 MiB)	Maximum response body size in bytes.
http.fetch.timeout_seconds	10	Dial and overall request timeout, in seconds.

http:
  fetch:
    max_bytes: 5242880
    timeout_seconds: 5

Dependencies

• SummerCMS modules: compass (config lookup).
• Third-party: none.
• Standard library: bytes, context, crypto/tls, encoding/json, errors, fmt, io, math, mime/multipart, net, net/http, net/netip, net/textproto, net/url, strings, syscall, time.

Testing

go test ./modules/fetchguard/...

The tests run against local net/http/httptest servers and cover the address classifier directly; they need no external services. To test code that calls a vendor through a fetchguard.Client, pass a context from fetchguard.WithTransport with a fake http.RoundTripper, such as the tide upstream fake that replays recorded vendor exchanges and asserts every request.