Files
summercms/.planning/phases/14-domain-jobs-and-external-integrations/14-01-PLAN.md
2026-10-03 18:56:43 +02:00

340 lines
46 KiB
Markdown

---
phase: 14-domain-jobs-and-external-integrations
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- modules/fetchguard/client.go
- modules/fetchguard/client_test.go
- modules/fetchguard/fetch.go
- modules/fetchguard/policy.go
- modules/fetchguard/example_test.go
- modules/fetchguard/README.md
- modules/tide/upstream.go
- modules/tide/upstream_test.go
- modules/tide/upstream_proxy.go
- modules/tide/upstream_proxy_test.go
- modules/tide/testdata/upstream/
- modules/tide/README.md
- modules/sunscreen/sunscreen.go
- modules/sunscreen/sunscreen_test.go
- modules/sunscreen/example_test.go
- modules/sunscreen/README.md
- modules/surf/recover_redaction_test.go
- modules/beachcomber/searchable.go
- modules/beachcomber/searchpage_test.go
- modules/beachcomber/typesense/engine.go
- modules/beachcomber/typesense/engine_test.go
- modules/beachcomber/README.md
- internal/build/build.go
- internal/build/build_test.go
- examples/hello/main.go
- cmd/summer/parity.go
- cmd/summer/main.go
- cmd/summer/main_test.go
- cmd/summer/parity_contract_test.go
- README.md
- docs/services/outbound-http.md
- docs/services/logging.md
- docs/services/parity-testing.md
- docs/services/search.md
- docs/console/utilities.md
- docs/setup/coming-from-wintercms.md
- docs/architecture/introduction.md
- ../fonoteka.go/main.go
- .planning/REQUIREMENTS.md
- .planning/ROADMAP.md
- .planning/PROJECT.md
autonomous: true
requirements: [INTG-01, INTG-02, API-08, SRCH-02]
estimate:
tokens: 350000
raw_tokens: 350000
tasks: 5
confidence: low
must_haves:
truths:
- "Per D-02 and the folded fetchguard todo, `fetchguard.NewClient(policy, cfg)` returns a guarded outbound client whose `Do`, `Send`, `Get`, `PostJSON`, `PutJSON` and `PostMultipart` send any method with JSON or multipart bodies and bearer headers, cap the response at MaxBytes (`too_large`), return the status and headers without judging them, and never follow a redirect in any mode (a 3xx comes back as a Result)."
- "Per D-05, `fetchguard.TrustedMode` is declared after `PublicOnlyMode` (existing numeric values unchanged); only TrustedMode accepts http and skips the host check and the dial-time private/reserved-IP guard; AllowHostsMode and PublicOnlyMode stay https-only with the dial guard on every new connection."
- "The transport override is code-only: it rides an unexported context key set by `fetchguard.WithTransport(ctx, rt)`; no Policy field, Client field, compass key, environment variable or request header can set it, and a reflection test proves Policy and Client export no http.RoundTripper."
- "Per D-15, `tide.LoadUpstream` reads `<fixture>.upstream.yaml` sidecars and `tide.NewUpstreamFake(sidecar, store)` answers the recorded responses offline while asserting each request Go sends (method, scheme, host, path, query, the compared headers with `{{name}}` placeholders expanded from the store, and the body compared as JSON semantically or by sha256 for base64 payloads); `Verify` fails on any mismatch, unconsumed exchange or extra request."
- "Per D-19, `summer parity:upstream` runs a loopback-only recording HTTPS proxy with a locally generated CA (key mode 0600, outside the fixtures tree), answers from a script file in script mode or forwards once in forward mode, and writes the sidecar with every secret replaced by a `{{name}}` var; an Authorization or X-Api-Key value that no var masks refuses the write."
- "Per the folded redacting-slog-handler todo, `sunscreen.Wrap` redacts the RedactCredentialsTap keys (api_key, apikey, authorization, bearer, password, secret, token, webhook_secret, admin_password, openai_api_key, anthropic_api_key, perplexity_api_key) case-insensitively in attrs, WithAttrs and nested groups, and scrubs Bearer, `sk-` and `x-api-key:` shapes from messages and string values; every generated application main installs it first through `sunscreen.InstallDefault(os.Stderr)`, and surf's recovered 500 echoes no panic text (SafeExceptionResponse behaviour pinned by a test)."
- "Per research Open Question 6 (resolved: add it), `beachcomber.DropIndex(ctx, engine, index)` reports whether the index existed through the optional `IndexDropper`, and the Typesense engine answers existed=false on a 404, so reindex can keep PHP's distinct already-absent message; `beachcomber.EnsureIndex` creates an empty index from its schema, as PHP's reindex does for an empty album table."
- "Per D-06, D-07, D-13 and D-14, REQUIREMENTS INTG-02 names Anthropic and OpenAI-compatible adapters over the guarded client, API-08 is feedback only, ROADMAP Phase 14 criteria 4-6 list the album Discogs and recognize routes and drop sitemap, its Repos line names summercms.go, sm-golem-plugin and sm-feedback-plugin, and PROJECT.md moves golem and feedback to their sm-*-plugin repos and drops feedback and sitemap from the application-plugin list, in a planning-docs-only commit."
- "Every changed module README and the affected docs pages name only identifiers that exist: `go test ./cmd/summer -run TestDocsTree` and `summer docs:build --check` pass, and no framework README or docs page names a consuming application."
artifacts:
- path: "modules/fetchguard/client.go"
provides: "NewClient, Client, Do, Send, Get, PostJSON, PutJSON, PostMultipart, FormField, FormFile, Bearer, WithTransport"
contains: "func WithTransport("
- path: "modules/tide/upstream.go"
provides: "UpstreamSidecar, UpstreamExchange, UpstreamRequest, UpstreamResponse, UpstreamPath, LoadUpstream, WriteUpstream, UpstreamFake, NewUpstreamFake, UpstreamCompareHeaders"
contains: "func NewUpstreamFake("
- path: "modules/tide/upstream_proxy.go"
provides: "UpstreamProxyConfig, UpstreamProxy, NewUpstreamProxy, EnsureParityCA, UpstreamScript, DefaultUpstreamProxyListen"
contains: "DefaultUpstreamProxyListen"
- path: "modules/sunscreen/sunscreen.go"
provides: "Wrap, InstallDefault, Scrub, Redacted, RedactedKeys"
contains: "func InstallDefault("
- path: "modules/beachcomber/searchable.go"
provides: "IndexDropper, DropIndex, IndexEnsurer, EnsureIndex"
contains: "IndexDropper"
- path: "cmd/summer/parity.go"
provides: "parity:upstream command"
contains: "parity:upstream"
key_links:
- from: "modules/tide/upstream.go"
to: "modules/fetchguard/client.go"
via: "UpstreamFake is the http.RoundTripper handed to fetchguard.WithTransport during replay"
pattern: "RoundTrip"
- from: "internal/build/build.go"
to: "modules/sunscreen/sunscreen.go"
via: "generated main calls sunscreen.InstallDefault(os.Stderr) before loading config"
pattern: "sunscreen.InstallDefault"
- from: "cmd/summer/parity.go"
to: "modules/tide/upstream_proxy.go"
via: "parity:upstream builds an UpstreamProxy and flushes the sidecar on shutdown"
pattern: "NewUpstreamProxy"
prohibitions:
- requirement_id: INTG-02
category: safety
statement: "The trusted (unguarded) outbound mode MUST NOT be selectable by configuration, request input or environment; only Go code that builds the policy can choose it"
status: resolved
verification: test
- requirement_id: INTG-01
category: privacy
statement: "A recorded upstream sidecar MUST NOT contain a live credential; an unmasked Authorization or X-Api-Key value refuses the write"
status: resolved
verification: test
---
## Phase Goal
ROADMAP Phase 14 goal (verbatim, not in user-story form): The domain-specific River jobs (CSV import write, Discogs match, wishlist digest), the reindex command, the Discogs client and AI cover recognition are ported on top of the Phase 11 jobs/realtime/search infrastructure and the Phase 13 API surface they serve.
This plan's slice: the framework gains what every later plan of the phase stands on. An application can call a vendor API through one guarded client, replay that call offline against what PHP really sent, record PHP's real upstream exchanges, keep credentials out of logs, and tell a dropped search index from an absent one. The planning docs say what Phase 14 now ships.
<objective>
Extend fetchguard into a guarded outbound client with a trusted mode and a code-only transport seam, add tide upstream sidecars with an asserting replay fake and the `summer parity:upstream` recording proxy, add the `sunscreen` redacting slog handler installed by every generated main, add `beachcomber.IndexDropper`, update module READMEs and docs, and reword REQUIREMENTS, ROADMAP and PROJECT per D-06, D-07, D-13 and D-14.
Purpose: Discogs, the Golem AI adapters and the G15Office job (plans 14-02 to 14-05) all send credentials to outside services through this client and are tested through these sidecars (D-02, D-05, D-15, D-19).
Output: framework packages with tests, READMEs and docs; regenerated application mains; reworded planning docs.
Repos: summercms.go (code and planning docs, separate commits) and fonoteka.go (only the regenerated `main.go`, its own commit). Commits are path-scoped; never add co-author tags.
</objective>
<execution_context>
@~/.claude/gsd-core/workflows/execute-plan.md
@~/.claude/gsd-core/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/STATE.md
@.planning/phases/14-domain-jobs-and-external-integrations/14-CONTEXT.md
@.planning/phases/14-domain-jobs-and-external-integrations/14-RESEARCH.md
@.planning/phases/14-domain-jobs-and-external-integrations/14-PATTERNS.md
@modules/fetchguard/fetch.go
@modules/fetchguard/policy.go
<interfaces>
- fetchguard today: `Fetch(ctx, rawURL, Policy, *compass.Config) (*Result, error)` GET only; `Policy{Mode, AllowHosts, MaxBytes, Timeout}` plus unexported `tlsConfig`, `skipReservedCheck`; `Mode` constants `AllowHostsMode`, `PublicOnlyMode`; `Result{Body, ContentType, StatusCode}`; `*Error{Reason, Err}` with reasons invalid_url, scheme, unresolvable, private_ip, network_error, too_large; helpers `resolveLimits`, `dialControl`, `mapTransportError`, `hostAllowed`, `isReservedOrPrivate` (ip.go).
- tide: `Store` and `OpenStore(path)` (variables.go, `{{name}}` substitution and reverse masking), `CentrifugoRecorder` (centrifugo.go, the fake-recorder pattern: options, mutex slice, LimitReader, never stores the auth secret), `Proxy`/`ProxyConfig`/`NewProxy` and `requireLoopbackAddr` (proxy.go), `DefaultCentrifugoListen = "127.0.0.1:8424"`, flows decoded with goccy/go-yaml DisallowUnknownField (fixture.go).
- cmd/summer: `toolCommands()` (main.go), `parityBroadcastsCommand()` and `requireFlag`, `varsOutsideDir` (parity.go), `TestToolCommandNames` expected list (main_test.go), `TestParityCommandContract` (parity_contract_test.go).
- internal/build: `generateMain(m Manifest)` writes the application main; `TestDocsCommandsMirrorGeneratedMain` only parses `commands :=` / `append(commands,` constructors.
- beachcomber: `Engine{Name, Configured, Upsert, Delete, Flush, SearchIDs}`, optional `PageSearcher` plus `SearchPage(ctx, e, index, q)` fallback (searchable.go:87-108); typesense `(*Engine).Flush` treats 404 as success (typesense/engine.go:205-215).
- surf: `recoverJSON`/`recoverBare` (router.go), `TestRecoverReturnsOpaqueJSON500` (router_test.go).
</interfaces>
</context>
## Artifacts this phase produces
(This plan's share.)
- fetchguard: `NewClient`, `Client`, `(*Client).Do`, `Send`, `Get`, `PostJSON`, `PutJSON`, `PostMultipart`, `FormField{Name, Value}`, `FormFile{Field, Filename, ContentType, Body}`, `Bearer`, `WithTransport`, `TrustedMode`, `Result.Header`.
- tide: `UpstreamSidecar`, `UpstreamExchange`, `UpstreamRequest`, `UpstreamResponse`, `UpstreamPath`, `LoadUpstream`, `WriteUpstream`, `UpstreamFake`, `NewUpstreamFake`, `(*UpstreamFake).RoundTrip`, `(*UpstreamFake).Verify`, `UpstreamCompareHeaders`, `UpstreamScript`, `UpstreamScriptResponse`, `UpstreamProxyConfig`, `UpstreamProxy`, `NewUpstreamProxy`, `(*UpstreamProxy).ListenAndServe`, `(*UpstreamProxy).Flush`, `EnsureParityCA`, `DefaultUpstreamProxyListen` (`127.0.0.1:8425`). Sidecar file format `<fixture>.upstream.yaml` (version 1).
- New module `modules/sunscreen` (import `git.golem15.com/golem15/summercms/modules/sunscreen`): `Wrap`, `InstallDefault`, `Scrub`, `Redacted`, `RedactedKeys`; README; root README modules row.
- beachcomber: `IndexDropper`, `DropIndex`, `IndexEnsurer`, `EnsureIndex`; typesense `(*Engine).DropIndex`, `(*Engine).EnsureIndex`.
- CLI: `summer parity:upstream` (`--listen`, `--ca-dir`, `--out`, `--mode`, `--script`, `--vars`).
- Docs: new `docs/services/logging.md`; updated outbound-http, parity-testing, search, console utilities, coming-from-wintercms, architecture introduction.
- Tests: `TestClientPostJSONThroughUpstreamFake`, `TestUpstreamFakeRejectsMismatchedRequest`, `TestTransportSeamIsCodeOnly`, `TestClientModes`, `TestClientNeverFollowsRedirects`, `TestClientMultipart`, `TestClientBodyCap`, `TestClientSchemeGuard`, `TestUpstreamProxyScriptMode`, `TestUpstreamProxyRefusesNonLoopback`, `TestWriteUpstreamRefusesUnmaskedCredential`, `TestUpstreamFakeHashesBase64Bodies`, `TestEnsureParityCA`, `TestRedactHandler`, `TestScrub`, `TestGenerateMainInstallsRedactingLogger`, `TestRecoverHidesPanicDetails`, `TestDropIndex`, `TestEngineDropIndex`, `TestEngineEnsureIndex`.
## Assumptions
- The phrase "unexported test-transport seam" in the confirmed plan split is implemented as an unexported context key type and unexported Client state, set only through the code-level `WithTransport`; nothing reachable from production input can set it.
- Sidecars carry no step index: exchanges are consumed in recorded order across the whole flow.
- A4 (research): consumers choose their timeouts (Discogs 10 s, AI 120 s, G15Office 30 s); the client has no vendor defaults.
<tasks>
<task type="tracer">
<name>Task 1: A guarded JSON POST leaves through fetchguard.Client and a tide upstream fake answers it offline while asserting the exact request</name>
<files>modules/fetchguard/client.go, modules/fetchguard/client_test.go, modules/fetchguard/fetch.go, modules/fetchguard/policy.go, modules/tide/upstream.go, modules/tide/upstream_test.go, modules/tide/testdata/upstream/</files>
<read_first>modules/fetchguard/fetch.go (Fetch, resolveLimits, dialControl, mapTransportError, the body cap at lines 85-96), modules/fetchguard/policy.go, modules/fetchguard/fetch_test.go (withTestLoopback), modules/tide/centrifugo.go (CentrifugoRecorder: options, mutex, LimitReader, secret never stored), modules/tide/variables.go (Store, placeholder expansion and reverse masking), modules/tide/fixture.go (goccy decode with DisallowUnknownField), modules/tide/diff.go (structural JSON compare with UseNumber), .planning/phases/14-domain-jobs-and-external-integrations/14-RESEARCH.md (sections "fetchguard: what the guarded client needs" and "How sidecars fit the existing layout")</read_first>
<action>Per D-02, D-05 and D-15 (thinnest path through both packages).
(1) fetchguard/client.go: `NewClient(policy Policy, cfg *compass.Config) (*Client, error)` resolves MaxBytes and Timeout once through resolveLimits and builds one http.Client: CheckRedirect returns http.ErrUseLastResponse, Transport with Proxy nil, the dial Control from dialControl(policy), TLSClientConfig from policy.tlsConfig, keep-alives on (the dial Control runs on every new connection). `(*Client).Do(req *http.Request) (*http.Response, error)` validates the URL exactly as Fetch does (invalid_url, https-only, AllowHosts in AllowHostsMode), then sends through the RoundTripper stored in the request context by `WithTransport` when present, otherwise through its own transport, and wraps the response body in a reader that fails with `*Error{Reason: ReasonTooLarge}` past MaxBytes. `(*Client).Send(req) (*Result, error)` reads the whole capped body. `(*Client).PostJSON(ctx, rawURL string, header http.Header, body any) (*Result, error)` marshals body, sets `Content-Type: application/json` first and then copies header (the RequestSender order). Result gains `Header http.Header` (additive field; Fetch fills it too). `WithTransport(ctx context.Context, rt http.RoundTripper) context.Context` stores rt under an unexported key type; its doc comment states it is the test and parity-replay seam and that no Policy field, config key, environment variable or header can set it. Refactor Fetch to share the transport builder without changing its behaviour or tests.
(2) tide/upstream.go: types `UpstreamSidecar{Version int; Exchanges []UpstreamExchange}`, `UpstreamExchange{Request UpstreamRequest; Response UpstreamResponse}`, `UpstreamRequest{Method, URL string; Headers map[string]string; Body string}`, `UpstreamResponse{Status int; Headers map[string]string; Body string}` with yaml tags in lower snake case; `UpstreamPath(fixturePath string) string` maps `x.yaml` to `x.upstream.yaml`; `LoadUpstream(path string) (UpstreamSidecar, error)` decodes with goccy/go-yaml DisallowUnknownField, requires version 1 and wraps fs.ErrNotExist when the file is absent. `UpstreamCompareHeaders` lists User-Agent, Accept, Content-Type, Authorization, X-Api-Key, Anthropic-Version, Anthropic-Beta. `NewUpstreamFake(s UpstreamSidecar, store *Store) *UpstreamFake` implements http.RoundTripper: it takes the next exchange in order, compares method, scheme, host and path, the parsed query (order-insensitive), each compared header present on either side after expanding `{{name}}` placeholders from store, and the body (JSON semantically with UseNumber when either Content-Type is JSON, otherwise bytes); a match returns the recorded status, headers and body without dialing; a mismatch returns an error naming the field and is remembered. `(*UpstreamFake).Verify() error` joins every mismatch, each unconsumed exchange and each extra request. Neutral hosts only (api.example.test).
(3) Tests: client_test.go `TestClientPostJSONThroughUpstreamFake` (load testdata/upstream/post_json.upstream.yaml, AllowHostsMode client for api.example.test, ctx from WithTransport with the fake, PostJSON returns the sidecar status, header and body, Verify is nil) and `TestTransportSeamIsCodeOnly` (reflection over Policy and Client finds no exported field of a type implementing http.RoundTripper; a request without the context override reaches the real transport, proven by an AllowHostsMode call to a host outside the list failing before any dial); upstream_test.go `TestUpstreamFakeRejectsMismatchedRequest` (wrong method, path, query value, User-Agent, Authorization after expansion, JSON body value, an extra request and an unconsumed exchange each make Verify fail with the field named).</action>
<verify>
<automated>go vet ./... &amp;&amp; go test ./modules/fetchguard ./modules/tide -count=1 -v -run '^(TestClientPostJSONThroughUpstreamFake|TestTransportSeamIsCodeOnly|TestUpstreamFakeRejectsMismatchedRequest)$' &amp;&amp; go test ./modules/fetchguard ./modules/tide -count=1</automated>
<fails_when>Any command exits non-zero; the verbose run prints "--- FAIL", "no tests to run" or "--- SKIP", or lacks "--- PASS" for TestClientPostJSONThroughUpstreamFake, TestTransportSeamIsCodeOnly and TestUpstreamFakeRejectsMismatchedRequest; the package runs report FAIL for an existing fetchguard or tide test.</fails_when>
</verify>
<acceptance_criteria>
- `grep -c 'func WithTransport(' modules/fetchguard/client.go` prints 1.
- `grep -c 'func NewUpstreamFake(' modules/tide/upstream.go` prints 1 and `grep -c 'func (f \*UpstreamFake) Verify() error' modules/tide/upstream.go` prints 1.
- `grep -cE 'ErrUseLastResponse' modules/fetchguard/client.go` prints at least 1.
- `ls modules/tide/testdata/upstream/post_json.upstream.yaml` succeeds and `grep -c 'version: 1' modules/tide/testdata/upstream/post_json.upstream.yaml` prints 1.
</acceptance_criteria>
<done>A JSON POST built by the guarded client is answered offline from a sidecar and its request is asserted field by field, proving the D-15 replay architecture before any vendor code exists.</done>
</task>
<task type="auto">
<name>Task 2: An application calls any vendor API through the guarded client (PUT, multipart, bearer, trusted operator endpoints) and the docs show how</name>
<files>modules/fetchguard/client.go, modules/fetchguard/client_test.go, modules/fetchguard/policy.go, modules/fetchguard/example_test.go, modules/fetchguard/README.md, README.md, docs/services/outbound-http.md, docs/setup/coming-from-wintercms.md, docs/architecture/introduction.md</files>
<read_first>modules/fetchguard/client.go (Task 1), modules/fetchguard/policy.go (Mode block lines 12-18), modules/fetchguard/ip.go, modules/fetchguard/README.md, modules/fetchguard/example_test.go, README.md (modules table), docs/services/outbound-http.md, docs/setup/coming-from-wintercms.md (line 41 row), docs/architecture/introduction.md (line 28), /media/nvme/dev/golem15/fonoteka/plugins/golem15/apparatus/classes/RequestSender.php (sendPostRequest, header order, multipart), CLAUDE.md "Documentation" section</read_first>
<action>Per D-02 and D-05.
(1) policy.go: add `TrustedMode` after `PublicOnlyMode` in the Mode const block so existing values keep 0 and 1. Its doc comment: for endpoints an operator configured in Go code or admin settings (for example a LAN model server); http and https allowed, no host check, no dial guard; still capped, still no redirects. client.go: in TrustedMode skip the scheme rule, the AllowHosts check and dialControl; in the other modes keep them. Add `PutJSON(ctx, rawURL, header, body)`, `Get(ctx, rawURL, header)`, `PostMultipart(ctx, rawURL string, header http.Header, fields []FormField, files []FormFile) (*Result, error)` with `FormField{Name, Value string}` and `FormFile{Field, Filename, ContentType string; Body io.Reader}` written in slice order through mime/multipart, and `Bearer(token string) string` returning `Bearer <token>`. Request bodies are not capped (callers bound their own inputs).
(2) Tests in client_test.go: `TestClientModes` (AllowHostsMode refuses a host outside the list with invalid_url; PublicOnlyMode refuses a loopback address at dial with private_ip; TrustedMode reaches an http httptest server on 127.0.0.1), `TestClientNeverFollowsRedirects` (a 302 is returned as a Result in all three modes and the Location target is never requested), `TestClientMultipart` (field and file order, filename, part Content-Type and bytes as received by an httptest server), `TestClientBodyCap` (MaxBytes plus one byte fails Send with too_large; Do's body reader fails the same way), `TestClientSchemeGuard` (http refused with scheme in both guarded modes). example_test.go: `ExampleClient_PostJSON` using WithTransport with a stub RoundTripper, no network.
(3) Docs in the same commit: fetchguard README (summary sentence now "Guarded outbound HTTP client and fetcher that blocks private and reserved addresses, enforces host, size and timeout limits, and offers an explicit trusted mode for operator-configured endpoints."; Features, Usage with Client, API reference for every new identifier, Testing names WithTransport); root README modules row reuses that sentence; docs/services/outbound-http.md gains "Calling a service API" (client per vendor, JSON, multipart, bearer, per-consumer timeouts, trusted mode only for operator endpoints, never for user-supplied URLs) and "Testing outbound calls" (WithTransport plus a link to parity-testing.md); coming-from-wintercms.md gains a row mapping a plugin's own curl request sender to `fetchguard.Client`; architecture/introduction.md keeps fetchguard in the HTTP row with the new wording. Never name a consuming application.</action>
<verify>
<automated>go vet ./... &amp;&amp; go test ./modules/fetchguard -count=1 -race -v -run '^(TestClientModes|TestClientNeverFollowsRedirects|TestClientMultipart|TestClientBodyCap|TestClientSchemeGuard|ExampleClient_PostJSON)$' &amp;&amp; go test ./modules/fetchguard -count=1 &amp;&amp; go test ./cmd/summer -count=1 -run '^(TestDocsTree|TestDocsCommandsMirrorGeneratedMain)$' &amp;&amp; go run ./cmd/summer docs:build --check</automated>
<fails_when>Any command exits non-zero; the verbose run prints "--- FAIL", "no tests to run", "--- SKIP" or "DATA RACE", or lacks "--- PASS" for any of the five client tests or ExampleClient_PostJSON; docs:build --check reports an unknown identifier, a broken link or a consuming-application name.</fails_when>
</verify>
<acceptance_criteria>
- `grep -nE '^\s+TrustedMode$' modules/fetchguard/policy.go` shows a line after the `PublicOnlyMode` line.
- `grep -c 'func (c \*Client) PostMultipart(' modules/fetchguard/client.go` prints 1 and `grep -c 'func Bearer(' modules/fetchguard/client.go` prints 1.
- `grep -c 'TrustedMode' modules/fetchguard/README.md` and `grep -c 'WithTransport' docs/services/outbound-http.md` each print at least 1.
- The fetchguard row of README.md contains "trusted mode for operator-configured endpoints".
</acceptance_criteria>
<done>The guarded client covers every request shape the Discogs, Golem and G15Office ports need, with the trusted mode reachable only from code, and the README and docs describe it.</done>
</task>
<task type="auto">
<name>Task 3: A developer records what the reference backend really sends upstream with summer parity:upstream and replays it offline</name>
<files>modules/tide/upstream.go, modules/tide/upstream_test.go, modules/tide/upstream_proxy.go, modules/tide/upstream_proxy_test.go, modules/tide/testdata/upstream/, modules/tide/README.md, cmd/summer/parity.go, cmd/summer/main.go, cmd/summer/main_test.go, cmd/summer/parity_contract_test.go, docs/console/utilities.md, docs/services/parity-testing.md</files>
<read_first>modules/tide/proxy.go (ProxyConfig, NewProxy, requireLoopbackAddr, varsOutsideFixtures, sessions), modules/tide/variables.go (Store, reverse masking, unclassified credential refusal), modules/tide/rules.go, modules/tide/upstream.go (Task 1), cmd/summer/parity.go (parityBroadcastsCommand, runParityBroadcasts, requireFlag, varsOutsideDir), cmd/summer/main.go (toolCommands), cmd/summer/main_test.go (TestToolCommandNames), cmd/summer/parity_contract_test.go, docs/console/utilities.md (parity table), docs/services/parity-testing.md, modules/tide/README.md, .planning/phases/14-domain-jobs-and-external-integrations/14-RESEARCH.md ("Recording PHP's side" paragraph and assumption A1)</read_first>
<action>Per D-15 and D-19.
(1) upstream.go: `WriteUpstream(path string, s UpstreamSidecar, store *Store) error` replaces every header value and body substring equal to a store variable value with `{{name}}`, refuses (error naming the header) an Authorization or X-Api-Key value that is not fully masked, replaces any JSON string value longer than 1024 characters that decodes as base64 with `{{sha256:<hex of the decoded bytes>}}`, and writes mode 0644. The fake compares such a placeholder by hashing the decoded value Go sent. Multipart bodies are stored as an ordered list of parts (name, filename, content type, sha256 of the bytes) and compared that way.
(2) upstream_proxy.go: `DefaultUpstreamProxyListen = "127.0.0.1:8425"`; `UpstreamProxyConfig{Listen, CADir, Out, Mode, Script, VarsPath string}`; `NewUpstreamProxy(cfg) (*UpstreamProxy, error)` requires a loopback Listen (requireLoopbackAddr), Mode `script` (default) or `forward`, Script in script mode, Out, and a CADir and VarsPath outside the directory that holds Out; `EnsureParityCA(dir string) (certPath string, err error)` creates or reuses an ECDSA P-256 self-signed CA (`parity-ca.pem` 0644, `parity-ca-key.pem` 0600) with crypto/x509 only; the proxy answers CONNECT, terminates TLS with a per-host leaf certificate signed by that CA, reads the decrypted request (body capped), and in script mode answers from `UpstreamScript{Responses []UpstreamScriptResponse{Method, Host, Path string; Response UpstreamResponse}}` (first unused entry whose method, host and path match; no match answers 599 and is recorded as an error), in forward mode sends the request once through a `fetchguard.Client` in PublicOnlyMode and returns the real response; every exchange is appended under a mutex; `ListenAndServe(ctx)` stops on ctx cancel; `Flush() error` writes Out with WriteUpstream.
(3) CLI: cmd/summer/parity.go `parityUpstreamCommand()` named `parity:upstream` with flags `--listen` (default tide.DefaultUpstreamProxyListen), `--ca-dir`, `--out`, `--mode` (default script), `--script`, `--vars`, using requireFlag; it prints the CA certificate path for the reference backend's curl and openssl CA settings, serves until interrupted, then flushes. Register it in toolCommands, add it to TestToolCommandNames' list with its flags, and add a contract case (non-loopback listen refused, vars inside the output directory refused) to TestParityCommandContract.
(4) Tests: upstream_proxy_test.go `TestUpstreamProxyScriptMode` (a Go client with Proxy set to the proxy URL and RootCAs from EnsureParityCA posts JSON with an Authorization header to https://api.example.test/v1/things, receives the scripted response, Flush writes a sidecar whose Authorization is `{{secret:example-token}}` and that LoadUpstream reads back), `TestUpstreamProxyRefusesNonLoopback`, `TestEnsureParityCA` (key mode 0600, second call reuses the files); upstream_test.go `TestWriteUpstreamRefusesUnmaskedCredential`, `TestUpstreamFakeHashesBase64Bodies` (a 2 KiB base64 image in a JSON body is stored as a sha256 placeholder and the fake accepts the same bytes and rejects one changed byte).
(5) Docs in the same commit: modules/tide/README.md (Features, Usage for sidecars, fake and proxy, API reference for every new identifier, CLI commands row, security rules: loopback only, 0600 CA key outside fixtures, masked credentials, forward mode only by explicit flag and never in CI); docs/console/utilities.md table row for `summer parity:upstream` with flags; docs/services/parity-testing.md section "Upstream exchanges" (sidecar format, recording through the proxy with the reference backend's HTTPS proxy and CA settings, offline replay through WithTransport). Neutral names only.</action>
<verify>
<automated>go vet ./... &amp;&amp; go test ./modules/tide -count=1 -race -v -run '^(TestUpstreamProxyScriptMode|TestUpstreamProxyRefusesNonLoopback|TestEnsureParityCA|TestWriteUpstreamRefusesUnmaskedCredential|TestUpstreamFakeHashesBase64Bodies|TestUpstreamFakeRejectsMismatchedRequest)$' &amp;&amp; go test ./modules/tide -count=1 &amp;&amp; go test ./cmd/summer -count=1 -v -run '^(TestToolCommandNames|TestParityCommandContract|TestDocsTree|TestDocsCommandsMirrorGeneratedMain)$' &amp;&amp; go run ./cmd/summer docs:build --check</automated>
<fails_when>Any command exits non-zero; a verbose run prints "--- FAIL", "no tests to run", "--- SKIP" or "DATA RACE", or lacks "--- PASS" for TestUpstreamProxyScriptMode, TestEnsureParityCA, TestWriteUpstreamRefusesUnmaskedCredential, TestUpstreamFakeHashesBase64Bodies and TestToolCommandNames; docs:build --check reports an unknown identifier or command.</fails_when>
</verify>
<acceptance_criteria>
- `grep -c '"parity:upstream"' cmd/summer/parity.go` prints at least 1 and `grep -c 'parityUpstreamCommand()' cmd/summer/main.go` prints 1.
- `grep -c 'parity:upstream' docs/console/utilities.md` prints at least 1 and `grep -c 'upstream.yaml' docs/services/parity-testing.md` prints at least 1.
- `grep -c '127.0.0.1:8425' modules/tide/upstream_proxy.go` prints at least 1.
- `grep -rc 'golang.org/x/' modules/tide/upstream_proxy.go` prints 0 (stdlib crypto only).
</acceptance_criteria>
<done>PHP's real outbound requests can be captured through a loopback MITM proxy into masked sidecars and replayed offline, with the command documented.</done>
</task>
<task type="auto">
<name>Task 4: Operators never see a credential in application logs, and a reindex can tell a dropped search index from an absent one</name>
<files>modules/sunscreen/sunscreen.go, modules/sunscreen/sunscreen_test.go, modules/sunscreen/example_test.go, modules/sunscreen/README.md, modules/surf/recover_redaction_test.go, internal/build/build.go, internal/build/build_test.go, examples/hello/main.go, modules/beachcomber/searchable.go, modules/beachcomber/searchpage_test.go, modules/beachcomber/typesense/engine.go, modules/beachcomber/typesense/engine_test.go, modules/beachcomber/README.md, README.md, docs/services/logging.md, docs/services/search.md, ../fonoteka.go/main.go</files>
<read_first>/media/nvme/dev/golem15/fonoteka/plugins/golem15/apparatus/classes/logging/RedactCredentialsTap.php, /media/nvme/dev/golem15/fonoteka/plugins/golem15/apparatus/classes/traits/SafeExceptionResponse.php, .planning/todos/pending/redacting-slog-handler.md, .planning/phases/14-domain-jobs-and-external-integrations/14-RESEARCH.md ("Redacting slog handler" and its code example), internal/build/build.go (generateMain lines 80-133), internal/build/build_test.go (TestGenerateMainRegistersCongaRuntimeCommands), modules/surf/router.go (recoverJSON, recoverBare), modules/surf/router_test.go (TestRecoverReturnsOpaqueJSON500), modules/beachcomber/searchable.go (PageSearcher, SearchPage), modules/beachcomber/typesense/engine.go (Flush, do, statusError), modules/beachcomber/README.md, docs/services/search.md, README.md (modules table format), CLAUDE.md "Documentation" section</read_first>
<action>Per the folded redacting-slog-handler todo and research Open Question 6.
(1) New module modules/sunscreen (package sunscreen): `Redacted = "[REDACTED]"`; `RedactedKeys` holds api_key, apikey, authorization, bearer, password, secret, token, webhook_secret, admin_password, openai_api_key, anthropic_api_key, perplexity_api_key (compared case-insensitively); `Scrub(s string) string` applies the PHP patterns: `Bearer` plus token becomes `Bearer [REDACTED]`, `sk-` followed by 20 or more alphanumerics becomes `sk-[REDACTED]`, `x-api-key:` plus value becomes `x-api-key: [REDACTED]` (case-insensitive where PHP is); `Wrap(next slog.Handler) slog.Handler` resolves each attr value (LogValuer), replaces values of redacted keys, recurses into groups, scrubs string values and error strings, scrubs the record message, redacts WithAttrs attrs before delegating and keeps WithGroup wrapped; `InstallDefault(w io.Writer)` calls slog.SetDefault with Wrap over slog.NewTextHandler(w, nil) and never wraps the existing default handler (that handler writes through the log package, which SetDefault redirects back into the new handler). README with the standard structure (H1, summary "Credential-redacting slog handler that keeps API keys, tokens and passwords out of application logs.", import line, Overview, Features, Usage, API reference, Dependencies, Testing) and a root README modules row reusing that sentence. Tests: `TestRedactHandler` (top-level, WithAttrs, nested group, mixed-case key, LogValuer, error value), `TestScrub` (each pattern plus a non-secret string unchanged); example_test.go `ExampleWrap`.
(2) internal/build/build.go: generateMain imports sunscreen and emits `sunscreen.InstallDefault(os.Stderr)` as the first statement of run; `TestGenerateMainInstallsRedactingLogger` asserts the line precedes compass.Load. Regenerate examples/hello/main.go and ../fonoteka.go/main.go with the built tool (build summer to a temporary path from this repo, run its build subcommand in each directory); commit the fonoteka.go main.go alone in fonoteka.go.
(3) modules/surf/recover_redaction_test.go `TestRecoverHidesPanicDetails`: a JSON-group and a raw-group handler panic with an error carrying an `sk-` key and a Bearer token; both responses equal the existing opaque bodies and contain neither secret, and a record logged through a sunscreen-wrapped handler contains neither (the SafeExceptionResponse check the todo asks for).
(4) beachcomber: `IndexDropper` interface with `DropIndex(ctx context.Context, index string) (existed bool, err error)` beside PageSearcher, and `DropIndex(ctx, e Engine, index string) (bool, error)` using it when implemented, else calling Flush and reporting existed true (documented). typesense `(*Engine).DropIndex` sends DELETE /collections/{index}; 2xx is existed true, 404 existed false, anything else the statusError. Also add the optional `IndexEnsurer` interface with `EnsureIndex(ctx, index string, schema map[string]any) error` and the helper `EnsureIndex(ctx, e Engine, index string, schema map[string]any) error` (a no-op for engines without it); typesense implements it with its existing collection-creation step, because Upsert of zero documents creates nothing and PHP's reindex creates the collection when the album table is empty. Tests `TestDropIndex` (fallback and interface paths), `TestEngineDropIndex` (httptest Typesense answering 200 then 404) and `TestEngineEnsureIndex` (creates once, a second call sends no create). README API reference and docs/services/search.md describe DropIndex and EnsureIndex.
(5) docs/services/logging.md (new, `section: services`): what is redacted, that generated mains install it, that plugins resolve `*slog.Logger` or fall back to slog.Default and log ids, never args, tokens or bodies. Neutral names only.</action>
<verify>
<automated>go vet ./... &amp;&amp; go test ./modules/sunscreen ./modules/surf ./modules/beachcomber/... ./internal/build -count=1 -race -v -run '^(TestRedactHandler|TestScrub|ExampleWrap|TestRecoverHidesPanicDetails|TestDropIndex|TestEngineDropIndex|TestEngineEnsureIndex|TestGenerateMainInstallsRedactingLogger)$' &amp;&amp; go test ./modules/sunscreen ./modules/surf ./modules/beachcomber/... ./internal/build -count=1 &amp;&amp; go test ./cmd/summer -count=1 -run '^(TestDocsTree|TestDocsCommandsMirrorGeneratedMain)$' &amp;&amp; go run ./cmd/summer docs:build --check &amp;&amp; go -C ../fonoteka.go vet ./... &amp;&amp; go -C ../fonoteka.go build ./...</automated>
<fails_when>Any command exits non-zero; the verbose run prints "--- FAIL", "no tests to run", "--- SKIP" or "DATA RACE", or lacks "--- PASS" for TestRedactHandler, TestScrub, TestRecoverHidesPanicDetails, TestDropIndex, TestEngineDropIndex, TestEngineEnsureIndex and TestGenerateMainInstallsRedactingLogger; docs:build --check reports a problem; the application no longer builds.</fails_when>
</verify>
<acceptance_criteria>
- `grep -c 'sunscreen.InstallDefault(os.Stderr)' internal/build/build.go` prints at least 1 and `grep -c 'sunscreen.InstallDefault(os.Stderr)' ../fonoteka.go/main.go` prints 1.
- `grep -c 'modules/sunscreen/README.md' README.md` prints 1.
- `grep -c 'IndexDropper' modules/beachcomber/README.md` prints at least 1.
- `ls docs/services/logging.md` succeeds and `grep -c 'section: services' docs/services/logging.md` prints 1.
</acceptance_criteria>
<done>Every application binary logs through the redacting handler, surf's 500 path provably hides panic text, and reindex has a framework call that distinguishes a dropped index from an absent one.</done>
</task>
<task type="auto">
<name>Task 5: The roadmap, requirements and project description say what Phase 14 ships (D-06, D-07, D-13, D-14)</name>
<files>.planning/REQUIREMENTS.md, .planning/ROADMAP.md, .planning/PROJECT.md</files>
<read_first>.planning/REQUIREMENTS.md (lines 28, 83, 105-106), .planning/ROADMAP.md (line 29 and the "### Phase 14:" section), .planning/PROJECT.md (lines 82, 120, 137), .planning/phases/14-domain-jobs-and-external-integrations/14-CONTEXT.md (D-06, D-07, D-09, D-13, D-14), .planning/notes/core-plugins-own-repos.md</read_first>
<action>Planning docs only, one commit in summercms.go containing only these three files; use Edit, never a whole-file Write.
(1) REQUIREMENTS per D-06 and D-13/D-14: INTG-02 reads "AI cover recognition through Anthropic and OpenAI-compatible adapters over the guarded client, with per-credential model and base URL overrides, plus the ai-credential/test route and the backend global vision model behind the AI resolver's admin tier" (the old SDK wording is removed). API-08 reads "Feedback submissions, widget config and the per-user hide preference from the stack feedback plugin (sitemap dropped for this application, D-14)". INTG-01 also lists `albums/match`, `albums/{id}/match`, `albums/{id}/apply-release` and `albums/import/discogs` (D-07). Leave status checkboxes and the traceability table untouched.
(2) ROADMAP per D-06, D-07, D-14: the Phase 14 summary bullet on line 29 drops "sitemap" and says feedback; Phase 14 `**Repos:**` becomes "fonoteka.go; summercms.go (14-01 framework helpers); sm-golem-plugin and sm-feedback-plugin (new core-plugin repos mounted as submodules)"; criterion 4 additionally lists `albums/match`, `albums/{id}/match`, `albums/{id}/apply-release` and `albums/import/discogs`; criterion 5 additionally lists `albums/recognize` on the JWT and personal-token groups and names the adapters as hand-rolled over the guarded client; criterion 6 reads "Feedback submissions, widget config and the hide preference work (sitemap dropped for this application, D-14); the oauth-client, prune-notifications and reindex commands all run correctly." Add one sentence under criterion 6 that `oauth-identities` GET/DELETE and token `GET /api/v1/fonoteka/me` stay pending (D-09, todo orphan-pending-routes).
(3) PROJECT.md per D-06: line 82 becomes feedback only; the "Two repositories" paragraph lists the application plugins as fonoteka and translate, and names the shared core plugins mounted as submodules: sm-user-plugin, sm-golem-plugin (`plugins/golem15/golem`) and sm-feedback-plugin (`plugins/golem15/feedback`); sitemap is no longer listed for the application (D-14).</action>
<verify>
<automated>test "$(grep -c 'Anthropic Go SDK' .planning/REQUIREMENTS.md)" = "0" &amp;&amp; grep -q 'OpenAI-compatible adapters over the guarded client' .planning/REQUIREMENTS.md &amp;&amp; grep -A16 '### Phase 14:' .planning/ROADMAP.md | grep -q 'albums/import/discogs' &amp;&amp; grep -A16 '### Phase 14:' .planning/ROADMAP.md | grep -q 'sm-feedback-plugin' &amp;&amp; grep -q 'sm-golem-plugin' .planning/PROJECT.md &amp;&amp; grep -q 'sm-feedback-plugin' .planning/PROJECT.md</automated>
<fails_when>Non-zero exit: REQUIREMENTS still carries the SDK wording or lacks the adapters wording, the Phase 14 section lacks albums/import/discogs or sm-feedback-plugin, or PROJECT.md lacks sm-golem-plugin or sm-feedback-plugin.</fails_when>
</verify>
<acceptance_criteria>
- `grep -n 'API-08' .planning/REQUIREMENTS.md | head -1` shows the feedback-only text and no "sitemap output".
- The traceability rows for JOBS-02, JOBS-03, SRCH-02, INTG-01, INTG-02, API-08, CLI-05 still read `Phase 14 | Pending`.
- `grep -A16 '### Phase 14:' .planning/ROADMAP.md | grep -c 'recognize'` prints at least 1.
- `git show --name-only --format= <sha of the rewording commit>` lists exactly .planning/PROJECT.md, .planning/REQUIREMENTS.md and .planning/ROADMAP.md.
</acceptance_criteria>
<done>The planning docs describe the Phase 14 scope as decided, in a commit separate from code.</done>
</task>
</tasks>
<threat_model>
## Trust Boundaries
| Boundary | Description |
|----------|-------------|
| Application → vendor host | Credentials and user data leave through the guarded client |
| Configuration and request input → client policy | Must never select the trusted mode or the transport |
| Reference backend → recording proxy | Real credentials pass a local MITM proxy during recording |
| Recorded sidecar → git | Fixtures are committed |
| Log records → log sink | Errors and attrs may carry secrets |
## STRIDE Threat Register
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|-----------|----------|-----------|----------|-------------|-----------------|
| T-14-01 | Tampering | fetchguard.Client guarded modes | high | mitigate | https-only, AllowHosts check, dial-time reserved-IP Control on every connection, redirects never followed; TestClientModes, TestClientNeverFollowsRedirects, TestClientSchemeGuard (Tasks 1-2). |
| T-14-02 | Elevation of Privilege | trusted mode and transport override | high | mitigate | TrustedMode only in Go code; transport only through WithTransport's unexported context key; TestTransportSeamIsCodeOnly (Task 1). |
| T-14-03 | Information Disclosure | application logs | high | mitigate | sunscreen.Wrap installed first in every generated main; redacted keys and scrub patterns; surf 500 hides panic text; TestRedactHandler, TestScrub, TestRecoverHidesPanicDetails, TestGenerateMainInstallsRedactingLogger (Task 4). |
| T-14-04 | Information Disclosure | committed sidecars | high | mitigate | WriteUpstream masks var values and refuses unmasked Authorization/X-Api-Key; TestWriteUpstreamRefusesUnmaskedCredential (Task 3); check_corpus --check-secrets scans sidecars from 14-02. |
| T-14-05 | Spoofing | recording proxy and CA | medium | mitigate | Loopback listen only, CA key 0600 outside the fixtures tree, forward mode only by explicit flag; TestUpstreamProxyRefusesNonLoopback, TestEnsureParityCA, TestParityCommandContract (Task 3). |
| T-14-06 | Denial of Service | vendor response bodies | medium | mitigate | MaxBytes cap on Do and Send; TestClientBodyCap (Task 2). |
| T-14-07 | Tampering | beachcomber.DropIndex | medium | mitigate | Deletes only the named index and reports existence; TestEngineDropIndex (Task 4). |
| T-14-SC | Tampering | package installs | low | accept | No npm, pip, cargo or Go module is added (D-02; RESEARCH Package Legitimacy Audit lists none); crypto/x509 and log/slog are stdlib. |
</threat_model>
<verification>
- summercms.go: `go vet ./... && go test ./... -count=1` green; `go test ./cmd/summer -run '^(TestDocsTree|TestDocsCommandsMirrorGeneratedMain)$'` and `go run ./cmd/summer docs:build --check` green.
- fonoteka.go: `go -C ../fonoteka.go vet ./... && go -C ../fonoteka.go test ./... -count=1 -short` green with the regenerated main.go.
</verification>
<success_criteria>
- The guarded client, sidecar fake, recording proxy, redacting handler and IndexDropper exist with tests, READMEs and docs.
- Generated mains install the redacting handler.
- REQUIREMENTS, ROADMAP and PROJECT carry D-06, D-07, D-13 and D-14 in a docs-only commit.
</success_criteria>
<output>
Create `.planning/phases/14-domain-jobs-and-external-integrations/14-01-SUMMARY.md` when done.
</output>