summercms/.planning/phases/14-domain-jobs-and-external-integrations/14-01-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, actuals, plan_head_before, plan_head_after, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, coverage, duration, completed, status

phase	plan	subsystem	tags	requires	provides	affects	actuals	plan_head_before	plan_head_after	tech-stack	key-files	key-decisions	patterns-established	requirements-completed	coverage	duration	completed	status
14-domain-jobs-and-external-integrations	01	infra	fetchguard tide parity slog redaction beachcomber typesense mitm-proxy x509	phase provides 11-jobs-realtime-and-search-infrastructure beachcomber engine contract and the Typesense engine phase provides 12-p-ytarium-api-collections-and-albums fetchguard.Fetch (GET-only SSRF guard)	fetchguard.Client (NewClient, Do, Send, Get, PostJSON, PutJSON, PostMultipart, Bearer, FormField, FormFile) with TrustedMode and the code-only WithTransport seam tide upstream sidecars (<fixture>.upstream.yaml), WriteUpstream masking, the asserting UpstreamFake summer parity:upstream loopback HTTPS recording proxy (EnsureParityCA, script and forward modes) sunscreen redacting slog handler, installed first by every generated main beachcomber IndexDropper/DropIndex and IndexEnsurer/EnsureIndex (Typesense implements both) REQUIREMENTS/ROADMAP/PROJECT reworded per D-06, D-07, D-13, D-14	14-02 discogs 14-03 jobs and reindex 14-04 sm-golem-plugin 14-05 sm-feedback-plugin 14-06 tests	tokens tasks commits 49200 5 6	06d6fdc758	7700c6da28	added patterns Vendor HTTP goes through one fetchguard.Client per vendor; tests hand it tide.NewUpstreamFake via fetchguard.WithTransport Upstream sidecars are recorded through summer parity:upstream and never hold a live credential Generated mains call sunscreen.InstallDefault(os.Stderr) before compass.Load	created modified modules/fetchguard/client.go modules/fetchguard/client_test.go modules/fetchguard/client_internal_test.go modules/tide/upstream.go modules/tide/upstream_test.go modules/tide/upstream_proxy.go modules/tide/upstream_proxy_test.go modules/tide/testdata/upstream/post_json.upstream.yaml modules/sunscreen/sunscreen.go modules/sunscreen/sunscreen_test.go modules/sunscreen/example_test.go modules/sunscreen/README.md modules/surf/recover_redaction_test.go docs/services/logging.md modules/fetchguard/fetch.go modules/fetchguard/policy.go modules/fetchguard/README.md modules/tide/README.md modules/beachcomber/searchable.go modules/beachcomber/typesense/engine.go modules/beachcomber/README.md internal/build/build.go examples/hello/main.go cmd/summer/parity.go cmd/summer/main.go README.md docs/services/outbound-http.md docs/services/parity-testing.md docs/services/search.md docs/console/utilities.md docs/setup/coming-from-wintercms.md docs/architecture/introduction.md ../fonoteka.go/main.go .planning/REQUIREMENTS.md .planning/ROADMAP.md .planning/PROJECT.md	fetchguard.Fetch ignores TrustedMode and keeps guarding as PublicOnlyMode; only Client honours it Multipart sidecar parts keep plain field values (masked) and only file parts as sha256, so a credential is never committed even as a hash Credential headers count as masked only when they end in a placeholder and the residue is a scheme word, optionally with one key= label (Bearer {{x}}, Discogs token={{x}}) UpstreamProxy.Flush refuses to write when any request failed (unscripted 599 or a forward error) sunscreen also redacts dashed sk- keys (sk-ant-...), which the PHP pattern misses requirements.mark-complete not run: INTG-01, INTG-02, API-08 and SRCH-02 stay Pending until 14-02..14-06 deliver them (Task 5 acceptance pins Pending)	Code-only test seam: an unexported context key set by an exported With* helper, proven unreachable from Policy/Client fields by reflection Optional engine capability interface plus a package-level helper with a fallback (PageSearcher/SearchPage, IndexDropper/DropIndex, IndexEnsurer/EnsureIndex)		id description requirement verification human_judgment D1 Guarded outbound client: any method, JSON and multipart bodies, bearer, capped body, never follows redirects, TrustedMode only from code INTG-02 kind ref status unit modules/fetchguard/client_test.go#TestClientModes, TestClientSchemeGuard, TestClientMultipart, TestClientBodyCap, TestClientPutJSONHeaderOrder, TestTransportSeamIsCodeOnly pass kind ref status unit modules/fetchguard/client_internal_test.go#TestClientNeverFollowsRedirects pass false id description requirement verification human_judgment D2 Upstream sidecars replay offline and assert the request Go sends INTG-01 kind ref status unit modules/fetchguard/client_test.go#TestClientPostJSONThroughUpstreamFake; modules/tide/upstream_test.go#TestUpstreamFakeRejectsMismatchedRequest, TestUpstreamFakeHashesBase64Bodies pass false id description requirement verification human_judgment D3 summer parity:upstream records masked sidecars through a loopback MITM proxy with a 0600 local CA key INTG-01 kind ref status unit modules/tide/upstream_proxy_test.go#TestUpstreamProxyScriptMode, TestUpstreamProxyRefusesNonLoopback, TestEnsureParityCA, TestUpstreamProxyMultipartAndForwardGuard; modules/tide/upstream_test.go#TestWriteUpstreamRefusesUnmaskedCredential pass kind ref status unit cmd/summer#TestToolCommandNames, TestParityCommandContract pass false id description verification human_judgment D4 Credential-redacting default logger in every generated main; surf 500 echoes no panic text kind ref status unit modules/sunscreen#TestRedactHandler, TestScrub, TestInstallDefault, ExampleWrap; internal/build#TestGenerateMainInstallsRedactingLogger; modules/surf#TestRecoverHidesPanicDetails pass false id description requirement verification human_judgment D5 beachcomber DropIndex reports already-absent indexes; EnsureIndex creates an empty index SRCH-02 kind ref status unit modules/beachcomber#TestDropIndex; modules/beachcomber/typesense#TestEngineDropIndex, TestEngineEnsureIndex pass false id description requirement verification human_judgment D6 Planning docs reworded per D-06, D-07, D-13, D-14 in a docs-only commit API-08 kind ref status other git show --name-only 7700c6d lists exactly PROJECT.md, REQUIREMENTS.md, ROADMAP.md; Task 5 grep checks pass false	25min	2026-10-03	complete

Phase 14 Plan 01: Framework helpers for vendor calls Summary

fetchguard grows a guarded vendor client with a code-only replay seam, tide records and replays upstream vendor exchanges through a loopback MITM proxy, sunscreen redacts credentials from every application log, and beachcomber can tell a dropped index from an absent one.

Performance

• Duration: 25 min
• Started: 2026-10-03T17:37:53Z
• Completed: 2026-10-03T18:02:30Z
• Tasks: 5
• Files modified: 41 in summercms.go, 1 in fonoteka.go

Accomplishments

• fetchguard.NewClient sends any method with JSON or multipart bodies and bearer headers, caps the response, returns status and headers unjudged and never follows a redirect. TrustedMode (declared after PublicOnlyMode) lifts the scheme, host and dial checks for operator endpoints, and only Go code can choose it. WithTransport is the test seam and lives only in a context value.
• tide sidecars (<fixture>.upstream.yaml) hold the vendor exchanges of a fixture. UpstreamFake replays them without dialing and asserts method, scheme, host, path, query, compared headers, and the body (JSON semantically, multipart part by part, base64 photos by sha256). Verify reports mismatches, extra requests and unconsumed exchanges, and never prints credential values.
• summer parity:upstream runs a loopback CONNECT proxy that terminates TLS with a local ECDSA parity CA (key mode 0600, outside the fixtures tree). It answers from a script or forwards once through a PublicOnlyMode client, then writes the masked sidecar. An unmasked Authorization or X-Api-Key refuses the write.
• New sunscreen module: Wrap redacts the RedactCredentialsTap keys and scrubs Bearer, sk- and x-api-key: shapes. Every generated main now calls sunscreen.InstallDefault(os.Stderr) first; the example and application mains were regenerated with it.
• beachcomber DropIndex and EnsureIndex (optional IndexDropper and IndexEnsurer), implemented by Typesense.
• REQUIREMENTS, ROADMAP and PROJECT now describe the Phase 14 scope as decided.

Task Commits

1. Task 1: guarded JSON POST answered by a tide upstream fake (tracer): 93b7142 (feat)
2. Task 2: PUT, multipart, bearer and trusted mode, plus docs: e6a6713 (feat)
3. Task 3: summer parity:upstream recording proxy and offline replay: ee0004f (feat)
4. Task 4: sunscreen handler, generated mains, surf redaction test: 7241704 (feat); beachcomber DropIndex/EnsureIndex: 58e6324 (feat); fonoteka.go regenerated main: 5738e82 (chore, fonoteka.go repo)
5. Task 5: planning docs reworded: 7700c6d (docs)

The tracer feedback gate after Task 1 re-ran <verify> end to end. It passed, so the expansion tasks went ahead.

Files Created/Modified

• modules/fetchguard/client.go: Client, the helpers, WithTransport, and the capped body reader
• modules/fetchguard/fetch.go, policy.go: shared transport builder, Result.Header, TrustedMode
• modules/tide/upstream.go: sidecar types, LoadUpstream/WriteUpstream, the asserting UpstreamFake
• modules/tide/upstream_proxy.go: the recording proxy and EnsureParityCA
• modules/sunscreen/*: the new module and its README
• modules/beachcomber/searchable.go, typesense/engine.go: index lifecycle helpers
• internal/build/build.go: the generated main installs sunscreen
• cmd/summer/parity.go, main.go: the parity:upstream command
• Docs: docs/services/logging.md (new), plus updates to outbound-http, parity-testing, search, console utilities, coming-from-wintercms and the architecture introduction

Decisions Made

See key-decisions in the frontmatter. The main ones:

• Fetch stays guarded even if given TrustedMode, so the one-shot user-URL fetcher can never be unguarded.
• Multipart sidecars keep plain field values (masked) and hash only file parts.
• Flush writes nothing when any request failed.
• Requirements are not marked complete, because this plan only lays the foundation for them.

Deviations from Plan

Auto-fixed Issues

1. [Rule 2 - Security] sunscreen also redacts dashed sk- keys

• Found during: Task 4
• Issue: PHP's pattern sk-[A-Za-z0-9]{20,} misses Anthropic keys (sk-ant-api03-...), and Phase 14 sends exactly those keys out.
• Fix: The pattern is now sk-[A-Za-z0-9_\-]{20,}. This redacts more than PHP and is invisible to parity.
• Files modified: modules/sunscreen/sunscreen.go
• Commit: 7241704

2. [Rule 2 - Security] Upstream mismatch messages never print credential values

• Found during: Task 1
• Issue: An Authorization or X-Api-Key mismatch error would echo the live token into CI logs.
• Fix: These two headers report "value differs". Request URLs in error messages drop the query, and long body diffs are clipped.
• Commit: 93b7142

3. [Rule 2 - Missing functionality] UpstreamPart type and MaxUpstreamBody (32 MiB)

• Found during: Task 3
• Issue: Recording multipart bodies as ordered parts needs an exported part type that the plan's artifact list did not name. Vision requests also carry base64 photos of up to about 14 MB, which exceeds tide's 8 MiB DefaultMaxBody.
• Fix: Added tide.UpstreamPart and tide.MaxUpstreamBody, both documented in the README.
• Commit: ee0004f

4. [Rule 3 - Blocking] Client tests live in an external test package plus one internal file

• Issue: tide imports fetchguard for the proxy's forward mode, so an internal fetchguard test that imports tide would be an import cycle.
• Fix: client_test.go is package fetchguard_test. The redirect test, which needs the unexported loopback hooks, is in client_internal_test.go.
• Commit: 93b7142, e6a6713

5. [CLAUDE.md - one logical change per commit] Task 4 split into two summercms.go commits

• Fix: sunscreen (7241704) and beachcomber (58e6324) are committed separately. The fonoteka.go main is committed alone in its own repo (5738e82).

Extra tests beyond the plan: TestLoadUpstreamAndPath, TestClientPutJSONHeaderOrder, TestNewClientNegativeLimitsUseDefaults, TestUpstreamProxyUnscriptedRequestFailsFlush, TestUpstreamProxyMultipartAndForwardGuard, TestInstallDefault.

Not changed: docs/architecture/introduction.md has no fetchguard wording in its HTTP row, only a link, so the only edit there is adding sunscreen to the Services row.

---

Total deviations: 5 (4 Rule 2/3 auto-fixes, 1 CLAUDE.md commit split). Impact: security hardening and test layout only, no scope creep.

Issues Encountered

• The local environment sets FORCE_COLOR=3, which fails two existing bonfire non-TTY colour tests. The /tmp tmpfs also hit its disk quota while linking test binaries. Both are environmental and unrelated to this plan. The full suite is green with FORCE_COLOR= and TMPDIR/GOTMPDIR pointed at ~/.cache/gotest-tmp.

Verification

• summercms.go: go vet ./... and go test ./... -count=1 are green. go test ./cmd/summer -run '^(TestDocsTree|TestDocsCommandsMirrorGeneratedMain)$' and summer docs:build --check pass ("no problems found").
• fonoteka.go: go vet ./..., go build ./... and go test ./... -count=1 -short are green with the regenerated main.go.
• Every per-task <verify> ran with -race where the plan asked. All named tests show --- PASS.

Known Stubs

None.

User Setup Required

None. No external service configuration required.

Next Phase Readiness

• 14-02 (Discogs), 14-04 (golem adapters) and 14-05 (G15Office job) can build one fetchguard.Client per vendor and test it through tide.NewUpstreamFake plus fetchguard.WithTransport.
• Recording real PHP exchanges needs php_parity.sh serve extended with HTTPS_PROXY=http://127.0.0.1:8425 and the parity CA as curl.cainfo/openssl.cafile (plan 14-02 scope). Assumption A1 (Guzzle/libcurl honour the env proxy) is still unverified against a live PHP run.
• 14-03 reindex can use beachcomber.EnsureIndex and beachcomber.DropIndex.

---

Phase: 14-domain-jobs-and-external-integrations Completed: 2026-10-03

Self-Check: PASSED

All created files exist; commits 93b7142, e6a6713, ee0004f, 7241704, 58e6324, 7700c6d (summercms.go) and 5738e82 (fonoteka.go) are present.