|
|
|
|
@@ -0,0 +1,269 @@
|
|
|
|
|
---
|
|
|
|
|
phase: 14-domain-jobs-and-external-integrations
|
|
|
|
|
plan: 01
|
|
|
|
|
subsystem: infra
|
|
|
|
|
tags: [fetchguard, tide, parity, slog, redaction, beachcomber, typesense, mitm-proxy, x509]
|
|
|
|
|
|
|
|
|
|
requires:
|
|
|
|
|
- phase: 11-jobs-realtime-and-search-infrastructure
|
|
|
|
|
provides: beachcomber engine contract and the Typesense engine
|
|
|
|
|
- phase: 12-p-ytarium-api-collections-and-albums
|
|
|
|
|
provides: fetchguard.Fetch (GET-only SSRF guard)
|
|
|
|
|
provides:
|
|
|
|
|
- fetchguard.Client (NewClient, Do, Send, Get, PostJSON, PutJSON, PostMultipart, Bearer, FormField, FormFile) with TrustedMode and the code-only WithTransport seam
|
|
|
|
|
- tide upstream sidecars (<fixture>.upstream.yaml), WriteUpstream masking, the asserting UpstreamFake
|
|
|
|
|
- summer parity:upstream loopback HTTPS recording proxy (EnsureParityCA, script and forward modes)
|
|
|
|
|
- sunscreen redacting slog handler, installed first by every generated main
|
|
|
|
|
- beachcomber IndexDropper/DropIndex and IndexEnsurer/EnsureIndex (Typesense implements both)
|
|
|
|
|
- REQUIREMENTS/ROADMAP/PROJECT reworded per D-06, D-07, D-13, D-14
|
|
|
|
|
affects: [14-02 discogs, 14-03 jobs and reindex, 14-04 sm-golem-plugin, 14-05 sm-feedback-plugin, 14-06 tests]
|
|
|
|
|
|
|
|
|
|
actuals:
|
|
|
|
|
tokens: 49200
|
|
|
|
|
tasks: 5
|
|
|
|
|
commits: 6
|
|
|
|
|
plan_head_before: 06d6fdc758c57528ae4f9fc3a4bdfbd1cacf9dfa
|
|
|
|
|
plan_head_after: 7700c6da28cf98db6ff54a0965a2a4b9f195e984
|
|
|
|
|
|
|
|
|
|
tech-stack:
|
|
|
|
|
added: []
|
|
|
|
|
patterns:
|
|
|
|
|
- "Vendor HTTP goes through one fetchguard.Client per vendor; tests hand it tide.NewUpstreamFake via fetchguard.WithTransport"
|
|
|
|
|
- "Upstream sidecars are recorded through summer parity:upstream and never hold a live credential"
|
|
|
|
|
- "Generated mains call sunscreen.InstallDefault(os.Stderr) before compass.Load"
|
|
|
|
|
|
|
|
|
|
key-files:
|
|
|
|
|
created:
|
|
|
|
|
- modules/fetchguard/client.go
|
|
|
|
|
- modules/fetchguard/client_test.go
|
|
|
|
|
- modules/fetchguard/client_internal_test.go
|
|
|
|
|
- modules/tide/upstream.go
|
|
|
|
|
- modules/tide/upstream_test.go
|
|
|
|
|
- modules/tide/upstream_proxy.go
|
|
|
|
|
- modules/tide/upstream_proxy_test.go
|
|
|
|
|
- modules/tide/testdata/upstream/post_json.upstream.yaml
|
|
|
|
|
- modules/sunscreen/sunscreen.go
|
|
|
|
|
- modules/sunscreen/sunscreen_test.go
|
|
|
|
|
- modules/sunscreen/example_test.go
|
|
|
|
|
- modules/sunscreen/README.md
|
|
|
|
|
- modules/surf/recover_redaction_test.go
|
|
|
|
|
- docs/services/logging.md
|
|
|
|
|
modified:
|
|
|
|
|
- modules/fetchguard/fetch.go
|
|
|
|
|
- modules/fetchguard/policy.go
|
|
|
|
|
- modules/fetchguard/README.md
|
|
|
|
|
- modules/tide/README.md
|
|
|
|
|
- modules/beachcomber/searchable.go
|
|
|
|
|
- modules/beachcomber/typesense/engine.go
|
|
|
|
|
- modules/beachcomber/README.md
|
|
|
|
|
- internal/build/build.go
|
|
|
|
|
- examples/hello/main.go
|
|
|
|
|
- cmd/summer/parity.go
|
|
|
|
|
- cmd/summer/main.go
|
|
|
|
|
- README.md
|
|
|
|
|
- docs/services/outbound-http.md
|
|
|
|
|
- docs/services/parity-testing.md
|
|
|
|
|
- docs/services/search.md
|
|
|
|
|
- docs/console/utilities.md
|
|
|
|
|
- docs/setup/coming-from-wintercms.md
|
|
|
|
|
- docs/architecture/introduction.md
|
|
|
|
|
- ../fonoteka.go/main.go
|
|
|
|
|
- .planning/REQUIREMENTS.md
|
|
|
|
|
- .planning/ROADMAP.md
|
|
|
|
|
- .planning/PROJECT.md
|
|
|
|
|
|
|
|
|
|
key-decisions:
|
|
|
|
|
- "fetchguard.Fetch ignores TrustedMode and keeps guarding as PublicOnlyMode; only Client honours it"
|
|
|
|
|
- "Multipart sidecar parts keep plain field values (masked) and only file parts as sha256, so a credential is never committed even as a hash"
|
|
|
|
|
- "Credential headers count as masked only when they end in a placeholder and the residue is a scheme word, optionally with one key= label (Bearer {{x}}, Discogs token={{x}})"
|
|
|
|
|
- "UpstreamProxy.Flush refuses to write when any request failed (unscripted 599 or a forward error)"
|
|
|
|
|
- "sunscreen also redacts dashed sk- keys (sk-ant-...), which the PHP pattern misses"
|
|
|
|
|
- "requirements.mark-complete not run: INTG-01, INTG-02, API-08 and SRCH-02 stay Pending until 14-02..14-06 deliver them (Task 5 acceptance pins Pending)"
|
|
|
|
|
|
|
|
|
|
patterns-established:
|
|
|
|
|
- "Code-only test seam: an unexported context key set by an exported With* helper, proven unreachable from Policy/Client fields by reflection"
|
|
|
|
|
- "Optional engine capability interface plus a package-level helper with a fallback (PageSearcher/SearchPage, IndexDropper/DropIndex, IndexEnsurer/EnsureIndex)"
|
|
|
|
|
|
|
|
|
|
requirements-completed: []
|
|
|
|
|
|
|
|
|
|
coverage:
|
|
|
|
|
- id: D1
|
|
|
|
|
description: "Guarded outbound client: any method, JSON and multipart bodies, bearer, capped body, never follows redirects, TrustedMode only from code"
|
|
|
|
|
requirement: INTG-02
|
|
|
|
|
verification:
|
|
|
|
|
- kind: unit
|
|
|
|
|
ref: "modules/fetchguard/client_test.go#TestClientModes, TestClientSchemeGuard, TestClientMultipart, TestClientBodyCap, TestClientPutJSONHeaderOrder, TestTransportSeamIsCodeOnly"
|
|
|
|
|
status: pass
|
|
|
|
|
- kind: unit
|
|
|
|
|
ref: "modules/fetchguard/client_internal_test.go#TestClientNeverFollowsRedirects"
|
|
|
|
|
status: pass
|
|
|
|
|
human_judgment: false
|
|
|
|
|
- id: D2
|
|
|
|
|
description: "Upstream sidecars replay offline and assert the request Go sends"
|
|
|
|
|
requirement: INTG-01
|
|
|
|
|
verification:
|
|
|
|
|
- kind: unit
|
|
|
|
|
ref: "modules/fetchguard/client_test.go#TestClientPostJSONThroughUpstreamFake; modules/tide/upstream_test.go#TestUpstreamFakeRejectsMismatchedRequest, TestUpstreamFakeHashesBase64Bodies"
|
|
|
|
|
status: pass
|
|
|
|
|
human_judgment: false
|
|
|
|
|
- id: D3
|
|
|
|
|
description: "summer parity:upstream records masked sidecars through a loopback MITM proxy with a 0600 local CA key"
|
|
|
|
|
requirement: INTG-01
|
|
|
|
|
verification:
|
|
|
|
|
- kind: unit
|
|
|
|
|
ref: "modules/tide/upstream_proxy_test.go#TestUpstreamProxyScriptMode, TestUpstreamProxyRefusesNonLoopback, TestEnsureParityCA, TestUpstreamProxyMultipartAndForwardGuard; modules/tide/upstream_test.go#TestWriteUpstreamRefusesUnmaskedCredential"
|
|
|
|
|
status: pass
|
|
|
|
|
- kind: unit
|
|
|
|
|
ref: "cmd/summer#TestToolCommandNames, TestParityCommandContract"
|
|
|
|
|
status: pass
|
|
|
|
|
human_judgment: false
|
|
|
|
|
- id: D4
|
|
|
|
|
description: "Credential-redacting default logger in every generated main; surf 500 echoes no panic text"
|
|
|
|
|
verification:
|
|
|
|
|
- kind: unit
|
|
|
|
|
ref: "modules/sunscreen#TestRedactHandler, TestScrub, TestInstallDefault, ExampleWrap; internal/build#TestGenerateMainInstallsRedactingLogger; modules/surf#TestRecoverHidesPanicDetails"
|
|
|
|
|
status: pass
|
|
|
|
|
human_judgment: false
|
|
|
|
|
- id: D5
|
|
|
|
|
description: "beachcomber DropIndex reports already-absent indexes; EnsureIndex creates an empty index"
|
|
|
|
|
requirement: SRCH-02
|
|
|
|
|
verification:
|
|
|
|
|
- kind: unit
|
|
|
|
|
ref: "modules/beachcomber#TestDropIndex; modules/beachcomber/typesense#TestEngineDropIndex, TestEngineEnsureIndex"
|
|
|
|
|
status: pass
|
|
|
|
|
human_judgment: false
|
|
|
|
|
- id: D6
|
|
|
|
|
description: "Planning docs reworded per D-06, D-07, D-13, D-14 in a docs-only commit"
|
|
|
|
|
requirement: API-08
|
|
|
|
|
verification:
|
|
|
|
|
- kind: other
|
|
|
|
|
ref: "git show --name-only 7700c6d lists exactly PROJECT.md, REQUIREMENTS.md, ROADMAP.md; Task 5 grep checks"
|
|
|
|
|
status: pass
|
|
|
|
|
human_judgment: false
|
|
|
|
|
|
|
|
|
|
duration: 25min
|
|
|
|
|
completed: 2026-10-03
|
|
|
|
|
status: complete
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# Phase 14 Plan 01: Framework helpers for vendor calls Summary
|
|
|
|
|
|
|
|
|
|
**fetchguard grows a guarded vendor client with a code-only replay seam, tide records and replays upstream vendor exchanges through a loopback MITM proxy, sunscreen redacts credentials from every application log, and beachcomber can tell a dropped index from an absent one.**
|
|
|
|
|
|
|
|
|
|
## Performance
|
|
|
|
|
|
|
|
|
|
- **Duration:** 25 min
|
|
|
|
|
- **Started:** 2026-10-03T17:37:53Z
|
|
|
|
|
- **Completed:** 2026-10-03T18:02:30Z
|
|
|
|
|
- **Tasks:** 5
|
|
|
|
|
- **Files modified:** 41 in summercms.go, 1 in fonoteka.go
|
|
|
|
|
|
|
|
|
|
## Accomplishments
|
|
|
|
|
|
|
|
|
|
- `fetchguard.NewClient` sends any method with JSON or multipart bodies and bearer headers, caps the response, returns status and headers unjudged and never follows a redirect. `TrustedMode` (declared after `PublicOnlyMode`) lifts the scheme, host and dial checks for operator endpoints, and only Go code can choose it. `WithTransport` is the test seam and lives only in a context value.
|
|
|
|
|
- tide sidecars (`<fixture>.upstream.yaml`) hold the vendor exchanges of a fixture. `UpstreamFake` replays them without dialing and asserts method, scheme, host, path, query, compared headers, and the body (JSON semantically, multipart part by part, base64 photos by sha256). `Verify` reports mismatches, extra requests and unconsumed exchanges, and never prints credential values.
|
|
|
|
|
- `summer parity:upstream` runs a loopback CONNECT proxy that terminates TLS with a local ECDSA parity CA (key mode 0600, outside the fixtures tree). It answers from a script or forwards once through a PublicOnlyMode client, then writes the masked sidecar. An unmasked Authorization or X-Api-Key refuses the write.
|
|
|
|
|
- New `sunscreen` module: `Wrap` redacts the RedactCredentialsTap keys and scrubs Bearer, `sk-` and `x-api-key:` shapes. Every generated main now calls `sunscreen.InstallDefault(os.Stderr)` first; the example and application mains were regenerated with it.
|
|
|
|
|
- beachcomber `DropIndex` and `EnsureIndex` (optional `IndexDropper` and `IndexEnsurer`), implemented by Typesense.
|
|
|
|
|
- REQUIREMENTS, ROADMAP and PROJECT now describe the Phase 14 scope as decided.
|
|
|
|
|
|
|
|
|
|
## Task Commits
|
|
|
|
|
|
|
|
|
|
1. **Task 1: guarded JSON POST answered by a tide upstream fake (tracer)**: `93b7142` (feat)
|
|
|
|
|
2. **Task 2: PUT, multipart, bearer and trusted mode, plus docs**: `e6a6713` (feat)
|
|
|
|
|
3. **Task 3: summer parity:upstream recording proxy and offline replay**: `ee0004f` (feat)
|
|
|
|
|
4. **Task 4: sunscreen handler, generated mains, surf redaction test**: `7241704` (feat); **beachcomber DropIndex/EnsureIndex**: `58e6324` (feat); **fonoteka.go regenerated main**: `5738e82` (chore, fonoteka.go repo)
|
|
|
|
|
5. **Task 5: planning docs reworded**: `7700c6d` (docs)
|
|
|
|
|
|
|
|
|
|
The tracer feedback gate after Task 1 re-ran `<verify>` end to end. It passed, so the expansion tasks went ahead.
|
|
|
|
|
|
|
|
|
|
## Files Created/Modified
|
|
|
|
|
|
|
|
|
|
- `modules/fetchguard/client.go`: Client, the helpers, WithTransport, and the capped body reader
|
|
|
|
|
- `modules/fetchguard/fetch.go`, `policy.go`: shared transport builder, `Result.Header`, `TrustedMode`
|
|
|
|
|
- `modules/tide/upstream.go`: sidecar types, LoadUpstream/WriteUpstream, the asserting UpstreamFake
|
|
|
|
|
- `modules/tide/upstream_proxy.go`: the recording proxy and EnsureParityCA
|
|
|
|
|
- `modules/sunscreen/*`: the new module and its README
|
|
|
|
|
- `modules/beachcomber/searchable.go`, `typesense/engine.go`: index lifecycle helpers
|
|
|
|
|
- `internal/build/build.go`: the generated main installs sunscreen
|
|
|
|
|
- `cmd/summer/parity.go`, `main.go`: the `parity:upstream` command
|
|
|
|
|
- Docs: `docs/services/logging.md` (new), plus updates to outbound-http, parity-testing, search, console utilities, coming-from-wintercms and the architecture introduction
|
|
|
|
|
|
|
|
|
|
## Decisions Made
|
|
|
|
|
|
|
|
|
|
See `key-decisions` in the frontmatter. The main ones:
|
|
|
|
|
- `Fetch` stays guarded even if given TrustedMode, so the one-shot user-URL fetcher can never be unguarded.
|
|
|
|
|
- Multipart sidecars keep plain field values (masked) and hash only file parts.
|
|
|
|
|
- Flush writes nothing when any request failed.
|
|
|
|
|
- Requirements are not marked complete, because this plan only lays the foundation for them.
|
|
|
|
|
|
|
|
|
|
## Deviations from Plan
|
|
|
|
|
|
|
|
|
|
### Auto-fixed Issues
|
|
|
|
|
|
|
|
|
|
**1. [Rule 2 - Security] sunscreen also redacts dashed `sk-` keys**
|
|
|
|
|
- **Found during:** Task 4
|
|
|
|
|
- **Issue:** PHP's pattern `sk-[A-Za-z0-9]{20,}` misses Anthropic keys (`sk-ant-api03-...`), and Phase 14 sends exactly those keys out.
|
|
|
|
|
- **Fix:** The pattern is now `sk-[A-Za-z0-9_\-]{20,}`. This redacts more than PHP and is invisible to parity.
|
|
|
|
|
- **Files modified:** modules/sunscreen/sunscreen.go
|
|
|
|
|
- **Commit:** 7241704
|
|
|
|
|
|
|
|
|
|
**2. [Rule 2 - Security] Upstream mismatch messages never print credential values**
|
|
|
|
|
- **Found during:** Task 1
|
|
|
|
|
- **Issue:** An Authorization or X-Api-Key mismatch error would echo the live token into CI logs.
|
|
|
|
|
- **Fix:** These two headers report "value differs". Request URLs in error messages drop the query, and long body diffs are clipped.
|
|
|
|
|
- **Commit:** 93b7142
|
|
|
|
|
|
|
|
|
|
**3. [Rule 2 - Missing functionality] `UpstreamPart` type and `MaxUpstreamBody` (32 MiB)**
|
|
|
|
|
- **Found during:** Task 3
|
|
|
|
|
- **Issue:** Recording multipart bodies as ordered parts needs an exported part type that the plan's artifact list did not name. Vision requests also carry base64 photos of up to about 14 MB, which exceeds tide's 8 MiB `DefaultMaxBody`.
|
|
|
|
|
- **Fix:** Added `tide.UpstreamPart` and `tide.MaxUpstreamBody`, both documented in the README.
|
|
|
|
|
- **Commit:** ee0004f
|
|
|
|
|
|
|
|
|
|
**4. [Rule 3 - Blocking] Client tests live in an external test package plus one internal file**
|
|
|
|
|
- **Issue:** tide imports fetchguard for the proxy's forward mode, so an internal fetchguard test that imports tide would be an import cycle.
|
|
|
|
|
- **Fix:** `client_test.go` is `package fetchguard_test`. The redirect test, which needs the unexported loopback hooks, is in `client_internal_test.go`.
|
|
|
|
|
- **Commit:** 93b7142, e6a6713
|
|
|
|
|
|
|
|
|
|
**5. [CLAUDE.md - one logical change per commit] Task 4 split into two summercms.go commits**
|
|
|
|
|
- **Fix:** sunscreen (7241704) and beachcomber (58e6324) are committed separately. The fonoteka.go main is committed alone in its own repo (5738e82).
|
|
|
|
|
|
|
|
|
|
**Extra tests beyond the plan:** TestLoadUpstreamAndPath, TestClientPutJSONHeaderOrder, TestNewClientNegativeLimitsUseDefaults, TestUpstreamProxyUnscriptedRequestFailsFlush, TestUpstreamProxyMultipartAndForwardGuard, TestInstallDefault.
|
|
|
|
|
|
|
|
|
|
**Not changed:** `docs/architecture/introduction.md` has no fetchguard wording in its HTTP row, only a link, so the only edit there is adding sunscreen to the Services row.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
**Total deviations:** 5 (4 Rule 2/3 auto-fixes, 1 CLAUDE.md commit split). **Impact:** security hardening and test layout only, no scope creep.
|
|
|
|
|
|
|
|
|
|
## Issues Encountered
|
|
|
|
|
|
|
|
|
|
- The local environment sets `FORCE_COLOR=3`, which fails two existing bonfire non-TTY colour tests. The /tmp tmpfs also hit its disk quota while linking test binaries. Both are environmental and unrelated to this plan. The full suite is green with `FORCE_COLOR=` and `TMPDIR`/`GOTMPDIR` pointed at `~/.cache/gotest-tmp`.
|
|
|
|
|
|
|
|
|
|
## Verification
|
|
|
|
|
|
|
|
|
|
- summercms.go: `go vet ./...` and `go test ./... -count=1` are green. `go test ./cmd/summer -run '^(TestDocsTree|TestDocsCommandsMirrorGeneratedMain)$'` and `summer docs:build --check` pass ("no problems found").
|
|
|
|
|
- fonoteka.go: `go vet ./...`, `go build ./...` and `go test ./... -count=1 -short` are green with the regenerated main.go.
|
|
|
|
|
- Every per-task `<verify>` ran with `-race` where the plan asked. All named tests show `--- PASS`.
|
|
|
|
|
|
|
|
|
|
## Known Stubs
|
|
|
|
|
|
|
|
|
|
None.
|
|
|
|
|
|
|
|
|
|
## User Setup Required
|
|
|
|
|
|
|
|
|
|
None. No external service configuration required.
|
|
|
|
|
|
|
|
|
|
## Next Phase Readiness
|
|
|
|
|
|
|
|
|
|
- 14-02 (Discogs), 14-04 (golem adapters) and 14-05 (G15Office job) can build one `fetchguard.Client` per vendor and test it through `tide.NewUpstreamFake` plus `fetchguard.WithTransport`.
|
|
|
|
|
- Recording real PHP exchanges needs `php_parity.sh serve` extended with `HTTPS_PROXY=http://127.0.0.1:8425` and the parity CA as `curl.cainfo`/`openssl.cafile` (plan 14-02 scope). Assumption A1 (Guzzle/libcurl honour the env proxy) is still unverified against a live PHP run.
|
|
|
|
|
- 14-03 reindex can use `beachcomber.EnsureIndex` and `beachcomber.DropIndex`.
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
*Phase: 14-domain-jobs-and-external-integrations*
|
|
|
|
|
*Completed: 2026-10-03*
|
|
|
|
|
|
|
|
|
|
## Self-Check: PASSED
|
|
|
|
|
|
|
|
|
|
All created files exist; commits 93b7142, e6a6713, ee0004f, 7241704, 58e6324, 7700c6d (summercms.go) and 5738e82 (fonoteka.go) are present.
|
Jakub Zych