From d86f27cc9cb98c361d1d67ab9fb43f1be628cdd2 Mon Sep 17 00:00:00 2001 From: Jakub Zych Date: Thu, 15 Oct 2026 17:41:58 +0200 Subject: [PATCH] docs(19.3-01): complete cooler cache and Redis rate-limit store plan --- .../19.3-01-SUMMARY.md | 256 ++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 .planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/19.3-01-SUMMARY.md diff --git a/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/19.3-01-SUMMARY.md b/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/19.3-01-SUMMARY.md new file mode 100644 index 0000000..88658b4 --- /dev/null +++ b/.planning/phases/19.3-framework-transport-and-infrastructure-for-wavepath/19.3-01-SUMMARY.md @@ -0,0 +1,256 @@ +--- +phase: 19.3-framework-transport-and-infrastructure-for-wavepath +plan: 01 +subsystem: infra +tags: [cooler, redis, go-redis, otter, cache, rate-limit, surf] + +requires: + - phase: 19.3 (planning) + provides: CONTEXT D-06..D-09/D-19, RESEARCH Pattern 1/2 (boot wiring, Lua fixed-window script) +provides: + - "decision note .planning/notes/2026-10-15-cooler-go-redis-otter.md (go-redis v9.23.0, otter v2.3.0, miniredis test-only, s3blob)" + - "module cooler: Cache (memory/otter + Redis drivers), RateLimitStore (Redis surf.Store), Install, FromApp, OpenRedis" + - "serve installs cooler before BuildRouter; BuildRouter uses a published surf.Store" + - "config keys redis.url, cache.driver, cache.prefix, cache.memory.max_entries, http.rate_limit.store" +affects: [19.3-02, 19.3-03, 19.3-04, 19.3-05] + +actuals: + tokens: 17150 + tasks: 2 + commits: 3 +plan_head_before: 7357525b0b4f00560dc9f200c67827f8f16e0795 +plan_head_after: fd7f204979f7ee73decf1e8ceb29bc99310c34ec + +tech-stack: + added: ["github.com/redis/go-redis/v9 v9.23.0", "github.com/maypok86/otter/v2 v2.3.0"] + patterns: + - "Structural interface (cooler.Attempter == surf.Store method set) so surf imports cooler, never the reverse" + - "serve-only service install: publish on backpack before BuildRouter; route:list/tests keep defaults" + - "Secret-safe Redis URL handling: fixed error naming the key, logs only the endpoint" + +key-files: + created: + - .planning/notes/2026-10-15-cooler-go-redis-otter.md + - modules/cooler/cooler.go + - modules/cooler/redis.go + - modules/cooler/ratelimit.go + - modules/cooler/memory.go + - modules/cooler/rediscache.go + - modules/cooler/smoke_test.go + - modules/cooler/example_test.go + - modules/cooler/README.md + - modules/surf/store_lookup_test.go + - docs/services/caching.md + modified: + - go.mod + - go.sum + - modules/surf/serve.go + - modules/surf/router.go + - modules/surf/README.md + - README.md + - docs/services/rate-limiting.md + - docs/setup/configuration.md + - docs/architecture/performance-and-scaling.md + +key-decisions: + - "Memory cache driver is otter v2 (bounded MaximumSize); a stdlib map was rejected as unbounded" + - "An unreachable Redis at start-up is a warning; a missing/malformed redis.url with a Redis driver selected is a boot error naming the key" + - "Key prefix: cache.prefix, else slug(app.name)+':' (non [a-z0-9] runs -> '_', trimmed), else 'summer:'; cache keys cache:, limiter keys rl:" + - "Remember degrades: a failed read still runs the callback and skips the write; a failed write is skipped; warnings at most once per minute" + - "Cache and rate limiter share one Redis client when both select Redis; FromApp builds lazily without a ping" + - "Redis Flush escapes glob characters of the prefix before SCAN MATCH so it matches only this app's cache keys" + +patterns-established: + - "cooler.FromApp for commands outside serve (dedicated job worker, console commands)" + - "Docker-gated smoke tests with testcontainers.Run(redis:7-alpine) and a testing.Short() skip" + +requirements-completed: [SC-2, SC-4, D-06, D-07, D-08, D-09, D-19] + +coverage: + - id: D1 + description: "Decision note naming go-redis, otter, miniredis (test-only) and s3blob lands before the first go.mod change" + requirement: D-07 + verification: + - kind: other + ref: "git merge-base --is-ancestor " + status: pass + human_judgment: false + - id: D2 + description: "Two serve instances sharing one Redis count one throttle:2,1 window (200, 200, 429 with Retry-After)" + requirement: D-09 + verification: + - kind: integration + ref: "modules/surf/store_lookup_test.go#TestRateLimitAcrossInstancesSmoke (Docker, redis:7-alpine)" + status: pass + human_judgment: false + - id: D3 + description: "BuildRouter uses a published surf.Store, otherwise MemoryStore; selecting Redis without install never dials" + requirement: D-09 + verification: + - kind: unit + ref: "modules/surf/store_lookup_test.go#TestBuildRouterStoreSmoke" + status: pass + human_judgment: false + - id: D4 + description: "redis.url parse errors never contain the URL, user or password; rediss:// parses with TLS" + requirement: D-09 + verification: + - kind: unit + ref: "modules/cooler/smoke_test.go#TestOpenRedisRedactsSmoke" + status: pass + human_judgment: false + - id: D5 + description: "cooler.Cache memory driver round trip (Set/Get/Has/Increment/Remember once/Forget/Flush/1 ms TTL, copies, max_entries refusal)" + requirement: D-08 + verification: + - kind: unit + ref: "modules/cooler/smoke_test.go#TestMemoryCacheSmoke" + status: pass + human_judgment: false + - id: D6 + description: "cooler.Cache Redis driver round trip; Flush leaves rl: and foreign keys in place" + requirement: D-08 + verification: + - kind: integration + ref: "modules/cooler/smoke_test.go#TestRedisCacheSmoke (Docker, redis:7-alpine)" + status: pass + human_judgment: false + - id: D7 + description: "Redis outage: rate limiter falls back to per-instance memory with a rate-limited warning" + requirement: D-09 + verification: + - kind: manual_procedural + ref: "ad-hoc test against redis://127.0.0.1:1 (fallback admitted 2 attempts, ~500 ms each); full test in plan 05" + status: pass + human_judgment: true + rationale: "Only an ad-hoc run proved the outage path; plan 05 owns the committed miniredis/outage tests" + - id: D8 + description: "cooler README, root table row, surf README, caching/rate-limiting/configuration/scaling docs" + requirement: SC-4 + verification: + - kind: other + ref: "go test ./cmd/summer -run TestDocsTree; summer docs:build --check" + status: pass + human_judgment: false + +duration: 14min +completed: 2026-10-15 +status: complete +--- + +# Phase 19.3 Plan 01: cooler cache and Redis rate-limit store Summary + +**New `cooler` module: a Redis fixed-window `surf.Store` (atomic Lua script, per-instance memory fallback) that `serve` publishes before the router, plus `cooler.Cache` with an otter v2 memory driver and a go-redis v9 driver under a per-app key prefix** + +## Performance + +- **Duration:** 14 min +- **Started:** 2026-10-15T15:27:12Z +- **Completed:** 2026-10-15T15:41:00Z +- **Tasks:** 2 (Task 1 tracer, Task 2 expansion) +- **Files modified:** 20 + +## Accomplishments + +- Decision note committed first (77dd11c), ahead of the go.mod change (fa4d89d): go-redis v9.23.0, otter v2.3.0, miniredis v2.40.0 test-only (plan 05, after a human check), s3blob behind a build tag (plan 03). +- `cooler.RateLimitStore`: one Lua script per attempt with `surf.MemoryStore` semantics (first attempt opens the window, denied attempts do not increment, the window never extends), keys `rl:`, 500 ms bound, fallback to a per-instance `surf.MemoryStore` with a warning at most once per minute. +- `serve` calls `installCooler` after `publishUploads` and before `BuildRouter`; `BuildRouter` takes `app.Lookup[Store]()` or falls back to `NewMemoryStore(2*time.Minute)`. `route:list` is unchanged and never dials Redis. +- `cooler.Cache` (Get, Set, Has, Forget, Remember, Increment, Flush). The memory driver uses otter, bounded by `cache.memory.max_entries` (default 10000). The Redis driver uses SET PX, EXISTS, UNLINK and INCRBY. Its Flush runs SCAN MATCH `cache:*` in batches of 500 and removes each batch with UNLINK. It never calls FLUSHDB. +- `cooler.Install` validates `cache.driver` / `http.rate_limit.store`, shares one Redis client, pings once (warning on failure) and publishes the Cache; `cooler.FromApp` serves jobs and commands lazily. +- Docs: new `docs/services/caching.md` (order 45), the section "Sharing limits across instances" in `docs/services/rate-limiting.md`, configuration rows, the scaling note, the cooler README in the standard structure, surf README rows, and the root table row. + +## Task Commits + +1. **Decision note (Task 1, D-07 ordering)** - `77dd11c` (docs) +2. **Task 1: Tracer: Redis surf.Store shared by two serve instances** - `fa4d89d` (feat) +3. **Task 2: cooler.Cache with otter memory and Redis drivers** - `fd7f204` (feat) + +**Plan metadata:** see the final docs(19.3-01) commit. + +## Identifiers added + +- cooler: `Cache` (methods `Get`, `Set`, `Has`, `Forget`, `Remember`, `Increment`, `Flush`), `Attempter`, `Handle` (`Close`, `Cache`, `RateLimitRedis`, `RateLimitStore`), `Install`, `FromApp`, `OpenRedis`, `NewMemoryCache`, `NewRedisCache`, `RateLimitStore` / `NewRateLimitStore` / `Attempt`, constants `RedisURLKey`, `CacheDriverKey`, `CachePrefixKey`, `CacheMaxEntriesKey`, `RateLimitStoreKey`, `DefaultMaxEntries`, `PingTimeout`, `AttemptTimeout`. +- surf: unexported `installCooler(ctx, app) (*cooler.Handle, error)`; `BuildRouter` store lookup. +- Tests: `TestBuildRouterStoreSmoke`, `TestRateLimitAcrossInstancesSmoke`, `TestOpenRedisRedactsSmoke`, `TestMemoryCacheSmoke`, `TestRedisCacheSmoke`, `ExampleFromApp`. + +## Docker smoke tests + +All run on this host (Docker daemon answered): `TestRateLimitAcrossInstancesSmoke` PASS, `TestRedisCacheSmoke` PASS (redis:7-alpine via testcontainers). The non-short `go test ./modules/cooler/... ./modules/surf/...` run is green. + +## Files Created/Modified + +- `.planning/notes/2026-10-15-cooler-go-redis-otter.md`: dependency decision note. +- `modules/cooler/cooler.go`: config keys, the Cache interface, Handle, Install, FromApp, the prefix helper, the shared warner and the Remember helper. +- `modules/cooler/redis.go`: OpenRedis (secret-safe parse errors, handshakes disabled, no dial). +- `modules/cooler/ratelimit.go`: the Lua fixed-window RateLimitStore with its fallback. +- `modules/cooler/memory.go`: the otter v2 memory driver. +- `modules/cooler/rediscache.go`: the Redis driver with a prefix-scoped Flush. +- `modules/cooler/smoke_test.go`, `modules/cooler/example_test.go`: smoke tests and the docs example. +- `modules/surf/serve.go`, `modules/surf/router.go`: `installCooler` and the store lookup. +- `modules/surf/store_lookup_test.go`: the surf smoke tests. +- READMEs and docs listed in key-files. + +## Decisions Made + +See key-decisions in the frontmatter. All are from the plan's "Claude's discretion" list, except the glob escaping in Flush and the trimmed slug, which are listed under deviations. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 3 - Blocking] The docs checker refuses Go code blocks without a `src=` reference** +- **Found during:** Task 2 (docs/services/caching.md) +- **Issue:** `TestDocsTree` failed with "go code block has no src= reference" on the caching guide's inline example. +- **Fix:** Added `modules/cooler/example_test.go` (`ExampleFromApp`, a verified Output) and pointed the guide at it with `src=modules/cooler/example_test.go#ExampleFromApp`, then ran `summer docs:sync`. +- **Files modified:** modules/cooler/example_test.go, docs/services/caching.md +- **Verification:** TestDocsTree and `docs:build --check` are green. +- **Committed in:** fd7f204 + +**2. [Rule 2 - Missing critical] SCAN MATCH treats the prefix as a pattern** +- **Found during:** Task 2 (rediscache.go Flush) +- **Issue:** A `cache.prefix` that contains `*`, `?`, `[` or `]` would let Flush match, and UNLINK, keys of other prefixes. +- **Fix:** `escapeGlob` escapes those characters before `SCAN MATCH cache:*`. +- **Files modified:** modules/cooler/rediscache.go +- **Verification:** TestRedisCacheSmoke (a foreign key and the `rl:` key survive Flush). +- **Committed in:** fd7f204 + +**3. [Rule 1 - Acceptance gate] The rate-limiting guide did not name `http.rate_limit.store`** +- **Found during:** Task 1 acceptance criteria (one grep line per file was required). +- **Issue:** The guide showed only the YAML form (`rate_limit: store: redis`). +- **Fix:** The guide now names the dotted key and its default. The fix was amended into the unpushed Task 1 commit. +- **Committed in:** fa4d89d + +**4. [Discretion refinement] Prefix slug and helper visibility** +- The slug of `app.name` also trims `_` at both ends (`Acme!` gives `acme:`, not `acme_:`). This is documented in the README and the caching guide. +- The prefix helper is unexported (`keyPrefix`) because the plan's artifact list names no exported prefix function. + +**5. [Interpretation] Remember after a failed read** +- When the read fails, Remember runs the callback and returns its value without attempting the write, because the backend that just failed would very likely fail the write too. When the read succeeds with a miss and the write then fails, the failed write is skipped with a warning. Both cases match the plan's degrade rule. + +--- + +**Total deviations:** 3 auto-fixed (1 blocking, 1 missing critical, 1 acceptance gate) plus 2 documented discretion choices. +**Impact on plan:** No scope creep. The identifiers, keys and behaviour match the plan. + +## Issues Encountered + +- During a Redis outage each `Attempt` waits up to `AttemptTimeout` (500 ms), because go-redis retries its dial within that bound before cooler falls back. go-redis also prints its own `connection pool: failed to dial` line for every failure, independent of cooler's once-a-minute warning. Both are within the plan (T-19.3-02 accepts a 500 ms bound). Plan 05 or a follow-up could add a short circuit-breaker and fewer dial retries if outage latency matters. + +## Known Stubs + +None. + +## User Setup Required + +None. Redis is opt-in through `http.rate_limit.store: redis` / `cache.driver: redis` and `redis.url`. + +## Next Phase Readiness + +- Plans 02, 03 and 04 can build on this commit. Plan 03 relies on the s3blob entry in the note. Plan 04 adds the site-cache purge to cooler. +- Plan 05 owns full unit coverage: a miniredis parity table for RateLimitStore against MemoryStore, the outage fallback, the Install/FromApp edge cases and the Flush glob escaping. + +--- +*Phase: 19.3-framework-transport-and-infrastructure-for-wavepath* +*Completed: 2026-10-15* + +## Self-Check: PASSED