docs(19.1): add validation strategy
This commit is contained in:
Jakub Zych
102
.planning/phases/19.1-chat-satellites/19.1-VALIDATION.md
Normal file
102
.planning/phases/19.1-chat-satellites/19.1-VALIDATION.md
Normal file
@@ -0,0 +1,102 @@
|
||||
---
|
||||
phase: "19.1"
|
||||
slug: "chat-satellites"
|
||||
# status lifecycle: draft (seeded by plan-phase) → validated (set by validate-phase §6)
|
||||
# audit-milestone §5.5 distinguishes NOT-VALIDATED (draft) from PARTIAL (validated + nyquist_compliant: false) (#2117)
|
||||
status: draft
|
||||
nyquist_compliant: false
|
||||
wave_0_complete: false
|
||||
created: "2026-10-17"
|
||||
---
|
||||
|
||||
# Phase 19.1 — Validation Strategy
|
||||
|
||||
> Per-phase validation contract for feedback sampling during execution. Source: `19.1-RESEARCH.md` § Validation Architecture.
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Framework** | Go `testing` + testcontainers-go v0.44.0 (Postgres); Docker tests skip under `-short` |
|
||||
| **Config file** | none at framework root — each plugin is its own module |
|
||||
| **Quick run command** | `go vet ./... && go test -short ./...` inside each touched plugin repo |
|
||||
| **Full suite command** | `go test ./...` (testcontainers Postgres + lighthouse memory + LiveKit Twirp fake + push HTTP fake) |
|
||||
| **Phase gate** | `scripts/check-phase19.1.sh` (create in last plan) requiring named tests across the five modules + the three decision-note paths |
|
||||
| **Estimated runtime** | quick ~30 s per repo; full Docker suite ~5–10 min per repo |
|
||||
|
||||
---
|
||||
|
||||
## Sampling Rate
|
||||
|
||||
- **After every task commit:** `go vet ./...` and `go test -short ./...` in **each** plugin module the task modified. A decision-note-only commit: `git grep` the locked import path and version in the note; no `go test`.
|
||||
- **After every plan wave:** `go test ./...` in every plugin the wave touched (Docker).
|
||||
- **Before `/msd-verify-work`:** `scripts/check-phase19.1.sh` green
|
||||
- **Max feedback latency:** 60 seconds for the quick run
|
||||
|
||||
Five repos means a task that finishes forum is not evidence for herald. Sample all 9 independent deliverables (5 plugins + 3 notes + 1 PHP map) at the phase gate; per-commit sample the subset in the diff.
|
||||
|
||||
---
|
||||
|
||||
## Per-Task Verification Map
|
||||
|
||||
Task IDs assume the research-recommended 6-plan split (confirm at the plan-count checkpoint). File Exists is Wave 0 until execute lands the tests.
|
||||
|
||||
| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status |
|
||||
|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------|
|
||||
| 19.1-01 | 01 | 1 | SC-19.1-1 | T-19.1-01 | community-join 10/min on friend request; IDOR 404 | integration | `go -C ../sm-userfriends-plugin test -count=1 -run TestHTTPTracer` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-01 | 01 | 1 | D-08 | — | festival group events; no chat hard-require | unit | `go -C ../sm-userfriends-plugin test -short -run TestNoChatHardRequire` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-02 | 02 | 2 | SC-19.1-1 | T-19.1-04 | XSS stripped; GFM table; ForumAccessResolver deny-all | integration | `go -C ../sm-forum-plugin test -count=1 -run TestHTTPTracer` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-02 | 02 | 2 | D-08 | — | forum does not import userfriends | unit | `go -C ../sm-forum-plugin test -short -run TestNoUserfriendsImport` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-03 | 03 | 3 | SC-19.1-2 | T-19.1-07 | no Golem import; actor from JWT (CMD-06) | unit | `go -C ../sm-chatbots-plugin test -short -run TestNoGolem` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-03 | 03 | 3 | D-05 | T-19.1-07 | /ping dispatch as JWT actor; help listed | integration | `go -C ../sm-chatbots-plugin test -count=1 -run TestHTTPTracer` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-04 | 04 | 4 | SC-19.1-2 | T-19.1-02 | webpush-go note names module+version before import | grep | `test -f .planning/notes/*webpush-go.md && grep -q SherClockHolmes/webpush-go .planning/notes/*webpush-go.md` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-04 | 04 | 4 | D-10 | T-19.1-02 | subscribe rejects http/private IP; PublicOnly send | unit | `go -C ../sm-herald-plugin test -short -run TestSubscribeEndpoint` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-04 | 04 | 4 | D-11 | T-19.1-02 | SendEvent enqueue in Tx; rollback sends 0 | integration | `go -C ../sm-herald-plugin test -count=1 -run TestSendEventRollback` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-05 | 05 | 5 | SC-19.1-2 | T-19.1-03 | LiveKit note names server-sdk-go/v2 + pion flag before import | grep | `test -f .planning/notes/*livekit*.md && grep -q livekit/server-sdk-go .planning/notes/*livekit*.md` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-05 | 05 | 5 | D-13 | T-19.1-03 | JWT claims golden vs PHP (HS256, sub=user id string, video grant) | unit | `go -C ../sm-chatvideo-plugin test -short -run TestLiveKitTokenClaims` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-05 | 05 | 5 | D-15 | T-19.1-05 | video:session:{id} participant allowed / stranger denied; prs | unit | `go -C ../sm-chatvideo-plugin test -short -run TestVideoAuthorizer` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-06 | 06 | 6 | SC-19.1-4 | — | PHP method maps + named tests in check-phase19.1.sh | unit | `bash scripts/check-phase19.1.sh` | ❌ W0 | ⬜ pending |
|
||||
| 19.1-06 | 06 | 6 | SC-19.1-1 | — | each baseline creates final PHP tables | unit | `go test -short ./updates -run TestBaseline` ×5 | ❌ W0 | ⬜ pending |
|
||||
|
||||
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
|
||||
|
||||
---
|
||||
|
||||
## Wave 0 Requirements
|
||||
|
||||
- [ ] Five repos: `go.mod` with `replace` to `../../../../summercms.go`, `plugin.go`, baseline, `harness_test.go` / `testmain_test.go` (`flag.Parse` + `testing.Short`)
|
||||
- [ ] `testdata/goldens/` per plugin (serializers, events, LiveKit JWT, forum HTML)
|
||||
- [ ] `acme` fixture plugin (test files only) where a context is needed (forum, userfriends adapter)
|
||||
- [ ] LiveKit `RoomAPI` + push `HTTPClient` fakes
|
||||
- [ ] Three decision notes (webpush-go, LiveKit, bluemonday) before the importing commits
|
||||
- [ ] D-17 markdown module (separate `/msd-quick` task) before forum execute
|
||||
- [ ] PHP→Go method map markdown in the last plan
|
||||
- [ ] `scripts/check-phase19.1.sh`
|
||||
|
||||
Existing fetchguard / lighthouse / seaglass / flare tests do **not** cover satellite behavior.
|
||||
|
||||
---
|
||||
|
||||
## Manual-Only Verifications
|
||||
|
||||
| Behavior | Requirement | Why Manual | Test Instructions |
|
||||
|----------|-------------|------------|-------------------|
|
||||
| Capture Wavepath serializer, event, LiveKit JWT, and forum HTML goldens | SC-19.1-1 | Needs a live PHP Wavepath backend | Record payloads from Wavepath; store under each plugin's `testdata/goldens/` |
|
||||
| Repo create / first push | Discretion | `tea repos create` and remote push need a human | `tea repos create --owner golem15 sm-<name>-plugin` then analog go.mod/replace; do not push until asked |
|
||||
| D-17 markdown module | D-17 | Separate `/msd-quick` in summercms.go | Forum plan blocks until `Render`/`Purify` exist; do not vendor goldmark/bluemonday in forum |
|
||||
| Live Wavepath MySQL schema dump | D-02 | This research session: `ERROR 1698` | Re-try with credentials at execute; until then schema SoT is PHP migrations |
|
||||
|
||||
---
|
||||
|
||||
## Validation Sign-Off
|
||||
|
||||
- [ ] All tasks have `<automated>` verify or Wave 0 dependencies
|
||||
- [ ] Sampling continuity: no 3 consecutive tasks without automated verify
|
||||
- [ ] Wave 0 covers all MISSING references
|
||||
- [ ] No watch-mode flags
|
||||
- [ ] Feedback latency < 60s
|
||||
- [ ] `nyquist_compliant: true` set in frontmatter
|
||||
|
||||
**Approval:** pending
|
||||
Reference in New Issue
Block a user