golem15/summercms
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(14.3-04): sign off the phase 14.3 security review and validation map

Browse Source 
- security review: T-14.3-20..26 added, every row pinned by a named test or gate stage
- validation: per-plan map with exact tests and commands, validated and nyquist_compliant
This commit is contained in:
Jakub Zych
2026-10-08 12:30:03 +02:00
parent 89f1ab2c44
commit d0af79b496
2 changed files with 82 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
.planning/phases/14.3-one-command-app-scaffolding-summer-new-and-summercms-io-new/14.3-SECURITY-REVIEW.md
View File

@@ -2,23 +2,26 @@
phase: 14.3
slug: one-command-app-scaffolding
status: reviewed
threats_total: 20
threats_closed: 20
threats_total: 27
threats_closed: 27
threats_open: 0
accepted_risks: 3
accepted_risks: 4
asvs_level: 1
created: 2026-10-08
reviewer: gsd-executor (14.3-02 Task 3, self-performed; see Reviewer Note)
reviewer: gsd-executor (14.3-02 Task 3 and 14.3-04 Task 3, self-performed; see Reviewer Note)
updated: 2026-10-08
---
# Phase 14.3 — Security Review (plans 01 and 02)
# Phase 14.3 — Security Review (plans 01 to 04)
> `summer new` (generator), the `curl | bash` bootstrap script in
> `modules/sunrise`, the site route `GET /new` and the D-20 acceptance harness.
> Every threat registered in 14.3-01-PLAN.md (T-14.3-01..10, SC) and
> 14.3-02-PLAN.md (T-14.3-11..19, SC) is mapped below to its source-level
> control and the evidence available now. Plan 04 adds rows T-14.3-20..26 and
> fills the "Pinned by (plan 04)" column with executed test names.
> 14.3-02-PLAN.md (T-14.3-11..19, SC), 14.3-03-PLAN.md (T-14.3-20..22) and
> 14.3-04-PLAN.md (T-14.3-23..26) is mapped below to its source-level control
> and evidence. The "Pinned by (plan 04)" column names the executed test or
> gate stage of `scripts/check-phase14.3.sh` that fails when the control
> breaks; `--security` checks this table.
**Date:** 2026-10-08
**Scope:** `git diff v0.1.12..HEAD -- internal/build cmd/summer modules/lagoon modules/sunrise scripts/check-phase14.3.sh` in `summercms.go` (15 files, +1520/-15), and the site plugin commit `2157bb9` (`script.go`, `plugin.go`, `routes_test.go`, `README.md`) in `sm-summercmsio-plugin`. Read alongside `../sm-summercmsio-app/deploy/nginx/summercms.io.conf`.
@@ -36,14 +39,14 @@ The plan names a security-review lane; this session had no separate auditor agen
## Verdict
20 threats in scope (T-14.3-01..19 and T-14.3-SC): 17 mitigated, 3 accepted (all low), **0 open**. No high finding; no code fix was needed.
27 threats in scope (T-14.3-01..26 and T-14.3-SC): 23 mitigated, 4 accepted (all low), **0 open**. No high finding; no code fix was needed. Every high row is pinned by a named test or gate stage (plan 04), and `bash scripts/check-phase14.3.sh --all` passed with these pins on 2026-10-08.
## ASVS L1
| ASVS | Applies | Control | Evidence |
|------|---------|---------|----------|
| V5 Input Validation | yes | `new.sh` `main`: `^[a-z][a-z0-9]*$` before any path use; `internal/build/new.go` `ValidateAppName`: same regex plus `token.IsKeyword`, `main`, `internal`; `--framework` and siblings accepted only when `go.mod` declares the framework module (`checkFramework`); `SUMMERCMS_REF` starting with `-` rejected (`target_commit`) | `TestNewAppSmoke`; refusal probe (bad name exit 2); plan 01 Task 2 rules check |
| V6 Cryptography | yes | `modules/lagoon/keygen.go` `GenerateAppKey` on `crypto/rand`, 32 bytes; `NewApp` `validateAppKey` rejects anything else | `TestGenerateAppKeyIs32ByteBase64`, `TestEncryptedKeyGeneratePrints32ByteBase64` |
| V6 Cryptography | yes | `modules/lagoon/keygen.go` `GenerateAppKey` on `crypto/rand`, 32 bytes; `NewApp` `validateAppKey` rejects anything else | `TestGenerateAppKey`, `TestEncryptedKeyGeneratePrints32ByteBase64`, `TestNewRequiresValidAppKey` |
| V10 Malicious Code | yes | Script embedded from the framework release (`sunrise.Script`); `curl -fsSL`; HTTPS default clone URL; release tags only; tags fetched without `--force`; ancestry check; no `eval`, no `sudo`, no dynamic code strings | `TestScriptTruncationRunsNothing`, `TestBootstrapEndToEnd`, refusal probe, `--acceptance` |
| V12 Files and Resources | yes | Script writes only `SUMMERCMS_DIR` (clone), `sm-<name>-app` (via `summer new`) and the Go install directory; parent must already exist; `NewApp` refuses an existing target (`os.Lstat`), creates with `os.Mkdir`, removes only what it created; `.env` 0600 | `TestScriptTruncationRunsNothing` (empty work dir), `--generator`, refusal probe |
| V14 Configuration | yes | Secrets only in `.env` (gitignored before `git add -A`); `.env.example` without a key; compose publishes `127.0.0.1:54320`; `/new` served `text/plain; charset=utf-8` with `X-Content-Type-Options: nosniff` | `--generator` (`git ls-files .env` empty, mode 600), `TestNewScriptRoute` |
@@ -52,29 +55,38 @@ The plan names a security-review lane; this session had no separate auditor agen
| Threat ID | Category | Component | Severity | Disposition | Control (file / function) | Evidence now | Status | Pinned by (plan 04) |
|-----------|----------|-----------|----------|-------------|---------------------------|--------------|--------|---------------------|
| T-14.3-01 | Tampering | app name, target path | high | mitigate | `internal/build/new.go` `ValidateAppName` before `filepath.Join(parent, appDirName(name))`; target is always `<parent>/sm-<name>-app` | `TestNewAppSmoke`; plan 01 rules check (RULES-OK) | mitigated | TestNewValidateAppName |
| T-14.3-02 | Information Disclosure | `.env`, first git commit | high | mitigate | `internal/build/new.go` `planNewApp` (`.env` mode 0600, `.gitignore` written with the files) and `initGit` (`git add -A` after `.gitignore` exists) | `scripts/check-phase14.3.sh --generator` asserts `git ls-files .env` empty and mode 600; acceptance run committed the app the same way | mitigated | TestNewEnvFileMode, TestNewGitFirstCommit |
| T-14.3-03 | Information Disclosure | app key generation | high | mitigate | `modules/lagoon/keygen.go` `GenerateAppKey` (`crypto/rand`); `internal/build/new.go` `validateAppKey` | `TestGenerateAppKeyIs32ByteBase64`; `--generator` (44-character key) | mitigated | TestGenerateAppKey, TestNewRequiresValidAppKey |
| T-14.3-04 | Tampering | existing directory, cleanup | high | mitigate | `internal/build/new.go` `NewApp`: `os.Lstat` refusal, `os.Mkdir` (not `MkdirAll`), `os.RemoveAll` only of the directory this run created; `new.sh` `main` also stops early when `sm-<name>-app` exists (`-e` or `-L`) | refusal probe: existing `sm-demo-app` stops the script before any clone, exit 1 | mitigated | TestNewRefusesExistingTarget, TestNewCleansUpOnFailure |
| T-14.3-05 | Information Disclosure | compose.yaml Postgres port | medium | mitigate | `internal/build/stubs/new.tmpl` compose: `127.0.0.1:54320:5432`, named volume only | `--generator` and `--acceptance` ran the compose file; template grep | mitigated | TestNewGolden |
| T-14.3-06 | Spoofing | framework auto-detection | medium | mitigate | `internal/build/new.go` `resolveFramework` / `checkFramework`: module path must be `git.golem15.com/golem15/summercms`; an explicit `--framework` never falls through; the script always passes `--framework` | `TestNewAppSmoke`; `--generator` (sibling detection) | mitigated | TestNewFrameworkResolution |
| T-14.3-07 | Information Disclosure | committed go.mod / go.work | low | mitigate | `internal/build/scaffold.go` `relModulePath`, relative `runGoWorkUse` | `--generator` (no absolute `use` line); `TestBootstrapEndToEnd` (relative `../summercms` replace) | mitigated | TestNewGolden |
| T-14.3-08 | Information Disclosure | compose.yaml dev credentials | low | accept | `summer`/`summer` is a documented local default for a loopback-only database; the encryption key stays in `.env` | template review | accepted | TestNewGolden |
| T-14.3-01 | Tampering | app name, target path | high | mitigate | `internal/build/new.go` `ValidateAppName` before `filepath.Join(parent, appDirName(name))`; target is always `<parent>/sm-<name>-app` | `TestNewAppSmoke`; plan 01 rules check (RULES-OK) | mitigated | TestNewValidateAppName, TestNewCommandViaCLI, TestScriptBehavior (name cases) |
| T-14.3-02 | Information Disclosure | `.env`, first git commit | high | mitigate | `internal/build/new.go` `planNewApp` (`.env` mode 0600, `.gitignore` written with the files) and `initGit` (`git add -A` after `.gitignore` exists) | `scripts/check-phase14.3.sh --generator` asserts `git ls-files .env` empty and mode 600; acceptance run committed the app the same way | mitigated | TestNewEnvFileMode, TestNewGitFirstCommit (.env untracked), TestNewGolden (dot-gitignore) |
| T-14.3-03 | Information Disclosure | app key generation | high | mitigate | `modules/lagoon/keygen.go` `GenerateAppKey` (`crypto/rand`); `internal/build/new.go` `validateAppKey` | `TestGenerateAppKey` (formerly `TestGenerateAppKeyIs32ByteBase64`); `--generator` (44-character key) | mitigated | TestGenerateAppKey, TestNewRequiresValidAppKey |
| T-14.3-04 | Tampering | existing directory, cleanup | high | mitigate | `internal/build/new.go` `NewApp`: `os.Lstat` refusal, `os.Mkdir` (not `MkdirAll`), `os.RemoveAll` only of the directory this run created; `new.sh` `main` also stops early when `sm-<name>-app` exists (`-e` or `-L`) | refusal probe: existing `sm-demo-app` stops the script before any clone, exit 1 | mitigated | TestNewRefusesExistingTarget, TestNewCleansUpOnFailure, TestNewRefusesTargetInsideFramework, TestScriptBehavior (existing app) |
| T-14.3-05 | Information Disclosure | compose.yaml Postgres port | medium | mitigate | `internal/build/stubs/new.tmpl` compose: `127.0.0.1:54320:5432`, named volume only | `--generator` and `--acceptance` ran the compose file; template grep | mitigated | TestNewGolden (compose.yaml) |
| T-14.3-06 | Spoofing | framework auto-detection | medium | mitigate | `internal/build/new.go` `resolveFramework` / `checkFramework`: module path must be `git.golem15.com/golem15/summercms`; an explicit `--framework` never falls through; the script always passes `--framework` | `TestNewAppSmoke`; `--generator` (sibling detection) | mitigated | TestNewFrameworkResolution, TestScriptBehavior (fresh clone passes --framework) |
| T-14.3-07 | Information Disclosure | committed go.mod / go.work | low | mitigate | `internal/build/scaffold.go` `relModulePath`, relative `runGoWorkUse` | `--generator` (no absolute `use` line); `TestBootstrapEndToEnd` (relative `../summercms` replace) | mitigated | TestNewGolden (go.mod, go.work), TestMakeAndAddPluginOnHelloCopy, --forbidden |
| T-14.3-08 | Information Disclosure | compose.yaml dev credentials | low | accept | `summer`/`summer` is a documented local default for a loopback-only database; the encryption key stays in `.env` | template review | accepted | TestNewGolden (compose.yaml) |
| T-14.3-09 | Information Disclosure | `summer serve` default `:8080` | low | accept | Existing surf default listens on all interfaces; next steps point at `127.0.0.1`; out of scope for this phase | review | accepted | — |
| T-14.3-10 | Repudiation | fallback git identity | low | accept | `internal/build/new.go` `initGit` / `gitIdentityMissing`: fallback only for the missing keys, first commit only, announced with a warning | `--generator` and `--acceptance` printed the warning | accepted | TestNewGitUsesConfiguredIdentity |
| T-14.3-10 | Repudiation | fallback git identity | low | accept | `internal/build/new.go` `initGit` / `gitIdentityMissing`: fallback only for the missing keys, first commit only, announced with a warning | `--generator` and `--acceptance` printed the warning | accepted | TestNewGitFirstCommit, TestNewGitUsesConfiguredIdentity |
| T-14.3-11 | Tampering | truncated download | high | mitigate | `modules/sunrise/new.sh`: top level holds only comments and function definitions; the last line is `main "$@"`; `main` parses arguments before any external command | `TestScriptTruncationRunsNothing` (every line-boundary prefix and every byte prefix of the final line, with stub `git`, `go`, `docker`, `curl`, `summer`; positive control proves the stubs record calls); `TestScriptShape` | mitigated | TestScriptTruncationRunsNothing, TestScriptShape |
| T-14.3-12 | Tampering | error body piped to bash | medium | mitigate | Documented one-liner uses `curl -fsSL` (no body on HTTP errors); `GET /new` answers only the embedded bytes (`script.go` `scriptHandler`); a cut-off body is covered by T-14.3-11 | `TestNewScriptRoute`; `TestBootstrapEndToEnd` runs the exact `curl -fsSL … \| bash -s -- demo` form | mitigated | TestScriptBehavior |
| T-14.3-12 | Tampering | error body piped to bash | medium | mitigate | Documented one-liner uses `curl -fsSL` (no body on HTTP errors); `GET /new` answers only the embedded bytes (`script.go` `scriptHandler`); a cut-off body is covered by T-14.3-11 | `TestNewScriptRoute`; `TestBootstrapEndToEnd` runs the exact `curl -fsSL … \| bash -s -- demo` form | mitigated | TestNewScriptRoute, TestBootstrapEndToEnd (--bootstrap) |
| T-14.3-13 | Spoofing | `/new` rendered as HTML | medium | mitigate | `sm-summercmsio-plugin` `script.go` `scriptHandler`: `Content-Type: text/plain; charset=utf-8`, `X-Content-Type-Options: nosniff` via `setCommonHeaders`; no request data reflected; nginx allows GET/HEAD only (`limit_except GET HEAD`) and gzips `text/plain` | `TestNewScriptRoute` (headers, strong ETag, 304), `TestRoutesAssemble` (`POST /new` 405) | mitigated | TestNewScriptRoute |
| T-14.3-14 | Tampering | moved upstream tag, non-fast-forward | high | mitigate | `new.sh` `checkout_framework`: `git fetch --quiet --tags origin` without `--force` (git refuses to clobber a local tag and the script stops); checkout only when `git merge-base --is-ancestor HEAD <target>`; otherwise exit 1 with the checkout unchanged | refusal probe: moved `v0.0.2` upstream → fetch refused, local tag kept at its original commit, HEAD unchanged; non-ancestor HEAD → "not a fast-forward", exit 1, HEAD unchanged | mitigated | TestScriptBehavior |
| T-14.3-15 | Tampering | user's existing checkout | high | mitigate | `new.sh` `check_checkout`: refuses a non-directory, a missing or foreign `go.mod` module, a non-git directory or a subdirectory of another repository, tracked changes (`git status --porcelain --untracked-files=no`), a branch (`git symbolic-ref -q HEAD`); the script contains no reset, clean, stash, `--force` or delete | refusal probe: branch checkout, dirty tracked file and non-framework directory each refused with exit 1 and left untouched | mitigated | TestScriptBehavior |
| T-14.3-16 | Elevation of Privilege | shell injection via name or overrides | high | mitigate | `new.sh` `main`: name regex before use; every expansion quoted (`shellcheck` clean); overrides used only as arguments (`git clone -- "$repo" "$dir"`, `-C "$dir"`); `SUMMERCMS_REF` rejected when it starts with `-`; no `eval`, `sudo` or dynamic code strings; `summer new` re-validates the name (`ValidateAppName`) | `grep -Ecw 'sudo\|eval'` = 0; `shellcheck`; refusal probe (bad name exit 2) | mitigated | TestScriptShellcheck, TestScriptBehavior |
| T-14.3-17 | Denial of Service | credential prompt hang with piped stdin | medium | mitigate | `new.sh` `main` exports `GIT_TERMINAL_PROMPT=0`; every `git`, `go`, `docker` and `summer` child runs with `</dev/null`; `main "$@"` is the last line so nothing follows in the pipe | `TestBootstrapEndToEnd` and `--acceptance` ran the script from a pipe | mitigated | TestScriptBehavior |
| T-14.3-18 | Tampering | supply chain of the served script | high | mitigate | `modules/sunrise/sunrise.go` `Script` embeds `new.sh` from the framework release the site binary is built against (the site's release build requires that tag); default clone URL is HTTPS; only `vX.Y.Z` tags are selected (`latest_tag`); the acceptance harness verifies the Go tarball against the sha256 published by go.dev | `TestScriptEmbedded`; `--acceptance` printed `go1.27.0.linux-amd64.tar.gz: OK` | mitigated | TestScriptEmbedded |
| T-14.3-19 | Elevation of Privilege | acceptance harness on host Docker socket | medium | mitigate | `scripts/check-phase14.3.sh` `stage_acceptance`: local harness only (never CI or the site); framework mounted read-only (`/fw:ro`); compose uses a named volume only; unique `COMPOSE_PROJECT_NAME`; inner trap `docker compose down -v`; host-side `acc_cleanup` removes the runner and every container, volume and network labelled with the project | after the run, no `phase143acc-*` container or volume remained | mitigated | TestScriptBehavior (acceptance stage in `--all`) |
| T-14.3-SC | Tampering | npm/pip/cargo installs | high | mitigate | No package-manager installs in the framework or the plugin; `git diff v0.1.12..HEAD -- go.mod go.sum` is empty and the plugin's `go.mod`/`go.sum` are unchanged; the container installs come from Debian's and Docker's signed apt repositories and a checksum-verified Go tarball | `git diff` empty; `--acceptance` | mitigated | Phase 14.3 gate forbidden-surface scan |
| T-14.3-14 | Tampering | moved upstream tag, non-fast-forward | high | mitigate | `new.sh` `checkout_framework`: `git fetch --quiet --tags origin` without `--force` (git refuses to clobber a local tag and the script stops); checkout only when `git merge-base --is-ancestor HEAD <target>`; otherwise exit 1 with the checkout unchanged | refusal probe: moved `v0.0.2` upstream → fetch refused, local tag kept at its original commit, HEAD unchanged; non-ancestor HEAD → "not a fast-forward", exit 1, HEAD unchanged | mitigated | TestScriptBehavior (moved upstream tag, non-fast-forward, re-run cases) |
| T-14.3-15 | Tampering | user's existing checkout | high | mitigate | `new.sh` `check_checkout`: refuses a non-directory, a missing or foreign `go.mod` module, a non-git directory or a subdirectory of another repository, tracked changes (`git status --porcelain --untracked-files=no`), a branch (`git symbolic-ref -q HEAD`); the script contains no reset, clean, stash, `--force` or delete | refusal probe: branch checkout, dirty tracked file and non-framework directory each refused with exit 1 and left untouched | mitigated | TestScriptBehavior (tracked change, branch, another module, not a git repository) |
| T-14.3-16 | Elevation of Privilege | shell injection via name or overrides | high | mitigate | `new.sh` `main`: name regex before use; every expansion quoted (`shellcheck` clean); overrides used only as arguments (`git clone -- "$repo" "$dir"`, `-C "$dir"`); `SUMMERCMS_REF` rejected when it starts with `-`; no `eval`, `sudo` or dynamic code strings; `summer new` re-validates the name (`ValidateAppName`) | `grep -Ecw 'sudo\|eval'` = 0; `shellcheck`; refusal probe (bad name exit 2) | mitigated | TestScriptShellcheck, TestScriptBehavior (metacharacter name, dash SUMMERCMS_REF) |
| T-14.3-17 | Denial of Service | credential prompt hang with piped stdin | medium | mitigate | `new.sh` `main` exports `GIT_TERMINAL_PROMPT=0`; every `git`, `go`, `docker` and `summer` child runs with `</dev/null`; `main "$@"` is the last line so nothing follows in the pipe | `TestBootstrapEndToEnd` and `--acceptance` ran the script from a pipe | mitigated | TestScriptBehavior (fresh clone: summer stdin not read although data is piped) |
| T-14.3-18 | Tampering | supply chain of the served script | high | mitigate | `modules/sunrise/sunrise.go` `Script` embeds `new.sh` from the framework release the site binary is built against (the site's release build requires that tag); default clone URL is HTTPS; only `vX.Y.Z` tags are selected (`latest_tag`); the acceptance harness verifies the Go tarball against the sha256 published by go.dev | `TestScriptEmbedded`; `--acceptance` printed `go1.27.0.linux-amd64.tar.gz: OK` | mitigated | TestScriptEmbedded, TestScriptBehavior (pre-release ignored, numeric tag order) |
| T-14.3-19 | Elevation of Privilege | acceptance harness on host Docker socket | medium | mitigate | `scripts/check-phase14.3.sh` `stage_acceptance`: local harness only (never CI or the site); framework mounted read-only (`/fw:ro`); compose uses a named volume only; unique `COMPOSE_PROJECT_NAME`; inner trap `docker compose down -v`; host-side `acc_cleanup` removes the runner and every container, volume and network labelled with the project | after the run, no `phase143acc-*` container or volume remained | mitigated | --acceptance (stage of --all) |
| T-14.3-20 | Tampering | one-liner shown on the page and in docs | medium | mitigate | HTTPS URL with `curl -fsSL`, taken verbatim from `vue-summercmsio-app/app/data/terminal.json`; docs fence in `docs/setup/installation.md` | plan 03 Task 1 and 3 verify chains | mitigated | TestTerminalScript, terminal.test.ts (pnpm test), TestTerminalCommands (--terminal) |
| T-14.3-21 | Denial of Service | gated terminal check leftovers | low | mitigate | `sm-summercmsio-app/terminal_check_test.go`: unique `COMPOSE_PROJECT_NAME`, `t.Cleanup` with `docker compose down -v`, bounded serve poll that stops the app binary and summer | plan 03 gated run left no container, volume or listener | mitigated | TestTerminalRunScript, TestTerminalCommands (--terminal) |
| T-14.3-22 | Information Disclosure | docs naming a consuming application | low | mitigate | `summer docs:build --check` forbidden-name rule; plan 03 grep on the edited pages | `docs:build --check` green | mitigated | --unit (docs:build --check), TestDocsTree |
| T-14.3-23 | Repudiation | phase gate | high | mitigate | `scripts/check-phase14.3.sh` `require_named_pass`: every required test needs a top-level `--- PASS:` line; a skipped required test, `no tests to run` or any failure fails the stage; `--all` runs each stage in its own process and prints `Phase 14.3 gate passed` only at the end | `--all` run 2026-10-08 | mitigated | --unit, --bootstrap, --terminal, --all |
| T-14.3-24 | Tampering | golden files | medium | mitigate | `internal/build/new_golden_test.go`: goldens are rewritten only with `-update`; `--unit` never passes it and fails when `internal/build/testdata` differs from the index after the run; goldens reviewed and committed with the tests (eed24b8) | one-byte golden edit made `TestNewGolden` fail (checked by hand, restored) | mitigated | TestNewGolden, TestNewGoldenNoPlugin, --unit |
| T-14.3-25 | Information Disclosure | fixed test key in goldens | low | accept | `goldenAppKey` is base64 of the bytes 0x00..0x1f, a deterministic non-secret test value used only by the golden tests | review | accepted | TestNewGolden |
| T-14.3-26 | Denial of Service | Docker resources from gate stages | low | mitigate | `--generator`, `--acceptance` and `--terminal` use unique compose project names and EXIT traps with `docker compose down -v`; `acc_cleanup` removes containers, volumes and networks by project label; `--all` runs each stage in its own process so every trap fires before the next stage | no `phase143*` or `summercms-terminal-*` container or volume remained after `--all` | mitigated | --generator, --acceptance, --terminal |
| T-14.3-SC | Tampering | npm/pip/cargo installs | high | mitigate | No package-manager installs in the framework or the plugin; `git diff v0.1.12..HEAD -- go.mod go.sum` is empty and the plugin's `go.mod`/`go.sum` are unchanged; the container installs come from Debian's and Docker's signed apt repositories and a checksum-verified Go tarball | `git diff` empty; `--acceptance` | mitigated | --forbidden (in --unit) |
## Refusal probe (2026-10-08)
Plan 04 turned every case below into an automated `TestScriptBehavior` subtest (25 cases, stub `go`/`docker`, real `git`, a bare fake framework remote with tags v0.1.0, v0.2.0, v0.10.0 and v1.0.0-rc1).
Fake framework repository with tags `v0.0.1`, `v0.0.2`; `SUMMERCMS_REPO=<fake>`; script run from a work directory holding `summercms`:
| Case | Result |

76
.planning/phases/14.3-one-command-app-scaffolding-summer-new-and-summercms-io-new/14.3-VALIDATION.md
View File

@@ -1,15 +1,16 @@
---
phase: "14.3"
slug: "one-command-app-scaffolding-summer-new-and-summercms-io-new"
status: draft
nyquist_compliant: false
wave_0_complete: false
status: validated
nyquist_compliant: true
wave_0_complete: true
created: "2026-10-08"
validated: "2026-10-08"
---
# Phase 14.3 — Validation Strategy
> Per-phase validation contract for feedback sampling during execution. Source: 14.3-RESEARCH.md § Validation Architecture.
> Per-phase validation contract for feedback sampling during execution. Source: 14.3-RESEARCH.md § Validation Architecture. Signed off by plan 04 after `bash scripts/check-phase14.3.sh --all` ended with `Phase 14.3 gate passed` on 2026-10-08.
---
@@ -19,9 +20,11 @@ created: "2026-10-08"
|----------|-------|
| **Framework** | Go stdlib `testing` (Go 1.27.0); `node:test` in `vue-summercmsio-app` |
| **Config file** | none |
| **Quick run command** | `go test ./internal/build/ ./cmd/summer/ ./modules/lagoon/ ./modules/<pkg>/ -short` |
| **Quick run command** | `go test ./internal/build/ ./cmd/summer/ ./modules/lagoon/ ./modules/sunrise/ -short` |
| **Full suite command** | `go vet ./... && go test ./... && go -C examples/hello test ./...` |
| **Estimated runtime** | ~40 seconds quick (incl. real builds), a few minutes full |
| **Phase gate** | `bash scripts/check-phase14.3.sh --all` (stages `--unit`, `--generator`, `--bootstrap`, `--terminal`, `--acceptance`, `--security`; final line `Phase 14.3 gate passed`) |
| **Golden update** | `go test ./internal/build/ -run '^(TestNewGolden|TestNewGoldenNoPlugin)$' -update`, then review and commit the diff; the gate never passes `-update` and fails when `internal/build/testdata` changes |
| **Estimated runtime** | ~20 seconds quick (incl. real builds), ~90 seconds full suite, ~6 minutes `--all` (container acceptance) |
---
@@ -29,26 +32,33 @@ created: "2026-10-08"
- **After every task commit:** quick run command
- **After every plan wave:** full suite + the plan's sibling-repo command (below)
- **Before `/gsd-verify-work` and before tagging `v0.1.13`:** full suite green, `go run ./cmd/summer docs:build --check` green, D-20 acceptance run recorded
- **Max feedback latency:** ~60 seconds
- **Before `/gsd-verify-work` and before tagging `v0.1.13`:** `bash scripts/check-phase14.3.sh --all` green
- **Max feedback latency:** ~60 seconds for the quick run
---
## Per-Plan Verification Map
| Plan | Behavior | Test Type | Automated Command |
|------|----------|-----------|-------------------|
| 01 | `summer new demo` in a temp parent with a `summercms` symlink to the repo builds | smoke (real build) | `go test ./internal/build/ -run 'TestNewApp' -count=1` |
| 01 | `summer new` wired in CLI with its flags | unit | `go test ./cmd/summer/ -run 'TestToolCommandNames' -count=1` |
| 01 | `GenerateAppKey` returns StdEncoding of 32 random bytes; `key:generate` uses it | unit | `go test ./modules/lagoon/ -run 'Key' -short -count=1` |
| 01 | `plugin:add` writes relative `go.work` paths | unit | `go test ./internal/build/ -run 'TestMakeAndAddPluginOnHelloCopy' -count=1` |
| 01 | `examples/hello` config gap fixed | integration | `go -C examples/hello test ./... -count=1` |
| 02 | script: shebang, `main "$@"` last line, `bash -n`, truncation runs nothing, tag selection on a temp repo | unit + gated e2e | `go test ./modules/<pkg>/ -count=1`; gated `SUMMERCMS_BOOTSTRAP_CHECK=1 go test ./modules/<pkg>/ -run TestBootstrapEndToEnd -count=1` |
| 02 | `GET /new` 200 `text/plain; charset=utf-8`, body = embedded script, HEAD ok, POST 405, nosniff | unit | `go -C ../sm-summercmsio-app/plugins/golem15/summercms test ./... -count=1` |
| 02 | D-20 container chain ends with starter route 200 | gated/manual | acceptance runner script or documented `docker run` command |
| 03 | docs tree valid, `summer new` known command, llms-full contains one-liner | unit | `go test ./cmd/summer -run 'TestDocsTree|TestDocsBuildRealTree' -count=1` and `go run ./cmd/summer docs:build --check` |
| 03 | landing page terminal data / tests | unit | `go -C ../sm-summercmsio-app test -run 'TestLoadTerminal|TestTerminalScript|TestTerminalEnv' .`; `pnpm -C ../sm-summercmsio-app/vue-summercmsio-app generate && pnpm -C ../sm-summercmsio-app/vue-summercmsio-app test` |
| 04 | golden output for every rendered file; name/denylist/existing-dir/framework-detection errors; relative replace; `--no-plugin`, `--no-git`; no-identity commit; cleanup on failure; real build (`-short` skip) | unit + golden | `go test ./internal/build/ -run 'TestNew' -count=1`; update goldens with `-update` |
| Plan | Behavior | Test Type | Automated Command | Status |
|------|----------|-----------|-------------------|--------|
| 01 | `summer new demo` in a temp parent with a `summercms` symlink builds and lists the starter route; `--no-plugin` builds | smoke (real build) | `go test ./internal/build/ -run '^(TestNewAppSmoke|TestNewNoPluginBuilds)$' -count=1` | ✅ |
| 01 | every file `summer new` writes, default and `--no-plugin` (go.mod relative replace, go.work, config/, .env, compose `127.0.0.1:54320`, .gitignore, CLAUDE.md, starter plugin, next steps) | golden | `go test ./internal/build/ -run '^(TestNewGolden|TestNewGoldenNoPlugin)$' -count=1` | ✅ |
| 01 | name rule and denylist; existing target (dir, file, symlink, dangling symlink); framework resolution order; target inside the checkout; cleanup on failure; key validation | unit | `go test ./internal/build/ -run '^(TestNewValidateAppName|TestNewRefusesExistingTarget|TestNewFrameworkResolution|TestNewRefusesTargetInsideFramework|TestNewCleansUpOnFailure|TestNewRequiresValidAppKey)$' -count=1` | ✅ |
| 01 | `.env` 0600; AGENTS.md symlink; first commit with fallback identity, `.env` untracked; configured identity; git missing is a warning; `--no-git` | unit | `go test ./internal/build/ -run '^(TestNewEnvFileMode|TestNewAgentsSymlink|TestNewGitFirstCommit|TestNewGitUsesConfiguredIdentity|TestNewGitMissingIsWarning|TestNewNoGit)$' -count=1` | ✅ |
| 01 | `plugin:add` writes relative `go.work` paths; plain `make:plugin` keeps the no-route stub | unit | `go test ./internal/build/ -run '^(TestMakeAndAddPluginOnHelloCopy|TestMakePluginRoutesUnchanged)$' -count=1` | ✅ |
| 01 | `summer new` wired in the CLI with its flags | unit | `go test ./cmd/summer/ -run '^(TestToolCommandNames|TestNewCommandViaCLI)$' -count=1` | ✅ |
| 01 | `GenerateAppKey` returns StdEncoding of 32 random bytes; `key:generate` uses it | unit | `go test ./modules/lagoon/ -run '^(TestGenerateAppKey|TestEncryptedKeyGeneratePrints32ByteBase64)$' -count=1` | ✅ |
| 01 | `examples/hello` config gap fixed | integration | `go -C examples/hello test ./... -count=1` | ✅ |
| 01 | D-12: generated app builds, migrates on PostgreSQL 16 and serves the starter route with no edits | e2e | `bash scripts/check-phase14.3.sh --generator` | ✅ |
| 02 | script: embedded bytes, shebang, `main "$@"` last line, `bash -n`, truncation runs nothing | unit | `go test ./modules/sunrise/ -run '^(TestScriptEmbedded|TestScriptShape|TestScriptTruncationRunsNothing)$' -count=1` | ✅ |
| 02 | script behaviour matrix (usage/name errors, existing app, Go missing/old/devel, git required, Docker warning, newest release tag, `SUMMERCMS_REF`, fast-forward re-runs, refusals with HEAD unchanged, stdin closed, cd line, PATH hint) and shellcheck | unit | `go test ./modules/sunrise/ -run '^(TestScriptBehavior|TestScriptShellcheck)$' -count=1` | ✅ |
| 02 | `curl \| bash` from an HTTP server, clone at HEAD, install into a GOBIN off PATH, build the app | gated e2e | `bash scripts/check-phase14.3.sh --bootstrap` (`SUMMERCMS_BOOTSTRAP_CHECK=1 go test ./modules/sunrise/ -run '^TestBootstrapEndToEnd$'`) | ✅ |
| 02 | `GET /new` 200 `text/plain; charset=utf-8`, body = embedded script, HEAD ok, POST 405, nosniff | unit | `go -C ../sm-summercmsio-app/plugins/golem15/summercms test ./... -count=1` (`TestNewScriptRoute`, `TestRoutesAssemble`) | ✅ |
| 02 | D-20 container chain ends with the starter route answering | e2e (Docker) | `bash scripts/check-phase14.3.sh --acceptance` | ✅ |
| 03 | docs tree valid, `summer new` a known command, llms outputs in sync | unit | `go test ./cmd/summer -run '^(TestDocsTree|TestDocsAIOutputsInSync)$' -count=1` and `go run ./cmd/summer docs:build --check` | ✅ |
| 03 | landing page terminal data and tests | unit | `go -C ../sm-summercmsio-app test ./... -count=1` (`TestLoadTerminal`, `TestTerminalScript`, `TestTerminalEnv`, `TestTerminalRunScript`); `pnpm -C ../sm-summercmsio-app/vue-summercmsio-app generate && pnpm -C ../sm-summercmsio-app/vue-summercmsio-app test` | ✅ |
| 03 | the six page commands run verbatim until the starter route answers | gated e2e | `bash scripts/check-phase14.3.sh --terminal` | ✅ |
| 04 | fail-closed gate: named PASS for every required test, forbidden surfaces, threat register | gate | `bash scripts/check-phase14.3.sh --all` | ✅ |
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
@@ -56,10 +66,10 @@ created: "2026-10-08"
## Wave 0 Requirements
- [ ] `internal/build/testdata/new/` golden files + `-update` flag (no golden infra exists yet)
- [ ] new `modules/<pkg>/` package with README and tests
- [ ] test helper creating `<tmp>/summercms` → repo-root symlink so the generated replace is deterministic (`../summercms`)
- [ ] optional acceptance runner script
- [x] `internal/build/testdata/new/` and `testdata/new-no-plugin/` golden files + `-update` flag (`internal/build/new_golden_test.go`)
- [x] new `modules/sunrise/` package with README and tests
- [x] test helper creating `<tmp>/summercms` → repo-root symlink so the generated replace is deterministic (`../summercms`): `newGoldenApp`, `parentWithFramework`
- [x] acceptance runner script: `scripts/check-phase14.3.sh --acceptance`
---
@@ -67,17 +77,17 @@ created: "2026-10-08"
| Behavior | Why Manual | Test Instructions |
|----------|------------|-------------------|
| `curl -fsSL https://summercms.io/new \| bash -s -- demo` against production | needs `v0.1.13` tag pushed and the site redeployed (user actions) | run D-20 chain in a clean `node:24-bookworm` container with Go 1.27 and the host Docker socket |
| `curl -fsSL https://summercms.io/new \| bash -s -- demo` against production | needs `v0.1.13` tagged and pushed, the site's require bumped and the site redeployed (user actions, D-28) | `curl -fsSI https://summercms.io/new` shows `200` and `text/plain; charset=utf-8`; in `sm-summercmsio-app`: `SUMMERCMS_TERMINAL_CHECK=1 go test -run TestTerminalCommands -count=1 -v .` with no overrides; in `summercms.go`: `bash scripts/check-phase14.3.sh --acceptance-production` |
---
## Validation Sign-Off
- [ ] All tasks have `<automated>` verify or Wave 0 dependencies
- [ ] Sampling continuity: no 3 consecutive tasks without automated verify
- [ ] Wave 0 covers all MISSING references
- [ ] No watch-mode flags
- [ ] Feedback latency < 60s
- [ ] `nyquist_compliant: true` set in frontmatter
- [x] All tasks have `<automated>` verify or Wave 0 dependencies
- [x] Sampling continuity: no 3 consecutive tasks without automated verify
- [x] Wave 0 covers all MISSING references
- [x] No watch-mode flags
- [x] Feedback latency < 60s
- [x] `nyquist_compliant: true` set in frontmatter
**Approval:** pending
**Approval:** approved 2026-10-08 (`bash scripts/check-phase14.3.sh --all` → `Phase 14.3 gate passed`)