docs(phase-34): add research and validation strategy
This commit is contained in:
Jakub Zych
702
.planning/phases/34-plugin-marketplace/34-RESEARCH.md
Normal file
702
.planning/phases/34-plugin-marketplace/34-RESEARCH.md
Normal file
@@ -0,0 +1,702 @@
|
||||
# Phase 34: Plugin marketplace - Research
|
||||
|
||||
**Researched:** 2026-10-29
|
||||
**Domain:** Go module/workspace mechanics for compiled plugins, git-based registry ingestion, a public JSON registry API, cabana/boardwalk admin screen, Nuxt SSR pages
|
||||
**Confidence:** HIGH for everything grounded in the code and the local probes below; MEDIUM for proposed shapes (registry JSON, DB schema, module name), which are design recommendations, not facts.
|
||||
|
||||
<user_constraints>
|
||||
## User Constraints (from CONTEXT.md)
|
||||
|
||||
### Locked Decisions
|
||||
|
||||
#### Manifest: plugin.yaml is generated, never hand-edited
|
||||
- **D-01:** `plugin.yaml` is generated by a new command `summer plugin:manifest` run in a plugin repo. `plugin.go` stays as it is (no change to `party.Plugin`, no breaking change to core plugins).
|
||||
- **D-02:** Every field is derived from the repo: `id` and `requires` from `plugin.go` (AST: the literals returned by `ID()` / `Requires()`), module path from `go.mod`, minimum framework version and minimum required-plugin versions from `go.mod`'s `require` lines, `name` and `description` from the README's H1 and its one-sentence summary (the standard module README structure). A plugin whose `ID()`/`Requires()` are not literal returns is an error that names the function.
|
||||
- **D-03:** Optional `--name="…"` / `--description="…"` override the README-derived text. Overrides persist in `plugin.yaml` and survive regeneration until changed or cleared (exact storage is planner's choice, e.g. an `overrides:` block).
|
||||
- **D-04:** `summer plugin:manifest --check` fails when the committed `plugin.yaml` is stale. It runs in each plugin repo's tests (a pact/test helper or `go test`-runnable check) and in the tag/changelog step (`scripts/changelog-post.py publish … --repo`). `plugin.yaml` is committed. — **Reversibility:** costly — the format is read by the registry ingester, `plugin:install` for git URLs, and every public plugin repo; changing it means regenerating all repos and migrating the ingester.
|
||||
- **D-05:** Version constraints are Go-style minimums: `framework: v0.4.0`, `requires: {golem15.translate: v0.2.0}`. No semver ranges (Go's minimal version selection; a major bump changes the module path).
|
||||
|
||||
#### Runtime metadata for the Installed tab
|
||||
- **D-06:** `summer build` reads each plugin's `plugin.yaml` and bakes name and description into `plugins.gen.go`. Plugins without a `plugin.yaml` show only their ID.
|
||||
- **D-07:** Version: module installs use the version from `go.mod`/build info; submodule or local-`replace` plugins get `git describe --tags` run in the plugin directory at build time (e.g. `v0.3.1-4-gabc123`), also baked into `plugins.gen.go`.
|
||||
|
||||
#### Install resolution
|
||||
- **D-08:** `summer plugin:install golem15.journal` picks the newest tag whose minimum framework version the app's `go.mod` already satisfies; `golem15.journal@v0.3.0` pins. If only tags needing a newer framework exist, it fails and names the framework version to move to (it does not bump the framework silently).
|
||||
- **D-09:** Missing required plugins are resolved transitively, the full list of additions is printed, and the user confirms; `--yes` skips the prompt and non-interactive runs require `--yes`. Dependencies install in the same mode as the requested plugin (module, or submodule under `--source`).
|
||||
- **D-10:** `summer plugin:update [id]` is in scope: module installs move to the newest compatible tag (`go get`), submodule installs are checked out at the newest compatible tag. Without an ID it updates all plugins it can resolve.
|
||||
- **D-11:** Git-URL installs are submodule-only in v1; no `--module`. Close the open question in `.planning/research/questions.md` as "not in v1". — **Reversibility:** reversible — adding `--module` later is additive.
|
||||
- **D-12:** A git-URL plugin's own repo is never sent to summercms.io, but its required plugin IDs (e.g. `golem15.translate`) are resolved through the registry like any ID install. `--offline` disables registry calls and then requires the dependencies to be installed already.
|
||||
- Both modes finish with the same registration `plugin:add` does (`summer.yaml`, `go.mod`, `go.work` where applicable), and module-installed plugins must work with the build steps that currently inspect plugin directories (`internal/build/leaf.go`, `internal/build/registry.go`) — resolve their directory from the module cache.
|
||||
|
||||
#### Registry ingestion and API (summercms.io, `golem15.summercms` site plugin)
|
||||
- **D-13:** Pull ingester: given a repo URL it clones anonymously, reads tags, each tag's `plugin.yaml`, and the README. It runs as a River periodic job (around hourly) and on an authenticated "ingest now" endpoint that `changelog-post.py publish --repo` calls. No forge webhooks.
|
||||
- **D-14:** The listing is admin-curated: a summercms.io admin adds a repo URL; the ingester fills the facts. A plugin is published only once it has a tag with a valid `plugin.yaml`; tags without one are skipped.
|
||||
- **D-15:** Every `vX.Y.Z` tag is stored as a version row with that tag's framework and plugin minimums (needed by D-08). README is stored for the latest tag only.
|
||||
- **D-16:** Public read-only JSON API, no auth, open CORS, cache headers: `GET /api/registry/v1/plugins?q=&category=` (list/search), `GET /api/registry/v1/plugins/{id}` (detail incl. README as markdown and rendered HTML), `GET /api/registry/v1/plugins/{id}/versions`. Consumed by the Nuxt marketplace pages, the CLI and cabana. — **Reversibility:** one-way — once released `summer` binaries call `/api/registry/v1`, its shape cannot change without a v2 path.
|
||||
- Presentation fields (category, featured, screenshots, icon, sort order) are edited in the summercms.io admin (from the decision note).
|
||||
|
||||
#### Admin Plugins screen
|
||||
- **D-17:** The SPA talks only to its own backend: cabana gets swag-annotated admin endpoints that call the registry with a timeout and short cache, and compute "update available" in Go against installed versions. No CSP/CORS change in the admin.
|
||||
- **D-18:** Config key `plugins.registry_url` (default `https://summercms.io`); empty disables registry calls, which hides the Marketplace tab and update badges while Installed still works. The CLI reads the same setting (allows a staging or self-hosted registry). Exact key name is planner's choice if it clashes with existing config.
|
||||
- **D-19:** New permission (a `system.plugins.view`-style key in cabana's permission set), granted to superusers by default.
|
||||
|
||||
### Claude's Discretion
|
||||
- Marketplace page layout and visuals on `vue-summercmsio-app` (list, search, category, detail); `/msd-ui-phase 34` can produce a UI contract if wanted.
|
||||
- Storage of D-03 overrides inside `plugin.yaml`.
|
||||
- Registry DB schema, ingester error handling, cache durations, periodic interval.
|
||||
- Whether older tags (before `plugin.yaml` existed) get backfilled; default per D-14 is to skip them.
|
||||
- Exact CLI output/prompt wording and the semantics of `plugin:update` with no ID when some plugins can't be resolved.
|
||||
|
||||
### Deferred Ideas (OUT OF SCOPE)
|
||||
- `--module` for git-URL (private) installs. Revisit if submodule-only proves limiting.
|
||||
- Already out of scope per the decision note: paid plugins, licensing, developer accounts, third-party submissions, themes, installing from the admin panel.
|
||||
- Reviewed Todos (not folded): 17 todos matched on keywords only (e.g. gate scripts naming apps, OIDC parity, MFA findings, menu plugin port). None concern the marketplace; none folded.
|
||||
</user_constraints>
|
||||
|
||||
<phase_requirements>
|
||||
## Phase Requirements
|
||||
|
||||
The roadmap lists "Requirements: TBD" and no requirement IDs were provided. The five success criteria are the acceptance surface; this table maps them so the planner can treat them as SC-1..SC-5.
|
||||
|
||||
| ID | Description | Research Support |
|
||||
|----|-------------|------------------|
|
||||
| SC-1 | Every public `sm-*-plugin` repo carries a framework-validated `plugin.yaml`; anonymous fetch works | §Manifest generation, §Finding F1 (13 plugins return a const, not a literal), §Plugin repo survey, anonymous `git ls-remote` probe |
|
||||
| SC-2 | summercms.io ingests manifest/README/versions on tag, serves the registry API, renders /marketplace; presentation fields in admin | §Ingester, §Registry API shape, §summercms.io layout, §Marketplace pages |
|
||||
| SC-3 | `plugin:install <id>` (module; `--source` submodule) and `plugin:install <git-url>` (submodule), with requirement checks and `plugin:add`-style registration | §Install resolution, §Findings F2–F5 (go.work replace, effective framework version, `ConfirmProceed`) |
|
||||
| SC-4 | Admin Plugins screen: Installed (versions, updates) + Marketplace (search, copy command) | §Admin screen, §Runtime metadata (D-06/D-07) |
|
||||
| SC-5 | Unit tests in the last plan | §Validation Architecture |
|
||||
</phase_requirements>
|
||||
|
||||
## Project Constraints (from CLAUDE.md)
|
||||
|
||||
From `summercms.go/CLAUDE.md`, `../CLAUDE.md` and `~/.claude/CLAUDE.md` (same authority as locked decisions):
|
||||
|
||||
- **No new dependency** without a research doc or a phase/decision note naming it (`.planning/research/STACK.md` is the decided set). This research recommends **zero new Go or npm dependencies** (see Standard Stack).
|
||||
- YAML is parsed with `goccy/go-yaml`, never `gopkg.in/yaml.v3`.
|
||||
- Standard library first.
|
||||
- Framework code never names a consuming application (READMEs, docs, examples use `acme`/`blog`). The registry default URL `https://summercms.io` is the framework's own site, not a consuming app, but **code comments and docs must not name `sm-summercmsio-app`**.
|
||||
- A new framework module gets a beach-themed name, a README in the standard structure (H1, one-sentence summary, import line, `**Name:**`, Overview, Features, Usage, API reference, optional Configuration/CLI, Dependencies, Testing) and a row in the root `README.md` table.
|
||||
- A change to a module's exported API, config keys, CLI commands or dependencies updates that module's README and the affected `docs/` pages **in the same commit**. Every identifier in docs must exist (`summer docs:build --check`, `go test ./cmd/summer -run TestDocsTree`). New commands must pass `TestCommandsDocumented` (description, long help, examples, described args/flags).
|
||||
- Core plugins (user, blog/journal, pages, payment, …) are used by many apps: **no breaking changes** unless asked. Plugin contracts (routes, payloads, tables, migration IDs, config keys) stay stable.
|
||||
- Plugin repos: commit and push the plugin first, then the app's submodule pointer bump as a separate commit. Never stage files inside a submodule from the app repo.
|
||||
- Lean planning: few, large plans; present plan count + one-line scopes and wait for confirmation; **unit tests are the last plan**; earlier plans may carry smoke tests.
|
||||
- State updates through `msd_run query state.*` handlers; g15office sync comment when the phase advances.
|
||||
- No co-author tags; one logical change per commit; planning docs and code in separate commits; stage files by explicit path (other sessions work concurrently, often in cabana — check for in-flight edits before touching it).
|
||||
- Don't push, tag or deploy without the user's go-ahead. Release tags only at phase completion, on the default branch (memory). Every pushed framework tag gets a /changelog post; plugin tags get /changelog/plugins posts.
|
||||
- Admin changes: rebuild `modules/boardwalk/dist`, commit it, pass `scripts/check-admin-dist.sh` before any tag (`.planning/notes/2026-10-07-embedded-admin-release-gate.md`).
|
||||
- Keep `go vet ./...` and `go test ./...` green at every commit. Docker-backed tests skip under `-short`; a `TestMain` starting containers must call `flag.Parse()` and check `testing.Short()`.
|
||||
|
||||
## Summary
|
||||
|
||||
The phase spans four repo groups, and most of its risk is in the Go module and workspace mechanics, not in the UI. Probes against the real repos and Go 1.27.2 turned up five facts the planner must design around. They are not visible from CONTEXT.md:
|
||||
|
||||
1. **13 of 22 plugins return a constant, not a literal, from `ID()`**, and two (`sm-pgstripe-plugin/subscriptions`, `sm-paymentgateway-plugin`) use cross-package constants. The existing `pluginIDFromGo` already fails on them (`plugin:add` cannot register `golem15.mfa` today; reproduced). D-02's "literal returns" needs a narrow, static constant resolver. Otherwise most public plugins cannot get a `plugin.yaml` without editing `plugin.go`.
|
||||
2. **A `--source` (submodule) install breaks the app's build**: every public plugin's committed `go.mod` carries `replace git.golem15.com/golem15/summercms => ../summercms.go`, and workspace modules' replaces apply. Reproduced; fixed by a `go.work`-level `replace` (verified).
|
||||
3. **Apps created by `summer new` require the framework at `v0.0.0`** with a local replace, so "the version the app's go.mod satisfies" (D-08) must be the *effective* framework version: the newest tag reachable in the replaced checkout.
|
||||
4. `bonfire.ConfirmProceed` returns **true** when not interactive, the opposite of D-09; `plugin:install` must not use it.
|
||||
5. Module installs work out of the box: `go get` succeeds in workspace mode, non-main-module replaces are ignored, and proxy.golang.org already serves the public plugins. `go list -json` resolves every plugin's directory (workspace, replace or module cache) in one call, so `leaf.go`/`registry.go` need no change and the D-06/D-07 metadata step has one resolver.
|
||||
|
||||
On summercms.io, the site plugin today is routes-only. It gains models, migrations, admin, a scheduled ingest command, a public API with a CORS profile, and an authenticated ingest endpoint. Production nginx denies non-GET outside `/_journal/api/v1/` and 404s `/backend`, and `queue.work_in_serve` is `false`, so the deploy configs change too. The marketplace pages fit the existing changelog Node SSR app (same prefix-routing pattern) better than the static `nuxt generate` landing.
|
||||
|
||||
**Primary recommendation:** put the plugin.yaml spec (parse, validate, generate, check, const-resolving ID/Requires reader) and the registry HTTP client in one new public framework module. The CLI, the build, cabana, plugin repo tests and the summercms.io ingester then share one implementation. Use `go list -json`, `go mod edit -json`, `go get`, `go work edit` and `git` through `os/exec`, as the framework already does, and add no new dependencies.
|
||||
|
||||
## Architectural Responsibility Map
|
||||
|
||||
| Capability | Primary Tier | Secondary Tier | Rationale |
|
||||
|------------|-------------|----------------|-----------|
|
||||
| `plugin.yaml` generation/check | Dev CLI (`summer`, framework public module) | Plugin repo `go test` | Derived from plugin source; must run inside plugin repos and the publish script |
|
||||
| Metadata baking (D-06/D-07) | Build tool (`internal/build`) | Runtime (`party`/cabana read it) | Only build time sees plugin dirs and git; runtime only reads baked values |
|
||||
| Registry data (plugins, versions, presentation) | API/Backend (summercms.io site plugin, Postgres) | — | Single source of truth for CLI, cabana, Nuxt |
|
||||
| Ingestion (clone, parse tags) | Backend job (conga scheduled command) | Backend HTTP (ingest-now endpoint) | Long-running, network-bound, must not run in browser or CLI |
|
||||
| Install/update resolution | Dev CLI | Registry API (read) | Mutates app files and go.mod/go.work/submodules: local only |
|
||||
| "Update available" computation | API/Backend (cabana, Go) | — | D-17: SPA talks only to its own backend |
|
||||
| Plugins screen UI | Browser (boardwalk SPA) | — | Read-only view + clipboard |
|
||||
| Marketplace pages | Frontend Server (Nuxt SSR, changelog Node app) | API (registry) | SEO-indexable, fresh data without rebuilding the embedded landing |
|
||||
| README → HTML | API/Backend (ingester, `breakwater.Render`) | — | Sanitize once at write time; Nuxt renders stored HTML |
|
||||
|
||||
## Standard Stack
|
||||
|
||||
No new dependency is needed. Everything below is already in the framework `go.mod`, in the decided set, or a standard tool already shelled out to.
|
||||
|
||||
### Core (existing, reused)
|
||||
| Component | Version | Purpose | Source of truth |
|
||||
|-----------|---------|---------|-----------------|
|
||||
| Go toolchain | 1.27 (`go 1.27.0` / `toolchain go1.27.0` lines; local go1.27.2) | `go list -json`, `go mod edit -json`, `go get`, `go work edit` via `os/exec` | `[VERIFIED: internal/build/scaffold.go:57-59]` `goVersionLine = "go 1.27.0"`, `toolchainLine = "toolchain go1.27.0"` |
|
||||
| `go/ast`, `go/parser`, `go/token` | stdlib | Const-resolving `ID()`/`Requires()` reader | already used in `internal/build/scaffold.go`, `registry.go`, `leaf.go` |
|
||||
| `github.com/goccy/go-yaml` | v1.19.2 | Parse and marshal `plugin.yaml` (struct field order = output order) | `[VERIFIED: go.mod]` |
|
||||
| `github.com/yuin/goldmark` | v1.8.6 | README AST walk: H1 text and first paragraph for name/description | `[VERIFIED: go.mod]`; already used by `internal/docsite`, `modules/breakwater` |
|
||||
| `modules/breakwater` (`breakwater.Render`) | in-tree | README markdown → sanitized HTML on summercms.io (GFM, raw HTML stripped, `http/https/mailto` links only, nofollow) | `[VERIFIED: modules/breakwater/breakwater.go:90]` |
|
||||
| `modules/fetchguard` (`fetchguard.NewClient`, `TrustedMode`) | in-tree | Registry HTTP client with size cap, timeout, no redirects; `WithTransport` test seam | `[VERIFIED: modules/fetchguard/client.go:40, policy.go:23-30]` |
|
||||
| `modules/compass` | in-tree | CLI reads `plugins.registry_url` from the app's `config/` + `SUMMER_` env | `[VERIFIED: modules/compass/README.md]` |
|
||||
| `modules/conga` + `pact.HasSchedule` | in-tree (River v0.47.0) | Hourly ingest as a scheduled console command | `[VERIFIED: modules/pact/capabilities.go:166-176]` |
|
||||
| `git` CLI | 2.56.0 locally; server: unverified | Ingester clone/tag reads; submodule installs; `git describe` | shelled out already in `internal/build/new.go:286` (`runGit`) |
|
||||
| swag | v1.16.6 via `go run` | cabana annotations → `admin/openapi/admin.json` → `schema.d.ts` | `[VERIFIED: scripts/check-admin-openapi.sh]` |
|
||||
|
||||
### Alternatives Considered
|
||||
| Instead of | Could Use | Tradeoff / why not |
|
||||
|------------|-----------|--------------------|
|
||||
| `go mod edit -json` for go.mod parsing | `golang.org/x/mod/modfile` | Not a dependency today (`go mod why -m golang.org/x/mod`: "main module does not need module golang.org/x/mod"); would need a decision note. `go mod edit -json` gives `Module.Path` and `Require[].Path/Version` (verified) with no dependency. |
|
||||
| Strict `vX.Y.Z` comparer (≈20 lines) | `golang.org/x/mod/semver` | Registry versions are strict `vX.Y.Z` (D-15), so a strict comparer is enough. Never feed `git describe` output (`v0.3.1-4-gabc`) into a semver comparer: semver reads it as a pre-release that sorts **before** `v0.3.1`. |
|
||||
| shell `git` in the ingester | `go-git` | New large dependency + decision note. Shell git is the framework's existing pattern; harden it with env (see Security). |
|
||||
| Forge HTTP API (raw files, tags) | — | Ties ingestion to one forge; D-13 locks "clones anonymously". |
|
||||
| Third Nuxt app for /marketplace | Extend the changelog SSR app | A third app means another supervisor program, port and nginx block. Reusing the changelog app adds one nginx prefix and reuses its fetch/swr/purge setup. Planner's discretion. |
|
||||
|
||||
**Installation:** none (no packages added).
|
||||
|
||||
## Package Legitimacy Audit
|
||||
|
||||
This phase installs **no external packages** (Go or npm). The audit is not applicable.
|
||||
|
||||
**Packages removed due to [SLOP] verdict:** none
|
||||
**Packages flagged as suspicious [SUS]:** none
|
||||
|
||||
## Key Findings (verified in this session)
|
||||
|
||||
### F1. Most plugins return a constant from `ID()`; the current reader fails on them
|
||||
`[VERIFIED: internal/build/scaffold.go:360-392]` `pluginIDFromGo` accepts only `*ast.BasicLit` of kind `token.STRING` in a `return` inside `func (…) ID()`. Probe (copy of that function run against real files):
|
||||
```
|
||||
sm-mfa-plugin/plugin.go => build: sm-mfa-plugin/plugin.go does not return a plugin id
|
||||
sm-journal-plugin/plugin.go => golem15.journal <nil>
|
||||
```
|
||||
Survey of `ID()`/`Requires()` in all sibling plugins:
|
||||
- Literal ID: faq, journal, menu, seo, sitemap, translate, user, summercms site plugin.
|
||||
- `return PluginID` with a same-file `const PluginID = "golem15.<name>"` (`[VERIFIED: grep of plugin.go files]`): chatbots, chatchannels, chat, chatvideo, forum, herald, mfa, oauthserver, paymentgatewaysubscriptions, pgstripe, sociallogin, userfriends.
|
||||
- Cross-package: `sm-paymentgateway-plugin/plugin.go:47` `const PluginID = payments.PluginID` (→ `payments/service.go:23` `const PluginID = "golem15.paymentgateway"`); `sm-pgstripe-plugin/subscriptions/plugin.go:83-85` `return []string{pgstripe.PluginID, pgsub.PluginID}` (one same-module package, one **other module**).
|
||||
- `Requires()` are literal string slices or `nil` everywhere else.
|
||||
|
||||
**Consequence:** D-02's strict "literal returns" would make 13 plugins fail `plugin:manifest`, and `plugin:add`/git-URL install (which "registers like plugin:add") already fails for them.
|
||||
**Recommendation (needs a quick user OK, since D-02 is locked):** a static resolver that still never runs code. It accepts (a) a string literal, (b) an identifier naming a package-level `const` whose value resolves recursively, in any non-test `.go` file of the same package directory, (c) `pkg.Ident` where `pkg`'s import path is in the same module (directory derived from module path) or in a required module (directory from `go list -m -json <module>` `.Dir`). Anything else (function call, var, concatenation with non-consts) is the D-02 error naming the function. Use the same resolver for `pluginIDFromGo` so `plugin:add` and `make:*` start working for const-ID plugins (a fix, not a behaviour break). The alternative, rewriting 13 `plugin.go` files to literal returns, touches core plugins and duplicates the exported `PluginID` constants.
|
||||
|
||||
### F2. `--source` (submodule) installs break the build unless go.work overrides plugin replaces
|
||||
All public plugin repos commit `replace git.golem15.com/golem15/summercms => ../summercms.go` (`[VERIFIED: grep "replace" sm-*-plugin/go.mod]`, e.g. `sm-menu-plugin/go.mod:176`). Several also replace sibling plugins (`sm-pgstripe-plugin/go.mod:184-186`, `sm-chatbots-plugin/go.mod:150-152`). In workspace mode these replaces apply. Probe: app + `go.work use ./plugins/golem15/menu` (clone of v0.1.1):
|
||||
```
|
||||
plugins/golem15/menu/plugin.go:17:2: git.golem15.com/golem15/summercms@v0.4.0 (replaced by ./plugins/golem15/summercms.go): reading plugins/golem15/summercms.go/go.mod: ... no such file or directory
|
||||
```
|
||||
After `go work edit -replace=git.golem15.com/golem15/summercms=git.golem15.com/golem15/summercms@v0.4.0`, `go build` succeeded (rc=0). A go.work replace overrides module replaces of the same path. The demo app already does this by hand (`sm-demo-app/go.work`: `replace git.golem15.com/golem15/summercms => ../summercms.go` plus three plugin replaces with the comment "pgstripe, pgsub, presentations, quizzes and feedback replace these three modules with paths that only exist in their own development layouts").
|
||||
**Rule for the installer:** after adding a submodule plugin to go.work, read its go.mod replaces (`go mod edit -json` → `Replace[]`). For every replaced module path `M` that the workspace does not already replace in go.work, add a go.work replace: to the app's own local replace target when the app's go.mod replaces `M` (framework dev layout), to the workspace dir of `M` when `M` is another installed submodule plugin, otherwise to `M@<version the app's go.mod requires>`. Whenever the app's framework requirement changes (`plugin:update`, update script), this pin must be refreshed. Document it as a pitfall.
|
||||
|
||||
### F3. The "app framework version" must be the effective one
|
||||
`[VERIFIED: internal/build/stubs/new.tmpl:7,9]` `require {{.Framework}} v0.0.0` and `replace {{.Framework}} => {{.Replace}}`. A `summer new` app satisfies no `framework: vX` minimum by its go.mod line. Other apps require real tags (`sm-demo-app/go.mod:23` `git.golem15.com/golem15/summercms v0.5.1`) but may still replace locally.
|
||||
**Algorithm:** `effectiveFramework(app)`. If the app's go.mod (or go.work) replaces the framework with a local directory, run `git -C <dir> describe --tags --abbrev=0 --match 'v[0-9]*'`. That is the newest tag reachable from the checkout's HEAD, so HEAD has at least its features. If that fails (no git, no tags), fall back to the go.mod require version and warn. Otherwise use the go.mod `require` version. Bake the same value into `plugins.gen.go` so cabana compares updates against what was actually built. At runtime `bonfire.BuildVersion().Framework` reports the go.mod require (`v0.0.0` for such apps) `[VERIFIED: modules/bonfire/version.go:31-51]`.
|
||||
|
||||
### F4. `ConfirmProceed` auto-confirms off a terminal
|
||||
`[VERIFIED: modules/bonfire/helpers.go:144-149]`: "Off a terminal, with -n or -q and in bonfire.Call it returns true without asking". D-09 requires the opposite. Use `bonfire.Interactive(out)` (`helpers.go:58`): interactive means `out.Confirm(question, false)`; non-interactive without `--yes` returns an error with a hint naming `--yes`.
|
||||
|
||||
### F5. Module installs and directory resolution work with stock Go
|
||||
- `go get git.golem15.com/golem15/sm-menu-plugin@v0.1.1` inside a go.work workspace succeeded and edited the cwd module's go.mod (`sm-menu-plugin v0.1.1 // indirect`, `summercms v0.4.0 // indirect`). The plugin's own `replace … => ../summercms.go` was ignored, as expected for a non-main module.
|
||||
- `GOPROXY=https://proxy.golang.org,direct` (local default) and `https://proxy.golang.org/git.golem15.com/golem15/sm-menu-plugin/@v/list` returned `v0.1.0 v0.1.1`. The public proxy and sum.golang.org already serve these modules, so users need no GOPRIVATE setup for public plugins.
|
||||
- `go list -e -json=ImportPath,Dir,Module <import paths…>` in `sm-demo-app` (0.37 s for 3 plugins) returned `Dir` and `Module{Path,Version,Replace,Main,Dir}` for a workspace module (`Main: true`), a go.work-replaced module (`Version: v0.1.13`, `Replace.Dir`) and the nested package `…/sm-pgstripe-plugin/subscriptions` (`Dir` = the subdirectory, `Module.Dir` = repo root). `go list -m -json` of a cached module gives `Dir: /home/jin/go/pkg/mod/git.golem15.com/golem15/sm-menu-plugin@v0.1.1`.
|
||||
- `[VERIFIED: internal/build/leaf.go:22-49]` `checkModelsLeaf` skips plugins whose dir is not found via `localModuleDir`/`defaultPluginDir` or is outside the app root. Module-cache plugins are skipped silently, which is correct: the leaf rule is a plugin-author check. **No change is needed in leaf.go.** `registry.go` (`refreshRegistry`) runs only from `make:*` on local plugins; it must never write into the module cache (read-only files).
|
||||
- **Conclusion:** use one helper, `resolvePluginDirs(appDir, manifest)` → `go list -e -json` over all `summer.yaml` import paths (with `applyGoWork`). The metadata step (D-06/D-07) uses it. The helper must tolerate failures (warn, ID-only metadata) so `summer build` never fails because of metadata.
|
||||
|
||||
### F6. summer.yaml `module:` is an import path, and one repo can hold two plugins
|
||||
`[VERIFIED: sm-demo-app/summer.yaml]` `- id: golem15.pgstripesubscriptions` / `module: git.golem15.com/golem15/sm-pgstripe-plugin/subscriptions` (no `subscriptions/go.mod`). The `sm-pgstripe-plugin` repo ships two plugins. **plugin.yaml must be per plugin package directory**, not per repo: `subscriptions/plugin.yaml` next to `subscriptions/plugin.go`. Each file carries the go.mod module path (what `go get` takes) and the package import path (what `summer.yaml` lists). The ingester finds them with `git ls-tree -r --name-only <tag>` filtered to `(^|/)plugin.yaml$`. Requires within the same module carry no separate minimum, because they ship in the same release.
|
||||
|
||||
### F7. README H1 is the plugin ID in 13 repos
|
||||
`[VERIFIED: head of each README.md]` H1 is `# golem15.<name>` in faq, journal, menu, mfa, oauthserver, paymentgateway, paymentgatewaysubscriptions, pgstripe, seo, sitemap, sociallogin, translate, user (and `pgstripe/subscriptions/README.md`). It is a human name in chatbots ("Chat bots"), chatchannels, chat, chatvideo, forum, herald, userfriends ("Friends"). The second paragraph is one sentence in all of them.
|
||||
**Recommendation:** when H1 equals the ID, `plugin:manifest` fails with a hint to pass `--name="…"` (persisted per D-03), rather than publishing "golem15.journal" as a display name or inventing a title-case transform. Rollout then needs one `--name` per such repo.
|
||||
|
||||
### F8. Go.mod requirements that cannot be installed as modules
|
||||
`sm-chatbots-plugin/go.mod:6` requires `git.golem15.com/golem15/sm-chat-plugin v0.0.0` (with a local replace). A consumer `go get` would fail on `v0.0.0`. `sm-userfriends-plugin/go.mod:6` uses a pseudo-version (fetchable). `plugin:manifest` should report a plugin requirement at `v0.0.0` as an error ("not a released version"), and the ingester should reject such tags as invalid (D-14). Pseudo-versions: allow, with a warning.
|
||||
|
||||
### F9. Untagged and private repos
|
||||
Anonymous `git ls-remote https://git.golem15.com/golem15/<repo>.git` (this session) succeeded for every sibling plugin repo plus quizzes, presentations, golem and feedback. `sm-summercmsio-plugin` is private, which is fine: it is a site plugin, not listed. Tag counts on origin: chatbots 0, chatchannels 0, chatvideo 0, forum 0, herald 0, userfriends 0, seo 1, quizzes 1, others 2–6. Untagged repos are not published until tagged (D-14). The go-get meta is served: `<meta name="go-import" content="git.golem15.com/golem15/sm-user-plugin git https://git.golem15.com/golem15/sm-user-plugin.git">`.
|
||||
|
||||
### F10. summercms.io deploy constraints
|
||||
- `[VERIFIED: sm-summercmsio-app/deploy/nginx/summercms.io.conf]` `location ^~ /backend { return 404; }`, and `limit_except GET HEAD { deny all; }` on `/`, `/changelog`, `/changelog/`. POST is allowed only under `location ^~ /_journal/api/v1/`. The ingest-now endpoint needs its own nginx location; the summercms.io admin (presentation fields) is reached through the existing SSH tunnel, like `changelog-post.py backfill`.
|
||||
- `[VERIFIED: sm-summercmsio-app/config/queue.yaml]` `work_in_serve: false` ("The site has no jobs, so serve does not start a worker"). The hourly scheduled ingest never runs unless this becomes `true`, or a separate `queue:work` supervisor program is added.
|
||||
- `[VERIFIED: sm-summercmsio-app/deploy/supervisor/summercms-io-changelog.conf]` changelog Node SSR on `NITRO_PORT="3095"`, `NUXT_JOURNAL_API_BASE="http://127.0.0.1:8095/_journal/api/v1"`.
|
||||
- `[VERIFIED: sm-summercmsio-app/plugins/golem15/summercms/links_test.go:45]` `var nodePaths = []string{"/changelog"}`. A landing link to `/marketplace` must be added there, or `TestLandingLinks` will try to resolve it through the binary.
|
||||
- Release builds fail on drift of committed `main.go`/`plugins.gen.go` (`sm-summercmsio-app/scripts/build.sh:160`, `sm-demo-app/scripts/build.sh:82`). The D-06/D-07 changes regenerate both files in every app, so each app needs one regeneration commit after upgrading.
|
||||
|
||||
## Architecture Patterns
|
||||
|
||||
### System Architecture Diagram
|
||||
|
||||
```
|
||||
plugin repo (sm-x-plugin) summercms.io (Go binary + Postgres) Nuxt SSR (changelog app, :3095)
|
||||
───────────────────────── ───────────────────────────────────── ───────────────────────────────
|
||||
plugin.go, go.mod, README.md admin (SSH tunnel): add repo URL, presentation ─┐
|
||||
│ summer plugin:manifest │ │
|
||||
▼ ▼ │
|
||||
plugin.yaml (committed) ── tag vX.Y.Z ──► [ingest] hourly scheduled cmd ◄── POST /api/registry/v1/ingest (g15_ token, scope registry)
|
||||
│ go test: --check │ git ls-remote (changed?) → bare partial clone ▲
|
||||
│ │ per vX.Y.Z tag: ls-tree → plugin.yaml(s) → validate │ changelog-post.py publish --repo
|
||||
│ │ latest tag: README → breakwater.Render │ (after plugin:manifest --check)
|
||||
│ ▼
|
||||
│ registry tables (plugins, versions, presentation, attachments)
|
||||
│ │
|
||||
│ ▼
|
||||
│ GET /api/registry/v1/plugins[?q,category] | /{id} | /{id}/versions (CORS *, Cache-Control)
|
||||
│ │ │ │
|
||||
│ ▼ ▼ ▼
|
||||
│ summer CLI (dev machine) cabana (app backend) /marketplace pages (SSR, swr)
|
||||
│ plugin:install / update GET /plugins (installed + updates)
|
||||
│ │ GET /plugins/marketplace?q=
|
||||
│ ▼ │
|
||||
│ ┌──── ID ──► resolve newest compatible ───┐ ▼
|
||||
│ │ (effective framework, deps) │ boardwalk SPA: Plugins screen
|
||||
│ ▼ ▼ (Installed | Marketplace, copy command)
|
||||
│ module: go get mod@v --source / git URL: clone → submodule add
|
||||
│ │ → go.work use + go.work replace overrides
|
||||
│ └──────────────► summer.yaml + go.mod (+ go.work) ◄─┘
|
||||
│ │
|
||||
└────────── module cache / submodule ─────┤ summer build: go list -json → plugin dirs
|
||||
│ read plugin.yaml, git describe → plugins.gen.go PluginInfo
|
||||
▼
|
||||
app binary: party publishes PluginInfo → cabana Installed tab
|
||||
```
|
||||
|
||||
### Recommended Project Structure (new/changed files)
|
||||
|
||||
```
|
||||
summercms.go/
|
||||
├── modules/<new-module>/ # NEW public module (name: beach-themed, e.g. "kiosk" — planner's choice)
|
||||
│ ├── manifest.go # plugin.yaml types, Parse, Validate, Marshal (goccy), schema version
|
||||
│ ├── derive.go # Derive(dir): go.mod via `go mod edit -json`, README via goldmark, ID/Requires resolver
|
||||
│ ├── resolve_ast.go # const-resolving ID()/Requires() reader (F1)
|
||||
│ ├── check.go # Check(dir) -> stale diff; CheckT(t, dir) helper for plugin repo tests
|
||||
│ ├── client.go # registry client over fetchguard (List, Get, Versions), JSON types
|
||||
│ ├── version.go # strict vX.Y.Z compare, git-describe base-tag parsing
|
||||
│ └── README.md
|
||||
├── internal/build/
|
||||
│ ├── build.go # generatePluginsGen + generateMain: PluginInfo, effective framework version
|
||||
│ ├── plugindirs.go # NEW: go list -json resolver (F5), git describe (D-07)
|
||||
│ ├── install.go # NEW: InstallModule, InstallSource, InstallGitURL, Update; go.work overrides (F2)
|
||||
│ ├── scaffold.go # AddPlugin gains a require-version option; pluginIDFromGo -> shared resolver
|
||||
│ └── manifest.go # unchanged API (SaveManifest drops summer.yaml comments — see Pitfalls)
|
||||
├── cmd/summer/main.go # plugin:manifest, plugin:install, plugin:update commands
|
||||
├── modules/party/ # Info type + publish/lookup of baked metadata (additive)
|
||||
├── modules/cabana/ # plugins.go (handlers), admin_openapi.go annotations, permission + settings entry
|
||||
├── modules/phrasebook/backend/lang/{en,pl}/lang.yaml # strings for screen + permission
|
||||
├── admin/src/views/PluginsView.vue, admin/src/app/router.ts, admin/src/state/useSettings.ts
|
||||
├── admin/openapi/admin.json, admin/src/api/schema.d.ts (generated), modules/boardwalk/dist (rebuilt)
|
||||
└── docs/… # console/scaffolding.md, architecture/go-modules-and-workspaces.md, a new plugins/marketplace page
|
||||
|
||||
sm-summercmsio-app/
|
||||
├── plugins/golem15/summercms/ # site plugin (submodule sm-summercmsio-plugin)
|
||||
│ ├── models/, updates/, controllers/ (admin YAML), console/ (registry:ingest), classes/ingest/, controllers/api/registry.go
|
||||
│ └── token_scopes.go, admin_permissions.go, lang/
|
||||
├── config/queue.yaml (work_in_serve), config/http.yaml (cors_profiles.registry), config/plugins.yaml (optional)
|
||||
├── deploy/nginx/summercms.io.conf (/marketplace → :3095, POST ingest), deploy/supervisor (if separate worker)
|
||||
├── scripts/changelog-post.py (+ test_changelog_post.py)
|
||||
└── vue-summercmsio-app/changelog/app/pages/marketplace/… (+ nuxt.config routeRules, runtimeConfig.registry)
|
||||
```
|
||||
|
||||
### Pattern 1: plugin.yaml format (recommendation; D-04 makes it costly to change)
|
||||
Include a schema version from day one, so the ingester and old `summer` binaries can branch later. Keep the generated text and the overrides separate.
|
||||
|
||||
```yaml
|
||||
# Generated by summer plugin:manifest. Do not edit; rerun the command.
|
||||
schema: 1
|
||||
id: golem15.journal
|
||||
module: git.golem15.com/golem15/sm-journal-plugin
|
||||
package: git.golem15.com/golem15/sm-journal-plugin # import path summer.yaml lists; differs for nested plugins (…/subscriptions)
|
||||
name: Journal
|
||||
description: The compiled journal plugin for SummerCMS applications: translatable posts and categories, tags, a settings singleton, and the tables a host migrate creates for a blog.
|
||||
framework: v0.5.1 # go.mod require of git.golem15.com/golem15/summercms
|
||||
requires:
|
||||
golem15.translate: v0.1.0 # Requires() IDs, minimum from go.mod require of that plugin's module
|
||||
overrides: # D-03: persisted --name/--description; absent when unused
|
||||
name: Journal
|
||||
```
|
||||
- `[ASSUMED]` field names (`schema`, `package`, `overrides`) are recommendations; the D-02/D-05 fields (`id`, module, `name`, `description`, `framework`, `requires`) come from CONTEXT.
|
||||
- Values in the example: `framework: v0.5.1` and `golem15.translate: v0.1.0` are quoted from `[VERIFIED: sm-journal-plugin/go.mod:6,8]` (`git.golem15.com/golem15/sm-translate-plugin v0.1.0`, `git.golem15.com/golem15/summercms v0.5.1`). The ID and Requires come from `[VERIFIED: sm-journal-plugin/plugin.go:44,46]` (`return "golem15.journal"`, `return []string{"golem15.translate"}`). The description is the README's second paragraph. Note that `name: Journal` is a `--name` override, because that README's H1 is `# golem15.journal` (F7).
|
||||
- Emit with `goccy/go-yaml` `Marshal` of an ordered struct (deterministic) plus the header comment. `--check` regenerates and compares bytes.
|
||||
- A requirement ID whose module is not in go.mod (soft dependency) gets an empty minimum (`""` = any). The ID→module mapping comes from resolving each required module's directory (`go list -m -json`) and reading its plugin ID with the F1 resolver.
|
||||
|
||||
### Pattern 2: Const-resolving ID reader (F1)
|
||||
```go
|
||||
// Source: pattern derived from internal/build/scaffold.go:360-392 (pluginIDFromGo), extended.
|
||||
// resolveString evaluates expr to a string constant without running code.
|
||||
func (r *resolver) resolveString(pkgDir string, file *ast.File, expr ast.Expr) (string, error) {
|
||||
switch e := expr.(type) {
|
||||
case *ast.BasicLit: // "golem15.journal"
|
||||
if e.Kind != token.STRING { return "", errNotConst }
|
||||
return strconv.Unquote(e.Value)
|
||||
case *ast.Ident: // PluginID -> package-level const in pkgDir
|
||||
return r.packageConst(pkgDir, e.Name)
|
||||
case *ast.SelectorExpr: // payments.PluginID, pgsub.PluginID
|
||||
imp := importPathFor(file, e.X) // from file.Imports, honouring named imports
|
||||
dir, err := r.dirForImport(imp) // same module: module root + suffix; else `go list -m -json` Dir
|
||||
if err != nil { return "", err }
|
||||
return r.packageConst(dir, e.Sel.Name)
|
||||
}
|
||||
return "", errNotConst // D-02 error names the function: "plugin.go: ID() must return a string literal or constant"
|
||||
}
|
||||
```
|
||||
`Requires()` accepts `nil`, or a `*ast.CompositeLit` of `[]string` whose elements each go through `resolveString`.
|
||||
|
||||
### Pattern 3: Plugin directory resolution (F5) and metadata baking (D-06/D-07)
|
||||
```go
|
||||
// go list -e -json=ImportPath,Dir,Module,Error <every summer.yaml module> in appDir, with applyGoWork(cmd, appDir).
|
||||
type listedPkg struct {
|
||||
ImportPath string
|
||||
Dir string
|
||||
Module *struct {
|
||||
Path, Version, Dir string
|
||||
Main bool
|
||||
Replace *struct{ Path, Dir, Version string }
|
||||
}
|
||||
}
|
||||
```
|
||||
Per plugin:
|
||||
- `plugin.yaml` is read from `Dir`; it is optional.
|
||||
- Version: if `Module.Main` (workspace module) or `Module.Replace` has a local `Dir`, run `git -C <Module.Dir> describe --tags --match 'v[0-9]*'` only when `git -C <Module.Dir> rev-parse --show-toplevel` equals `Module.Dir` (its own repo or submodule). Otherwise it is an in-app plugin with no version. Without `--dirty` (see Pitfall 6). Else (module cache): `Module.Version`.
|
||||
- Source: `module` | `submodule` | `local`.
|
||||
|
||||
Generated additions to `plugins.gen.go` (and one line in `generateMain`). The type lives in `party`; adding it is additive:
|
||||
```go
|
||||
// PluginInfo is the plugin metadata summer build read from plugin.yaml and git.
|
||||
var PluginInfo = []party.Info{
|
||||
{ID: "golem15.journal", Module: "git.golem15.com/golem15/sm-journal-plugin", Name: "…", Description: "…", Version: "v0.2.5", Source: "module"},
|
||||
}
|
||||
// FrameworkVersion is the effective framework version the build used (F3).
|
||||
const FrameworkVersion = "v0.5.3"
|
||||
```
|
||||
`[VERIFIED: internal/build/build.go:125]` the generated main calls `party.Activate(app, PluginIDs)`; add a publish of the info (e.g. `party.Describe(app, PluginInfo, FrameworkVersion)`) before it. `[ASSUMED]` names `party.Info`/`party.Describe`.
|
||||
|
||||
### Pattern 4: Install resolution (D-08/D-09/D-12)
|
||||
```
|
||||
install(arg, flags):
|
||||
app := FindAppDir; m := LoadManifest; fw := effectiveFramework(app) // F3
|
||||
if arg is git URL (https://, ssh://, scp-like user@host:path; never starting with "-"):
|
||||
tmp := git clone -- <url> plugins/.summer-install-<rand> // GIT_ALLOW_PROTOCOL=https:ssh
|
||||
pick tag: newest vX.Y.Z whose `git show <tag>:<path>/plugin.yaml` framework <= fw; else default-branch HEAD with a warning
|
||||
checkout tag; read plugin.yaml(s); validate; target := plugins/<vendor>/<name>; rename tmp -> target
|
||||
deps := plugin.yaml requires (resolved via registry unless --offline; --offline: must already be in summer.yaml)
|
||||
else (ID[@vX.Y.Z]):
|
||||
if registry disabled or --offline: error with hint
|
||||
versions := GET /plugins/{id}/versions; choose pinned or newest with framework <= fw
|
||||
none compatible: error "golem15.x vA.B.C needs framework >= vF; update the framework (current vE)"
|
||||
deps := chosen.requires
|
||||
closure: for each dep not in summer.yaml -> choose newest compatible with framework AND >= dep minimum;
|
||||
dep already installed -> check installed version >= minimum, else error naming `summer plugin:update <dep>`
|
||||
print additions (id@version, mode); if !--yes: Interactive? Confirm : error "non-interactive run needs --yes" (F4)
|
||||
apply:
|
||||
module mode: one `go get mod1@v1 mod2@v2 …` (applyGoWork; MVS sees the full set), then append summer.yaml entries
|
||||
source mode: per plugin: git clone/checkout tag -> `git submodule add <url> <path>` (adopts the clone) ->
|
||||
`git submodule absorbgitdirs` -> AddPlugin-equivalent with require version = tag -> go.work replace overrides (F2)
|
||||
finish: print "run summer build"
|
||||
```
|
||||
- `git submodule add` adopts an existing clone ("Adding existing repo at 'plugins/golem15/menu' to the index"). `git submodule absorbgitdirs` then moved `.git` into `.git/modules/…`, and `git submodule status` shows `86564746… plugins/golem15/menu (v0.1.1)` (verified in scratch).
|
||||
- Not a git repo (`summer new --no-git`): `git submodule add` fails. Use a plain clone and a warning, or error with a hint; planner's choice.
|
||||
- `AddPlugin` today writes `-require=<mod>@v0.0.0` and a go.mod `replace` (`[VERIFIED: internal/build/scaffold.go:394-406]`). For registry `--source` installs, pass the real tag as the require version, so a `GOWORK=off` release build (`sm-demo-app/scripts/build.sh` uses `GOWORK=off … go build`) and pin checks match. Keep `plugin:add`'s `v0.0.0` default unchanged.
|
||||
- `plugin:update [id]`: modules use `go get mod@newest-compatible`; submodules use `git -C dir fetch --tags`, `git checkout <tag>` and `go mod edit -require=mod@tag`. Without an ID: every summer.yaml plugin that resolves (registry ID, or a git-URL submodule via its own remote's tags). Report the skipped ones and exit 0 unless one that resolved failed (discretion).
|
||||
|
||||
### Pattern 5: cabana Plugins endpoints (D-17/D-19)
|
||||
- Routes inside the existing `r.GroupRaw(api, []string{"backend", "locale.from-principal"}, …)` block (`[VERIFIED: modules/cabana/http.go:316-339]`): `g.Get("/plugins", s.pluginsInstalled)` and `g.Get("/plugins/marketplace", s.pluginsMarketplace)`. Both are two-segment literal paths, so they do not collide with `/{vendor}/{plugin}/{controller}` (three segments) or `/settings/{code}`.
|
||||
- Permission: existing framework permissions use the `backend.*` namespace: `[VERIFIED: modules/cabana/backend_screens.go:18,21]` `PermissionManageUsers = "backend.manage_users"`, `PermissionManageRoles = "backend.manage_roles"`, declared with `Tab: "backend::lang.permissions.tab_system"` (`backend_screens.go:84-85`), no `Roles`. Recommend `backend.view_plugins` (D-19 allows the exact key) declared in `backendScreens.Permissions()`. Superusers pass every check (`modules/cabana/permissions_test.go:31` superuser case allows `a.b.c`), so "granted to superusers by default" needs no code.
|
||||
- Settings link: `pact.SettingsItem` link entries only target an admin **controller** (`[VERIFIED: modules/pact/capabilities.go:881-898]`), and the SPA maps `entry.controller` to a list route (`admin/src/state/useSettings.ts:34-39`). A custom view needs an additive field, e.g. `Path string` on `cabana.SettingsEntry` (`modules/cabana/navigation.go:23-35`) set by cabana's own entry, with the SPA's `settingsPath` preferring `entry.path`. Do not add it to `pact` unless plugins need it.
|
||||
- Registry calls: `fetchguard.NewClient(fetchguard.Policy{Mode: fetchguard.TrustedMode, MaxBytes: …}, app.Config)` (no redirects, so `plugins.registry_url` must be the final origin). Timeout of about 3–5 s, in-process TTL cache (list/search about 5 min, per-ID versions about 15 min), concurrency limit of about 4 for per-installed-plugin version lookups. Registry errors give HTTP 200 with `registry: {status: "unavailable"}` so the Installed tab still renders.
|
||||
- Update rule: `latest := newest version with framework <= FrameworkVersion`; compare it with the installed base tag (strip `-N-gHASH` from `git describe`); `update_available = latest > base`.
|
||||
- swag annotations in `admin_openapi.go`, then `scripts/check-admin-openapi.sh`; update `TestPhase09PermissionMatrix`/`TestPhase09ContractInventory` (`modules/cabana/phase09_contract_test.go`, `security_coverage_test.go`), which fail when routes and annotations diverge.
|
||||
|
||||
### Pattern 6: boardwalk view
|
||||
- New route in `admin/src/app/router.ts`, e.g. `{ path: '/system/plugins', name: 'plugins', component: PluginsView, meta: { shell: true } }`, placed before the `/:vendor/:plugin/:controller` patterns. A two-segment path cannot collide with them anyway.
|
||||
- Tabs: `reka-ui` is already a dependency (`admin/package.json` `"reka-ui": "2.9.10"`), so use its Tabs primitives. The copy button uses the existing pattern in `admin/src/components/account/ApiTokenCreateDialog.vue:260-263` (`navigator.clipboard.writeText`, with a fallback when `navigator.clipboard` is missing).
|
||||
- Data through the typed `api` client (`admin/src/api/client.ts`) only. Render registry text as text; do not render README HTML in the admin in v1.
|
||||
- Strings in `modules/phrasebook/backend/lang/{en,pl}/lang.yaml`. Then `npm --prefix admin run build`, commit `modules/boardwalk/dist`, `scripts/check-admin-dist.sh`.
|
||||
|
||||
### Pattern 7: summercms.io registry (site plugin `golem15.summercms`)
|
||||
- Follow `sm-faq-plugin` as the template: `updates/registry.go` `Register`/`All`, raw-SQL `CREATE TABLE` migrations with IDs like `202610110001_create_golem15_faq_faqs`, `pact.HasMigrations`/`HasModels`/`HasAdminControllers`/`HasPermissions`/`HasNavigation`, embedded admin YAML (`[VERIFIED: sm-faq-plugin/updates/*.go, admin*.go]`). Tables: `golem15_summercms_registry_plugins`, `golem15_summercms_registry_versions` (schema is discretion, see below).
|
||||
- Ingest auth mirrors the Journal API: `cabana.ResolveBackend(app, r, "golem15.summercms.*")` + `cabana.Allows(pr, []string{"golem15.summercms.registry_ingest"})` (`[VERIFIED: sm-journal-plugin/controllers/api/auth.go:83-104]`), plus a `pact.HasTokenScopes` scope (`sm-journal-plugin/token_scopes.go` pattern) such as `registry`, with `Permissions: []string{"golem15.summercms.*"}` and `Methods: ["POST"]`.
|
||||
- Scheduled ingest: a console command `summercms:registry-ingest [id]` + `Schedule() []pact.ScheduledCommand{{Command: "summercms:registry-ingest", Cadence: pact.Every(time.Hour)}}` (README example shape `[VERIFIED: modules/conga/README.md:119-125]`). Requires `queue.work_in_serve: true` (F10).
|
||||
- Public API: `r.Group("/api/registry/v1", surf.Use("throttle:<bucket>", "cors:registry"), …)` with a `http.cors_profiles.registry` profile (`allowed_origins: ['*']`, GET/HEAD, no credentials) per `[VERIFIED: modules/surf/README.md:31]` named CORS profiles. Set `Cache-Control: public, max-age=300` (discretion) and an ETag. The site plugin's `GET /` catch-all is less specific, so `/api/registry/v1/…` wins.
|
||||
|
||||
### Proposed registry API shape (D-16 is one-way: decide carefully)
|
||||
`[ASSUMED]`: this is a design proposal for user/planner sign-off, not an existing contract.
|
||||
```jsonc
|
||||
// GET /api/registry/v1/plugins?q=journal&category=content
|
||||
{ "data": [ { "id": "golem15.journal", "name": "Journal", "description": "…", "module": "git.golem15.com/golem15/sm-journal-plugin",
|
||||
"package": "git.golem15.com/golem15/sm-journal-plugin", "repository": "https://git.golem15.com/golem15/sm-journal-plugin.git",
|
||||
"category": "content", "featured": true, "icon_url": "https://summercms.io/storage/uploads/…",
|
||||
"latest_version": "v0.2.5", "latest_framework": "v0.5.1", "updated_at": "2026-10-29T10:00:00Z" } ],
|
||||
"meta": { "total": 1 } }
|
||||
// GET /api/registry/v1/plugins/{id} -> { "data": { …list fields, "readme_markdown": "…", "readme_html": "…", "screenshots": [{"url","caption"}] } }
|
||||
// GET /api/registry/v1/plugins/{id}/versions -> { "data": [ { "version": "v0.2.5", "framework": "v0.5.1",
|
||||
// "requires": { "golem15.translate": "v0.1.0" }, "module": "…", "package": "…", "tagged_at": "…" } ] } // newest first
|
||||
```
|
||||
Forward-compatibility rules: always use an envelope object (`data`, `meta`); clients ignore unknown fields; only additive changes within v1; 404 → `{"error": {...}}` for unknown IDs. Per-version `module`/`package` let a later major version change the module path (`/v2`).
|
||||
|
||||
### Anti-Patterns to Avoid
|
||||
- **Running plugin code to read its ID** (`go run` a probe that blank-imports the plugin): it executes init() of untrusted code and contradicts D-02's static intent.
|
||||
- **Writing into the module cache** (e.g. `refreshRegistry` on a cached plugin): the files are read-only and shared.
|
||||
- **`bonfire.ConfirmProceed` for the install prompt** (F4).
|
||||
- **Comparing `git describe` strings with semver**: a pre-release sorts below its base tag.
|
||||
- **Calling the registry from the SPA directly**: violates D-17 and CSP.
|
||||
- **Prerendering /marketplace into the embedded landing**: new plugins would need a binary release to appear.
|
||||
- **Calling `go get` with registry-provided strings without validation**: validate the module path (`validateModulePath`, which rejects a leading `-`, `[VERIFIED: internal/build/manifest.go:270-292]`) and the version (`^v\d+\.\d+\.\d+$`) first.
|
||||
|
||||
## Don't Hand-Roll
|
||||
|
||||
| Problem | Don't Build | Use Instead | Why |
|
||||
|---------|-------------|-------------|-----|
|
||||
| go.mod parsing (requires, replaces) | Extending the line scanner `parseGoMod` (`scaffold.go:494-535`) to handle `require (` blocks | `go mod edit -json` | Verified structured output; handles every go.mod syntax form |
|
||||
| Locating plugin source dirs | Guessing `plugins/<vendor>/<name>` or `$GOMODCACHE/<path>@<ver>` (case-encoding of uppercase letters!) | `go list -e -json` / `go list -m -json` | Handles workspace, replace, nested packages, module-cache path escaping |
|
||||
| Adding go.work entries | Text edits of go.work | `go work use` (already `runGoWorkUse`), `go work edit -replace=` | Verified override behaviour |
|
||||
| Markdown → HTML for READMEs | goldmark + a custom policy | `breakwater.Render` | Existing decided policy (GFM, safe, nofollow) |
|
||||
| Outbound HTTP to the registry | `http.DefaultClient` | `fetchguard.Client` (TrustedMode) | Size caps, timeouts, no redirects, test transport seam |
|
||||
| Submodule plumbing | Hand-writing `.gitmodules` | `git submodule add` + `absorbgitdirs` | Verified adoption of an existing clone |
|
||||
| YAML writing | `fmt` with manual quoting (descriptions contain `:` and backticks) | `goccy/go-yaml` `Marshal` | Correct quoting; deterministic field order |
|
||||
| Rate limiting/CORS on the public API | Custom middleware | `throttle:` + `cors:<profile>` surf middleware | Built-in and tested |
|
||||
|
||||
**Key insight:** every hard part (module resolution, MVS, workspace overrides, quoting, sanitizing) already exists either in the Go toolchain or in a framework module. The phase is integration work. Custom parsers in this area break on the first unusual go.mod or README.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Pitfall 1: const IDs (F1)
|
||||
**What goes wrong:** `plugin:manifest` and git-URL installs fail for 13 public plugins.
|
||||
**How to avoid:** the static const resolver, shared with `pluginIDFromGo`.
|
||||
**Warning signs:** "does not return a plugin id" on `sm-mfa-plugin`.
|
||||
|
||||
### Pitfall 2: submodule installs inherit dev replaces (F2)
|
||||
**What goes wrong:** `go build` fails with "replaced by ./plugins/golem15/summercms.go … no such file".
|
||||
**How to avoid:** go.work replace overrides for every local replace in an installed submodule's go.mod; refresh the framework pin when the framework requirement moves.
|
||||
|
||||
### Pitfall 3: `v0.0.0` framework requirement in `summer new` apps (F3)
|
||||
**What goes wrong:** every install reports "needs a newer framework", or the version check silently passes against a meaningless number.
|
||||
**How to avoid:** `effectiveFramework` via `git describe --abbrev=0` of the replaced checkout. Bake it for cabana.
|
||||
|
||||
### Pitfall 4: summer.yaml comments are lost
|
||||
`[VERIFIED: internal/build/manifest.go:146-181]` `SaveManifest` rewrites the file from `module`, `binary`, `plugins`, `build.tags`. `sm-demo-app/summer.yaml` and `sm-bm-app/summer.yaml` carry comments ("# summercms_mcp links the MCP endpoint …") that `plugin:add`/`plugin:install` would delete. Either append plugin entries with a comment-preserving edit (goccy AST, or text insertion after the last `plugins:` item), or document it. Prefer preserving: the CLI is meant for real apps.
|
||||
|
||||
### Pitfall 5: `go get` marks new plugins `// indirect`
|
||||
Until `plugins.gen.go` imports them, the go.mod entry is `// indirect` (verified). It is harmless for builds. `plugin:install` could run `summer build` (regenerates imports) and then `go mod tidy`, or just tell the user to run `summer build`. Do not run `go mod tidy` before plugins.gen.go imports the module: tidy would remove it.
|
||||
|
||||
### Pitfall 6: `git describe` churn in committed `plugins.gen.go`
|
||||
Apps commit `plugins.gen.go`, and release builds refuse drift (F10). Baking `git describe` output changes the file whenever a plugin submodule moves. That is acceptable because submodule bumps are commits that should include the regenerated file. Never use `--dirty`: an uncommitted change inside a submodule would make `summer build` rewrite a tracked file and break release builds. Give an in-app plugin (not its own repo) an empty version, not the app's describe.
|
||||
|
||||
### Pitfall 7: proxy/sumdb immutability
|
||||
proxy.golang.org caches every fetched version forever. A moved or retagged plugin tag gives users a checksum mismatch. Plugin tags must never be moved. Add this to the release docs.
|
||||
|
||||
### Pitfall 8: framework-version inflation from the `--check` test helper
|
||||
If plugin repos import a new framework helper in `go test`, their go.mod must require the framework release that contains it. By D-02 that becomes their `framework:` minimum (Go's MVS would enforce it anyway). See Open Question 2.
|
||||
|
||||
### Pitfall 9: summercms.io jobs never run
|
||||
`queue.work_in_serve: false` (F10). Without flipping it, or adding a `queue:work` supervisor program, the hourly ingest does nothing in production while every local test passes.
|
||||
|
||||
### Pitfall 10: nginx blocks the ingest POST and the admin
|
||||
Add `location = /api/registry/v1/ingest { limit_except POST { deny all; } proxy_pass http://127.0.0.1:8095; … }`. Keep the presentation admin behind the SSH tunnel. `scripts/check-deploy.sh` validates nginx/supervisor syntax.
|
||||
|
||||
### Pitfall 11: CORS preflight
|
||||
nginx `limit_except GET HEAD` denies OPTIONS. Simple GETs without custom headers need no preflight; keep the registry API GET-only and header-free for browsers.
|
||||
|
||||
### Pitfall 12: concurrent sessions in cabana and internal/build
|
||||
Phases 20 and 32.1 are in flight. Check `git status`/`git log -- modules/cabana` before editing, and stage explicit paths.
|
||||
|
||||
### Pitfall 13: ingestion trusts plugin.yaml identity
|
||||
A repo could claim another plugin's ID or a module path that is not its own. At ingest: (a) ID unique across listed repos (first-listed wins; conflict is an error on the newer listing); (b) `module` must equal the repo URL's host+path without `.git` (core-plugins rule: module path = repo path), so `go get <module>` fetches the same code the listing shows.
|
||||
|
||||
## Code Examples
|
||||
|
||||
### Hardened git for the ingester (stdlib exec)
|
||||
```go
|
||||
// Source: env semantics from https://git-scm.com/docs/git (GIT_ALLOW_PROTOCOL, GIT_TERMINAL_PROMPT,
|
||||
// GIT_CONFIG_NOSYSTEM, GIT_CONFIG_GLOBAL), pattern of internal/build/new.go runGit.
|
||||
func gitCmd(ctx context.Context, dir string, args ...string) *exec.Cmd {
|
||||
cmd := exec.CommandContext(ctx, "git", args...)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(minimalEnv(), // PATH, HOME=<tmp>, no inherited GIT_*
|
||||
"GIT_TERMINAL_PROMPT=0",
|
||||
"GIT_ALLOW_PROTOCOL=https", // ingester: https only (tests may add "file")
|
||||
"GIT_CONFIG_NOSYSTEM=1",
|
||||
"GIT_CONFIG_GLOBAL=/dev/null",
|
||||
)
|
||||
return cmd
|
||||
}
|
||||
// clone: git clone --bare --filter=blob:none --no-tags? (keep tags) -- <url> <tmp> (fallback without --filter if the server refuses)
|
||||
// tags: git for-each-ref --format=%(refname:short)%09%(objectname)%09%(creatordate:iso-strict) refs/tags
|
||||
// files: git ls-tree -r --name-only <tag> ; git show <tag>:<path>/plugin.yaml ; git show <latest>:<path>/README.md
|
||||
// cheap change check before cloning: git ls-remote --tags -- <url>
|
||||
```
|
||||
Never check out the tree, never `--recurse-submodules`, cap blob sizes read (`io.LimitReader`), and remove the temp dir. Always put `--` before URLs and paths.
|
||||
|
||||
### CLI git (user's own credentials, private SSH allowed)
|
||||
Same pattern but keep the user's global config (credential helpers, `insteadOf`), set `GIT_TERMINAL_PROMPT=0` only when not interactive, and `GIT_ALLOW_PROTOCOL=https:ssh` (test seam for `file`).
|
||||
|
||||
### Effective framework version (F3)
|
||||
```go
|
||||
// app go.mod: go mod edit -json -> Require[] (summercms version), Replace[] (local dir?)
|
||||
if dir := localFrameworkReplace(app); dir != "" {
|
||||
out, err := exec.CommandContext(ctx, "git", "-C", dir, "describe", "--tags", "--abbrev=0", "--match", "v[0-9]*").Output()
|
||||
if err == nil { return strings.TrimSpace(string(out)), nil }
|
||||
}
|
||||
return requireVersion(app, "git.golem15.com/golem15/summercms"), nil
|
||||
```
|
||||
|
||||
### changelog-post.py hook (D-04, D-13)
|
||||
- Before posting a plugin release: require that `git -C REPO describe --exact-match HEAD` equals the tag and the worktree is clean, then run `summer plugin:manifest --check <dir>`, using `summer` from PATH or else `go -C FRAMEWORK run ./cmd/summer plugin:manifest --check <abs REPO dir>` (so the command needs an optional `[dir]` argument). Do the same for nested plugin dirs (any `plugin.yaml` in the repo).
|
||||
- After a successful publish: `POST {SITE}/api/registry/v1/ingest` with `{"repository": <origin https URL>}` and the same token. A 403 (token lacks the new scope) or a network failure prints a warning ("the hourly ingest will pick it up") and does not fail the publish. Unit-test the new pure helpers in `scripts/test_changelog_post.py` (stdlib unittest; `call` is replaced with a failing stub there).
|
||||
|
||||
## State of the Art
|
||||
|
||||
| Old Approach | Current Approach | When Changed | Impact |
|
||||
|--------------|------------------|--------------|--------|
|
||||
| Plugins only as local dirs + `replace` (`plugin:add`) | Plugins as plain Go module requires, resolved by `go list` | This phase | Build steps must stop assuming `plugins/<vendor>/<name>` exists |
|
||||
| `go get` refused in workspace mode (old Go) | `go get` works in workspace mode, edits cwd module | Verified on Go 1.27.2 | No `GOWORK=off` dance needed for module installs |
|
||||
| README H1 = plugin ID (13 repos) | Human name via `--name` override | This phase | One-time `--name` per repo during rollout |
|
||||
|
||||
**Deprecated/outdated:** the `.planning/notes/core-plugins-own-repos.md` statement that plugin go.mod replaces the framework with `../../../../summercms.go` is outdated. The repos use `../summercms.go` (sibling layout) `[VERIFIED: sm-*-plugin/go.mod]`.
|
||||
|
||||
## Plan Decomposition (recommendation for the planner)
|
||||
|
||||
Repo/dependency order, tracer-first, few large plans, tests last (project rule):
|
||||
|
||||
1. **Manifest foundation + build metadata (summercms.go):** the new public module (plugin.yaml types, const-resolving reader, derive from go.mod/README, marshal, check, test helper), `summer plugin:manifest [--check] [--name] [--description] [dir]`, `pluginIDFromGo` switched to the resolver, `go list` plugin-dir resolver, D-06/D-07 baking + effective framework version, `party.Info`. Module README, root table row, docs. Smoke: generate plugin.yaml for journal (literal), mfa (const) and pgstripe/subscriptions (cross-module const).
|
||||
2. **Plugin repo rollout (each public `sm-*-plugin`):** generate and commit `plugin.yaml` (`--name` where H1 = ID), add the check test, fix `v0.0.0` plugin requirements where they block (F8), app pointer bumps as separate commits. Tags remain **user-gated**, and the framework tag that contains the helper must exist before plugin tags whose go.mod requires it (Open Question 2). This plan is mostly mechanical; it could be merged into plan 1 if the user prefers fewer plans.
|
||||
3. **Registry on summercms.io (sm-summercmsio-app, site plugin, vue app):** models/migrations, admin (repos + presentation, attachments), ingester service + scheduled command + ingest endpoint + token scope, public API (CORS profile, cache), config (`work_in_serve`, CORS profile), nginx/supervisor, `/marketplace` pages in the changelog SSR app, landing header link + `nodePaths`, changelog-post.py hooks. Smoke: ingest a local fixture repo, `curl` the API.
|
||||
4. **Install/update CLI (summercms.go):** registry client usage, `plugin:install` (ID module / `--source` / git URL), `plugin:update`, `plugins.registry_url` config read, `--yes`/`--offline`, go.work overrides (F2), summer.yaml comment preservation, docs. Smoke against an `httptest` registry.
|
||||
5. **Admin Plugins screen (summercms.go):** cabana endpoints + permission + settings link + swag/openapi regen + contract tests update + PluginsView (tabs, copy) + lang + dist rebuild + gate.
|
||||
6. **Unit tests (all repos):** see the Validation Architecture.
|
||||
|
||||
Plans 4 and 5 both depend only on plan 1 and the API shape (fixtures), not on plan 3 being deployed. They can follow plan 1 in either order; plan 5 touches cabana, where other sessions are active.
|
||||
|
||||
## Assumptions Log
|
||||
|
||||
| # | Claim | Section | Risk if Wrong |
|
||||
|---|-------|---------|---------------|
|
||||
| A1 | Extending D-02 to resolve package-level and cross-package string constants is acceptable to the user | F1, Pattern 2 | If refused, 13 core plugins need `plugin.go` edits (literal returns) |
|
||||
| A2 | plugin.yaml field names `schema`, `package`, `overrides` and one file per plugin package dir | Pattern 1, F6 | Format is costly to change after rollout (D-04) |
|
||||
| A3 | Registry JSON shape in "Proposed registry API shape" | Pattern 7 | One-way after release (D-16) |
|
||||
| A4 | Marketplace pages live in the existing changelog Node SSR app, URL `/marketplace/<vendor>/<name>` | Architecture, Alternatives | A separate app/URL scheme changes nginx and supervisor work |
|
||||
| A5 | `git` is installed on the production server (rome) for the ingester | Environment | Ingest fails in production; needs install or a go-git decision note |
|
||||
| A6 | Permission key `backend.view_plugins` and SPA path `/system/plugins` | Pattern 5/6 | Cosmetic; D-19 left the exact key open |
|
||||
| A7 | Ingest endpoint uses a new token scope (`registry`); the publish token may lack it, so ingest-now is best-effort | Pattern 7, changelog hook | User must mint a token with the new scope for instant ingestion (never done by the agent) |
|
||||
| A8 | `party.Info` / `party.Describe` names and placement in `party` | Pattern 3 | Naming only |
|
||||
| A9 | Which repos are "public plugins" to list (quizzes, presentations, golem, feedback exist publicly but have no sibling checkout; six chat-family repos are untagged) | F9 | Wrong listing scope |
|
||||
| A10 | Proposed new module name (e.g. `kiosk`) | Structure | Naming only |
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Constant IDs vs D-02's "literal returns" (A1).**
|
||||
- What we know: 13/22 plugins return `PluginID`; two use cross-package constants; `plugin:add` already fails on them.
|
||||
- Recommendation: approve the static const resolver; it keeps D-02's "no code execution" intent.
|
||||
2. **How the plugin repos' `--check` runs in `go test` without inflating framework minimums.**
|
||||
- Options: (a) import a framework helper. Each plugin's go.mod then requires the release containing it, which becomes its `framework:` minimum (likely the phase's release tag), and plugin tags wait for that framework tag. (b) The test shells out to `summer` from PATH or `go run git.golem15.com/golem15/summercms/cmd/summer@<pinned>` (the framework go.mod has no replace, so `go run pkg@version` works) and skips under `-short` or when unavailable. No go.mod change, but weaker and network-bound.
|
||||
- Recommendation: (a) for simplicity, provided the user accepts that every listed plugin then requires the new framework release. Otherwise (b) plus the mandatory check in `changelog-post.py`.
|
||||
3. **Which repos are listed at launch (A9)**, and whether untagged repos get a first tag in this phase (tags are user-gated).
|
||||
4. **Ingest token scope (A7)**: new `registry` scope (user mints a token), or accept the existing `journal`-scoped token for the ingest endpoint (lower friction, scope semantics blur).
|
||||
5. **`summer.yaml` comment preservation**: implement now (recommended) or document the loss.
|
||||
6. **plugin:install in a non-git app directory** (`summer new --no-git`) with `--source`: clone without a submodule, or refuse.
|
||||
|
||||
## Environment Availability
|
||||
|
||||
| Dependency | Required By | Available | Version | Fallback |
|
||||
|------------|------------|-----------|---------|----------|
|
||||
| Go toolchain | everything Go; `go list/get/mod edit/work edit` | ✓ | go1.27.2 | — |
|
||||
| git | submodule installs, `git describe`, ingester, tests with fixture repos | ✓ (local) | 2.56.0 | — |
|
||||
| git on rome (production) | ingester | unknown | — | install git, or a go-git decision note (A5) |
|
||||
| Network to git.golem15.com, proxy.golang.org | install integration checks, rollout | ✓ | anonymous HTTPS verified | `GOPROXY=file://` fixture for tests |
|
||||
| Docker | testcontainers Postgres (registry DB tests, cabana tests) | ✓ (`docker ps` ok in this session) | client 29.9.1 | `sudo -u golem` if the socket is denied; `-short` skips |
|
||||
| Node / npm | admin SPA build/tests | ✓ | node v24.22.0, npm 12.2.1 | — |
|
||||
| pnpm | vue-summercmsio-app | ✓ | 12.14.0 | — |
|
||||
| Python 3 | changelog-post.py tests | ✓ | 3.14.8 | — |
|
||||
| swag | openapi gate | ✓ via `go run …@v1.16.6` (network) | v1.16.6 | — |
|
||||
|
||||
**Missing dependencies with no fallback:** none locally. Production `git` is unverified (A5): plan a deploy-time check (`command -v git` in DEPLOY.md / smoke) and fail ingestion with a clear error if `exec.LookPath("git")` fails.
|
||||
|
||||
## Validation Architecture
|
||||
|
||||
### Test Framework
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Framework | Go `testing` (+ testcontainers-go v0.44.0 for Postgres); Vitest 3.2.7 (+ browser mode) for admin; `node --test` for vue-summercmsio-app; stdlib `unittest` for scripts |
|
||||
| Config file | none for Go; `admin/vitest.config.ts`, `admin/vitest.browser.config.ts` |
|
||||
| Quick run command | `go test -short ./internal/build ./cmd/summer ./modules/<new-module> ./modules/party ./modules/cabana` |
|
||||
| Full suite command | `go vet ./... && go test ./... && npm --prefix admin run test && npm --prefix admin run typecheck && scripts/check-admin-openapi.sh --check && scripts/check-admin-dist.sh && summer docs:build --check` |
|
||||
|
||||
### Phase Requirements → Test Map
|
||||
| Req | Behavior | Test Type | Automated Command | File Exists? |
|
||||
|-----|----------|-----------|-------------------|-------------|
|
||||
| SC-1 | Derive: literal, same-pkg const, cross-pkg const (same module + other module), non-const → error naming function | unit (temp dirs, `go mod edit -json`; fixture module tree) | `go test -short ./modules/<new> -run TestDerive` | ❌ Wave 0 |
|
||||
| SC-1 | README H1/summary extraction; H1 == ID → `--name` hint; overrides persist across regeneration | unit | `go test -short ./modules/<new> -run TestReadme` | ❌ |
|
||||
| SC-1 | `--check` detects stale file; byte-stable output | unit | `go test -short ./modules/<new> -run TestCheck` | ❌ |
|
||||
| SC-1 | Validate rejects bad ID, bad module, `v0.0.0` plugin requirement, unknown schema | unit | `go test -short ./modules/<new> -run TestValidate` | ❌ |
|
||||
| SC-1 | `plugin:add` registers a const-ID plugin (regression for F1) | unit (temp app copy, like `TestMakeAndAddPluginOnHelloCopy`) | `go test -short ./internal/build -run TestAddPluginConstID` | ❌ |
|
||||
| SC-1 | Each plugin repo's own check | plugin repo test | `go -C ../sm-<name>-plugin test -run TestPluginManifest ./...` | ❌ per repo |
|
||||
| D-06/07 | plugins.gen.go includes PluginInfo/FrameworkVersion; go list resolution for workspace/replace/module-cache/nested; git describe only when own repo; no `--dirty` | unit (golden strings) + fixture git repo | `go test -short ./internal/build -run 'TestGeneratePluginInfo|TestPluginDirs|TestDescribe'` | ❌ |
|
||||
| SC-2 | Ingester: fixture bare repo (git init, tags with/without plugin.yaml, nested plugin) via `file://` (test-only protocol) → version rows, latest README, skips invalid tags, module/repo mismatch rejected, ID conflict rejected | integration (Docker Postgres) | `go -C ../sm-summercmsio-app/plugins/golem15/summercms test -run TestIngest ./...` | ❌ |
|
||||
| SC-2 | Registry API: list/search/category/detail/versions shapes, CORS header, Cache-Control, 404 envelope | integration (httptest + Postgres) | `… test -run TestRegistryAPI ./...` | ❌ |
|
||||
| SC-2 | Ingest endpoint: 401 without token, 403 wrong scope, 200 with scope | integration | `… test -run TestIngestEndpoint ./...` | ❌ |
|
||||
| SC-2 | changelog-post.py: ingest call + check invocation helpers, 403 tolerated | unit | `python3 -B scripts/test_changelog_post.py` | ✅ (extend) |
|
||||
| SC-2 | Marketplace pages render from fixture API | Node SSR smoke | `pnpm --dir vue-summercmsio-app test` (+ new marketplace fixture check) | ❌ |
|
||||
| SC-2 | Landing `/marketplace` link is a node path | unit (gated build) | `SUMMERCMS_REQUIRE_BUILD=1 go -C plugins/golem15/summercms test -run TestLandingLinks ./...` | ✅ (extend `nodePaths`) |
|
||||
| SC-3 | Resolution: newest compatible, pin, no compatible → names framework, transitive closure, installed-too-old dep, cycle guard | unit (pure, fake registry) | `go test -short ./internal/build -run TestResolveInstall` | ❌ |
|
||||
| SC-3 | Effective framework version: require vs local replace + describe | unit (fixture git repo) | `go test -short ./internal/build -run TestEffectiveFramework` | ❌ |
|
||||
| SC-3 | Module install end-to-end: `GOPROXY=file://` fixture proxy (stdlib `archive/zip` builds `@v/list,.info,.mod,.zip`), with `GOPROXY=file://<dir> GOSUMDB=off GOFLAGS=-modcacherw GOMODCACHE=<t.TempDir()>` set only in the test env `[ASSUMED: exact env set to confirm while writing the test]` | integration (no network) | `go test ./internal/build -run TestInstallModule` | ❌ |
|
||||
| SC-3 | `--source`/git-URL install: fixture repo via `file://`, submodule added, go.work use + replace override, `go build` passes | integration | `go test ./internal/build -run TestInstallSource` | ❌ |
|
||||
| SC-3 | Non-interactive without `--yes` errors; `--offline` with missing deps errors | unit (bonfire.Call root) | `go test -short ./cmd/summer -run TestPluginInstallPrompts` | ❌ |
|
||||
| SC-3 | Commands documented | existing gate | `go test ./cmd/summer -run 'TestCommandsDocumented|TestDocsTree'` | ✅ |
|
||||
| SC-4 | Installed endpoint: metadata, versions, update flag (describe base vs registry), registry down → still 200 | unit (fake registry via `fetchguard.WithTransport`) | `go test -short ./modules/cabana -run TestPluginsEndpoints` | ❌ |
|
||||
| SC-4 | Permission matrix + contract inventory include new routes | existing tests | `go test -short ./modules/cabana -run 'TestPhase09PermissionMatrix|TestPhase09ContractInventory'` | ✅ (update) |
|
||||
| SC-4 | PluginsView: tabs, search, copy button (clipboard stub), hidden Marketplace when registry disabled | vitest | `npm --prefix admin run test -- PluginsView` | ❌ |
|
||||
| SC-4 | OpenAPI and dist gates | scripts | `scripts/check-admin-openapi.sh --check && scripts/check-admin-dist.sh` | ✅ |
|
||||
|
||||
### Sampling Rate
|
||||
- **Per task commit:** `go vet ./... && go test -short <touched packages>` (+ `npm --prefix admin run test` for SPA tasks)
|
||||
- **Per wave merge:** `go test ./...` (Docker) in summercms.go; site plugin `go test ./...`; admin gates
|
||||
- **Phase gate:** full suite green in all touched repos, plus a `scripts/check-phase34.sh` (convention: `scripts/check-phaseN.sh`) before `/msd-verify-work`
|
||||
|
||||
### Wave 0 Gaps
|
||||
- [ ] `modules/<new>/…_test.go`: derive/readme/check/validate, client (fetchguard transport fake)
|
||||
- [ ] `internal/build/install_test.go`, `plugindirs_test.go`: fixture helpers to create a git repo with tags and a `file://` GOPROXY tree (stdlib only)
|
||||
- [ ] site plugin `TestMain` with testcontainers (must call `flag.Parse()` and honour `testing.Short()`)
|
||||
- [ ] `admin/tests/…PluginsView` spec
|
||||
- [ ] `scripts/check-phase34.sh`
|
||||
|
||||
## Security Domain
|
||||
|
||||
`security_enforcement` is not set in `.planning/config.json`, so it is enabled by default.
|
||||
|
||||
### Applicable ASVS Categories
|
||||
|
||||
| ASVS Category | Applies | Standard Control |
|
||||
|---------------|---------|-----------------|
|
||||
| V2 Authentication | yes (ingest endpoint) | `cabana.ResolveBackend` (backend JWT or `g15_` personal token) |
|
||||
| V3 Session Management | no new sessions | existing admin cookie session for cabana endpoints |
|
||||
| V4 Access Control | yes | `backend.view_plugins` (or chosen key) on cabana endpoints; `golem15.summercms.registry_ingest` + token scope on ingest; admin-only repo curation |
|
||||
| V5 Input Validation | yes | plugin.yaml schema validation (ID regex, `validateModulePath`, strict `vX.Y.Z`), git URL validation (scheme allow-list, no leading `-`), query params (`q` length cap, category slug regex), registry JSON validated by the CLI before any exec |
|
||||
| V6 Cryptography | no new crypto | sumdb verification of module installs stays on (do not set GONOSUMDB for public plugins) |
|
||||
| V10 Malicious code | partially | installing a plugin compiles third-party code into the binary. This is inherent and out of scope, but the CLI prints exactly what it will add and requires confirmation (D-09) |
|
||||
| V12 Files and Resources | yes | ingester: temp dirs, bare clones, no checkout/hooks/submodules, size caps; CLI never writes outside the app root (`underRoot`) |
|
||||
| V14 Configuration | yes | `plugins.registry_url` operator-trusted (TrustedMode), https by default, no redirects |
|
||||
|
||||
### Known Threat Patterns
|
||||
|
||||
| Pattern | STRIDE | Standard Mitigation |
|
||||
|---------|--------|---------------------|
|
||||
| Option/argument injection via repo URL or module/version strings into `git`/`go` | Tampering/Elevation | exec without shell, `--` before positional args, validators rejecting leading `-`, strict version regex |
|
||||
| `ext::`/`file::` transport abuse via an attacker-supplied git URL | Elevation | `GIT_ALLOW_PROTOCOL=https:ssh` (CLI) / `https` (ingester) `[CITED: git-scm.com/docs/git]` |
|
||||
| Malicious repo content during ingest (huge blobs, symlinks, hooks) | DoS/Tampering | bare partial clone, read only via `git show`/`ls-tree`, `io.LimitReader`, timeouts, no checkout |
|
||||
| README XSS on the marketplace | Tampering | `breakwater.Render` at ingest (strips raw HTML, http/https/mailto only, nofollow); Nuxt renders only stored sanitized HTML |
|
||||
| Registry poisoning (repo claims another ID or another module) | Spoofing | admin-curated listing; ID uniqueness; module path must equal repo path (Pitfall 13) |
|
||||
| Registry outage or slowness breaking the admin or CLI | DoS | cabana timeouts + cache + degraded 200; CLI clear error, `--offline` for git-URL installs |
|
||||
| Ingest endpoint abuse (forced clones) | DoS | token + permission, throttle bucket, per-repo single-flight, `ls-remote` change check before cloning |
|
||||
| Supply chain: moved tags | Tampering | sumdb catches mismatch; never move tags (Pitfall 7); submodule installs pin the commit in the parent repo |
|
||||
| Open CORS leaking private data | Information disclosure | registry API serves only public data, no credentials, GET only |
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (HIGH confidence)
|
||||
- Codebase files read this session: `internal/build/{manifest,build,leaf,registry,scaffold,artifact}.go`, `cmd/summer/main.go`, `cmd/summer/commands_documented_test.go`, `modules/bonfire/{version,helpers}.go`, `modules/party/registry.go`, `modules/cabana/{backend_screens,navigation,http,admin_openapi}.go`, `modules/pact/capabilities.go`, `modules/fetchguard/{client,policy}.go`, `modules/breakwater/breakwater.go`, `modules/seaglass/seaglass.go`, `modules/compass/README.md`, `modules/conga/README.md`, `admin/src/app/router.ts`, `admin/src/views/{AccountView,SettingsIndexView}.vue`, `admin/src/api/client.ts`, `admin/package.json`, `scripts/check-admin-{openapi,dist}.sh`, `internal/build/stubs/new.tmpl`
|
||||
- Sibling repos: all `sm-*-plugin/{plugin.go,go.mod,README.md}`, `sm-faq-plugin` structure, `sm-journal-plugin/{routes.go,controllers/api/auth.go,token_scopes.go}`, `sm-pgstripe-plugin/subscriptions`, `sm-demo-app/{summer.yaml,go.work,scripts/build.sh}`, `sm-summercmsio-app/{CLAUDE.md,summer.yaml,config/queue.yaml,deploy/nginx,deploy/supervisor,scripts/changelog-post.py,scripts/build.sh}`, site plugin `plugin.go`/`static.go`/`links_test.go`, `vue-summercmsio-app/{package.json,nuxt.config.ts,changelog/nuxt.config.ts}`
|
||||
- Probes this session: `pluginIDFromGo` on mfa/journal; `go get` in workspace; `go list -json` in sm-demo-app; `go list -m -json` module cache Dir; proxy.golang.org `@v/list`; `go.work` replace override; `git submodule add` adoption + `absorbgitdirs`; anonymous `git ls-remote` for 23 repos; go-get meta tag; `go mod edit -json`; `go mod why -m golang.org/x/mod`
|
||||
|
||||
### Secondary (MEDIUM confidence)
|
||||
- https://git-scm.com/docs/git: environment variables `GIT_ALLOW_PROTOCOL`, `GIT_TERMINAL_PROMPT`, `GIT_CONFIG_NOSYSTEM`, `GIT_CONFIG_GLOBAL` (fetched this session)
|
||||
|
||||
### Tertiary (LOW confidence)
|
||||
- None used for decisions. The proposed API shape, schema and names are marked `[ASSUMED]`.
|
||||
|
||||
## Metadata
|
||||
|
||||
**Confidence breakdown:**
|
||||
- Standard stack: HIGH. All components exist in-tree; no additions.
|
||||
- Go module/workspace mechanics (F2, F3, F5): HIGH. Reproduced locally on Go 1.27.2.
|
||||
- Plugin repo survey (F1, F6–F9): HIGH. Read every repo this session.
|
||||
- Architecture/API shape/schema: MEDIUM. These are design recommendations needing sign-off (A2, A3).
|
||||
- Pitfalls: HIGH for the code-grounded ones; MEDIUM for production (rome git, A5).
|
||||
|
||||
**Research date:** 2026-10-29
|
||||
**Valid until:** 2026-11-28 (stable toolchain; re-check plugin repo tags and the go.mod replace layouts before rollout)
|
||||
81
.planning/phases/34-plugin-marketplace/34-VALIDATION.md
Normal file
81
.planning/phases/34-plugin-marketplace/34-VALIDATION.md
Normal file
@@ -0,0 +1,81 @@
|
||||
---
|
||||
phase: "34"
|
||||
slug: "plugin-marketplace"
|
||||
# status lifecycle: draft (seeded by plan-phase) → validated (set by validate-phase §6)
|
||||
# audit-milestone §5.5 distinguishes NOT-VALIDATED (draft) from PARTIAL (validated + nyquist_compliant: false) (#2117)
|
||||
status: draft
|
||||
nyquist_compliant: false
|
||||
wave_0_complete: false
|
||||
created: "2026-10-29"
|
||||
---
|
||||
|
||||
# Phase 34 — Validation Strategy
|
||||
|
||||
> Per-phase validation contract for feedback sampling during execution.
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Framework** | go test (+ testcontainers Postgres); Vitest for admin; node --test for vue-summercmsio-app; unittest for scripts |
|
||||
| **Config file** | none for Go; admin/vitest.config.ts |
|
||||
| **Quick run command** | `go vet ./... && go test -short ./internal/build ./cmd/summer ./modules/party ./modules/cabana` |
|
||||
| **Full suite command** | `go test ./... && npm --prefix admin run test && scripts/check-admin-openapi.sh --check && scripts/check-admin-dist.sh && summer docs:build --check` |
|
||||
| **Estimated runtime** | ~180 seconds |
|
||||
|
||||
---
|
||||
|
||||
## Sampling Rate
|
||||
|
||||
- **After every task commit:** Run the quick run command
|
||||
- **After every plan wave:** Run the full suite command (per repo touched)
|
||||
- **Before `/msd-verify-work`:** Full suite must be green
|
||||
- **Max feedback latency:** 60 seconds
|
||||
|
||||
---
|
||||
|
||||
## Per-Task Verification Map
|
||||
|
||||
| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status |
|
||||
|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------|
|
||||
| _filled by the planner/executor per task; see 34-RESEARCH.md § Validation Architecture for the SC-1..SC-4 test map_ | | | | | | | | | ⬜ pending |
|
||||
|
||||
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
|
||||
|
||||
---
|
||||
|
||||
## Wave 0 Requirements
|
||||
|
||||
- [ ] manifest module tests: derive / README / check / validate
|
||||
- [ ] `internal/build` fixture helpers: git repo with tags, `file://` GOPROXY tree (stdlib only)
|
||||
- [ ] site plugin `TestMain` with testcontainers (`flag.Parse()` + `testing.Short()`)
|
||||
- [ ] admin PluginsView vitest spec
|
||||
- [ ] `scripts/check-phase34.sh`
|
||||
|
||||
*If none: "Existing infrastructure covers all phase requirements."*
|
||||
|
||||
---
|
||||
|
||||
## Manual-Only Verifications
|
||||
|
||||
| Behavior | Requirement | Why Manual | Test Instructions |
|
||||
|----------|-------------|------------|-------------------|
|
||||
| Production ingest on summercms.io (git present on server, hourly job runs) | SC-2 | Needs the deployed host | Deploy, add a repo in the admin, wait for/trigger ingest, `curl /api/registry/v1/plugins` |
|
||||
| Marketplace pages look right | SC-2 | Visual | Open /marketplace list, search, category and a detail page |
|
||||
|
||||
*If none: "All phase behaviors have automated verification."*
|
||||
|
||||
---
|
||||
|
||||
## Validation Sign-Off
|
||||
|
||||
- [ ] All tasks have `<automated>` verify or Wave 0 dependencies
|
||||
- [ ] Sampling continuity: no 3 consecutive tasks without automated verify
|
||||
- [ ] Wave 0 covers all MISSING references
|
||||
- [ ] No watch-mode flags
|
||||
- [ ] Feedback latency < 60s
|
||||
- [ ] `nyquist_compliant: true` set in frontmatter
|
||||
|
||||
**Approval:** pending
|
||||
Reference in New Issue
Block a user