golem15/summercms
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(33-10): chromedp decision note and STACK row

Browse Source
This commit is contained in:
Jakub Zych
2026-11-04 15:43:58 +01:00
parent e245b54ad9
commit e9504b1ece
2 changed files with 43 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
.planning/notes/2026-10-28-chromedp-rendered-crawl.md Normal file
View File

@@ -0,0 +1,42 @@
# Decision: the site audit's rendered comparison uses chromedp, optionally
**Date:** 2026-10-28 (dependency approved 2026-11-04)
**Status:** Accepted
**Context:** Phase 33 (SEO output, audit and content, D-70), 33-CONTEXT.md "Rendered comparison (Jakub, 2026-10-28): optional chromedp", 33-RESEARCH.md Alternatives considered and Package Legitimacy Audit, plan 33-06 Task 4 (approval), plan 33-10 Task 4 (P33-55)
## Decision
The site audit's crawler (`sm-seo-plugin`, package `classes/crawl`) always checks the raw, server-rendered HTML. It compares a page's raw HTML with its JavaScript-rendered DOM only when an operator configures a Chrome or Chromium binary; without one every crawl records the comparison as skipped and says so (the crawl preview P6, `GET /_seo/api/v1/crawls/{id}` and the MCP tool `seo_get_crawl` answer `render {status: skipped}`).
- **`github.com/chromedp/chromedp` pinned at exactly v0.21.0.** Published 2026-10-11 (proxy.golang.org, origin `github.com/chromedp/chromedp` at tag `v0.21.0`, commit `7862adfd`), MIT. Approved by Jakub on 2026-11-04 (33-06 Task 4). v0.21.1 and v0.22.0 exist but were under two weeks old at approval and are deliberately not used. Its go.mod requires `github.com/chromedp/cdproto` v0.157.8 and `github.com/go-json-experiment/json`; nothing else.
- **Why this library.** A maintained pure-Go client of the Chrome DevTools Protocol: no cgo, no Node runtime, no bundled browser. It drives an operator-installed Chrome or Chromium binary (RESEARCH Alternatives considered: a raw-HTML-only crawl misses content that appears only after JavaScript; a Node headless browser would add a second runtime to a single-binary deployment).
- **What is compared** (D-70, P33-55): same-host links, an H1, or a substantial amount of text (at least 50 words and 20 % more) that appear only after JavaScript are reported as `content_js` with `details.check: rendered`. The raw HTML against the CMS record stays `details.check: cms` and runs on every crawl.
- **Chrome is optional and operator-installed.** Production servers have none by default. The executable path is the Go config key `golem15.seo.crawl_chrome_path` (environment-overridable like every key); `golem15.seo.crawl_render_max` (default 200) caps the pages rendered per crawl, in crawl order. An empty path skips the comparison; a relative path, or one that is not an executable regular file, records the comparison as failed with a note. The crawl itself never fails because of the renderer.
- **The path comes from config only, never from an admin setting or MCP.** Any holder of `golem15.seo.manage` can write the SEO settings and call the SEO tools; a writable executable path would let them make the server run an arbitrary binary (T-33-69).
- **Plugin-only.** Only `sm-seo-plugin` requires chromedp. The framework go.mod does not (D-90: the framework gets no SEO code), and `grep chromedp go.mod` in summercms.go stays empty.
## The guard (P33-55)
Headless Chrome executes untrusted page JavaScript on the server, so it is held to the crawler's `fetchguard` limits:
- **Resolver pinning.** Before Chrome starts, the site host and its `www` twin are resolved and every address is checked with `fetchguard.IsPrivateAddr`; a private or reserved address refuses to start the renderer (render status failed, the note names the address) unless `golem15.seo.crawl_trusted` is true. The checked addresses go into `--host-resolver-rules` as `MAP <host> <address>` lines after a leading `MAP * ~NOTFOUND`, so every other host resolves to nothing and Chrome cannot re-resolve the site host to another address.
- **Request interception.** Every request is paused through the DevTools Fetch domain and continued only when it is `https` (`http` only in trusted mode) to the site host or its twin, or a `data:` or `blob:` URL; every other request fails with `BlockedByClient`. Third-party scripts, fonts, analytics and link-local metadata addresses are therefore never loaded.
- **No state.** Each crawl uses a fresh temporary profile (`os.MkdirTemp`) removed when the renderer closes; no cookies, Authorization header, HTTP credentials or extensions; downloads denied. The crawler never signs in.
- **Limits.** One tab renders at a time, 20 s per page, the DOM read with `OuterHTML` is capped at 5 MiB, and Chrome is closed when the crawl finishes, fails or is cancelled.
- **Sandbox.** chromedp's default allocator flags are used without additions that weaken Chrome's sandbox (no `--no-sandbox`).
## Rejected alternatives
- **Raw HTML only.** No dependency and no browser on the server, but content and links that appear only after JavaScript go unreported, which D-70 asks the audit to find.
- **`github.com/go-rod/rod`.** Also a pure-Go DevTools client, but it downloads a browser by itself by default and adds a larger API surface; chromedp's allocator takes an explicit executable path, which keeps the binary under operator control.
- **A Node headless browser (Puppeteer, Playwright).** A second runtime next to the single binary.
## Package legitimacy
The package-legitimacy seam cannot check Go modules, so chromedp was gated by the blocking human check of plan 33-06 Task 4 and approved by Jakub (v0.21.0) before any `go get`. v0.21.0 resolves through proxy.golang.org with the origin above.
## Consequences
- `sm-seo-plugin/go.mod` requires `github.com/chromedp/chromedp v0.21.0` (and, through it, `github.com/chromedp/cdproto`); applications that mount the plugin get it through the plugin, and chromedp is compiled in whether or not a Chrome path is configured.
- A site whose crawl should compare rendered pages installs Chrome or Chromium on the server and sets `golem15.seo.crawl_chrome_path`; nothing else changes.
- If the library ever needs replacing, only `classes/crawl/render_chrome.go` changes: the crawler talks to the `crawl.Renderer` seam.

1
.planning/research/STACK.md
View File

@@ -47,6 +47,7 @@ This extends `.planning/research/go-ecosystem.md` (2026-09-16 GitHub-API-verifie
| air-verse/air | v1.67.4 (Aug 1, 2026), actively maintained | Dev watch-rebuild loop | This is the "watch loop that rebuilds the compiled-plugin binary on change" the plugin architecture already calls for. Config via `.air.toml`; point `cmd`/`bin` at `go build -o ./tmp/summer ./cmd/summer && ./tmp/summer serve`. |
| golangci-lint | v2.13.2 (Aug 27, 2026) | Linting | v2's config schema (`version: "2"` in `.golangci.yml`) is a breaking change from v1 — do not copy a v1 config from an older Go project without migrating it. |
| `github.com/markbates/goth` (core plus providers google, facebook, github, azureadv2; never gothic) | v1.82.0 | OAuth 2.0 client for social sign-in in sm-sociallogin-plugin | Decided in .planning/notes/2026-10-20-sociallogin-goth.md (Phase 19.5 D-02); plugin-only, the framework go.mod does not require it. |
| `github.com/chromedp/chromedp` | v0.21.0 (Oct 11, 2026; pinned exactly, newer tags deliberately not used) | Optional raw-versus-rendered comparison of the SEO site audit in sm-seo-plugin, driving an operator-installed Chrome or Chromium (`golem15.seo.crawl_chrome_path`) | Decided in .planning/notes/2026-10-28-chromedp-rendered-crawl.md (Phase 33 D-70, P33-55); pure Go, no cgo or Node; plugin-only, the framework go.mod does not require it. |
| `github.com/jimsmart/grobotstxt` | v1.0.3 | Google-faithful robots.txt matching (longest match, allow wins ties, `*` and `$`, matching line) for the robots.txt tester and the noindex-blocked warning in sm-seo-plugin | Decided in .planning/notes/2026-10-27-robots-txt-grobotstxt.md (Phase 32 D-99); plugin-only, the framework go.mod does not require it. |
| `charm.land/lipgloss/v2` | v2.0.6 | Console styles, layout and tables (bonfire palette, diary console handler) | Decided in .planning/notes/2026-10-21-console-charm-stack.md (Phase 19.8 D-02); imported only by modules/bonfire and modules/diary |
| `charm.land/bubbletea/v2` | v2.1.0 | Runtime for the live spinner, process and progress views | Decided in .planning/notes/2026-10-21-console-charm-stack.md (Phase 19.8 D-02); imported only by modules/bonfire and modules/diary |