diff --git a/.planning/notes/2026-10-27-robots-txt-grobotstxt.md b/.planning/notes/2026-10-27-robots-txt-grobotstxt.md new file mode 100644 index 00000000..a490bd11 --- /dev/null +++ b/.planning/notes/2026-10-27-robots-txt-grobotstxt.md @@ -0,0 +1,33 @@ +# Decision: the robots.txt tester uses grobotstxt + +**Date:** 2026-10-27 +**Status:** Accepted +**Context:** Phase 32 (SEO foundation, D-22, D-24, D-99), 32-RESEARCH.md Standard Stack and Package Legitimacy Audit, plan 32-04 Task 4 + +## Decision + +The robots.txt tester of `sm-seo-plugin` (element E5, widget action `robots_test`) and the check that warns when robots.txt blocks a noindex address (D-24) match URLs with `github.com/jimsmart/grobotstxt`. + +- **`github.com/jimsmart/grobotstxt` pinned at v1.0.3.** Published 2022-03-16 (proxy.golang.org), Apache-2.0, origin `github.com/jimsmart/grobotstxt` at tag `v1.0.3`. It is the newest tag. +- **Why this library.** It is a function-for-function Go port of Google's C++ robots.txt parser and matcher (`google/robotstxt`), so the tester answers what Googlebot does: + - the longest matching rule wins, and `Allow` wins a tie with `Disallow`; + - `*` matches any run of characters and `$` anchors the end; + - user-agent groups are chosen as Google chooses them (the most specific group, else `*`); + - `RobotsMatcher.MatchingLine()` gives the line number of the deciding rule (0 when no rule matched), which E5 shows to the editor. +- **What the plugin calls:** `NewRobotsMatcher()`, `(*RobotsMatcher).AgentAllowed(robotsBody, userAgent, uri)` and `MatchingLine()`. Nothing else, and only in `sm-seo-plugin/classes/robots.go`. +- **Plugin-only.** Only `sm-seo-plugin` requires it. The framework `go.mod` does not, and `grep jimsmart/grobotstxt go.mod` in summercms.go stays empty. +- **Imported packages:** the library imports the standard library only (`go list -deps` on a scratch module lists `github.com/jimsmart/grobotstxt` as its single non-stdlib package). Its go.mod requires ginkgo and gomega for its own tests; they reach a consumer's go.sum and are never compiled into a binary. + +## Rejected alternatives + +- **`github.com/temoto/robotstxt` v1.1.2** (2021, MIT). Older, with its own parser, no matching-line API, and no documented Google longest-match semantics. +- **A hand-written RFC 9309 matcher** (about 200 lines). Standard library first and no stale dependency, but protocol code the project prefers to take from a maintained library, and Google's behaviour has edge cases the port already covers. RFC 9309 is frozen (2022), so a 2022 release of a faithful port is not a staleness risk in practice. + +## Package legitimacy + +The package-legitimacy seam cannot check Go modules, so grobotstxt was gated by a blocking human check (plan 32-03 Task 4) and approved by Jakub before any `go get`. The orchestrator verified that v1.0.3 resolves through proxy.golang.org; the licence file in the module is Apache-2.0. + +## Consequences + +- `sm-seo-plugin/go.mod` requires `github.com/jimsmart/grobotstxt v1.0.3`; applications that mount the plugin get it through the plugin. +- If the library ever needs replacing, the tester's answer shape (allowed or blocked, the line and its text, blocked by a bot switch) is the plugin's own, so only `classes/robots.go` changes. diff --git a/.planning/research/STACK.md b/.planning/research/STACK.md index b80bd078..f9ff963d 100644 --- a/.planning/research/STACK.md +++ b/.planning/research/STACK.md @@ -47,6 +47,7 @@ This extends `.planning/research/go-ecosystem.md` (2026-09-16 GitHub-API-verifie | air-verse/air | v1.67.4 (Aug 1, 2026), actively maintained | Dev watch-rebuild loop | This is the "watch loop that rebuilds the compiled-plugin binary on change" the plugin architecture already calls for. Config via `.air.toml`; point `cmd`/`bin` at `go build -o ./tmp/summer ./cmd/summer && ./tmp/summer serve`. | | golangci-lint | v2.13.2 (Aug 27, 2026) | Linting | v2's config schema (`version: "2"` in `.golangci.yml`) is a breaking change from v1 — do not copy a v1 config from an older Go project without migrating it. | | `github.com/markbates/goth` (core plus providers google, facebook, github, azureadv2; never gothic) | v1.82.0 | OAuth 2.0 client for social sign-in in sm-sociallogin-plugin | Decided in .planning/notes/2026-10-20-sociallogin-goth.md (Phase 19.5 D-02); plugin-only, the framework go.mod does not require it. | +| `github.com/jimsmart/grobotstxt` | v1.0.3 | Google-faithful robots.txt matching (longest match, allow wins ties, `*` and `$`, matching line) for the robots.txt tester and the noindex-blocked warning in sm-seo-plugin | Decided in .planning/notes/2026-10-27-robots-txt-grobotstxt.md (Phase 32 D-99); plugin-only, the framework go.mod does not require it. | | `charm.land/lipgloss/v2` | v2.0.6 | Console styles, layout and tables (bonfire palette, diary console handler) | Decided in .planning/notes/2026-10-21-console-charm-stack.md (Phase 19.8 D-02); imported only by modules/bonfire and modules/diary | | `charm.land/bubbletea/v2` | v2.1.0 | Runtime for the live spinner, process and progress views | Decided in .planning/notes/2026-10-21-console-charm-stack.md (Phase 19.8 D-02); imported only by modules/bonfire and modules/diary | | `charm.land/bubbles/v2` | v2.2.1 | Spinner and progress components | Decided in .planning/notes/2026-10-21-console-charm-stack.md (Phase 19.8 D-02); imported only by modules/bonfire and modules/diary |