diff --git a/README.md b/README.md index 47b2b88..198c74c 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ WinterCMS theme components such as `journalPost` and `journalPosts` are a later ## Features -- **Posts.** `models.Post` maps `golem15_journal_posts`. Translatable fields are title, content, content_html, excerpt, metadata, and slug (indexed). Fillable does not include `redactor_id`, `content_html`, or `user_id`. Attachments `featured_images` and `content_images` use `system_files` under the same MorphName. YAML admin uses `mltext`, `mltextarea` and the plugin's own `mlblogeditor` field type for content (see [Admin: Blog Editor field types](#admin-blog-editor-field-types)); an admin save stores the sanitized HTML in both `content` and `content_html`. `classes.FormatHTML` regenerates `content_html` on the other write paths (`journal:import-posts`, Journal API writes). It renders markdown and passes raw HTML in `content` through the renderer (Winter parity), then sanitizes the result with a `bluemonday` allow-list: bluemonday's UGC policy plus `footer`, `class`, schema.org microdata, the `color`, `font-size`, `font-weight` and `text-align` styles, goldmark's footnote roles, `dir="auto"` and two pasted-transcript `data-message-*` attributes. Like Winter's `Html::clean`, it strips what the policy does not allow instead of refusing the save: script, style, iframe, object, embed, form, base and meta elements, comments, event handler attributes, and URLs other than http, https, mailto and relative ones. HTML the policy keeps whole is stored byte for byte, so imported bodies survive a save. A tag that repeats an attribute is never kept whole: the sanitized output, which keeps the first value, is stored. On the API and import paths `content` itself is stored as submitted. Translated `content_html` goes through the same policy on the API and `journal:import-posts` write paths (`classes.FormatTranslation`): when the translation carries a non-blank `content`, `content_html` is rendered from it and a supplied `content_html` is replaced, as the host column is always derived; a translation with only `content_html` is sanitized with `classes.SanitizeHTML`. Winter stored a supplied translated `content_html` verbatim. The API response shapes and status codes are unchanged. +- **Posts.** `models.Post` maps `golem15_journal_posts`. Translatable fields are title, content, content_html, excerpt, metadata, and slug (indexed). Fillable does not include `redactor_id`, `content_html`, or `user_id`. Attachments `featured_images` and `content_images` use `system_files` under the same MorphName. YAML admin uses `mltext`, `mltextarea` and the plugin's own `mlblogeditor` field type for content (see [Admin: Blog Editor field types](#admin-blog-editor-field-types)); an admin save stores the sanitized HTML in both `content` and `content_html`. `classes.FormatHTML` regenerates `content_html` on the other write paths (`journal:import-posts`, Journal API writes). It renders markdown and passes raw HTML in `content` through the renderer (Winter parity), then sanitizes the result with a `bluemonday` allow-list: bluemonday's UGC policy plus `footer`, `class`, schema.org microdata, the `color`, `font-size`, `font-weight` and `text-align` styles, goldmark's footnote roles, `dir="auto"`, two pasted-transcript `data-message-*` attributes, and exactly four rules for the Blog Editor's own output (Phase 16.2): `data-embed` matching `^youtube$` and `data-id` matching `^[A-Za-z0-9_-]{11}$` on `figure` (the embed placeholder), `alt` with any text on `img`, and `start` matching `^[0-9]+$` on `ol`. No iframe, other data attribute, style property or URL scheme is allowed for the editor; `TestEditorOutputPassesPolicy` checks every entry of the editor corpus against the policy. Like Winter's `Html::clean`, it strips what the policy does not allow instead of refusing the save: script, style, iframe, object, embed, form, base and meta elements, comments, event handler attributes, and URLs other than http, https, mailto and relative ones. HTML the policy keeps whole is stored byte for byte, so imported bodies survive a save. Admin saves strip the same way and name what they removed (`classes.SanitizeHTMLReport`). A tag that repeats an attribute is never kept whole: the sanitized output, which keeps the first value, is stored. On the API and import paths `content` itself is stored as submitted. Translated `content_html` goes through the same policy on the API and `journal:import-posts` write paths (`classes.FormatTranslation`): when the translation carries a non-blank `content`, `content_html` is rendered from it and a supplied `content_html` is replaced, as the host column is always derived; a translation with only `content_html` is sanitized with `classes.SanitizeHTML`. Winter stored a supplied translated `content_html` verbatim. The API response shapes and status codes are unchanged. - **Categories and tags.** `models.Category` is translatable (name, description, slug). It also stores `content`, an HTML host column that is not translatable, edited in the admin with the `blogeditor` field type. Fillable is name, slug, code, description, content and parent_id; nest_* columns exist in DDL and are not fillable. `models.Tag` is not translatable. Fillable is name, slug, description. Pivots are `golem15_journal_posts_categories` and `golem15_journal_posts_tags`. - **Settings.** Singleton row ID=1 on `golem15_journal_settings`, not PHP `system_settings`. `search_use_typesense` defaults to false. A fresh save never contacts Typesense. - **Public HTTP.** Anonymous GET `/_journal/api/v1/posts`, `/posts/{slug}`, `/categories`, `/tags`, `/rss`, `/authors/{author}`, and `/authors/{author}/posts`. Writes under the same prefix require a cabana backend JWT (`aud=backend`). @@ -159,7 +159,7 @@ The element follows the admin field type element contract: the `value` property - **Embeds.** The embed button takes a YouTube URL (`youtube.com/watch?v=`, `youtu.be/`, `youtube.com/shorts/`, `youtube.com/embed/`) and stores exactly `
`, where `ID` matches `^[A-Za-z0-9_-]{11}$`. The admin shows a card and never an iframe; the site layer expands the placeholder. Another provider is one entry in `EMBED_PROVIDERS` (`embed.ts`) and one in the layer. - **Links.** The editor creates and keeps links only to `http`, `https`, `mailto` and relative URLs (`isPolicyUrl` in `extensions.ts`), the schemes both sanitizer policies keep; the scheme is read after removing control characters and spaces. A created link has `href` only; stored links keep their own `target` and `rel`, which the server strips on an edited save. - **Images.** An image chosen with the file picker (JPEG, PNG, GIF, WebP) or dropped on the editor is handed to the admin page as a bubbling `summer-upload` event with `detail {id, file}`. The admin uploads it with its own session and answers on the element with `summer-upload-result` `{id, url}` or `{id, error}`. On a URL the editor asks for the alt text (required) and an optional caption and inserts ``, or `
` with `` and `
` when a caption is given. An error, or no answer within 60 seconds, is shown in the status line and nothing is inserted. -- **Sanitizing.** The element never sanitizes. Every admin save goes through `classes.SanitizeHTML`, which strips what the policy does not allow. +- **Sanitizing.** The element never sanitizes. Every admin save goes through `classes.SanitizeHTMLReport`, which strips what the policy does not allow and reports it; the policy keeps all of the editor's own output. The element is built by the plugin's own JS build into `assets/dist/blogeditor.js` and `assets/dist/blogeditor.css`. Both files are committed and embedded through `admin.go`, so a host application needs no Node: @@ -242,9 +242,11 @@ Editor images are uploaded by the admin through the field's upload route and sto | `classes.MediaError` | An upload `StoreMedia` refuses, with the request `Field` and the public route's `Message`. | | `classes.MediaMaxBytes` | The upload size limit, 10240 KiB. | | `classes.MediaFolderPattern` | The characters an upload folder may hold. | -| `classes.SanitizeHTML` | Strips from HTML (not markdown) what the `FormatHTML` allow-list does not allow; HTML the policy keeps whole is returned byte for byte. | +| `classes.SanitizeHTML` | Strips from HTML (not markdown) what the `FormatHTML` allow-list does not allow; HTML the policy keeps whole is returned byte for byte. The string half of `SanitizeHTMLReport`. | +| `classes.SanitizeHTMLReport` | `SanitizeHTML` plus an `HTMLRemovals` report of what the policy removed; the report is empty exactly when the input comes back unchanged. | +| `classes.HTMLRemovals` | Removed element and attribute names (distinct, first seen, at most ten each) with exact `ElementCount` and `AttributeCount`, removed `Comments`, and `Other` for any other change; `Empty` reports no removal. | | `classes.FormatTranslation` | Sets `content_html` in one locale's translated attributes: rendered from a non-blank `content` with `FormatHTML`, otherwise a supplied `content_html` sanitized with `SanitizeHTML`. | -| `classes.CheckHTML` | Gate for HTML stored as is, such as category content: refuses HTML that the `FormatHTML` allow-list would strip or change, including any tag that repeats an attribute, instead of stripping it. | +| `classes.CheckHTML` | Predicate: an error when the `FormatHTML` allow-list would strip or change the HTML, including any tag that repeats an attribute. No write path refuses with it any more (admin saves strip and report); it is the counterpart of the site layer's gate and is used by the policy tests. | ## CLI commands diff --git a/classes/editor_policy_test.go b/classes/editor_policy_test.go new file mode 100644 index 0000000..455ba4b --- /dev/null +++ b/classes/editor_policy_test.go @@ -0,0 +1,253 @@ +package classes + +import ( + "encoding/json" + "os" + "slices" + "strings" + "testing" + + xhtml "golang.org/x/net/html" +) + +// editorPlaceholder is the Blog Editor's YouTube embed placeholder. +const editorPlaceholder = `
` + +// editorCorpus reads plan 02's editor-corpus.json, the editor's own output. +// A missing or empty file fails the test; it never skips. +func editorCorpus(t *testing.T) []struct{ Name, HTML string } { + t.Helper() + raw, err := os.ReadFile("../tests/blogeditor/fixtures/editor-corpus.json") + if err != nil { + t.Fatalf("editor corpus: %v", err) + } + var entries []struct { + Name string `json:"name"` + HTML string `json:"html"` + } + if err := json.Unmarshal(raw, &entries); err != nil { + t.Fatalf("editor corpus: %v", err) + } + if len(entries) == 0 { + t.Fatal("editor corpus is empty") + } + out := make([]struct{ Name, HTML string }, 0, len(entries)) + for _, e := range entries { + if e.Name == "" || e.HTML == "" { + t.Fatalf("editor corpus entry %+v has no name or html", e) + } + out = append(out, struct{ Name, HTML string }{e.Name, e.HTML}) + } + return out +} + +// tagAttrs returns the attribute names of every start tag named tag in doc, +// one slice per tag, repeats included. +func tagAttrs(doc, tag string) [][]string { + var out [][]string + z := xhtml.NewTokenizer(strings.NewReader(doc)) + for { + tt := z.Next() + if tt == xhtml.ErrorToken { + return out + } + if tt != xhtml.StartTagToken && tt != xhtml.SelfClosingTagToken { + continue + } + raw := rawAttrNames(z.Raw()) + if z.Token().Data == tag { + out = append(out, raw) + } + } +} + +// TestEditorOutputPassesPolicy pins Phase 16.2 D-05 (amended): the HTML +// policy keeps every piece of markup the Blog Editor writes, byte for byte +// and with an empty removal report, and its four editor-output additions +// (figure data-embed and data-id, img alt, ol start) admit nothing else. +func TestEditorOutputPassesPolicy(t *testing.T) { + for _, e := range editorCorpus(t) { + t.Run("corpus "+e.Name, func(t *testing.T) { + if got := SanitizeHTML(e.HTML); got != e.HTML { + t.Fatalf("SanitizeHTML changed the editor's output:\n got %q\nwant %q", got, e.HTML) + } + if _, report := SanitizeHTMLReport(e.HTML); !report.Empty() { + t.Fatalf("report = %+v, want empty", report) + } + if err := CheckHTML(e.HTML); err != nil { + t.Fatalf("CheckHTML = %v", err) + } + }) + } + + t.Run("placeholder is kept", func(t *testing.T) { + if got := SanitizeHTML(editorPlaceholder); got != editorPlaceholder { + t.Fatalf("SanitizeHTML(placeholder) = %q", got) + } + out, err := FormatHTML("Intro\n\n" + editorPlaceholder + "\n\nOutro") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(out, editorPlaceholder) { + t.Fatalf("FormatHTML dropped the placeholder: %q", out) + } + }) + + stripped := []struct { + name, in, want string + }{ + {"other provider", `
`, `
`}, + {"provider with a suffix", `
`, `
`}, + {"id of 10 characters", `
`, `
`}, + {"id of 12 characters", `
`, `
`}, + {"id with a quote", `
`, `
`}, + {"id with a space", `
`, `
`}, + {"id with an angle bracket", `
`, `
`}, + {"embed attributes on div", `
x
`, `
x
`}, + {"embed attributes on img", ``, ``}, + {"embed attributes on p", `

x

`, `

x

`}, + {"other data attribute on figure", `
`, editorPlaceholder}, + {"data attribute alone on figure", `
`, `
`}, + {"onload on the placeholder", `
`, editorPlaceholder}, + {"iframe next to the placeholder", editorPlaceholder + ``, editorPlaceholder}, + {"start that is not a number", `
  1. a
`, `
  1. a
`}, + {"negative start", `
  1. a
`, `
  1. a
`}, + {"start on ul", ``, ``}, + {"alt on p", `

y

`, `

y

`}, + } + for _, tc := range stripped { + t.Run("strips "+tc.name, func(t *testing.T) { + got, report := SanitizeHTMLReport(tc.in) + if got != tc.want { + t.Fatalf("SanitizeHTMLReport(%q) = %q, want %q", tc.in, got, tc.want) + } + if report.Empty() { + t.Fatalf("report is empty for a stripped input %q", tc.in) + } + if CheckHTML(tc.in) == nil { + t.Fatalf("CheckHTML(%q) = nil, want an error", tc.in) + } + }) + } + + t.Run("alt keeps any text as one attribute", func(t *testing.T) { + for _, in := range []string{ + `Turkey Tail: czy to "grzyb odporności"?`, + `x" onerror="alert(1)`, + `x" onerror="alert(1)`, + `<script>alert(1)</script>`, + } { + got, report := SanitizeHTMLReport(in) + if got != in || !report.Empty() { + t.Fatalf("SanitizeHTMLReport(%q) = %q, %+v, want it unchanged", in, got, report) + } + out := htmlPolicy.Sanitize(in) + if attrs := tagAttrs(out, "img"); len(attrs) != 1 || !slices.Equal(attrs[0], []string{"src", "alt"}) { + t.Fatalf("policy output %q has img attributes %v, want [src alt]", out, attrs) + } + } + }) + + t.Run("ol start keeps a number", func(t *testing.T) { + const in = `
  1. c
` + if got := SanitizeHTML(in); got != in { + t.Fatalf("SanitizeHTML(%q) = %q", in, got) + } + }) +} + +// TestSanitizeHTMLReport pins the removal report the Blog Editor turns into +// admin notices (D-10 amended). +func TestSanitizeHTMLReport(t *testing.T) { + cases := []struct { + name string + in, out string + elements []string + elementCount int + attributes []string + attributeCount int + comments int + other bool + }{ + {name: "clean", in: `

Plain link

`, out: `

Plain link

`}, + {name: "style element", in: `

a

`, out: `

a

`, elements: []string{"style"}, elementCount: 1}, + {name: "onclick attribute", in: `

x

`, out: `

x

`, attributes: []string{"onclick"}, attributeCount: 1}, + {name: "comment", in: `

a

`, out: `

a

`, comments: 2}, + {name: "repeated attribute", in: `

x

`, out: `

x

`, attributes: []string{"class"}, attributeCount: 1}, + {name: "style loses a property", in: `

x

`, out: `

x

`, attributes: []string{"style"}, attributeCount: 1}, + {name: "javascript link", in: `xy`, out: `xy`, elements: []string{"a"}, elementCount: 1}, + { + name: "more than ten elements", + in: `
`, + out: ``, + elements: []string{"script", "iframe", "object", "embed", "form", "input", "button", "select", "textarea", "base"}, + elementCount: 13, + }, + { + name: "more than ten attributes", + in: `

x

`, + out: `

x

`, + attributes: []string{"onclick", "onload", "onerror", "onmouseover", "onfocus", "onblur", "onkeyup", "onkeydown", "onchange", "onsubmit"}, + attributeCount: 12, + }, + {name: "doctype", in: "

x

", out: "

x

", other: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, r := SanitizeHTMLReport(tc.in) + if got != tc.out { + t.Fatalf("output = %q, want %q", got, tc.out) + } + if !slices.Equal(r.Elements, tc.elements) || r.ElementCount != tc.elementCount || + !slices.Equal(r.Attributes, tc.attributes) || r.AttributeCount != tc.attributeCount || + r.Comments != tc.comments || r.Other != tc.other { + t.Fatalf("report = %+v", r) + } + if r.Empty() != (got == tc.in) { + t.Fatalf("Empty() = %v but output changed = %v", r.Empty(), got != tc.in) + } + }) + } + + t.Run("cut lists end the notice names with an ellipsis", func(t *testing.T) { + _, r := SanitizeHTMLReport(cases[7].in) + notices := removalNotices(r) + if len(notices) != 1 || notices[0].Count != 13 || !strings.HasSuffix(notices[0].Params["names"], ", …") || + !strings.HasPrefix(notices[0].Params["names"], "script, iframe, ") { + t.Fatalf("notices = %#v", notices) + } + _, r = SanitizeHTMLReport(`

x

`) + notices = removalNotices(r) + if len(notices) != 2 || notices[0].Params["names"] != "onclick" || notices[1].Message != editorRemovedComments || notices[1].Count != 1 { + t.Fatalf("notices = %#v", notices) + } + _, r = SanitizeHTMLReport("

x

") + if notices = removalNotices(r); len(notices) != 1 || notices[0].Message != editorMarkupChanged || notices[0].Count != 0 { + t.Fatalf("notices = %#v", notices) + } + }) + + t.Run("report is empty exactly when the output equals the input", func(t *testing.T) { + inputs := []string{} + for _, tc := range unsafePayloads { + inputs = append(inputs, tc.src) + } + fixture, err := os.ReadFile("../tests/blogeditor/fixtures/imported-post.html") + if err != nil { + t.Fatal(err) + } + inputs = append(inputs, string(fixture)) + for _, e := range editorCorpus(t) { + inputs = append(inputs, e.HTML) + } + for _, in := range inputs { + got, r := SanitizeHTMLReport(in) + if r.Empty() != (got == in) { + t.Fatalf("SanitizeHTMLReport(%q): Empty() = %v, output %q", in, r.Empty(), got) + } + if got != SanitizeHTML(in) { + t.Fatalf("SanitizeHTMLReport and SanitizeHTML disagree on %q", in) + } + } + }) +} diff --git a/classes/format_html.go b/classes/format_html.go index b1f23ed..fe94dd5 100644 --- a/classes/format_html.go +++ b/classes/format_html.go @@ -54,6 +54,22 @@ func newHTMLPolicy() *bluemonday.Policy { p.AllowAttrs("dir").Matching(regexp.MustCompile(`(?i)^(ltr|rtl|auto)$`)).Globally() p.AllowAttrs("data-message-author-role", "data-message-id"). Matching(regexp.MustCompile(`^[A-Za-z0-9_-]+$`)).OnElements("div") + // Blog Editor output (Phase 16.2 D-05), exactly what the editor writes + // and nothing more; plan 02's editor-corpus.json pins it + // (TestEditorOutputPassesPolicy) and the site layer mirrors these rules. + // The embed placeholder
: + // the provider name and the 11-character video id, on figure only. The + // layer expands it; no iframe is ever stored. + p.AllowAttrs("data-embed").Matching(regexp.MustCompile(`^youtube$`)).OnElements("figure") + p.AllowAttrs("data-id").Matching(regexp.MustCompile(`^[A-Za-z0-9_-]{11}$`)).OnElements("figure") + // The editor's alt field takes free text, which UGC's Paragraph pattern + // cuts at ':', '?', '"' and more. bluemonday accepts an attribute when + // any of its rules matches and escapes the value on output; alt carries + // no URL. + p.AllowAttrs("alt").OnElements("img") + // TipTap's ordered-list input rule writes start when a list does not + // begin at 1. + p.AllowAttrs("start").Matching(regexp.MustCompile(`^[0-9]+$`)).OnElements("ol") return p } @@ -310,10 +326,12 @@ func textOf(v any) string { } } -// CheckHTML gates HTML that is stored as is, such as category content. It -// uses the FormatHTML allow-list but refuses instead of stripping: doc is -// refused when the policy would remove or change any element, attribute or -// text in it. +// CheckHTML reports whether the FormatHTML allow-list would remove or +// change any element, attribute or text in doc: it returns an error when it +// would, nil when the policy keeps doc whole. Since Phase 16.2 no admin or +// API write path refuses with it (admin saves strip and report through +// SanitizeHTMLReport, D-10 amended); it is the predicate counterpart of the +// site layer's isSafeJournalHtml, and the policy cross-check tests use it. func CheckHTML(doc string) error { if _, changed := sanitizeHTML(doc); changed { return errors.New("journal: HTML contains elements, attributes or URLs that are not allowed") diff --git a/controllers/posts_editor_smoke_test.go b/controllers/posts_editor_smoke_test.go index 010b2b8..57425bc 100644 --- a/controllers/posts_editor_smoke_test.go +++ b/controllers/posts_editor_smoke_test.go @@ -146,6 +146,17 @@ func TestBlogEditorTracerSmoke(t *testing.T) { } }) + t.Run("decode keeps the embed placeholder", func(t *testing.T) { + const placeholder = `

Film:

` + got, err := decode(t, placeholder) + if err != nil { + t.Fatal(err) + } + if got.Value != placeholder || len(got.Notices) != 0 { + t.Fatalf("Decode(placeholder) = %#v", got) + } + }) + t.Run("decode strips and reports", func(t *testing.T) { got, err := decode(t, `

a

x`) if err != nil {