From cb1c6882138fe71e816656ff03691a6497f081a5 Mon Sep 17 00:00:00 2001 From: Jakub Zych Date: Sat, 10 Oct 2026 12:08:51 +0200 Subject: [PATCH] feat(admin): category content on the Blog Editor and editor image uploads - category content is the blogeditor type (uploadFolder categories): it is stripped and reported like post content; the 422 CheckHTML gate is retired and blank content still becomes NULL - blogeditor and mlblogeditor Upload: magic-byte check for JPEG, PNG, GIF and WebP matching the extension, safe lowercase names, storage through classes.StoreMedia in the field's uploadFolder - TestBlogEditorUploadSmoke, TestCategoriesAdminContent asserts the strip, README describes the field types, admin saves and editor uploads --- README.md | 46 +++++- classes/blog_editor.go | 137 +++++++++++++++- controllers/categories.go | 18 +-- controllers/posts_editor_smoke_test.go | 206 +++++++++++++++++++++++++ journal_api_categories_test.go | 56 ++++--- lang/en/lang.yaml | 2 +- lang/pl/lang.yaml | 2 +- models/category/fields.yaml | 3 +- 8 files changed, 417 insertions(+), 53 deletions(-) diff --git a/README.md b/README.md index f9ac560..47b2b88 100644 --- a/README.md +++ b/README.md @@ -18,8 +18,8 @@ WinterCMS theme components such as `journalPost` and `journalPosts` are a later ## Features -- **Posts.** `models.Post` maps `golem15_journal_posts`. Translatable fields are title, content, content_html, excerpt, metadata, and slug (indexed). Fillable does not include `redactor_id`, `content_html`, or `user_id`. Attachments `featured_images` and `content_images` use `system_files` under the same MorphName. YAML admin uses `mltext`/`mlmarkdown`. `classes.FormatHTML` regenerates `content_html` on every write path (admin save, `journal:import-posts`, Journal API writes). It renders markdown and passes raw HTML in `content` through the renderer (Winter parity), then sanitizes the result with a `bluemonday` allow-list: bluemonday's UGC policy plus `footer`, `class`, schema.org microdata, the `color`, `font-size`, `font-weight` and `text-align` styles, goldmark's footnote roles, `dir="auto"` and two pasted-transcript `data-message-*` attributes. Like Winter's `Html::clean`, it strips what the policy does not allow instead of refusing the save: script, style, iframe, object, embed, form, base and meta elements, comments, event handler attributes, and URLs other than http, https, mailto and relative ones. HTML the policy keeps whole is stored byte for byte, so imported bodies survive a save. A tag that repeats an attribute is never kept whole: the sanitized output, which keeps the first value, is stored. `content` itself is stored as submitted. Translated `content_html` goes through the same policy on the API and `journal:import-posts` write paths (`classes.FormatTranslation`): when the translation carries a non-blank `content`, `content_html` is rendered from it and a supplied `content_html` is replaced, as the host column is always derived; a translation with only `content_html` is sanitized with `classes.SanitizeHTML`. Winter stored a supplied translated `content_html` verbatim. The API response shapes and status codes are unchanged. -- **Categories and tags.** `models.Category` is translatable (name, description, slug). It also stores `content`, an HTML host column that is not translatable, edited in the admin as a textarea. Fillable is name, slug, code, description, content and parent_id; nest_* columns exist in DDL and are not fillable. `models.Tag` is not translatable. Fillable is name, slug, description. Pivots are `golem15_journal_posts_categories` and `golem15_journal_posts_tags`. +- **Posts.** `models.Post` maps `golem15_journal_posts`. Translatable fields are title, content, content_html, excerpt, metadata, and slug (indexed). Fillable does not include `redactor_id`, `content_html`, or `user_id`. Attachments `featured_images` and `content_images` use `system_files` under the same MorphName. YAML admin uses `mltext`, `mltextarea` and the plugin's own `mlblogeditor` field type for content (see [Admin: Blog Editor field types](#admin-blog-editor-field-types)); an admin save stores the sanitized HTML in both `content` and `content_html`. `classes.FormatHTML` regenerates `content_html` on the other write paths (`journal:import-posts`, Journal API writes). It renders markdown and passes raw HTML in `content` through the renderer (Winter parity), then sanitizes the result with a `bluemonday` allow-list: bluemonday's UGC policy plus `footer`, `class`, schema.org microdata, the `color`, `font-size`, `font-weight` and `text-align` styles, goldmark's footnote roles, `dir="auto"` and two pasted-transcript `data-message-*` attributes. Like Winter's `Html::clean`, it strips what the policy does not allow instead of refusing the save: script, style, iframe, object, embed, form, base and meta elements, comments, event handler attributes, and URLs other than http, https, mailto and relative ones. HTML the policy keeps whole is stored byte for byte, so imported bodies survive a save. A tag that repeats an attribute is never kept whole: the sanitized output, which keeps the first value, is stored. On the API and import paths `content` itself is stored as submitted. Translated `content_html` goes through the same policy on the API and `journal:import-posts` write paths (`classes.FormatTranslation`): when the translation carries a non-blank `content`, `content_html` is rendered from it and a supplied `content_html` is replaced, as the host column is always derived; a translation with only `content_html` is sanitized with `classes.SanitizeHTML`. Winter stored a supplied translated `content_html` verbatim. The API response shapes and status codes are unchanged. +- **Categories and tags.** `models.Category` is translatable (name, description, slug). It also stores `content`, an HTML host column that is not translatable, edited in the admin with the `blogeditor` field type. Fillable is name, slug, code, description, content and parent_id; nest_* columns exist in DDL and are not fillable. `models.Tag` is not translatable. Fillable is name, slug, description. Pivots are `golem15_journal_posts_categories` and `golem15_journal_posts_tags`. - **Settings.** Singleton row ID=1 on `golem15_journal_settings`, not PHP `system_settings`. `search_use_typesense` defaults to false. A fresh save never contacts Typesense. - **Public HTTP.** Anonymous GET `/_journal/api/v1/posts`, `/posts/{slug}`, `/categories`, `/tags`, `/rss`, `/authors/{author}`, and `/authors/{author}/posts`. Writes under the same prefix require a cabana backend JWT (`aud=backend`). - **Author slug.** Plugin migration adds `backend_users.golem15_bloghub_author_slug`. The column stores the public author slug that the author routes resolve. @@ -86,7 +86,7 @@ GET /_journal/api/v1/authors/jane-doe/posts?page=2 `content` is the category's HTML body for a category landing page. It is stored and returned as is: it is not markdown and the API does not render, trim or sanitize it. It is `null` when unset, and it is the same in every locale, because it is a host column and not translatable. A consumer that renders it as HTML must apply its own gate or sanitizer first. -The admin Categories form refuses content that fails `classes.CheckHTML` with a `422` on `content`, and stores blank content as `null`. Rows written by SQL or by import commands bypass that gate, so consumers keep their own gate. +The admin Categories form edits content with the `blogeditor` field type: a save strips what the HTML policy does not allow, names what it removed in the response's `meta.notices`, and is never refused with a `422` because of markup. Blank content is stored as `null`. Rows written by SQL or by import commands are not sanitized by the form, so consumers keep their own gate. ```json {"data":[{"id":4,"name":"Acme news","slug":"news","description":null,"content":"

Acme news

Pillar copy

","parent_id":null,"nest_depth":0,"post_count":3}]} @@ -174,6 +174,37 @@ After changing `assets/src`, run `npm run build` and commit `assets/dist`. `tests/blogeditor/fixtures/editor-corpus.json` holds the editor's own output, built through editor commands: an embed, images with and without a caption, a table with a header row, links, headings, lists (one starting at 3), a blockquote, inline marks, a code block, a horizontal rule, a hard break, and one document combining them. `tests/blogeditor/corpus.test.ts` rebuilds it and fails when the committed file differs; `UPDATE_CORPUS=1 npm test` rewrites it. The Go policy test and the site layer check the same entries against both sanitizer policies, so entry names stay stable. +### Admin: Blog Editor field types + +The plugin registers two admin form field types (`journal.Plugin.FormFieldTypes`, built by `classes.BlogEditorTypes`) that render the Blog Editor element. Any controller's `fields.yaml` may use them: + +| Type | Value | Options | +|------|-------|---------| +| `blogeditor` | One HTML string in a host column (category content). | `uploadFolder` | +| `mlblogeditor` | One HTML string per content locale, stored like `mlmarkdown`: the default locale in the host column, the others as translations (post content). | `htmlColumn`, `uploadFolder` | + +`htmlColumn` names the translatable attribute that holds the rendered HTML of a markdown value (`content_html` for posts). `uploadFolder` is the folder under `/journal/` that editor images are stored in (`posts`, `categories`; `editor` when unset); it follows the public upload route's folder rule (at most 100 letters, digits, `_`, `-` or `/`, no `..`) and is checked at boot. + +```yaml +content: + label: golem15.journal::lang.post.content + type: mlblogeditor + htmlColumn: content_html + uploadFolder: posts +``` + +What an admin save stores: + +- Each submitted locale's value goes through `classes.SanitizeHTMLReport`. A value the policy keeps whole is stored byte for byte. Anything the policy does not allow is removed, the save goes through, and the response's `meta.notices` names what was removed per field and locale (for example `2 elements were removed: script, iframe`), in English or Polish. A save is never refused because of markup; a value that is not text is a `422` on the field. +- For posts, the stored HTML goes to both `content` and `content_html` of that locale: the host columns through the `FormBeforeCreate`/`FormBeforeUpdate` hooks, the other locales through `FormAfterCreate`/`FormAfterUpdate` and the translation writer. The admin path does not run goldmark. +- A locale whose content the administrator did not change is not written, so an imported post keeps both columns byte for byte until its body is edited. +- A post whose `content` does not start with `<` (`classes.LooksLikeHTML`) is a markdown post: the editor opens it with that locale's `content_html`, or with `FormatHTML(content)` when that is empty. Its first edited save stores HTML in both columns. +- Category content follows the same rule and is no longer refused with a `422`. + +The public Journal API is unchanged: `POST` and `PUT /_journal/api/v1/posts` still take markdown `content` and store `FormatHTML` output in `content_html`. + +Editor images are uploaded by the admin through the field's upload route and stored by `classes.StoreMedia`, the same storage as `POST /_journal/api/v1/media/upload`: the key is `/journal//` with the same numbered suffix when the name is taken and the same public URL. An editor upload must be a JPEG, PNG, GIF or WebP file whose first bytes sniff as the type its extension names (no SVG), at most 10240 KiB. Its name is lowercased and every character outside `a-z`, `0-9`, `.`, `_` and `-` becomes `-`, so the inserted `src` needs no escaping. The public upload route keeps its own naming, request fields, response and messages. + ## API reference | Identifier | Description | @@ -185,6 +216,7 @@ After changing `assets/src`, run `npm run build` and commit `assets/dist`. | `journal.Plugin.Models` | Models registered by this plugin. | | `journal.Plugin.Commands` | Registers `journal:export-posts` and `journal:import-posts`. | | `journal.Plugin.Routes` | Public GETs (posts, categories, tags, RSS, authors) and backend-Bearer writes under `/_journal/api/v1`. | +| `journal.Plugin.FormFieldTypes` | Registers the `blogeditor` and `mlblogeditor` admin field types. | | `journal.Plugin.Buckets` | Limiter buckets `journal-public-api` and `journal-api` (Max 120). | | `models.Post` | Post row; `TableName` is `golem15_journal_posts`. | | `models.Post.MorphName` | PHP class string `Golem15\Journal\Models\Post`. | @@ -202,6 +234,14 @@ After changing `assets/src`, run `npm run build` and commit `assets/dist`. | `models.Settings` | Singleton settings row; `TableName` is `golem15_journal_settings`. | | `models.All` | Models registered by this plugin. | | `updates.All` | gormigrate steps registered by this plugin. | +| `classes.BlogEditorTypes` | The `blogeditor` and `mlblogeditor` field types: Decode through `SanitizeHTMLReport` with removal notices, Load with the markdown fallback, list text and the image upload. | +| `classes.BlogEditorElement` | The editor's custom element tag, `golem15-journal-blogeditor`. | +| `classes.LooksLikeHTML` | Whether stored content is HTML: its first non-space character is `<`. | +| `classes.StoreMedia` | Stores an uploaded image under `/journal//` and returns its `StoredMedia`; shared by the public upload route and the editor upload. | +| `classes.StoredMedia` | A stored media object's public `URL` and `Path`. | +| `classes.MediaError` | An upload `StoreMedia` refuses, with the request `Field` and the public route's `Message`. | +| `classes.MediaMaxBytes` | The upload size limit, 10240 KiB. | +| `classes.MediaFolderPattern` | The characters an upload folder may hold. | | `classes.SanitizeHTML` | Strips from HTML (not markdown) what the `FormatHTML` allow-list does not allow; HTML the policy keeps whole is returned byte for byte. | | `classes.FormatTranslation` | Sets `content_html` in one locale's translated attributes: rendered from a non-blank `content` with `FormatHTML`, otherwise a supplied `content_html` sanitized with `SanitizeHTML`. | | `classes.CheckHTML` | Gate for HTML stored as is, such as category content: refuses HTML that the `FormatHTML` allow-list would strip or change, including any tag that repeats an attribute, instead of stripping it. | diff --git a/classes/blog_editor.go b/classes/blog_editor.go index 3fcff14..ba33063 100644 --- a/classes/blog_editor.go +++ b/classes/blog_editor.go @@ -1,9 +1,13 @@ package classes import ( + "bytes" "context" "errors" "fmt" + "io" + "net/http" + "path" "regexp" "strings" "unicode" @@ -11,6 +15,7 @@ import ( "git.golem15.com/golem15/summercms/modules/backpack" "git.golem15.com/golem15/summercms/modules/pact" + "gocloud.dev/blob" xhtml "golang.org/x/net/html" ) @@ -21,6 +26,7 @@ const BlogEditorElement = "golem15-journal-blogeditor" // Phrase keys of the Blog Editor field types. const ( editorContentInvalid = "golem15.journal::lang.editor.content_invalid" + editorImageInvalid = "golem15.journal::lang.editor.image_invalid" editorRemovedElements = "golem15.journal::lang.editor.removed_elements" editorRemovedAttributes = "golem15.journal::lang.editor.removed_attributes" editorRemovedComments = "golem15.journal::lang.editor.removed_comments" @@ -30,9 +36,28 @@ const ( // blogListValueRunes is how many runes of text a list column shows. const blogListValueRunes = 120 +// blogUploadFolder is the upload folder of a field without uploadFolder. +const blogUploadFolder = "editor" + var ( - blogHTMLColumnPattern = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`) - blogUploadFolderPattern = regexp.MustCompile(`^[A-Za-z0-9_\-/]*$`) + blogHTMLColumnPattern = regexp.MustCompile(`^[A-Za-z_][A-Za-z0-9_]*$`) + blogUploadExtPattern = regexp.MustCompile(`^\.[a-z0-9]+$`) + // blogImageTypes maps an upload extension to the content type its first + // bytes must sniff as. + blogImageTypes = map[string]string{ + "jpg": "image/jpeg", + "jpeg": "image/jpeg", + "png": "image/png", + "gif": "image/gif", + "webp": "image/webp", + } + // blogImageExtensions is the extension given to a name without one. + blogImageExtensions = map[string]string{ + "image/jpeg": "jpg", + "image/png": "png", + "image/gif": "gif", + "image/webp": "webp", + } ) // BlogEditorTypes returns the Blog Editor admin field types: blogeditor for @@ -41,14 +66,17 @@ var ( // keeps (SanitizeHTMLReport) and reports what it removed as notices; it // never refuses a string because of its markup. Load serves HTML content as // it is and a markdown value (one that does not start with '<') as the -// rendered HTML of the htmlColumn option. app is asked for the booted -// application when a type needs it at request time; it may be nil in tests. +// rendered HTML of the htmlColumn option. Upload stores an image the editor +// inserts through StoreMedia, in the media bucket the app publishes. app is +// asked for the booted application at request time; it may be nil in tests, +// where Upload then fails. // // Options: htmlColumn (mlblogeditor only) names the translatable attribute // that holds the rendered HTML of a markdown value; uploadFolder is the -// folder under /journal/ that editor image uploads are stored in. +// folder under /journal/ that editor image uploads are stored in ("editor" +// when unset). func BlogEditorTypes(app func() *backpack.App) []pact.FormFieldType { - _ = app // used by the editor image upload + upload := uploadBlogImage(app) return []pact.FormFieldType{ { Name: "blogeditor", @@ -60,6 +88,7 @@ func BlogEditorTypes(app func() *backpack.App) []pact.FormFieldType { Decode: decodeBlogHTML, Load: loadBlogHTML, ListValue: blogListValue, + Upload: upload, }, { Name: "mlblogeditor", @@ -72,6 +101,7 @@ func BlogEditorTypes(app func() *backpack.App) []pact.FormFieldType { Decode: decodeBlogHTML, Load: loadBlogHTML, ListValue: blogListValue, + Upload: upload, }, } } @@ -106,7 +136,7 @@ func checkBlogOptions(options map[string]any) error { // checkUploadFolder applies the public media upload route's folder rule. func checkUploadFolder(folder string) error { - if len(folder) > 100 || !blogUploadFolderPattern.MatchString(folder) || strings.Contains(folder, "..") { + if !validMediaFolder(folder) { return errors.New("uploadFolder must be at most 100 letters, digits, '_', '-' or '/' without '..'") } return nil @@ -259,3 +289,96 @@ func blogListValue(_ context.Context, value any) string { runes := []rune(text) return strings.TrimRightFunc(string(runes[:blogListValueRunes]), unicode.IsSpace) + "…" } + +// uploadBlogImage is the Blog Editor's Upload: it stores an image the +// editor inserts through StoreMedia, in the field's uploadFolder, and +// returns its URL. The file's first bytes must sniff as a JPEG, PNG, GIF or +// WebP image of the type its extension names (a name without an extension +// gets the sniffed one); anything else is refused with +// golem15.journal::lang.editor.image_invalid. The stored name is +// safeUploadName's. A MediaError is refused with its message; a missing +// application or bucket is an opaque error. +func uploadBlogImage(app func() *backpack.App) func(context.Context, pact.FormFieldUpload) (pact.FormFieldUploadResult, error) { + return func(ctx context.Context, in pact.FormFieldUpload) (pact.FormFieldUploadResult, error) { + var a *backpack.App + if app != nil { + a = app() + } + if a == nil { + return pact.FormFieldUploadResult{}, errors.New("journal: editor upload: application is not configured") + } + bucket, ok := a.Lookup[*blob.Bucket]() + if !ok || bucket == nil { + return pact.FormFieldUploadResult{}, errors.New("journal: editor upload: media bucket is not configured") + } + if in.Body == nil { + return pact.FormFieldUploadResult{}, &pact.FormFieldError{Messages: []string{editorImageInvalid}} + } + head := make([]byte, 512) + n, err := io.ReadFull(in.Body, head) + if err != nil && !errors.Is(err, io.EOF) && !errors.Is(err, io.ErrUnexpectedEOF) { + return pact.FormFieldUploadResult{}, fmt.Errorf("journal: editor upload: read: %w", err) + } + head = head[:n] + sniffed := http.DetectContentType(head) + ext, known := blogImageExtensions[sniffed] + if !known { + return pact.FormFieldUploadResult{}, &pact.FormFieldError{Messages: []string{editorImageInvalid}} + } + name := safeUploadName(in.Filename) + if named := strings.TrimPrefix(path.Ext(name), "."); named == "" { + name += "." + ext + } else if want, allowed := blogImageTypes[named]; allowed && want != sniffed { + return pact.FormFieldUploadResult{}, &pact.FormFieldError{Messages: []string{editorImageInvalid}} + } + folder, _ := in.Options["uploadFolder"].(string) + if folder == "" { + folder = blogUploadFolder + } + stored, err := StoreMedia(ctx, bucket, folder, name, io.MultiReader(bytes.NewReader(head), in.Body)) + if err != nil { + var refused *MediaError + if errors.As(err, &refused) && refused != nil { + return pact.FormFieldUploadResult{}, &pact.FormFieldError{Messages: []string{refused.Message}} + } + return pact.FormFieldUploadResult{}, err + } + return pact.FormFieldUploadResult{URL: stored.URL}, nil + } +} + +// safeUploadName is the stored name of an editor upload: the base name, +// lowercased, every character outside a-z, 0-9, '.', '_' and '-' turned +// into '-' with runs collapsed and no leading or trailing '-' or '.', and +// the extension kept. An empty name becomes "image". The result needs no +// escaping in an src attribute. +func safeUploadName(filename string) string { + base := path.Base(strings.ReplaceAll(filename, `\`, "/")) + ext := strings.ToLower(path.Ext(base)) + if !blogUploadExtPattern.MatchString(ext) { + ext = "" + } + stem := strings.ToLower(strings.TrimSuffix(base, path.Ext(base))) + if ext == "" { + stem = strings.ToLower(base) + } + var b strings.Builder + dash := false + for _, r := range stem { + ok := (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') || r == '.' || r == '_' || r == '-' + if !ok || r == '-' { + if !dash { + b.WriteByte('-') + dash = true + } + continue + } + b.WriteRune(r) + dash = false + } + stem = strings.Trim(b.String(), "-.") + if stem == "" { + stem = "image" + } + return stem + ext +} diff --git a/controllers/categories.go b/controllers/categories.go index 794ae1a..c1851a1 100644 --- a/controllers/categories.go +++ b/controllers/categories.go @@ -5,7 +5,6 @@ import ( "errors" "strings" - "git.golem15.com/golem15/sm-journal-plugin/classes" "git.golem15.com/golem15/sm-journal-plugin/models" "git.golem15.com/golem15/summercms/modules/backpack" "git.golem15.com/golem15/summercms/modules/cabana" @@ -61,21 +60,20 @@ func (categoriesAdminController) FormBeforeUpdate(ctx context.Context, model any return prepareCategoryWrite(cat) } -// prepareCategoryWrite gates the category HTML body (D-21): blank content is -// stored as NULL, and content that fails classes.CheckHTML is a 422 on -// content. Any other content is stored exactly as submitted. +// prepareCategoryWrite stores blank category content (white space only) as +// NULL and accepts any other content it receives. Since Phase 16.2 the +// content field is the blogeditor type, whose Decode already stored the HTML +// policy's output (classes.SanitizeHTMLReport) and told the administrator +// what it removed, so an admin save is never refused because of markup +// (D-10 amended, see the bluemonday decision note). Rows written by SQL or +// by the import commands do not pass through here and are not sanitized by +// it. func prepareCategoryWrite(cat *models.Category) error { if cat.Content == nil { return nil } if strings.TrimSpace(*cat.Content) == "" { cat.Content = nil - return nil - } - if err := classes.CheckHTML(*cat.Content); err != nil { - return &cabana.ValidationError{Details: map[string]any{ - "content": []string{err.Error()}, - }} } return nil } diff --git a/controllers/posts_editor_smoke_test.go b/controllers/posts_editor_smoke_test.go index 93bdda3..010b2b8 100644 --- a/controllers/posts_editor_smoke_test.go +++ b/controllers/posts_editor_smoke_test.go @@ -1,6 +1,7 @@ package controllers import ( + "bytes" "context" "errors" "os" @@ -13,8 +14,10 @@ import ( "git.golem15.com/golem15/summercms/modules/backpack" "git.golem15.com/golem15/summercms/modules/bouncer" "git.golem15.com/golem15/summercms/modules/cabana" + "git.golem15.com/golem15/summercms/modules/lagoon/attach" "git.golem15.com/golem15/summercms/modules/pact" "git.golem15.com/golem15/summercms/modules/phrasebook" + "gocloud.dev/blob/memblob" "gorm.io/gorm" ) @@ -334,3 +337,206 @@ func TestBlogEditorTracerSmoke(t *testing.T) { } }) } + +// pngBytes, jpegBytes and htmlBytes are file heads http.DetectContentType +// recognizes. +var ( + pngBytes = append([]byte("\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR"), make([]byte, 64)...) + jpegBytes = append([]byte("\xff\xd8\xff\xe0\x00\x10JFIF\x00"), make([]byte, 64)...) + gifBytes = append([]byte("GIF89a"), make([]byte, 64)...) + webpBytes = append([]byte("RIFF\x24\x00\x00\x00WEBPVP8 "), make([]byte, 64)...) + htmlBytes = []byte("") + svgBytes = []byte(``) +) + +// TestBlogEditorUploadSmoke covers plan 03 Task 2: the media storage shared +// by the public upload route and the editor Upload, the editor Upload's +// magic-byte check and safe names, and category content on blogeditor. +func TestBlogEditorUploadSmoke(t *testing.T) { + bucket := memblob.OpenBucket(nil) + t.Cleanup(func() { _ = bucket.Close() }) + app := backpack.New(nil) + if err := app.Publish(bucket); err != nil { + t.Fatal(err) + } + ctx := context.Background() + stored := func(t *testing.T, key string) []byte { + t.Helper() + data, err := bucket.ReadAll(ctx, key) + if err != nil { + t.Fatalf("read %s: %v", key, err) + } + return data + } + mediaError := func(t *testing.T, err error, message string) { + t.Helper() + var refused *classes.MediaError + if !errors.As(err, &refused) || refused.Field != "file" || refused.Message != message { + t.Fatalf("error = %v, want MediaError on file %q", err, message) + } + } + + t.Run("StoreMedia keeps the public route's layout", func(t *testing.T) { + got, err := classes.StoreMedia(ctx, bucket, "photos", "cover.png", bytes.NewReader(pngBytes)) + if err != nil { + t.Fatal(err) + } + if got.Path != "/journal/photos/cover.png" || got.URL != attach.PublicURL("journal/photos/cover.png") { + t.Fatalf("StoreMedia = %+v", got) + } + if !bytes.Equal(stored(t, "journal/photos/cover.png"), pngBytes) { + t.Fatal("stored bytes differ") + } + again, err := classes.StoreMedia(ctx, bucket, "/journal/photos/", `C:\up\cover.png`, bytes.NewReader(pngBytes)) + if err != nil { + t.Fatal(err) + } + if again.Path != "/journal/photos/cover-1.png" { + t.Fatalf("second StoreMedia path = %q, want the unique suffix", again.Path) + } + }) + + t.Run("StoreMedia refuses other extensions and large files", func(t *testing.T) { + _, err := classes.StoreMedia(ctx, bucket, "", "logo.svg", bytes.NewReader(svgBytes)) + mediaError(t, err, "The file must be an image.") + big := append(append([]byte{}, pngBytes...), make([]byte, classes.MediaMaxBytes)...) + _, err = classes.StoreMedia(ctx, bucket, "", "big.png", bytes.NewReader(big)) + mediaError(t, err, "The file may not be greater than 10240 kilobytes.") + if exists, _ := bucket.Exists(ctx, "journal/big.png"); exists { + t.Fatal("a refused file was stored") + } + if _, err := classes.StoreMedia(ctx, nil, "", "a.png", bytes.NewReader(pngBytes)); err == nil { + t.Fatal("StoreMedia without a bucket = nil error") + } else if errors.As(err, new(*classes.MediaError)) { + t.Fatalf("StoreMedia without a bucket = %v, want an opaque error", err) + } + }) + + types := classes.BlogEditorTypes(func() *backpack.App { return app }) + for _, ft := range types { + if ft.Upload == nil { + t.Fatalf("%s has no Upload", ft.Name) + } + } + upload := func(ft pact.FormFieldType, options map[string]any, name string, body []byte) (pact.FormFieldUploadResult, error) { + return ft.Upload(ctx, pact.FormFieldUpload{Field: "content", Options: options, Filename: name, Body: bytes.NewReader(body)}) + } + refused := func(t *testing.T, err error, message string) { + t.Helper() + var fe *pact.FormFieldError + if !errors.As(err, &fe) || len(fe.Messages) != 1 || fe.Messages[0] != message { + t.Fatalf("error = %v, want FormFieldError %q", err, message) + } + } + posts := map[string]any{"htmlColumn": "content_html", "uploadFolder": "posts"} + + t.Run("editor upload stores a sniffed image under a safe name", func(t *testing.T) { + got, err := upload(types[1], posts, "Zdjęcie 1.PNG", pngBytes) + if err != nil { + t.Fatal(err) + } + if got.URL != attach.PublicURL("journal/posts/zdj-cie-1.png") { + t.Fatalf("URL = %q", got.URL) + } + if !bytes.Equal(stored(t, "journal/posts/zdj-cie-1.png"), pngBytes) { + t.Fatal("stored bytes differ from the upload") + } + for name, body := range map[string][]byte{"a.jpg": jpegBytes, "b.jpeg": jpegBytes, "c.gif": gifBytes, "d.webp": webpBytes} { + if _, err := upload(types[0], map[string]any{"uploadFolder": "categories"}, name, body); err != nil { + t.Fatalf("upload %s: %v", name, err) + } + } + got, err = upload(types[0], nil, "...", pngBytes) + if err != nil { + t.Fatal(err) + } + if !strings.HasPrefix(got.URL, attach.PublicURL("journal/editor/")) { + t.Fatalf("default folder URL = %q, want journal/editor/", got.URL) + } + }) + + t.Run("editor upload refuses what does not sniff as its image type", func(t *testing.T) { + const invalid = "golem15.journal::lang.editor.image_invalid" + _, err := upload(types[1], posts, "x.png", htmlBytes) + refused(t, err, invalid) + _, err = upload(types[1], posts, "logo.svg", svgBytes) + refused(t, err, invalid) + _, err = upload(types[1], posts, "photo.jpg", pngBytes) + refused(t, err, invalid) + _, err = upload(types[1], posts, "photo.svg", pngBytes) + refused(t, err, "The file must be an image.") + big := append(append([]byte{}, pngBytes...), make([]byte, classes.MediaMaxBytes)...) + _, err = upload(types[1], posts, "big.png", big) + refused(t, err, "The file may not be greater than 10240 kilobytes.") + }) + + t.Run("editor upload without a bucket is an opaque error", func(t *testing.T) { + bare := classes.BlogEditorTypes(func() *backpack.App { return backpack.New(nil) }) + _, err := bare[1].Upload(ctx, pact.FormFieldUpload{Field: "content", Options: posts, Filename: "a.png", Body: bytes.NewReader(pngBytes)}) + if err == nil || errors.As(err, new(*pact.FormFieldError)) { + t.Fatalf("error = %v, want an opaque error", err) + } + nilApp := classes.BlogEditorTypes(nil) + if _, err := nilApp[1].Upload(ctx, pact.FormFieldUpload{Field: "content", Filename: "a.png", Body: bytes.NewReader(pngBytes)}); err == nil { + t.Fatal("upload without an app = nil error") + } + }) + + t.Run("category content compiles as blogeditor", func(t *testing.T) { + schema, err := cabana.CompileFormWithTypes("golem15.journal", categoriesAdminController{}, os.DirFS(".."), classes.BlogEditorTypes(nil)) + if err != nil { + t.Fatal(err) + } + content := formField(t, schema, "content") + if content.Type != "blogeditor" || content.Element != classes.BlogEditorElement || content.Translatable { + t.Fatalf("content = type %q element %q translatable %v", content.Type, content.Element, content.Translatable) + } + if content.TypeOptions["uploadFolder"] != "categories" || content.Size != "huge" { + t.Fatalf("content typeOptions = %#v size %q", content.TypeOptions, content.Size) + } + }) + + plain := types[0] + t.Run("category load serves the stored HTML", func(t *testing.T) { + html := "

Pillar

" + for _, tc := range []struct { + value any + want string + }{{&html, html}, {(*string)(nil), ""}, {nil, ""}, {"plain text", "plain text"}} { + got, err := plain.Load(ctx, pact.FormFieldValue{Field: "content", Options: map[string]any{"uploadFolder": "categories"}, Default: true, Value: tc.value}) + if err != nil || got != tc.want { + t.Fatalf("Load(%#v) = %#v, %v, want %q", tc.value, got, err, tc.want) + } + } + }) + + t.Run("category decode strips with a notice", func(t *testing.T) { + got, err := plain.Decode(ctx, pact.FormFieldValue{Field: "content", Default: true, Value: `

x

`}) + if err != nil { + t.Fatal(err) + } + if got.Value != "

x

" || len(got.Notices) != 1 { + t.Fatalf("Decode = %#v", got) + } + if n := got.Notices[0]; n.Message != "golem15.journal::lang.editor.removed_attributes" || n.Count != 1 || n.Params["names"] != "onclick" { + t.Fatalf("notice = %#v", n) + } + }) + + t.Run("category hooks map blank to NULL and accept markup", func(t *testing.T) { + ctl := categoriesAdminController{} + blank := " \n\t" + cat := &models.Category{Content: &blank} + if err := ctl.FormBeforeUpdate(ctx, cat); err != nil || cat.Content != nil { + t.Fatalf("blank: content = %v, err = %v, want NULL", cat.Content, err) + } + body := `

x

` + cat = &models.Category{Content: &body} + if err := ctl.FormBeforeCreate(ctx, cat); err != nil { + t.Fatalf("FormBeforeCreate refused markup: %v", err) + } + if cat.Content == nil || *cat.Content != body { + t.Fatalf("content = %v, want it as received", cat.Content) + } + }) +} diff --git a/journal_api_categories_test.go b/journal_api_categories_test.go index 3e81e95..5397196 100644 --- a/journal_api_categories_test.go +++ b/journal_api_categories_test.go @@ -7,6 +7,7 @@ import ( "reflect" "sort" "strconv" + "strings" "testing" "time" @@ -156,8 +157,8 @@ func TestCategoriesAdminContent(t *testing.T) { continue } contentAt = i - if f.Type != "textarea" || f.Size != "huge" { - t.Fatalf("content field = %+v, want textarea size huge", f) + if f.Type != "blogeditor" || f.Size != "huge" { + t.Fatalf("content field = %+v, want blogeditor size huge", f) } } if contentAt < 1 || form.Fields[contentAt-1].Name != "description" { @@ -172,6 +173,9 @@ func TestCategoriesAdminContent(t *testing.T) { "slug": slug, "content": safe, }) + if notices := adminNotices(t, rec); notices != nil { + t.Fatalf("safe create carries meta.notices: %s", rec.Body.String()) + } var created struct { ID uint `json:"id"` } @@ -194,55 +198,47 @@ func TestCategoriesAdminContent(t *testing.T) { t.Fatalf("API content = %#v, want %q", rows["acme-news"]["content"], safe) } - assertContentRefused := func(t *testing.T, rec *httptest.ResponseRecorder) { + // assertContentStripped checks a save the blogeditor Decode stripped + // (D-10 amended, D-19): the status, and one meta.notices entry for + // content whose message names what was removed. + assertContentStripped := func(t *testing.T, rec *httptest.ResponseRecorder, status int, removed string) { t.Helper() - if rec.Code != http.StatusUnprocessableEntity { - t.Fatalf("status = %d, want 422: %s", rec.Code, rec.Body.String()) + if rec.Code != status { + t.Fatalf("status = %d, want %d: %s", rec.Code, status, rec.Body.String()) } - var body struct { - Error struct { - Code string `json:"code"` - Details map[string]any `json:"details"` - } `json:"error"` - } - if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil { - t.Fatalf("422 body %s: %v", rec.Body.String(), err) - } - if body.Error.Code != "validation_failed" { - t.Fatalf("error code = %q, want validation_failed", body.Error.Code) - } - if _, ok := body.Error.Details["content"]; !ok { - t.Fatalf("details do not name content: %s", rec.Body.String()) + notices := adminNotices(t, rec) + if len(notices) != 1 || notices[0].Field != "content" || !strings.Contains(notices[0].Message, removed) { + t.Fatalf("meta.notices = %#v, want one content notice naming %s", notices, removed) } } - t.Run("unsafe create refused", func(t *testing.T) { + t.Run("unsafe create stripped", func(t *testing.T) { rec := env.call(t, http.MethodPost, adminAPI(adminCategoriesPath), token, map[string]any{ "name": map[string]any{"en": "Unsafe", "pl": "Niebezpieczna"}, "slug": map[string]any{"en": "unsafe-acme", "pl": "niebezpieczna-acme"}, "content": "

x

", }) - assertContentRefused(t, rec) - var n int64 - if err := env.gdb.Model(&models.Category{}).Where("slug = ?", "unsafe-acme").Count(&n).Error; err != nil { - t.Fatal(err) + assertContentStripped(t, rec, http.StatusCreated, "onclick") + var cat models.Category + if err := env.gdb.Where("slug = ?", "unsafe-acme").Take(&cat).Error; err != nil { + t.Fatalf("unsafe-acme row: %v", err) } - if n != 0 { - t.Fatalf("unsafe-acme rows = %d, want 0", n) + if cat.Content == nil || *cat.Content != "

x

" { + t.Fatalf("stored content = %#v, want

x

", cat.Content) } }) itemPath := adminAPI(adminCategoriesPath + "/" + strconv.FormatUint(uint64(created.ID), 10)) - t.Run("unsafe update refused", func(t *testing.T) { + t.Run("unsafe update stripped", func(t *testing.T) { rec := env.call(t, http.MethodPut, itemPath, token, map[string]any{ "name": name, "slug": slug, "content": "x", }) - assertContentRefused(t, rec) - if got := storedContent(t); got == nil || *got != safe { - t.Fatalf("stored content after refused update = %#v, want %q", got, safe) + assertContentStripped(t, rec, http.StatusOK, "removed: a") + if got := storedContent(t); got == nil || *got != "x" { + t.Fatalf("stored content after stripped update = %#v, want x", got) } }) diff --git a/lang/en/lang.yaml b/lang/en/lang.yaml index a89dcb2..ece1f4f 100644 --- a/lang/en/lang.yaml +++ b/lang/en/lang.yaml @@ -111,7 +111,7 @@ category: name_placeholder: New category name description: Description content: "Content" - content_comment: "HTML shown on the category page above its post list. Unsafe HTML (scripts, frames, event handlers, javascript, vbscript and data URLs) is refused." + content_comment: "HTML shown on the category page above its post list. Unsafe HTML (scripts, frames, event handlers, javascript, vbscript and data URLs) is removed on save, and the form names what was removed." slug: Slug slug_placeholder: new-category-slug posts: Posts diff --git a/lang/pl/lang.yaml b/lang/pl/lang.yaml index 94e4e72..1be4249 100644 --- a/lang/pl/lang.yaml +++ b/lang/pl/lang.yaml @@ -75,7 +75,7 @@ category: name_placeholder: Nazwa nowej kategorii description: Opis content: "Treść" - content_comment: "Kod HTML wyświetlany na stronie kategorii nad listą wpisów. Niebezpieczny HTML (skrypty, ramki, atrybuty zdarzeń, adresy javascript, vbscript i data) jest odrzucany." + content_comment: "Kod HTML wyświetlany na stronie kategorii nad listą wpisów. Niebezpieczny HTML (skrypty, ramki, atrybuty zdarzeń, adresy javascript, vbscript i data) jest usuwany przy zapisie, a formularz podaje, co usunięto." slug: Alias slug_placeholder: alias-nowej-kategorii posts: Posty diff --git a/models/category/fields.yaml b/models/category/fields.yaml index 05f90c3..9ef0f1b 100644 --- a/models/category/fields.yaml +++ b/models/category/fields.yaml @@ -25,6 +25,7 @@ fields: content: label: golem15.journal::lang.category.content comment: golem15.journal::lang.category.content_comment - type: textarea + type: blogeditor + uploadFolder: categories size: huge span: full