diff --git a/README.md b/README.md index 9ba8046..50a9c61 100644 --- a/README.md +++ b/README.md @@ -32,10 +32,13 @@ The frontend (a Nuxt layer or app) asks `GET /_seo/api/v1/resolve?url=...` for t - **AI bots.** Switches for OAI-SearchBot, Claude-SearchBot, PerplexityBot (search), ChatGPT-User, Claude-User (user fetch), GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta-ExternalAgent (training), all allowed by default. A bot switched off gets `User-agent: ` / `Disallow: /`. - **Verification files** stored in the settings are served by `GET /_seo/api/v1/verification/{name}` for the frontend to map at the site root. - **Non-production lock.** Outside the `production` environment every resolve answer is `noindex, nofollow` with `staging: true`, and `/robots.txt` is `User-agent: *` / `Disallow: /` with `X-Robots-Tag: noindex, nofollow`, whatever is stored. It depends on the environment only; no setting turns it off. -- **History** (`golem15_seo_revisions`): every SEO write (the SEO tab, rules, redirects, robots.txt, settings) is recorded with author, time, via (admin, MCP, or auto for address changes) and the changed fields before and after. Deleted records keep their values in history. +- **History** (`golem15_seo_revisions`): every SEO write (the SEO tab, rules, redirects, robots.txt, settings, the 404 log) is recorded with author, time, via (admin, MCP, import, auto for address changes, undo) and the changed fields before and after. Deleted records keep their values in history. +- **Undo** of one change or of a whole batch, from the History preview (record actions **Undo this change** and **Undo the whole batch**) and through MCP. An undo is refused when a later change touched the same values (`history.undo_stale`, naming who and when), when it would create a redirect loop (`history.undo_loop`) or when the record is gone (`history.undo_gone`); an undo is itself recorded. +- **Batches** (D-12): a list of changes (SEO by URL, redirects, URL rules) applied all or nothing under one batch id. A dry run answers the diff and the conflicts (loops, chains, noindex on a published record's address, a canonical pointing to a redirected or noindex address) and writes nothing. +- **CSV import and export** of redirects, URL rules and the SEO of records (by URL), in the **Import and export** screen (element `golem15-seo-transfer`) and through MCP. Files are UTF-8 with a BOM, comma or semicolon; exported cells that start a formula (`=`, `+`, `-`, `@`, tab, CR) are prefixed with an apostrophe, which the import drops again. An import is previewed row by row (new, changed, unchanged, refused, with notes) and applied as one batch with via `import`, or not at all while a row is refused. The screen's apply token is an HMAC-SHA256 under the app key over the file and a fingerprint of the data the preview read; it is refused with `transfer.stale` when the data changed. The screen previews files up to 60 KB and downloads exports up to 250 KB; MCP imports take up to 900 KB per call and exports come in pages of 1000 rows. - **Record events.** A deleted record loses its entries after commit; a created record gets its address snapshot; an updated record whose address changed gets its 301. - **Entry generation.** The `golem15.seo.generate_entries` job (conga) creates missing entries with the current address of every published record and refreshes the effective-value cache; editor values are never changed. -- **Field types** `seotext`, `mlseotext`, `seorobots`, `mlseorobots`, `seoimage` and `robotstxt`, with the custom elements `golem15-seo-text`, `golem15-seo-robots`, `golem15-seo-serp` (the result preview widget), `golem15-seo-image`, `golem15-seo-robotstxt` (the robots.txt editor) and `golem15-seo-robotstester` (the tester widget). +- **Field types** `seotext`, `mlseotext`, `seorobots`, `mlseorobots`, `seoimage` and `robotstxt`, with the custom elements `golem15-seo-text`, `golem15-seo-robots`, `golem15-seo-serp` (the result preview widget), `golem15-seo-image`, `golem15-seo-robotstxt` (the robots.txt editor), `golem15-seo-robotstester` (the tester widget) and `golem15-seo-transfer` (the import and export widget). - **One permission**, `golem15.seo.manage`, for the SEO tab, rules, redirects, the 404 log, robots.txt, settings and history, in the admin and through MCP. - **Management API and MCP.** Every SEO admin operation has a JSON route under `/_seo/api/v1` and an MCP tool bound to it (pier, prefix `seo_`). Tools address pages by URL: a change goes to the record behind the URL, else to an exact URL rule, and the answer says where. A read grant sees only the read tools. `AdminOperations` maps each admin operation to its tool, and a test fails when one has none. - **Issue report.** Missing and duplicate titles and descriptions, URL rules overriding a record, redirect chains, canonicals pointing to a noindex or redirected address, and noindex addresses robots.txt blocks, each with the tool that fixes it. @@ -148,13 +151,17 @@ The management routes live under `/_seo/api/v1` (bucket `seo-api`, 300 requests | `show_robots`, `update_robots`, `test_robots` | `GET/PUT /robots`, `GET /robots/test?url=&agent=` | robots.txt, the AI bot switches and the tester on the saved file. | | `show_settings`, `update_settings` | `GET/PUT /settings` | The SEO settings and, under `types`, the per-type settings. | | `list_history`, `show_history` | `GET /history`, `GET /history/{id}` | History, filtered by `via`, `kind` and `batch`. | +| `undo_change`, `undo_batch` | `POST /history/{id}/undo`, `POST /history/batches/{batch}/undo` | Undo; refusals are `409` with `undo_stale`, `undo_loop`, `undo_gone` or `already_undone`. | +| `apply_batch` | `POST /batch` | Changes (`set_seo`, `create_redirect`, `update_redirect`, `delete_redirect`, `create_rule`, `update_rule`, `delete_rule`) all or nothing; `dry_run` answers the diff and the conflicts. A refused change is `422` (`redirect_loop` with the change and the path, or `batch_refused`). | +| `export_csv` | `GET /export?dataset=&delimiter=&page=` | One page of 1000 rows as CSV text. | +| `import_csv` | `POST /import` | A CSV file of a dataset; `dry_run` previews it. A file over 900 KB is `413`; `missing_column`, `unreadable` and `import_refused` are `422`. | | `list_issues` | `GET /issues` | The CMS-data issue report with a count per kind. | Lists take `page`, `per_page` (at most 200) and `search`, and answer `{"data": [...], "page", "per_page", "total"}`. GET tools are read-only; the deletes and `gone_not_found` are destructive. ### Admin -- Navigation item **SEO** (icon `search`), landing on **Redirects**, with the side menu **Redirects**, **404 log**, **URL rules**, **robots.txt** and **History**. +- Navigation item **SEO** (icon `search`), landing on **Redirects**, with the side menu **Redirects**, **404 log**, **URL rules**, **robots.txt**, **History** and **Import and export**. - Settings tile **SEO settings** (`seo_settings`), a link to the settings singleton controller. - Every SEO screen and the SEO tab require `golem15.seo.manage`. Without it an administrator sees no SEO tab, and a posted SEO value changes nothing. @@ -174,6 +181,10 @@ The element bundle (`assets/dist/seo.js` and `seo.css`) is built from `assets/sr | `classes.Owner`, `classes.URLSEOWrite`, `classes.URLSEOResult` | The owner of a URL (`classes.WrittenToRecord` or `classes.WrittenToRule`) and a write addressed by URL. | | `classes.Issue` | One finding of the issue report (`classes.IssueMissingTitle` and the other kinds). | | `classes.WithTx` | Puts a transaction on the context so every read of the service sees it. | +| `classes.BatchChange`, `classes.BatchOptions`, `classes.BatchResult`, `classes.BatchRefusedError` | A batch: `Service.RunBatch` runs it as a dry run or all or nothing (`classes.WithBatch` marks the history rows). | +| `classes.UndoRefusal`, `classes.Undoable` | `Service.Undo` and `Service.UndoBatch` and their refusals. | +| `classes.ReadCSV`, `classes.WriteCSV`, `classes.Export`, `classes.ImportRequest`, `classes.ImportResult` | CSV files and `Service.Export` and `Service.Import`; `Service.TransferToken` and `Service.ReadTransferToken` sign and verify the screen's apply token. | +| `controllers.TransferControllerID`, `controllers.TransferAction`, `controllers.HistoryActionUndo`, `controllers.HistoryActionUndoBatch` | The import and export screen, its widget action and the History record actions. | | `classes.Resolver` | `Resolve(ctx, url)` answers a `classes.Result` (actions `ok`, `not_found`, `redirect`, `gone`) with a source per value in `Result.Sources`; `ResolveRecord` resolves a record at its own address; `Strict` checks a path against the record's address. | | `classes.RedirectSet`, `classes.NewRedirectSet`, `classes.LoadRedirectSet` | A compiled redirect rule set: `Match` and `Follow` (`classes.Chain`, at most `classes.MaxRedirectHops`). | | `classes.RedirectCheck`, `classes.LoopError`, `classes.ProblemsError` | The save-time simulation's answer and the refusals of `SaveRedirect`. | @@ -222,4 +233,4 @@ go test ./... # with Docker: the Postgres smoke tests npm ci && npm run typecheck && bash scripts/check-dist.sh ``` -The smoke tests boot the plugin with fake content plugins (`acme.pages`, and `acme.blog` with a category segment for strict URLs) and exercise the SEO tab, the settings tabs, the resolver precedence, history, generation, the URL rule form, redirects (engine, loops and chains, address changes from the admin and from events), the 404 log, normalisation and strict URLs, robots.txt and verification files through the admin and public APIs, and the management API as an MCP client (`TestMCPSetURLSeoTracer`, `TestManagementAPI`, `TestIssues`). `TestEveryAdminOperationHasTool` and `TestMCPReadToolsAreGET` run without Docker. +The smoke tests boot the plugin with fake content plugins (`acme.pages`, and `acme.blog` with a category segment for strict URLs) and exercise the SEO tab, the settings tabs, the resolver precedence, history, generation, the URL rule form, redirects (engine, loops and chains, address changes from the admin and from events), the 404 log, normalisation and strict URLs, robots.txt and verification files through the admin and public APIs, and the management API as an MCP client (`TestMCPSetURLSeoTracer`, `TestManagementAPI`, `TestIssues`), batches, undo and CSV (`TestBatch`, `TestUndo`, `TestCSVTransfer`, and the `mcp` and `import` cases of `TestLoopRefused`). `TestEveryAdminOperationHasTool` and `TestMCPReadToolsAreGET` run without Docker. diff --git a/admin_navigation.go b/admin_navigation.go index 62d04b0..bbbb8c5 100644 --- a/admin_navigation.go +++ b/admin_navigation.go @@ -52,6 +52,13 @@ func (*Plugin) Navigation() []pact.NavigationItem { Permissions: []string{controllers.PermissionManage}, Controller: controllers.HistoryControllerID, }, + { + Code: "transfer", + Label: "golem15.seo::lang.nav.transfer", + Icon: "download", + Permissions: []string{controllers.PermissionManage}, + Controller: controllers.TransferControllerID, + }, }, }} } diff --git a/batch_test.go b/batch_test.go new file mode 100644 index 0000000..a48df0f --- /dev/null +++ b/batch_test.go @@ -0,0 +1,322 @@ +package seo_test + +import ( + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +// snapshot dumps every SEO table the batch and undo write, so a dry run +// can be checked to leave rows and values exactly as they were. +func (e *testEnv) snapshot(t *testing.T) string { + t.Helper() + var b strings.Builder + for _, q := range []string{ + `SELECT id, source, match, target, status, origin, note FROM golem15_seo_redirects ORDER BY id`, + `SELECT id, url, match, title, description, canonical, robots FROM golem15_seo_rules ORDER BY id`, + `SELECT id, model_type, record_id, locale, title, description, canonical, robots, og_image FROM golem15_seo_entries ORDER BY id`, + `SELECT id, via, subject_kind, subject_id, batch_id, undone_at FROM golem15_seo_revisions ORDER BY id`, + `SELECT id, content, bots::text FROM golem15_seo_robots ORDER BY id`, + } { + rows, err := e.sql.QueryContext(t.Context(), q) + if err != nil { + t.Fatal(err) + } + cols, _ := rows.Columns() + n := 0 + for rows.Next() { + vals := make([]any, len(cols)) + ptrs := make([]any, len(cols)) + for i := range vals { + ptrs[i] = &vals[i] + } + if err := rows.Scan(ptrs...); err != nil { + t.Fatal(err) + } + fmt.Fprintf(&b, "%v\n", vals) + n++ + } + _ = rows.Close() + fmt.Fprintf(&b, "-- %d rows\n", n) + } + return b.String() +} + +// redirectChange is a create_redirect change of a batch request. +func redirectChange(source, target string) map[string]any { + return map[string]any{"op": "create_redirect", "redirect": map[string]any{"source": source, "target": target}} +} + +// conflictKinds lists the kinds of a batch answer's conflicts. +func conflictKinds(body map[string]any) map[string]bool { + out := map[string]bool{} + list, _ := body["conflicts"].([]any) + for _, c := range list { + m, _ := c.(map[string]any) + out[fmt.Sprint(m["kind"])] = true + } + return out +} + +// TestBatch checks D-12 (SC-4): a dry run answers the diff and the +// conflicts (a chain, noindex on a published record's address, a +// canonical pointing to a redirect the batch creates) and leaves every +// row as it was; a batch with a loop is refused as a whole with 422 naming +// the change; an applied batch writes every change under one batch id. +func TestBatch(t *testing.T) { + env := bootEnv(t) + token := env.admin(t, true) + env.createPage(t, token, map[string]any{"title": "Page A", "slug": "a", "excerpt": "About A", "published": true}) + env.createPage(t, token, map[string]any{"title": "Page B", "slug": "b", "excerpt": "About B", "published": true}) + agent := env.principal(t, env.admin(t, false, "golem15.seo.manage")) + + changes := []any{ + redirectChange("/r1", "/pages/a"), + redirectChange("/r2", "/pages/b"), + redirectChange("/r3", "/r1"), + map[string]any{"op": "set_seo", "url": "/pages/a", "values": map[string]any{"robots": "noindex"}}, + map[string]any{"op": "set_seo", "url": "/pages/b", "values": map[string]any{"canonical": "/r2"}}, + } + + before := env.snapshot(t) + dry := env.mcpExpect(t, http.StatusOK, http.MethodPost, "/_seo/api/v1/batch", agent, map[string]any{"dry_run": true, "changes": changes}) + if after := env.snapshot(t); after != before { + t.Fatalf("a dry run changed the data:\nbefore:\n%s\nafter:\n%s", before, after) + } + if dry["dry_run"] != true || dry["applied"] != false || dry["count"] != float64(5) { + t.Fatalf("dry run = %v", dry) + } + items, _ := dry["changes"].([]any) + if len(items) != 5 { + t.Fatalf("dry run changes = %v", dry["changes"]) + } + for i, item := range items { + if diff, _ := item.(map[string]any)["diff"].([]any); len(diff) == 0 { + t.Fatalf("change %d has no diff: %v", i, item) + } + } + kinds := conflictKinds(dry) + for _, k := range []string{"redirect_chain", "noindex_published", "canonical_redirected"} { + if !kinds[k] { + t.Fatalf("dry run conflicts %v lack %s: %v", kinds, k, dry["conflicts"]) + } + } + + // A loop refuses the whole batch. + loop := env.mcp(t, http.MethodPost, "/_seo/api/v1/batch", agent, map[string]any{"changes": []any{ + redirectChange("/l0", "/pages/a"), + redirectChange("/l1", "/l2"), + redirectChange("/l2", "/l1"), + }}) + if loop.Code != http.StatusUnprocessableEntity || !strings.Contains(loop.Body.String(), `"error":"redirect_loop"`) || !strings.Contains(loop.Body.String(), `"change":2`) { + t.Fatalf("loop batch = %d %s, want 422 redirect_loop on change 2", loop.Code, loop.Body.String()) + } + if after := env.snapshot(t); after != before { + t.Fatalf("a refused batch wrote:\n%s", after) + } + invalid := env.mcp(t, http.MethodPost, "/_seo/api/v1/batch", agent, map[string]any{"changes": []any{ + redirectChange("/ok", "/pages/a"), + map[string]any{"op": "set_seo", "url": "/pages/a", "values": map[string]any{"robots": "sideways"}}, + }}) + if invalid.Code != http.StatusUnprocessableEntity || !strings.Contains(invalid.Body.String(), `"change":1`) { + t.Fatalf("invalid batch = %d %s, want 422 naming change 1", invalid.Code, invalid.Body.String()) + } + if after := env.snapshot(t); after != before { + t.Fatalf("a refused batch wrote:\n%s", after) + } + if rec := env.mcp(t, http.MethodPost, "/_seo/api/v1/batch", agent, map[string]any{"changes": []any{map[string]any{"op": "explode"}}}); rec.Code != http.StatusUnprocessableEntity { + t.Fatalf("unknown op = %d, want 422", rec.Code) + } + + applied := env.mcpExpect(t, http.StatusOK, http.MethodPost, "/_seo/api/v1/batch", agent, map[string]any{"changes": changes}) + batch, _ := applied["batch_id"].(string) + if applied["applied"] != true || batch == "" { + t.Fatalf("apply = %v", applied) + } + var rows []struct { + Via string + BatchID string + } + if err := env.db.Raw(`SELECT via, COALESCE(batch_id, '') AS batch_id FROM golem15_seo_revisions ORDER BY id`).Scan(&rows).Error; err != nil { + t.Fatal(err) + } + in := 0 + for _, r := range rows { + if r.BatchID == batch { + in++ + if r.Via != "mcp" { + t.Fatalf("batch revision via %s, want mcp", r.Via) + } + } + } + if in != 5 { + t.Fatalf("revisions in batch %s = %d, want 5 (%+v)", batch, in, rows) + } + if got := env.resolve(t, "/r3"); got["action"] != "redirect" { + t.Fatalf("resolve /r3 = %v, want the applied redirect", got) + } + if hist := env.mcpExpect(t, http.StatusOK, http.MethodGet, "/_seo/api/v1/history?batch="+batch, agent, nil); hist["total"] != float64(5) { + t.Fatalf("history?batch = %v", hist) + } +} + +// historyIDs returns the ids of the revisions of a subject kind, oldest +// first. +func (e *testEnv) historyIDs(t *testing.T, kind string) []string { + t.Helper() + var ids []uint + if err := e.db.Raw(`SELECT id FROM golem15_seo_revisions WHERE subject_kind = ? ORDER BY id`, kind).Scan(&ids).Error; err != nil { + t.Fatal(err) + } + out := make([]string, len(ids)) + for i, id := range ids { + out[i] = fmt.Sprint(id) + } + return out +} + +// entryTitle reads a page's stored SEO title in en. +func (e *testEnv) entryTitle(t *testing.T, id string) string { + t.Helper() + var title string + if err := e.db.Raw(`SELECT COALESCE(MAX(title), '') FROM golem15_seo_entries WHERE model_type = ? AND record_id = ? AND locale = 'en'`, pagesModel, id).Scan(&title).Error; err != nil { + t.Fatal(err) + } + return title +} + +// TestUndo checks D-13 undo (T-32-27): one change is undone with its +// before value and an undo revision; an undo that would overwrite a newer +// change, recreate a loop or touch a deleted record is refused with +// history.undo_stale, undo_loop and undo_gone; a batch is undone as a +// whole; the history screen offers both as record actions. +func TestUndo(t *testing.T) { + env := bootEnv(t) + token := env.admin(t, true) + pageA := env.createPage(t, token, map[string]any{"title": "Page A", "slug": "a", "excerpt": "About A", "published": true}) + pageB := env.createPage(t, token, map[string]any{"title": "Page B", "slug": "b", "excerpt": "About B", "published": true}) + agent := env.principal(t, env.admin(t, false, "golem15.seo.manage")) + set := func(url, field, value string) { + env.mcpExpect(t, http.StatusOK, http.MethodPut, "/_seo/api/v1/seo", agent, map[string]any{"url": url, "values": map[string]any{field: value}}) + } + undo := func(id string) *httptest.ResponseRecorder { + return env.mcp(t, http.MethodPost, "/_seo/api/v1/history/"+id+"/undo", agent, map[string]any{}) + } + + // One change. + set("/pages/a", "title", "One") + r1 := env.historyIDs(t, "entry")[0] + if rec := undo(r1); rec.Code != http.StatusOK { + t.Fatalf("undo = %d %s", rec.Code, rec.Body.String()) + } + if got := env.entryTitle(t, pageA); got != "" { + t.Fatalf("title after undo = %q, want the value before", got) + } + var vias []string + if err := env.db.Raw(`SELECT via FROM golem15_seo_revisions WHERE subject_kind = 'entry' ORDER BY id`).Scan(&vias).Error; err != nil { + t.Fatal(err) + } + if len(vias) != 2 || vias[1] != "undo" { + t.Fatalf("entry revisions after undo = %v, want an undo revision", vias) + } + if rec := undo(r1); rec.Code != http.StatusConflict { + t.Fatalf("second undo = %d, want 409", rec.Code) + } + + // Stale: a newer change of the same field. + set("/pages/a", "title", "Two") + ids := env.historyIDs(t, "entry") + r2 := ids[len(ids)-1] + set("/pages/a", "title", "Three") + rec := undo(r2) + if rec.Code != http.StatusConflict || !strings.Contains(rec.Body.String(), `"error":"undo_stale"`) || !strings.Contains(rec.Body.String(), "MCP: ") || !strings.Contains(rec.Body.String(), "Nothing was undone") { + t.Fatalf("stale undo = %d %s, want 409 undo_stale naming the author", rec.Code, rec.Body.String()) + } + if got := env.entryTitle(t, pageA); got != "Three" { + t.Fatalf("title after a refused undo = %q, want Three", got) + } + + // Loop: undoing a target change would recreate a loop. + created := env.mcpExpect(t, http.StatusCreated, http.MethodPost, "/_seo/api/v1/redirects", agent, map[string]any{"source": "/u1", "target": "/u2"}) + u1 := fmt.Sprint(created["data"].(map[string]any)["id"]) + env.mcpExpect(t, http.StatusOK, http.MethodPut, "/_seo/api/v1/redirects/"+u1, agent, map[string]any{"target": "/u3"}) + ids = env.historyIDs(t, "redirect") + retarget := ids[len(ids)-1] + env.mcpExpect(t, http.StatusCreated, http.MethodPost, "/_seo/api/v1/redirects", agent, map[string]any{"source": "/u2", "target": "/u1"}) + rec = undo(retarget) + if rec.Code != http.StatusConflict || !strings.Contains(rec.Body.String(), `"error":"undo_loop"`) || !strings.Contains(rec.Body.String(), "/u1") { + t.Fatalf("loop undo = %d %s, want 409 undo_loop with the path", rec.Code, rec.Body.String()) + } + + // Gone: the record was deleted. + set("/pages/b", "description", "Gone soon") + ids = env.historyIDs(t, "entry") + onB := ids[len(ids)-1] + env.expect(t, http.StatusOK, http.MethodDelete, adminAPI("/acme/pages/pages/"+pageB), token, nil) + rec = undo(onB) + if rec.Code != http.StatusConflict || !strings.Contains(rec.Body.String(), `"error":"undo_gone"`) { + t.Fatalf("gone undo = %d %s, want 409 undo_gone", rec.Code, rec.Body.String()) + } + + // A batch is undone as a whole. + applied := env.mcpExpect(t, http.StatusOK, http.MethodPost, "/_seo/api/v1/batch", agent, map[string]any{"changes": []any{ + redirectChange("/b1", "/pages/a"), + redirectChange("/b2", "/pages/a"), + map[string]any{"op": "set_seo", "url": "/pages/a", "values": map[string]any{"title": "Batched"}}, + }}) + batch := applied["batch_id"].(string) + before := env.redirectCount(t) + out := env.mcpExpect(t, http.StatusOK, http.MethodPost, "/_seo/api/v1/history/batches/"+batch+"/undo", agent, map[string]any{}) + if out["count"] != float64(3) { + t.Fatalf("batch undo = %v, want 3", out) + } + if n := env.redirectCount(t); n != before-2 { + t.Fatalf("redirects after the batch undo = %d, want %d", n, before-2) + } + if got := env.entryTitle(t, pageA); got != "Three" { + t.Fatalf("title after the batch undo = %q, want Three", got) + } + var open int64 + if err := env.db.Raw(`SELECT COUNT(*) FROM golem15_seo_revisions WHERE batch_id = ? AND undone_at IS NULL`, batch).Scan(&open).Error; err != nil || open != 0 { + t.Fatalf("batch revisions not undone = %d err=%v", open, err) + } + if rec := env.mcp(t, http.MethodPost, "/_seo/api/v1/history/batches/"+batch+"/undo", agent, map[string]any{}); rec.Code != http.StatusNotFound { + t.Fatalf("second batch undo = %d, want 404", rec.Code) + } + + // The history screen's record actions. + set("/pages/a", "title", "Four") + ids = env.historyIDs(t, "entry") + last := ids[len(ids)-1] + api := adminAPI("/golem15/seo/history") + show := env.expect(t, http.StatusOK, http.MethodGet, api+"/"+last, token, nil).Body.String() + if !strings.Contains(show, `"undo"`) || strings.Contains(show, `"undo_batch"`) { + t.Fatalf("history preview actions = %s, want undo and no undo_batch for an unbatched change", show) + } + res := env.expect(t, http.StatusOK, http.MethodPost, api+"/"+last+"/actions/undo", token, map[string]any{}) + if !strings.Contains(res.Body.String(), "Change undone.") { + t.Fatalf("undo action = %s", res.Body.String()) + } + set("/pages/a", "title", "Five") + set("/pages/a", "title", "Six") + ids = env.historyIDs(t, "entry") + stale := ids[len(ids)-2] + if rec := env.call(t, http.MethodPost, api+"/"+stale+"/actions/undo", token, map[string]any{}); rec.Code != http.StatusForbidden || !strings.Contains(rec.Body.String(), "has changed since") { + t.Fatalf("stale undo action = %d %s, want 403 with history.undo_stale", rec.Code, rec.Body.String()) + } + applied = env.mcpExpect(t, http.StatusOK, http.MethodPost, "/_seo/api/v1/batch", agent, map[string]any{"changes": []any{redirectChange("/b3", "/pages/a"), redirectChange("/b4", "/pages/a")}}) + var inBatch uint + if err := env.db.Raw(`SELECT MIN(id) FROM golem15_seo_revisions WHERE batch_id = ?`, applied["batch_id"]).Scan(&inBatch).Error; err != nil { + t.Fatal(err) + } + show = env.expect(t, http.StatusOK, http.MethodGet, api+"/"+fmt.Sprint(inBatch), token, nil).Body.String() + if !strings.Contains(show, `"undo_batch"`) { + t.Fatalf("batched entry preview = %s, want undo_batch", show) + } + res = env.expect(t, http.StatusOK, http.MethodPost, api+"/"+fmt.Sprint(inBatch)+"/actions/undo_batch", token, map[string]any{}) + if !strings.Contains(res.Body.String(), "Batch undone.") { + t.Fatalf("undo_batch action = %s", res.Body.String()) + } +} diff --git a/classes/batch.go b/classes/batch.go new file mode 100644 index 0000000..27b49db --- /dev/null +++ b/classes/batch.go @@ -0,0 +1,475 @@ +package classes + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + "fmt" + "strings" + + "git.golem15.com/golem15/sm-seo-plugin/models" + "git.golem15.com/golem15/summercms/modules/lagoon" + "gorm.io/gorm" +) + +// MaxBatchChanges caps the changes of one batch. +const MaxBatchChanges = 2000 + +// The operations of a batch change. +const ( + OpSetSEO = "set_seo" + OpCreateRedirect = "create_redirect" + OpUpdateRedirect = "update_redirect" + OpDeleteRedirect = "delete_redirect" + OpCreateRule = "create_rule" + OpUpdateRule = "update_rule" + OpDeleteRule = "delete_rule" +) + +// The conflict kinds a batch reports (D-12). +const ( + ConflictLoop = "redirect_loop" + ConflictChain = "redirect_chain" + ConflictNoindexPublished = "noindex_published" + ConflictCanonicalRedirected = "canonical_redirected" + ConflictCanonicalNoindex = "canonical_noindex" +) + +// RedirectFields are the fields of a redirect write; a nil field is left +// as it is (or takes the default on a create). +type RedirectFields struct { + Source *string `json:"source,omitempty"` + Target *string `json:"target,omitempty"` + Status *int `json:"status,omitempty"` + Match *string `json:"match,omitempty"` + Query *string `json:"query,omitempty"` + IgnoreCase *bool `json:"ignore_case,omitempty"` + TrailingSlash *bool `json:"trailing_slash,omitempty"` + Note *string `json:"note,omitempty"` +} + +// Apply writes the fields that are set onto r. +func (f RedirectFields) Apply(r *models.Redirect) { + if f.Source != nil { + r.Source = strings.TrimSpace(*f.Source) + } + if f.Target != nil { + r.Target = strings.TrimSpace(*f.Target) + } + if f.Status != nil { + r.Status = models.RedirectStatus(*f.Status) + } + if f.Match != nil { + r.Match = strings.TrimSpace(*f.Match) + } + if f.Query != nil { + r.Query = strings.TrimSpace(*f.Query) + } + if f.IgnoreCase != nil { + r.IgnoreCase = *f.IgnoreCase + } + if f.TrailingSlash != nil { + r.TrailingSlash = *f.TrailingSlash + } + if f.Note != nil { + r.Note = strings.TrimSpace(*f.Note) + } +} + +// RuleFields are the fields of a URL rule write; a nil field is left as +// it is. +type RuleFields struct { + URL *string `json:"url,omitempty"` + Match *string `json:"match,omitempty"` + Title *string `json:"title,omitempty"` + Description *string `json:"description,omitempty"` + Canonical *string `json:"canonical,omitempty"` + Robots *string `json:"robots,omitempty"` + OGTitle *string `json:"og_title,omitempty"` + OGDescription *string `json:"og_description,omitempty"` + OGImage *string `json:"og_image,omitempty"` + Note *string `json:"note,omitempty"` +} + +// Apply writes the fields that are set onto r. +func (f RuleFields) Apply(r *models.Rule) { + for column, v := range map[string]*string{ + "url": f.URL, "match": f.Match, "title": f.Title, "description": f.Description, + "canonical": f.Canonical, "robots": f.Robots, "og_title": f.OGTitle, + "og_description": f.OGDescription, "og_image": f.OGImage, "note": f.Note, + } { + if v != nil { + SetRuleValue(r, column, strings.TrimSpace(*v)) + } + } +} + +// BatchChange is one change of a batch (D-12): set_seo addresses a page +// by URL (D-11); the redirect and rule operations create, update (by id) +// or delete (by id). +type BatchChange struct { + Op string `json:"op"` + ID uint `json:"id,omitempty"` + URL string `json:"url,omitempty"` + Locale string `json:"locale,omitempty"` + Values map[string]string `json:"values,omitempty"` + ForceRule bool `json:"force_rule,omitempty"` + Redirect *RedirectFields `json:"redirect,omitempty"` + Rule *RuleFields `json:"rule,omitempty"` +} + +// DiffEntry is one changed object of a batch change: the history row the +// change wrote, its before and after values. +type DiffEntry struct { + Kind string `json:"kind"` + Subject string `json:"subject"` + What string `json:"what"` + Before map[string]any `json:"before"` + After map[string]any `json:"after"` +} + +// Conflict is a finding of a batch (D-12): a loop refuses the batch, the +// other kinds are warnings. +type Conflict struct { + Change int `json:"change"` + Kind string `json:"kind"` + URL string `json:"url,omitempty"` + Path []string `json:"path,omitempty"` + Target string `json:"target,omitempty"` + Message string `json:"message"` + Refused bool `json:"refused,omitempty"` +} + +// BatchItem is the outcome of one change: what it wrote (the diff), where +// a URL change went, its conflicts, and the refusal when it was refused. +type BatchItem struct { + Index int `json:"index"` + Op string `json:"op"` + Owner *Owner `json:"owner,omitempty"` + Diff []DiffEntry `json:"diff"` + Conflicts []Conflict `json:"conflicts,omitempty"` + Error string `json:"error,omitempty"` + Message string `json:"message,omitempty"` + Errors map[string][]string `json:"errors,omitempty"` + Path []string `json:"path,omitempty"` +} + +// BatchResult is a batch's answer: the batch id its history rows carry, +// whether it was a dry run and whether it was applied, every change's +// outcome, all conflicts, and how many changes wrote something. +type BatchResult struct { + BatchID string `json:"batch_id"` + DryRun bool `json:"dry_run"` + Applied bool `json:"applied"` + Changes []BatchItem `json:"changes"` + Conflicts []Conflict `json:"conflicts"` + Count int `json:"count"` +} + +// Refused returns the first refused change, nil when none was. +func (r BatchResult) Refused() *BatchItem { + for i := range r.Changes { + if r.Changes[i].Error != "" { + return &r.Changes[i] + } + } + return nil +} + +// BatchRefusedError refuses a batch with a refused change: nothing was +// written (D-12, all-or-nothing). +type BatchRefusedError struct{ Result BatchResult } + +func (e *BatchRefusedError) Error() string { + if item := e.Result.Refused(); item != nil { + return fmt.Sprintf("golem15.seo: batch change %d refused: %s", item.Index, item.Error) + } + return "golem15.seo: batch refused" +} + +// ErrStale refuses an apply whose data changed after its preview. +var ErrStale = errors.New("golem15.seo: the data changed after the preview") + +// errDryRun rolls a dry run back on purpose. +var errDryRun = errors.New("golem15.seo: dry run") + +// BatchOptions are a batch's mode: DryRun rolls everything back after the +// diff and conflicts are collected; Via overrides the writer history +// records (import); BatchID names the batch (a new id when empty). +type BatchOptions struct { + DryRun bool + Via string + BatchID string +} + +// NewBatchID returns a random batch id. +func NewBatchID() string { + var b [16]byte + if _, err := rand.Read(b[:]); err != nil { + panic(err) + } + return hex.EncodeToString(b[:]) +} + +// RunBatch runs changes as one batch (D-12, RESEARCH Pattern 8): every +// change goes through the service inside one transaction, each change's +// diff (the history rows it wrote) and conflicts are collected, and the +// conflicts that depend on the whole batch (a canonical pointing to a +// redirect another change creates) are checked once every change is in +// place. A dry run then rolls back on purpose; a refused change (a loop, +// an invalid value) rolls everything back and answers a +// *BatchRefusedError; otherwise the batch commits with one batch id on +// every history row. +func (s *Service) RunBatch(ctx context.Context, changes []BatchChange, opts BatchOptions) (BatchResult, error) { + return s.runBatch(ctx, opts, func(context.Context, *gorm.DB) ([]BatchChange, error) { return changes, nil }) +} + +// runBatch is RunBatch with the changes built inside the transaction +// (the import maps its rows against the data it applies to). +func (s *Service) runBatch(ctx context.Context, opts BatchOptions, build func(ctx context.Context, tx *gorm.DB) ([]BatchChange, error)) (BatchResult, error) { + gdb, err := s.GormDB() + if err != nil { + return BatchResult{}, err + } + res := BatchResult{BatchID: opts.BatchID, DryRun: opts.DryRun, Changes: []BatchItem{}, Conflicts: []Conflict{}} + if res.BatchID == "" { + res.BatchID = NewBatchID() + } + err = lagoon.Transaction(ctx, gdb, func(ctx context.Context, tx *gorm.DB) error { + ctx = WithTx(WithBatch(ctx, res.BatchID), tx) + if opts.Via != "" { + ctx = WithVia(ctx, opts.Via) + } + changes, err := build(ctx, tx) + if err != nil { + return err + } + if len(changes) > MaxBatchChanges { + return &ProblemsError{Problems: map[string][]models.Problem{"changes": {{Key: "golem15.seo::lang.api.batch_too_large", Params: map[string]string{"max": fmt.Sprint(MaxBatchChanges)}}}}} + } + for i, ch := range changes { + item, err := s.batchChange(ctx, tx, i, ch, res.BatchID) + if err != nil { + return err + } + res.Changes = append(res.Changes, item) + } + for i := range res.Changes { + late, err := s.batchConflicts(ctx, tx, i, changes[i], res.Changes[i]) + if err != nil { + return err + } + res.Changes[i].Conflicts = append(res.Changes[i].Conflicts, late...) + } + for _, item := range res.Changes { + res.Conflicts = append(res.Conflicts, item.Conflicts...) + if len(item.Diff) > 0 { + res.Count++ + } + } + if res.Refused() != nil { + return &BatchRefusedError{Result: res} + } + if opts.DryRun { + return errDryRun + } + return nil + }) + switch { + case errors.Is(err, errDryRun): + return res, nil + case err != nil: + var refused *BatchRefusedError + if errors.As(err, &refused) { + refused.Result.Applied = false + } + return res, err + } + res.Applied = true + return res, nil +} + +// batchChange runs one change under a savepoint, so a refused change +// leaves the transaction usable for the next one, and reads the history +// rows it wrote as its diff. +func (s *Service) batchChange(ctx context.Context, tx *gorm.DB, index int, ch BatchChange, batchID string) (BatchItem, error) { + item := BatchItem{Index: index, Op: ch.Op, Diff: []DiffEntry{}} + var last uint + if err := tx.Raw(`SELECT COALESCE(MAX(id), 0) FROM golem15_seo_revisions`).Scan(&last).Error; err != nil { + return item, err + } + point := fmt.Sprintf("seo_change_%d", index) + if err := tx.SavePoint(point).Error; err != nil { + return item, err + } + err := s.applyChange(ctx, tx, index, ch, &item) + if err != nil { + if rb := tx.RollbackTo(point).Error; rb != nil { + return item, rb + } + var loop *LoopError + var problems *ProblemsError + switch { + case errors.As(err, &loop): + item.Error, item.Path = ConflictLoop, loop.Check.Path + item.Message = s.LoopMessage(ctx, loop.Check) + item.Conflicts = append(item.Conflicts, Conflict{Change: index, Kind: ConflictLoop, Path: loop.Check.Path, Message: item.Message, Refused: true}) + case errors.As(err, &problems): + item.Error = "validation_failed" + item.Errors = map[string][]string{} + var msgs []string + for field, list := range problems.Problems { + for _, p := range list { + msg := s.PhraseCtx(ctx, p.Key, p.Params) + item.Errors[field] = append(item.Errors[field], msg) + msgs = append(msgs, field+": "+msg) + } + } + item.Message = strings.Join(msgs, "; ") + case errors.Is(err, ErrNoRecord): + item.Error, item.Message = "not_found", s.PhraseCtx(ctx, "golem15.seo::lang.api.change_not_found", nil) + case errors.Is(err, ErrBadURL): + item.Error, item.Message = "bad_url", err.Error() + case errors.Is(err, errUnknownOp): + item.Error, item.Message = "unknown_op", s.PhraseCtx(ctx, "golem15.seo::lang.api.op_unknown", map[string]string{"op": ch.Op}) + default: + return item, err + } + return item, nil + } + if err := tx.Exec("RELEASE SAVEPOINT " + point).Error; err != nil { + return item, err + } + var rows []models.Revision + if err := tx.Where("id > ? AND batch_id = ?", last, batchID).Order("id ASC").Find(&rows).Error; err != nil { + return item, err + } + for _, r := range rows { + item.Diff = append(item.Diff, DiffEntry{Kind: r.SubjectKind, Subject: r.SubjectID, What: r.What, Before: r.Before, After: r.After}) + } + return item, nil +} + +// errUnknownOp refuses a change with an unknown operation. +var errUnknownOp = errors.New("golem15.seo: unknown batch operation") + +// applyChange runs one change through the service. +func (s *Service) applyChange(ctx context.Context, tx *gorm.DB, index int, ch BatchChange, item *BatchItem) error { + switch ch.Op { + case OpSetSEO: + res, err := s.SetURLSEO(ctx, tx, URLSEOWrite{URL: ch.URL, Locale: ch.Locale, Values: ch.Values, ForceRule: ch.ForceRule}) + if err != nil { + return err + } + owner := res.Owner + item.Owner = &owner + return nil + case OpCreateRedirect, OpUpdateRedirect: + r := models.NewRedirect() + r.Origin = RedirectOrigin(ctx) + if ch.Op == OpUpdateRedirect { + var err error + if r, err = LoadRedirect(tx, ch.ID); err != nil { + return err + } + } + if ch.Redirect != nil { + ch.Redirect.Apply(&r) + } + check, err := s.SaveRedirect(ctx, tx, &r) + if err != nil { + return err + } + if check.Chain { + notice := ChainNotice(check)[0] + item.Conflicts = append(item.Conflicts, Conflict{Change: index, Kind: ConflictChain, URL: r.Source, Path: check.Path, Target: check.Final, Message: s.PhraseCtx(ctx, notice.Message, notice.Params)}) + } + return nil + case OpDeleteRedirect: + r, err := LoadRedirect(tx, ch.ID) + if err != nil { + return err + } + return s.DeleteRedirect(ctx, tx, r) + case OpCreateRule, OpUpdateRule: + r := models.Rule{Match: models.MatchExact} + if ch.Op == OpUpdateRule { + var err error + if r, err = LoadRule(tx, ch.ID); err != nil { + return err + } + } + if ch.Rule != nil { + ch.Rule.Apply(&r) + } + return s.SaveRule(ctx, tx, &r) + case OpDeleteRule: + r, err := LoadRule(tx, ch.ID) + if err != nil { + return err + } + return s.DeleteRule(ctx, tx, r) + } + return errUnknownOp +} + +// batchConflicts are the warnings that depend on the whole batch, checked +// once every change is in place: noindex on the address of a published +// record (the records the sitemaps list; Phase 33's sitemap filter +// refines it) and a canonical pointing to a redirected or noindex +// address. +func (s *Service) batchConflicts(ctx context.Context, tx *gorm.DB, index int, ch BatchChange, item BatchItem) ([]Conflict, error) { + if item.Error != "" { + return nil, nil + } + var out []Conflict + var robots, canonical, url string + switch ch.Op { + case OpSetSEO: + robots, canonical, url = ch.Values["robots"], ch.Values["canonical"], ch.URL + if item.Owner != nil { + url = item.Owner.URL + } + case OpCreateRule, OpUpdateRule: + if ch.Rule == nil { + return nil, nil + } + if ch.Rule.Robots != nil { + robots = *ch.Rule.Robots + } + if ch.Rule.Canonical != nil { + canonical = *ch.Rule.Canonical + } + if ch.Rule.URL != nil { + url = *ch.Rule.URL + } + default: + return nil, nil + } + if strings.Contains(strings.ToLower(robots), "noindex") && url != "" && !strings.Contains(url, "*") { + owner, _, err := s.FindOwner(ctx, tx, url, false) + if err != nil && !errors.Is(err, ErrBadURL) { + return nil, err + } + if err == nil && owner.WrittenTo == WrittenToRecord && owner.Published { + out = append(out, Conflict{Change: index, Kind: ConflictNoindexPublished, URL: owner.URL, + Message: s.PhraseCtx(ctx, "golem15.seo::lang.api.conflict_noindex_published", map[string]string{"url": owner.URL})}) + } + } + if strings.TrimSpace(canonical) != "" { + kind, target, err := s.canonicalProblem(ctx, canonical) + if err != nil { + return nil, err + } + switch kind { + case IssueCanonicalRedirected: + out = append(out, Conflict{Change: index, Kind: ConflictCanonicalRedirected, URL: url, Target: target, + Message: s.PhraseCtx(ctx, "golem15.seo::lang.transfer.row_canonical", map[string]string{"target": target})}) + case IssueCanonicalNoindex: + out = append(out, Conflict{Change: index, Kind: ConflictCanonicalNoindex, URL: url, + Message: s.PhraseCtx(ctx, "golem15.seo::lang.status.canonical_noindex", map[string]string{"url": canonical})}) + } + } + return out, nil +} diff --git a/classes/csv.go b/classes/csv.go new file mode 100644 index 0000000..14e7def --- /dev/null +++ b/classes/csv.go @@ -0,0 +1,145 @@ +package classes + +import ( + "bytes" + "encoding/csv" + "errors" + "io" + "strings" +) + +// BOM is the UTF-8 byte order mark export files start with, so Excel +// opens Polish characters correctly; import strips it. +const BOM = "\ufeff" + +// ErrUnreadable refuses a file that is not CSV with a header row +// (transfer.unreadable). +var ErrUnreadable = errors.New("golem15.seo: the file could not be read as CSV") + +// CSVDelimiter returns the delimiter rune of a setting: ";" or ",", comma +// by default. +func CSVDelimiter(d string) rune { + if strings.TrimSpace(d) == ";" { + return ';' + } + return ',' +} + +// formulaStart reports whether a cell would be read as a formula by a +// spreadsheet: it starts with = + - @, a tab or a carriage return. +func formulaStart(s string) bool { + if s == "" { + return false + } + switch s[0] { + case '=', '+', '-', '@', '\t', '\r': + return true + } + return false +} + +// escapeCell prefixes a formula cell with an apostrophe (CSV formula +// injection, T-32-24). +func escapeCell(s string) string { + if formulaStart(s) { + return "'" + s + } + return s +} + +// unescapeCell drops the apostrophe escapeCell added, so an exported file +// imports unchanged. +func unescapeCell(s string) string { + if strings.HasPrefix(s, "'") && formulaStart(s[1:]) { + return s[1:] + } + return s +} + +// WriteCSV writes header and rows as UTF-8 with a BOM and the delimiter, +// every formula cell escaped. +func WriteCSV(header []string, rows [][]string, delimiter string) (string, error) { + var b bytes.Buffer + b.WriteString(BOM) + w := csv.NewWriter(&b) + w.Comma = CSVDelimiter(delimiter) + if err := w.Write(header); err != nil { + return "", err + } + for _, row := range rows { + cells := make([]string, len(row)) + for i, c := range row { + cells[i] = escapeCell(c) + } + if err := w.Write(cells); err != nil { + return "", err + } + } + w.Flush() + return b.String(), w.Error() +} + +// CSVRow is one data row of a file: its line number and its cells by +// column name. +type CSVRow struct { + Line int + Fields map[string]string +} + +// ReadCSV reads a file with a header row: the BOM is stripped, the +// delimiter is the one given (detected from the header when empty), the +// column names are trimmed and lowercased and the apostrophe of escaped +// formula cells is dropped. Rows whose cells are all empty are skipped. +// A file that is not CSV, has no header or was read with the wrong +// delimiter answers ErrUnreadable. +func ReadCSV(data, delimiter string) ([]string, []CSVRow, error) { + data = strings.TrimPrefix(data, BOM) + if strings.TrimSpace(data) == "" { + return nil, nil, ErrUnreadable + } + first, _, _ := strings.Cut(data, "\n") + comma := CSVDelimiter(delimiter) + if strings.TrimSpace(delimiter) == "" && strings.Count(first, ";") > strings.Count(first, ",") { + comma = ';' + } + r := csv.NewReader(strings.NewReader(data)) + r.Comma = comma + r.FieldsPerRecord = -1 + header, err := r.Read() + if err != nil { + return nil, nil, ErrUnreadable + } + for i, h := range header { + header[i] = strings.ToLower(strings.TrimSpace(h)) + } + if len(header) == 1 && strings.ContainsAny(header[0], ",;") { + return nil, nil, ErrUnreadable + } + var rows []CSVRow + for { + rec, err := r.Read() + if errors.Is(err, io.EOF) { + break + } + if err != nil { + return nil, nil, ErrUnreadable + } + line, _ := r.FieldPos(0) + row := CSVRow{Line: line, Fields: map[string]string{}} + empty := true + for i, cell := range rec { + if i >= len(header) || header[i] == "" { + continue + } + cell = unescapeCell(strings.TrimSpace(cell)) + if cell != "" { + empty = false + } + row.Fields[header[i]] = cell + } + if !empty { + rows = append(rows, row) + } + } + return header, rows, nil +} diff --git a/classes/history.go b/classes/history.go index 2a8a8c4..932ccce 100644 --- a/classes/history.go +++ b/classes/history.go @@ -18,6 +18,21 @@ import ( type viaKey struct{} +type batchKey struct{} + +// WithBatch marks ctx as part of batch id: every history row written +// through it carries the batch id, so the batch can be listed and undone +// as a whole (D-12, D-13). +func WithBatch(ctx context.Context, id string) context.Context { + return context.WithValue(ctx, batchKey{}, id) +} + +// BatchOf is the batch id of ctx, "" outside a batch. +func BatchOf(ctx context.Context) string { + id, _ := ctx.Value(batchKey{}).(string) + return id +} + // WithVia marks ctx as a write through via (import, auto, undo), which // history records instead of admin or mcp. func WithVia(ctx context.Context, via string) context.Context { @@ -182,10 +197,14 @@ func (s *Service) author(ctx context.Context, db *gorm.DB, via string) (*uint, s } // Record appends one history row per change whose values differ, inside -// db (the caller's transaction): author and via from ctx, What as given, -// before and after of the changed fields only. +// db (the caller's transaction): author, via and batch from ctx, What as +// given, before and after of the changed fields only. func (s *Service) Record(ctx context.Context, db *gorm.DB, changes ...Change) error { via := ViaOf(ctx) + var batch *string + if id := BatchOf(ctx); id != "" { + batch = &id + } var authorID *uint var authorLabel string authored := false @@ -199,6 +218,7 @@ func (s *Service) Record(ctx context.Context, db *gorm.DB, changes ...Change) er authored = true } row := models.Revision{ + BatchID: batch, AuthorID: authorID, AuthorLabel: authorLabel, Via: via, diff --git a/classes/transfer.go b/classes/transfer.go new file mode 100644 index 0000000..eeafdf2 --- /dev/null +++ b/classes/transfer.go @@ -0,0 +1,802 @@ +package classes + +import ( + "bytes" + "compress/gzip" + "context" + "crypto/hmac" + "crypto/rand" + "crypto/sha256" + "encoding/base64" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "sort" + "strconv" + "strings" + "sync" + "time" + + "git.golem15.com/golem15/sm-seo-plugin/models" + "git.golem15.com/golem15/summercms/modules/lagoon" + "gorm.io/gorm" +) + +// The datasets of import and export (P-27, UI-SPEC C13). +const ( + DatasetRedirects = "redirects" + DatasetRules = "rules" + DatasetRecords = "records" +) + +// The size caps of import and export (RESEARCH Pitfall 14, T-32-29). +const ( + // AdminImportLimit is the largest file the import screen previews + // (UI-SPEC Q-8 assumption: 60 KB, under the 64 KiB action payload). + AdminImportLimit = 60 << 10 + // MCPImportLimit is the largest file one MCP import takes; larger files + // are imported in chunks. + MCPImportLimit = 900 << 10 + // ExportActionLimit is the largest export the screen downloads in one + // action answer (the action data is capped at 256 KiB). + ExportActionLimit = 250 << 10 + // ExportPageRows is the number of rows of one export page through MCP. + ExportPageRows = 1000 + // PreviewRowCap is the number of preview rows the screen shows. + PreviewRowCap = 200 +) + +// DatasetColumns are each dataset's columns in file order; the first one +// is required. +var DatasetColumns = map[string][]string{ + DatasetRedirects: {"source", "match", "target", "status", "query", "ignore_case", "trailing_slash", "note"}, + DatasetRules: {"url", "match", "title", "description", "canonical", "robots", "og_title", "og_description", "og_image", "note"}, + DatasetRecords: {"url", "locale", "title", "description", "keyphrase", "canonical", "robots", "og_title", "og_description", "og_image"}, +} + +// TransferError refuses an import or export as a whole: Code is +// unreadable, missing_column, dataset_unknown, too_large or stale, Key the +// phrase key of its message. +type TransferError struct { + Code string + Key string + Params map[string]string +} + +func (e *TransferError) Error() string { return "golem15.seo: transfer " + e.Code } + +// ImportRefusedError refuses an apply while a row is refused; nothing was +// written. +type ImportRefusedError struct{ Result ImportResult } + +func (e *ImportRefusedError) Error() string { return "golem15.seo: import refused" } + +// Export is one export answer: the file name, the CSV text (UTF-8 with a +// BOM) and the page of the rows it holds. +type Export struct { + Dataset string `json:"dataset"` + Filename string `json:"filename"` + CSV string `json:"csv"` + Page int `json:"page"` + Pages int `json:"pages"` + Total int `json:"total"` +} + +// boolCell writes a bool cell. +func boolCell(b bool) string { + if b { + return "1" + } + return "0" +} + +// parseBool reads a bool cell: 1, true, yes, on, tak and their opposites. +func parseBool(s string) (bool, bool) { + switch strings.ToLower(strings.TrimSpace(s)) { + case "1", "true", "yes", "on", "tak", "y": + return true, true + case "0", "false", "no", "off", "nie", "n": + return false, true + } + return false, false +} + +// datasetRows reads every row of a dataset in file order. +func datasetRows(db *gorm.DB, dataset string) ([][]string, error) { + var out [][]string + switch dataset { + case DatasetRedirects: + var rows []models.Redirect + if err := db.Order("id ASC").Find(&rows).Error; err != nil { + return nil, err + } + for _, r := range rows { + out = append(out, []string{r.Source, r.Match, r.Target, strconv.Itoa(int(r.Status)), r.Query, boolCell(r.IgnoreCase), boolCell(r.TrailingSlash), r.Note}) + } + case DatasetRules: + var rows []models.Rule + if err := db.Order("url ASC, id ASC").Find(&rows).Error; err != nil { + return nil, err + } + for _, r := range rows { + out = append(out, []string{r.URL, r.Match, r.Title, r.Description, r.Canonical, r.Robots, r.OGTitle, r.OGDescription, r.OGImage, r.Note}) + } + case DatasetRecords: + var rows []models.Entry + if err := db.Where("url <> ''").Order("url ASC, locale ASC").Find(&rows).Error; err != nil { + return nil, err + } + for _, e := range rows { + out = append(out, []string{e.URL, e.Locale, e.Title, e.Description, e.Keyphrase, e.Canonical, e.Robots, e.OGTitle, e.OGDescription, e.OGImage}) + } + default: + return nil, &TransferError{Code: "dataset_unknown", Key: "golem15.seo::lang.transfer.dataset_unknown"} + } + return out, nil +} + +// Export writes one page of a dataset as CSV with the delimiter (D-13): +// UTF-8 with a BOM and formula cells prefixed with an apostrophe. perPage +// 0 exports every row. +func (s *Service) Export(ctx context.Context, dataset, delimiter string, page, perPage int) (Export, error) { + db, err := s.DB(ctx) + if err != nil { + return Export{}, err + } + rows, err := datasetRows(db, dataset) + if err != nil { + return Export{}, err + } + out := Export{Dataset: dataset, Total: len(rows), Page: 1, Pages: 1, + Filename: "seo-" + dataset + "-" + time.Now().UTC().Format("2006-01-02") + ".csv"} + if perPage > 0 { + out.Pages = (len(rows) + perPage - 1) / perPage + if out.Pages == 0 { + out.Pages = 1 + } + if page < 1 { + page = 1 + } + out.Page = page + start := (page - 1) * perPage + if start > len(rows) { + start = len(rows) + } + end := min(start+perPage, len(rows)) + rows = rows[start:end] + } + out.CSV, err = WriteCSV(DatasetColumns[dataset], rows, delimiter) + return out, err +} + +// ImportRequest is one import: the dataset, the file and its delimiter +// (detected from the header when empty), whether to apply, the writer +// history records, and for an apply from a preview token the fingerprint +// the preview saw. +type ImportRequest struct { + Dataset string + CSV string + Delimiter string + DryRun bool + Fingerprint string +} + +// PreviewRow is one row of the import preview (UI-SPEC C13): the file +// line, the change (new, changed, unchanged or refused), the address, a +// changed field with its before and after value, and the note; Warning +// marks a row applied with a warning. +type PreviewRow struct { + Line int `json:"line"` + Change string `json:"change"` + Address string `json:"address"` + Field string `json:"field"` + Before string `json:"before"` + After string `json:"after"` + Note string `json:"note"` + Warning bool `json:"warning,omitempty"` +} + +// ImportSummary counts the rows of an import by change. +type ImportSummary struct { + New int `json:"new"` + Changed int `json:"changed"` + Unchanged int `json:"unchanged"` + Refused int `json:"refused"` + Warning int `json:"warning"` +} + +// ImportResult is an import's answer: the summary, the preview rows +// (refused first), the number of file rows, the number of changes an +// apply writes, the fingerprint of the data the preview read, and after +// an apply the batch it wrote. +type ImportResult struct { + Dataset string `json:"dataset"` + Summary ImportSummary `json:"summary"` + Rows []PreviewRow `json:"rows"` + Total int `json:"total"` + Count int `json:"count"` + Applied bool `json:"applied"` + BatchID string `json:"batch_id,omitempty"` + Fingerprint string `json:"-"` +} + +// planned is one file row mapped to a change. +type planned struct { + row CSVRow + address string + status string // new, changed, unchanged, refused + change int // index in the batch, -1 for none + diffs [][3]string + notes []string + state any +} + +// Import maps a file's rows to changes and runs them as one batch with +// via import (D-12, D-13, P-27): a preview is a dry run answering every +// row's change, the notes (loops refused, chains, canonicals that +// redirect, rows that create a URL rule) and the fingerprint of the data +// it read; an apply runs the preview again, refuses with +// *ImportRefusedError while a row is refused, with transfer.stale when +// the fingerprint differs from req.Fingerprint, and applies every change +// or none. +func (s *Service) Import(ctx context.Context, req ImportRequest) (ImportResult, error) { + header, rows, err := ReadCSV(req.CSV, req.Delimiter) + if err != nil { + return ImportResult{}, &TransferError{Code: "unreadable", Key: "golem15.seo::lang.transfer.unreadable"} + } + columns, ok := DatasetColumns[req.Dataset] + if !ok { + return ImportResult{}, &TransferError{Code: "dataset_unknown", Key: "golem15.seo::lang.transfer.dataset_unknown"} + } + if !contains(header, columns[0]) { + return ImportResult{}, &TransferError{Code: "missing_column", Key: "golem15.seo::lang.transfer.missing_column", Params: map[string]string{"column": columns[0]}} + } + preview, err := s.importPass(ctx, req.Dataset, header, rows, true, req.Fingerprint) + if err != nil || req.DryRun { + return preview, err + } + if preview.Summary.Refused > 0 { + return preview, &ImportRefusedError{Result: preview} + } + if preview.Count == 0 { + preview.Applied = true + return preview, nil + } + return s.importPass(ctx, req.Dataset, header, rows, false, preview.Fingerprint) +} + +// contains reports whether list holds v. +func contains(list []string, v string) bool { + for _, x := range list { + if x == v { + return true + } + } + return false +} + +// importPass maps and runs the rows once. +func (s *Service) importPass(ctx context.Context, dataset string, header []string, rows []CSVRow, dryRun bool, expect string) (ImportResult, error) { + res := ImportResult{Dataset: dataset, Total: len(rows), Rows: []PreviewRow{}} + var plan []planned + batch, err := s.runBatch(ctx, BatchOptions{DryRun: dryRun, Via: models.ViaImport}, func(ctx context.Context, tx *gorm.DB) ([]BatchChange, error) { + var changes []BatchChange + var err error + plan, changes, err = s.planImport(ctx, tx, dataset, header, rows) + if err != nil { + return nil, err + } + res.Fingerprint = fingerprint(dataset, plan) + if expect != "" && res.Fingerprint != expect { + return nil, ErrStale + } + return changes, nil + }) + var refused *BatchRefusedError + switch { + case errors.Is(err, ErrStale): + return res, &TransferError{Code: "stale", Key: "golem15.seo::lang.transfer.stale"} + case errors.As(err, &refused): + batch = refused.Result + case err != nil: + return res, err + } + s.previewRows(ctx, &res, plan, batch) + if refused != nil && !dryRun { + return res, &ImportRefusedError{Result: res} + } + if !dryRun { + res.Applied, res.BatchID = true, batch.BatchID + } + return res, nil +} + +// fingerprint hashes the data an import read: each row's address and the +// state of the object it changes, so an apply sees whether anything +// changed since its preview (T-32-25). +func fingerprint(dataset string, plan []planned) string { + h := sha256.New() + h.Write([]byte(dataset)) + for _, p := range plan { + raw, _ := json.Marshal([]any{p.row.Line, p.address, p.state}) + h.Write(raw) + } + return hex.EncodeToString(h.Sum(nil)) +} + +// cell returns a row's cell and whether the file has the column. +func cell(row CSVRow, column string) (string, bool) { + v, ok := row.Fields[column] + return v, ok +} + +// planImport maps rows to batch changes against the data in tx. +func (s *Service) planImport(ctx context.Context, tx *gorm.DB, dataset string, header []string, rows []CSVRow) ([]planned, []BatchChange, error) { + var plan []planned + var changes []BatchChange + add := func(p planned, ch *BatchChange) { + p.change = -1 + if ch != nil && p.status != "refused" { + p.change = len(changes) + changes = append(changes, *ch) + } + plan = append(plan, p) + } + for _, row := range rows { + switch dataset { + case DatasetRedirects: + p, ch, err := s.planRedirect(ctx, tx, row) + if err != nil { + return nil, nil, err + } + add(p, ch) + case DatasetRules: + p, ch, err := s.planRule(tx, row) + if err != nil { + return nil, nil, err + } + add(p, ch) + case DatasetRecords: + p, ch, err := s.planRecord(ctx, tx, row) + if err != nil { + return nil, nil, err + } + add(p, ch) + } + } + return plan, changes, nil +} + +// planRedirect maps a redirects row: the redirect with the same source and +// match is updated, else one is created. +func (s *Service) planRedirect(ctx context.Context, tx *gorm.DB, row CSVRow) (planned, *BatchChange, error) { + source, _ := cell(row, "source") + match, _ := cell(row, "match") + if match == "" { + match = models.RedirectExact + } + p := planned{row: row, address: source} + f := RedirectFields{Source: &source, Match: &match} + for _, column := range []string{"target", "query", "note"} { + if v, ok := cell(row, column); ok { + v := v + switch column { + case "target": + f.Target = &v + case "query": + if v != "" { + f.Query = &v + } + case "note": + f.Note = &v + } + } + } + if v, ok := cell(row, "status"); ok && v != "" { + n, err := strconv.Atoi(v) + if err != nil { + p.status = "refused" + p.notes = append(p.notes, s.PhraseCtx(ctx, "golem15.seo::lang.transfer.row_invalid", map[string]string{"field": "status", "message": v})) + return p, nil, nil + } + f.Status = &n + } + for _, column := range []string{"ignore_case", "trailing_slash"} { + v, ok := cell(row, column) + if !ok || v == "" { + continue + } + b, valid := parseBool(v) + if !valid { + p.status = "refused" + p.notes = append(p.notes, s.PhraseCtx(ctx, "golem15.seo::lang.transfer.row_invalid", map[string]string{"field": column, "message": v})) + return p, nil, nil + } + if column == "ignore_case" { + f.IgnoreCase = &b + } else { + f.TrailingSlash = &b + } + } + var rows []models.Redirect + if err := tx.Where("source = ? AND match = ?", strings.TrimSpace(source), strings.TrimSpace(match)).Limit(1).Find(&rows).Error; err != nil { + return p, nil, err + } + if len(rows) == 0 { + p.status = "new" + after := models.NewRedirect() + f.Apply(&after) + p.diffs = append(p.diffs, [3]string{"target", "", redirectSummary(after)}) + return p, &BatchChange{Op: OpCreateRedirect, Redirect: &f}, nil + } + cur := rows[0] + p.state = RedirectValues(cur) + next := cur + f.Apply(&next) + next.Normalize() + before, after := RedirectValues(cur), RedirectValues(next) + for _, column := range DatasetColumns[DatasetRedirects] { + if !sameValue(before[column], after[column]) { + p.diffs = append(p.diffs, [3]string{column, text(jsonNorm(before[column])), text(jsonNorm(after[column]))}) + } + } + if len(p.diffs) == 0 { + p.status = "unchanged" + return p, nil, nil + } + p.status = "changed" + return p, &BatchChange{Op: OpUpdateRedirect, ID: cur.ID, Redirect: &f}, nil +} + +// redirectSummary is a new redirect in the preview: "→ target (301)". +func redirectSummary(r models.Redirect) string { + if r.Status == models.Status410 { + return "410" + } + return fmt.Sprintf("→ %s (%d)", r.Target, int(r.Status)) +} + +// planRule maps a rules row: the rule with the same address and match is +// updated, else one is created. +func (s *Service) planRule(tx *gorm.DB, row CSVRow) (planned, *BatchChange, error) { + url, _ := cell(row, "url") + match, _ := cell(row, "match") + match = models.NormalizeMatch(match) + p := planned{row: row, address: url} + f := RuleFields{URL: &url, Match: &match} + values := map[string]*string{} + for _, column := range DatasetColumns[DatasetRules][2:] { + if v, ok := cell(row, column); ok { + v := strings.TrimSpace(v) + if column == "robots" && v != "" { + if normal, err := NormalizeRobots(v); err == nil { + v = normal + } + } + values[column] = &v + } + } + f.Title, f.Description, f.Canonical, f.Robots = values["title"], values["description"], values["canonical"], values["robots"] + f.OGTitle, f.OGDescription, f.OGImage, f.Note = values["og_title"], values["og_description"], values["og_image"], values["note"] + var rows []models.Rule + if err := tx.Where("url = ? AND match = ?", strings.TrimSpace(url), match).Limit(1).Find(&rows).Error; err != nil { + return p, nil, err + } + if len(rows) == 0 { + p.status = "new" + for _, column := range DatasetColumns[DatasetRules][2:] { + if v := values[column]; v != nil && *v != "" { + p.diffs = append(p.diffs, [3]string{column, "", *v}) + } + } + return p, &BatchChange{Op: OpCreateRule, Rule: &f}, nil + } + cur := rows[0] + p.state = RuleValues(cur) + for column, v := range values { + if old := ruleColumn(cur, column); old != *v { + p.diffs = append(p.diffs, [3]string{column, old, *v}) + } + } + sortDiffs(p.diffs, DatasetColumns[DatasetRules]) + if len(p.diffs) == 0 { + p.status = "unchanged" + return p, nil, nil + } + p.status = "changed" + return p, &BatchChange{Op: OpUpdateRule, ID: cur.ID, Rule: &f}, nil +} + +// ruleColumn reads one column of a rule. +func ruleColumn(r models.Rule, column string) string { + switch column { + case "url": + return r.URL + case "match": + return r.Match + case "note": + return r.Note + } + return r.Value(column) +} + +// sortDiffs orders diffs by the dataset's column order. +func sortDiffs(diffs [][3]string, columns []string) { + pos := map[string]int{} + for i, c := range columns { + pos[c] = i + } + sort.SliceStable(diffs, func(i, j int) bool { return pos[diffs[i][0]] < pos[diffs[j][0]] }) +} + +// planRecord maps a records row by URL (D-11): the record behind the URL +// in the row's locale, else an exact URL rule (transfer.row_rule). +func (s *Service) planRecord(ctx context.Context, tx *gorm.DB, row CSVRow) (planned, *BatchChange, error) { + url, _ := cell(row, "url") + locale, _ := cell(row, "locale") + p := planned{row: row, address: url} + owner, _, err := s.FindOwner(ctx, tx, url, false) + if errors.Is(err, ErrBadURL) { + p.status = "refused" + p.notes = append(p.notes, s.PhraseCtx(ctx, "golem15.seo::lang.transfer.row_invalid", map[string]string{"field": "url", "message": err.Error()})) + return p, nil, nil + } + if err != nil { + return p, nil, err + } + p.address = owner.URL + values := map[string]string{} + for _, column := range DatasetColumns[DatasetRecords][2:] { + if v, ok := cell(row, column); ok { + v = strings.TrimSpace(v) + if column == "robots" && v != "" { + if normal, err := NormalizeRobots(v); err == nil { + v = normal + } + } + values[column] = v + } + } + current := map[string]string{} + exists := false + if owner.WrittenTo == WrittenToRecord { + if locale == "" { + locale = owner.Locale + } + typ, _ := s.Registry().ContentType(owner.Type) + e, found, err := LoadEntry(tx, typ.Content.Model, owner.ID, locale) + if err != nil { + return p, nil, err + } + exists = true + for _, column := range EntryColumns { + if found { + current[column] = EntryValue(e, column) + } else { + current[column] = "" + } + } + p.state = []any{owner.Type, owner.ID, locale, current} + } else { + p.notes = append(p.notes, s.PhraseCtx(ctx, "golem15.seo::lang.transfer.row_rule", nil)) + if owner.RuleID != 0 { + r, err := LoadRule(tx, owner.RuleID) + if err != nil { + return p, nil, err + } + exists = true + for _, column := range EntryColumns { + current[column] = r.Value(column) + } + p.state = RuleValues(r) + } + if values["keyphrase"] == "" { + delete(values, "keyphrase") + } + } + changed := map[string]string{} + for column, v := range values { + if current[column] != v { + changed[column] = v + p.diffs = append(p.diffs, [3]string{column, current[column], v}) + } + } + sortDiffs(p.diffs, DatasetColumns[DatasetRecords]) + switch { + case len(changed) == 0: + p.status = "unchanged" + return p, nil, nil + case exists: + p.status = "changed" + default: + p.status = "new" + } + return p, &BatchChange{Op: OpSetSEO, URL: url, Locale: locale, Values: changed}, nil +} + +// previewRows fills the summary, the preview rows and the count from the +// plan and the batch it ran. +func (s *Service) previewRows(ctx context.Context, res *ImportResult, plan []planned, batch BatchResult) { + var refused, others []PreviewRow + for _, p := range plan { + status, notes := p.status, append([]string{}, p.notes...) + warning := false + if p.change >= 0 && p.change < len(batch.Changes) { + item := batch.Changes[p.change] + if item.Error != "" { + status = "refused" + switch { + case item.Error == ConflictLoop: + notes = append(notes, s.PhraseCtx(ctx, "golem15.seo::lang.transfer.row_loop", map[string]string{"path": strings.Join(item.Path, " → ")})) + case len(item.Errors) > 0: + fields := make([]string, 0, len(item.Errors)) + for f := range item.Errors { + fields = append(fields, f) + } + sort.Strings(fields) + for _, f := range fields { + notes = append(notes, s.PhraseCtx(ctx, "golem15.seo::lang.transfer.row_invalid", map[string]string{"field": f, "message": strings.Join(item.Errors[f], " ")})) + } + default: + notes = append(notes, item.Message) + } + } + for _, c := range item.Conflicts { + switch c.Kind { + case ConflictLoop: + case ConflictChain: + warning = true + notes = append(notes, s.PhraseCtx(ctx, "golem15.seo::lang.transfer.row_chain", map[string]string{"path": strings.Join(c.Path, " → ")})) + default: + warning = true + notes = append(notes, c.Message) + } + } + } + switch status { + case "new": + res.Summary.New++ + case "changed": + res.Summary.Changed++ + case "unchanged": + res.Summary.Unchanged++ + case "refused": + res.Summary.Refused++ + } + if warning && status != "refused" { + res.Summary.Warning++ + } + if status == "new" || status == "changed" { + res.Count++ + } + note := strings.Join(notes, " ") + base := PreviewRow{Line: p.row.Line, Change: status, Address: p.address, Note: note, Warning: warning && status != "refused"} + var rows []PreviewRow + if status == "changed" || status == "new" { + for _, d := range p.diffs { + r := base + r.Field, r.Before, r.After = d[0], d[1], d[2] + rows = append(rows, r) + } + } + if len(rows) == 0 { + rows = []PreviewRow{base} + } + if status == "refused" { + refused = append(refused, rows...) + } else { + others = append(others, rows...) + } + } + res.Rows = append(refused, others...) +} + +// The transfer token (T-32-25): the preview's dataset, delimiter, file and +// fingerprint, gzip-compressed and signed with HMAC-SHA256 under the app +// key, so the apply needs no server-side state and a forged or edited +// token is refused. +type transferToken struct { + Dataset string `json:"d"` + Delimiter string `json:"l"` + CSV string `json:"c"` + Fingerprint string `json:"f"` +} + +var ( + fallbackKeyOnce sync.Once + fallbackKey []byte +) + +// tokenKey is the HMAC key of transfer tokens: derived from the app key, +// or a random key per process when the app has none (tokens then do not +// survive a restart). +func (s *Service) tokenKey() []byte { + if cfg := s.Config(); cfg != nil { + if primary, _, err := lagoon.LoadAppKey(cfg); err == nil && len(primary) > 0 { + mac := hmac.New(sha256.New, primary) + mac.Write([]byte("golem15.seo.transfer")) + return mac.Sum(nil) + } + } + fallbackKeyOnce.Do(func() { + fallbackKey = make([]byte, 32) + if _, err := rand.Read(fallbackKey); err != nil { + panic(err) + } + }) + return fallbackKey +} + +// TransferToken signs a preview for its apply. +func (s *Service) TransferToken(dataset, delimiter, csv, fp string) (string, error) { + raw, err := json.Marshal(transferToken{Dataset: dataset, Delimiter: delimiter, CSV: csv, Fingerprint: fp}) + if err != nil { + return "", err + } + var b bytes.Buffer + zw := gzip.NewWriter(&b) + if _, err := zw.Write(raw); err != nil { + return "", err + } + if err := zw.Close(); err != nil { + return "", err + } + mac := hmac.New(sha256.New, s.tokenKey()) + mac.Write(b.Bytes()) + return base64.RawURLEncoding.EncodeToString(b.Bytes()) + "." + base64.RawURLEncoding.EncodeToString(mac.Sum(nil)), nil +} + +// ErrBadToken refuses a transfer token that is malformed or not signed by +// this application. +var ErrBadToken = errors.New("golem15.seo: the import token is not valid") + +// ReadTransferToken verifies a token and returns the import it applies: +// the preview's file with the fingerprint the apply must find. +func (s *Service) ReadTransferToken(token string) (ImportRequest, error) { + body, sig, ok := strings.Cut(strings.TrimSpace(token), ".") + if !ok { + return ImportRequest{}, ErrBadToken + } + payload, err := base64.RawURLEncoding.DecodeString(body) + if err != nil { + return ImportRequest{}, ErrBadToken + } + got, err := base64.RawURLEncoding.DecodeString(sig) + if err != nil { + return ImportRequest{}, ErrBadToken + } + mac := hmac.New(sha256.New, s.tokenKey()) + mac.Write(payload) + if !hmac.Equal(got, mac.Sum(nil)) { + return ImportRequest{}, ErrBadToken + } + zr, err := gzip.NewReader(bytes.NewReader(payload)) + if err != nil { + return ImportRequest{}, ErrBadToken + } + raw, err := io.ReadAll(io.LimitReader(zr, 4<<20)) + if err != nil { + return ImportRequest{}, ErrBadToken + } + var t transferToken + if err := json.Unmarshal(raw, &t); err != nil { + return ImportRequest{}, ErrBadToken + } + return ImportRequest{Dataset: t.Dataset, Delimiter: t.Delimiter, CSV: t.CSV, Fingerprint: t.Fingerprint}, nil +} + +// ChoiceCtx translates a plural phrase for count in the request locale, +// with :count set. +func (s *Service) ChoiceCtx(ctx context.Context, key string, count int, params map[string]string) string { + tr := s.translator() + if tr == nil { + return key + } + p := map[string]string{"count": strconv.Itoa(count)} + for k, v := range params { + p[k] = v + } + return tr.Choice(ctx, key, count, p) +} diff --git a/classes/undo.go b/classes/undo.go new file mode 100644 index 0000000..fab4641 --- /dev/null +++ b/classes/undo.go @@ -0,0 +1,540 @@ +package classes + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "reflect" + "sort" + "strconv" + "strings" + "time" + + "git.golem15.com/golem15/sm-seo-plugin/models" + "git.golem15.com/golem15/summercms/modules/bouncer" + "git.golem15.com/golem15/summercms/modules/pact" + "gorm.io/gorm" +) + +// The undo refusals (D-13, T-32-27). +const ( + UndoStale = "undo_stale" + UndoLoop = "undo_loop" + UndoGone = "undo_gone" + UndoDone = "already_undone" + UndoUnsupported = "undo_unsupported" +) + +// UndoRefusal refuses an undo: Code is one of the Undo* constants, Key the +// phrase key of its message and Params the message's parameters. Nothing +// was undone. +type UndoRefusal struct { + Code string + Key string + Params map[string]string +} + +func (e *UndoRefusal) Error() string { return "golem15.seo: " + e.Code } + +// LoadRevision returns history entry id; ErrNoRecord when there is none. +func LoadRevision(db *gorm.DB, id uint) (models.Revision, error) { + var rows []models.Revision + if err := db.Where("id = ?", id).Limit(1).Find(&rows).Error; err != nil { + return models.Revision{}, err + } + if len(rows) == 0 { + return models.Revision{}, ErrNoRecord + } + return rows[0], nil +} + +// Undoable reports whether a history entry can be offered an undo: not +// undone yet and of a kind undo restores. +func Undoable(r models.Revision) bool { + if r.UndoneAt != nil { + return false + } + switch r.SubjectKind { + case models.SubjectEntry, models.SubjectRule, models.SubjectRedirect, models.SubjectRobots, models.SubjectSettings, models.SubjectNotFound: + return true + } + return false +} + +// Undo restores the values before history entry id inside tx (D-13): the +// undo is itself recorded (via undo) and the entry is marked undone. It +// is refused with an *UndoRefusal when the entry is undone already, when a +// later change touched the same values (undo_stale, naming who and when), +// when the restore would create a redirect loop (undo_loop) or when the +// record it belongs to is gone (undo_gone). +func (s *Service) Undo(ctx context.Context, tx *gorm.DB, id uint) ([]uint, error) { + rev, err := LoadRevision(tx, id) + if err != nil { + return nil, err + } + if rev.UndoneAt != nil { + return nil, &UndoRefusal{Code: UndoDone, Key: "golem15.seo::lang.history.undo_done"} + } + if err := s.undoOne(WithVia(ctx, models.ViaUndo), tx, rev); err != nil { + return nil, err + } + return []uint{rev.ID}, nil +} + +// UndoBatch undoes every change of batch that is not undone yet, newest +// first, inside tx: all of them or none (D-13). ErrNoRecord when the batch +// has nothing left to undo. +func (s *Service) UndoBatch(ctx context.Context, tx *gorm.DB, batch string) ([]uint, error) { + var revs []models.Revision + if err := tx.Where("batch_id = ? AND undone_at IS NULL", batch).Order("id DESC").Find(&revs).Error; err != nil { + return nil, err + } + if len(revs) == 0 { + return nil, ErrNoRecord + } + ctx = WithVia(ctx, models.ViaUndo) + ids := make([]uint, 0, len(revs)) + for _, rev := range revs { + if err := s.undoOne(ctx, tx, rev); err != nil { + return nil, err + } + ids = append(ids, rev.ID) + } + return ids, nil +} + +// BatchSize counts the changes of batch that are not undone yet. +func BatchSize(db *gorm.DB, batch string) (int64, error) { + var n int64 + err := db.Model(&models.Revision{}).Where("batch_id = ? AND undone_at IS NULL", batch).Count(&n).Error + return n, err +} + +// jsonNorm brings a value to its JSON form, so a stored history value +// and a current one compare equal when they encode the same. +func jsonNorm(v any) any { + raw, err := json.Marshal(plainValue(v)) + if err != nil { + return fmt.Sprint(v) + } + var out any + if json.Unmarshal(raw, &out) != nil { + return string(raw) + } + if s, ok := out.(string); ok { + return strings.TrimSpace(s) + } + return out +} + +// sameValue reports whether two values encode the same; nil and "" are +// the same empty value. +func sameValue(a, b any) bool { + na, nb := jsonNorm(a), jsonNorm(b) + if na == nil { + na = "" + } + if nb == nil { + nb = "" + } + return reflect.DeepEqual(na, nb) +} + +// subjectState is the current values of a history subject; exists is +// false when the object is gone. +type subjectState struct { + values map[string]any + exists bool + // recordGone is true for an entry whose record no longer exists. + recordGone bool + typ pact.MenuItemType + model string + recordID string + locale string +} + +// currentState reads the current values of rev's subject. +func (s *Service) currentState(ctx context.Context, tx *gorm.DB, rev models.Revision) (subjectState, error) { + var st subjectState + switch rev.SubjectKind { + case models.SubjectEntry: + parts := strings.SplitN(rev.SubjectID, "|", 3) + if len(parts) != 3 { + return st, &UndoRefusal{Code: UndoUnsupported, Key: "golem15.seo::lang.history.undo_unsupported"} + } + st.model, st.recordID, st.locale = parts[0], parts[1], parts[2] + reg := s.Registry() + typ, ok := pact.MenuItemType{}, false + if reg != nil { + typ, ok = reg.ContentTypeForModel(st.model) + } + if !ok || typ.Content == nil { + st.recordGone = true + return st, nil + } + st.typ = typ + rec, err := reg.Describe(ctx, typ.Key, st.recordID, s.UnprefixedLocale()) + if err != nil { + return st, err + } + if rec == nil { + st.recordGone = true + return st, nil + } + e, found, err := LoadEntry(tx, st.model, st.recordID, st.locale) + if err != nil { + return st, err + } + st.exists = true + st.values = map[string]any{} + for _, c := range EntryColumns { + v := "" + if found { + v = EntryValue(e, c) + } + st.values[c] = v + } + case models.SubjectRule: + id, _ := strconv.ParseUint(rev.SubjectID, 10, 64) + r, err := LoadRule(tx, uint(id)) + if errors.Is(err, ErrNoRecord) { + return st, nil + } + if err != nil { + return st, err + } + st.exists, st.values = true, RuleValues(r) + case models.SubjectRedirect: + id, _ := strconv.ParseUint(rev.SubjectID, 10, 64) + r, err := LoadRedirect(tx, uint(id)) + if errors.Is(err, ErrNoRecord) { + return st, nil + } + if err != nil { + return st, err + } + st.exists, st.values = true, RedirectValues(r) + case models.SubjectRobots: + r, err := LoadRobots(tx) + if err != nil { + return st, err + } + st.exists, st.values = true, RobotsValues(r) + case models.SubjectSettings: + if key, ok := strings.CutPrefix(rev.SubjectID, "type:"); ok { + ts, err := LoadTypeSetting(tx, key) + if err != nil { + return st, err + } + st.exists, st.values = true, TypeSettingValues(ts) + } else { + row, err := LoadSettings(tx) + if err != nil { + return st, err + } + st.exists, st.values = true, SettingsValues(row) + } + case models.SubjectNotFound: + id, _ := strconv.ParseUint(rev.SubjectID, 10, 64) + n, err := LoadNotFound(tx, uint(id)) + if errors.Is(err, ErrNoRecord) { + return st, nil + } + if err != nil { + return st, err + } + st.exists, st.values = true, NotFoundValues(n) + default: + return st, &UndoRefusal{Code: UndoUnsupported, Key: "golem15.seo::lang.history.undo_unsupported"} + } + return st, nil +} + +// staleRefusal names the newest later change of field on rev's subject. +func (s *Service) staleRefusal(ctx context.Context, tx *gorm.DB, rev models.Revision, field string) error { + author, when := "—", "" + var later []models.Revision + if err := tx.Where("subject_kind = ? AND subject_id = ? AND id > ?", rev.SubjectKind, rev.SubjectID, rev.ID).Order("id DESC").Find(&later).Error; err != nil { + return err + } + for _, l := range later { + if _, ok := l.After[field]; ok || len(l.After) == 0 { + if l.AuthorLabel != "" { + author = l.AuthorLabel + } + when = l.CreatedAt.UTC().Format("2006-01-02 15:04") + break + } + } + return &UndoRefusal{Code: UndoStale, Key: "golem15.seo::lang.history.undo_stale", Params: map[string]string{ + "field": s.fieldLabel(ctx, field), "author": author, "time": when, + }} +} + +// fieldLabel is the label of a history field in the request locale. +func (s *Service) fieldLabel(ctx context.Context, key string) string { + base, locale, _ := strings.Cut(key, ".") + for _, p := range models.RuleProperties { + if p.Column == base { + label := s.PhraseCtx(ctx, p.Label, nil) + if locale != "" { + label += " (" + locale + ")" + } + return label + } + } + return key +} + +// undoOne restores the values before rev and marks rev undone. +func (s *Service) undoOne(ctx context.Context, tx *gorm.DB, rev models.Revision) error { + st, err := s.currentState(ctx, tx, rev) + if err != nil { + return err + } + if rev.SubjectKind == models.SubjectEntry && st.recordGone { + label := s.PhraseCtx(ctx, "golem15.seo::lang.history.record", nil) + if st.typ.Label != "" { + label = s.PhraseCtx(ctx, st.typ.Label, nil) + } + return &UndoRefusal{Code: UndoGone, Key: "golem15.seo::lang.history.undo_gone", Params: map[string]string{"type": label}} + } + created := len(rev.Before) == 0 && len(rev.After) > 0 + deleted := len(rev.After) == 0 && len(rev.Before) > 0 + if deleted { + if st.exists && rev.SubjectKind != models.SubjectEntry { + return s.staleRefusal(ctx, tx, rev, firstKey(rev.Before)) + } + } else { + if !st.exists { + return &UndoRefusal{Code: UndoGone, Key: "golem15.seo::lang.history.undo_gone", Params: map[string]string{"type": s.PhraseCtx(ctx, kindLabels[rev.SubjectKind], nil)}} + } + keys := make([]string, 0, len(rev.After)) + for k := range rev.After { + keys = append(keys, k) + } + sort.Strings(keys) + for _, k := range keys { + cur, ok := st.values[k] + if !ok { + return &UndoRefusal{Code: UndoUnsupported, Key: "golem15.seo::lang.history.undo_unsupported"} + } + if !sameValue(cur, rev.After[k]) { + return s.staleRefusal(ctx, tx, rev, k) + } + } + } + if err := s.restore(ctx, tx, rev, st, created, deleted); err != nil { + var loop *LoopError + if errors.As(err, &loop) { + return &UndoRefusal{Code: UndoLoop, Key: "golem15.seo::lang.history.undo_loop", Params: map[string]string{"path": loop.Check.PathText()}} + } + return err + } + now := time.Now().UTC() + updates := map[string]any{"undone_at": now} + if p, ok := bouncer.User(ctx); ok && p.ID != 0 { + updates["undone_by_id"] = p.ID + } else if p, ok := mcpPrincipal(ctx); ok && p.ID != 0 { + updates["undone_by_id"] = p.ID + } + return tx.Model(&models.Revision{}).Where("id = ?", rev.ID).Updates(updates).Error +} + +// kindLabels are the phrase keys of the history subjects in the undo_gone +// refusal. +var kindLabels = map[string]string{ + models.SubjectEntry: "golem15.seo::lang.history.record", + models.SubjectRule: "golem15.seo::lang.history.kind_rule", + models.SubjectRedirect: "golem15.seo::lang.history.kind_redirect", + models.SubjectRobots: "golem15.seo::lang.history.kind_robots", + models.SubjectSettings: "golem15.seo::lang.history.kind_settings", + models.SubjectNotFound: "golem15.seo::lang.history.kind_notfound", +} + +// firstKey is the first key of m in order, "" for none. +func firstKey(m map[string]any) string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + if len(keys) == 0 { + return "" + } + return keys[0] +} + +// text reads a history value as text. +func text(v any) string { + switch t := v.(type) { + case nil: + return "" + case string: + return t + case float64: + return strconv.FormatFloat(t, 'f', -1, 64) + case bool: + return strconv.FormatBool(t) + } + return fmt.Sprint(v) +} + +// restore writes the values before rev through the service, so the undo +// is checked (loops) and recorded like any write. +func (s *Service) restore(ctx context.Context, tx *gorm.DB, rev models.Revision, st subjectState, created, deleted bool) error { + switch rev.SubjectKind { + case models.SubjectEntry: + values := RecordValues{} + for k := range rev.After { + values[k] = map[string]string{st.locale: text(rev.Before[k])} + } + if deleted { + for k, v := range rev.Before { + values[k] = map[string]string{st.locale: text(v)} + } + } + _, _, err := s.SaveRecord(ctx, tx, RecordWrite{Type: st.typ, ID: st.recordID, Values: values}) + return err + case models.SubjectRule: + id, _ := strconv.ParseUint(rev.SubjectID, 10, 64) + switch { + case created: + r, err := LoadRule(tx, uint(id)) + if err != nil { + return err + } + return s.DeleteRule(ctx, tx, r) + case deleted: + r := models.Rule{} + for k, v := range rev.Before { + SetRuleValue(&r, k, text(v)) + } + return s.SaveRule(ctx, tx, &r) + } + r, err := LoadRule(tx, uint(id)) + if err != nil { + return err + } + for k := range rev.After { + SetRuleValue(&r, k, text(rev.Before[k])) + } + return s.SaveRule(ctx, tx, &r) + case models.SubjectRedirect: + id, _ := strconv.ParseUint(rev.SubjectID, 10, 64) + switch { + case created: + r, err := LoadRedirect(tx, uint(id)) + if err != nil { + return err + } + return s.DeleteRedirect(ctx, tx, r) + case deleted: + r := models.NewRedirect() + for k, v := range rev.Before { + setRedirectValue(&r, k, v) + } + _, err := s.SaveRedirect(ctx, tx, &r) + return err + } + r, err := LoadRedirect(tx, uint(id)) + if err != nil { + return err + } + for k := range rev.After { + setRedirectValue(&r, k, rev.Before[k]) + } + _, err = s.SaveRedirect(ctx, tx, &r) + return err + case models.SubjectRobots: + var w RobotsWrite + if v, ok := rev.Before["content"]; ok { + content := text(v) + w.Content = &content + } + if bots, ok := rev.Before["bots"].(map[string]any); ok { + w.Bots = map[string]bool{} + for token, on := range bots { + b, _ := on.(bool) + w.Bots[token] = b + } + } + _, _, err := s.SaveRobots(ctx, tx, w) + return err + case models.SubjectSettings: + if key, ok := strings.CutPrefix(rev.SubjectID, "type:"); ok { + w := TypeSettingWrite{} + for k := range rev.After { + v := rev.Before[k] + switch k { + case "inject": + b, _ := v.(bool) + w.Inject = &b + case "title_template": + t := text(v) + w.TitleTemplate = &t + case "description_template": + t := text(v) + w.DescriptionTemplate = &t + case "robots": + t := text(v) + w.Robots = &t + } + } + _, err := s.SaveSettings(ctx, tx, SettingsWrite{Types: map[string]TypeSettingWrite{key: w}}) + return err + } + values := map[string]any{} + for k := range rev.After { + values[k] = rev.Before[k] + } + _, err := s.SaveSettings(ctx, tx, SettingsWrite{Values: values}) + return err + case models.SubjectNotFound: + if !deleted { + return &UndoRefusal{Code: UndoUnsupported, Key: "golem15.seo::lang.history.undo_unsupported"} + } + now := time.Now().UTC() + row := models.NotFound{URL: text(rev.Before["url"]), Referer: text(rev.Before["referer"]), FirstSeen: &now, LastHit: &now} + if h, ok := rev.Before["hits"].(float64); ok { + row.Hits = int64(h) + } + if err := tx.Create(&row).Error; err != nil { + return err + } + return s.Record(ctx, tx, Change{ + Kind: models.SubjectNotFound, + SubjectID: strconv.FormatUint(uint64(row.ID), 10), + What: s.Phrase("golem15.seo::lang.history.what_notfound", map[string]string{"url": row.URL}), + After: NotFoundValues(row), + }) + } + return &UndoRefusal{Code: UndoUnsupported, Key: "golem15.seo::lang.history.undo_unsupported"} +} + +// setRedirectValue writes one history value onto a redirect. +func setRedirectValue(r *models.Redirect, key string, v any) { + switch key { + case "source": + r.Source = text(v) + case "match": + r.Match = text(v) + case "target": + r.Target = text(v) + case "status": + if n, err := strconv.Atoi(text(v)); err == nil { + r.Status = models.RedirectStatus(n) + } + case "query": + r.Query = text(v) + case "ignore_case": + r.IgnoreCase, _ = v.(bool) + case "trailing_slash": + r.TrailingSlash, _ = v.(bool) + case "origin": + r.Origin = text(v) + case "note": + r.Note = text(v) + } +} diff --git a/controllers/admin_registry.go b/controllers/admin_registry.go index 27f654c..718c043 100644 --- a/controllers/admin_registry.go +++ b/controllers/admin_registry.go @@ -34,6 +34,7 @@ func AdminControllers(app func() *backpack.App) []pact.AdminController { rulesAdminController{app: app}, robotsAdminController{app: app}, historyAdminController{app: app}, + transferAdminController{app: app}, } } diff --git a/controllers/api/batch.go b/controllers/api/batch.go new file mode 100644 index 0000000..e67d96a --- /dev/null +++ b/controllers/api/batch.go @@ -0,0 +1,61 @@ +package api + +import ( + "context" + "errors" + "net/http" + + "git.golem15.com/golem15/sm-seo-plugin/classes" +) + +// ApplyBatchInput is the body of POST /_seo/api/v1/batch. +type ApplyBatchInput struct { + DryRun bool `json:"dry_run,omitempty" jsonschema:"answer the diff and the conflicts without writing anything"` + Changes []classes.BatchChange `json:"changes" jsonschema:"the changes, applied in order: op set_seo (url, locale, values, force_rule), create_redirect and update_redirect (id, redirect), delete_redirect (id), create_rule and update_rule (id, rule), delete_rule (id)"` +} + +// ApplyBatch serves POST /_seo/api/v1/batch (D-12, P-26): the changes run +// in one transaction. With dry_run the answer is every change's diff and +// the conflicts (redirect loops and chains, noindex on a published +// record's address, a canonical pointing to a redirected or noindex +// address) and nothing is written. Without it the batch is applied as a +// whole under one batch id, which history and undo_batch use. A refused +// change refuses the whole batch with 422: redirect_loop with the change +// and the loop's path, or batch_refused with the change's errors. +func ApplyBatch(service func() *classes.Service) http.HandlerFunc { + return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) { + var in ApplyBatchInput + if err := decode(r, 4*maxBody, &in); err != nil { + return 0, nil, err + } + res, err := svc.RunBatch(ctx, in.Changes, classes.BatchOptions{DryRun: in.DryRun}) + var refused *classes.BatchRefusedError + if errors.As(err, &refused) { + return 0, nil, &apiError{Status: http.StatusUnprocessableEntity, Body: BatchRefusal(refused.Result)} + } + if err != nil { + return 0, nil, err + } + return http.StatusOK, res, nil + }) +} + +// BatchRefusal is the 422 answer of a refused batch: the first refused +// change, its error, path or field errors, and every change's outcome. +func BatchRefusal(res classes.BatchResult) map[string]any { + item := res.Refused() + out := map[string]any{"error": "batch_refused", "changes": res.Changes} + if item == nil { + return out + } + out["change"] = item.Index + out["message"] = item.Message + if item.Error == classes.ConflictLoop { + out["error"] = "redirect_loop" + out["path"] = item.Path + } + if len(item.Errors) > 0 { + out["errors"] = item.Errors + } + return out +} diff --git a/controllers/api/history.go b/controllers/api/history.go index 62a7ded..714fb56 100644 --- a/controllers/api/history.go +++ b/controllers/api/history.go @@ -2,10 +2,12 @@ package api import ( "context" + "errors" "net/http" "git.golem15.com/golem15/sm-seo-plugin/classes" "git.golem15.com/golem15/sm-seo-plugin/models" + "gorm.io/gorm" ) // ListHistoryInput is the query of GET /_seo/api/v1/history. @@ -93,3 +95,64 @@ func ShowHistory(service func() *classes.Service) http.HandlerFunc { return http.StatusOK, map[string]any{"data": RevisionJSON(rows[0])}, nil }) } + +// HistoryBatchInput names one batch. +type HistoryBatchInput struct { + Batch string `json:"batch" path:"batch" jsonschema:"the batch id"` +} + +// undoAnswer maps an undo refusal to 409 with its code and message. +func undoAnswer(ctx context.Context, svc *classes.Service, err error) error { + var refusal *classes.UndoRefusal + if errors.As(err, &refusal) { + return &apiError{Status: http.StatusConflict, Body: map[string]any{ + "error": refusal.Code, + "message": svc.PhraseCtx(ctx, refusal.Key, refusal.Params), + }} + } + return err +} + +// UndoChange serves POST /_seo/api/v1/history/{id}/undo (D-13): the values +// before the change are restored and the undo is recorded (via undo). It +// answers 409 undo_stale when a later change touched the same values, +// undo_loop when the restore would create a redirect loop, undo_gone when +// the record is gone and already_undone; nothing is undone then. +func UndoChange(service func() *classes.Service) http.HandlerFunc { + return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) { + id, err := pathID(r, "id") + if err != nil { + return 0, nil, err + } + var ids []uint + err = write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error { + var err error + ids, err = svc.Undo(ctx, tx, id) + return err + }) + if err != nil { + return 0, nil, undoAnswer(ctx, svc, err) + } + return http.StatusOK, map[string]any{"undone": ids, "count": len(ids)}, nil + }) +} + +// UndoBatch serves POST /_seo/api/v1/history/batches/{batch}/undo: every +// change of the batch not undone yet is undone in one transaction, newest +// first, or none is (the same refusals as one change). 404 when nothing of +// the batch is left to undo. +func UndoBatch(service func() *classes.Service) http.HandlerFunc { + return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) { + batch := r.PathValue("batch") + var ids []uint + err := write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error { + var err error + ids, err = svc.UndoBatch(ctx, tx, batch) + return err + }) + if err != nil { + return 0, nil, undoAnswer(ctx, svc, err) + } + return http.StatusOK, map[string]any{"undone": ids, "count": len(ids)}, nil + }) +} diff --git a/controllers/api/redirects.go b/controllers/api/redirects.go index 0cbfe18..460f5f7 100644 --- a/controllers/api/redirects.go +++ b/controllers/api/redirects.go @@ -4,7 +4,6 @@ import ( "context" "net/http" "strconv" - "strings" "git.golem15.com/golem15/sm-seo-plugin/classes" "git.golem15.com/golem15/sm-seo-plugin/models" @@ -51,47 +50,6 @@ type UpdateRedirectInput struct { Note *string `json:"note,omitempty" jsonschema:"an internal note"` } -// RedirectBody is the decoded body of a redirect write; the batch and the -// import use it too. -type RedirectBody struct { - Source *string `json:"source,omitempty"` - Target *string `json:"target,omitempty"` - Status *int `json:"status,omitempty"` - Match *string `json:"match,omitempty"` - Query *string `json:"query,omitempty"` - IgnoreCase *bool `json:"ignore_case,omitempty"` - TrailingSlash *bool `json:"trailing_slash,omitempty"` - Note *string `json:"note,omitempty"` -} - -// Apply writes the fields sent onto r. -func (b RedirectBody) Apply(r *models.Redirect) { - if b.Source != nil { - r.Source = strings.TrimSpace(*b.Source) - } - if b.Target != nil { - r.Target = strings.TrimSpace(*b.Target) - } - if b.Status != nil { - r.Status = models.RedirectStatus(*b.Status) - } - if b.Match != nil { - r.Match = strings.TrimSpace(*b.Match) - } - if b.Query != nil { - r.Query = strings.TrimSpace(*b.Query) - } - if b.IgnoreCase != nil { - r.IgnoreCase = *b.IgnoreCase - } - if b.TrailingSlash != nil { - r.TrailingSlash = *b.TrailingSlash - } - if b.Note != nil { - r.Note = strings.TrimSpace(*b.Note) - } -} - // RedirectJSON is a redirect in an answer. func RedirectJSON(r models.Redirect) map[string]any { out := classes.RedirectValues(r) @@ -175,7 +133,7 @@ func ShowRedirect(service func() *classes.Service) http.HandlerFunc { // validation_failed for invalid fields. The origin is mcp through MCP. func CreateRedirect(service func() *classes.Service) http.HandlerFunc { return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) { - var body RedirectBody + var body classes.RedirectFields if err := decode(r, maxBody, &body); err != nil { return 0, nil, err } @@ -203,7 +161,7 @@ func UpdateRedirect(service func() *classes.Service) http.HandlerFunc { if err != nil { return 0, nil, err } - var body RedirectBody + var body classes.RedirectFields if err := decode(r, maxBody, &body); err != nil { return 0, nil, err } diff --git a/controllers/api/rules.go b/controllers/api/rules.go index 06e05ee..7a30893 100644 --- a/controllers/api/rules.go +++ b/controllers/api/rules.go @@ -3,7 +3,6 @@ package api import ( "context" "net/http" - "strings" "git.golem15.com/golem15/sm-seo-plugin/classes" "git.golem15.com/golem15/sm-seo-plugin/models" @@ -53,33 +52,6 @@ type UpdateRuleInput struct { Note *string `json:"note,omitempty" jsonschema:"an internal note"` } -// ruleBody is the decoded body of a rule write. -type ruleBody struct { - URL *string `json:"url"` - Match *string `json:"match"` - Title *string `json:"title"` - Description *string `json:"description"` - Canonical *string `json:"canonical"` - Robots *string `json:"robots"` - OGTitle *string `json:"og_title"` - OGDescription *string `json:"og_description"` - OGImage *string `json:"og_image"` - Note *string `json:"note"` -} - -// apply writes the fields sent onto r. -func (b ruleBody) apply(r *models.Rule) { - for column, v := range map[string]*string{ - "url": b.URL, "match": b.Match, "title": b.Title, "description": b.Description, - "canonical": b.Canonical, "robots": b.Robots, "og_title": b.OGTitle, - "og_description": b.OGDescription, "og_image": b.OGImage, "note": b.Note, - } { - if v != nil { - classes.SetRuleValue(r, column, strings.TrimSpace(*v)) - } - } -} - // RuleJSON is a rule in an answer. func RuleJSON(r models.Rule) map[string]any { out := classes.RuleValues(r) @@ -142,12 +114,12 @@ func ShowRule(service func() *classes.Service) http.HandlerFunc { // an invalid or taken address or invalid values. func CreateRule(service func() *classes.Service) http.HandlerFunc { return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) { - var body ruleBody + var body classes.RuleFields if err := decode(r, maxBody, &body); err != nil { return 0, nil, err } rule := models.Rule{Match: models.MatchExact} - body.apply(&rule) + body.Apply(&rule) err := write(ctx, svc, func(ctx context.Context, tx *gorm.DB) error { return svc.SaveRule(ctx, tx, &rule) }) @@ -165,7 +137,7 @@ func UpdateRule(service func() *classes.Service) http.HandlerFunc { if err != nil { return 0, nil, err } - var body ruleBody + var body classes.RuleFields if err := decode(r, maxBody, &body); err != nil { return 0, nil, err } @@ -175,7 +147,7 @@ func UpdateRule(service func() *classes.Service) http.HandlerFunc { if rule, err = classes.LoadRule(tx, id); err != nil { return err } - body.apply(&rule) + body.Apply(&rule) return svc.SaveRule(ctx, tx, &rule) }) if err != nil { diff --git a/controllers/api/transfer.go b/controllers/api/transfer.go new file mode 100644 index 0000000..a4df7fd --- /dev/null +++ b/controllers/api/transfer.go @@ -0,0 +1,127 @@ +package api + +import ( + "context" + "errors" + "net/http" + "strconv" + "strings" + + "git.golem15.com/golem15/sm-seo-plugin/classes" +) + +// ExportInput is the query of GET /_seo/api/v1/export. +type ExportInput struct { + Dataset string `json:"dataset" query:"dataset" jsonschema:"redirects, rules or records (the SEO of records, addressed by URL)"` + Delimiter string `json:"delimiter,omitempty" query:"delimiter" jsonschema:"the CSV delimiter: , or ; (the SEO settings' delimiter by default)"` + Page int `json:"page,omitempty" query:"page" jsonschema:"the page of 1000 rows, from 1"` +} + +// ImportInput is the body of POST /_seo/api/v1/import. +type ImportInput struct { + Dataset string `json:"dataset" jsonschema:"redirects, rules or records"` + CSV string `json:"csv" jsonschema:"the file: a header row with the export's column names, then the rows; at most 900 KB per call"` + Delimiter string `json:"delimiter,omitempty" jsonschema:"the CSV delimiter: , or ; (detected from the header when omitted)"` + DryRun bool `json:"dry_run,omitempty" jsonschema:"preview the rows (new, changed, unchanged, refused, with notes) without writing anything"` +} + +// mcpPreviewRows caps the preview rows an import answers through the API. +const mcpPreviewRows = 1000 + +// transferRefusal maps an import or export refusal: 422 with the code and +// the message (and the column of missing_column), 409 for a stale +// preview, 422 import_refused with the preview while a row is refused. +func transferRefusal(ctx context.Context, svc *classes.Service, err error) error { + var te *classes.TransferError + var refused *classes.ImportRefusedError + switch { + case errors.As(err, &te): + body := map[string]any{"error": te.Code, "message": svc.PhraseCtx(ctx, te.Key, te.Params)} + if c := te.Params["column"]; c != "" { + body["column"] = c + } + status := http.StatusUnprocessableEntity + if te.Code == "stale" { + status = http.StatusConflict + } + return &apiError{Status: status, Body: body} + case errors.As(err, &refused): + res := refused.Result + capRows(&res, mcpPreviewRows) + return &apiError{Status: http.StatusUnprocessableEntity, Body: map[string]any{ + "error": "import_refused", "message": svc.PhraseCtx(ctx, "golem15.seo::lang.transfer.refused_blocks", nil), + "summary": res.Summary, "rows": res.Rows, "total": res.Total, + }} + } + return err +} + +// capRows keeps the first max preview rows. +func capRows(res *classes.ImportResult, max int) { + if len(res.Rows) > max { + res.Rows = res.Rows[:max] + } +} + +// delimiterOf is the request's delimiter, else the settings' one. +func delimiterOf(ctx context.Context, svc *classes.Service, given string) string { + if d := strings.TrimSpace(given); d == "," || d == ";" { + return d + } + db, err := read(ctx, svc) + if err != nil { + return "," + } + row, err := classes.LoadSettings(db) + if err != nil || row.CSVDelimiter == "" { + return "," + } + return row.CSVDelimiter +} + +// ExportCSV serves GET /_seo/api/v1/export?dataset=&delimiter=&page= (D-13, +// P-27): one page of 1000 rows of a dataset as CSV text, UTF-8 with a BOM, +// formula cells prefixed with an apostrophe, with the page count. +func ExportCSV(service func() *classes.Service) http.HandlerFunc { + return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) { + q := r.URL.Query() + page, _ := strconv.Atoi(q.Get("page")) + out, err := svc.Export(ctx, q.Get("dataset"), delimiterOf(ctx, svc, q.Get("delimiter")), page, classes.ExportPageRows) + if err != nil { + return 0, nil, transferRefusal(ctx, svc, err) + } + return http.StatusOK, out, nil + }) +} + +// ImportCSV serves POST /_seo/api/v1/import (D-13, P-27): with dry_run the +// preview of every row; without it the rows are applied as one batch (via +// import) or, while a row is refused (a loop, an invalid value), nothing +// is (422 import_refused). A file over 900 KB is 413; an unreadable file, +// a missing column or an unknown dataset 422. +func ImportCSV(service func() *classes.Service) http.HandlerFunc { + return manage(service, func(ctx context.Context, svc *classes.Service, r *http.Request) (int, any, error) { + var in ImportInput + if err := decode(r, 2*maxBody, &in); err != nil { + return 0, nil, err + } + if len(in.CSV) > classes.MCPImportLimit { + return 0, nil, &apiError{Status: http.StatusRequestEntityTooLarge, Body: map[string]any{ + "error": "too_large", "message": svc.PhraseCtx(ctx, "golem15.seo::lang.transfer.too_large_mcp", nil), + }} + } + delimiter := strings.TrimSpace(in.Delimiter) + if delimiter != "," && delimiter != ";" { + delimiter = "" + } + res, err := svc.Import(ctx, classes.ImportRequest{Dataset: in.Dataset, CSV: in.CSV, Delimiter: delimiter, DryRun: in.DryRun}) + if err != nil { + return 0, nil, transferRefusal(ctx, svc, err) + } + capRows(&res, mcpPreviewRows) + return http.StatusOK, map[string]any{ + "dry_run": in.DryRun, "applied": res.Applied, "batch_id": res.BatchID, + "summary": res.Summary, "rows": res.Rows, "total": res.Total, "count": res.Count, + }, nil + }) +} diff --git a/controllers/history.go b/controllers/history.go index 23898a6..75faedc 100644 --- a/controllers/history.go +++ b/controllers/history.go @@ -8,8 +8,10 @@ import ( "sort" "strings" + "git.golem15.com/golem15/sm-seo-plugin/classes" "git.golem15.com/golem15/sm-seo-plugin/models" "git.golem15.com/golem15/summercms/modules/backpack" + "git.golem15.com/golem15/summercms/modules/cabana" "git.golem15.com/golem15/summercms/modules/pact" "gorm.io/gorm" ) @@ -20,10 +22,15 @@ const HistoryControllerID = "golem15.seo.history" // errHistoryReadOnly refuses writes through the history form. var errHistoryReadOnly = errors.New("golem15.seo: history is read-only") +// The history preview's record actions (UI-SPEC C12). +const ( + HistoryActionUndo = "undo" + HistoryActionUndoBatch = "undo_batch" +) + // historyAdminController lists and previews the SEO change history // (D-13). History is written by the plugin's service only: the form saves -// and deletes nothing. Undo arrives as record actions with the management -// API. +// and deletes nothing. The preview undoes one change or a whole batch. type historyAdminController struct { app func() *backpack.App } @@ -227,15 +234,88 @@ func (c historyAdminController) PartialData(ctx context.Context, name string, re return view, nil } +// AdminRecordActions are the preview's undo actions (UI-SPEC C12): undo +// applies while the change is not undone; undo_batch while the change +// belongs to a batch with changes not undone. Both confirm without the +// danger variant (an undo is itself undoable); a refusal (stale, loop, +// gone) is a 403 whose message the admin shows as a toast. +func (c historyAdminController) AdminRecordActions() []pact.AdminRecordAction { + return []pact.AdminRecordAction{ + { + Name: HistoryActionUndo, + Label: "golem15.seo::lang.history.undo", + Confirm: "golem15.seo::lang.history.undo_confirm", + Permissions: []string{PermissionManage}, + Applies: func(_ context.Context, record any) (bool, error) { + r, ok := record.(*models.Revision) + return ok && r != nil && classes.Undoable(*r), nil + }, + Run: func(ctx context.Context, in pact.AdminRecordActionInput) (pact.AdminRecordActionResult, error) { + return c.undo(ctx, in.Record, false) + }, + }, + { + Name: HistoryActionUndoBatch, + Label: "golem15.seo::lang.history.undo_batch", + Confirm: "golem15.seo::lang.history.undo_batch_confirm", + Permissions: []string{PermissionManage}, + Applies: func(ctx context.Context, record any) (bool, error) { + r, ok := record.(*models.Revision) + if !ok || r == nil || r.BatchID == nil || *r.BatchID == "" { + return false, nil + } + db, err := database(ctx, c.app) + if err != nil { + return false, err + } + n, err := classes.BatchSize(db, *r.BatchID) + return n > 0, err + }, + Run: func(ctx context.Context, in pact.AdminRecordActionInput) (pact.AdminRecordActionResult, error) { + return c.undo(ctx, in.Record, true) + }, + }, + } +} + +// undo runs an undo inside the action's transaction. +func (c historyAdminController) undo(ctx context.Context, record any, batch bool) (pact.AdminRecordActionResult, error) { + r, ok := record.(*models.Revision) + svc, err := classes.NewService(appOf(c.app)) + if !ok || r == nil || err != nil { + return pact.AdminRecordActionResult{}, classes.ErrNotBooted + } + db, err := database(ctx, c.app) + if err != nil { + return pact.AdminRecordActionResult{}, err + } + done := "golem15.seo::lang.history.undone" + if batch && r.BatchID != nil { + _, err = svc.UndoBatch(ctx, db, *r.BatchID) + done = "golem15.seo::lang.history.batch_undone" + } else { + _, err = svc.Undo(ctx, db, r.ID) + } + var refusal *classes.UndoRefusal + if errors.As(err, &refusal) { + return pact.AdminRecordActionResult{}, &cabana.ForbiddenError{Message: svc.PhraseCtx(ctx, refusal.Key, refusal.Params)} + } + if err != nil { + return pact.AdminRecordActionResult{}, err + } + return pact.AdminRecordActionResult{Message: done}, nil +} + var ( - _ pact.AdminController = historyAdminController{} - _ pact.AdminPermissioned = historyAdminController{} - _ pact.AdminRecordSource = historyAdminController{} - _ pact.FormBeforeCreate = historyAdminController{} - _ pact.FormBeforeUpdate = historyAdminController{} - _ pact.FormBeforeDelete = historyAdminController{} - _ pact.ListExtendRecords = historyAdminController{} - _ pact.ListRowStates = historyAdminController{} - _ pact.FilterOptions = historyAdminController{} - _ pact.AdminPartialData = historyAdminController{} + _ pact.HasAdminRecordActions = historyAdminController{} + _ pact.AdminController = historyAdminController{} + _ pact.AdminPermissioned = historyAdminController{} + _ pact.AdminRecordSource = historyAdminController{} + _ pact.FormBeforeCreate = historyAdminController{} + _ pact.FormBeforeUpdate = historyAdminController{} + _ pact.FormBeforeDelete = historyAdminController{} + _ pact.ListExtendRecords = historyAdminController{} + _ pact.ListRowStates = historyAdminController{} + _ pact.FilterOptions = historyAdminController{} + _ pact.AdminPartialData = historyAdminController{} ) diff --git a/controllers/history/config_form.yaml b/controllers/history/config_form.yaml index e90775f..223fad8 100644 --- a/controllers/history/config_form.yaml +++ b/controllers/history/config_form.yaml @@ -1,8 +1,13 @@ -# History preview (UI-SPEC C12): read-only with the P5 diff. +# History preview (UI-SPEC C12): read-only with the P5 diff, and the undo +# of the change or of its whole batch. name: golem15.seo::lang.history.name form: ~/plugins/golem15/seo/models/revision/fields.yaml modelClass: Golem15\SEO\Models\Revision defaultRedirect: golem15/seo/history +preview: {} + +recordActions: [undo, undo_batch] + messages: preview: golem15.seo::lang.history.preview diff --git a/controllers/transfer.go b/controllers/transfer.go new file mode 100644 index 0000000..a954b69 --- /dev/null +++ b/controllers/transfer.go @@ -0,0 +1,199 @@ +package controllers + +import ( + "context" + "encoding/json" + "errors" + "strings" + + "git.golem15.com/golem15/sm-seo-plugin/classes" + "git.golem15.com/golem15/sm-seo-plugin/models" + "git.golem15.com/golem15/summercms/modules/backpack" + "git.golem15.com/golem15/summercms/modules/cabana" + "git.golem15.com/golem15/summercms/modules/pact" + "gorm.io/gorm" +) + +// TransferControllerID is the import and export screen (UI-SPEC C13). +const TransferControllerID = "golem15.seo.transfer" + +// TransferAction is the files widget's action (E6): ops export, preview +// and apply. +const TransferAction = "seo_transfer" + +// transferAdminController is the import and export singleton (D-12, D-13, +// P-27) on the settings row: the delimiter and the E6 files widget. +type transferAdminController struct { + app func() *backpack.App +} + +func (transferAdminController) ID() string { return TransferControllerID } +func (transferAdminController) ModelName() string { return `Golem15\SEO\Models\Transfer` } +func (transferAdminController) ConfigDir() string { return "controllers/transfer" } +func (transferAdminController) NewRecord() any { return &models.Transfer{} } +func (transferAdminController) RequiredPermissions() []string { + return []string{PermissionManage} +} + +func (c transferAdminController) service() *classes.Service { + svc, err := classes.NewService(appOf(c.app)) + if err != nil { + return nil + } + return svc +} + +// SingletonRecord returns the settings row as the screen's record; an +// unset delimiter shows semicolon when the site's default locale is pl +// (Excel in Polish), else comma. +func (c transferAdminController) SingletonRecord(ctx context.Context, db *gorm.DB) (any, error) { + if db == nil { + return nil, errors.New("golem15.seo: database is not configured") + } + if _, err := classes.SettingsRow(db); err != nil { + return nil, err + } + var rows []models.Transfer + if err := db.Where("id = ?", 1).Limit(1).Find(&rows).Error; err != nil || len(rows) == 0 { + return nil, errors.New("golem15.seo: settings row is missing") + } + row := rows[0] + if row.CSVDelimiter == "" { + row.CSVDelimiter = "," + if svc := c.service(); svc != nil && strings.HasPrefix(svc.DefaultLocale(ctx, db), "pl") { + row.CSVDelimiter = ";" + } + } + return &row, nil +} + +// FormBeforeUpdate refuses a delimiter other than comma or semicolon. +func (c transferAdminController) FormBeforeUpdate(ctx context.Context, model any) error { + row, ok := model.(*models.Transfer) + if !ok || row == nil { + return nil + } + if row.CSVDelimiter != "," && row.CSVDelimiter != ";" { + return &cabana.ValidationError{Details: map[string]any{"csv_delimiter": []string{Phrase(ctx, translator(c.app), "golem15.seo::lang.api.value_invalid", nil)}}} + } + return nil +} + +// AdminJS loads the SEO element bundle (E6) on the screen. +func (transferAdminController) AdminJS() []string { return []string{"assets/dist/seo.js"} } + +// AdminCSS loads the SEO element styles on the screen. +func (transferAdminController) AdminCSS() []string { return []string{"assets/dist/seo.css"} } + +// transferPayload is the widget's request. +type transferPayload struct { + Op string `json:"op"` + Dataset string `json:"dataset"` + CSV string `json:"csv"` + Token string `json:"token"` +} + +// AdminActions registers the files widget's action. +func (c transferAdminController) AdminActions() []pact.AdminAction { + return []pact.AdminAction{{ + Name: TransferAction, + Label: "golem15.seo::lang.transfer.preview", + Permissions: []string{PermissionManage}, + Run: c.run, + }} +} + +// refusal is an op's answer that the element shows above its actions: a +// refusal it can explain, sent as data so the preview stays on screen. +func refusal(op, code, message string) pact.AdminActionResult { + return pact.AdminActionResult{Data: map[string]any{"op": op, "error": code, "message": message}} +} + +// run serves the widget (UI-SPEC C13): export answers {filename, csv}; +// preview answers {token, summary, rows (the first 200, refused first), +// total, count}; apply re-runs the preview of the token's file, refuses +// with transfer.stale when the data changed since, and applies every row +// as one import batch. +func (c transferAdminController) run(ctx context.Context, in pact.AdminActionInput) (pact.AdminActionResult, error) { + svc := c.service() + if svc == nil { + return pact.AdminActionResult{}, classes.ErrNotBooted + } + var p transferPayload + if len(in.Payload) > 0 { + if err := json.Unmarshal(in.Payload, &p); err != nil { + return pact.AdminActionResult{}, &cabana.ValidationError{Details: map[string]any{in.Field: []string{"golem15.seo::lang.transfer.unreadable"}}} + } + } + delimiter, _ := in.Values["csv_delimiter"].(string) + if delimiter != ";" { + delimiter = "," + } + phrase := func(key string, params map[string]string) string { return svc.PhraseCtx(ctx, key, params) } + failed := func(op string, err error) (pact.AdminActionResult, error) { + var te *classes.TransferError + var refused *classes.ImportRefusedError + switch { + case errors.As(err, &te): + return refusal(op, te.Code, phrase(te.Key, te.Params)), nil + case errors.As(err, &refused): + return refusal(op, "refused", phrase("golem15.seo::lang.transfer.refused_blocks", nil)), nil + } + return pact.AdminActionResult{}, err + } + switch p.Op { + case "export": + out, err := svc.Export(ctx, p.Dataset, delimiter, 0, 0) + if err != nil { + return failed("export", err) + } + if len(out.CSV) > classes.ExportActionLimit { + return refusal("export", "too_large", phrase("golem15.seo::lang.transfer.export_too_large", nil)), nil + } + return pact.AdminActionResult{Data: map[string]any{"op": "export", "filename": out.Filename, "csv": out.CSV}}, nil + case "preview": + if len(p.CSV) > classes.AdminImportLimit { + return refusal("preview", "too_large", phrase("golem15.seo::lang.transfer.too_large", nil)), nil + } + res, err := svc.Import(ctx, classes.ImportRequest{Dataset: p.Dataset, CSV: p.CSV, Delimiter: delimiter, DryRun: true}) + if err != nil { + return failed("preview", err) + } + token, err := svc.TransferToken(p.Dataset, delimiter, p.CSV, res.Fingerprint) + if err != nil { + return pact.AdminActionResult{}, err + } + rows := res.Rows + if len(rows) > classes.PreviewRowCap { + rows = rows[:classes.PreviewRowCap] + } + return pact.AdminActionResult{Data: map[string]any{ + "op": "preview", "token": token, "dataset": p.Dataset, "summary": res.Summary, + "rows": rows, "total": res.Total, "count": res.Count, + }}, nil + case "apply": + req, err := svc.ReadTransferToken(p.Token) + if err != nil { + return refusal("apply", "token", phrase("golem15.seo::lang.transfer.token_invalid", nil)), nil + } + res, err := svc.Import(ctx, req) + if err != nil { + return failed("apply", err) + } + return pact.AdminActionResult{ + Message: svc.ChoiceCtx(ctx, "golem15.seo::lang.transfer.applied", res.Count, nil), + Data: map[string]any{"op": "apply", "applied": true, "count": res.Count, "batch_id": res.BatchID}, + }, nil + } + return pact.AdminActionResult{}, &cabana.ValidationError{Details: map[string]any{in.Field: []string{"golem15.seo::lang.transfer.unreadable"}}} +} + +var ( + _ pact.AdminController = transferAdminController{} + _ pact.AdminPermissioned = transferAdminController{} + _ pact.AdminRecordSource = transferAdminController{} + _ pact.AdminSingleton = transferAdminController{} + _ pact.FormBeforeUpdate = transferAdminController{} + _ pact.HasAdminActions = transferAdminController{} + _ pact.AdminClientAssets = transferAdminController{} +) diff --git a/controllers/transfer/config_form.yaml b/controllers/transfer/config_form.yaml new file mode 100644 index 0000000..4cf8ec9 --- /dev/null +++ b/controllers/transfer/config_form.yaml @@ -0,0 +1,14 @@ +# Import and export singleton (UI-SPEC C13): the footer Save stores the +# delimiter only. +name: golem15.seo::lang.transfer.title +form: ~/plugins/golem15/seo/models/transfer/fields.yaml +modelClass: Golem15\SEO\Models\Transfer +defaultRedirect: golem15/seo/transfer + +update: + redirect: golem15/seo/transfer + redirectClose: golem15/seo/redirects + +messages: + update: golem15.seo::lang.transfer.title + saved: backend::lang.messages.form.saved diff --git a/lang/en/lang.yaml b/lang/en/lang.yaml index b63634e..05e0ab6 100644 --- a/lang/en/lang.yaml +++ b/lang/en/lang.yaml @@ -8,6 +8,7 @@ nav: seo: SEO history: History rules: URL rules + transfer: Import and export permissions: tab: SEO manage: Manage SEO @@ -184,6 +185,23 @@ history: diff_before: Before diff_after: After list_empty: No SEO changes recorded yet. Every change from the admin, MCP or an import appears here and can be undone. + undo: Undo this change + undo_confirm: Undo this change? The values before it are restored, and the undo is recorded as a new change. + undone: Change undone. + undo_batch: Undo the whole batch + undo_batch_confirm: Undo every change of this batch? It is undone as a whole or not at all. + batch_undone: Batch undone. + undo_stale: "“:field” has changed since: :author changed it on :time. Undoing would overwrite that. Nothing was undone." + undo_loop: "Undoing would create a redirect loop: :path. Nothing was undone." + undo_gone: The :type this change belongs to no longer exists. Nothing was undone. + undo_done: This change is already undone. + undo_unsupported: This change cannot be undone here. Nothing was undone. + record: record + kind_rule: URL rule + kind_redirect: redirect + kind_robots: robots.txt + kind_settings: SEO settings + kind_notfound: 404 log entry jobs: generate_entries: 'SEO: generate entries' effective: @@ -298,3 +316,32 @@ api: bot_unknown: This is not an AI bot of the robots.txt screen. value_invalid: This value is not valid here. type_unknown: No content type with this key is registered. + batch_too_large: "A batch takes at most :max changes." + change_not_found: The record this change names does not exist. + op_unknown: "Unknown operation “:op”." + conflict_noindex_published: "“:url” is a published page listed in the sitemaps; noindex removes it from search results." +transfer: + title: Import and export + delimiter: CSV delimiter + delimiter_comma: Comma (,) + delimiter_semicolon: Semicolon (;), as Excel uses in Polish + delimiter_comment: Files are UTF-8 with a byte order mark, so Excel opens Polish characters correctly. + widget: Files + preview: Preview import + applied: + one: "Applied :count change as one batch. You can undo it in History." + other: "Applied :count changes as one batch. You can undo them in History." + stale: The data changed after the preview. Preview the file again before applying. + unreadable: The file could not be read as CSV. Check the delimiter and that the first row holds the column names. + missing_column: "The file has no “:column” column. Download an export to see the expected columns." + dataset_unknown: Choose redirects, URL rules or SEO of records. + too_large: This file is larger than 60 KB. Split it into smaller files, or import it through MCP. + too_large_mcp: This file is larger than 900 KB. Import it in smaller parts. + export_too_large: This export is too large to download here. Export it through MCP, page by page. + refused_blocks: Fix the refused rows in the file and preview it again. Nothing is applied while a row is refused. + token_invalid: This preview is no longer valid. Preview the file again. + row_loop: "Creates a redirect loop: :path" + row_chain: "Creates a chain: :path" + row_canonical: "The canonical URL redirects to “:target”" + row_rule: "No record at this address: a URL rule is created" + row_invalid: ":field: :message" diff --git a/lang/pl/lang.yaml b/lang/pl/lang.yaml index 6be1dd3..b762a68 100644 --- a/lang/pl/lang.yaml +++ b/lang/pl/lang.yaml @@ -8,6 +8,7 @@ nav: seo: SEO history: Historia zmian rules: Reguły adresów + transfer: Import i eksport permissions: tab: SEO manage: Zarządzanie SEO @@ -190,6 +191,23 @@ history: diff_before: Przed diff_after: Po list_empty: Nie zarejestrowano jeszcze zmian SEO. Każda zmiana z panelu, MCP lub importu pojawi się tutaj i będzie można ją cofnąć. + undo: Cofnij tę zmianę + undo_confirm: Cofnąć tę zmianę? Przywrócone zostaną wcześniejsze wartości, a cofnięcie zostanie zapisane jako nowa zmiana. + undone: Cofnięto zmianę. + undo_batch: Cofnij całą partię + undo_batch_confirm: Cofnąć wszystkie zmiany z tej partii? Cofa się całość albo nic. + batch_undone: Cofnięto partię. + undo_stale: "„:field” zmieniło się od tego czasu: :author zmienił je :time. Cofnięcie nadpisałoby tę zmianę. Nic nie cofnięto." + undo_loop: "Cofnięcie utworzyłoby pętlę przekierowań: :path. Nic nie cofnięto." + undo_gone: ":type, którego dotyczy ta zmiana, już nie istnieje. Nic nie cofnięto." + undo_done: Ta zmiana jest już cofnięta. + undo_unsupported: Tej zmiany nie można tu cofnąć. Nic nie cofnięto. + record: Rekord + kind_rule: Reguła adresu + kind_redirect: Przekierowanie + kind_robots: robots.txt + kind_settings: Ustawienia SEO + kind_notfound: Wpis dziennika 404 jobs: generate_entries: 'SEO: generowanie wpisów' effective: @@ -308,3 +326,34 @@ api: bot_unknown: To nie jest bot AI z ekranu robots.txt. value_invalid: Ta wartość jest tu nieprawidłowa. type_unknown: Nie zarejestrowano typu treści o tym kluczu. + batch_too_large: "Partia może mieć najwyżej :max zmian." + change_not_found: Rekord wskazany w tej zmianie nie istnieje. + op_unknown: "Nieznana operacja „:op”." + conflict_noindex_published: "„:url” to opublikowana strona z map witryny; noindex usunie ją z wyników wyszukiwania." +transfer: + title: Import i eksport + delimiter: Separator CSV + delimiter_comma: Przecinek (,) + delimiter_semicolon: Średnik (;), jak w polskim Excelu + delimiter_comment: Pliki są w UTF-8 ze znacznikiem BOM, więc Excel poprawnie otwiera polskie znaki. + widget: Pliki + preview: Podgląd importu + applied: + one: "Zastosowano :count zmianę jako jedną partię. Możesz ją cofnąć w Historii zmian." + few: "Zastosowano :count zmiany jako jedną partię. Możesz je cofnąć w Historii zmian." + many: "Zastosowano :count zmian jako jedną partię. Możesz je cofnąć w Historii zmian." + other: "Zastosowano :count zmiany jako jedną partię. Możesz je cofnąć w Historii zmian." + stale: Dane zmieniły się po podglądzie. Wyświetl podgląd pliku ponownie przed zastosowaniem. + unreadable: Nie udało się odczytać pliku jako CSV. Sprawdź separator i czy pierwszy wiersz zawiera nazwy kolumn. + missing_column: "W pliku brakuje kolumny „:column”. Pobierz eksport, aby zobaczyć oczekiwane kolumny." + dataset_unknown: Wybierz przekierowania, reguły adresów lub SEO rekordów. + too_large: Ten plik jest większy niż 60 KB. Podziel go na mniejsze pliki lub zaimportuj przez MCP. + too_large_mcp: Ten plik jest większy niż 900 KB. Zaimportuj go w mniejszych częściach. + export_too_large: Ten eksport jest za duży, by pobrać go tutaj. Wyeksportuj go przez MCP, strona po stronie. + refused_blocks: Popraw odrzucone wiersze w pliku i ponownie wyświetl podgląd. Nic nie zostanie zastosowane, dopóki jakiś wiersz jest odrzucony. + token_invalid: Ten podgląd jest już nieważny. Wyświetl podgląd pliku ponownie. + row_loop: "Tworzy pętlę przekierowań: :path" + row_chain: "Tworzy łańcuch: :path" + row_canonical: "Adres kanoniczny przekierowuje na „:target”" + row_rule: "Pod tym adresem nie ma rekordu: zostanie utworzona reguła adresu" + row_invalid: ":field: :message" diff --git a/mcp.go b/mcp.go index eff8f66..364652c 100644 --- a/mcp.go +++ b/mcp.go @@ -274,6 +274,60 @@ func (p *Plugin) MCP() pact.MCP { Permissions: []string{mcpPermission}, }, + // Undo (D-13). + { + Name: "undo_change", + Title: "Undo an SEO change", + Description: "Restores the values before one SEO change from history; the undo is itself recorded. Refused with undo_stale when a later change touched the same values (naming who and when), undo_loop when the restore would create a redirect loop, undo_gone when the record is gone.", + Guard: mcpGuard, + Method: http.MethodPost, + Path: "/_seo/api/v1/history/{id}/undo", + Input: apicontrollers.HistoryIDInput{}, + Permissions: []string{mcpPermission}, + }, + { + Name: "undo_batch", + Title: "Undo a batch", + Description: "Undoes every change of a batch or an import (by batch id) as a whole, or none of them, with the refusals of undo_change.", + Guard: mcpGuard, + Method: http.MethodPost, + Path: "/_seo/api/v1/history/batches/{batch}/undo", + Input: apicontrollers.HistoryBatchInput{}, + Permissions: []string{mcpPermission}, + }, + + // Bulk changes and CSV (D-12, D-13). + { + Name: "apply_batch", + Title: "Apply a batch of SEO changes", + Description: "Applies a list of changes (set_seo by URL, redirect and rule creates, updates and deletes) all or nothing. With dry_run it answers the diff and the conflicts (redirect loops and chains, noindex on a published page, a canonical pointing to a redirected page) and writes nothing; a loop refuses the whole batch.", + Guard: mcpGuard, + Method: http.MethodPost, + Path: "/_seo/api/v1/batch", + Input: apicontrollers.ApplyBatchInput{}, + Permissions: []string{mcpPermission}, + }, + { + Name: "export_csv", + Title: "Export SEO data as CSV", + Description: "Exports redirects, URL rules or the SEO of records (by URL) as CSV, UTF-8 with a BOM, 1000 rows per page; cells that start a formula are prefixed with an apostrophe.", + Guard: mcpGuard, + Method: http.MethodGet, + Path: "/_seo/api/v1/export", + Input: apicontrollers.ExportInput{}, + Permissions: []string{mcpPermission}, + }, + { + Name: "import_csv", + Title: "Import SEO data from CSV", + Description: "Imports redirects, URL rules or the SEO of records (rows address pages by URL; no record means a URL rule) from CSV of at most 900 KB. With dry_run it previews every row as new, changed, unchanged or refused with notes; otherwise every row is applied as one batch, or nothing while a row is refused (a loop, an invalid value).", + Guard: mcpGuard, + Method: http.MethodPost, + Path: "/_seo/api/v1/import", + Input: apicontrollers.ImportInput{}, + Permissions: []string{mcpPermission}, + }, + // Audit (D-13, D-14 CMS part). { Name: "list_issues", diff --git a/models/transfer.go b/models/transfer.go new file mode 100644 index 0000000..1bfae15 --- /dev/null +++ b/models/transfer.go @@ -0,0 +1,24 @@ +package models + +import "time" + +// Transfer is the import and export screen's view of the settings row +// (id 1 on golem15_seo_settings, UI-SPEC C13): the CSV delimiter its +// footer Save stores. It is not listed in All, because the row's model is +// Settings. +type Transfer struct { + ID uint `gorm:"column:id;primaryKey"` + CSVDelimiter string `gorm:"column:csv_delimiter"` + UpdatedAt *time.Time `gorm:"column:updated_at"` +} + +func (Transfer) TableName() string { return "golem15_seo_settings" } + +// MorphName names the screen's record. +func (Transfer) MorphName() string { return `Golem15\SEO\Models\Transfer` } + +// Fillable is the delimiter. +func (Transfer) Fillable() []string { return []string{"csv_delimiter"} } + +// Rules is empty: the controller checks the delimiter. +func (Transfer) Rules() map[string]string { return map[string]string{} } diff --git a/models/transfer/fields.yaml b/models/transfer/fields.yaml new file mode 100644 index 0000000..5af1bd4 --- /dev/null +++ b/models/transfer/fields.yaml @@ -0,0 +1,20 @@ +# Import and export (UI-SPEC C13): the CSV delimiter the footer Save +# stores, and the files widget (E6) with the export, the import preview and +# the apply, all through the seo_transfer action. +fields: + csv_delimiter: + label: golem15.seo::lang.transfer.delimiter + comment: golem15.seo::lang.transfer.delimiter_comment + type: dropdown + span: left + options: + ',': golem15.seo::lang.transfer.delimiter_comma + ';': golem15.seo::lang.transfer.delimiter_semicolon + + _transfer: + label: golem15.seo::lang.transfer.widget + type: widget + widget: golem15-seo-transfer + action: seo_transfer + fill: [csv_delimiter] + span: full diff --git a/operations.go b/operations.go index f888ddb..fd9fc3b 100644 --- a/operations.go +++ b/operations.go @@ -3,7 +3,8 @@ package seo import "git.golem15.com/golem15/sm-seo-plugin/controllers" // The kinds of an admin operation (AdminOperation.Kind). Record, bulk and -// widget actions add ":" and the action's name. +// widget actions add ":" and the action's name, and a widget action with +// several ops "#" and the op. const ( // OpList is a controller's list. OpList = "list" @@ -92,4 +93,15 @@ var AdminOperations = []AdminOperation{ // History. {controllers.HistoryControllerID, OpList, "list_history"}, {controllers.HistoryControllerID, OpShow, "show_history"}, + {controllers.HistoryControllerID, OpRecordAction + ":" + controllers.HistoryActionUndo, "undo_change"}, + {controllers.HistoryControllerID, OpRecordAction + ":" + controllers.HistoryActionUndoBatch, "undo_batch"}, + + // Import and export: the delimiter is a settings column; the files + // widget's ops export, preview and apply. + {controllers.TransferControllerID, OpShow, "show_settings"}, + {controllers.TransferControllerID, OpUpdate, "update_settings"}, + {controllers.TransferControllerID, OpWidgetAction + ":" + controllers.TransferAction, "import_csv"}, + {controllers.TransferControllerID, OpWidgetAction + ":" + controllers.TransferAction + "#export", "export_csv"}, + {controllers.TransferControllerID, OpWidgetAction + ":" + controllers.TransferAction + "#preview", "import_csv"}, + {controllers.TransferControllerID, OpWidgetAction + ":" + controllers.TransferAction + "#apply", "import_csv"}, } diff --git a/redirects_test.go b/redirects_test.go index 6e7687d..0eca876 100644 --- a/redirects_test.go +++ b/redirects_test.go @@ -5,6 +5,7 @@ import ( "context" "encoding/json" "errors" + "fmt" "net/http" "net/http/httptest" "sort" @@ -423,6 +424,40 @@ func TestLoopRefused(t *testing.T) { if rows := env.revisions(t, "redirect"); len(rows) < 4 { t.Fatalf("redirect history rows = %d, want every admin save recorded", len(rows)) } + + // The same loop is refused from MCP (a tool error with the path) and + // from a CSV import (a refused row that blocks the apply), SC-3. + t.Run("mcp", func(t *testing.T) { + agent := env.principal(t, env.admin(t, false, "golem15.seo.manage")) + before := env.redirectCount(t) + rec := env.mcp(t, http.MethodPost, "/_seo/api/v1/redirects", agent, map[string]any{"source": "/lb", "target": "/la"}) + if rec.Code != http.StatusUnprocessableEntity || !strings.Contains(rec.Body.String(), `"error":"redirect_loop"`) || !strings.Contains(rec.Body.String(), `"/lb"`) { + t.Fatalf("MCP loop = %d %s, want 422 redirect_loop with the path", rec.Code, rec.Body.String()) + } + rec = env.mcp(t, http.MethodPost, "/_seo/api/v1/batch", agent, map[string]any{"changes": []any{map[string]any{"op": "create_redirect", "redirect": map[string]any{"source": "/lb", "target": "/la"}}}}) + if rec.Code != http.StatusUnprocessableEntity || !strings.Contains(rec.Body.String(), `"error":"redirect_loop"`) { + t.Fatalf("MCP batch loop = %d %s", rec.Code, rec.Body.String()) + } + if after := env.redirectCount(t); after != before { + t.Fatalf("redirects after the MCP loop = %d, want %d", after, before) + } + }) + t.Run("import", func(t *testing.T) { + agent := env.principal(t, env.admin(t, false, "golem15.seo.manage")) + before := env.redirectCount(t) + file := "source,target\n/lb,/la\n" + preview := env.mcpExpect(t, http.StatusOK, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "redirects", "csv": file, "delimiter": ",", "dry_run": true}) + if !strings.Contains(fmt.Sprint(preview["rows"]), "Creates a redirect loop: /lb → /la → /lb") { + t.Fatalf("import preview = %v, want transfer.row_loop", preview) + } + rec := env.mcp(t, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "redirects", "csv": file, "delimiter": ","}) + if rec.Code != http.StatusUnprocessableEntity || !strings.Contains(rec.Body.String(), `"error":"import_refused"`) { + t.Fatalf("import apply = %d %s, want 422 import_refused", rec.Code, rec.Body.String()) + } + if after := env.redirectCount(t); after != before { + t.Fatalf("redirects after the refused import = %d, want %d", after, before) + } + }) } // report posts a 404 report with token ("" sends none). diff --git a/routes.go b/routes.go index 3e6d354..ac4ae5c 100644 --- a/routes.go +++ b/routes.go @@ -65,6 +65,11 @@ func (p *Plugin) Routes(r pact.Router) error { g.Put("/settings", apicontrollers.UpdateSettings(p.Service), api) g.Get("/history", apicontrollers.ListHistory(p.Service), api) g.Get("/history/{id}", apicontrollers.ShowHistory(p.Service), api) + g.Post("/history/{id}/undo", apicontrollers.UndoChange(p.Service), api) + g.Post("/history/batches/{batch}/undo", apicontrollers.UndoBatch(p.Service), api) + g.Get("/export", apicontrollers.ExportCSV(p.Service), api) + g.Post("/import", apicontrollers.ImportCSV(p.Service), api) + g.Post("/batch", apicontrollers.ApplyBatch(p.Service), api) g.Get("/issues", apicontrollers.ListIssues(p.Service), api) }) return nil diff --git a/transfer_test.go b/transfer_test.go new file mode 100644 index 0000000..b430464 --- /dev/null +++ b/transfer_test.go @@ -0,0 +1,175 @@ +package seo_test + +import ( + "encoding/json" + "fmt" + "net/http" + "strings" + "testing" +) + +const bom = "\ufeff" + +// transferAction calls the import and export screen's widget action. +func (e *testEnv) transferAction(t *testing.T, token string, delimiter string, payload map[string]any) map[string]any { + t.Helper() + rec := e.expect(t, http.StatusOK, http.MethodPost, adminAPI("/golem15/seo/transfer/widgets/_transfer"), token, map[string]any{ + "record_id": 1, + "values": map[string]any{"csv_delimiter": delimiter}, + "payload": payload, + }) + var out struct { + Data struct { + Data map[string]any `json:"data"` + Message string `json:"message"` + } `json:"data"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil { + t.Fatalf("decode %s: %v", rec.Body.String(), err) + } + if out.Data.Data == nil { + out.Data.Data = map[string]any{} + } + out.Data.Data["_message"] = out.Data.Message + return out.Data.Data +} + +// summary reads an import answer's summary as counts. +func summary(t *testing.T, body map[string]any) map[string]int { + t.Helper() + raw, _ := body["summary"].(map[string]any) + out := map[string]int{} + for k, v := range raw { + f, _ := v.(float64) + out[k] = int(f) + } + return out +} + +// TestCSVTransfer checks D-13 import and export (P-27, T-32-24/25/29): +// the export is UTF-8 with a BOM, the chosen delimiter and formula cells +// prefixed with an apostrophe, and importing it back changes nothing; +// missing columns and unreadable files are refused; a looping row is +// refused and blocks the apply; a records row without a record becomes a +// URL rule; the admin's apply token is refused once the data changed and +// applies one import batch when fresh; size caps hold on both paths. +func TestCSVTransfer(t *testing.T) { + env := bootEnv(t) + token := env.admin(t, true) + env.createPage(t, token, map[string]any{"title": "Page A", "slug": "a", "excerpt": "About A", "published": true}) + agent := env.principal(t, env.admin(t, false, "golem15.seo.manage")) + env.mcpExpect(t, http.StatusCreated, http.MethodPost, "/_seo/api/v1/redirects", agent, map[string]any{"source": "/e1", "target": "/t1", "note": "=cmd"}) + env.mcpExpect(t, http.StatusCreated, http.MethodPost, "/_seo/api/v1/redirects", agent, map[string]any{"source": "/e2", "target": "/t2", "status": 302}) + + exp := env.mcpExpect(t, http.StatusOK, http.MethodGet, "/_seo/api/v1/export?dataset=redirects&delimiter=%3B", agent, nil) + csv, _ := exp["csv"].(string) + if !strings.HasPrefix(csv, bom+"source;match;target;status;query;ignore_case;trailing_slash;note") { + t.Fatalf("export = %q, want a BOM and the semicolon header", csv) + } + if !strings.Contains(csv, "'=cmd") || strings.Contains(csv, ";=cmd") { + t.Fatalf("export = %q, want the formula cell prefixed with an apostrophe", csv) + } + if !strings.HasPrefix(fmt.Sprint(exp["filename"]), "seo-redirects-") { + t.Fatalf("filename = %v", exp["filename"]) + } + + preview := env.mcpExpect(t, http.StatusOK, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "redirects", "csv": csv, "delimiter": ";", "dry_run": true}) + if s := summary(t, preview); s["unchanged"] != 2 || s["new"] != 0 || s["changed"] != 0 || s["refused"] != 0 { + t.Fatalf("re-import of the export = %v, want 2 unchanged", preview) + } + + // Refusals of the file as a whole. + rec := env.mcp(t, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "redirects", "csv": "target;status\n/x;301\n", "delimiter": ";", "dry_run": true}) + if rec.Code != http.StatusUnprocessableEntity || !strings.Contains(rec.Body.String(), `"error":"missing_column"`) || !strings.Contains(rec.Body.String(), "source") { + t.Fatalf("missing column = %d %s", rec.Code, rec.Body.String()) + } + rec = env.mcp(t, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "redirects", "csv": "source;target\n\"/x;/y\n", "delimiter": ";", "dry_run": true}) + if rec.Code != http.StatusUnprocessableEntity || !strings.Contains(rec.Body.String(), `"error":"unreadable"`) { + t.Fatalf("unreadable = %d %s", rec.Code, rec.Body.String()) + } + if rec := env.mcp(t, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "nope", "csv": "a\n"}); rec.Code != http.StatusUnprocessableEntity { + t.Fatalf("unknown dataset = %d, want 422", rec.Code) + } + + // A looping row is refused and nothing is applied. + looping := "source,target\n/new1,/t1\n/e1x,/e1\n/t1,/new1\n" + preview = env.mcpExpect(t, http.StatusOK, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "redirects", "csv": looping, "delimiter": ",", "dry_run": true}) + if s := summary(t, preview); s["refused"] != 1 || s["new"] != 2 { + t.Fatalf("looping preview = %v", preview) + } + if !strings.Contains(fmt.Sprint(preview["rows"]), "Creates a redirect loop") { + t.Fatalf("looping rows = %v, want transfer.row_loop", preview["rows"]) + } + before := env.redirectCount(t) + rec = env.mcp(t, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "redirects", "csv": looping, "delimiter": ","}) + if rec.Code != http.StatusUnprocessableEntity || !strings.Contains(rec.Body.String(), `"error":"import_refused"`) { + t.Fatalf("looping apply = %d %s, want 422 import_refused", rec.Code, rec.Body.String()) + } + if n := env.redirectCount(t); n != before { + t.Fatalf("redirects after a refused import = %d, want %d", n, before) + } + + // Records address pages by URL; no record means a URL rule. + records := "url;locale;title;robots\n/pages/a;en;Imported;noindex\n/no-record;;Rule title;\n" + preview = env.mcpExpect(t, http.StatusOK, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "records", "csv": records, "delimiter": ";", "dry_run": true}) + if s := summary(t, preview); s["changed"] != 1 || s["new"] != 1 { + t.Fatalf("records preview = %v", preview) + } + if !strings.Contains(fmt.Sprint(preview["rows"]), "No record at this address") { + t.Fatalf("records rows = %v, want transfer.row_rule", preview["rows"]) + } + applied := env.mcpExpect(t, http.StatusOK, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "records", "csv": records, "delimiter": ";"}) + if applied["applied"] != true || applied["count"] != float64(2) { + t.Fatalf("records apply = %v", applied) + } + if got := env.resolve(t, "/no-record"); got["action"] != "ok" || got["seo"].(map[string]any)["title"] != "Rule title" { + t.Fatalf("resolve /no-record = %v, want the imported rule", got) + } + + // The admin screen: a stale token is refused, a fresh one applies. + if fields := env.formSchema(t, token, "golem15/seo/transfer"); len(fields) < 2 { + t.Fatalf("transfer form = %v", fields) + } + file := "source;target;status\n/admin1;/pages/a;301\n/e2;/t2-new;302\n" + prev := env.transferAction(t, token, ";", map[string]any{"op": "preview", "dataset": "redirects", "csv": file}) + tok, _ := prev["token"].(string) + if tok == "" || summary(t, prev)["new"] != 1 || summary(t, prev)["changed"] != 1 { + t.Fatalf("admin preview = %v", prev) + } + env.mcpExpect(t, http.StatusCreated, http.MethodPost, "/_seo/api/v1/redirects", agent, map[string]any{"source": "/admin1", "target": "/elsewhere"}) + stale := env.transferAction(t, token, ";", map[string]any{"op": "apply", "token": tok}) + if stale["error"] != "stale" || !strings.Contains(fmt.Sprint(stale["message"]), "The data changed after the preview") { + t.Fatalf("stale apply = %v, want transfer.stale", stale) + } + forged := env.transferAction(t, token, ";", map[string]any{"op": "apply", "token": tok + "x"}) + if forged["error"] == nil { + t.Fatalf("forged token = %v, want a refusal", forged) + } + prev = env.transferAction(t, token, ";", map[string]any{"op": "preview", "dataset": "redirects", "csv": file}) + done := env.transferAction(t, token, ";", map[string]any{"op": "apply", "token": prev["token"]}) + if done["applied"] != true || !strings.Contains(fmt.Sprint(done["_message"]), "Applied 2 changes as one batch") { + t.Fatalf("apply = %v", done) + } + var vias []string + if err := env.db.Raw(`SELECT DISTINCT via FROM golem15_seo_revisions WHERE batch_id = ?`, done["batch_id"]).Scan(&vias).Error; err != nil || len(vias) != 1 || vias[0] != "import" { + t.Fatalf("import batch vias = %v err=%v, want import", vias, err) + } + var origin string + if err := env.db.Raw(`SELECT origin FROM golem15_seo_redirects WHERE source = '/e2'`).Scan(&origin).Error; err != nil || origin != "mcp" { + t.Fatalf("an updated redirect keeps its origin: %q err=%v", origin, err) + } + + // Export through the screen, and the size caps. + out := env.transferAction(t, token, ",", map[string]any{"op": "export", "dataset": "rules"}) + if c := fmt.Sprint(out["csv"]); !strings.HasPrefix(c, bom+"url,match,title") || !strings.Contains(c, "/no-record") { + t.Fatalf("admin export = %v", out) + } + big := "source;target\n" + strings.Repeat("/a-long-source-address;/a-long-target-address\n", 1350) + if tooLarge := env.transferAction(t, token, ";", map[string]any{"op": "preview", "dataset": "redirects", "csv": big}); tooLarge["error"] != "too_large" { + t.Fatalf("admin import over 60 KB = %v, want too_large", tooLarge) + } + huge := "source;target\n" + strings.Repeat("/a-long-source-address;/a-long-target-address\n", 21000) + if rec := env.mcp(t, http.MethodPost, "/_seo/api/v1/import", agent, map[string]any{"dataset": "redirects", "csv": huge, "dry_run": true}); rec.Code != http.StatusRequestEntityTooLarge { + t.Fatalf("MCP import over 900 KB = %d, want 413", rec.Code) + } +}