Commit Graph

8 Commits

Author SHA1 Message Date
Jakub Zych
49cc241f58 feat: internal links widget with counts and suggestions
- E10 golem15-seo-links in the SEO tab after the graph preview (types
  with a body): inbound and outbound counts from the last finished crawl,
  the orphan warning, up to 10 pages to link to (Copy address) and 10
  that could link here
- classes/linkgraph.go builds the report from the crawl's pages and
  links; the crawler's orphan suggestions share NormalText
- GET /_seo/api/v1/links and seo_get_link_suggestions; widget action
  seo_links mapped in AdminOperations
- element bundle rebuilt
2026-11-04 16:40:16 +01:00
Jakub Zych
b553fc2937 feat: rendered comparison through headless Chrome
- github.com/chromedp/chromedp v0.21.0 (decision note
  2026-10-28-chromedp-rendered-crawl.md): with golem15.seo.crawl_chrome_path
  the crawl renders up to golem15.seo.crawl_render_max pages (200) and
  compares them with the raw HTML; without a path it records skipped
- guard: site host and www twin resolved and checked with
  fetchguard.IsPrivateAddr, pinned through --host-resolver-rules with a
  final MAP * ~NOTFOUND; Fetch-domain interception fails every request
  outside the site (third-party scripts never load); fresh temporary
  profile removed on Close; downloads denied; one tab, 20 s, 5 MiB DOM
- a relative or non-executable path, a private address or a Chrome that
  does not start records render_status failed; the crawl still finishes
- config keys documented; README Rendered comparison section
- TestChromeRenderer (runs with a local Chromium) and TestRendererGuards
2026-11-04 15:58:36 +01:00
Jakub Zych
fc53d93b9d feat: site audit settings and the nightly crawl
- Site audit tab (UI-SPEC C20): crawl_schedule, crawl_max_pages
  (10-50 000), crawl_delay (100-5000 ms), crawl_keep (1-50) and
  generic_anchors seeded with the en and pl lists; ranges refuse with
  lagoon::validation.between on the screen and through the API
- the crawler reads its limits and generic anchors from the settings;
  crawls beyond crawl_keep are deleted with their pages and issues
- seo:crawl [--scheduled] and the nightly schedule entry at 02:30
  (P33-40): off does nothing, weekly runs on Mondays in app.timezone
- TestCrawlSchedule
2026-11-04 15:41:54 +01:00
Jakub Zych
c4f0b4f94c feat: Site audit and Crawls screens with their MCP tools
- golem15.seo.issues (Site audit): current issues errors first, labels,
  summary and fix columns, filters, P7 issue_details, Check this address
  again, Ignore, Stop ignoring, bulk Ignore, ignored rows badged
- golem15.seo.crawls (Crawls): newest first, P6 crawl_summary with one
  status callout, staging and rendered-comparison lines, stats strip and
  per-kind table; Cancel; a running crawl refuses delete
- Start crawl toolbar action on both, refusing already_running and
  no_site_url; navigation Site audit (gauge) and Crawls (globe)
- routes and MCP tools start/cancel/list/get/delete crawls, list_issues
  with filters (open by default), recheck_url, ignore and unignore
- AdminOperations maps both controllers; toolbar actions get their own
  operation kind
- TestCrawlAdmin, TestCrawlTools, TestIssueKindLabels
2026-11-04 15:34:22 +01:00
Jakub Zych
edc03cae4b feat: orphan pages and merged CMS issues with fix routes
- orphan_page for listed indexable pages no crawled page links to, with
  pages mentioning the title or focus keyphrase as suggestions (D-82)
- CMS-data image_alt and h1_in_content from the content descriptors
  (TopHeading 2 only), stored with the crawl as source cms (D-85, D-50)
- seo-page names Tools edit seo_update_page; every stored issue carries
  a tool or fix code with details.what (P33-38)
- TestIssueMerge (ignored fingerprints survive the next crawl, fixed
  pages lose their issues) and TestStagingCrawl (Pitfall 12)
2026-11-04 15:18:01 +01:00
Jakub Zych
8c632331cc feat: crawler checks for JSON-LD, hreflang, links, images, raw HTML and the rendered-comparison seam
- jsonld_invalid, jsonld_multiple, and jsonld_required/recommended from
  schema.Validate (the E9 rules); publisher nodes route to settings
- hreflang_error and hreflang_reciprocal over the fetched alternates
- broken_link and redirected_link with the linking pages, generic_anchor
  against the en and pl seed lists
- image_alt, image_size and featured_lazy
- content_js check cms: the record title in the H1, half its words and
  its primary parent's link in the raw HTML
- compareRendered (pure): links, an H1 or text only after JavaScript as
  content_js check rendered; the Renderer seam and render bookkeeping
- TestRenderSkipped: skipped without a renderer, failed with its note
2026-11-04 15:13:09 +01:00
Jakub Zych
f2c93aeabe feat: crawler checks for status, sitemaps, canonical, titles and headings
- page_status, sitemap_error, sitemap_noindex, sitemap_blocked and
  sitemap_missing against the fetched sitemap files and robots.txt
- canonical_missing and canonical_error, missing and duplicate titles and
  descriptions over the crawl
- h1_missing, h1_multiple, heading_skipped, heading_empty and
  h1_title_mismatch; content headings route to the record's tool
- the staging lock's noindex is never counted (Pitfall 12)
- TestCrawlerChecks on the planted site with a clean control page
2026-11-04 15:10:22 +01:00
Jakub Zych
c2b61bf8e9 feat: crawler finds a page with two H1 headings and stores the issue (tracer)
- golem15_seo_crawls, crawl pages, links, issues and ignores (P33-37)
- classes/crawl: fetchguard client on the site host and its www twin,
  sitemap index and robots.txt seeds, two workers with a delay, limits
- h1_missing and h1_multiple with the P33-38 fix route
- POST /_seo/api/v1/crawls queues the golem15.seo.crawl job; GET /issues
  answers the stored issues with id, severity, source, fix, found_at,
  ignored and crawl_id
2026-11-04 15:05:45 +01:00