Commit Graph

3 Commits

Author SHA1 Message Date
Jakub Zych
3fdb44bce2 fix: the crawler and the Chrome guard keep to the site's port
Fetcher.Allowed and Key, the rendered comparison and the Chrome request
guard accept an address only on the site's effective port (explicit,
else 443 for https and 80 for http), so page code cannot make headless
Chrome reach other services on the site host's address and a link to
another port is not reported as a page of the site.
2026-11-04 22:56:02 +01:00
Jakub Zych
cfeb3dff12 fix: bound opening a Chrome tab
Render bounds opening the tab and rendering the page with one deadline
(TabOpenTimeout 5 s plus RenderTimeout 20 s), so a hung Chrome no longer
holds the crawl's render lock until the job timeout.
2026-11-04 22:53:40 +01:00
Jakub Zych
b553fc2937 feat: rendered comparison through headless Chrome
- github.com/chromedp/chromedp v0.21.0 (decision note
  2026-10-28-chromedp-rendered-crawl.md): with golem15.seo.crawl_chrome_path
  the crawl renders up to golem15.seo.crawl_render_max pages (200) and
  compares them with the raw HTML; without a path it records skipped
- guard: site host and www twin resolved and checked with
  fetchguard.IsPrivateAddr, pinned through --host-resolver-rules with a
  final MAP * ~NOTFOUND; Fetch-domain interception fails every request
  outside the site (third-party scripts never load); fresh temporary
  profile removed on Close; downloads denied; one tab, 20 s, 5 MiB DOM
- a relative or non-executable path, a private address or a Chrome that
  does not start records render_status failed; the crawl still finishes
- config keys documented; README Rendered comparison section
- TestChromeRenderer (runs with a local Chromium) and TestRendererGuards
2026-11-04 15:58:36 +01:00