From 835bf3f74c7f4584eb9854bf20e724593e76b182 Mon Sep 17 00:00:00 2001 From: Jakub Zych Date: Wed, 4 Nov 2026 15:58:44 +0100 Subject: [PATCH] docs(33-10): resolver rule order in the chromedp note --- .planning/notes/2026-10-28-chromedp-rendered-crawl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.planning/notes/2026-10-28-chromedp-rendered-crawl.md b/.planning/notes/2026-10-28-chromedp-rendered-crawl.md index 9cbd6aee..7035921f 100644 --- a/.planning/notes/2026-10-28-chromedp-rendered-crawl.md +++ b/.planning/notes/2026-10-28-chromedp-rendered-crawl.md @@ -19,7 +19,7 @@ The site audit's crawler (`sm-seo-plugin`, package `classes/crawl`) always check Headless Chrome executes untrusted page JavaScript on the server, so it is held to the crawler's `fetchguard` limits: -- **Resolver pinning.** Before Chrome starts, the site host and its `www` twin are resolved and every address is checked with `fetchguard.IsPrivateAddr`; a private or reserved address refuses to start the renderer (render status failed, the note names the address) unless `golem15.seo.crawl_trusted` is true. The checked addresses go into `--host-resolver-rules` as `MAP
` lines after a leading `MAP * ~NOTFOUND`, so every other host resolves to nothing and Chrome cannot re-resolve the site host to another address. +- **Resolver pinning.** Before Chrome starts, the site host and its `www` twin are resolved and every address is checked with `fetchguard.IsPrivateAddr`; a private or reserved address refuses to start the renderer (render status failed, the note names the address) unless `golem15.seo.crawl_trusted` is true. The checked addresses go into `--host-resolver-rules` as `MAP
` lines (one address per host, IPv4 first) followed by a final `MAP * ~NOTFOUND`, so every other host resolves to nothing and Chrome cannot re-resolve the site host to another address. Chrome applies the first rule that matches a host, so the catch-all must come last: with a leading `MAP * ~NOTFOUND` the site host itself fails with `ERR_NAME_NOT_RESOLVED` (checked against Chromium during 33-10; P33-55 had it leading). - **Request interception.** Every request is paused through the DevTools Fetch domain and continued only when it is `https` (`http` only in trusted mode) to the site host or its twin, or a `data:` or `blob:` URL; every other request fails with `BlockedByClient`. Third-party scripts, fonts, analytics and link-local metadata addresses are therefore never loaded. - **No state.** Each crawl uses a fresh temporary profile (`os.MkdirTemp`) removed when the renderer closes; no cookies, Authorization header, HTTP credentials or extensions; downloads denied. The crawler never signs in. - **Limits.** One tab renders at a time, 20 s per page, the DOM read with `OuterHTML` is capped at 5 MiB, and Chrome is closed when the crawl finishes, fails or is cancelled.