golem15/summercms
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(33-10): resolver rule order in the chromedp note

Browse Source
This commit is contained in:
Jakub Zych
2026-11-04 15:58:44 +01:00
parent e9504b1ece
commit 835bf3f74c
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.planning/notes/2026-10-28-chromedp-rendered-crawl.md
View File

@@ -19,7 +19,7 @@ The site audit's crawler (`sm-seo-plugin`, package `classes/crawl`) always check
Headless Chrome executes untrusted page JavaScript on the server, so it is held to the crawler's `fetchguard` limits:
- **Resolver pinning.** Before Chrome starts, the site host and its `www` twin are resolved and every address is checked with `fetchguard.IsPrivateAddr`; a private or reserved address refuses to start the renderer (render status failed, the note names the address) unless `golem15.seo.crawl_trusted` is true. The checked addresses go into `--host-resolver-rules` as `MAP <host> <address>` lines after a leading `MAP * ~NOTFOUND`, so every other host resolves to nothing and Chrome cannot re-resolve the site host to another address.
- **Resolver pinning.** Before Chrome starts, the site host and its `www` twin are resolved and every address is checked with `fetchguard.IsPrivateAddr`; a private or reserved address refuses to start the renderer (render status failed, the note names the address) unless `golem15.seo.crawl_trusted` is true. The checked addresses go into `--host-resolver-rules` as `MAP <host> <address>` lines (one address per host, IPv4 first) followed by a final `MAP * ~NOTFOUND`, so every other host resolves to nothing and Chrome cannot re-resolve the site host to another address. Chrome applies the first rule that matches a host, so the catch-all must come last: with a leading `MAP * ~NOTFOUND` the site host itself fails with `ERR_NAME_NOT_RESOLVED` (checked against Chromium during 33-10; P33-55 had it leading).
- **Request interception.** Every request is paused through the DevTools Fetch domain and continued only when it is `https` (`http` only in trusted mode) to the site host or its twin, or a `data:` or `blob:` URL; every other request fails with `BlockedByClient`. Third-party scripts, fonts, analytics and link-local metadata addresses are therefore never loaded.
- **No state.** Each crawl uses a fresh temporary profile (`os.MkdirTemp`) removed when the renderer closes; no cookies, Authorization header, HTTP credentials or extensions; downloads denied. The crawler never signs in.
- **Limits.** One tab renders at a time, 20 s per page, the DOM read with `OuterHTML` is capped at 5 MiB, and Chrome is closed when the crawl finishes, fails or is cancelled.