Commit Graph

6 Commits

Author SHA1 Message Date
Jakub Zych
3fdb44bce2 fix: the crawler and the Chrome guard keep to the site's port
Fetcher.Allowed and Key, the rendered comparison and the Chrome request
guard accept an address only on the site's effective port (explicit,
else 443 for https and 80 for http), so page code cannot make headless
Chrome reach other services on the site host's address and a link to
another port is not reported as a page of the site.
2026-11-04 22:56:02 +01:00
Jakub Zych
9de6b2ec83 feat: noindex paths from the route report leave the sitemaps (tracer)
The route report takes an optional noindex list of static paths. When
the key is present the stored set (golem15_seo_noindex_paths) is
replaced, and the sitemap filter leaves those paths out with the reason
noindex; a changed set asks the sitemap for a debounced rebuild through
contract.Rebuilder. A listed noindex address without a CMS record now
names sitemap_update_settings, or fix: code without the sitemap plugin.
2026-11-04 22:44:26 +01:00
Jakub Zych
edc03cae4b feat: orphan pages and merged CMS issues with fix routes
- orphan_page for listed indexable pages no crawled page links to, with
  pages mentioning the title or focus keyphrase as suggestions (D-82)
- CMS-data image_alt and h1_in_content from the content descriptors
  (TopHeading 2 only), stored with the crawl as source cms (D-85, D-50)
- seo-page names Tools edit seo_update_page; every stored issue carries
  a tool or fix code with details.what (P33-38)
- TestIssueMerge (ignored fingerprints survive the next crawl, fixed
  pages lose their issues) and TestStagingCrawl (Pitfall 12)
2026-11-04 15:18:01 +01:00
Jakub Zych
8c632331cc feat: crawler checks for JSON-LD, hreflang, links, images, raw HTML and the rendered-comparison seam
- jsonld_invalid, jsonld_multiple, and jsonld_required/recommended from
  schema.Validate (the E9 rules); publisher nodes route to settings
- hreflang_error and hreflang_reciprocal over the fetched alternates
- broken_link and redirected_link with the linking pages, generic_anchor
  against the en and pl seed lists
- image_alt, image_size and featured_lazy
- content_js check cms: the record title in the H1, half its words and
  its primary parent's link in the raw HTML
- compareRendered (pure): links, an H1 or text only after JavaScript as
  content_js check rendered; the Renderer seam and render bookkeeping
- TestRenderSkipped: skipped without a renderer, failed with its note
2026-11-04 15:13:09 +01:00
Jakub Zych
f2c93aeabe feat: crawler checks for status, sitemaps, canonical, titles and headings
- page_status, sitemap_error, sitemap_noindex, sitemap_blocked and
  sitemap_missing against the fetched sitemap files and robots.txt
- canonical_missing and canonical_error, missing and duplicate titles and
  descriptions over the crawl
- h1_missing, h1_multiple, heading_skipped, heading_empty and
  h1_title_mismatch; content headings route to the record's tool
- the staging lock's noindex is never counted (Pitfall 12)
- TestCrawlerChecks on the planted site with a clean control page
2026-11-04 15:10:22 +01:00
Jakub Zych
c2b61bf8e9 feat: crawler finds a page with two H1 headings and stores the issue (tracer)
- golem15_seo_crawls, crawl pages, links, issues and ignores (P33-37)
- classes/crawl: fetchguard client on the site host and its www twin,
  sitemap index and robots.txt seeds, two workers with a delay, limits
- h1_missing and h1_multiple with the P33-38 fix route
- POST /_seo/api/v1/crawls queues the golem15.seo.crawl job; GET /issues
  answers the stored issues with id, severity, source, fix, found_at,
  ignored and crawl_id
2026-11-04 15:05:45 +01:00