fix: root files answer 503 while the backend has not answered

- rootFileTargets gives robots.txt, sitemap.xml and llms.txt their backend
  target also in state unknown without a last answer, so the failing fetch
  answers 503 with Retry-After: 60 instead of the app's 404 (L4 WR-08)
- an older backend (absent) still passes through; sitemap files and
  verification files still need a site answer
- expect-seo-down.txt expects the 503s and states the staging reason

README.md

@@ -29,7 +29,7 @@ The layer's only module is its own local one (`lib/seo-module.ts`); it pulls in
 - **One JSON-LD graph (v0.2.0).** The graph holds the CMS nodes (the answer's `jsonld`) plus the nodes pages and layers add with `useSchemaNode(node)`, merged by `@id`: a node whose `@id` the CMS already has fills in only the keys the CMS node lacks; a new `@id` (or none) is appended. No node at all, no script. unhead escapes `<` inside the JSON, so a value cannot end the script element.
 - **Robots and staging.** A `staging` answer (the backend outside `SUMMER_ENV=production`) makes every page `noindex, nofollow`, known or not. The `X-Robots-Tag` response header follows the same value, or the page's `x_robots_tag`; a redirect carries none.
 - **404 log.** Each page view answered 404 (a GET that accepts `text/html`, from an unknown route or a page that throws a 404) is reported once with `POST <apiBase>/notfound`, `{"url", "referer"}` and `Authorization: Bearer <notfoundToken>`, after the response, so it never delays it. Without a token nothing is reported.
-- **Root files.** With a backend that answers `GET <apiBase>/site` (v0.2.0), `/robots.txt`, `/sitemap.xml`, `/llms.txt` and `/sitemap-<name>.xml` (served from the backend's `/_sitemap/files/`) are proxied to `backendBase`, and so is each one-segment `.html`, `.txt` or `.xml` file the site answer lists in `root_files`: a verification file (`<apiBase>/verification/<name>`) or the IndexNow key file (`/_sitemap/indexnow/<key>.txt`). The backend's status, content type and `X-Robots-Tag` are passed on, with `X-Content-Type-Options: nosniff`; a backend that does not answer or answers 5xx gives 503 with `Retry-After: 60`, so crawlers retry instead of reading a missing robots.txt as "everything allowed". Any other path passes through to the app. The site answer is cached for 300 seconds and cleared by the purge; while the backend is down the last one seen is used. With a backend whose `/site` answers 404 (before v0.2.0, or no SEO plugin), or one not reached since the server started, robots.txt and the sitemaps are not proxied (route them to the backend in the site's proxy) and verification files keep the v0.1.1 lookup: a request for a single-segment `.html`, `.txt` or `.xml` file (`/google0123.html`, `/BingSiteAuth.xml`) is looked up under `GET <apiBase>/verification/<name>` and served with `X-Content-Type-Options: nosniff`, a sandboxing CSP and `X-Robots-Tag: noindex` when the plugin has it; otherwise it falls through to the app's routes. `robots.txt`, `sitemap.xml`, `sitemap.xsl`, `rss.xml`, `feed.xml`, `atom.xml` and `llms.txt` are never looked up that way. `rootFiles: false` turns both off.
+- **Root files.** With a backend that answers `GET <apiBase>/site` (v0.2.0), `/robots.txt`, `/sitemap.xml`, `/llms.txt` and `/sitemap-<name>.xml` (served from the backend's `/_sitemap/files/`) are proxied to `backendBase`, and so is each one-segment `.html`, `.txt` or `.xml` file the site answer lists in `root_files`: a verification file (`<apiBase>/verification/<name>`) or the IndexNow key file (`/_sitemap/indexnow/<key>.txt`). The backend's status, content type and `X-Robots-Tag` are passed on, with `X-Content-Type-Options: nosniff`; a backend that does not answer or answers 5xx gives 503 with `Retry-After: 60`, so crawlers retry instead of reading a missing robots.txt as "everything allowed". Any other path passes through to the app. The site answer is cached for 300 seconds and cleared by the purge; while the backend is down the last one seen is used. While the backend has not answered since the server started, `/robots.txt`, `/sitemap.xml` and `/llms.txt` still go to it and answer 503 with `Retry-After: 60` when it fails, so a crawler retries instead of reading the app's 404 as an allow-all robots.txt (a staging site's `Disallow: /` lives on the backend); sitemap files and verification files need a site answer. With a backend whose `/site` answers 404 (before v0.2.0, or no SEO plugin), robots.txt and the sitemaps are not proxied (route them to the backend in the site's proxy); there, and while the backend has not answered, verification files keep the v0.1.1 lookup: a request for a single-segment `.html`, `.txt` or `.xml` file (`/google0123.html`, `/BingSiteAuth.xml`) is looked up under `GET <apiBase>/verification/<name>` and served with `X-Content-Type-Options: nosniff`, a sandboxing CSP and `X-Robots-Tag: noindex` when the plugin has it; otherwise it falls through to the app's routes. `robots.txt`, `sitemap.xml`, `sitemap.xsl`, `rss.xml`, `feed.xml`, `atom.xml` and `llms.txt` are never looked up that way. `rootFiles: false` turns both off.
 - **Site cache purge (v0.2.0).** `POST /api/__site__/cache/purge` (moved here from vue-journal-layer, vue-menu-layer and vue-faq-layer, which no longer ship a copy since they extend this layer; same contract) removes every Nitro route-cache entry and empties the layer's resolve and site caches, so an admin save shows on the next page view. The backend calls it on loopback after the save commits, with `Authorization: Bearer <cachePurge.token>` and `{"kind","slug"}`; the answer is `{"purged": <n>}`. It answers 404 without a token of at least 32 characters and for any proxied request (X-Forwarded-For, X-Real-IP or Forwarded) unless `cachePurge.allowProxied` is true, and 401 for a missing or wrong bearer. `/api/__site__/**` is never route-cached. While another layer still ships its own copy of the route and that copy answers, `server/plugins/seo-cache-purge.ts` empties the SEO caches after any 200 purge (also after the journal's deprecated `/api/__journal__/cache/purge`).
 - **Build guard (v0.2.0, D-70).** An indexable page must render on the server. `nuxt build` (and `nuxt generate`) fails with `[vue-seo-layer] indexable routes must render on the server (D-70): <routes>` when a route rule sets `ssr: false`, a page is a `.client.vue` page or the app sets `ssr: false`, unless the route is declared in `seo.noindexRoutes` (or the app sets `seo.indexable: false`). The guard reads the rules after every layer and module merged them: `routeRules`, `nitro.routeRules` and the rules of each layer; `experimental.inlineRouteRules` is refused, since `defineRouteRules` in a page could turn server rendering off where the guard cannot see it. `nuxt dev` and `nuxt prepare` only warn.
 - **Noindex routes (v0.2.0, D-23).** A route-rule pattern in `seo.noindexRoutes` (`'/p/**'`, `'/konto'`, `'/x/*'`) gets `X-Robots-Tag: noindex, nofollow` through its route rule (unless the app set its own header there) and a `robots` meta tag. A page whose meta sets `requiresAuth: true` gets `noindex, nofollow`, and one whose meta sets `seo: { robots: '…' }` gets that value, as meta tag and, during SSR, header, without any SEO code in the page; this is how apps mark signed-in and auth pages. These values win over the CMS's, except on staging. No other page is ever marked noindex by the layer.

server/lib/site.ts

@@ -9,8 +9,9 @@
 //            files looked up by name, robots.txt and the sitemaps left to
 //            the site's proxy)
 //   unknown  the backend did not answer: the last ok answer is used when
-//            there is one, else the layer behaves as with absent (as
-//            v0.1.1 did with the backend down); nothing is cached
+//            there is one; else robots.txt, sitemap.xml and llms.txt still
+//            go to the backend (whose failure is a 503 with Retry-After) and
+//            every other path behaves as with absent; nothing is cached
 //
 // Nitro auto-imports ($fetch, useRuntimeConfig) are used inside the
 // functions only, so `node --test` can import this file.
@@ -79,18 +80,23 @@ export interface RootTarget {
 /**
  * Where a root file request goes on the backend, in the order to try (the
  * first answer that is not 404 is served), or null when the layer does not
- * serve the path. Only with a site answer (state ok, possibly the last one
+ * serve the path. With a site answer (state ok, possibly the last one
  * seen): robots.txt, sitemap.xml, llms.txt and sitemap-<name>.xml always,
  * any other one-segment .html, .txt or .xml file when root_files lists it
- * (a verification file, then the IndexNow key file). Without one (an older
- * backend, or a backend never reached since start) every path passes
- * through, as in v0.1.1. Nothing from the request but the matched name
- * reaches the URL.
+ * (a verification file, then the IndexNow key file). With a backend never
+ * reached since start (unknown, no last answer): robots.txt, sitemap.xml
+ * and llms.txt only, so the failing fetch answers 503 with Retry-After and
+ * a crawler never reads the app's 404 as an allow-all robots.txt (a staging
+ * site's Disallow lives on the backend, L4 WR-08); every other path passes
+ * through. With an older backend (absent, the site answer is 404) every
+ * path passes through, as in v0.1.1. Nothing from the request but the
+ * matched name reaches the URL.
  */
 export function rootFileTargets(path: string, site: SeoSiteState, backendBase: string, apiBase: string): RootTarget[] | null {
-  if (site.state !== 'ok') return null
+  if (site.state === 'absent') return null
   const pathname = path.split('?')[0] ?? ''
   if (FIXED.has(pathname)) return [{ kind: 'root', url: `${backendBase}${pathname}` }]
+  if (site.state !== 'ok') return null
   const sitemap = SITEMAP_FILE.exec(pathname)
   if (sitemap) return [{ kind: 'sitemap', url: `${backendBase}/_sitemap/files/${sitemap[1]}` }]
   const file = ROOT_FILE.exec(pathname)

server/middleware/seo-root.ts

@@ -7,12 +7,16 @@
 // of reading a missing robots.txt as "everything allowed". Any other path
 // passes through.
 //
-// Only once the backend has given its site answer (SEO v0.2.0; the last
-// one seen is kept while the backend is down). With a backend before v0.2.0
-// (the site answer is 404), or one not reached since the server started,
-// nothing is proxied here, as in v0.1.1: robots.txt and the sitemaps stay
-// with the site's proxy (or fall through to the app) and verification
-// files keep the v0.1.1 lookup (seo-verification.ts). Off with
+// Once the backend has given its site answer (SEO v0.2.0; the last one
+// seen is kept while the backend is down) every file above is proxied.
+// While the backend has not answered since the server started, only
+// robots.txt, sitemap.xml and llms.txt are: the failing fetch answers 503
+// with Retry-After, so a staging site never exposes the app's 404 as an
+// allow-all robots.txt (L4 WR-08). With a backend before v0.2.0 (the site
+// answer is 404) nothing is proxied here, as in v0.1.1: robots.txt and the
+// sitemaps stay with the site's proxy (or fall through to the app); there
+// and while the backend is unknown, verification files keep the v0.1.1
+// lookup (seo-verification.ts). Off with
 // the module option seo.rootFiles false or the layer switched off. The
 // backend URLs come from the server-only bases and an allow-listed name only
 // (rootFileTargets), never from the request.

testing/expect-seo-down.txt

@@ -1,10 +1,15 @@
 # The CMS unreachable since the server started (NUXT_SEO_API_BASE on a
-# closed port): pages render normally with their own head (fail open) and,
-# with no site answer ever seen, root files fall through to the app as in
-# v0.1.1 (a backend failure after a site answer is 503, expect-seo.txt).
+# closed port): pages render normally with their own head (fail open).
+# robots.txt, sitemap.xml and llms.txt answer 503 with Retry-After, so a
+# crawler retries: falling through to the app would serve a 404 that reads
+# as an allow-all robots.txt, also on a staging site whose Disallow lives
+# on the backend (L4 WR-08). With no site answer ever seen, sitemap files
+# and verification files still fall through to the app as in v0.1.1.
 200 /about contains=<title>Own about | Playground</title> notcontains=About us | Acme noheader=X-Robots-Tag count=application/ld+json:0
 200 /plain contains=<title>Plain page | Playground</title>
-404 /robots.txt noheader=Retry-After
+503 /robots.txt header=Retry-After: 60
+503 /sitemap.xml header=Retry-After: 60
+503 /llms.txt header=Retry-After: 60
 404 /sitemap-posts-1.xml
 404 /google0123.html
 # The HTML sitemap with the backend down is a 503 (never an empty page).

tests/rootFiles.test.ts

@@ -15,11 +15,23 @@ test('rootFileTargets: robots.txt, sitemap.xml, llms.txt and sitemap files go to
   assert.deepEqual(rootFileTargets('/sitemap-posts-1.xml', ok, B, A), [{ kind: 'sitemap', url: `${B}/_sitemap/files/sitemap-posts-1.xml` }])
 })
 
-test('rootFileTargets: without a site answer (older backend, or never reached) nothing is served, as in v0.1.1', () => {
-  for (const state of [{ state: 'absent' }, { state: 'unknown' }] as SeoSiteState[]) {
-    for (const p of ['/robots.txt', '/sitemap.xml', '/llms.txt', '/sitemap-posts-1.xml', '/google1234.html']) {
-      assert.equal(rootFileTargets(p, state, B, A), null, `${state.state} ${p}`)
-    }
+test('rootFileTargets: an older backend (absent) passes every path through, as in v0.1.1', () => {
+  const absent: SeoSiteState = { state: 'absent' }
+  for (const p of ['/robots.txt', '/sitemap.xml', '/llms.txt', '/sitemap-posts-1.xml', '/google1234.html']) {
+    assert.equal(rootFileTargets(p, absent, B, A), null, p)
+  }
+})
+
+test('rootFileTargets: a backend never reached since start (unknown) still owns the fixed root files (L4 WR-08)', () => {
+  // The fetch then fails and the middleware answers 503 with Retry-After,
+  // so a crawler never reads the app's 404 as an allow-all robots.txt.
+  const unknown: SeoSiteState = { state: 'unknown' }
+  assert.deepEqual(rootFileTargets('/robots.txt', unknown, B, A), [{ kind: 'root', url: `${B}/robots.txt` }])
+  assert.deepEqual(rootFileTargets('/sitemap.xml?x=1', unknown, B, A), [{ kind: 'root', url: `${B}/sitemap.xml` }])
+  assert.deepEqual(rootFileTargets('/llms.txt', unknown, B, A), [{ kind: 'root', url: `${B}/llms.txt` }])
+  // sitemap files and verification files still need a site answer
+  for (const p of ['/sitemap-posts-1.xml', '/google1234.html', `/${KEY}`, '/random.txt']) {
+    assert.equal(rootFileTargets(p, unknown, B, A), null, p)
   }
 })