golem15/vue-seo-layer
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: root files follow the last site answer, a 404 included, while the backend is down

Browse Source 
An SEO-less site's robots.txt, sitemap.xml and llms.txt keep reaching
the app during an outage longer than the cache TTL; a backend that never
answered still gets the 503.

quick-261104-xae, review WR-04.
This commit is contained in:
Jakub Zych
2026-11-05 00:23:19 +01:00
parent 973142a714
commit b69647fecd
4 changed files with 125 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -29,7 +29,7 @@ The layer's only module is its own local one (`lib/seo-module.ts`); it pulls in
- **One JSON-LD graph (v0.2.0).** The graph holds the CMS nodes (the answer's `jsonld`) plus the nodes pages and layers add with `useSchemaNode(node)`, merged by `@id`: a node whose `@id` the CMS already has fills in only the keys the CMS node lacks; a new `@id` (or none) is appended. No node at all, no script. unhead escapes `<` inside the JSON, so a value cannot end the script element.
- **Robots and staging.** A `staging` answer (the backend outside `SUMMER_ENV=production`) makes every page `noindex, nofollow`, known or not. The `X-Robots-Tag` response header follows the same value, or the page's `x_robots_tag`; a redirect carries none.
- **404 log.** Each page view answered 404 (a GET that accepts `text/html`, from an unknown route or a page that throws a 404) is reported once with `POST <apiBase>/notfound`, `{"url", "referer"}` and `Authorization: Bearer <notfoundToken>`, after the response, so it never delays it. Without a token nothing is reported.
- **Root files.** With a backend that answers `GET <apiBase>/site` (v0.2.0), `/robots.txt`, `/sitemap.xml`, `/llms.txt` and `/sitemap-<name>.xml` (served from the backend's `/_sitemap/files/`) are proxied to `backendBase`, and so is each one-segment `.html`, `.txt` or `.xml` file the site answer lists in `root_files`: a verification file (`<apiBase>/verification/<name>`) or the IndexNow key file (`/_sitemap/indexnow/<key>.txt`). The backend's status, content type and `X-Robots-Tag` are passed on, with `X-Content-Type-Options: nosniff`; a backend that does not answer or answers 5xx gives 503 with `Retry-After: 60`, so crawlers retry instead of reading a missing robots.txt as "everything allowed". Any other path passes through to the app. The site answer is cached for 300 seconds and cleared by the purge; while the backend is down the last one seen is used. While the backend has not answered since the server started, `/robots.txt`, `/sitemap.xml` and `/llms.txt` still go to it and answer 503 with `Retry-After: 60` when it fails, so a crawler retries instead of reading the app's 404 as an allow-all robots.txt (a staging site's `Disallow: /` lives on the backend); sitemap files and verification files need a site answer. With a backend whose `/site` answers 404 (before v0.2.0, or no SEO plugin), robots.txt and the sitemaps are not proxied (route them to the backend in the site's proxy); there, and while the backend has not answered, verification files keep the v0.1.1 lookup: a request for a single-segment `.html`, `.txt` or `.xml` file (`/google0123.html`, `/BingSiteAuth.xml`) is looked up under `GET <apiBase>/verification/<name>` and served with `X-Content-Type-Options: nosniff`, a sandboxing CSP and `X-Robots-Tag: noindex` when the plugin has it; otherwise it falls through to the app's routes. `robots.txt`, `sitemap.xml`, `sitemap.xsl`, `rss.xml`, `feed.xml`, `atom.xml` and `llms.txt` are never looked up that way. `rootFiles: false` turns both off.
- **Root files.** With a backend that answers `GET <apiBase>/site` (v0.2.0), `/robots.txt`, `/sitemap.xml`, `/llms.txt` and `/sitemap-<name>.xml` (served from the backend's `/_sitemap/files/`) are proxied to `backendBase`, and so is each one-segment `.html`, `.txt` or `.xml` file the site answer lists in `root_files`: a verification file (`<apiBase>/verification/<name>`) or the IndexNow key file (`/_sitemap/indexnow/<key>.txt`). The backend's status, content type and `X-Robots-Tag` are passed on, with `X-Content-Type-Options: nosniff`; a backend that does not answer or answers 5xx gives 503 with `Retry-After: 60`, so crawlers retry instead of reading a missing robots.txt as "everything allowed". Any other path passes through to the app. The site answer is cached for 300 seconds and cleared by the purge; while the backend is down the last answer it gave is used, a 404 included, so on a site whose backend has no SEO plugin robots.txt and the sitemaps keep coming from the app during an outage. While the backend has not answered since the server started or since the last purge, `/robots.txt`, `/sitemap.xml` and `/llms.txt` still go to it and answer 503 with `Retry-After: 60` when it fails, so a crawler retries instead of reading the app's 404 as an allow-all robots.txt (a staging site's `Disallow: /` lives on the backend); sitemap files and verification files need a site answer. With a backend whose `/site` answers 404 (before v0.2.0, or no SEO plugin), robots.txt and the sitemaps are not proxied (route them to the backend in the site's proxy); there, and while the backend has not answered, verification files keep the v0.1.1 lookup: a request for a single-segment `.html`, `.txt` or `.xml` file (`/google0123.html`, `/BingSiteAuth.xml`) is looked up under `GET <apiBase>/verification/<name>` and served with `X-Content-Type-Options: nosniff`, a sandboxing CSP and `X-Robots-Tag: noindex` when the plugin has it; otherwise it falls through to the app's routes. `robots.txt`, `sitemap.xml`, `sitemap.xsl`, `rss.xml`, `feed.xml`, `atom.xml` and `llms.txt` are never looked up that way. `rootFiles: false` turns both off.
- **Site cache purge (v0.2.0).** `POST /api/__site__/cache/purge` (moved here from vue-journal-layer, vue-menu-layer and vue-faq-layer, which no longer ship a copy since they extend this layer; same contract) removes every Nitro route-cache entry and empties the layer's resolve and site caches, so an admin save shows on the next page view. The backend calls it on loopback after the save commits, with `Authorization: Bearer <cachePurge.token>` and `{"kind","slug"}`; the answer is `{"purged": <n>}`. It answers 404 without a token of at least 32 characters and for any proxied request (X-Forwarded-For, X-Real-IP or Forwarded) unless `cachePurge.allowProxied` is true, and 401 for a missing or wrong bearer. `/api/__site__/**` is never route-cached. While another layer still ships its own copy of the route and that copy answers, `server/plugins/seo-cache-purge.ts` empties the SEO caches after any 200 purge (also after the journal's deprecated `/api/__journal__/cache/purge`).
- **Build guard (v0.2.0, D-70).** An indexable page must render on the server. `nuxt build` (and `nuxt generate`) fails with `[vue-seo-layer] indexable routes must render on the server (D-70): <routes>` when a route rule sets `ssr: false`, a page is a `.client.vue` page or the app sets `ssr: false`, unless the route is declared in `seo.noindexRoutes` (or the app sets `seo.indexable: false`). The guard reads the rules after every layer and module merged them: `routeRules`, `nitro.routeRules` and the rules of each layer; `experimental.inlineRouteRules` is refused, since `defineRouteRules` in a page could turn server rendering off where the guard cannot see it. `nuxt dev` and `nuxt prepare` only warn.
- **Noindex routes (v0.2.0, D-23).** A route-rule pattern in `seo.noindexRoutes` (`'/p/**'`, `'/konto'`, `'/x/*'`) gets `X-Robots-Tag: noindex, nofollow` through its route rule (unless the app set its own header there) and a `robots` meta tag. A page whose meta sets `requiresAuth: true` gets `noindex, nofollow`, and one whose meta sets `seo: { robots: '…' }` gets that value, as meta tag and, during SSR, header, without any SEO code in the page; this is how apps mark signed-in and auth pages. These values win over the CMS's, except on staging. No other page is ever marked noindex by the layer.

32
server/lib/site.ts
View File

@@ -8,10 +8,12 @@
// v0.2.0): the layer keeps its v0.1.1 behaviour (verification
// files looked up by name, robots.txt and the sitemaps left to
// the site's proxy)
// unknown the backend did not answer: the last ok answer is used when
// there is one; else robots.txt, sitemap.xml and llms.txt still
// go to the backend (whose failure is a 503 with Retry-After) and
// every other path behaves as with absent; nothing is cached
// unknown the backend did not answer, or answered something other than
// a 200 object or a 404: the last definite answer (ok or absent)
// seen since the server started or the last purge is then used;
// with none, robots.txt, sitemap.xml and llms.txt still go to the
// backend (a failure is a 503 with Retry-After) and every other
// path behaves as with absent; nothing is cached
//
// Nitro auto-imports ($fetch, useRuntimeConfig) are used inside the
// functions only, so `node --test` can import this file.
@@ -24,13 +26,15 @@ export const SITE_CACHE_TTL_MS = 300_000
export type SeoSiteState = { state: 'ok'; site: SeoSiteAnswer } | { state: 'absent' } | { state: 'unknown' }
const cache = createLru<SeoSiteState>({ max: 1, ttlMs: SITE_CACHE_TTL_MS })
let lastOk: SeoSiteState | null = null
// The clock is read at call time, so a test can move Date.now.
const cache = createLru<SeoSiteState>({ max: 1, ttlMs: SITE_CACHE_TTL_MS, now: () => Date.now() })
/** The last definite answer (ok or absent), used while the backend is unknown. */
let lastKnown: SeoSiteState | null = null
/** Empties the site cache (the purge calls it through clearSeoCaches). */
export function clearSiteCache(): void {
cache.clear()
lastOk = null
lastKnown = null
}
export function siteCacheSize(): number {
@@ -58,8 +62,8 @@ export async function siteAnswer(cfg: SeoServerConfig, headers: Record<string, s
},
(s) => s.state !== 'unknown',
)
if (got.state === 'ok') lastOk = got
if (got.state === 'unknown' && lastOk) return lastOk
if (got.state !== 'unknown') lastKnown = got
else if (lastKnown) return lastKnown
return got
}
@@ -80,16 +84,18 @@ export interface RootTarget {
/**
* Where a root file request goes on the backend, in the order to try (the
* first answer that is not 404 is served), or null when the layer does not
* serve the path. With a site answer (state ok, possibly the last one
* seen): robots.txt, sitemap.xml, llms.txt and sitemap-<name>.xml always,
* serve the path. With a site answer (state ok, possibly the remembered
* one while the backend is down): robots.txt, sitemap.xml, llms.txt and sitemap-<name>.xml always,
* any other one-segment .html, .txt or .xml file when root_files lists it
* (a verification file, then the IndexNow key file). With a backend never
* reached since start (unknown, no last answer): robots.txt, sitemap.xml
* and llms.txt only, so the failing fetch answers 503 with Retry-After and
* a crawler never reads the app's 404 as an allow-all robots.txt (a staging
* site's Disallow lives on the backend, L4 WR-08); every other path passes
* through. With an older backend (absent, the site answer is 404) every
* path passes through, as in v0.1.1. Nothing from the request but the
* through. With an older backend or one without the SEO plugin (absent,
* the site answer is 404, possibly the remembered one while the backend is
* down) every path passes through, as in v0.1.1. Unknown here means the
* backend has not answered since start or the last purge. Nothing from the request but the
* matched name reaches the URL.
*/
export function rootFileTargets(path: string, site: SeoSiteState, backendBase: string, apiBase: string): RootTarget[] | null {

10
server/middleware/seo-root.ts
View File

@@ -7,10 +7,12 @@
// of reading a missing robots.txt as "everything allowed". Any other path
// passes through.
//
// Once the backend has given its site answer (SEO v0.2.0; the last one
// seen is kept while the backend is down) every file above is proxied.
// While the backend has not answered since the server started, only
// robots.txt, sitemap.xml and llms.txt are: the failing fetch answers 503
// Once the backend has given its site answer (SEO v0.2.0) every file above
// is proxied. The last answer, ok or absent, is kept while the backend is
// down, so on a backend without the SEO plugin the files keep reaching the
// app during an outage. Only while the backend has not answered since the
// server started or the last purge are robots.txt, sitemap.xml and
// llms.txt proxied by their fixed names: the failing fetch answers 503
// with Retry-After, so a staging site never exposes the app's 404 as an
// allow-all robots.txt (L4 WR-08). With a backend before v0.2.0 (the site
// answer is 404) nothing is proxied here, as in v0.1.1: robots.txt and the

101
tests/rootFiles.test.ts
View File

@@ -1,6 +1,6 @@
import assert from 'node:assert/strict'
import test from 'node:test'
import { rootFileTargets, siteState, type SeoSiteState } from '../server/lib/site.ts'
import test, { type TestContext } from 'node:test'
import { clearSiteCache, rootFileTargets, siteAnswer, siteState, SITE_CACHE_TTL_MS, type SeoSiteState } from '../server/lib/site.ts'
import { seoBackendBase, seoServerConfig } from '../server/lib/config.ts'
const B = 'http://127.0.0.1:8096'
@@ -84,3 +84,100 @@ test('seoServerConfig: defaults, aliases and tokens', () => {
assert.equal(seoServerConfig({ seo: { notfoundToken: 't' } }).routeReportToken, 't')
assert.equal(seoServerConfig({ seo: { notfoundToken: 't', routeReportToken: 'r' } }).routeReportToken, 'r')
})
// siteAnswer with a stubbed Nitro $fetch and a moved clock. Each answer is
// a {status, _data} pair or an Error the fetch throws.
type Reply = { status: number; _data: unknown } | Error
function stubSite(t: TestContext) {
const g = globalThis as Record<string, unknown>
const prev = g.$fetch
const replies: Reply[] = []
const state = { calls: 0, clock: 1_000_000 }
g.$fetch = {
raw: async () => {
state.calls++
const r = replies.shift()
if (!r) throw new Error('no reply queued')
if (r instanceof Error) throw r
return r
},
}
t.mock.method(Date, 'now', () => state.clock)
clearSiteCache()
const ask = async (r: Reply): Promise<SeoSiteState> => {
replies.push(r)
return siteAnswer(seoServerConfig({}), {})
}
const expire = () => {
state.clock += SITE_CACHE_TTL_MS + 1
}
const restore = () => {
clearSiteCache()
g.$fetch = prev
}
return { state, ask, expire, restore }
}
const down = () => new Error('connect ECONNREFUSED')
test('siteAnswer: a remembered absent outlives the cache while the backend is down (WR-04)', async (t) => {
const s = stubSite(t)
try {
assert.equal((await s.ask({ status: 404, _data: { error: 'not_found' } })).state, 'absent')
s.expire()
for (const r of [down(), { status: 500, _data: {} }, { status: 200, _data: '<html>' }, { status: 401, _data: {} }] as Reply[]) {
const before = s.state.calls
const got = await s.ask(r)
assert.equal(got.state, 'absent', String(r instanceof Error ? r.message : r.status))
assert.equal(s.state.calls, before + 1, 'each call fetches again')
for (const p of ['/robots.txt', '/sitemap.xml', '/llms.txt']) assert.equal(rootFileTargets(p, got, B, A), null, p)
}
} finally {
s.restore()
}
})
test('siteAnswer: the latest definite answer wins (WR-04)', async (t) => {
const s = stubSite(t)
try {
assert.equal((await s.ask({ status: 200, _data: { root_files: [] } })).state, 'ok')
s.expire()
assert.equal((await s.ask({ status: 404, _data: {} })).state, 'absent')
s.expire()
assert.equal((await s.ask(down())).state, 'absent')
s.expire()
assert.equal((await s.ask({ status: 200, _data: { root_files: [] } })).state, 'ok')
s.expire()
const got = await s.ask(down())
assert.equal(got.state, 'ok')
assert.deepEqual(rootFileTargets('/robots.txt', got, B, A), [{ kind: 'root', url: `${B}/robots.txt` }])
} finally {
s.restore()
}
})
test('siteAnswer: a backend that never answered stays unknown and owns the fixed root files', async (t) => {
const s = stubSite(t)
try {
const got = await s.ask(down())
assert.equal(got.state, 'unknown')
assert.deepEqual(rootFileTargets('/robots.txt', got, B, A), [{ kind: 'root', url: `${B}/robots.txt` }])
const before = s.state.calls
assert.equal((await s.ask(down())).state, 'unknown')
assert.equal(s.state.calls, before + 1, 'unknown is never cached')
} finally {
s.restore()
}
})
test('siteAnswer: a purge forgets the remembered answer', async (t) => {
const s = stubSite(t)
try {
assert.equal((await s.ask({ status: 404, _data: {} })).state, 'absent')
clearSiteCache()
assert.equal((await s.ask(down())).state, 'unknown')
} finally {
s.restore()
}
})