fix: root files follow the last site answer, a 404 included, while the backend is down

An SEO-less site's robots.txt, sitemap.xml and llms.txt keep reaching
the app during an outage longer than the cache TTL; a backend that never
answered still gets the 503.

quick-261104-xae, review WR-04.
This commit is contained in:
Jakub Zych
2026-11-05 00:23:19 +01:00
parent 973142a714
commit 3868f3d74d
4 changed files with 125 additions and 20 deletions

View File

@@ -7,10 +7,12 @@
// of reading a missing robots.txt as "everything allowed". Any other path
// passes through.
//
// Once the backend has given its site answer (SEO v0.2.0; the last one
// seen is kept while the backend is down) every file above is proxied.
// While the backend has not answered since the server started, only
// robots.txt, sitemap.xml and llms.txt are: the failing fetch answers 503
// Once the backend has given its site answer (SEO v0.2.0) every file above
// is proxied. The last answer, ok or absent, is kept while the backend is
// down, so on a backend without the SEO plugin the files keep reaching the
// app during an outage. Only while the backend has not answered since the
// server started or the last purge are robots.txt, sitemap.xml and
// llms.txt proxied by their fixed names: the failing fetch answers 503
// with Retry-After, so a staging site never exposes the app's 404 as an
// allow-all robots.txt (L4 WR-08). With a backend before v0.2.0 (the site
// answer is 404) nothing is proxied here, as in v0.1.1: robots.txt and the