From 9933283be3f493393954128d6bf559c47c0fd897 Mon Sep 17 00:00:00 2001 From: Benoit TRAVERS Date: Tue, 30 Jun 2026 11:37:41 +0200 Subject: [PATCH] chore(website): add root robots.txt advertising all sitemaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit robots.txt is per-host and only read from the domain root, which only this landing repo serves (btravstack.github.io/robots.txt). Allow all crawling and point crawlers at all four sitemaps — the hub plus each project's docs (amqp-contract / temporal-contract / unthrown), all on this one domain. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/website/public/robots.txt | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 apps/website/public/robots.txt diff --git a/apps/website/public/robots.txt b/apps/website/public/robots.txt new file mode 100644 index 0000000..ec35e93 --- /dev/null +++ b/apps/website/public/robots.txt @@ -0,0 +1,14 @@ +# https://btravstack.github.io/robots.txt +# btravstack is one site on one domain: this landing page is the hub, and each +# project's docs live under a sub-path (/amqp-contract/, /temporal-contract/, +# /unthrown/). robots.txt is per-host and only read from the domain root, so +# this single file governs crawling for all of them. + +User-agent: * +Allow: / + +# Sitemaps — the hub plus every project's docs. +Sitemap: https://btravstack.github.io/sitemap.xml +Sitemap: https://btravstack.github.io/amqp-contract/sitemap.xml +Sitemap: https://btravstack.github.io/temporal-contract/sitemap.xml +Sitemap: https://btravstack.github.io/unthrown/sitemap.xml