RobotsTagMiddleware

struct RobotsTagMiddleware : AsyncMiddleware

Adds an X-Robots-Tag header to every response, asking search engines, archivers, and AI crawlers not to index or reuse the content. Complements GET /robots.txt and the robots <meta> tag in trunk.html; well-behaved crawlers honor all three.