feat: add various robots.txt and YAML configurations for user agent handling and crawl delays

2026-06-11 06:47:59 +00:00 · 2025-06-13 13:31:47 -04:00
parent 5a85cd41a2
commit 714496e317
14 changed files with 231 additions and 356 deletions
@@ -0,0 +1,5 @@
+# Simple robots.txt test
+User-agent: *
+Disallow: /admin/
+Disallow: /private
+Allow: /public