feat: add various robots.txt and YAML configurations for user agent handling and crawl delays

2026-06-11 06:47:59 +00:00 · 2025-06-13 13:31:47 -04:00
parent 5a85cd41a2
commit 714496e317
14 changed files with 231 additions and 356 deletions
@@ -0,0 +1,2 @@
+# Empty robots.txt (comments only)
+# No actual rules