arian/anubis-mirror
1
0
Fork 0
mirror of https://github.com/TecharoHQ/anubis.git synced 2026-04-18 14:24:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(robots2policy): handle multiple user agents under one block (#925)

Browse Source
This commit is contained in:
Jason Cameron
2025-09-06 22:35:19 -04:00
committed by GitHub
parent 87c2f1e0e6
commit 82099d9e05
8 changed files with 208 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
cmd/robots2policy/testdata/consecutive.robots.txt vendored Normal file
View File

@@ -0,0 +1,25 @@
# Test consecutive user agents that should be grouped into any: blocks
User-agent: *
Disallow: /admin
Crawl-delay: 10
# Multiple consecutive user agents - should be grouped
User-agent: BadBot
User-agent: SpamBot
User-agent: EvilBot
Disallow: /
# Single user agent - should be separate
User-agent: GoodBot
Disallow: /private
# Multiple consecutive user agents with crawl delay
User-agent: SlowBot1
User-agent: SlowBot2
Crawl-delay: 5
# Multiple consecutive user agents with specific path
User-agent: SearchBot1
User-agent: SearchBot2
User-agent: SearchBot3
Disallow: /search