This website requires JavaScript.
Explore
Help
Sign In
arian
/
anubis-mirror
Watch
1
Star
0
Fork
0
You've already forked anubis-mirror
mirror of
https://github.com/TecharoHQ/anubis.git
synced
2026-04-10 18:48:44 +00:00
Code
Issues
Packages
Projects
Releases
Wiki
Activity
Files
7a74dddb3e9e6b8b94d6a38584484159d5c3a786
anubis-mirror
/
cmd
/
robots2policy
History
Jason Cameron
7a74dddb3e
docs: add crawl delay weight adjustment and deny user agents option to robots2policy CLI
2025-06-13 14:14:50 -04:00
..
testdata
feat: add various robots.txt and YAML configurations for user agent handling and crawl delays
2025-06-13 13:40:49 -04:00
main.go
docs: add crawl delay weight adjustment and deny user agents option to robots2policy CLI
2025-06-13 14:14:50 -04:00
robots2policy_test.go
test: add comprehensive tests for robots2policy conversion and parsing
2025-06-13 13:40:49 -04:00
