Jason Cameron
|
5730801928
|
feat: add batch processing for robots.txt files to generate Anubis CEL policies
|
2025-06-13 16:56:22 -04:00 |
|
Jason Cameron
|
c1a6daf6d2
|
fix: add validation for generated Anubis rules from robots.txt
|
2025-06-13 16:56:05 -04:00 |
|
Jason Cameron
|
60031f8b89
|
style: improve alignment of struct fields in AnubisRule and OGTagCache
|
2025-06-13 16:35:33 -04:00 |
|
Jason Cameron
|
5aa7d8e0d6
|
fix: reorder fields in AnubisRule struct for better alignment
|
2025-06-13 16:34:43 -04:00 |
|
Jason Cameron
|
5ee537bc22
|
fix: correct flag check in main function
|
2025-06-13 16:32:51 -04:00 |
|
Jason Cameron
|
2ed783c1a6
|
refactor: use preexisting structs in config
|
2025-06-13 16:29:15 -04:00 |
|
Xe Iaso
|
332320791a
|
fix(robots2policy): use sigs.k8s.io/yaml
Signed-off-by: Xe Iaso <me@xeiaso.net>
|
2025-06-13 14:40:23 -04:00 |
|
Jason Cameron
|
2dd2a2fa8c
|
Update cmd/robots2policy/main.go
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
|
2025-06-13 14:26:04 -04:00 |
|
Jason Cameron
|
31c0351c04
|
Update cmd/robots2policy/main.go
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
|
2025-06-13 14:25:54 -04:00 |
|
Jason Cameron
|
7a74dddb3e
|
docs: add crawl delay weight adjustment and deny user agents option to robots2policy CLI
|
2025-06-13 14:14:50 -04:00 |
|
Jason Cameron
|
c48029b48e
|
fix: update example URL in usage instructions for robots2policy CLI
|
2025-06-13 13:44:05 -04:00 |
|
Jason Cameron
|
d1947cadd2
|
test: add comprehensive tests for robots2policy conversion and parsing
|
2025-06-13 13:40:49 -04:00 |
|
Jason Cameron
|
714496e317
|
feat: add various robots.txt and YAML configurations for user agent handling and crawl delays
|
2025-06-13 13:40:49 -04:00 |
|
Jason Cameron
|
5a85cd41a2
|
feat: implement crawl delay handling as weight adjustment in Anubis rules
|
2025-06-13 13:40:48 -04:00 |
|
Jason Cameron
|
9ad2d42b9f
|
feat: add robots2policy CLI utility to convert robots.txt to Anubis challenge policies
|
2025-06-13 13:40:48 -04:00 |
|