arian/anubis-mirror
1
0
Fork 0
mirror of https://github.com/TecharoHQ/anubis.git synced 2026-04-17 05:44:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add various robots.txt and YAML configurations for user agent handling and crawl delays

Browse Source
This commit is contained in:
Jason Cameron
2025-06-13 13:31:47 -04:00
parent 5a85cd41a2
commit 714496e317
14 changed files with 231 additions and 356 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
cmd/robots2policy/testdata/wildcards.yaml vendored Normal file
View File

@@ -0,0 +1,16 @@
- name: robots-txt-policy-disallow-1
action: CHALLENGE
expression:
single: path.matches("^/search.*")
- name: robots-txt-policy-disallow-2
action: CHALLENGE
expression:
single: path.matches("^/.*/private")
- name: robots-txt-policy-disallow-3
action: CHALLENGE
expression:
single: path.matches("^/file.\\.txt")
- name: robots-txt-policy-disallow-4
action: CHALLENGE
expression:
single: path.matches("^/admin/.*.action=delete")