arian/anubis-mirror
1
0
Fork 0
mirror of https://github.com/TecharoHQ/anubis.git synced 2026-04-15 12:58:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: add crawl delay weight adjustment and deny user agents option to robots2policy CLI

Browse Source
This commit is contained in:
Jason Cameron
2025-06-13 14:14:50 -04:00
parent 995abcbe68
commit 7a74dddb3e
2 changed files with 12 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
docs/docs/admin/robots2policy.mdx
View File

@@ -34,13 +34,15 @@ robots2policy -input robots.txt -action DENY -format json
## Options
| Flag | Description | Default |
|-----------|--------------------------------------------------------------------|---------------------|
| `-input` | robots.txt file path or URL (use `-` for stdin) | *required* |
| `-output` | Output file (use `-` for stdout) | stdout |
| `-format` | Output format: `yaml` or `json` | `yaml` |
| `-action` | Action for disallowed paths: `ALLOW`, `DENY`, `CHALLENGE`, `WEIGH` | `CHALLENGE` |
| `-name` | Policy name prefix | `robots-txt-policy` |
| Flag | Description | Default |
|-----------------------|--------------------------------------------------------------------|---------------------|
| `-input` | robots.txt file path or URL (use `-` for stdin) | *required* |
| `-output` | Output file (use `-` for stdout) | stdout |
| `-format` | Output format: `yaml` or `json` | `yaml` |
| `-action` | Action for disallowed paths: `ALLOW`, `DENY`, `CHALLENGE`, `WEIGH` | `CHALLENGE` |
| `-name` | Policy name prefix | `robots-txt-policy` |
| `-crawl-delay-weight` | Weight adjustment for crawl-delay rules | `3` |
| `-deny-user-agents` | Action for blacklisted user agents | `DENY` |
## Example