arian/anubis-mirror
1
0
Fork 0
mirror of https://github.com/TecharoHQ/anubis.git synced 2026-04-10 18:48:44 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: add crawl delay weight adjustment and deny user agents option to robots2policy CLI

Browse Source
This commit is contained in:
Jason Cameron
2025-06-13 14:14:50 -04:00
parent 995abcbe68
commit 7a74dddb3e
2 changed files with 12 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
cmd/robots2policy/main.go
View File

@@ -39,17 +39,17 @@ type Weight struct {
}
type Challenge struct {
Difficulty int `yaml:"difficulty,omitempty" json:"difficulty,omitempty"`
Algorithm string `yaml:"algorithm,omitempty" json:"algorithm,omitempty"`
Difficulty int `yaml:"difficulty,omitempty" json:"difficulty,omitempty"`
ReportAs int `yaml:"report_as,omitempty" json:"report_as,omitempty"`
}
type AnubisRule struct {
Name string `yaml:"name" json:"name"`
Action string `yaml:"action" json:"action"`
Expression map[string]interface{} `yaml:"expression,omitempty" json:"expression,omitempty"`
Challenge *Challenge `yaml:"challenge,omitempty" json:"challenge,omitempty"`
Weight *Weight `yaml:"weight,omitempty" json:"weight,omitempty"`
Name string `yaml:"name" json:"name"`
Action string `yaml:"action" json:"action"`
}
func main() {

16
docs/docs/admin/robots2policy.mdx
View File

@@ -34,13 +34,15 @@ robots2policy -input robots.txt -action DENY -format json
## Options
| Flag | Description | Default |
|-----------|--------------------------------------------------------------------|---------------------|
| `-input` | robots.txt file path or URL (use `-` for stdin) | *required* |
| `-output` | Output file (use `-` for stdout) | stdout |
| `-format` | Output format: `yaml` or `json` | `yaml` |
| `-action` | Action for disallowed paths: `ALLOW`, `DENY`, `CHALLENGE`, `WEIGH` | `CHALLENGE` |
| `-name` | Policy name prefix | `robots-txt-policy` |
| Flag | Description | Default |
|-----------------------|--------------------------------------------------------------------|---------------------|
| `-input` | robots.txt file path or URL (use `-` for stdin) | *required* |
| `-output` | Output file (use `-` for stdout) | stdout |
| `-format` | Output format: `yaml` or `json` | `yaml` |
| `-action` | Action for disallowed paths: `ALLOW`, `DENY`, `CHALLENGE`, `WEIGH` | `CHALLENGE` |
| `-name` | Policy name prefix | `robots-txt-policy` |
| `-crawl-delay-weight` | Weight adjustment for crawl-delay rules | `3` |
| `-deny-user-agents` | Action for blacklisted user agents | `DENY` |
## Example