* Implement FCrDNS and other DNS features
* Redesign DNS cache and methods
* Fix DNS cache
* Rename regexSafe arg
* Alter verifyFCrDNS(addr) behaviour
* Remove unused dnsCache field from Server struct
* Upd expressions docs
* Update docs/docs/CHANGELOG.md
Signed-off-by: Xe Iaso <[email protected]>
* refactor(dns): simplify FCrDNS logging
* docs: clarify verifyFCrDNS behavior
Add a note to the documentation for `verifyFCrDNS` to clarify that it returns true when no PTR records are found for the given IP address.
* fix(dns): Improve FCrDNS error handling and tests
The `VerifyFCrDNS` function previously ignored errors returned from reverse DNS lookups. This could lead to incorrect passes when a DNS failure (other than a simple 'not found') occurred. This change ensures that any error from a reverse lookup will cause the FCrDNS check to fail.
The test suite for FCrDNS has been updated to reflect this change. The mock DNS lookups now simulate both 'not found' errors and other generic DNS errors. The test cases have been updated to ensure that the function behaves correctly in both scenarios, resolving a situation where two test cases were effectively duplicates.
* docs: Update FCrDNS documentation and spelling
Corrected a typo in the `verifyFCrDNS` function documentation.
Additionally, updated the spelling exception list to include new terms and remove redundant entries.
* chore: update spelling
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Co-authored-by: Xe Iaso <[email protected]>
* test: add httpdebug tool
Signed-off-by: Xe Iaso <[email protected]>
* fix(data/clients/git): more strictly match the git client
Signed-off-by: Xe Iaso <[email protected]>
* fix(default-config): make the default config far less paranoid
This uses a variety of heuristics to make sure that clients that claim
to be browsers are more likely to behave like browsers. Most of these
are based on the results of a lot of reverse engineering and data
collection from honeypot servers.
Signed-off-by: Xe Iaso <[email protected]>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib): implement request weight
Replaces #608
This is a big one and will be what makes Anubis a generic web
application firewall. This introduces the WEIGH option, allowing
administrators to have facets of request metadata add or remove
"weight", or the level of suspicion. This really makes Anubis weigh
the soul of requests.
Signed-off-by: Xe Iaso <[email protected]>
* fix(lib): maintain legacy challenge behavior
Signed-off-by: Xe Iaso <[email protected]>
* fix(lib): make weight have dedicated checkers for the hashes
Signed-off-by: Xe Iaso <[email protected]>
* feat(data): convert some rules over to weight points
Signed-off-by: Xe Iaso <[email protected]>
* docs: document request weight
Signed-off-by: Xe Iaso <[email protected]>
* fix(CHANGELOG): spelling error
Signed-off-by: Xe Iaso <[email protected]>
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
* docs: fix links to challenge information
Signed-off-by: Xe Iaso <[email protected]>
* docs(policies): fix formatting
Signed-off-by: Xe Iaso <[email protected]>
* fix(config): make default weight adjustment 5
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
* Split up AI filtering files
Create aggressive/moderate/permissive policies to allow administrators to choose their AI/LLM stance.
Aggressive policy matches existing default in Anubis.
Removes `Google-Extended` flag from `ai-robots-txt.yaml` as it doesn't exist in requests.
Rename `ai-robots-txt.yaml` to `ai-catchall.yaml` as the file is no longer a copy of the source repo/file.
* chore: spelling
* chore: fix embeds
* chore: fix data includes
* chore: fix file name typo
* chore: Ignore READMEs in configs
* chore(lib/policy/config): go tool goimports -w
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Co-authored-by: Xe Iaso <[email protected]>
* Define OpenAI bot ALLOW policies
Allows OpenAI bots to be allowlisted at the choice of the Anubis administrator. None are enabled by default.
* Define MistralAI bot ALLOW policy
* chore: spelling
* feat(lib/policy): add support for CEL checkers
This adds the ability for administrators to use Common Expression
Language[0] (CEL) for more advanced check logic than Anubis previously
offered.
These can be as simple as:
```yaml
- name: allow-api-routes
action: ALLOW
expression:
and:
- '!(method == "HEAD" || method == "GET")'
- path.startsWith("/api/")
```
or get as complicated as:
```yaml
- name: allow-git-clients
action: ALLOW
expression:
and:
- userAgent.startsWith("git/") || userAgent.contains("libgit") || userAgent.startsWith("go-git") || userAgent.startsWith("JGit/") || userAgent.startsWith("JGit-")
- >
"Git-Protocol" in headers && headers["Git-Protocol"] == "version=2"
```
Internally these are compiled and evaluated with cel-go[1]. This also
leaves room for extensibility should that be desired in the future. This
will intersect with #338 and eventually intersect with TLS fingerprints
as in #337.
[0]: https://cel.dev/
[1]: https://github.com/google/cel-go
Signed-off-by: Xe Iaso <[email protected]>
* feat(data/apps): add API route allow rule for non-HEAD/GET
Signed-off-by: Xe Iaso <[email protected]>
* docs: document expression syntax
Signed-off-by: Xe Iaso <[email protected]>
* fix: fixes in review
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
