Previously Anubis constructed challenge strings from request metadata.
This was a good idea in spirit, but has turned out to be a very bad idea
in practice. This new flow reuses the Store facility to dynamically
create challenge values with completely random data.
This is a fairly big rewrite of how Anubis processes challenges. Right
now it defaults to using the in-memory storage backend, but on-disk
(boltdb) and valkey-based adaptors will come soon.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Fix cookieDynamicDomain option not being set in Options struct
* Fix using wrong cookie name when using dynamic cookie domains
* Adjust testcases for new cookie option structs
* Add known words to expect.txt and change typo in Zombocom
* Cleanup expect.txt
* Add changes to changelog
* Bump versions of grpc and apimachinery
* Fix testcases and add additional condition for dynamic cookie domain
* lib/localization: implement localization system
Locale files are placed in lib/localization/locales/. If you add a
locale, update manifest.json with available locales.
* Exclude locales from check spelling
* tests(lib/localization): add comprehensive translations test
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(challenge/metarefresh): enable localization
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix: use simple syntax for localization in templ
Also localize CELPHASE into French according to the wishes of the
artist.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore:(js): fix forbidden patterns
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: add goi18n to tools
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/localization): dynamically determine the list of supported languages
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* feat: dynamic cookie domains
Replaces #685
I was having weird testing issues when trying to merge #685, so I
rewrote it from scratch to be a lot more minimal.
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(xess): remove unused xess templates
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore(checker): remove unused staticHashChecker implementation
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: add pinact and deadcode to go tools (pinact is used for the gha pinning)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: update Docker and kubectl actions to latest versions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: update Homebrew action from master to main in workflow files
See df537ec97f
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: remove unused go-colorable and tools dependencies from go.sum
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: update postcss-import and other dependencies to latest versions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: update Docusaurus dependencies to version 3.8.1
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: downgrade playwright and playwright-core to version 1.52.0
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Closes#564
This one is really dumb. Take a seat and listen to my tale of woe.
While @victorvalenca was working on #693 we ran into a strange issue.
The tests would consistently pass on Firefox but instantly failed on
Chrome. After adding increasingly desperate debugging logs to the mix,
we found out that somehow Chrome was randomizing the contents of its
Accept-Language header. This was making the challenge string get
calculated differently, thus making things spuriously fail. I cannot
figure out what causes Chrome to do this other than you being in an
environment where you have more than one "system language" set.
Either way, this should finally fix this issue and bring peace to the
land forever*.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(config): opengraph passthrough configuration
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(ogtags): use config.OpenGraph for configuration
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: wire up ogtags config in most of the app
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(ogtags): return default tags if they are supplied
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: make OpenGraph legal so we have some sanity in reviewing
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): use OpenGraph.Enabled
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): load default config file if one is not specified in spawnAnubis
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(config): fix ST1005
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document open graph defaults and its new home in the policy file
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(installation): point to weight threshold new home
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: rename default to override
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(default-config): add off-by-default opengraph settings to bot policy file
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(anubis): make build
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix build
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat: replace cidranger with bart improving performance by 3-20x
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* perf: replace cidranger with bart for IP range checking
- Replace cidranger.Ranger with bart.Lite in RemoteAddrChecker
- Use netip.ParsePrefix instead of net.ParseCIDR for modern IP handling
- Improve performance: 3-20x faster lookups with zero heap allocations
- Update imports to use github.com/gaissmai/bart and net/netip
- Remove cidranger dependency from go.mod
Benchmark results:
- IPv4 lookups: 4x faster (15.58ns vs 63.25ns, 0 vs 2 allocs)
- IPv6 lookups: 3x faster (26.51ns vs 76.96ns, 0 vs 2 allocs)
- Insertions: 20x faster (976ns vs 19,191ns)
- Large tables: 14x faster (5.2ns vs 74.85ns)
* docs: clarify CHANGELOG to not give false impressions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* perf: optimize string concatenation in RemoteAddrChecker hash generation
Replace fmt.Fprintln with strings.Join for 7x faster performance:
- Before: 935.1 ns/op, 784 B/op, 22 allocs/op
- After: 133.2 ns/op, 192 B/op, 1 alloc/op
The hash is used for JWT cookie validation and error code generation.
Comma separation provides the same deterministic uniqueness as newlines
but with significantly better performance during policy initialization.
* chore: remove accidentally commited string benchmark
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: apply Copilot suggestions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix: reference the right var name
i cannot write a merge commit
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: fix formatting in .air.toml and installation.mdx
* feat: add --strip-base-prefix flag to modify request paths when forwarding
Closes: #638
* refactor: apply structpacking (betteralign)
* fix: add validation for strip-base-prefix and base-prefix configuration
* fix: improve request path handling by cloning request and modifying URL path
* chore: remove integration tests as they are too annoying to debug on my system
* feat(lib): implement request weight
Replaces #608
This is a big one and will be what makes Anubis a generic web
application firewall. This introduces the WEIGH option, allowing
administrators to have facets of request metadata add or remove
"weight", or the level of suspicion. This really makes Anubis weigh
the soul of requests.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): maintain legacy challenge behavior
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): make weight have dedicated checkers for the hashes
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(data): convert some rules over to weight points
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document request weight
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(CHANGELOG): spelling error
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: fix links to challenge information
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(policies): fix formatting
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(config): make default weight adjustment 5
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(deps): update dependencies in go.mod and go.sum
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor: rename variables for clarity in anubis.go and main.go
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(checker): handle error when inserting IP range in ranger
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(tests): simplify boolean checks in header and URL value tests
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(api): remove unused /test-error endpoint and restrict /make-challenge to development
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* build(deps): update golang-set to v2.8.0 in go.sum
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Update metadata
check-spelling run (pull_request) for json/stuff
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
* feat(lib/challenge): HTTP meta refresh challenge method
Closes#95
This challenge method enables users that don't (or won't) support
JavaScript to pass Anubis challenges. It works by using HTML meta
refresh directives to ensure that the client is a browser.
This is OFF by default. In order to enable it, an administrator MUST
choose to make the default challenge method `metarefresh`.
TODO(Xe):
- [ ] Documentation on this challenge method
- [ ] Amend wording around Anubis being a proof of work proxy in the docs
- [ ] Add configuration file syntax for the default challenge method and settings
- [ ] Test with early customers
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib/challenge/metarefresh): use this value of err
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: add metarefresh challenge info, Web AI Firewall Utility
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Split up AI filtering files
Create aggressive/moderate/permissive policies to allow administrators to choose their AI/LLM stance.
Aggressive policy matches existing default in Anubis.
Removes `Google-Extended` flag from `ai-robots-txt.yaml` as it doesn't exist in requests.
Rename `ai-robots-txt.yaml` to `ai-catchall.yaml` as the file is no longer a copy of the source repo/file.
* chore: spelling
* chore: fix embeds
* chore: fix data includes
* chore: fix file name typo
* chore: Ignore READMEs in configs
* chore(lib/policy/config): go tool goimports -w
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* feat(lib): annotate cookies with what rule was passed
Anubis JWTs now contain a policyRule claim with the cryptographic hash
of the rule that it passed. This is intended to help with a future move
away from proof of work being the default.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix cookie storage logic
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
For some reason, Google Chrome will randomly send a "full"
Accept-Language header, and other times it will send a "partial"
Accept-Language header. This makes the challenge construction
inconsistent.
This commit fixes this issue by only considering up to the first five
characters of the Accept-Language header when making a challenge string.
Signed-off-by: Xe Iaso <me@xeiaso.net>
This seems counter-intuitive at first glance, but let me cook.
One of the problems with Anubis is that the rule matching is super
deterministic. This means that attackers can figure out what patterns
they are hitting and change things to bypass them.
The randInt function lets you have rulesets behave nondeterministically.
This is a very easy way to hang yourself, but can be great to
psychologically mess with scraper operators. Consider this rule:
```yaml
- name: deny-lightpanda-sometimes
action: DENY
expression:
all:
- userAgent.matches("LightPanda")
- randInt(16) >= 4
```
It would match about 75% of the time.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(expression): add validation for empty ExpressionOrList
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(imports): block empty file imports with improved error checking logic
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* docs(expression): improve validation to error on empty CEL expressions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Closes#531
This changes `anubis_challenges_issued` to be a vector counter that
records the challenge issuance method.
Signed-off-by: Xe Iaso <me@xeiaso.net>
Closes#520
For some reason, Chrome and Firefox are very picky over what they use to
match cookies that need to be deleted. Listen to me for my tale of woe:
The basic problem here is that cookies were an early hack added on the
side of the HTTP spec and they're basically impossible to upgrade or
change because who knows what relies on the exact behavior cookies use.
As a result, cookies don't just match by name, but by every setting that
exists on them. You can also have two cookies with the same name but
different values. This spec is a nightmare lol.
Even more fun: browsers will make up values for cookies if they aren't
set, meaning that getting a challenge token at `/docs` is semantically
different than a challenge token you got from `/`.
This PR fixes this issue by explicitly setting the "make sure cookie
support is working" cookie's path to `/`, meaning that it will always be
sent. Additionally, cookies are expired by setting the expiry time to
one minute in the past.
Hopefully this will fix it. I'm testing this locally and it seems to
work fine.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib): ensure that clients store cookies
If a client is misconfigured and does not store cookies, then they can
get into a proof of work death spiral with Anubis. This fixes the
problem by setting a test cookie whenever the user gets hit with a
challenge page. If the test cookie is not there at challenge pass time,
then they are blocked. Administrators will also get a log message
explaining that the user intentionally broke cookie support and that this
behavior is not an Anubis bug.
Additionally, this ensures that clients being shown a challenge support
gzip-compressed responses by showing the challenge page at gzip level 1.
This level is intentionally chosen in order to minimize system impacts.
The ClearCookie function is made more generic to account for cookie
names as an argument. A correlating SetCookie function was also added to
make it easier to set cookies.
* chore(lib): clean up test code
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Previously this made ClearCookie always clear cookies by name even when
CookieDomain was set. This change fixes this and adds tests to make sure
that this doesn't happen again.
Signed-off-by: Xe Iaso <me@xeiaso.net>
Also properly re-brand the cookies so that some of the /x/ heritage is
lost.
This will invalidate existing cookies and probably affects tests.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor: reorder import statements in fetch.go and fetch_test.go
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix: optimize struct field alignment to reduce memory usage
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat(lib/policy): add support for CEL checkers
This adds the ability for administrators to use Common Expression
Language[0] (CEL) for more advanced check logic than Anubis previously
offered.
These can be as simple as:
```yaml
- name: allow-api-routes
action: ALLOW
expression:
and:
- '!(method == "HEAD" || method == "GET")'
- path.startsWith("/api/")
```
or get as complicated as:
```yaml
- name: allow-git-clients
action: ALLOW
expression:
and:
- userAgent.startsWith("git/") || userAgent.contains("libgit") || userAgent.startsWith("go-git") || userAgent.startsWith("JGit/") || userAgent.startsWith("JGit-")
- >
"Git-Protocol" in headers && headers["Git-Protocol"] == "version=2"
```
Internally these are compiled and evaluated with cel-go[1]. This also
leaves room for extensibility should that be desired in the future. This
will intersect with #338 and eventually intersect with TLS fingerprints
as in #337.
[0]: https://cel.dev/
[1]: https://github.com/google/cel-go
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(data/apps): add API route allow rule for non-HEAD/GET
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document expression syntax
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix: fixes in review
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
