* docs: split nginx configuration files to their own directory
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add nginx config smoke test based on the config in the docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
TIL docker clients don't include the Accept header all the time. I would
have thought they did that. Oops.
Closes: #1346
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat: first implementation of honeypot logic
This is a bit of an experiment, stick with me.
The core idea here is that badly written crawlers are that: badly
written. They look for anything that contains `<a href="whatever" />`
tags and will blindly use those values to recurse. This takes advantage
of that by hiding a link in a `<script>` tag like this:
```html
<script type="ignore"><a href="/bots-only">Don't click</a></script>
```
Browsers will ignore it because they have no handler for the "ignore"
script type.
This current draft is very unoptimized (it takes like 7 seconds to
generate a page on my tower), however switching spintax libraries will
make this much faster.
The hope is to make this pluggable with WebAssembly such that we force
administrators to choose a storage method. First we crawl before we
walk.
The AI involvement in this commit is limited to the spintax in
affirmations.txt, spintext.txt, and titles.txt. This generates a bunch
of "pseudoprofound bullshit" like the following:
> This Restoration to Balance & Alignment
>
> There's a moment when creators are being called to realize that the work
> can't be reduced to results, but about energy. We don't innovate products
> by pushing harder, we do it by holding the vision. Because momentum can't
> be forced, it unfolds over time when culture are moving in the same
> direction. We're being invited into a paradigm shift in how we think
> about innovation. [...]
This is intended to "look" like normal article text. As this is a first
draft, this sucks and will be improved upon.
Assisted-by: GLM 4.6, ChatGPT, GPT-OSS 120b
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(honeypot/naive): optimize hilariously
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(honeypot/naive): attempt to automatically filter out based on crawling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): use mazeGen instead of bsGen
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: add honeypot docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(test): go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: fix spelling metadata
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Implement FCrDNS and other DNS features
* Redesign DNS cache and methods
* Fix DNS cache
* Rename regexSafe arg
* Alter verifyFCrDNS(addr) behaviour
* Remove unused dnsCache field from Server struct
* Upd expressions docs
* Update docs/docs/CHANGELOG.md
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor(dns): simplify FCrDNS logging
* docs: clarify verifyFCrDNS behavior
Add a note to the documentation for `verifyFCrDNS` to clarify that it returns true when no PTR records are found for the given IP address.
* fix(dns): Improve FCrDNS error handling and tests
The `VerifyFCrDNS` function previously ignored errors returned from reverse DNS lookups. This could lead to incorrect passes when a DNS failure (other than a simple 'not found') occurred. This change ensures that any error from a reverse lookup will cause the FCrDNS check to fail.
The test suite for FCrDNS has been updated to reflect this change. The mock DNS lookups now simulate both 'not found' errors and other generic DNS errors. The test cases have been updated to ensure that the function behaves correctly in both scenarios, resolving a situation where two test cases were effectively duplicates.
* docs: Update FCrDNS documentation and spelling
Corrected a typo in the `verifyFCrDNS` function documentation.
Additionally, updated the spelling exception list to include new terms and remove redundant entries.
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* fix(config): deprecate the report_as field for challenges
This was a bad idea when it was added and it is irresponsible to
continue to have it. It causes more UX problems than it fixes with
slight of hand.
Closes: #1310Closes: #1307
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(policy): use the new logger for config validation messages
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin/thresholds): remove this report_as setting
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: add dependabot cooldown
One of the things I need to worry about with Anubis is the idea that
could pwn a dependency and then get malicious code into prod without
realizing it, a-la Jia Tan. Given that Anubis relies on tools like
Dependabot to manage updating dependencies (good for other reasons),
it makes sense to have Dependabot have a 7 day cooldown for new
versions of dependencies.
This follows the advice from Yossarian on their blog at [1]. Thanks
for the post and easy to copy/paste snippets!
[1]: https://blog.yossarian.net/2025/11/21/We-should-all-be-using-dependency-cooldowns
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor: move lib/policy/config to lib/config
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor: don't set global loggers anymore
Ref #864
You were right @kotx, it is a bad idea to set the global logger
instance.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(config): add log sink support
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(test): go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin/policies): add logging block documentation
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(cmd/anubis): revert this change, it's meant to be its own PR
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add file logging smoke test
Assisted-by: GLM 4.6 via Claude Code
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix: don't expose the old log file time format string
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(internal): add ListOr[T any] type
This is a utility type that lets you decode a JSON T or list of T as a
single value. This will be used with Redis Sentinel config so that you
can specify multiple sentinel addresses.
Ref TecharoHQ/botstopper#24
Assisted-by: GLM 4.6 via Claude Code
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(store/valkey): add Redis(R) Sentinel support
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
check-spelling run (pull_request) for Xe/redis-sentinel
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* chore(store/valkey): remove pointless comments
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document the Redis™ Sentinel configuration options
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(store/valkey): Redis™ Sentinel doesn't require a password
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
* test: testcontainers improvements
Use the endpoint feature to get the connection URL for the container.
There are cases where localhost is not the correct one, for example when DOCKER_HOST is set to another machine.
Also, don't specify the external port for the mapping so a random unused port is used, in cases when there is already Valkey/Redis running as a container and port mapped externally on 6379.
* also remove this hack, doesn't seem necessary.
* fix(localization): correct formatting of Swedish loading message
* fix(main): correct formatting and improve readability in main.go
* fix(challenge): add difficulty and policy rule hash to challenge metadata
* docs(challenge): fix panic when validating challenges in privacy-mode browsers
* (feat) Add cluster support to redis/vaultkey store
* (chore) Update CHANGELOG.md
* (fix) Disable maintenance notification on the Valkey store
* (fix) Valkey text fix and allow maintnotifications in spelling.
* fix(data): add services folder to embedded filesystem
Also includes a regression test to ensure this does not happen again.
Assisted-By: GLM 4.6 via Claude Code
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: add asset build verification workflow
A CI pass that fails if generated files are out of date.
* chore: npm run assets
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(data): add ruleset to explicitly allow Docker / OCI clients
Fixes#1252
This is technically a regression as these clients used to work in Anubis
v1.22.0, however it is allowable to make this opt-in as most websites do not
expect to be serving Docker / OCI registry client traffic.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/gh-1252/docker-registry-client-fix
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* test(docker-registry): export the right envvars
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: add simdjson dependency for homebrew node
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: install go/node without homebrew
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: use right github commit variable
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: remove simdjson dependency
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: install ko with an action
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: add OCI registry caveat docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Tencent Cloud's abuse team reached out to me recently and asked for this
rule to be removed. Prior attempts to reach out to them to report
abusive traffic have failed, thus leading to this IP space block as a
last resort to try and maintain uptime for systems administrators.
Unfortunately, it's difficult for Tencent's abuse team to take action if
there is a blanket block like this. Let's see if this doesn't cause too
much grief.
* test(nginx-external-auth): bring up to code standards
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): close open redirect when in subrequest mode
Closes GHSA-cf57-c578-7jvv
Previously Anubis had an open redirect in subrequest auth mode due to an
insufficent fix in GHSA-jhjj-2g64-px7c. This patch adds additional
validation at several steps of the flow to prevent open redirects in
subrequest auth mode as well as implements automated testing to prevent
this from occuring in the future.
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Recent models like GPT-5 have broken these instructions. As such, I
don't think that it's worth having these around anymore. I think that
longer term it may be better to have a policy of having people disclaim
which models they use in commit footers rather than having a "don't use
this tool" policy, which people are just going to work around and
ignore.
* fix!(policy/checker): make List and-like
This has the potential to break user configs.
Anubis lets you stack multiple checks at once with blocks like this:
```yaml
name: allow-prometheus
action: ALLOW
user_agent_regex: ^prometheus-probe$
remote_addresses:
- 192.168.2.0/24
```
Previously, this only returned ALLOW if _any one_ of the conditions
matched. This behaviour has changed to only return ALLOW if _all_ of the
conditions match.
I have marked this as a potentially breaking change because I'm
absolutely certain that someone is relying on this behaviour due to
spacebar heating. If this bites you, please let me know ASAP.
Signed-off-by: Xe Iaso <me@xeiaso.net>
Assisted-by: GPT-OSS 120b on local hardware
* fix(policy/checker): more explicit short-circuit
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(default-config): block tencent cloud by default
This is what happens when you don't have an abuse contact.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Closes#1206
This can cause Anubis to have other issues, but at the very least these
issues are at the Anubis level, not the level of your target service so
it's less bad.
* Update nn.json
Signed-off-by: Sunniva Løvstad <github@turtle.garden>
* Update nn.json (2)
Signed-off-by: Sunniva Løvstad <github@turtle.garden>
* Change awkward wording
Proof of Work → work-proof, that is confirmation that someone is real through work (the computer works)
Signed-off-by: Sunniva Løvstad <github@turtle.garden>
---------
Signed-off-by: Sunniva Løvstad <github@turtle.garden>
* fix(lib): show error message detail when hitting some common flows
Instead of giving the user nothing to go off of, this patch gives them
an opaque blob of ROT-13 encoded base64. The logic is that if you are
smart enough to figure out how to decode this, you're probably smart
enough to either fix your broken client or give it to the adminstrator.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/show-error-state
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
* feat(data): add default-config macro
Closes#1152
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add default-config-macro smoke test
This uses an AI generated python script to diff the contents of the bots
field of the default configuration file and the
data/meta/default-config.yaml file. It emits a patch showing what needs
to be changed.
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add httpdebug tool
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(data/clients/git): more strictly match the git client
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(default-config): make the default config far less paranoid
This uses a variety of heuristics to make sure that clients that claim
to be browsers are more likely to behave like browsers. Most of these
are based on the results of a lot of reverse engineering and data
collection from honeypot servers.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
* fix(lib): enable multiple consecutive slash support
Closes#754Closes#808Closes#815
Apparently more applications use multiple slashes in a row than I
thought. There is no easy way around this other than to do this hacky
fix to avoid net/http#ServeMux's URL cleaning.
* test(double_slash): add sourceware case
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix tests for double slash fix
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Signed-off-by: Xe Iaso <me@xeiaso.net>
With this change, setting targetSNI to 'auto' causes anubis to
use the request host name as the SNI name, allowing multiple sites
to use the same anubis instance and same backend, while still securely
connecting to the backend via https.
See https://github.com/TecharoHQ/anubis/issues/424
* feat(lib/challenge): expose ResponseWriter to challenge issuers
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(metarefresh): randomly use the Refresh header
There are several ways to trigger an automatic refresh without
JavaScript. One of them is the "meta refresh" method[1], but the other
is with the Refresh header[2]. Both are semantically identical and
supported with browsers as old as Chrome version 1.
Given that they are basically the same thing, this patch makes Anubis
randomly select between them by using the challenge random data's first
character. This will fire about 50% of the time.
I expect this to have no impact. If this works out fine, then I will
implement some kind of fallback logic for the fast challenge such that
admins can opt into allowing clients with a no-js configuration to pass
the fast challenge. This needs to bake in the oven though.
[1]: https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/meta/http-equiv
[2]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Refresh
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(metarefresh): simplify random logic
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
If Anubis is not shut down correctly and there are leftover socket
files, Anubis will refuse to start.
As "checkpath -D" currently does not work as expected
(https://github.com/OpenRC/openrc/issues/335), simply use "rm -rf"
before starting Anubis.
Signed-off-by: Anna @CyberTailor <cyber@sysrq.in>
Also, will allow to set cookie `SameSite` mode on command line or
environment. Note that `None` mode will be forced to ``Lax`` if
cookie is set to not be secure.
Signed-off-by: Valentin Lab <valentin.lab@kalysto.org>
* fix(decaymap): fix lock convoy
Ref #1103
This uses the actor pattern to delay deletion instead of making things
fight over a lock. It also properly fixes locking logic to prevent the
convoy problem.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Add option for difficulty JWT field
* Add DIFFICULTY_IN_JWT option to docs
* Add missing_required_forwarded_headers to lt translation via Google Translate
* docs(CHANGELOG): move CHANGELOG entry to the top
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* security: npm audit fix for GHSA-hfm8-9jrf-7g9w et. al
Closes#1097
I'm not sure that this is required, but I'd sleep better at night not
finding out that it is required the hard way.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: bump postcss version
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(challenge): demote temporal assurance to 80% instead of 95%
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(challenge/preact): wait a little longer to be extra safe
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(challenge/metarefresh): wait a little longer to be extra safe
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(CHANGELOG): add fix notes
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Some admins have noticed that clients are not waiting the right amount
of time in order to access a resource protected by the metarefresh
challenge. This patch adds a check to make sure that clients have waited
at least 95% (difficulty times 950 milliseconds instead of difficulity
times 1000 milliseconds) of the time they should.
If this scales, maybe time is the best way to go for Anubis in the near
future instead of anything else computational.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(localization): Add Vietnamese translation
* feat(localization): Add Vietnamese language translation
* feat(localization): Add record to CHANGELOG.md
* feat(localization): Add test case for Vietnamese
* internal/log: Implement logging of HOST when using subrequest auth
The host header wouldn't be set on subrequest auth, so we need to look for X-Forwarded-Host header when logging requests.
* chore: add changelog entry
---------
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Co-authored-by: Xe Iaso <xe.iaso@techaro.lol>
* fix(worker): constrain nonce value to be a whole integer
Closes#1043
Sometimes the worker could get into a strange state where it has a
decimal nonce, but the server assumes that the nonce can only be a whole
number. This patch constrains the nonce to be a whole number on the
worker end by detecting if the nonce is a decimal number and then
truncating away the decimal portion.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(algorithms/fast): truncate decimal place on number of threads
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update installation.mdx to include a link to the Caddy docs
Signed-off-by: Julian Krieger <julian.krieger@hm.edu>
* Update CHANGELOG.md to include documentation changes
Signed-off-by: Julian Krieger <julian.krieger@hm.edu>
---------
Signed-off-by: Julian Krieger <julian.krieger@hm.edu>
* fix(default-config): block Huawei Cloud
Closes#978
Huawei Cloud has been egregious about its scraping. All attempts to
contact their abuse team have failed. If you work for Huawei Cloud,
please raise this issue internally and get the scraping to just stop.
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): ensure issued challenges don't get double-spent
Closes#1002
TL;DR: challenge IDs were not validated at time of token issuance. A
dedicated attacker could solve a challenge once and reuse it across
multiple sessons in order to mint additional tokens.
With the advent of store based challenge issuance in #749, this means
that these challenge IDs are only good for 30 minutes. Websites using
the most recent version of Anubis have limited exposure to this problem.
Websites using older versions of Anubis have a much more increased
exposure to this problem and are encouraged to keep this software
updated as often and as frequently as possible.
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Add option for customizable explanation text
* Add changes to CHANGELOG.md
* Replace custom explanation text in favor of static simplified text
Also includes translations for the simple_explanation using Google
Translate as a placeholder so tests pass.
---------
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Co-authored-by: Xe Iaso <xe.iaso@techaro.lol>
This was causing confusion and less technical users were thinking that
websites had been intruded upon, causing them to send me horrible things
over email.
All non-English strings were amended using Google Translate. Please fix
the localization as appropriate.
* Add JWTRestrictionHeader funktionality
* Add JWTRestrictionHeader to docs
* Move JWT_RESTRICTION_HEADER from advanced section to normal one
* Add rull request URL to Changelog
* Set default value of JWT_RESTRICTION_HEADER to X-Real-IP
* refactor: make challenge pages return the challenge component
This means that challenge pages will return only the little bit that
actually matters, not the entire component.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): move Anubis version info to be implicitly in the footer
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): embed challenge ID into generated pages
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): make tests pass
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/policy/config): amend tests
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix tests again
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
* chore(web/js): delete proof-of-work-slow.mjs
This code has served its purpose and now needs to be retired to the
great beyond. There is no replacement for this, the fast implementation
will be used instead.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(web): handle building multiple JS entrypoints and web workers
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(web): rewrite frontend worker handling
This completely rewrites how the proof of work challenge works based on
feedback from browser engine developers and starts the process of making
the proof of work function easier to change out.
- Import @aws-crypto/sha256-js to use in Firefox as its implementation
of WebCrypto doesn't jump directly from highly optimized browser
internals to JIT-ed JavaScript like Chrome's seems to.
- Move the worker code to `web/js/worker/*` with each worker named after
the hashing method and hash method implementation it uses.
- Update bench.mjs to import algorithms the new way.
- Delete video.mjs, it was part of a legacy experiment that I never had
time to finish.
- Update LibreJS comment to add info about the use of
@aws-crypto/sha256-js.
- Also update my email to my @techaro.lol address.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): don't hard dep webcrypto anymore
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(lib/policy): start the deprecation process for slow
This mostly adds a warning, but the "slow" method is in the process of
being removed. Warn admins with slog.Warn.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(web/js): allow running Anubis in non-secure contexts
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/purge-slow
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Previously the X-Forwarded-For middleware could return two commas in a
row. This is a regression test to make sure that doesn't happen again.
Imports a patch previously exclusive to Botstopper.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): block XSS attacks via nonstandard URLs
This could allow an attacker to craft an Anubis pass-challenge URL that
forces a redirect to nonstandard URLs, such as the `javascript:` scheme
which executes arbitrary JavaScript code in a browser context when the
user clicks the "Try again" button.
Release-status: cut
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
The TLS termination server sets X-Real-IP to be used by the back-end, but the back-end configuration example doesn't actually extract it so nginx logs (and back-end processing) fails to log or use the visiting IP in any way (it just states `unix:` if using a unix socket like in the example given, or the local IP if forwarded over TCP).
Adding real_ip_header to the config will fix this.
Signed-off-by: Moonchild <moonchild@palemoon.org>
* fix(lib): fix challenge issuance logic
Fixes#869
v1.21.0 changed the core challenge flow to maintain information about
challenges on the server side instead of only doing them via stateless
idempotent generation functions and relying on details to not change.
There was a subtle bug introduced in this change: if a client has an
unknown challenge ID set in its test cookie, Anubis will clear that
cookie and then throw an HTTP 500 error.
This has been fixed by making Anubis throw a new challenge page instead.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): you win this time spell check
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update pt-BR.json
While current version is good enough as a machine translation, it is not natural enough for what a reader would like to read while browsing sites made by native devs - including the subtle nuances from the original English version, now incorporated to the translation instead of plain, literal translations with questionable meanings.
Signed-off-by: HQuest <hquest@gmail.com>
* fix(locales/pt-BR): anubis is from Canada
CA is the ISO country code for Canada, but also the US state code for California.
Co-authored-by: Victor Fernandes <victorvalenca@gmail.com>
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: HQuest <hquest@gmail.com>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Victor Fernandes <victorvalenca@gmail.com>
Fixes#877
Continued from #879, event loop thrashing can cause stack space
exhaustion on ia32 systems. Previously this would thrash the event loop
in Firefox and Firefox derived browsers such as Pale Moon. I suspect
that this is the ultimate root cause of the bizarre irreproducible bugs
that Pale Moon (and maybe Cromite) users have been reporting since at
least #87 was merged.
The root cause is an invalid boolean statement:
```js
// send a progress update every 1024 iterations. since each thread checks
// separate values, one simple way to do this is by bit masking the
// nonce for multiples of 1024. unfortunately, if the number of threads
// is not prime, only some of the threads will be sending the status
// update and they will get behind the others. this is slightly more
// complicated but ensures an even distribution between threads.
if (
(nonce > oldNonce) | 1023 && // we've wrapped past 1024
(nonce >> 10) % threads === threadId // and it's our turn
) {
postMessage(nonce);
}
```
The logic here looks fine but is subtly wrong as was reported in #877
by a user in the Pale Moon community. Consider the following scenario:
`nonce` is a counter that increments by the worker count every loop.
This is intended to spread the load between CPU cores as such:
| Iteration | Worker ID | Nonce |
| :-------- | :-------- | :---- |
| 1 | 0 | 0 |
| 1 | 1 | 1 |
| 2 | 0 | 2 |
| 3 | 1 | 3 |
And so on.
The incorrect part of this is the boolean logic, specifically the part
with the bitwise or `|`. I think the intent was to use a logical or
(`||`), but this had the effect of making the `postMessage` handler fire
on every iteration. The intent of this snippet (as the comment clearly
indicates) is to make sure that the main event loop is only updated with
the worker status every 1024 iterations per worker. This had the
opposite effect, causing a lot of messages to be sent from workers to
the parent JavaScript context.
This is bad for the event loop.
Instead, I have ripped out that statement and replaced it with a much
simpler increment only counter that fires every 1024 iterations.
Additionally, only the first thread communicates back to the parent
process. This does mean that in theory the other workers could be ahead
of the first thread (posting a message out of a worker has a nonzero
cost), but in practice I don't think this will be as much of an issue as
the current behaviour is.
The root cause of the stack exhaustion is likely the pressure caused by
all of the postMessage futures piling up. Maybe the larger stack size in
64 bit environments is causing this to be fine there, maybe it's some
combination of newer hardware in 64 bit systems making this not be as
much of a problem due to it being able to handle events fast enough to
keep up with the pressure.
Either way, thanks much to @wolfbeast and the Pale Moon community for
finding this. This will make Anubis faster for everyone!
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Possible fix for #877
In some cases, the parallel solution finder in Anubis could cause
all of the worker promises to leak due to the fact the promises
were being improperly terminated. A recursion bomb happens in the
following scenario:
1. A worker sends a message indicating it found a solution to the proof
of work challenge.
2. The `onmessage` handler for that worker calls `terminate()`
3. Inside `terminate()`, the parent process loops through all other
workers and calls `w.terminate()` on them.
4. It's possible that terminating a worker could lead to the `onerror`
event handler.
5. This would create a recursive loop of `onmessage` -> `terminate` ->
`onerror` -> `terminate` -> `onerror` and so on.
This infinite recursion quickly consumes all available stack space, but
this has never been noticed in development because all of my computers
have at least 64Gi of ram provisioned to them under the axiom paying for
more ram is cheaper than paying in my time spent having to work around
not having enough ram. Additionally, ia32 has a smaller base stack size,
which means that they will run into this issue much sooner than users on
other CPU architectures will.
The fix adds a boolean `settled` flag to prevent termination from
running more than once.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add i18n smoke test
Makes sure that all of the languages that Anubis supports show up when
the challenge page is sent to a client.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(i18n): build anubis so that the smoke test doesn't backoff timeout
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
This fixes a bug that was introduced in 68b653b0, in which the call
to metricsServer was passed a plain context.Background without
signal handling.
This commit adds back in the signal handling for the metrics server,
as well as for the Thoth client and storage backend.
Closes: #853
Signed-off-by: Emily Rowlands <emily@erowl.net>
* feat(anubis): add /healthz route to metrics server
Also add health check test for Docker Compose and update documentation
for health checking Anubis with Docker Compose.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Uptime Robot is a commonly used service for tracking service
interruptions. Additional policy definitions may be beneficial for
services that do publish their IP addresses in use. The list is
additionally aggregated to slightly shorten it.
Signed-off-by: Marcel Bischoff <marcel@herrbischoff.com>
This is not used yet, but it will be part of a larger strategy around
adding/removing weight based on JA4H (and other) fingerprint matches
with Thoth.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): fix race condition when rendering multiple challenge pages at once
Closes#832
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): make try again button work
Looks like the intent of this was "try the solution again". This fix
makes the client try the challenge again.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): don't block a user if they have an invalid challenge cookie
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
This was causing issues with git clone against highly loaded servers. I
thought that this would be pretty innocuous, but I guess I was wrong.
Oops!
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Fix centered-div class usage in index.templ
There was a redundant <center> tag around a div with centered-div class. Well, not so redundant because a typo in the class attribute caused it to not apply.
Removed another <center> tag and replaced by a div.centered-div for consistency.
Signed-off-by: Jesús Martínez Novo <martineznovo@gmail.com>
* Fix centered-div class usage in index.templ (continuation)
Template needed to be compiled into go code...
---------
Signed-off-by: Jesús Martínez Novo <martineznovo@gmail.com>
* docs(known-instances): add rpmfusion.org to known instances
Signed-off-by: Lothar Serra Mari <mail@serra.me>
* docs(known-instances): add wiki.freepascal.org to known instances
Signed-off-by: Lothar Serra Mari <mail@serra.me>
---------
Signed-off-by: Lothar Serra Mari <mail@serra.me>
* correct gitea.botPolicies extension to be yaml, not json
while Anubis probably doesn't care about the extension, and would parse a JSON file just fine too, the rest of the page talks about `gitea.botPolicies.yaml`, so let's be consistent
Signed-off-by: Evgeni Golov <evgeni@golov.de>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Evgeni Golov <evgeni@golov.de>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* docs(known-instances): add clew.se to known instances
Signed-off-by: Lothar Serra Mari <mail@serra.me>
* docs(known-instances): add tumfatig.net to known instances
Signed-off-by: Lothar Serra Mari <mail@serra.me>
---------
Signed-off-by: Lothar Serra Mari <mail@serra.me>
* feat(lib/policy/expressions): add system load average to bot expression inputs
This lets Anubis dynamically react to system load in order to
increase and decrease the required level of scrutiny. High load? More
scrutiny required. Low load? Less scrutiny required.
* docs: spell system correctly
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/load-average
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* fix(default-config): don't enable low load average feature by default
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
* Add translation for Traditional Chinese
* Add translation for Traditional Chinese: test
* Add translation for Traditional Chinese: Add PR number to CHANGELOG
* Add translation for Traditional Chinese: test: remove empty lines
* Add translation for Traditional Chinese: test: remove empty lines
* docs(known-instances): add Duke University to known instances
Signed-off-by: Lothar Serra Mari <mail@serra.me>
* docs(known-instances): add fabulous.systems to known instances
Signed-off-by: Lothar Serra Mari <mail@serra.me>
* docs(known-instances): add coinhoards.org to known instances
Signed-off-by: Lothar Serra Mari <mail@serra.me>
* chore(spelling): exempt the known instances page
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Lothar Serra Mari <mail@serra.me>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* feat(decaymap): add Delete method
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(lib/challenge): refactor Validate to take ValidateInput
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib): implement store interface
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib/store): all metapackage to import all store implementations
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(policy): import all store backends
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib): use new challenge creation flow
Previously Anubis constructed challenge strings from request metadata.
This was a good idea in spirit, but has turned out to be a very bad idea
in practice. This new flow reuses the Store facility to dynamically
create challenge values with completely random data.
This is a fairly big rewrite of how Anubis processes challenges. Right
now it defaults to using the in-memory storage backend, but on-disk
(boltdb) and valkey-based adaptors will come soon.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(decaymap): fix documentation typo
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(lib): fix SA4004
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/store): make generic storage interface test adaptor
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(decaymap): invert locking process for Delete
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib/store): add bbolt store implementation
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(devcontainer): adapt to docker compose, add valkey service
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): make challenges live for 30 minutes by default
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(lib/store): implement valkey backend
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/store/valkey): disable tests if not using docker
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/policy/config): ensure valkey stores can be loaded
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/store-interface
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* chore(devcontainer): remove port forwards because vs code handles that for you
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(default-config): add a nudge to the storage backends section of the docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(docs): listen on 0.0.0.0 for dev container support
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(policy): document storage backends
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG and internal links
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin/policies): don't start a sentence with as
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: fixes found in review
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
This may seem strange, but allowlisting common crawl means that scrapers
have less incentive to scrape because they can just grab the data from
common crawl instead of scraping it again.
I'm gonna be totally honest here, I'm still not sure why #564 is still
an issue. This is really confusing and I'm going to totally throw out
how Anubis issues challenges and redo it with Valkey (#201, #622).
The problem seems to be that I assume that the makeChallenge function in
package lib is idempotent for the same client. I have no idea why this
would be inconsistent, but for some reason it is and I'm just at a loss
for words as to why this is happening.
This stops the bleeding by improving the UX as a stopgap.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Add cookie prefix option
* Add explaination comment for TestCookieName
* Rename TestCookieName value from cookie-test-if-you-block-this-anubis-wont-work to cookie-verification
* Add changes to CHANGELOG.md
* Add values to CookieName and TestCookieName in anubis.go required for testcases
* Fix cookieDynamicDomain option not being set in Options struct
* Fix using wrong cookie name when using dynamic cookie domains
* Adjust testcases for new cookie option structs
* Add known words to expect.txt and change typo in Zombocom
* Cleanup expect.txt
* Add changes to changelog
* Bump versions of grpc and apimachinery
* Fix testcases and add additional condition for dynamic cookie domain
* lib/localization: implement localization system
Locale files are placed in lib/localization/locales/. If you add a
locale, update manifest.json with available locales.
* Exclude locales from check spelling
* tests(lib/localization): add comprehensive translations test
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(challenge/metarefresh): enable localization
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix: use simple syntax for localization in templ
Also localize CELPHASE into French according to the wishes of the
artist.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore:(js): fix forbidden patterns
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: add goi18n to tools
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib/localization): dynamically determine the list of supported languages
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* feat: dynamic cookie domains
Replaces #685
I was having weird testing issues when trying to merge #685, so I
rewrote it from scratch to be a lot more minimal.
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(xess): remove unused xess templates
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore(checker): remove unused staticHashChecker implementation
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* feat: add pinact and deadcode to go tools (pinact is used for the gha pinning)
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: update Docker and kubectl actions to latest versions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: update Homebrew action from master to main in workflow files
See df537ec97f
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: remove unused go-colorable and tools dependencies from go.sum
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: update postcss-import and other dependencies to latest versions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: update Docusaurus dependencies to version 3.8.1
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* chore: downgrade playwright and playwright-core to version 1.52.0
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Closes#564
This one is really dumb. Take a seat and listen to my tale of woe.
While @victorvalenca was working on #693 we ran into a strange issue.
The tests would consistently pass on Firefox but instantly failed on
Chrome. After adding increasingly desperate debugging logs to the mix,
we found out that somehow Chrome was randomizing the contents of its
Accept-Language header. This was making the challenge string get
calculated differently, thus making things spuriously fail. I cannot
figure out what causes Chrome to do this other than you being in an
environment where you have more than one "system language" set.
Either way, this should finally fix this issue and bring peace to the
land forever*.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(config): opengraph passthrough configuration
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(ogtags): use config.OpenGraph for configuration
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: wire up ogtags config in most of the app
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(ogtags): return default tags if they are supplied
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: make OpenGraph legal so we have some sanity in reviewing
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): use OpenGraph.Enabled
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): load default config file if one is not specified in spawnAnubis
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(config): fix ST1005
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document open graph defaults and its new home in the policy file
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(installation): point to weight threshold new home
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: rename default to override
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(default-config): add off-by-default opengraph settings to bot policy file
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(anubis): make build
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix build
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat: replace cidranger with bart improving performance by 3-20x
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* perf: replace cidranger with bart for IP range checking
- Replace cidranger.Ranger with bart.Lite in RemoteAddrChecker
- Use netip.ParsePrefix instead of net.ParseCIDR for modern IP handling
- Improve performance: 3-20x faster lookups with zero heap allocations
- Update imports to use github.com/gaissmai/bart and net/netip
- Remove cidranger dependency from go.mod
Benchmark results:
- IPv4 lookups: 4x faster (15.58ns vs 63.25ns, 0 vs 2 allocs)
- IPv6 lookups: 3x faster (26.51ns vs 76.96ns, 0 vs 2 allocs)
- Insertions: 20x faster (976ns vs 19,191ns)
- Large tables: 14x faster (5.2ns vs 74.85ns)
* docs: clarify CHANGELOG to not give false impressions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* perf: optimize string concatenation in RemoteAddrChecker hash generation
Replace fmt.Fprintln with strings.Join for 7x faster performance:
- Before: 935.1 ns/op, 784 B/op, 22 allocs/op
- After: 133.2 ns/op, 192 B/op, 1 alloc/op
The hash is used for JWT cookie validation and error code generation.
Comma separation provides the same deterministic uniqueness as newlines
but with significantly better performance during policy initialization.
* chore: remove accidentally commited string benchmark
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: apply Copilot suggestions
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix: reference the right var name
i cannot write a merge commit
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Revert "docs/blog: remove (#273)"
This reverts commit df3509ec99.
* chore: intro to the blog post
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(known-instances): add bugs.scummvm.org and gitlab.postmarketos.org
Signed-off-by: Lothar Serra Mari <mail@serra.me>
* chore: clean uri
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Lothar Serra Mari <mail@serra.me>
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Co-authored-by: Jason Cameron <git@jasoncameron.dev>
* refactor(ogtags): optimize URL construction and memory allocations
* test(ogtags): add benchmarks and memory usage tests for OGTagCache
* refactor(ogtags): optimize OGTags subsystem to reduce allocations and improve request runtime by up to 66%
* Update docs/docs/CHANGELOG.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* refactor(ogtags): optimize URL string construction to reduce allocations
* Update internal/ogtags/ogtags.go
Co-authored-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
* test(ogtags): add fuzz tests for getTarget and extractOGTags functions
* fix(ogtags): update memory calculation logic
Prev it would say that we had allocated 18pb
=== RUN TestMemoryUsage
mem_test.go:107: Memory allocated for 10k getTarget calls: 18014398509481904.00 KB
mem_test.go:135: Memory allocated for 1k extractOGTags calls: 18014398509481978.00
Now it's fixed with
=== RUN TestMemoryUsage
mem_test.go:109: Memory allocated for 10k getTarget calls:
mem_test.go:110: Total: 630.56 KB (0.62 MB)
mem_test.go:111: Per operation: 64.57 bytes
mem_test.go:140: Memory allocated for 1k extractOGTags calls:
mem_test.go:141: Total: 328.17 KB (0.32 MB)
mem_test.go:142: Per operation: 336.05 bytes
* refactor(ogtags): optimize meta tag extraction for improved performance
* Update metadata
check-spelling run (pull_request) for json/ogmem
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* chore: update CHANGELOG for recent optimizations and version bump
* refactor: improve URL construction and meta tag extraction logic
* style: cleanup fuzz tests
---------
Signed-off-by: Jason Cameron <jasoncameron.all@gmail.com>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* style: fix formatting in .air.toml and installation.mdx
* feat: add --strip-base-prefix flag to modify request paths when forwarding
Closes: #638
* refactor: apply structpacking (betteralign)
* fix: add validation for strip-base-prefix and base-prefix configuration
* fix: improve request path handling by cloning request and modifying URL path
* chore: remove integration tests as they are too annoying to debug on my system
* feat(lib): implement request weight
Replaces #608
This is a big one and will be what makes Anubis a generic web
application firewall. This introduces the WEIGH option, allowing
administrators to have facets of request metadata add or remove
"weight", or the level of suspicion. This really makes Anubis weigh
the soul of requests.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): maintain legacy challenge behavior
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): make weight have dedicated checkers for the hashes
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(data): convert some rules over to weight points
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document request weight
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(CHANGELOG): spelling error
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: fix links to challenge information
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(policies): fix formatting
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(config): make default weight adjustment 5
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(deps): update dependencies in go.mod and go.sum
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor: rename variables for clarity in anubis.go and main.go
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(checker): handle error when inserting IP range in ranger
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* fix(tests): simplify boolean checks in header and URL value tests
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor(api): remove unused /test-error endpoint and restrict /make-challenge to development
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* build(deps): update golang-set to v2.8.0 in go.sum
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* Update metadata
check-spelling run (pull_request) for json/stuff
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
* feat(lib/challenge): HTTP meta refresh challenge method
Closes#95
This challenge method enables users that don't (or won't) support
JavaScript to pass Anubis challenges. It works by using HTML meta
refresh directives to ensure that the client is a browser.
This is OFF by default. In order to enable it, an administrator MUST
choose to make the default challenge method `metarefresh`.
TODO(Xe):
- [ ] Documentation on this challenge method
- [ ] Amend wording around Anubis being a proof of work proxy in the docs
- [ ] Add configuration file syntax for the default challenge method and settings
- [ ] Test with early customers
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib/challenge/metarefresh): use this value of err
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: add metarefresh challenge info, Web AI Firewall Utility
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Gives us many nice things like:
* Windows support for yeet (modulo TecharoHQ/yeet#29)
* Removes the dependency on /bin/sh or /bin/bash thanks to
mvdan.cc/sh/v3
* Checksum-compliant reproducible builds by default
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Split up AI filtering files
Create aggressive/moderate/permissive policies to allow administrators to choose their AI/LLM stance.
Aggressive policy matches existing default in Anubis.
Removes `Google-Extended` flag from `ai-robots-txt.yaml` as it doesn't exist in requests.
Rename `ai-robots-txt.yaml` to `ai-catchall.yaml` as the file is no longer a copy of the source repo/file.
* chore: spelling
* chore: fix embeds
* chore: fix data includes
* chore: fix file name typo
* chore: Ignore READMEs in configs
* chore(lib/policy/config): go tool goimports -w
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* Define OpenAI bot ALLOW policies
Allows OpenAI bots to be allowlisted at the choice of the Anubis administrator. None are enabled by default.
* Define MistralAI bot ALLOW policy
* chore: spelling
* Add Applebot definition
Adds Apple's search indexing bot, and allowlists it by default.
Allowlisted by default because it is equivalent to Googlebot/Bingbot. Remove Applebot from `ai-robots-txt.yaml` for the same reasons.
Remove `Applebot-Extended` from `ai-robots-txt.yaml` as it has no effect.
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* feat(lib): annotate cookies with what rule was passed
Anubis JWTs now contain a policyRule claim with the cryptographic hash
of the rule that it passed. This is intended to help with a future move
away from proof of work being the default.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix cookie storage logic
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(bench): await benchmark loop and adjust outline styles in templates
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* refactor: remove unused showContinueBar function and clean up video error handling
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
* style: format code for consistency and readability using prettier
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
---------
Signed-off-by: Jason Cameron <git@jasoncameron.dev>
For some reason, Google Chrome will randomly send a "full"
Accept-Language header, and other times it will send a "partial"
Accept-Language header. This makes the challenge construction
inconsistent.
This commit fixes this issue by only considering up to the first five
characters of the Accept-Language header when making a challenge string.
Signed-off-by: Xe Iaso <me@xeiaso.net>
Closes#565
The page already had the version number embedded into it, but that was
not printed to the page. This prints the version number set at compile
time to the page.
Signed-off-by: Xe Iaso <me@xeiaso.net>
This seems counter-intuitive at first glance, but let me cook.
One of the problems with Anubis is that the rule matching is super
deterministic. This means that attackers can figure out what patterns
they are hitting and change things to bypass them.
The randInt function lets you have rulesets behave nondeterministically.
This is a very easy way to hang yourself, but can be great to
psychologically mess with scraper operators. Consider this rule:
```yaml
- name: deny-lightpanda-sometimes
action: DENY
expression:
all:
- userAgent.matches("LightPanda")
- randInt(16) >= 4
```
It would match about 75% of the time.
Signed-off-by: Xe Iaso <me@xeiaso.net>
Anubis offers a [development container](https://containers.dev/) image in order to make it easier to contribute to the project. This image is based on [Xe/devcontainer-base/go](https://github.com/Xe/devcontainer-base/tree/main/src/go), which is based on Debian Bookworm with the following customizations:
- [Fish](https://fishshell.com/) as the shell complete with a custom theme
- [Go](https://go.dev) at the most recent stable version
- [Node.js](https://nodejs.org/en) at the most recent stable version
- [Atuin](https://atuin.sh/) to sync shell history between your host OS and the development container
- [Docker](https://docker.com) to manage and build Anubis container images from inside the development container
- [Ko](https://ko.build/) to build production-ready Anubis container images
- [Neovim](https://neovim.io/) for use with Git
This development container is tested and known to work with [Visual Studio Code](https://code.visualstudio.com/). If you run into problems with it outside of VS Code, please file an issue and let us know what editor you are using.
Anubis [weighs the soul of your connection](https://en.wikipedia.org/wiki/Weighing_of_souls) using a proof-of-work challenge in order to protect upstream resources from scraper bots.
Anubis is a Web AI Firewall Utility that [weighs the soul of your connection](https://en.wikipedia.org/wiki/Weighing_of_souls) using one or more challenges in order to protect upstream resources from scraper bots.
This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.
@@ -29,7 +66,7 @@ Anubis is a bit of a nuclear response. This will result in your website being bl
In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you.
If you want to try this out, connect to [anubis.techaro.lol](https://anubis.techaro.lol).
If you want to try this out, visit the Anubis documentation site at [anubis.techaro.lol](https://anubis.techaro.lol).
Techaro follows the [Semver 2.0 scheme](https://semver.org/).
## Supported Versions
Techaro strives to support the two most recent minor versions of Anubis. Patches to those versions will be published as patch releases.
## Reporting a Vulnerability
Email security@techaro.lol with details on the vulnerability and reproduction steps. You will get a response as soon as possible.
Please take care to send your email as a mixed plaintext and HTML message. Messages with GPG signatures or that are plaintext only may be blocked by the spam filter.
bindNetwork=flag.String("bind-network","tcp","network family to bind HTTP to, e.g. unix, tcp")
challengeDifficulty=flag.Int("difficulty",anubis.DefaultDifficulty,"difficulty of the challenge")
cookieDomain=flag.String("cookie-domain","","if set, the top-level domain that the Anubis cookie will be valid for")
cookieDynamicDomain=flag.Bool("cookie-dynamic-domain",false,"if set, automatically set the cookie Domain value based on the request domain")
cookieExpiration=flag.Duration("cookie-expiration-time",anubis.CookieDefaultExpirationTime,"The amount of time the authorization cookie is valid for")
cookiePrefix=flag.String("cookie-prefix",anubis.CookieName,"prefix for browser cookies created by Anubis")
cookiePartitioned=flag.Bool("cookie-partitioned",false,"if true, sets the partitioned flag on Anubis cookies, enabling CHIPS support")
difficultyInJWT=flag.Bool("difficulty-in-jwt",false,"if true, adds a difficulty field in the JWT claims")
useSimplifiedExplanation=flag.Bool("use-simplified-explanation",false,"if true, replaces the text when clicking \"Why am I seeing this?\" with a more simplified text for a non-tech-savvy audience.")
forcedLanguage=flag.String("forced-language","","if set, this language is being used instead of the one from the request's Accept-Language header")
hs512Secret=flag.String("hs512-secret","","secret used to sign JWTs, uses ed25519 if not set")
cookieSecure=flag.Bool("cookie-secure",true,"if true, sets the secure flag on Anubis cookies")
cookieSameSite=flag.String("cookie-same-site","None","sets the same site option on Anubis cookies, will auto-downgrade None to Lax if cookie-secure is false. Valid values are None, Lax, Strict, and Default.")
ed25519PrivateKeyHex=flag.String("ed25519-private-key-hex","","private key used to sign JWTs, if not set a random one will be assigned")
ed25519PrivateKeyHexFile=flag.String("ed25519-private-key-hex-file","","file name containing value for ed25519-private-key-hex")
metricsBind=flag.String("metrics-bind",":9090","network address to bind metrics to")
@@ -55,10 +67,12 @@ var (
policyFname=flag.String("policy-fname","","full path to anubis policy document (defaults to a sensible built-in policy)")
redirectDomains=flag.String("redirect-domains","","list of domains separated by commas which anubis is allowed to redirect to. Leaving this unset allows any domain.")
slogLevel=flag.String("slog-level","INFO","logging level (see https://pkg.go.dev/log/slog#hdr-Levels)")
stripBasePrefix=flag.Bool("strip-base-prefix",false,"if true, strips the base prefix from requests forwarded to the target server")
target=flag.String("target","http://localhost:3923","target to reverse proxy to, set to an empty string to disable proxying when only using auth request")
targetSNI=flag.String("target-sni","","if set, the value of the TLS handshake hostname when forwarding requests to the target")
targetSNI=flag.String("target-sni","","if set, TLS handshake hostname when forwarding requests to the target, if set to auto, use Host header")
targetHost=flag.String("target-host","","if set, the value of the Host header when forwarding requests to the target")
targetInsecureSkipVerify=flag.Bool("target-insecure-skip-verify",false,"if true, skips TLS validation for the backend")
targetDisableKeepAlive=flag.Bool("target-disable-keepalive",false,"if true, disables HTTP keep-alive for the backend")
healthcheck=flag.Bool("healthcheck",false,"run a health check against Anubis")
useRemoteAddress=flag.Bool("use-remote-address",false,"read the client's IP address from the network request, useful for debugging and running Anubis on bare metal")
debugBenchmarkJS=flag.Bool("debug-benchmark-js",false,"respond to every request with a challenge for benchmarking hashrate")
@@ -67,6 +81,15 @@ var (
ogCacheConsiderHost=flag.Bool("og-cache-consider-host",false,"enable or disable the use of the host in the Open Graph tag cache")
extractResources=flag.String("extract-resources","","if set, extract the static resources to the specified folder")
webmasterEmail=flag.String("webmaster-email","","if set, displays webmaster's email on the reject page for appeals")
publicUrl=flag.String("public-url","","the externally accessible URL for this Anubis instance, used for constructing redirect URLs (e.g., for forwardAuth).")
xffStripPrivate=flag.Bool("xff-strip-private",true,"if set, strip private addresses from X-Forwarded-For")
customRealIPHeader=flag.String("custom-real-ip-header","","if set, read remote IP from header of this name (in case your environment doesn't set X-Real-IP header)")
thothInsecure=flag.Bool("thoth-insecure",false,"if set, connect to Thoth over plain HTTP/2, don't enable this unless support told you to")
thothURL=flag.String("thoth-url","","if set, URL for Thoth, the IP reputation database for Anubis")
thothToken=flag.String("thoth-token","","if set, API token for Thoth, the IP reputation database for Anubis")
jwtRestrictionHeader=flag.String("jwt-restriction-header","X-Real-IP","If set, the JWT is only valid if the current value of this header matched the value when the JWT was created")
log.Fatalf("failed to generate ed25519 key: %v",err)
}
slog.Warn("generating random key, Anubis will have strange behavior when multiple instances are behind the same load balancer target, for more information: see https://anubis.techaro.lol/docs/admin/installation#key-generation")
lg.Warn("generating random key, Anubis will have strange behavior when multiple instances are behind the same load balancer target, for more information: see https://anubis.techaro.lol/docs/admin/installation#key-generation")
slog.Warn("REDIRECT_DOMAINS is not set, Anubis will only redirect to the same domain a request is coming from, see https://anubis.techaro.lol/docs/admin/configuration/redirect-domains")
lg.Warn("REDIRECT_DOMAINS is not set, Anubis will only redirect to the same domain a request is coming from, see https://anubis.techaro.lol/docs/admin/configuration/redirect-domains")
# # The imprint page that will be linked to at the footer of every Anubis page.
# page:
# # The HTML <title> of the page
# title: Imprint and Privacy Policy
# # The HTML contents of the page. The exact contents of this page can
# # and will vary by locale. Please consult with a lawyer if you are not
# # sure what to put here
# body: >-
# <p>Last updated: June 2025</p>
# <h2>Information that is gathered from visitors</h2>
# <p>In common with other websites, log files are stored on the web server saving details such as the visitor's IP address, browser type, referring page and time of visit.</p>
# <p>Cookies may be used to remember visitor preferences when interacting with the website.</p>
# <p>Where registration is required, the visitor's email and a username will be stored on the server.</p>
# <!-- ... -->
# Open Graph passthrough configuration, see here for more information:
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
