* Add Wikimedia Foundation citoid services file
Wikimedia Foundation runs a service called citoid which retrieves citation metadata from urls in order to create formatted citations.
This file contains the ip ranges allocated to the WMF (https://wikitech.wikimedia.org/wiki/IP_and_AS_allocations) from which the services make requests, as well as regex for the User-Agents from both services used to generate citations (citoid, and Zotero's translation-server which citoid makes requests to as well in order to generate the metadata).
Signed-off-by: Marielle Volz <marielle.volz@gmail.com>
* Add Wikimedia Citoid crawler to allowed list
Signed-off-by: Marielle Volz <marielle.volz@gmail.com>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Marielle Volz <marielle.volz@gmail.com>
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
PR #1451 added `CONTRIBUTING.md`, but the commit message guidelines
there conflict with the ones in `developer/code-quality.md`. Since
`CONTRIBUTING.md` is newer, presumably the guidelines there are what's
expected from new commits. But after removing that section from
`code-quality.md`, there's not much content left, so this commit just
deletes the file entirely.
Signed-off-by: Max Chernoff <git@maxchernoff.ca>
* refactor(http): split long line in respondWithStatus
Signed-off-by: Max Chernoff <git@maxchernoff.ca>
* feat(http): set `Cache-Control: no-store` on error responses
Since #132, Anubis has set `Cache-Control: no-store` on challenge
responses. However, this does not apply to deny responses, meaning that
if Anubis is configured to block certain user agents and is behind a
caching reverse proxy, this error page will be cached and served to all
subsequent requests, even those with an allowed user agent. This commit
configures the error page responder to also set the `Cache-Control`
header, meaning that deny and challenge responses will now both have the
same behaviour.
Signed-off-by: Max Chernoff <git@maxchernoff.ca>
* chore(spelling): add new words to allowlist
Signed-off-by: Max Chernoff <git@maxchernoff.ca>
* chore(actions): bump Go version to fix govulncheck errors
Signed-off-by: Max Chernoff <git@maxchernoff.ca>
---------
Signed-off-by: Max Chernoff <git@maxchernoff.ca>
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Co-authored-by: Xe Iaso <xe.iaso@techaro.lol>
* ci: purge govulncheck, it's less signal than i hoped
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci(go): use go stable
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: use go stable
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(web): include base prefix in generated URLs
Forgot to add the base prefix to these URLs. Committed a fix for this
and added a test to ensure this does not repeat. Oops!
Closes: #1402
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: split nginx configuration files to their own directory
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add nginx config smoke test based on the config in the docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat: first implementation of honeypot logic
This is a bit of an experiment, stick with me.
The core idea here is that badly written crawlers are that: badly
written. They look for anything that contains `<a href="whatever" />`
tags and will blindly use those values to recurse. This takes advantage
of that by hiding a link in a `<script>` tag like this:
```html
<script type="ignore"><a href="/bots-only">Don't click</a></script>
```
Browsers will ignore it because they have no handler for the "ignore"
script type.
This current draft is very unoptimized (it takes like 7 seconds to
generate a page on my tower), however switching spintax libraries will
make this much faster.
The hope is to make this pluggable with WebAssembly such that we force
administrators to choose a storage method. First we crawl before we
walk.
The AI involvement in this commit is limited to the spintax in
affirmations.txt, spintext.txt, and titles.txt. This generates a bunch
of "pseudoprofound bullshit" like the following:
> This Restoration to Balance & Alignment
>
> There's a moment when creators are being called to realize that the work
> can't be reduced to results, but about energy. We don't innovate products
> by pushing harder, we do it by holding the vision. Because momentum can't
> be forced, it unfolds over time when culture are moving in the same
> direction. We're being invited into a paradigm shift in how we think
> about innovation. [...]
This is intended to "look" like normal article text. As this is a first
draft, this sucks and will be improved upon.
Assisted-by: GLM 4.6, ChatGPT, GPT-OSS 120b
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(honeypot/naive): optimize hilariously
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(honeypot/naive): attempt to automatically filter out based on crawling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): use mazeGen instead of bsGen
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: add honeypot docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(test): go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: fix spelling metadata
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Implement FCrDNS and other DNS features
* Redesign DNS cache and methods
* Fix DNS cache
* Rename regexSafe arg
* Alter verifyFCrDNS(addr) behaviour
* Remove unused dnsCache field from Server struct
* Upd expressions docs
* Update docs/docs/CHANGELOG.md
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor(dns): simplify FCrDNS logging
* docs: clarify verifyFCrDNS behavior
Add a note to the documentation for `verifyFCrDNS` to clarify that it returns true when no PTR records are found for the given IP address.
* fix(dns): Improve FCrDNS error handling and tests
The `VerifyFCrDNS` function previously ignored errors returned from reverse DNS lookups. This could lead to incorrect passes when a DNS failure (other than a simple 'not found') occurred. This change ensures that any error from a reverse lookup will cause the FCrDNS check to fail.
The test suite for FCrDNS has been updated to reflect this change. The mock DNS lookups now simulate both 'not found' errors and other generic DNS errors. The test cases have been updated to ensure that the function behaves correctly in both scenarios, resolving a situation where two test cases were effectively duplicates.
* docs: Update FCrDNS documentation and spelling
Corrected a typo in the `verifyFCrDNS` function documentation.
Additionally, updated the spelling exception list to include new terms and remove redundant entries.
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: Xe Iaso <me@xeiaso.net>
* chore: add dependabot cooldown
One of the things I need to worry about with Anubis is the idea that
could pwn a dependency and then get malicious code into prod without
realizing it, a-la Jia Tan. Given that Anubis relies on tools like
Dependabot to manage updating dependencies (good for other reasons),
it makes sense to have Dependabot have a 7 day cooldown for new
versions of dependencies.
This follows the advice from Yossarian on their blog at [1]. Thanks
for the post and easy to copy/paste snippets!
[1]: https://blog.yossarian.net/2025/11/21/We-should-all-be-using-dependency-cooldowns
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor: move lib/policy/config to lib/config
Signed-off-by: Xe Iaso <me@xeiaso.net>
* refactor: don't set global loggers anymore
Ref #864
You were right @kotx, it is a bad idea to set the global logger
instance.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(config): add log sink support
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore(test): go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs(admin/policies): add logging block documentation
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(cmd/anubis): revert this change, it's meant to be its own PR
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: go mod tidy
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add file logging smoke test
Assisted-by: GLM 4.6 via Claude Code
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix: don't expose the old log file time format string
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(internal): add ListOr[T any] type
This is a utility type that lets you decode a JSON T or list of T as a
single value. This will be used with Redis Sentinel config so that you
can specify multiple sentinel addresses.
Ref TecharoHQ/botstopper#24
Assisted-by: GLM 4.6 via Claude Code
Signed-off-by: Xe Iaso <me@xeiaso.net>
* feat(store/valkey): add Redis(R) Sentinel support
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
check-spelling run (pull_request) for Xe/redis-sentinel
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* chore(store/valkey): remove pointless comments
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: document the Redis™ Sentinel configuration options
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(store/valkey): Redis™ Sentinel doesn't require a password
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
* (feat) Add cluster support to redis/vaultkey store
* (chore) Update CHANGELOG.md
* (fix) Disable maintenance notification on the Valkey store
* (fix) Valkey text fix and allow maintnotifications in spelling.
* ci: add asset build verification workflow
A CI pass that fails if generated files are out of date.
* chore: npm run assets
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(data): add ruleset to explicitly allow Docker / OCI clients
Fixes#1252
This is technically a regression as these clients used to work in Anubis
v1.22.0, however it is allowable to make this opt-in as most websites do not
expect to be serving Docker / OCI registry client traffic.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/gh-1252/docker-registry-client-fix
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
* test(docker-registry): export the right envvars
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: add simdjson dependency for homebrew node
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: install go/node without homebrew
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: use right github commit variable
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: remove simdjson dependency
Signed-off-by: Xe Iaso <me@xeiaso.net>
* ci: install ko with an action
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: add OCI registry caveat docs
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
Recent models like GPT-5 have broken these instructions. As such, I
don't think that it's worth having these around anymore. I think that
longer term it may be better to have a policy of having people disclaim
which models they use in commit footers rather than having a "don't use
this tool" policy, which people are just going to work around and
ignore.
* feat(default-config): block tencent cloud by default
This is what happens when you don't have an abuse contact.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: update spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): show error message detail when hitting some common flows
Instead of giving the user nothing to go off of, this patch gives them
an opaque blob of ROT-13 encoded base64. The logic is that if you are
smart enough to figure out how to decode this, you're probably smart
enough to either fix your broken client or give it to the adminstrator.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* Update metadata
check-spelling run (pull_request) for Xe/show-error-state
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
on-behalf-of: @check-spelling <check-spelling-bot@check-spelling.dev>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Signed-off-by: check-spelling-bot <check-spelling-bot@users.noreply.github.com>
* feat(data): add default-config macro
Closes#1152
Signed-off-by: Xe Iaso <me@xeiaso.net>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test: add default-config-macro smoke test
This uses an AI generated python script to diff the contents of the bots
field of the default configuration file and the
data/meta/default-config.yaml file. It emits a patch showing what needs
to be changed.
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(lib): enable multiple consecutive slash support
Closes#754Closes#808Closes#815
Apparently more applications use multiple slashes in a row than I
thought. There is no easy way around this other than to do this hacky
fix to avoid net/http#ServeMux's URL cleaning.
* test(double_slash): add sourceware case
Signed-off-by: Xe Iaso <me@xeiaso.net>
* test(lib): fix tests for double slash fix
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Signed-off-by: Xe Iaso <me@xeiaso.net>
