* Add Wikimedia Foundation citoid services file
Wikimedia Foundation runs a service called citoid which retrieves citation metadata from urls in order to create formatted citations.
This file contains the ip ranges allocated to the WMF (https://wikitech.wikimedia.org/wiki/IP_and_AS_allocations) from which the services make requests, as well as regex for the User-Agents from both services used to generate citations (citoid, and Zotero's translation-server which citoid makes requests to as well in order to generate the metadata).
Signed-off-by: Marielle Volz <[email protected]>
* Add Wikimedia Citoid crawler to allowed list
Signed-off-by: Marielle Volz <[email protected]>
* chore: update spelling
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Marielle Volz <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
Co-authored-by: Xe Iaso <[email protected]>
TIL docker clients don't include the Accept header all the time. I would
have thought they did that. Oops.
Closes: #1346
Signed-off-by: Xe Iaso <[email protected]>
* Implement FCrDNS and other DNS features
* Redesign DNS cache and methods
* Fix DNS cache
* Rename regexSafe arg
* Alter verifyFCrDNS(addr) behaviour
* Remove unused dnsCache field from Server struct
* Upd expressions docs
* Update docs/docs/CHANGELOG.md
Signed-off-by: Xe Iaso <[email protected]>
* refactor(dns): simplify FCrDNS logging
* docs: clarify verifyFCrDNS behavior
Add a note to the documentation for `verifyFCrDNS` to clarify that it returns true when no PTR records are found for the given IP address.
* fix(dns): Improve FCrDNS error handling and tests
The `VerifyFCrDNS` function previously ignored errors returned from reverse DNS lookups. This could lead to incorrect passes when a DNS failure (other than a simple 'not found') occurred. This change ensures that any error from a reverse lookup will cause the FCrDNS check to fail.
The test suite for FCrDNS has been updated to reflect this change. The mock DNS lookups now simulate both 'not found' errors and other generic DNS errors. The test cases have been updated to ensure that the function behaves correctly in both scenarios, resolving a situation where two test cases were effectively duplicates.
* docs: Update FCrDNS documentation and spelling
Corrected a typo in the `verifyFCrDNS` function documentation.
Additionally, updated the spelling exception list to include new terms and remove redundant entries.
* chore: update spelling
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Co-authored-by: Xe Iaso <[email protected]>
* fix(config): deprecate the report_as field for challenges
This was a bad idea when it was added and it is irresponsible to
continue to have it. It causes more UX problems than it fixes with
slight of hand.
Closes: #1310Closes: #1307
Signed-off-by: Xe Iaso <[email protected]>
* fix(policy): use the new logger for config validation messages
Signed-off-by: Xe Iaso <[email protected]>
* docs(admin/thresholds): remove this report_as setting
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
* fix(data): add services folder to embedded filesystem
Also includes a regression test to ensure this does not happen again.
Assisted-By: GLM 4.6 via Claude Code
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Tencent Cloud's abuse team reached out to me recently and asked for this
rule to be removed. Prior attempts to reach out to them to report
abusive traffic have failed, thus leading to this IP space block as a
last resort to try and maintain uptime for systems administrators.
Unfortunately, it's difficult for Tencent's abuse team to take action if
there is a blanket block like this. Let's see if this doesn't cause too
much grief.
* feat(default-config): block tencent cloud by default
This is what happens when you don't have an abuse contact.
Signed-off-by: Xe Iaso <[email protected]>
* chore: update spelling
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
* feat(data): add default-config macro
Closes#1152
Signed-off-by: Xe Iaso <[email protected]>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <[email protected]>
* test: add default-config-macro smoke test
This uses an AI generated python script to diff the contents of the bots
field of the default configuration file and the
data/meta/default-config.yaml file. It emits a patch showing what needs
to be changed.
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
* test: add httpdebug tool
Signed-off-by: Xe Iaso <[email protected]>
* fix(data/clients/git): more strictly match the git client
Signed-off-by: Xe Iaso <[email protected]>
* fix(default-config): make the default config far less paranoid
This uses a variety of heuristics to make sure that clients that claim
to be browsers are more likely to behave like browsers. Most of these
are based on the results of a lot of reverse engineering and data
collection from honeypot servers.
Signed-off-by: Xe Iaso <[email protected]>
* docs: update CHANGELOG
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
* fix(default-config): block Huawei Cloud
Closes#978
Huawei Cloud has been egregious about its scraping. All attempts to
contact their abuse team have failed. If you work for Huawei Cloud,
please raise this issue internally and get the scraping to just stop.
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Uptime Robot is a commonly used service for tracking service
interruptions. Additional policy definitions may be beneficial for
services that do publish their IP addresses in use. The list is
additionally aggregated to slightly shorten it.
Signed-off-by: Marcel Bischoff <[email protected]>
This was causing issues with git clone against highly loaded servers. I
thought that this would be pretty innocuous, but I guess I was wrong.
Oops!
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib/policy/expressions): add system load average to bot expression inputs
This lets Anubis dynamically react to system load in order to
increase and decrease the required level of scrutiny. High load? More
scrutiny required. Low load? Less scrutiny required.
* docs: spell system correctly
Signed-off-by: Xe Iaso <[email protected]>
* Update metadata
check-spelling run (pull_request) for Xe/load-average
Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>
* fix(default-config): don't enable low load average feature by default
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: check-spelling-bot <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
* feat(decaymap): add Delete method
Signed-off-by: Xe Iaso <[email protected]>
* chore(lib/challenge): refactor Validate to take ValidateInput
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib): implement store interface
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib/store): all metapackage to import all store implementations
Signed-off-by: Xe Iaso <[email protected]>
* chore(policy): import all store backends
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib): use new challenge creation flow
Previously Anubis constructed challenge strings from request metadata.
This was a good idea in spirit, but has turned out to be a very bad idea
in practice. This new flow reuses the Store facility to dynamically
create challenge values with completely random data.
This is a fairly big rewrite of how Anubis processes challenges. Right
now it defaults to using the in-memory storage backend, but on-disk
(boltdb) and valkey-based adaptors will come soon.
Signed-off-by: Xe Iaso <[email protected]>
* chore(decaymap): fix documentation typo
Signed-off-by: Xe Iaso <[email protected]>
* chore(lib): fix SA4004
Signed-off-by: Xe Iaso <[email protected]>
* test(lib/store): make generic storage interface test adaptor
Signed-off-by: Xe Iaso <[email protected]>
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
* fix(decaymap): invert locking process for Delete
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib/store): add bbolt store implementation
Signed-off-by: Xe Iaso <[email protected]>
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
* chore: go mod tidy
Signed-off-by: Xe Iaso <[email protected]>
* chore(devcontainer): adapt to docker compose, add valkey service
Signed-off-by: Xe Iaso <[email protected]>
* fix(lib): make challenges live for 30 minutes by default
Signed-off-by: Xe Iaso <[email protected]>
* feat(lib/store): implement valkey backend
Signed-off-by: Xe Iaso <[email protected]>
* test(lib/store/valkey): disable tests if not using docker
Signed-off-by: Xe Iaso <[email protected]>
* test(lib/policy/config): ensure valkey stores can be loaded
Signed-off-by: Xe Iaso <[email protected]>
* Update metadata
check-spelling run (pull_request) for Xe/store-interface
Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>
* chore(devcontainer): remove port forwards because vs code handles that for you
Signed-off-by: Xe Iaso <[email protected]>
* docs(default-config): add a nudge to the storage backends section of the docs
Signed-off-by: Xe Iaso <[email protected]>
* chore(docs): listen on 0.0.0.0 for dev container support
Signed-off-by: Xe Iaso <[email protected]>
* docs(policy): document storage backends
Signed-off-by: Xe Iaso <[email protected]>
* docs: update CHANGELOG and internal links
Signed-off-by: Xe Iaso <[email protected]>
* docs(admin/policies): don't start a sentence with as
Signed-off-by: Xe Iaso <[email protected]>
* chore: fixes found in review
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Signed-off-by: check-spelling-bot <[email protected]>
This may seem strange, but allowlisting common crawl means that scrapers
have less incentive to scrape because they can just grab the data from
common crawl instead of scraping it again.
* Fix cookieDynamicDomain option not being set in Options struct
* Fix using wrong cookie name when using dynamic cookie domains
* Adjust testcases for new cookie option structs
* Add known words to expect.txt and change typo in Zombocom
* Cleanup expect.txt
* Add changes to changelog
* Bump versions of grpc and apimachinery
* Fix testcases and add additional condition for dynamic cookie domain
* feat(lib): implement request weight
Replaces #608
This is a big one and will be what makes Anubis a generic web
application firewall. This introduces the WEIGH option, allowing
administrators to have facets of request metadata add or remove
"weight", or the level of suspicion. This really makes Anubis weigh
the soul of requests.
Signed-off-by: Xe Iaso <[email protected]>
* fix(lib): maintain legacy challenge behavior
Signed-off-by: Xe Iaso <[email protected]>
* fix(lib): make weight have dedicated checkers for the hashes
Signed-off-by: Xe Iaso <[email protected]>
* feat(data): convert some rules over to weight points
Signed-off-by: Xe Iaso <[email protected]>
* docs: document request weight
Signed-off-by: Xe Iaso <[email protected]>
* fix(CHANGELOG): spelling error
Signed-off-by: Xe Iaso <[email protected]>
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
* docs: fix links to challenge information
Signed-off-by: Xe Iaso <[email protected]>
* docs(policies): fix formatting
Signed-off-by: Xe Iaso <[email protected]>
* fix(config): make default weight adjustment 5
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Xe Iaso