It seems that https://github.com/check-spelling/check-spelling/ has been
archived and this causes CI to fail. I guess we will have to keep a
beter eye on our speeling from now on until a solution can be found.
This is unfortunate, but such is life.
Signed-off-by: Xe Iaso <[email protected]>
This adds support for enabling the HttpOnly flag for cookies. Setting
this default option value to false makes it a conservative,
backwards-compatible change.
By using the HttpOnly flag, sites stay working even when using strict
cookie consent management tools, which is frequently used in EU sites to
comply with GDPR and ePrivacy directive.
The tests for setting cookies have been merged into a single
table-driven test structure, adding a test case for toggling the
HttpOnly option, while also adding a proper assertion for the custom
expiration option.
Assisted-by: Claude Opus 4.8 via Claude Code
Signed-off-by: Timon de Groot <[email protected]>
cmd/anubis: call uuid.EnableRandPool() at the top of main. The pool
batches crypto/rand reads internally, dramatically reducing per-call
syscall overhead for UUID generation. UUIDs are produced on every
issued challenge (NewV7, 3.7 times faster, down to zero allocation) and on
every challenge page render (NewString, 1.6 times faster, 1 fewer allocation).
The pool is non-cryptographic-key material, PoW challenge bytes and signing
keys still go directly to crypto/rand.
lib/anubis.go: three trivial optimizations in issueChallenge and
maybeReverseProxy, reducing the amount of allocations by 2%, which isn't much
but since the changes are trivial:
- fmt.Sprintf("%x", randomData) -> hex.EncodeToString(randomData)
- cache uuid.UUID.String() once instead of calling it three times
- fmt.Sprintf("ogtags:allow:%s%s", ...) -> string concat
Signed-off-by: jvoisin <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
Co-authored-by: Xe Iaso <[email protected]>
gzip.NewWriterLevel allocates fresh deflate window and hash table
buffers (~1.18 MiB) on every request. This commit pools them in a closure-local
sync.Pool so each middleware instance reuses its writers.
The level is validated once at setup (NewWriterLevel against
io.Discard); pooled writers are reset to io.Discard on Put so the
pool doesn't pin response writers between requests.
Only call site is RenderIndex (lib/http.go), which serves the
challenge page, so this directly cuts the per-challenge allocation
footprint.
I benchmarked the change using the following benchmark,
put in the commit message instead of in a file since it's pretty much useless
outside of this particular change.
```
package internal
import (
"io"
"net/http"
"net/http/httptest"
"testing"
)
func BenchmarkGzipMiddleware(b *testing.B) {
payload := make([]byte, 4096)
for i := range payload {
payload[i] = byte(i)
}
inner := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write(payload)
})
h := GzipMiddleware(1, inner)
b.ReportAllocs()
b.RunParallel(func(pb *testing.PB) {
req := httptest.NewRequest(http.MethodGet, "/", nil)
req.Header.Set("Accept-Encoding", "gzip")
for pb.Next() {
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
io.Copy(io.Discard, rec.Body)
}
})
}
```
The results are pretty nice:
Benchmarks (Linux arm64, count=10, benchstat, vs origin/main):
GzipMiddleware-8 sec/op 158.8µs ± 4% -> 5.2µs ± 3% -96.72% (p=0.000)
GzipMiddleware-8 B/op 1180.6 KiB -> 1.9 KiB -99.84% (p=0.000)
GzipMiddleware-8 allocs/op 32 -> 13 -59.38% (p=0.000)
Signed-off-by: jvoisin <[email protected]>
This is based on private evaluation of a prerelease security product.
I cannot comment further other than I am impressed by its output.
This commit is a squash of several commits. The impactful commits
have details underneath markdown heading twos.
## fix(metrics): don't expose pprof by default
pprof[1] is the Go standard library profiling toolkit. It is invaluable
for diagnosing how Go programs perform in the wild. However it also is
able to expose secret data set with command line flags. This is not
ideal and should be mitigated by correctly configured firewall rules. We
don't live in a world where people correctly configure firewall rules,
so we have to fix things for people. Welcome to 2026.
[1]: https://pkg.go.dev/runtime/pprof
Ref: AWOO-001
## fix(honeypot/naive): cap r9k delay to one second
Otherwise this can get unbounded, which can cause problems with lesser
HTTP proxies such as Apache.
Ref: AWOO-002
## fix(policy): mend an edge case with subrequest auth and query strings
This fixes an unlikely edge case where using subrequest auth and query
strings with path based filtering can cause reality to differ from
administrator intent. This effectively strips the query string from
subrequest auth checks. This deficiency should be fixed in the future.
Ref: AWOO-004
## fix(expressions): mend possible nil pointer deref edge case
If Anubis just started up, load averages may not be set in memory. This
can cause a nil pointer dereference which could fail requests with weird
errors until the async thread sets the load averages.
Ref: AWOO-005
## fix(lib): mend case where domainless redirects could allow cross-domain redirects
Ref: AWOO-009
## fix(expressions): validate randInt bounds before rand.IntN
Non-positive or platform-overflowing arguments to the CEL randInt
helper used to reach rand.IntN unchecked, surfacing a CEL evaluator
error during request processing when policies passed
attacker-influenced values (e.g. contentLength). Reject non-positive
bounds and detect int narrowing explicitly, returning a typed CEL
error in both cases.
Ref: AWOO-010
Signed-off-by: Xe Iaso <[email protected]>
Currently the honeypotting feature has no limits or delays anywhere and
uses that to feed an internal greylist of IP networks. This can cause
issues such as in #1613 where Claude's crawler seemed to pick up on it
and egress data at over one megabit per second until the administrator
noticed and blocked the address range.
This takes a different approach by inspiration of how the classic #xkcd
IRC bot Robot9000 works. The first time a given IPv4 /24 or IPv6 /48
visits a honepot page, Anubis sleeps for 1 millisecond. The second it
sleeps for two milliseconds. The third is four milliseconds and so on.
The goal of this is to make the scraping inherently self-limiting such
that the scrapers go off in their own corner where they won't really
hurt anyone.
Let's see if this works out according to keikaku.
Ref: https://github.com/TecharoHQ/anubis/issues/1613
Signed-off-by: Xe Iaso <[email protected]>
* fix(policy): correctly wire subrequest mode through CEL/path checkers
Previously Anubis only checked for the X-Original-Url when using
subrequest mode. This header is used by the example nginx config to pass
the request path through from the original client request to Anubis in
order to do path-based filtering.
According to facts and circumstances, Traefik hardcodes its own
headers[1]:
```text
httpdebug-1 | GET /.within.website/x/cmd/anubis/api/check
httpdebug-1 | X-Forwarded-Method: GET
httpdebug-1 | X-Forwarded-Proto: http
httpdebug-1 | X-Forwarded-Server: b9a5d299c929
httpdebug-1 | X-Forwarded-Port: 8080
httpdebug-1 | X-Forwarded-Uri: /
httpdebug-1 | X-Real-Ip: 172.18.0.1
httpdebug-1 | Accept-Encoding: gzip
httpdebug-1 | User-Agent: curl/8.20.0
httpdebug-1 | Accept: */*
httpdebug-1 | X-Forwarded-For: 172.18.0.1
httpdebug-1 | X-Forwarded-Host: localhost:8080
```
As a result, this means that path-based filtering did not work.
This commit fixes this issue by amending how path based checking logic
works:
* For CEL based checks, this pipes through the `subrequestMode` flag from
main and alters the behaviour if either `X-Original-Url` or
`X-Forwarded-Url` are found. These values are currently hardcoded for
convenience but probably need to be made configurable in the policy
file at a future date.
* For path-based checks, this uses the existing `subrequestMode` flag
from main and adds `X-Forwarded-Url` to the list of headers it checks.
A smoke test was added to make sure that traefik in this mode continues
to work. Thank you https://github.com/flifloo for filing a detailed
issue with the relevant configuration fragments. Those configuration
fragments formed the core of this smoke test.
[1]: https://doc.traefik.io/traefik/v3.4/middlewares/http/forwardauth/
Closes: https://github.com/TecharoHQ/anubis/issues/1628
Signed-off-by: Xe Iaso <[email protected]>
Co-Authored-By: flifloo <[email protected]>
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Co-authored-by: flifloo <[email protected]>
* fix: patch GHSA-6wcg-mqvh-fcvg
PR https://github.com/TecharoHQ/anubis/pull/1015 added the ability for
reverse proxies using Anubis in subrequest auth mode to look at the path
of a request as there are many rules in the wild that rely on checking
the path. This is how access to things like robots.txt or anything in the
.well-known directory is unaffected by Anubis.
However this logic was also enabled for non-subrequest deployments of Anubis,
meaning that a specially crafted request could include a /.well-known/
path in it and then get around Anubis with little effort.
This fix gates the logic behind a new plumbed variable named subrequestMode
that only fires when Anubis is running in subrequest auth mode. This
properly contains that workaround so that the logic does not fire in
most deployments.
Signed-off-by: Xe Iaso <[email protected]>
* chore: spelling
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Xe Iaso <[email protected]>
Using the User-Agent as a filtering vector for the honeypot maze was a
decent idea, however in practice it can become a DoS vector by a
malicious client adding a lot of points to Google Chrome's User-Agent
string. In practice it also seems that the worst offenders use vanilla
Google Chrome User-Agent strings as well, meaning that this backfires
horribly.
Gotta crack a few eggs to make omlettes.
Closes: #1580
Signed-off-by: Xe Iaso <[email protected]>
* l10n(de): revise and update German translations
Improve strings for clarity and naturalness, update Wikipedia link to German version.
Signed-off-by: Michael <[email protected]>
* test(localization): fix tests
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Michael <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
Co-authored-by: Xe Iaso <[email protected]>
* Resolve#1193
Address documentation and error message issues around REDIRECT_DOMAINS and required keywords in bot specifications.
* Add CHANGELOG entry
* fix: enable CEL iterators
Signed-off-by: Jason Cameron <[email protected]>
* test: add unit tests for CELChecker map iteration
Signed-off-by: Jason Cameron <[email protected]>
* fix: implement map iterators for HTTPHeaders and URLValues to resolve CEL internal errors
Signed-off-by: Jason Cameron <[email protected]>
* fix: replace checker.NewMapIterator with newMapIterator for HTTPHeaders and URLValues
Signed-off-by: Jason Cameron <[email protected]>
---------
Signed-off-by: Jason Cameron <[email protected]>
When displayed in Japanese, the `バージョン` (version) is in the middle, but the version number is at the end, so it is displayed strangely. Improve this.
**"version_info":**
```
このウェブサイトはAnubisバージョンで動作しています
```
to
```
このウェブサイトはAnubisで動作しています バージョン
```
Signed-off-by: BALLOON | FU-SEN <[email protected]>
* Add Wikimedia Foundation citoid services file
Wikimedia Foundation runs a service called citoid which retrieves citation metadata from urls in order to create formatted citations.
This file contains the ip ranges allocated to the WMF (https://wikitech.wikimedia.org/wiki/IP_and_AS_allocations) from which the services make requests, as well as regex for the User-Agents from both services used to generate citations (citoid, and Zotero's translation-server which citoid makes requests to as well in order to generate the metadata).
Signed-off-by: Marielle Volz <[email protected]>
* Add Wikimedia Citoid crawler to allowed list
Signed-off-by: Marielle Volz <[email protected]>
* chore: update spelling
Signed-off-by: Xe Iaso <[email protected]>
---------
Signed-off-by: Marielle Volz <[email protected]>
Signed-off-by: Xe Iaso <[email protected]>
Co-authored-by: Xe Iaso <[email protected]>
