This commit addresses two allocation issues:
- make([]string, 0, 4) was immediately orphaned by strings.Split
whenever an X-Forwarded-For header is present (the common
behind-a-proxy case), wasting an allocation per request.
- append([]string{ip}, forwardedList...) prepended a freshly
allocated slice on every loop iteration in O(n²), defeating the
pre-sized make two lines above.
This commit builds the chain with cheap appends into the pre-sized slice
and flip it once with slices.Reverse instead.
On a local benchmark with a 5-hop chain: 512->384 B/op (-25%), 12->7 allocs/op
(-42%), ~20% faster. As computeXFFHeader runs on every proxied request via the
XForwardedForUpdate middleware, it's an interesting optimization.
Signed-off-by: jvoisin <julien.voisin@dustri.org>
GzipMiddleware pooled its gzip.Writer in a closure-local sync.Pool
(#1654). That works only when the middleware is built once, but its
sole call site (RenderIndex) reconstructs it on every request because
the wrapped handler embeds the request-specific challenge page. The
pool was therefore created empty, used once, and garbage collected on
every challenge render, never reusing a writer. The setup-time
gzip.NewWriterLevel validation allocated a second throwaway ~1.18 MiB
writer per request on top of that.
This commit hoists the pools to a process-global sync.Map keyed by compression
level so the deflate buffers survive reconstruction, and validate the level
with a cheap range check (gzip.HuffmanOnly..BestCompression) instead of
allocating a writer to probe for an error.
Measured with the middleware constructed per request, which is the production
pattern:
B/op 1,210,345 -> 1,930 -99.84%
allocs/op 37 -> 14 -62%
ns/op ~31µs -> ~5.9µs ~-81%
To prove that this fixes a real issue, tests are added, as the middleware was
untested:
- gzip round-trip and non-gzip passthrough correctness
- TestGzipMiddlewareInvalidLevel: construction-time range-check panic
- TestGzipMiddlewarePoolSurvivesReconstruction: asserts via
testing.AllocsPerRun that per-request construction stays ~14 allocs,
not the ~37 of a defeated pool
- BenchmarkGzipMiddleware: per-request construction pattern
Signed-off-by: jvoisin <julien.voisin@dustri.org>
incrementNetwork called internal.SHA256sum(network) twice with the
same input; once for the Get key and once for the Set key. Hoist it
into a single local so the hash is computed once.
This path is gated behind a per-request sleep, so the runtime impact
is negligible; however, it lowers a tiny bit the cost of running anubis
CPU-wise, increasing also a tiny bit the cost asymmetry for crawlers.
Signed-off-by: jvoisin <julien.voisin@dustri.org>
The JA4H middleware computed a fingerprint for every request, even though
the resulting X-Http-Fingerprint-JA4H header is only consumed when a policy
references it. Benchmarks in #834 showed this added ~17µs and 59 allocs to
each request on the hot path.
Detect whether any bot rule references the header (via a headers_regex key
or a CEL expression) at config-parse time, expose it as ParsedConfig.NeedJA4H,
and only install the internal.JA4H middleware when it's actually needed.
Threshold expressions can't access request headers, so only bot rules are
scanned.
Refs: #834
Signed-off-by: jvoisin <julien.voisin@dustri.org>
It seems that https://github.com/check-spelling/check-spelling/ has been
archived and this causes CI to fail. I guess we will have to keep a
beter eye on our speeling from now on until a solution can be found.
This is unfortunate, but such is life.
Signed-off-by: Xe Iaso <me@xeiaso.net>
This adds support for enabling the HttpOnly flag for cookies. Setting
this default option value to false makes it a conservative,
backwards-compatible change.
By using the HttpOnly flag, sites stay working even when using strict
cookie consent management tools, which is frequently used in EU sites to
comply with GDPR and ePrivacy directive.
The tests for setting cookies have been merged into a single
table-driven test structure, adding a test case for toggling the
HttpOnly option, while also adding a proper assertion for the custom
expiration option.
Assisted-by: Claude Opus 4.8 via Claude Code
Signed-off-by: Timon de Groot <timon.degroot@team.blue>
cmd/anubis: call uuid.EnableRandPool() at the top of main. The pool
batches crypto/rand reads internally, dramatically reducing per-call
syscall overhead for UUID generation. UUIDs are produced on every
issued challenge (NewV7, 3.7 times faster, down to zero allocation) and on
every challenge page render (NewString, 1.6 times faster, 1 fewer allocation).
The pool is non-cryptographic-key material, PoW challenge bytes and signing
keys still go directly to crypto/rand.
lib/anubis.go: three trivial optimizations in issueChallenge and
maybeReverseProxy, reducing the amount of allocations by 2%, which isn't much
but since the changes are trivial:
- fmt.Sprintf("%x", randomData) -> hex.EncodeToString(randomData)
- cache uuid.UUID.String() once instead of calling it three times
- fmt.Sprintf("ogtags:allow:%s%s", ...) -> string concat
Signed-off-by: jvoisin <julien.voisin@dustri.org>
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Co-authored-by: Xe Iaso <xe.iaso@techaro.lol>
gzip.NewWriterLevel allocates fresh deflate window and hash table
buffers (~1.18 MiB) on every request. This commit pools them in a closure-local
sync.Pool so each middleware instance reuses its writers.
The level is validated once at setup (NewWriterLevel against
io.Discard); pooled writers are reset to io.Discard on Put so the
pool doesn't pin response writers between requests.
Only call site is RenderIndex (lib/http.go), which serves the
challenge page, so this directly cuts the per-challenge allocation
footprint.
I benchmarked the change using the following benchmark,
put in the commit message instead of in a file since it's pretty much useless
outside of this particular change.
```
package internal
import (
"io"
"net/http"
"net/http/httptest"
"testing"
)
func BenchmarkGzipMiddleware(b *testing.B) {
payload := make([]byte, 4096)
for i := range payload {
payload[i] = byte(i)
}
inner := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write(payload)
})
h := GzipMiddleware(1, inner)
b.ReportAllocs()
b.RunParallel(func(pb *testing.PB) {
req := httptest.NewRequest(http.MethodGet, "/", nil)
req.Header.Set("Accept-Encoding", "gzip")
for pb.Next() {
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
io.Copy(io.Discard, rec.Body)
}
})
}
```
The results are pretty nice:
Benchmarks (Linux arm64, count=10, benchstat, vs origin/main):
GzipMiddleware-8 sec/op 158.8µs ± 4% -> 5.2µs ± 3% -96.72% (p=0.000)
GzipMiddleware-8 B/op 1180.6 KiB -> 1.9 KiB -99.84% (p=0.000)
GzipMiddleware-8 allocs/op 32 -> 13 -59.38% (p=0.000)
Signed-off-by: jvoisin <julien.voisin@dustri.org>
This is based on private evaluation of a prerelease security product.
I cannot comment further other than I am impressed by its output.
This commit is a squash of several commits. The impactful commits
have details underneath markdown heading twos.
## fix(metrics): don't expose pprof by default
pprof[1] is the Go standard library profiling toolkit. It is invaluable
for diagnosing how Go programs perform in the wild. However it also is
able to expose secret data set with command line flags. This is not
ideal and should be mitigated by correctly configured firewall rules. We
don't live in a world where people correctly configure firewall rules,
so we have to fix things for people. Welcome to 2026.
[1]: https://pkg.go.dev/runtime/pprof
Ref: AWOO-001
## fix(honeypot/naive): cap r9k delay to one second
Otherwise this can get unbounded, which can cause problems with lesser
HTTP proxies such as Apache.
Ref: AWOO-002
## fix(policy): mend an edge case with subrequest auth and query strings
This fixes an unlikely edge case where using subrequest auth and query
strings with path based filtering can cause reality to differ from
administrator intent. This effectively strips the query string from
subrequest auth checks. This deficiency should be fixed in the future.
Ref: AWOO-004
## fix(expressions): mend possible nil pointer deref edge case
If Anubis just started up, load averages may not be set in memory. This
can cause a nil pointer dereference which could fail requests with weird
errors until the async thread sets the load averages.
Ref: AWOO-005
## fix(lib): mend case where domainless redirects could allow cross-domain redirects
Ref: AWOO-009
## fix(expressions): validate randInt bounds before rand.IntN
Non-positive or platform-overflowing arguments to the CEL randInt
helper used to reach rand.IntN unchecked, surfacing a CEL evaluator
error during request processing when policies passed
attacker-influenced values (e.g. contentLength). Reject non-positive
bounds and detect int narrowing explicitly, returning a typed CEL
error in both cases.
Ref: AWOO-010
Signed-off-by: Xe Iaso <xe.iaso@techaro.lol>
Currently the honeypotting feature has no limits or delays anywhere and
uses that to feed an internal greylist of IP networks. This can cause
issues such as in #1613 where Claude's crawler seemed to pick up on it
and egress data at over one megabit per second until the administrator
noticed and blocked the address range.
This takes a different approach by inspiration of how the classic #xkcd
IRC bot Robot9000 works. The first time a given IPv4 /24 or IPv6 /48
visits a honepot page, Anubis sleeps for 1 millisecond. The second it
sleeps for two milliseconds. The third is four milliseconds and so on.
The goal of this is to make the scraping inherently self-limiting such
that the scrapers go off in their own corner where they won't really
hurt anyone.
Let's see if this works out according to keikaku.
Ref: https://github.com/TecharoHQ/anubis/issues/1613
Signed-off-by: Xe Iaso <me@xeiaso.net>
* fix(policy): correctly wire subrequest mode through CEL/path checkers
Previously Anubis only checked for the X-Original-Url when using
subrequest mode. This header is used by the example nginx config to pass
the request path through from the original client request to Anubis in
order to do path-based filtering.
According to facts and circumstances, Traefik hardcodes its own
headers[1]:
```text
httpdebug-1 | GET /.within.website/x/cmd/anubis/api/check
httpdebug-1 | X-Forwarded-Method: GET
httpdebug-1 | X-Forwarded-Proto: http
httpdebug-1 | X-Forwarded-Server: b9a5d299c929
httpdebug-1 | X-Forwarded-Port: 8080
httpdebug-1 | X-Forwarded-Uri: /
httpdebug-1 | X-Real-Ip: 172.18.0.1
httpdebug-1 | Accept-Encoding: gzip
httpdebug-1 | User-Agent: curl/8.20.0
httpdebug-1 | Accept: */*
httpdebug-1 | X-Forwarded-For: 172.18.0.1
httpdebug-1 | X-Forwarded-Host: localhost:8080
```
As a result, this means that path-based filtering did not work.
This commit fixes this issue by amending how path based checking logic
works:
* For CEL based checks, this pipes through the `subrequestMode` flag from
main and alters the behaviour if either `X-Original-Url` or
`X-Forwarded-Url` are found. These values are currently hardcoded for
convenience but probably need to be made configurable in the policy
file at a future date.
* For path-based checks, this uses the existing `subrequestMode` flag
from main and adds `X-Forwarded-Url` to the list of headers it checks.
A smoke test was added to make sure that traefik in this mode continues
to work. Thank you https://github.com/flifloo for filing a detailed
issue with the relevant configuration fragments. Those configuration
fragments formed the core of this smoke test.
[1]: https://doc.traefik.io/traefik/v3.4/middlewares/http/forwardauth/
Closes: https://github.com/TecharoHQ/anubis/issues/1628
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-Authored-By: flifloo <flifloo@gmail.com>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Co-authored-by: flifloo <flifloo@gmail.com>
* fix: patch GHSA-6wcg-mqvh-fcvg
PR https://github.com/TecharoHQ/anubis/pull/1015 added the ability for
reverse proxies using Anubis in subrequest auth mode to look at the path
of a request as there are many rules in the wild that rely on checking
the path. This is how access to things like robots.txt or anything in the
.well-known directory is unaffected by Anubis.
However this logic was also enabled for non-subrequest deployments of Anubis,
meaning that a specially crafted request could include a /.well-known/
path in it and then get around Anubis with little effort.
This fix gates the logic behind a new plumbed variable named subrequestMode
that only fires when Anubis is running in subrequest auth mode. This
properly contains that workaround so that the logic does not fire in
most deployments.
Signed-off-by: Xe Iaso <me@xeiaso.net>
* chore: spelling
Signed-off-by: Xe Iaso <me@xeiaso.net>
---------
Signed-off-by: Xe Iaso <me@xeiaso.net>
Using the User-Agent as a filtering vector for the honeypot maze was a
decent idea, however in practice it can become a DoS vector by a
malicious client adding a lot of points to Google Chrome's User-Agent
string. In practice it also seems that the worst offenders use vanilla
Google Chrome User-Agent strings as well, meaning that this backfires
horribly.
Gotta crack a few eggs to make omlettes.
Closes: #1580
Signed-off-by: Xe Iaso <me@xeiaso.net>
dependabot[bot]