docs(k8s): document that Kubernetes support needs a non-default storage backend

Closes: #1602
Signed-off-by: Xe Iaso <me@xeiaso.net>

docs/docs/CHANGELOG.md

@@ -39,6 +39,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fix a race in the bbolt store where the asynchronous cleanup scheduled by an expired read could delete a value that had just been refreshed; the delete now only fires when the key still carries the same expired generation it observed.
 - Marginally increase the performances of requests processing
 - Marginally improve the performances of PoW validation
+- Significantly improve the performances of the gzip middleware
 
 ## v1.25.0: Necron
 

docs/docs/admin/environments/kubernetes.mdx

@@ -131,11 +131,27 @@ Then point your Ingress to the Anubis port:
               name: anubis
 ```
 
+## Storage
+
+By default, Anubis stores all of its data in memory. This memory is not shared between pods. If you have multiple instances of Anubis without the data being [stored outside of memory](../policies.mdx#storage-backends) and a [shared cookie key](../installation.mdx#key-generation), you will run into [unexpected behaviour](https://github.com/TecharoHQ/anubis/issues/1602) when user traffic traverses between pods.
+
+Based on the deployment of your Kubernetes cluster, here are the preferable storage backends to pick from:
+
+| Backend  | Pro                                                             | Con                                                                                          |
+| :------- | :-------------------------------------------------------------- | :------------------------------------------------------------------------------------------- |
+| `bbolt`  | Only requires a ReadWriteOnce PVC.                              | Does not support more than one Anubis pod.                                                   |
+| `memory` | Requires no configuration.                                      | Process memory is not shared between pods.                                                   |
+| `s3api`  | Great if your cluster includes Rook/Ceph to use RADOS directly. | Potentially higher latency unless you use a store like [Tigris](https://www.tigrisdata.com). |
+| `valkey` | Trivial to configure in your cluster.                           | If your Redis/Valkey server is down, Anubis is going to have issues.                         |
+
+Pick your poison accordingly. Many production deployments use the `s3api` and `valkey` backends without issue. Single node deployments can get away with either `memory` or `bbolt` depending on the facts and circumstances of the deployment.
+
 ## Envoy Gateway
 
 If you are using envoy-gateway, the `X-Real-Ip` header is not set by default, but Anubis does require it. You can resolve this by adding the header, either on the specific `HTTPRoute` where Anubis is listening, or on the `ClientTrafficPolicy` to apply it to any number of Gateways:
 
 HTTPRoute:
+
 ```yaml
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
@@ -160,6 +176,7 @@ spec:
 ```
 
 Applying to any number of Gateways:
+
 ```yaml
 apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: ClientTrafficPolicy

docs/docs/admin/policies.mdx

@@ -219,8 +219,11 @@ Anubis offers the following storage backends:
 
 - [`memory`](#memory) -- A simple in-memory hashmap
 - [`bbolt`](#bbolt) -- An on-disk key/value store backed by [bbolt](https://github.com/etcd-io/bbolt), an embedded key/value database for Go programs
+- [`s3api`](#s3api) -- Amazon S3 based storage or another compatible object store
 - [`valkey`](#valkey) -- A remote in-memory key/value database backed by [Valkey](https://valkey.io/) (or another database compatible with the [RESP](https://redis.io/docs/latest/develop/reference/protocol-spec/) protocol)
 
+:::warning
+
 If no storage backend is set in the policy file, Anubis will use the [`memory`](#memory) backend by default. This is equivalent to the following in the policy file:
 
 ```yaml
@@ -229,6 +232,10 @@ store:
   parameters: {}
 ```
 
+This means that all session data that is required for the challenge mechanism to work is stored **IN PROCESS MEMORY** that is **NOT** shared between instances of Anubis. If you set up Anubis with multiple instances using the `memory` storage backend, your users will sometimes get "Administrator has misconfigured Anubis" error messages when it cannot look up the aforementioned session data.
+
+:::
+
 ### `memory`
 
 The memory backend is an in-memory cache. This backend works best if you don't use multiple instances of Anubis or don't have mutable storage in the environment you're running Anubis in.

internal/gzip.go

@@ -2,11 +2,28 @@ package internal
 
 import (
 	"compress/gzip"
+	"io"
 	"net/http"
 	"strings"
+	"sync"
 )
 
 func GzipMiddleware(level int, next http.Handler) http.Handler {
+	// Validate the level once at setup; gzip.NewWriterLevel only fails for
+	// invalid levels and we'd rather panic now than mid-request.
+	if _, err := gzip.NewWriterLevel(io.Discard, level); err != nil {
+		panic(err)
+	}
+
+	// Per-middleware pool of *gzip.Writer. Each entry carries ~40 KiB of
+	// deflate buffers; reusing them avoids that allocation on every request.
+	pool := sync.Pool{
+		New: func() any {
+			gz, _ := gzip.NewWriterLevel(io.Discard, level)
+			return gz
+		},
+	}
+
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		if !strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
 			next.ServeHTTP(w, r)
@@ -14,11 +31,13 @@ func GzipMiddleware(level int, next http.Handler) http.Handler {
 		}
 
 		w.Header().Set("Content-Encoding", "gzip")
-		gz, err := gzip.NewWriterLevel(w, level)
+		gz := pool.Get().(*gzip.Writer)
-		if err != nil {
+		gz.Reset(w)
-			panic(err)
+		defer func() {
-		}
+			gz.Close()
-		defer gz.Close()
+			gz.Reset(io.Discard)
+			pool.Put(gz)
+		}()
 
 		grw := gzipResponseWriter{ResponseWriter: w, sink: gz}
 		next.ServeHTTP(grw, r)