v1.18.0: Varis zos Galvus

The big ticket feature in this release is [CEL expression matching support](https://anubis.techaro.lol/docs/admin/configuration/expressions). This allows you to tailor your approach for the individual services you are protecting.

These can be as simple as:

```yaml
- name: allow-api-requests
  action: ALLOW
  expression:
    all:
      - '"Accept" in headers'
      - 'headers["Accept"] == "application/json"'
      - 'path.startsWith("/api/")'
```

Or as complicated as:

```yaml
- name: allow-git-clients
  action: ALLOW
  expression:
    all:
      - >-
        (
          userAgent.startsWith("git/") ||
          userAgent.contains("libgit") ||
          userAgent.startsWith("go-git") ||
          userAgent.startsWith("JGit/") ||
          userAgent.startsWith("JGit-")
        )
      - '"Git-Protocol" in headers'
      - headers["Git-Protocol"] == "version=2"
```

The docs have more information, but here's a tl;dr of the variables you have access to in expressions:

| Name            | Type                  | Explanation                                                                                                                               | Example                                                      |
| :-------------- | :-------------------- | :---------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------- |
| `headers`       | `map[string, string]` | The [headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers) of the request being processed.                        | `{"User-Agent": "Mozilla/5.0 Gecko/20100101 Firefox/137.0"}` |
| `host`          | `string`              | The [HTTP hostname](https://web.dev/articles/url-parts#host) the request is targeted to.                                                  | `anubis.techaro.lol`                                         |
| `method`        | `string`              | The [HTTP method](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Methods) in the request being processed.                    | `GET`, `POST`, `DELETE`, etc.                                |
| `path`          | `string`              | The [path](https://web.dev/articles/url-parts#pathname) of the request being processed.                                                   | `/`, `/api/memes/create`                                     |
| `query`         | `map[string, string]` | The [query parameters](https://web.dev/articles/url-parts#query) of the request being processed.                                          | `?foo=bar` -> `{"foo": "bar"}`                               |
| `remoteAddress` | `string`              | The IP address of the client.                                                                                                             | `1.1.1.1`                                                    |
| `userAgent`     | `string`              | The [`User-Agent`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent) string in the request being processed. | `Mozilla/5.0 Gecko/20100101 Firefox/137.0`                   |

This will be made more elaborate in the future. Give me time. This is a [simple, lovable, and complete](https://longform.asmartbear.com/slc/) implementation of this feature so that administrators can get hacking ASAP.

Other changes:

- Use CSS variables to deduplicate styles
- Fixed native packages not containing the stdlib and botPolicies.yaml
- Change import syntax to allow multi-level imports
- Changed the startup logging to use JSON formatting as all the other logs do.
- Added the ability to do [expression matching with CEL](./admin/configuration/expressions.mdx)
- Add a warning for clients that don't store cookies
- Disable Open Graph passthrough by default ([#435](https://github.com/TecharoHQ/anubis/issues/435))
- Clarify the license of the mascot images ([#442](https://github.com/TecharoHQ/anubis/issues/442))
- Started Suppressing 'Context canceled' errors from http in the logs ([#446](https://github.com/TecharoHQ/anubis/issues/446))

Signed-off-by: Xe Iaso <me@xeiaso.net>

.github/FUNDING.yml

@@ -1 +1,2 @@
-patreon: cadey
+patreon: cadey
+github: xe

.github/workflows/docker-pr.yml

@@ -12,10 +12,10 @@ permissions:
 
 jobs:
   build:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     steps:
       - name: Checkout code
-        uses: actions/checkout@v4
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
         with:
           fetch-tags: true
           fetch-depth: 0
@@ -25,7 +25,7 @@ jobs:
         uses: Homebrew/actions/setup-homebrew@master
 
       - name: Setup Homebrew cellar cache
-        uses: actions/cache@v4
+        uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
         with:
           path: |
             /home/linuxbrew/.linuxbrew/Cellar
@@ -47,7 +47,7 @@ jobs:
 
       - name: Docker meta
         id: meta
-        uses: docker/metadata-action@v5
+        uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 # v5.7.0
         with:
           images: ghcr.io/techarohq/anubis
 

.github/workflows/docker.yml

@@ -18,10 +18,10 @@ permissions:
 
 jobs:
   build:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     steps:
       - name: Checkout code
-        uses: actions/checkout@v4
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
         with:
           fetch-tags: true
           fetch-depth: 0
@@ -31,7 +31,7 @@ jobs:
         uses: Homebrew/actions/setup-homebrew@master
 
       - name: Setup Homebrew cellar cache
-        uses: actions/cache@v4
+        uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
         with:
           path: |
             /home/linuxbrew/.linuxbrew/Cellar
@@ -52,7 +52,7 @@ jobs:
           brew bundle
 
       - name: Log into registry 
-        uses: docker/login-action@v3
+        uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
         with:
           registry: ghcr.io
           username: techarohq
@@ -60,7 +60,7 @@ jobs:
 
       - name: Docker meta
         id: meta
-        uses: docker/metadata-action@v5
+        uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 # v5.7.0
         with:
           images: ghcr.io/techarohq/anubis
 
@@ -74,8 +74,8 @@ jobs:
           SLOG_LEVEL: debug
       
       - name: Generate artifact attestation
-        uses: actions/attest-build-provenance@v2
+        uses: actions/attest-build-provenance@db473fddc028af60658334401dc6fa3ffd8669fd # v2.3.0
         with:
           subject-name: ghcr.io/techarohq/anubis
           subject-digest: ${{ steps.build.outputs.digest }}
-          push-to-registry: true
+          push-to-registry: true

.github/workflows/docs-deploy.yml

@@ -13,18 +13,18 @@ permissions:
 
 jobs:
   build:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
 
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
         with:
           persist-credentials: false
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
 
       - name: Log into registry 
-        uses: docker/login-action@v3
+        uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
         with:
           registry: ghcr.io
           username: techarohq
@@ -32,13 +32,13 @@ jobs:
 
       - name: Docker meta
         id: meta
-        uses: docker/metadata-action@v5
+        uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 # v5.7.0
         with:
           images: ghcr.io/techarohq/anubis/docs
 
       - name: Build and push
         id: build
-        uses: docker/build-push-action@v6
+        uses: docker/build-push-action@14487ce63c7a62a4a324b0bfb37086795e31c6c1 # v6.16.0
         with:
           context: ./docs
           cache-to: type=gha
@@ -49,14 +49,14 @@ jobs:
           push: true
 
       - name: Apply k8s manifests to aeacus
-        uses: actions-hub/kubectl@master
+        uses: actions-hub/kubectl@e81783053d902f50d752d21a6d99cf9689a652e1 # v1.33.0
         env:
           KUBE_CONFIG: ${{ secrets.AEACUS_KUBECONFIG }}
         with:
           args: apply -k docs/manifest
 
       - name: Apply k8s manifests to aeacus
-        uses: actions-hub/kubectl@master
+        uses: actions-hub/kubectl@e81783053d902f50d752d21a6d99cf9689a652e1 # v1.33.0
         env:
           KUBE_CONFIG: ${{ secrets.AEACUS_KUBECONFIG }}
         with:

.github/workflows/docs-test.yml

@@ -0,0 +1,39 @@
+name: Docs test build
+
+on:
+  pull_request:
+    branches: [ "main" ]
+
+permissions:
+  contents: read
+  actions: write
+
+jobs:
+  build:
+    runs-on: ubuntu-24.04
+
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        with:
+          persist-credentials: false
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@902fa8ec7d6ecbf8d84d538b9b233a880e428804 # v5.7.0
+        with:
+          images: ghcr.io/techarohq/anubis/docs
+
+      - name: Build and push
+        id: build
+        uses: docker/build-push-action@14487ce63c7a62a4a324b0bfb37086795e31c6c1 # v6.16.0
+        with:
+          context: ./docs
+          cache-to: type=gha
+          cache-from: type=gha
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          platforms: linux/amd64
+          push: false

.github/workflows/go.yml

@@ -13,9 +13,9 @@ permissions:
 jobs:
   go_tests:
     #runs-on: alrest-techarohq
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         persist-credentials: false
 
@@ -28,7 +28,7 @@ jobs:
       uses: Homebrew/actions/setup-homebrew@master
 
     - name: Setup Homebrew cellar cache
-      uses: actions/cache@v4
+      uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
       with:
         path: |
           /home/linuxbrew/.linuxbrew/Cellar
@@ -49,7 +49,7 @@ jobs:
         brew bundle
 
     - name: Setup Golang caches
-      uses: actions/cache@v4
+      uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
       with:
         path: |
           ~/.cache/go-build
@@ -59,22 +59,21 @@ jobs:
           ${{ runner.os }}-golang-
 
     - name: Cache playwright binaries
-      uses: actions/cache@v4
+      uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
       id: playwright-cache
       with:
         path: |
           ~/.cache/ms-playwright
         key: ${{ runner.os }}-playwright-${{ hashFiles('**/go.sum') }}
 
-    - name: install playwright browsers
-      run: |
-        npx --yes playwright@1.50.1 install --with-deps
-        npx --yes playwright@1.50.1 run-server --port 9001 &
-
     - name: install node deps
       run: |
         npm ci
-        npm run assets
+
+    - name: install playwright browsers
+      run: |
+        npx --no-install playwright@1.52.0 install --with-deps
+        npx --no-install playwright@1.52.0 run-server --port 9001 &
 
     - name: Build
       run: npm run build
@@ -82,6 +81,11 @@ jobs:
     - name: Test
       run: npm run test
 
-    - uses: dominikh/staticcheck-action@v1
+    - name: Lint with staticcheck
+      uses: dominikh/staticcheck-action@fe1dd0c3658873b46f8c9bb3291096a617310ca6 # v1.3.1
       with:
         version: "latest"
+
+    - name: Govulncheck
+      run: |
+        go tool govulncheck ./...

.github/workflows/package-builds-stable.yml

@@ -11,9 +11,9 @@ permissions:
 jobs:
   package_builds:
     #runs-on: alrest-techarohq
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         persist-credentials: false
         fetch-tags: true
@@ -28,7 +28,7 @@ jobs:
       uses: Homebrew/actions/setup-homebrew@master
 
     - name: Setup Homebrew cellar cache
-      uses: actions/cache@v4
+      uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
       with:
         path: |
           /home/linuxbrew/.linuxbrew/Cellar
@@ -49,7 +49,7 @@ jobs:
         brew bundle
 
     - name: Setup Golang caches
-      uses: actions/cache@v4
+      uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
       with:
         path: |
           ~/.cache/go-build
@@ -64,8 +64,9 @@ jobs:
 
     - name: Build Packages
       run: |
-        wget https://github.com/Xe/x/releases/download/v1.13.4/yeet_1.13.4_amd64.deb -O var/yeet.deb
+        wget https://github.com/TecharoHQ/yeet/releases/download/v0.2.2/yeet_0.2.2_amd64.deb -O var/yeet.deb
         sudo apt -y install -f ./var/yeet.deb
+        rm ./var/yeet.deb
         yeet
 
     - name: Upload released artifacts
@@ -78,4 +79,4 @@ jobs:
         cd var
         for file in *; do
           gh release upload $RELEASE $file
-        done
+        done

.github/workflows/package-builds-unstable.yml

@@ -13,9 +13,9 @@ permissions:
 jobs:
   package_builds:
     #runs-on: alrest-techarohq
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
       with:
         persist-credentials: false
         fetch-tags: true
@@ -30,7 +30,7 @@ jobs:
       uses: Homebrew/actions/setup-homebrew@master
 
     - name: Setup Homebrew cellar cache
-      uses: actions/cache@v4
+      uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
       with:
         path: |
           /home/linuxbrew/.linuxbrew/Cellar
@@ -51,7 +51,7 @@ jobs:
         brew bundle
 
     - name: Setup Golang caches
-      uses: actions/cache@v4
+      uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
       with:
         path: |
           ~/.cache/go-build
@@ -66,11 +66,12 @@ jobs:
 
     - name: Build Packages
       run: |
-        wget https://github.com/Xe/x/releases/download/v1.13.4/yeet_1.13.4_amd64.deb -O var/yeet.deb
+        wget https://github.com/TecharoHQ/yeet/releases/download/v0.2.2/yeet_0.2.2_amd64.deb -O var/yeet.deb
         sudo apt -y install -f ./var/yeet.deb
+        rm ./var/yeet.deb
         yeet
 
-    - uses: actions/upload-artifact@v4
+    - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
       with:
         name: packages
-        path: var/*
+        path: var/*

.github/workflows/zizmor.yml

@@ -11,17 +11,17 @@ on:
 jobs:
   zizmor:
     name: zizmor latest via PyPI
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     permissions:
       security-events: write
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v4
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
         with:
           persist-credentials: false
 
       - name: Install the latest version of uv
-        uses: astral-sh/setup-uv@v5
+        uses: astral-sh/setup-uv@6b9c6063abd6010835644d4c2e1bef4cf5cd0fca # v6.0.1
 
       - name: Run zizmor 🌈
         run: uvx zizmor --format sarif . > results.sarif 
@@ -29,7 +29,7 @@ jobs:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} 
 
       - name: Upload SARIF file
-        uses: github/codeql-action/upload-sarif@v3
+        uses: github/codeql-action/upload-sarif@60168efe1c415ce0f5521ea06d5c2062adbeed1b # v3.28.17
         with:
           sarif_file: results.sarif
           category: zizmor

.github/zizmor.yml

@@ -0,0 +1,5 @@
+rules:
+  unpinned-uses:
+    config:
+      policies:
+        Homebrew/actions/*: any

.vscode/settings.json

@@ -0,0 +1,15 @@
+{
+  "github.copilot.enable": {
+    "*": false,
+    "plaintext": false,
+    "markdown": false,
+    "mdx": false,
+    "json": false,
+    "scminput": false,
+    "yaml": false,
+    "go": false,
+    "zig": false,
+    "javascript": false,
+    "properties": false
+  }
+}

Makefile

@@ -1,27 +1,32 @@
-NODE_MODULES = node_modules
-VERSION := $(shell cat ./VERSION)
+VERSION= $(shell cat ./VERSION)
+GO?= go
+NPM?= npm
 
 .PHONY: build assets deps lint prebaked-build test
 
-assets:
-	npm run assets
-
-deps:
-	npm ci
-	go mod download
-
-build: deps
-	npm run build
-	@echo "Anubis is now built to ./var/anubis"
-
 all: build
 
-lint:
-	go vet ./...
-	go tool staticcheck ./...
+deps:
+	$(NPM) ci
+	$(GO) mod download
+
+assets: PATH:=$(PWD)/node_modules/.bin:$(PATH)
+assets: deps
+	$(GO) generate ./...
+	./web/build.sh
+	./xess/build.sh
+
+build: assets
+	$(GO) build -o ./var/anubis ./cmd/anubis
+	@echo "Anubis is now built to ./var/anubis"
+
+lint: assets
+	$(GO) vet ./...
+	$(GO) tool staticcheck ./...
+	$(GO) tool govulncheck ./...
 
 prebaked-build:
-	go build -o ./var/anubis -ldflags "-X 'github.com/TecharoHQ/anubis.Version=$(VERSION)'" ./cmd/anubis
+	$(GO) build -o ./var/anubis -ldflags "-X 'github.com/TecharoHQ/anubis.Version=$(VERSION)'" ./cmd/anubis
 
-test:
-	npm run test
+test: assets
+	$(GO) test ./...

README.md

@@ -10,11 +10,21 @@
 ![language count](https://img.shields.io/github/languages/count/TecharoHQ/anubis)
 ![repo size](https://img.shields.io/github/repo-size/TecharoHQ/anubis)
 
-Anubis [weighs the soul of your connection](https://en.wikipedia.org/wiki/Weighing_of_souls) using a sha256 proof-of-work challenge in order to protect upstream resources from scraper bots.
+## Sponsors
 
-Installing and using this will likely result in your website not being indexed by some search engines. This is considered a feature of Anubis, not a bug.
+Anubis is brought to you by sponsors and donors like:
 
-This is a bit of a nuclear response, but AI scraper bots scraping so aggressively have forced my hand. I hate that I have to do this, but this is what we get for the modern Internet because bots don't conform to standards like robots.txt, even when they claim to.
+[![Distrust](./docs/static/img/sponsors/distrust-logo.webp)](https://distrust.co)
+[![Terminal Trove](./docs/static/img/sponsors/terminal-trove.webp)](https://terminaltrove.com/?utm_campaign=github&utm_medium=referral&utm_content=anubis&utm_source=abgh)
+[![canine.tools](./docs/static/img/sponsors/caninetools-logo.webp)](https://canine.tools)
+
+## Overview
+
+Anubis [weighs the soul of your connection](https://en.wikipedia.org/wiki/Weighing_of_souls) using a proof-of-work challenge in order to protect upstream resources from scraper bots.
+
+This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.
+
+Anubis is a bit of a nuclear response. This will result in your website being blocked from smaller scrapers and may inhibit "good bots" like the Internet Archive. You can configure [bot policy definitions](./docs/docs/admin/policies.mdx) to explicitly allowlist them and we are working on a curated set of "known good" bots to allow for a compromise between discoverability and uptime.
 
 In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you.
 
@@ -28,8 +38,22 @@ For live chat, please join the [Patreon](https://patreon.com/cadey) and ask in t
 
 ## Star History
 
-[![Star History Chart](https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date)](https://www.star-history.com/#TecharoHQ/anubis&Date)
+<a href="https://www.star-history.com/#TecharoHQ/anubis&Date">
+ <picture>
+   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date&theme=dark" />
+   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date" />
+   <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date" />
+ </picture>
+</a>
 
 ## Packaging Status
 
-[![Packaging status](https://repology.org/badge/vertical-allrepos/anubis-anti-crawler.svg)](https://repology.org/project/anubis-anti-crawler/versions)
+[![Packaging status](https://repology.org/badge/vertical-allrepos/anubis-anti-crawler.svg?columns=3)](https://repology.org/project/anubis-anti-crawler/versions)
+
+## Contributors
+
+<a href="https://github.com/TecharoHQ/anubis/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=TecharoHQ/anubis" />
+</a>
+
+Made with [contrib.rocks](https://contrib.rocks).

VERSION

@@ -1 +1 @@
-1.15.1
+1.18.0

anubis.go

@@ -1,6 +1,8 @@
-// Package Anubis contains the version number of Anubis.
+// Package anubis contains the version number of Anubis.
 package anubis
 
+import "time"
+
 // Version is the current version of Anubis.
 //
 // This variable is set at build time using the -X linker flag. If not set,
@@ -11,9 +13,18 @@ var Version = "devel"
 // access.
 const CookieName = "within.website-x-cmd-anubis-auth"
 
+// CookieDefaultExpirationTime is the amount of time before the cookie/JWT expires.
+const CookieDefaultExpirationTime = 7 * 24 * time.Hour
+
+// BasePrefix is a global prefix for all Anubis endpoints. Can be emptied to remove the prefix entirely.
+var BasePrefix = ""
+
 // StaticPath is the location where all static Anubis assets are located.
 const StaticPath = "/.within.website/x/cmd/anubis/"
 
+// APIPrefix is the location where all Anubis API endpoints are located.
+const APIPrefix = "/.within.website/x/cmd/anubis/api/"
+
 // DefaultDifficulty is the default "difficulty" (number of leading zeroes)
 // that must be met by the client in order to pass the challenge.
 const DefaultDifficulty = 4

cmd/anubis/main.go

@@ -20,7 +20,6 @@ import (
 	"os"
 	"os/signal"
 	"path/filepath"
-	"regexp"
 	"strconv"
 	"strings"
 	"sync"
@@ -28,6 +27,7 @@ import (
 	"time"
 
 	"github.com/TecharoHQ/anubis"
+	"github.com/TecharoHQ/anubis/data"
 	"github.com/TecharoHQ/anubis/internal"
 	libanubis "github.com/TecharoHQ/anubis/lib"
 	botPolicy "github.com/TecharoHQ/anubis/lib/policy"
@@ -38,10 +38,12 @@ import (
 )
 
 var (
+	basePrefix               = flag.String("base-prefix", "", "base prefix (root URL) the application is served under e.g. /myapp")
 	bind                     = flag.String("bind", ":8923", "network address to bind HTTP to")
 	bindNetwork              = flag.String("bind-network", "tcp", "network family to bind HTTP to, e.g. unix, tcp")
 	challengeDifficulty      = flag.Int("difficulty", anubis.DefaultDifficulty, "difficulty of the challenge")
 	cookieDomain             = flag.String("cookie-domain", "", "if set, the top-level domain that the Anubis cookie will be valid for")
+	cookieExpiration         = flag.Duration("cookie-expiration-time", anubis.CookieDefaultExpirationTime, "The amount of time the authorization cookie is valid for")
 	cookiePartitioned        = flag.Bool("cookie-partitioned", false, "if true, sets the partitioned flag on Anubis cookies, enabling CHIPS support")
 	ed25519PrivateKeyHex     = flag.String("ed25519-private-key-hex", "", "private key used to sign JWTs, if not set a random one will be assigned")
 	ed25519PrivateKeyHexFile = flag.String("ed25519-private-key-hex-file", "", "file name containing value for ed25519-private-key-hex")
@@ -50,15 +52,17 @@ var (
 	socketMode               = flag.String("socket-mode", "0770", "socket mode (permissions) for unix domain sockets.")
 	robotsTxt                = flag.Bool("serve-robots-txt", false, "serve a robots.txt file that disallows all robots")
 	policyFname              = flag.String("policy-fname", "", "full path to anubis policy document (defaults to a sensible built-in policy)")
+	redirectDomains          = flag.String("redirect-domains", "", "list of domains separated by commas which anubis is allowed to redirect to. Leaving this unset allows any domain.")
 	slogLevel                = flag.String("slog-level", "INFO", "logging level (see https://pkg.go.dev/log/slog#hdr-Levels)")
-	target                   = flag.String("target", "http://localhost:3923", "target to reverse proxy to")
+	target                   = flag.String("target", "http://localhost:3923", "target to reverse proxy to, set to an empty string to disable proxying when only using auth request")
 	healthcheck              = flag.Bool("healthcheck", false, "run a health check against Anubis")
 	useRemoteAddress         = flag.Bool("use-remote-address", false, "read the client's IP address from the network request, useful for debugging and running Anubis on bare metal")
 	debugBenchmarkJS         = flag.Bool("debug-benchmark-js", false, "respond to every request with a challenge for benchmarking hashrate")
 	ogPassthrough            = flag.Bool("og-passthrough", false, "enable Open Graph tag passthrough")
 	ogTimeToLive             = flag.Duration("og-expiry-time", 24*time.Hour, "Open Graph tag cache expiration time")
+	ogCacheConsiderHost      = flag.Bool("og-cache-consider-host", false, "enable or disable the use of the host in the Open Graph tag cache")
 	extractResources         = flag.String("extract-resources", "", "if set, extract the static resources to the specified folder")
-	webmasterEmail		 = flag.String("webmaster-email", "", "if set, displays webmaster's email on the reject page for appeals")
+	webmasterEmail           = flag.String("webmaster-email", "", "if set, displays webmaster's email on the reject page for appeals")
 )
 
 func keyFromHex(value string) (ed25519.PrivateKey, error) {
@@ -75,7 +79,7 @@ func keyFromHex(value string) (ed25519.PrivateKey, error) {
 }
 
 func doHealthCheck() error {
-	resp, err := http.Get("http://localhost" + *metricsBind + "/metrics")
+	resp, err := http.Get("http://localhost" + *metricsBind + anubis.BasePrefix + "/metrics")
 	if err != nil {
 		return fmt.Errorf("failed to fetch metrics: %w", err)
 	}
@@ -118,7 +122,10 @@ func setupListener(network string, address string) (net.Listener, string) {
 
 		err = os.Chmod(address, os.FileMode(mode))
 		if err != nil {
-			listener.Close()
+			err := listener.Close()
+			if err != nil {
+				log.Printf("failed to close listener: %v", err)
+			}
 			log.Fatal(fmt.Errorf("could not change socket mode: %w", err))
 		}
 	}
@@ -174,14 +181,10 @@ func main() {
 
 	internal.InitSlog(*slogLevel)
 
-	if *healthcheck {
-		if err := doHealthCheck(); err != nil {
+	if *extractResources != "" {
+		if err := extractEmbedFS(data.BotPolicies, ".", *extractResources); err != nil {
 			log.Fatal(err)
 		}
-		return
-	}
-
-	if *extractResources != "" {
 		if err := extractEmbedFS(web.Static, "static", *extractResources); err != nil {
 			log.Fatal(err)
 		}
@@ -189,9 +192,14 @@ func main() {
 		return
 	}
 
-	rp, err := makeReverseProxy(*target)
-	if err != nil {
-		log.Fatalf("can't make reverse proxy: %v", err)
+	var rp http.Handler
+	// when using anubis via Systemd and environment variables, then it is not possible to set targe to an empty string but only to space
+	if strings.TrimSpace(*target) != "" {
+		var err error
+		rp, err = makeReverseProxy(*target)
+		if err != nil {
+			log.Fatalf("can't make reverse proxy: %v", err)
+		}
 	}
 
 	policy, err := libanubis.LoadPoliciesOrDefault(*policyFname, *challengeDifficulty)
@@ -199,30 +207,29 @@ func main() {
 		log.Fatalf("can't parse policy file: %v", err)
 	}
 
-	fmt.Println("Rule error IDs:")
+	ruleErrorIDs := make(map[string]string)
 	for _, rule := range policy.Bots {
 		if rule.Action != config.RuleDeny {
 			continue
 		}
 
-		hash, err := rule.Hash()
-		if err != nil {
-			log.Fatalf("can't calculate checksum of rule %s: %v", rule.Name, err)
-		}
-
-		fmt.Printf("* %s: %s\n", rule.Name, hash)
+		hash := rule.Hash()
+		ruleErrorIDs[rule.Name] = hash
 	}
-	fmt.Println()
 
 	// replace the bot policy rules with a single rule that always benchmarks
 	if *debugBenchmarkJS {
-		userAgent := regexp.MustCompile(".")
 		policy.Bots = []botPolicy.Bot{{
-			Name:      "",
-			UserAgent: userAgent,
-			Action:    config.RuleBenchmark,
+			Name:   "",
+			Rules:  botPolicy.NewHeaderExistsChecker("User-Agent"),
+			Action: config.RuleBenchmark,
 		}}
 	}
+	if *basePrefix != "" && !strings.HasPrefix(*basePrefix, "/") {
+		log.Fatalf("[misconfiguration] base-prefix must start with a slash, eg: /%s", *basePrefix)
+	} else if strings.HasSuffix(*basePrefix, "/") {
+		log.Fatalf("[misconfiguration] base-prefix must not end with a slash")
+	}
 
 	var priv ed25519.PrivateKey
 	if *ed25519PrivateKeyHex != "" && *ed25519PrivateKeyHexFile != "" {
@@ -233,12 +240,12 @@ func main() {
 			log.Fatalf("failed to parse and validate ED25519_PRIVATE_KEY_HEX: %v", err)
 		}
 	} else if *ed25519PrivateKeyHexFile != "" {
-		hex, err := os.ReadFile(*ed25519PrivateKeyHexFile)
+		hexFile, err := os.ReadFile(*ed25519PrivateKeyHexFile)
 		if err != nil {
 			log.Fatalf("failed to read ED25519_PRIVATE_KEY_HEX_FILE %s: %v", *ed25519PrivateKeyHexFile, err)
 		}
 
-		priv, err = keyFromHex(string(bytes.TrimSpace(hex)))
+		priv, err = keyFromHex(string(bytes.TrimSpace(hexFile)))
 		if err != nil {
 			log.Fatalf("failed to parse and validate content of ED25519_PRIVATE_KEY_HEX_FILE: %v", err)
 		}
@@ -251,17 +258,35 @@ func main() {
 		slog.Warn("generating random key, Anubis will have strange behavior when multiple instances are behind the same load balancer target, for more information: see https://anubis.techaro.lol/docs/admin/installation#key-generation")
 	}
 
+	var redirectDomainsList []string
+	if *redirectDomains != "" {
+		domains := strings.Split(*redirectDomains, ",")
+		for _, domain := range domains {
+			_, err = url.Parse(domain)
+			if err != nil {
+				log.Fatalf("cannot parse redirect-domain %q: %s", domain, err.Error())
+			}
+			redirectDomainsList = append(redirectDomainsList, strings.TrimSpace(domain))
+		}
+	} else {
+		slog.Warn("REDIRECT_DOMAINS is not set, Anubis will only redirect to the same domain a request is coming from, see https://anubis.techaro.lol/docs/admin/configuration/redirect-domains")
+	}
+
 	s, err := libanubis.New(libanubis.Options{
-		Next:              rp,
-		Policy:            policy,
-		ServeRobotsTXT:    *robotsTxt,
-		PrivateKey:        priv,
-		CookieDomain:      *cookieDomain,
-		CookiePartitioned: *cookiePartitioned,
-		OGPassthrough:     *ogPassthrough,
-		OGTimeToLive:      *ogTimeToLive,
-		Target:            *target,
-		WebmasterEmail:     *webmasterEmail,
+		BasePrefix:           *basePrefix,
+		Next:                 rp,
+		Policy:               policy,
+		ServeRobotsTXT:       *robotsTxt,
+		PrivateKey:           priv,
+		CookieDomain:         *cookieDomain,
+		CookieExpiration:     *cookieExpiration,
+		CookiePartitioned:    *cookiePartitioned,
+		OGPassthrough:        *ogPassthrough,
+		OGTimeToLive:         *ogTimeToLive,
+		RedirectDomains:      redirectDomainsList,
+		Target:               *target,
+		WebmasterEmail:       *webmasterEmail,
+		OGCacheConsidersHost: *ogCacheConsiderHost,
 	})
 	if err != nil {
 		log.Fatalf("can't construct libanubis.Server: %v", err)
@@ -276,15 +301,15 @@ func main() {
 		wg.Add(1)
 		go metricsServer(ctx, wg.Done)
 	}
-
 	go startDecayMapCleanup(ctx, s)
 
 	var h http.Handler
 	h = s
 	h = internal.RemoteXRealIP(*useRemoteAddress, *bindNetwork, h)
 	h = internal.XForwardedForToXRealIP(h)
+	h = internal.XForwardedForUpdate(h)
 
-	srv := http.Server{Handler: h}
+	srv := http.Server{Handler: h, ErrorLog: internal.GetFilteredHTTPLogger()}
 	listener, listenerUrl := setupListener(*bindNetwork, *bind)
 	slog.Info(
 		"listening",
@@ -297,6 +322,9 @@ func main() {
 		"debug-benchmark-js", *debugBenchmarkJS,
 		"og-passthrough", *ogPassthrough,
 		"og-expiry-time", *ogTimeToLive,
+		"base-prefix", *basePrefix,
+		"cookie-expiration-time", *cookieExpiration,
+		"rule-error-ids", ruleErrorIDs,
 	)
 
 	go func() {
@@ -318,12 +346,20 @@ func metricsServer(ctx context.Context, done func()) {
 	defer done()
 
 	mux := http.NewServeMux()
-	mux.Handle("/metrics", promhttp.Handler())
+	mux.Handle(anubis.BasePrefix+"/metrics", promhttp.Handler())
 
-	srv := http.Server{Handler: mux}
+	srv := http.Server{Handler: mux, ErrorLog: internal.GetFilteredHTTPLogger()}
 	listener, metricsUrl := setupListener(*metricsBindNetwork, *metricsBind)
 	slog.Debug("listening for metrics", "url", metricsUrl)
 
+	if *healthcheck {
+		log.Println("running healthcheck")
+		if err := doHealthCheck(); err != nil {
+			log.Fatal(err)
+		}
+		return
+	}
+
 	go func() {
 		<-ctx.Done()
 		c, cancel := context.WithTimeout(context.Background(), 5*time.Second)
@@ -349,7 +385,7 @@ func extractEmbedFS(fsys embed.FS, root string, destDir string) error {
 			return err
 		}
 
-		destPath := filepath.Join(destDir, relPath)
+		destPath := filepath.Join(destDir, root, relPath)
 
 		if d.IsDir() {
 			return os.MkdirAll(destPath, 0o700)

cmd/containerbuild/main.go

@@ -123,15 +123,15 @@ func parseImageList(imageList string) ([]image, error) {
 		// reg.xeiaso.net/techaro/anubis:latest
 		// repository: reg.xeiaso.net/techaro/anubis
 		// tag:        latest
-		parts := strings.SplitN(img, ":", 2)
+		index := strings.LastIndex(img, ":")
 		result = append(result, image{
-			repository: parts[0],
-			tag:        parts[1],
+			repository: img[:index],
+			tag:        img[index+1:],
 		})
 	}
 
 	if len(result) == 0 {
-		return nil, fmt.Errorf("no images provided, bad flags??")
+		return nil, fmt.Errorf("no images provided, bad flags")
 	}
 
 	return result, nil

data/apps/allow-api-routes.yaml

@@ -0,0 +1,6 @@
+- name: allow-api-routes
+  action: ALLOW
+  expression:
+    all:
+    - '!(method == "HEAD" || method == "GET")'
+    - path.startsWith("/api/")

data/apps/gitea-rss-feeds.yaml

@@ -0,0 +1,7 @@
+# By Aibrew: https://github.com/TecharoHQ/anubis/discussions/261#discussioncomment-12821065
+- name: gitea-feed-atom
+  action: ALLOW
+  path_regex: ^/[.A-Za-z0-9_-]{1,256}?[./A-Za-z0-9_-]*\.atom$
+- name: gitea-feed-rss
+  action: ALLOW
+  path_regex: ^/[.A-Za-z0-9_-]{1,256}?[./A-Za-z0-9_-]*\.rss$

data/botPolicies.json

@@ -1,398 +1,29 @@
 {
   "bots": [
     {
-      "name": "amazonbot",
-      "user_agent_regex": "Amazonbot",
-      "action": "DENY"
+      "import": "(data)/bots/_deny-pathological.yaml"
     },
     {
-      "name": "googlebot",
-      "user_agent_regex": "\\+http\\://www\\.google\\.com/bot\\.html",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "2001:4860:4801:10::/64",
-        "2001:4860:4801:11::/64",
-        "2001:4860:4801:12::/64",
-        "2001:4860:4801:13::/64",
-        "2001:4860:4801:14::/64",
-        "2001:4860:4801:15::/64",
-        "2001:4860:4801:16::/64",
-        "2001:4860:4801:17::/64",
-        "2001:4860:4801:18::/64",
-        "2001:4860:4801:19::/64",
-        "2001:4860:4801:1a::/64",
-        "2001:4860:4801:1b::/64",
-        "2001:4860:4801:1c::/64",
-        "2001:4860:4801:1d::/64",
-        "2001:4860:4801:1e::/64",
-        "2001:4860:4801:1f::/64",
-        "2001:4860:4801:20::/64",
-        "2001:4860:4801:21::/64",
-        "2001:4860:4801:22::/64",
-        "2001:4860:4801:23::/64",
-        "2001:4860:4801:24::/64",
-        "2001:4860:4801:25::/64",
-        "2001:4860:4801:26::/64",
-        "2001:4860:4801:27::/64",
-        "2001:4860:4801:28::/64",
-        "2001:4860:4801:29::/64",
-        "2001:4860:4801:2::/64",
-        "2001:4860:4801:2a::/64",
-        "2001:4860:4801:2b::/64",
-        "2001:4860:4801:2c::/64",
-        "2001:4860:4801:2d::/64",
-        "2001:4860:4801:2e::/64",
-        "2001:4860:4801:2f::/64",
-        "2001:4860:4801:31::/64",
-        "2001:4860:4801:32::/64",
-        "2001:4860:4801:33::/64",
-        "2001:4860:4801:34::/64",
-        "2001:4860:4801:35::/64",
-        "2001:4860:4801:36::/64",
-        "2001:4860:4801:37::/64",
-        "2001:4860:4801:38::/64",
-        "2001:4860:4801:39::/64",
-        "2001:4860:4801:3a::/64",
-        "2001:4860:4801:3b::/64",
-        "2001:4860:4801:3c::/64",
-        "2001:4860:4801:3d::/64",
-        "2001:4860:4801:3e::/64",
-        "2001:4860:4801:40::/64",
-        "2001:4860:4801:41::/64",
-        "2001:4860:4801:42::/64",
-        "2001:4860:4801:43::/64",
-        "2001:4860:4801:44::/64",
-        "2001:4860:4801:45::/64",
-        "2001:4860:4801:46::/64",
-        "2001:4860:4801:47::/64",
-        "2001:4860:4801:48::/64",
-        "2001:4860:4801:49::/64",
-        "2001:4860:4801:4a::/64",
-        "2001:4860:4801:4b::/64",
-        "2001:4860:4801:4c::/64",
-        "2001:4860:4801:50::/64",
-        "2001:4860:4801:51::/64",
-        "2001:4860:4801:52::/64",
-        "2001:4860:4801:53::/64",
-        "2001:4860:4801:54::/64",
-        "2001:4860:4801:55::/64",
-        "2001:4860:4801:56::/64",
-        "2001:4860:4801:60::/64",
-        "2001:4860:4801:61::/64",
-        "2001:4860:4801:62::/64",
-        "2001:4860:4801:63::/64",
-        "2001:4860:4801:64::/64",
-        "2001:4860:4801:65::/64",
-        "2001:4860:4801:66::/64",
-        "2001:4860:4801:67::/64",
-        "2001:4860:4801:68::/64",
-        "2001:4860:4801:69::/64",
-        "2001:4860:4801:6a::/64",
-        "2001:4860:4801:6b::/64",
-        "2001:4860:4801:6c::/64",
-        "2001:4860:4801:6d::/64",
-        "2001:4860:4801:6e::/64",
-        "2001:4860:4801:6f::/64",
-        "2001:4860:4801:70::/64",
-        "2001:4860:4801:71::/64",
-        "2001:4860:4801:72::/64",
-        "2001:4860:4801:73::/64",
-        "2001:4860:4801:74::/64",
-        "2001:4860:4801:75::/64",
-        "2001:4860:4801:76::/64",
-        "2001:4860:4801:77::/64",
-        "2001:4860:4801:78::/64",
-        "2001:4860:4801:79::/64",
-        "2001:4860:4801:80::/64",
-        "2001:4860:4801:81::/64",
-        "2001:4860:4801:82::/64",
-        "2001:4860:4801:83::/64",
-        "2001:4860:4801:84::/64",
-        "2001:4860:4801:85::/64",
-        "2001:4860:4801:86::/64",
-        "2001:4860:4801:87::/64",
-        "2001:4860:4801:88::/64",
-        "2001:4860:4801:90::/64",
-        "2001:4860:4801:91::/64",
-        "2001:4860:4801:92::/64",
-        "2001:4860:4801:93::/64",
-        "2001:4860:4801:94::/64",
-        "2001:4860:4801:95::/64",
-        "2001:4860:4801:96::/64",
-        "2001:4860:4801:a0::/64",
-        "2001:4860:4801:a1::/64",
-        "2001:4860:4801:a2::/64",
-        "2001:4860:4801:a3::/64",
-        "2001:4860:4801:a4::/64",
-        "2001:4860:4801:a5::/64",
-        "2001:4860:4801:c::/64",
-        "2001:4860:4801:f::/64",
-        "192.178.5.0/27",
-        "192.178.6.0/27",
-        "192.178.6.128/27",
-        "192.178.6.160/27",
-        "192.178.6.192/27",
-        "192.178.6.32/27",
-        "192.178.6.64/27",
-        "192.178.6.96/27",
-        "34.100.182.96/28",
-        "34.101.50.144/28",
-        "34.118.254.0/28",
-        "34.118.66.0/28",
-        "34.126.178.96/28",
-        "34.146.150.144/28",
-        "34.147.110.144/28",
-        "34.151.74.144/28",
-        "34.152.50.64/28",
-        "34.154.114.144/28",
-        "34.155.98.32/28",
-        "34.165.18.176/28",
-        "34.175.160.64/28",
-        "34.176.130.16/28",
-        "34.22.85.0/27",
-        "34.64.82.64/28",
-        "34.65.242.112/28",
-        "34.80.50.80/28",
-        "34.88.194.0/28",
-        "34.89.10.80/28",
-        "34.89.198.80/28",
-        "34.96.162.48/28",
-        "35.247.243.240/28",
-        "66.249.64.0/27",
-        "66.249.64.128/27",
-        "66.249.64.160/27",
-        "66.249.64.224/27",
-        "66.249.64.32/27",
-        "66.249.64.64/27",
-        "66.249.64.96/27",
-        "66.249.65.0/27",
-        "66.249.65.128/27",
-        "66.249.65.160/27",
-        "66.249.65.192/27",
-        "66.249.65.224/27",
-        "66.249.65.32/27",
-        "66.249.65.64/27",
-        "66.249.65.96/27",
-        "66.249.66.0/27",
-        "66.249.66.128/27",
-        "66.249.66.160/27",
-        "66.249.66.192/27",
-        "66.249.66.224/27",
-        "66.249.66.32/27",
-        "66.249.66.64/27",
-        "66.249.66.96/27",
-        "66.249.68.0/27",
-        "66.249.68.128/27",
-        "66.249.68.32/27",
-        "66.249.68.64/27",
-        "66.249.68.96/27",
-        "66.249.69.0/27",
-        "66.249.69.128/27",
-        "66.249.69.160/27",
-        "66.249.69.192/27",
-        "66.249.69.224/27",
-        "66.249.69.32/27",
-        "66.249.69.64/27",
-        "66.249.69.96/27",
-        "66.249.70.0/27",
-        "66.249.70.128/27",
-        "66.249.70.160/27",
-        "66.249.70.192/27",
-        "66.249.70.224/27",
-        "66.249.70.32/27",
-        "66.249.70.64/27",
-        "66.249.70.96/27",
-        "66.249.71.0/27",
-        "66.249.71.128/27",
-        "66.249.71.160/27",
-        "66.249.71.192/27",
-        "66.249.71.224/27",
-        "66.249.71.32/27",
-        "66.249.71.64/27",
-        "66.249.71.96/27",
-        "66.249.72.0/27",
-        "66.249.72.128/27",
-        "66.249.72.160/27",
-        "66.249.72.192/27",
-        "66.249.72.224/27",
-        "66.249.72.32/27",
-        "66.249.72.64/27",
-        "66.249.72.96/27",
-        "66.249.73.0/27",
-        "66.249.73.128/27",
-        "66.249.73.160/27",
-        "66.249.73.192/27",
-        "66.249.73.224/27",
-        "66.249.73.32/27",
-        "66.249.73.64/27",
-        "66.249.73.96/27",
-        "66.249.74.0/27",
-        "66.249.74.128/27",
-        "66.249.74.160/27",
-        "66.249.74.192/27",
-        "66.249.74.32/27",
-        "66.249.74.64/27",
-        "66.249.74.96/27",
-        "66.249.75.0/27",
-        "66.249.75.128/27",
-        "66.249.75.160/27",
-        "66.249.75.192/27",
-        "66.249.75.224/27",
-        "66.249.75.32/27",
-        "66.249.75.64/27",
-        "66.249.75.96/27",
-        "66.249.76.0/27",
-        "66.249.76.128/27",
-        "66.249.76.160/27",
-        "66.249.76.192/27",
-        "66.249.76.224/27",
-        "66.249.76.32/27",
-        "66.249.76.64/27",
-        "66.249.76.96/27",
-        "66.249.77.0/27",
-        "66.249.77.128/27",
-        "66.249.77.160/27",
-        "66.249.77.192/27",
-        "66.249.77.224/27",
-        "66.249.77.32/27",
-        "66.249.77.64/27",
-        "66.249.77.96/27",
-        "66.249.78.0/27",
-        "66.249.78.32/27",
-        "66.249.79.0/27",
-        "66.249.79.128/27",
-        "66.249.79.160/27",
-        "66.249.79.192/27",
-        "66.249.79.224/27",
-        "66.249.79.32/27",
-        "66.249.79.64/27",
-        "66.249.79.96/27"
-      ]
+      "import": "(data)/bots/ai-robots-txt.yaml"
     },
     {
-      "name": "bingbot",
-      "user_agent_regex": "\\+http\\://www\\.bing\\.com/bingbot\\.htm",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "157.55.39.0/24",
-        "207.46.13.0/24",
-        "40.77.167.0/24",
-        "13.66.139.0/24",
-        "13.66.144.0/24",
-        "52.167.144.0/24",
-        "13.67.10.16/28",
-        "13.69.66.240/28",
-        "13.71.172.224/28",
-        "139.217.52.0/28",
-        "191.233.204.224/28",
-        "20.36.108.32/28",
-        "20.43.120.16/28",
-        "40.79.131.208/28",
-        "40.79.186.176/28",
-        "52.231.148.0/28",
-        "20.79.107.240/28",
-        "51.105.67.0/28",
-        "20.125.163.80/28",
-        "40.77.188.0/22",
-        "65.55.210.0/24",
-        "199.30.24.0/23",
-        "40.77.202.0/24",
-        "40.77.139.0/25",
-        "20.74.197.0/28",
-        "20.15.133.160/27",
-        "40.77.177.0/24",
-        "40.77.178.0/23"
-      ]
+      "import": "(data)/crawlers/_allow-good.yaml"
     },
     {
-      "name": "qwantbot",
-      "user_agent_regex": "\\+https\\://help\\.qwant\\.com/bot/",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "91.242.162.0/24"
-      ]
+      "import": "(data)/bots/aggressive-brazilian-scrapers.yaml"
     },
     {
-      "name": "kagibot",
-      "user_agent_regex": "\\+https\\://kagi\\.com/bot",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "216.18.205.234/32",
-        "35.212.27.76/32",
-        "104.254.65.50/32",
-        "209.151.156.194/32"
-      ]
-    },
-    {
-      "name": "marginalia",
-      "user_agent_regex": "search\\.marginalia\\.nu",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "193.183.0.162/31",
-        "193.183.0.164/30",
-        "193.183.0.168/30",
-        "193.183.0.172/31",
-        "193.183.0.174/32"
-      ]
-    },
-    {
-      "name": "mojeekbot",
-      "user_agent_regex": "http\\://www\\.mojeek\\.com/bot\\.html",
-      "action": "ALLOW",
-      "remote_addresses": [
-        "5.102.173.71/32"
-      ]
-    },
-    {
-      "name": "us-artificial-intelligence-scraper",
-      "user_agent_regex": "\\+https\\://github\\.com/US-Artificial-Intelligence/scraper",
-      "action": "DENY"
-    },
-    {
-      "name": "well-known",
-      "path_regex": "^/.well-known/.*$",
-      "action": "ALLOW"
-    },
-    {
-      "name": "favicon",
-      "path_regex": "^/favicon.ico$",
-      "action": "ALLOW"
-    },
-    {
-      "name": "robots-txt",
-      "path_regex": "^/robots.txt$",
-      "action": "ALLOW"
-    },
-    {
-      "name": "lightpanda",
-      "user_agent_regex": "^Lightpanda/.*$",
-      "action": "DENY"
-    },
-    {
-      "name": "headless-chrome",
-      "user_agent_regex": "HeadlessChrome",
-      "action": "DENY"
-    },
-    {
-      "name": "headless-chromium",
-      "user_agent_regex": "HeadlessChromium",
-      "action": "DENY"
-    },
-    {
-      "name": "generic-bot-catchall",
-      "user_agent_regex": "(?i:bot|crawler)",
-      "action": "CHALLENGE",
-      "challenge": {
-        "difficulty": 16,
-        "report_as": 4,
-        "algorithm": "slow"
-      }
+      "import": "(data)/common/keep-internet-working.yaml"
     },
     {
       "name": "generic-browser",
-      "user_agent_regex": "Mozilla",
+      "user_agent_regex": "Mozilla|Opera",
       "action": "CHALLENGE"
     }
   ],
-  "dnsbl": false
+  "dnsbl": false,
+  "status_codes": {
+    "CHALLENGE": 200,
+    "DENY": 200
+  }
 }

data/botPolicies.yaml

@@ -0,0 +1,61 @@
+## Anubis has the ability to let you import snippets of configuration into the main
+## configuration file. This allows you to break up your config into smaller parts
+## that get logically assembled into one big file.
+##
+## Of note, a bot rule can either have inline bot configuration or import a
+## bot config snippet. You cannot do both in a single bot rule.
+##
+## Import paths can either be prefixed with (data) to import from the common/shared
+## rules in the data folder in the Anubis source tree or will point to absolute/relative
+## paths in your filesystem. If you don't have access to the Anubis source tree, check
+## /usr/share/docs/anubis/data or in the tarball you extracted Anubis from.
+
+bots:
+# Pathological bots to deny
+- # This correlates to data/bots/deny-pathological.yaml in the source tree
+  # https://github.com/TecharoHQ/anubis/blob/main/data/bots/deny-pathological.yaml
+  import: (data)/bots/_deny-pathological.yaml
+- import: (data)/bots/aggressive-brazilian-scrapers.yaml
+
+# Enforce https://github.com/ai-robots-txt/ai.robots.txt
+- import: (data)/bots/ai-robots-txt.yaml
+
+# Search engine crawlers to allow, defaults to:
+#   - Google (so they don't try to bypass Anubis)
+#   - Bing
+#   - DuckDuckGo
+#   - Qwant
+#   - The Internet Archive
+#   - Kagi
+#   - Marginalia
+#   - Mojeek
+- import: (data)/crawlers/_allow-good.yaml
+
+# Allow common "keeping the internet working" routes (well-known, favicon, robots.txt)
+- import: (data)/common/keep-internet-working.yaml
+
+# # Punish any bot with "bot" in the user-agent string
+# # This is known to have a high false-positive rate, use at your own risk
+# - name: generic-bot-catchall
+#   user_agent_regex: (?i:bot|crawler)
+#   action: CHALLENGE
+#   challenge:
+#     difficulty: 16  # impossible
+#     report_as: 4    # lie to the operator
+#     algorithm: slow # intentionally waste CPU cycles and time
+
+# Generic catchall rule
+- name: generic-browser
+  user_agent_regex: >-
+    Mozilla|Opera
+  action: CHALLENGE
+
+dnsbl: false
+
+# By default, send HTTP 200 back to clients that either get issued a challenge
+# or a denial. This seems weird, but this is load-bearing due to the fact that
+# the most aggressive scraper bots seem to really, really, want an HTTP 200 and
+# will stop sending requests once they get it.
+status_codes:
+  CHALLENGE: 200
+  DENY: 200 

data/bots/_deny-pathological.yaml

@@ -0,0 +1,3 @@
+- import: (data)/bots/cloudflare-workers.yaml
+- import: (data)/bots/headless-browsers.yaml
+- import: (data)/bots/us-ai-scraper.yaml

data/bots/aggressive-brazilian-scrapers.yaml

@@ -0,0 +1,28 @@
+- name: deny-aggressive-brazilian-scrapers
+  action: DENY
+  expression:
+    any:
+    # Internet Explorer should be out of support
+    - userAgent.contains("MSIE")
+    # Trident is the Internet Explorer browser engine
+    - userAgent.contains("Trident")
+    # Opera is a fork of chrome now
+    - userAgent.contains("Presto")
+    # Windows CE is discontinued
+    - userAgent.contains("Windows CE")
+    # Windows 95 is discontinued
+    - userAgent.contains("Windows 95")
+    # Windows 98 is discontinued
+    - userAgent.contains("Windows 98")
+    # Windows 9.x is discontinued
+    - userAgent.contains("Win 9x")
+    # Amazon does not have an Alexa Toolbar.
+    - userAgent.contains("Alexa Toolbar")
+- name: challenge-aggressive-brazilian-scrapers
+  action: CHALLENGE
+  expression:
+    any:
+    # This is not released, even Windows 11 calls itself Windows 10
+    - userAgent.contains("Windows NT 11.0")
+    # iPods are not in common use
+    - userAgent.contains("iPod")

data/bots/ai-robots-txt.yaml

@@ -0,0 +1,4 @@
+- name: "ai-robots-txt"
+  user_agent_regex: >-
+    AI2Bot|Ai2Bot-Dolma|aiHitBot|Amazonbot|anthropic-ai|Applebot|Applebot-Extended|Brightbot 1.0|Bytespider|CCBot|ChatGPT-User|Claude-Web|ClaudeBot|cohere-ai|cohere-training-data-crawler|Cotoyogi|Crawlspace|Diffbot|DuckAssistBot|FacebookBot|Factset_spyderbot|FirecrawlAgent|FriendlyCrawler|Google-Extended|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iaskspider/2.0|ICC-Crawler|ImagesiftBot|img2dataset|imgproxy|ISSCyberRiskCrawler|Kangaroo Bot|meta-externalagent|Meta-ExternalAgent|meta-externalfetcher|Meta-ExternalFetcher|NovaAct|OAI-SearchBot|omgili|omgilibot|Operator|PanguBot|Perplexity-User|PerplexityBot|PetalBot|Scrapy|SemrushBot-OCOB|SemrushBot-SWA|Sidetrade indexer bot|TikTokSpider|Timpibot|VelenPublicWebCrawler|Webzio-Extended|YouBot
+  action: DENY

data/bots/cloudflare-workers.yaml

@@ -0,0 +1,4 @@
+- name: cloudflare-workers
+  headers_regex:
+    CF-Worker: .*
+  action: DENY

data/bots/headless-browsers.yaml

@@ -0,0 +1,9 @@
+- name: lightpanda
+  user_agent_regex: ^LightPanda/.*$
+  action: DENY
+- name: headless-chrome
+  user_agent_regex: HeadlessChrome
+  action: DENY
+- name: headless-chromium
+  user_agent_regex: HeadlessChromium
+  action: DENY

data/bots/irc-bots/archlinux-phrik.yaml

@@ -0,0 +1,9 @@
+# phrik in the Arch Linux IRC channels
+- name: archlinux-phrik
+  action: ALLOW
+  expression:
+    all:
+    - remoteAddress == "159.69.213.214"
+    - userAgent == "Mozilla/5.0 (compatible; utils.web Limnoria module)"
+    - '"X-Http-Version" in headers'
+    - headers["X-Http-Version"] == "HTTP/1.1"

data/bots/irc-bots/gentoo-chat.yaml

@@ -0,0 +1,9 @@
+# chat in the gentoo IRC channels
+- name: gentoo-chat
+  action: ALLOW
+  expression:
+    all:
+    - remoteAddress == "45.76.166.57"
+    - userAgent == "Mozilla/5.0 (Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0"
+    - '"X-Http-Version" in headers'
+    - headers["X-Http-Version"] == "HTTP/1.1"

data/bots/us-ai-scraper.yaml

@@ -0,0 +1,3 @@
+- name: us-artificial-intelligence-scraper
+  user_agent_regex: \+https\://github\.com/US-Artificial-Intelligence/scraper
+  action: DENY

data/clients/git.yaml

@@ -0,0 +1,14 @@
+- name: allow-git-clients
+  action: ALLOW
+  expression:
+    all:
+    - >
+      (  
+        userAgent.startsWith("git/") ||
+        userAgent.contains("libgit") ||
+        userAgent.startsWith("go-git") ||
+        userAgent.startsWith("JGit/") ||
+        userAgent.startsWith("JGit-")
+      )
+    - '"Git-Protocol" in headers'
+    - headers["Git-Protocol"] == "version=2"

data/clients/go-get.yaml

@@ -0,0 +1,7 @@
+- name: go-get
+  action: ALLOW
+  expression:
+    all:
+    - userAgent.startsWith("Go-http-client/")
+    - '"go-get" in query'
+    - query["go-get"] == "1"

data/common/allow-api-like.yaml

@@ -0,0 +1,6 @@
+- name: allow-api-routes
+  action: ALLOW
+  expression:
+    all:
+    - '!(method == "HEAD" || method == "GET")'
+    - path.startsWith("/api/")

data/common/allow-private-addresses.yaml

@@ -0,0 +1,15 @@
+- name: ipv4-rfc-1918
+  action: ALLOW
+  remote_addresses:
+  - 10.0.0.0/8
+  - 172.16.0.0/12
+  - 192.168.0.0/16
+  - 100.64.0.0/10
+- name: ipv6-ula
+  action: ALLOW
+  remote_addresses:
+  - fc00::/7
+- name: ipv6-link-local
+  action: ALLOW
+  remote_addresses:
+  - fe80::/10

data/common/json-api.yaml

@@ -0,0 +1,7 @@
+- name: allow-api-requests
+  action: ALLOW
+  expression:
+    all:
+      - '"Accept" in headers'
+      - 'headers["Accept"] == "application/json"'
+      - 'path.startsWith("/api/")'

data/common/keep-internet-working.yaml

@@ -0,0 +1,13 @@
+# Common "keeping the internet working" routes
+- name: well-known
+  path_regex: ^/.well-known/.*$
+  action: ALLOW
+- name: favicon
+  path_regex: ^/favicon.ico$
+  action: ALLOW
+- name: robots-txt
+  path_regex: ^/robots.txt$
+  action: ALLOW
+- name: sitemap
+  path_regex: ^/sitemap.xml$
+  action: ALLOW

data/common/rfc-violations.yaml

@@ -0,0 +1,3 @@
+- name: no-user-agent-string
+  action: DENY
+  expression: userAgent == ""

data/crawlers/_allow-good.yaml

@@ -0,0 +1,8 @@
+- import: (data)/crawlers/googlebot.yaml
+- import: (data)/crawlers/bingbot.yaml
+- import: (data)/crawlers/duckduckbot.yaml
+- import: (data)/crawlers/qwantbot.yaml
+- import: (data)/crawlers/internet-archive.yaml
+- import: (data)/crawlers/kagibot.yaml
+- import: (data)/crawlers/marginalia.yaml
+- import: (data)/crawlers/mojeekbot.yaml

data/crawlers/bingbot.yaml

@@ -0,0 +1,34 @@
+- name: bingbot
+  user_agent_regex: \+http\://www\.bing\.com/bingbot\.htm
+  action: ALLOW
+  # https://www.bing.com/toolbox/bingbot.json
+  remote_addresses: [
+    "157.55.39.0/24",
+    "207.46.13.0/24",
+    "40.77.167.0/24",
+    "13.66.139.0/24",
+    "13.66.144.0/24",
+    "52.167.144.0/24",
+    "13.67.10.16/28",
+    "13.69.66.240/28",
+    "13.71.172.224/28",
+    "139.217.52.0/28",
+    "191.233.204.224/28",
+    "20.36.108.32/28",
+    "20.43.120.16/28",
+    "40.79.131.208/28",
+    "40.79.186.176/28",
+    "52.231.148.0/28",
+    "20.79.107.240/28",
+    "51.105.67.0/28",
+    "20.125.163.80/28",
+    "40.77.188.0/22",
+    "65.55.210.0/24",
+    "199.30.24.0/23",
+    "40.77.202.0/24",
+    "40.77.139.0/25",
+    "20.74.197.0/28",
+    "20.15.133.160/27",
+    "40.77.177.0/24",
+    "40.77.178.0/23"
+  ]

data/crawlers/duckduckbot.yaml

@@ -0,0 +1,275 @@
+- name: duckduckbot
+  user_agent_regex: DuckDuckBot/1\.1; \(\+http\://duckduckgo\.com/duckduckbot\.html\)
+  action: ALLOW
+  # https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot
+  remote_addresses: [
+    "57.152.72.128/32",
+    "51.8.253.152/32",
+    "40.80.242.63/32",
+    "20.12.141.99/32",
+    "20.49.136.28/32",
+    "51.116.131.221/32",
+    "51.107.40.209/32",
+    "20.40.133.240/32",
+    "20.50.168.91/32",
+    "51.120.48.122/32",
+    "20.193.45.113/32",
+    "40.76.173.151/32",
+    "40.76.163.7/32",
+    "20.185.79.47/32",
+    "52.142.26.175/32",
+    "20.185.79.15/32",
+    "52.142.24.149/32",
+    "40.76.162.208/32",
+    "40.76.163.23/32",
+    "40.76.162.191/32",
+    "40.76.162.247/32",
+    "40.88.21.235/32",
+    "20.191.45.212/32",
+    "52.146.59.12/32",
+    "52.146.59.156/32",
+    "52.146.59.154/32",
+    "52.146.58.236/32",
+    "20.62.224.44/32",
+    "51.104.180.53/32",
+    "51.104.180.47/32",
+    "51.104.180.26/32",
+    "51.104.146.225/32",
+    "51.104.146.235/32",
+    "20.73.202.147/32",
+    "20.73.132.240/32",
+    "20.71.12.143/32",
+    "20.56.197.58/32",
+    "20.56.197.63/32",
+    "20.43.150.93/32",
+    "20.43.150.85/32",
+    "20.44.222.1/32",
+    "40.89.243.175/32",
+    "13.89.106.77/32",
+    "52.143.242.6/32",
+    "52.143.241.111/32",
+    "52.154.60.82/32",
+    "20.197.209.11/32",
+    "20.197.209.27/32",
+    "20.226.133.105/32",
+    "191.234.216.4/32",
+    "191.234.216.178/32",
+    "20.53.92.211/32",
+    "20.53.91.2/32",
+    "20.207.99.197/32",
+    "20.207.97.190/32",
+    "40.81.250.205/32",
+    "40.64.106.11/32",
+    "40.64.105.247/32",
+    "20.72.242.93/32",
+    "20.99.255.235/32",
+    "20.113.3.121/32",
+    "52.224.16.221/32",
+    "52.224.21.53/32",
+    "52.224.20.204/32",
+    "52.224.21.19/32",
+    "52.224.20.249/32",
+    "52.224.20.203/32",
+    "52.224.20.190/32",
+    "52.224.16.229/32",
+    "52.224.21.20/32",
+    "52.146.63.80/32",
+    "52.224.20.227/32",
+    "52.224.20.193/32",
+    "52.190.37.160/32",
+    "52.224.21.23/32",
+    "52.224.20.223/32",
+    "52.224.20.181/32",
+    "52.224.21.49/32",
+    "52.224.21.55/32",
+    "52.224.21.61/32",
+    "52.224.19.152/32",
+    "52.224.20.186/32",
+    "52.224.21.27/32",
+    "52.224.21.51/32",
+    "52.224.20.174/32",
+    "52.224.21.4/32",
+    "51.104.164.109/32",
+    "51.104.167.71/32",
+    "51.104.160.177/32",
+    "51.104.162.149/32",
+    "51.104.167.95/32",
+    "51.104.167.54/32",
+    "51.104.166.111/32",
+    "51.104.167.88/32",
+    "51.104.161.32/32",
+    "51.104.163.250/32",
+    "51.104.164.189/32",
+    "51.104.167.19/32",
+    "51.104.160.167/32",
+    "51.104.167.110/32",
+    "20.191.44.119/32",
+    "51.104.167.104/32",
+    "20.191.44.234/32",
+    "51.104.164.215/32",
+    "51.104.167.52/32",
+    "20.191.44.22/32",
+    "51.104.167.87/32",
+    "51.104.167.96/32",
+    "20.191.44.16/32",
+    "51.104.167.61/32",
+    "51.104.164.147/32",
+    "20.50.48.159/32",
+    "40.114.182.172/32",
+    "20.50.50.130/32",
+    "20.50.50.163/32",
+    "20.50.50.46/32",
+    "40.114.182.153/32",
+    "20.50.50.118/32",
+    "20.50.49.55/32",
+    "20.50.49.25/32",
+    "40.114.183.251/32",
+    "20.50.50.123/32",
+    "20.50.49.237/32",
+    "20.50.48.192/32",
+    "20.50.50.134/32",
+    "51.138.90.233/32",
+    "40.114.183.196/32",
+    "20.50.50.146/32",
+    "40.114.183.88/32",
+    "20.50.50.145/32",
+    "20.50.50.121/32",
+    "20.50.49.40/32",
+    "51.138.90.206/32",
+    "40.114.182.45/32",
+    "51.138.90.161/32",
+    "20.50.49.0/32",
+    "40.119.232.215/32",
+    "104.43.55.167/32",
+    "40.119.232.251/32",
+    "40.119.232.50/32",
+    "40.119.232.146/32",
+    "40.119.232.218/32",
+    "104.43.54.127/32",
+    "104.43.55.117/32",
+    "104.43.55.116/32",
+    "104.43.55.166/32",
+    "52.154.169.50/32",
+    "52.154.171.70/32",
+    "52.154.170.229/32",
+    "52.154.170.113/32",
+    "52.154.171.44/32",
+    "52.154.172.2/32",
+    "52.143.244.81/32",
+    "52.154.171.87/32",
+    "52.154.171.250/32",
+    "52.154.170.28/32",
+    "52.154.170.122/32",
+    "52.143.243.117/32",
+    "52.143.247.235/32",
+    "52.154.171.235/32",
+    "52.154.171.196/32",
+    "52.154.171.0/32",
+    "52.154.170.243/32",
+    "52.154.170.26/32",
+    "52.154.169.200/32",
+    "52.154.170.96/32",
+    "52.154.170.88/32",
+    "52.154.171.150/32",
+    "52.154.171.205/32",
+    "52.154.170.117/32",
+    "52.154.170.209/32",
+    "191.235.202.48/32",
+    "191.233.3.202/32",
+    "191.235.201.214/32",
+    "191.233.3.197/32",
+    "191.235.202.38/32",
+    "20.53.78.144/32",
+    "20.193.24.10/32",
+    "20.53.78.236/32",
+    "20.53.78.138/32",
+    "20.53.78.123/32",
+    "20.53.78.106/32",
+    "20.193.27.215/32",
+    "20.193.25.197/32",
+    "20.193.12.126/32",
+    "20.193.24.251/32",
+    "20.204.242.101/32",
+    "20.207.72.113/32",
+    "20.204.242.19/32",
+    "20.219.45.67/32",
+    "20.207.72.11/32",
+    "20.219.45.190/32",
+    "20.204.243.55/32",
+    "20.204.241.148/32",
+    "20.207.72.110/32",
+    "20.204.240.172/32",
+    "20.207.72.21/32",
+    "20.204.246.81/32",
+    "20.207.107.181/32",
+    "20.204.246.254/32",
+    "20.219.43.246/32",
+    "52.149.25.43/32",
+    "52.149.61.51/32",
+    "52.149.58.139/32",
+    "52.149.60.38/32",
+    "52.148.165.38/32",
+    "52.143.95.162/32",
+    "52.149.56.151/32",
+    "52.149.30.45/32",
+    "52.149.58.173/32",
+    "52.143.95.204/32",
+    "52.149.28.83/32",
+    "52.149.58.69/32",
+    "52.148.161.87/32",
+    "52.149.58.27/32",
+    "52.149.28.18/32",
+    "20.79.226.26/32",
+    "20.79.239.66/32",
+    "20.79.238.198/32",
+    "20.113.14.159/32",
+    "20.75.144.152/32",
+    "20.43.172.120/32",
+    "20.53.134.160/32",
+    "20.201.15.208/32",
+    "20.93.28.24/32",
+    "20.61.34.40/32",
+    "52.242.224.168/32",
+    "20.80.129.80/32",
+    "20.195.108.47/32",
+    "4.195.133.120/32",
+    "4.228.76.163/32",
+    "4.182.131.108/32",
+    "4.209.224.56/32",
+    "108.141.83.74/32",
+    "4.213.46.14/32",
+    "172.169.17.165/32",
+    "51.8.71.117/32",
+    "20.3.1.178/32",
+    "52.149.56.151/32",
+    "52.149.30.45/32",
+    "52.149.58.173/32",
+    "52.143.95.204/32",
+    "52.149.28.83/32",
+    "52.149.58.69/32",
+    "52.148.161.87/32",
+    "52.149.58.27/32",
+    "52.149.28.18/32",
+    "20.79.226.26/32",
+    "20.79.239.66/32",
+    "20.79.238.198/32",
+    "20.113.14.159/32",
+    "20.75.144.152/32",
+    "20.43.172.120/32",
+    "20.53.134.160/32",
+    "20.201.15.208/32",
+    "20.93.28.24/32",
+    "20.61.34.40/32",
+    "52.242.224.168/32",
+    "20.80.129.80/32",
+    "20.195.108.47/32",
+    "4.195.133.120/32",
+    "4.228.76.163/32",
+    "4.182.131.108/32",
+    "4.209.224.56/32",
+    "108.141.83.74/32",
+    "4.213.46.14/32",
+    "172.169.17.165/32",
+    "51.8.71.117/32",
+    "20.3.1.178/32"
+  ]

data/crawlers/googlebot.yaml

@@ -0,0 +1,263 @@
+- name: googlebot
+  user_agent_regex: \+http\://www\.google\.com/bot\.html
+  action: ALLOW
+  # https://developers.google.com/static/search/apis/ipranges/googlebot.json
+  remote_addresses: [
+    "2001:4860:4801:10::/64",
+    "2001:4860:4801:11::/64",
+    "2001:4860:4801:12::/64",
+    "2001:4860:4801:13::/64",
+    "2001:4860:4801:14::/64",
+    "2001:4860:4801:15::/64",
+    "2001:4860:4801:16::/64",
+    "2001:4860:4801:17::/64",
+    "2001:4860:4801:18::/64",
+    "2001:4860:4801:19::/64",
+    "2001:4860:4801:1a::/64",
+    "2001:4860:4801:1b::/64",
+    "2001:4860:4801:1c::/64",
+    "2001:4860:4801:1d::/64",
+    "2001:4860:4801:1e::/64",
+    "2001:4860:4801:1f::/64",
+    "2001:4860:4801:20::/64",
+    "2001:4860:4801:21::/64",
+    "2001:4860:4801:22::/64",
+    "2001:4860:4801:23::/64",
+    "2001:4860:4801:24::/64",
+    "2001:4860:4801:25::/64",
+    "2001:4860:4801:26::/64",
+    "2001:4860:4801:27::/64",
+    "2001:4860:4801:28::/64",
+    "2001:4860:4801:29::/64",
+    "2001:4860:4801:2::/64",
+    "2001:4860:4801:2a::/64",
+    "2001:4860:4801:2b::/64",
+    "2001:4860:4801:2c::/64",
+    "2001:4860:4801:2d::/64",
+    "2001:4860:4801:2e::/64",
+    "2001:4860:4801:2f::/64",
+    "2001:4860:4801:31::/64",
+    "2001:4860:4801:32::/64",
+    "2001:4860:4801:33::/64",
+    "2001:4860:4801:34::/64",
+    "2001:4860:4801:35::/64",
+    "2001:4860:4801:36::/64",
+    "2001:4860:4801:37::/64",
+    "2001:4860:4801:38::/64",
+    "2001:4860:4801:39::/64",
+    "2001:4860:4801:3a::/64",
+    "2001:4860:4801:3b::/64",
+    "2001:4860:4801:3c::/64",
+    "2001:4860:4801:3d::/64",
+    "2001:4860:4801:3e::/64",
+    "2001:4860:4801:40::/64",
+    "2001:4860:4801:41::/64",
+    "2001:4860:4801:42::/64",
+    "2001:4860:4801:43::/64",
+    "2001:4860:4801:44::/64",
+    "2001:4860:4801:45::/64",
+    "2001:4860:4801:46::/64",
+    "2001:4860:4801:47::/64",
+    "2001:4860:4801:48::/64",
+    "2001:4860:4801:49::/64",
+    "2001:4860:4801:4a::/64",
+    "2001:4860:4801:4b::/64",
+    "2001:4860:4801:4c::/64",
+    "2001:4860:4801:50::/64",
+    "2001:4860:4801:51::/64",
+    "2001:4860:4801:52::/64",
+    "2001:4860:4801:53::/64",
+    "2001:4860:4801:54::/64",
+    "2001:4860:4801:55::/64",
+    "2001:4860:4801:56::/64",
+    "2001:4860:4801:60::/64",
+    "2001:4860:4801:61::/64",
+    "2001:4860:4801:62::/64",
+    "2001:4860:4801:63::/64",
+    "2001:4860:4801:64::/64",
+    "2001:4860:4801:65::/64",
+    "2001:4860:4801:66::/64",
+    "2001:4860:4801:67::/64",
+    "2001:4860:4801:68::/64",
+    "2001:4860:4801:69::/64",
+    "2001:4860:4801:6a::/64",
+    "2001:4860:4801:6b::/64",
+    "2001:4860:4801:6c::/64",
+    "2001:4860:4801:6d::/64",
+    "2001:4860:4801:6e::/64",
+    "2001:4860:4801:6f::/64",
+    "2001:4860:4801:70::/64",
+    "2001:4860:4801:71::/64",
+    "2001:4860:4801:72::/64",
+    "2001:4860:4801:73::/64",
+    "2001:4860:4801:74::/64",
+    "2001:4860:4801:75::/64",
+    "2001:4860:4801:76::/64",
+    "2001:4860:4801:77::/64",
+    "2001:4860:4801:78::/64",
+    "2001:4860:4801:79::/64",
+    "2001:4860:4801:80::/64",
+    "2001:4860:4801:81::/64",
+    "2001:4860:4801:82::/64",
+    "2001:4860:4801:83::/64",
+    "2001:4860:4801:84::/64",
+    "2001:4860:4801:85::/64",
+    "2001:4860:4801:86::/64",
+    "2001:4860:4801:87::/64",
+    "2001:4860:4801:88::/64",
+    "2001:4860:4801:90::/64",
+    "2001:4860:4801:91::/64",
+    "2001:4860:4801:92::/64",
+    "2001:4860:4801:93::/64",
+    "2001:4860:4801:94::/64",
+    "2001:4860:4801:95::/64",
+    "2001:4860:4801:96::/64",
+    "2001:4860:4801:a0::/64",
+    "2001:4860:4801:a1::/64",
+    "2001:4860:4801:a2::/64",
+    "2001:4860:4801:a3::/64",
+    "2001:4860:4801:a4::/64",
+    "2001:4860:4801:a5::/64",
+    "2001:4860:4801:c::/64",
+    "2001:4860:4801:f::/64",
+    "192.178.5.0/27",
+    "192.178.6.0/27",
+    "192.178.6.128/27",
+    "192.178.6.160/27",
+    "192.178.6.192/27",
+    "192.178.6.32/27",
+    "192.178.6.64/27",
+    "192.178.6.96/27",
+    "34.100.182.96/28",
+    "34.101.50.144/28",
+    "34.118.254.0/28",
+    "34.118.66.0/28",
+    "34.126.178.96/28",
+    "34.146.150.144/28",
+    "34.147.110.144/28",
+    "34.151.74.144/28",
+    "34.152.50.64/28",
+    "34.154.114.144/28",
+    "34.155.98.32/28",
+    "34.165.18.176/28",
+    "34.175.160.64/28",
+    "34.176.130.16/28",
+    "34.22.85.0/27",
+    "34.64.82.64/28",
+    "34.65.242.112/28",
+    "34.80.50.80/28",
+    "34.88.194.0/28",
+    "34.89.10.80/28",
+    "34.89.198.80/28",
+    "34.96.162.48/28",
+    "35.247.243.240/28",
+    "66.249.64.0/27",
+    "66.249.64.128/27",
+    "66.249.64.160/27",
+    "66.249.64.224/27",
+    "66.249.64.32/27",
+    "66.249.64.64/27",
+    "66.249.64.96/27",
+    "66.249.65.0/27",
+    "66.249.65.128/27",
+    "66.249.65.160/27",
+    "66.249.65.192/27",
+    "66.249.65.224/27",
+    "66.249.65.32/27",
+    "66.249.65.64/27",
+    "66.249.65.96/27",
+    "66.249.66.0/27",
+    "66.249.66.128/27",
+    "66.249.66.160/27",
+    "66.249.66.192/27",
+    "66.249.66.224/27",
+    "66.249.66.32/27",
+    "66.249.66.64/27",
+    "66.249.66.96/27",
+    "66.249.68.0/27",
+    "66.249.68.128/27",
+    "66.249.68.32/27",
+    "66.249.68.64/27",
+    "66.249.68.96/27",
+    "66.249.69.0/27",
+    "66.249.69.128/27",
+    "66.249.69.160/27",
+    "66.249.69.192/27",
+    "66.249.69.224/27",
+    "66.249.69.32/27",
+    "66.249.69.64/27",
+    "66.249.69.96/27",
+    "66.249.70.0/27",
+    "66.249.70.128/27",
+    "66.249.70.160/27",
+    "66.249.70.192/27",
+    "66.249.70.224/27",
+    "66.249.70.32/27",
+    "66.249.70.64/27",
+    "66.249.70.96/27",
+    "66.249.71.0/27",
+    "66.249.71.128/27",
+    "66.249.71.160/27",
+    "66.249.71.192/27",
+    "66.249.71.224/27",
+    "66.249.71.32/27",
+    "66.249.71.64/27",
+    "66.249.71.96/27",
+    "66.249.72.0/27",
+    "66.249.72.128/27",
+    "66.249.72.160/27",
+    "66.249.72.192/27",
+    "66.249.72.224/27",
+    "66.249.72.32/27",
+    "66.249.72.64/27",
+    "66.249.72.96/27",
+    "66.249.73.0/27",
+    "66.249.73.128/27",
+    "66.249.73.160/27",
+    "66.249.73.192/27",
+    "66.249.73.224/27",
+    "66.249.73.32/27",
+    "66.249.73.64/27",
+    "66.249.73.96/27",
+    "66.249.74.0/27",
+    "66.249.74.128/27",
+    "66.249.74.160/27",
+    "66.249.74.192/27",
+    "66.249.74.32/27",
+    "66.249.74.64/27",
+    "66.249.74.96/27",
+    "66.249.75.0/27",
+    "66.249.75.128/27",
+    "66.249.75.160/27",
+    "66.249.75.192/27",
+    "66.249.75.224/27",
+    "66.249.75.32/27",
+    "66.249.75.64/27",
+    "66.249.75.96/27",
+    "66.249.76.0/27",
+    "66.249.76.128/27",
+    "66.249.76.160/27",
+    "66.249.76.192/27",
+    "66.249.76.224/27",
+    "66.249.76.32/27",
+    "66.249.76.64/27",
+    "66.249.76.96/27",
+    "66.249.77.0/27",
+    "66.249.77.128/27",
+    "66.249.77.160/27",
+    "66.249.77.192/27",
+    "66.249.77.224/27",
+    "66.249.77.32/27",
+    "66.249.77.64/27",
+    "66.249.77.96/27",
+    "66.249.78.0/27",
+    "66.249.78.32/27",
+    "66.249.79.0/27",
+    "66.249.79.128/27",
+    "66.249.79.160/27",
+    "66.249.79.192/27",
+    "66.249.79.224/27",
+    "66.249.79.32/27",
+    "66.249.79.64/27",
+    "66.249.79.96/27"
+  ]

data/crawlers/internet-archive.yaml

@@ -0,0 +1,8 @@
+- name: internet-archive
+  action: ALLOW
+  # https://ipinfo.io/AS7941
+  remote_addresses: [
+    "207.241.224.0/20",
+    "208.70.24.0/21",
+    "2620:0:9c0::/48"
+  ]

data/crawlers/kagibot.yaml

@@ -0,0 +1,10 @@
+- name: kagibot
+  user_agent_regex: \+https\://kagi\.com/bot
+  action: ALLOW
+  # https://kagi.com/bot
+  remote_addresses: [
+    "216.18.205.234/32",
+    "35.212.27.76/32",
+    "104.254.65.50/32",
+    "209.151.156.194/32"
+  ]

data/crawlers/marginalia.yaml

@@ -0,0 +1,11 @@
+- name: marginalia
+  user_agent_regex: search\.marginalia\.nu
+  action: ALLOW
+  # Received directly over email
+  remote_addresses: [
+    "193.183.0.162/31",
+    "193.183.0.164/30",
+    "193.183.0.168/30",
+    "193.183.0.172/31",
+    "193.183.0.174/32"
+  ]

data/crawlers/mojeekbot.yaml

@@ -0,0 +1,5 @@
+- name: mojeekbot
+  user_agent_regex: \+https\://www\.mojeek\.com/bot\.html
+  action: ALLOW
+  # https://www.mojeek.com/bot.html
+  remote_addresses: [ "5.102.173.71/32" ]

data/crawlers/qwantbot.yaml

@@ -0,0 +1,5 @@
+- name: qwantbot
+  user_agent_regex: \+https\://help\.qwant\.com/bot/
+  action: ALLOW
+  # https://help.qwant.com/wp-content/uploads/sites/2/2025/01/qwantbot.json
+  remote_addresses: [ "91.242.162.0/24" ]

data/embed.go

@@ -3,6 +3,6 @@ package data
 import "embed"
 
 var (
-	//go:embed botPolicies.json
+	//go:embed botPolicies.yaml botPolicies.json all:apps all:bots all:clients all:common all:crawlers
 	BotPolicies embed.FS
 )

docs/blog/2019-05-28-first-blog-post.md

@@ -1,12 +0,0 @@
----
-slug: first-blog-post
-title: First Blog Post
-authors: [slorber, yangshun]
-tags: [hola, docusaurus]
----
-
-Lorem ipsum dolor sit amet...
-
-<!-- truncate -->
-
-...consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet

docs/blog/2019-05-29-long-blog-post.md

@@ -1,44 +0,0 @@
----
-slug: long-blog-post
-title: Long Blog Post
-authors: yangshun
-tags: [hello, docusaurus]
----
-
-This is the summary of a very long blog post,
-
-Use a `<!--` `truncate` `-->` comment to limit blog post size in the list view.
-
-<!-- truncate -->
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet
-
-Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque elementum dignissim ultricies. Fusce rhoncus ipsum tempor eros aliquam consequat. Lorem ipsum dolor sit amet

docs/blog/2021-08-01-mdx-blog-post.mdx

@@ -1,24 +0,0 @@
----
-slug: mdx-blog-post
-title: MDX Blog Post
-authors: [slorber]
-tags: [docusaurus]
----
-
-Blog posts support [Docusaurus Markdown features](https://docusaurus.io/docs/markdown-features), such as [MDX](https://mdxjs.com/).
-
-:::tip
-
-Use the power of React to create interactive blog posts.
-
-:::
-
-{/* truncate */}
-
-For example, use JSX to create an interactive button:
-
-```js
-<button onClick={() => alert('button clicked!')}>Click me!</button>
-```
-
-<button onClick={() => alert('button clicked!')}>Click me!</button>

docs/blog/2021-08-26-welcome/index.md

@@ -1,29 +0,0 @@
----
-slug: welcome
-title: Welcome
-authors: [slorber, yangshun]
-tags: [facebook, hello, docusaurus]
----
-
-[Docusaurus blogging features](https://docusaurus.io/docs/blog) are powered by the [blog plugin](https://docusaurus.io/docs/api/plugins/@docusaurus/plugin-content-blog).
-
-Here are a few tips you might find useful.
-
-<!-- truncate -->
-
-Simply add Markdown files (or folders) to the `blog` directory.
-
-Regular blog authors can be added to `authors.yml`.
-
-The blog post date can be extracted from filenames, such as:
-
-- `2019-05-30-welcome.md`
-- `2019-05-30-welcome/index.md`
-
-A blog post folder can be convenient to co-locate blog post images:
-
-![Docusaurus Plushie](./docusaurus-plushie-banner.jpeg)
-
-The blog supports tags as well!
-
-**And if you don't want a blog**: just delete this directory, and use `blog: false` in your Docusaurus config.

docs/blog/authors.yml

@@ -1,23 +0,0 @@
-yangshun:
-  name: Yangshun Tay
-  title: Front End Engineer @ Facebook
-  url: https://github.com/yangshun
-  image_url: https://github.com/yangshun.png
-  page: true
-  socials:
-    x: yangshunz
-    github: yangshun
-
-slorber:
-  name: Sébastien Lorber
-  title: Docusaurus maintainer
-  url: https://sebastienlorber.com
-  image_url: https://github.com/slorber.png
-  page:
-    # customize the url of the author page at /blog/authors/<permalink>
-    permalink: '/all-sebastien-lorber-articles'
-  socials:
-    x: sebastienlorber
-    linkedin: sebastienlorber
-    github: slorber
-    newsletter: https://thisweekinreact.com

docs/blog/tags.yml

@@ -1,19 +0,0 @@
-facebook:
-  label: Facebook
-  permalink: /facebook
-  description: Facebook tag description
-
-hello:
-  label: Hello
-  permalink: /hello
-  description: Hello tag description
-
-docusaurus:
-  label: Docusaurus
-  permalink: /docusaurus
-  description: Docusaurus tag description
-
-hola:
-  label: Hola
-  permalink: /hola
-  description: Hola tag description

docs/docs/CHANGELOG.md

@@ -11,37 +11,150 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-- Added support for native Debian, Red Hat, and tarball packaging strategies including installation and use directions.
-- A prebaked tarball has been added, allowing distros to build Anubis like they could in v1.15.x.
-- The placeholder Anubis mascot has been replaced with a design by [CELPHASE](https://bsky.app/profile/celphase.bsky.social).
-- Added a periodic cleanup routine for the decaymap that removes expired entries, ensuring stale data is properly pruned.
+## v1.18.0: Varis zos Galvus
+
+The big ticket feature in this release is [CEL expression matching support](https://anubis.techaro.lol/docs/admin/configuration/expressions). This allows you to tailor your approach for the individual services you are protecting.
+
+These can be as simple as:
+
+```yaml
+- name: allow-api-requests
+  action: ALLOW
+  expression:
+    all:
+      - '"Accept" in headers'
+      - 'headers["Accept"] == "application/json"'
+      - 'path.startsWith("/api/")'
+```
+
+Or as complicated as:
+
+```yaml
+- name: allow-git-clients
+  action: ALLOW
+  expression:
+    all:
+      - >-
+        (  
+          userAgent.startsWith("git/") ||
+          userAgent.contains("libgit") ||
+          userAgent.startsWith("go-git") ||
+          userAgent.startsWith("JGit/") ||
+          userAgent.startsWith("JGit-")
+        )
+      - '"Git-Protocol" in headers'
+      - headers["Git-Protocol"] == "version=2"
+```
+
+The docs have more information, but here's a tl;dr of the variables you have access to in expressions:
+
+| Name            | Type                  | Explanation                                                                                                                               | Example                                                      |
+| :-------------- | :-------------------- | :---------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------- |
+| `headers`       | `map[string, string]` | The [headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers) of the request being processed.                        | `{"User-Agent": "Mozilla/5.0 Gecko/20100101 Firefox/137.0"}` |
+| `host`          | `string`              | The [HTTP hostname](https://web.dev/articles/url-parts#host) the request is targeted to.                                                  | `anubis.techaro.lol`                                         |
+| `method`        | `string`              | The [HTTP method](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Methods) in the request being processed.                    | `GET`, `POST`, `DELETE`, etc.                                |
+| `path`          | `string`              | The [path](https://web.dev/articles/url-parts#pathname) of the request being processed.                                                   | `/`, `/api/memes/create`                                     |
+| `query`         | `map[string, string]` | The [query parameters](https://web.dev/articles/url-parts#query) of the request being processed.                                          | `?foo=bar` -> `{"foo": "bar"}`                               |
+| `remoteAddress` | `string`              | The IP address of the client.                                                                                                             | `1.1.1.1`                                                    |
+| `userAgent`     | `string`              | The [`User-Agent`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent) string in the request being processed. | `Mozilla/5.0 Gecko/20100101 Firefox/137.0`                   |
+
+This will be made more elaborate in the future. Give me time. This is a [simple, lovable, and complete](https://longform.asmartbear.com/slc/) implementation of this feature so that administrators can get hacking ASAP.
+
+Other changes:
+
+- Use CSS variables to deduplicate styles
+- Fixed native packages not containing the stdlib and botPolicies.yaml
+- Change import syntax to allow multi-level imports
+- Changed the startup logging to use JSON formatting as all the other logs do.
+- Added the ability to do [expression matching with CEL](./admin/configuration/expressions.mdx)
+- Add a warning for clients that don't store cookies
+- Disable Open Graph passthrough by default ([#435](https://github.com/TecharoHQ/anubis/issues/435))
+- Clarify the license of the mascot images ([#442](https://github.com/TecharoHQ/anubis/issues/442))
+- Started Suppressing 'Context canceled' errors from http in the logs ([#446](https://github.com/TecharoHQ/anubis/issues/446))
+
+## v1.17.1: Asahi sas Brutus: Echo 1
+
+- Added customization of authorization cookie expiration time with `--cookie-expiration-time` flag or envvar
+- Updated the `OG_PASSTHROUGH` to be true by default, thereby allowing Open Graph tags to be passed through by default
+- Added the ability to [customize Anubis' HTTP status codes](./admin/configuration/custom-status-codes.mdx) ([#355](https://github.com/TecharoHQ/anubis/issues/355))
+
+## v1.17.0: Asahi sas Brutus
+
+- Ensure regexes can't end in newlines ([#372](https://github.com/TecharoHQ/anubis/issues/372))
+- Add documentation for default allow behavior (implicit rule)
+- Enable [importing configuration snippets](./admin/configuration/import.mdx) ([#321](https://github.com/TecharoHQ/anubis/pull/321))
+- Refactor check logic to be more generic and work on a Checker type
+- Add more AI user agents based on the [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) project
+- Embedded challenge data in initial HTML response to improve performance
+- Added support to use Nginx' `auth_request` directive with Anubis
+- Added support to allow to restrict the allowed redirect domains
+- Whitelisted [DuckDuckBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/) in botPolicies
+- Improvements to build scripts to make them less independent of the build host
+- Improved the Open Graph error logging
+- Added `Opera` to the `generic-browser` bot policy rule
+- Added FreeBSD rc.d script so can be run as a FreeBSD daemon
+- Allow requests from the Internet Archive
+- Added example nginx configuration to documentation
+- Added example Apache configuration to the documentation [#277](https://github.com/TecharoHQ/anubis/issues/277)
+- Move per-environment configuration details into their own pages
+- Added support for running anubis behind a prefix (e.g. `/myapp`)
+- Added headers support to bot policy rules
+- Moved configuration file from JSON to YAML by default
+- Added documentation on how to use Anubis with Traefik in Docker
+- Improved error handling in some edge cases
+- Disable `generic-bot-catchall` rule because of its high false positive rate in real-world scenarios
+- Moved all CSS inline to the Xess package, changed colors to be CSS variables
+- Set or append to `X-Forwarded-For` header unless the remote connects over a loopback address [#328](https://github.com/TecharoHQ/anubis/issues/328)
+- Fixed mojeekbot user agent regex
+- Added support for running anubis behind a base path (e.g. `/myapp`)
+- Reduce Anubis' paranoia with user cookies ([#365](https://github.com/TecharoHQ/anubis/pull/365))
+- Added support for Open Graph passthrough while using unix sockets
+- The Open Graph subsystem now passes the HTTP `HOST` header through to the origin
+- Updated the `OG_PASSTHROUGH` to be true by default, thereby allowing Open Graph tags to be passed through by default
+
+## v1.16.0
+
+Fordola rem Lupis
+
+> I want to make them pay! All of them! Everyone who ever mocked or looked down on me -- I want the power to make them pay!
+
+The following features are the "big ticket" items:
+
+- Added support for native Debian, Red Hat, and tarball packaging strategies including installation and use directions
+- A prebaked tarball has been added, allowing distros to build Anubis like they could in v1.15.x
+- The placeholder Anubis mascot has been replaced with a design by [CELPHASE](https://bsky.app/profile/celphase.bsky.social)
+- Verification page now shows hash rate and a progress bar for completion probability
+- Added support for [Open Graph tags](https://ogp.me/) when rendering the challenge page. This allows for social previews to be generated when sharing the challenge page on social media platforms ([#195](https://github.com/TecharoHQ/anubis/pull/195))
+- Added support for passing the ed25519 signing key in a file with `-ed25519-private-key-hex-file` or `ED25519_PRIVATE_KEY_HEX_FILE`
+
+The other small fixes have been made:
+
+- Added a periodic cleanup routine for the decaymap that removes expired entries, ensuring stale data is properly pruned
 - Added a no-store Cache-Control header to the challenge page
 - Hide the directory listings for Anubis' internal static content
-- Changed `--debug-x-real-ip-default` to `--use-remote-address`, getting the IP address from the request's socket address instead.
+- Changed `--debug-x-real-ip-default` to `--use-remote-address`, getting the IP address from the request's socket address instead
 - DroneBL lookups have been disabled by default
 - Static asset builds are now done on demand instead of the results being committed to source control
 - The Dockerfile has been removed as it is no longer in use
 - Developer documentation has been added to the docs site
 - Show more errors when some predictable challenge page errors happen ([#150](https://github.com/TecharoHQ/anubis/issues/150))
-- Verification page now shows hash rate and a progress bar for completion probability.
-- Added the `--debug-benchmark-js` flag for testing proof-of-work performance during development.
+- Added the `--debug-benchmark-js` flag for testing proof-of-work performance during development
 - Use `TrimSuffix` instead of `TrimRight` on containerbuild
 - Fix the startup logs to correctly show the address and port the server is listening on
 - Add [LibreJS](https://www.gnu.org/software/librejs/) banner to Anubis JavaScript to allow LibreJS users to run the challenge
 - Added a wait with button continue + 30 second auto continue after 30s if you click "Why am I seeing this?"
-- Fixed a typo in the challenge page title.
-- Disabled running integration tests on Windows hosts due to it's reliance on posix features (see [#133](https://github.com/TecharoHQ/anubis/pull/133#issuecomment-2764732309)).
-- Added support for passing the ed25519 signing key in a file with `-ed25519-private-key-hex-file` or `ED25519_PRIVATE_KEY_HEX_FILE`.
+- Fixed a typo in the challenge page title
+- Disabled running integration tests on Windows hosts due to it's reliance on posix features (see [#133](https://github.com/TecharoHQ/anubis/pull/133#issuecomment-2764732309))
 - Fixed minor typos
-- Added a Makefile to enable comfortable workflows for downstream packagers.
+- Added a Makefile to enable comfortable workflows for downstream packagers
 - Added `zizmor` for GitHub Actions static analysis
 - Fixed most `zizmor` findings
 - Enabled Dependabot
 - Added an air config for autoreload support in development ([#195](https://github.com/TecharoHQ/anubis/pull/195))
-- Added support for [OpenGraph tags](https://ogp.me/) when rendering the challenge page. This allows for social previews to be generated when sharing the challenge page on social media platforms ([#195](https://github.com/TecharoHQ/anubis/pull/195))
-- Added an `--extract-resources` flag to extract static resources to a local folder.
-- Add noindex flag to all Anubis pages ([#227](https://github.com/TecharoHQ/anubis/issues/227)).
+- Added an `--extract-resources` flag to extract static resources to a local folder
+- Add noindex flag to all Anubis pages ([#227](https://github.com/TecharoHQ/anubis/issues/227))
 - Added `WEBMASTER_EMAIL` variable, if it is present then display that email address on error pages ([#235](https://github.com/TecharoHQ/anubis/pull/235), [#115](https://github.com/TecharoHQ/anubis/issues/115))
+- Hash pinned all GitHub Actions
 
 ## v1.15.1
 
@@ -124,7 +237,7 @@ Livia sas Junius
   [#21](https://github.com/TecharoHQ/anubis/pull/21)
 - Don't overflow the image when browser windows are small (eg. on phones)
   [#27](https://github.com/TecharoHQ/anubis/pull/27)
-- Lower the default difficulty to 4 from 5
+- Lower the default difficulty to 5 from 4
 - Don't duplicate work across multiple threads [#36](https://github.com/TecharoHQ/anubis/pull/36)
 - Documentation has been moved to https://anubis.techaro.lol/ with sources in docs/
 - Removed several visible AI artifacts (e.g., 6 fingers) [#37](https://github.com/TecharoHQ/anubis/pull/37)
@@ -167,4 +280,4 @@ Livia sas Junius
   ([fd6903a](https://github.com/TecharoHQ/anubis/commit/fd6903aeed315b8fddee32890d7458a9271e4798)).
 - Footer links on the check page now point to Techaro's brand
   ([4ebccb1](https://github.com/TecharoHQ/anubis/commit/4ebccb197ec20d024328d7f92cad39bbbe4d6359))
-- Anubis was imported from [Xe/x](https://github.com/Xe/x).
+- Anubis was imported from [Xe/x](https://github.com/Xe/x)

docs/docs/admin/configuration/custom-status-codes.mdx

@@ -0,0 +1,19 @@
+# Custom status codes for Anubis errors
+
+Out of the box, Anubis will reply with `HTTP 200` for challenge and denial pages. This is intended to make AI scrapers have a hard time with your website because when they are faced with a non-200 response, they will hammer the page over and over until they get a 200 response. This behavior may not be desirable, as such Anubis lets you customize what HTTP status codes are returned when Anubis throws challenge and denial pages.
+
+This is configured in the `status_codes` block of your [bot policy file](../policies.mdx):
+
+```yaml
+status_codes:
+  CHALLENGE: 200
+  DENY: 200
+```
+
+To match CloudFlare's behavior, use a configuration like this:
+
+```yaml
+status_codes:
+  CHALLENGE: 403
+  DENY: 403
+```

docs/docs/admin/configuration/expressions.mdx

@@ -0,0 +1,150 @@
+# Expression-based rule matching
+
+Most of the Anubis matchers let you match individual parts of a request and only those parts in isolation. In order to defend a service in depth, you often need the ability to match against multiple aspects of a request. Anubis implements [Common Expression Language (CEL)](https://cel.dev) to let administrators define these more advanced rules. This allows you to tailor your approach for the individual services you are protecting.
+
+As an example, here is a rule that lets you allow JSON API requests through Anubis:
+
+```yaml
+- name: allow-api-requests
+  action: ALLOW
+  expression:
+    all:
+      - '"Accept" in headers'
+      - 'headers["Accept"] == "application/json"'
+      - 'path.startsWith("/api/")'
+```
+
+This is an advanced feature and as such it is easy to get yourself in trouble with it. Use this with care.
+
+## Common Expression Language (CEL)
+
+CEL is an expression language made by Google as a part of their access control lists system. As programs grow more complicated and users have the need to express more complicated security requirements, they often want the ability to just run a small bit of code to check things for themselves. CEL expressions are built for this. They are implicitly sandboxed so that they cannot affect the system they are running in and also designed to evaluate as fast as humanly possible.
+
+Imagine a CEL expression as the contents of an `if` statement in JavaScript or the `WHERE` clause in SQL. Consider this example expression:
+
+```python
+userAgent == ""
+```
+
+This is roughly equivalent to the following in JavaScript:
+
+```js
+if (userAgent == "") {
+  // Do something
+}
+```
+
+Using these expressions, you can define more elaborate rules as facts and circumstances demand. For more information about the syntax and grammar of CEL, take a look at [the language specification](https://github.com/google/cel-spec/blob/master/doc/langdef.md).
+
+## How Anubis uses CEL
+
+Anubis uses CEL to let administrators create complicated filter rules. Anubis has several modes of using CEL:
+
+- Validating requests against single expressions
+- Validating multiple expressions and ensuring at least one of them are true (`any`)
+- Validating multiple expressions and ensuring all of them are true (`all`)
+
+The common pattern is that every Anubis expression returns `true`, `false`, or raises an error.
+
+### Single expressions
+
+A single expression that returns either `true` or `false`. If the expression returns `true`, then the action specified in the rule will be taken. If it returns `false`, Anubis will move on to the next rule.
+
+For example, consider this rule:
+
+```yaml
+- name: no-user-agent-string
+  action: DENY
+  expression: userAgent == ""
+```
+
+For this rule, if a request comes in without a [`User-Agent` string](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent) set, Anubis will deny the request and return an error page.
+
+### `any` blocks
+
+An `any` block that contains a list of expressions. If any expression in the list returns `true`, then the action specified in the rule will be taken. If all expressions in that list return `false`, Anubis will move on to the next rule.
+
+For example, consider this rule:
+
+```yaml
+- name: known-banned-user
+  action: DENY
+  expression:
+    any:
+      - remoteAddress == "8.8.8.8"
+      - remoteAddress == "1.1.1.1"
+```
+
+For this rule, if a request comes in from `8.8.8.8` or `1.1.1.1`, Anubis will deny the request and return an error page.
+
+#### `all` blocks
+
+An `all` block that contains a list of expressions. If all expressions in the list return `true`, then the action specified in the rule will be taken. If any of the expressions in the list returns `false`, Anubis will move on to the next rule.
+
+For example, consider this rule:
+
+```yaml
+- name: go-get
+  action: ALLOW
+  expression:
+    all:
+      - userAgent.startsWith("Go-http-client/")
+      - '"go-get" in query'
+      - query["go-get"] == "1"
+```
+
+For this rule, if a request comes in matching [the signature of the `go get` command](https://pkg.go.dev/cmd/go#hdr-Remote_import_paths), Anubis will allow it through to the target.
+
+## Variables exposed to Anubis expressions
+
+Anubis exposes the following variables to expressions:
+
+| Name            | Type                  | Explanation                                                                                                                               | Example                                                      |
+| :-------------- | :-------------------- | :---------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------- |
+| `headers`       | `map[string, string]` | The [headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers) of the request being processed.                        | `{"User-Agent": "Mozilla/5.0 Gecko/20100101 Firefox/137.0"}` |
+| `host`          | `string`              | The [HTTP hostname](https://web.dev/articles/url-parts#host) the request is targeted to.                                                  | `anubis.techaro.lol`                                         |
+| `method`        | `string`              | The [HTTP method](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Methods) in the request being processed.                    | `GET`, `POST`, `DELETE`, etc.                                |
+| `path`          | `string`              | The [path](https://web.dev/articles/url-parts#pathname) of the request being processed.                                                   | `/`, `/api/memes/create`                                     |
+| `query`         | `map[string, string]` | The [query parameters](https://web.dev/articles/url-parts#query) of the request being processed.                                          | `?foo=bar` -> `{"foo": "bar"}`                               |
+| `remoteAddress` | `string`              | The IP address of the client.                                                                                                             | `1.1.1.1`                                                    |
+| `userAgent`     | `string`              | The [`User-Agent`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent) string in the request being processed. | `Mozilla/5.0 Gecko/20100101 Firefox/137.0`                   |
+
+Of note: in many languages when you look up a key in a map and there is nothing there, the language will return some "falsy" value like `undefined` in JavaScript, `None` in Python, or the zero value of the type in Go. In CEL, if you try to look up a value that does not exist, execution of the expression will fail and Anubis will return an error.
+
+In order to avoid this, make sure the header or query parameter you are testing is present in the request with an `all` block like this:
+
+```yaml
+- name: challenge-wiki-history-page
+  action: CHALLENGE
+  all:
+    - 'path == "/index.php"'
+    - '"title" in query'
+    - '"action" in query'
+    - 'query["action"] == "history"
+```
+
+This rule throws a challenge if and only if all of the following conditions are true:
+
+- The URL path is `/index.php`
+- The URL query string contains a `title` value
+- The URL query string contains an `action` value
+- The URL query string's `action` value is `"history"`
+
+So given an HTTP request like this:
+
+```text
+GET /index.php?title=Index&action=history HTTP/1.1
+User-Agent: Mozilla/5.0 Gecko/20100101 Firefox/137.0
+Host: wiki.int.techaro.lol
+X-Real-Ip: 8.8.8.8
+```
+
+Anubis would return a challenge because all of those conditions are true.
+
+## Functions exposed to Anubis expressions
+
+There are currently no functions from the Anubis runtime exposed to expressions. This will change in the future.
+
+## Life advice
+
+Expressions are very powerful. This is a benefit and a burden. If you are not careful with your expression targeting, you will be liable to get yourself into trouble. If you are at all in doubt, throw a `CHALLENGE` over a `DENY`. Legitimate users can easily work around a `CHALLENGE` result with a [proof of work challenge](../../design/why-proof-of-work.mdx). Bots are less likely to be able to do this.

docs/docs/admin/configuration/import.mdx

@@ -0,0 +1,186 @@
+# Importing configuration rules
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+Anubis has the ability to let you import snippets of configuration into the main configuration file. This allows you to break up your config into smaller parts that get logically assembled into one big file.
+
+EG:
+
+<Tabs>
+<TabItem value="json" label="JSON">
+
+```json
+{
+  "bots": [
+    {
+      "import": "(data)/bots/ai-robots-txt.yaml"
+    },
+    {
+      "import": "(data)/bots/cloudflare-workers.yaml"
+    }
+  ]
+}
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML" default>
+
+```yaml
+bots:
+  # Pathological bots to deny
+  - # This correlates to data/bots/ai-robots-txt.yaml in the source tree
+    import: (data)/bots/ai-robots-txt.yaml
+  - import: (data)/bots/cloudflare-workers.yaml
+```
+
+</TabItem>
+</Tabs>
+
+Of note, a bot rule can either have inline bot configuration or import a bot config snippet. You cannot do both in a single bot rule.
+
+<Tabs>
+<TabItem value="json" label="JSON">
+
+```json
+{
+  "bots": [
+    {
+      "import": "(data)/bots/ai-robots-txt.yaml",
+      "name": "generic-browser",
+      "user_agent_regex": "Mozilla|Opera\n",
+      "action": "CHALLENGE"
+    }
+  ]
+}
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML" default>
+
+```yaml
+bots:
+  - import: (data)/bots/ai-robots-txt.yaml
+    name: generic-browser
+    user_agent_regex: >
+      Mozilla|Opera
+    action: CHALLENGE
+```
+
+</TabItem>
+</Tabs>
+
+This will return an error like this:
+
+```text
+config is not valid:
+config.BotOrImport: rule definition is invalid, you must set either bot rules or an import statement, not both
+```
+
+Paths can either be prefixed with `(data)` to import from the [the data folder in the Anubis source tree](https://github.com/TecharoHQ/anubis/tree/main/data) or anywhere on the filesystem. If you don't have access to the Anubis source tree, check /usr/share/docs/anubis/data or in the tarball you extracted Anubis from.
+
+## Importing from imports
+
+You can also import from an imported file in case you want to import an entire folder of rules at once.
+
+<Tabs>
+<TabItem value="json" label="JSON">
+
+```json
+{
+  "bots": [
+    {
+      "import": "(data)/bots/_deny-pathological.yaml"
+    }
+  ]
+}
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML" default>
+
+```yaml
+bots:
+  - import: (data)/bots/_deny-pathological.yaml
+```
+
+</TabItem>
+</Tabs>
+
+This lets you import an entire ruleset at once:
+
+```yaml
+# (data)/bots/_deny-pathological.yaml
+- import: (data)/bots/cloudflare-workers.yaml
+- import: (data)/bots/headless-browsers.yaml
+- import: (data)/bots/us-ai-scraper.yaml
+```
+
+Use this with care, you can easily get yourself into a state where Anubis recursively imports things for eternity if you are not careful. The best way to use this is to make a "root import" named `_everything.yaml` or `_allow-good.yaml` so they sort to the top. Name your meta-imports after the main verb they are enforcing so that you can glance at the configuration file and understand what it's doing.
+
+## Writing snippets
+
+Snippets can be written in either JSON or YAML, with a preference for YAML. When writing a snippet, write the bot rules you want directly at the top level of the file in a list.
+
+Here is an example snippet that allows [IPv6 Unique Local Addresses](https://en.wikipedia.org/wiki/Unique_local_address) through Anubis:
+
+<Tabs>
+<TabItem value="json" label="JSON">
+
+```json
+[
+  {
+    "name": "ipv6-ula",
+    "action": "ALLOW",
+    "remote_addresses": ["fc00::/7"]
+  }
+]
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML" default>
+
+```yaml
+- name: ipv6-ula
+  action: ALLOW
+  remote_addresses:
+    - fc00::/7
+```
+
+</TabItem>
+</Tabs>
+
+## Extracting Anubis' embedded filesystem
+
+You can always extract the list of rules embedded into the Anubis binary with this command:
+
+```text
+anubis --extract-resources=static
+```
+
+This will dump the contents of Anubis' embedded data to a new folder named `static`:
+
+```text
+static
+├── apps
+│   └── gitea-rss-feeds.yaml
+├── botPolicies.json
+├── botPolicies.yaml
+├── bots
+│   ├── ai-robots-txt.yaml
+│   ├── cloudflare-workers.yaml
+│   ├── headless-browsers.yaml
+│   └── us-ai-scraper.yaml
+├── common
+│   ├── allow-private-addresses.yaml
+│   └── keep-internet-working.yaml
+└── crawlers
+    ├── bingbot.yaml
+    ├── duckduckbot.yaml
+    ├── googlebot.yaml
+    ├── internet-archive.yaml
+    ├── kagibot.yaml
+    ├── marginalia.yaml
+    ├── mojeekbot.yaml
+    └── qwantbot.yaml
+```

docs/docs/admin/configuration/open-graph.mdx

@@ -5,14 +5,15 @@ title: Open Graph Configuration
 
 # Open Graph Configuration
 
-This page provides detailed information on how to configure [OpenGraph tag](https://ogp.me/) passthrough in Anubis. This enables social previews of resources protected by Anubis without having to exempt each scraper individually.
+This page provides detailed information on how to configure [Open Graph tag](https://ogp.me/) passthrough in Anubis. This enables social previews of resources protected by Anubis without having to exempt each scraper individually.
 
 ## Configuration Options
 
-| Name             | Description                                               | Type     | Default | Example                 |
-|------------------|-----------------------------------------------------------|----------|---------|-------------------------|
-| `OG_PASSTHROUGH` | Enables or disables the Open Graph tag passthrough system | Boolean  | `false` | `OG_PASSTHROUGH=true`   |
-| `OG_EXPIRY_TIME` | Configurable cache expiration time for Open Graph tags    | Duration | `24h`   | `OG_EXPIRY_TIME=1h`     |
+| Name                     | Description                                               | Type     | Default | Example                       |
+| ------------------------ | --------------------------------------------------------- | -------- | ------- | ----------------------------- |
+| `OG_PASSTHROUGH`         | Enables or disables the Open Graph tag passthrough system | Boolean  | `true`  | `OG_PASSTHROUGH=true`         |
+| `OG_EXPIRY_TIME`         | Configurable cache expiration time for Open Graph tags    | Duration | `24h`   | `OG_EXPIRY_TIME=1h`           |
+| `OG_CACHE_CONSIDER_HOST` | Enables or disables the use of the host in the cache key  | Boolean  | `false` | `OG_CACHE_CONSIDER_HOST=true` |
 
 ## Usage
 
@@ -21,6 +22,7 @@ To configure Open Graph tags, you can set the following environment variables, e
 ```sh
 export OG_PASSTHROUGH=true
 export OG_EXPIRY_TIME=1h
+export OG_CACHE_CONSIDER_HOST=false
 ```
 
 ## Implementation Details
@@ -33,6 +35,8 @@ When `OG_PASSTHROUGH` is enabled, Anubis will:
 
 The cache expiration time is controlled by `OG_EXPIRY_TIME`.
 
+When `OG_CACHE_CONSIDER_HOST` is enabled, Anubis will include the host in the cache key for Open Graph tags. This ensures that tags are cached separately for different hosts.
+
 ## Example
 
 Here is an example of how to configure Open Graph tags in your Anubis setup:
@@ -40,8 +44,19 @@ Here is an example of how to configure Open Graph tags in your Anubis setup:
 ```sh
 export OG_PASSTHROUGH=true
 export OG_EXPIRY_TIME=1h
+export OG_CACHE_CONSIDER_HOST=false
 ```
 
-With these settings, Anubis will cache Open Graph tags for 1 hour and pass them through to the challenge page.
+With these settings, Anubis will cache Open Graph tags for 1 hour and pass them through to the challenge page, not considering the host in the cache key.
+
+## When to Enable `OG_CACHE_CONSIDER_HOST`
+
+In most cases, you would want to keep `OG_CACHE_CONSIDER_HOST` set to `false` to avoid unnecessary cache fragmentation. However, there are some scenarios where enabling this option can be beneficial:
+
+1. **Multi-Tenant Applications**: If you are running a multi-tenant application where different tenants are hosted on different subdomains, enabling `OG_CACHE_CONSIDER_HOST` ensures that the Open Graph tags are cached separately for each tenant. This prevents one tenant's Open Graph tags from being served to another tenant's users.
+
+2. **Different Content for Different Hosts**: If your application serves different content based on the host, enabling `OG_CACHE_CONSIDER_HOST` ensures that the correct Open Graph tags are cached and served for each host. This is useful for applications that have different branding or content for different domains or subdomains.
+
+3. **Security and Privacy Concerns**: In some cases, you may want to ensure that Open Graph tags are not shared between different hosts for security or privacy reasons. Enabling `OG_CACHE_CONSIDER_HOST` ensures that the tags are cached separately for each host, preventing any potential leakage of information between hosts.
 
 For more information, refer to the [installation guide](../installation).

docs/docs/admin/configuration/redirect-domains.mdx

@@ -0,0 +1,94 @@
+---
+title: Redirect Domain Configuration
+---
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+Anubis has an HTTP redirect in the middle of its check validation logic. This redirect allows Anubis to set a cookie on validated requests so that users don't need to pass challenges on every page load.
+
+This flow looks something like this:
+
+```mermaid
+sequenceDiagram
+  participant User
+  participant Challenge
+  participant Validation
+  participant Backend
+
+  User->>+Challenge: GET /
+  Challenge->>+User: Solve this challenge
+  User->>+Validation: Here's the solution, send me to /
+  Validation->>+User: Here's a cookie, go to /
+  User->>+Backend: GET /
+```
+
+However, in some cases a sufficiently dedicated attacker could trick a user into clicking on a validation link with a solution pre-filled out. For example:
+
+```mermaid
+sequenceDiagram
+  participant Hacker
+  participant User
+  participant Validation
+  participant Evil Site
+
+  Hacker->>+User: Click on yoursite.com with this solution
+  User->>+Validation: Here's a solution, send me to evilsite.com
+  Validation->>+User: Here's a cookie, go to evilsite.com
+  User->>+Evil Site: GET evilsite.com
+```
+
+If this happens, Anubis will throw an error like this:
+
+```text
+Redirect domain not allowed
+```
+
+## Configuring allowed redirect domains
+
+By default, Anubis will limit redirects to be on the same HTTP Host that Anubis is running on (EG: requests to yoursite.com cannot redirect outside of yoursite.com). If you need to set more than one domain, fill the `REDIRECT_DOMAINS` environment variable with a comma-separated list of domain names that Anubis should allow redirects to.
+
+:::note
+
+These domains are _an exact string match_, they do not support wildcard matches.
+
+:::
+
+<Tabs>
+  <TabItem value="env-file" label="Environment file" default>
+
+```shell
+# anubis.env
+
+REDIRECT_DOMAINS="yoursite.com,secretplans.yoursite.com"
+# ...
+```
+
+  </TabItem>
+  <TabItem value="docker-compose" label="Docker Compose">
+
+```yaml
+services:
+  anubis-nginx:
+    image: ghcr.io/techarohq/anubis:latest
+    environment:
+      REDIRECT_DOMAINS: "yoursite.com,secretplans.yoursite.com"
+      # ...
+```
+
+  </TabItem>
+  <TabItem value="k8s" label="Kubernetes">
+
+Inside your Deployment, StatefulSet, or Pod:
+
+```yaml
+- name: anubis
+  image: ghcr.io/techarohq/anubis:latest
+  env:
+    - name: REDIRECT_DOMAINS
+      value: "yoursite.com,secretplans.yoursite.com"
+    # ...
+```
+
+  </TabItem>
+</Tabs>

docs/docs/admin/configuration/subrequest-auth.mdx

@@ -0,0 +1,144 @@
+---
+title: Subrequest Authentication
+---
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+Anubis can act in one of two modes:
+
+1. Reverse proxy (the default): Anubis sits in the middle of all traffic and then will reverse proxy it to its destination. This is the moral equivalent of a middleware in your favorite web framework.
+2. Subrequest authentication mode: Anubis listens for requests and if they don't pass muster then they are forwarded to Anubis for challenge processing. This is the equivalent of Anubis being a sidecar service.
+
+## Nginx
+
+Anubis can perform [subrequest authentication](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-subrequest-authentication/) with the `auth_request` module in Nginx. In order to set this up, keep the following things in mind:
+
+The `TARGET` environment variable in Anubis must be set to a space, eg:
+
+<Tabs>
+  <TabItem value="env-file" label="Environment file" default>
+
+```shell
+# anubis.env
+
+TARGET=" "
+# ...
+```
+
+  </TabItem>
+  <TabItem value="docker-compose" label="Docker Compose">
+
+```yaml
+services:
+  anubis-nginx:
+    image: ghcr.io/techarohq/anubis:latest
+    environment:
+      TARGET: " "
+      # ...
+```
+
+  </TabItem>
+  <TabItem value="k8s" label="Kubernetes">
+
+Inside your Deployment, StatefulSet, or Pod:
+
+```yaml
+- name: anubis
+  image: ghcr.io/techarohq/anubis:latest
+  env:
+    - name: TARGET
+      value: " "
+    # ...
+```
+
+  </TabItem>
+</Tabs>
+
+In order to configure this, you need to add the following location blocks to each server pointing to the service you want to protect:
+
+```nginx
+location /.within.website/ {
+    # Assumption: Anubis is running in the same network namespace as
+    # nginx on localhost TCP port 8923
+    proxy_pass http://127.0.0.1:8923;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header Host $http_host;
+    proxy_pass_request_body off;
+    proxy_set_header content-length "";
+    auth_request off;
+}
+
+location @redirectToAnubis {
+    return 307 /.within.website/?redir=$scheme://$host$request_uri;
+    auth_request off;
+}
+```
+
+This sets up `/.within.website` to point to Anubis. Any requests that Anubis rejects or throws a challenge to will be sent here. This also sets up a named location `@redirectToAnubis` that will redirect any requests to Anubis for advanced processing.
+
+Finally, add this to your root location block:
+
+```nginx
+location / {
+    # diff-add
+    auth_request /.within.website/x/cmd/anubis/api/check;
+    # diff-add
+    error_page 401 = @redirectToAnubis;
+}
+```
+
+This will check all requests that don't match other locations with Anubis to ensure the client is genuine.
+
+This will make every request get checked by Anubis before it hits your backend. If you have other locations that don't need Anubis to do validation, add the `auth_request off` directive to their blocks:
+
+```nginx
+location /secret {
+    # diff-add
+    auth_request off;
+
+    # ...
+}
+```
+
+Here is a complete example of an Nginx server listening over TLS and pointing to Anubis:
+
+<details>
+  <summary>Complete example</summary>
+
+```nginx
+# /etc/nginx/conf.d/nginx.local.cetacean.club.conf
+
+server {
+  listen 443 ssl;
+  listen [::]:443 ssl;
+  server_name         nginx.local.cetacean.club;
+  ssl_certificate     /etc/techaro/pki/nginx.local.cetacean.club/tls.crt;
+  ssl_certificate_key /etc/techaro/pki/nginx.local.cetacean.club/tls.key;
+  ssl_protocols       TLSv1.2 TLSv1.3;
+  ssl_ciphers         HIGH:!aNULL:!MD5;
+
+  proxy_set_header X-Real-IP $remote_addr;
+  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+
+  location /.within.website/ {
+    proxy_pass http://localhost:8923;
+    auth_request off;
+  }
+
+  location @redirectToAnubis {
+    return 307 /.within.website/?redir=$scheme://$host$request_uri;
+    auth_request off;
+  }
+
+  location / {
+    auth_request /.within.website/x/cmd/anubis/api/check;
+    error_page 401 = @redirectToAnubis;
+    root /usr/share/nginx/html;
+    index index.html index.htm;
+  }
+}
+```
+
+</details>

docs/docs/admin/default-allow-behavior.mdx

@@ -0,0 +1,92 @@
+---
+title: Default allow behavior
+---
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+# Default allow behavior
+
+Anubis is designed to be as unintrusive as possible to your existing infrastructure.
+
+By default, it allows all traffic unless a request matches a rule that explicitly denies or challenges it.
+
+Only requests matching a DENY or CHALLENGE rule are blocked or challenged. All other requests are allowed. This is called "the implicit rule".
+
+## Example: Minimal policy
+
+If your policy only blocks a specific bot, all other requests will be allowed:
+
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
+```json
+{
+  "bots": [
+    {
+      "name": "block-amazonbot",
+      "user_agent_regex": "Amazonbot",
+      "action": "DENY"
+    }
+  ]
+}
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+- name: block-amazonbot
+  user_agent_regex: Amazonbot
+  action: DENY
+```
+
+</TabItem>
+</Tabs>
+
+## How to deny by default
+
+If you want to deny all traffic except what you explicitly allow, add a catch-all deny rule at the end of your policy list. Make sure to add ALLOW rules for any traffic you want to permit before this rule.
+
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
+```json
+{
+  "bots": [
+    {
+      "name": "allow-goodbot",
+      "user_agent_regex": "GoodBot",
+      "action": "ALLOW"
+    },
+    {
+      "name": "catch-all-deny",
+      "path_regex": ".*",
+      "action": "DENY"
+    }
+  ]
+}
+```
+
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+- name: allow-goodbot
+  user_agent_regex: GoodBot
+  action: ALLOW
+- name: catch-all-deny
+  path_regex: .*
+  action: DENY
+```
+
+</TabItem>
+</Tabs>
+
+## Final remarks
+
+- Rules are evaluated in order; the first match wins.
+- The implicit allow rule is always last and cannot be removed.
+- Use your logs to monitor what traffic is being allowed by default.
+
+See [Policy Definitions](./policies) for more details on writing rules.

docs/docs/admin/environments/_category_.json

@@ -0,0 +1,8 @@
+{
+  "label": "Environments",
+  "position": 20,
+  "link": {
+    "type": "generated-index",
+    "description": "Detailed information about individual environments (such as HTTP servers, platforms, etc.) Anubis is known to work with."
+  }
+}

docs/docs/admin/environments/apache.mdx

@@ -0,0 +1,151 @@
+# Apache
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
+Anubis is intended to be a filter proxy. The way to integrate this is to break your configuration up into two parts: TLS termination and then HTTP routing. Consider this diagram:
+
+```mermaid
+---
+title: Apache as tls terminator and HTTP router
+---
+
+flowchart LR
+    T(User Traffic)
+    subgraph Apache 2
+        TCP(TCP 80/443)
+        US(TCP 3001)
+    end
+
+    An(Anubis)
+    B(Backend)
+
+    T --> |TLS termination| TCP
+    TCP --> |Traffic filtering| An
+    An --> |Happy traffic| US
+    US --> |whatever you're doing| B
+```
+
+Effectively you have one trip through Apache to do TLS termination, a detour through Anubis for traffic scrubbing, and then going to the backend directly. This final socket is what will do HTTP routing.
+
+:::note
+
+These examples assume that you are using a setup where your nginx configuration is made up of a bunch of files in `/etc/httpd/conf.d/*.conf`. This is not true for all deployments of Apache. If you are not in such an environment, append these snippets to your `/etc/httpd/conf/httpd.conf` file.
+
+:::
+
+## Dependencies
+
+Install the following dependencies for proxying HTTP:
+
+<Tabs>
+  <TabItem value="rpm" label="Red Hat / RPM" default>
+
+```text
+dnf -y install mod_proxy_html
+```
+
+  </TabItem>
+  <TabItem value="deb" label="Debian / Ubuntu / apt">
+
+```text
+apt-get install -y libapache2-mod-proxy-html libxml2-dev
+```
+
+  </TabItem>
+</Tabs>
+
+## Configuration
+
+Assuming you are protecting `anubistest.techaro.lol`, you need the following server configuration blocks:
+
+1. A block on port 80 that forwards HTTP to HTTPS
+2. A block on port 443 that terminates TLS and forwards to Anubis
+3. A block on port 3001 that actually serves your websites
+
+```text
+# Plain HTTP redirect to HTTPS
+<VirtualHost *:80>
+       ServerAdmin your@email.here
+       ServerName anubistest.techaro.lol
+       DocumentRoot /var/www/anubistest.techaro.lol
+       ErrorLog /var/log/httpd/anubistest.techaro.lol_error.log
+       CustomLog /var/log/httpd/anubistest.techaro.lol_access.log combined
+       RewriteEngine on
+       RewriteCond %{SERVER_NAME} =anubistest.techaro.lol
+       RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
+</VirtualHost>
+
+# HTTPS listener that forwards to Anubis
+<VirtualHost *:443>
+       ServerAdmin your@email.here
+       ServerName anubistest.techaro.lol
+       DocumentRoot /var/www/anubistest.techaro.lol
+       ErrorLog /var/log/httpd/anubistest.techaro.lol_error.log
+       CustomLog /var/log/httpd/anubistest.techaro.lol_access.log combined
+
+       SSLCertificateFile /etc/letsencrypt/live/anubistest.techaro.lol/fullchain.pem
+       SSLCertificateKeyFile /etc/letsencrypt/live/anubistest.techaro.lol/privkey.pem
+       Include /etc/letsencrypt/options-ssl-apache.conf
+
+       # These headers need to be set or else Anubis will
+       # throw an "admin misconfiguration" error.
+       RequestHeader set "X-Real-Ip" expr=%{REMOTE_ADDR}
+       RequestHeader set X-Forwarded-Proto "https"
+
+       ProxyPreserveHost On
+
+       ProxyRequests Off
+       ProxyVia Off
+
+       # Replace 9000 with the port Anubis listens on
+       ProxyPass / http://[::1]:9000/
+       ProxyPassReverse / http://[::1]:9000/
+</VirtualHost>
+</IfModule>
+
+# Actual website config
+<VirtualHost *:3001>
+       ServerAdmin your@email.here
+       ServerName anubistest.techaro.lol
+       DocumentRoot /var/www/anubistest.techaro.lol
+       ErrorLog /var/log/httpd/anubistest.techaro.lol_error.log
+       CustomLog /var/log/httpd/anubistest.techaro.lol_access.log combined
+</VirtualHost>
+```
+
+Make sure to add a separate configuration file for the listener on port 3001:
+
+```text
+# /etc/httpd/conf.d/listener-3001.conf
+
+Listen 127.0.0.1:3001
+```
+
+This can be repeated for multiple sites. Anubis does not care about the HTTP `Host` header and will happily cope with multiple websites via the same instance.
+
+Then reload your Apache config and load your website. You should see Anubis protecting your apps!
+
+```text
+sudo systemctl reload httpd.service
+```
+
+## Troubleshooting
+
+Here are some answers to questions that came in in testing:
+
+### I'm running on a Red Hat distribution and Apache is saying "service unavailable" for every page load
+
+If you see a "Service unavailable" error on every page load and run a Red Hat derived distribution, you are missing a `selinux` setting. The exact command will be in a journalctl log message like this:
+
+```text
+*****  Plugin catchall_boolean (89.3 confidence) suggests   ******************
+
+If you want to allow HTTPD scripts and modules to connect to the network using TCP.
+Then you must tell SELinux about this by enabling the 'httpd_can_network_connect' boolean.
+
+Do
+setsebool -P httpd_can_network_connect 1
+```
+
+This will fix the error immediately.

docs/docs/admin/environments/caddy.mdx

@@ -0,0 +1,71 @@
+# Caddy
+
+To use Anubis with Caddy, stick Anubis between Caddy and your backend. For example, consider this application setup:
+
+```mermaid
+---
+title: Caddy with Anubis in the middle
+---
+
+flowchart LR
+    T(User Traffic)
+    TCP(TCP 80/443)
+    An(Anubis)
+    B(Backend)
+    Blocked
+
+    T --> TCP
+    TCP --> |Traffic filtering| An
+    An --> |Happy traffic| B
+    An --> |Malicious traffic| Blocked
+```
+
+Instead of your traffic going directly to your backend, it takes a detour through Anubis. Anubis filters out the "bad" traffic and passes the "good" traffic to the backend.
+
+To set up Anubis with Docker compose and Caddy, start with a `docker-compose` configuration like this:
+
+```yaml
+services:
+  caddy:
+    image: caddy:2
+    ports:
+      - 80:80
+      - 443:443
+      - 443:443/udp
+    volumes:
+      - ./conf:/etc/caddy
+      - caddy_config:/config
+      - caddy_data:/data
+
+  anubis:
+    image: ghcr.io/techarohq/anubis:latest
+    pull_policy: always
+    environment:
+      BIND: ":3000"
+      TARGET: http://httpdebug:3000
+
+  httpdebug:
+    image: ghcr.io/xe/x/httpdebug
+    pull_policy: always
+
+volumes:
+  caddy_data:
+  caddy_config:
+```
+
+And then put the following in `conf/Caddyfile`:
+
+```Caddyfile
+# conf/Caddyfile
+
+yourdomain.example.com {
+  tls your@email.address
+
+  reverse_proxy http://anubis:3000 {
+		header_up X-Real-Ip {remote_host}
+		header_up X-Http-Version {http.request.proto}
+	}
+}
+```
+
+If you want to protect multiple services with Anubis, you will need to either start multiple instances of Anubis (Anubis requires less than 32 MB of ram on average) or set up a two-tier routing setup where TLS termination is done with one instance of Caddy and the actual routing to services is done with another instance of Caddy. See the [nginx](./nginx.mdx) or [Apache](./apache.mdx) documentation to get ideas on how you would do this.

docs/docs/admin/environments/docker-compose.mdx

@@ -0,0 +1,26 @@
+# Docker compose
+
+Docker compose is typically used in concert with other load balancers such as [Apache](./apache.mdx) or [Nginx](./nginx.mdx). Below is a minimal example showing you how to set up an instance of Anubis listening on host port 8080 that points to a static website containing data in `./www`:
+
+```yaml
+services:
+  anubis-nginx:
+    image: ghcr.io/techarohq/anubis:latest
+    environment:
+      BIND: ":8080"
+      DIFFICULTY: "4"
+      METRICS_BIND: ":9090"
+      SERVE_ROBOTS_TXT: "true"
+      TARGET: "http://nginx"
+      POLICY_FNAME: "/data/cfg/botPolicy.yaml"
+      OG_PASSTHROUGH: "true"
+      OG_EXPIRY_TIME: "24h"
+    ports:
+      - 8080:8080
+    volumes:
+      - "./botPolicy.yaml:/data/cfg/botPolicy.yaml:ro"
+  nginx:
+    image: nginx
+    volumes:
+      - "./www:/usr/share/nginx/html"
+```

docs/docs/admin/environments/kubernetes.mdx

@@ -0,0 +1,128 @@
+# Kubernetes
+
+When setting up Anubis in Kubernetes, you want to make sure that you thread requests through Anubis kinda like this:
+
+```mermaid
+---
+title: Anubis embedded into workload pods
+---
+
+flowchart LR
+    T(User Traffic)
+
+    IngressController(IngressController)
+
+    subgraph Service
+        AnPort(Anubis Port)
+        BPort(Backend Port)
+    end
+
+    subgraph Pod
+        An(Anubis)
+        B(Backend)
+    end
+
+    T -->  IngressController
+    IngressController --> AnPort
+    AnPort --> An
+    An --> B
+```
+
+Anubis is lightweight enough that you should be able to have many instances of it running without many problems. If this is a concern for you, please check out [ingress-anubis](https://github.com/jaredallard/ingress-anubis?ref=anubis.techaro.lol).
+
+This example makes the following assumptions:
+
+- Your target service is listening on TCP port `5000`.
+- Anubis will be listening on port `8080`.
+
+Adjust these values as facts and circumstances demand.
+
+Create a secret with the signing key Anubis should use for its responses:
+
+```
+kubectl create secret generic anubis-key \
+  --namespace default \
+  --from-literal=ED25519_PRIVATE_KEY_HEX=$(openssl rand -hex 32)
+```
+
+Attach Anubis to your Deployment:
+
+```yaml
+containers:
+  # ...
+  - name: anubis
+    image: ghcr.io/techarohq/anubis:latest
+    imagePullPolicy: Always
+    env:
+      - name: "BIND"
+        value: ":8080"
+      - name: "DIFFICULTY"
+        value: "4"
+      - name: ED25519_PRIVATE_KEY_HEX
+        valueFrom:
+          secretKeyRef:
+            name: anubis-key
+            key: ED25519_PRIVATE_KEY_HEX
+      - name: "METRICS_BIND"
+        value: ":9090"
+      - name: "SERVE_ROBOTS_TXT"
+        value: "true"
+      - name: "TARGET"
+        value: "http://localhost:5000"
+      - name: "OG_PASSTHROUGH"
+        value: "true"
+      - name: "OG_EXPIRY_TIME"
+        value: "24h"
+    resources:
+      limits:
+        cpu: 750m
+        memory: 256Mi
+      requests:
+        cpu: 250m
+        memory: 256Mi
+    securityContext:
+      runAsUser: 1000
+      runAsGroup: 1000
+      runAsNonRoot: true
+      allowPrivilegeEscalation: false
+      capabilities:
+        drop:
+          - ALL
+      seccompProfile:
+        type: RuntimeDefault
+```
+
+Then add a Service entry for Anubis:
+
+```yaml
+# ...
+spec:
+  ports:
+    # diff-add
+    - protocol: TCP
+      # diff-add
+      port: 8080
+      # diff-add
+      targetPort: 8080
+      # diff-add
+      name: anubis
+```
+
+Then point your Ingress to the Anubis port:
+
+```yaml
+  rules:
+  - host: git.xeserv.us
+    http:
+      paths:
+      - pathType: Prefix
+        path: "/"
+        backend:
+          service:
+            name: git
+            port:
+              # diff-remove
+              name: http
+              # diff-add
+              name: anubis
+```

docs/docs/admin/environments/nginx.mdx

@@ -0,0 +1,166 @@
+# Nginx
+
+Anubis is intended to be a filter proxy. The way to integrate this with nginx is to break your configuration up into two parts: TLS termination and then HTTP routing. Consider this diagram:
+
+```mermaid
+---
+title: Nginx as tls terminator and HTTP router
+---
+
+flowchart LR
+    T(User Traffic)
+    subgraph Nginx
+        TCP(TCP 80/443)
+        US(Unix Socket or
+another TCP port)
+    end
+
+    An(Anubis)
+    B(Backend)
+
+    T --> |TLS termination| TCP
+    TCP --> |Traffic filtering| An
+    An --> |Happy traffic| US
+    US --> |whatever you're doing| B
+```
+
+Instead of your traffic going right from TLS termination into the backend, it takes a detour through Anubis. Anubis filters out the "bad" traffic and then passes the "good" traffic to another socket that Nginx has open. This final socket is what you will use to do HTTP routing.
+
+Effectively, you have two roles for nginx: TLS termination (converting HTTPS to HTTP) and HTTP routing (distributing requests to the individual vhosts). This can stack with something like Apache in case you have a legacy deployment. Make sure you have the right [TLS certificates configured](https://code.kuederle.com/letsencrypt/) at the TLS termination level.
+
+:::note
+
+These examples assume that you are using a setup where your nginx configuration is made up of a bunch of files in `/etc/nginx/conf.d/*.conf`. This is not true for all deployments of nginx. If you are not in such an environment, append these snippets to your `/etc/nginx/nginx.conf` file.
+
+:::
+
+Assuming that we are protecting `anubistest.techaro.lol`, here's what the server configuration file would look like:
+
+```nginx
+# /etc/nginx/conf.d/server-anubistest-techaro-lol.conf
+
+# HTTP - Redirect all HTTP traffic to HTTPS
+server {
+  listen 80;
+  listen [::]:80;
+
+  server_name anubistest.techaro.lol;
+
+  location / {
+    return 301 https://$host$request_uri;
+  }
+}
+
+# TLS termination server, this will listen over TLS (https) and then
+# proxy all traffic to the target via Anubis.
+server {
+  # Listen on TCP port 443 with TLS (https) and HTTP/2
+  listen 443 ssl http2;
+  listen [::]:443 ssl http2;
+
+  location / {
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_pass http://anubis;
+  }
+
+  server_name anubistest.techaro.lol;
+
+  ssl_certificate     /path/to/your/certs/anubistest.techaro.lol.crt;
+  ssl_certificate_key /path/to/your/certs/anubistest.techaro.lol.key;
+}
+
+# Backend server, this is where your webapp should actually live.
+server {
+  listen unix:/run/nginx/nginx.sock;
+
+  server_name anubistest.techaro.lol;
+  root "/srv/http/anubistest.techaro.lol";
+  index index.html;
+
+  # Your normal configuration can go here
+  # location .php { fastcgi...} etc.
+}
+```
+
+:::tip
+
+You can copy the `location /` block into a separate file named something like `conf-anubis.inc` and then include it inline to other `server` blocks:
+
+```nginx
+# /etc/nginx/conf.d/conf-anubis.inc
+
+# Forward to anubis
+location / {
+  proxy_set_header Host $host;
+  proxy_set_header X-Real-IP $remote_addr;
+  proxy_pass http://anubis;
+}
+```
+
+Then in a server block:
+
+<details>
+<summary>Full nginx config</summary>
+
+```nginx
+# /etc/nginx/conf.d/server-mimi-techaro-lol.conf
+
+server {
+  # Listen on 443 with SSL
+  listen 443 ssl http2;
+  listen [::]:443 ssl http2;
+
+  # Slipstream via Anubis
+  include "conf-anubis.inc";
+
+  server_name mimi.techaro.lol;
+
+  ssl_certificate     /path/to/your/certs/mimi.techaro.lol.crt;
+  ssl_certificate_key /path/to/your/certs/mimi.techaro.lol.key;
+}
+
+server {
+  listen unix:/run/nginx/nginx.sock;
+
+  server_name mimi.techaro.lol;
+  root "/srv/http/mimi.techaro.lol";
+  index index.html;
+
+  # Your normal configuration can go here
+  # location .php { fastcgi...} etc.
+}
+```
+
+</details>
+
+:::
+
+Create an upstream for Anubis.
+
+```nginx
+# /etc/nginx/conf.d/upstream-anubis.conf
+
+upstream anubis {
+  # Make sure this matches the values you set for `BIND` and `BIND_NETWORK`.
+  # If this does not match, your services will not be protected by Anubis.
+
+  # Try anubis first over a UNIX socket
+  server unix:/run/anubis/nginx.sock;
+  #server 127.0.0.1:8923;
+
+  # Optional: fall back to serving the websites directly. This allows your
+  # websites to be resilient against Anubis failing, at the risk of exposing
+  # them to the raw internet without protection. This is a tradeoff and can
+  # be worth it in some edge cases.
+  #server unix:/run/nginx.sock backup;
+}
+```
+
+This can be repeated for multiple sites. Anubis does not care about the HTTP `Host` header and will happily cope with multiple websites via the same instance.
+
+Then reload your nginx config and load your website. You should see Anubis protecting your apps!
+
+```text
+sudo systemctl reload nginx.service
+```

docs/docs/admin/environments/traefik.mdx

@@ -0,0 +1,215 @@
+---
+id: traefik
+title: Traefik
+---
+
+
+:::note
+
+ This only talks about integration through Compose,
+ but it also applies to docker cli options.
+
+:::
+
+Currently, Anubis doesn't have any Traefik middleware,
+so you need to manually route it between Traefik and your target service.
+This routing is done per labels in Traefik.
+
+In this example, we will use 4 Containers:
+
+- `traefik` - the Traefik instance
+- `anubis` - the Anubis instance
+- `target` - our service to protect (`traefik/whoami` in this case)
+- `target2` - a second service that isn't supposed to be protected (`traefik/whoami` in this case)
+
+There are 3 steps we need to follow:
+
+1. Create a new exclusive Traefik endpoint for Anubis
+2. Pass all unspecified requests to Anubis
+3. Let Anubis pass all verified requests back to Traefik on its exclusive endpoint
+
+## Diagram of Flow
+
+This is a small diagram depicting the flow.
+Keep in mind that `8080` or `80` can be anything depending on your containers.
+
+```mermaid
+flowchart LR
+user[User]
+traefik[Traefik]
+anubis[Anubis]
+target[Target]
+
+user-->|:443 - Requesting Service|traefik
+traefik-->|:8080 - Passing to Anubis|anubis
+anubis-->|:3923 - Passing back to Traefik|traefik
+traefik-->|:80 - Passing to the target|target
+```
+
+## Create an Exclusive Anubis Endpoint in Traefik
+
+There are 2 ways of registering a new endpoint in Traefik.
+Which one to use depends on how you configured your Traefik so far.
+
+**CLI Options:**
+
+```yml
+--entrypoints.anubis.address=:3923
+```
+
+**traefik.yml:**
+
+```yml
+entryPoints:
+  anubis:
+    address: ":3923"
+```
+
+It is important that the specified port isn't actually reachable from the outside,
+but only exposed in the Docker network.
+Exposing the Anubis port on Traefik directly will allow direct unprotected access to all containers behind it.
+
+## Passing all unspecified Web Requests to Anubis
+
+There are cases where you want Traefik to still route some requests without protection, just like before.
+To achieve this, we can register Anubis as the default handler for non-protected requests.
+
+We also don't want users to get SSL Errors during the checking phase,
+thus we also need to let Traefik provide SSL Certs for our endpoint.
+This example expects an TLS cert resolver called `le`.
+
+We also expect there to be an endpoint called `websecure` for HTTPS in this example.
+
+This is an example of the required labels to configure Traefik on the Anubis container:
+
+```yml
+labels:
+  - traefik.enable=true # Enabling Traefik
+  - traefik.docker.network=traefik # Telling Traefik which network to use
+  - traefik.http.routers.anubis.priority=1 # Setting Anubis to the lowest priority, so it only takes the slack
+  - traefik.http.routers.anubis.rule=PathRegexp(`.*`) # Wildcard match every path
+  - traefik.http.routers.anubis.entrypoints=websecure # Listen on HTTPS
+  - traefik.http.services.anubis.loadbalancer.server.port=8080 # Telling Traefik to which port it should route requests
+  - traefik.http.routers.anubis.service=anubis # Telling Traefik to use the above specified port
+  - traefik.http.routers.anubis.tls.certresolver=le # Telling Traefik to resolve a Cert for Anubis
+```
+
+## Passing all Verified Requests Back Correctly to Traefik
+
+To pass verified requests back to Traefik,
+we only need to configure Anubis using its environment variables:
+
+```yml
+environment:
+  - BIND=:8080
+  - TARGET=http://traefik:3923
+```
+
+## Full Example Config
+
+Now that we know how to pass all requests back and forth, here is the example.
+This example contains 2 services: one that is protected and the other one that is not.
+
+**compose.yml**
+
+```yml
+services:
+  traefik:
+    image: traefik:v3.3
+    ports:
+      - 80:80
+      - 443:443
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - ./letsencrypt:/letsencrypt
+      - ./traefik.yml:/traefik.yml:ro
+    networks:
+      - traefik
+    labels:
+      # Enable Traefik
+      - traefik.enable=true
+      - traefik.docker.network=traefik
+      # Redirect any HTTP to HTTPS
+      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
+      - traefik.http.routers.web.rule=PathPrefix(`/`)
+      - traefik.http.routers.web.entrypoints=web
+      - traefik.http.routers.web.middlewares=redirect-to-https
+      - traefik.http.routers.web.tls=false
+
+  anubis:
+    image: ghcr.io/techarohq/anubis:main
+    environment:
+      # Telling Anubis, where to listen for Traefik
+      - BIND=:8080
+      # Telling Anubis to point to Traefik via the Docker network
+      - TARGET=http://traefik:3923
+    networks:
+      - traefik
+    labels:
+      - traefik.enable=true # Enabling Traefik
+      - traefik.docker.network=traefik # Telling Traefik which network to use
+      - traefik.http.routers.anubis.priority=1 # Setting Anubis to the lowest priority, so it only takes the slack
+      - traefik.http.routers.anubis.rule=PathRegexp(`.*`) # wildcard match anything
+      - traefik.http.routers.anubis.entrypoints=websecure # Listen on HTTPS
+      - traefik.http.services.anubis.loadbalancer.server.port=8080 # Telling Traefik to which port it should route requests
+      - traefik.http.routers.anubis.service=anubis # Telling Traefik to use the above specified port
+      - traefik.http.routers.anubis.tls.certresolver=le # Telling Traefik to resolve a Cert for Anubis
+
+  # Protected by Anubis
+  target:
+    image: traefik/whoami:latest
+    networks:
+      - traefik
+    labels:
+      - traefik.enable=true # Enabling Traefik
+      - traefik.docker.network=traefik # Telling Traefik which network to use
+      - traefik.http.routers.target.rule=Host(`example.com`) # Only Matching Requests for example.com
+      - traefik.http.routers.target.entrypoints=anubis # Listening on the exclusive Anubis Network
+      - traefik.http.services.target.loadbalancer.server.port=80 # Telling Traefik where to receive requests
+      - traefik.http.routers.target.service=target # Telling Traefik to use the above specified port
+
+  # Not Protected by Anubis
+  target2:
+    image: traefik/whoami:latest
+    networks:
+      - traefik
+    labels:
+      - traefik.enable=true # Enabling Traefik
+      - traefik.docker.network=traefik # Telling Traefik which network to use
+      - traefik.http.routers.target2.rule=Host(`another.com`) # Only Matching Requests for example.com
+      - traefik.http.routers.target2.entrypoints=websecure # Listening on the exclusive Anubis Network
+      - traefik.http.services.target2.loadbalancer.server.port=80 # Telling Traefik where to receive requests
+      - traefik.http.routers.target2.service=target2 # Telling Traefik to use the above specified port
+      - traefik.http.routers.target2.tls.certresolver=le # Telling Traefik to resolve a Cert for this Target
+
+networks:
+  traefik:
+    name: traefik
+```
+
+**traefik.yml**
+
+```yml
+api:
+  insecure: false # shouldn't be enabled in prod
+
+entryPoints:
+  # Web
+  web:
+    address: ":80"
+  websecure:
+    address: ":443"
+  # Anubis
+  anubis:
+    address: ":3923"
+
+certificatesResolvers:
+  le:
+    acme:
+      tlsChallenge: {}
+      email: "admin@example.com"
+      storage: "/letsencrypt/acme.json"
+
+providers:
+  docker: {}
+```

docs/docs/admin/installation.mdx

@@ -4,6 +4,9 @@ title: Setting up Anubis
 
 import RandomKey from "@site/src/components/RandomKey";
 
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
 Anubis is meant to sit between your reverse proxy (such as Nginx or Caddy) and your target service. One instance of Anubis must be used per service you are protecting.
 
 <center>
@@ -38,32 +41,76 @@ The Docker image runs Anubis as user ID 1000 and group ID 1000. If you are mount
 
 Anubis has very minimal system requirements. I suspect that 128Mi of ram may be sufficient for a large number of concurrent clients. Anubis may be a poor fit for apps that use WebSockets and maintain open connections, but I don't have enough real-world experience to know one way or another.
 
+## Native packages
+
+For more detailed information on installing Anubis with native packages, please read [the native install directions](./native-install.mdx).
+
 ## Environment variables
 
 Anubis uses these environment variables for configuration:
 
-| Environment Variable           | Default value           | Explanation                                                                                                                                                                                                                                                                              |
-| :----------------------------- | :---------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `BIND`                         | `:8923`                 | The network address that Anubis listens on. For `unix`, set this to a path: `/run/anubis/instance.sock`                                                                                                                                                                                  |
-| `BIND_NETWORK`                 | `tcp`                   | The address family that Anubis listens on. Accepts `tcp`, `unix` and anything Go's [`net.Listen`](https://pkg.go.dev/net#Listen) supports.                                                                                                                                               |
-| `COOKIE_DOMAIN`                | unset                   | The domain the Anubis challenge pass cookie should be set to. This should be set to the domain you bought from your registrar (EG: `techaro.lol` if your webapp is running on `anubis.techaro.lol`). See [here](https://stackoverflow.com/a/1063760) for more information.               |
-| `COOKIE_PARTITIONED`           | `false`                 | If set to `true`, enables the [partitioned (CHIPS) flag](https://developers.google.com/privacy-sandbox/cookies/chips), meaning that Anubis inside an iframe has a different set of cookies than the domain hosting the iframe.                                                           |
-| `DIFFICULTY`                   | `5`                     | The difficulty of the challenge, or the number of leading zeroes that must be in successful responses.                                                                                                                                                                                   |
-| `ED25519_PRIVATE_KEY_HEX`      | unset                   | The hex-encoded ed25519 private key used to sign Anubis responses. If this is not set, Anubis will generate one for you. This should be exactly 64 characters long. See below for details.                                                                                               |
-| `ED25519_PRIVATE_KEY_HEX_FILE` | unset                   | Path to a file containing the hex-encoded ed25519 private key. Only one of this or its sister option may be set.                                                                                                                                                                         |
-| `METRICS_BIND`                 | `:9090`                 | The network address that Anubis serves Prometheus metrics on. See `BIND` for more information.                                                                                                                                                                                           |
-| `METRICS_BIND_NETWORK`         | `tcp`                   | The address family that the Anubis metrics server listens on. See `BIND_NETWORK` for more information.                                                                                                                                                                                   |
-| `OG_EXPIRY_TIME`               | `24h`                   | The expiration time for the Open Graph tag cache.                                                                                                                                                                                                                                        |
-| `OG_PASSTHROUGH`               | `false`                 | If set to `true`, Anubis will enable Open Graph tag passthrough.                                                                                                                                                                                                                         |
-| `POLICY_FNAME`                 | unset                   | The file containing [bot policy configuration](./policies.md). See the bot policy documentation for more details. If unset, the default bot policy configuration is used.                                                                                                                |
-| `SERVE_ROBOTS_TXT`             | `false`                 | If set `true`, Anubis will serve a default `robots.txt` file that disallows all known AI scrapers by name and then additionally disallows every scraper. This is useful if facts and circumstances make it difficult to change the underlying service to serve such a `robots.txt` file. |
-| `SOCKET_MODE`                  | `0770`                  | _Only used when at least one of the `*_BIND_NETWORK` variables are set to `unix`._ The socket mode (permissions) for Unix domain sockets.                                                                                                                                                |
-| `TARGET`                       | `http://localhost:3923` | The URL of the service that Anubis should forward valid requests to. Supports Unix domain sockets, set this to a URI like so: `unix:///path/to/socket.sock`.                                                                                                                             |
-| `USE_REMOTE_ADDRESS`           | unset                   | If set to `true`, Anubis will take the client's IP from the network socket. For production deployments, it is expected that a reverse proxy is used in front of Anubis, which pass the IP using headers, instead.                                                                        |
-| `WEBMASTER_EMAIL`              | unset                   | If set, shows a contact email address when rendering error pages. This email address will be how users can get in contact with administrators.                                                                                                                                           |
+| Environment Variable           | Default value           | Explanation                                                                                                                                                                                                                                                                                                          |
+| :----------------------------- | :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `BASE_PREFIX`                  | unset                   | If set, adds a global prefix to all Anubis endpoints. For example, setting this to `/myapp` would make Anubis accessible at `/myapp/` instead of `/`. This is useful when running Anubis behind a reverse proxy that routes based on path prefixes.                                                                  |
+| `BIND`                         | `:8923`                 | The network address that Anubis listens on. For `unix`, set this to a path: `/run/anubis/instance.sock`                                                                                                                                                                                                              |
+| `BIND_NETWORK`                 | `tcp`                   | The address family that Anubis listens on. Accepts `tcp`, `unix` and anything Go's [`net.Listen`](https://pkg.go.dev/net#Listen) supports.                                                                                                                                                                           |
+| `COOKIE_DOMAIN`                | unset                   | The domain the Anubis challenge pass cookie should be set to. This should be set to the domain you bought from your registrar (EG: `techaro.lol` if your webapp is running on `anubis.techaro.lol`). See this [stackoverflow explanation of cookies](https://stackoverflow.com/a/1063760) for more information.      |
+| `COOKIE_EXPIRATION_TIME`       | `168h`                  | The amount of time the authorization cookie is valid for.                                                                                                                                                                                                                                                            |
+| `COOKIE_PARTITIONED`           | `false`                 | If set to `true`, enables the [partitioned (CHIPS) flag](https://developers.google.com/privacy-sandbox/cookies/chips), meaning that Anubis inside an iframe has a different set of cookies than the domain hosting the iframe.                                                                                       |
+| `DIFFICULTY`                   | `4`                     | The difficulty of the challenge, or the number of leading zeroes that must be in successful responses.                                                                                                                                                                                                               |
+| `ED25519_PRIVATE_KEY_HEX`      | unset                   | The hex-encoded ed25519 private key used to sign Anubis responses. If this is not set, Anubis will generate one for you. This should be exactly 64 characters long. See below for details.                                                                                                                           |
+| `ED25519_PRIVATE_KEY_HEX_FILE` | unset                   | Path to a file containing the hex-encoded ed25519 private key. Only one of this or its sister option may be set.                                                                                                                                                                                                     |
+| `METRICS_BIND`                 | `:9090`                 | The network address that Anubis serves Prometheus metrics on. See `BIND` for more information.                                                                                                                                                                                                                       |
+| `METRICS_BIND_NETWORK`         | `tcp`                   | The address family that the Anubis metrics server listens on. See `BIND_NETWORK` for more information.                                                                                                                                                                                                               |
+| `OG_EXPIRY_TIME`               | `24h`                   | The expiration time for the Open Graph tag cache.                                                                                                                                                                                                                                                                    |
+| `OG_PASSTHROUGH`               | `false`                 | If set to `true`, Anubis will enable Open Graph tag passthrough.                                                                                                                                                                                                                                                     |
+| `OG_CACHE_CONSIDER_HOST`       | `false`                 | If set to `true`, Anubis will consider the host in the Open Graph tag cache key.                                                                                                                                                                                                                                     |
+| `POLICY_FNAME`                 | unset                   | The file containing [bot policy configuration](./policies.mdx). See the bot policy documentation for more details. If unset, the default bot policy configuration is used.                                                                                                                                           |
+| `REDIRECT_DOMAINS`             | unset                   | If set, restrict the domains that Anubis can redirect to when passing a challenge.<br/><br/>If this is unset, Anubis may redirect to any domain which could cause security issues in the unlikely case that an attacker passes a challenge for your browser and then tricks you into clicking a link to your domain. |
+| `SERVE_ROBOTS_TXT`             | `false`                 | If set `true`, Anubis will serve a default `robots.txt` file that disallows all known AI scrapers by name and then additionally disallows every scraper. This is useful if facts and circumstances make it difficult to change the underlying service to serve such a `robots.txt` file.                             |
+| `SOCKET_MODE`                  | `0770`                  | _Only used when at least one of the `*_BIND_NETWORK` variables are set to `unix`._ The socket mode (permissions) for Unix domain sockets.                                                                                                                                                                            |
+| `TARGET`                       | `http://localhost:3923` | The URL of the service that Anubis should forward valid requests to. Supports Unix domain sockets, set this to a URI like so: `unix:///path/to/socket.sock`.                                                                                                                                                         |
+| `USE_REMOTE_ADDRESS`           | unset                   | If set to `true`, Anubis will take the client's IP from the network socket. For production deployments, it is expected that a reverse proxy is used in front of Anubis, which pass the IP using headers, instead.                                                                                                    |
+| `WEBMASTER_EMAIL`              | unset                   | If set, shows a contact email address when rendering error pages. This email address will be how users can get in contact with administrators.                                                                                                                                                                       |
 
 For more detailed information on configuring Open Graph tags, please refer to the [Open Graph Configuration](./configuration/open-graph.mdx) page.
 
+### Using Base Prefix
+
+The `BASE_PREFIX` environment variable allows you to run Anubis behind a path prefix. This is useful when:
+
+- You want to host multiple services on the same domain
+- You're using a reverse proxy that routes based on path prefixes
+- You need to integrate Anubis with an existing application structure
+
+For example, if you set `BASE_PREFIX=/myapp`, Anubis will:
+
+- Serve its challenge page at `/myapp/` instead of `/`
+- Serve its API endpoints at `/myapp/.within.website/x/cmd/anubis/api/` instead of `/.within.website/x/cmd/anubis/api/`
+- Serve its static assets at `/myapp/.within.website/x/cmd/anubis/` instead of `/.within.website/x/cmd/anubis/`
+
+When using this feature with a reverse proxy:
+
+1. Configure your reverse proxy to route requests for the specified path prefix to Anubis
+2. Set the `BASE_PREFIX` environment variable to match the path prefix in your reverse proxy configuration
+3. Ensure that your reverse proxy preserves the path when forwarding requests to Anubis
+
+Example with Nginx:
+
+```nginx
+location /myapp/ {
+    proxy_pass http://anubis:8923/myapp;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+}
+```
+
+With corresponding Anubis configuration:
+
+```
+BASE_PREFIX=/myapp
+```
+
 ### Key generation
 
 To generate an ed25519 private key, you can use this command:
@@ -76,113 +123,18 @@ Alternatively here is a key generated by your browser:
 
 <RandomKey />
 
-## Docker compose
+## Next steps
 
-Add Anubis to your compose file pointed at your service:
+To get Anubis filtering your traffic, you need to make sure it's added to your HTTP load balancer or platform configuration. See the [environments category](/docs/category/environments) for detailed information on individual environments.
 
-```yaml
-services:
-  anubis-nginx:
-    image: ghcr.io/techarohq/anubis:latest
-    environment:
-      BIND: ":8080"
-      DIFFICULTY: "5"
-      METRICS_BIND: ":9090"
-      SERVE_ROBOTS_TXT: "true"
-      TARGET: "http://nginx"
-      POLICY_FNAME: "/data/cfg/botPolicy.json"
-      OG_PASSTHROUGH: "true"
-      OG_EXPIRY_TIME: "24h"
-    ports:
-      - 8080:8080
-    volumes:
-      - "./botPolicy.json:/data/cfg/botPolicy.json:ro"
-  nginx:
-    image: nginx
-    volumes:
-      - "./www:/usr/share/nginx/html"
-```
+- [Apache](./environments/apache.mdx)
+- [Docker compose](./environments/docker-compose.mdx)
+- [Kubernetes](./environments/kubernetes.mdx)
+- [Nginx](./environments/nginx.mdx)
+- [Traefik](./environments/traefik.mdx)
 
-## Kubernetes
+:::note
 
-This example makes the following assumptions:
+Anubis loads its assets from `/.within.website/x/xess/` and `/.within.website/x/cmd/anubis`. If you do not reverse proxy these in your server config, Anubis won't work.
 
-- Your target service is listening on TCP port `5000`.
-- Anubis will be listening on port `8080`.
-
-Attach Anubis to your Deployment:
-
-```yaml
-containers:
-  # ...
-  - name: anubis
-    image: ghcr.io/techarohq/anubis:latest
-    imagePullPolicy: Always
-    env:
-      - name: "BIND"
-        value: ":8080"
-      - name: "DIFFICULTY"
-        value: "5"
-      - name: "METRICS_BIND"
-        value: ":9090"
-      - name: "SERVE_ROBOTS_TXT"
-        value: "true"
-      - name: "TARGET"
-        value: "http://localhost:5000"
-      - name: "OG_PASSTHROUGH"
-        value: "true"
-      - name: "OG_EXPIRY_TIME"
-        value: "24h"
-    resources:
-      limits:
-        cpu: 500m
-        memory: 128Mi
-      requests:
-        cpu: 250m
-        memory: 128Mi
-    securityContext:
-      runAsUser: 1000
-      runAsGroup: 1000
-      runAsNonRoot: true
-      allowPrivilegeEscalation: false
-      capabilities:
-        drop:
-          - ALL
-      seccompProfile:
-        type: RuntimeDefault
-```
-
-Then add a Service entry for Anubis:
-
-```yaml
-# ...
-spec:
-  ports:
-    # diff-add
-    - protocol: TCP
-      # diff-add
-      port: 8080
-      # diff-add
-      targetPort: 8080
-      # diff-add
-      name: anubis
-```
-
-Then point your Ingress to the Anubis port:
-
-```yaml
-  rules:
-  - host: git.xeserv.us
-    http:
-      paths:
-      - pathType: Prefix
-        path: "/"
-        backend:
-          service:
-            name: git
-            port:
-              # diff-remove
-              name: http
-              # diff-add
-              name: anubis
-```
+:::

docs/docs/admin/native-install.mdx

@@ -5,6 +5,8 @@ title: Installing Anubis with a native package
 import Tabs from "@theme/Tabs";
 import TabItem from "@theme/TabItem";
 
+Download the package for your system from [the most recent release on GitHub](https://github.com/TecharoHQ/anubis/releases).
+
 Install the Anubis package using your package manager of choice:
 
 <Tabs>
@@ -84,20 +86,20 @@ Once it's installed, make a copy of the default configuration file `/etc/anubis/
 sudo cp /etc/anubis/default.env /etc/anubis/gitea.env
 ```
 
-Copy the default bot policies file to `/etc/anubis/gitea.botPolicies.json`:
+Copy the default bot policies file to `/etc/anubis/gitea.botPolicies.yaml`:
 
 <Tabs>
 <TabItem value="debrpm" label="Debian or Red Hat" default>
 
 ```text
-sudo cp /usr/share/doc/anubis/botPolicies.json /etc/anubis/gitea.botPolicies.json
+sudo cp /usr/share/doc/anubis/botPolicies.yaml /etc/anubis/gitea.botPolicies.yaml
 ```
 
 </TabItem>
 <TabItem value="tarball" label="Tarball">
 
 ```text
-sudo cp ./doc/botPolicies.json /etc/anubis/gitea.botPolicies.json
+sudo cp ./doc/botPolicies.yaml /etc/anubis/gitea.botPolicies.yaml
 ```
 
 </TabItem>
@@ -112,7 +114,7 @@ BIND_NETWORK=tcp
 DIFFICULTY=4
 METRICS_BIND=[::1]:8240
 METRICS_BIND_NETWORK=tcp
-POLICY_FNAME=/etc/anubis/gitea.botPolicies.json
+POLICY_FNAME=/etc/anubis/gitea.botPolicies.yaml
 TARGET=http://localhost:3000
 ```
 
@@ -129,3 +131,8 @@ curl http://localhost:8240/metrics
 ```
 
 Then set up your reverse proxy (Nginx, Caddy, etc.) to point to the Anubis port. Anubis will then reverse proxy all requests that meet the policies in `/etc/anubis/gitea.botPolicies.json` to the target service.
+
+For more details on particular reverse proxies, see here:
+
+- [Apache](./environments/apache.mdx)
+- [Nginx](./environments/nginx.mdx)

docs/docs/admin/policies.mdx

@@ -2,15 +2,25 @@
 title: Policy Definitions
 ---
 
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+
 Out of the box, Anubis is pretty heavy-handed. It will aggressively challenge everything that might be a browser (usually indicated by having `Mozilla` in its user agent). However, some bots are smart enough to get past the challenge. Some things that look like bots may actually be fine (IE: RSS readers). Some resources need to be visible no matter what. Some resources and remotes are fine to begin with.
 
 Bot policies let you customize the rules that Anubis uses to allow, deny, or challenge incoming requests. Currently you can set policies by the following matches:
 
 - Request path
 - User agent string
+- HTTP request header values
+- [Importing other configuration snippets](./configuration/import.mdx)
+
+As of version v1.17.0 or later, configuration can be written in either JSON or YAML.
 
 Here's an example rule that denies [Amazonbot](https://developer.amazon.com/en/amazonbot):
 
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
 ```json
 {
   "name": "amazonbot",
@@ -19,15 +29,37 @@ Here's an example rule that denies [Amazonbot](https://developer.amazon.com/en/a
 }
 ```
 
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+- name: amazonbot
+  user_agent_regex: Amazonbot
+  action: DENY
+```
+
+</TabItem>
+</Tabs>
+
 When this rule is evaluated, Anubis will check the `User-Agent` string of the request. If it contains `Amazonbot`, Anubis will send an error page to the user saying that access is denied, but in such a way that makes scrapers think they have correctly loaded the webpage.
 
 Right now the only kinds of policies you can write are bot policies. Other forms of policies will be added in the future.
 
 Here is a minimal policy file that will protect against most scraper bots:
 
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
 ```json
 {
   "bots": [
+    {
+      "name": "cloudflare-workers",
+      "headers_regex": {
+        "CF-Worker": ".*"
+      },
+      "action": "DENY"
+    },
     {
       "name": "well-known",
       "path_regex": "^/.well-known/.*$",
@@ -52,9 +84,35 @@ Here is a minimal policy file that will protect against most scraper bots:
 }
 ```
 
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+bots:
+  - name: cloudflare-workers
+    headers_regex:
+      CF-Worker: .*
+    action: DENY
+  - name: well-known
+    path_regex: ^/.well-known/.*$
+    action: ALLOW
+  - name: favicon
+    path_regex: ^/favicon.ico$
+    action: ALLOW
+  - name: robots-txt
+    path_regex: ^/robots.txt$
+    action: ALLOW
+  - name: generic-browser
+    user_agent_regex: Mozilla
+    action: CHALLENGE
+```
+
+</TabItem>
+</Tabs>
+
 This allows requests to [`/.well-known`](https://en.wikipedia.org/wiki/Well-known_URI), `/favicon.ico`, `/robots.txt`, and challenges any request that has the word `Mozilla` in its User-Agent string. The [default policy file](https://github.com/TecharoHQ/anubis/blob/main/data/botPolicies.json) is a bit more cohesive, but this should be more than enough for most users.
 
-If no rules match the request, it is allowed through.
+If no rules match the request, it is allowed through. For more details on this default behavior and its implications, see [Default allow behavior](./default-allow-behavior.mdx).
 
 ## Writing your own rules
 
@@ -72,6 +130,11 @@ Name your rules in lower case using kebab-case. Rule names will be exposed in Pr
 
 Rules can also have their own challenge settings. These are customized using the `"challenge"` key. For example, here is a rule that makes challenges artificially hard for connections with the substring "bot" in their user agent:
 
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
+This rule has been known to have a high false positive rate in testing. Please use this with care.
+
 ```json
 {
   "name": "generic-bot-catchall",
@@ -85,6 +148,25 @@ Rules can also have their own challenge settings. These are customized using the
 }
 ```
 
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+This rule has been known to have a high false positive rate in testing. Please use this with care.
+
+```yaml
+# Punish any bot with "bot" in the user-agent string
+- name: generic-bot-catchall
+  user_agent_regex: (?i:bot|crawler)
+  action: CHALLENGE
+  challenge:
+    difficulty: 16 # impossible
+    report_as: 4 # lie to the operator
+    algorithm: slow # intentionally waste CPU cycles and time
+```
+
+</TabItem>
+</Tabs>
+
 Challenges can be configured with these settings:
 
 | Key          | Example  | Description                                                                                                                                                                                    |
@@ -99,6 +181,9 @@ The `remote_addresses` field of a Bot rule allows you to set the IP range that t
 
 For example, you can allow a search engine to connect if and only if its IP address matches the ones they published:
 
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
 ```json
 {
   "name": "qwantbot",
@@ -108,8 +193,25 @@ For example, you can allow a search engine to connect if and only if its IP addr
 }
 ```
 
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+- name: qwantbot
+  user_agent_regex: \+https\://help\.qwant\.com/bot/
+  action: ALLOW
+  # https://help.qwant.com/wp-content/uploads/sites/2/2025/01/qwantbot.json
+  remote_addresses: ["91.242.162.0/24"]
+```
+
+</TabItem>
+</Tabs>
+
 This also works at an IP range level without any other checks:
 
+<Tabs>
+<TabItem value="json" label="JSON" default>
+
 ```json
 {
   "name": "internal-network",
@@ -118,6 +220,19 @@ This also works at an IP range level without any other checks:
 }
 ```
 
+</TabItem>
+<TabItem value="yaml" label="YAML">
+
+```yaml
+name: internal-network
+action: ALLOW
+remote_addresses:
+  - 100.64.0.0/10
+```
+
+</TabItem>
+</Tabs>
+
 ## Risk calculation for downstream services
 
 In case your service needs it for risk calculation reasons, Anubis exposes information about the rules that any requests match using a few headers:
@@ -126,6 +241,6 @@ In case your service needs it for risk calculation reasons, Anubis exposes infor
 | :---------------- | :--------------------------------------------------- | :--------------- |
 | `X-Anubis-Rule`   | The name of the rule that was matched                | `bot/lightpanda` |
 | `X-Anubis-Action` | The action that Anubis took in response to that rule | `CHALLENGE`      |
-| `X-Anubis-Status` | The status and how strict Anubis was in its checks   | `PASS-FULL`      |
+| `X-Anubis-Status` | The status and how strict Anubis was in its checks   | `PASS`           |
 
 Policy rules are matched using [Go's standard library regular expressions package](https://pkg.go.dev/regexp). You can mess around with the syntax at [regex101.com](https://regex101.com), make sure to select the Golang option.

docs/docs/design/how-anubis-works.mdx

@@ -37,7 +37,7 @@ flowchart TD
     ValidateChallenge -- If anything is wrong --> Fail
 ```
 
-### Challenge presentation
+## Challenge presentation
 
 Anubis decides to present a challenge using this logic:
 
@@ -89,7 +89,7 @@ work valid?"}
     PresentChallenge -- Back again for another cycle --> Request
 ```
 
-### Proof of passing challenges
+## Proof of passing challenges
 
 When a client passes a challenge, Anubis sets an HTTP cookie named `"within.website-x-cmd-anubis-auth"` containing a signed [JWT](https://jwt.io/) (JSON Web Token). This JWT contains the following claims:
 
@@ -102,7 +102,7 @@ When a client passes a challenge, Anubis sets an HTTP cookie named `"within.webs
 
 This ensures that the token has enough metadata to prove that the token is valid (due to the token's signature), but also so that the server can independently prove the token is valid. This cookie is allowed to be set without triggering an EU cookie banner notification; but depending on facts and circumstances, you may wish to disclose this to your users.
 
-### Challenge format
+## Challenge format
 
 Challenges are formed by taking some user request metadata and using that to generate a SHA-256 checksum. The following request headers are used:
 
@@ -115,6 +115,6 @@ Challenges are formed by taking some user request metadata and using that to gen
 
 This forms a fingerprint of the requestor using metadata that any requestor already is sending. It also uses time as an input, which is known to both the server and requestor due to the nature of linear timelines. Depending on facts and circumstances, you may wish to disclose this to your users.
 
-### JWT signing
+## JWT signing
 
 Anubis uses an ed25519 keypair to sign the JWTs issued when challenges are passed. Anubis will generate a new ed25519 keypair every time it starts. At this time, there is no way to share this keypair between instance of Anubis, but that will be addressed in future versions.

docs/docs/developer/building-anubis.md

@@ -12,6 +12,12 @@ These instructions may work, but for right now they are informative for downstre
 
 If you are doing a build entirely from source, here's what you need to do:
 
+:::info
+
+If you maintain a package for Anubis v1.15.x or older, you will need to update your package build. You may want to use one of the half-baked tarballs if your distro/environment of choice makes it difficult to use npm.
+
+:::
+
 ### Tools needed
 
 In order to build a production-ready binary of Anubis, you need the following packages in your environment:

docs/docs/funding.md

@@ -7,4 +7,4 @@ Anubis is provided to the public for free in order to help advance the common go
 
 If you want to run an unbranded or white-label version of Anubis, please [contact Xe](https://xeiaso.net/contact) to arrange a contract. This is not meant to be "contact us" pricing, I am still evaluating the market for this solution and figuring out what makes sense.
 
-You can donate to the project [on Patreon](https://patreon.com/cadey).
+You can donate to the project [on Patreon](https://patreon.com/cadey) or via [GitHub Sponsors](https://github.com/sponsors/Xe).

docs/docs/index.mdx

@@ -15,14 +15,57 @@ title: Anubis
 ![language count](https://img.shields.io/github/languages/count/TecharoHQ/anubis)
 ![repo size](https://img.shields.io/github/repo-size/TecharoHQ/anubis)
 
-Anubis [weighs the soul of your connection](https://en.wikipedia.org/wiki/Weighing_of_souls) using a sha256 proof-of-work challenge in order to protect upstream resources from scraper bots.
+## Sponsors
+
+Anubis is brought to you by sponsors and donors like:
+
+[![Distrust](/img/sponsors/distrust-logo.webp)](https://distrust.co)
+[![Terminal Trove](/img/sponsors/terminal-trove.webp)](https://terminaltrove.com/?utm_campaign=github&utm_medium=referral&utm_content=anubis&utm_source=abgh)
+[![canine.tools](/img/sponsors/caninetools-logo.webp)](https://canine.tools)
+
+## Overview
+
+Anubis [weighs the soul of your connection](https://en.wikipedia.org/wiki/Weighing_of_souls) using a proof-of-work challenge in order to protect upstream resources from scraper bots.
 
 This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.
 
-Anubis is a bit of a nuclear response. This will result in your website being blocked from smaller scrapers and may inhibit "good bots" like the Internet Archive. You can configure [bot policy definitions](./admin/policies.md) to explicitly allowlist them and we are working on a curated set of "known good" bots to allow for a compromise between discoverability and uptime.
+Anubis is a bit of a nuclear response. This will result in your website being blocked from smaller scrapers and may inhibit "good bots" like the Internet Archive. You can configure [bot policy definitions](https://anubis.techaro.lol/docs/admin/policies) to explicitly allowlist them and we are working on a curated set of "known good" bots to allow for a compromise between discoverability and uptime.
+
+In most cases, you should not need this and can probably get by using Cloudflare to protect a given origin. However, for circumstances where you can't or won't use Cloudflare, Anubis is there for you.
 
 ## Support
 
 If you run into any issues running Anubis, please [open an issue](https://github.com/TecharoHQ/anubis/issues/new?template=Blank+issue) and include all the information I would need to diagnose your issue.
 
-For live chat, please join the [Patreon](https://patreon.com/cadey) and ask in the Patron discord in the channel `#anubis`.
+For live chat, please join the [Patreon](https://patreon.com/cadey) or join [GitHub Sponsors](https://github.com/sponsors/Xe) and ask in the Patron discord in the channel `#anubis`.
+
+## Star History
+
+<a href="https://www.star-history.com/#TecharoHQ/anubis&Date">
+  <picture>
+    <source
+      media="(prefers-color-scheme: dark)"
+      srcSet="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date&theme=dark"
+    />
+    <source
+      media="(prefers-color-scheme: light)"
+      srcSet="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date"
+    />
+    <img
+      alt="Star History Chart"
+      src="https://api.star-history.com/svg?repos=TecharoHQ/anubis&type=Date"
+    />
+  </picture>
+</a>
+
+## Packaging Status
+
+[![Packaging status](https://repology.org/badge/vertical-allrepos/anubis-anti-crawler.svg?columns=3)](https://repology.org/project/anubis-anti-crawler/versions)
+
+## Contributors
+
+<a href="https://github.com/TecharoHQ/anubis/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=TecharoHQ/anubis" />
+</a>
+
+Made with [contrib.rocks](https://contrib.rocks).

docs/docs/user/frequently-asked-questions.mdx

@@ -0,0 +1,20 @@
+# Frequently Asked Questions
+
+## Why can't you just put details about the proof of work challenge into the challenge page so I don't need to run JavaScript?
+
+A common question is something along the lines of "why can't you give me a shell script to run the challenge on my laptop so that I don't have to enable JavaScript". Malware has been known to show an interstitial that [asks the user to paste something into their run box on Windows](https://www.malwarebytes.com/blog/news/2025/03/fake-captcha-websites-hijack-your-clipboard-to-install-information-stealers), which will then make that machine a zombie in a botnet.
+
+It would be in very bad taste to associate a security product such as Anubis with behavior similar to what malware uses. This would destroy user trust in the product and potentially result in reputational damage for the contributors. When at all possible, we want to avoid this happening.
+
+Technically inclined users are easily able to understand how the proof of work check works by either reading the JavaScript on the page or [reading the source code of the JavaScript program](https://github.com/TecharoHQ/anubis/tree/main/web/js). Please note that the format of the challenges and the algorithms used to solve them are liable to change without notice and are not considered part of the public API of Anubis. When such a change occurs, this will break your workarounds.
+
+If [sufficient funding is raised](https://github.com/TecharoHQ/anubis/discussions/278), a browser extension that packages the proof of work checks and looks for Anubis challenge pages to solve them will be created.
+
+## Why does Anubis use [Web Workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) to do its proof of work challenge?
+
+Anubis uses [Web Workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) to do its proof of work challenge for two main reasons:
+
+1. The proof of work operation is a lot of serially blocking calls. If you do serially blocking calls in JavaScript, some browsers will hang and not respond to user input. This is bad user experience. Using a Web Worker allows the browser to do this computation in the background so your browser will not hang.
+2. Web Workers allow you to do multithreaded execution of JavaScript code. This lets Anubis run its checks in parallel across all your system cores so that the challenge can complete as fast as possible. In the last decade, most CPU advancements have come from making cores and code extremely parallel. Using Web Workers lets Anubis take advantage of your hardware as much as possible so that the challenge finishes as fast as possible.
+
+If you use a browser extension such as [JShelter](https://jshelter.org/), you will need to [modify your JShelter configuration](./known-broken-extensions.md#jshelter) to allow Anubis' proof of work computation to complete.

docs/docs/user/known-broken-extensions.md

@@ -3,17 +3,47 @@ title: List of known browser extensions that can break Anubis
 ---
 
 This page contains a list of all of the browser extensions that are known to break Anubis' functionality and their associated GitHub issues, along with instructions on how to work around the issue.
+
 ## [JShelter](https://jshelter.org/)
 
-| Extension    | JShelter                                      |
-| :----------- | :-------------------------------------------- |
-| Website      | [jshelter.org](https://jshelter.org/)         |
-| GitHub issue | https://github.com/TecharoHQ/anubis/issues/25 |
+| Extension    | JShelter                                                                                                                                           |
+| :----------- | :------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Website      | [jshelter.org](https://jshelter.org/)                                                                                                              |
+| GitHub issue | https://github.com/TecharoHQ/anubis/issues/25                                                                                                      |
+| Be aware of  | [What are Web Workers, and what are the threats that I face?](https://jshelter.org/faq/#what-are-web-workers-and-what-are-the-threats-that-i-face) |
 
-Workaround steps:
+### Workaround steps (recommended):
+
+1. Click on the JShelter badge icon (typically in the toolbar next to your navigation bar; if you cannot locate the icon, see [this question](https://jshelter.org/faq/#can-i-see-a-jshelter-badge-icon-next-to-my-navigation-bar-i-want-to-interact-with-the-extension-easily-and-avoid-going-through-settings)).
+2. Expand JavaScript Shield settings by clicking on the `Modify` button.
+3. Click on the `Detail tweaks of JS shield for this site` button.
+4. Click and drag the `WebWorker` slider to the left until `Remove` is replaced by the `Unprotected`.
+5. Refresh the page, for example, by clicking on the `Refresh page` button at the top of the JShelter pop up window.
+6. You might want to restore the Worker settings once you go through the challenge.
+
+### Workaround steps (alternative if you do not want to dig in JShelter's pop up):
+
+1. Click on the JShelter badge icon (typically in the toolbar next to your navigation bar; if you cannot locate the icon, see [this question](https://jshelter.org/faq/#can-i-see-a-jshelter-badge-icon-next-to-my-navigation-bar-i-want-to-interact-with-the-extension-easily-and-avoid-going-through-settings)).
+2. Expand JavaScript Shield settings by clicking on the `Modify` button.
+3. Choose "Turn JavaScript Shield off"
+4. Refresh the page, for example, by clicking on the `Refresh page` button at the top of the JShelter pop up window.
+
+:::note
+
+Taking these actions will remove all protections of JavaScript Shield for all pages at the visited web site. You might want review and amend your JavaScript shield settings once you go through the challenge based on your operational security model.
+
+:::
+
+### Workaround steps (alternative if you do not like JShelter's pop up):
 
 1. Open JShelter extension settings
 2. Click on JS Shield details
 3. Enter in the domain for a website protected by Anubis
 4. Choose "Turn JavaScript Shield off"
 5. Hit "Add to list"
+
+:::note
+
+Taking these actions will remove all protections of JavaScript Shield for all pages at the visited web site. You might want review and amend your JavaScript shield settings once you go through the challenge based on your operational security model.
+
+:::

docs/docs/user/known-instances.md

@@ -0,0 +1,56 @@
+---
+title: List of known websites using Anubis
+---
+
+This page contains a non-exhaustive list with all websites using Anubis.
+
+- <details>
+  <summary>The Linux Foundation</summary>
+  - https://git.kernel.org/
+  - https://lore.kernel.org/
+  </details>
+- https://gitlab.gnome.org/
+- https://scioly.org/
+- https://bugs.winehq.org/
+- https://svnweb.freebsd.org/
+- https://trac.ffmpeg.org/
+- https://xeiaso.net/
+- https://source.puri.sm/
+- https://git.enlightenment.org/
+- https://superlove.sayitditto.net/
+- https://linktaco.com/
+- https://jaredallard.dev/
+- https://dev.sanctum.geek.nz/
+- https://canine.tools/
+- https://git.lupancham.net/
+- https://dev.haiku-os.org
+- http://code.hackerspace.pl/
+- https://wiki.archlinux.org/
+- https://git.devuan.org/
+- https://hydra.nixos.org/
+- https://codeberg.org/
+- https://www.cfaarchive.org/
+- https://gitlab.freedesktop.org/
+- <details>
+  <summary>FreeCAD</summary>
+  - https://forum.freecad.org/
+  - https://wiki.freecad.org/
+  </details>
+- <details>
+  <summary>ScummVM</summary>
+  - https://forums.scummvm.org/
+  - https://wiki.scummvm.org/
+  </details>
+- <details>
+  <summary>Sourceware</summary>
+  - https://sourceware.org/cgit
+  - https://sourceware.org/glibc/wiki
+  - https://builder.sourceware.org/testruns/
+  - https://patchwork.sourceware.org/
+  - https://gcc.gnu.org/bugzilla/
+  - https://gcc.gnu.org/cgit
+  </details>
+- <details>
+  <summary>The United Nations</summary>
+  - https://policytoolbox.iiep.unesco.org/
+  </details>

docs/docusaurus.config.ts

@@ -70,6 +70,9 @@ const config: Config = {
   ],
 
   themeConfig: {
+    colorMode: {
+      respectPrefersColorScheme: true,
+    },
     // Replace with your project's social card
     image: 'img/docusaurus-social-card.jpg',
     navbar: {
@@ -125,10 +128,6 @@ const config: Config = {
         {
           title: 'More',
           items: [
-            {
-              label: 'Blog',
-              to: '/blog',
-            },
             {
               label: 'GitHub',
               href: 'https://github.com/TecharoHQ/anubis',

docs/package-lock.json

@@ -8512,9 +8512,9 @@
       }
     },
     "node_modules/estree-util-value-to-estree": {
-      "version": "3.3.2",
-      "resolved": "https://registry.npmjs.org/estree-util-value-to-estree/-/estree-util-value-to-estree-3.3.2.tgz",
-      "integrity": "sha512-hYH1aSvQI63Cvq3T3loaem6LW4u72F187zW4FHpTrReJSm6W66vYTFNO1vH/chmcOulp1HlAj1pxn8Ag0oXI5Q==",
+      "version": "3.3.3",
+      "resolved": "https://registry.npmjs.org/estree-util-value-to-estree/-/estree-util-value-to-estree-3.3.3.tgz",
+      "integrity": "sha512-Db+m1WSD4+mUO7UgMeKkAwdbfNWwIxLt48XF2oFU9emPfXkIu+k5/nlOj313v7wqtAPo0f9REhUvznFrPkG8CQ==",
       "license": "MIT",
       "dependencies": {
         "@types/estree": "^1.0.0"
@@ -10093,9 +10093,9 @@
       }
     },
     "node_modules/http-proxy-middleware": {
-      "version": "2.0.7",
-      "resolved": "https://registry.npmjs.org/http-proxy-middleware/-/http-proxy-middleware-2.0.7.tgz",
-      "integrity": "sha512-fgVY8AV7qU7z/MmXJ/rxwbrtQH4jBQ9m7kp3llF0liB7glmFeVZFBepQb32T3y8n8k2+AEYuMPCpinYW+/CuRA==",
+      "version": "2.0.9",
+      "resolved": "https://registry.npmjs.org/http-proxy-middleware/-/http-proxy-middleware-2.0.9.tgz",
+      "integrity": "sha512-c1IyJYLYppU574+YI7R4QyX2ystMtVXZwIdzazUIPIJsHuWNd+mho2j+bKoHftndicGj9yh+xjd+l0yj7VeT1Q==",
       "license": "MIT",
       "dependencies": {
         "@types/http-proxy": "^1.17.8",

docs/src/css/custom.css

@@ -6,13 +6,13 @@
 
 /* You can override the default Infima variables here. */
 :root {
-  --ifm-color-primary: #2e8555;
-  --ifm-color-primary-dark: #29784c;
-  --ifm-color-primary-darker: #277148;
-  --ifm-color-primary-darkest: #205d3b;
-  --ifm-color-primary-light: #33925d;
-  --ifm-color-primary-lighter: #359962;
-  --ifm-color-primary-lightest: #3cad6e;
+  --ifm-color-primary: #ff5630;
+  --ifm-color-primary-dark: #ad422a;
+  --ifm-color-primary-darker: #8f3521;
+  --ifm-color-primary-darkest: #592115;
+  --ifm-color-primary-light: #ff7152;
+  --ifm-color-primary-lighter: #ff9178;
+  --ifm-color-primary-lightest: #ffb09e;
   --ifm-code-font-size: 95%;
   --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1);
   --code-block-diff-add-line-color: #ccffd8;
@@ -21,16 +21,17 @@
 
 /* For readability concerns, you should choose a lighter palette in dark mode. */
 [data-theme="dark"] {
-  --ifm-color-primary: #25c2a0;
-  --ifm-color-primary-dark: #21af90;
-  --ifm-color-primary-darker: #1fa588;
-  --ifm-color-primary-darkest: #1a8870;
-  --ifm-color-primary-light: #29d5b0;
-  --ifm-color-primary-lighter: #32d8b4;
-  --ifm-color-primary-lightest: #4fddbf;
-  --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);
-  --code-block-diff-add-line-color: #216932;
-  --code-block-diff-remove-line-color: #8b423b;
+  --ifm-color-primary: #e64a19;
+  --ifm-color-primary-dark: #b73a12;
+  --ifm-color-primary-darker: #8c2c0e;
+  --ifm-color-primary-darkest: #5a1e0a;
+  --ifm-color-primary-light: #eb6d45;
+  --ifm-color-primary-lighter: #f09178;
+  --ifm-color-primary-lightest: #f5b5a6;
+  --ifm-code-font-size: 95%;
+  --docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.25);
+  --code-block-diff-add-line-color: #2d5a2c;
+  --code-block-diff-remove-line-color: #5a2d2c;
 }
 
 .code-block-diff-add-line {

go.mod

@@ -1,28 +1,33 @@
 module github.com/TecharoHQ/anubis
 
-go 1.24.2
+go 1.24.0
+
+toolchain go1.24.2
 
 require (
-	github.com/a-h/templ v0.3.857
+	github.com/a-h/templ v0.3.865
 	github.com/facebookgo/flagenv v0.0.0-20160425205200-fcd59fca7456
 	github.com/golang-jwt/jwt/v5 v5.2.2
-	github.com/playwright-community/playwright-go v0.5001.0
-	github.com/prometheus/client_golang v1.21.1
+	github.com/google/cel-go v0.25.0
+	github.com/playwright-community/playwright-go v0.5200.0
+	github.com/prometheus/client_golang v1.22.0
 	github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a
 	github.com/yl2chen/cidranger v1.0.2
-	golang.org/x/net v0.38.0
+	golang.org/x/net v0.40.0
+	k8s.io/apimachinery v0.33.0
 )
 
 require (
+	cel.dev/expr v0.23.1 // indirect
 	github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c // indirect
 	github.com/a-h/parse v0.0.0-20250122154542-74294addb73e // indirect
 	github.com/andybalholm/brotli v1.1.0 // indirect
+	github.com/antlr4-go/antlr/v4 v4.13.0 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/cli/browser v1.3.0 // indirect
-	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
-	github.com/deckarep/golang-set/v2 v2.6.0 // indirect
+	github.com/deckarep/golang-set/v2 v2.7.0 // indirect
 	github.com/facebookgo/ensure v0.0.0-20160127193407-b4ab57deab51 // indirect
 	github.com/facebookgo/stack v0.0.0-20160209184415-751773369052 // indirect
 	github.com/facebookgo/subset v0.0.0-20150612182917-8dac2c3c4870 // indirect
@@ -30,26 +35,36 @@ require (
 	github.com/fsnotify/fsnotify v1.7.0 // indirect
 	github.com/go-jose/go-jose/v3 v3.0.4 // indirect
 	github.com/go-stack/stack v1.8.1 // indirect
-	github.com/klauspost/compress v1.17.11 // indirect
+	github.com/kr/text v0.2.0 // indirect
 	github.com/mattn/go-colorable v0.1.13 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/natefinch/atomic v1.0.1 // indirect
-	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
 	github.com/prometheus/client_model v0.6.1 // indirect
 	github.com/prometheus/common v0.62.0 // indirect
 	github.com/prometheus/procfs v0.15.1 // indirect
+	github.com/stoewer/go-strcase v1.2.0 // indirect
+	golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa // indirect
 	golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 // indirect
 	golang.org/x/mod v0.24.0 // indirect
-	golang.org/x/sync v0.12.0 // indirect
-	golang.org/x/sys v0.31.0 // indirect
-	golang.org/x/tools v0.31.0 // indirect
-	google.golang.org/protobuf v1.36.4 // indirect
+	golang.org/x/sync v0.14.0 // indirect
+	golang.org/x/sys v0.33.0 // indirect
+	golang.org/x/telemetry v0.0.0-20240522233618-39ace7a40ae7 // indirect
+	golang.org/x/text v0.25.0 // indirect
+	golang.org/x/tools v0.32.0 // indirect
+	golang.org/x/vuln v1.1.4 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20240826202546-f6391c0de4c7 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20240826202546-f6391c0de4c7 // indirect
+	google.golang.org/protobuf v1.36.5 // indirect
 	honnef.co/go/tools v0.6.1 // indirect
+	sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 // indirect
+	sigs.k8s.io/yaml v1.4.0 // indirect
 )
 
 tool (
 	github.com/a-h/templ/cmd/templ
+	golang.org/x/tools/cmd/goimports
 	golang.org/x/tools/cmd/stringer
+	golang.org/x/vuln/cmd/govulncheck
 	honnef.co/go/tools/cmd/staticcheck
 )

go.sum

@@ -1,11 +1,15 @@
+cel.dev/expr v0.23.1 h1:K4KOtPCJQjVggkARsjG9RWXP6O4R73aHeJMa/dmCQQg=
+cel.dev/expr v0.23.1/go.mod h1:hLPLo1W4QUmuYdA72RBX06QTs6MXw941piREPl3Yfiw=
 github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c h1:pxW6RcqyfI9/kWtOwnv/G+AzdKuy2ZrqINhenH4HyNs=
 github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
 github.com/a-h/parse v0.0.0-20250122154542-74294addb73e h1:HjVbSQHy+dnlS6C3XajZ69NYAb5jbGNfHanvm1+iYlo=
 github.com/a-h/parse v0.0.0-20250122154542-74294addb73e/go.mod h1:3mnrkvGpurZ4ZrTDbYU84xhwXW2TjTKShSwjRi2ihfQ=
-github.com/a-h/templ v0.3.857 h1:6EqcJuGZW4OL+2iZ3MD+NnIcG7nGkaQeF2Zq5kf9ZGg=
-github.com/a-h/templ v0.3.857/go.mod h1:qhrhAkRFubE7khxLZHsBFHfX+gWwVNKbzKeF9GlPV4M=
+github.com/a-h/templ v0.3.865 h1:nYn5EWm9EiXaDgWcMQaKiKvrydqgxDUtT1+4zU2C43A=
+github.com/a-h/templ v0.3.865/go.mod h1:oLBbZVQ6//Q6zpvSMPTuBK0F3qOtBdFBcGRspcT+VNQ=
 github.com/andybalholm/brotli v1.1.0 h1:eLKJA0d02Lf0mVpIDgYnqXcUn0GqVmEFny3VuID1U3M=
 github.com/andybalholm/brotli v1.1.0/go.mod h1:sms7XGricyQI9K10gOSf56VKKWS4oLer58Q+mhRPtnY=
+github.com/antlr4-go/antlr/v4 v4.13.0 h1:lxCg3LAv+EUK6t1i0y1V6/SLeUi0eKEKdhQAlS8TVTI=
+github.com/antlr4-go/antlr/v4 v4.13.0/go.mod h1:pfChB/xh/Unjila75QW7+VU4TSnWnnk9UTnmpPaOR2g=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
@@ -14,11 +18,12 @@ github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UF
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/cli/browser v1.3.0 h1:LejqCrpWr+1pRqmEPDGnTZOjsMe7sehifLynZJuqJpo=
 github.com/cli/browser v1.3.0/go.mod h1:HH8s+fOAxjhQoBUAsKuPCbqUuxZDhQ2/aD+SzsEfBTk=
+github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
-github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/deckarep/golang-set/v2 v2.6.0 h1:XfcQbWM1LlMB8BsJ8N9vW5ehnnPVIw0je80NsVHagjM=
-github.com/deckarep/golang-set/v2 v2.6.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/deckarep/golang-set/v2 v2.7.0 h1:gIloKvD7yH2oip4VLhsv3JyLLFnC0Y2mlusgcvJYW5k=
+github.com/deckarep/golang-set/v2 v2.7.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4=
 github.com/facebookgo/ensure v0.0.0-20160127193407-b4ab57deab51 h1:0JZ+dUmQeA8IIVUMzysrX4/AKuQwWhV2dYQuPZdvdSQ=
 github.com/facebookgo/ensure v0.0.0-20160127193407-b4ab57deab51/go.mod h1:Yg+htXGokKKdzcwhuNDwVvN+uBxDGXJ7G/VN1d8fa64=
 github.com/facebookgo/flagenv v0.0.0-20160425205200-fcd59fca7456 h1:CkmB2l68uhvRlwOTPrwnuitSxi/S3Cg4L5QYOcL9MBc=
@@ -37,11 +42,21 @@ github.com/go-stack/stack v1.8.1 h1:ntEHSVwIt7PNXNpgPmVfMrNhLtgjlmnZha2kOpuRiDw=
 github.com/go-stack/stack v1.8.1/go.mod h1:dcoOX6HbPZSZptuspn9bctJ+N/CnF5gGygcUP3XYfe4=
 github.com/golang-jwt/jwt/v5 v5.2.2 h1:Rl4B7itRWVtYIHFrSNd7vhTiz9UpLdi6gZhZ3wEeDy8=
 github.com/golang-jwt/jwt/v5 v5.2.2/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
+github.com/google/cel-go v0.25.0 h1:jsFw9Fhn+3y2kBbltZR4VEz5xKkcIFRPDnuEzAGv5GY=
+github.com/google/cel-go v0.25.0/go.mod h1:hjEb6r5SuOSlhCHmFoLzu8HGCERvIsDAbxDAyNU/MmI=
+github.com/google/go-cmdtest v0.4.1-0.20220921163831-55ab3332a786 h1:rcv+Ippz6RAtvaGgKxc+8FQIpxHgsF+HBzPyYL2cyVU=
+github.com/google/go-cmdtest v0.4.1-0.20220921163831-55ab3332a786/go.mod h1:apVn/GCasLZUVpAJ6oWAuyP7Ne7CEsQbTnc0plM3m+o=
 github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
-github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
-github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
-github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
-github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
+github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
+github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
+github.com/google/renameio v0.1.0 h1:GOZbcHa3HfsPKPlmyPyN2KEohoMXOhdMbHrvbpl2QaA=
+github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
+github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
+github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
+github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
+github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
+github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
 github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
 github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
 github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
@@ -55,23 +70,27 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/natefinch/atomic v1.0.1 h1:ZPYKxkqQOx3KZ+RsbnP/YsgvxWQPGxjC0oBt2AhwV0A=
 github.com/natefinch/atomic v1.0.1/go.mod h1:N/D/ELrljoqDyT3rZrsUmtsuzvHkeB/wWjHV22AZRbM=
-github.com/playwright-community/playwright-go v0.5001.0 h1:EY3oB+rU9cUp6CLHguWE8VMZTwAg+83Yyb7dQqEmGLg=
-github.com/playwright-community/playwright-go v0.5001.0/go.mod h1:kBNWs/w2aJ2ZUp1wEOOFLXgOqvppFngM5OS+qyhl+ZM=
+github.com/playwright-community/playwright-go v0.5200.0 h1:z/5LGuX2tBrg3ug1HupMXLjIG93f1d2MWdDsNhkMQ9c=
+github.com/playwright-community/playwright-go v0.5200.0/go.mod h1:UnnyQZaqUOO5ywAZu60+N4EiWReUqX1MQBBA3Oofvf8=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
-github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/prometheus/client_golang v1.21.1 h1:DOvXXTqVzvkIewV/CDPFdejpMCGeMcbGCQ8YOmu+Ibk=
-github.com/prometheus/client_golang v1.21.1/go.mod h1:U9NM32ykUErtVBxdvD3zfi+EuFkkaBvMb09mIfe0Zgg=
+github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
+github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
 github.com/prometheus/client_model v0.6.1 h1:ZKSh/rekM+n3CeS952MLRAdFwIKqeY8b62p8ais2e9E=
 github.com/prometheus/client_model v0.6.1/go.mod h1:OrxVMOVHjw3lKMa8+x6HeMGkHMQyHDk9E3jmP2AmGiY=
 github.com/prometheus/common v0.62.0 h1:xasJaQlnWAeyHdUBeGjXmutelfJHWMRr+Fg4QszZ2Io=
 github.com/prometheus/common v0.62.0/go.mod h1:vyBcEuLSvWos9B1+CyL7JZ2up+uFzXhkqml0W5zIY1I=
 github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
 github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
+github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
+github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
 github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a h1:iLcLb5Fwwz7g/DLK89F+uQBDeAhHhwdzB5fSlVdhGcM=
 github.com/sebest/xff v0.0.0-20210106013422-671bd2870b3a/go.mod h1:wozgYq9WEBQBaIJe4YZ0qTSFAMxmcwBhQH0fO0R34Z0=
+github.com/stoewer/go-strcase v1.2.0 h1:Z2iHWqGXH00XYgqDmNgQbIBxf3wrNq0F3feEy0ainaU=
+github.com/stoewer/go-strcase v1.2.0/go.mod h1:IBiWB2sKIp3wVVQ3Y035++gc+knqhUQag1KpM8ahLw8=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
 github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
@@ -81,6 +100,8 @@ github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5t
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
+golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa h1:FRnLl4eNAQl8hwxVVC17teOw8kdjVDVAiFMtgUdTSRQ=
+golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa/go.mod h1:zk2irFbV9DP96SEBUUAy67IdHUaZuSnrz1n472HUCLE=
 golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 h1:1P7xPZEwZMoBoz0Yze5Nx2/4pxj6nw9ZqHWXqP0iRgQ=
 golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678/go.mod h1:AbB0pIl9nAr9wVwH+Z2ZpaocVmF5I4GyWCDIsVjR0bk=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
@@ -92,13 +113,13 @@ golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
 golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
-golang.org/x/net v0.38.0 h1:vRMAPTMaeGqVhG5QyLJHqNDwecKTomGeqbnfZyKlBI8=
-golang.org/x/net v0.38.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8=
+golang.org/x/net v0.40.0 h1:79Xs7wF06Gbdcg4kdCCIQArK11Z1hr5POQ6+fIYHNuY=
+golang.org/x/net v0.40.0/go.mod h1:y0hY0exeL2Pku80/zKK7tpntoX23cqL3Oa6njdgRtds=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.12.0 h1:MHc5BpPuC30uJk597Ri8TV3CNZcTLu6B6z4lJy+g6Jw=
-golang.org/x/sync v0.12.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
+golang.org/x/sync v0.14.0 h1:woo0S4Yywslg6hp4eUFjTVOyKt0RookbpAHG4c1HmhQ=
+golang.org/x/sync v0.14.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -109,8 +130,10 @@ golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
-golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
+golang.org/x/sys v0.33.0 h1:q3i8TbbEz+JRD9ywIRlyRAQbM0qF7hu24q3teo2hbuw=
+golang.org/x/sys v0.33.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
+golang.org/x/telemetry v0.0.0-20240522233618-39ace7a40ae7 h1:FemxDzfMUcK2f3YY4H+05K9CDzbSVr2+q/JKN45pey0=
+golang.org/x/telemetry v0.0.0-20240522233618-39ace7a40ae7/go.mod h1:pRgIJT+bRLFKnoM1ldnzKoxTIn14Yxz928LQRYYgIN0=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
@@ -122,19 +145,35 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
 golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
+golang.org/x/text v0.25.0 h1:qVyWApTSYLk/drJRO5mDlNYskwQznZmkpV2c8q9zls4=
+golang.org/x/text v0.25.0/go.mod h1:WEdwpYrmk1qmdHvhkSTNPm3app7v4rsT8F2UD6+VHIA=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
 golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
-golang.org/x/tools v0.31.0 h1:0EedkvKDbh+qistFTd0Bcwe/YLh4vHwWEkiI0toFIBU=
-golang.org/x/tools v0.31.0/go.mod h1:naFTU+Cev749tSJRXJlna0T3WxKvb1kWEx15xA4SdmQ=
+golang.org/x/tools v0.32.0 h1:Q7N1vhpkQv7ybVzLFtTjvQya2ewbwNDZzUgfXGqtMWU=
+golang.org/x/tools v0.32.0/go.mod h1:ZxrU41P/wAbZD8EDa6dDCa6XfpkhJ7HFMjHJXfBDu8s=
+golang.org/x/vuln v1.1.4 h1:Ju8QsuyhX3Hk8ma3CesTbO8vfJD9EvUBgHvkxHBzj0I=
+golang.org/x/vuln v1.1.4/go.mod h1:F+45wmU18ym/ca5PLTPLsSzr2KppzswxPP603ldA67s=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-google.golang.org/protobuf v1.36.4 h1:6A3ZDJHn/eNqc1i+IdefRzy/9PokBTPvcqMySR7NNIM=
-google.golang.org/protobuf v1.36.4/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
+google.golang.org/genproto/googleapis/api v0.0.0-20240826202546-f6391c0de4c7 h1:YcyjlL1PRr2Q17/I0dPk2JmYS5CDXfcdb2Z3YRioEbw=
+google.golang.org/genproto/googleapis/api v0.0.0-20240826202546-f6391c0de4c7/go.mod h1:OCdP9MfskevB/rbYvHTsXTtKC+3bHWajPdoKgjcYkfo=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20240826202546-f6391c0de4c7 h1:2035KHhUv+EpyB+hWgJnaWKJOdX1E95w2S8Rr4uWKTs=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20240826202546-f6391c0de4c7/go.mod h1:UqMtugtsSgubUsoxbuAoiCXvqvErP7Gf0so0mK9tHxU=
+google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
+google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
+gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 honnef.co/go/tools v0.6.1 h1:R094WgE8K4JirYjBaOpz/AvTyUu/3wbmAoskKN/pxTI=
 honnef.co/go/tools v0.6.1/go.mod h1:3puzxxljPCe8RGJX7BIy1plGbxEOZni5mR2aXe3/uk4=
+k8s.io/apimachinery v0.33.0 h1:1a6kHrJxb2hs4t8EE5wuR/WxKDwGN1FKH3JvDtA0CIQ=
+k8s.io/apimachinery v0.33.0/go.mod h1:BHW0YOu7n22fFv/JkYOEfkUYNRN0fj0BlvMFWA7b+SM=
+sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 h1:/Rv+M11QRah1itp8VhT6HoVx1Ray9eB4DBr+K+/sCJ8=
+sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3/go.mod h1:18nIHnGi6636UCz6m8i4DhaJ65T6EruyzmoQqI2BVDo=
+sigs.k8s.io/yaml v1.4.0 h1:Mk1wCc2gy/F0THH0TAp1QYyJNzRm2KCLy3o5ASXVI5E=
+sigs.k8s.io/yaml v1.4.0/go.mod h1:Ejl7/uTz7PSA4eKMyQCUTnhZYNmLIl+5c2lQPGR2BPY=

internal/headers.go

@@ -1,15 +1,29 @@
 package internal
 
 import (
+	"errors"
+	"fmt"
 	"log/slog"
 	"net"
 	"net/http"
+	"net/netip"
 	"strings"
 
 	"github.com/TecharoHQ/anubis"
 	"github.com/sebest/xff"
 )
 
+// TODO: move into config
+type XFFComputePreferences struct {
+	StripPrivate  bool
+	StripLoopback bool
+	StripCGNAT    bool
+	StripLLU      bool
+	Flatten       bool
+}
+
+var CGNat = netip.MustParsePrefix("100.64.0.0/10")
+
 // UnchangingCache sets the Cache-Control header to cache a response for 1 year if
 // and only if the application is compiled in "release" mode by Docker.
 func UnchangingCache(next http.Handler) http.Handler {
@@ -65,6 +79,115 @@ func XForwardedForToXRealIP(next http.Handler) http.Handler {
 	})
 }
 
+// XForwardedForUpdate sets or updates the X-Forwarded-For header, adding
+// the known remote address to an existing chain if present
+func XForwardedForUpdate(next http.Handler) http.Handler {
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		defer next.ServeHTTP(w, r)
+
+		pref := XFFComputePreferences{
+			StripPrivate:  true,
+			StripLoopback: true,
+			StripCGNAT:    true,
+			Flatten:       true,
+			StripLLU:      true,
+		}
+
+		remoteAddr := r.RemoteAddr
+		origXFFHeader := r.Header.Get("X-Forwarded-For")
+
+		if remoteAddr == "@" {
+			// remote is a unix socket
+			// do not touch chain
+			return
+		}
+
+		xffHeaderString, err := computeXFFHeader(remoteAddr, origXFFHeader, pref)
+		if err != nil {
+			slog.Debug("computing X-Forwarded-For header failed", "err", err)
+			return
+		}
+
+		if len(xffHeaderString) == 0 {
+			r.Header.Del("X-Forwarded-For")
+		} else {
+			r.Header.Set("X-Forwarded-For", xffHeaderString)
+		}
+	})
+}
+
+var (
+	ErrCantSplitHostParse = errors.New("internal: unable to net.SplitHostParse")
+	ErrCantParseRemoteIP  = errors.New("internal: unable to parse remote IP")
+)
+
+func computeXFFHeader(remoteAddr string, origXFFHeader string, pref XFFComputePreferences) (string, error) {
+	remoteIP, _, err := net.SplitHostPort(remoteAddr)
+	if err != nil {
+		return "", fmt.Errorf("%w: %w", ErrCantSplitHostParse, err)
+	}
+	parsedRemoteIP, err := netip.ParseAddr(remoteIP)
+	if err != nil {
+		return "", fmt.Errorf("%w: %w", ErrCantParseRemoteIP, err)
+	}
+
+	origForwardedList := make([]string, 0, 4)
+	if origXFFHeader != "" {
+		origForwardedList = strings.Split(origXFFHeader, ",")
+		for i := range origForwardedList {
+			origForwardedList[i] = strings.TrimSpace(origForwardedList[i])
+		}
+	}
+	origForwardedList = append(origForwardedList, parsedRemoteIP.String())
+	forwardedList := make([]string, 0, len(origForwardedList))
+	// this behavior is equivalent to
+	// ingress-nginx "compute-full-forwarded-for"
+	// https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#compute-full-forwarded-for
+	//
+	// this would be the correct place to strip and/or flatten this list
+	//
+	// strip - iterate backwards and eliminate configured trusted IPs
+	// flatten - only return the last element to avoid spoofing confusion
+	//
+	// many applications handle this in different ways, but
+	// generally they'd be expected to do these two things on
+	// their own end to find the first non-spoofed IP
+	for i := len(origForwardedList) - 1; i >= 0; i-- {
+		segmentIP, err := netip.ParseAddr(origForwardedList[i])
+		if err != nil {
+			// can't assess this element, so the remainder of the chain
+			// can't be trusted. not a fatal error, since anyone can
+			// spoof an XFF header
+			slog.Debug("failed to parse XFF segment", "err", err)
+			break
+		}
+		if pref.StripPrivate && segmentIP.IsPrivate() {
+			continue
+		}
+		if pref.StripLoopback && segmentIP.IsLoopback() {
+			continue
+		}
+		if pref.StripLLU && segmentIP.IsLinkLocalUnicast() {
+			continue
+		}
+		if pref.StripCGNAT && CGNat.Contains(segmentIP) {
+			continue
+		}
+		forwardedList = append([]string{segmentIP.String()}, forwardedList...)
+	}
+	var xffHeaderString string
+	if len(forwardedList) == 0 {
+		xffHeaderString = ""
+		return xffHeaderString, nil
+	}
+	if pref.Flatten {
+		xffHeaderString = forwardedList[len(forwardedList)-1]
+	} else {
+		xffHeaderString = strings.Join(forwardedList, ",")
+	}
+	return xffHeaderString, nil
+}
+
 // NoStoreCache sets the Cache-Control header to no-store for the response.
 func NoStoreCache(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -73,7 +196,7 @@ func NoStoreCache(next http.Handler) http.Handler {
 	})
 }
 
-// Do not allow browsing directory listings in paths that end with /
+// NoBrowsing prevents directory browsing by returning a 404 for any request that ends with a "/".
 func NoBrowsing(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		if strings.HasSuffix(r.URL.Path, "/") {

internal/log.go

@@ -0,0 +1,59 @@
+package internal
+
+import (
+	"fmt"
+	"log"
+	"log/slog"
+	"net/http"
+	"os"
+	"strings"
+)
+
+func InitSlog(level string) {
+	var programLevel slog.Level
+	if err := (&programLevel).UnmarshalText([]byte(level)); err != nil {
+		fmt.Fprintf(os.Stderr, "invalid log level %s: %v, using info\n", level, err)
+		programLevel = slog.LevelInfo
+	}
+
+	leveler := &slog.LevelVar{}
+	leveler.Set(programLevel)
+
+	h := slog.NewJSONHandler(os.Stderr, &slog.HandlerOptions{
+		AddSource: true,
+		Level:     leveler,
+	})
+	slog.SetDefault(slog.New(h))
+}
+
+func GetRequestLogger(r *http.Request) *slog.Logger {
+	return slog.With(
+		"user_agent", r.UserAgent(),
+		"accept_language", r.Header.Get("Accept-Language"),
+		"priority", r.Header.Get("Priority"),
+		"x-forwarded-for",
+		r.Header.Get("X-Forwarded-For"),
+		"x-real-ip", r.Header.Get("X-Real-Ip"),
+	)
+}
+
+// ErrorLogFilter is used to suppress "context canceled" logs from the http server when a request is canceled (e.g., when a client disconnects).
+type ErrorLogFilter struct {
+	Unwrap *log.Logger
+}
+
+func (elf *ErrorLogFilter) Write(p []byte) (n int, err error) {
+	logMessage := string(p)
+	if strings.Contains(logMessage, "context canceled") {
+		return len(p), nil // Suppress the log by doing nothing
+	}
+	if elf.Unwrap != nil {
+		return elf.Unwrap.Writer().Write(p)
+	}
+	return len(p), nil
+}
+
+func GetFilteredHTTPLogger() *log.Logger {
+	stdErrLogger := log.New(os.Stderr, "", log.LstdFlags) // essentially what the default logger is.
+	return log.New(&ErrorLogFilter{Unwrap: stdErrLogger}, "", 0)
+}

internal/log_test.go

@@ -0,0 +1,46 @@
+package internal
+
+import (
+	"bytes"
+	"log"
+	"strings"
+	"testing"
+)
+
+func TestErrorLogFilter(t *testing.T) {
+	var buf bytes.Buffer
+	destLogger := log.New(&buf, "", 0)
+	errorFilterWriter := &ErrorLogFilter{Unwrap: destLogger}
+	testErrorLogger := log.New(errorFilterWriter, "", 0)
+
+	// Test Case 1: Suppressed message
+	suppressedMessage := "http: proxy error: context canceled"
+	testErrorLogger.Println(suppressedMessage)
+
+	if buf.Len() != 0 {
+		t.Errorf("Suppressed message was written to output. Output: %q", buf.String())
+	}
+	buf.Reset()
+
+	// Test Case 2: Allowed message
+	allowedMessage := "http: another error occurred"
+	testErrorLogger.Println(allowedMessage)
+
+	output := buf.String()
+	if !strings.Contains(output, allowedMessage) {
+		t.Errorf("Allowed message was not written to output. Output: %q", output)
+	}
+	if !strings.HasSuffix(output, "\n") {
+		t.Errorf("Allowed message output is missing newline. Output: %q", output)
+	}
+	buf.Reset()
+
+	// Test Case 3: Partially matching message (should be suppressed)
+	partiallyMatchingMessage := "Some other log before http: proxy error: context canceled and after"
+	testErrorLogger.Println(partiallyMatchingMessage)
+
+	if buf.Len() != 0 {
+		t.Errorf("Partially matching message was written to output. Output: %q", buf.String())
+	}
+	buf.Reset()
+}

internal/ogtags/cache.go

@@ -8,23 +8,26 @@ import (
 )
 
 // GetOGTags is the main function that retrieves Open Graph tags for a URL
-func (c *OGTagCache) GetOGTags(url *url.URL) (map[string]string, error) {
+func (c *OGTagCache) GetOGTags(url *url.URL, originalHost string) (map[string]string, error) {
 	if url == nil {
 		return nil, errors.New("nil URL provided, cannot fetch OG tags")
 	}
-	urlStr := c.getTarget(url)
+
+	target := c.getTarget(url)
+	cacheKey := c.generateCacheKey(target, originalHost)
+
 	// Check cache first
-	if cachedTags := c.checkCache(urlStr); cachedTags != nil {
+	if cachedTags := c.checkCache(cacheKey); cachedTags != nil {
 		return cachedTags, nil
 	}
 
-	// Fetch HTML content
-	doc, err := c.fetchHTMLDocument(urlStr)
+	// Fetch HTML content, passing the original host
+	doc, err := c.fetchHTMLDocumentWithCache(target, originalHost, cacheKey)
 	if errors.Is(err, syscall.ECONNREFUSED) {
 		slog.Debug("Connection refused, returning empty tags")
 		return nil, nil
-	} else if errors.Is(err, ErrNotFound) {
-		// not even worth a debug log...
+	} else if errors.Is(err, ErrOgHandled) {
+		// Error was handled in fetchHTMLDocument, return empty tags
 		return nil, nil
 	}
 	if err != nil {
@@ -35,17 +38,28 @@ func (c *OGTagCache) GetOGTags(url *url.URL) (map[string]string, error) {
 	ogTags := c.extractOGTags(doc)
 
 	// Store in cache
-	c.cache.Set(urlStr, ogTags, c.ogTimeToLive)
+	c.cache.Set(cacheKey, ogTags, c.ogTimeToLive)
 
 	return ogTags, nil
 }
 
+func (c *OGTagCache) generateCacheKey(target string, originalHost string) string {
+	var cacheKey string
+
+	if c.ogCacheConsiderHost {
+		cacheKey = target + "|" + originalHost
+	} else {
+		cacheKey = target
+	}
+	return cacheKey
+}
+
 // checkCache checks if we have the tags cached and returns them if so
-func (c *OGTagCache) checkCache(urlStr string) map[string]string {
-	if cachedTags, ok := c.cache.Get(urlStr); ok {
+func (c *OGTagCache) checkCache(cacheKey string) map[string]string {
+	if cachedTags, ok := c.cache.Get(cacheKey); ok {
 		slog.Debug("cache hit", "tags", cachedTags)
 		return cachedTags
 	}
-	slog.Debug("cache miss", "url", urlStr)
+	slog.Debug("cache miss", "url", cacheKey)
 	return nil
 }

internal/ogtags/cache_test.go

@@ -4,12 +4,13 @@ import (
 	"net/http"
 	"net/http/httptest"
 	"net/url"
+	"reflect"
 	"testing"
 	"time"
 )
 
 func TestCheckCache(t *testing.T) {
-	cache := NewOGTagCache("http://example.com", true, time.Minute)
+	cache := NewOGTagCache("http://example.com", true, time.Minute, false)
 
 	// Set up test data
 	urlStr := "http://example.com/page"
@@ -17,18 +18,19 @@ func TestCheckCache(t *testing.T) {
 		"og:title":       "Test Title",
 		"og:description": "Test Description",
 	}
+	cacheKey := cache.generateCacheKey(urlStr, "example.com")
 
 	// Test cache miss
-	tags := cache.checkCache(urlStr)
+	tags := cache.checkCache(cacheKey)
 	if tags != nil {
 		t.Errorf("expected nil tags on cache miss, got %v", tags)
 	}
 
 	// Manually add to cache
-	cache.cache.Set(urlStr, expectedTags, time.Minute)
+	cache.cache.Set(cacheKey, expectedTags, time.Minute)
 
 	// Test cache hit
-	tags = cache.checkCache(urlStr)
+	tags = cache.checkCache(cacheKey)
 	if tags == nil {
 		t.Fatal("expected non-nil tags on cache hit, got nil")
 	}
@@ -67,7 +69,7 @@ func TestGetOGTags(t *testing.T) {
 	defer ts.Close()
 
 	// Create an instance of OGTagCache with a short TTL for testing
-	cache := NewOGTagCache(ts.URL, true, 1*time.Minute)
+	cache := NewOGTagCache(ts.URL, true, 1*time.Minute, false)
 
 	// Parse the test server URL
 	parsedURL, err := url.Parse(ts.URL)
@@ -76,7 +78,8 @@ func TestGetOGTags(t *testing.T) {
 	}
 
 	// Test fetching OG tags from the test server
-	ogTags, err := cache.GetOGTags(parsedURL)
+	// Pass the host from the parsed test server URL
+	ogTags, err := cache.GetOGTags(parsedURL, parsedURL.Host)
 	if err != nil {
 		t.Fatalf("failed to get OG tags: %v", err)
 	}
@@ -95,13 +98,15 @@ func TestGetOGTags(t *testing.T) {
 	}
 
 	// Test fetching OG tags from the cache
-	ogTags, err = cache.GetOGTags(parsedURL)
+	// Pass the host from the parsed test server URL
+	ogTags, err = cache.GetOGTags(parsedURL, parsedURL.Host)
 	if err != nil {
 		t.Fatalf("failed to get OG tags from cache: %v", err)
 	}
 
 	// Test fetching OG tags from the cache (3rd time)
-	newOgTags, err := cache.GetOGTags(parsedURL)
+	// Pass the host from the parsed test server URL
+	newOgTags, err := cache.GetOGTags(parsedURL, parsedURL.Host)
 	if err != nil {
 		t.Fatalf("failed to get OG tags from cache: %v", err)
 	}
@@ -120,3 +125,116 @@ func TestGetOGTags(t *testing.T) {
 
 	}
 }
+
+// TestGetOGTagsWithHostConsideration tests the behavior of the cache with and without host consideration and for multiple hosts in a theoretical setup.
+func TestGetOGTagsWithHostConsideration(t *testing.T) {
+	var loadCount int // Counter to track how many times the test route is loaded
+
+	// Create a test server
+	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		loadCount++ // Increment counter on each request to the server
+		w.Header().Set("Content-Type", "text/html")
+		w.Write([]byte(`
+			<!DOCTYPE html>
+			<html>
+			<head>
+				<meta property="og:title" content="Test Title" />
+				<meta property="og:description" content="Test Description" />
+			</head>
+			<body><p>Content</p></body>
+			</html>
+		`))
+	}))
+	defer ts.Close()
+
+	parsedURL, err := url.Parse(ts.URL)
+	if err != nil {
+		t.Fatalf("failed to parse test server URL: %v", err)
+	}
+
+	expectedTags := map[string]string{
+		"og:title":       "Test Title",
+		"og:description": "Test Description",
+	}
+
+	testCases := []struct {
+		name                string
+		ogCacheConsiderHost bool
+		requests            []struct {
+			host              string
+			expectedLoadCount int // Expected load count *after* this request
+		}
+	}{
+		{
+			name:                "Host Not Considered - Same Host",
+			ogCacheConsiderHost: false,
+			requests: []struct {
+				host              string
+				expectedLoadCount int
+			}{
+				{"host1", 1}, // First request, miss
+				{"host1", 1}, // Second request, same host, hit (host ignored)
+			},
+		},
+		{
+			name:                "Host Not Considered - Different Host",
+			ogCacheConsiderHost: false,
+			requests: []struct {
+				host              string
+				expectedLoadCount int
+			}{
+				{"host1", 1}, // First request, miss
+				{"host2", 1}, // Second request, different host, hit (host ignored)
+			},
+		},
+		{
+			name:                "Host Considered - Same Host",
+			ogCacheConsiderHost: true,
+			requests: []struct {
+				host              string
+				expectedLoadCount int
+			}{
+				{"host1", 1}, // First request, miss
+				{"host1", 1}, // Second request, same host, hit
+			},
+		},
+		{
+			name:                "Host Considered - Different Host",
+			ogCacheConsiderHost: true,
+			requests: []struct {
+				host              string
+				expectedLoadCount int
+			}{
+				{"host1", 1}, // First request, miss
+				{"host2", 2}, // Second request, different host, miss
+				{"host2", 2}, // Third request, same as second, hit
+				{"host1", 2}, // Fourth request, same as first, hit
+			},
+		},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			loadCount = 0 // Reset load count for each test case
+			cache := NewOGTagCache(ts.URL, true, 1*time.Minute, tc.ogCacheConsiderHost)
+
+			for i, req := range tc.requests {
+				ogTags, err := cache.GetOGTags(parsedURL, req.host)
+				if err != nil {
+					t.Errorf("Request %d (host: %s): unexpected error: %v", i+1, req.host, err)
+					continue // Skip further checks for this request if error occurred
+				}
+
+				// Verify tags are correct (should always be the same in this setup)
+				if !reflect.DeepEqual(ogTags, expectedTags) {
+					t.Errorf("Request %d (host: %s): expected tags %v, got %v", i+1, req.host, expectedTags, ogTags)
+				}
+
+				// Verify the load count to check cache hit/miss behavior
+				if loadCount != req.expectedLoadCount {
+					t.Errorf("Request %d (host: %s): expected load count %d, got %d (cache hit/miss mismatch)", i+1, req.host, req.expectedLoadCount, loadCount)
+				}
+			}
+		})
+	}
+}

internal/ogtags/fetch.go

@@ -1,9 +1,11 @@
 package ogtags
 
 import (
+	"context"
 	"errors"
 	"fmt"
 	"golang.org/x/net/html"
+	"io"
 	"log/slog"
 	"mime"
 	"net"
@@ -11,57 +13,79 @@ import (
 )
 
 var (
-	ErrNotFound = errors.New("page not found") /*todo: refactor into common errors lib? */
-	emptyMap    = map[string]string{}          // used to indicate an empty result in the cache. Can't use nil as it would be a cache miss.
+	ErrOgHandled = errors.New("og: handled error") // used to indicate that the error was handled and should not be logged
+	emptyMap     = map[string]string{}             // used to indicate an empty result in the cache. Can't use nil as it would be a cache miss.
 )
 
-func (c *OGTagCache) fetchHTMLDocument(urlStr string) (*html.Node, error) {
-	resp, err := c.client.Get(urlStr)
+// fetchHTMLDocumentWithCache fetches the HTML document from the given URL string,
+// preserving the original host header.
+func (c *OGTagCache) fetchHTMLDocumentWithCache(urlStr string, originalHost string, cacheKey string) (*html.Node, error) {
+	req, err := http.NewRequestWithContext(context.Background(), "GET", urlStr, nil)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create http request: %w", err)
+	}
+
+	// Set the Host header to the original host
+	if originalHost != "" {
+		req.Host = originalHost
+	}
+
+	// Add proxy headers
+	req.Header.Set("X-Forwarded-Proto", "https")
+	req.Header.Set("User-Agent", "Anubis-OGTag-Fetcher/1.0") // For tracking purposes
+
+	// Send the request
+	resp, err := c.client.Do(req)
 	if err != nil {
 		var netErr net.Error
 		if errors.As(err, &netErr) && netErr.Timeout() {
 			slog.Debug("og: request timed out", "url", urlStr)
-			c.cache.Set(urlStr, emptyMap, c.ogTimeToLive/2) // Cache empty result for half the TTL to not spam the server
+			c.cache.Set(cacheKey, emptyMap, c.ogTimeToLive/2) // Cache empty result for half the TTL to not spam the server
 		}
 		return nil, fmt.Errorf("http get failed: %w", err)
 	}
-	// this defer will call MaxBytesReader's Close, which closes the original body.
-	defer resp.Body.Close()
+
+	// Ensure the response body is closed
+	defer func(Body io.ReadCloser) {
+		err := Body.Close()
+		if err != nil {
+			slog.Debug("og: error closing response body", "url", urlStr, "error", err)
+		}
+	}(resp.Body)
 
 	if resp.StatusCode != http.StatusOK {
 		slog.Debug("og: received non-OK status code", "url", urlStr, "status", resp.StatusCode)
-		c.cache.Set(urlStr, emptyMap, c.ogTimeToLive) // Cache empty result for non-successful status codes
-		return nil, ErrNotFound
+		c.cache.Set(cacheKey, emptyMap, c.ogTimeToLive) // Cache empty result for non-successful status codes
+		return nil, fmt.Errorf("%w: page not found", ErrOgHandled)
 	}
 
 	// Check content type
 	ct := resp.Header.Get("Content-Type")
 	if ct == "" {
-		// assume non html body
 		return nil, fmt.Errorf("missing Content-Type header")
 	} else {
 		mediaType, _, err := mime.ParseMediaType(ct)
 		if err != nil {
-			// Malformed Content-Type header
-			return nil, fmt.Errorf("invalid Content-Type '%s': %w", ct, err)
+			slog.Debug("og: malformed Content-Type header", "url", urlStr, "contentType", ct)
+			return nil, fmt.Errorf("%w malformed Content-Type header: %w", ErrOgHandled, err)
 		}
 
 		if mediaType != "text/html" && mediaType != "application/xhtml+xml" {
-			return nil, fmt.Errorf("unsupported Content-Type: %s", mediaType)
+			slog.Debug("og: unsupported Content-Type", "url", urlStr, "contentType", mediaType)
+			return nil, fmt.Errorf("%w unsupported Content-Type: %s", ErrOgHandled, mediaType)
 		}
 	}
 
-	resp.Body = http.MaxBytesReader(nil, resp.Body, c.maxContentLength)
+	resp.Body = http.MaxBytesReader(nil, resp.Body, maxContentLength)
 
 	doc, err := html.Parse(resp.Body)
 	if err != nil {
 		// Check if the error is specifically because the limit was exceeded
 		var maxBytesErr *http.MaxBytesError
 		if errors.As(err, &maxBytesErr) {
-			slog.Debug("og: content exceeded max length", "url", urlStr, "limit", c.maxContentLength)
-			return nil, fmt.Errorf("content too large: exceeded %d bytes", c.maxContentLength)
+			slog.Debug("og: content exceeded max length", "url", urlStr, "limit", maxContentLength)
+			return nil, fmt.Errorf("content too large: exceeded %d bytes", maxContentLength)
 		}
-		// parsing error (e.g., malformed HTML)
 		return nil, fmt.Errorf("failed to parse HTML: %w", err)
 	}
 

internal/ogtags/fetch_test.go

@@ -2,6 +2,7 @@ package ogtags
 
 import (
 	"fmt"
+	"golang.org/x/net/html"
 	"io"
 	"net/http"
 	"net/http/httptest"
@@ -78,8 +79,8 @@ func TestFetchHTMLDocument(t *testing.T) {
 			}))
 			defer ts.Close()
 
-			cache := NewOGTagCache("", true, time.Minute)
-			doc, err := cache.fetchHTMLDocument(ts.URL)
+			cache := NewOGTagCache("", true, time.Minute, false)
+			doc, err := cache.fetchHTMLDocument(ts.URL, "anything")
 
 			if tt.expectError {
 				if err == nil {
@@ -105,9 +106,9 @@ func TestFetchHTMLDocumentInvalidURL(t *testing.T) {
 		t.Skip("test requires theoretical network egress")
 	}
 
-	cache := NewOGTagCache("", true, time.Minute)
+	cache := NewOGTagCache("", true, time.Minute, false)
 
-	doc, err := cache.fetchHTMLDocument("http://invalid.url.that.doesnt.exist.example")
+	doc, err := cache.fetchHTMLDocument("http://invalid.url.that.doesnt.exist.example", "anything")
 
 	if err == nil {
 		t.Error("expected error for invalid URL, got nil")
@@ -117,3 +118,9 @@ func TestFetchHTMLDocumentInvalidURL(t *testing.T) {
 		t.Error("expected nil document for invalid URL, got non-nil")
 	}
 }
+
+// fetchHTMLDocument allows you to call fetchHTMLDocumentWithCache without a duplicate generateCacheKey call
+func (c *OGTagCache) fetchHTMLDocument(urlStr string, originalHost string) (*html.Node, error) {
+	cacheKey := c.generateCacheKey(urlStr, originalHost)
+	return c.fetchHTMLDocumentWithCache(urlStr, originalHost, cacheKey)
+}

internal/ogtags/integration_test.go

@@ -93,7 +93,7 @@ func TestIntegrationGetOGTags(t *testing.T) {
 			expectError:  false,
 		},
 		{
-			name:         "Non-existent page",
+			name:         "Nonexistent page",
 			path:         "/not-found",
 			query:        "",
 			expectedTags: nil,
@@ -104,7 +104,7 @@ func TestIntegrationGetOGTags(t *testing.T) {
 	for _, tc := range testCases {
 		t.Run(tc.name, func(t *testing.T) {
 			// Create cache instance
-			cache := NewOGTagCache(ts.URL, true, 1*time.Minute)
+			cache := NewOGTagCache(ts.URL, true, 1*time.Minute, false)
 
 			// Create URL for test
 			testURL, _ := url.Parse(ts.URL)
@@ -112,7 +112,8 @@ func TestIntegrationGetOGTags(t *testing.T) {
 			testURL.RawQuery = tc.query
 
 			// Get OG tags
-			ogTags, err := cache.GetOGTags(testURL)
+			// Pass the host from the test URL
+			ogTags, err := cache.GetOGTags(testURL, testURL.Host)
 
 			// Check error expectation
 			if tc.expectError {
@@ -139,7 +140,8 @@ func TestIntegrationGetOGTags(t *testing.T) {
 			}
 
 			// Test cache retrieval
-			cachedOGTags, err := cache.GetOGTags(testURL)
+			// Pass the host from the test URL
+			cachedOGTags, err := cache.GetOGTags(testURL, testURL.Host)
 			if err != nil {
 				t.Fatalf("failed to get OG tags from cache: %v", err)
 			}

internal/ogtags/ogtags.go

@@ -1,51 +1,111 @@
 package ogtags
 
 import (
+	"context"
+	"log/slog"
+	"net"
 	"net/http"
 	"net/url"
+	"strings"
 	"time"
 
 	"github.com/TecharoHQ/anubis/decaymap"
 )
 
+const (
+	maxContentLength = 16 << 20        // 16 MiB in bytes, if there is a reasonable reason that you need more than this...Why?
+	httpTimeout      = 5 * time.Second /*todo: make this configurable?*/
+)
+
 type OGTagCache struct {
-	cache            *decaymap.Impl[string, map[string]string]
-	target           string
-	ogPassthrough    bool
-	ogTimeToLive     time.Duration
-	approvedTags     []string
-	approvedPrefixes []string
-	client           *http.Client
-	maxContentLength int64
+	cache               *decaymap.Impl[string, map[string]string]
+	targetURL           *url.URL
+	ogCacheConsiderHost bool
+	ogPassthrough       bool
+	ogTimeToLive        time.Duration
+	approvedTags        []string
+	approvedPrefixes    []string
+	client              *http.Client
 }
 
-func NewOGTagCache(target string, ogPassthrough bool, ogTimeToLive time.Duration) *OGTagCache {
+func NewOGTagCache(target string, ogPassthrough bool, ogTimeToLive time.Duration, ogTagsConsiderHost bool) *OGTagCache {
 	// Predefined approved tags and prefixes
 	// In the future, these could come from configuration
 	defaultApprovedTags := []string{"description", "keywords", "author"}
 	defaultApprovedPrefixes := []string{"og:", "twitter:", "fediverse:"}
-	client := &http.Client{
-		Timeout: 5 * time.Second, /*make this configurable?*/
+
+	var parsedTargetURL *url.URL
+	var err error
+
+	if target == "" {
+		// Default to localhost if target is empty
+		parsedTargetURL, _ = url.Parse("http://localhost")
+	} else {
+		parsedTargetURL, err = url.Parse(target)
+		if err != nil {
+			slog.Debug("og: failed to parse target URL, treating as non-unix", "target", target, "error", err)
+			// If parsing fails, treat it as a non-unix target for backward compatibility or default behavior
+			// For now, assume it's not a scheme issue but maybe an invalid char, etc.
+			// A simple string target might be intended if it's not a full URL.
+			parsedTargetURL = &url.URL{Scheme: "http", Host: target} // Assume http if scheme missing and host-like
+			if !strings.Contains(target, "://") && !strings.HasPrefix(target, "unix:") {
+				// If it looks like just a host/host:port (and not unix), prepend http:// (todo: is this bad...? Trace path to see if i can yell at user to do it right)
+				parsedTargetURL, _ = url.Parse("http://" + target) // fetch cares about scheme but anubis doesn't
+			}
+		}
 	}
 
-	const maxContentLength = 16 << 20 // 16 MiB in bytes
+	client := &http.Client{
+		Timeout: httpTimeout,
+	}
+
+	// Configure custom transport for Unix sockets
+	if parsedTargetURL.Scheme == "unix" {
+		socketPath := parsedTargetURL.Path // For unix scheme, path is the socket path
+		client.Transport = &http.Transport{
+			DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
+				return net.Dial("unix", socketPath)
+			},
+		}
+	}
 
 	return &OGTagCache{
-		cache:            decaymap.New[string, map[string]string](),
-		target:           target,
-		ogPassthrough:    ogPassthrough,
-		ogTimeToLive:     ogTimeToLive,
-		approvedTags:     defaultApprovedTags,
-		approvedPrefixes: defaultApprovedPrefixes,
-		client:           client,
-		maxContentLength: maxContentLength,
+		cache:               decaymap.New[string, map[string]string](),
+		targetURL:           parsedTargetURL, // Store the parsed URL
+		ogPassthrough:       ogPassthrough,
+		ogTimeToLive:        ogTimeToLive,
+		ogCacheConsiderHost: ogTagsConsiderHost, // todo: refactor to be a separate struct
+		approvedTags:        defaultApprovedTags,
+		approvedPrefixes:    defaultApprovedPrefixes,
+		client:              client,
 	}
 }
 
+// getTarget constructs the target URL string for fetching OG tags.
+// For Unix sockets, it creates a "fake" HTTP URL that the custom dialer understands.
 func (c *OGTagCache) getTarget(u *url.URL) string {
-	return c.target + u.Path
+	if c.targetURL.Scheme == "unix" {
+		// The custom dialer ignores the host, but we need a valid http URL structure.
+		// Use "unix" as a placeholder host. Path and Query from original request are appended.
+		fakeURL := &url.URL{
+			Scheme:   "http", // Scheme must be http/https for client.Get
+			Host:     "unix", // Arbitrary host, ignored by custom dialer
+			Path:     u.Path,
+			RawQuery: u.RawQuery,
+		}
+		return fakeURL.String()
+	}
+
+	// For regular http/https targets
+	target := *c.targetURL // Make a copy
+	target.Path = u.Path
+	target.RawQuery = u.RawQuery
+	return target.String()
+
 }
 
 func (c *OGTagCache) Cleanup() {
-	c.cache.Cleanup()
+	if c.cache != nil {
+		c.cache.Cleanup()
+	}
 }

internal/ogtags/ogtags_test.go

@@ -1,7 +1,16 @@
 package ogtags
 
 import (
+	"context"
+	"errors"
+	"fmt"
+	"net"
+	"net/http"
 	"net/url"
+	"os"
+	"path/filepath"
+	"reflect"
+	"strings"
 	"testing"
 	"time"
 )
@@ -29,14 +38,23 @@ func TestNewOGTagCache(t *testing.T) {
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			cache := NewOGTagCache(tt.target, tt.ogPassthrough, tt.ogTimeToLive)
+			cache := NewOGTagCache(tt.target, tt.ogPassthrough, tt.ogTimeToLive, false)
 
 			if cache == nil {
 				t.Fatal("expected non-nil cache, got nil")
 			}
 
-			if cache.target != tt.target {
-				t.Errorf("expected target %s, got %s", tt.target, cache.target)
+			// Check the parsed targetURL, handling the default case for empty target
+			expectedURLStr := tt.target
+			if tt.target == "" {
+				// Default behavior when target is empty is now http://localhost
+				expectedURLStr = "http://localhost"
+			} else if !strings.Contains(tt.target, "://") && !strings.HasPrefix(tt.target, "unix:") {
+				// Handle case where target is just host or host:port (and not unix)
+				expectedURLStr = "http://" + tt.target
+			}
+			if cache.targetURL.String() != expectedURLStr {
+				t.Errorf("expected targetURL %s, got %s", expectedURLStr, cache.targetURL.String())
 			}
 
 			if cache.ogPassthrough != tt.ogPassthrough {
@@ -50,6 +68,45 @@ func TestNewOGTagCache(t *testing.T) {
 	}
 }
 
+// TestNewOGTagCache_UnixSocket specifically tests unix socket initialization
+func TestNewOGTagCache_UnixSocket(t *testing.T) {
+	tempDir := t.TempDir()
+	socketPath := filepath.Join(tempDir, "test.sock")
+	target := "unix://" + socketPath
+
+	cache := NewOGTagCache(target, true, 5*time.Minute, false)
+
+	if cache == nil {
+		t.Fatal("expected non-nil cache, got nil")
+	}
+
+	if cache.targetURL.Scheme != "unix" {
+		t.Errorf("expected targetURL scheme 'unix', got '%s'", cache.targetURL.Scheme)
+	}
+	if cache.targetURL.Path != socketPath {
+		t.Errorf("expected targetURL path '%s', got '%s'", socketPath, cache.targetURL.Path)
+	}
+
+	// Check if the client transport is configured for Unix sockets
+	transport, ok := cache.client.Transport.(*http.Transport)
+	if !ok {
+		t.Fatalf("expected client transport to be *http.Transport, got %T", cache.client.Transport)
+	}
+	if transport.DialContext == nil {
+		t.Fatal("expected client transport DialContext to be non-nil for unix socket")
+	}
+
+	// Attempt a dummy dial to see if it uses the correct path (optional, more involved check)
+	dummyConn, err := transport.DialContext(context.Background(), "", "")
+	if err == nil {
+		dummyConn.Close()
+		t.Log("DialContext seems functional, but couldn't verify path without a listener")
+	} else if !strings.Contains(err.Error(), "connect: connection refused") && !strings.Contains(err.Error(), "connect: no such file or directory") {
+		// We expect connection refused or not found if nothing is listening
+		t.Errorf("DialContext failed with unexpected error: %v", err)
+	}
+}
+
 func TestGetTarget(t *testing.T) {
 	tests := []struct {
 		name     string
@@ -66,24 +123,39 @@ func TestGetTarget(t *testing.T) {
 			expected: "http://example.com",
 		},
 		{
-			name:     "With complex path",
-			target:   "http://example.com",
-			path:     "/pag(#*((#@)ΓΓΓΓe/Γ",
-			query:    "id=123",
-			expected: "http://example.com/pag(#*((#@)ΓΓΓΓe/Γ",
+			name:   "With complex path",
+			target: "http://example.com",
+			path:   "/pag(#*((#@)ΓΓΓΓe/Γ",
+			query:  "id=123",
+			// Expect URL encoding and query parameter
+			expected: "http://example.com/pag%28%23%2A%28%28%23@%29%CE%93%CE%93%CE%93%CE%93e/%CE%93?id=123",
 		},
 		{
 			name:     "With query and path",
 			target:   "http://example.com",
 			path:     "/page",
 			query:    "id=123",
-			expected: "http://example.com/page",
+			expected: "http://example.com/page?id=123",
+		},
+		{
+			name:     "Unix socket target",
+			target:   "unix:/tmp/anubis.sock",
+			path:     "/some/path",
+			query:    "key=value&flag=true",
+			expected: "http://unix/some/path?key=value&flag=true", // Scheme becomes http, host is 'unix'
+		},
+		{
+			name:     "Unix socket target with ///",
+			target:   "unix:///var/run/anubis.sock",
+			path:     "/",
+			query:    "",
+			expected: "http://unix/",
 		},
 	}
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			cache := NewOGTagCache(tt.target, false, time.Minute)
+			cache := NewOGTagCache(tt.target, false, time.Minute, false)
 
 			u := &url.URL{
 				Path:     tt.path,
@@ -98,3 +170,86 @@ func TestGetTarget(t *testing.T) {
 		})
 	}
 }
+
+// TestIntegrationGetOGTags_UnixSocket tests fetching OG tags via a Unix socket.
+func TestIntegrationGetOGTags_UnixSocket(t *testing.T) {
+	tempDir := t.TempDir()
+
+	socketPath := filepath.Join(tempDir, "anubis-test.sock")
+
+	// Ensure the socket does not exist initially
+	_ = os.Remove(socketPath)
+
+	// Create a simple HTTP server listening on the Unix socket
+	listener, err := net.Listen("unix", socketPath)
+	if err != nil {
+		t.Fatalf("Failed to listen on unix socket %s: %v", socketPath, err)
+	}
+	defer func(listener net.Listener, socketPath string) {
+		if listener != nil {
+			if err := listener.Close(); err != nil && !errors.Is(err, net.ErrClosed) {
+				t.Logf("Error closing listener: %v", err)
+			}
+		}
+
+		if _, err := os.Stat(socketPath); err == nil {
+			if err := os.Remove(socketPath); err != nil {
+				t.Logf("Error removing socket file %s: %v", socketPath, err)
+			}
+		}
+	}(listener, socketPath)
+
+	server := &http.Server{
+		Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Set("Content-Type", "text/html")
+			fmt.Fprintln(w, `<!DOCTYPE html><html><head><meta property="og:title" content="Unix Socket Test" /></head><body>Test</body></html>`)
+		}),
+	}
+	go func() {
+		if err := server.Serve(listener); err != nil && !errors.Is(err, http.ErrServerClosed) {
+			t.Logf("Unix socket server error: %v", err)
+		}
+	}()
+	defer func(server *http.Server, ctx context.Context) {
+		err := server.Shutdown(ctx)
+		if err != nil {
+			t.Logf("Error shutting down server: %v", err)
+		}
+	}(server, context.Background()) // Ensure server is shut down
+
+	// Wait a moment for the server to start
+	time.Sleep(100 * time.Millisecond)
+
+	// Create cache instance pointing to the Unix socket
+	targetURL := "unix://" + socketPath
+	cache := NewOGTagCache(targetURL, true, 1*time.Minute, false)
+
+	// Create a dummy URL for the request (path and query matter)
+	testReqURL, _ := url.Parse("/some/page?query=1")
+
+	// Get OG tags
+	// Pass an empty string for host, as it's irrelevant for unix sockets
+	ogTags, err := cache.GetOGTags(testReqURL, "")
+
+	if err != nil {
+		t.Fatalf("GetOGTags failed for unix socket: %v", err)
+	}
+
+	expectedTags := map[string]string{
+		"og:title": "Unix Socket Test",
+	}
+
+	if !reflect.DeepEqual(ogTags, expectedTags) {
+		t.Errorf("Expected OG tags %v, got %v", expectedTags, ogTags)
+	}
+
+	// Test cache retrieval (should hit cache)
+	// Pass an empty string for host
+	cachedTags, err := cache.GetOGTags(testReqURL, "")
+	if err != nil {
+		t.Fatalf("GetOGTags (cache hit) failed for unix socket: %v", err)
+	}
+	if !reflect.DeepEqual(cachedTags, expectedTags) {
+		t.Errorf("Expected cached OG tags %v, got %v", expectedTags, cachedTags)
+	}
+}

internal/ogtags/parse_test.go

@@ -12,7 +12,7 @@ import (
 // TestExtractOGTags updated with correct expectations based on filtering logic
 func TestExtractOGTags(t *testing.T) {
 	// Use a cache instance that reflects the default approved lists
-	testCache := NewOGTagCache("", false, time.Minute)
+	testCache := NewOGTagCache("", false, time.Minute, false)
 	// Manually set approved tags/prefixes based on the user request for clarity
 	testCache.approvedTags = []string{"description"}
 	testCache.approvedPrefixes = []string{"og:"}
@@ -189,7 +189,7 @@ func TestIsOGMetaTag(t *testing.T) {
 
 func TestExtractMetaTagInfo(t *testing.T) {
 	// Use a cache instance that reflects the default approved lists
-	testCache := NewOGTagCache("", false, time.Minute)
+	testCache := NewOGTagCache("", false, time.Minute, false)
 	testCache.approvedTags = []string{"description"}
 	testCache.approvedPrefixes = []string{"og:"}
 

internal/slog.go

@@ -1,24 +0,0 @@
-package internal
-
-import (
-	"fmt"
-	"log/slog"
-	"os"
-)
-
-func InitSlog(level string) {
-	var programLevel slog.Level
-	if err := (&programLevel).UnmarshalText([]byte(level)); err != nil {
-		fmt.Fprintf(os.Stderr, "invalid log level %s: %v, using info\n", level, err)
-		programLevel = slog.LevelInfo
-	}
-
-	leveler := &slog.LevelVar{}
-	leveler.Set(programLevel)
-
-	h := slog.NewJSONHandler(os.Stderr, &slog.HandlerOptions{
-		AddSource: true,
-		Level:     leveler,
-	})
-	slog.SetDefault(slog.New(h))
-}

internal/test/playwright_test.go

@@ -52,6 +52,24 @@ var (
 			realIP:    placeholderIP,
 			userAgent: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.6099.28 Safari/537.36",
 		},
+		{
+			name:      "Amazonbot",
+			action:    actionDeny,
+			realIP:    placeholderIP,
+			userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)",
+		},
+		{
+			name:      "Amazonbot",
+			action:    actionDeny,
+			realIP:    placeholderIP,
+			userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)",
+		},
+		{
+			name:      "PerplexityAI",
+			action:    actionDeny,
+			realIP:    placeholderIP,
+			userAgent: "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)",
+		},
 		{
 			name:      "kagiBadIP",
 			action:    actionChallenge,
@@ -80,7 +98,7 @@ const (
 	actionChallenge action = "CHALLENGE"
 
 	placeholderIP     = "fd11:5ee:bad:c0de::"
-	playwrightVersion = "1.50.1"
+	playwrightVersion = "1.51.1"
 )
 
 type action string
@@ -101,6 +119,9 @@ func doesNPXExist(t *testing.T) {
 }
 
 func run(t *testing.T, command string) string {
+	if testing.Short() {
+		t.Skip("skipping integration smoke testing in short mode")
+	}
 	t.Helper()
 
 	shPath, err := exec.LookPath("sh")
@@ -244,6 +265,132 @@ func TestPlaywrightBrowser(t *testing.T) {
 	}
 }
 
+func TestPlaywrightWithBasePrefix(t *testing.T) {
+	if os.Getenv("DONT_USE_NETWORK") != "" {
+		t.Skip("test requires network egress")
+		return
+	}
+
+	t.Skip("NOTE(Xe)\\ these tests require HTTPS support in #364")
+
+	doesNPXExist(t)
+	startPlaywright(t)
+
+	pw := setupPlaywright(t)
+	basePrefix := "/myapp"
+	anubisURL := spawnAnubisWithOptions(t, basePrefix)
+
+	// Reset BasePrefix after test
+	t.Cleanup(func() {
+		anubis.BasePrefix = ""
+	})
+
+	browsers := []playwright.BrowserType{pw.Chromium}
+
+	for _, typ := range browsers {
+		t.Run(typ.Name()+"/basePrefix", func(t *testing.T) {
+			browser, err := typ.Connect(buildBrowserConnect(typ.Name()), playwright.BrowserTypeConnectOptions{
+				ExposeNetwork: playwright.String("<loopback>"),
+			})
+			if err != nil {
+				t.Fatalf("could not connect to remote browser: %v", err)
+			}
+			defer browser.Close()
+
+			ctx, err := browser.NewContext(playwright.BrowserNewContextOptions{
+				AcceptDownloads: playwright.Bool(false),
+				ExtraHttpHeaders: map[string]string{
+					"X-Real-Ip": "127.0.0.1",
+				},
+				UserAgent: playwright.String("Mozilla/5.0 (X11; Linux x86_64; rv:136.0) Gecko/20100101 Firefox/136.0"),
+			})
+			if err != nil {
+				t.Fatalf("could not create context: %v", err)
+			}
+			defer ctx.Close()
+
+			page, err := ctx.NewPage()
+			if err != nil {
+				t.Fatalf("could not create page: %v", err)
+			}
+			defer page.Close()
+
+			// Test accessing the base URL with prefix
+			_, err = page.Goto(anubisURL+basePrefix, playwright.PageGotoOptions{
+				Timeout: pwTimeout(testCases[0], time.Now().Add(5*time.Second)),
+			})
+			if err != nil {
+				pwFail(t, page, "could not navigate to test server with base prefix: %v", err)
+			}
+
+			// Check if challenge page is displayed
+			image := page.Locator("#image[src*=pensive], #image[src*=happy]")
+			err = image.WaitFor(playwright.LocatorWaitForOptions{
+				Timeout: pwTimeout(testCases[0], time.Now().Add(5*time.Second)),
+			})
+			if err != nil {
+				pwFail(t, page, "could not wait for challenge image: %v", err)
+			}
+
+			isVisible, err := image.IsVisible()
+			if err != nil {
+				pwFail(t, page, "could not check if challenge image is visible: %v", err)
+			}
+			if !isVisible {
+				pwFail(t, page, "challenge image not visible")
+			}
+
+			// Complete the challenge
+			// Wait for the challenge to be solved
+			anubisTest := page.Locator("#anubis-test")
+			err = anubisTest.WaitFor(playwright.LocatorWaitForOptions{
+				Timeout: pwTimeout(testCases[0], time.Now().Add(30*time.Second)),
+			})
+			if err != nil {
+				pwFail(t, page, "could not wait for challenge to be solved: %v", err)
+			}
+
+			// Verify the challenge was solved
+			content, err := anubisTest.TextContent(playwright.LocatorTextContentOptions{})
+			if err != nil {
+				pwFail(t, page, "could not get text content: %v", err)
+			}
+
+			var tm int64
+			if _, err := fmt.Sscanf(content, "%d", &tm); err != nil {
+				pwFail(t, page, "unexpected output: %s", content)
+			}
+
+			// Check if the timestamp is reasonable
+			now := time.Now().Unix()
+			if tm < now-60 || tm > now+60 {
+				pwFail(t, page, "unexpected timestamp in output: %d not in range %d±60", tm, now)
+			}
+
+			// Check if cookie has the correct path
+			cookies, err := ctx.Cookies()
+			if err != nil {
+				pwFail(t, page, "could not get cookies: %v", err)
+			}
+
+			var found bool
+			for _, cookie := range cookies {
+				if cookie.Name == anubis.CookieName {
+					found = true
+					if cookie.Path != basePrefix+"/" {
+						t.Errorf("cookie path is wrong, wanted %s, got: %s", basePrefix+"/", cookie.Path)
+					}
+					break
+				}
+			}
+
+			if !found {
+				t.Errorf("Cookie %q not found", anubis.CookieName)
+			}
+		})
+	}
+}
+
 func buildBrowserConnect(name string) string {
 	u, _ := url.Parse(*playwrightServer)
 
@@ -357,14 +504,14 @@ func pwFail(t *testing.T, page playwright.Page, format string, args ...any) erro
 }
 
 func pwTimeout(tc testCase, deadline time.Time) *float64 {
-	max := *playwrightMaxTime
+	maxTime := *playwrightMaxTime
 	if tc.isHard {
-		max = *playwrightMaxHardTime
+		maxTime = *playwrightMaxHardTime
 	}
 
 	d := time.Until(deadline)
-	if d <= 0 || d > max {
-		return playwright.Float(float64(max.Milliseconds()))
+	if d <= 0 || d > maxTime {
+		return playwright.Float(float64(maxTime.Milliseconds()))
 	}
 	return playwright.Float(float64(d.Milliseconds()))
 }
@@ -410,6 +557,10 @@ func setupPlaywright(t *testing.T) *playwright.Playwright {
 }
 
 func spawnAnubis(t *testing.T) string {
+	return spawnAnubisWithOptions(t, "")
+}
+
+func spawnAnubisWithOptions(t *testing.T, basePrefix string) string {
 	t.Helper()
 
 	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -436,6 +587,7 @@ func spawnAnubis(t *testing.T) string {
 		Policy:         policy,
 		ServeRobotsTXT: true,
 		Target:         "http://" + host + ":" + port,
+		BasePrefix:     basePrefix,
 	})
 	if err != nil {
 		t.Fatalf("can't construct libanubis.Server: %v", err)

internal/xff_test.go

@@ -0,0 +1,179 @@
+package internal
+
+import (
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+)
+
+func TestXForwardedForUpdateIgnoreUnix(t *testing.T) {
+	var remoteAddr = ""
+	var xff = ""
+
+	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		remoteAddr = r.RemoteAddr
+		xff = r.Header.Get("X-Forwarded-For")
+		w.WriteHeader(http.StatusOK)
+	})
+
+	r := httptest.NewRequest(http.MethodGet, "/", nil)
+
+	r.RemoteAddr = "@"
+
+	w := httptest.NewRecorder()
+
+	XForwardedForUpdate(h).ServeHTTP(w, r)
+
+	if r.RemoteAddr != remoteAddr {
+		t.Errorf("wanted remoteAddr to be %s, got: %s", r.RemoteAddr, remoteAddr)
+	}
+
+	if xff != "" {
+		t.Error("handler added X-Forwarded-For when it should not have")
+	}
+}
+
+func TestXForwardedForUpdateAddToChain(t *testing.T) {
+	var xff = ""
+	const expected = "1.1.1.1"
+
+	h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		xff = r.Header.Get("X-Forwarded-For")
+		w.WriteHeader(http.StatusOK)
+	})
+
+	srv := httptest.NewServer(XForwardedForUpdate(h))
+
+	r, err := http.NewRequest(http.MethodGet, srv.URL, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	r.Header.Set("X-Forwarded-For", "1.1.1.1,10.20.30.40")
+
+	if _, err := srv.Client().Do(r); err != nil {
+		t.Fatal(err)
+	}
+
+	if xff != expected {
+		t.Logf("expected: %s", expected)
+		t.Logf("got:      %s", xff)
+		t.Error("X-Forwarded-For header was not what was expected")
+	}
+}
+
+func TestComputeXFFHeader(t *testing.T) {
+	for _, tt := range []struct {
+		name          string
+		remoteAddr    string
+		origXFFHeader string
+		pref          XFFComputePreferences
+		result        string
+		err           error
+	}{
+		{
+			name:          "StripPrivate",
+			remoteAddr:    "127.0.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1",
+			pref: XFFComputePreferences{
+				StripPrivate: true,
+			},
+			result: "1.1.1.1,127.0.0.1",
+		},
+		{
+			name:          "StripLoopback",
+			remoteAddr:    "127.0.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1,127.0.0.1",
+			pref: XFFComputePreferences{
+				StripLoopback: true,
+			},
+			result: "1.1.1.1,10.0.0.1",
+		},
+		{
+			name:          "StripCGNAT",
+			remoteAddr:    "100.64.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1,100.64.0.1",
+			pref: XFFComputePreferences{
+				StripCGNAT: true,
+			},
+			result: "1.1.1.1,10.0.0.1",
+		},
+		{
+			name:          "StripLinkLocalUnicastIPv4",
+			remoteAddr:    "169.254.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1,169.254.0.1",
+			pref: XFFComputePreferences{
+				StripLLU: true,
+			},
+			result: "1.1.1.1,10.0.0.1",
+		},
+		{
+			name:          "StripLinkLocalUnicastIPv6",
+			remoteAddr:    "169.254.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1,fe80::",
+			pref: XFFComputePreferences{
+				StripLLU: true,
+			},
+			result: "1.1.1.1,10.0.0.1",
+		},
+		{
+			name:          "Flatten",
+			remoteAddr:    "127.0.0.1:80",
+			origXFFHeader: "1.1.1.1,10.0.0.1,fe80::,100.64.0.1,169.254.0.1",
+			pref: XFFComputePreferences{
+				StripPrivate:  true,
+				StripLoopback: true,
+				StripCGNAT:    true,
+				StripLLU:      true,
+				Flatten:       true,
+			},
+			result: "1.1.1.1",
+		},
+		{
+			name:          "TrimSpaces",
+			remoteAddr:    "127.0.0.1:80",
+			origXFFHeader: "1.1.1.1, 10.0.0.1, fe80::, 100.64.0.1, 169.254.0.1",
+			pref: XFFComputePreferences{
+				StripPrivate:  true,
+				StripLoopback: true,
+				StripCGNAT:    true,
+				StripLLU:      true,
+				Flatten:       true,
+			},
+			result: "1.1.1.1",
+		},
+		{
+			name:       "invalid-ip-port",
+			remoteAddr: "fe80::",
+			err:        ErrCantSplitHostParse,
+		},
+		{
+			name:       "invalid-remote-ip",
+			remoteAddr: "anubis:80",
+			err:        ErrCantParseRemoteIP,
+		},
+		{
+			name:       "no-xff-dont-panic",
+			remoteAddr: "127.0.0.1:80",
+			pref: XFFComputePreferences{
+				StripPrivate:  true,
+				StripLoopback: true,
+				StripCGNAT:    true,
+				StripLLU:      true,
+				Flatten:       true,
+			},
+		},
+	} {
+		t.Run(tt.name, func(t *testing.T) {
+			result, err := computeXFFHeader(tt.remoteAddr, tt.origXFFHeader, tt.pref)
+			if err != nil && !errors.Is(err, tt.err) {
+				t.Errorf("computeXFFHeader got the wrong error, wanted %v but got: %v", tt.err, err)
+			}
+
+			if result != tt.result {
+				t.Errorf("computeXFFHeader returned the wrong result, wanted %q but got: %q", tt.result, result)
+			}
+		})
+	}
+}